Comparison - Health Data Query Systems

Introduction:

Importance of data analysis - Data collection, analysis, and dissemination are core public health functions. Historically, data were analyzed with generic statistical software. However, that process is unreliable and tedious, and only available to a few expert users. Static published reports are helpful, but give the user no choice for customizing the analyses, or an easy way to incorporate the results into reports. Static online electronic reports are more helpful, but again give the user no choice for customization. Web-based query systems make best use of public health data sets. Therefore, many agencies have tried to develop web-based query systems.

Purpose of comparison - Vitalnet is a professional data analysis solution, with both desktop and web-based front ends. The purpose of this evaluation is to objectively compare the capabilities of different Vitalnet platforms with each other, and possibly with one or more prominent state-developed web-based query systems.


Methods:

Systems included - We included several fully functioning Vitalnet desktop and web-based platforms in the comparison. For comparison, we might include one or more long-established state-developed web systems. We selected mortality data as a convenient data set for comparing systems.

Positive Capabilities - To compare the systems, we identified 59 "positive capabilities" that can be clearly marked "YES" or "NO". We used criteria in the 2005 report "Current Status and Desirable Functionalities of State Web-based Data Query Systems", by Friedman and Parrish, as a starting point. We tried to only include positives that: 1) are not trivial (eg, only include the most important chart customizations), and 2) can be objectively scored "YES/NO" (eg, important yet subjective criteria such as "ease of use" were excluded).

Negative characteristics - We also identified 3 significant "negative characteristics", obvious faults that should not be present. We tried to only include negatives that: 1) are not trivial, and 2) can be objectively scored "YES/NO".

Scoring the criteria - We made a concerted effort to find every positive capability in the systems. For example, we wanted to include "SVG maps" as a capability promised by one of the State systems, but were unable to produce that capability. For each system, we added a point for each "positive capability" provided, no action (zero change) if the "positive capability" was missing, and subtracted a point for each "negative characteristic" present.


Summary Table of Results:

	System	Overall Score		System Environment
VPC	VitalPro for Win32		97%	For Windows PC
STD	VitalWeb Standard		83%	For web browser
WZD	VitalWeb Wizard		88%	For web browser
AJX	VitalWeb Ajax		97%	For web browser

Detailed Table of Results:

Positive Capabilities	VPC	STD	WZD	AJX
Age groups - Can use 5-year, 10-year, 20-year?	✓	✓	✓	✓
Age groups - Can exclude groups from analysis?	✓	✓	✓	✓
Age groups - Can use any needed single group?	✓	✓	✓	✓
Age groups - Can use any needed set of groups?	✓	·	·	✓
Bar charts - Can make horizontal bar charts?	✓	✓	✓	✓
Bar charts - Can make vertical bar charts?	✓	✓	✓	✓
Bar charts - Can make stacked bar charts?	✓	✓	✓	✓
Bar charts - Change height / width? 2 · 4 / 0.3 · 0.6	✓	✓	✓	✓
Bar charts - Can change bar color? A · B · C	✓	✓	✓	✓
Bar charts - Can include confidence intervals?	✓	✓	✓	✓
Bar charts - Can include / exclude grid lines?	✓	✓	✓	✓
Confidence intervals - Makes 80%, 95%, 99%?	✓	✓	✓	✓
Data export - Makes Excel file for data	✓	✓	✓	✓
Data export - Makes DBF file for data	✓	✓	✓	✓
Data export - Makes text data table	✓	✓	✓	✓
Geographic - Includes regional groupings?	✓	✓	✓	✓
Geographic - Can compare "Area Sets"	✓	·	·	✓
ICD codes - Directly does leading cause analysis?	✓	✓	✓	✓
ICD codes - Can compare "ICD Sets"	✓	·	·	✓
ICD codes - Includes complete ICD system	✓	·	·	✓
Interface - Includes context-sensitive help?	✓	✓	✓	✓
Interface - Always "remembers" settings?	✓	✓	✓	✓
Interface - Shows output in separate window?	✓	✓	✓	✓
Line charts - can make line charts	✓	✓	✓	✓
Line charts - can make stacked line charts?	·	·	·	·
Line charts - can change chart height	✓	✓	✓	✓
Line charts - can change line color	✓	✓	✓	✓
Line charts - can include / exclude grid lines	✓	✓	✓	✓
Line charts - can include sizeable point symbols	✓	✓	✓	✓
Maps - Can print (or omit) area labels?	✓	✓	✓	✓
Maps - Does Jenks Natural Breaks	✓	✓	✓	✓
Maps - Makes DBF file for GIS layer	✓	✓	✓	✓
Maps - Has all Brewer color palettes	✓	✓	✓	✓
Maps - Makes PDF format maps	✓	✓	✓	✓
Multi-Tables - Makes table for each row variable	✓	·	✓	✓
Multi-Tables - Makes table for each statistic	✓	·	✓	✓
Multi-Tables - Makes table for each area	✓	·	✓	✓
Pie charts - can make pie charts	✓	✓	✓	✓
Pie charts - can change slice colors? A · B · C	✓	✓	✓	✓
Pie charts - can change slice rotation? 12 · 3 · 6	✓	✓	✓	✓
Pie charts - can change label mode? LL · LO · LG	✓	✓	✓	✓
Pie charts - can change pie radius? 2.0 · 2.5 · 3.0	✓	✓	✓	✓
Pie charts - can toggle slice percents? UL · AL · OP	✓	✓	✓	✓
Race groups - Can compare "Race Sets"	✓	·	·	✓
Statistics - does coefficient of variation?	·	·	·	·
Statistics - Does mean age of death	✓	✓	✓	✓
Statistics - Does standardized mortality ratio	✓	✓	✓	✓
Statistics - Does years of potential life lost	✓	✓	✓	✓
Suppression - Can suppress low counts on table	✓	✓	✓	✓
Suppression - Can suppress low counts on map	✓	✓	✓	✓
Suppression - Can change suppress level?  3  ·  5	✓	✓	✓	✓
Tables - Can display row / column percents	✓	✓	✓	✓
Tables - Can change precision?  1  ·  3	✓	✓	✓	✓
Tables - Can sort for any row variable	✓	✓	✓	✓
Tables - Correctly shows unknowns?  1  ·  2	✓	✓	✓	✓
Tables - Output has URL for future reference	✓	✓	✓	✓
Years - Can use 2-year and 3-year ranges	✓	✓	✓	✓
Years - Analyzes for significant time trend	✓	✓	✓	✓
Years - Can make time trend maps	✓	✓	✓	✓
Negative Characteristics	VPC	STD	WZD	AJX
Interface - has obvious bugs (eg, bad links)?	·	·	·	·
Charts - often look very bad (eg, 30 pie slices)?	·	·	·	·
Charts - often are misleading (eg, rate pie)?	·	·	·	·
Score: Out of 59 possible (1/4/2018)	57	49	52	57
Percent: Out of 100 percent possible	97	83	88	97

Discussion:

In an objective comparison of capabilities, the Vitalnet systems score very high. Vitalnet offers the opportunity for any agency to avoid or get out of the expensive, frustrating, and failure-prone software development business. Vitalnet allows the agency to get best value, provide better services, make operations smoother, and lower costs.

There are possible limitations or room for expansion of the comparisons, including:

1) Ease of use is not measured: Ease of use is a top priority with Vitalnet, including extensive user testing. However, controlled usability comparisons between the systems have not been carried out. This kind of user analysis is subjective and fraught with great difficulty. Readers are encouraged to try Vitalnet and come to their own conclusions.

2) Output quality is not measured: Vitalnet output is clean, documented, publication-ready, and correct. However, output quality is difficult to quantify. An objective comparison would require great expertise and care. Readers are encouraged to look at Vitalnet output and come to their own conclusions.

3) Criteria are not prioritized: Priorities could possibly be assigned to the different capabilities. For example, "makes YPLL" might be worth 2 points, and "does mean age of death" 1 point. Or perhaps vice versa.

4) Always possible we missed something: We took care to use basic capabilities, and verify that capabilities were present (or absent). But some may choose to include other criteria, or exclude various capabilities. Please let us know if any suggestions or comments.

Regardless of the possible improvements to this kind of analysis, the basic point is clear, that Vitalnet provides the needed capabilities for analyzing / disseminating large, complex data sets.