Table of Contents:
Introduction ·
Methods ·
Results ·
Discussion
Summary Table of Results ·
Detailed Table of Results
Introduction:
Data collection, analysis, and dissemination are core public health functions.
Historically, data were analyzed with generic statistical software.
However, that process is unreliable and tedious, and only available to a few expert users.
Published reports are valuable, but give the user no choice for customizing the analyses.
Web-based query systems can make much better use of public health data sets.
Therefore, many agencies have tried to develop web-based query systems.
Vitalnet is a commercial data analysis solution, with both
desktop and web-based front ends.
The purpose of this evaluation is to objectively compare the capabilities
of Vitalnet with the two most prominent and well-funded state-developed
web-based query systems.
Methods:
Systems included -
For completeness, we included both Vitalnet desktop and web-based platforms in the comparison.
It should be noted that VitalPro for Windows is under development,
and VitalPro for Unix is mostly used for testing.
For comparison, we included the two most well-funded and long-established
State-developed web systems.
We selected mortality data as a convenient data set for comparing systems.
Systems excluded -
We did not include generic statistical software (such as SAS or SPSS),
because they require an expert to run and need
extensive programming to be able to produce population-based rates.
The only other commercial software for this purpose we are aware of
merely displays pre-tabulated spreadsheet-type displays,
so we excluded from the analysis as too limited.
Positive Capabilities -
To compare the systems, we identified 65 "positive capabilities" that can be
clearly marked "YES" or "NO".
We used criteria in the 2005 report
"Current Status and Desirable Functionalities of State Web-based Data Query Systems",
by Friedman and Parrish, as a starting point.
We only included capabilities that:
1) are not trivial,
2) can be objectively scored "YES/NO",
and
3) are in at least one, but not all, of the systems compared.
For example, 1) only the most important chart customizations were included,
as others might be considered trivial, 2) subjective criteria such as
"ease of use" cannot be objectively scored "YES/NO" and were excluded,
and 3) "counts" was excluded because all systems scored "YES".
Negative characteristics -
We also identified 3 significant "negative characteristics",
obvious faults that should not be present.
We only included negatives that:
1) are significant (not trivial),
2) can be objectively scored "YES/NO",
and
3) are in at least one of the systems compared.
Scoring the criteria -
We made a concerted effort to find every positive capability in the systems.
For example, we wanted to include "SVG maps" as a capability promised
by one of the State systems, but were unable to produce that capability.
For each system,
we added a point for each "positive capability" provided,
and subtracted a point for each "negative characteristic" present.
Results:
Please refer to
Summary Table of Results
and
Detailed Table of Results below.
97% -
VitalPro for Windows
(VFW)
(desktop)
under development, scored highest.
92% -
VitalWeb Ajax
(AJX)
(browser)
scored second highest.
The ICD system is under development.
89% -
VitalPro for DOS
(VPD)
(desktop)
was close behind.
It does not make maps.
89% -
VitalWeb Wizard
(WZD)
(browser)
makes maps and charts,
and runs step-by-step.
85% -
VitalWeb Standard
(STD)
(browser)
does not make multi-tables.
62% -
VitalPro for UNIX
(VPU)
(desktop)
is mostly used for testing.
31% -
State system #1
(browser)
scored much lower.
It lacks many needed capabilities.
9% -
State system #2
(browser)
scored very low.
It only includes a few needed capabilities.
Discussion:
In an objective comparison of capabilities,
the Vitalnet systems score much higher than
the two most prominent and well-funded state systems.
Vitalnet offers the opportunity for any agency to avoid or get
out of get out of the expensive,
frustrating, and failure-prone software development business.
Vitalnet allows the agency to get best value, provide better services,
make operations smoother, and lower costs.
There are possible limitations or room for expansion of the comparisons, including:
1) Ease of use is not measured:
Ease of use is a top priority with Vitalnet, including extensive user testing.
However, controlled usability comparisons between the systems have not been carried out.
This kind of user analysis is subjective and fraught with great difficulty.
Readers are encouraged to try Vitalnet
and come to their own conclusions.
2) Output quality is not measured:
The Vitalnet output is clean, documented, publication-ready, and correct.
The output from both the other systems have obvious deficiencies.
However, output quality is difficult to quantify.
An objective comparison would require great expertise and care.
Readers are encouraged to look at
Vitalnet output
and come to their own conclusions.
3) Criteria are not prioritized:
Priorities could be assigned to the different capabilities.
For example, "makes YPLL" might be worth 2 points, and "does mean age of death" 1 point.
Or perhaps vice versa.
In any case, the general trend would remain that the Vitalnet systems
far outperform the others.
Summary Table of Results:
| |
System |
Overall Score |
System Environment |
| A |
VitalPro for DOS |
 |
89 |
For Windows PC
|
| B |
VitalPro for Unix |
 |
62 |
For Linux PC
|
| C |
VitalPro for Windows |
 |
97 |
For Windows PC
|
| D |
VitalWeb Standard |
 |
85 |
For web browser
|
| E |
VitalWeb Wizard |
 |
89 |
For web browser
|
| F |
VitalWeb Ajax |
 |
92 |
For web browser
|
| G |
Prominent, well-funded state system #1 |
 |
9 |
For web browser |
| H |
Prominent, well-funded state system #2 |
 |
31 |
For web browser |
Detailed Table of Results:
| Positive Capabilities |
VitalPro |
VitalWeb |
State |
| A |
B |
C |
D |
E |
F |
G |
H |
| Age groups - can use (0-4, ..., 80+) (5-year)? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| Age groups - can use (0-9, ..., 80+) (10-year)? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| Age groups - can use (0-4, 5-9) (5-year subset)? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| Age groups - can use (0-8, 9+) (arbitrary groups)? |
+ |
+ |
+ |
· |
· |
+ |
· |
+ |
| Age groups - can use 3-7 (arbitrary single group)? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| Bar charts - can make horizontal bar charts? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| Bar charts - can make vertical bar charts? |
+ |
· |
+ |
+ |
+ |
+ |
· |
+ |
| Bar charts - can make stacked bar charts? |
+ |
· |
+ |
+ |
+ |
+ |
· |
+ |
| Bar charts - change height? 2 · 4 width? 0.3 · 0.6 |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Bar charts - can change bar color? A · B · C |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Bar charts - can include confidence intervals? |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Bar charts - can include grid lines on chart? |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Confidence intervals - change level? 80 · 95 · 99 |
+ |
+ |
+ |
+ |
+ |
+ |
+ |
· |
| Confidence intervals - documents methods? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Data export - makes CSV or Excel data table? |
+ |
+ |
+ |
+ |
+ |
+ |
+ |
· |
| Data export - makes DBF data table? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Data export - makes text data table? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Geographic - can compare region and county? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Geographic - can compare any A+B and C? |
+ |
+ |
+ |
· |
· |
+ |
· |
· |
| Geographic - includes regional groupings? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| ICD codes - can compare any A+B and C? |
+ |
+ |
+ |
· |
· |
· |
· |
· |
| ICD codes - directly does sorted leading causes? |
+ |
+ |
+ |
+ |
+ |
· |
· |
+ |
| ICD codes - includes complete ICD system? |
+ |
+ |
+ |
· |
· |
· |
· |
+ |
| Interface - includes context-sensitive help manual? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Interface - retains settings if switches statistic? |
+ |
+ |
+ |
+ |
+ |
+ |
+ |
· |
| Interface - retains settings if switches to maps? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Interface - shows output in own window? |
+ |
+ |
+ |
+ |
+ |
+ |
+ |
· |
| Line charts - can make line charts? |
+ |
· |
+ |
+ |
+ |
+ |
· |
+ |
| Line charts - can make stacked line charts? |
· |
· |
· |
· |
· |
· |
· |
+ |
| Line charts - can change chart height? |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Line charts - can change line color? |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Line charts - can include grid lines? |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Line charts - can include sizeable point symbols? |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Maps - can toggle area labels? |
· |
· |
+ |
+ |
+ |
+ |
· |
· |
| Maps - makes data layer for GIS software? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Maps - has all Brewer color palettes? |
· |
· |
+ |
+ |
+ |
+ |
· |
· |
| Maps - makes PDF format? |
· |
· |
+ |
+ |
+ |
+ |
· |
· |
| Multi-Tables - makes table for each row variable? |
+ |
+ |
+ |
· |
+ |
+ |
· |
· |
| Multi-Tables - makes table for each statistic? |
+ |
+ |
+ |
· |
+ |
+ |
· |
· |
| Multi-Tables - makes table for each area? |
+ |
+ |
+ |
· |
+ |
+ |
· |
· |
| Pie charts - can make pie charts? |
+ |
· |
+ |
+ |
+ |
+ |
· |
+ |
| Pie charts - can change slice colors? A · B · C |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Pie charts - can change slice rotation? 12 · 3 · 6 |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Pie charts - can change label mode? LL · LO · LG |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Pie charts - can change pie radius? 2.0 · 2.5 · 3.0 |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Pie charts - can toggle slice percents? UL · AL · OP |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Race groups - can compare any A+B and C? |
+ |
+ |
+ |
· |
· |
+ |
· |
+ |
| Statistics - can set age adjustment year? 70 · 00 |
+ |
+ |
+ |
+ |
+ |
+ |
+ |
· |
| Statistics - does coefficient of variation? |
· |
· |
· |
· |
· |
· |
· |
+ |
| Statistics - does mean age of death? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| Statistics - does standardized mortality ratio? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Statistics - does years of potential life lost? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Suppression - can suppress low counts on table? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| Suppression - can suppress low counts on map? |
· |
· |
+ |
+ |
+ |
+ |
· |
+ |
| Suppression - can change level? 3 · 5 |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Tables - can display column percents? |
+ |
+ |
+ |
+ |
+ |
+ |
+ |
· |
| Tables - can display row percents? |
+ |
+ |
+ |
+ |
+ |
+ |
+ |
· |
| Tables - can change precision? 1 · 3 |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Tables - can sort for any row variable? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Tables - correctly shows unknowns? 1 · 2 |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| Tables - output has URL for future reference? |
+ |
· |
+ |
+ |
+ |
+ |
· |
· |
| Years - can use (00-01, 02-03, ...) (2-year)? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| Years - can use (00-02, 03-05, ...) (3-year)? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
+ |
| Years - does time trend significance analysis? |
+ |
+ |
+ |
+ |
+ |
+ |
· |
· |
| Years - can make time trend maps? |
· |
· |
+ |
+ |
+ |
+ |
· |
· |
| Negative Characteristics |
A |
B |
C |
D |
E |
F |
G |
H |
| Interface - has obvious bugs (eg, bad links)? |
· |
· |
· |
· |
· |
· |
— |
· |
| Charts - often look very bad (eg, 30 pie slices)? |
· |
· |
· |
· |
· |
· |
· |
— |
| Charts - often are misleading (eg, rate pie)? |
· |
· |
· |
· |
· |
· |
· |
— |
| Count: Out of 65 possible (3/15/2011) |
58 |
40 |
63 |
55 |
58 |
60 |
6 |
20 |
| Percent: Out of 100 possible |
89 |
62 |
97 |
85 |
89 |
92 |
9 |
31 |
|