Validation of Using Low Number of Jackknife Replicates

Purpose - We recently developed a Vitalnet module for analyzing BRFSS data. The new program makes BRFSS data analysis much better, easier, and more reliable.

Due to the complex survey design, confidence intervals (CI) for BRFSS data are non-trivial. Several CI methods are available for BRFSS data. We chose "Jackknife Replication" (JR) mainly because 1) JR can calculate a CI for any outcome, including medians, and 2) JR seemed easier to understand and explain. Probably because of inability to run the computation-intensive JR method fast enough, most BRFSS analyses using SAS and SUDAAN typically use Taylor Series linearization (TS) instead.

We have figured out how to make the JR method run fast, in most cases. Nevertheless, to speed things further, we would like to allow the user to use a lower number of replicates. The purpose of this study was to estimate what affect this has on the accuracy of the method.

Methods - A series of analyses were done for 2005, Texas, _RFBING3 (binge drinker), % Yes, by sex, 95% CI, jackknife replicate method, as shown in this sample table. Each analysis used a different number of maximum replicates. If the number of maximum replicates is greater than the number of valid surveys in a cell, the number of replicates is changed to the lower number. Each "diff" in the table is the difference between the confidence limit with reduced max_reps and the confidence limit with unlimited max_reps. For example, 21.01 - 20.95 = 0.06 (a very small difference). If the differences are small enough, then it is OK to use a lower number of replicates.

max_reps	Male				Female
max_reps	CI lo	diff	CI hi	diff	CI lo	diff	CI hi	diff
50	21.01	0.06	25.36	0.06	4.85	0.12	7.03	0.12
100	21.08	0.13	25.29	0.13	4.87	0.10	7.01	0.10
200	20.99	0.04	25.38	0.04	4.96	0.01	6.92	0.01
400	20.96	0.01	25.41	0.01	4.97	0.00	6.90	0.00
800	20.92	0.03	25.45	0.03	4.98	0.01	6.92	0.01
1600	20.93	0.02	25.44	0.02	4.96	0.01	6.90	0.01
3200	20.95	0.00	25.42	0.00	4.98	0.01	6.91	0.01
6400	20.95	0.00	25.42	0.00	4.97	0.00	6.91	0.00
Unlimited	20.95	0.00	25.42	0.00	4.97	0.00	6.91	0.00

Results - The Jackknife CI for small numbers of replicates is very similar to those for unlimited replicates. The largest differences (shown in red) were only about 0.1 and occurred with low replicates. The differences are less than or about the same as differences between different CI calculation methods, as implemented in various BRFSS analysis systems, which are about 0.1 for large numbers and are about 0.5 for small numbers.

Conclusions - For exploratory data analysis, to speed up Jackknife CI calculation, the user may select a low number of replicates, with very little loss in accuracy. For published results, it is recommended to select more replicates, as this gives a more accurate result.