Validate Age-Adjusted BRFSS Percents


The age-adjusted BRFSS statistics within Vitalnet appear to be correct. But to be on the safe side, we need to compare with independently produced statistics. This proved to be a little difficult, due to the rarity of published BRFSS age-adjusted statistics.





Alaska - The Alaska BRFSS program is one of the few States that publishes age-adjusted statistics. Alaska runs two BRFSS surveys. The second survey is called "Alaska Modified BRFSS" or "Alaska Supplemental BRFSS". From the 2008 Alaska BRFSS documentation:

"For some indicators BRFSS data were combined with data from a second survey, the Alaska Modified BRFSS. This survey was state developed and is funded by the Alaska Tobacco Prevention and Control Program and focuses largely on tobacco use and attitudes. It has been collected in Alaska since 2004. The Modified BRFSS uses the same sample design, data collection methodology, and has the same sample size goal of 2,500 surveys collected per year as the BRFSS. Combining the Modified BRFSS survey with the BRFSS where possible allows for a larger sample size for analysis and more stable estimates. Also note that when the combined BRFSS and Modified BRFSS data were used, estimates will not match those provided by the CDC for those indicators."

So we chose those questions that were NOT asked on the Modified BRFSS. Of 32 risk factors in the 2008 Annual Report, 19 were asked on the Modified BRFSS. That still left us with 12 potential risk factors for the comparison:

prostate cancer (40+) - 2008
HIV/AIDS test (18-64) - 2008
seatbelt use - 2008
breast cancer screening (40+) - 2008
cervical cancer screening - 2008
colorectal cancer screening (50+) - 2008
prostate cancer screening (40+) - 2008
oral cancer exams - 2008
annual dental visit - 2008
hepatitis B vaccination - 2008
influenza vaccination (65+) - 2008 and 2009
pneumonia vaccination (65+) - 2008 and 2009

CHECKUP_____1X_1YR_ cannot be compared for 2009, because AK does not include NEVER in denominator.

Another complication was that the Alaska age-adjusted rates used the following age distribution: 18-24, 25-34, 35-44, 45-64, 65+. This is distribution #9 from Klein and Schoenborn (2001). So for purposes of the comparison, we built distribution #9 into Vitalnet software.

There was a further complication that some of the 12 risk factors not asked on the Modified BRFSS were limited to a certain age group. For example, mammmography screening question is only asked on women age 40 and older. The age-adjustment in Vitalnet is currently set up to use all age groups. In the future, we modify the age-adjustment to allow just a few age groups. But for purposes of this initial testing, we cannot use mammography (and the others listed below) for the comparison:

prostate cancer (40+) - 2008
HIV/AIDS test (18-64) - 2008 and 2009
breast cancer screening (40+) - 2008
colorectal cancer screening (50+) - 2008
prostate cancer screening (40+) - 2008
influenza vaccination (65+) - 2008 and 2009
pneumonia vaccination (65+) - 2008 and 2009

Of the 12 risk factors not on the Modified BRFSS, 5 were asked of the total population, so potentially could be compared:

seatbelt use - 2008
cervical cancer screening - 2008
oral cancer exams - 2008
annual dental visit - 2008
fruit and vegetable consumption - 2009
heavy drinking - 2009
VIGCAT_________NOW_ - 2009
_RFCHOL_______EVER_ - 2009
BPHIGH_NP_____EVER_ - 2009
binge drinking - 2009
hepatitis B vaccination - 2008

Seatbelt use: Not currently imported into Vitalnet. So could not be tested.

Cervical cancer screening: Had a Pap test in past 3 years? The weighted %yes differed between Vitalnet and the Alaska report. So could not do the comparison. Alaska reported 1089 yes. Vitalnet reported 1079 yes. No way to resolve this difference!

Oral cancer exam: Not currently imported into Vitalnet. So could not be tested.

Annual dental visit: Visited a dentist within past year? Vitalnet reports 65.3 weighted percent yes, based on 1682 yes responses. Alaska reports 66.3 weighted # yes. It was unclear the number of yes responses.

Vaccination for Hepatitis B: Not currently imported into Vitalnet. So could not be tested.

When the 2009 report became available, there were a lot more more variables that were not part of the Modified BRFSS that could be used for comparison.

Another complication was that Alaska did not report age-adjusted results before 2008. Also, before 2008, BRFSS percent results were only reported as whole numbers. This is fine for reporting, but not enough precision for validation testing. Therefore, validation was limited to 2008 and 2009.

As of 2012-02-16, there were nine totally independent validated comparisons between AK and Vitalnet, and zero discrepancies.
To increase the number of independent comparisons, analyses were also broken down by sex. This resulted in a total of 25 comparisons. Each comparison matched.





Utah - Utah IBIS does age-adjusted BRFSS statistics.

An advantage of comparing with the Utah data is that Utah has two decimal places. The extra precision makes it simpler to verify that the results are exactly correct. For example, the two results 2.11 and 2.12 would appear the same at the one decimal point level, but are clearly different with the extra precision. As a first step, I compared HLTHPLAN_______NOW_ weighted % yes between Vitalnet and Utah IBIS. Has health care coverage or insurance? The results matched exactly:

  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", PCT_Y, 2005, ND, "84.85", "84.85", 4497, 999999, 999999, 999999, VN_US___, UT_QUERY, },
  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", PCT_Y, 2006, ND, "84.97", "84.97", 4566, 999999, 999999, 999999, VN_US___, UT_QUERY, },
  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", PCT_Y, 2007, ND, "84.71", "84.71", 4508, 999999, 999999, 999999, VN_US___, UT_QUERY, },
  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", PCT_Y, 2008, ND, "85.35", "85.35", 4742, 999999, 999999, 999999, VN_US___, UT_QUERY, },
  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", PCT_Y, 2009, ND, "84.95", "84.95", 9055, 999999, 999999, 999999, VN_US___, UT_QUERY, },
  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", PCT_Y, 2010, ND, "84.07", "84.07", 9011, 999999, 999999, 999999, VN_US___, UT_QUERY, },

Next, for Utah, the interface said that there were 8 age groups for the age adjustment. I looked through the web site, and could not find any documentation for exactly which age groups were used, Sent email to Jennifer Wrathall asking for clarification about AA std age groups. Utah staff responded and clarified the age groups, and updated their documentation,

Utah AA age groups

Unfortunately, the Utah distribution is not one of the standard NCHS distributions. For most BRFSS age-adjusted results, UT uses the following age groups:

Age Group U.S. 2000 Population Projection (in thousands) Weight
===============================================
18 - 24 Years 26,258 0.1288
25 - 34 Years 37,233 0.1827
35 - 44 Years 44,659 0.2191
45 - 54 Years 37,030 0.1817
55 - 64 Years 23,961 0.1175
65 - 74 Years 18,136 0.0889
75 - 84 Years 12,315 0.0604
85 Years and Over 4,259 0.0209
===============================================
Total, 18 years and older 203,851 1

The Utah age groupings and age-adjustment standard populations were programmed into Vitalnet. The results slightly differed. In each case, the Utah IBIS results were slightly lower: The results:

  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", AA_PY, 2005, ND, "86.45", "86.42", 4473, 4473, 5097, 5097, VN_US___, UT_QUERY, },
  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", AA_PY, 2006, ND, "86.49", "86.46", 4537, 4537, 5161, 5161, VN_US___, UT_QUERY, },
  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", AA_PY, 2007, ND, "86.08", "86.05", 4487, 4487, 5042, 5042, VN_US___, UT_QUERY, },
  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", AA_PY, 2008, ND, "86.79", "86.76", 4712, 4712, 5283, 5283, VN_US___, UT_QUERY, },
  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", AA_PY, 2009, ND, "86.34", "86.31", 8986, 8986, 10056, 10056, VN_US___, UT_QUERY, },
  { FUT_ALL_FutVarNdx______FUT_BF_HLTHPLAN_______NOW_, "UT", AA_PY, 2010, ND, "85.44", "85.41", 8925, 8925, 10041, 10041, VN_US___, UT_QUERY, },

Note that the discrepancy would not have shown up even if the Utah results had only one decimal point precision, at least for these six comparisons. In other words, if the Utah results had one decimal point precision, it is possible they might have appeared to match with the Vitalnet results.

Another 6 comparisons were done for 2005-2010 PERSDOC________NOW_ BRFSS question. Another 6 comparisons were done for 2005-2010 CHECKUP_____1X_1YR_ BRFSS question. Again, the Utah results were slightly less than the Vitalnet results. One of the results for these additional 12 would have shown up as different, if the Utah results had been limited to one decimal place. There is no way to generalize about the number of decimal places needed to do the validation, since it depends on how similar the results are, except to say that more decimal places are useful than fewer for this validation. Of course, in reporting results in practice, excessive precision with extra decimal places is not a good practice, and gives an unrealistic picture of the accuracy. But for doing validation at the algorithmic level, high precision is a desirable thing.

To determine which age-adjusted result might be correct in this case, I carried out a calculation by hand for one example case: HLTHPLAN_______NOW_ for 2005-2010. The additional years of data were used to generate enough counts so that no cells were suppressed in UT age results. The goal was to see which result, 86.26 or 86.23 (IBIS) was correct. When using the actual populations in the standard population, 86.45 (Vitalnet result) was correct. It was suspected that perhaps the UT software was using the population proportions instead, and that may have accounted for the slight difference. However, in the spreadsheet, the same 86.26 result was obtained when doing the age adjustment with the population proportions.

          Weighted Standard Population Utah
 Age Group % Yes Population Proportion Weighted %
 =====================================================================
 18 to 24 72.8438 26,258 1,912,732.5004 0.1288 9.3823 72.84 9.381792
 25 to 34 79.9478 37,233 2,976,696.4374 0.1827 14.6065 79.95 14.606865
 35 to 44 85.5785 44,659 3,821,850.2315 0.2191 18.7502 85.58 18.750578
 45 to 54 88.4653 37,030 3,275,870.0590 0.1817 16.0741 88.47 16.074999
 55 to 64 91.3527 23,961 2,188,902.0447 0.1175 10.7339 91.35 10.733625
 65 to 74 98.4396 18,136 1,785,300.5856 0.0889 8.7513 98.44 8.751316
 75 to 84 98.4049 12,315 1,211,856.3435 0.0604 5.9437 98.40 5.94336
 85 to 99+ 96.3758 4,259 410,464.5322 0.0209 2.0143 96.38 2.014342
 =====================================================================
 All 203,851 17,583,672.7343 1.0000 86.2563 86.2569

86.2575 <- Vitalnet Result of using standard population
86.2563 <- Vitalnet Result of using population proportions
86.2569 <- IBIS Result of using population proportions

There is no use further speculating as the source of the discrepancy. The Utah results by age group agreed with those produced by Vitalnet. The only way to resolve the discrepancy would be to look at the Utah source code (not possible) or have another source compare the results.





New Mexico NM does AA statistics with an IBIS system. For Is overweight or obese (BMI 25+)? Weighted %yes results agree exactly:

2005 60.36 3,231
2006 59.74 3,879
2007 60.82 3,924
2008 59.94 3,734
2009 61.78 5,322
2010 60.70 4,188

NM does AA statistics with an IBIS system. For Ever had diabetes (excluding pregnancy)? Weighted %yes results agree exactly:

2005 7.27 503
2006 7.33 615
2007 7.82 656
2008 7.86 648
2009 8.61 1,038
2010 8.49 871

NM does AA statistics with an IBIS system. For Ever had diabetes (excluding pregnancy)? AA Weighted %yes results DO NOT agree exactly:

2005 7.12 501
2006 7.08 613
2007 7.59 652
2008 7.70 646
2009 8.36 1,032
2010 8.14 865

Sent email to Vivian Heye asking for clarification about AA std age groups. She referred to Lois Haggard and Wayne Honey.

New Mexico age adjusted results have one decimal place. 45 comparisons were done. All matched between the NM and Vitalnet results.

What about age adjustment for mammography? Just 40+. First, let's look at cervical screening. That includes all ages. Alaska 2008 weighted % yes for Pap test within past three years is 83.6%. The same figure, age-adjusted, is 80.4%. Alaska 2008 weighted % yes for mammogram within past two years is 67.6%. The same figure, age-adjusted, is 67.8%. They use three categories: 40-49, 50-64, and 65+. For 2006, CDC says 73.4% of females GE 40 had a mammogram within past two years. So I'm confused. CDC does not have 2008 Alaska data results.

Try age-adjustment for mammography using Utah data. Utah IBIS says 68.21% for 2006. Vitalnet says 73.16%. CDC pub says 68.2%.

Journals to consider:
American Journal of Public Health
American Journal of Preventive Medicine

Articles: BRFSS trends free full text - full population - exclude population such as 18-44

http://www.ncbi.nlm.nih.gov/pubmed/21918494
1: Centers for Disease Control and Prevention (CDC). State-specific trends in lung cancer incidence and smoking--United States, 1999-2008. MMWR Morb Mortal Wkly Rep. 2011 Sep 16;60(36):1243-7. PubMed PMID: 21918494.
http://www.cdc.gov/mmwr/preview/mmwrhtml/mm6036a3.htm?s_cid=mm6036a3_w
Uses AA - NO - used for cancer incidence rates

MMWR Weekly November 20, 2009 / 58(45);1259-1263
Estimated County-Level Prevalence of Diabetes and Obesity --- United States, 2007
http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5845a2.htm
Uses AA - YES

http://www.ncbi.nlm.nih.gov/pubmed/21993341
1: Centers for Disease Control and Prevention (CDC). Prevalence of coronary heart disease--United States, 2006-2010. MMWR Morb Mortal Wkly Rep. 2011 Oct 14;60(40):1377-81. PubMed PMID: 21993341.
http://www.cdc.gov/mmwr/preview/mmwrhtml/mm6040a1.htm?s_cid=mm6040a1_w
Uses AA - YES

http://www.ncbi.nlm.nih.gov/pubmed/22005613
1: Hayes DK, Fan AZ, Smith RA, Bombard JM. Trends in selected chronic conditions and behavioral risk factors among women of reproductive age, behavioral risk factor surveillance system, 2001-2009. Prev Chronic Dis. 2011 Nov;8(6):A120. Epub 2011 Oct 17. Erratum in: Prev Chronic Dis. 2012 Jan;9:E33. PubMed PMID: 22005613; PubMed Central PMCID: PMC3221562.
Uses AA - NO

http://www.ncbi.nlm.nih.gov/pubmed/21779530
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3133759/#B17
1: Shao Q, Chin KV. Survey of American food trends and the growing obesity epidemic. Nutr Res Pract. 2011 Jun;5(3):253-9. Epub 2011 Jun 21. PubMed PMID: 21779530; PubMed Central PMCID: PMC3133759.
Uses AA - NO

http://www.ncbi.nlm.nih.gov/pubmed/21672409
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3136983/
1: Zhang Z, Zhang L, Penman A, May W. Using small-area estimation method to calculate county-level prevalence of obesity in Mississippi, 2007-2009. Prev Chronic Dis. 2011 Jul;8(4):A85. Epub 2011 Jun 15. PubMed PMID: 21672409; PubMed Central PMCID: PMC3136983.
Uses AA - NO

http://www.ncbi.nlm.nih.gov/pubmed/21462724
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3281295/
1: Hardman K, Hunt KJ, Carter RE, Jenkins C, Hill R, Lackland DT. Diabetes management and vaccination rates in the Southeast United States, 2000 through 2007. Ethn Dis. 2011 Winter;21(1):13-9. PubMed PMID: 21462724; PubMed Central PMCID: PMC3281295.
Uses AA - YES

http://www.ncbi.nlm.nih.gov/pubmed/20829745
http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5935a1.htm?s_cid=mm5935a1_w
1: Centers for Disease Control and Prevention (CDC). State-specific trends in fruit and vegetable consumption among adults --- United States, 2000-2009. MMWR Morb Mortal Wkly Rep. 2010 Sep 10;59(35):1125-30. PubMed PMID: 20829745.
Uses AA - YES

http://www.ncbi.nlm.nih.gov/pubmed/20730503
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2988151/?tool=pubmed
1: Zhu J, Brawarsky P, Lipsitz S, Huskamp H, Haas JS. Massachusetts health reform and disparities in coverage, access and health status. J Gen Intern Med. 2010 Dec;25(12):1356-62. Epub 2010 Aug 21. PubMed PMID: 20730503; PubMed Central PMCID: PMC2988151.
Uses AA - NO - for the situations where warranted

http://www.ncbi.nlm.nih.gov/pubmed/20394695
1: Harris CD, Pan L, Mukhtar Q. Changes in receiving preventive care services among US adults with diabetes, 1997-2007. Prev Chronic Dis. 2010 May;7(3):A56. Epub 2010 Apr 15. PubMed PMID: 20394695; PubMed Central PMCID: PMC2879988.
Uses AA - YES

http://www.ncbi.nlm.nih.gov/pubmed/19894079
1: Brown DW. Smoking prevalence among US veterans. J Gen Intern Med. 2010 Feb;25(2):147-9. Epub 2009 Nov 6. PubMed PMID: 19894079; PubMed Central PMCID: PMC2837499.
Uses AA - YES

http://www.ncbi.nlm.nih.gov/pubmed/19564169
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2714952/?tool=pubmed
1: Salomon JA, Nordhagen S, Oza S, Murray CJ. Are Americans feeling less healthy? The puzzle of trends in self-rated health. Am J Epidemiol. 2009 Aug 1;170(3):343-51. Epub 2009 Jun 29. PubMed PMID: 19564169; PubMed Central PMCID: PMC2714952.
Uses AA - YES

http://www.ncbi.nlm.nih.gov/pubmed/19282813
http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5809a1.htm
1: Centers for Disease Control and Prevention (CDC). State-specific prevalence and trends in adult cigarette smoking--United States, 1998-2007. MMWR Morb Mortal Wkly Rep. 2009 Mar 13;58(9):221-6. PubMed PMID: 19282813.
Uses AA - NO

http://www.ncbi.nlm.nih.gov/pubmed/16971886
http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5536a1.htm
1: Centers for Disease Control and Prevention (CDC). State-specific prevalence of obesity among adults--United States, 2005. MMWR Morb Mortal Wkly Rep. 2006 Sep 15;55(36):985-8. PubMed PMID: 16971886.
Uses AA - NO

http://www.ncbi.nlm.nih.gov/pubmed/18556598
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2446467/?tool=pubmed
1: Kabir Z, Connolly GN, Clancy L, Koh HK, Capewell S. Coronary heart disease deaths and decreased smoking prevalence in Massachusetts, 1993-2003. Am J Public Health. 2008 Aug;98(8):1468-9. Epub 2008 Jun 12. PubMed PMID: 18556598; PubMed Central PMCID: PMC2446467.
Uses AA - YES

http://www.ncbi.nlm.nih.gov/pubmed/18341771
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2396974/?tool=pubmed
1: Kabir Z, Connolly GN, Clancy L, Koh HK, Capewell S. Coronary heart disease deaths and decreased smoking prevalence in Massachusetts, 1993-2003. Am J Public Health. 2008 Aug;98(8):1468-9. Epub 2008 Jun 12. PubMed PMID: 18556598; PubMed Central PMCID: PMC2446467.
Uses AA - NO - confuses age-adjustment with survey adjustment

http://www.ncbi.nlm.nih.gov/pubmed/18268146
http://www.healthmetricsandevaluation.org/publications/summaries/trends-and-cardiovascular-mortality-effects-state-level-blood-pressure-and-un
1: Ezzati M, Oza S, Danaei G, Murray CJ. Trends and cardiovascular mortality effects of state-level blood pressure and uncontrolled hypertension in the United States. Circulation. 2008 Feb 19;117(7):905-14. Epub 2008 Feb 11. PubMed PMID: 18268146.
Uses AA - YES

http://www.ncbi.nlm.nih.gov/pubmed/18701879
http://www.cdc.gov/mmwr/preview/mmwrhtml/ss5707a1.htm
1: Kilmer G, Roberts H, Hughes E, Li Y, Valluru B, Fan A, Giles W, Mokdad A, Jiles R; Centers for Disease Control and Prevention (CDC). Surveillance of certain health behaviors and conditions among states and selected local areas--Behavioral Risk Factor Surveillance System (BRFSS), United States, 2006. MMWR Surveill Summ. 2008 Aug 15;57(7):1-188. PubMed PMID: 18701879.
Uses AA - NO

http://www.ncbi.nlm.nih.gov/pubmed/21976118
http://www.cdc.gov/mmwr/preview/mmwrhtml/mm6039a4.htm?s_cid=mm6039a4_w
1: Centers for Disease Control and Prevention. Vital signs: alcohol-impaired driving among adults--United States, 2010. MMWR Morb Mortal Wkly Rep. 2011 Oct 7;60(39):1351-6. PubMed PMID: 21976118.
Uses AA - NO

http://www.ncbi.nlm.nih.gov/pubmed/17399752
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2864630/?tool=pubmed
1: Sturm R. Increases in morbid obesity in the USA: 2000-2005. Public Health. 2007 Jul;121(7):492-6. Epub 2007 Mar 30. PubMed PMID: 17399752; PubMed Central PMCID: PMC2864630.
Uses AA - YES

http://www.ncbi.nlm.nih.gov/pubmed/17357360
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1820442/
1: Harper S, Lynch J. Trends in socioeconomic inequalities in adult health behaviors among U.S. states, 1990-2004. Public Health Rep. 2007 Mar-Apr;122(2):177-89. PubMed PMID: 17357360; PubMed Central PMCID: PMC1820442.
Uses AA - NO

Different state standards:

7 7 // NM bf
8 8 // AK bf
9 9 // UT bf
10 10 // MO bf

AA_STD_AGE_GRPS_STR "Klein and Shoenborn #10: 18-, 30-, 40-, 50-, ..., 80+ (NM)"
AA_STD_AGE_GRPS_STR "Klein and Shoenborn #9: 18-, 25-, 35-, 45-, 65+ (AK)"
AA_STD_AGE_GRPS_STR "Klein and Schoenborn: 18-, 25-, 35-, 45-, ..., 85+ (UT)"
AA_STD_AGE_GRPS_STR "Klein and Schoenborn: 18-, 25-, 35-, 45-, ..., 65+ (MO)"

Healthy People Objectives from BRFSS