Validity of using an electronic medical record for assessing quality of care in an outpatient setting

From Clinfowiki
Jump to: navigation, search

One of the frequently cited advantages of an electronic medical record (EMR) is its use in assessment of healthcare quality measures. [1] EMRs often capture a broader range of data than traditional administrative billing systems, and contain aggregate data on all patients of the organization, as opposed to separate, payer-specific sources. Yet few studies have validated EMR-based strategies on data retrieved for assessment of healthcare quality. One such assessment is the HEDIS 2004 quality measure on diagnosis of bacterial pharyngitis (strep throat) in children, which mandates confirmatory diagnostic testing for Group A Streptococcal infection(GAS) before prescribing antibiotics. The purpose of this study was to compare information retrieval using an EMR-based versus a traditional administrative data-based strategy, while assessing compliance with the HEDIS 2004 pharyngitis quality measure within a pediatric outpatient clinic.


The investigators sought to identify all visits to the Yale-New Haven Hospital Primary Care Center by children ages 3-18 yr diagnosed with pharyngitis within a one year period. The first search method used the outpatient EMR system (GE Logician version 5.2), which captured all outpatient encounters and prescriptions written at the clinic, yet was not used as an office management system for administrative billing functions. Although data entry was somewhat structured, much of the clinician entered data was ultimately saved as one large textual field. The EMR was first searched for all documents within the date ranges that contained “throat” and “phary” strings (note: this pulls both normal and abnormal exams of the throat as “throat” is a key word in the PE section). Next, selected fields (using field delimiter characters within the text block) of these documents were downloaded into SAS for further analysis, including birthdate, date of encounter and the entire text of the medical note. After isolating notes for children within the specified age range, the “assessment” portion of the remaining documents’ text was searched for full and partial spelling, as well as misspelling of “pharyngitis”, “tonsillitis”, “scarlet fever” and “strep throat”, excluding records with a co-existing diagnosis requiring antibiotics (e.g. otitis media). Diagnostic testing for GAS was assessed by both evidence of a diagnostic lab result, and by whether an order was placed for the diagnostic test. Prescriptions for antibiotics at the time of the visit were captured by the EMR.


The second search strategy utilized a data search of the administrative billing database. ICD-9 codes generated from forms completed by the clinician within the clinic at the time of the encounter, and hospital billing codes for GAS diagnostic testing were searched for specific target codes. The administrative billing database did not capture prescription information; therefore the EMR system was used to investigate prescriptions written during those encounters identified by the administrative billing data. A reference group of encounters consisted of all records identified by either the EMR or administrative billing searches, which were manually reviewed by a pediatrician to confirm that the encounter was a true episode of pharyngitis. GAS testing and prescription identification were performed using the EMR search approaches.


Recall, precision and specificity of the two search methodologies in finding cases of pharyngitis were measured. The study compared the proportion of episodes of pharyngitis in which an antibiotic was prescribed with and without diagnostic testing for GAS. Both nonparametric paired analysis and Z-statistic testing that accounted for interdependence of samples were used for comparisons.


A total of 479 possible episodes of pharyngitis were identified by the EMR- and administrative data-based strategies, of which 391 (82%) were confirmed by manual review as true episodes of pharyngitis. The EMR-strategy had a greater recall (96%) vs. the administrative data-based strategy (62%), which was statistically significant (p< 0.0001). The precision of each strategy, however, was not significantly different (87% EMR vs. 86%). The administrative data-based strategy had a higher specificity (55% vs. 34% EMR), meaning that it found fewer erroneous episodes than the EMR-based approach.


Diagnostic testing for GAS was ordered in 77% of the true episodes of pharyngitis within the reference group, similar to that found using the administrative-based strategy (76%). Results of diagnostic testing for GAS varied significantly from the EMR-based strategy, however, depending on method of identification; evaluating laboratory results yielded a rate of 33%, as opposed to 71% when examining orders placed for the diagnostic test, both significantly less than that found with the other strategies. The strategies also differed as to the proportion of encounters in which an antibiotic was prescribed (23% reference, 20% EMR, 30% administrative dataset). Although the different strategies found varying numbers of encounters that had antibiotics prescribed (the ultimate denominator for the HEDIS measure), the percentage of those encounters that also included diagnostic testing for GAS was not significantly different (84% reference, 85% EMR, 82% administrative dataset).


Comment: There are a couple of issues with the study design and analysis that suggest some caution in interpretation of the results. The reference group of encounters was derived from the EMR and administrative data-based search strategies, and may not have captured all encounters with true pharyngitis. Thus, the true recall rate of each of these strategies from this population is not known. In addition, there was significant interdependence of the EMR and administrative data-based strategies (e.g. the administrative data strategy required use of the EMR for prescription analysis), which may have influenced the similar success rates at retrieving encounters with the diagnosis of pharyngitis with antibiotics prescribed and diagnostic testing performed. Finally, no comment was made on the consistency of documentation of the assessment and diagnosis within the EMR, although the investigators did search the UMLS thesaurus for all appropriate terms for streptococcal pharyngitis and found no other strings or phrases suggested beyond those used.

This study illustrated the need for validation of EMR-based data retrieval strategies. Although data was entered into the EMR in a somewhat structured manner, it was saved as a block of text which then required a complex text search algorithm, yet in the end was more customizable than the administrative data-based strategy which performed standard searches through highly structured and coded data. Furthermore, the choice of search strategy for evidence of diagnostic testing within the EMR-based approach affected the results (using lab results vs. orders for diagnostic test). Given the different recall rates and specificity of data retrieval between the two search strategies, one should take care when comparing previous quality assessments to results from an EMR without prior validation of the EMR search strategy.

--Jeffrey Merrill

15:32, 9 November 2006 (CST)