Evaluating health information exchange

From Clinfowiki
Jump to: navigation, search

Evaluating public health uses of health information exchange Jason S. Shapiro, Journal of Biomedical Informatics 40 (2007) S46-S49

Shapiro states that while the potential public health benefits of health information exchange (HIE) are clear there is little supporting evidence in the current literature. He notes that while there are several health use cases for HIE only a few have been implemented and even fewer have been evaluated.

The author lists six potential public health use cases for HIE (as described by Mostashari et al.)

  1. Mandated reporting of laboratory diagnoses: Shapiro contends mandated reports would be timely and complete if direct electronic reporting to state health departments via HIEs occurred. He proposes using the volume of relevant data (expressed as the ratio of relevant data reported to all relevant data in a laboratories database) as the primary evaluation measure. He also proposes pre- and post-implementation studies of efficiency, completeness and timeliness.
  2. Mandated reporting of physician-based diagnoses: Shapiro argues that HIEs would increase mandated physician reporting by automatic screening of data such as ICD-9 diagnostic codes, CPT codes and medications. However he acknowledges that the data so obtained would then have to be manually reviewed to exclude erroneous or non-relevant cases. He argues that with maturation of HIE systems more advanced informatics would lead to improved system recall. Unlike the first use case he does not see recall as a viable evaluation because of the cost and time constraints of determining all relevant data in the database. He advocates using precision (which would look at the number of erroneous or non-relevant cases that are included.)
  3. Public health investigation: Shapiro points out that in this use case the patient who has a reportable disease is already known and the health investigator need only access the HIE for additional information needed for the investigation rather than using more time consuming methods (making phone calls, travelling or obtaining paper records.) The ease with which this can be accomplished is dependent on the particular HIE (question of full electronic access vs. simple results retrieval.) Shapiro recommends using qualitative or semi-qualitative studies such as surveys, semi-structured interviews or observational techniques for evaluation.
  4. Disease based non-reportable laboratory data: HIEs will help centralize data and improve accessibility for health departments helping to guide public heath messages and rule out harmless epidemiological causes.
  5. Antibiotic-resistant organism surveillance: Culture resistant patterns would be transmitted through the HIE system to the health department and this information could then be used to construct community-wide antibiograms or notify clinicians about patients previously diagnosed with infections caused by an antibiotic resistant organism. Shapiro recommends evaluating rates of local resistance patterns as well as of nosocomial infections pre- and post- implementation.
  1. Population-level quality monitoring: HIEs would be used to monitor secondary and tertiary prevention of chronic diseases, e.g., rates of hemoglobin A1C levels for diabetes control. Quality metrics pre- and post- implementation could then be used to monitor preventive care.

Comments

Overall a good discussion of public health uses of HIE. However, I found approaches to evaluation in disease-based non-reportable data to be the least robust of all the use cases discussed. Also the evaluation of mandated reporting of physician-based diagnoses suggested by Shapiro involves manual review which would be time consuming.

Evaluating public health uses of health information exchange

Jason S. Shapiro




The potential of health information exchange (HIE) to improve the public health and public health activities may seem obvious, but current literature provides little evidence proving these effects. This article provides general evaluation methods for measuring the impact of HIE on public health in six use cases. For each use case the author gives a brief description for the case then discusses potential approaches to the evaluation of these use cases.


Mandated reporting of laboratory diagnoses.

  • Measuring the volume of relevant data transmitted using Electronic laboratory reporting directly to health departments would be a primary measure of success in this use case (i.e. recall): relevant ∩ retrieved/relevant. If feasible, a pre- and post-implementation study, could be conducted to demonstrate any change in the frequency of mandated reporting of laboratory diagnoses using HIE laboratory data.


Additional metrics of interest could look at changes in efficiency, completeness and timeliness of reporting.


Mandated reporting of physician-based diagnoses

Recall would be difficult to measure in this use case since measuring all relevant cases for the denominator would be time consuming and expensive. Another measure of interest is to evaluate the performance of the system (i.e. the precision): relevant ∩ retrieved/retrieved.

  • Measurement of precision would require a log file capturing all potential clinician reportable diseases detected by the system, followed by manual review to determine which cases are relevant.

Another useful evaluative measure might be to determine the volume of reporting pre- and post-implementation in much the same way that was suggested above for mandated reporting of laboratory diagnoses, as well as the efficiency, completeness and timeliness of reporting.

Public health investigation

Because implementation of this use case will vary depending on the extent to which the HIE incorporates clinical data, recommendation of a specific evaluation is difficult. One possibility is to do a qualitative or semi-qualitative study of the public health investigator’s experience through surveys, semi-structured interviews or observational techniques. In addition if this were done across multiple HIE projects, analysis might permit the development of a set of standard practices for HIE implementations to help identify the data elements and user interface features that are most essential to this use case.


Disease-based non-reportable laboratory data

Because this use case will rely on the development of faster, less expensive, and more accessible assays to clinicians, and these are being developed and deployed in parallel with HIE systems, they will likely act as a confounder, making it difficult to construct an evaluation plan that incorporates a pre-implementation phase.

Of most interest here is the gathering and analysis of empirical data to make new discoveries regarding the epidemiology of these common pathogens.


Antibiotic-resistant organism surveillance

Since the intervention here is the dissemination of a community-wide antibiogram, A study evaluating the rates of local antibiotic resistance patterns before and after the implementation of a community- wide antibiogram would be of interest. Similarly, studies could be done pre- and post-implementation on a hospital level to see if an antibiotic-resistant organism (ARO) notification system leads to earlier identification and isolation of ARO-infected patients, and to see if hospital rates of nosocomial ARO infections decrease.


Population-level quality monitoring

Again, this use case would require a pre- and post implementation study. A number of organizations have developed, or are developing, standard sets of quality measures that could be used for evaluation.


Conclusion

This paper describes preliminary suggestions for measuring the impact of HIE on public health in specific use cases. There are other secondary and tertiary benefits to improved public health that would be much more difficult to measure.

During the early phases of development of regional HIE, the projects will be limited in scope due to only partial penetration of local markets and therefore only partial data capture. Measures likely to be affected during this early phase are efficiency measures and costs. These early measures may be used to calculate a return on investment of the initial implementation costs.

As these HIE systems mature and begin to share data with one another, and a truly interoperable nationwide health information network (NHIN) begins to coalesce, quality and safety effects will begin to accrue and be measurable.


Limitations and Strengths

This article provides general evaluation methods for measuring the impact of HIE on public health in only six use cases and in two of this six cases , the author mention that it is difficult to construct an evaluation plan.

The good thing here is that he describe the use cases in some details and he also states some other benefits of the suggested evaluation measures in guiding and encouraging the process of HIE development and implementation.


by Ahmed Mahmoud


Evaluating public health uses of health information exchange Jason S. Shapiro

The potential of health information exchange (HIE) to improve the public health and public health activities may seem obvious, but current literature provides little evidence proving these effects. This article provides general evaluation methods for measuring the impact of HIE on public health in six use cases. For each use case the author gives a brief description for the case then discusses potential approaches to the evaluation of these use cases.

Mandated reporting of laboratory diagnoses

Measuring the volume of relevant data transmitted using Electronic laboratory reporting directly to health departments would be a primary measure of success in this use case (i.e. recall): relevant ∩ retrieved/ relevant If feasible, a pre- and post-implementation study, could be conducted to demonstrate any change in the frequency of mandated reporting of laboratory diagnoses using HIE laboratory data. Additional metrics of interest could look at changes in efficiency, completeness and timeliness of reporting.

Mandated reporting of physician-based diagnoses

Recall would be difficult to measure in this use case since measuring all relevant cases for the denominator would be time consuming and expensive. Another measure of interest is to evaluate the performance of the system (i.e. the precision): relevant ∩ retrieved /retrieved Measurement of precision would require a log file capturing all potential clinician reportable diseases detected by the system, followed by manual review to determine which cases are relevant. Another useful evaluative measure might be to determine the volume of reporting pre- and post-implementation in much the same way that was suggested above for mandated reporting of laboratory diagnoses, as well as the efficiency, completeness and timeliness of reporting.

Public health investigation

Because implementation of this use case will vary depending on the extent to which the HIE incorporates clinical data, recommendation of a specific evaluation is difficult. One possibility is to do a qualitative or semi-qualitative study of the public health investigator’s experience through surveys, semi-structured interviews or observational techniques. In addition if this were done across multiple HIE projects, analysis might permit the development of a set of standard practices for HIE implementations to help identify the data elements and user interface features that are most essential to this use case.

Disease-based non-reportable laboratory data

Because this use case will rely on the development of faster, less expensive, and more accessible assays to clinicians, and these are being developed and deployed in parallel with HIE systems, they will likely act as a confounder, making it difficult to construct an evaluation plan that incorporates a pre-implementation phase. Of most interest here is the gathering and analysis of empirical data to make new discoveries regarding the epidemiology of these common pathogens.

Antibiotic-resistant organism surveillance

Since the intervention here is the dissemination of a community-wide antibiogram, A study evaluating the rates of local antibiotic resistance patterns before and after the implementation of a community- wide antibiogram would be of interest. Similarly, studies could be done pre- and post-implementation on a hospital level to see if an antibiotic-resistant organism (ARO) notification system leads to earlier identification and isolation of ARO-infected patients, and to see if hospital rates of nosocomial ARO infections decrease.

Population-level quality monitoring

Again, this use case would require a pre- and post implementation study. A number of organizations have developed, or are developing, standard sets of quality measures that could be used for evaluation.

Conclusion

This paper describes preliminary suggestions for measuring the impact of HIE on public health in specific use cases. There are other secondary and tertiary benefits to improved public health that would be much more difficult to measure. During the early phases of development of regional HIE, the projects will be limited in scope due to only partial penetration of local markets and therefore only partial data capture. Measures likely to be affected during this early phase are efficiency measures and costs. These early measures may be used to calculate a return on investment of the initial implementation costs. As these HIE systems mature and begin to share data with one another, and a truly interoperable nationwide health information network (NHIN) begins to coalesce, quality and safety effects will begin to accrue and be measurable.

Limitations and Strengths

This article provides general evaluation methods for measuring the impact of HIE on public health in only six use cases and in two of this six cases , the author mention that it is difficult to construct an evaluation plan. The good thing here is that he describe the use cases in some details and he also states some other benefits of the suggested evaluation measures in guiding and encouraging the process of HIE development and implementation.

Reviewed by Ahmed Mahmoud

Evaluating public health uses of health information exchange

Pages S46-S49 Jason S. Shapiro


The paper focuses on providing general evaluation methods for 6 different use cases representing the use of health information exchange (HIE) in public health aspects

Let’s give a look at the different use cases and proposed evaluation method

Mandating reporting of laboratory diagnosis

Where reporting is done electronically based on the results of ordered laboratory tests The author recommended the calculation of Recall as a measure for evaluating the impact of the use of HIE in reporting such laboratory diagnosis before and after implementing the HIE public health reporting functionality


Mandating reporting of physician based diagnosis

Reporting by physicians independently to Health Department about diagnosis or risk factors utilizing ICD9-CM and CPT codes The author pointed out the difficulty in adopting the Recall as a measure and suggested the use of Precision

Public health investigation

Where the use of computerized queries replaces the traditional investigational activities The author pointed out the difficulty of specifying certain measure(s) for this Because it depends on the nature of Clinical Information System and data scope, amount exposed and granulated data. As a solution to this the author recommended to have qualitative or semi qualitative studies and surveys.


Disease-based non-reportable laboratory data

Reporting certain diseases of interest for biosurveillance It is difficult to specify certain evaluation plan for such data because the discovery of such data is dependent on development of new, cheap and fast assays to help clinicians. Therefore an evaluation plan should be constructed in parallel to HIE construction based on community specific parameters.


Antibiotic-resistant organism surveillance

The detection takes place through electronic reporting of microbiology culture resistance results. Evaluation may be done by estimating the rate of resistance before and after system implementation


Population level quality monitoring

It emphasizes the public health involvement in secondary prevention of chronic diseases by consistent reporting rate of preventive measures (ex. rate of colonoscopy) Pre and post-implementation studies will be best in this case by the use of standard sets of quality measures


Comments

Reading the title and introduction is quite enough to realize how proactive the author was in choosing such topic where all people are focusing on bringing HIE initiatives to life, such vision allows us not to focus on implementation and loose our main goal which is improving healthcare quality for the whole population.

The author selected six different use cases, and I think that those are the most important use cases for the use of HIE in public health activities and the he started to describe them in quite well organized matter by describing the use case and providing the suggested evaluation approach, although the description was quite clear I felt the using diagrams here would be useful and saves time and get reader’s attention as the Chinese proverb says” A picture is better than thousands of words”

For those use cases where specifying a certain evaluation approach is difficult the author was really brilliant in providing reasons for this difficulty and simple roadmap for the adoption of other approaches suitable to different situation.

The author hit an important area of the use case of interest for biosurveillance as this one of the hottest topics that gained a lot of debates, but I was eager to have detailed information about this use case and it is impact on the rapid and early detection of bioterrorisim attacks so as to emphasize the concept of considering such use case for future design of HIE and Clinical Information Systems.


Reviewed By

Mohamed Aly Abd-Elmoneam

Clinical Information Systems BMI-512 winter 2008


L. Bernard-Pantin