View source for Evaluating Clinical Decision Support Systems:Monitoring CPOE Order Check Override Rates in the Department of Veterans Affairs’ Computerized Patient Record System

A review of research article (2008) titled "Evaluating Clinical Decision Support Systems: Monitoring CPOE Order Check Override Rates in the Department of Veterans Affairs’ Computerized Patient Record System"<ref name="Lin">Lin, C.P., Payne, T. H., Nichol, W. P., Hoey, P. J., Anderson, C. L., Gennari, J. H. Evaluating Clinical Decision Support Systems: Monitoring CPOE Order Check Override Rates in the Department of Veterans Affairs’ Computerized Patient Record System.2008.Journal of the American Medical Informatics Association Volume 15 (5), 620-626. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2528033/</ref> by Lin et al.

== First review ==

===Objectives===

To reevaluate and compare the VA Veterans Affairs Puget Sound-[http://www.pugetsound.va.gov/ VA Puget Sound] Health Care System’s computerized practitioner order entry [[CPOE]] system generated critical order checks over ride rates in 2001 to that of 2006. A secondary objective was to assess the impact of system changes related to topical medication order checks.

===Introduction===

VA Puget Sound Health System health care providers had been using CPOE which has an inbuilt order checking to mitigate the potential medication errors in orders in view of patient safety since 1997. Most of the times the order checks alerts with “high severity” were overridden by healthcare providers in view of clinical irrelevance. A study was conducted in 2001 to analyze the various factors which affect the order checking overrides. A follow up study had been conducted to reassess if the changes in the [[Computerized Patient Record System]] (CPRS) order check rules have an influence on the overridden rates of order checks.

===Design and Setting===

The Computerized Patient Record System (CPRS) part of the larger [[Veterans Health Information Systems and Technology Architecture (VistA)]] had been used by VA Puget Sound for note entry, results review and order entry. Critical overridden order checks were analyzed for patients from VA Puget Sound Health Care System Hospitals following VA centrally developed and controlled National Drug File (NDF) and few locally developed Drug files. 
VA CPOE also classifies order checks as “critical” or “significant” were high mainly for drug-allergy and drug-drug order checks and also few other types. To be classified as critical, the interaction must be identified in the manufacturer’s black box warning, or be well documented in the literature to cause significant sequelae. Significant drug interactions do not meet the critical [[drug-drug interaction]] criteria but are still thought to have substantial clinical importance.

===Methods===

Retrospective analysis by post-hoc logging into system for order activity for two time 3 day periods were Wednesday, January 4, 2006 14:11 to Friday, January 6, 2006 15:46 (Period1) and from Monday, January 9, 2006 08:41 to Wednesday, January 11, 2006 10:30 (Period 2).
*Inclusion criteria: Direct practitioner entry in the ordering package.
*Exclusion criteria: Orders entered through the Pharmacy, Lab or Radiology packages.
Lin et al., defined override rate as the percentage of distinct orders receiving a high severity, critical order check that are signed. 



===Results===

Chi-square contingency table test was applied to compare results from the 2001 and 2006 studies.	
Eight different types of critical order checks identified were Drug-Drug Interaction, [[Drug-allergy interaction]], Clozapine appropriateness, Procedure uses intravenous contrast media - abnormal biochemistry result/no creatinine results within 30 days, Metformin - no serum creatinine, Patient has no allergy assessment, Patient allergic to contrast media, Procedure uses intravenous contrast media and patient is taking metformin.
The percentages of overridden high severity order checks had increased from 0.5% in 2001 to 2.5% in 2006. Drug-Drug order checks override rate percentage declined by a percentage in 2006 (87%) than in 2001(88%) with rate being still over 85%. But the percentage of Drug-Allergy Order Checks escalated a difference of 12% from 2001(69%) to 2006(81%).Pearson’s chi-square contingency table test calculated that overall there had been a statistically significant change in the rate of critical order checks from 2001 to 2006. A slight decrease in critical drug-drug overrides on Topical form medications was observed from 29% in 2001 to 25.9% in 2006.

===Discussion===
 
Lin et al., study highlighted that the override rates were due to diverse factors. It strongly agrees with  Kuperman et al. who recommended that drug knowledge base designers need to provide the necessary tools to understand, customize and share rule information and that organizations need to create policy and procedure infrastructure to support the use of these tools.
System behavior should also be easily monitored, and ease of evaluation and the development of built-in evaluation tools should be accessible in system design.
Abookire et al. study highlighted that periodic evaluation of system operators to identify the unexpected effects on order checking and particularly after the introduction of new policies, or updates, or changes in the system. Further studies will be interesting in this aspect. 

===Limitations===

The retrospective analysis applied in this study, sampling the data orders at different times during the year, identifying the factor or combination of factors, both technical and social, may have contributed to new system behaviors including significantly higher drug allergy order check and override rates. So, Lin et al., could not control for many possible changes in the environment and so were not certain about the cause of the overridden rates. 

===Conclusion===

Lin et al were successful in finding the quantitative data but still need to assume few factors which could have  contributed to such high percentages of overridden rates of critically high rates  particularly life threatening order checks for drug-allergy orders which might have been due to policy changes and changes in rule bases and drug files.
Simultaneous Analysis of order check systems both qualitatively observational  with quantitative order checks monitoring to better understand clinical decision making and the interactions physicians have with information and decision support systems. These outcomes must be clinically relevant for correlation.


===Comments===
More studies on overridden rates qualitatively and quantitatively on patient outcomes and educating the physicians about the documented clinically relevant data of the importance of order checks, might decrease the overridden rates in future.
Related study at the VA Puget Sound Healthcare System  [[A qualitative cross-site study of physician order entry|A qualitative cross-site study of physician order entry]]

=Second Review=
==Background==
The purpose of this study was to identify and measure the number of override rates in 2006 for computerized practitioner order entry (CPOE) for the Veteran Affairs Puget Sound Health Care System. Alerts are set in place to reduce errors and provide information over drug-allergies and drug-drug interactions <ref name="van"> van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of Drug Safety Alerts in Computerized Physician Order Entry. J Am Med Inform Assoc. 2006;13(2):138–47. http://www.ncbi.nlm.nih.gov/pubmed/16357358 </ref>. A previous study conducted in 2001 would compare results. 

==Methods==
A post-hoc logging program helped identify and analyze ordering data to then measure the number of orders, order check types, and order check overrides by order check type. Pearson’s chi-square tests were used to compare results from 2006 to previous study in 2001.

==Results==
The study reviewed 37,040 orders that generated '''908 (2.5%''') critical order checks and identified an '''74/85 (87%)''' override rate for drug-drug critical alerts compared to '''95/108 (88%)''' in the 2001 study '''(X2=0.04, df=1,p=0.85)'''. The study also identified a '''341/420 (81%)''' override rate compared to '''72/105 (69%)''' in 2001 for drug-allergies '''(X2=7.97, df=1,p=0.005)'''. Of these override rates, there were '''420/37040 (1.13%)''' orders generated compared to '''105/42621 (0.25%)''' during a drug-allergy order check in 2001.

==Conclusion==
The override rates of these orders generated including drug-drug and drug-allergy order checks were high. From the 2001 study to the current 2006 study, there was a significant increase in the frequency of drug-allergy order checks. For purposes of clinical computing systems frequent monitoring of override rates and study further physician action during ordering and decision support.

==Comments==
I have been interested in the varying VA’s [[EHR]]s as it has provided insight for a wide range of uses. It was an interesting read to look at override rates of the alerts put in place for efficiency and patient safety. There were policy and drug file changes between 2006 and 2001 overrides that would have led to believe the need for less overrides in 2006. It indicates the need for continued monitoring.

==References==
<references/>

[[Category: CDS]]
[[Category: Reviews]]
[[Category: CPOE]]