Intelligent information: a national system for monitoring clinical performance Bottle A Aylin P
Review of - Bottle A, Aylin P. Intelligent information: a national system for monitoring clinical performance. Health Serv Res. 2008 Feb;43(1 Pt 1):10-31.
The objective of the paper was to discuss the use of statistical process control charts to monitor clinical performance and what the impact of that monitoring can be to the quality of care provided.
The “US approach” p. 11 and the “UK approach” p. 14 to measuring and monitoring quality are discussed in respect sections as well as the “Role and Users of Out Tool” p.16 of their monitoring tool. This section states and then illustrates how their tool is complementary to government-led performance monitoring.
The “Real-World Examples of Use” section describes scenarios involving the following aspects of the tool. All the real world examples and source data are derived from English NHS hospitals including but not limited to: Walsall Hospital NHS Trust and Bradford Teaching Hospitals NHS Trust. Note that a trust can include several hospitals. The real world examples are organized around the functionality listed below:
- Procedure-Specific Mortality Alarms
- Disease-Specific Readmission Alarms
- Mortality Alarms Involving Multisite Hospitals
- Length of Stay Alarms Involving Mutiple Hospitals
“The Use of the Tool in Practice: Following up a suspected high outcome rate” p.19 goes into detailed discussion of the typical users workflow in his or her use of the system including logging on, requesting desired analysis and how he or she might act on the results of that analysis. This includes screen shots of the “Online Monitoring System” front page seen after logging in. The section also details the importance of differentiating between relevant, actionable alerts and “false alarms”.
The “Tools Methodology sections” p. 21-26 discusses the data sources, availability and quality of the data, the outcomes measured, the use of statistical process control charts, estimation of the expected risk of each outcome, benchmarks and setting thresholds and future improvements to the system. The tool uses 9 years of NHS hospitals admission data covering acute and community hospital trusts. Additionally there is a monthly submission from each trust via a data warehouse, the NHS-Clearing Service from April 2005 resulting in data that is about 6 weeks out of data at any given time.
Coded Data used
- The diagnosis codes used are ICD10 (as opposed to ICD9 used in the US) that is then assigned to one of 259 clinically meaningfully groupings using AHRQ’s CCS classifications.
- 12 operation fields use U.K. OPCS4 codes. No standard from grouping OPCS4 codes exists so groups were created with the expect advise of various professional bodies.
The statistical process control chart that the article states was the “most powerful test for detecting unacceptably high rates for a given false-positive rate” (Mosutakides 1986) was the CUMSUM, log-likelihood cumulative sum.
Benchmarking was done against the trust national average. With the specific aim of the chart detecting twice or over the national odds for poor performance and half or under the national odds for good performance.
The article concludes that they system created “allows the monitoring of clinical outcomes with a short time lag”. It states the tools potential to actionable event driven decision support to quality administrators that could result in improved quality of care. This would allow for audits that could reveal both problems and good practice more proactively then the U.K. Healthcare Commission “annual health check” currently used for this type of auditing and benchmarking.
For myself personally I would be eager to see a version of this tool that is or could be applied to in the U.S. and the article does mention that as a possibility as health information technology becomes broadly implemented here and the complexities of the private-public model are addressed.
Reviewed by Christine Klein for BMI512 Spring 2008
