Difference between revisions of "Garber"

Revision as of 15:04, 8 February 2012

Performance of a Web-based Clinical Decision Support System System for Internists Mark Graber and Ashlei Matthew

Question

Does a proprietary Clinical Decision Support System help find the correct diagnosis without extensive data input.

Background

The goal is to measure the sensitivity and speed of “Isabel” web-based CDS Isabel” (Isabel Healthcare Inc, USA) a second generation web-based CDS system. The goal of “Isabel” is to provide CDS to Internist by entering key findings (usually 3-6) or complete clinical narrative and with 2-3 seconds up to 1-2 minutes with outcome judged by presence of the correct diagnosis minutes in the first 30 diagnosis suggested by Isabel with 10 displayed per page. For a basis of medical knowledge, Isabel uses two key textbooks == Oxford Textbooks of Medicine 4th Edition and the Oxford Textbook of Geriatric Medicine and 46 major journals including General and Subspecialty Medicine and Toxicology.

Methods

61 consecutive “Case Records of Massachusetts General Hospital” New England Journal of medicine Vol. 350 == 166-1163204 V 353 189-198,205; only patients > 10 years of age and discussed actual diagnosis based on clinical and physical findings were Included as test cases. A total of 50 case reports were used to test Isabel. Two methods of data entry were used entering 3-6 key findings or full-text of the case report was copied and pasted into Isabel.

Results

When entering 3-6 findings : 48 of 50 cases the correct diagnosis was included in the top 30 suggestions. For 2 out of 50 of the cases not found, the correct diagnosis was not in the Isabel database; therefore, there was no way for Isabel to suggest the diagnosis. When copying and pasting the whole text, Isabel’s top 30 diagnosis list included the correct diagnosis 37 out of 50 cases. For 51% of those with correct diagnosis was found in the first 10 diagnosis displayed. For 71% of the test cases, the correct diagnosis was included in the first 20 diagnosis displayed on two separate pages. Manual entering data required ~ 1 minute and copy and paste took about 5 seconds.

Discussion

Isabel was able to suggest the correct diagnosis with the first 30 choices 37 out of 50 times or 48 out 50 times. Negative findings in clinical test narrative were missed for example “No chest Pain” was translated to “ chest pain“ and misguided Isabel. Commentary by Reviewer Timothy H Hartzog MD, FAAP ==

This article reads more like a glossy sales presentation than a journal articles. The underlying technology is never discussed; the methods used are skewed by non-blinded testers-who knew the correct diagnosis before testing the system.

NEJM case reports tend to be centered around rare and unusual cases not the typical cases; Rare cases tend to have unique finding which will Increase the chances of finding a match. Second weakness, the study’s choice of success being 1 positive match in one of the top 30 displayed diagnosis. This translates into of specificity of only 3 % which is not much better than randomly choosing 30 diagnosis/pages from any large medicine textbook. Third, there is no rank-order which is a problem because ordering test to cover 30 diagnoses would be an unnecessary waste of resources and pain for the patient. Fourth, weaknesses are there is no indication of the spread of diagnosis or were the other 29 diagnosis even a reasonable choice, were diagnosis grouped by organ system or other methodology.

Timothy Hartzog MD, FAAP Pediatric Hospitalis Medical university of South Carolina Medical Director of Information Technology hartzogt@musc.edu