Receiver Operating Characteristic (ROC) curve

Receiver operator characteristic (ROC) analysis is a quantitative method applicable to a binary classification that generally will have been determined from continuous data based on an established threshold (cut-off) value. [1] However, discrete binary categories can also be used in ROC analysis.

Description

Drawing an ROC curve involves plotting sensitivity (on the y-axis) versus 1-specificity (on the x-axis). This can also be thought of as a plot of the fraction of true positives [true positives/(true positives + false negatives)] versus the fraction of false positives [false positives/(true negatives + false positives)]. [2]

This graphical approach of ROC analysis makes it relatively easy to grasp the inter-relationships between the sensitivity and specificity of a particular measurement. [3] [4] [5] Consequently, when medical diagnostic tests are developed, ROC analysis is helpful in determining the threshold value at which the test result is deemed to be abnormal and in comparing the performance of a new test to that of prior diagnostic tests.[6]

In addition, calculations of the area under the ROC curve provide a summary measure of the accuracy of the diagnostic test,[7] [8] which can also be thought of as the ability of the test to correctly classify (or discriminate between) those with and without the disease. For a perfect test, the ROC area under the curve (AUC) would be 1.0 whereas for a test that is no better than chance the AUC would be 0.5.

A review of software programs for ROC analysis is available [9] as are links to ROC analysis software. [10] [11]

History

ROC analysis had its beginnings in observations made in Britain during World War II when radar receiver operators were being assessed on their ability to differentiate signal (e.g., enemy aircraft) from noise (e.g., flocks of birds). Not only did individual radar operators differ in their skills, but changes in the radar receiver gain levels could influence signal to noise ratios. Subsequent developments of the concept of ROC (which has also been referred to as the relative operating characteristic) were an outgrowth of work in statistical decision theory and psychology. [12]

Since that time, ROC analysis has been used in a number of fields including engineering, quality control (e.g., materials testing), weather forecasting, and psychology (particularly psychophysics). [13] Its use in medicine to assess diagnostic test performance was first described by Lusted in 1971. [14]

Principal Use

A common use in medicine involves assessing performance of diagnostic tests, including radiologic evaluations. However, use of ROC curves remains important in each of the areas listed above.

Advantages

[15] [16]

• Simple graphical approach facilitates visual interpretation of data
• Represents accuracy as a composite measure over the entire range of the test
• Independent of prevalence, which simplifies sampling
• Threshold determination can consider all possible cut-offs
• Permits calculation of useful summary measures (e.g., area under the curve)
• Permits comparison of two or more curves (e.g., comparing new test with prior standard)

Shortcomings

[17] [18]

• Calculation can be cumbersome without specialized computer software
• Design of studies to use ROC curves in defining test accuracies and disease thresholds can be complicated
• Large sample sizes may be needed to generate smooth ROC curves

Examples in Informatics

Within informatics there are many examples of the use of ROC curves, particularly in the realms of medical decision analysis and text retrieval and processing. [19] [20] [21] [22] [23]

Sources

1. Wikipedia. Receiver Operating Characteristic. Accessed on February 29, 2008 at: http://en.wikipedia.org/wiki/Receiver_operating_characteristic
2. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993 Apr;39(4):561-77. Accessed on February 29, 2008 at: http://www.clinchem.org/cgi/reprint/39/4/561
3. Wise Interface for Statistics Education. ROC Curves. [24]
4. The magnificent ROC. [25]
5. Obuchowski NA. Receiver operating characteristic curves and their use in radiology. Radiology. 2003 Oct;229(1):3-8. [26]
6. Obuchowski NA, Lieber ML, Wians FH Jr. ROC curves in clinical chemistry: uses, misuses, and possible solutions. Clin Chem. 2004 Jul;50(7):1118-25. Accessed on February 29, 2008 at: http://www.clinchem.org/cgi/content/full/50/7/1118
7. Med-Calc. ROC curve analysis: introduction. [27]
8. Stephan C, Wesseling S, Schink T, Jung K. Comparison of eight computer programs for receiver-operating characteristic analysis. Clin Chem. 2003 Mar;49(3):433-9. http://www.clinchem.org/cgi/content/full/49/3/433#T3
9. Software programs available from the Kurt Rossmann Laboratories. Accessed on February 29, 2008 at: http://www-radiology.uchicago.edu/krl/KRL_ROC/software_index6.htm
10. ROC Analysis: Web-based Calculator for ROC Curves. Accessed on February 29, 2008 at: http://www.rad.jhmi.edu/jeng/javarad/roc/JROCFITi.html
11. Swets JA. The relative operating characteristic in psychology. Science, 1973 Dec. 7;182(4116): 990-1000. [28]
12. Swets JA. Measuring the accuracy of diagnostic systems. Science. 1988 Jun 3;240(4857):1285-93. Accessed on February 29, 2008 at: http://www.sciencemag.org/cgi/reprint/240/4857/1285.pdf?ijkey=5144d06a071e0c18afdb3d9437a532852adea2c9
13. Lusted LB. Signal detectability and medical decision-making. Science 1971; 171:1217-1219. [29]
14. Lasko TA, Bhagwat JG, Zou KH, Ohno-Machado L. The use of receiver operating characteristic curves in biomedical informatics. J Biomed Inform. 2005 Oct;38(5):404-15. Accessed on February 29, 2008. Abstract located at: http://www.ncbi.nlm.nih.gov/pubmed/16198999
15. Wieland SC, Brownstein JS, Berger B, Mandl KD. Automated real time constant-specificity surveillance for disease outbreaks. BMC Med Inform Decis Mak. 2007 Jun 13;7:15. [30]
16. Aphinyanaphongs Y, Tsamardinos I, Statnikov A, Hardin D, Aliferis CF. Text categorization models for high-quality article retrieval in internal medicine.

J Am Med Inform Assoc. 2005 Mar-Apr;12(2):207-16. Accessed on February 29, 2008 at: http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pubmed&pubmedid=15561789

1. Dexheimer JW, Brown LE, Leegon J, Aronsky D. Comparing decision support methodologies for identifying asthma exacerbations. Medinfo. 2007;12(Pt 2):880-4. [31]
2. Hripcsak G, Friedman C, Alderson PO, DuMouchel W, Johnson SB, Clayton PD. Unlocking clinical data from narrative reports: a study of natural language processing. Ann Intern Med. 1995 May 1;122(9):681-8. [32]