Return on investment

From Clinfowiki
Jump to: navigation, search

Return on investment (ROI) analysis is a quantitative method of evaluation. It is used to assess the potential earnings outcome from potential investment or business project. ROI analysis can be used to assist in making business decisions about the future. The practice of using ROI to determine whether or not to move ahead with a project is called ROI Analysis. ROI is defined as the gain from an investment minus the cost of the investment all divided by the cost of the investment.

History

Return on investment analysis appears to have been in use for an extremely long time.

As published on the Project Management Institute (PMI) website by Gary R. Heerkens (MBA, CBM, PMP), "ROI is an evaluation of the incremental financial benefit a company expects to receive for a given amount of incremental expenditure. The term incremental has considerable significance here. ROI calculations are based entirely upon the economic change (both positive and negative) that would result from approving a particular project." [1]

Descriptions for the following calculations were defined for the following:

  • Net present value (NPV)
  • Internal rate of return (IRR)
  • Payback period [1]

Principal use

Measuring the future gain from a given project or investment. In this case, the attempt to quantitatively assess the future performance of an IT project is the principal use.

One could approach the ROI from the perspective of the Institute of Medicine Report, Crossing the Quality Chasm

  1. Safe: Reducing adverse drug events, inappropriate testing
  2. Effective: Reducing drug costs through appropriate prescribing
  3. Efficient: Reducing drug, laborotory, or radiologic utilization
  4. Timely: Reducing wait times
  5. Patient-centered: Reducing length-of-stay while hospitalized
  6. Equitable: Provides data to demonstrate equal delivery

Patient Costs

Not only does the use of EMRs reduce the cost for physicians, but it also will reduce patient costs as well. With a move to an EMR, the physician will no longer have some of the previous time and labor intensive tasks such as transcription of documents and chart pulling, storage, and refilling thus passing on the savings to the patient. [2]

Advantages

Return on investment analysis is a performance measure which allows for a quantitative measure to be applied to a business case decision. This can help remove emotion from a decision by having a standard measure to compare one investment to another.

A study conducted over a five year period found that the use of electronic medical records had significant cost saving, with an estimated net benefit of $86,000 per provider. “Benefits accrue primarily from savings in drug expenditures, improved utilization of radiology tests, better capture of charges, and decreased billing errors.” [3]

Shortcomings

All potential variables can be difficult to enter into the analysis. Return on investment analysis can be difficult to include all necessary variables and can not predict potential new variables, or how the investment may trigger unintended consequences. If the return on investment analysis is performed for a small pilot project, the analysis doesn’t always hold true for larger projects.

Evaluating success of EMR implementation

Dean Sittig, professor at UT Houston's School of Biomedical Informatics, has suggested a new set of criteria for determining for an EMR implementation. Based on Koch's Postulates and Hill's criteria for causation, these criteria are designed specifically for EMR evaluation.

  • Must have the hardware and software available before the effect is identified.
    • Need to at least estimate state of affairs before system is implemented…manual review
  • Show that clinicians are actually using the system that could produce the effect.
  • Show that the effect increases with increasing availability and usage of the system.
  • Show that all obvious “alternative explanations” for the effect are false.
  • Show the effect goes away when the system goes away.
  • Show that a similar effect occurs when a similar system is installed and used at a similar facility.


A survey of approximately 400 Healthcare IT News readers was conducted on EHR systems. Survey results on nine vendors were reported on September 14, 2015. Readers shared what they enjoyed and what they would change.[4]

A review from the U.S. Department of Health and Human Services - Health Information Technology discusses how ROI can be achieved. It summarizes results conducted by other studies. " [5]

Lessons Learned

Lessons learned on ROI were identified in "How to Drive ROI in Your Healthcare Improvement Projects Executive Report" published by the Health Catalyst: [6]

  • Measure the hard ROI, document the soft ROI
  • Push departments to focus on ROI, document changes and embrace process change
  • Build clinical champions to document and show outcomes
  • Continuously report to leadership on outcomes
  • Establish baseline and metrics prior to interventions
  • Obtain operational buyin. [6]

Examples in Informatics

Return on Investment for a Computerized Physician Order Entry System Rainu Kaushal, Ashish K. Jha, Calvin Franz, John Glaser, Kanaka D. Shetty, Tonushree Jaggi, Blackford Middleton, Gilad J. Kuperman, Ramin Khorasani, Milenko Tanasijevic, David W. Bates Brigham and Women's Hospital CPOE Working Group J. Am. Med. Inform. Assoc. 2006;13(3):261-266. PrePrint published May 1, 2006; doi:10.1197/jamia.M1984

Comments on Return on Investment (ROI) As It Applies to Clinical Systems Mark E. Frisse J. Am. Med. Inform. Assoc. 2006;13(3):365-367. PrePrint published May 1, 2006; doi:10.1197/jamia.M2072 [Full Text] [PDF]

S. Wang A cost-benefit analysis of electronic medical records in primary care. The American Journal of Medicine, Volume 114, Issue 5, Pages 397-403

Return on Investment (ROI) Estimates

While barriers of determining actual ROI for EMR implementations exist, companies such as Dr. Cloud EMR are providing EMR and EHR ROI estimates based on each practice's details. This however does not suggest that it is entirely accurate and is only an estimate. DrCloudEMR is built by DrCloud Healthcare Solutions Inc, a wholly owned EnSoftek, Inc. subsidiary. [65] There are 2 main postulates for ROI which KOSH’s postulate and Sir Austin Bradford Hill’s criteria for Causation. Kosh’s postulate for CIS is i. The system or feature must be present in every case in which the benefit is observed. ii. The system must be isolated from the organization. iii. The benefit must be reproduced when the system is implemented in a new organization. iv. We must demonstrate that the system was used in the new organization. Hill’s Criteria for Causation includes (a) Strength of Association (b) Consistency of findings (c) Specificity of Association (d) Temporality (e) Dose-response (f) Plausibility (g) Coherence (h) Experimental Evidence and Analogy.

(a) Strength of Association tells us that the greater the change observed, the more likely the association is to be causal (e.g. If a EHR system is implemented and the CPOE feature greatly reduces medication errors, we could say that the implementation of the system had a causal effect on the reduction of medication errors and the strength of association is great).

(b) Consistency of Findings explains that if a change has been observed by different groups in different places with different circumstances and systems, the change is valid, so to speak. For example, if Company A (London, England, UK) implements System A , Company B (Houston, TX, USA) implements System B, and Company C (Guadalajara, Jalisco, Mexico) implements System C, and all three companies reduce medication errors using their respective systems, we can, again say that the CPOE feature of EHR systems can help reduce medication errors. It is important to note that the more consistent findings amongst different groups in different places, the better.

(c) Specificity of Association requires us to ask if there are any other factors which may have affected the change that we've observed. In regards to medication errors being reduced, one would have to ask if CPOE was the only factor involved. If errors could have been reduced due to other mechanisms in place besides CPOE alerts (e.g. better workflow in departments, new policies, etc.), the specificity of association could be considered weak. Weak does not imply wrong, but it does mean that more research has to be initiated.

(d) Temporality addresses the evaluation after an EHR system is implemented. Temporality asks us "were there any changes AFTER the system was implemented?" Usually this is harder to prove due to lack of data prior to EHR implementation, however, Sittig rates temporality as "strong."

(e) Dose-Response asks if the size of changes are directly correlated with the increase of system use (e.g. were medication errors greatly reduced due to the use of many medication alerts in the EHR system?). Usually, there is a strong and direct correlation between system use and the reduction of medication errors, as one example of a dose response in an EHR system.

(f) Plausibility must be shown; There must be some way to demonstrate that the EHR system was used the way it was intended to deliver certain results (e.g. Physicians must have used clinical support decisions the way the EHR system intended to reduce medication errors, in order to demonstrate plausibility.)

(g) Coherence simply states that changes caused by EHR systems should be caused by other EHR systems elsewhere. So, if medication errors are reduced by the use of one EHR system and that happens with the use of many other EHR systems, coherence exists.

(h) Experimental Evidence and Analogy is proving that when the system is not used properly or at all, that certain changes stop. So, if an EHR system is not being used properly or at all (after initial proper use), does a rise in medication errors resume? Experimental evidence is hard to obtain after EHR implementation because it requires not using the system for quite some time (which many would view as wasted money).

ROI Healthcare Project Tools

In the Health Catalyst's Executive Report "How to Drive ROI in Your Healthcare Improvement Projects", a four-step process and tools for ROI analysis were identified. [7]

  • Define the project and the business need
    • State proposal clearly
    • Justify the business need
  • Begin to quantify ROI
    • Identify all costs
    • Estimate benefits
    • Identify direct benefits
    • Identify indirect or intangible benefits and set improvement targets
    • Identify all revenue opportunities
    • Document assumptions
    • Perform a sensitivity analysis
    • Identify risks and alternatives
  • Recruit and train, plan, and implement
  • Evaluate costs, revenue and direct benefits
    • Ensure nothing else has changed
    • Perform Financial ROI calculations
    • Review ROI calculations with the team
    • Make adjustments
    • Monitor and ensure sustained results [7]

DESCRIPTION

Return on Investment (ROI) analysis is a quantitative approach that can be used to build a financial business case for any project. The term means that decision makers evaluate the investment by comparing the magnitude and timing of expected gains to the investment costs. Decision makers will also look for ways to improve ROI by reducing costs, increasing gains, or accelerating gains (1). Furthermore, ROI analysis can be used to make a decision about a future IT investment. For a prospective analysis, estimates of anticipated cost and performance are based on assumptions about the future (1,2). Another ROI approach involves a retrospective analysis that can show actual performance data about the IT project’s costs and returns (1). ROI analysis in general is a diverse collection of methods, skills, tools, activities, and ideas. They all may be useful for assessing the relative value over time of some investment. These methods are not, however, a single formula or predetermined calculation that will yield a simple yes-or-no answer to the question of how to invest. Consequently, a meaningful analysis of returns on investment in information technology is far easier said than done. Choices about how to conduct an ROI analysis should be based understandings about:

  • The strategic objective(s) of the analysis,
  • The place of the proposed IT investment in the overall project
  • How the analysis should be done (i.e., what data and methods of analysis are best suited to those objectives) (1,4).

HISTORY: As far as I can research this is technique has been used as long as costs are involved.

PRINICIPAL USE: Defining and measuring the costs and returns from IT investments. Stated another way: the measurement of the difference between the costs of and benefits from an investment (2).

ADVANTAGES: A methodology to help understand and assess the potential cost/investment needed for a project. While there may shortcomings as stated below, cost analysis must at least be done and ROI can provide a basic infrastructure.

SHORTCOMINGS: Measuring impact Assessing/analyzing scale/scope of project Often pilot projects do not reflect the true cost when the project is expanded on a larger scale Not much of an ROI literature base (3).

EXAMPLES IN INFORMATICS: 1. J Med Syst. 2006 Jun;30(3):159-68. Reviewing the benefits and costs of electronic health records and associated patient safety technologies. Describes the challenges in measuring return on investment (ROI) and reviews published ROI studies on health IT, including EHRs. Concluded that articles examining these benefits are much more common than studies examining ROI itself and additional research utilizing broader perspectives and multidisciplinary techniques will be needed before a better understanding of ROI from health IT is achieved.

2. J Assoc Acad Minor Phys. 2002 Jul;13(3):61-5. Return on investment analysis for a computer-based patient record in the outpatient clinic setting. The high cost of CPR implementation has been a major barrier to widespread acceptance of these systems. Describes a framework to evaluate the costs and benefits of implementing CPR systems in outpatient clinical settings. Return on investment (ROI), a, is one method to evaluate the economic implications of CPR. Supported the idea that understanding the ROI framework will enable physicians to make informed strategic decisions regarding purchase and implementation of CPR systems in their practices.

3. Am J Med. 2003 Apr 1;114(5):397-403. A cost-benefit analysis of electronic medical records in primary care. Electronic medical record systems improve the quality of patient care and decrease medical errors, but their financial effects have not been as well documented. The purpose of this study was to estimate the net financial benefit or cost of implementing electronic medical record systems in primary care. Implementation of an electronic medical record system in primary care can result in a positive financial return on investment to the health care organization. Electronic prescribing software is cost-effective for all size practices with a more rapid return on investment in larger practices.

  1. J Am Med Inform Assoc. 2006 May-Jun;13(3):261-6. 2006 Feb 24.

Return on investment for a computerized physician order entry system. Concluded hospitals may be able to save money and improve patient safety by investing in CPOE systems.

Challenges to ROI

However, calculating ROI is not a simple task. Issues that have hampered ROI studies and affected their validity include:

  • Pressure to justify expense
  • Shoddy collection of "before" comparison data after the implementation
  • Application of multiple simple statistical tests (the more statistical tests you run, the more likely you are to find something significant)

References

  1. 1.0 1.1 Gary R. Heerkens (MBA, CBM, PMP).The Mysteries Of ROI Revealed.http://www.pmi.org/learning/concept-defined-return-on-investments-3861
  2. Medical Practice Efficiencies & Cost Savings http://www.healthit.gov/providers-professionals/medical-practice-efficiencies-cost-savings
  3. The American Journal of Medicine. 2003 http://www.amjmed.com/article/S0002-9343(03)00057-3/fulltext
  4. 2015 survey results: Speaking out about EHRs. http://www.healthcareitnews.com/news/survey-results-speaking-out-about-ehrs
  5. How can a Return on Investment arise from an EHR implementation. http://www.hrsa.gov/healthit/toolbox/Childrenstoolbox/DevelopingPediatricEHRs/roifromehr.html
  6. 6.0 6.1 How to Drive ROI in Your Healthcare Improvement Projects.https://www.healthcatalyst.com/wp-content/uploads/2013/11/Executive-Report-Driving-ROI-final.pdf
  7. 7.0 7.1 B.Brown, L.H.Falk.How to Drive ROI in Your Healthcare Improvement Projects. https://www.healthcatalyst.com/how-to-drive-roi-in-your-healthcare-improvement-projects-html/2/
  1. Cresswell, A. Return on Investment In Information Technology: A Guide for Managers August 2004.
  2. J Assoc Acad Minor Phys. 2002 Jul;13(3):61-5.
  3. J Med Syst. 2006 Jun; 30(3):159-68.
  4. Speaking out about EHRs. http://www.healthcareitnews.com/news/survey-results-speaking-out-about-ehrs
  5. How can a Return on Investment arise from an EHR implementation. http://www.hrsa.gov/healthit/toolbox/Childrenstoolbox/DevelopingPediatricEHRs/roifromehr.html

Amit Shah, MD As001