Assessing performance of an Electronic Health Record (EHR) using Cognitive Task Analysis

From Clinfowiki
Jump to: navigation, search

This is a review of a paper submitted by Himali Saitwal, Xuan Feng, Muhammad Walji, Vimla Patel and Jiajie Zhang. [1]

Research question:

Can Task Analysis methods effectively evaluate the usability of an Electronic Health Record (EHR) system and suggest areas of improvement in the user interface?


This study evaluated the usability of an Electronic Health Record (EHR) military user interface for AHLTA (Armed Forces Health Longitudinal Technology Application), using the Cognitive Task Analysis (CTA) method called GOMS (Goals, Operators, Methods, and Selection rules) and an associated technique called KLM (Keystroke Level Model). The results showed, on the average, a user needed 106 steps to complete a task. To perform14 tasks, they needed about 22 min (independent of system response time) for data entry, of which 11 min are spent on more effortful mental operators.[1]


According to the authors, “Many health information system projects fail due to lack of systematic consideration of human-centered computing issues such as usability, workflow, organizational change, and process reengineering. This paper evaluates the usability of the AHLTA (Armed forces Health Longitudinal Technology Application) EHR (Electronic Health Record) system that is used by the US military which connects 412 medical clinics, 414 dental clinics and 65 military hospitals. According to Dr. Casscells, assistant secretary of defense for health affairs, AHLTA is difficult to learn, and once you have learned it, it is cumbersome and difficult to navigate. Clinician users of AHLTA also report seeing fewer patients and having longer workdays, largely because of the extra time needed to use the system. Two major factors that lead to sluggish performance of this EHR system are complexity of the GUI (Graphical User Interface) and system response time. This paper analyzes the complexity of the GUI of AHLTA independent of its system response time.” [1]


The overall approach used to evaluate the user interface of AHLTA was to conduct a GOMS (Goals, Operators, Methods, and Selection rules) analysis that identified all the sub-tasks of a given task and classify them into mental or physical operators. GOMS was performed on a set of 14 common tasks that were identified by AHLTA users. “Inter-rater reliability was then calculated to determine agreement between two evaluators who independently conducted GOMS analysis for each task. Finally, one associated GOMS analysis technique – KLM – was used to predict the execution time required to perform each given task.” [1]


“The total number of steps for a given task ranged from a minimum of 41 for “Review Coding of Medical Encounter” to a maximum of 466 for “entering vital signs”. The mean number of steps for a task was 106.” [1] These values reflect the time a user has to spend interacting with AHLTA and does not include the time in examining a patient. Further, these estimates are based on the assumption that the users are expert computer users. Execution time for the 14 tasks ranged from a low of 35 s to high of 6.5 min. [1]


This paper identified the following factors that limited the performance of AHLTA: (1) a large number of steps are needed to complete a task; (2) a long execution time by the user is required and (3) a high percentage of mental workload by its’ operators makes it tiring to operate. The inter-rater reliability analysis showed that the method used in this study can be reliably used for evaluating the usability of the health information systems. The results show that for a given patient, if the physician has to perform all necessary (14 in all) tasks, it takes around 22 min, which makes the system ineffective for users. [1]


The results from the Cognitive Task Analysis methods can effectively evaluate the usability of an Electronic Health Record (EHR) system and suggest areas of improvement in the user interface up to a point. The end user is the ultimate judge of the EHR design. There are many roles a user plays. A military physician does not normally take a patient’s vital signs. It is normally conducted a nurse or medical assistant but a EHR system that takes 466 steps to enter vitals is a bit much in my opinion. The trouble I saw with clinicians using AHLTA was that it kept crashing. This required the clinicians to record the medical record data on paper and re-input the data into AHLTA when the system was up and running again. I know a patient does not want to be bothered while a physician takes 22 minutes of point of entry time just to input data. Between the medical clerks, nurses/medical assistants, physicians, other clinicians, medical technicians, and medical administrators, it takes a team effort to evaluate and treat a patient.


  1. 1.0 1.1 1.2 1.3 1.4 1.5 1.6 Saitwal, H., Feng, X., Walji, M., Patel, V., Zhang, J. (2010), Assessing performance of an Electronic Health Record (EHR) using Cognitive Task Analysis International Journal of Medical Informatics. 2010 Jul;79(7):501-6. doi: 10.1016/j.ijmedinf.2010.04.001. Epub 2010 May 7. (Accessed on 23 Oct 2015)