Evaluation and Design Methodologies

Evaluation Methods in Informatics

This is a collection of short descriptions of evaluation methods used in informatics research.

Ethnography

Cognitive ethnography is one of the tools which could be used to study effective manners for instruction of BMI in professional schools. A study to delineate the present state of informatics curricula at healthcare professional schools, the present status of BMI knowledge held by students, at the beginning and completion of their healthcare professional education, and a systematic analysis of what should be included in a standardized, structured BMI curriculum to meet the Liaison Committee on Medical Education (LCME) standards would benefit from the use of cognitive ethnography.

Critical Incident Technique

Critical Incident Technique (CIT) is a form of criterion sampling. [Patton, 2002] CIT relies upon interviews of subjects who fall into a defined category or categories and who carry out defined work tasks. Recurring interview topics are recorded as incidents, and the incidents are analyzed so to develop a psychological profile of the subjects. The goal of CIT research is to understand the weaknesses involved with a particular task and to provide solutions to resolve those weaknesses. [Wikipedia, 2007]

Ethnomethodology

Ethnomethodology is a sociological discipline which focuses on the ways in which people make sense of their world, display this understanding to others, and produce the mutually shared social order in which they live. It is distinct from traditional sociology, and does not seek to compete with it, or provide remedies for any of its practices. Furthermore, ethnomethodology is concerned with the"how" (the methods) by which that social order is produced, and shared (1,2). It seeks to describe the practices (the methods) these individuals use in their actual descriptions of those settings. Ethnomethology may ask, how do people perceive understand and explain the world in which they live? Specifically, with patient care, it may ask, are these perceptions and beliefs about the world changed when we become ill? Ethnomethodology is particularly concerned with cultural differences in explanations offered, especially the influence of social norms on the communication process. This cultural context is particularly significant when a patient tries to understand or the doctor tries to explain a condition, for example, a disease or newly diagnosed cancer.

Protocol Analysis

Protocol Analysis uses verbal descriptions of though processes and tasks to generate data about a given scenario or cognitive act. The primary technique used in protocol analysis is to ask subjects to "think-aloud" while performing a task. These verbal descriptions provide a set of explicit procedures that more thoroughly describe the solution to a task than other introspective techniques. According to K.A. Ericsson, "the central assumption of protocol analysis is that it possible to instruct subjects to verbalize their thoughts in a manner that doesn’t alter the sequence of thoughts mediating the completion of a task, and can therefore be accepted as valid data on thinking. Based on their theoretical analysis, Ericsson and Simon (1993) argued that the closest connection between thinking and verbal reports is found when subjects verbalize thoughts generated during task completion."