Protagonists: Henry Ford, Nikola Tesla $...

transcript

11-‐04-‐18

Cognitive Task Analysis Carolina Wannheden, Doctoral student Health Informatics Center, LIME

carolina.wannheden@ki.se

”The expert’s bill”

Protagonists: Henry Ford, Nikola Tesla

$ 10,000

Marking wall: $ 1 Knowing where to mark: $ 9,999

2011-04-18 carolina.wannheden@ki.se

11-‐04-‐18

Discuss in pairs…

1. What is it that Tesla knows, and how does he know it?

2. How can we grasp this knowledge?

3. Why would we want to grasp this knowledge?

Learning Outcomes

You should be able to

Explain when and why to perform a Cognitive Task Analysis Explain what distinguishes an expert and when knowledge-based

systems may be useful Explain different knowledge structures Apply the Critical Decision Method for the aquisition of expert

knowledge to inform the design of a CDSS

11-‐04-‐18

The purpose of CTA…

…is to understand how people think and what they know (cognitive) to achieve some particular goal (task)…

…in order to Analyze incidents Develop training material Develop memory aids, decision aids, expert systems

When to use knowledge-based systems

When real experts are

Scarce Expensive Inconsistent Unavailable on a routine basis

11-‐04-‐18

Case from week 1

Read the case and reflect on…

1. What makes the anesthesiologist’s decision challenging?

2. Why does the anesthesiologist choose the general anesthesia?

3. How would you describe the context? Which factors affect her decision-making?

4. What type of knowledge does the anesthesiologist need in order to make the right decision? Can you distinguish different types of knowledge?

11-‐04-‐18

Experts know how, not just what!

Conceptual/factual knowledge (what) General anesthesia is for… Regional anesthesia is for…

Procedural knowledge (how) IF X, then provide the appropriate anesthesia and operate.

(Structural knowledge Guideline A is applicable for patients aged 15 or more)

11-‐04-‐18

Key aspects of CTA

1. Knowledge acquisition (KA) 2. Data analysis 3. Knowledge representation

What CTA tries to capture

What people are thinking about What they are paying attention to The strategies they are using in making decisions What they are trying to accomplish What information they discard What they know about the way a process works

(Crandall, Klein, Hoffman, 2006)

11-‐04-‐18

Techniques for knowledge acquisition

Interviews + Require a minimum level of resources + Can be performed in a relatively short time frame + Can yield a significant amount of qualitative knowledge - Lack of quantitative data - Bias due to selection of questions by researcher - Elicited knowledge may not correspond to what expert actually

does Think-aloud protocols Observations

Ethnographic evaluations to collect information in context Group techniques (e.g. brainstorming)

Data analysis methods

Protocol and discourse analysis Elicit knowledge from individuals while they are engaged in

problem-solving or reasoning tasks Determine conceptual entities and relationships between them

Concept mapping Node-link structures of knowledge Concept maps can support the formation of consensus among

experts Verification and validation

Verification: fulfillment of perceived requirements (to define design) Validation: fulfillment of realized requirements (upon

implementation)

11-‐04-‐18

Knowledge Representation

Narrative formats Chronologies Data organizers Process diagrams Concept maps

Decision Requirements Table

Treatment phase

Decision challenge

Cue/ Information

Strategy or practice

Novice Traps

Giving anesthesia before surgery

Choose adequate type of anesthesia

Age, medical history

11-‐04-‐18

Challenges in acquiring expert knowledge

Complex and resource-intensive Identification and access to domain expert with

Sufficient domain knowledge Interest in participating in knowledge acquisition process Minimal bias

(Greenes, 2006)

Reasoning biases

Poor estimation of probabilities (Probability bias) Use terms like ”suggests”, ”supports”, ”goes against”, ”often”,

”evokes the possibility” to describes uncertainty

Estimation bias Recency bias (mistaking for frequeny) Anchor judgments on initial estimates Familiarity or stereotypic frequency over objective frequency Overestimate frequency of rare events

(Greenes, 2006)

11-‐04-‐18

Critical Decision Method (CDM)

CDM was created to learn from specific incidents (Hoffman et al., 1998)

Described well in chapter 5, Working Minds: A Practitioner's Guide to Cognitive Task Analysis (Crandall, Klein, Hoffman, 2006)

The CDM interview

Intensive in-depth interview (duration ~2 hrs) to elicit cognitive functions such as decision making, planning, sensemaking within a specific challenging incident

Conducted by 2 researchers 1. Primary facilitator (and note-taker) 2. Note-taker and time-keeper

Conducted in 4 sweeps (phases) 1. Incident identification 2. Constructing a timeline 3. Deepening 4. ”What if” queries

11-‐04-‐18

Sweep 1: Incident identification

Goal: Try to identify an incident that will contain cognitive components beyond background and routine procedural knowledge Nonroutine, challenging events The participant has to have a role as a ”doer/decision maker” The participant’s decision making should have had a direct impact

on the outcome Critical event, time pressure

Ask the participant to provide a brief account of the story, from beginning to end

Sweep 2: Constructing a timeline

Goal: Get a clear, refined, and verified overview of the incident structure, identifying key events and segments.

The interviewer diagrams the sequence of events on a timeline Identify critical points/”decision points” Try to note sequence and duration of events, actions, perceptions,

thoughts, decisions

11-‐04-‐18

Sweep 3: Deepening

Goal: Get inside the expert’s head: ”[W]hat did they know, when did they know it, how did they know, and what did they do with what they knew?” (Crandall et al., 2006)

Based on the timeline probe critical points for the participant’s Perceptions Expectations Goals Judgments Confusions, uncertainties, concerns Options Information needed and used

Sweep 4: ”What if” queries

Goal: Illuminate expert-novice differences and potential vulnerabilities for error in the domain

The interviewer poses hypotheticals about the event What if a novice had been at charge? What if [key feature] had been different? What training might have been an advantage? What knowledge, information, tools/technologies could have

helped?

11-‐04-‐18

Discuss in your groups…

Would the Critical Decision Method be an appropriate method to elicit knowledge for the CDSS you intend to develop?

Would it be a feasible method?

A final quote

”For now, […], it is the direct interaction among experts, and between experts and knowledge engineers, that will serve a crucial role in assuring the development of high quality and accepted knowledge bases that in turn enable the development and effective use of decision support systems.” (Greenes, 2006)

11-‐04-‐18

Protagonists: Henry Ford, Nikola Tesla $...

Documents