Towards improved user-product testing with cognitively enhanced

scenariosWilfred van der Vegte

Delft University of Technology

(2011 figures)• Founded in 1842• 17,250 students• 12 undergraduate

programmes• 33 post-graduate

programmes• 2,540 academic staff• 8 faculties (departments)

Faculty ofIndustrial Design Engineering

• Founded in 1969

• Largest university-based design* school worldwide(*product design, consumer durables)

• 1,980 students

• >4,700 alumni (MSc level)

• 3 departments,94 scientific staff

Where I work

Where I work

Section of Computer-Aided Design Engineering• ICT in product design• Historically: CAx• Currently: Cyber-physical systems, ubiquitous

computing, smart products:how to apply technology in products & in product design

• Headed by Prof. dr. mult. Imre Horváth• ~20 people, ~10 permanent scientific staff

Towards improved user-product testing with cognitively enhanced

scenariosWilfred van der Vegte

Typical engineering simulations

Involving humans in product simulations

Real humans(interactive simulation)

Virtual humans(computer models)

computer simulation

product model

computer simulation

product model

human model

human model

perception

decision-making(brain)

action (muscles)

motor control(brain, CNS)

Does not require human

subjects

?

meta

bolis

m

Simulating use with virtual humans

virtual human

perception

decision-making(brain)

action (muscles)

motor control(brain, CNS)

do we have to simulate

all of this?No ...let’s

simplify

scenario bundle(by designer)

approximate

approximate

already solvedby others

meta

bolis

m

The scenario bundle

• Designer’s conjecture of human actions /decision-making

• Graphical notation

• Possible paths through use processconnecting interactions

• Contains multiple options(courses of use process)

• Controls a physics simulation of the product

• Is specified as an automaton(←in this case a statechart)

hold retract align

reach

reorient

release

hesitate

place

clasp

move

join

unwrap

enter

wait

hesitate

switch off

switch on

reach

push

drop

lift

catch

drop

throw

guide

break

locate

grasp

reach

pull

linger

hold

close

loosenreach

carry

retrieve

wait

carrylift

mix

pull

followenter

reduce

unfold

open

insert

increase

tighten

hold

hold

push

adjust rotate

Example: snack dispenser

sim

ula

tion

scen

ari

o

bu

nd

le

pro

gra

mm

ing

of

pro

du

ct

Advantages

• Use process of the product can be tested if only a computer model is available

• No human subjects needed

• Designer can do what-if studies

• One scenario, multiple varieties of the product design

• Varieties of how the product is programmed

• Varieties in human behaviour, e.g., hesitation

• Varieties of scenario: rearrange blocks & arrows

• Building blocks of scenarios are reusable in other design projects

Key limitations

• Simulation of physics computationally demanding and not (yet) very stable

• Algorithms for realistic simulation of low-level human motor control exist, but have not been included

• Lack of realism in human information processing (thinking, reasoning, ...): only what the designer preconceives is included

But ... Are animations of physics and

human motor behavior always needed?

No! Often only the course of values of specific variables over time is important!(power consumption, temperature, speed, ...)

Demonstrative example: espresso machine with power-save function for water heating

Investigation of• boiler temperature• energy consumptionover a typical use process in which several coffee-making sessions take place, over several hours / days / weeks /...with given settings for power saving (reduce power by x% after non-use over Δt = t1 and by 100% after another Δt = t2)

Make_one_cup/ size=choosefrom(2)

Push_Buttonwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A Release_Button

entry:button=none

Wait_for_coffee

after (wait,sec)

[pump_sound== 1]

[pump_sound== 0]

[size== 1]

Remove_cup

Building a scenario bundle for using the espresso machine

Make_one_cup/ , intervention=choosefrom(3)size=choosefrom(2)

Push_Buttonwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A Release_Button

entry:button=none

Wait_to_halt_serving

Espr/desired_volume= arbitrary(55 ,1 , 200 , 17.5 )

Lungo/desired_volume= arbitrary ( 128 ,1, 250 , 30 )Wait_for_coffee

Push_Button_againwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A

Release_Button_again/button=none

after (wait,sec)

[pump_sound== 1]

[intervention== 2]

[pump_sound== 0][perceived_cup_vol>desired_volume||perceived_res_content== 0]

after (wait,sec) [pump_sound== 0]

[size== 1]

[size== 1 ]

[size== 1]

Remove_cup

Building a scenario bundle for using the espresso machine

Make_one_cup/ , intervention=choosefrom(3)size=choosefrom(2)

Keep_pressedEspr/desired_volume= arbitrary (55 ,1, 200 , 17.5 )

Lungo/desired_volume= arbitrary( 128 ,1, 250 ,30 )

Push_Buttonwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A Release_Button

entry:button=none

Wait_to_halt_serving

Espr/desired_volume= arbitrary(55 ,1 , 200 , 17.5 )

Lungo/desired_volume= arbitrary ( 128 ,1, 250 , 30 )Wait_for_coffee Release1_Button

entry:button=none

Push_Button_againwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A

Release_Button_again/button=none

after (wait,sec) [intervention== 3]

[perceived_cup_vol>desired_volume||perceived_res_content== 0]

[pump_sound== 1]

[intervention== 2]

[pump_sound== 0][perceived_cup_vol>desired_volume||perceived_res_content== 0]

[pump_sound== 0 ]

after (wait,sec) [pump_sound== 0]

[size== 1 ]

[size== 1]

[size== 1 ]

[size== 1]

Remove_cup

Building a scenario bundle for using the espresso machine

Make_one_cup/ , intervention=choosefrom(3)size=choosefrom(2)

Keep_pressedEspr/desired_volume= arbitrary (55 ,1, 200 , 17.5 )

Lungo/desired_volume= arbitrary( 128 ,1, 250 ,30 )

Push_Buttonwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A Release_Button

entry:button=none

Wait_to_halt_serving

Espr/desired_volume= arbitrary(55 ,1 , 200 , 17.5 )

Lungo/desired_volume= arbitrary ( 128 ,1, 250 , 30 )Wait_for_coffee Release1_Button

entry:button=none

Push_Button_againwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A

Release_Button_again/button=none

after (wait,sec)[pump_sound== 0 ]

{wait= 0.1 * arbitrary(2,1,4,1)}

after (wait,sec) [intervention== 3]

[perceived_cup_vol>desired_volume||perceived_res_content== 0]

after (wait,sec)

[pump_sound== 0]{wait= 0.1 * arbitrary (2 ,1 ,4 ,1 )} [pump_sound== 1]

[intervention== 2]

[perceived_res_content== 0] [pump_sound== 0]

[pump_sound== 0][perceived_cup_vol>desired_volume||perceived_res_content== 0]

[pump_sound== 0 ]

after (wait,sec) [pump_sound== 0]

[size== 1 ]

[size== 1]

[size== 1 ]

[size== 1]

Remove_cup

Building a scenario bundle for using the espresso machine

Make_n_tot_cupsentry:n_tot= choosefrom (5 ),n_cups= 0

Eject_capsulecapsule_inserted= 0Insert_capsule_and_place_cup

capsule_inserted= 1busy= arbitrary (4 ,3 ,6 ,2 )

Refill_reservoir/supply= arbitrary ( 800 , 10 , 1000 ,500 )just_filled= 1 ,busy= arbitrary (10 ,6, 20 ,6)

Make_one_cup/ , intervention=choosefrom(3)size=choosefrom(2) , just_filled=0

Keep_pressedEspr/desired_volume= arbitrary (55 ,1, 200 , 17.5 )

Lungo/desired_volume= arbitrary( 128 ,1, 250 ,30 )

Push_Buttonwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A Release_Button

entry:button=none

Wait_to_halt_serving

Espr/desired_volume= arbitrary(55 ,1 , 200 , 17.5 )

Lungo/desired_volume= arbitrary ( 128 ,1, 250 , 30 )Wait_for_coffee Release1_Button

entry:button=none

Push_Button_againwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A

Release_Button_again/button=none

Remove_cup/n_cups++

[perceived_res_content< 0.1 ]

after(5 ,sec)

after (busy, sec)[capsule_inserted== 1]

after (busy,sec)

after (wait,sec)[pump_sound== 0 ]

{wait= 0.1 * arbitrary(2,1,4,1)}

after (wait,sec) [intervention== 3]

[perceived_cup_vol>desired_volume||perceived_res_content== 0]

after (wait,sec)

[pump_sound== 0]{wait= 0.1 * arbitrary (2 ,1 ,4 ,1 )} [pump_sound== 1]

[intervention== 2]

[perceived_res_content== 0] [pump_sound== 0]

[pump_sound== 0][perceived_cup_vol>desired_volume||perceived_res_content== 0]

[pump_sound== 0 ]

after (wait,sec) [pump_sound== 0]

[size== 1 ]

[size== 1]

[size== 1 ]

[size== 1]

Remove_cup

Building a scenario bundle for using the espresso machine

User

Models_and_specifications

User User_cupboard

Espresso_machine Supplier

Espresso_machine_logic Supplier_processing

Courier_service

Espresso_machine_physics

PumpShipping

Boiler Cup_filling

Prepare_orders

Track_consumption_remotelyBoiler_thermostatCoffee_serving

Reservoir

Boiler

Detect_power_change

Compute_T_water

Compute_input_time

heating_or_coolingduring:input_time=t-t1exit:input_time=0

compute_P_change/P_change=P_heat-P_prevT_last=T_water

store_previous_value/P_prev=P_heatP_change=0

no_heating_or_cooling

Exponentialduring: T_water=T_amb+P_heat*k_1...-(T_amb+P_heat*k_1-T_last)*ml.exp(-(k_3/m_water)*input_time)

T_water = T_amb + P_heat*k1 - k2*exp[-(k3/m_water)*t];k_2 = 1-(T_amb_P_heat*k_1-T_last)

Constant_and_cold

[P_change>0]/t1=t/T_water=T_ambT_last=T_amb

[hasChanged(P_heat)==1]

[P_change<0]/t1=t

[P_change>0]/t1=t

[P_heat==0&&...T_water-T_amb<0.5]/T_water=T_amb

[P_heat>0]

Pump 2

ON/P_pump=59

OFF/discharge=0,P_pump=0,empty_sound=0

Run 2

detect_capsule 1absent/flow=5.86

present/flow=4.29

build_up_pressure/ deliver/discharge=flow

run_dry/discharge=0,empty_sound=1

[pump==1] [pump==0]

[capsule_inserted==1]

[capsule_inserted==0]

[capsule_inserted==0]

after(2.5,sec)

[reservoir_content<5]

To compensate for the missing 3D physics simulation, we have now used a hybrid automaton (discrete+continuous)

In addition, the statechart has become a timed hybrid automaton (THA) to efficiently deal with timing (latency, delays, time-outs,...)

User

Scenario_network

Go_through_m_tot_sessions_of_coffee_makingentry:m_tot= choosefrom (5 ),m_sessions= 0

Fill_reservoir/supply= arbitrary ( 900 , 10 , 1000 , 650 )just_filled= 1

Wait_for_light

Make_n_tot_cupsentry:n_tot= choosefrom (5 ),n_cups= 0

Eject_capsulecapsule_inserted= 0Insert_capsule_and_place_cup

capsule_inserted= 1busy= arbitrary (4 ,3 ,6 ,2 )

Refill_reservoir/supply= arbitrary ( 800 , 10 , 1000 ,500 )just_filled= 1 ,busy= arbitrary (10 ,6, 20 ,6)

Make_one_cup/ , intervention=choosefrom(3)size=choosefrom(2) , just_filled=0

Keep_pressedEspr/desired_volume= arbitrary (55 ,1, 200 , 17.5 )

Lungo/desired_volume= arbitrary( 128 ,1, 250 ,30 )

Push_Buttonwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A Release_Button

entry:button=none

Wait_to_halt_serving

Espr/desired_volume= arbitrary(55 ,1 ,200 , 17.5 )

Lungo/desired_volume= arbitrary ( 128 ,1, 250 , 30 )Wait_for_coffee Release1_Button

entry:button=none

Push_Button_againwait= 0.1 * arbitrary (2,1,4,1)

Lungo/button=B

Espr/button=A

Release_Button_again/button=none

Remove_cup/n_cups++

Take_short_break

Break/wait= arbitrary( 100 , 30 , 200 ,200 )go_further= choosefrom (3)

Release_Button_Aentry:button=none

Push_Button_A/button=ACheckLight

Empty_reservoir_and_take_long_break/supply= 0 ,just_filled= 0 ,switch_status= 0wait= arbitrary ( 200 ,100 , 1000 , 300 )

[m_sessions==m_tot]after (wait,sec)

after (0.1 ,sec)/switch_status= 1

[blinking== 0]

[switch_status== 1 ]

[blinking== 0]

[n_cups==n_tot]/m_sessions++

[blinking== 0]

[perceived_res_content< 0.1 ]

after(5 ,sec)

after (busy, sec)[capsule_inserted== 1]

after (busy,sec)

after (wait,sec)[pump_sound== 0 ]

{wait= 0.1 * arbitrary(2,1,4,1)}

after (wait,sec) [intervention== 3]

[perceived_cup_vol>desired_volume||perceived_res_content== 0]

after (wait,sec)

[pump_sound== 0]{wait= 0.1 * arbitrary (2 ,1 ,4 ,1 )} [pump_sound== 1]

[intervention== 2]

[perceived_res_content== 0] [pump_sound== 0]

[pump_sound== 0][perceived_cup_vol>desired_volume||perceived_res_content== 0]

[pump_sound== 0 ]

after (wait,sec) [pump_sound== 0]

[size== 1 ]

[size== 1]

[size== 1 ]

[size== 1]

after ( 0.2 ,sec)[go_further> 1]after (wait,sec) [blinking== 1]

Remove_cup

User

Energy consumption (kWh)

Reservoir content (ml)

Boiler water temperature (K)

Power- save mode

Power- save mode

Serving lungos

Serving espressos

User takes break

User takes break

Thermostatcontrols temperature

Thermostat controls temperature

User switches machine off

Time (s)

User refills reservoir

These simulation outcomes can be

generated up to 5000 faster than real time

Revisiting the limitations

• Simulation of physics has been simplified (no more 3D), and is fast and reliable

• Low-level human motor control is disregarded, still the whole use process can be simulated (in this case)

• Still lack of realism in human information processing (thinking, reasoning, ...): only what the designer preconceives is included

So ... how to increase realism in human information processing (thinking, reasoning, ...)?Aspects of human information processing to be simulated

1. Logic of decision making:under which condition what action is taken?e.g. “IF cup is full THEN retrieve cup from machine”:straightforward execution of ‘normal’ use, assuming a particular history of preceding events.

• But can a simulation predict a user acting according tothe production rule“IF cup is full THEN stick finger in it”? →unlikely!

So ... how to increase realism in human information processing (thinking, reasoning, ...)?Aspects of human information processing to be simulated

1. Logic of decision making:under which condition what action is taken?e.g. “IF cup is full THEN retrieve cup from machine”:straightforward execution of ‘normal’ use (instructions), assuming a particular history of events, including required preceding actions.

• We can howevertry to generatetypical aberrationsfrom ‘regular use’:the so-callederror phenotypes

taxonomy accordingto Hollnagel→

action in wrong place

action at wrong time

action of wrong type

action not included in current plans

repetition

reversal

omission

delay

premature action

replacement

insertion

intrusion

restart

jumping

undershoot

sidetracking

capture

branching

overshoot

error mode simple phenotype( applies to one action)

complex phenotype( applies to multiple

interconnected actions)

So ... how to increase realism in human information processing (thinking, reasoning, ...)?Aspects of human information processing to be

simulated

2. Processing time:How long does it take to accomplisha given action, taking into accountaspects such as memory retrieval,memory capacity, learning,multitasking, distraction, etc.

• These aspects can be simulatedusing cognitive architecturessuch as ACT-R

• A cognitive architecture is

• a blueprint of the human mind• based on findings from brain science• filled with psychologically validated

task models expressed as production rules

central production system

retrieval buffer

declarative module goal buffer

intentional module

external world

visualbuffer

motorbuffer

visualmodule

motormodule

(basal ganglia )(ventrolateral

prefrontal cortex )

(temporal cortex / hippocampus)

(dorsolateral prefrontal cortex)

(not identified)

(parietal cortex)

(motor cortex)

(occipital cortex)

(motor cortex / cerebel-

lum)

central production

systemretrieval buffer

declarative module goal buffer

intentional module

external world

visual

buffer

motor

buffer

visual

module

motor

module

How to realize simulations with cognitively enhanced scenarios (CES)

central

produc -

tion

system

re -

trieval

buffer

declar -

ative

module

goal

buffer

inte

nti

on

al

mo

du

le

ex

tern

al

wo

rld

visual

buffer

motor

buffer

visual

module

motor

module

inte

nti

on

al

mo

du

le

ex

tern

al

wo

rld

How to realize simulations with cognitively enhanced scenarios (CES)

central

produc -

tion

system

re -

trieval

buffer

declar -

ative

module

goal

buffer

visual

buffer

motor

buffer

visual

module

motor

module

Scenario bundle (THA)(human actions)

THA of the ‘world’(product, environment)

Cognitive Architecture (ACT-R)

action in wrong place

action at wrong time

action of wrong type

action not included in current plans

repetition

reversal

omission

delay

premature action

replacement

insertion

intrusion

restart

jumping

undershoot

sidetracking

capture

branching

overshoot

error mode simple phenotype(applies to one action)

complex phenotype(applies to multiple

interconnected actions)Human-ErrorPhenotype Generator

Operation of ship locks –our first real-life application?

usually combined with a movable bridgevarious types of boats and skippers

gates, traffic lights, leveling, etc., are allman-controlled

usually 2-3 chambers

operator works remotely, usingmultiple monitors

Operation of ship locks –our first real-life application?

The Dutch Government Agency of Public Works & Water Management is developing a new system of centralized control rooms for locks, movable bridges and other objects With CES simulations the

system can be pre-tested• without human subjects• batchwise, systematically

varying parameters (weather, traffic density, ...)

• to reveal incidents theoretically happening once in 10/100/1,000 years

• that cannot be discovered through interactive testing

Potential problems:• Multitasking, but also

boredom, can cause cognitive overload, cognitive lock-up& other error-invoking mental phenomena

• Errors can cause injury, collisions or even flooding

Envisioned setup for first application

central

produc -

tion

system

re -

trieval

buffer

declar -

ative

module

goal

buffer

visual

buffer

motor

buffer

visual

module

motor

module

Scenario bundle (THA)(human actions)

THA of the ‘world’(product, environment)

Cognitive Architecture (ACT-R)

action in wrong place

action at wrong time

action of wrong type

action not included in current plans

repetition

reversal

omission

delay

premature action

replacement

insertion

intrusion

restart

jumping

undershoot

sidetracking

capture

branching

overshoot

error mode simple phenotype(applies to one action)

complex phenotype(applies to multiple

interconnected actions)Human-ErrorPhenotype Generator

Conclusions

Simulation with cognitively enhanced scenarios (CES)

• can help evaluating human-system interactions where many variations can influence the outcomes

• based on psychologically validated knowledge on the workings of the human brain

• benefits from simplifications & shortcuts:simulation speed is determined by slowest element in simulation loop → avoid complex models & algorithms