LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
GaLAGame and Learning Alliance
The European Network of Excellence on Serious Games
WP3 – SG Application Fields
Michela Mortara, Chiara Eva Catalano, Bianca Falcidieno
CNR – IMATIMarco Luccini CEDEP
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
Outline
• Overview of WP3– mission and objectives
• Synthesis of Year 1 – activities and outcomes
• Year 2– Focus, strategy, outcomes
– Plan for user studies
• Plan for year 3 (4)
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
Overview of WP3
• Mission: Understand the current practices, needs and desiderata of customers and developers of SGs in different application domains
• 6 specific Tasks, or Special Interest Groups
– Business and Management (T3.1, CEDEP)– Engineering and Manufacturing (T3.2, POLIMI)– Health and Fitness (T3.3, HWU)– Security, Safety & Crisis Management (T3.4, NURC)– Humanities & Heritage (T3.5, CNR)– Personal & Social Learning & Ethics (T3.6, INESC-ID)
• 1 Harmonization & Integration Task (T3.7, CEDEP)
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
Synthesis of Year 1
Activity: surveying…
• SG proposition and diffusion dynamics• Dissemination channels• Key stakeholders• Barriers to the adoption • Requirements• Open issues
Outcomes:
• Collection of SGs in the application domains• Best practices in adoption/development• Recommendations for maximizing SG diffusion
STAR
questionnaires, interviews
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
Main findings of year 1
• High demand of evidence based practice
• Lack of a formal methodology to assessgame effectiveness perceived as the mainbarrier to SG adoption
• Scepticism about SG effectiveness
• Low adoption, small market
• Uncertain return of investment
• Few skilled developers, low budget projects
Scepticism about SG
effectiveness
Low adoption,
small market
Uncertain return of
investment
Few developers, low budget
projects
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
Focus of Year 2
“The panel recommends that this WP develop a plan to design and conduct one or more empirical studies of SGs now in use which are generally regarded as effective but which lack empirical evidence of impact. Such a research plan could be developed in year two and conducted in year three”
Based on the 1st year outcomes and the reviewers’ recommendation, the focus of year 2 has been on the learning impact of SGs.
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
Strategy
SG descrip
tion
AS – game
testing and evaluation
London – game
workshops
SG collection (year 1) SG monitoring (year 2)
• Game workshops
• Game testing and evaluation
• Collaborative SG descriptions
• Educational objectives & exploitation styles
• Technological features
• Availability to test
• Success– Diffusion / Popularity
– Awards
• Learning Impact– Evidence from the literature
– Recognised potential
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
Strategy
SG descrip
tion
AS – game
testing and evaluation
London – game
workshops
SG collection (year 1) SG monitoring (year 2)
• Game workshops
• Game evaluation and testing
• Collaborative SG descriptions
For each SIG:
• Best practices wrt maximizing the learning impact
• Current/best practices in the assessment methodology
• Identification of the SGs for user studies
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
BPs and recommendations on how to maximize and evaluate the Learning Impact [T3.7]
• Minimum cognitive overload
• Facilitate the learning
Minimizing the cognitive overload
• R1• Engage the learner
experientially/constructively• Situate the learning
Maximizing the learning experience
• Clear Evaluation strategies
• Comprehensive evaluation plan
Designing learning evaluation as an Impact project
• R3
• Flexibility, reusability, exploitation
• Facilitate the learning
Design the SG as an integral part of a learning program
R1 R2
R3 R4
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
BPs and recommendations on how to maximize and evaluate the Learning Impact [T3.7]
• Minimum cognitive overload
• Facilitate the learning
Minimizing the cognitive overload
R1
• User friendly game interface & Intuitive navigation (Playing History, Plantville, Siqur)
• No previous knowledge required, gradually increasing difficult learning tasks (Shortfall, Thiatro)
• Use facilitation when the embedded complexity is relevant
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
BPs and recommendations on how to maximize and evaluate the Learning Impact [T3.7]
• R1• Situate the learning• Engage the learner
experientially/constructively
Maximizing the learning experience
R2
• Minimization of cognitive overload a necessary condition• A suitable environment and proper interactions to fit the context of use
at best• Immersive 3D environments (Thiatro, Boarders’ Ahoy!, TARGET,
Innov8)• Interaction among players (EagleRacing)
• Engage learners in building their new knowledge (Icura, MeTycoon)• Fun but also emotional tension and frustration
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
BPs and recommendations on how to maximize and evaluate the Learning Impact [T3.7]
• Clear Evaluation strategies
• Comprehensive evaluation plan
Designing learning evaluation as an Impact project
• R3
• Facilitate the learning
Design the SG as an integral part of a learning program
R3 R4
• Identify as early as possible the learning goals and the kinds of data to be collected in order to identify the suitable learning metrics.
• If the game is part of an educational strategy the stakeholders will be involved for the design and evaluation phase
• Facilitation to raise attention, provide learning guidance and consolidate the learning
• Maximization of the learning impact and effective evaluation
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
Plan for user studies – objectives
Learning theories and
models
best practice
s & recommendation
s
Plan for user
studies• Selected SGs generally regarded as
effective
• Go beyond available studies (if any)
• Validate the GALA evaluation (description template -WP1)
• Evaluate SG effectiveness per se / with respect to or coupled with other means
• Assess long term retention / actionability
• Not only cognitive gain, higher levels in Bloom’s / Kirkpatrick’s models
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
User studies
• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural
awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
User studies
• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural
awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)
Is part of some company specific programs and other education programs at CEDEP and at
INSEAD
The evaluation will benefit of a PGLC, never done before
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
User studies
• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural
awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)
Developed and adopted at POLIMI
Summative evaluation including scenario questions
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
User studies
• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural
awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)
Largely adopted and well studied
BUT will be evaluated from the point of view of game mechanics and narrative
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
User studies
• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural
awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)
Is part of education programs at CEDEP and at INSEAD and at some company specific programs.The evaluation will benefit of a PGLC, never done
before
Both considered BP for CA in defence operations – different technological solutions
Planned a comparative study on different user profiles
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
User studies
• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural
awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)
Largely adopted in Danish schools.
Evaluation at DIS (Genova) in march, to be repeated over time
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
User studies
• EagleRacing (collaboration dynamics)• MetaVals (finance)• Seconds (supply chain management)• Cosiga (product development process)• SBCE (lean manufacturing)• Re-Mission (support to treatment)• Squire’s Quest (food and nutrition)• Afghanistan pre-deployment course (defence, cultural
awareness) • Boarders Ahoy! (defence, ship boarding operations)• Playing History – The Plague (History)• ICURA (cultural awareness, japanese etiquette)• Stop Disaster! (risk awareness)• Darfur is Dying (human rights)
Large diffusion (supported by U.N.)
positive perception of learning impact but no empirical studies yet
Evaluation of experimental vs control group
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
Further SIG activities (WP3 Handbook)
• Set-up and maintain dedicated SIG web pages (as suggested by the reviewers) [joint effort with WP8]
• Contribute to SG description
• Publish scientific papers [16 ref. proc, 2 ref. journals] , organize Special Issues [1] and Workshops [3]
• Attract Associate Partners
• Monitoring SG proposition, stakeholders, dissemination channels
LuxembourgJanuary 28, 2013
European Commission Information Society and
Media
2nd Review Meeting
Next steps
• Main efforts in year 3: – Conduct the user studies– Disseminate the outcomes– Recommendations for maximizing the total value of SGs:
revision, update, validation through the user studies – Continue Evaluating/describing SGs, Monitoring SG proposition,
Updating the SIG pages
• Outlook to year 4:– describing and prioritizing the kinds of research that is needed to
measure the learning impact (reviewers’ rec.) from the application perspective
– SIG Final Thematic Workshops