Zurich July 2006
Josie Taylor, Professor of Learning TechnologyCentre for Research in Computing
Institute of Educational Technology The Open University
Methods for studying learning, collaboration and technology use in mobile environments
Zurich July 2006
Aim of talk
• Key issue: – methodological challenges of trying to satisfy various
stakeholders when evaluating learning and technology use in informal settings.
• A method – for representing user activity (practices, strategies and
conflicts) that emerge when interacting with technological systems in an informal mobile learning setting (learning?)
– semiotic and technological space.
– cultural historical activity theory, and develops Engestrom's (1987) extended model of human activity.
– Task Models
Zurich July 2006
Acknowledgements
• Mike Sharples• Patrick McAndrew• Giasemi Vavoula• Claire O’Malley
Zurich July 2006
Model of Mobile Learning:Research from MOBIlearn
• Sharples, Taylor & Vavoula (in press) propose a theory of learning in the mobile age as:
‘the processes of coming to know through conversations across multiple contexts amongst people and personal interactive technologies’
Zurich July 2006
A theory of learning in the mobile age
• “Conversation and context are essential constructs for understanding mobile learning, and offering implications for the ownership of learning and the integration of mobile learning with conventional education.”
• Draws on Pask’s (1976) Conversation Theory, as does Laurillard (2002)
Zurich July 2006
Partner provides facility for practical model building and problem solving
Partner acts to build models and solve problems
Learner acts to build models and solve problems
Sets goalsAdjusts model
ActsModifies actions
How questions and responses
Partner demonstrates or elicits models and problem solutions
Learner demonstrates understanding of models and
problem solutions
Offers theories and ideasRe-describes theories
Offers conceptions and explanationsRe-describes conceptions
Why questions and responses
Partner provides facility for mediating agreements
Level of Actions
Level of Descriptions
Conversational Framework, Laurillard 2002
Zurich July 2006
A theory of learning in the mobile age
• The focus of our investigation is not the learner, nor their technology, but the communicative interaction between these to advance knowing.
• Conversation is the driving process of learning. It is the means by which we negotiate differences, understand each other’s experiences and form transiently stable interpretations of the world.
• Sharples, M., Taylor, J., and Vavoula, G., (in press) A Theory of Learning for the Mobile Age, in R. Andrews and C. Haythornthwaite, Handbook of e-learning Research, Sage Publications.
Zurich July 2006
Research in progress
Ways of capturing and analysing this learning– Based in activity theory– Use scenarios to develop mobile learning
activities– Run trials to test both the activity and the
technology
Zurich July 2006
Additional Issues
• Communication in large multi-site, multi-national projects building systems (Carroll: 1995 – Scenarios; Taylor & Evans, 2005)
• Evaluators need to move between stakeholders, responding appropriately to each, as well as serving the needs of the end users
• Stakeholders: funders, system designers, system builders, educators, domain specialists, curators, teachers, companies, end users…
Zurich July 2006
MOBIlearn Project
• Project Info:– 33 month EU-funded with 15+ partners– Circa 8m Euros– 200+ members of SIG
• Goals:– Designing an architecture for pedagogically sound
mobile learning environments– Implementing an instantiation of the architecture with
current technologies– Evaluate
Zurich July 2006
3 MOBIlearn Scenarios:3 types of learning
• Museum – visitors to Art Gallery (informal)
• MBA – professionals in full time work engaged in study
(formal curriculum, work based)
• First Aid Training – volunteers in full time work needing training in
situ (voluntary, curriculum, work-based)
Zurich July 2006
Internally initiated
Internally structured
Formal learning - not within scope for MOBIlearn
Externally structured
MuseumInformal learning
Externally initiated
First Aid Voluntary learning
MBA Resource
based learning
Adapted from Livingstone 2001
Zurich July 2006
Scenario Development Process
Zurich July 2006
Scenarios
Guidelines
Literature review
Functional Requirements
•Pilot studies
•Qnr studies
•Interviews
•Diary studies
UML
Non functional requirements
System Design
Trials
Software Development Process
Zurich July 2006
Development of Task Model• Based in AT – a tool to:
– Represent learning in mobile settings– Capture the complexity of the setting and
situation– Capture aspects of the dialectical process of
appropriation both technical and semiotic– Communicate with various stakeholders as
part of evaluation
Zurich July 2006
Activity Theory (AT)
• Cultural Historical Activity Theory (Leont’ev, 1978)
• Grounded in Vygotsky’s (1978) theorising about the social-cultural development of human mind (developmental studies of higher mental functioning)
Zurich July 2006
Extended model of human activity (Engeström, 1987)
Subject Object
CommunityRules Division of
Labour
Tools
Outcome
Zurich July 2006
Rules Community Division of labour
Control Context Communication
Reflecting social rules and control in technological sense
Not just human community but also technological setting
Negotiating both with other humans and technology
Zurich July 2006
Semiotic Layer – Human/task focused
Zurich July 2006
Technological Layer – Implementation focused
Zurich July 2006
Interaction between semiotic and technological layers
Zurich July 2006
A pretty version
Zurich July 2006
Purpose:
• To hold a very complex situation in tension– Looking for conflicts between layers– Looking for support between layers– Identifying relationships and interactions– Allowing systematic manipulation– Allowing multiple views to be represented
Zurich July 2006
Example: First Aid Training
• First aiders at the Open University• Need to keep training up to date and
refreshed, but not many opportunities to practise skills
• All are full-time workers, so training could take place in workplace
• Communication/collaboration is as important as access to content of training
Zurich July 2006
First Aid Training Scenario: players
• Task leader:–Configures task–Interacts with participants–Provides guidance
• Task participants –act singly and in pairs to complete task–Briefed and de-briefed as a whole group
Zurich July 2006
Task: Participant view
• Initial on-line chat (group)• On-line Quiz (individual)• Challenge is set (pairs)• G1: On-line Brainstorming (pairs)• G2: On-line Brainstorming (pairs)• Get together on-line and vote (group)
Zurich July 2006
Agenda for the First Aid Trial
Zurich July 2006
Task: Task Leader view
• Set up content and direct participants to it• Construct on-line agenda• Oversee activity during chats• Set tools to appropriate states (e.g. close
the vote at the end of voting)
Zurich July 2006
Technological Tool (PDA and mobile phones)
Semiotic Tool(First Aid Training)
Semiotic Subject (First Aider)
Semiotic Object ( knowledge & skills)
Changed Object (revised knowledge & skills)
Control
Technological
(usability of device)
Semiotic
(constraints & protocols in first aid)
ContextTechnological
(network facilities, wireless)Semiotic
(community of first aiders)
Communication
Technological
(texting, uploading pictures & text)
Semiotic
(conversation; support of co-workers)
Task Model for First Aid Scenario
Technological Subject (User)
Technological Object (access to information)
Zurich July 2006
Task: Evaluator view
• Developing scenario, constructing task models and feeding into requirements or design
• Designing evaluation instruments (e.g. observation schedules, questionnaires etc)
Zurich July 2006
Iterative feedback
GenericRequirements
Semiotic Model
Technical Model
DesignFeedback
ScenarioTrials
Zurich July 2006
Semiotic Tool(Collaborative: Working together to
assess situations)
Semiotic Subject (First Aider)
Semiotic Object (G1: online brainstorming)
Changed Object (improved performance leading to G2)
Control
Semiotic
(Share; don’t diss
other person’s pov; be
constructive)
ContextSemiotic
(partner for activity; larger group)
Communication
Semiotic
(conversation; discussion;
Agreement; resolution of argument)
Semiotic model for First Aid Scenario: brainstorming
Zurich July 2006
Technological Tool Workspaces: agenda, brainstorm, chat, vote
Control
Moderated, user selections, Many choices, different levels,
task-based
Context
Wireless, fixed location, group work, divided groups
Communication
Guided, group and moderator, chats
Technical Model for First Aid Trial
Technological Subject
User at work
Technological Object
Work through agenda
Zurich July 2006
Post Trial Findings
• Technical:– Tool choice allows unified working– Usability issues are reduced by providing a
task focus– Performance on MOBIlearn system is
adequate when working on real tasks– Task-based control is available but does not
ensure shared views
Zurich July 2006
Post Trial Findings
• Semiotic:– Moderating requires control information to and from
other users– Alerting is needed to bring synchronicity– Mobility needs the user to be mobile not just the
tools
Zurich July 2006
Technological Tool Workspaces: agenda,brainstorm, chat, vote
Control
Moderated, user selections, Many choices, different levels,
task-based
Context
Wireless, fixed location, group work, divided groups
Communication
Guided, group and moderator, chats
From trial: Support from technology
Technological Subject
User at work
Technological Object
Work through agenda
Zurich July 2006
Technological Tool Workspaces: agenda, brainstorm, chat, vote
Control
Moderated, user selections, Many choices, different levels,
task-based
Context
Wireless, fixed location, groupwork, divided groups
Communication
Guided, group and moderator, chats
From trial: Conflicts in technology
Technological Subject
User at work
Technological Object
Work through agenda
Zurich July 2006
Technological Tool Workspaces: agenda, brainstorm, chat, vote
Control
Moderated, user selections, Many choices, different levels,
task-based
Context
Wireless, fixed location, group work, divided groups
Communication
Guided, group and moderator, chats
From trial: Conflicts in technology
Technological Subject
User at work
Technological Object
Work through agenda
Zurich July 2006
Semiotic Tool(First Aid Training)
Semiotic Subject (First Aider)
Semiotic Object ( knowledge & skills)
Changed Object (revised knowledge & skills)
Control
Alerting
Context
Mobility
Communication
Moderator role
Task Issues for First Aid Scenario
Zurich July 2006
40
Museum Scenario
• Uffizi museum in Florence• Groups of users looking at Botticelli
paintings• Variety of handheld devices and phones
– Tablet PCs; pocket PCs; phones; PDAs
• Data collected by Italian partners– Observation studies– Videotape of movement through gallery– Pre/post questionnaires (satisfaction) 40
Zurich July 2006
Museum Scenario data
• Good multimedia facilities and high levels of usability in the tablet computers facilitated the semiotic level:– associations with other personal ‘devices’ (i.e.
diaries and calendars)
• Semiotic context has within it relatively large numbers of young women – levels of communication and sharing
increased
Zurich July 2006
Task Model Example: Museum Scenario
Technological Tool:(PDA and mobile phones, pocket PCs, tablet PCs)
Semiotic Tool(Learning about paintings)
Semiotic Subject (Museum Visitor)
Semiotic Object ( knowledge & skills)
Changed Object (revised knowledge & skills)
Control
Technological
(usability of device)
Semiotic
(constraints & protocols in
museum)
ContextTechnological
(network facilities, wireless)Semiotic
(community of visitors)
Communication
Technological
(texting, uploading pictures & text)
Semiotic
(conversation; support of co-visitors)
Technological Subject (User)
Technological Object (access to information)
Zurich July 2006
Museum Scenario data
• Younger participants were keen on the idea of using the Chat service to save, download and print conversations so that they had a textual photograph to remind them of their visit to the museum.
• Technological aspects of the scenario were supporting and augmenting the semiotic activities, providing a
much richer experience for visitors.
Zurich July 2006
Support in the Museum Scenario
Technological Tool (Pocket PC/ Notebook)
Semiotic Tool(Learning about
paintings)
Semiotic Subject (museum visitor)
Semiotic Object ( knowledge & skills)
Changed Object (revised knowledge & skills)
ControlTechnological
(usability of device - good)Semiotic
(Social Rules: diary functions/increased sharing)
ContextTechnological
(interesting content well presented)Semiotic
(Young people/young women)
CommunicationTechnological
(good comms facilities; chat)Semiotic
(exchange of information/opinion)
Technological Subject (User)
Technological Object (access to information)
Zurich July 2006
Conflict in the Museum Scenario
• In the case of the other technologies (mobile telephones and PDA’s) we find that the relative lack of usability in the technological domain inhibits all these
developments in the semiotic.
Zurich July 2006
Conflict in the Museum Scenario
Technological Tool (mobile phone/PDA)
Semiotic Tool(Learning about
paintings)
Semiotic Subject (museum visitor)
Semiotic Object ( knowledge & skills)
Changed Object (revised knowledge & skills)
ControlTechnological
(usability of device -poor)Semiotic
(Social Rules: little interaction with exhibits, system or other
visitors)
ContextTechnological
(difficult interface, poor search)Semiotic
(community: no engagement)
CommunicationTechnological
(good comms; chat)Semiotic
(communications not used)
Technological Subject (User)
Technological Object (access to information)
Zurich July 2006
Conclusions • Work-in-progress
• The task model helps to provide a structured space within which to investigate success or failure
• Enables evaluators :– to disambiguate specific technology from
functionality– to separate human issues from technological
issues, and put them back together again– to try several different implementation
solutions in comparable circumstances
Zurich July 2006
Conclusions (2)
• Understanding the dialectic is very important– Process level – appropriation and its impact
on behaviour– Specific technology level – what the tools are
offering in terms of functionalities and how the user makes the best use of them to achieve their goal (or not!)
• Focus on the activity keeps the learner to the fore
Zurich July 2006
Conclusions (3)
• So far the task models are proving useful for evaluators
• Enables us to say more about what might be going on in a given activity
• Enables us to pinpoint potential sources of conflict within and between levels
• Provides a way of representing learning and interaction with technology
Zurich July 2006
Future Work
• Computer support for diagrams• Drilling deeper into scenarios
– How deep is deep enough?– Is more useful data yielded?– Are triangles the best representation?– If so, why exactly?