Internet based assessments via collaborative problem solving: The Assessment and Teaching of 21st Century

Skills (ATC21S™) Project

Patrick GriffinExecutive Director, ATC21S

Patrick Griffin, University of Melbourne, Executive DirectorATC21S

Michael Stevenson, Cisco Vice President - Global Education, Former ATC21S Board Chair (2009–2010)

Shelly Esque, Intel Vice President - Legal and Corporate Affairs , Current ATC21S Board Chair (2011–2012)

Anthony Salcito, Microsoft Vice President - Education, Former ATC21S Board Chair (2010–2011)

Esther Care, University of Melbourne, ATC21S International Research Coordinator

Ministerial representatives from…

– Australia:

– Finland:

– Singapore

– USA:

– Costa Rica:

– Netherlands:

EXECUTIVE BOARD

PROJECT GOVERNANCE

PROJECT GOVERNANCE

ADVISORY BOARD

Patrick Griffin, University of Melbourne, Executive Director ATC21S (Chair)

Andreas Schleicher, Organization for Economic Co-operation and Development (OECD)

Seamus Hegarty, International Association for the Evaluation of Educational Achievement (IEA)

Irina Bokova (Director General UNESCO)

Ray Adams, Technical Director PISA 2003-2012

Marc Durando, European Schoolnet

Esther Care, University of Melbourne, ATC21S International Research Coordinator

Stuart Elliott, National Academy of Sciences

David Forster, International Testing Commission

Robin Horn, World Bank

Eugenio Eduardo Severin, Inter-American Development Bank

National project managers from each founder country and associate country

Task force member from each company:

Katrina Reynan, Director, Cisco

Martina Roth, Director, Intel

Greg Butler, Director, Microsoft

Traditional assessments may not be suited to measure many 21st century skills

Goal is to develop new assessment approaches matched to new C21 skills and to advise systems, schools and teachers on the use of assessment data to help students develop higher order performances

 

Focus: the Assessment and Teaching of 21st Century Skills

Phase 1Conceptualise C21 skills

and education output needs

Phase 2 Skill

Identification and

hypotheses

Phase 3 Development

and coding via coglabs

Phase 4 Pilot studies and trials

Phase 5 Dissemination

scale and policy

CONCEPTUALIZING THE SKILLS

Assemble experts Define methods

Explore practical and technical needs in the classroom Create a new framework

Five working groups were established: Defining 21st-century skills: Ms. Senta Raizen, WestEd Methodological issues: Dr. Mark Wilson, University of

California, Berkeley Technological issues: Dr. Beno Csapo, University of

Szeged, Hungary Classrooms and formative evaluation: Dr. John

Bransford, University of Washington, and Dr. Marlene

Scardamalia, University of Toronto Policy frameworks and new assessments: Dr. Linda

Darling-Hammond, Stanford University

21ST-CENTURY SKILLS DEFINEDATC21S started by internationally defining 21st-century skills as

four broad categories.

Ways of thinking

Ways of working

Tools for working

Living in the world

Collaborative Problem Solving

Learning through Digital

Networking

Problem Solving

Critical thinking

Problem solving

Learning to learn

Metacognition

Communication

collaboration

Information Literacy

ICT Literacy

citizenship

Life and Career

Personal responsibility

Social Responsibility

hypotheses

generalisations

Rules

Patterns

Elements

Cognitive Social Intellectual Capital

Social Capital

Producer

Consumer

Task Regulation

Knowledge Building

Social Regulation

Participation

Perspective

Ways of thinking

Ways of working

Tools for working

Living in the world

Collaborative Problem Solving

Learning through Digital

Networking

Problem Solving

Critical thinking

Problem solving

Learning to learn

Metacognition

Communication

collaboration

Information Literacy

ICT Literacy

citizenship

Life and Career

Personal responsibility

Social Responsibility

hypotheses

generalisations

Rules

Patterns

Elements

Cognitive Social Intellectual Capital

Social Capital

Producer

Consumer

Task Regulation

Knowledge Building

Social Regulation

Participation

Perspective

Collaborative problem-solving

ATC21S Assessment and Teaching of 21st Century SkillsThe University of Melbourne • Cisco • Intel • Microsoft • www.atc21s.org

Components

Collaborative problem solving

Social skills

Participation

Perspective taking

Social regulation

Cognitive skills

Task regulation

Knowledge building

Ways of thinking

Ways of working

Tools for working

Living in the world

Collaborative Problem Solving

Learning through Digital

Networking

Problem Solving

Critical thinking

Problem solving

Learning to learn

Metacognition

Communication

collaboration

Information Literacy

ICT Literacy

citizenship

Life and Career

Personal responsibility

Social Responsibility

hypotheses

generalisations

Rules

Patterns

Elements

Cognitive Social Intellectual Capital

Social Capital

Producer

Consumer

Task Regulation

Knowledge Building

Social Regulation

Participation

Perspective

Participation skills over levelsLow Peripheral participation

Low subjective responsibility for outcomes of collaboration, leading to lurking behavior

Simple epistemological beliefs (knowledge is perceived as fixed and to be transmitted from teacher/textbook to learner)

Middle Activity in scaffolded environments Responding to cues in communication Medium subjective responsibility for outcomes of

collaboration Developed epistemological beliefs (knowledge is

perceived as fixed, but can be elaborated through communication and collaboration)

High Initiating and promoting interaction Activating and scaffolding others in participation Ensuring equal participation rates among group members High subjective responsibility for outcomes of

collaboration Sophisticated epistemological beliefs (knowledge is

perceived as fluid, constructed, and inherently social/collaborative in nature)

Perspective taking skills

Low Low levels of empathy High egocentric bias Social projection (expectation of others as highly similar to

oneself) Ignoring contributions from others Contributions are not tailored to participants

Middle Medium levels of empathy Medium level of egocentric bias Receptive ability (being able to understand what others want to

convey, e.g. from overhearing) Contributions from others are taken into account Contributions are moderately tailored to recipients

High High levels of empathy Low or no egocentric bias Contributions from others are embraced and contextualized with

respect to collaborators’ opinions and skills Eliciting contributions from others (e.g. through questions) Contributions are tailored to recipients (audience design)

Social regulation skills

Low Low tolerance for ambiguity Competitive or individualistic social value orientation Low readiness to negotiate joint understanding Tendency to withdraw after conflict arises

Middle Cooperative social value orientation Attempts to negotiate joint understanding Conflicts will be avoided Initiation of compromises

High Pro-social attitudes Strategies for conflict resolution Conflicts are regarded as productive tensions Initiation of successful compromises

Task regulation skills

Low Trial and error hypothesis testing Unorganized sequence of solution attempts Little or no goal setting Variety of taskwork mental models will be ignored

Middle Forward search through a problem space Organized sequence of solution attempts Setting of unspecific goals Variety of taskwork mental models will be taken into

accountHigh Reflective regulation

Forward and backward search through a problem space Strategic oversight over collaborative strategy Setting of specific goals Variety of taskwork mental models will be harnessed

productively

Knowledge building skills

Low Knowledge telling Sharing of information Isolated contributions Lack of argumentation patterns

Middle Critical analysis of information Building on input from others Adding artifacts Forming of incomplete arguments

High Knowledge transforming Integration and synthesis of multiple artifacts Forming of complete, proper arguments (explanatory

coherence)

Developin assessment Tasks for ATC21S

Founder countries

Associate and founder countries

Draft -- Concept check for reality

Panel – cognitive laboratory for codes

Pilot – for administration

Trials – for calibration

Tasks and Assignments to assess ICT Literacy

Poetry(Graphic Organizers, Creating and Listening to Audio and video)

Arctic Trek (Collaborative Notebook, Information Foraging)

ATC21S

Twenty first century skills

Collecting the Data

Coding and scoring

Cognitive Behaviour

U2L001A = presence of chat before any action

Social Behaviour

U2L00 4A = All positions hav been covered (providing player had access to 3 balls)

U2L006A = sewuential placement of balls -6 combinationsLMRLMRRMPRMLRMLLMRLMRRMLLLMMRRRRMMLL

Calibration and Interpretation

Element Amplification Low Middle HighParticipation 1.Action Activity within environment No or very little activity in

environmentActivity in scaffolded context Activity in scaffolded and

unscaffolded contexts2.Interaction Interacting with, prompting and

responding to the contributions of others

Acknowledges communication

Responding to cues in communication

Initiating and promoting interaction or activity

3.Task completion Undertaking and completing a task or part of a task individually

Maintains presence only (lurking)

Identifies and attempts the task

Perseveres in task as indicated by repeated attempts or multiple strategies

Perspective taking 4.Responding adaptive responsiveness

Ignoring, accepting or adapting contributions of others

Contributions or prompts from others are ignored

Contributions or prompts from others are taken into account

Contributions or prompts of others are adapted and incorporated appropriately

5.Audience awarenessMutual modelling

Awareness of how to adapt behaviour to increase suitability for others

Contributions are not tailored to participants

Contributions are modified for recipient understanding in the light of feedback

Contributions are tailored to recipients (audience design)

Social regulation6.Negotiation Achieving a resolution or

reaching compromiseNo attempts to negotiate joint understanding

Comments on differences in perspective

Negotiates through differences in perspective

7.MetamemorySelf concept

Recognising own strengths and weaknesses

Notes own performance Comments on own performance in terms of appropriateness or adequacy

Comments on own performance in terms of appropriateness or adequacy in the context of the task

8.Transactive memory Recognising strengths and weaknesses of others

Notes performance of others

Comments on performance of others in terms of appropriateness or adequacy

Comments on performance of others in terms of appropriateness or adequacy in the context of the task

9.responsibility Initiative Assuming responsibility for ensuring aspects of task are completed by the group

Undertakes activities largely independently of others

Completes activities and reports to others

Assumes group responsibility as indicated by use of second person plural or accepting others’ contribution

11. Resource management Managing resources or people to complete a task

Uses resources (or directs people) without consultation.

Suggests that people or resources be used in part of a task

Allocates people or resources to a task through to completion

Element Amplification Low Middle High

Knowledge Building

20. Knowledge acquisition Follow path to gain knowledge Acquires knowledge as a result of being given it directly

Deliberate single actions to acquire knowledge

Knowledge acquired through multiple purposeful actions

12. Rules “If …then” (planning and executing)

Formulating a course of action to address a problem or task

Activity is undertaken with little or no prior formulation for a course of action

Identifies short sequences of actions for a specific task

Identifies potential multiple sequence routes for a complex task

19. Relationships (representing and formulating)

Making connections between elements of knowledge

Focused on (acts/shares) isolated pieces of information

Building on input and information from others

Integrates and synthesises of multiple pieces of information

18. Hypothesis “what if…” (reflecting and monitoring

Changing from one line of reasoning or course of action to another as information or circumstances change

Maintains a single line of approach

tries multiple options in light of new information or lack of progress

Reconstructs and reorganizes understanding of the problem in light of ne information or opinion.

Task Regulation

15. Collecting Elements Explore and understand Does not recognize the need for further information

Identifies the need for specific information related o immediate activity

Identifies need for varied information related to multiple activities.

13. Systematicity Implementing possible solutions to a monitoring progress

Trial and error hypothesis testing in an unorganized sequence of solution attempts

Forward search through a problem space with an organized sequence of solution attempts

Forward and backward search through a problem space with reflective solution attempts

17. Tolerance for ambiguity

Accepting ambiguous situations and exploring options within these

Maintains only a presence in situations where there is ambiguity

notes ambiguity and suggests options

Explores the problem space

14. Organising (Problem analysis)

Analysing and defining a problem in familiar language (i.e. Making the problem more manageable and meaningful)

Problem is stated as initially represented either explicitly or implicitly

Problem is divided into sub problems

Problem is divided into sub problems and their inter dependence is recognised

Dimensionality

- cognitive

Cognitive Strand Level Descriptions

Level 5 Refined strategic application & problem solving At this level, students can recognise previous errors and transfer and use that knowledge to more complex sub task pages. They recognise resources which they concluse have caused previous errors and avoid these. They are developing a level of confidence in dealing with the task and this allows them to correct or override their partner’s responses/answers/ actions. Students share appropriate resources with their partner and use consistent strategies that have they have identified as effective in previous steps in the problem solution. Any planning and most strategies being implemented are within a team context.

Level 4 Systematic and methodical At this level, students begin to solve problems through a collaborative process and use more effective co-working strategies. By now they can simplify the problem and review previous sub task implementations several times to determine whether the rule is transferable in subsequent task pages. Exchanging information between partners and using the shared information efficiently . Their sequencing and trial and error actions require few attempts and are hence completed in an optimal amount of time/attempts. Actions appear to be well thought out and planned and each action appears purposeful They can identify cause and effect and use suitable strategies to gain a correct path solution. The student directs and guides the partner in appropriate actions and resources.

Level 3 Strategic Planning Organising & Executing At this level the student can complete lower difficulty tasks correctly and independently. They are able to complete the final step or sub task and assist their partner in this process. They increase their working together by planning strategies of working, goal setting and sharing their resources with their partner. The student tends to focus on the relevant resources and disregards those that posed no benefit in previous trials. The student is able to conclude mutually agreeable answers when required. In multi-page tasks the student teds to identify successful approaches and resources and apply the same rules repeatedly. The student continues to intensify the sequential investigations and systematic behaviour in subsequent task pages that have increased in difficulty. This leads to attempts to test hypotheses together with the partner and to the development of increasingly thorough trial and error behaviour.

Level 2 directed approach Systematic Trial & Error At this level, the student identifies possible cause and effect of actions and demonstrated an initial understanding of the task concept and begins testing hypotheses and rules. The student uses successively enhanced sequential trials or systematic exploration to increase this knowledge. they may try to alter the techniques they use to investigate or explore the change to the task they are making. They start to share information with their partner, and to identify differences of resources and information that each partner has. They seek information about their partners resource set and provide information about their own to the partner. They reorganise information and begin to distinguish between the resources relevant the irrelevant to the problem space.

Level 1 Trial and Error Exploration Students at this level are primarily exploring the task environment, clicking on various resources often in a random fashion. They may engage in singular / unilateral approaches to trial and error in an attempt to build knowledge of the problem space. They participate individually in the activity and attempt to problem solve through an apparent unsystematic guessing approach. Their strategy is limited to following the specific instructions provided unable to proceed without instructions. They tend to repeat errors or reproduce unproductive actions with no clear indication of advancing through the task for several attempts. They may acknowledge their partner, in the context of discussing the problem space in a descriptive context or for the purpose of gaining more information.

social

Social Strand Level 5- Cooperation, Shared Goals and Appreciation

At this level, the student works collaboratively with partner for knowledge building and problem solving solutions. The student engages with the partner early in the task and there is discussion of planning and strategy. Feedback from partner is incorporated and/or adapted to identify solution paths or modify incorrect ones. Where there are conflicts (e.g. missing information or inaccurate information from partner), the student manages them successfully to reach common understanding / agreement with partner before proceeding on a possible solution path.

Level 4 Mutual Commitment At this level, the student works collaboratively with partner throughout the problem solving process. He/She promotes meaningful interaction with partner, exchanging information/ resources and approaches as well as incorporating ideas from partner. The student also reports his/her own activities to partner. Differences in points of view (e.g. inaccurate information / irrelevant resources) are managed much more effectively and there are attempts to reach consensus to progress on the activity.

Level 3 Resolving Differences At this level, the student becomes aware of the partner role early in the collaborative problem solving process and recognises the need to engage with partner for the problem solving task. The student initiates and promotes interaction with partner; sharing resources, information and ideas. The student attempts to clarify differences (e.g. irrelevant resources from partner) in order to reach agreement (not necessarily consensus) to progress on the task. The student acts as though they are aware of his/her own strengths and challenges in collaborative problem solving.

Level 2 Awareness of Partnership At this level, the student works independently on some parts of the activity but is also beginning to develop an awareness of partner role which can impact on the chances of completing the task successfully. He/she interacts with partner and requests resources / information. The student also starts to tailor communication with partner to improve mutual understanding of the communications exchange. The student may have conflicts with his/her partner but there is an attempt to clarify differences.

Level 1 Clarification and Independence At this level, the student is working independently on the activity using only his/her own ideas. There is limited interaction with partner and this is mainly prompted by task instructions. The student acknowledges communication cues by partner. However, the student’s communication with partner elicits requests for clarification and restatement Disagreements lead to conflicts that are not resolved by the end of the activity.

Developmental Progressions

Social Strand Cognitive StrandLevel 5- Cooperation, Shared

Goals and AppreciationLevel 5 Refined strategic

application & problem solving

Level 4 Mutual Commitment Level 4 Systematic and methodical

Level 3 Resolving DifferencesLevel 3 Strategic Planning

Organising & Executing

Level 2 Awareness of Partnership

Level 2 directed approach Systematic Trial & Error

Level 1 Clarification and Independence

Level 1 Trial and Error Exploration

Feedback and Reporting

Reports

Reporting to StudentsLearning Readiness

Real time reports will be available, based on empirically developed learning progressions. They will identify a point of readiness to learn for each student.

This type of report will be linked to teaching interventions associated with readiness to learn as indicated by the black bar in the spine of the chart.

Instructional groups

Teacher development

Professional development

modules

1. Defining and Assessing 21st Century Skills

2. Using a Developmental Model

3. ATC21S Assessments: Getting Started

4. Interpreting Reports5. Teaching and

Learning 21st Century Skills

The System

An integrated system– Developmental progressions – standards referenced – Assessment tasks with automatic scoring– Teaching intervention and professional development– Reports for students, teachers and systems

Next stepsAccess and Use

Validation studiesExpanded resources – mainstream

curriculumOther 21st century skills

• ATC21S Assessment and Teaching of 21st Century Skills• The University of Melbourne • Cisco • Intel • Microsoft • www.atc21s.org

Transfer and Utility

To what degree will 21st century skills such as collaborative problem solving or ICT Literacy in Learning Networks facilitate student learning?

– Are these skills teachable and learnable?– Will the skills enhance learning in the classroom?

To what degree might enhanced 21st century skills contribute to adaptiveness of graduate students to the workforce?

ATC21s New Users?

Follow up!

Volume 2: The methods

Volume 3: The re

search

Implications

Conventional Assessment

• Correctness if the important part of the response

• Response can indicate reasoning ability of a specific level

• Difficult to interpret various incorrect responses

• Additipnal data is lost in the focus on the correct response

Future Applications

Task design

I. Design tasks to capture specific components of cognitive ability

I. working memoryII. Processing speedIII. Pattern recognitionIV. Systematicity’

II. Background data capture allows more user friendly and engaging asssessments

I. Reduce test anxietyII. counteract teaching to the testIII. Capture multiple variablesIV. Reduce test time needV. Increase assessment efficiencyVI. Better, richer information for teachers

III. Applications in serious game industry

Roll Out and Dissemination

What is in the system?

• Two models - cloud and portable

• Prototype assessment tasks

• Developmental progressions

• Reporting and feedback for schools, teachers and students

• Professional development modules

• Specifications for developers

Portable and Cloud Systems

Portable version Cloud version

Local adaptations Centralised control/management

Decision-making autonomy Centralised decision-making

Customisation enabledEmbed or link within existing systems

Local adaptationsAddition of tasks

No customisation

Local support structure Centralised support structure

Flexible registration system Rigid registration system

FlashLess susceptible to differences across

browsersNot currently supported on iPad*

HTML 5More susceptible to differences across

browsers

PISA 2015

• Collaborative problem solving to be tested in OECD countries in 2015. PISA will test human to computer interaction

• Collaborative Problem solving involving human to human maybe in 2018