2 2012-03-19 First steps towards consolidation from GDocs and other input
3 2012-03-23 Feedback integrated from partners
4 2012-03-27 Internal review comments considered
5 2012-03-30 Layout adaptations - Final version for submission
Impressum
Full project title: Next Generation Teaching, Education and Learning for Life
Grant Agreement No: 285114
Workpackage Leader: Susan Bull, BHAM
Project Co-ordinator: Harald Mayer, JRS
Scientific Project Leader: Peter Reimann, MTO
Acknowledgement: The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 258114.
Disclaimer: This document does not represent the opinion of the European Community, and the European Community is not responsible for any use that might be made of its content.
This document contains material, which is the copyright of certain NEXT-TELL consortium parties, and may not be reproduced or copied without permission. All NEXT-TELL consortium parties have agreed to full publication of this document. The commercial use of any information contained in this document may require a license from the proprietor of that information.
Neither the NEXT-TELL consortium as a whole, nor a certain party of the NEXT-TELL consortium warrant that the information contained in this document is capable of use, nor that use of the information is free from risk, and does not accept any liability for loss or damage suffered by any person using this information.
2.1 Purpose of this Document ........................................................................................................................ 2
2.2 Scope of this Document ............................................................................................................................ 2
2.3 Status of this Document ............................................................................................................................ 2
2.4 Related Documents ................................................................................................................................... 2
3 Goals and specifications for ECAAD method and tools v.2 (Task 2.1) .......................................................... 3
3.1 Observations on teachers ......................................................................................................................... 3
3.2 Lessons learned and Year 2 development work ....................................................................................... 3 3.2.1 General conclusions for the development of ECAAD method and tool ...................................................................................... 3 3.2.2 Differentiating between user roles (personas) ........................................................................................................................... 5
4 Appraisal and Assessment in STEM (Task 2.2) ............................................................................................. 9
4.1 Observations on teachers specific to STEM teaching ............................................................................... 9
4.2 Lessons learned and Year 2 development work ....................................................................................... 9 4.2.1 21st Century competencies integrated into STEM learning ......................................................................................................... 9 4.2.2 Non-immersive formative assessment: Developing the ProNiFa approach further ................................................................. 12 4.2.3 Outlook .................................................................................................................................................................................... 14
5 Appraisal and Assessment in TESL (Task 2.3) ............................................................................................ 16
5.1 Observations on teachers specific to TESL teaching ............................................................................... 16
5.2 Lessons learned and Year 2 development work ..................................................................................... 16 5.2.1 Integrating Second Language Education with 21st Century skills: Written English................................................................... 16 5.2.2 Integrating Second Language Education with 21st Century skills: Spoken English ................................................................... 19 5.2.3 Observations on learning English in immersive environments ................................................................................................. 19 5.2.4 Scenario 1: The Chatterdale Mystery ....................................................................................................................................... 20 5.2.5 Scenario 2: Team-building TV-show ......................................................................................................................................... 22 5.2.6 Summary of C21 assessment integrated into TESL learning: ................................................................................................... 24
6 Learning activity design modeling: Integration with the ECAAD learning ecosystem (Task 2.4) ................ 25
6.1 Observations on teachers ....................................................................................................................... 25
6.2 Lessons learned and Year 2 development work ..................................................................................... 25
7 Privacy and Trust Framework (T2.5) ......................................................................................................... 29
The ECAAD methodology (ECAAD stands for Evidence-centered Activity and Assessment Design) comprises the planning of ICT-enhanced learning activity sequences, and the design of ICT-supported formative assessment methods that can be integrated into the learning activity sequences. Thus, ECAAD provides three elements for the other NEXT-TELL layers: (a) it provides a ‘script’ for the learning activities and the (mostly embedded) assessment processes that are to be realized in the Learning Environment, (b) it provides information on what aspects of students’ learning activities need to be tracked and which aspects of students’ work need to be stored in the e-portfolio, and (c) it provides information on how to transform data on students’ activities and products into information about students’ competencies that can be rendered in the Open Learner Model.
Studies conducted in Year 1 and informal observations are telling us that few teachers that were introduced to ECAAD v.1 (both method and tool) saw an immediate use for their current (lesson planning) practices. Our studies identified a number of reasons for this, ranging from time constraints (no time for detailed planning) to unfamiliarity with the pedagogical methods that underlie individualized planning and formative assessment, such as domain analysis and the development of (individual) learning progressions. Also, only some of the teachers we worked with routinely collaborate in teams of teachers to plan localized curricula and lessons.
In Sections 3-6 of this document we outline our revised approach in light of these findings. Section 3 and 6 describe how we plan to modify and extend the general approach to lesson and assessment planning, as well as the tool support. In a nutshell, we will accommodate the need for more directly applicable ICT-enhanced “lesson plans” by providing teachers with models of lessons that foster 21
st Century learning skills, and we will
extend the method and tool support by distributing the development of activity and assessment models over a number of people/roles, with a more focused role for teachers. We will also, as originally planned, extend the integration of the modeling components into the operational environment: the learning tools that we make available in NEXT-TELL. Sections 4 and 5 detail how the learning environments and assessment methods that we specifically develop for Science, Technology, Math and Engineering Education (STEM) and for Teaching English as a Second Language (TESL) will be extended from v.1, and how in these two areas we will relate more explicitly to 21
The document specifies version 2 of the ECAAD method and tool support in terms of modifications to and extensions of Version 1 (see D2.2). ECAAD stands for Evidence-centred Activity and Assessment Design. The main activities to be supported comprise on the most general level domain analysis (what is the relevant knowledge in a particular domain, how is it learned and used, how do you know it when you see it), domain modeling (which aspects of students activities need to be recorded and How do these get transformed into statements about students proficiencies), and the delivery of assessment either in explicit form (e.g. through a quiz engine) or in embedded form (e.g., the analysis of part of a log file comprising a student's interaction with a simulation program). The ECAAD framework explicates the interrelationships among substantive arguments from the domain, assessment designs, and operational processes of assessment delivery and embedding. This document further specifies the method and the tool support for conducting ECAAD analysis and planning, its application in STEM and TESL learning, and further steps to integrating the modeler/planner with the learning environment provided in NEXT-TELL.
2.2 Scope of this Document
This document provides an update to specifications for ECAAD method and tools, not the method and tools itself (those will be covered in D2.4).
2.3 Status of this Document
This is the final version of D2.3
2.4 Related Documents
Before reading this document it is recommended to be familiar with the following documents:
3 Goals and specifications for ECAAD method and tools v.2
(Task 2.1)
3.1 Observations on teachers
Most observations regarding lesson and assessment planning were made on German teachers (see D6.3). As explained there in more detail, German Gymnasium teachers in particular have a real need for individualizing their teachings to adequately react to the different performance of students.
Based on their lesson plans they prepared for one of the workshops, it seems that assessment (in a formative manner which can be seen as a pre-condition for an individualized instruction) is not in focus of their teachings although they use self- and peer-assessment. These teachers use some sort of on-going formative assessment rather intuitively and are not used to make their intuitive assessments explicit. The teachers had some problems to state what kind of information relevant for teaching and learning they gather during their lessons. This might lead to the assumption that they are inexperienced with the planning of formative assessment and therefore of an individualized instruction (the very detailed teaching plan they showed us as an example from their teacher education did not comprise individualization).
“Time issues” – which are always mentioned by teachers in a manifold way (example in this workshop “We cannot [...] make a teaching plan for each lesson we are teaching in a week”) – also lead to some challenges because the planning of a detailed individualized instruction takes more preparation time as long as one is inexperienced.
The German teachers did not like the idea of modeling their lessons, neither with the ECAAD planner nor just on spreadsheets. For these teachers “ECAAD modeling” their lesson is much too detailed. This was probably true because they did not see the need in finding assessment triggers or assessment methods. In their classroom lessons, they would do it automatically (but not with ICT support) without explicit planning. Although the school and teachers want to teach more individually, use more ICT, and also like the OLM idea, the teachers were not yet ready to plan assessments before hand with ECAAD. It was, however, also clear that they could not really explicate at this stage what they need in order to make their teaching more individualized. Furthermore, although these teachers already use ICT to some extent, their integrated Technological Pedagogical Content Knowledge (TPCK) might not be experienced enough to see that implementing formative e-assessment needs more preparation than a traditional classroom lesson without formative e-assessment. Being equipped with hardware is necessary but not sufficient for formative e-assessment.
Regarding plan re-use, the conclusion of our teachers was that it is not possible to adopt the plans of another teacher. Even if it was possible, it would not help reduce planning time. Nevertheless, it might help to have a detailed lesson plan with all needed materials inclusive. The aspect of having some sort of online repository for plans and materials seemed interesting to them for getting new ideas but they were not so fond of the idea to put their own plans and materials into the ECAAD planner. The conclusion of the teachers in this workshop was that adopting another teacher’s lesson plan is not practical is somewhat surprising, given that exchanging lesson plans is a frequent and pervasive practice is countries such as the United States. Observations on Norwegian teachers yielded that they are more used than German teachers to detailed local planning of curriculum and instruction, and use spreadsheets for that purpose routinely.
As NEXT-TELL wants to support teachers in their workflow, we need to consider carefully how to bridge the gap from where teachers are right now and where NEXT-TELL might lead them.
3.2 Lessons learned and Year 2 development work
3.2.1 General conclusions for the development of ECAAD method and tool
From these observations, we draw two main conclusions regarding the design of ECAAD method and tools:
1. Because most teachers’ lesson preparation practices to not follow a systematic process of mapping
learning progressions, including formative assessment, and articulating individual learning plans, nor
do they include the use of ICT for these purposes, the ECAAD planner needs to provide
templates/models that scaffold these elements for teachers.
2. For the same reasons, and because the benefits of systematic planning do not occur to the individual
teacher directly, it is necessary to refine the user roles and to implement a more distributed way of
producing models: The labour gets divided between teacher/student, pedagogical expert, assessment
expert, and technical expert.
From research in Year 1 it has become clear that most teachers expect a “technology project” such as NEXT-TELL to provide “ready made” solutions for specific curricular content. The approach we favour, where pedagogical decisions are left to the teacher, and where the method and tool support takes the form of a systematic way of designing online learning activities and embedded formative assessments does not, in many cases, meet their (presented) primary need. However, since the high-level decision to keep teachers in control of curriculum, pedagogy, and assessment has been made on the basis of research and with a view that sustained use of ICT in classrooms needs to include the teacher as an active appropriator of technology rather than as a conduit for pre-fabricated (and quickly out-dated/superseded) content modules, we are not ready to give up that position. What in the short term might be meeting teachers’ immediate ICT needs may in the long term become an obstacle to sustained use and innovation.
There is a middle way, though, and it happens to meet also the need to foster and assess 21st
Century skills: While we still expect the teacher to plan for the subject-matter specific curriculum and assessment aspects, we will be providing pedagogical support as well as technical support for extending subject matter teaching so that it provides opportunities to acquire or extend competencies related to collaboration, communication, self-guided learning, complex problem solving, and ICT skills. In this manner, we respect teachers’ competencies with regard to their disciplinary teaching, but we add concrete support for extending the disciplinary teaching to the more general 21
st C competencies--an area for which teachers arguably have not been prepared.
This strategy provides also an answer to a challenge that will increasingly arise in school systems: As the requirements for developing these skills in schools become increasingly part of educational policies (Darling-Hammond, 2011)), the question arises who will teach the necessary knowledge and how will students be provided with opportunities for practicing these skills and improving upon them? And this on top of an already over-crowded curriculum! Furthermore, research on learning strategy development and meta-cognitive skills development tells us that such competencies should be taught and practiced integrated into disciplinary content, rather than as stand-alone courses. It seems evident that developing these competencies will be all teachers’ task, although they are not prepared for this at this stage, and although the manner classroom learning is organised is not conducive to acquiring and practicing most of the 21C competencies. Approaches are needed where these competencies are taught and fostered across classrooms, teachers, and disciplines. This is what NEXT-TELL can contribute to, also mobilising the potentials of ICTs.
For this purpose, the ECAAD planner will be further developed to allow for the rapid integration of learning and assessment activities that address 21C skills into subject matter teaching. The development of these competencies over time and across school disciplines will be visualized as part of the work in WP3 and WP4. The ECAAD plans will be of a kind so that they can be rapidly implemented in the NEXT-TELL learning ecosystem.
Part of the development work will hence focus on providing for teachers learning sequence models in the ECAAD model library into which they can integrate specific subject matter teaching. For instance, if a math teacher foresees letting students work in a group on problems, it should be easy for her to provide these problems on a platform such as Google Docs or Moodle, and to add collaboration activities. Similarly, when an English teacher wants to engage students in writing an essay, it should be straightforward for him to embed this into a peer review process or turn it into a collaborative writing activity. What the project will provide is pedagogical and technical models that can be extended by specific subject matter content and subject matter specific activities. Following the roles model described in the next section, it will be the task of the pedagogical expert (teacher) and the assessment expert (initially a project member) to jointly develop a lesson model that
combines subject matter content with opportunities for practicing 21C skills and for integrating assessment of the development of these skills. First models will be provided for:
Collaborative learning, including team and leadership skills development.
Self-guided learning, including planning of learning and evidencing learning through e-portfolios.
As a consequence of this adaptation to user needs, the five assessment models for STEM and TESL respectively, described in the DOW as to be developed “in close cooperation with teachers”, will now be primarily designed by the project, with teachers adding disciplinary subject matter content.
In addition to this general re-orientation, v.2 of the ECAAD method and tools will address the following issues, all based on observations on teachers as end users:
Reduce Complexity: the complexity of applying the method and tool on an operational/day-to-day basis by teachers and educators is a challenging tasks due to resources that can be allocated to the task. The main issue lies within the argumentation of added value of a concise plan from a teacher’s but also learner’s perspective. The level of formalization should still allow operational views and sharing/collaboration aspects (machine interpretation, search, browse, execute).
Provide critical mass for collaboration/reuse: the reuse of models and plans within the planner and argued added-value through sharing of knowledge is limited by a) cultural difference in participating countries and b) the availability of a critical mass within the online repository of the tool.
Show operationalization: An important aspect within the added-value discussion is to show the operational usage of the model. The current understanding of planning for enabling transparent views on steps performed and evidence on pedagogical decisions taken by the educator is not regarded as a convincing argument.
Enable personalization/customization: different disciplines, different cultural background/countries need a custom approach in defining the ECCAD planning method as well as tool environment. Such concepts are available as discussed in V1 of this document (Metamodeling approach) but need to be made transparent.
Distinct between different roles/levels of motivation: a clear distinction is needed between roles and different motivational levels within the target group of educators. This approach is not limited to the ECCAD environment, but should be considered throughout the components/services developed in NEXT-TELL.
These observations confirm findings such as reported in Falconer et al. (2007), documenting strict limitations to the level of detail and extent of formalisation teachers are ready to provide for specifying learning designs. Based upon these observations a role-based/persona based approach is proposed as the target for V2 of the ECCAD method and tool.
3.2.2 Differentiating between user roles (personas)
Developing personas for the ECCAD environment and defining roles accordingly that represent the functional perspective of the method and tool will support different end-users using the planner’s method and also tool functionality. The personas establish the basis for an identification of functionality; in addition, the levels of expertise and motivational aspects form the second dimension in following analysis.
1. Learner/Student: the role of the learner with respect to ECAAD is currently not in scope from a planning perspective but is targeted in the executional view (WP3). It is planned to consider this role in detailed within year 3 of NEXT-TELL.
From a planning perspective, different interaction models are envisioned for V3 of the method and tools moving from a consumer perspective (models are developed by teachers/experts, students “follow” the models”) to a consumer/provider interaction/negotiation view. From a methodology point of view an additional preparing step is prepared that allows the student to a) propose a certain approach on how to learn, what to learn, what means to use, b) start a negotiation process with the assessing entity and c) “release” the negotiated plan to be followed on an individual basis.
For V2, the student persona is a consuming one, continuous and transparent review/viewing capability as well as suggestion functionality is provided within the planner.
2. Pedagogical expert: the following table presents the requirements derived from the observation and the proposed approach introducing expertise and motivational levels. These levels have to be understood as accumulating, meaning that the expert user has all functionality available described before.
An important aspect from a method perspective is that the view on the method is limited in order to reduce complexity. For the pedagogical expert, only the learning progression map as well as for specific cases the learning activity model is available, meaning that all other cross-linked information is used as available from the repository.
# Functionality Expertise/
Motivation
1 Browse learning progressions: building upon the online repository provided in V1 of the method and tool to store/share models developed by others, functionality to easily view and assess the models is provided. This browsing functionality includes the following:
Search: full-text and through annotations
Read graphically: similar as the viewer currently available, enhanced by commenting features
Read textual: read in a textual/checklist format
Export to standard formats for print: transform models into readable formats such as Microsoft Word or PDF
Basic
2 Interact: a discussion functionality is provided that allows to
a) Contact the creator/expert for the specific design
b) Enable communication through web-means (commenting, chat, blogging)
c) Enable sharing through an interaction approach
Basic
3 Trigger refinement of progression and assessment expert: to support the collaborative development and interaction between different experts, a hand-over mechanism and support environment is established. The hand-over is made available by operationalization the method itself and providing delegation/task management features.
Basic
4 Instantiate learning progression templates: as a base functionality, adaption possibilities are offered. This means that the pedagogical experts views models in the repository (using the functionality above), “copies” and adapts. This means that the model stays the same, but the context is updated accordingly. An open issue with respect to the reuse is a quality assessment of the templates provided (maturity levels, peer evaluation).
Medium
5 Re-/New design through templates: in order to design new plans, functionality to create models and plans using spread sheet input is provided. The functionality is built upon the following approach:
a) Export a template from the repository (if a template seems appropriate) or download form for fill-in
6 Re/New design through modeling shell: In addition to the functionality above, expert users can directly interact with the modeling shell and create models within the environment.
Expert
7 Full access to ECAAD method within tool: For advanced users and experts, the limitation of the view is removed and a full-fetched view is provided
Expert
8 Animate models: as a preview to execution, animation of progression, quality checks are introduced
Expert
3. Assessment expert: The persona of the assessment expert and its requirements follows the same approach as for the pedagogical expert. Within the method framework presented in V1, the focus lies within the assessment column (namely assessment map and for specific cases the assessment sequence model). The assessment expert is a domain expert in the domain and capable to diagnose complex skills and competences plus related assessment concepts.
# Functionality Expertise/
Motivation
1 Review of learning progression and diagnosis support: this aspect is provided through specific functionality such as analysis and mapping/discovery within the repository of available/similar concepts.
Basic
2 Interact: a discussion functionality is provided that allows to
a) Contact the creator/expert for the specific design
b) Enable communication through web-means (commenting, chat, blogging)
Enable sharing through an interaction approach
Basic
3 Trigger assessment method binding: as a functionality to match patterns with existing implemented methods and bind them for execution.
Medium
4 Re-/New design through templates: see above Medium
5 Re/New-design through modeling shell: see above Expert
6 Full access to ECAAD method within tool: see above Expert
7 Animate models: see above Expert
4. Implementation expert: The implementation expert is a technical expert with respect to the method and tool and has access to all functionality mentioned above plus additional technical infrastructure. The implementation expert is regarded as a service desk in the sense of a support unit. Services offered by the implementation expert are the following
Knowledge transfer/Training: Enhance the level of participants of other groups through F2F and online
courses
Workshop support: provide support in workshop and meetings of stakeholders
Development support: support the stakeholders in the “translation” and formalisation of input into
The requirements as defined above are subject to a detailed elicitation process and technical validation for implementation in V2 of the tool environment and organisational environment.
4.1 Observations on teachers specific to STEM teaching
The results of the Workshop “Designing for Assessment” conducted by TUG and reported in D6.3 suggest a variety of user requirements for the development of effective teaching and assessment tools. By asking teachers to reflect on their current teaching, assessing and feedback methods needs, wishes, critical aspects as well as action needs could be elicited concerning to the methodological and technological draft of NEXT-TELL. The key message is that such teaching and assessing tools only have the possibility to be used by a cross-section of teachers when they are simple, easy to use and when they have an added value for an individualized teaching and assessing. Another relevant and interesting aspect resulting from discussions with teachers is that there is a wish for an infrastructure that allows on the one hand to link all the scattered bits and pieces and on the other hand to integrate existing materials (e.g., plans, tools, methods, learning goals, learning objects, and tests) in a fast and especially easy way.
4.2 Lessons learned and Year 2 development work
In order to address that demands coming from the side of teachers/schools/students, we will pursue two development directions. On the one hand the continuous development of the CbKST-based ProNIFA tool – as a software tool but also in terms of a conceptual approach to non-numerical formative assessment – and on the other hand the expansion of the “working with data” use case scenario towards a broader integration of the 21st century skills, focusing on (complex) problem solving, learning to learn, and collaboration/team work. We start with the later.
4.2.1 21st
Century competencies integrated into STEM learning
Complex problem solving and PISA
Especially with regard to STEM teaching and assessing ways of thinking, i.e. problem solving, critical thinking, and decision making play an important role. They appear in various forms in a number of large-scale international assessments with the most prominent one PISA. PISA is primarily focused on problem solving tasks that are related to mathematics and science.
In PISA problem competencies are defined as “…an individual’s capacity to use cognitive processes to confront and resolve real, cross-disciplinary situations where the solution path is not immediately obvious and where the content areas or curricular areas that might be applicable are not within a single subject area of mathematics, science or reading” (OECD, 2003, p.15).
In order to assess students’ ability to solve problems in real-life situations beyond the specific context of schools subject areas, in the context of PISA an assessment approach and framework was established, including items that are situated in real-life contexts; that are not resolvable through the application of routine solutions; and where connections between multiple content areas are required. With this in mind PISA focused on three different types of problems i) decision making (i.e., choosing among alternatives under constraints) ii) system analysis and design (i.e., identifying the relationships between part of a system and/or designing a system to express the relationships between parts), and iii) trouble shooting (i.e., diagnosing and correcting a faulty or underperforming system or mechanism). These different aspects of assessment enable to collect evidence of students’ knowledge and skills associated with the problem-solving process.
In particular, students should be able to demonstrate that they could:
Understand the problem
o Understanding text, diagrams, formulas, and/or tabular information and drawing inferences from them
o Relating information from various sources
o Demonstrating understanding of relevant concepts
o Using information from students’ background knowledge to understand the information given
Characterize the problem
o Identifying variables in the problem and their interrelationships
o Making decisions about the relevance or irrelevance of variables
o Constructing hypotheses
o Retrieving, organizing, considering and critically evaluating contextual information
Represent the problem
o Constructing tabular, graphical, symbolic or verbal representations
o Applying a given external representation to the solution of the problem
o Shifting between representational formats
Solve the problem
o Making decisions (decision making)
o Analysing a system or designing a system (system analysis and design)
o Diagnosing and proposing a solution (trouble shooting)
Reflect on the solution
o Examining solutions and looking for additional information or clarification
o Evaluating solutions from different perspectives through restructure them and making them more socially or technically acceptable
o Justifying solutions
Communicate the problem solution
o Selecting appropriate media and representations to express and to communicate solutions to an outside audience
Students’ performance can be separated into four different levels of problem-solving competence on a problem solving scale: 1) weak or emergent problem solvers [below Level 1], 2) basic problem solvers [Level 1], 3) reasoning, decision-making problem solvers [Level 2], and 4) reflective, communicative problem solvers [Level 3].
Below Level 1: Weak or emergent problem solvers
Students whose performance is below level 1 are not able to solve easy problem-solving items. They have difficulties in applying necessary cognitive processes to identify, characterize, and represent the problems. These difficulties relate to all three kinds of problem tasks (i.e., in decision making, system analysis and design, and trouble shooting).
Level 1: Basic problem solvers
Basic problem solvers are able to solve such problem tasks that only contain single information or information that is well-structured and well-defined. They do not successfully deal with problems which are multi-faceted that include more than one data source. However they understand the nature of the problem and are able to retrieve such kind of information that is directly related to the major topic of the problem. Additionally they are able to transform the given information and present it in different ways (e.g., taking information from a table to create a graph).
Level 2: Reasoning, decision-making problem solvers
Reasoning, decision-making problem solvers are able to apply different types of reasoning in order to analyse situations where several alternatives and constraints have to be solved. Students at level 2 have the capacity to combine and synthesis information from a variety of sources by combining different kinds of representations (e.g., formalized language, numerical and graphical information), and by coping with representations which are
unfamiliar to them (e.g., statements in a programming language). Additionally reasoning, decision-making problem solvers are able to draw conclusions and to make decisions among a variety of alternatives.
Level 3: Reflective, communicative problem solvers
Reflective, communicative problem solvers are able to do not only understand and analyse a situation where several alternatives and constraints are given and draw correct inferences based on two or more sources of information but also to reflect on underlying relationships and relate these to the solution. Furthermore students at this level solve problems systematically by constructing and using their own representation. These representations enable them on the one hand to solve the given problems and on the other hand to verify their solutions with regard to all requirements of the problem. These students are able to communicate and justifying their results and solutions in an accurate written and/or visual form.
The problem-solving scale used in PISA and its four different levels of expertise is derived from an analysis of components and processes typically necessary to solve problems.
In order foster complex problem solving as part of the 21C skills portfolio, and in order to make this relevant for schools, we will develop computer-based variants of complex problem solving items (with connections to the STEM curriculum) that students can use to practice the respective problem solving skills, and will develop on-line materials that can be used by teachers and/or the students themselves (self-guided learning) to develop the respectively needed competencies and skills. The practice problem-solving environment will be made available in the NEXT-TELL learning ecosystem, with Google Spreadsheets as the central component. Formative assessment will be provided by means of cbKST diagnostic methodology (Kickmeier-Rust & Albert, 2011), implemented in ProNiFa, with competency visualisations also made available also in the OLM.
While STEM learning affords many opportunities for acquiring and practicing complex problem solving competencies, it is also conducive to fostering self-guided learning (SGL), and collaboration/team work. In our approach to 21C learning, we consider SGL as a key component, because it provides us with the means to develop (individual) learning plans that can be used for feed-forward, in addition to feed-back. That is to say, teachers and students will be invited And the fact that we employ web-based tools for STEM learning in NEXT-TELL facilitates the integration (and assessment) of collaborative learning activities.
21C competency area Assessment focus Y2 Y3 Y4
Self-guided learning (1) Planning learning x x x
(2) Documenting & communicating learning in an e-portfolio.
x x x
Complex Problem Solving
(3) Analysing complex systems and processes
x x x
(4) Modeling complex systems and processes
x x
Collaboration & team work
(5) Manage group projects and meetings
x x
In order to develop these assessment methods, we will follow the approach to assessment design suggested with the Evidence-Centered Assessment Design methodology (Mislevy, 2006) and elaborated in the PADI project (Riscontente et al., 2007). Fully automated diagnosis/assessment will be provided for the methods (and by the time) marked with X, partially automated, but fully modeled methods for the others.
4.2.2 Non-immersive formative assessment: Developing the ProNiFa approach further
ProNIFA stands for probabilistic non-invasive formative assessment and is developed in the context of the NEXT-TELL project. The tool, in essence establishes a handy user interface for CbKST-related services and functionalities. The services are running on a service and cover, broadly speaking, CbKST-related computation tasks, such as updating of probability distributions over competence structures. In addition to that, ProNIFA provides several authoring, analysis, and visualization features. The tool is a Windows application that utilizes various interfaces and links to online-based contents.
Figure 1: Screen shots of the ProNIFA tool
A distinct feature in the context of formative assessment is the multi-source approach. ProNIFA allows connecting the analysis features to a broad range of sources of evidence. This refers to direct interfaces (for example to Google Docs) and it refers to connecting, automatically or manually, to certain log files. The only requisite is the availability of log files on the local computer, through HTTP access or via FTP. Using this level of connectivity, multiple sources can be merged and can contribute to a holistic analysis of learners’ achievements and activity levels. As an example, ProNIFA enables a teacher to use the results of a Moodle test, exercises done in Google Spreadsheets, and the commitment displayed in a virtual meeting in a chat, to conduct a semi-automated appraisal of students.
A central goal for v.2 of ProNiFa is to develop a number of alternative competency visusalisations, and to make these available through the OLM. A number of different types of visualisations will be considered, discussed next.
Hasse diagrams. Hasse diagrams enable a complete view to, potentially, huge structures. Insofar, they are ideal for capturing the large competence spaces occurring in the context of CbKST-based assessment and recommendations. Very briefly, a Hasse diagram shows all possible (admissible) competence or knowledge states. By the logic of CbKST, each learner is, with a certain likelihood, in one of the competence states. This allows coding the state likelihoods for example by colors and thereby visualizing areas and set of states with high (or vice versa low) probabilities. The simplest approach would by highlighting the competence state for a specific learner with the highest probability. The same coding principle can be used for multiple learners. This allows for identifying various sub-groups in a class, outliers, the best learners, and so on.
A second aspect comes from the edges of the graph. Since the diagram reads from bottom to top, the edges indicate the “learning path” of a learner. Depending on the domain, we can monitor and represent each learning step from a first initial competence state to the current state. In the context of formative assessment, such information elucidates efforts of the learners, learning strategies, perhaps used learning materials, but also the efficacy of the teachers.
Finally, a Hasse diagram offers the visualization of two very distinct concepts, the inner and out fringes. The inner fringe indicates what a learner can do / know at the moment. This is a clear hypothesis of which test/assessment items this learner can master with a certain probability. Such information may be used to generate effective and individualized tests. The concept of the out fringe indicates what competency should or can be reasonably taught to a specific learner as a next step. This provides a teacher with clear recommendation about future teaching on an individualized basis.
Figure 2: Visualizations of learning using Hasse diagrams in ProNIFA: The left panel shows two distinct competence states, a starting state and the present state (the upper). The red line indicates the chosen
learning paths. The right panel shows the most likely competence state of a learner (green) and the distribution in the entire class. The orange circle, as an example, indicates 50% of the students, the
yellow 75%.
Pixel Clouds. Pixel clouds are a similar concept of representing ability on an individual or group level. In principle, the pixel clouds depicts each competence state (or skill/competency) as a single pixel. Each of the competence states is assigned a probability value which is color coded. The brighter a pixel is the higher is the corresponding probability, vice versa, the darker a pixel is the lower is the corresponding probability. The difficultly (or in other terms the structural location) of a competence or competence state in given by the position in the Euclidean space, ordered from left to right. This type of visualization has the great advantage that huge competency spaces can be grasped with a single sight. Despite maybe huge spaces, important information for teachers can be displayed on a single screen without the need for zooming. As show in the following figure, by this means also temporal information can be illustrated easily and quickly.
Figure 3: Pixel cloud illustrating an assessment process. With each piece of evidence (in an ECAAD sense), the picture of a learner’s competencies is getting clearer and more stable.
Problem Spaces. A problem space is a formal and complete description of all possible solutions steps for a specific problem, represented from the starting state to the final (desired) state. All steps are represented on the basis of their admissibility according to the corresponding set of rules. The figure below illustrates the famous problem (game) the “Tower of Hanoi”. In the context of formative assessment, this type of visualization can be used to illustrate the progress of students in problem solving situations. This might be on a small scale (within a short term problem, e.g., a test item) or on a large scale (e.g., in terms of a medium to long term project). It shall be highlighted that the term “problem” refers to a broad notion of task, a problem might well be a mathematical test item which is based on a set of calculation rules. The type of illustration is highly intuitive and, maybe more importantly, allows not only a snap shot of the present state but also indicates chosen solution paths.
Figure 4: The image shows the problem space for the famous Tower of Hanoi problem/game.
4.2.3 Outlook
While years 1 and 2 were characterized by the focus on assessment and interpretation algorithms, year 3 must be increasingly in the focus of two distinct aspects. Based on the teacher workshops and the concrete applications of ProNIFA, we will develop and integrate a broader authoring functionality as well as a broader link to a range of other Next-tell related tools. More concretely, in order to empower teachers to use the ProNIFA as a package right away, it is necessary to have features that enable defining working states (e.g., the problem solving states) as well as interpretation rules and heuristics. This authoring, however, is not trivial since it must cover all sorts of connected educational tools (in the concrete context of NEXT-TELL this may be Moodle, e-Portfolios, second Life, Google Docs, etc.). As a second (but strongly related) focus, in year 3 we need to concrete on the development of various APIs and trigger mechanisms in order to facilitate an
automatized real-time data exchange between software tools, assessment algorithms running behind ProNIFA, and visualization tools (the OLM at the far end).
Teaching English as a Second Language is approached in two areas: written English, and spoken English. While in Year 1 the focus was on spoken English, by the end of Year 2 there will also be an approach in place to support the development and assessment of writing in English.
5.1 Observations on teachers specific to TESL teaching
Norwegian teachers are already using ICT to support their formative assessment. For example, teachers at Nordahl Greig upper secondary school have had positive experience with giving formative assessment while students are writing texts in Google docs. Others are using videos that students make of their own learning for student self-reflection and formative feedback. Initial experience with immersive, environments, on the other hand, has been mixed. While the potential as a tool where students can practice and develop their speaking and conversational skills is recognised, the organisational and technical issues in supporting and enabling the environments is taxing (see below). This has implications for how we need to prepare and support scenarios build on immersive environments.
5.2 Lessons learned and Year 2 development work
5.2.1 Integrating Second Language Education with 21st
Century skills: Written English
For written English, two scenarios will be further developed to connect the assessments to the OLM (work has already begun on this) and to establish relations between writing and development of 21C competencies.
Process writing and continuous assessment in Google Docs
The main idea of Process Writing is the aspect of students receiving feedback on their texts, from their peers and the teacher, during the writing process. The students use Google Docs to produce a text and invite peers and the teacher to comment on the document during the writing process. When completed, the student submits a final text for summative assessment.
The Norwegian teacher who uses this approach has had positive experience with this assessment method. When the teacher asked the students to evaluate various aspects of their English class, with respect to their own learning and personal development, there were surprisingly many students who gave a high score on continuous assessment in Google Docs. Prior to this evaluation the teacher was unaware that the students thought that continuous assessment promoted learning as much as they did. The students made a clear and very conscious distinction between continuous assessment and getting a comment and a mark on a finished text.
Task. The students are to create an English text that is 2 A4 pages long. The topic of the text is chosen by the students, but needs to be anchored in 1-3 competency aims from any other subject than English in the curriculum. The text will be created in Google docs and the students will give feedback to one peer.
Procedure. The students are given three deadlines. The first draft is to be completed for the first deadline and will be formatively assessed by the teacher. The second draft should be ready for the next deadline and one peer gives formative feedback. A third deadline gives the date when the text is to be submitted for final assessment and grading. Only the third deadline is compulsory.
In order to give formative peer feedback the students are given a list of criteria for self- and peer assessment (see below).
Procedure for students / Formal requirements for students:
1. The student creates a Google document ,which will be shared with the teacher and a classmate.
2. The student marks the document with name and class.
3. The students are given a link to the curriculum for all subjects in their educational programme.
4. The student has a high degree of freedom in selecting a topic, but the text need to be anchored in 1-3 competency aims from the curriculum. The student may base the presentation on the competency aims in any subject, but not English.
5. Once the student has selected a topic, the student also needs to devise a research question. The teacher must approve this before commencing work on the project.
6. The student should find an informative and engaging title.
7. The first week and the second week the student works on the individual in-depth study topic chosen from a course within their own education programme (e.g., biology, social science, etc).
8. The student should get time to work with the project in all English lessons for two weeks. Working on this project will also be homework for this period.
9. The final document, which must be a minimum of 2 A4 pages long, is saved in a file format that is possible to save and up-load to It’s Learning (or the Learning Management System at the school).
Assessment criteria: task specific. The assessment criteria are part of the assessment method. The teacher reported that without the criteria the students’ feedback was not concrete enough and therefore difficult for the student to use in order to improve the text. Therefore, the following criteria are given to the assessor.
1. Do you/your peer answer your research question?
2. Do you/your peer use relevant sources?
3. Do you/your peer manage to engage the reader’s interest throughout?
4. Do you demonstrate understanding of the competence aims?
5. Are the competency aims reflected throughout your/your peer’s presentation?
6. Do you/your peer demonstrate a personal and creative approach to the topic?
Assessment in terms of C21 competencies. Project-based, continuous writing tasks, such as the one just sketched, that combine subject matter learning with writing (in a foreign language) afford numerous opportunities for practising and formatively assessing 21C competencies, in addition to learning about subject matters and about writing (written communication). In particular these 21C competency areas can be practiced:
Information literacy
Self-guided learning (learning to learn, metacognition)
As part of the work in T2.3, we will develop models of online writing tasks that explicate the relation to these competencies, and provide examples of how to embed assessments into these tasks that can be used to update measures of these competencies in the OLM. For instance, in order to describe teamwork competencies, the framework developed by Salas et al. (2005) (see also Kay et al., 2006) will be used as a basis to discern sub-competencies of the capacity to work in a team. The writing environment of choice will be Google Docs because this affords easy to use tools not only for writing but also for synchronous and asynchronous communication around a jointly authored document.
Using Repertory Grid for collaborative learning.
The repertory grid technique available in NEXT-TELL through the work by partner CBS in form of the RGFA tool can be used in various forms in the context of the teaching of English as a Second Language. In the course of Year 2, this technique and the RGFA tool will be used in two forms: As a learning activity (“Odd one out”) and as diagnostic means to help teachers decide which students should work together in collaborative learning activities (“Assigning students to groups”).
Odd-One-Out. The following describes the steps of Odd-One-Out in STEM and TESL. Students participate in creative problem-solving that challenges them to think, reflect, reason, communicate and give feedback while learning and developing their critical thinking skills. By using the Repertory Grid tool, different subject terms are approached. The Odd-One-Out forces the student to think and communicate on how one learns something, as well as making one aware of using knowledge to reason and explain the chosen relation between subject terms, in addition to reflect on peer solutions.
In the odd-one-out strategy a list of several concepts are given to the students to put in the Repertory Grid. The students sit in groups and each student has to remove one of the concepts using the Repertory Grid tool and this removal needs to be justified for the peers in the group (there is no correct answer, i.e., any of the concept could be removed). The teacher’s role is to encourage these thinking and learning skills by asking the students to formulate in their own words how they came up with their solutions, in what way they cooperated with the other students and the teacher, the role of their co-learners for own learning process, and what they have learned by doing the task.
The teacher should use the Repertory Grid tool to look for misconceptions in order to help in the discussion.
1. The student is given the learning goals and a description of the odd-one-out strategy is presented.
2. The student is presented for four concepts.
3. By using the Repertory Grid tool the students make choices of which concept to remove.
4. The student should develop an explanation to explain how the concepts fit together and why to choose the one to remove. The students need to show and weigh the evidence and come to a conclusion to justify the odd-one-out choices.
5. The students have a discussion/debate in their group where they argue for their explanation to the odd-one-out choices.
6. During the debate each student writes notes on the feedback they receive to be used for later reflection.
7. The student draws a conclusions by reflecting on the debate using the debate notes and feedback given them.
8. Each student revises the explanation for the odd-one-out choices.
Assigning Students to Groups. A “classical” problem for group learning pedagogy is to decide on group membership: How should students be allocated to groups? While sometimes ad hoc criteria suffice, from a pedagogical point of view the allocation should follow a more meaningful rationale. The Repertory Grid Technique offers the means to group students by similarities (or differences) in their conceptualization of a topic. With RGT as implemented in the NEXT-TELL ecosystem with the RGFA tool, it is rather straightforward to identify which students have similar views of a topic (“freedom of speech”, say) and then to use that information for pedagogical decision making. For instance, a teacher may want to form teams where there is
certain homogeneity of views, or the teacher may want to maximize the difference in views in each team, so as to foster debate. Whichever the case, RGT can offer diagnostic information on student’s current views of a topic that go way beyond simple polls as afforded by audience response systems (“clickers”). Of course, RGT can also be used to monitor changes in views/beliefs/knowledge effected by learning and group work. In Year 2, we will have developed and tested with teachers the use of RGFA for the purpose of informing group allocation decisions.
5.2.2 Integrating Second Language Education with 21st
Century skills: Spoken English
In the following section we describe some design ideas behind scenarios that we build in our virtual environment. In these scenarios a lot of live spoken interaction is going on, especially when the team members do not share a common mother tongue and when they need to exchange different information collaboration is facilitated. The scenarios must not be seen as stand-alone, but need embedding in prior activities. For ideas on that, please refer to D6.3.
5.2.3 Observations on learning English in immersive environments
Since the use of immersive technologies such as Second Life and Open Sim in classrooms faces many challenges, we begin this section with a reflection of experiences from use in NEXT-TELL schools so far.
Organizational challenges, such as accessing Web-services, installing local Software and handling Hardware accessories like headsets are the most severe obstacles. Depending on the participatory school, multiple layers of approval (up to 4) are required for using any given tool.
A second finding is, that for specific learning scenarios only small windows of opportunity are available, and missing the windows requires a redesign of the scenario to meet changed requirements from the teacher and curricular points of view. This also drains on precious teacher/student time.
Our finding in this respect is, that a substantial lead-time is needed when planning any scenario/session.
UniRes observed in Norway that getting the teacher to “own” the experience is a challenge with SL/OS. It is a severe demand on teacher when s/he does not feel competent in SL/OS. Thus, there is a huge demand on supporting this type of scenario. Loss of control is an issue with some teachers, an loss of connection is one for others (as indicated in the cognitive density questionnaires provided to the Norwegian teachers).
Spoken Language training is increasing, as the technology allow for more simultaneous conversation strands to happen in parallel than in a pure classroom setting where it would become noise. This is because using headsets conversations can be organized very efficiently. The setting also creates incentives and positive feedback for those students who engage. One of the students in Norway told us that it was better in our scenario than in the classroom since there they usually were given a text and a role and they just read the text when it was their turn, i.e., there was no spontaneous conversation.
21st century skills are addressed by design activities where a strong team collaboration and interaction between participants from different countries and cultures is necessary. In addition to learning results this pan-European tandem constellation creates a huge positive impact on motivation for many students. This can be observed by the creativity sparkled through the interaction which can not be observed at the same level in isolated settings.
Next steps are to solidify and detail the findings of our learning scenario. This requires more data collection and analysis. More data collection points will be automated and integrated into the learning scenarios evaluated in the test-beds. More collected data will also reflect in extended modeling for the assessment part, which is currently done manually. Teacher feedback is helping us to refine the focus on what questions to ask and what data to collect in the first place.
The Chatterdale Mystery is a mystery quest in OpenSim in which teams of students have to find out what happened to the population of Chatterdale and find the lost Prof. Jones. They get some information at the beginning (an eMail from their boss and a map of Chatterdale) and they find hints as well as 2 survivors in Chatterdale itself.
Design:
Technical:
o Location: Chatterdale, a virtual English city on the virtyou-Grid (OpenSim) http://virtyou.com/welcome/
o Voice chat: 2 options: in-world voice support or teamspeak
Story:
o An interesting story that fits to the age group of 12/13 years old students
o An attractive goal to achieve (find Prof. Jones)
o Rewards: Students may pick up some items (e.g. clothes) for their avatar (those items are in the cave, where they find Prof. Jones)
Teams
o Students work in teams. This implies the chance to set up internationally mixed teams (as we did within our Austro-Norwegian cooperation) - a great chance for students to practice English all through the activity
o Asymmetric team assignments: Each team had a slightly different task to solve - but all were related to the mysterious disappearance of the people of Chatterdale.
o Asymmetric information within the teams: To make it more challenging, different team members (e.g. the Austrian and the Norwegian team members) can be given different information.
Communication
o Internationally mixed teams (especially with asymmetric information) enforces tight communication and cooperation from the beginning on
o Actors: There are 2 surviving Chatterdalers and Prof. Jones. Those are played by actors. The students have to talk to them and draw the right conclusions in order to be able to solve their assignment.
Data capturing
o Video data
o Audio data (especially easily to be captured by using team speak)
Please refer to deliverable 6.3 for more information about the school trials. There you also find more information (checklists) about how to set up such a trial and which team-building activities (using Moodle) we did before.
Assessment:
The aims described in the Curriculum, which includes the CEFR and some 21st Century skills have been itemized and numbered, in this table the reference numbers are included. NPCs are actors with special roles in the environments, usually native speakers.
OLM: curriculum aims / CEFR / 21st Century Skills
assessment measurement evidence provided by
Student understands the assignment (reading B1 CEFR#109, AT#3)
Students start looking for the right place / person (time for start moving / arriving)
automated locator/greeter script (triggered when entering target area)
Main point was identified and addressed in conversation with NPCs (NPC feedback)
NPC’s feedback dialog prompted when student leaves
Approach and Ask NPCs for help (speaking A1 or more CEFR#116ff, AT#4+#19, AT#63)
Is student approaching NPCs and talk to them? (Location sensor A after B)
automated locator/greeter script, chat time statistics
Who is talking how much? (timing) chat time statistics*
Level of language used? NPC’s feedback dialog prompted when student leaves
Interaction sequence (chat, talk, etc. approximate timing and intensity)
analysis on all interaction data (movements, chat)*
In this table the left column “OLM: curriculum aims / CEFR / 21st Century Skills” links the assessed items to the skills modeled in OLM. Currently 3 skill-maps are referred to: the Austrian curriculum for second languages in secondary level; the CEFR and 21st Century skills. More skill-maps might be added later.
The middle column “assessment measurement” is describing what data we are looking for.
The right column “evidence provided by” is describing how we collect the data.
5.2.5 Scenario 2: Team-building TV-show
We consider team building as an essential prerequisite for successful teamwork - in virtual environments as well as in real. Therefore we developed this team-building scenario. It supports a team-building process where teams should be built out of 2 different groups (e.g. Austrian and Italian students). The basic idea is to visualise the process by positioning the current team’s gondola: If the yellow gondola is docked at the space with the Italian candidates, then (obviously), the next step is, that one Italian candidate is picked by the yellow team. In the next step the blue gondola would dock at the space with the Austrian candidates, then the yellow at the Austrians and the blue at the Italians.
After one “round” we have one Italian and one Austrian in each team. This very general setup can be used for all kinds of selection-tasks. In our trial, the candidates had to answer questions, which they prepared in Moodle before.
The aims described in the Curriculum, which includes the CEFR and some 21st Century skills have been itemized and numbered, in this table the reference numbers are included. NPCs are actors with special roles in the environments, usually native speakers.
curriculum aims / CEFR / 21st Century Skills
assessment measurement evidence provided by
Student understands the assignment (reading B1 CEFR#109, AT#3)
Students start looking for the right place / person (time for start moving / arriving)
automated locator/greeter script (triggered when entering target area)
speak out clearly, make it easy for others to answer: spoken production C2 (CEFR#137, 21st skill#27)
Candidates can answer to the question of the moderator (a student too who is asking)
audio/video record of TV show (web-stream)*
Understand the question (hearing A1ff CEFR#94, AT#2, AT#63)
Did the candidate react/answer to the said approprately? (Moderator feedback, the moderator might be another student)
audio/video record of TV show (web-stream); moderator’s reply; chat analysis*
spoken production B1 (CEFR#130, 21st skill#27, AT#80 bzw. AT#99)
Feedback on language and answer from peers (understanding, like answer)
“Like” Button (Audience); NPC’s feedback dialog prompted when moderator shift focus to next candidate
Feedback on language from teacher: Level, suggestions
repeat and correct the spoken production (-> C2 CEFR#137)
compare Student and Moderator's version of the answer, eventually repeat after class
live experience; audio/video record*
*) This evidence data needs currently one step of manual processing and is only partly automated yet.
In this table the left column “OLM: curriculum aims / CEFR / 21st Century Skills” links the assessed items to the skills modeled in OLM. Currently 3 skill-maps are reffered to: the Austrian curriculum for second languages in secondary level; the CEFR and 21st Century skills. More skill-maps might be added later. The middle column “assessment measurement” is describing what data we are looking for. The right column “evidence provided by” is describing how we collect the data.
5.2.6 Summary of C21 assessment integrated into TESL learning:
21C competency area Assessment focus Y2 Y3 Y4
Self-guided learning (1) Planning learning X X X
(2) Documenting & communicating learning in an e-portfolio.
X X X
Communication (3) Write different texts, including collaboratively written ones.
X X X
(4) Listen to and understand various spoken messages in a variety of communicative situations and to speak concisely and clearly.
X X X
Collaboration & team work
(5) Manage group projects and meetings
X X
In order to develop these assessment methods, we will follow the approach to assessment design suggested with the Evidence-Centered Assessment Design methodology (Mislevy, 2006) and elaborated in the PADI project (Riscontente et al., 2007). Fully automated diagnosis/assessment will be provided for the methods (and by the time) marked with X, and partially automatized but fully modeled methods for the others. Note that the methods developed here will be different from those in STEM (see Section 4.2.1), even where the same competence area is addressed.
6 Learning activity design modeling: Integration with the
ECAAD learning ecosystem (Task 2.4)
6.1 Observations on teachers
For observations on teachers, see D6.3 and the summary provided in the document here in Section 3.1.
6.2 Lessons learned and Year 2 development work
As already indicated in Section 3.2.2, a persona approach is introduced in V2 of ECAAD method and tool specification. This approach must be reflected in the ECAAD modeling environment: start page, user interface, available functions, etc. The differences in the user interface will also influence availability of functionalities for the integration with other NEXT-TELL tools. In next chapters, a short description of interfaces, together with sample use cases for each persona is provided.
Integration of Google Docs and ECAAD planner
Based on the first user trials, Google Docs is commonly used by teachers. It seems to be much more intuitive for teachers to write and share e.g. curriculum in Google Docs than to model it using ECAAD planner. Therefore, Google Docs could you used as an alternative representation of graphical models, gaining more acceptance from end-users. For the ECAAD planner users this would mean possibility to display, edit and print models not only as a graphic but also in more textual form (e.g. tabular view)
Figure – sample use case for Google Docs and ECAAD Planner integration
The following functional extensions should be made available
# Functionality
1 Display graphical models in textual form (e.g. tabular view) for certain personas
2 Import of semi-structured data from Google Docs to ECAAD planner
- definition of import templates to support different models, classes and attributes
- mechanism for adding/overwriting model content if data already exist in the tool
- definition of export templates for different users/scenarios (e.g. curriculum publishing)
4 Show/hide import/export functionality only for certain personas
Integration of LMS and ECAAD planner
For LMS (e.g. Moodle) and ECAAD planner integration scenario, also different, more simple user interface should be made available. Typically, a pedagogical (or assessment) expert would make a draft of a learning course – not necessarily using graphical models, but e.g. text-based data input. This would be than exported to LMS system for further processing using SCORM format.
Figure – sample use case for LMS and ECAAD Planner integration
# Functionality
1 SCORM export of models from ECAAD planner
2 Show SCORM export functionality only for certain personas
Integration of cbKST and ECAAD planner
For competence diagnosis, cbKST tools are used. A sample use case assumes, that the assessment experts define a set of competencies in ECAAD planner (again, using much simpler interface as it is in R1 of tools). These competencies are exported to cbKST tools where competence diagnosis takes place. Results, e.g. in form of priorities for competencies are important back to ECAAD planner and update existing data.
Figure – sample use case for cbKST and ECAAD Planner integration
# Functionality
1 Export of data from ECAAD planner to cbKST
2 Import of data from cbKST to ECAAD planner
3 Show/hide import/export functionality only for certain personas
Integration of OLM and ECAAD planner
In this scenario, also different personas trigger the data exchange between both tools. E.g., if the learning process should be visualised in OLM, a pedagogical expert should have a rather simple interface allowing the export to OLM. On the other hand, if process instance data should be visualised in OLM, a more advanced interface for data exchange could be available, as this task will be performed by the implementation expert.
Figure – sample use case for OLM and ECAAD Planner integration
# Functionality
1 Export of data from ECAAD planner to cbKST
2 Export of data (or data pointers in Mahara) from Activity stepper to OLM
3 Show/hide import/export functionality only for certain personas
Integration of OpenSim and ECAAD planner
In this scenario, no technical integration will be developed, focusing rather on process documentation in form of instructions for OpenSim users (e.g. from the ECAAD planner of user-friendly documentation is exported which explains steps performed in a lesson).
# Functionality
1 Export of process data from ECAAD planner to Word/PDF documentation in textual and user-friendly form (e.g. tabular view)
3 Show/hide export functionality only for certain personas
As a result of the first phase of task 2.5 “Privacy and Trust Framework” deliverable D2.10 documents the requirements with respect to privacy and trust in NEXT-TELL regarding the environment provided to students, teachers, parents and related stakeholders.
In light of our own research with teachers, as well as in light of other research on teachers as authors of pedagogical planning artifacts, the two main conclusions we draw are:
1. NEXT-TELL needs to provide teachers with immediate benefits (“time savers”) in order to make them
engage with method and tools. The immediate benefit we will provide in v.2 is to provide teachers
with a number of “ready-made” learning activity models that, once disciplinary subject matter content
has been added, can be used for developing and assessing specific 21st
Century competencies,
embedded into IT-based learning solutions made available to teachers by NEXT-TELL.
2. The task of modeling ICT-integrated learning and assessment needs to be performed by a team with
multiple roles and areas of expertise. In order to support this team work, ECAAD v.2 will differentiate
the modeling methods into four roles, and design the tool support to fit these roles.
Darling-Hammond, L. (2011). Policy frameworks for new assessments. ACTS21 White Paper. (Available at http://atc21s.org/wp-content/uploads/2011/11/5-Policy-Framework.pdf)
Falconer, I., Beetham, H., Oliver, R., Lockyer, L., & Littlejohn, A. (2007). Mod4L Final Report: Representing learning designs. Glasgow: Caledonian University.
Kay, J., Maisonneuve, N., Yacef, K., & Reimann, P. (2006). The Big Five and Visualisations for Team Work Activity. In M. Ikeda, K. D. Ashley & T.-W. Chan (Eds.), Proceedings of Intelligent Tutoring Systems (ITS06) (pp. 197-206). Heidelberg: Springer.
Kickmeier-Rust, M. D., & Albert, D. (2011). Micro adaptivity: Protecting immersion in didactically adaptive digital educational games. Journal of Computer Assisted Learning, 26, 95-105.
Mislevy, R. J., & Riscontente, M. M. (2006). Evidence-centered assessment design. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test design (pp. 61-90). Mahwah, NJ: Lawrence Erlbaum.
OECD (2003). The PISA 2003 Assessment Framework: Mathematics, Reading, Science and Problem Solving Knowledge and Skills. OECD, Paris.
OECD (2004). Problem Solving for Tomorrow’s World. First measures of Cross-Curricular Competencies from PISA 2003. OECD, Paris.
Riscontente, M. M., Mislevy, R. J., & Hamel, L. (2007). An introduction to PADI task templates. (Principled assessment designs for inquiry technical report 3) Principled assessment designs for inquiry technical report 3.
Salas, E., Sims, D. E., & Burke, C. S. (2005). Is there a "Big Five" in teamwork? Small Group Research, 36(5), 555-599.
EFL 'English as a Foreign Language'; EFL refers to learning English in a non-English-speaking region, such as studying English in an Asian or Latin American nation. Typically, EFL is learned as part of a student's school curriculum or for career purposes if working for an international corporation.
ENA Epistemic Network Analysis
ESL English as a Second Language; refers to learning English in the target language environment
HCI Human Computer Interaction
ICT Information and Communication Technology
IT Information Technology
LEPP Longitudinal Evaluation of Performance in Psychology (2nd generation e-portfolio)
NEXT-TELL Next Generation Teaching, Education and Learning for Life
PADI The PADI project aims to provide a practical, theory-based approach to developing quality assessments of science inquiry by combining developments in cognitive psychology and research on science inquiry with advances in measurement theory and technology.
RA Requirement Analysis
RDS Researcher-led Design Study
SRI Stanford Research Institute
STEM The Science, Technology, Engineering, and Mathematics (STEM) fields are collectively considered core technological underpinnings of an advanced society, according to both the National Research Council and the National Science Foundation
TDS Teacher-led Design Study
TEL Technology Enhanced Learning
TESL Teaching English as Second Language
TISL Teachers Inquiry into Students Learning
Acknowledgement: The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 258114.
