Date post: | 19-Dec-2015 |
Category: |
Documents |
View: | 216 times |
Download: | 1 times |
Davinia Hernández-Leo, Eloy D. Villasclaras,Davinia Hernández-Leo, Eloy D. Villasclaras,Juan I. Asensio-Pérez, Yannis DimitriadisJuan I. Asensio-Pérez, Yannis Dimitriadis
GSIC / EMIC groupGSIC / EMIC groupUniversity of Valladolid, SpainUniversity of Valladolid, Spain
TENCompetence Workshop
Barcelona, June 22nd 2007 http://gsic.tel.uva.es
Some findings Some findings from the evaluation of from the evaluation of Collage authoring toolCollage authoring tool
2
Our approach (I)Our approach (I)
ChallengesChallenges1. Designing scripts that potentially elicits the
desired interactions
How can How can teachersteachers create “ create “potentially effectivepotentially effective” ” ““computer-interpretablecomputer-interpretable” collaboration scripts ” collaboration scripts according to the necessities of their situation?according to the necessities of their situation?
3. Hiding computational representations of scripts to teachers
2. Formalizing the scripts so that they are computer-interpretable
3
Our approach (II)Our approach (II)
1. Not easy to design a script that potentially elicits the desired interactions
Reusing the “essence” of commonly used CSCL scripts (formulated as patterns)CLFPs, Collaborative Learning Flow Patterns:Broadly accepted techniques repetitively used by practitioners when structuring the flow of types of (collaborative) learning activities (E.g. Jigsaw, Pyramid (Snowball) CLFPs)
4
Our approach (IV)Our approach (IV)
2. Formalizing the scripts so that they are computer-interpretable
IMS Learning Design (LD)IMS Learning Design (LD)
5
Our approach (V)Our approach (V)
3. Computational representations are not familiar to educators
Integrate in authoring tools “a pattern-based design process for CSCL macro-scripts
computationally represented with IMS LD” which allows teachers to create their own computer-interpretable script by reusing CLFPs as visual LD templates that can be assembled and refined
COLLAGECOLLAGE (COLaborative LeArning desiGn Editor)(COLaborative LeArning desiGn Editor)
(Based on Reload Editor. IMS-LD level A compliant)
6
Evaluation methodology (I)Evaluation methodology (I)
Multicase studyMulticase study (Stake, 2005)(Stake, 2005)
Authentic experiences with different audiences (contextual issues, achievement Authentic experiences with different audiences (contextual issues, achievement of educational benefits, impact of software systems)of educational benefits, impact of software systems)
Cross-case analysisCross-case analysis
Not aiming at accurate generalizations but at valid conclusionsNot aiming at accurate generalizations but at valid conclusions
Mixed method combining qualitative and quantitative data gathering Mixed method combining qualitative and quantitative data gathering
techniquestechniques (Martínez et al., 2003)(Martínez et al., 2003)
Triangulation: Comparative analysis of the data obtained with the different Triangulation: Comparative analysis of the data obtained with the different techniquestechniques
7
Evaluation methodology (II)Evaluation methodology (II)
Multicase studyMulticase study (Stake, 2005)(Stake, 2005)
QUITAIN: The proposed pattern-based design process for CSCL macro-scripts computationally represented with IMS LD
ISSUE: Does the design process implementedin Collage facilitate the reuse of CLFPs
in the creation of particularized LD-represented CSCL scripts in a way
that allows teachers to focus on the CSCL critical elements?
Creating CSCL scripts based on
CLFPs using Collage
ISSUE: Can we useCollage for creating a script representing a scenario proposed
by a third-party?
Solving a third-party scenario
ISSUE: Can we use CSCL scripts
created with Collage in real situations ?
Putting into practice a CSCL
script created with Collage
“Collage workshops” Case Study
Lab. uni. of Valladolidand Cádiz, Spain
Hands-onsessions
Mixed method Questionnaires
ObservationsFocus groups
Mini-cases
“Planet game” Case Study
ICALT conference
Pre-workPresentingDiscussing
Achieved CSCL script, papers, discussion
“Network Managment” Case Study
Two f2f anda distancesession
Lab. Uni. OfValladolid
Mixed method
8
Evaluation methodology (III)Evaluation methodology (III)
Two main experiences with the target audienceTwo main experiences with the target audience (teachers with interest in CL and CS)(teachers with interest in CL and CS)
University of Cádiz, Spain [University of Cádiz, Spain [UCAUCA], ], University of Valladolid, Spain [University of Valladolid, Spain [UVAUVA]]
Creation of the script (for Network Management course)Creation of the script (for Network Management course)
Two smaller experiencesTwo smaller experiences (mini-cases): (mini-cases): CSCL practitioners [CSCL practitioners [GSIC/EMICGSIC/EMIC], ],
Educational technologist [Educational technologist [UNFOLDUNFOLD]] Several different scriptsSeveral different scripts
DataData: Scripts, questionnaires, observations, discussion : Scripts, questionnaires, observations, discussion groupsgroups
9
Sample creation of a script (I)Sample creation of a script (I)
““CTM2” script CTM2” script (applied in the “Network Management” case (applied in the “Network Management” case study)study)
Optional undergraduate course on Network Management Optional undergraduate course on Network Management technologiestechnologies
TeacherTeacher
I want to design a collaboration script that guides the students in the
collaborative understanding of a collaborative understanding of a complex long technical papercomplex long technical paper that that
can be divided into 3 different can be divided into 3 different sectionssections (3 versions of a network (3 versions of a network management protocol). I want the management protocol). I want the
students to discuss and reach students to discuss and reach agreement on the main ideas of the agreement on the main ideas of the
paper…paper…
(Evaluation methodology)(Evaluation methodology)
10
Sample creation of a script (II)Sample creation of a script (II)
Checking educational Checking educational benefits, types of benefits, types of problems, complexityproblems, complexity
Selecting the CLFPs
Reading information and Reading information and examplesexamples
(Evaluation methodology)(Evaluation methodology)
11
Authoring a CLFP-based
LD
Combining the Combining the CLFPsCLFPs
Sample creation of a script (III)Sample creation of a script (III)(Evaluation methodology)(Evaluation methodology)
12
Sample creation of a script (IV)Sample creation of a script (IV)
Authoring a CLFP-based
LD
(Evaluation methodology)(Evaluation methodology)
Refining the Refining the
CLFPsCLFPs
13
Evaluation methodology (IV)Evaluation methodology (IV)
ISSUE: Does the design process implemented in Collage facilitate the reuse of CLFPs in the creation of particularized LD-represented CSCL scripts in a way that allows teachers to focus on the CSCL critical elements?
Pattern-Pattern-based based design design processprocess
Focus on Focus on CSCL CSCL critical critical elementselements
Use of Use of CollageCollage
Characteristics of Characteristics of the potential the potential audienceaudience
Four topics of interestFour topics of interest
14
Evaluation resultsEvaluation results
Pattern-based Pattern-based design processdesign process
Is the selection of the Is the selection of the CLFP-based LD templates CLFP-based LD templates and their representation and their representation useful and satisfactory?useful and satisfactory?
Does the design process achieves a satisfactory trade-off Does the design process achieves a satisfactory trade-off between the reuse of the CLFPs and the creation of between the reuse of the CLFPs and the creation of scripts contextualized according to the situational needs?scripts contextualized according to the situational needs?
Support data
“… a minimum formation of patterns is necessary, for which Collage is helpful [UVA-focus]”
Selection phase is critical and promotes the understanding of the patterns
“Collage systematizes the selection of the patterns [UCA-quest-final]”
All the GSIC/EMIC, 5 UNFOLD, 3 UVA and 10 UCA partici-pants agree on that the CLFPs are significant. 1 UVA and 3 UCA interviewees state that they are other relevant techniques but these are OK. The rest (2 UNFOLD, 1 UVA and 1 UCA participants) indicate that they do not know any CL technique.
The CLFPs are significant but there are more well-known strategies (e.g. for assessment)
Support data
“… I miss a pattern that I employ in my classes, which is randomly selecting a member of a groupto explain what the group has done [UVA-quest-final]”
The interviewees’ ideas of the CLFPs coincide with what is presented in Collage
Support data
“… I think that the patterns are perfectly transferredto the user workspace, reproducing the needed roles and activities for their execution [GSIC/EMIC]”
The combinations of patterns provides design flexibility
Support data
“… it would be nice if other patterns could be added.However, as patterns can be combined, thesealready offer quite a lot of flexibility [UNFOLD]”
Support data
“… no example is directly transferable…”“… the examples are useful especially if results areavailable… as in a community of practice…”[UVA-focus]
Pattern-based templates are probably more useful in the processof customizing a new situation thanready-to-run templates, but completeexamples are also useful
Support data
“… no example is directly transferable…”“… the examples are useful especially if results areavailable… as in a community of practice…”[UVA-focus]
Satisfactory trade-off between flexibility, keeping the essence captured in the CLFPs, hiding LD-specific technologicaldetails and providing a clear (but limited) set of design options
“… it saves the teacher a lot of specification workload [UVA-quest-final]”“I do not find the process too contrained since it enablesthe combination of patterns… and I think that it is easierfor a novice to have an already structured model [UVA-focus]”
Support data
15
Evaluation resultsEvaluation results
Use of Use of CollageCollage
Can the teachers use successfully Collage?Can the teachers use successfully Collage?
How can CollageHow can Collagebe improved?be improved?
Support data
Most of the participants find Collage user-friendly and intuitive and are able to create partly completed examples during the workshops. Very few problems appear by they point out the importance of applying the scripts in real situations.
“I am satisfactorily astonished by the user-friendliness of the tool; I thought it was going to be more difficult [UCA-quest-final]”“.. the participants create the designs without difficulties… [UVA-observer]”
Collage is easier to use, specific to CL andit is the first IMS-LD editor providing pattern-based templates
“… at last, patterns in practice!” “… less complicated and more intuitive than Reload on its own…”“makes Reload much easier to use for specific purposes”[UNFOLD]
Support data
Incorporating more CLFPs and other types of patterns (also non-pattern based elements), which would provide more flexible design options. (Level B and C of IMS-LD) “Preliminary view” of the script. The integration of Collage with instantiation and delivery systems should support flexibility.
“… it may happen that advanced users cannot specifythings that they would like [UVA-quest-final]”
“… the incorporation of activities that are not part of a pattern but that may be complementary [GSIC/EMIC]”“… if some students abandon the course in the third
session, then I need to re-structure [UVA-focus]”
Support data
16
Evaluation resultsEvaluation results
CharacteristicsCharacteristicsof the potential of the potential audienceaudience
[UCA-focus-group]
Which are the Which are the characteristics andcharacteristics andmotivations of themotivations of thepotential audiencepotential audienceof Collage?of Collage?
-Interested in designing CL processes to be used with an LMS in f2f, distant or blended situations-teachers, novice or with experience in CL-some experience and interest in the use of ICT,but it is not necessary to be expert (IMS-LD) technologist
“… it is going to be necessary to use this type of tools in our educational system. The EEES will also push to use these tools [UCA-quest-final].”
“… for non-experts on CL this is really an eye-opener [UNFOLD]”
Support data
17
Discussion (I)Discussion (I)
A systematic multicase study A systematic multicase study Focus on Focus on Collage workshopsCollage workshops (2 experiences (2 experiences
with target audience and 2 mini-cases), using with target audience and 2 mini-cases), using a mixed evaluation methoda mixed evaluation method
Other two cases:Other two cases:
ICALT 2006 workshop (third-party scenario)ICALT 2006 workshop (third-party scenario)
Experience with students in a real situation Experience with students in a real situation (Hernández-Leo et al., IEEE Transactions on (Hernández-Leo et al., IEEE Transactions on Education, to be published in November 2007)Education, to be published in November 2007)
18
Discussion (II)Discussion (II)
Positive results with respect to:Positive results with respect to: Reuse and contextualizationReuse and contextualization Focus on CSCL critical elementsFocus on CSCL critical elements Support of practitioners (teachers)Support of practitioners (teachers)
But also pointers for several But also pointers for several enhancements and new research effortsenhancements and new research efforts
http://gsic.tel.uva.es/collagehttp://gsic.tel.uva.es/collage
Davinia Hernández-LeoDavinia Herná[email protected]@tel.uva.es
20
Evaluation resultsEvaluation results
Focus on CSCLFocus on CSCLcritical elementscritical elements
Does the design process:Does the design process:help to determine the help to determine the learning objectives, task-type and expected interactionlearning objectives, task-type and expected interaction that will be developed? that will be developed?
help to understand and determine the help to understand and determine the structure regarding the structure regarding the flow of activities and hierarchy flow of activities and hierarchy of groupsof groups??
support also the definition of support also the definition of group-size, resource distribution, computer support and structure within group-size, resource distribution, computer support and structure within activitiesactivities?? The “selection phase” helps to determine
the objectives related to CL that will be promoted and to select the task-type that will be solved by the students
Support data“… it helps to determine the CLFP more suitable to foster particular objectives [UVA-quest-final]”“… it helps to envisage how the interactions willdevelp [UCA-quest-final]”
Many arguments support that the design process is very satisfactory concerning this element
Support data“I think that the most important aspect of Collage is that ithelps to structure and systemtize the different types of activities… Collage has helped me to structure what I already do but I do not organize well [UCA-focus]”
Support data“… it hightlights the interrelations among activities”“… the structure of the (learning) process helps to check the roles that each group plays in each moment” [UCA-quest-final] “… Collage helps to conceptualize the groups [UVA-focus]“
I helps to specify these elements, but it doesNot provide suggestion of recommendedcontent or tools and the eventual structure within activities is textually-defined (or supported by tools)
“… Since Collage guides the design process…, we are pushed to reflect on each element that shapes the activity [UVA-quest-final]”“the design implies the previous determination of allthese elements and their organization [UCA-quest-final]”
Support data