TELOS TESTBED – 2006TELOS TESTBED – 2006EvaluatingEvaluating Learning Design Learning Design

ToolsTools

Karin Lundgren-CayrolKarin Lundgren-CayrolIsabelle SavardIsabelle SavardMarcello MainaMarcello MainaClaire BanvilleClaire Banville

En collaboration avec:En collaboration avec:Theme 6.3 France Henri & Josianne BasqueTheme 6.3 France Henri & Josianne Basque

OutlineOutline• 2006 - Testbed Context, Goals and Objectives2006 - Testbed Context, Goals and Objectives• General Evaluation Plan & Formative General Evaluation Plan & Formative

Evaluation DesignEvaluation Design• Participants Participants • Hands-on Evaluation ScenarioHands-on Evaluation Scenario• ResultsResults

– Background questionnaireBackground questionnaire– Testbed Evaluation Scenario Testbed Evaluation Scenario – Final QuestionnaireFinal Questionnaire

• Preliminary conclusionsPreliminary conclusions• Connected Student Thesis WorkConnected Student Thesis Work

– Isabelle Savard (Isabelle Savard (Poster78 : Cultural Diversity and Poster78 : Cultural Diversity and Reuse of Learning Objects)Reuse of Learning Objects)

– Marcelo Maina (Marcelo Maina (Adapting MISA to IMS LearningAdapting MISA to IMS Learning Design)Design)

• QuestionsQuestions

GoalGoalGoalGoalTo identify strengths and weaknesses of the TELOS Instructional Design Interface

potential Tools

Cycle3: IDLD METHOD

Objectives 2006Objectives 2006Objectives 2006Objectives 2006• To To test toolstest tools that are scheduled to be that are scheduled to be

incorporated into the TELOS instructional incorporated into the TELOS instructional design Interface. This year:design Interface. This year:– MOT+LD editorMOT+LD editor– Canadian LD Canadian LD RepositoryRepository– Learning Design MethodologyLearning Design Methodology

• To elicit information about To elicit information about practitioners ways practitioners ways and needsand needs in terms of facilitating their in terms of facilitating their instructional design tasks.instructional design tasks.

• To To send results to concerned teamssend results to concerned teams about their about their tools in regards to their integration into TELOS.tools in regards to their integration into TELOS.– Team 6.3 – ID’s opinions, tasks and perceived competency Team 6.3 – ID’s opinions, tasks and perceived competency

profileprofile– Team 2.1 and 6.2 – Revisions of processes, tools and guidesTeam 2.1 and 6.2 – Revisions of processes, tools and guides

Protocol & Testbed Protocol & Testbed CycleCycle

ReviseTELOS

Core

ReviseLKMS

LKMS

ReviseLKMA

LKMA

ReviseLKMA

LKMP

Identify needsIdentify needs Create testbed scenario

Create testbed scenario

DisseminateDisseminateCarry outtestbed

Carry outtestbed

Analyse results

Analyse results

Create testbed protocol

Create testbed protocol

demandsdemandsdemandsdemandsdemandsdemands

Context 2006 Context 2006 Context 2006 Context 2006

TELOS consists of four different interfaces and toolsets according to actor

Engineer (TELOS Kernel) Technician (LKMS) Instructional Designer (LKMA) Learner and Teacher Interfaces (LKMP)

Call for ParCall for ParticipantsticipantsCall for ParCall for Participantsticipants

Requirements: Experience in designing online learning /

e-learning Good computer knowledge (Windows) Experience with web-based technology

TTestbed Agenda & Instrumentsestbed Agenda & Instruments Complete Background Information

concerning (15 min) Work Context Technical Profile and Tool Usage ID Experience and Practice

Carry out the Evaluation Scenario Demo Session (1h30) Hands-on (5 h)

Complete a Final Questionnaire (30min) Participate in a Focus Group (1h)

Let’s visit the Let’s visit the IDLDIDLD site siteAnd carry out this And carry out this

EVALUATION ScenarioEVALUATION Scenario

http://www.idld.orghttp://www.idld.org

Some Preliminary ResultsSome Preliminary Results

CCompletion Ratesompletion RatesCCompletion Ratesompletion Rates

Background questionnaireBackground questionnaire (N=11) 9 Mauritians (Validation of instrument)

11 Canadians

Demo sessionDemo session (N=9) 1 in LORIT; 6 = ClickToMeet; 2 by LORIT Webcast & chat; 1 used 2 abandon

Evaluation ScenarioEvaluation Scenario (N = 6 - 1) 2 abandon; 1 finished but did not return it; 3 about half; 1 all but adapting UoL; 1 all the tasks

Final Questionnaire (Final Questionnaire (N = 6)4 most questions2 less than half

Focus Group Focus Group = cancelled

Background questionnaireBackground questionnaire (N=11) 9 Mauritians (Validation of instrument)

11 Canadians

Demo sessionDemo session (N=9) 1 in LORIT; 6 = ClickToMeet; 2 by LORIT Webcast & chat; 1 used 2 abandon

Evaluation ScenarioEvaluation Scenario (N = 6 - 1) 2 abandon; 1 finished but did not return it; 3 about half; 1 all but adapting UoL; 1 all the tasks

Final Questionnaire (Final Questionnaire (N = 6)4 most questions2 less than half

Focus Group Focus Group = cancelled

Background Information - Background Information - ContextContext

Mauritius (N=9) F2F interviews allowed validation of BQ All worked at the University (7 ID; 2 tutors) 3 designed in English, 2 in French, and 4 in Both languages 5 had LO repository access

Canada (N=11) Geographically distributed (contact by email and phone) 8 University, 2 College and 1 Government 4 Prof/Teach; 5 ID; 1 Tutor ; 1 Training Manager 7 designed in English, 1 in French, & 3 Both languages 2 LO access; 2 under development; 6 NO &1 didn’t know

Background Information – Background Information – Context Context (1)(1)

Variablevalue

MauritiusN=9

CanadaN=11

Age group20 to 3031 to 50

50+

540

434

Mother TongueEnglishFrenchOther

11

7 Creole

63

2 Chinese

ProfessionProfessor/Teacher

Instructional Designer Trainer/Tutor

Training Manager

0720

4511

Context (2)Context (2)Variable Value

Mauritius Canada

OrganizationUniversity

CollegeHigh SchoolGovernment

9000

8201

Design LanguageEnglishFrench

Both

324

633

LO Repository AvailabilityYesNo

Don’t know

5/800

281

Tendency:Tendency: Instructional designers work in any discipline, teachers referred to their domain

expertise.

Background Information – Background Information – Canadians Tool UsageCanadians Tool Usage

10/11 mostly used Windows 6/11 used IE, 4 Mozilla & 1 Netscape

Solve computer problems (pick one method)

MAURITIUS(N=9)

Most of the time

Some-times

Rarely Never

Read manuals 1 1

Online help 1 2

Web Forums 0

Human resources

7Solve computer problems For each method indicate your preferred use

Canada(N=11)

Most of

the time

Sometimes

Rarely Never T

Read manuals 4 5 0 2 11

Online help 8 1 1 1 11

Web Forums 4 4 2 1 11

Human resources

8 2 1 0 11

Preferred Help Method Preferred Help Method

Type method to learnCana

da(n-11)

Mauritius

(n=9)

TotalN=20

Read printed manuals 2 2 4 (20%)

Explore on my own 4 2 6 (30%)

Ask an expert 2 2 4 (20%)

Take a training course 2 2 4 (20%

Use a guided tour 1 1 2 (20%

Give up and leave it 0 0 0 (0%)

Conclusion: Telos must provide a plethora

of advisory systems as well as having

human resources available through the use of sophisticated communication tools.

Tool Usage (1)Tool Usage (1)Software Name Canada

(N=11)Mauritius

(N=9)

Often Never Often Never

Word processor 11 0 9 0

Spreadsheets 4 1 5 0

PowerPoint 4 1 7 0

Communication Tools

Videoconference 7 2 0*(3) 4

Textual forums 7 2 5 1

Community of practice 5 4 2 3

News Group 4 7 2 4

Chat 7 3 6 1

Wiki 6 2 - -

Blog 5 3 - -

Tool Usage (2)Tool Usage (2)

Software CanadaN=11

Often Never

MauritiusN=9

Often Never

Concept Mapping Software 5 5 3 3

Internet Search Tools 11 0 9 0

HTML Autorhing Software 8 0 6 1

Webbased platform with LOR 3 2 1 4

without LOR 8 1 4 2

Tool Usage (3) LO and Tool Usage (3) LO and RepositoryRepository

Type of usage

Software NameCanada N = 11

IHELPMOT+L

D RELOAD PALOMA MERLOT

Never heard about it

5 4 4 7 3

I know about it 3 3 2 0 2

Seen demo 0 1 0 0 0

Used once Use regularly

1 2 2 4 2

2 1 3 0 4

ConclusionConclusion:

Most LO, IMS LD related tools are not widely known.

Background Information – Background Information – ID Experience and Practice ID Experience and Practice (1)(1)

Canada 5/11 had more than 6 years 3/11 had between 3 and 6 years 3/11 less than 2 years

Instructional Design Method C M %

Formal method 1 0 9%

Adaptation of formal method 4 5 36%

Imposed by institution 2 2 18%

My own method 3 2 27%

Total 10 9

ConclusionConclusion: TELOS must be able to provide flexible instructional design environments.

Mauritius 1/9 had more than 6 years 7/9 had between 3 and 6 years 1/9 less than 2 years

16/20 more than 3 years

Reusing LO’sReusing LO’s

ConclusionConclusion: : LO’s are used in different ways LO’s are used in different ways and are starting to be a well known and are starting to be a well known concepts.concepts.

Use existing LO’s in the following manner

CanadaN=11

MauritiusN=9

TotalN=20

To create a new course 1 (9%) 6 (55%) 7 (35%)

To adapt an existing course

3 (27%) 0 (0%) 3 (18%)

Both 7 (64%) 3 (30%) 10 (50%) How do you use existing LOs

CanadaN=11

MauritiusN=9

TotalN=20

As is 1 0 1 (5%)

As an inspiration 4 0 4 (20%)

All possibilities 6 9 15 (75%)

Validation process and Validation process and resultsresults

Validation process in Mauritius (Savard, 2006)Validation process in Mauritius (Savard, 2006)• First version of the background First version of the background

questionnairequestionnaire• Semi-structured interviewsSemi-structured interviews to find out their to find out their

regular ID process and tasksregular ID process and tasks• Similar to Similar to Gagné’s ADDIEGagné’s ADDIE Instructional Instructional

Design Model.Design Model.

ResultsResults::• Comment on each question which led toComment on each question which led to

– changes to Que’s 5, 10 & 21changes to Que’s 5, 10 & 21– addition Que’s 17, 19 & 22 (ADDIE)addition Que’s 17, 19 & 22 (ADDIE)

ADDIEADDIE - Instructional Design - Instructional Design TaskTask

• AAnalysisnalysis

• DDesignesign

• DDevelopment/Productionevelopment/Production

• IImplementationmplementation

• EEvaluationvaluation

AADDIEDDIE - - Instructional Design Instructional Design TaskTask

•AAnalysisnalysis

ConclusionsConclusions

In general, the Analysis task is already done when a In general, the Analysis task is already done when a demand comes to Mauritians Instructional Designer’s.demand comes to Mauritians Instructional Designer’s.

They often have to reformulate objectives.They often have to reformulate objectives.

For the Canadians, the Analysis is an important task.For the Canadians, the Analysis is an important task.

AADDDIEDIE - - Instructional Design Instructional Design TaskTask

• DDesignesign

ConclusionsConclusions: : • Instructional Structure is an important task for both Instructional Structure is an important task for both

ID from Mauritius and Canada.ID from Mauritius and Canada.

ADADDDIIEE - - Instructional Design Instructional Design TaskTask

• DDevelopment/Productionevelopment/Production

Conclusions:Conclusions:• Material production is an important task in both Mauritius and Material production is an important task in both Mauritius and

Canada.Canada.• In Mauritius the delivery and maintenance plan seem not to be In Mauritius the delivery and maintenance plan seem not to be

their their responsibilityresponsibility..

ADDADDIIEE - - Instructional Design Instructional Design TaskTask

• IImplementationmplementation

• Conclusions:Conclusions:• Small field testing doesnt take too much Small field testing doesnt take too much

time for ID from both contries.time for ID from both contries.

ADDIADDIEE - - Instructional Design Instructional Design TaskTask

• EEvaluationvaluation

ConclusionsConclusions• The course evaluation is not a major concern in The course evaluation is not a major concern in

Mauritius.Mauritius.• 70% of Canadians do carry out this task70% of Canadians do carry out this task

ID Competencies (ibstpi)ID Competencies (ibstpi)

• Four groups of Competency statements for Four groups of Competency statements for instructional designersinstructional designers::– A- Professional FoundationsA- Professional Foundations– B- Planning and AnalysisB- Planning and Analysis– C- Design and DevelopmentC- Design and Development– D- Implementation and ManagementD- Implementation and Management

• Validate which competencies were Validate which competencies were the most important in their job the most important in their job situation.situation.

http://www.ibstpi.org/

ID Competencies (ibstpi)ID Competencies (ibstpi)ResultsResults

A. Professional FoundationsA. Professional Foundations MaMauu

CaCann

TotalTotal

1- 1- CommunicateCommunicate effectively in visual, oral effectively in visual, oral and written form.and written form.

99 1111 2020

2- Apply current research and theory to the 2- Apply current research and theory to the practice of instructional design.practice of instructional design.

88 77 1515

3- Update and improve one’s knowledge, 3- Update and improve one’s knowledge, skills and attitudes pertaining to skills and attitudes pertaining to instructional design and related fields. instructional design and related fields.

88 44 1212

4- Apply fundamental research skills to 4- Apply fundamental research skills to instructional design projects. instructional design projects.

66 44 1010

5- Identify and resolve 5- Identify and resolve ethical and legalethical and legal implications of design in the work place. implications of design in the work place.

55 33 88

ID Competencies (ibstpi) - ResultsID Competencies (ibstpi) - Results

B. Planning and AnalysisB. Planning and Analysis MaMauu

CaCann

TotalTotal

6- Conduct a needs assessment. 6- Conduct a needs assessment. 77 66 1313

7- Design a curriculum or program. 7- Design a curriculum or program. 77 66 1313

8- Select and use a variety of techniques for 8- Select and use a variety of techniques for determining instructional content. determining instructional content.

77 44 1111

9- Identify and describe target population 9- Identify and describe target population characteristics. characteristics.

88 55 1313

10- Analyze the characteristics of the 10- Analyze the characteristics of the environment. environment.

55 77 1212

11- Analyze the characteristics of existing 11- Analyze the characteristics of existing and emerging technologies and their use in and emerging technologies and their use in an instructional environment. an instructional environment.

88 66 1414

12- Reflect upon the elements of a situation 12- Reflect upon the elements of a situation before finalizing design solutions and before finalizing design solutions and strategies. strategies.

88 66 1414

ID Competencies (ibstpi)ID Competencies (ibstpi)ResultsResults

C. Design and DevelopementC. Design and Developement MaMauu

CaCann

TotalTotal

13- Select, modify, or create a design and 13- Select, modify, or create a design and development model appropriate for a given development model appropriate for a given project. project.

77 99 1616

14- Select and use a variety of techniques to 14- Select and use a variety of techniques to define and sequence the instructional define and sequence the instructional

content and strategies.content and strategies.

77 77 1414

15- Select or modify existing instructional 15- Select or modify existing instructional materials. materials.

99 77 1616

16- Develop instructional materials. 16- Develop instructional materials. 99 55 141417- Design instruction that reflects an 17- Design instruction that reflects an understanding of the diversity of learners understanding of the diversity of learners and groups of learners. and groups of learners.

66 44 1010

18- Evaluate and assess instruction and its 18- Evaluate and assess instruction and its impact. impact.

88 55 1313

ID Competencies (ibstpi)ID Competencies (ibstpi)ResultsResults

D. Implementation and D. Implementation and ManagementManagement

MaMauu

CaCann

TotalTotal

19- Plan and manage instructional design 19- Plan and manage instructional design projects. projects.

44 88 1212

20- Promote collaboration, partnerships and 20- Promote collaboration, partnerships and relationships among the participants in a relationships among the participants in a design project. design project.

77 77 1414

21- Apply business skills to managing 21- Apply business skills to managing instructional design. instructional design.

22 55 77

22- Design instructional management 22- Design instructional management systems. systems.

33 44 77

23- Provide for the effective implementation 23- Provide for the effective implementation of instructional products and programs. of instructional products and programs.

66 44 1010

Evaluation ScenarioEvaluation Scenario Variable Yes No T

Intro to IMS LD motivating 3 2 5

Methodological Guide sufficient 5 0 5

Preferred type of resource to learn new subjectsPapers

PresentationsGuided Tours

041

5

Trouble occurred during Download of MOT+ Installation of MOT+

10

23

3

Modeling technique sufficient to get started 3 0 3

Troubles occurred accessing PALOMA 1 2 3

Narrative description Useful for reuse

Helps understand graphical UoL33

00 3

Rubric's Evaluation Grid is valuable for reuse 3 0 3

ID Principles doc is valuable for reuse 2 1 3

Comments and SuggestionsComments and Suggestions

• Experience too Experience too time consumingtime consuming because of because of all the documents to be studiedall the documents to be studied

• Several separated Demo Sessions are Several separated Demo Sessions are necessary necessary

• IMS LD very IMS LD very complexcomplex • MOTPlusMOTPlus has has significant advantagessignificant advantages • GraphicalGraphical modeling is interesting modeling is interesting• All tools were very interestingAll tools were very interesting

Final Questionnaire Final Questionnaire

IDLD PORTALIDLD PORTALFeature (n = 6 to 7)Feature (n = 6 to 7) PositivPositiv

eeAnsweAnswe

rsrs

NeutrNeutralal

NegativNegative e AnswersAnswers

OrganisationOrganisation SimpleSimple

ClearClear

AttractiveAttractive

ComprehensiveComprehensive

MotivatingMotivating

44

77

77

44

66

22

00

00

22

00

Interface Text Interface Text

Uses clear VocabularyUses clear Vocabulary

Well adapted to target audienceWell adapted to target audience

Is free of jargonIs free of jargon

Highlights the activityHighlights the activity

Is motivatingIs motivating

44

44

44

44

33 11

33

22

33

22

33Conclusion: Very divided opinions, we need more evaluators!!

MOT+LD and Graphical MOT+LD and Graphical ModelingModeling

Usability (N=4)Usability (N=4) PositivePositiveAnswerAnswerss

NeutralNeutral NegativNegative e AnswerAnswerss

Well designedWell designed 22 11 11

Responds to my needsResponds to my needs 22 11 00

Is usefulIs useful 44 00

Is functionalIs functional 44 00

Is transparentIs transparent 33 11

Opens new design possibilitiesOpens new design possibilities 22 22 00

I could easily go from one model to I could easily go from one model to another without being lost or another without being lost or confused confused

22 22

The links are helpful and pertinent The links are helpful and pertinent 33 11 00

Sub-models are a very good way of Sub-models are a very good way of

viewing detailed informationviewing detailed information 44 00

ConclusionConclusion: At least useful and functional for those who tried

PALOMA Resource ManagerPALOMA Resource Manager

Usability (n=4 or 5)Usability (n=4 or 5) ++++ NeutrNeutralal

----

Well designedWell designed 22 22

Responds to my needsResponds to my needs 11 22 11

Is usefulIs useful 22 22 00

Is functionalIs functional 44 00

Is easy to useIs easy to use 22 11 11

Has easy to understand terminologyHas easy to understand terminology 33 11

Asset to my organisationAsset to my organisation 00 33 11

Facilitates reuseFacilitates reuse 22 22

Motivates me to share resourcesMotivates me to share resources 22 11 11

Has a very helpful user guideHas a very helpful user guide 33 11

I would suggest this tool to my colleguesI would suggest this tool to my collegues 11 11 22Suggestions: Need a more user-friendly version We need more reliable federated search protocols Advanced search is a bit confused

Conclusions: Divided opinions and Needs more evaluators!!

Some Preliminary Some Preliminary ConclusionsConclusions

• The ADDIE Model and ibstpi competencyThe ADDIE Model and ibstpi competency– Generally practiced by ID’s (few differences)Generally practiced by ID’s (few differences)– Questions whether Planning and Evaluation is Questions whether Planning and Evaluation is

an ID task?an ID task?• TELOS must be able to provideTELOS must be able to provide

– Sophisticated communication tools to simulate Sophisticated communication tools to simulate human resource contacthuman resource contact

– flexible instructional design environmentsflexible instructional design environments• Confirms complexity of the IMS LD conceptsConfirms complexity of the IMS LD concepts

– Need to adjust vocabularyNeed to adjust vocabulary– Provide transparent & flexible toolsProvide transparent & flexible tools– Provide more examples in the Canadian LD Provide more examples in the Canadian LD

RespositoryRespository