Aied99 a toolstalk_murray

AIED July 99, Murray ITS Authoring Tools Survey 1

ITS Authoring Tools: an Overview of the state of the art

Tom MurrayUniversity of Massachusetts &

Hampshire College, Amherst, MA

www.cs.umass.edu/~tmurray

• References in Murray 1999, IJAIED 10(1): Authoring Intelligent Tutoring Systems: An analysis of the state of the art


OR:Cottage industry forms as thousands build intelligent tutoring systems in

their basements--NOT YET!

OR:ITS construction:

How Easy Can It Be?


What is an ITS, such that one can be “authored?”– Any CBI system that separates content

(what) from strategy (how)

– Usually makes inferences about “what the student knows”

– I.E. Contains a “model” of domain, strategy, and/or student

• --> We’re talking about pretty basic ITSs


Purposes of ITS Authoring Tools

• (Caveat: Authoring shells vs. authoring tools)

• Cost-effective production of ITSs• Decreased skill threshold for authors• Insure good quality by content validation or

constraining the ITS to a particular model• Allow more participation of practicing educators in

ITS design and evaluation• Provide a test bed for evaluation of alternative

strategy or content models


How many ITS authoring tools have been built?

29 projects

CATEGORY PROJECTS/SYSTEMS1 Curriculum Sequencing and

PlanningDOCENT, IDE, ISD Expert, Expert CML

2 Tutoring Strategies Eon, GTE, REDEEM, SmartTrainer AT3 Device Simulation and

Equipment TrainingDIAG, RIDES, MITT-Writer, ICAT,SIMQUEST, XAIDA

4 Domain Expert System Demonstr8, D3 Trainer, Training Express5 Multiple Knowledge Types CREAM-Tools, DNA, ID-Expert, IRIS,

XAIDA6 Special Purpose IDLE-Tool/IMap, LAT7 Intelligent/adaptive

HypermediaCALAT, GETMAS, InterBook,MetaLinks


How many ITS authoring tools …MORE

29 + 17 = 46 systems

~ 1982 to present

CALAT CAIRNEYDEMONSTR8 TDK, PUPSDOCENT StudyEon KAFITSID EXPERT Electronic Trainer, ISD-ExpertIDLE-Tool IMAP, INDIE, GBS-architecturesREDEEM COCARIDES IMTS, RAPIDS, and see DIAGSIMQUEST SMISLESmart-Trainer AT FITS

Precursor systems

• References in Murray 1999, IJAIED 10(1): Authoring Intelligent Tutoring Systems: An analysis of the state of the art


Overview: Multiple perspectives describing the field

• What kinds of ITSs have been authored? • Authoring the Interface, Domain, Teaching, and

Student Models• What Authoring/Knowledge Acquisition Methods

Have Been Used?• How Are Authoring Systems Designed? (Design

Tradeoffs & Open Issues)• Pragmatics and Use (Are ITS authoring systems

“real?”)


What kinds of ITSs have been authored?

• Both pedagogy-oriented and performance-oriented ITSs

• Seven Types of ITSs

• Tools constrain ITSs


Seven Categories of Authored ITSs

• Strengths, Limits, Variations, student perspective

• Categories 3, 4, & 6 are mostly “performance-oriented”

CATEGORY PROJECTS/SYSTEMS1 Curriculum Sequencing and

PlanningDOCENT, IDE, ISD Expert, Expert CML

2 Tutoring Strategies Eon, GTE, REDEEM, SmartTrainer AT3 Device Simulation and

Equipment TrainingDIAG, RIDES, MITT-Writer, ICAT,SIMQUEST, XAIDA

4 Domain Expert System Demonstr8, D3 Trainer, Training Express5 Multiple Knowledge Types CREAM-Tools, DNA, ID-Expert, IRIS,

XAIDA6 Special Purpose IDLE-Tool/IMap, LAT7 Intelligent/adaptive

HypermediaCALAT, GETMAS, InterBook,MetaLinks


1. Curriculum Sequencing and Planning

Systems: DOCENT, IDE, ISD Expert, Expert CML• Basic and early historical systems• Separates content from presentation and sequencing• Rules, constraints, or strategies for “intelligently”

sequencing content--at the “macro level” (topic level)

• Usually low fidelity interfaces, canned content, simple student models


2. Tutoring StrategiesSystems: REDEEM, Eon, GTE, Smart Trainer AT

#1 above PLUS:• Micro-level and explicit tutoring strategies

– Instructional primitives for hints, explanations, examples. reviews, feedback…

– Instruction can have a more dialogue or conversational feel

• Some include multiple teaching strategies and meta-strategies

• Often have low fidelity interfaces, canned content, simple student models


Tutoring strategies category: Example

REDEEM Genetics Tutor

Content from (ToolBook based) CAI courseware


3. Device Simulation and Equipment Training

Systems: DIAG, RIDES, MITT-Writer, ICAT, SIMQUEST, XAIDA

• Micro-world/simulation-based learning environments• Most focus on equipment/device operation and

maintenance procedures• Building the simulation is time consuming, but much

of the “tutoring” then comes for free.


Examples from RIDES Tutors


4. Domain Expert System

Systems: Demonstr8, D3 Trainer, Training Express• Deep/runnable models of problem solving expertise• Fine grained student diagnosis and modeling• Building an expert system is very difficult -- but then

instruction can come “for free”


D3s Medical Tutor


Demonstr8’s Subtraction Tutor


5. Multiple Knowledge Types

Systems: CREAM-Tools, DNA, ID-Expert, IRIS, XAIDA

• “Gagne Hypothesis:” There are different types of knowledge --> Each has its own instructional methods and representational formalism

• Template-like framework for decomposing content into facts, concepts, and procedures

• Many based on instructional design theory principles• Limited so far to relatively simple facts, concepts,

procedures


6. Special Purpose

Systems: IDLE-Tool/IMap, LEAP Authoring Tool

• Build tutors for a particular type of task• Can provide strong authoring guidance and

constraints• Design and pedagogical principles can be enforced• The task, interface, and pedagogy must fit relatively

inflexibly to the given model


Example: IDLE-Tool:Sickle Cell Counselor


7. Intelligent/Adaptive Hypermedia

Systems: CALAT, GETMAS, InterBook, MetaLinks

• Similar to Category #1 but also deals with Navigation and (dis)orientation issues

• Accessibility and UI uniformity benefits associated with the WWW

• Limited interactivity and learning environment fidelity

• Potential for making inferences from large numbers of students


Example: InterBook


Authoring the Interface, Domain, Teaching, and Student Models

• Interface• Domain Model

– Curriculum knowledge structures

– Simulations of Devices and Phenomena

– Expert Systems

• Teaching Model• Student Model


1. Authoring the Interface• Systems with interface authoring tools:

RIDES (below), Eon, SIMQUEST RIDES


Example 2

Eon’s Interface Editor

EON


2. Authoring the Domain model

Curriculum knowledge and structures

Simulations/models of devices and

phenomena

Domain Expertise models (expert system)


Authoring Curriculum Knowledge and Structures

• Topics/KUs• Relationships

(e.g. prerequisite)

• Knowl. Type(concept, procedure…)

• Objectives• Importance• Difficulty

Eon


Example: IRIS


Example: CREAM Tools


Authoring Simulations of Devices and Phenomena

XAIDA


Example 2 RIDES


Authoring Domain Expertise (Expert systems): D3 Trainer


3. Authoring the Teaching Model Example: REDEEM


Authoring the Teaching Model Example 2: Eon


4. Authoring the Student Model

Eon’s SM Editor


What Authoring/Knowledge Acquisition Methods Have Been Used?

• 1. Scaffolding knowledge articulation with models• 2. Embedded knowledge and default knowledge• 3. Knowledge management• 4. Knowledge visualization• 5. Knowledge elicitation and work flow

management• 6. Knowledge and design validation• 7. Knowledge re-use• 8. Automated knowledge creation


1. Scaffolding knowledge articulation with models

• Ex. 1: Templates: IDLE-Tools

• Ex. 2: Ontology-Aware tools: SmartTrainer AT


REDEEM

2. Embedded knowledge and default knowledge


3. Knowledge management

• Topics/KUs• Lesson Objectives• Interface objects & screens• Exercises, examples, pictures• Teaching Strategy actions

CALAT


4. Knowledge visualization

• LEAP-AT


5. Knowledge elicitation and work flow management

• Author: “What do I do next?” “ Where do I start?”• Prompts in ID-Expert and DNA: “Which of the

following describes what the student will learn: a. What is is? B. How to do it? C. How does it work?”

• Top down vs opportunistic design– DNA: Semi-structured interactive dialog has

prompts with choices• REDEEM: Agenda mechanism for authoring tasks


6. Knowledge and design validation

• Opportunistic & Open ended --> more flexibility & more errors

• Constraint-based advice: – “The estimated time for all Lesson-2 topics exceeeds the

estimated time for Lesson-2”

– “The engine maintenance procedure has no sub-steps defined”

– “Lesson-3 objectives include procedural and conceptual knowledge, but there are no conceptual topics linked to Lesson-3.”


7. Knowledge re-use

• Libraries of Content, Graphics, Strategies, etc.

• Flexible reconfiguration of components


8. Automated knowledge creation• Example-Based programming

– Inferring a general procedure/rule from an example procedure/rule

DEMONSTR8


Ex.2: Automated knowledge creation

faultyRU1 Abnormal

Outcome A1

4

1

24

5

Indicator A

Indicator B

NORMAL

NORMAL

Outcome A2

AbnormalOutcome B1

AbnormalOutcome B2

Outcome B3

3faultyRU2

faultyRU3

faultyRU1

faultyRU2

faultyRU3

1

6 ALWAYS5 USUALLY4 VERY_OFTEN3 OFTEN AS NOT2 SOMETIMES1 RARELY0 NEVER

1

6

2

3

5

DIAG


Suggestions for a Full-Featured Authoring System

• Visual reification of conceptual and structural elements

• Assistance: design steps or agenda; constraint-based validation

• Content reusability and object libraries• Scriptable and customizable• WYSIWIG editing, Opportunistic design, Easy

design/test iteration (interpreted vs compiled), Reasonable default values– for rapid prototyping


How Are Authoring Systems Designed?

Design Tradeoffs & Open Issues

• The space of design tradeoffs

• General vs. special purpose authoring systems

• Who are the authors?

• Who should author ITS instructional strategies?

• Meta-Level Authoring


The Space of Design Tradeoffs

DomainModel

TutoringStrategy

StudentModel

LearningEnvironment

Power/ Breadth

Flexibility Depth

LearnabilityUsability Productivity

Fidelity

Cost

[The design space has 24 (6x4)

independent dimensions or axes.]

DomainModel

TutoringStrategy

StudentModel

LearningEnvironment

Power/ Breadth

Flexibility Depth

LearnabilityUsability Productivity

Fidelity

Cost

[The design space has 24 (6x4)

independent dimensions or axes.]


General vs. special purpose authoring systems

• E.G. special purpose systems: LAT and IDLE-Tool– Greater usability, fidelity, depth -- but only for design

goals that match the tools.

– Does the “demand” balance the inflexibility?

– How to make more customizable while maintaining ease of use?

• Types of abstraction/specialization?– 1. Real-world tasks

– 2. Abstract tasks

– 3. Knowledge types


Abstracting ITSs for special purpose authoring systems• 1. Abstracting real-world tasks:

Investigate & Decide; Evidence-Based reporting; Run an Organization

• 2. “Abstract tasks:” Equipment operation & maintenance (RIDES); Conversational Grammars (customer service; LAT)

• 3. Knowledge types: Facts, concepts, procedures, principles (CREAM-Tools, DNA, ID-Expert, XAIDA)


Who are the authors? What level of skill & training should be expected?

• Authoring skill sets: instructional design, classroom pragmatics, graphics/UI, domain knowledge, knowledge engineering, script-level programming...

• IDLE, XAIDA, REDEEM: try to allow authoring by teachers and “off the street” domain experts with minimal training


Suggested authoring scenario

• Effort level: Building an ITS is more like writing a book than creating a greeting card!

• Skill level: Skill level equivalent: Accounting applications, CAD, spreadsheet macros, 3-D modeling, advanced Photoshop…-- Special training but reasonable

• Sophistication level: Authors need to look at the big picture and do ongoing quality assessment of what they build

• ITSs are built by design teams, not individuals (distributed skill sets)


Who should specify/author ITS instructional strategies?

PROS CONSTeachers

PRACTICALPractical experience Not good at articulating or

abstracting expertiseInstructional Designers

ANALYTICTheories are widely used in

some circlesLimited to basic knowledge types

that are easily representedPsychologists

THEORETICALKnow “how the mind works” Use 'first principles'—only useful

for simple knowledge structuresEducational researchers

EMPIRICALEmpirical studies of tutoring

and classroomsAfter many years still don't

agree on muchComputer scientists

(ACTUAL?!)...end up building the systems… “Isn’t it just all common sense?”…

Domain Experts(I.E. NO acquisition of

instructional knowledge

Experts just show how they do atask & authoring tool infers the

instructional methods

Fixed instructional method


Meta-Level Authoring

• Custom/extensible interface widgets

• Customizable descriptive vocabulary

• Pre-configured tutoring strategies and student models

Eon


Use & Pragmatics (Are ITS authoring systems “real?”)

• Authoring system Use

• Authoring system Productivity

• Authoring system Evaluation


Authoring Tool Use

Examples: • XAIDA domains: equipment operation and maintenance, algebra, medicine, computer literacy, biology

• IDLE-Tool: three informal trials with 21, 8, 8 grad student and grade school teacher authors

1. Early prototypes andproofs of concept

D3 Trainer, Demonstr8, DIAG, IRIS,Expert-CML, SmartTrainer AT

2. Evaluated or usedprototypes

CREAM-Tools, DNA, Eon, GTE,IDLE-Tool, LAT

3. Moderately evaluatedor used

ISD-Expert/Training Express,REDEEM, SIMQUEST, XAIDA

4. Heavily used(relatively)

IDE, CALAT, RIDES


(Relatively) Heavily Used Authoring Tools

• Build a dozen or more ITSs• Many ITSs used in real educational settings• Robust enough for use independent of original design

team

• RIDES: many project spin-offs and diverse domains• CALAT: over 300 Web-based courses (used at NTT)


Authoring Tool Productivity• For traditional CAI: Estimated 300:1 ratio of

development to instruction time• ITS authoring: goals and some spotty evidence

– ID-Expert’s goal: 30:1

– XAIDA’s goal 10:1; evidence of a first time user at 16:1

– KAFITS Physics tutor w/ six hours of instruction: 100:1

– CALAT: ITS development in about the same time as traditional instruction

– REDEEM: 2:1 to segment CAI content & make intelligent

• Implication: AI Knowledge Representation does provide ITSs with inherent efficiencies


Authoring Tool Evaluations

• Existence proofs: Usability; Productivity; Breadth

• Examples:– XAIDA

– REDEEM

– IDLE-Tools, COCA, LAT, KAFITS, DNA


Evaluations of XAIDA

• Eight authoring field studies with average of 10 instructor participants each

• 13 studies of students using the built tutors

Data: • Learnability: abilities assessment, self-report skills,

cognitive assessment, task-based performance

• Acceptability: open-ended questionnaire

• Productivity: use analysis

• Usability: questionnaire


XAIDA EvaluationValence of Comments

Across Training

0%

20%

40%

60%

80%

100%

Beforetraining

End of Day1

End of Day2

End of Day3

Aftertraining

Neutral

Negative

Positive

Frequency of Comments


Proficiency Using XAIDA

1

2

3

4

5

6

7

8

9

10

Before training End of Day 1 End of Day 2 End of Day 3 End of Day 4 After trainingNOVICE

EXPERT


Task Time SpentTraining 2 hoursCourse familiarisation 1 hourDescribing pages and sections 4 hoursReflection points & non-computer-basedtasks

1 hour

Authoring questions 2 hoursClassifying students 15 minsDeveloping teaching strategies 15 minsRelating students to sections 15 minsRelating students to strategies 5 minsTotal 10 hours 50 mins

REDEEM Evaluation:• 1 SME author, 3 teacher authors, 7 “virtual students”• Data: authoring sub-task time, variations among

authors, appreciation of added “intelligence”

Time spent by teacher practitioner:


Some Formative Evaluation Results

(And see Productivity and Use above):

• Authors’ cognitive model of the domain had a structure closer to SME’s after tool use (XAIDA)

• Considerable difference between authors in content structure, strategy specification, categorization of students (REDEEM)

• Teacher reactions in general positive but difficulty with complex relationships among content pieces

• Teachers thought AI technology could simulate reasonable teaching strategies(COCOA)

• Tools needed to give users abstract view of the content (IDLE-Tools)


Some Formative Evaluation Results (cont.)

• Including examples for design steps/information was very helpful (IDLE-Tools)

• Graphic representations for knowledge elicitation much less error-prone than text-based (LAT)

• Overestimated of the level of expertise authors would gain in a short amount of time (LAT)

• Authors have difficulty conceptualizing non-linear, modular content (KAFTIS)

• Comparing automated knowledge elicitation to coded-by-hand task analysis: automated method covered most of the domain knowledge in a small fraction of the time (DNA)


Summary

• Many types of ITSs have been “authored”• Wide variety of knowledge acquisition and authoring

methods have been used. Too early to know when each is most appropriate.

• Some tools have significant use and a few are in commercial or near-commercial form

• Promising results in from evaluations of usability and productivity, with more rigorous evaluations just starting

• What are the foreseeable limits?


Conclusions--How Easy Can It Be?

• There are limits!

• Limited use of cookie-cutter special purpose authoring tools-- too restrictive for most authors

• Limited ability to reduce ITS authoring to easy, small, independent steps (recipes)

• Authors need to think about the big picture and need skills and tools to do this


...Back to the Future

• Customizability requirements will usually lead to the author specifying BEHAVIORS (choices, rules, algorithms) as well as static information

• This requires ability to RUN, test, and modify these behaviors

• This is (simple) PROGRAMMING• Debugging skills and tools will be needed! (Tracing,

stepping, inspecting states, etc.)


------------------------------------


ITS Authoring Tools: an Overview of the state of the art

Tom MurrayUniversity of Massachusetts &

Hampshire College, Amherst, MA

www.cs.umass.edu/~tmurray


People #1

CALAT (&CAIRNEY)

Kiyama, M., Ishiuchi, S., Ikeda, K., Tsujimoto, M. & Fukuhara, Y.(1997).

CREAM-TOOLS Frasson, C., Nkambou, R., Gauthier, G., Rouane, K. (1998).Nkambou, R., Gauthier, R., & Frasson, M.C. (1996).

D3-TRAINER Reinhardt, B., Schewe, S. (1995).DEMONSTR8 (&TDK, PUPS)

Blessing, S.B. (1997). Anderson, J. R. & Pelletier, R. (1991).Anderson, J. & Skwarecki, E. (1986).

DIAG Towne, D.M. (1997).EON (& KAFITS) Murray, T. (1998,1996).IDLE-Tool (&IMAP, GBS-archits)

Bell, B. (1999). Jona, M. & Kass, A. (1997).

INTERBOOK (&ElM-Art)

Brusilovsky, P., Schwartz, E., & Weber, G. (1996).

IRIS Arruarte, A., Fernandez-Castro, I., Ferrero, B. & Greer, J. (1997).LAT (LEAPAuthoring Tool)

Sparks, R. Dooley, S., Meiskey, L. & Blumenthal, R. (1999).Dooley, S., Meiskey, L., Blumenthal, R., & Sparks, R. (1995).

REDEEM (&COCA)

Major, N., Ainsworth, S. & Wood, D. (1997). Major, N.P. & Reichgelt,H (1992).

RIDES (& IMTS,RAPIDS, DIAG)

Munro, A., Johnson, M.C., Pizzini, Q.A., Surmon, D.S., Towne, D.M,& Wogulis, J.L. (1997). Towne, D.M., Munro, A., (1988).

Smart TrainerAT (& FITS)

Jin, L, Chen, W., Hayashi, Y., Ikeda, M. Mizoguchi, R. (1999); Ikeda,M. & Mizoguchi, R. (1994)

XAIDA Hsieh, P., Halff, H, Redfield, C. (1999). Wenzel, B., Dirnberger, M.,Hsieh, P., Chudanov, T., & Halff, H. (1998).


People #2

DNA/SMART Shute, V.J. (1998).DOCENT (&Study)

Winne P.H. (1991). Winne, P. & Kramer, L. (1988).

EXPERT-CML Jones, M. & Wipond, K. (1991).GETMAS Wong, W.K. & Chan, T.W. (1997).GTE Van Marcke, K. (1998,1992).ID EXPERT (&ElectronicTrainer)

Merrill, M.D., & ID2 Research Group (1998). Merrill, M. D. (1987).

IDE (& IDEInterpreter)

Russell, D. (1988). Russell, D., Moran, T. & Jordan, D. (1988).

MetaLinks Murray, T., Condit, C., & Haaugsjaa, E. (1998).SIMQUEST (&SMISLE)

Jong, T. de & vanJoolingen, W.R. (1998). Van Joolingen, W.R. &Jong, T. de (1996).

TRAININGEXPRESS

Clancey, W. & Joerger, K. (1988).


Date post:	05-Dec-2014
Category:	Education
Upload:	perspegrity5
View:	55 times
Download:	2 times