Modeling complex tasks in the context of educational systems Tutorial B2 - 10h30-12h Pierre...

Modeling complex tasks in the context of educational systems

Tutorial B2 - 10h30-12h

Pierre Tchounikine

2Tutorial - ICCE/ICCAI ’2000 - Pierre Tchounikine

Pierre Tchounikine

[email protected]

Professor of computer science - University of Le Mans (France)

Director of the LIUM lab.

Knowledge engineering & educational systems

Personal Research interests:

knowledge engineering (modeling of complex tasks)

intelligent advisory systems

collaborative systems


Introduction (AI and educational systems)

From the very first steps of Artificial Intelligence (AI), the construction of educational systems has been considered as a « natural » field

educational systems should be able to

solve complex exercises and present their solving to students

manage students

understand students’ actions

manage interactions

etc.



AI in education encountered the same difficulties as in the other application fields

things are much more complicated than expected

what can be done is far away from human competence

Problems are amplified by the socio-cultural feeling on AI / education

“attempting to model the very specific type of interactions that exist between a teacher and a student ?”

“attempting to replace human teachers by machines ?”



Many objectives that were initially tackled by AI techniques are no longer considered or approached by other means

from natural language understanding to interactive interfaces

from tutoring systems to collaborative environments

etc.

However, modeling complex tasks remains a task that appears recurrently when constructing educational systems

(and studying problem-solving in the context of educational system still participates in the evolution of AI)


Introduction (From AI to Knowledge Engineering)

Modeling complex tasks has essentially been studied in the context of problem-solving systems (“expert systems”)

the first steps (70’s-80’s) appeared promising

but the systems appeared nevertheless very limited

different works attempted to by-pass the apparent problems that limited the competence of these systems ...

knowledge representation language (production rules, frames, etc.)

inference-mechanisms power

verification of knowledge-bases coherence

etc.

… with very limited results



Researchers finally concluded that what went wrong was not only in how the knowledge is represented or manipulated, but mainly on how it is acquired

(which conditionates what can be manipulated and how)

The focus is on “Knowledge Engineering” (KE)

Constructing problem-solving systems is now seen essentially as a modeling problem



KE has developed engineering approaches

modeling approaches (how to tackle the modeling)

technical approaches (how to operationalize the model)

How can these approaches be applied to modeling complex tasks in the context of educational systems ?


Structure of the presentation

Part I: Modeling complex tasks, different contexts

Part II: From the production-rules paradigm to knowledge engineering approaches

Part III: The Task-Method paradigm




Part II: From the production rules paradigm to knowledge engineering approaches



Modeling complex tasks: different contexts

Prototypical context of modeling a complex task in an educational system: present how the system solves a (type of) problem

(implicitly: how the student should solve the problem)

examples:

solving a mathematical problem

constructing a program

etc.

However, other contexts require modeling a complex task !



Modeling a competence in order to present it to the student

present the task that is to be realized by the student (a scenario)

present how the system solves a (type of) problem / how students should

Modeling some of the educational system functionalities

students’ actions diagnosis

management of interactions

Modeling the task that is to be realized by the teacher

context: framework to construct educational systems

explicit what is to be realized by the teacher, construct advisory systems













Modeling a scenario

A scenario defines a way of using the domain knowledge to be learned by the student

two things must be modeled:

the domain knowledge (what is to be learned)

the task (what is to be done by the student)


A knowledge model (modeled with the MOT language)

Searchingfor a job

Diagnose thesituation

Plan the job search

Revise the situation

C

C

Execute the job search

plan

C

C

State the job objective

C

Job profile

I/P

I/P

Job objective

I/PI/P

Search plan and materials

I/P I/P

Job search results

I/P

I/P

(courtesy G.Paquette)


A scenario: a complex task

Read about job

descriptions

Job description sample

I/P

Read about the National

Occupational Classification

Matrix

Job description forms

Internet site NOC

Job Futures (volume 1)

CareerDirections

I/P

Your Occupational

Group

Fill out the job description

form

P

File your job descriptions

P

Define your job family

PDefine

your job category

Define your skill level

P

P

I/P

I/P

I/P

Consult publications

I/P

I/P I/P

Define your skill type

PI/P I/P

P

I/P

Employer file

I/P

(courtesy G.Paquette)













Modeling a problem-solving competence / student

Basic idea: problem-solving is generally taught by

leaning by seeing

learning by doing

an educational system that teaches problem-solving must be embodied with an “ideal” problem-solving competence and

use this competence to solve problems and present this competence as a “possible” model for the student (tutoring)

use this competence as a reference to interpret (diagnose) the students’ actions and help them (coaching)


General principles

the “ideal” problem-solving to be presented is not that of an expert

careful definition by domain experts + teachers + pedagogues

when presented to students, the problem-solving must be

presented in the context of concrete exercises

presented at an abstract level

dissociation strategy / domain / exercise knowledge

when used as a reference, the modeled competence must

enable the matching of students’ actions / system competence

enable different problem-solving (and not only the “ideal” one)


Example: a mathematical problem-solving competenceResolution

of an exercise

CC CC

general approach

Problem analysis

Formalisation of the mathematical optimisation

situation

Solution of a linear

programming problem

Solution analysis

CC

CC

formalise an optimisation problem

variables definition

definition of the nature of the objective

function writing out the

objective function

constraints definition

methods of table

C CC

graphic methodmethods of table

applied to the dual

definition of the feability

domainisoquant tracing

solution determination


Example: a domain level

nb-variables=1

linear-programming-pb

multi-variable-pb

variable-analysis-pb

optimisation-pb transition-pb

nb-variables>2

Abstract facts Facts related to the exercise

type-of-pb

Strategy facts

one-variable-pb

optimisation-pb-with-contraints

IS-Avp

Eq

X>2

nb-variables=2

one-explicit-constraint several-explicit-constraint

X2 certain , impossiblelikely , unlikely

Icertain , impossiblelikely , unlikely

I Icertain , impossiblelikely , unlikely

certain , certain

X2

certain , impossiblelikely , unlikelycertain , true Iu true , certain

IS-Apc













Modeling student’s actions diagnosis

The student’s actions must be analyzed on the basis of the ideal problem solving and in respect to their coherence, their pertinence / problem, etc.

example: the fact that a student cuts through the “ideal” problem-solving can be accepted if it does not disable some future possibilities

analysis of the influence at the strategic level (can it influence the choice of some future tasks ?)

analysis of the influence at the domain level (will some intermediate facts be missed, is this a problem ?)

the diagnosis is not a simple matching students-results / system- results, it must be modeled as a complex task


Example of a diagnosis model

Influence of not achieving an Activity

Actanalyze the activity post-conditions

(strategic level)analyze the intermediate results

(domain level)

identify the

post-conditions

identify the

influence of these post-conditions on the solving

identify if these

post-conditions can be produced elsewhere

...













Modeling the management of interactions

Managing the interaction with the student is a difficult task, that require opportunistic decisions

The management of interactions can be seen as a problem- solving task:

the system’s action is the result of a reasoning-process that takes into consideration:

the diagnosis of the students’ production and situation

the expected productions, the expected process

an interaction strategy


Example of an interaction model

Manage the interaction

Act ReactObserve

The teacher's method

Elaborate a diagnosis

Make Comments on the diagnosis

Propose an exercise

Recall lesson

Make comments on a diagnosis a diagnosis has been elaborated

Comment all the results using the same strategy

Select a strategy for interacting Comment some results













Modeling what is to be realized by the teacher

Constructing an educational system is a difficult job

a certain number of works attempt to model how to construct educational systems

Example : The MISA methodology (G.Paquette, Licef)

“a methodology that helps content experts who are designing a course or a learning activity to deal with complex didactic engineering tasks such as knowledge distribution into modules or statement of learning objectives”


Modeling what is to be realized by the teacher

One of the 150 Tasks of the MISA methodology:

444TechnologicalInfrastructure

4.9 Define communication networks and

tools

320Instructional

Scenarios

440Didactic kits and

their users

4.7 Define didactic kits and users

230Media selection

principles

642Implementation

plan

6.2 Plan the implementation of the learning system

I/P

I/P

4.6 Plan the validation tests 436

Validation PlanI/P

430Models of learning

materials

I/P

I/P

I/P

I/P

I/P

I/P

420Learning material

properties

I/P 640Evolution and

maintenance plan of the learning system

6.1 Plan the evolution and maintenance

I/PI/PI/PI/PI/PI/P

I/P

446 Delivery Services

4.10 Define delivery services I/PI/P

I/P

442Delivery model

I/P

I/P


Modeling educational tasks in a framework

When a workbench is associated with a methodology it can be useful to construct an advisory system

a just-in-time first-line "intelligent" help (suggestion, explanation) based on the interactions between the user and the host system

analyze the teacher actions / methodology

provide some assistance

not very different from modeling an “ideal” problem-solving and providing some coaching to the student


Analysis (1): differentiate

Modeling knowledge that can “easily” be identified

constructing a learning scenario

Needs:

a modeling tool that helps expliciting and representing the process

Modeling knowledge that is not “easily” accessible

eliciting the problem-solving competence to be presented to the students

Needs:

an approach to identify the competence to be modeled

a modeling tool to explicit the model


Analysis (2): differentiate

Modeling with the objective of structuring a process

a pedagogical scenario, an engineering methodology

Needs:

a modeling tool such as MOT: a set of epistemological primitives and a visual interface

Modeling in order to operationalize and use run-time

running an ideal problem-solving, analyzing the student’s actions, managing interactions

Needs:

a modeling approach

an operationalization language


Synthesis (1): modeling a reference competence

idea : embody a system with a competence that serves as a reference to help the system user

examples: tutoring system / ideal solving, coaching system / ideal solving, advisory system / methodology, etc.

Objectives to consider : a model that

corresponds to the considered competence

is defined at an abstract level

can easily be described / modified by non computer-science specialists

denotes the ideal solving and some latitude “around” it

is explicit and can be analyzed by a meta-level (reflective) layer


Synthesis (2): modeling system’s functionalities

idea: consider “managing” capacities as problem-solving tasks and model them explicitly

examples: diagnosis of the students’ actions, management of interactions

Objectives to consider

easy description / modification by non computer-science specialists

meta-level (reflective) capacities capacity to analyze the considered object-model (students diagnosis)

opportunistic capacities (interaction management)

explicit description, at an abstract level





in the context of modeling and operationalizing a reference problem-solving competence



General ideas

The system must embed a problem-solving competence that is carefully defined for the pedagogical objectives

solving the problem is not sufficient !

The problem-solving competence must be

presented in the context of concrete exercises

presented at an abstract level

represented explicitly

The problem-solving competence must be analyzable


The “structural correspondence principle”

An important constraint about how to operationalize the system:

to every notion of the modeled competence corresponds a structure of the system

the system structure must denote the competence it models

(the principles of the model must not get lost in the implementation)

capacity to present the competence

capacity to analyze the competence, for instance to

analyze / present what the system can do

analyze / present how it can realize some objectives

diagnose students’ actions


From the transcription to the modeling paradigm

Construction of the conceptual model

Instanciation of the conceptual model

Analysis of the expert knowledge Operationalization

TRANSCRIPTION

domain knowledge + control knowledge

(« what type of knowledge and how to manage it » at an abstract level)

MODELIZATION


The transcription paradigm: a naive vision

teachers provide the required knowledge (helped by knowledge-engineers)

knowledge is translated into (for example) production rules

the adequate competence is defined by constructing the knowledge- base through “test and repair” cycles:

identify some knowledge

transcribe the knowledge into a particular formalism

e.g. production rules: if < conditions> then <new facts, actions>

test


The transcription paradigm: underlying principles

teachers can easily provide the knowledge that is to be represented as the “ideal” problem solving

knowledge can be represented by low-level formalisms such as production rules

adding new rules increases the system competence

production rules can easily be added or removed

the overall strategy “emerges” from the knowledge-base interactions


The transcription paradigm: problems

In fact, the transcription paradigm conducts to the acquisition of knowledge on the basis of low-level implementation-dependent features

Two main problems:

identifying the problem-solving competence

representing explicitly the identified competence


The transcription paradigm: problems

teachers often cannot easily provide adequate knowledge (!)

dissociating knowledge relative to the exercise and the “rationale” (the abstract general competence) is not of the teachers’ usual task !

knowledge-bases based on low-level formalisms become intractable

testing and refining the system competence becomes impossible

the strategy does not “naturally” emerge from a knowledge-base

“compilation” of knowledge in order to obtain the apparent problem-solving behavior of the teacher

non-respect of the “structural correspondence principle”


The modeling paradigm

Knowledge acquisition is a modeling problem:

The objective is to identify a “conceptual model”:

an implementation independent description of the problem-solving strategy, using conceptual primitives that capture the expertise in an adapted way

a model that can be used to communicate by different entities (not a mental model)

a model that guides knowledge acquisition

Theoretical background: Alan Newel’s “knowledge level” theory


The modeling paradigmAn example of a (Kads) model of expertise: heuristic and systematic diagnosis


The modeling paradigmAn example of a (Kads) model of expertise: heuristic classification

strategy level

inference level

domain level


A non generic (but abstract) conceptual model

Resolution of an

exercise

CC CC

general approach

Problem analysis


situation


programming problem

Solution analysis

CC

CC





objective function


methods of table

C CC


applied to the dual





KA as a modeling problem: advantages

From the point of view of constructing the system

defining the model is made tractable

the elicitation of domain knowledge is guided by the model and not by implementation features

From the point of view of using the system

the problem-solving is described at an abstract level

different levels of knowledge are clearly dissociated

the respect of the structural correspondence principle is made easier


Elaboration of the conceptual model

The key-point is the elaboration of the conceptual model

Two approaches can be dissociated

the top-down approach: refining generic problem-solving structures

(the leading point of view for constructing expert systems, but not necessarily the best for educational systems)

bottom-up approach: modeling by data-abstraction


Refining generic problem-solving structures

idea: the model is defined by adapting generic problem-solving methods to the considered domain

characteristic: modeling is viewed as an interpretative process

generic patterns of frequently observed knowledge-use are used to interpret the problem-solving to be modeled

justification:

there is no need to reinvent things at every new system

presenting teachers with abstract models helps them to achieve a “rational reconstruction” of a prescriptive problem-solving model

approach:

domain experts / teachers elicit enough knowledge to allow the knowledge-engineer to propose an abstract model


Refining generic problem-solving structures

Major works:

W.J.Clancey’s work on “Heuristic Classification” as the abstract rationale behind Mycin

B.Chandrasekaran’s Generic Tasks theory

CommonKads methodology (Europe)

Protege (Stanford)

Examples of generic strategies:

simple classification, heuristic diagnosis (single fault, multiple fault), monitoring, prediction, synthesis, abductive hypothesis assembly

(cover and differentiate), etc.


Modeling by data-abstraction

idea: the model is defined by data-abstraction from domain knowledge and observed problem-solving

characteristic: modeling is viewed as a constructive process

justification:

in an educational system one must take into account pedagogical aspects that can only be elicited from teachers’ activities

approach:

domain experts / teachers are asked to solve concrete problems

the knowledge-engineer elicits and synthesizes this material and abstracts a first model

the final model is defined by interaction between the knowledge-engineer and the teacher


Generic models / data-abstraction: criteria

Adequacy to the competence to be learned

the more rational model is not necessarily the more pedagogically suitable

take into consideration the students’ knowledge and pb-solving model

be careful that the solving is reproducible by the student

Capacity to present / accept other ways to solve the problem

adaptation of the strategy and the knowledge according to what students know and do not know

acceptance of variants of the “ideal” competence

Capacity to explicit the system actions

putting into evidence why an action is adequate or not adequate

comparison of different possible actions


Refining generic structures: Pro’s and Con’s

Pro ’s:

reuse of available abstract models

Con ’s

generic models are purely “rational” : they have not been constructed in order to take into account aspects such as solving by different means or using the expertise for other uses than problem-solving

generic models are expressed in a generic vocabulary (e.g. “abstract”, “refine”, or “specify”) and suppose a certain point of view on the domain

...


Refining generic structures: Pro’s and Con’s

Con ’s

…

an interpretative process necessarily introduces a bias that does not allow the respect of idiosyncratic aspects of teaching problem-solving in the considered domain

definition of a model different from that of the teacher

projection of teacher model on one of the models provided by the considered library of models

using a model different from the one that is used in the rest of the teaching would imply modifying all the pedagogy around the software, which is generally not possible


Data abstraction: Pro’s and Con’s

Pro ’s:

modeling by data-abstraction facilitates the respect of the specificities that appear in the problem-solving proposed by teachers and pedagogues

Con ’s

abstracting from concrete performances is a difficult process

a danger is to define a descriptive model of the source of expertise, when we want a prescriptive model

it is difficult to control the adequacy of the model with the observed performances

the process depends on the reliability of the teachers and pedagogues


Synthesis (1)

Opposite strengths and weaknesses

modeling by refining generic models facilitates the definition of rational systematic models

modeling by data-abstraction facilitates taking into consideration idiosyncratic aspects of teachers’ problem-solving and other uses of the model

An opposition that is generally connected to how one considers the “intrinsic nature” of human teachers

Approaches that can be mixed

taking into account of problem-solving specificities of the domain by data-abstraction and then use of generic models to guide the “rationalisation” of the model


Synthesis (2)

What is the most suitable approach for a particular project must be studied according to different considerations such as

what can be obtained from the teachers (compiled knowledge or abstract description of a pedagogic problem-solving expertise)

nature of the problem-solving to be taught and how one intends to teach it (e.g. interpretable as a typical process such as diagnostic or very specific process and pedagogical specificities to be taken into consideration)

the constraints under which the system is constructed (e.g. keeping close to the teachers’ usual behavior or not)

what features are necessary for the different intended uses of the model (presentation of different problem-solvings, comparison of different alternatives strategies, etc.)







The Task-Method paradigm

A formalism that can be used to model and implement systems

Tasks and Methods are conceptual primitives that

enable representing a conceptual model at an abstract level

facilitate the operationalization process

facilitate the respect of the structural correspondence principle


The Task-Method paradigm: principles

idea: represent control knowledge in a way that denotes A.Newel’s rationality principle

what do I have to do ?

how can I do it ?

Principle 1: dissociate

Tasks (the description of what is to be done) what

Methods (means to realize Tasks) how

Principle 2: dynamic selection

Tasks (what is to be done) and Methods (how to achieve what is to be done) are selected dynamically, at run-time, according to the context


The Task-Method paradigm: a typical Task structure

Name name of the Task

Post-conditions

expressed with a domain-level representation language

Expectedresults

list of possible results


Input Context description of when the Task can be achieved


AssociatedMethods

list of Methods that can realize the Task- Decomposition methods- Action methods


The Task-Method paradigm: a typical Method structure

Name name of the Method

Resources what knowledge is necessary to perform the method


Results results produced by the method


Input Context description of when the Method can be achieved


StructureDecomposition methods: list of sub-Tasks

Action methods: knowledge base, interaction with theuser, implicit process


The Task-Method paradigm: principles

The general algorithm : dynamic selection of tasks and methods

No Task

A Task

Identify possible Methods

Identify applicable Methods

Activate a Method

END

Select a Task to achieveSelect a Method

Evaluate the state of a Task


The Task-Method paradigm: example

Resolution of an

exercise

CC CC

general approach

Problem analysis


situation


programming problem

Solution analysis

CC

CC





objective function


methods of table

C CC


applied to the dual




a task

a method

several methods for a task


The Task-Method paradigm: example

nb-variables=1

linear-programming-pb

multi-variable-pb

variable-analysis-pb

optimisation-pb transition-pb

nb-variables>2

Abstract facts Facts related to the exercise

type-of-pb

Strategy facts

one-variable-pb

optimisation-pb-with-contraints

IS-Avp

Eq

X>2

nb-variables=2

one-explicit-constraint several-explicit-constraint

X2 certain , impossiblelikely , unlikely

Icertain , impossiblelikely , unlikely

I Icertain , impossiblelikely , unlikely

certain , certain

X2

certain , impossiblelikely , unlikelycertain , true Iu true , certain

IS-Apc


The Task-Method paradigm: a possible Task

Name identify-the-type-of-problem

Post-conditions type-of-problem , defined

Expectedresults

optimisation-problem (true , false)transition-problem (true , false)…

Input Context number-of-variables , knownnumber-of-constraints , known…

AssociatedMethods

…


The Task-Method paradigm: basic advantages

dissociation problem-solving competence / domain knowledge

the general strategy is defined by the Tasks and Methods description

the domain knowledge is represented separately

modularity: tasks and methods can be added / removed easily

dynamicity: dynamic selection of tasks and methods

explicitness: tasks and methods can be presented / analyzed

easy respect of the structural correspondence principle


The Task-Method paradigm: different uses

Modeling of a reference problem-solving competence

explicit representation, flexibility

Modeling of interaction processes

opportunistic behavior


T&M: modeling a problem-solving competence

explicit representation of the problem-solving competence

adaptability to the context (selection mechanisms)

different levels of interactions with the student (that correspond to different competencies): the student selects the methods - selects the tasks

the student performs the methods (defines the methods’ outputs)

capacity to indicate to a student the implications of his choices

indicate that if some result is not produced now one will not be able to achieve Method xxx because some knowledge will be missing

flexibility (introduction of pedagogical features in the model) ...


T&M: flexibility / problem-solving competence

The task-method explicit representation facilitates the introduction of pedagogical features in the model

introduction of new slots that denote pedagogical features in the Task (resp. Method) structure

modification of the selection mechanisms in order to take these notions into account



Example: accepting “variants” of the ideal problem-solving

the “ideal” problem-solving proposed by teachers usually decomposes the process into many “micro-steps”

a student that goes further than what is expected by the teacher (e.g., given a Task, produces more facts that what is strictly necessary) can cut straightforward in the Task-Method decomposition defined by the teacher

the system must be able to accept such a behavior

cutting through the Task-Method structure must be an explicit process



Tackling this classical problem within the Task-Method paradigm:

Methods are described so as to enable the production of more than what is strictly necessary

Tasks are associated with knowledge that correspond to “interpreting the problem-solving state” (i.e., what has been produced by the methods) in order to make students aware of the general strategy

A distinction is introduced between

necessary interpretations: what is strictly necessary

advised interpretations: how the teacher would manage

possible interpretations: everything that is possible



The problem-solving model is modified in order to allow a distinction between different behaviors:

the “not enough” student: required knowledge has not been produced

the “just in time” student: everything that is required but nothing more

the “teacher like” (“system like”) student

the “exhaustive” student: more than interesting



This distinction can be used for different interactions:

allow a student to cut through the teacher process, e.g. not to realize a Task because its output context is already obtained by some preceding Tasks

explain to the student why some missing results will conduct to a problem when some future tasks will be considered

etc.

NB: such interactions require a complex diagnosis !


T&M: modeling interaction

Manage the interaction

Act ReactObserve

The teacher's method

Elaborate a diagnosis

Make Comments on the diagnosis

Propose an exercise

Recall lesson

a Taska decomposition method

a sub Task


T&M: modeling interaction

Make comments on a diagnosis A diagnosis has been elaborated.

Comment all the results using the same strategy

Comment on correct, incorrect, and then on missing results We wish to make comments in the CIM order

Select a strategy for interacting

Comment on the remarkable situation #1 There is a C but anticipated result AND an I initial one AND these results are related together

Comment on the results in a standard way No remarkable situation has been detected.

Comment on some results

a Task a decomposition method

two alternative methods from which the best / context

will be selected opportunistically at run-time

(dynamic selection of methods)


Operationalization the Task-Method paradigm

Predefined Task-Method languages

advantage: propose predefined structures and the corresponding selection mechanisms

risk: structures that do not correspond to the model (some notions cannot be explicitly denoted and must be “compiled”)

Operationalization from scratch

advantage: the different notions of the model (including the ones added for pedagogical features) can be explicitly denoted

disadvantage: implementation cost

(in both cases the connection with the domain layer must be studied carefully)


Synthesis

Different contexts require the modeling of complex tasks

Complex tasks must be modeled at an abstract level

According to the context one can

use a generic problem-solving strategy

construct a specific strategy by data-abstraction

The respect of the structural correspondence principle is an important feature

The Task-Method paradigm presents different advantages and is the more generally used


Basic works used in this presentation

MOT, a modeling language http//www.licef.teluq.uquebec.ca

Kads, the European methodology http//www.commonkads.uva.nl

Protege, creating and modifying reusable ontologies and problem-solving methods

http://www.smi.stanford.edu/projects/protege/

Bibliography:

EKAW-KAW-PKAW workshops

AI-Ed, ITS and ICCE conferences

Contact: [email protected]

http://www-ic2.univ-lemans.fr/~tchou

Date post:	26-Dec-2015
Category:	Documents
Upload:	moris-lamb
View:	221 times
Download:	0 times

Modeling complex tasks in the context of educational systems Tutorial B2 - 10h30-12h Pierre...

Documents