Competency Modeling for E-Learning

PAWS Lab Work onCompetencies and Student ModelingPeter Brusilovsky

School of Information Sciences

University of Pittsburgh, [email protected]

http://www.sis.pitt.edu/~peterb

Agenda

• Overview• ADAPT2 architecture• Original student modeling in CUMULATE

– Example, DB Exploratorium• Problems and solutions

– Multi-ontology issue – introduce ontology server

– Efficiency – pull to push switch• Cross-systems, cross-ontology, and cross-

domain modeling

University of Pittsburgh - PAWS Lab 3

Main Stages of Our Work

• Centralized user modeling (1990-1998)• Multi-system personalization based on ADAPT2 (2003-

2007)– CUMULATE 1: Single domain model (one system, one model)

(2003-2006) – CUMULATE 2: Parallel independent modeling using 2 models

(2004-2014)• Cross-domain mapping for cold start (2007)

– C to Java• Single domain guided evidence mapping (2008-2010)

– Topic to concept mapping for Java– Constraints to concepts mapping for SQL

• Single domain automatic mapping (2010-2012)

User Model

Collects informationabout individual user

Provides adaptation effect

AdaptiveSystem

User Modeling side

Adaptation side

Centralized Single System Modeling

Classic loop user modeling - adaptation in adaptive systems

University of Pittsburgh - PAWS Lab

KT Architecture

• Learning experiences are delivered by various [adaptive, smart] re-usable activities residing on distributed activity servers

• A portal provides single log-in and singe access point to all content

• A student modeling server maintains a centralized student model

• A value-added service could work as intermediary between “dumb” learning content and portal

• Brusilovsky, P. (2004) KnowledgeTree: A distributed architecture for adaptive e-learning. In: Proceedings of 13th International World Wide Web Conference, WWW 2004, New York, NY, 17-22 May, 2004, ACM Press, pp. 104-113

KT Architecture

Portal

ActivityServer

Student Modeling Server

Value-addedService

Making it Open

• There are no other requirements to the components than an ability to support standard protocols

• Any new activity server can be used as long as it complies to the protocols

• The architecture allows for different portals and value added services to co-exist as long as they support protocols

• Multiple student model servers allowed

Protocols

• Portal/service activity server/service– Request activities, respond with a list of

relevant activities, start activity• Portal/service/activity server student

model server– Report information about student, request

information about student• Student model server portal service activity server– Transparent chain of authentication

A student model server CUMULATE

Competencies-Based Modeling

• Lower level of student model has a flow of content-level events– Which content was used, who used, results (0-1)

• Each content item is connected to knowledge units– Topic-based modeling: coarse grain units, each content

“belongs” to topic (1->N), based on topic network– Concept-based modeling: fine grain units, each content is

indexed with related concepts, based on ontology• An inference agent processes events in the context of

KU connections and maintains up-to-date KU-Level model

• Cumulate allows multiple independent inference agents– Agents for different modeling approaches (i.e, BMA, BKT)– Agents that model content on different levels

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N103

0

27

4

Concept-Level Knowledge Model


Example: Database Exploratorium• Knowledge Tree

portal for content access

• Three kinds of activities– Examples– Problems– SQL Lab

• Central user model serverCUMULATE

• Two levels of modeling– Topics (teacher)– Concepts (ontology)

• Both levels are used independently for adaptation

Brusilovsky, P., Sosnovsky, S., Lee, D., Yudelson, M., Zadorozhny, V., and Zhou, X. (2010) Learning SQL programming with interactive tools: from integration to personalization. ACM Transactions on Computing Education 9 (4), Article No. 19, pp. 1-15.

SQ

L O

nto

log

yWe created C, SQLand Java Ontologies

Two-level adaptation in DBE

Moving to many systems and ontologies


Problems with KT

• We started the integration of adaptive systems produced by other groups…

• Multiple ontologies (domain models)– Two systems complement each other, but use

different domain models for content indexing• Complex user modeling mechanisms

– User modeling server can’t replicate same level of inference student models from events

Cross-System Knowledge Modeling

http://adapt2.sis.pitt.edu/kt/

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept Nyesno

no

noyes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept Nyesno

no

noyes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept Nyesno

no

noyes

yes


Missing links

The Approach: Ontology-Based Cross-System Personalization


Connect DM(ontologies)

UM of C knowledge

JavaC

UM ofJava

knowledge

How we started – from C to Java• Manual vs. Automatic

ontology mapping• Knowledge mapping using

ontology mapping• Compare predicted and

demonstrated knowledge• Automatic mapping is

comparable with manual• Overall gain for translated

knowledge is not high• We got concerned about

model to model mapping• Started exploring evidence

mapping

Sosnovsky, S., Dolog, P., Henze, N., Brusilovsky, P., and Nejdl, W. (2007) Translation of overlay models of student knowledge for relative domains based on domain ontology mapping. 13th International Conference on Artificial Intelligent in Education, AI-ED 2007, Marina Del Rey, CA, July 9-13, 2007, IOS, pp. 289-296

How we can deal with multiple competency organizations?• Content should be separated from its

content-metadata, i.e., ontology indexing or topic categorization

• The same smart content item could be classified under different topic networks or indexed using different ontologies

• We need to maintain and use multiple descriptions for the same item and multiple user models!

Solution: Ontology Server

• Ontology Server as a new component in the new ADAPT2

architecture• Ontology server maintains one specific domain ontology• Ontology Server collects metadata about everything

related to this ontology– Content-level metadata for all resources indexed with this

ontology– Overlay student models for all students that are modeled with

this ontology• A Student modeling server can use several ontology

servers in parallel to perform modeling in different ontologies

• Brusilovsky, P., Sosnovsky, S., and Yudelson, M. (2005) Ontology-based framework for user model interoperability in distributed learning environments. In: World Conference on E-Learning, E-Learn 2005, pp. 2851-2855.

Multiple Ontologies in ADAPT2

• The new architecture ADAPT2 allows the use of multiple ontologies for content and student modeling

• Each ontology is maintained by a dedicated ontology server

• Ontology server is handling all requests related with the ontology - about the ontology itself, learning activities, and users

Summary

• Learning activities are separated from its content metadata

• An activity server’s duty is to maintain and serve an activity (URI invocation)

• Each activity can be indexed in terms of several ontologies

• An ontology server (not activity server!) stores content metadata for all activities indexed in terms of this ontology

Ontology server

An ontology server support inference level of UM server

SEDONA: UM exchange with ontology servers

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept Nyesno

no

noyes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept Nyesno

no

noyes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept Nyesno

no

noyes

yes

Ontology A

Ontology B


Practical Experience

• Implemented first version of an Ontology server Sedona

• Addressed more urgent student model efficiency issue

• Fully redesigned CUMULATE server, moved from pull to push, very efficient

• Ontology server as a unit has never been adapted to new CUMULATE, instead CUMULATE started to perform some of its functions

• Decided to collect more cross-ontology experience to redesign all Sedona functions properly

• Continued with a series of cross-ontology modeling experiments

SEDONA: UM Exchange

• Ontology server is an exchange point for concept-level overlay student models that are based on the stored ontology

• Each UM server or adaptive system that can deduce student knowledge in terms of this ontology reports it to the server

• Each adaptive system that need to know the level of student knowledge for concepts of this ontology can query the ontology server


Lightweight event-based centralized user modeling

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept Nyesno

no

noyes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept Nyesno

no

noyes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept Nyesno

no

noyes

yes

Central UM

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept Nyesno

no

noyes

yes


Sosnovsky, S., Brusilovsky, P., Yudelson, M., Mitrovic, A., Mathews, M., and Kumar, A. (2009) Semantic Integration of Adaptive Educational Systems. In: T. Kuflik, S. Berkovsky, F. Carmagnola, D. Heckmann and A. Krüger (eds.): Advances in Ubiquitous User Modelling. Lecture Notes in Computer Science, Vol. 5830, pp. 134-158.

• Student side:– Use systems in parallel (any order, any

combination)– No extra overhead (single sign-on,

single place to access)• System side:

– Integrated environment > (system1 + system2)

– Each system should try to increase the quality of user modeling and adaptation

What we Consider as True Integration


Explored Cases

• QuizJet integration with Problets in Java domain– One source KI to many target KI mapping– Brusilovsky, P., Sosnovsky, S., Yudelson, M., Kumar, A., and Hsiao,

I.-H. (2008) User Model Integration in a Distributed Adaptive E-Learning System. Workshop on User Model Integration at the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems.

• SQL Exploratorium integration with SQL tutor in SQL domain– Many to many KI mapping from source to target domain– Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., and

Yudelson, M. (2008) Ontology-based integration of adaptive educational systems. 16th International Conference on Computers in Education (ICCE’2008), Taipei, Taiwan, October, 27-31, 2008, pp. 11-18

Java Problets: The Interface

Sample progra

m

Student’sanswer

Help

Questiontext

System’sfeedback

Java Problets: Domain Model• Problets implement traditional overlay user

modeling to adapt to student’s performance The domain

model of a problet is a concept map enhanced with learning objectives, that combine pedagogical and domain knowledge

QuizJET (1):System Description• QuizJet (Java Evaluation Toolkit) is a system for

authoring and delivery of online self-assessment quizzes for Java programming language

• A typical QuizJET problem is a sample program (consisting of one or several classes), that a student needs to evaluate and provide an answer a follow-up question

• QuizJET generates problems by substituting a numerical value in the program template with a randomized parameter

• Upon receiving a student’s answer QuizJET provides a feedback indicating the correctness of the answer and the right answer (if the student’s attempt was not successful)

QuizJET (2):Student Interface• Students can access QuizJET problems through

the KnowledgeTree portal

Topics in the course

Activities available for the current

topic

Problem text

Problem's classes

QuizJET’s feedback

QuizJET (3): Domain Model• Java Ontology

specifies about 500classes connectedwith 3 types of relations: subClassOf,partOf/hasPart, and related

• About 300 classes areavailable for indexing

• A class can play one of two roles in the problemindex: prerequisite or outcome


Domain Model Integration• Main problem: different modeling paradigms

– A learning objective models application of a concepts in the certain context

– Extra classes from the Java ontology have been used for context modeling

– Weights are assigned to prevent too aggressive propagation of classes responsible for context modeling

• Example:– This learning objective models a situation when the

conditional part of the if-else statement is a relational expression evaluated into true value

Evidence-based UM integration in CUMULATE


• An example of semantic integration of two working adaptive systems relaying on very different domain models

• Many to many KI mapping from source to target domain– Topology constructed by domain experts– Data could be used to improve the mapping

Integrating SQL Tutor and SQL Exploratorium


SQL-Exploratorium


SQL-Tutor

Goal: Integrated Environment

http://www.sis.pitt.edu/~paws/ont/SQL.owl

SQL Explorer: SQL Ontology


SQL-Tutor: Constraints


• Constraints and Concepts are too difficult to map them

• A typical constraint models syntactic or semantic relation between several concepts

• Manual connect constraint to concepts with somedegree (small-1, medium-2, or large-3)

Domain Model Mapping


• Solution to SQL-Tutor problem, triggers a number of constraints satisfied and or violated

• Mapping model calculates knowledge update for every concepts related to every triggered constrained:

• The updates are reported to SQL-Exploratorium’s user modeling server

Evidence-Based Modeling


Architecture

• University of Pittsburgh, 2 courses: undergraduate and graduate

• ½ of semester• 42 students tried SQL-KnoT, 18 –

SQL-Tutor• Out of 103 sessions of using SQL-

KnoT 66 co-located with SQL-Tutor usage

Evaluation


• Questionnaire (21 students)– I1 / I2: Overall, I like the interface of

SQL-KnoT/SQL-Tutor. – U1 / U2: SQL-KnoT/SQL-Tutor is a useful

learning tool.– C1 / C2: SQL-KnoT/SQL-Tutor problems

challenged me intellectually.

Results

49

Evaluating and improving mapping:SQL Exploratorium and SQL Tutor• Authoring constraint mapping is time

consuming• How we can evaluate weights?• How we can improve mapping?


SQL KnoT and SQL-Tutor (2)• 6 experts (2 teachers, 2 GSA, 2

practitioners)• 1012 constraint-concept relations: strong

(1/1), medium (2/3), weak (1/3)• Usage log of 3544 SQL-Tutor problem-

solving attempts of 38 users• Dataset specific subset

– 282 constraints, 576 relations, 61 concepts


51

Fitting The Source(Constraint) Model

• Experts only need to produce relations b/w KIs – the rest is automatic


University of Pittsburgh - PAWS Lab 52

References on cross-system modelingSosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., Yudelson, M., Brusilovsky, V., and

Sharma, D. (2008) Towards integration of adaptive educational systems: mapping domain models to ontologies. Proceedings of 6th International Workshop on Ontologies and Semantic Web for E-Learning (SWEL'2008) in conjunction with ITS'2008, Montreal, Canada, June 23, 2008.

Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., and Yudelson, M. (2008) Ontology-based integration of adaptive educational systems. Proceedings of 16th International Conference on Computers in Education (ICCE’2008), Taipei, Taiwan, October, 27-31, 2008, pp. 11-18.

Brusilovsky, P., Sosnovsky, S., Yudelson, M., Kumar, A., and Hsiao, I.-H. (2008) User Model Integration in a Distributed Adaptive E-Learning Systems. Proceedings of Workshop on User Model Integration at the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (AH'2008), Hannover, Germany, July 29, 2008.

Brusilovsky, P., Mitrovic, A., Sosnovsky, S., Mathews, M., Yudelson, M., Lee, D., and Zadorozhny, V. (2009) Database exploratorium: a semantically integrated adaptive educational system. In: Proceedings of Ubiquitous User Modeling Workshop at the 17th International Conference on User Modeling, Adaptation, and Personalization (UMAP 2009), Trento, Italy, June 22, 2009

Sosnovsky, S., Brusilovsky, P., Yudelson, M., Mitrovic, A., Mathews, M., and Kumar, A. (2009) Semantic Integration of Adaptive Educational Systems. In: T. Kuflik, S. Berkovsky, F. Carmagnola, D. Heckmann and A. Krüger (eds.): Advances in Ubiquitous User Modelling. Lecture Notes in Computer Science, Vol. 5830, pp. 134-158

53

Automatic Ontology Mapping

• SQL Integration demonstrated using expert-authored and automatically-tuned domain ontology mapping we can do efficient cross-system personalization with two conceptualizations (ontologies) in the same domain

• Expert labor is expensive. Could we do automatic mapping between two ontologies in the same domain?

• The case is explored in– Sosnovsky, S., Brusilovsky, P., and Hsiao, I.-H. (2012) Adaptation

"in the Wild": Ontology-based Personalization of Open-Corpus Learning Material. In: Proceedings of 7th European Conference on Technology Enhanced Learning (EC-TEL 2012), Saarbrücken, Germany, pp. 425-431.

– Sosnovsky, S. (2011). Ontology-based Open-Corpus Personalization for E-Learning PhD Thesis, University of Pittsburgh.

9/26/2010

54

What Happened with auto-mapping?

University of Pittsburgh - PAWS Lab Sergey Sosnovsky PhD Thesis

55

OOPS Interface: Reading Phase

content of the chosen

topic

Navigation links to the next and

the previous topics

Feedback/exit

buttons

University of Pittsburgh - PAWS Lab Sergey Sosnovsky PhD Thesis

Date post:	08-Sep-2014
Category:	Technology
Upload:	peter-brusilovsky
View:	228 times
Download:	4 times