Cross-System Personalization for College Students
Peter Brusilovsky with Sergey Sosnovsky
Michael Yudelson
Shaghayegh Sahebi
Chirayu Wongchokprasitti
Sharon Hsiao
User Model
Collects information about individual user
Provides adaptation effect
Adaptive System
User Modeling side
Adaptation side
User-Adaptive Systems Then…
Classic loop user modeling - adaptation in adaptive systems
University of Pittsburgh - PAWS Lab
… and now
University of Pittsburgh - PAWS Lab
Personalization Challenges in the New Context • How an adaptive system can benefit from
information about users collected by other systems? – What is the framework for UM integration? – Will it improve cold-start situation? – Will it improve parallel use of multiple systems
• Can we do it for different types of user models? – Knowledge model – Interest model
University of Pittsburgh - PAWS Lab 4
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N 10 3
0
2 7
4
Concept-Level Knowledge Model
University of Pittsburgh - PAWS Lab
Cross-System Knowledge Modeling
http://adapt2.sis.pitt.edu/kt/
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N yes no
no
no yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N yes no
no
no yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N yes no
no
no yes
yes
University of Pittsburgh - PAWS Lab
Missing links
The Approach: Ontology-Based Cross-System Personalization
University of Pittsburgh - PAWS Lab
Connect DM (ontologies)
Main Stages of Our Work
• Centralized user modeling (1990-1998) • Multi-system personalization based on single
domain model: ADAPT2 (2003-2007) • Cross-domain mapping for cold start (2007)
– C to Java • Single domain guided evidence mapping
(2008-2010) – Topic to concept mapping for Java – Constraints to concepts mapping for SQL
• Single domain automatic mapping (2010-2012) University of Pittsburgh - PAWS Lab 8
UM of C knowledge
Java C
UM of Java
knowledge
How we started – from C to Java
• Manual vs. Automatic ontology mapping
• Knowledge mapping using ontology mapping
• Compare predicted and demonstrated knowledge
• Automatic mapping is comparable with manual
• Overall gain for translated knowledge is not high
• We got concerned about model to model mapping
• Started exploring evidence mapping
University of Pittsburgh - PAWS Lab
SEDONA: UM exchange with ontology servers
http://adapt2.sis.pitt.edu/kt/
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N yes no
no
no yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N yes no
no
no yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N yes no
no
no yes
yes
Ontology A
Ontology B
University of Pittsburgh - PAWS Lab
SEDONA: UM Exchange
• Ontology server is an exchange point for concept-level overlay student models that are based on the stored ontology
• Each UM server or adaptive system that can deduce student knowledge in terms of this ontology reports it to the server
• Each adaptive system that need to know the level of student knowledge for concepts of this ontology can query the ontology server
University of Pittsburgh - PAWS Lab
Lightweight event-based centralized user modeling
http://adapt2.sis.pitt.edu/kt/
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N yes no
no
no yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N yes no
no
no yes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N yes no
no
no yes
yes
Central UM Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N yes no
no
no yes
yes
University of Pittsburgh - PAWS Lab
• Student side: – Use systems in parallel (any order, any
combination) – No extra overhead (single sign-on, single
place to access) • System side:
– Integrated environment > (system1 + system2)
– Each system should try to increase the quality of user modeling and adaptation
Goal: True Integration
University of Pittsburgh - PAWS Lab
Java Problets: The Interface
Sample program
Student’s answer
Help
Question text
System’s feedback
Java Problets: Domain Model • Problets implement traditional overlay user modeling to
adapt to student’s performance
l The domain model of a problet is a concept map enhanced with learning objectives, that combine pedagogical and domain knowledge
QuizJET (1): System Description • QuizJet (Java Evaluation Toolkit) is a system for authoring
and delivery of online self-assessment quizzes for Java programming language
• A typical QuizJET problem is a sample program (consisting of one or several classes), that a student needs to evaluate and provide an answer a follow-up question
• QuizJET generates problems by substituting a numerical value in the program template with a randomized parameter
• Upon receiving a student’s answer QuizJET provides a feedback indicating the correctness of the answer and the right answer (if the student’s attempt was not successful)
QuizJET (2): Student Interface • Students can access QuizJET problems through the
KnowledgeTree portal
Topics in the course
Activities available for the
current topic
Problem text
Problem's classes
QuizJET’s feedback
QuizJET (3): Domain Model • Java Ontology
specifies about 500 classes connected with 3 types of relations: subClassOf, partOf/hasPart, and related
• About 300 classes are available for indexing
• A class can play one of two roles in the problem index: prerequisite or outcome
University of Pittsburgh - PAWS Lab
Domain Model Integration • Main problem: different modeling paradigms
– A learning objective models application of a concepts in the certain context
– Extra classes from the Java ontology have been used for context modeling
– Weights are assigned to prevent too aggressive propagation of classes responsible for context modeling
• Example: – This learning objective models a situation when the conditional part
of the if-else statement is a relational expression evaluated into true value
Evidence-based UM integration in CUMULATE
University of Pittsburgh - PAWS Lab
SQL-Exploratorium
University of Pittsburgh - PAWS Lab
SQL-Tutor
Goal: Integrated Environment
http://www.sis.pitt.edu/~paws/ont/SQL.owl
SQL Explorer: SQL Ontology
University of Pittsburgh - PAWS Lab
SQL-Tutor: Constraints
University of Pittsburgh - PAWS Lab
• Constraints and Concepts are too difficult to map them automatically
• A typical constraint models syntactic or semantic relation between several concepts
• Manual connect constraint to concepts with some degree (small-1, medium-2, or large-3)
Domain Model Mapping
University of Pittsburgh - PAWS Lab
• Solution to SQL-Tutor problem, triggers a number of constraints satisfied and or violated
• Mapping model calculates knowledge update for every concepts related to every triggered constrained:
• The updates are reported to SQL-Exploratorium’s user modeling server
Evidence-Based Modeling
University of Pittsburgh - PAWS Lab
Architecture
• University of Pittsburgh, 2 courses: undergraduate and graduate
• ½ of semester
• 42 students tried SQL-KnoT, 18 – SQL-Tutor
• Out of 103 sessions of using SQL-KnoT 66 co-located with SQL-Tutor usage
Evaluation
University of Pittsburgh - PAWS Lab
• Questionnaire (21 students) – I1 / I2: Overall, I like the interface of SQL-
KnoT/SQL-Tutor. – U1 / U2: SQL-KnoT/SQL-Tutor is a useful
learning tool. – C1 / C2: SQL-KnoT/SQL-Tutor problems
challenged me intellectually.
Results
• An example of semantic integration of two working adaptive systems relaying on very different domain models
• Students used the systems together during single sessions and liked the opportunity
• More evaluation is needed to verify the effect of integration of user modeling accuracy and adaptation
• It is interesting to evaluate the combined adaption (adaptive navigation from SQL-Exploratorium followed by intelligent coaching from SQL-Tutor)
What was presented in the past UbiqUM
University of Pittsburgh - PAWS Lab
Discussion
+ Experts only need to produce relations b/w KIs – the rest is automatic
+ Relations can be removed (strength=0)
- Cannot add relations
University of Pittsburgh - PAWS Lab
References: Past UbiqUM papers Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., Yudelson, M., Brusilovsky, V., and Sharma, D.
(2008) Towards integration of adaptive educational systems: mapping domain models to ontologies. Proceedings of 6th International Workshop on Ontologies and Semantic Web for E-Learning (SWEL'2008) in conjunction with ITS'2008, Montreal, Canada, June 23, 2008.
Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., and Yudelson, M. (2008) Ontology-based integration of adaptive educational systems. Proceedings of 16th International Conference on Computers in Education (ICCE’2008), Taipei, Taiwan, October, 27-31, 2008, pp. 11-18.
Brusilovsky, P., Sosnovsky, S., Yudelson, M., Kumar, A., and Hsiao, I.-H. (2008) User Model Integration in a Distributed Adaptive E-Learning Systems. Proceedings of Workshop on User Model Integration at the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (AH'2008), Hannover, Germany, July 29, 2008.
Brusilovsky, P., Mitrovic, A., Sosnovsky, S., Mathews, M., Yudelson, M., Lee, D., and Zadorozhny, V. (2009) Database exploratorium: a semantically integrated adaptive educational system. In: Proceedings of Ubiquitous User Modeling Workshop at the 17th International Conference on User Modeling, Adaptation, and Personalization (UMAP 2009), Trento, Italy, June 22, 2009
Sosnovsky, S., Brusilovsky, P., Yudelson, M., Mitrovic, A., Mathews, M., and Kumar, A. (2009) Semantic Integration of Adaptive Educational Systems. In: T. Kuflik, S. Berkovsky, F. Carmagnola, D. Heckmann and A. Krüger (eds.): Advances in Ubiquitous User Modelling. Lecture Notes in Computer Science, Vol. 5830, pp. 134-158
University of Pittsburgh - PAWS Lab 33
Evaluating and improving mapping: SQL Exploratorium and SQL Tutor • Authoring constraint mapping is time consuming
• How we can evaluate weights?
• How we can improve mapping?
34
147 "You have used some names
in the WHERE clause that are not
from this database."
207 "You need to specify the join
condition in FROM!"
Join
FROM Clause
Attribute
Table
Database
ConceptsConstraintsw=1
w=2/3
w=2/3
w=2/3
w=1/3
University of Pittsburgh - PAWS Lab
SQL KnoT and SQL-Tutor (2) • 6 experts (2 teachers, 2 GSA, 2 practitioners)
• 1012 constraint-concept relations: strong (1/1), medium (2/3), weak (1/3)
• Usage log of 3544 SQL-Tutor problem-solving attempts of 38 users
• Dataset specific subset – 282 constraints, 576 relations, 61 concepts
University of Pittsburgh - PAWS Lab
Fitting The Source (Constraint) Model
• Experts only need to produce relations b/w KIs – the rest is automatic
36 University of Pittsburgh - PAWS Lab
What was not presented at UbiqUM
• Sosnovsky, S., Dolog, P., Henze, N., Brusilovsky, P., and Nejdl, W. (2007) Translation of overlay models of student knowledge for relative domains based on domain ontology mapping. 13th International Conference on Artificial Intelligent in Education, AI-ED 2007, Marina Del Rey, CA, July 9-13, 2007, IOS, pp. 289-296
• Yudelson, M., Brusilovsky, P., Mitrovic, A., and Mathews, M. (2010) Using Numeric Optimization To Refine Semantic User Model Integration Of Adaptive Educational Systems. Proceedings of the Third International Conference on Educational Data Mining (EDM 2010), Pittsburgh, PA, June 11-13, 2010, pp. 221-230.
• Sosnovsky, S. (2011) Ontology-based Open-Corpus Personalization for e-Learning. PhD Thesis, University of Pittsburgh
University of Pittsburgh - PAWS Lab
What Happened with auto-mapping?
38 University of Pittsburgh - PAWS Lab Sergey Sosnovsky PhD Thesis
OOPS Interface: Reading Phase
39
content of the chosen topic
Navigation links to the next and the previous topics
Feedback/exit buttons
University of Pittsburgh - PAWS Lab Sergey Sosnovsky PhD Thesis
Cross System Interest Modeling
• CoMeT: a social system for sharing information about research colloquia in Pittsburgh
• Models user research interests by observing bookmarking and sharing behavior
• Cold start problem – can’t recommend with no bookmarks
• Can we seed user profiles using other systems that represent user research interests? – Paper bookmarking systems – CuteULike
University of Pittsburgh - PAWS Lab
CoMeT: Collaborative Management of Talks (try pittcomet.info)
University of Pittsburgh - PAWS Lab 41
Traditional Interest Modeling with Keywords
• Document model – a bag of words represented as a vector in keywords
vector space with TF.IDF weighting scheme
University of Pittsburgh - PAWS Lab
W1
W2
W3
W4
W5
W6
D1 0 1 0 0 0 0
D2 .5 0 0 .5 0 0
D3 .12 .13 0 .25 .5 0
D4 .25 0 .25 0 .25 .25
Talks/Papers
Keywords
Recommending Talks to Users • User model
– A combination of vectors of “interesting documents” – Possibly weighted by the rare of interest
• K-nearest neighbor method – recommend top K closest documents to user profile
University of Pittsburgh - PAWS Lab 43
D1 D2 D3 D4
U1 1 0 0 0
U2 .25 0 .5 .25
U3 0 .5 .25 .25
W1
W2
w3
D1 0 1 0
D2 0 0 .5
D3 0 1 0
D4 0 0 .5
U: User Profiles in Talks/Papers Space
D: Documents in Keywords Space
users
Documents Keywords
w1 w2
w3
U1 1 0 1
U2 .25 0.5
.37
U3 0 .25
.37
UP: User Profiles in Keywords Space
users
Keywords
Recommendation with Additional Sources of Information • Sources of information about user interests:
– Standard information: Keywords of bookmarked talks in CoMeT
– Tags of talks in CoMeT – Keywords of bookmarked papers from CiteULike – Tags of papers in CiteULike (CUL)
• Explore the impact of additional sources
• Also explore different models for fusion of tags and keywords
University of Pittsburgh - PAWS Lab 44
Document Representation Models
• Control condition: Keywords Only (KO) – Keywords extracted from documents’ titles and abstracts
• Keywords+n*Tags (KnT) – Keywords extracted from documents’ titles and abstracts +
tags assigned to documents
• Keywords Concatenated by Tags (KCT) – Keywords extracted from documents’ titles and abstracts +
tags assigned to documents
University of Pittsburgh - PAWS Lab 45
Keywords+n*Tags (KnT) Model • Each document: a bag of words containing :
– document’s abstract, title and tags
• Tags: regular keywords – Each tag appears n times
• Merge CUL and CoMeT data in this model: same as KO
University of Pittsburgh - PAWS Lab 46
W1 W2 W3/T1
W4/T2 T3 T4
D1 0 1 1 0 0 0
D2 1 0 3 5 0 0
D3 1 2 3 0 1 0
D4 2 0 5 0 2 1
Keywords
Talks/Papers
Common Keywords & Tags Tag
s
Keywords: w1, w2, w3, w2
Tags: T1, T3
D3
n=2
W3=T1 W4=T2
Keywords Concatenated by Tags (KCT) Model • Tags: a separated source of information
• Each document: a bag of keywords and a bag of tags – Concatenating keywords and tags vectors – TF.IDF weightening scheme
University of Pittsburgh - PAWS Lab 47
W1 W2 W3 W4 T1 T2 T3 T4
D1 0 1 1 0 0 0 0 0
D2 1 0 3 1 0 2 0 0
D3 1 2 1 0 1 0 1 0
D4 2 3 3 0 1 0 2 1
Keywords
Talks/Papers
Tags
W3=T1 W4=T2
Keywords: w1, w2, w3, w2
Tags: T1, T3
D3
Merging CUL and CoMeT Data
University of Pittsburgh - PAWS Lab 48
w1 w2
T1 T2
P1 1 0 0 0
P2 .25 0 .5 .25
P3 0 .5 .25 .25
W2
W3
T1
C1 0 1 0
C2 0 0 .5
Dc: CUL Papers’ Matrix Dt: CoMeT Talks’ Matrix W1 w2 W3 T1 T2
C1 0 0 1 0 0
C2 0 0 0 .5 0
P1 1 0 0 0 0
P2 .25 0 0 .5 .25
P3 0 .5 0 .25 .25
k
m+i
e
l+j
K+e
l+m+i+j-o-p
D: Merged Documents’ Matrix
k- the number of CiteULike papers m- the number of keywords used in CiteULike papers i- the number of tags used in CiteULike papers e- total number of talks in CoMeT l- total number of keywords in CoMeT j- total number of tags in CoMeT o- the number of common keywords between two CoMeT and CiteULike systems P- the number of common tags between two CoMeT and CiteULike systems
Experimental Results
• User study: – 8 real users of both CoMeT and CiteULike systems
• Questionnaire for each recommended talk: – Is this talk related to your interest? (yes/no question) – How interesting this talk to you? (in 5-point scale) – If the talk is related to your interests, how novel is this talk
to you? (in 5-step scale) • Measures:
– Relevance: precision by yes/no answers – Interest: nDCG by 5-point scale – Novelty: averaged the novelty ratings (Non-relevant = zero
novelty) University of Pittsburgh - PAWS Lab 49
Precision and Novelty for different number of recommendations
University of Pittsburgh - PAWS Lab 50
Conclusion
• Including another reliable user profile → increase precision of recommendations;
• Using CiteULike data for all models – Increased Relevance of recommended documents – Decreased novelty for KO model
• CiteULike: adding, reviewing and rating related papers to their research field • CoMeT: information about talks happening within a specific time given on a particular
date users bookmark a more novel, less relevant talk
• Adding tags – Increased novelty of recommendations (both using CoMeT and CUL data) – Increased relatedness in larger number of recommendations
• Injection of keywords from another source of data: more reliable than including tags for relevance
• Including tags from various sources of information: more reliable for interestingness or novelty
University of Pittsburgh - PAWS Lab 51
Back to the start
One user, many models of the same user
University of Pittsburgh - PAWS Lab
User Model
Adaptive System
Let’s look from the other side
Many users, many models of different users
User Model
User Model
User Model
University of Pittsburgh - PAWS Lab
Treemap Group UM for Java Programming
University of Pittsburgh - PAWS Lab
University of Pittsburgh - PAWS Lab
The more the students compared to their peers, the higher post-quiz scores they received (r= 0.34 p=0.004)
Progressor for Java Programming
University of Pittsburgh - PAWS Lab
Progressor+ for rich content
University of Pittsburgh - PAWS Lab
And some more references
• Sahebi, S., Wongchokprasitti, C., and Brusilovsky, P. (2010) Recommending research colloquia: a study of several sources for user profiling. In: Proceedings of the 1st International Workshop on Information Heterogeneity and Fusion in Recommender Systems (HetRec 2010) at the 2010 ACM conference on Recommender systems, RecSys '10, Barcelona, Spain, ACM, pp. 32-38
• Brusilovsky, P., Hsiao, I.-H., and Folajimi, Y. (2011) QuizMap: Open Social Student Modeling and Adaptive Navigation Support with TreeMaps. Proceedings of 6th European Conference on Technology Enhanced Learning (ECTEL 2011), Palermo, Italy, Sptember 20-23, 2011, Springer-Verlag, pp. 71-82.
• Hsiao, I.-H., Bakalov, F., Brusilovsky, P., and König-Ries, B. (2011) Open Social Student Modeling: Visualizing Student Models with Parallel Introspective Views. In: Proceedings of 19th International Conference on User Modeling, Adaptation, and Personalization, UMAP 2011, Girona, Spain, Springer-Verlag, pp. 171-182.
• Hsiao, I.-H., Guerra, J., Parra, D., Bakalov, F., König-Ries, B., and Brusilovsky, P. (2012) Comparative Social Visualization for Personalized E-learning. Proceedings of the Working Conference on Advanced Visual Interfaces, AVI 2012, Capri, Italy, ACM Press, pp. 303-307.
University of Pittsburgh - PAWS Lab 58