+ All Categories
Home > Documents > LinkedUp kickoff meeting session 4

LinkedUp kickoff meeting session 4

Date post: 12-Jan-2015
Category:
Upload: hendrik-drachsler
View: 1,425 times
Download: 1 times
Share this document with a friend
Description:
 
59
LinkedUp kickoff / Session 4: Evaluation Framework Criteria and Indicator Hendrik Drachsler & Slavi Stoyanov
Transcript
Page 1: LinkedUp kickoff meeting session 4

LinkedUp kickoff / Session 4: Evaluation Framework Criteria and Indicator

Hendrik Drachsler & Slavi Stoyanov

Page 2: LinkedUp kickoff meeting session 4

Agenda •  05-minute Introduction (Hendrik) •  20-minute Presentations on experiences, best practices

(Philippe Cudré-Mauroux) (Nikolaus Forgo)

•  10-minute Plenary Discussion on lessons learned for LinkedUp (All)

•  15-minute Presentation on Group Concept Mapping (Hendrik) •  15-minute Presentation of the initial version of the evaluation

framework + examples for educational and usability evaluation criteria and suitable methods (Hendrik)

•  25-minute Plenary discussion on suitable evaluation criteria, methods, and experts that should be involved in the development of the evaluation framework

Page 3: LinkedUp kickoff meeting session 4

Objectives of the session

1.  Legal/privacy aspects of open data sharing 2.  Awareness about the evaluation task 3.  Knowing the GCM method 4.  Collection of suitable evaluation indicators

Nikolaus Forgo

Page 4: LinkedUp kickoff meeting session 4

4

An example: Evaluation of TEL RecSys

The idea is to pick from my previous list 20-50 movies that share similar audience with “Taken”, then how much I will like depend on how much I liked those early movies – In short: I tend to watch this movie because I have watched those movies … or – People who have watched those movies also liked this movie (Amazon style)

probabilistic combination of – Item-based method – User-based method – Matrix Factorization – (May be) content-based method

Page 5: LinkedUp kickoff meeting session 4

5

RecSysTEL Eval. criteria

Kirkpatrick model by Manouselis et al. 2010

Combine approach by Drachsler et al. 2008

1. Accuracy 2. Coverage 3. Precision 4. Recall

1. Reaction of learner 2. Learning improved 3. Behaviour 4. Results

1. Effectiveness of learning 2. Efficiency of learning 3. Drop out rate 4. Satisfaction

1. Accuracy 2. Coverage 3. Precision 4. Recall

Page 6: LinkedUp kickoff meeting session 4

6

Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011). Recommender Systems in Technology Enhanced Learning. In P. B. Kantor, F. Ricci, L. Rokach, & B. Shapira (Eds.), Recommender Systems Handbook (pp. 387-415). Berlin: Springer.

TEL RecSys::Review study

Conclusions:

Half of the systems (11/20) still at design or prototyping stage only 9 systems evaluated through trials with human users.

Page 7: LinkedUp kickoff meeting session 4

7

The TEL recommender research is a bit like this...

We need to design for each domain an appropriate recommender system that fits the goals, tasks,

and particular constraints#

Page 8: LinkedUp kickoff meeting session 4

8

But...

Kaptain Koboldhttp://www.flickr.com/photos/kaptainkobold/3203311346/

“The performance results of different research efforts in recommender systems are hardly comparable.” (Manouselis et al., 2010)

TEL recommender experiments lack transparency and standardization. They need to be repeatable to test: •  Validity •  Verification •  Compare results

Page 9: LinkedUp kickoff meeting session 4

9 9

Data-driven Research and Learning Analytics#

Hendrik Drachsler (a), Katrien Verbert (b)##(a) CELSTEC, Open University of the Netherlands#(b) Dept. Computer Science, K.U.Leuven, Belgium##

EATEL-

Page 10: LinkedUp kickoff meeting session 4
Page 11: LinkedUp kickoff meeting session 4

11

#Drachsler, H., Bogers, T., Vuorikari, R., Verbert, K., Duval, E., Manouselis, N., Beham, G., Lindstaedt, S., Stern, H., Friedrich, M., & Wolpers, M. (2010). Issues and Considerations regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning. Presentation at the 1st Workshop Recommnder Systems in Technology Enhanced Learning (RecSysTEL) in conjunction with 5th European Conference on Technology Enhanced Learning (EC-TEL 2010): Sustaining TEL: From Innovation to Learning and Practice. September, 28, 2010, Barcelona, Spain.##

TEL RecSys::Evaluation/datasets

Page 12: LinkedUp kickoff meeting session 4

12 17

5. Dataset FrameworkFormal Datasets

Informal

Data A Data B Data C

Algorithms: Algoritmen A Algoritmen B Algoritmen C

Models: Learner Model A Learner Model B

Measured attributes: Attribute A Attribute B Attribute C

Algorithms: Algoritmen D Algoritmen E

Models: Learner Model C Learner Model E

Measured attributes: Attribute A Attribute B Attribute C

Algorithms: Algoritmen B Algoritmen D

Models: Learner Model A Learner Model C

Measured attributes: Attribute A Attribute B Attribute C

dataTEL evaluation model

42

Page 13: LinkedUp kickoff meeting session 4

13 17

5. Dataset FrameworkFormal Datasets

Informal

Data A Data B Data C

Algorithms: Algoritmen A Algoritmen B Algoritmen C

Models: Learner Model A Learner Model B

Measured attributes: Attribute A Attribute B Attribute C

Algorithms: Algoritmen D Algoritmen E

Models: Learner Model C Learner Model E

Measured attributes: Attribute A Attribute B Attribute C

Algorithms: Algoritmen B Algoritmen D

Models: Learner Model A Learner Model C

Measured attributes: Attribute A Attribute B Attribute C

dataTEL evaluation model

42

In LinkedUp we have the opportunity to apply a structured approach to develop a community accepted evaluation framework. 1.  Top-Down by a literature study 2.  Bottom-up by GCM with experts in the field

Page 14: LinkedUp kickoff meeting session 4

WP2: Literature review 1. Literature review of suitable evaluation approaches and criteria 2. Review of comprising initiatives such as LinkedEducation, MULCE, E3FPLE and the SIG dataTEL

 

Page 15: LinkedUp kickoff meeting session 4

25/05/12 15 Stefan Dietze

•  Group Concept Mapping resembles the Post-it notes problem solving technique and Delphi method

•  GCM involves participants in a few simple activities (generating, sorting and rating of ideas) that most people are used to.

GCM is different in two substantial ways: 1. Robust analysis (MDS and HCA) GCM takes up the original participants contribution and then quantitatively aggregate it to show their collective view (as thematic clusters) 2. Visualisation GCM presents the results from the analysis as conceptual maps and other graphical representations (pattern matching and go-zones).

Hendrik Drachsler

WP2: Group Concept Mapping

Page 16: LinkedUp kickoff meeting session 4

•  innovations in way network is delivered •  (investigate) corporate/structural alignment •  assist in the development of non-traditional partnerships (Rehab with the

Medicine Community) •  expand investigation and knowledge of PSN'S/PSO's •  continue STHCS sponsored forums on public health issues (medicine

managed care forum) •  inventory assets of all participating agencies (providers, Venn Diagrams) •  access additional funds for telemedicine expansion •  better utilization of current technological bridge •  continued support by STHCS to member facilities •  expand and encourage utilization of interface programs to strengthen the

viability and to improve the health care delivery system (ie teleconference) •  discussion with CCHN

...organize the issues...

brainstorm

Work quickly and effectively

under pressure

49

Organize the work when

directions are not specific.

39

Decide how to manage

multiple tasks. 20 Manage resources effectively.

4

sort

rate

Page 17: LinkedUp kickoff meeting session 4

Binary, square similarity matrix

Sort for one participant

Representation

Total square similarity matrix

across participants

Page 18: LinkedUp kickoff meeting session 4

Multidimensional Scaling

Output: An n-dimensional mapping of the entities

56 55

54

53

52

51

50

49 48

47

46

45

44

43

42

41

40 39

38

37 36

35

34

33

32

31

30 29

28

27 26

25

24 23 22

21 20 19

18 17 16

15

14

13

12

11

10

9

8

7

6

5

4

3

1

Input: A square matrix of relationships among a set of entities

!5 !1 !2 !4 !0 !1 !1 !3 !1 !0!!1 !5 !0 !0 !0 !1 !0 !0 !2 !0!!2 !0 !5 !3 !0 !0 !0 !0 !0 !0!!4 !0 !3 !5 !0 !0 !0 !0 !0 !0!!0 !0 !0 !0 !5 !0 !0 !2 !0 !0!!1 !1 !0 !0 !0 !5 !0 !0 !4 !0!!1 !0 !0 !0 !0 !0 !5 !0 !0 !0!!3 !0 !0 !0 !2 !0 !0 !5 !0 !0!!1 !2 !0 !0 !0 !4 !0 !0 !5 !0!!0 !0 !0 !0 !0 !0 !0 !0 !0 !5 !!

Page 19: LinkedUp kickoff meeting session 4

•  innovations in way network is delivered •  (investigate) corporate/structural alignment •  assist in the development of non-traditional partnerships (Rehab with the

Medicine Community) •  expand investigation and knowledge of PSN'S/PSO's •  continue STHCS sponsored forums on public health issues (medicine

managed care forum) •  inventory assets of all participating agencies (providers, Venn Diagrams) •  access additional funds for telemedicine expansion •  better utilization of current technological bridge •  continued support by STHCS to member facilities •  expand and encourage utilization of interface programs to strengthen the

viability and to improve the health care delivery system (ie teleconference) •  discussion with CCHN

Work quickly and effectively under

pressure 49

Organize the work when directions are

not specific. 39

Decide how to manage multiple

tasks. 20

Manage resources effectively. 4

sort

rate

brainstorm

organize

Management Financing

Regionalization

STHCS as model

Community & Consumer Views

Information Services Technology

…”map” the issues...

Page 20: LinkedUp kickoff meeting session 4

Management Financing

Regionalization

Mission & Ideology

Community & Consumer Views

Information Services Technology

•  innovations in way network is delivered •  (investigate) corporate/structural alignment •  assist in the development of non-traditional partnerships (Rehab with the

Medicine Community) •  expand investigation and knowledge of PSN'S/PSO's •  continue STHCS sponsored forums on public health issues (medicine

managed care forum) •  inventory assets of all participating agencies (providers, Venn Diagrams) •  access additional funds for telemedicine expansion •  better utilization of current technological bridge •  continued support by STHCS to member facilities •  expand and encourage utilization of interface programs to strengthen the

viability and to improve the health care delivery system (ie teleconference) •  discussion with CCHN

Work quickly and effectively under

pressure 49

Organize the work when directions are

not specific. 39

Decide how to manage multiple

tasks. 20

Manage resources effectively. 4

sort

rate

brainstorm

organize

Management Financing

Regionalization

STHCS as model

Community & Consumer Views

Information Services Technology

map

...prioritize the issues...

Page 21: LinkedUp kickoff meeting session 4

A Cluster Map

Technology in education

Tools and services enhancing learning

Open education and resources

Assessment, accreditation and qualifications

Globalisation of education

Roles of institutions

Individual and profession driven education

Role of teacher

Life-long learning

Formal education goes informal

Individual and social nature of learning

Epistemological and ontological bases of pedagogical methods

Page 22: LinkedUp kickoff meeting session 4

A cluster rating map

Technology in education

Tools and services enhancing learning

Open education and resources

Assessment, accreditation and qualifications

Globalisation of education

Roles of institutions

Individual and profession driven education

Role of teacher

Life-long learning

Formal education goes informal

Individual and social nature of learning

Epistemological and ontological bases of pedagogical methods

Cluster Legend Layer Value 1 3,21 to 3,38 2 3,38 to 3,55 3 3,55 to 3,72 4 3,72 to 3,89 5 3,89 to 4,06

Page 23: LinkedUp kickoff meeting session 4

Pattern Matching Values

r = -.5

Importance Feasibility

4.06

3.21

3.91

3.15

Formal education goes informal Technology in education

Individual and profession driven educationOpen education and resources

Globalisation of educationRole of teacher

Epistemological and ontological bases of pedagogical methodsRoles of institutions

Roles of institutionsGlobalisation of education

Role of teacherAssessment, accreditation and qualifications

Individual and social nature of learningTools and services enhancing learning

Assessment, accreditation and qualificationsEpistemological and ontological bases of pedagogical methods

Life-long learningLife-long learning

Technology in educationFormal education goes informal

Tools and services enhancing learningIndividual and profession driven education

Open education and resourcesIndividual and social nature of learning

Page 24: LinkedUp kickoff meeting session 4

Pattern Matching Groups

r = .81

Technical Science Social science

4

3.52

4.09

3.03

Technology in educationGlobalisation of education

Open education and resourcesTechnology in education

Roles of institutionsRole of teacher

Role of teacherOpen education and resources

Assessment, accreditation and qualificationsRoles of institutions

Globalisation of educationEpistemological and ontological bases of pedagogical methods

Tools and services enhancing learningAssessment, accreditation and qualifications

Epistemological and ontological bases of pedagogical methodsFormal education goes informal

Formal education goes informal Life-long learning

Life-long learningTools and services enhancing learning

Individual and social nature of learningIndividual and profession driven education

Individual and profession driven educationIndividual and social nature of learning

Page 25: LinkedUp kickoff meeting session 4

GCM in CELSTEC

•  Characteristics of Adaptive Learning Content Management System •  Success and failure factors for ICT project in higher education •  The future of education •  Mobile learning •  Handover training interventions •  Framework of digital competence •  Effect of TV programmes on minority groups •  LLL Limburg •  ICT and foreign language learning •  Language technologies for LLL •  Development of learning outcomes of interdisciplinary module on Creativity

and Innovation •  Part of a software and development methodology •  5 PhD projects

Page 26: LinkedUp kickoff meeting session 4

Handover

•  105 statements about handover training interventions

•  Sorting on similarity in meaning •  Rating on importance and feasibility

Page 27: LinkedUp kickoff meeting session 4

A point map

Page 28: LinkedUp kickoff meeting session 4

A cluster map

Page 29: LinkedUp kickoff meeting session 4

Clusters’ labels

Page 30: LinkedUp kickoff meeting session 4

Use Cases – Evaluation Framework – LinkedUp Challenge

Core questions: 1. What are relevant indicators (e.g., scalability, drop-

out)? 2. How to measure and benchmark? 3. How to allow comparability across diversity of

submissions?

4. We want to be as specific as possible!

Page 31: LinkedUp kickoff meeting session 4

Looking at the DoW

Page 32: LinkedUp kickoff meeting session 4

Looking at the DoW

Page 33: LinkedUp kickoff meeting session 4

Two Objectives Perfect world

LinkedUp Challenge

Open to all kinds of submissions

Practical (easy to use)

Efficient (time saving)

Transparent

Accurate

Academic complete

Address diverse target groups

Transparent

Page 34: LinkedUp kickoff meeting session 4

Rating Map feasibility

Page 35: LinkedUp kickoff meeting session 4

Rating Map importance

Page 36: LinkedUp kickoff meeting session 4
Page 37: LinkedUp kickoff meeting session 4
Page 38: LinkedUp kickoff meeting session 4

Go-Zone Assessment, accreditation and qualifications

3.612.27 4.732.18

4.82

Importance

Feas

ibil

ity

3.6

2

6

9

20

47

48

67

7284

87

88

99

104

113

118

136

145

147

197

r = .07

Page 39: LinkedUp kickoff meeting session 4

Concept Mapping (Sorting input)

To organize the issues

Concept Mapping Process

Measurement (Rating input)

To observe expectations and results

Pattern Matching and Go Zones

To link expectations and results, importance and capacity

Page 40: LinkedUp kickoff meeting session 4

X X X X

Page 41: LinkedUp kickoff meeting session 4

Concept System Brainstorming

Page 42: LinkedUp kickoff meeting session 4

Concept System instruction for sorting

Page 43: LinkedUp kickoff meeting session 4

Concept System Sorting

Page 44: LinkedUp kickoff meeting session 4

Concept System Rating

Page 45: LinkedUp kickoff meeting session 4

Online Consultation IPTS •  delphi study – 79 experts

•  common understanding / mapping of digital competence

•  online brainstorm

•  “A digitally competent person…”

Page 46: LinkedUp kickoff meeting session 4

Procedure data analysis

•  identify unique statements (134) •  Sort statements (Websort.net)

•  Plan b: workshop 17 experts

Page 47: LinkedUp kickoff meeting session 4
Page 48: LinkedUp kickoff meeting session 4
Page 49: LinkedUp kickoff meeting session 4
Page 50: LinkedUp kickoff meeting session 4
Page 51: LinkedUp kickoff meeting session 4
Page 52: LinkedUp kickoff meeting session 4
Page 53: LinkedUp kickoff meeting session 4

Next day

•  feedback initial solution

•  15 clusters

•  4 groups: add label / describe / rearrange

Page 54: LinkedUp kickoff meeting session 4

•  analyse results

•  14 Cluster – 125 statements

•  second consultation round: •  total group •  comment / rate

•  analyse feedback

Page 55: LinkedUp kickoff meeting session 4
Page 56: LinkedUp kickoff meeting session 4

Evaluation / USP

•  Flexibility

•  Rich data

•  Intuitive yet robust method

•  Collective view (≠ consensus)

Page 57: LinkedUp kickoff meeting session 4

Usability evaluation

Page 58: LinkedUp kickoff meeting session 4

Your turn …

http://bit.ly/Linkedup

Questions – Suggestions – Open Discussion

please add anything that comes to your mind in the open Gdoc

Page 59: LinkedUp kickoff meeting session 4

59

This silde is available at:http://www.slideshare.com/Drachsler Email: [email protected]

Skype: celstec-hendrik.drachsler

Blogging at: http://www.drachsler.de

Twittering at: http://twitter.com/HDrachsler

Thank you for attending this lecture!


Recommended