8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 1/47

Participatory Learning and Assessment in Competency-Based

Contexts

Daniel T. Hickey

 Associate Professor of Learning Sciences

Center for Research on Learning and Technology

Indiana University

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 2/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 3/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 4/47

Research Context

4

2009 in Sakai

@IU

2012 in

CourseBuilder

BOOC

2013 in Canvas

@ IU & IUHS

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 5/47

Use public contexts givemeaning to knowledge tools

Reward productive 

disciplinary engagement

Grade artifacts through localreflections

Let individuals assess theirunderstanding privately

Measure aggregated

achievement discreetly 5

James

Greeno

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 6/47

• Ed Psych 540: Learning &

Cognition in Education

• Challenging to Teach

 – Diverse experience & ambitions

among students

 – Ambitious but busy instructor!

 – Started adjuncts in 2011

• Typical Accountability

 – Use textbook item bank

 – Include a pretest

• Online in Sakai in 2009

 – Canvas in 2015

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 7/47

7

The P540 Team

Gina Howard

Lauren Smith

Andi Rehak

Xinyi Shen

Joshua Quick

Firat Soylu

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 8/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 9/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 10/47

• Educational Assessment BOOC

 – P507 Assessment in Schools embedded for credit

• Offered to 500 students for free in 2013 with a grant

from Google using CourseBuilder

 – 460 registered, 160 started, 60 finished, 9 in P507• Offered a smaller streamlined version in 2014

 – Added videos and quizzed and automated key features

 – 60 started, 22 completed, 12 in P507

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 11/47

Use “public” contexts to give

meaning to knowledge tools

• Problematize knowledge from

learners’ perspective 

 – Publically open up the discipline

• Learners define (and redefine) a

personally-meaningful context

 – Can be disciplinary problem, project,

investigation, etc.

 – Embodies their experience, interests,

& aspirations

RogersHall

11

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 12/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 13/47

 

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 14/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 15/47

15

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 16/47

16

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 17/47

 

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 18/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 19/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 20/47

 

20

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 21/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 22/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 23/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 24/47

 

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 25/47

Reward productive 

disciplinary engagement• DE involves declarative knowledge and

cultural practices

• PDE connects knowledge and practices,makes connections, brings in resources,

etc.

• Support PDE so it can be rewarded

 – Comment directly & publically on wikifolios

 – Work in the open and encourage “lurking”

 – No grades or mandatory posts

25

RandiEngle

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 26/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 27/47

 

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 28/47

 

28

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 29/47

 

29

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 30/47

 

30

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 31/47

Grade artifacts through

“local” reflections 

• Wikifolios generate lots of text andcomments

 – Grading artifacts and commentsundermines participation

• Grade reflections on PDE.

 – Critical engagement (suitability of context

for learning concepts)  – Collaborative engagement (what you

learned from others) 

 – Consequential engagement(consequences of new knowledge) 

31

MelissaGresalfi

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 32/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 33/47

 

33

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 34/47

Let individuals assess 

understanding privately 

• Offer ungraded open-ended

items and performance tasks

• Cover all of the big ideas in the

assignment

• Use to re-engage rather thanremediate.

 – Items themselves have little formative

value beyond the correct answer34

PamelaMoss

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 35/47

35

M hi t

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 36/47

Measure achievement

discreetly

• Standards-oriented tests

 – Measure gains using pretests, compare

instruction, document improvement

 – Primary student function is motivatingprior engagement

• Refine tests (but don’t teach to them) 

 – Use the item analysis routines in your LMS

• Protect test security

 – Withhold item-level feedback

• Randomly selected timed items

 – Proctor or use unsearchable items

Me! 

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 37/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 38/47

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 39/47

1. Wikifolios averaged8.9 comments & 95words/comment

2. Groupwikis forumsaveraged 61 posts

3. Groupwiki averaged40 comments & 92words/comment

4. 2062 commentsincluded 182 badges

0

5

10

15

1 2 3 4 5 6&7 8 9 10 11 12 13 14 15

Average Number Comments per Wikifolio

Unthreaded

Threaded

0

20

40

60

80

100

Forum Posts by Group andComments by Groupwiki

Forum

Posts

Groupwiki

Comments

A t & Di t

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 40/47

Agreement & Disagreement

in Badges vs. Comments

010

20

30

40

50

60

70

80

90100

Agreement Disagreement Perspective

Change

Reinforcement

   P   e   r   c   e   n   t   a   g   e    F   r

   e   q   u   e   i   n   c   y

Content (potentially co-ccuring)

Comments

Badges

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 41/47

P540 Exam Scores (percent correct)

010

20

30

40

50

60

70

80

90

100

MC Pretest

(25 items, 50

minutes, 0

points)

MC Midterm

(40 minutes,

20 items, 10

points)

Essay Midterm

(50 minutes, 5

items 5 points)

MC Final

(60 minutes,

30 items 9

points)

Essay Final

(120 Minutes,

6 items, 6

points)

Mean Exam Scores (Percent Correct)

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 42/47

Course Evaluation Results

0 1 2 3 4 5

My instructor is WELL-PREPARED for classmeetings.

My instructor treats students with

RESPECT

Overall, I would rate this INSTRUCTOR as

outstanding.

Overall, I would rate the quality of this

COURSE as outstanding.

14 Other Sections This Course

Campus-Wide Items

(11/15, 5-point Likert, SD to SA)

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 43/47

Course Evaluation Results

(11/15, 5-point Likert, SD to SA)

0 1 2 3 4 5

This distance education format helped me feel IN

CONTROL of my own learning

The online technology used in this course was

RELIABLE

The ONLINE FORMAT was well suited for this course.

The level of INTERACTION in this distance coursewas sufficient.

I would take ANOTHER ONLINE COURSE from the

School of Education

The instructor used the available TECHNOLOGY well.

School-Wide Online Learning Items

14 Other Sections This Course

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 44/47

Issues with Semi-Formal (Cohorted)

Course Format

 Assumes regular instructor effort

Is not what online learners expect

Not flexible

  Can’t get behind 

Led many students to drop out of the BOOC

Led many IUHS students to stay with

distance ed courses

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 45/47

Moving to Self-Paced Courses

Not simple to get interaction Need to find like-minded peers

Need to find peers working on same unit

Need a large number of learners Currently creating a self-paced BOOC

New participant list shows current learners

New wikifolio feature allowing archiving

Learners can look at archived work

Currently creating self-paced IUHS courses

Modules with exams and Canvabadge

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 46/47

Moving to Open Learning Resources

Need to automate many features  Automated registration and homepage

Real-time participatory feedback

Exam scoring and badges issuer Relying extensively on open ed resources

Use ranking feature with OERs

Pending NSF proposals for IUHS and IU

Q ti ?

8/9/2019 Participatory Learning and Assessment in Competency-Based Online Learning (253586350)

http://slidepdf.com/reader/full/participatory-learning-and-assessment-in-competency-based-online-learning-253586350 47/47

Questions?

Thank you very much!More info at www.RemediatingAssessment.blogspot.com 

Hickey, D. T. (in press). Situative approaches to aligning summative, formative, and

transformative assessment functions.  Assessment in Education: Principles,

 Policies, and Practices.

Hickey, D. T., & Rehak, A. (2013). Wikifolios and participatory assessment for

engagement, understanding, and achievement in online courses.  Journal of

 Educational Media and Hypermedia, 22 (4), 229-263.

Hickey, D. T., & Zuiker, S. J. (2012). Multi-level assessment for discourse,

understanding, and achievement in innovative learning contexts.  The Journal of

the Learning Sciences, 22, (4) 1-65Hickey, D. T., Kelly, T. A, & Shen, X. (2014). Small to big before massive: Scaling up

 participatory learning and assessment. Proceedings of the Fourth International

Conference on Learning Analytics and Knowledge, Indianapolis, IN (pp. 93-97)