Streamlined OrchestrationInvestigating the Impact of Organised Orchestration on Teaching

Lighton PhiriSupervisors:A/Prof. Hussein SulemanProf. Dr. Christoph Meinel

March 15, 2016

Introduction

■ Exploring technology-driven orchestration◻ Understand whether and/or how organisation of learning activities

influences educators’ effectiveness■ Thesis goal

◻ Investigate the impact of organised orchestration on teaching◻ Investigate successful use of an orchestration workbench

2

Motivation

Supporting educators considered one the of most effective ways of improving the quality of education

However, contemporary orchestration is:1) Challenging

◻ Heterogenous activities—instructional and non-instructional◻ Timing constraints—typical lecture session lasts ~45 minutes

2) Ad hoc◻ Specialised tools used for managing activities◻ Generic tools appropriated to achieve desired goal

3

Thesis statement

Streamlined orchestration—attainable through explicit organisation of enactment activities using an orchestration workbench—could potentially make educators more effective.

Research questions:1) Does an orchestration workbench enable educators to

become more effective?2) What is the impact of an orchestration workbench on

educators’ teaching experience?

4

Contributions

5

Organised orchestration: streamlined technology-driven orchestration.

■ Prototype UIs and workflows◻ Workbench UIs and workflows for attaining organised

orchestration■ Case study results

◻ Case studies demonstrating approach in authentic educational settings

■ Experimental studies◻ Analysis and results of experiments conducted to demonstrate

the effect on organised orchestration on teaching

Evaluation plan (1)

‘5+3’ Conceptual Orchestration Framework

■ Management◻ Coordination of learning

activities◻ Regulation of activities to

guide learning activities■ Flexibility

◻ Flexibility of tool■ Assessment

◻ Learning experience

6

Evaluation plan (2)

7

# Evaluation Technique

1. Case studies

2. Experiment in the large

3. Targeted group

4. Survey of existing tools

5. Captive audience

6. General random trial

7. Survey of teaching staff

8. UCT Lecture recording analysis

Study 1: Expert reviews

■ Expert review sessions held with potential EdTech early adopters◻ Individuals integrating technology

in the classroom◻ Understand potential study

environment; recruit participants■ Eight teaching staff interviewed■ Main outcomes

◻ Scale of technology integration◻ Varying learning models

8

Department #

Architecture & Planning 2

CILT 1

Chemical Engineering 1

Computer Science 3

Mechanical Engineering 1

8

Study 2: Flipped class case study

■ Computer Architecture second year course◻ 175 registered students◻ Three core-activities conducted during lecture sessions

■ Study aimed at assessing feasibility and potential◻ Extent of prototype tool use◻ Tool effect on learners’ learning experience

9

Activity Orchestration Orchestration Level

Demonstration Live demonstration Class

Discussion Interactive discussions Class/Individual

Timed quiz Timed quiz session Individual

Orchestration tool usage pattern

■ Nine orchestration tools used during duration of course ◻ Most tools used to render content◻ Specialised tools once-off tasks

■ On average prototype orchestration tool used most of the time◻ Used 66.72% on average◻ Context switching occurred an

average of two times with noticeable time during switchover

10

Tool Freq. Duration

Prototype 10 00:30:31

VideoGlide 8 00:07:56

Firefox 1 00:21:29

Impress 1 00:38:26

Evince 1 00:00:50

QtSpim 1 00:12:47

Robotic Arm 1 00:01:24

TextEditor 1 00:02:07

VirtualBox 1 00:00:58

Learner experience

■ Tool helped organise activities

■ Static sequencing (activity listing) found helpful

■ Timer useful to average performers

11

■ ”I did not really notice the tool”

■ ”I found the classroom experience fun”

Study findings

Feasibility of organised orchestration■ Facilitated neutral flow of activities■ Results from learner survey indicate tool was impact

neutral

Potential to facilitate improved learning outcomes■ Learner survey suggests orchestration workbench has

the potential to positively impact learning experience

12

Study 3: Ad hoc vs. organised orchestration

13

PortableApps—ad hoc orchestrationWorkbench prototype interface—organised orchestration

PortableApps Prototype Workbench

Study objectives

14

How does organised orchestration, using an orchestration workbench, compare with ad hoc orchestration?

■ Hypothesis #1: Workbench is more effective◻ H0: Workbench effectiveness is comparable to ad hoc approach◻ Ha: Workbench is more effective than ad hoc approach

■ Hypothesis #2: Workbench results in more positive effect teaching experience◻ H0: Effect of workbench on teaching experience is comparable to

ad hoc approach◻ Ha: Workbench has a positive effect on teaching experience

Experimental variables & conditions

15

■ Within group laboratory study◻ 29 participants from General Education and Training department

at Cape Peninsula University of Technology■ IV: orchestration technique■ DV: time on task; AttrakDiff 2 dimensions■ Control variables: ISP level; Teaching experience &

Computing experience

RQ AH Factor Variable Scale Description

RQ1 H1Task speed Tt Min Task time

Success PQ [-3 – 3] AttrakDiff 2

RQ2 H2 User experience HQ-I; HQ-S; ATT [-3 – 3] AttrakDiff 2

Experimental procedure

16

Experimental procedure involved random blocks■ Learning scenario

◻ CAPS Grade 5 Teacher Guide■ Orchestration techniques

◻ PortableApps: Ad hoc approach◻ Prototype: Workbench approach

■ Data collection◻ Time on tasks (effectiveness)◻ AttrakDiff 2 ‘Comparison A–B’

investigation (effectiveness and user experience)

◻ General comments

Experimental results: Time on tasks

■ Time of tasks (paired student t-test)◻ p-value = 0.1349

■ Orchestration 14.7% faster using workbench

17

p > 0.05—no statistically significant difference for all four dimensions

Grp 1 Grp 2 ISP Teaching Computing

Workbench 5.14 4.33 ISP 2, 4; N/A 2; 3; 4+ times 2; 3+ years

PortableApps 7.00 4.11 — — —

Experimental results: AttrakDiff 2 (2)

AttrakDiff 2 portfolio-presentation

■ Both orchestration approaches fall within the same character region

■ Workbench approach falls closer to desired character region

18

NOTE: Missing confidence rectangles ***

Experimental results: AttrakDiff 2 (3)

AttrakDiff 2 dimensions

■ PQ (Wilcoxon test)◻ p-value = 0.5465

■ HQ-I (paired student t-test)◻ p-value = 0.08498

■ HQ-S (paired student t-test)◻ p-value = 0.8886

■ ATT (Wilcoxon test)◻ p-value = 0.1634

19

p > 0.05—no statistically significant difference for all four dimensions

Experimental results: AttrakDiff 2 (4)

20

Ad hoc analysis of AttrakDiff 2 dimensions performed by analysing dimension word-pairs.

Experimental results: AttrakDiff 2 (5)

21

Ad hoc analysis of AttrakDiff 2 dimensions performed by analysing dimension word-pairs.

Study findings

Effectiveness■ Learning activities orchestrated faster using workbench■ Participants’ perceived success more pronounced with

workbench

User experience■ Higher mean score for HQ-I and ATT dimensions

Validity of results■ No statistical significance

◻ Only 29 of planned 52 participants took part in study◻ Study to be replicated with additional participants

22

Future directions

23

Further explore the correlation between the use of an orchestration workbench and positive teaching and learning experience

■ Dynamic pre-session management◻ IMS Simple Sequencing standard

■ Evaluate approach in authentic educational settings

Year #3+ Plan

24

Planned activities

25

Milestone Milestone description Timescale

1. Experiment #2 Replicate study May 2016

2. Experiment #3 CSC1010H case study May 2016

3. Experiment #4 Large-scale experiment August 2016

4. : : :

5. : : :

6. Thesis draft Draft manuscript February 2017

7. Thesis manuscript Final manuscript April 2017

Bibliography

[1] Jeremy Roschelle et al. “Classroom Orchestration: Synthesis”. Computers & Education, 6a9:523–526, 2013.

[2] Pierre Dillenbourg. “Design for Classroom Orchestration”. Computers & Education, 69:485–492, 2013.

[3] Pierre Dillenbourg and Patrick Jermann. “Technology for classroom orchestration”. New Science of Learning. 525–552, 2011.

[4] Luis P. Prieto et al. “Orchestrating technology enhanced learning: a literature review and a conceptual framework”. International Journal of Technology Enhanced Learning, 3(6):583, 2011.

[5] Juan A. Munoz-Cristobal et al. “Supporting Teacher Orchestration in Ubiquitous Learning Environments: A Study in Primary Education”. TLT, 83–97, 2014.

26