Post on 30-Dec-2015
transcript
SOLSTICE Conference 2015
4th & 5th June 2015
Technology enhanced formative assessment: Participant experiences, relative efficiencies and tutor learning curves
Dr W. Rod Cullen, Manchester Metropolitan University
Formative assessment...“... is designed to provide learners with feedback on progress and inform development, but does not contribute to the overall assessment.”
QAA 2000
“I can’t get my students to do
work that doesn’t get a mark”
“My students don’t use the
feedback I give them – sometimes
they don’t even pick it up”
“That all looks very impressive,
but how long is it going to take me
to do that?”
Session outlineNarrative account of evolution of “Enhancing Learning Teaching and Assessment with Technology” unit (ELTAT)•Delivery model•Feedback methods•Feedback efficiencies•Participant experiences•Conclusions and recommendations
A brief history of ELTATDate Unit Title Credit Weeks Tutor(s)
2007-09 Designing Courses for VLEs (DCVLEs)
10 4 WRC
2009-10 Designing Courses for VLEs (DCVLEs)
10 4 WRC, SG
2011-13 Design Effective Blended and Online Learning (DEBOL)
10 4 WRC, SG with guest appearances from NR and MR
2014 Enhancing Learning Teaching and Assessment with Technology (ELTAT)
30 12 WRC, SG, NR, MR
Learner Profile• Range of backgrounds
– New and experienced academic staff– Academic support staff– Technical support staff
• Majority working full-time but some part-time• Differing assessment requirements
– PG Cert Academic Practice– MA Academic Practice– CPD for credits– CPD non-credits
• Just like real students!
Formative delivery model
Online Preparation
In ClassSession 1
Onlineformativeactivity
Online Preparation
In ClassSession 2
Onlineformativeactivity
Week 1 Week 2
Feedback
Mini (e)Portfolio
Feedback
Data CollectionTutor Activity – since 20071. Kept a detailed
feedback activity log2. Worked out time spent
engaged in each feedback activity.
End-to-end process and each stage: Download, Annotate, Write/ Record, Publish, Upload
Participant experience – 2007-2014
1. Interviewed participants about experiences
2. End of unit survey3. Submission data4. Attendance5. Informal anecdotal
evidence
Ongoing Opportunistic Action Research
Feedback methods (DCVLEs)Methods DetailsWritten Annotated comments on Submission MS
Word + Written Comments
Audio Annotated comments on Submission MS Word + Audio recording
Video Screen Capture
Annotated comments on Submission MS Word + Video Screen Capture
Producing formative feedback1. Getting up to speed with new technology (Pebblepad)
2. Technical problems with codecs and files sizes (common with web-based video)
3. Initially uncomfortable with recording process
Issues for Susan
4. Got up to speed with audio format more quickly
Cullen W. R. (2011)
Per
cen
tag
e
Attendance and engagement with formative tasks 2007-10
1. Reasonably high overall attendance
2. Generally high (70% +) submission rates for formative tasks
4. Consistently lower submission rates in Week 4
MASummative Assessment
3. Use of the feedback is embedded in the online prep and F2F activities
Ongoing development2011 Programme Review:Designing Effective Online and Blended Learning (DEBOL)
• Change in unit title• Neil and Mark – contributions as specialist lecturers
2013 Major Programme Review:Enhancing Learning Teaching and Assessment with Technology (ELTAT)
• 30 Credits • Much longer delivery – 12 weeks• More time to complete tasks (and provide feedback)• Team approach to formative feedback – Tutors allocated group
of student to work with throughout the course
Feedback methodsFormative Activities
Learning outcomes review
Complete LTA matrix review
Revise LTA matrix
Assessment strategy
Revised LTA matrix 2
Monitoring and evaluation plan
Timeline
ELTAT - 2014Feedback provision
Moodle Assignment; MS Word Comments +Audio recording
Moodle Assignment; MS Word Comments +Video Screen capture
Moodle Assignment; MS Word Comments + Written Proforma
Turnitin Assignment; Quickmarks, General Comment + Audio recording
Moodle Assignment; MS Word Comments +Audio recording
Moodle Assignment; MS Word Comments +Audio recording
Moodle Assignment; MS Word Comments +Audio recording
Producing formative feedback1. Rod + Susan very similar times
2. Neil + Mark learning the ropes
3. Pronounced for video and audio
About the task as much as the technology
4. Similar times for written feedback – new task
5. Similar times for TII (Written + Audio)
Team overview1. For team as a whole consistent ≈ 55 mins per submissions (end-to-end)
2. Variation in what the time is required for
3. Download times minimal
4. Upload of audio and video = minimal time but 2-3 mins more than written.
5. No down/upload for TII
6. Additional “publish” stage for audio
7. Reading through and annotating the submissions is most time consuming activity
Efficiency of productionWritten feedback:
•Word count tool in MS Word
•Divided by the total time (end-to-end) to produce the feedback
Screen capture and audio feedback:
•Counted spoken words in 1 minute of recorded feedback = est. speaking rate.
•Length audio and video recording X speaking rate = estimate of the total number of words in each piece of feedback.
•Divided by the total time (end-to-end) to produce the feedback
Turnitin: •Pasted the general text comments into a word document – Word count tool.
•Length of audio feedback X tutors speaking rate = estimate of number of word spoken.
•Work count + Words spoken = estimate of the total number of words in each piece of feedback.
•Divided by the total time (end-to-end) to produce the feedback
Annotations (comments in MS word and Quickmarks in Turnitin) not included
Estimated Feedback words/per minute1. Audio and Video screen capture – learning curve?
Task as well as tech
2. Written feedback - Personal preferences and expertise is an influence
3. TII least efficient – multi-media + suspect we made more comments in the text (Quickmarks)
5. Comparing efficiencies across technologies is more complex than anticipated
4. Mark has felt need to reflect on his feedback on all tasks
Student perspectives
Prefer audio and video to written feedback
Personal and Engaging
Understandable
Annotations important
50:50 split between audio and video
Utilised differently
Video more impact
Audio more reflective
• Learning curves associated with – “Technical skills”– Feelings of self-consciousness– Familiarity with requirements of the formative task
• Technical aspects not as time consuming as cognitive• Audio feedback can be produced quickly and simply• Model of delivery can influence engagement – need to add
value to completing formative tasks• Participants indicated a preference for audio and video screen
captures– Linked to characteristics of the task– Possible novelty aspect
ConclusionsTake home messages
To embed formative assessment into a curriculum •Design it into delivery model – make the value of the feedback clear•Align with summative assessment and signpost this to students•Select feedback technology best matched to the type of tasks•Be prepared for the learning curve
Feedback methodsOnline Formative Activities
Learning outcomes review
Complete LTA matrix
Revise LTA matrix
Activity Design Plan
Monitoring and evaluation plan
Timeline
2011-14Feedback provision
Moodle Assignment; Track Changes/annotated script MS Word
Moodle Assignment; Track Changes/annotated script MS Word
Moodle Assignment; Annotated MS Word doc + Video screen capture
Moodle Assignment; Annotated MS Word doc + Audio recording
Moodle Assignment; Annotated MS Word doc +Audio recording
Moodle Assignment; Annotated MS Word doc +Audio recording
2008Feedback provision
Online discussion forum; Track changes/ annotated script MS Word
Annotated MS Word doc + Written Comments
Annotated MS Word doc + Video screen capture
Annotated MS Word doc + Video screen
Annotated MS Word doc + Video screen
Annotated MS Word doc + Video screen
2009 and 2010Feedback provision
Written proformas via Pebblepad
Written proformas via Pebblepad
Annotated MS Word doc + Video screen capture
Annotated MS Word doc + Audio recording
Annotated MS Word doc +Audio recording
Annotated MS Word doc +Audio recording
Data Collection• All tutors kept details feedback activity logs relevant to
the technologies being usedTrack Changes + Audio recording
Track Changes + Screen Capture
Track Changes + Written Proforma
Turnitin (Quickmarks, Comments, Audio)
Download Download Download
Annotation (Track Changes)
Annotation (Track Changes)
Annotation (Track Changes)
Annotation (Quickmarks)
General Text Comment
Recording Recording Recording
Publishing
Upload Upload Upload
Logistical considerations• DCVLEs & DEBOL versions
Core Option 10 credits – Nominal 100 hrs• Very limited F2F contact (4 x 3 hours sessions - 12 hours
in total)
• ELTAT• 30 Credits – Nominal 300 hours• More F2F • Over a long period