Date post: | 28-Mar-2015 |
Category: |
Documents |
Upload: | valeria-polley |
View: | 215 times |
Download: | 1 times |
The Role of Peer Review in Supporting the Sustainability of Technology-Enhanced
Learning Environments
Pantelis M. Papadopoulos, United Nations UniversityAntonio Cerone, United Nations University
2 / 24
Overview
Sustainability in Educational Technology
Research domain and argument
The peer review process
Collaboration script
Three studies on peer reviewThe role of coercionFree-selectionIndirect feedback
3 / 24
Sustainability in Educational Technology
Multiple definitions of sustainability in ETOrganizational, financial, technical, institutional, etc
We focus on pedagogy, using sustainability to refer toLife cycle of a technology-enhanced learning environment (TELE)Ability of a TELE to sustain an active group of students
Even with institutional support and financial viability, a TELE may fail, if it does not capture the interest of the students
Link between sustainability and student engagement
4 / 24
Research domain and argument
DomainPeer review method in computer supported collaborating learning (CSLC)
Student engagement can be affected byIndividual factors: learning profile, intrinsic motives, etcLearning activity characteristics: peer interaction, scaffolding method, etc
Our argumentPeer review method can (a) enhance student engagement, (b) support the creation of sustainable a learning community, and consequently support the sustainability of the TELE used by the community
Literature already reports multiple benefits from peer review in many levels
5 / 24
The peer review method
Four major phasesProducing initial work
Students work individually/collaboratively to produce the initial draft
Assigning reviewersThe teacher/system assigns peer work to reviewers
Review submissionReview comments are sent back to the authors
Revision and final version The author revises the initial draft according to reviews and creates the final version
6 / 24
Collaboration script
Didactical script guiding students into meaningful learning interactionsEven in scripted collaboration, there is a distance between the prescribed task and actual implemented activity
External script: the activity as prescribed by the teacher
Internal script: the mental representation of the script that the group builds from teacher's prescription
Actual script: the actual task and interactions that learners engage into
First Study
Peer review and script coercion
8 / 24
First Study: Method
Two groups of juniors majoring in Informatics studied in a TELE20 students in Low Coercion condition22 students in High Coercion conditionWe randomly assigned students into same-sex dyads
Collaboration scriptAll students worked individually to produce the initial draftStudents in dyads reviewed each other's work following a set of guidelinesStudents had to discuss, reach a consensus, and form a final common answer
Low Coercion group: submitting reviews in the TELE was optionalHigh Coercion group: submitting reviews in the TELE was mandatory
9 / 24
First Study: Test results
Five phases: pre-test, individual study, collaboration, post-test, interviewPre-test: t-test results showed that the two groups were comparable regarding prior domain knowledge (p>0.05)Post-test: ANCOVA results showed that the High Coercion group significantly outperformed the Low Coercion group (p<0.05)
High Coercion Low Coercion
M SD n M SD NPre-test 2.24 (0.71) 22 2.13 (0.59) 20Post-test 7.42 (1.30) 22 6.36 (0.83) 20
10 / 24
First Study: Collaboration patterns
Analysis of collaboration patterns based on: (a) student statements in the interviews, (b) comparison of individual and collaborative answers in the TELE, and (c) the TELE log files
Collaboration patterns: Ideal, moderate, weak
Only 1 out of 10 Low Coercion dyads submitted written reviews in the TELEThe others said that they shared review comments during discussionEven if review comments were shared, they would be unstructured and scattered in the discussion
Low Coercion group: 2 dyads in the "ideal", 5 dyads in the "moderate", and 3 in the "weak"High Coercion group: 5 dyads in the "ideal", and 6 dyads in the "moderate"
11 / 24
First Study: Conclusions
Script coercion can be used by the teacher as a mean to engage students into the activity
When left to decide, almost all the students in the Low Coercion group opted-out of the review process as described in the guidelines
Students in the High Coercion group demonstrated better collaboration patterns and post-test performance
Second Study
The free-selection protocol
13 / 24
Second Study: Method
Two groups of sophomores majoring in Informatics and Telecommunications Engineering studied in a TELE
20 students in Assigned-Pairs (pre-defined dyads by the teacher)22 students in Free-Selection
Collaboration scriptAll students worked individually to produce the initial draftStudents reviewed each other's drafts through a double-blinded processStudents had to review the initial submissions taking into account review comments
14 / 24
Second Study: Review conditions
Assigned-PairLearners in an author-reviewer dyad are assigned exclusively to each otherWe randomly assigned students into dyads
Free-SelectionLearners have access to all the submission made by the rest of the class
More points of view accessible
They can read and review as many submission they like
min. at least one review shift to dyad format
15 / 24
Second Study: Test results
Five phases: pre-test, study, review & revise, post-test, interviewPre-test: t-test results showed that the two groups were comparable regarding prior domain knowledge (p>0.05)Post-test: ANCOVA results showed that the Assigned-Pair group significantly outperformed the Free-Selection group (p<0.05)
Assigned-Pair Free-Selection
M SD n M SD NPre-test 2.69 (1.07) 20 2.59 (0.83) 17Post-test 7.71 (0.95) 20 8.43 (0.81) 17
16 / 24
Second Study: Student attitudes
Students in the Free-Selection group studied on average 8 and reviewed 2 out of the 16 drafts submitted by their peers
Two students did not receive reviews and we asked selected students to provide the missing reviews
Interviews: Two main trends in selecting drafts by FS studentsFind good answers to be able to give nice commentsFind weak answers to be able to provide more meaningful reviews
Interviews: FS student submit more reviews than asked because:It was a good exercise for them to clarify their thoughtsAfter a while it was easy to do and it would increase the possibility of everyone receiving at least one review ( community culture)
17 / 24
Second Study: Conclusions
FS students…Took advantage of the multiple points of view offered in their peers' drafts
Engaged deeper in the activity voluntarily far exceeding the minimum effort requirements
Explicitly reported a community culture in submitting more reviews
Had a more positive opinion about the activity than the AP students, despite the fact that the latter had to work less
Third Study
What if there is only indirect feedback?
19 / 24
Third Study: Method
Based on the findings of the Second StudyIndirect feedback: Students get new insights by reading others' draftsCan the lack of review comments be address by indirect feedback and self-review?
Two groups of sophomores majoring in Informatics and Telecommunications Engineering studied in a TELE
Free-selection protocol20 students in Self-Review (their drafts were excluded from review and they had to fill in a self-review form before revising their drafts)18 students in Peer-Review (they all received at least one review)
Collaboration script: same as in Second Study
20 / 24
Third Study: Test results
Five phases: pre-test, study, review & revise, post-test, interviewThe two groups were comparable both in pre-test and post-test (p>0.05)
Self Review Peer Review
M SD n M SD NPre-test 2. 20 (1.09) 20 2.04 (1.06) 18Post-test 8.19 (1.37) 20 8.13 (1.40) 18
21 / 24
Third Study: Student attitudes
Same attitudes as in Second Study
Students demonstrated strategies that far exceeded the minimum effort requirement
Interviews: Same two trends in selecting draftsFind good answers to be able to give nice commentsFind weak answers to be able to provide more meaningful reviews
22 / 24
Third Study: Conclusions
The study tried to provide evidence that reading other's draft can be equally beneficial to getting peer feedback
We supported non-reviewed students with a self-review process and results showed that they were comparable to typical Free-Selection students with peer reviews
Since receiving peer review is not the only way to receive feedback, it is easier for a student to participate in a learning community that applies the Free-Selection protocol
Even if the student does not get comments from peers, the previously submitted answers can provide valuable feedbackSelf-review can help the student perform comparative analysis and identify weak points in his own work
23 / 24
General Remarks
Peer review process can be an effective tool for structuring the interaction between students
When used right, the degree of coercion in scripted collaboration can result to deeper engagement and consequently to better learning outcomes
Free-selection protocol allows for more freedom, while it maintains a minimum level of engagement
FS protocol has resulted in increased student effort in both studies it was used
The community culture of working more to benefit others also appeared in the two studies with FS protocol
The indirect feedback in FS makes the protocol even appropriate for learning communities
24 / 24
Thank you!
http://to.ly/fyqO