Online peer assessment: an exploratory case study in a higher education civil engineering course

SPEE Special Session "Talking about Teaching 2012"

(TAT'12)26 - 28 September 2012, Villach, Austria

Online peer assessment: an exploratory case study in

a higher education civil engineering course

Gordon GramsDep. Letters, Arts and Communication

UTAD - University of Trás-os-Montes e Alto DouroVila Real, [email protected]

Daniela PedrosaFaculty of Psychology and Educational Sciences

UC - University of CoimbraCoimbra, [email protected]

Gonçalo Cruz, Ana MaiaPro-Chancellery fo Innovation & Information Management

UTAD - University of Trás-os-Montes e Alto DouroVila Real, Portugal

{goncaloc, margaridam}@utad.pt

Caroline DominguezDep. Engineering

C-Made/UTAD - University of Trás-os-Montes e Alto DouroVila Real, Portugal

[email protected]



I. Purpose of the study

II. Caracterization of the activity

III. Research methodology

IV. Evaluation

V. Discussion and limitations

VI. Conclusions and future work

Agenda






IV. Evaluation



Agenda

What we are

talking

about? What was our

aim?

What literature says?



I. Purpose of the study – “What we are talking about?”

Civil engineering students were challenged to reinforce a set of cognitive and personal competencies, using a web-based collaborative writing environment and a peer assessment approach..

It’s all about learning...





It’s all about teaching...





It’s all about pedagogy...



I. Purpose of the study – “What literature says?”

• Effective communication skills is considered to be one of the main goals of engineering curricula.

• Students’ resistance to peer review, but only a few conclusions on which factors lay behind that resistance..

• Little understanding on the impact of students’ perceptions of peer reviewed work.

• When feedback is given only by peers it seems to be less enforcing.

• Feedback provides guidelines when it is specific and clear.



I. Purpose of the study – “What literature says?”

Adapted from Nelson, M. M., & Schunn, C. D. (2009). The nature of feedback: How different types of peer feedback affect writing performance. Instructional Science, 27(4), 375-401.



I. Purpose of the study – “What was our aim?”

a. Examine students’ attitudes towards online peer assessment

b. Evaluate students’ written communication skills• The type of feedback implemented by student-reviewers.

• The overall students’ appreciation: activity tasks; web-based learning environment; learning support materials; and received feedback.• The related factors with students’ use and appreciation of received feedback.






IV. Evaluation



Agenda

The context and the related competencies

The activity

tasks

The learning support materials

What was the approach?

The web-

based

learning

environment



II. Caracterization of the activity – The context and the related competencies

Midcourse level (3rd semester of a 6-semester program) of a Bachelor of Civil Engineering course, focusing only on one component of the syllabus of Management of Construction Enterprises Curricular Unit.Familiarize students in the exercise of the SWOT (Strengths, Weaknesses, Opportunities, and Threats) economic strategic analysis approach.

Students had to identify and characterize the opportunities and threats that construction companies (and other agents) faces according its economic, technological, legal, political and social-cultural environment.

Reinforce/develop writing skills, synthesis, analysis, critical thinking, autonomy, interpersonal communication, use of ICT and group work.



II. Caracterization of the activity – The activity tasks

1st

week

•Select an economic article from one available source, dealing with recent news of an economic phenomenon;

•Perform a summary of the article;

•Analyze the article, identifying and classifying the type of variables;

•Evaluate them in terms of Opportunities and Threats;

•Give his/her own opinion on the reported news;

For the authors




2nd

week

•Read the document and give written feedback with suggestions for improvement

•Evaluate the colleague’s work on a 0 to 1 grading scale, taking into account evidence of the summary, the identification of all the variables of the economic environment, the conclusions in terms of opportunities and threats, and the elaboration of a well-grounded final individual commentary.

For the reviewers




2nd

week

•Made a revision of the work done by the author and the reviewer colleague.

•Evaluate both of them with the same grading scale (0 to 1).

For the teacher




3rd

week

•Make voluntary improvements to his/her original document.

For the authors




4th

week

•Give a final grade to the authors.

For the teacher




2 Cycles during a semester

The final grade was the average from the teacher’s final grades given to both assignments.

One third of the total grade of the curricular unit (in a grading scalefrom 0 to 20 values).



II. Caracterization of the activity – The web-based learning environment












II. Caracterization of the activity – The learning support materials

Guidelines




Management of the activity (1)




Management of the activity (2)






IV. Evaluation



Agenda

Which instruments? For what data?Which steps?



III. Research methodology – “Which instruments? For what data?”

Two different instruments

1. Questionnaire

2. Students’ assignments and grades





1. Questionnaire

46 quantitative questions:

•12 were related to the characterization of students’ profile (sex, age, previous digital skills);

•17 with the pedagogical approach adopted (execution of the tasks, facility of use the online environment, time availability and support material);

•6 dealt with the students’ perception about the quality of the feedback (its value and impact to the writing improvement);

•10 on the acquired skills and the general satisfaction with the activity;





1. Questionnaire

4 qualitative open questions allowed students to state why they found (or not) important the suggestions made by the peer-reviewers and by the teacher, and why they used (or not) these suggestions to improve their own work.

After the data collection, a simple descriptive statistical analysis was made for the quantitative questions, limited to counting frequencies of the response modalities for each question.

For the qualitative questions, the nature of each response was verified through a structured content analysis grid.





2. Students’ assignments and grades

The type of feedback made by student-reviewers and by the teacher was analyzed in each document, using the feedback model adapted from Nelson & Schunn (2009). An analysis grid corresponding to a checklist of analyzed criteria was elaborated.

A total of 39 assignments (41%) of 81 were selected and analyzed. They all had feedback given, both by peers and by the teacher.

Finally, the average of the intermediate grades given by reviewers and teachers was calculated for additional information.



III. Research methodology – “Which steps?”

1. Sample characterization;

2. Quantitative and qualitative analysis of student perceptions;

3. Description of the type of feedback found in the final documents with a comparison of the grades given by student-reviewers and the teacher;

Three main steps






IV. Evaluation



Agenda

Use of the Google Docs environment

Characterizati

on of the

sample

Students’

perceptions

based on

quantitative

data

Type of

feedback and

grades given

by students

Students’ perceptions based on qualitative data



IV. Evaluation – Characterization of the sample

55% of the students were male.

Previously, most of the students used social networks and e-mail on a daily base (73% and 77%, respectively).

Only one student already knew about or had previously used Google Docs or a similar online environment.

95% of the students performed all the proposed mandatory tasks.

23 students from a total of 47(49%)...



IV. Evaluation – Use of the GoogleDocs environment

All of the students (100%) considered Google Docs to be a useful and efficient platform for the proposed tasks;

85% agreed that Google Docs made communication between colleagues and the teacher easier;

77% of the students evaluated the support documents for the activity (tables and guideline document) positively, allowing better organization, coordination and management of the activity;

55% considered the time available to perform the activity to be long enough;;



IV. Evaluation – Students’ perceptions based on quantitative data

You can see the detailed analysis here:

Cruz, G.; Dominguez, C.; Pedrosa, D.; Maia, A. (2012). A avaliação entre pares com recurso ao Google Docs: um estudo de caso exploratório num curso de Licenciatura em Engenharia Civil, in Livro de Atas - Conferência Ibérica em Inovação na Educação com TIC – Bragança.



IV. Evaluation – Students’ perceptions based on qualitative data


















IV. Evaluation – Type of feedback and grades given by students















IV. Evaluation



Agenda

What we can

draw?What we can improve?



V. Discussion and limitations – “What we can draw?”

The use of an online environment did not represent any difficulty for students, contrary to some literature. The support material prepared proved to be very important for the realization of the activity.

Students have a satisfactory level of written communication skills. However, some negative perceptions of feedback remain due to the lack of clarity in some cases, non specification of problems and their location, as well as no provision of solutions and explanations.

Thus, from a pedagogical point of view, it seems necessary to perform one or more training sessions with students on “what are the characteristics of a good feedback.” These in-class practice evaluation sessions can use a demonstration of good practices, models, key aspects, etc., with discussion on how to improve the arguments made .




Another factor which appears to influence the use (or non use) of peer or teacher feedback is the evaluation grades.

The small number of students who used the feedback to improve their work in the analyzed documents (= 3) and the high average (0.86 in a scale from 0 to 1) of the intermediate and final grades assigned by the teacher to each document leads us to consider the importance of testing the following question in future research:

How does the intermediate grade evaluation by the teacher influence the use of peer feedback by the student author for improving his/her work?




The final important factor in our analysis appears to be teacher status, specifically the overvaluation of teacher feedback compared with the reviewers,' as other studies already have demonstrated.

However, when we look at the grades assigned by the teacher and the peers, there was not a significant difference.

These results lead us to question the need to perform graded assessment of work at the intermediate level by both the teacher and peers (although this would require awareness and credibility of their own grading ability from the students, contrary to their current beliefs).



V. Discussion and limitations – “What we can improve?”

The large number of responses listed in the sub-category of “general perceptions” did not allow a good comprehension of the students' reasons for appreciating or using feedback. This results from a limitation of the research methodology, which was based exclusively in the use of a single instrument of data collection (in this case, the questionnaire). To overcome this problem, we suggest the use of other instruments and / or data collection methods such as personal or focus group interviews.

No data correlation between the instruments (questionnaire was anonymous).

No blind peer-review.

The adopted grading scale (students didn’t know what are the difference, for example, between a 0.7 grade to a 0.8).






IV. Evaluation



Agenda

What’s the

final remarks?

What’s the next step?



VI. Conclusions and future work – “What’s the final remarks?”

Digital competencies do not represent any difficulty for the realization of the activity. However, good preparation and support guidelines are essential for its success.

Relevant factors underlying the students' appreciation and usefulness of received feedback, in particular by the peer reviewers, were identified: type of feedback, influence of the teacher status (versus peers), and grades.

The written communication skills seems to be quite developed and applied, although they can be improved at the level of summarization and provision of solutions. Thus, it is important to spend more time in developing skills on how to deliver good feedback, improving effective communication, for example by demonstrating successful models and case studies on feedback as well as the benefits of this type of activity.

The low rate of voluntary feedback leads us to question the influence of intermediate grading in the process, which could be better investigated in future research.



VI. Conclusions and future work – “What’s the next step?”

We are deepen the analysis on peer feedback, moving the research to other dimensions such as the effect of online peer assessment on critical thinking, and designing an research proposal will allow for follow up of each student’s results during the next academic semester.

We are actually doing that..

3 Groups of students: Grades + Comments; Only grades; Only comments;Cornell Critical Thinking Test (X) (1985) - to apply as a Pre- and Pos-testEnnis Critical Thinking Taxonomy (1985) - to measure the learning outcomesEnnis FRISCO Approach (1985) - to promote critical thinking competencies



Thanks for listeningHope to hear from you...

@GCruzMatos or @elearning_utad

Facebook.com/goncalocruzmatos

Msu Speiser

[email protected]

Date post:	27-May-2015
Category:	Education
Upload:	goncalo-cruz-matos
View:	460 times
Download:	0 times