+ All Categories
Home > Documents > PEER EVALUATION SYSTEM FOR MOOC PEAS: Peer Expert ...€¦ · peer evaluation system for mooc peas:...

PEER EVALUATION SYSTEM FOR MOOC PEAS: Peer Expert ...€¦ · peer evaluation system for mooc peas:...

Date post: 25-Sep-2020
Category:
Upload: others
View: 1 times
Download: 0 times
Share this document with a friend
of 1 /1
Contributors: Jaspal Singh Kshitij Jain Nikhita Vedula Priyanka Mathur Saumya Agrawal Priya Agrawal SELF EVALUATION Occurs when learners assess their own performance. Advantages -Students learn to: ? Objectively reflect on and critically evaluate their own progress and skill development. ? Identify gaps in their understanding and capabilities. ? Discern how to improve their performance. ? Learn independently and think critically. EXPERT EVALUATION Traditional grading technique in which the instructor examines the solution of the student. Advantages- ? Assures correct evaluation. ? highly reliable. ? higher satisfaction among students. AUTOGRADE Makes use of external auto grader service. Advantages- ? Highly efficient. ? ? Accurate for checking program codes and multiple choice questions(MCQ). ? ? Faster results. PEER EVALUATION Process in which students or their peers' grade assignments or tests based on a teacher's benchmarks. Advantages - ? Empower students to take responsibility for, and manage, their own learning. ? Enable students to learn to assess and to develop life-long assessment skills. ? Enhance students' learning through knowledge diffusion and exchange of ideas. ? Motivate students to engage with course material more deeply. Features of peer evaluation in PEAS ? Calibration ? Optimal Distribution of reviewers is done to maintain the uniformity of the review process. ? Peer Review is the grading process where each reviewer grades the peer in accordance to the rubrics. ? Normalisation ? Incentive Mechanism to promote honest grading. Calibration The calibration method is employed in the beginning when no user profile is maintained. The different calibration techniques supported are : l Peer calibration l Expert calibration l No calibration (random grouping) Normalization PEAS uses a normalization method to ensure fair grades, which is used to eliminate problems of student being generous and biased in giving scores. ü ? SELF EVALUATION The main objective of this module is to promote learning among students. There are two different self grading models in PEAS: ? Pure self (for learning purpose only) Hybrid self (for learning as well as grading) Hybrid Self Evaluation ? Normalization-It is used to prevent self over grading . ? Incentive mechanism-Incentive to ensure students follow rubrics guidelines and do honest grading. ü EXPERT EVALUATION Here, the instructor of the course can himself evaluate the students' submissions. PEAS uses Instructor grading for: ? Solving discrepancy in case of marks issue/student dissatisfaction. ? Providing training dataset to Machine Learning Algorithms. ? Provides reliable grades to students. ü AUTOGRADER PEAS has an auto grading module which makes use of external auto grader service for grading assignments. ? Uses Discern API of edX to use its EASE auto grading library for essays. ? Incorporates programming language compilers to auto grade programming codes. FUTURE SCOPE ? a separate grading module. ? PEAS API will be released as an open source grading interface. PEAS can be integrated with edX as Objective: ? To provide highly reliable/accurate assessment. ? To allocate a balanced and limited workload across students and course staff. ? It should be scalable to class sizes of tens or thousands of students. ? PEAS applies broadly to a diverse collection of problem settings. Our evaluation system PEAS ? Provides highly reliable/accurate assessment. ? Allocates a balanced and limited workload across students and course staff. ? Scalable to class sizes of tens or thousands of students. TABLE 2: COMPARISON OF PEAS SELF GRADING WITH EDX ORA PEER EVALUATION SYSTEM FOR MOOC PEAS: Peer Expert Autograde Self FEATURES FEATURES EDX ORA PEAS SCALABLE FOR A MOOC ß ß EXPERT EVALUATION FOR STUDENT GRADES ü ß TRAINING DATA SET FOR MACHINE LEARNING ü ü SOLVING DISCREPANCY ß ü TABLE 3: COMPARISON OF PEAS EXPERT GRADING WITH EDX ORA FEATURES EDX ORA PEAS CALLIBRATION ü ü COMPULSORY CALIBRATION ü ß NORMALISATION ß ü INCENTIVE ß ü OPTIMAL DISTRIBUTION ß ü PEER REVIEW ü ü TABLE 1: COMPARISON OF PEAS PEER GRADING WITH EDX ORA PEER EVALUATION MOOC Evaluation strategies ü In peer assessment, PEAS reliably measures the performance of students without the need for expert's intervention. PEER EVALUATION Fundamental Research Group, Dept of CSE, IIT Bombay Summer Internship 2013(9th May to 7th July) email: [email protected] design by: Bhairav
Transcript
Page 1: PEER EVALUATION SYSTEM FOR MOOC PEAS: Peer Expert ...€¦ · peer evaluation system for mooc peas: peer expert autograde self features features edx ora peas scalable for a mooc ß

Contributors: Jaspal Singh Kshitij Jain Nikhita Vedula Priyanka Mathur Saumya Agrawal Priya Agrawal

SELFEVALUATION

Occurs when learners assess their own

performance. Advantages -Students learn to:?Objectively reflect on and critically evaluate their own progress and skill development.?Identify gaps in their understanding and capabilities.?Discern how to improve their performance.?Learn independently

and think critically.

EXPERT EVALUATIONTraditional grading technique in which the

instructor examines the solution of the student.Advantages-?Assures correct evaluation.?highly reliable.?higher satisfaction among students.

AUTOGRADE

Makes use of external auto grader service.

Advantages-

?Highly efficient.?

?Accurate for checking program codes and multiple choice

questions(MCQ).?

?Faster results.

PEER EVALUATION

Process in which students or their peers' grade assignments or tests based on a teacher's benchmarks.

Advantages -?Empower students to take responsibility for, and manage, their own learning.

?Enable students to learn to assess and to develop life-long assessment skills.?Enhance students' learning through knowledge diffusion and

exchange of ideas.?Motivate students to engage with course

material more deeply.

Features of peer evaluation in PEAS?Calibration?Optimal Distribution of reviewers is done to maintain the uniformity of the review process.?Peer Review is the grading process where each reviewer grades the peer in accordance to the rubrics.?Normalisation?Incentive Mechanism to promote honest grading.

Calibration The calibration method is employed in the beginning when no user profile is maintained. The different calibration techniques supported are : lPeer calibration lExpert calibration lNo calibration (random grouping)

Normalization PEAS uses a normalization method to ensure fair grades, which is used to eliminate problems of student being generous and biased in giving scores.

ü

?

SELF EVALUATIONThe main objective of this module is to promote learning among students. There are two different self grading models in PEAS:

? Pure self (for learning purpose only) Hybrid self (for learning as well as grading)

Hybrid Self Evaluation?Normalization-It is used to prevent self over grading .?Incentive mechanism-Incentive to ensure students follow rubrics guidelines and do honest grading.

ü EXPERT EVALUATIONHere, the instructor of the course can himself evaluate the students' submissions. PEAS uses Instructor grading for:?Solving discrepancy in case of marks issue/student dissatisfaction.?Providing training dataset to Machine Learning Algorithms.?Provides reliable grades to students.

ü AUTOGRADER

PEAS has an auto grading module which makes use of external auto grader service for grading assignments. ? Uses Discern API of edX to use its EASE auto grading library for essays.? Incorporates programming language compilers to auto grade programming codes.

FUTURE SCOPE

?

a separate grading module.?PEAS API will be released as an open source grading interface.

PEAS can be integrated with edX as

Objective:?To provide highly reliable/accurate assessment.?To allocate a balanced and limited workload across students and course staff.?It should be scalable to class sizes of tens or thousands of students.?PEAS applies broadly to a diverse collection of problem settings.

Our evaluation system PEAS

?Provides highly reliable/accurate assessment.?Allocates a balanced and limited workload across students and course staff.?Scalable to class sizes of tens or thousands of students.

TABLE 2: COMPARISON OF PEAS SELF GRADING WITH EDX ORA

PEER EVALUATION SYSTEM FOR MOOC PEAS: Peer Expert Autograde Self

FEATURES

FEATURES EDX ORA PEAS

SCALABLE FOR A MOOC û û

EXPERT EVALUATION FOR

STUDENT GRADES

ü û

TRAINING DATA SET FOR

MACHINE LEARNING

ü ü

SOLVING DISCREPANCY û ü

TABLE 3: COMPARISON OF PEAS EXPERT GRADING WITH EDX ORA

FEATURES EDX ORA PEAS

CALLIBRATION ü ü

COMPULSORY CALIBRATION ü û NORMALISATION û ü INCENTIVE û ü

OPTIMAL DISTRIBUTION û ü

PEER REVIEW ü ü

TABLE 1: COMPARISON OF PEAS PEER GRADING WITH EDX ORA

PEER EVALUATION

MOOCEvaluationstrategies

ü

In peer assessment, PEAS reliably measures the performance of students without the need for expert's intervention.

PEER EVALUATION

Fundamental Research Group, Dept of CSE, IIT Bombay

Summer Internship 2013(9th May to 7th July) email: [email protected]

des

ign

by: B

hair

av

Recommended