Date post: | 17-Jun-2015 |
Category: |
Technology |
Upload: | ricardo-queiros |
View: | 598 times |
Download: | 3 times |
Programming Exercises Evaluation Systems:An Interoperability Survey
Ricardo Queirós and José Paulo LealFaculty of Sciences of University of Porto
4th International Conference on Computer Supported Education (CSEDU’12)Porto, Portugal
April 16, 2012
Introduction Related Work Interoperability Survey Synthesis Conclusion
Outline
1 Introduction
2 Related Work
3 Interoperability Survey
4 Synthesis
5 Conclusion
Queirós & Leal PES: An Interoperability Survey 16-04-2012 2 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
xxxContext
Learning programming requires solving programming exercises.Manual assessment of exercises:
Time consuming - teachers need to assess a large number ofexercises (e.g. large classes)Error prone - hinders the consistency and accuracy of assessmentresults
Existent automatic evaluation systems aretoo genericfocused on simple domains (e.g. quizzes)
Appearance of
Programming Exercises Evaluation Systems (PES)
Queirós & Leal PES: An Interoperability Survey 16-04-2012 3 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
xxxProgramming Exercises Evaluation Systems
Automatic tools tograde students’ programming exercisesgive feedback on the quality of students’ solutions
Used on different learning scenarios:Curricular (e.g. practical classes, assignments and exams)Competitive (e.g. programming contests)
IOI - for secondary school studentsACM-ICPC - for university studentsIEEExtreme - for IEEE student members
Examples:AutoGrader, BOSS, Hustoj, Mooshak, WEB-CAT, etc.
Queirós & Leal PES: An Interoperability Survey 16-04-2012 4 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
xxxExisting surveys coverage
Several surveys enumerates and compares PES features...how the analysis of the code is madehow the tests are definedhow grades are calculated
...but neglects the interoperability feature. Organized at two levels:contentcommunication
Queirós & Leal PES: An Interoperability Survey 16-04-2012 5 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
xxxExisting surveys coverage
Content - how a programming exercise is describedas a Learning Objectcomposed by assets such as statement, test cases, feedback, etc.
communication - how it should be shared among systems:
LearningObjectsRepository
LearningManagementSystem
ProgrammingEvaluationSystem
1
2
3
4LO reference, Learner reference,
attemp t
LO reference LO
Evaluatio n repo rt
Queirós & Leal PES: An Interoperability Survey 16-04-2012 6 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
xxxProject
Work descriptionAn interoperability survey on existing PES
Main goalto gather information on the interoperability features of existent PES
Benefitsfills the gap found in most surveys on PES interoperability featureshelps in the PES selection to integrate in e-Learning environments
Queirós & Leal PES: An Interoperability Survey 16-04-2012 7 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Evolution of assessment tools
Evolution of # systems
Evolution of features
Punched cards
Support for one language (e.g. Algol)
No feedback
No administration facilities
Command-line interfaces (manual operation of scripts)
Support for few languages (e.g. C, JAVA)
Limited feedback
Content management
Static analysis
Student automated testing
Grading-support system
Competitive learning (e.g. contests)
Web based interfaces
Multi-languages (Prolog, SQL, FORTRAN)
Richer and incremental feedback
Course/student administration facilities
Sophisticated testing approaches
Automatic test generation
Plagiarism detection
Service-oriented
Integration with LMS
Early Assessment Syst ems (1960 - 1985) Tool Oriented Systems (1985 - 2000) Web-Oriented Syst ems (2000 - ... )
Figure: Petcha use cases.
Queirós & Leal PES: An Interoperability Survey 16-04-2012 8 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Recent surveys
Five surveys:Douce et al. (2005)
Methodology: details features of PES organized in 3 generationsTrend: evaluation of GUI programs, meta-testing (evaluation of thestudents’ tests), SOA and use of interoperability standards
Kirsti AlaMutka (2005)Methodology: organizes PES in dynamic and static evaluatorsTrend: content and communication standardization.
Liang et al. (2009)Methodology: details dynamic and static analysis methods of PESTrends: security, algorithms for automatic generation of test dataand content standardization
Ihantola et al. (2010)Methodology: discuss PES (2006-2010) features (e.g. testsdefinition, resubmission policies and security)Trends: integration with LMS and assessment of web applications
Romli et al. (2010)Methodology: approaches for test data generationTrends: test data generation techniques, interoperability and security
Queirós & Leal PES: An Interoperability Survey 16-04-2012 9 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Interoperability analysis
Survey on PES interoperability featuresApproach for the selection of
the sample - 15 toolsbased on its effective use (exercises and users management)selected tools: AutoGrader, BOSS2, CourseMaker, CTPracticals,DOMJudge, EduComponents, GAME, HUSTOJ, Moe, Mooshak,Peach3, Submit!, USACO, Verkkoke, Web-CAT
the criteria - 3 facetsbased on surveys trends and our backgroundselected facets: programming exercises, users and assessment results
Queirós & Leal PES: An Interoperability Survey 16-04-2012 10 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Interoperability analysis
These 3 facets are synchronized with PES main objective
to evaluate a user’s attempt to solve a programming exercise andproduce an assessment result.
For each facet each PES is evaluated by 3 interoperability maturity levels:
Level 0 - manual configuration of dataLevel 1 - data import/exportLevel 2 - services invocation
Queirós & Leal PES: An Interoperability Survey 16-04-2012 11 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Programming Exercises
Programming Exercises facetLevel 0 - manual configuration of exercisesLevel 1 - import/export of exercisesLevel 2 - integration with repository services
Systems Level 0 Level 1 Level 2AutoGrader F - -BOSS2 F - -
CourseMaker F - -CTPraticals F - -DOMJudge F - -
EduComponents F - -GAME F - -
HUSTOJ F P -Moe F P -
Mooshak F F FPeach3 F P -Submit! F - -USACO F - -Verkkoke F F -Web-CAT F F P
All systems support the configuration of exercises6 tools export exercises; 3 bidirectional support; few systems useexercises formats2 tools support communication with repositories through SOA
Queirós & Leal PES: An Interoperability Survey 16-04-2012 12 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Users
Users facetLevel 0 - manual configuration of users;Level 1 - import/export of users;Level 2 - integration with user directories services (authentication)and AMS (authorization)
Systems Level 0 Level 1 Level 2AutoGrader F F PBOSS2 F - -
CourseMaker F - -CTPraticals F F PDOMJudge F F P
EduComponents F F PGAME F - -
HUSTOJ F - -Moe F - -
Mooshak F F PPeach3 F - -Submit! F - -USACO F F -Verkkoke F F -Web-CAT F F -
All systems support the manual configuration of users8 tools allow the import/export of users in non-standard formats5 tools communicates with authentication services (LDAP)
Queirós & Leal PES: An Interoperability Survey 16-04-2012 13 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Assessment results
Assessment results facetLevel 0 - visualization of evaluation resultsLevel 1 - export of assessment resultsLevel 2 - integration with LMS
Systems Level 0 Level 1 Level 2AutoGrader F F PBOSS2 F - -
CourseMaker F - -CTPraticals F F PDOMJudge F F -
EduComponents F F PGAME F - -
HUSTOJ F - -Moe F - -
Mooshak F F -Peach3 F F -Submit! F - -USACO F - -Verkkoke F F PWeb-CAT F F -
All systems present the evaluation results to usersThe majority allows its exportation in non-standard formats4 systems support the communication with LMS
Queirós & Leal PES: An Interoperability Survey 16-04-2012 14 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Interoperability facets coverage
Mooshak, Web-CAT, Verkkoke offer the best interoperability levelsHalf of the systems studied did not reach 50% of the maturity rateThere are a lot to do regarding PES interoperability
39%
44%
50%
61%
67%
67%
67%
72%
72%
83%
Moe
USACO
Peach3
DOMJudge
AutoGrader
CTPracticals
EduComponents
Verkkoke
Web-CAT
Mooshak
Sy
ste
ms
33%
33%
33%
33%
39%
39%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
BOSS2
CourseMaker
GAME
Submit!
HUSTOJ
Moe
% of maturity
Queirós & Leal PES: An Interoperability Survey 16-04-2012 15 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Interoperability features coverage
There is no specific trend on interoperability featuresNevertheless, programming exercises facet presents the lower resultsNeed to standardize:
the description of programming exercisesthe communication of PES with other systems (e.g. repositories)
53%43% 44%
30%
40%
50%
60%
70%
80%
90%
100%
% o
f co
ve
rag
e
not covered
covered
46%56% 55%
0%
10%
20%
30%
programming exercises user assessment results
Interoperability facets
Queirós & Leal PES: An Interoperability Survey 16-04-2012 16 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Conclusion
This work presents an interoperability survey on PES15 tools were surveyed based on 3 interoperability trend facets
programming exercisesusersassessment results
Issues detected on most PES regarding interoperabilitythe lack of standards to
describe programming exercises and assessment resultscommunicate with other e-Learning systems (e.g. LMS, AMS, LOR)
Survey benefitsis the first interoperability study on existing PEScan base the
selection of existing PES in e-Learning environmentsdesign of interoperability requirements for the creation of new PES
Queirós & Leal PES: An Interoperability Survey 16-04-2012 17 / 18
Introduction Related Work Interoperability Survey Synthesis Conclusion
FakeTitle1Questions
Thanks for your attention
Ricardo Queiró[email protected]
Queirós & Leal PES: An Interoperability Survey 16-04-2012 18 / 18