WP4.1 e-‐Assessment Field tests
D 4.1.4 Results on the use of SEB with LMS
D 4.1.5 Results on the use of SEB on students computers
D 4.1.6 Results of the tests of SEB on public computers
at UniGe
University of Geneva,
Laurent Moccozet
Sept. 12, 2013
WP 4.1; D4.1.4-‐6 Learning Infrastructure UniGe
Page 2 of 7
I) Tests with LMS 1) The “certificat de formation continue en obstétrique, médecine maternelle et fœtale" is a certificate in continuous education offered at the University of Geneva: http://www.unige.ch/formcont/pathologiegrossesse.html
It is a continuing education program for physicians, midwives and nurses practicing in the field of obstetrics, pediatrics, neonatology, pedopsychiatry, anesthesia, anatomical pathology and general medicine. The training is planned in 9 modules from January to September 2013 and a final work for a total of 10 ECTS credits. Each module articulates theoretical knowledge and practical skills. The certificate currently enrolls 16 students.
A Dokeos platform was opened in January 2013. The features that were used were: the calendar (with dates of course), the description of the course and teaching materials (course presentation mostly) and the regulation studies. There was no use of forum. The satisfaction evaluation was made through the platform, but relayed on e mails students. The Dokeos platform (then moved to Chamilo) was little used by the students. When preparing for the exam online, some students had never connected.
The assessment takes the form of a MCQ evaluating the knowledge and skills acquired during the training modules. Each module includes several speakers. The program coordinator has asked speakers for questions for each module. He endorsed all the issues with the program director, professor in the faculty of medicine, in relation to the skills and knowledge expected. The result was implemented as a 60 questions MCQ on Moodle. The choice of Moodle instead of Dokeos/Chamilo has been imposed by the requirement for K’ type questions. This type of question is not available on the Chamilo platform in Geneva.
The MCQ takes place in a computers room usually used for seminars and workshops. The room is not freely accessible to students. The room contains 60 computers (see photo). It is particularly suited for conducting online assessments as each computer is inside a cubicle. PCs are running Windows XP and boot up from a Rembo server. This configuration will change in September 2013 and PCs will be updated to Windows 7 with a new boot server system.
WP 4.1; D4.1.4-‐6 Learning Infrastructure UniGe
Page 3 of 7
SEB has been pre-‐installed on the disk image that the computers in the room are loading when students logging in. The SEB icon is directly available on the desktop so that students can directly click on the icon to launch SEB. The link launched by SEB corresponds to an MCQ implemented with Moodle.
At the beginning of the test, students receive a paper with step-‐by-‐step guidelines to launch SEB and achieve the MCQ. Students log in on the computer, launch SEB from the desktop, log in to Moodle and start the MCQ and enter the MCQ password. They complete the MCQ and submit it. The MCQ was setup with a limited amount of time and only one possible trial. In case of problem, each student has received a paper copy of the MCQ for backup.
During the test we have encountered some problems: when some students were saving their answers, we have noticed that some answers changed when the MCQ page was refreshed. This was confirmed by comparing the answers of the paper copy with the online one. We are currently testing the setup to identify the problem.
Less critical issues include:
-‐ For a long MCQ (including 60 questions for the experiment), it is not a good idea to organize all the questions inside a single page.
-‐ The timer that is continuously displayed on the screen makes students anxious. -‐ Setting the duration of the exam (2 hours) in the setup of the MCQ does not seem
appropriate, as it is not possible to tune the duration if there is any problem. It should be more flexible to ask students to stop and submit the MCQ once the exam duration is over.
-‐ Same remark regarding the availability of the quiz. It should be opened much before and much after the expected time.
It appears that for students that are not in a computer science curriculum, online assessments make them anxious. It was important for the test to provide them with guidelines and take some time at the beginning of the session to explain them the process. It was also important to have people available in the room to answer technical questions or address technical problems.
Remark:
WP 4.1; D4.1.4-‐6 Learning Infrastructure UniGe
Page 4 of 7
Having the SEB icon directly available on the computer desktop is a good idea for the test itself. Students immediately identify the program and launch it. However there is a need to remove it outside exam period in order to avoid that any curious student could launch SEB and get blocked.
Further investigations:
Test the same setups with the new computers configuration; progressively extend these setups to bigger classes: 1) less than 50 to test the load with Moodle/Chamilo; 2) more than 50 to test how to manage the rotation of sub-‐groups in the room; organize a standard setup for online quiz assessment with guidelines and support documents for teachers.
This experiment is currently conducted in collaboration with Suzanne de Jonckheere from the continuing education office at the University of Geneva and Prof. Jean-‐Marie Pellegrinelli, head of clinic at the University Hospital of Geneva and program coordinator for the certificate.
2) The Faculty of Medicine is used to online assessments organized with a dedicated framework: Campus. However, it appears that this framework is not adapted for some topics that would require more freedom than what Campus can currently afford. We are currently experimenting how SEB and Chamilo or Moodle could be used.
For some topics such as histology, students are trained with histological sections. A virtual microscope has been developed and is made available online to students (http://vslwww.unige.ch/). A new system will be available for September 2013. This has many advantages over real microscopes: students can train from any place at any time where they have an Internet access; they can directly annotate histological sections…
The current exam sessions are organized as face-‐to-‐face oral assessments, with in 2nd year: 170-‐175 students and in 3rd year: 150-‐155 students. Each exam session takes 4 half–days with 32 students per half-‐day and mobilizes many teachers.
The objective is to move to a simulation based online assessment organized with a MCQ and a virtual microscope using a computer room with 128 seats and 64 computers (with polarized screens) that is currently used for online assessments.
WP 4.1; D4.1.4-‐6 Learning Infrastructure UniGe
Page 5 of 7
Figure 1 -‐ Virtual microscope homepage
Figure 2 -‐ An example of histological section
For the online assessment, students should be given access to the virtual microscope at the same time as they
Two setups are currently investigated:
-‐ Students answer the questions using a dedicated library of sections made available through the virtual microscope.
-‐ Students answer the questions using a dedicated library of sections made available through the virtual microscope and may have for some questions to justify their answer by capturing a snapshot of a section (and submit it with the answer to the question).
WP 4.1; D4.1.4-‐6 Learning Infrastructure UniGe
Page 6 of 7
The configuration that is currently investigated consists in using SEB with Chamilo (or Moodle). A dedicated course is created in Chamilo with a learning path that gives simultaneously access to the virtual microscope as a link and to the MCQ. Snapshots can be submitted by giving access to the assignment submission space of the course in Chamilo.
The configuration could not be tested in real conditions during exam sessions. Tests will be conducted in year 2013/2014.
This experiment is conducted in collaboration with Prof. Annelise Wohlwend at the University Hospital of Geneva.
II) Tests with students’ computers The course “Introduction à la programmation” is an introductory course to programming in Java. It takes place in the first year of the Information Systems bachelor. There are between 20 to 30 students. The course is supported with an LMS: Dokeos/Chamilo. Students all work with their personal laptops during the course and the seminars. The programming language is Java and programming is done with BlueJ, an open source cross platform IDE. The programming environment is installed on each personal laptop. The course and seminar takes place in a room with wireless electrical plugs available at each place so that students can easily charge their laptops. The tables can be moved and placed so that students cannot read others’ screen.
Different setups have been evaluated:
1) Open textbook exam: The goal is to avoid students having to print the course notes on paper for the exam, when course notes are authorized during a written exam or written MCQ. SEB is configured so that students are given access to an online dropbox giving access to all the documents in PDF format. They perform the exam on paper, either as a quiz or as an essay.
2) Online MCQ with open textbook: The goal is to avoid students having to print the course notes on paper for the exam, when course notes are authorized during the completion of the MCQ. Students are given access to a starting page that includes two links. The first one gives access to an online dropbox containing all the authorized documents in PDF formats and the second one gives access to an MCQ in Dokeos/Chamilo. Students can browse the notes when answering the questions.
3) Open book exam and online essay: The goals are 1) to avoid students having to print the course notes on paper for the exam, when course notes are authorized during the essay 2) to get controlled and readable answers to open questions. Students are given access to a starting page that includes two links. The first one gives access to an online dropbox containing all the authorized documents in PDF formats and the second one gives access to an exercise in Dokeos/Chamilo. The exercise is composed of a list of open questions. Students can browse the notes when answering the questions.
4) Open book exam and programming exercise: The goals are 1) to avoid students having to print the course notes on paper for the exam, when course notes are authorized during the essay 2) to get students writing a program and submit it to the course LMS. Students are given access to a starting page that includes two links. The first
WP 4.1; D4.1.4-‐6 Learning Infrastructure UniGe
Page 7 of 7
one gives access to an online dropbox containing all the authorized documents and the exercise statement in PDF formats with a java template and the second one gives access to the assignments submission in Dokeos/Chamilo. Students have access to their local programming environment. They submit their final program on Dokeos/Chamilo once it is completed. This setup has only been evaluated with the SEB pre version 2 for Mac.
Another setup has been tested by creating a learning path in Chamilo. The documents, the MCQ and all required resources are inserted inside the learning path. Students can navigate through the different components during the exam session.
These setups have not been formally evaluated. The project has started too late for the exam session of February 2013 and there were no students for the second exam session in August 2013. They have been informally tested with students and teaching assistants. They will be formally applied during the next semester in Autumn 2013.
Remarks:
Students have a great variety of laptops in terms of models, ages, OS and OS versions. Therefore it is important to perform a pre-‐installation before the exam, to check that everything is working (particularly that students can read the PDF notes) and to train students to get used to the environment.
A few “spare” laptops have to be prepared in case there is any problem with students’ ones.
Further investigations:
Progressively extend this setup to bigger classes to test the wireless connexion requirements; extend the different scenarios to students who are not in computer science to evaluate the difficulty of the local installation of SEB; develop guidelines and support documents for teachers who would like to test the setup.
III) Test with public computers Public computers are managed the same way as the computers used for the first test with LMS. SEB can be pre-‐installed on the disk image that the computers in the room are loading when students logging in. The SEB icon can be made directly available on the desktop so that students can click on the icon to launch SEB. No real experiment has been conducted, but the setups used for the first experiment with LMS have been tested with good results. The main problem with the public computers concerns the setup of the room itself. Students can easily see the screens of some of the computers around them. Therefore further investigations will be necessary to reorganize the rooms and eventually build movable partitions to enclose computer screens. A similar setup is currently being used at the Faculty of Medicine with the public computers available at the library that are used for online examinations (with another framework then SEB).
