IN DEGREE PROJECT TECHNOLOGY,FIRST CYCLE, 15 CREDITS

, STOCKHOLM SWEDEN 2021

A Comparative Analysis on Formative vs Summative Assessment in Remote Education Caused by the COVID-19 Pandemic

VILMA JALAVA

KTH ROYAL INSTITUTE OF TECHNOLOGYSCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

A Comparative Analysis onFormative vs SummativeAssessment in RemoteEducation Caused by theCOVID-19 Pandemic

VILMA JALAVA

Master in Computer ScienceDate: June 9, 2021Supervisor: Richard GlasseyExaminer: Pawel HermanSchool of Electrical Engineering and Computer ScienceSwedish title: En jämförande analys av formativ kontra- summativbedömning i distansutbilningen orsakad av COVID-19-pandemin

iv

AbstractDue to the COVID-19 Pandemic, educational institutions around the globe hadto switch to distance education. The purpose of this study was to investigatewhether there were any notable differences in grades received by first-yearcomputer science students at KTH who had online education vs those fromprevious years who did the same courses on campus. Specifically, the studylooked at one course which used formative assessment, a programming courseand one course which used summative assessment, a mathematics course. Theanalysis was done by gathering data from course results from the students as-sessed during the pandemic and comparing it to the average results receivedby students pre-pandemic using Pearson’s chi-square analysis. The resultsshowed there were no notable differences for the formative course and thatthe students had performed as expected. However, the results for the summa-tive course were a lot lower than those expected. One factor that could haveaffected this result was how the structure of the seminaries and bonus points forthe exam had been changed for remote education. Another factor was demon-strated by a previous study about Computer Science students at KTHs expe-riences with the distance education performed by PhD student Emma Riese.She found out that a majority of students felt that the pandemic had a negativeeffect on their studies. Factors such as finding the general quality of educationhaving declined, combined with a lack of motivation and self-discipline, couldalso play a part. However, limited data resulting in some assumptions havingto be drawn combined with a limited span of courses and students included inthe analysis makes it hard to draw any factual conclusions. Further research isneeded to address the weaknesses of the study.

v

SammanfattningI sambandmedCOVID-19-pandemin fick utbildningsinstitutioner världen överanpassa sig till distansutbildning. Syftet med denna studie var att undersökaom det fanns några anmärkningsvärda skillnader i betygen hos första årets Da-tateknikstudenter på KTH som hade distansundervisning jämfört med studen-ter från tidigare år som gjorde samma kurser på campus. Specifikt undersöktestudien en kurs som använde formativ bedömning, en programmeringskursoch en kurs som använde summativ bedömning, en matematikkurs. Analysengjordes genom att samla in data från kursresultaten från studenterna som exa-minerades under pandemin och jämföra demmed de genomsnittliga resultatensom studenter fick före pandemin med användning av Chitvåfördelning. Re-sultaten visade att det inte fanns några anmärkningsvärda skillnader för denformativa kursen och att eleverna hade presterat som förväntat. Resultaten förden summerande kursen var dock mycket lägre än förväntat. En faktor somkan ha påverkat detta resultat var hur seminariernas struktur och bonuspoängför examen hade ändrats för distansundervisning. En annan faktor demonstre-rades av en tidigare studie på KTHs Datatekniksstudenters upplevelse av di-stansstudier utförd av doktoranden Emma Riese. Hon kom fram till att en stormajoritet av studenterna kände att pandemin hade en negativ effekt på derasstudier. Faktorer som studenterna nämnde enligt hennes studie var till exempelen minskning i den allmänna kvaliteten på utbildningen. Detta i kombinationmed den rapporterade brist på motivation och självdisciplin som studenternakände, kan spela roll i resultaten denna studie kom fram till. Men med begrän-sad data som resulterade i att vissa antaganden behövde dras i kombinationmed ett begränsat spann av kurser och studenter som ingick i analysen gördet svårt att dra några faktiska slutsatser. Ytterligare forskning behövs för atthantera svagheterna i studien.

Contents

1 Introduction 11.1 Research Question . . . . . . . . . . . . . . . . . . . . . . . 21.2 Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Background 42.1 Formative vs Summative assessment . . . . . . . . . . . . . . 42.2 Adaptation to remote study . . . . . . . . . . . . . . . . . . . 62.3 Programming - DD1337 . . . . . . . . . . . . . . . . . . . . 62.4 Algebra and Geometry - SF1624 . . . . . . . . . . . . . . . . 72.5 Pearsons Chi-squared test . . . . . . . . . . . . . . . . . . . . 82.6 Definition: Zoom . . . . . . . . . . . . . . . . . . . . . . . . 82.7 Previous Research . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Method 143.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . 153.2 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 173.3 Scope and Limitations . . . . . . . . . . . . . . . . . . . . . 17

4 Results 194.1 Formative Assessment Analysis . . . . . . . . . . . . . . . . 194.2 Summative course analysis . . . . . . . . . . . . . . . . . . . 22

5 Discussion 255.1 Key Findings . . . . . . . . . . . . . . . . . . . . . . . . . . 255.2 Comparison with previous studies . . . . . . . . . . . . . . . 265.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.4 Error analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 275.5 Future research . . . . . . . . . . . . . . . . . . . . . . . . . 28

vi

CONTENTS vii

6 Conclusion 29

7 Bibliography 31

Chapter 1

Introduction

With the sudden breakout of the COVID-19 pandemic, schools and univer-sities worldwide needed to quickly adjust their methods of teaching to suitonline education [1][2]. According to the United Nations Educational, Scien-tific and Cultural Organization, around 72.4% of all students (1.3 billion) inschools and universities, registered in schools and universities were not able toattend in-person classes due to COVID-19 [3]. One of the universities makingthis switch was the Royal Institute of Technology, KTH, located in Stockholm,Sweden. While some of the KTH courses were required to significantly changetheir teaching forms and examination methods, others changed nearly nothing.By learning how to adapt more quickly and efficiently to new circumstancesand adjusting to online communication when possible, the negative impact ofsimilar events on education could decrease tremendously.

One of the biggest challenges which universities faced when switching to on-line education was in the method of examination [4]. Courses that had previ-ously relied on having a final exam taken on campus could no longer conductthis and had to resort to alternative methods[5]. This came with challengessince a change in examination methods can result in a sudden change in theaverage grade received in a course. For instance, preventing cheating becomesmore complicated when an exam is not being conducted in person, resultingin an inflation of grades. On the other hand, the changed examination canalso prove to be more difficult, and there are more external factors that affecta student’s performance. It may be more challenging for some students to fo-cus at home, especially if they do not live alone. Internet connection can varyamong students, giving some students a disadvantage. There might be a lackof motivation when studying due to the pandemic in general; these can all re-

1

2 CHAPTER 1. INTRODUCTION

sult in the average grade for a class dropping severely. Therefore this studyset out to discover if students that studied a course during the pandemic hadany advantages or disadvantages to those who did it on campus, more specifi-cally whether there were significant changes in the average grades received[6].

This study looks at the results from two different types of examination, for-mative and summative, from before and during the pandemic and tries to de-termine which type of examination provided the most similar results to theon-campus examination. Comparing the results’ distribution of these methodsbefore and during the pandemic could prove significant in understanding whysome examination methods work better than others in different circumstances.By altering the examination method depending on the given situation and theavailable resources, there can be a fairer grading with less worry of bias or un-fair examination to students who have to do the examination remotely. Thereis also a possibility that when education goes back to being the way it was pre-pandemic, a hybrid solution can be implemented where some of the pros withonline education are combined with the standard, on-campus examination [7].

1.1 Research QuestionThis study compares examination results pre-pandemic to the results receivedduring the pandemic in order to answer the question: How has the methodof examination, formative versus summative, for first-year computer scienceuniversity students at KTH affected the result distribution when changed todistance learning?

1.2 AimThis project aims to help teachers construct future examinations so that the ex-aminations’ level stays constant no matter if it remotely or in person, allowingschools and universities to change between the examination methods with lessworry for unjust assessments.

CHAPTER 1. INTRODUCTION 3

1.3 ApproachBy comparing course results from students who have taken a course duringpre-pandemic years with results from students who did it online, we were ableto analyse how different courses with different examination methods (forma-tive and summative) have adapted and changed with the introduction of thepandemic and the effect it has had on the final results as well as analysingstudents’ opinions about the structure of said courses with the help of a sur-vey sent out every year. For the year 2020, an extra section was added to thissurvey which asked students specific questions about how the pandemic hasaffected their education and personal wellbeing, which might help gain insightinto external factors impacting the results.

Chapter 2

Background

This section describes the adaptation to remote education made by KTH uni-versity, the types of assessment focused on, the analysed courses, their meth-ods of Examination, and the formula used for the analysis. Lastly, this sectionpresents the previous research related to this area to which our results are com-pared.

2.1 Formative vs Summative assessmentTwo of the main methods which teachers use to assess the learning of a studentis via formative and summative assessment [8][9][10]. The superior assess-ment method is often debated, to the point of Alice Man Sze Lau comparingthe debate to "a situation that increasingly calls to mind the famous slogan inGeorge Orwell’s Animal Farm – ’Four legs good, two legs bad." in her article[11].

Formative assessment is done by having students complete continuous assign-ments [12]. The focus is on providing ongoing feedback for students abouttheir progress and helping teachers identify the areas that need more work,leading to improved student success if correctly done. Some main advantagesof formative assessment include taking into account the individual process ofeach student and their amount of error put in, focuses on understanding morethan memorising, students have to be active in their learning understand theirstrengths and weaknesses.

4

CHAPTER 2. BACKGROUND 5

Summative assessment is often done by having the students complete a highstakes final exam at the end of the course to see that they have understoodeverything and apply it [13]. The faculty can also use the feedback from aSummative examination to determine if anything needs to be changed. Thestudent does not give any definite feedback about how they are doing until thevery end, where they find out whether they passed or have to do the courseagain. Some main advantages for summative assessment include that it takesplace at certain intervals when achievement has to be reported and thereforenot needing continuous involvement from a teacher; there is little room for un-just assessment since students are all judged by the same criteria, it involveseasy quality assurance procedures and is easily compared to the curriculumseeing that student have an understanding of that which the course has cov-ered.

For years there has been debate about whether one method is superior to theother. A study performed by Katie A. Hendrickson at Ohio University anal-ysed how Finland’s lack of high-stakes examinations compared to the UnitedStates’ heavy use of standardised tests correlated with Finland outperformingthe US andmost of the western world in the PISA tests, encouraging the UnitedStates educational system to take note of this[14]. On the other hand, as statedin "Assessment Matters In Higher Education", there needs to be some assess-ment done on students, especially in higher education, to ensure that they havelearned all that is needed to pass said course [15]. Summative assessment is apractical assessment method that enables multiple students to be examined atthe same time by one examiner and is the primary type of examination usedwhen examining, for instance, students’ knowledge in math due to its effectivemethod to determine whether students remember what they have been taughtand can use this knowledge to solve advanced problems.

6 CHAPTER 2. BACKGROUND

2.2 Adaptation to remote studyAt the beginning of the COVID-19 outbreak in Sweden, in March 2020, KTHshut down its campus completely and immediately switched over to remotelearning in all courses. Since there was little to no forewarning to this sud-den shutdown, KTH and many other universities had to go into a state calledemergency remote teaching[16]. This is defined as a type of temporary shiftof the education to an alternate delivery model due to crisis circumstances.However, as it became clear that education would still be held remotely duringthe autumn, educational facilities had the whole summer to plan.

At KTH, some courses had an easier time with the switch, changing almostnothing except having lectures on zoom instead of on-campus whilst othercourses had to make more drastic changes. The most significant differenceseems to have been for courses with a written closed-book exam as their ex-amination method. There have been two main ways that this change was im-plemented. Either the examination was changed to an open book exam but thedifficulty of the questions were changed or the examination was kept the samewith supervision over Zoom to ensure that it is done in an intended mannerwithout cheating.

2.3 Programming - DD1337In the chosen Formative course, Programming (course code DD1337) [17],students receive a new assignment every week to complete, receiving instantfeedback. Passing an assignment means submitting a solution to GitHub[18],a hosting provider for software development and version control often usedby programmers, in time and going to the mandatory practice session. Froma total of 9 assignments, a student needs to pass at least five assignments topass the course. After passing five, each additional assignment that the studentpasses bumps their grade up by one letter; this is demonstrated by figure 2.1below.

CHAPTER 2. BACKGROUND 7

Figure 2.1: Criteria for each grade in the course DD1337

2.4 Algebra and Geometry - SF1624The chosen Summative course, Algebra andGeometry (course code SF1624)[19],has one final exam as the only examination. The course offers optional lecturesand practice sessions but relies mainly on self-studies. The exam consists ofsix exercises, each worth six points. Part A consists of the first two assign-ments. The following two assignments constitute part B, and the last two as-signments part C.

There are six optional seminars for the student to complete, granting themone point per seminar passed, which work as bonus points for the exam. Pass-ing all seminars and receiving six bonus points grants the student full pointsfor one exercise. A total of sixteen points are needed for the lowest passinggrade, E. Figure 2.2 shows the number of points needed for each grade.

Figure 2.2: Criteria for each grade in the course SF1624

In order to obtain full points on a problem, it is required that: the whole solu-tion is well presented and easy to follow, all steps in the calculation have to bepresented clearly. Providing an answer without good enough motivation willreceive zero points.

8 CHAPTER 2. BACKGROUND

2.5 Pearsons Chi-squared testPearson’s Chi-Squared test, also known as the chi-squared goodness of fittest[20], is a statistical hypothesis test that investigates the validity of a hy-pothesis[21]. It is often used to evaluate whether sample data represent thetotal population, i.e. to test how likely the observed distribution of data fitswith the expected distribution. In this case, it will determine how well theobserved results from this pandemic year coincide with the expected valuethat is the average result from the previous years. The P-Value states whetherthe observed values are statistically significant or not. Statistically significantmeans that a result means that the null hypothesis can be rejected. The nullhypothesis, in this case, being a hypothesis of how well our expected valuescoincide with the actual results. The P-Value will give a number between 0and 1, where the closer to 1, the better the values coincide. Usually, a P-Valueless than 0.05 leads to rejection of the null hypothesis:

(χ2 ) =∑ (Observed−Expected)2

Expected

Where O stands for the observed (actual) value and E represents the expectedvalue. The expected value is subtracted from the observed value to find thedifference between them. Then to get rid of negative differences, the differenceis squared. After this, the squared difference is divided by the expected valueto normalise the larger and smaller values. Lastly, we have to sum up all thesecalculated values for each cell.

2.6 Definition: ZoomZoom Video Communications, Inc. (Zoom) is a California-based companyoffering a cloud-based video conferencing service allowing users to virtuallymeet with others, either by video or audio-only [22]. Due to the rise of theCOVID-19 pandemic, the usage of Zoom’s software increased significantly,and many Universities decided to use it as their leading platform for remoteeducation, one of those being KTH.

CHAPTER 2. BACKGROUND 9

2.7 Previous ResearchIn April 2021, a study was conducted at KTH titled “Computer Science Ma-jors’ Experiences of Their Distance Education Caused by the COVID-19 Pan-demic” [23]. This study looked at a comparative survey taken by 794 Com-puter Science students at KTH University who have studied at KTH both dur-ing on-campus education and remote learning. The students were asked tocompare their traditional on-campus education to the distance education theyrecently had to adapt to on multiple aspects in the survey, ranging from stressand motivation to quality of education and ability to collaborate with otherstudents. Although there were some differences between the grades, and eventhough most students preferred their lectures to be recorded as videos to allowpausing and fast-forwarding or going back, the results were quite conclusivethat most students preferred the on-campus setting in all aspects. Aspects suchas flexibility and reduced commuting timewere also advantageouswith remotelearning, but some students found that the added flexibility had a negative ef-fect on their discipline. Figure 2.3 shows more precisely how students feltabout different aspects. As the figure demonstrates, with all the graphs mainlybeing on the left side, most students found that on-campus education suitedthem better.

The results by Riese are supported by a study done by Son C et al. titled "Ef-fects of COVID-19 on College Students’ Mental Health in the United States:Interview Survey Study", where 195 students at a large public university inthe United States were surveyed about the effects of the pandemic on theirmental health [24]. 71% of the students found that their stress and anxietyhad increased due to the COVID-19 pandemic; the main factors affecting theparticipants’ mental health negatively are shown, as illustrated by Figure 2.4.Most of the categories impacted at least 54% of the participants, the secondbiggest problem being difficulty concentrating (89%), and 81% of studentsmentioning some concerns on their academic performance being similar tothe results in the study by Riese and Kann.

Son C et al. point out that since university students comprise a population con-sidered particularly vulnerable to mental health concerns due to factors suchas a difficult economic situation outside of the pandemic, there is an urgentneed to develop strategies and use more resources for student mental health.

10 CHAPTER 2. BACKGROUND

Figure 2.3: Likert scale data from questions regarding course activities

CHAPTER 2. BACKGROUND 11

Figure 2.4: Students’ ratings on mental health aspects in an order of negativeimpacts (mild, moderate, and severe)

A study published in 2005 by the Department of Educational Psychology atBall State University looked at whether online examination affected students’exam anxiety and performance [25]. The first aspect of the study looked atwhether students performed differently and had less anxiety with summativeexaminations online than those performed with pencil and paper. Secondly,the study looked at how students’ experiences differ based on the availabilityof online formative assessment (practice quizzes). The study found that on-line examination had no effect on students’ performances regarding summativeassessment but reported a slightly lower level of test anxiety. Regarding for-mative assessment, the authors found that students performed slightly better inan online environment. The figure 2.5 demonstrates how low-scoring studentsheld significantly higher levels of cognitive test anxiety than both the average-and high-scoring students and that low-scoring students held higher levels ofperceived test threat than the high-scoring students. This suggests externalfactors effecting test results in students.

12 CHAPTER 2. BACKGROUND

Figure 2.5: Means and StandardDeviations on Test Perception and PreparationMeasures: Assessment Format and Prior Performance

CHAPTER 2. BACKGROUND 13

At the University of Ilorin in Nigeria, a similar study was done to researchstudent attitudes towards computer-based testing. One of the aptitude testswas changed to be held on computers [26]. Two thousand two hundred ninestudents were then sent surveys, and focus group discussions were held. Theresults showed that most students had a positive attitude towards the onlineexamination. More than half of the respondents even stated preferring it topaper and pencil tests. However, some reported negative impacts of computer-based testing, such as a shortage of computers, slow network, and difficultyreading on the screen.

Chapter 3

Method

In order to answer the research question, a univariate analysis was made on theaverage passing grade for two courses received by students in years when ex-aminations were done on-campus and compared to the average grade fromwhen examinations were held online. The following section describes themethods used to answer the research question. Section 3.1 explains how thedata was collected, and section 3.2 explains how the data analysis was con-ducted.

The comparison was made using the Chi-square test described in section 2.5.The only grades looked at will be first-year computer science students whopassed the examination due to it being impossible to tell how many of the stu-dents participated in the examination but received an F or dropped the coursewithout doing the examination.

14

CHAPTER 3. METHOD 15

3.1 Data CollectionData for the summative course was extracted through Ladok [27], a studentadministration system used by KTH. Due to difficulty gathering data (furtherexplained in section 3.3) we were only able to gather the raw data for one ex-amination, the one held remotely in January 2021. Figure 3.1 depicts the rawdata. As the figure demonstrates, for students who did not pass the exam, itwas impossible to see whether they failed the exam or decided to drop thecourse before the exam. The data also show students who passed the examduring the re-examination in April, but these students will also be disregardedin the analysis.

Figure 3.1: Raw data received for SF1624

The data from the summative examinations before the pandemic was extractedby looking at the statistics posted by the Department of Mathematics at KTHUniversity https://kthgrumatte.webfactional.com/stat/SF1624/tenres/ since on Ladok it was not clear how many of the examswere taken by Computer Science students for the previous years and howmanyof the exams were from students on other programs. This data includes those

16 CHAPTER 3. METHOD

who received a failing grade, so the percentages were recalculated to showcasehow many of the passing grades were a particular grade. Since this data onlyshows how many percent of students got each grade and there was no way ofgetting information about how many students wrote the exam, an assumptionwas made that around 137 computer science students received a passing gradefor each exam. This assumption was made based on data from the results in2021 extracted from Ladok.

The data gathering for the formative course was very straightforward sinceour supervisor is the examiner for the analysed course. Therefore, he was ableto extract all the results received in the course for every year from 2020 until2016. Figure 3.2 shows how the raw data looked.

Figure 3.2: Raw data received for DD1337

CHAPTER 3. METHOD 17

3.2 Data AnalysisThe courses that were focused on were one course that uses formative assess-ment and one course that uses summative assessment—using a 5 point systemwhere A=5.0, B=4.5, C=4.0, D=3.5, E=3.0. Calculating the average grades ofthe courses pre-pandemic gives an estimate about how many students shouldreceive a specific grade during the pandemic results. A comparison was madebetween the actual results and the expected results with the Chi-squared anal-ysis and see how much the results stray to determine what type of examinationhad a smaller deviation from the pre-pandemic results. The further from 1the chi-squared value is, the average change in distribution between the pre-pandemic results to the mid-pandemic is, the more successful the type of ex-amination is since it is more similar to the original. Then a conclusion canbe drawn of what works well and what does not, and improvement points forcourses can be suggested with more considerable result distribution changes.

These results were be compared to the study described in section 2.7, wherean analysis was made on a survey of students broad experiences (not specificfor any course) with distance education, motivation, ability to collaborate withother students. The survey results were used to find reasons why results maydiffer between students from these two years that do not exclusively have to dowith the examination.

3.3 Scope and LimitationsThis study focused on the effect of this on Computer Science students sinceit could be expected that the students are familiar with using their computersfor educational purposes. Therefore, struggling to adapt to new technologyshould not significantly have impacted the results. The results might look en-tirely different for students who have not previously used computers as muchin their education. Due to difficulty gathering examination data, this study fo-cused only on first-year students, making the scope small. There might there-fore be a difference in results for students in the later years since they had alltheir education on campus and suddenly had to change in contrast to first-yearstudents who did not have any previous experience to compare to. The datacollection for the summative course proved quite challenging. Emails weresent to multiple Examiners in the Mathematics department without receivingany response. We also wrote to the Mathematics institution, who contacted

18 CHAPTER 3. METHOD

the examiners again on our behalf. This time around, a response was receivedfrom the examiners saying they were unable to help. After some further dig-ging, it came to our knowledge that this data is supposed to be public for all tosee. However, the study was quite behind on schedule due to the issues men-tioned earlier, so there was only time to gather data and analyse one formativeand one summative course.

Chapter 4

Results

In this section, the results from the analysis are presented. The examinationresults from two courses were analysed from the most recent one ending in2021 back up to three or four years back in time.

4.1 Formative Assessment AnalysisFor the formative course, anonymised grades for every student from the lastfive years were looked at. Since the course runs from August until January, theyears 2016-2019 were held on campus, while the course was held remotely in2020. In figure 4.1, the number of students receiving each passing grade wascompiled for each year. The students who received a failing grade or whodropped out of the course were omitted.

Figure 4.1: Contigency table containing results from DD1337

19

20 CHAPTER 4. RESULTS

In figure 4.2 below, the percent of students who received each grade in allthe pre-pandemic year and the remote education year are demonstrated. Thefigures show that the formative course has a very high average grade everyyear—a majority of students who receive a passing grade complete all theassignments and receive an A. As figure 4.3 demonstrates, there is not a verybig difference in the average grades received by students during education oncampus and students who studied the course during distance education.

(a) result averages for the years 2016-2019

(b) results 2020

Figure 4.2: DD1337 Grade percentages

CHAPTER 4. RESULTS 21

Figure 4.3: Comparison of grade averages for DD1337

Figure 4.4 compares the expected amount of students receiving each gradebased on the years 2016-2019 and compares it to the actual amount of studentsthat received that grade in 2020. These are used in the chi-squared formula andgive p=0,967818789 which is very close to 1 meaning that the result is veryclose to the expectation.

Figure 4.4: Table for the chi-squared analysis

22 CHAPTER 4. RESULTS

4.2 Summative course analysisFor the Summative course, the gathering of data was slightly more of a chal-lenge. The examiner changes from year to year, and they were not willing tocooperate. The only exact grades obtained were those for the exam held in Jan-uary 2021 over Zoom. For the results from the previous years, an assumptionwas made that a similar amount of students passed the exam each year, usingstatistics from https://kthgrumatte.webfactional.com, recal-culated not to include those who received a failing grade. Figure 4.5 showshow the data was represented on the website. Only data from the students ofCDATE1 was looked at since those are the first-year computer science stu-dents.

Figure 4.5: Statistics for exams represented on the KTH website

After putting together the number of students who received a specific gradebased on the assumptions that around 137 students passed the examinations,the statistics can be seen in figure 4.6.

Figure 4.6: Contigency table containing results from SF1624

CHAPTER 4. RESULTS 23

Further visualised in figure 4.7 in the summative course, there is quite a sig-nificant difference with more students receiving an E than usually. In figure4.8, it is quite clearly shown that all grades above E had a lower amount ofstudents receive it than usual.

(a) average results for the years 2017-2020

(b) results received 2021

Figure 4.7: SF1624 Grade percentages

24 CHAPTER 4. RESULTS

Figure 4.8: Comparison of grade averages for SF1624

Figure 4.9 compares the expected amount of students receiving each gradebased on the years 2017-2020 and compares it to the actual amount of studentsthat received that grade in 2021. These are used in the chi-squared formula andgive p=2,0142E-08 which is very far from 1 and indicates a large deviation inthe examination results.

Figure 4.9: Table for the chi-squared analysis

Chapter 5

Discussion

In this section, the results are analyzed and discussed with the goal of answer-ing the research question comparing Formative and Summative examinationsduring distance education.

5.1 Key FindingsThe results demonstrate the formative course having very similar results whenheld online as it had been on-campus. Therefore, adjusting to distance ed-ucation was straightforward, did not cause any significant issues. Since thehand-ins of the assignments in this course were done online even before re-mote education, it can be imagined that no significant changes had to be done.The main difference seemed to be that the practise sessions were now held re-motely via Zoom instead of classrooms.

However, for the summative course, the deviation of the result was substantial.The exam was structured similarly as before; the only difference was it helddone over Zoom instead of in a classroom, so the reason for the drop in theresults might have more to do with factors other than the examination itself.One big difference with the remote implementation of the course was collect-ing bonus points for the exam. Students could attend a seminar every weekin pre-pandemic years. During the seminar the students had to complete oneexercise and if they answered correctly one bonus point was received for theexam, receiving all six bonus points meant students could skip one exercisefrom the first three. However, due to the pandemic making KTH unable tohost these tests, they changed to more challenging assignments that the stu-dents had all week to work on. The most notable change with this change was

25

26 CHAPTER 5. DISCUSSION

that the bonus points now only counted towards the first exercise and not anyof the first three. So if a student completed the first exercise of the exam but notthe second, the bonus points could go to the second exercise in pre-pandemicyears, but in the pandemic years it could not. This might have affected theaverage grade at least slightly since usually, the first exercise is the easiest, andthe exercises get progressively more difficult.

5.2 Comparison with previous studiesThe study by Riese referenced in section 2.7 looked at surveys that KTH stu-dents had filled out about their personal experiences with distance education.This shows that in most aspects, distance education has affected the studentsnegatively. The main positive aspects was “Increased Flexibility”, where stu-dents reported that they were able to sleep more and adapt their studying basedon their daily schedules. They also reported being able to pause and rewind thevideo lectures resulting in them understanding more of the material coveredin the lectures. According to the study, the second positive aspect was “LessCommuting”, where students reported saving time and money.

For the negative aspects, the students had a lot more to say. Factors such aslack of social interactions, decreased motivation, concentration and study dis-cipline, lack of structure, and inferior quality of the education. In a coursethat relies heavily on self-studying and discipline, such as the summative one,the pandemic and distance education could have had a detrimental effect onstudent performance. This is supported by the study by Son C et al. however,since the study also found students general health to suffer from the pandemic,this was found to have a negative effect on their performance at school.

However, the study performed by Cassady, J.C et al. and the one by Adeyinka,T et al. showed opposite results, that students performed at the same levelor even slightly better when performing examinations online and seemed toprefer them. A reason for this might be that during this study, writing theexaminations online was a choice by the participating students and not some-thing which was forced upon them by circumstances. Another similar reasonmight be that the studies were performed before the pandemic, and thereforeworrying for the students own and their loved ones physical health was not afactor affecting mental health.

CHAPTER 5. DISCUSSION 27

5.3 LimitationsIn this study, there were multiple limiting factors, forcing it to scale back sig-nificantly. The biggest constraint was the difficulty in gathering the neededdata. To make a more large-scale study, multiple courses should have beenanalysed. The original plan was to analyse at least three formative and threesummative courses, preferably more. However, this was not possible dueto the difficulties gathering data and the unwillingness to cooperate with theMathematics department’s Examiners. Having only one summative course ofanalysing meant that only one formative course was analysed to give a faircomparison.

Another constraint, which was amplified by the previous, was the time con-straint. A significant amount of time was spent trying to contact examiners andwaiting for a response which never arrived. To add to this, I lost my projectpartner halfway through. This meant that the plan had to be restructured, andthe project was pushed back in its timeline. These factors made it clear thatcutbacks had to be made to the study. Making it on a smaller scale than in-tended.

5.4 Error analysisA significant source of error in the results is also all the assumptions that had tobe made in the summative courses. There was no information as to how manystudents took the exam for each year before 2021. Therefore, an assumptionhad to be made that the same number of students passed the exam each year.Therefore the results that state that the average grade was lower than usualcan be false. However, had more students than usual passed, implicating thatthe summative course did better with distance education, this study would notshow this.

Another potential factor of error that favoured the formative course is thecourse chosen for the analysis. As the results demonstrated, it is a coursein which students are likely to earn a very high grade which could affect theconsistency of the grades. It is also worth mentioning that since the formativecourse was a programming course and the study looked at computer sciencestudents, it can be expected that they have a higher interest in that subject andare therefore inclined to do better.

28 CHAPTER 5. DISCUSSION

5.5 Future researchAs the previous section expresses, this study was very generalizing, and a fur-ther study should be performed, building on this study and doing more exten-sive research. The main improvement would be to have a more considerableamount of data to look at, with more different types of courses explored andnot only Computer Science students but also students that do not usually usetheir computers as much in their on-campus education. Another study thatshould be made is with the factors that might have caused students to performworse during distance education which has more to do with the external fac-tors than the examination. As some previous studies suggested, the pandemichas had a significant negative impact on student mental health. More researchneeds to be done here to help students and prevent it from getting worse.

Chapter 6

Conclusion

This analysis regarding the Formative course showed that switching to dis-tance education had no noticeable effect on the course results. However, it isessential to keep in mind that the course analysed is quite an easy course toget a high grade according to the average grade distribution. It is also a coursewhich students might be more inclined to have an interest in, having chosento study Computer Science as their primary field. Therefore more formativecourses should be analysed for the results to be more statistically significantand see if the pattern is the same or shifts following how challenging the courseis.

The summative course results showed a considerable dip in grades, implicat-ing that the transition had been less successful than the formative. Potentialreasons for this include change of distribution of the bonus points described insection 5.1. Other reasons for this, supported by previous studies on studentmental health during the pandemic and students’ opinions on distance edu-cation, included factors such as lack of educational quality or general lack ofstructure and motivation experienced by students due to the quarantine.

However, due to the limitations of data and the assumptions that had to bemade due to this, a definite conclusion can not be drawn. Further courseswould have to be analysed to be more statistically significant. This study canbe built upon with more data to determine if any modification would need tobe done in the summative courses or if their grades should be raised due tohaving a more rigorous examination.

29

30 CHAPTER 6. CONCLUSION

In conclusion, the switch to distance education has room for improvements incourses with summative examination to make results more equal to the resultsreceived pre-pandemic, however since it is hard to know how much of thiswas actually due to the examination and how much was due to external factorsaffecting students well being negative, further studies should be done so thatthis effect can be made as small as possible.

Chapter 7

Bibliography

[1] European Universities Initiative, 2020. Survey on the impact of COVID-19on European Universities. European Commission, p.1.

[2] Ali, W., 2020. Online and Remote Learning in Higher Education Insti-tutes: A Necessity in light of COVID-19 Pandemic. Higher Education Studies,10(3), p.16.

[3] Al iimyan, K., 2020. The Education System Transformation (TEST), Howwill the education sector change In the Middle East after Corona? MiddleEast Forbes, 27.

[4] James, F., 2020. The Challenges and Advantages of Conducting ExamsDuring the COVID-19 Crisis - QS. [online] QS. Available at: <https://www.qs.com/the-challenges-and-advantages-of-conducting-exams-during-the-covid-19-crisis/> [Accessed19 May 2021].

[5] KTH., 2021. Remote examination | KTH. [online] Available at: <https://www.kth.se/en/student/kurs/tentamen/tentamen-pa-distans> [Accessed 12 May 2021].

[6] Bocij, P. Greasley, A., 1995. CanComputer-Based Testing AchieveQualityand Efficiency in Assessment?. International Journal of Educational Telecom-munications, 1(1),. Charlottesville, VA: Association for the Advancementof Computing in Education (AACE). Retrieved June 1, 2021 from https://www.learntechlib.org/primary/p/88018/

31

32 CHAPTER 7. BIBLIOGRAPHY

[7] Passerini, K. and Granger, M., 2000. A developmental model for distancelearning using the Internet. Computers Education, 34(1), pp.1-15.

[8] Dixson, D. and Worrell, F., 2016. Formative and Summative Assessmentin the Classroom. Theory Into Practice, 55(2), pp.153-159.

[9] Wynne Harlen Mary James., 1997 Assessment and Learning: differencesand relationships between formative and summative assessment. Assessmentin Education: Principles, Policy Practice, 4:3, 365-379

[10] Morgan, C., 2004. The student assessment handbook. London: Rout-ledgeFalmer, pp.18-20.

[11] Alice Man Sze Lau (2016) ‘Formative good, summative bad?’ – A re-view of the dichotomy in assessment literature. Journal of Further and HigherEducation, 40:4, 509-525, DOI: 10.1080/0309877X.2014.984600

[12] Boston, Carol (2002) The Concept of Formative Assessment. PracticalAssessment, Research, and Evaluation: Vol. 8 , Article 9. DOI: https://doi.org/10.7275/kmcq-dj31 Available at: https://scholarworks.umass.edu/pare/vol8/iss1/9

[13] Peter T. Knight (2002) Summative Assessment inHigher Education: Prac-tices in disarray. Studies inHigher Education, 27:3, 275-286, DOI: 10.1080/03075070220000662

[14] Hendrickson, K.A., 2012. Assessment in Finland: A Scholarly Reflectionon One Country’s Use of Formative, Summative, and Evaluative Practices.Mid-Western Educational Researcher, 25.

[15] Brown Jr, J., 1999. Assessment matters in higher education. McGraw-Hill Education (UK).

[16] C. Hodges, S. Moore, B. Lockee, T. Trust, and A. Bond, 2020 The dif-ference between emergency remote teaching and online learning. EducauseReview, vol. 27, pp. 1–12, 2020.

[17] Kth.se. 2021. [online] Available at: <https://www.kth.se/student/kurser/kurs/kursplan/DD1337-20191.pdf?lang=en>

CHAPTER 7. BIBLIOGRAPHY 33

[Accessed 27 May 2021].

[18] En.wikipedia.org. 2021. GitHub - Wikipedia. [online] Available at:<https://en.wikipedia.org/wiki/GitHub> [Accessed 1 June2021].

[19] Kth.se. 2021. [online] Available at: <https://www.kth.se/student/kurser/kurs/kursplan/SF1624-20192.pdf?lang=en>[Accessed 27 May 2021].

[20] Cochran, W. The 2 Test of Goodness of Fit. The Annals of MathematicalStatistics, vol. 23, no. 3, 1952, pp. 315–345. JSTOR, www.jstor.org/stable/2236678. Accessed 2 June 2021.

[21] Nihan, S., 2020. Karl Pearsons chi-square tests. Educational Researchand Reviews, 15(9), pp.575-580.

[22] En.wikipedia.org. 2021. Zoom Video Communications - Wikipedia. [on-line] Available at: <https://en.wikipedia.org/wiki/Zoom_Video_Communications> [Accessed 1 June 2021].

[23] Riese, E. and Kann, V., 2021. Computer Science Majors’ Experiences ofTheir Distance Education Caused by the COVID-19 Pandemic.

[24] Son C, Hegde S, Smith A,Wang X, Sasangohar F Effects of COVID-19 onCollege Students’ Mental Health in the United States: Interview Survey Study.J Med Internet Res 2020;22(9):e21279 URL: https://www.jmir.org/2020/9/e21279 DOI: 10.2196/21279

[25] Cassady, J.C. and Gridley, B.E. (2005). The Effects of Online Formativeand Summative Assessment on Test Anxiety and Performance. The Journal ofTechnology, Learning and Assessment, [online] 4(1). Available at: https://ejournals.bc.edu/index.php/jtla/article/view/1648[Accessed 12 Feb. 2021].

[26] Adeyinka, Tella Bashorun, M.. (2012). Attitude of Undergraduate Stu-dents Towards Computer-Based Test (CBT). International Journal of Informa-tion andCommunication Technology Education. 8. 33-45. 10.4018/jicte.2012040103.

34 CHAPTER 7. BIBLIOGRAPHY

[27] En.wikipedia.org. 2021. Ladok - Wikipedia. [online] Available at: <https://en.wikipedia.org/wiki/Ladok> [Accessed 29 May 2021].

www.kth.se

TRITA -EECS-EX-2021:476