TLC/37/08

Executive Summary of Business for Teaching and Learning Committee

Title Postgraduate External Examiner Reports overview analysis for 2015/16

Author Carmel Oxley-King, Academic Development and Quality Enhancement Manager(T) 01273 878130 (E) c.oxley@sussex.ac.uk

Type Report

Date March 2017

Strategic context Confirmation of academic standards of postgraduate awards

Summary University of Sussex reports71 out of 94 External Examiner reports expected were received. This represents a 76% response rate.

Apart from the 94 expected reports there were an additional 9 reports not received due to resignations. TLC is asked to note that all External Examiners who had resigned had completed approval of assessments prior to their resignation.

Essential reading None

Risk analysis Low risk

Resource implications None

Consultation Not applicable

Future actions To be determined by UTLC and ACR

Effective date of introduction

N/A

Recommendation University Teaching and Learning Committee is asked to discuss the postgraduate External Examiner Reports analysis of institutional issues for 2015/16. The Committee is asked to note that School-level issues were considered as part of the annual course review and enhancement event which took place in January and included PGT.

1

1. Summary and Analysis of Postgraduate External Examiners Reports 2015/16

Of the 94 reports expected by the Academic Development and Quality Enhancement Office 71 completed reports were received. These were distributed widely throughout the institution in line with agreed procedures. This gives us a response rate of 76% (this excludes reports from 9 external examiners who had resigned). This compared to an 88% response rate in 2014/15. This excludes BSMS reports which JARB receives on behalf of the University of Sussex and the University of Brighton. Actions plans in response to School level issues will be considered by the School TLCs and responses sent directly by the Chair of the Progression and Award Board (PAB) to External Examiners. The reports and the action plans are reviewed during the Annual Course Review event. A proposed response to institutional issues that have been raised in External Examiner Reports have been set out below in italics.

Sections 2-4 provide a summary of data extracted from the External Examiner reports 2015/16 and areas of good practice amongst Schools.Institutional Issues for 2015/16 (refer to relevant section for more detail)

Individual response to External Examiners regarding institutional issues raised in reports – PVC (T&L) to respond to institutional issues raised by individual external examiners. Going forwards to consider use of ESEF to enable submission of external examiner resport and enable institutional issues to be extracted for response by PVC. (Currently external examiners are sent the PGT summary at the start of the next academic year to inform them of the University’s response to issues raised. For School level issues, the Deputy Chair of the PAB sends the External Examiner the action plan approved by STLC.

Concerns regarding application of the discretionary regulations - University to provide some guidance on application of the discretionary regulations. The prescribed method of moderation is disliked by some External Examiners in particular Schools – During 2015/16 the University addressed this by

publishing (i) FAQs to support External Examiners in understanding the prescribed process for moderation and (ii) by scheduling induction for new Externals in January in addition to the briefings that are scheduled annually in advance of the PABs. Attendance at the latter has been low but for those who attended understanding of the moderation process has improved. Targeted briefings for External Examiners and academics will be arranged for Schools with ongoing concerns.

Feedback varies greatly across and within Schools (occasionally too generic, too brief or unclear, not aligned to mark and marking criteria – Consider providing guidance to Schools. Also University to review ESEF to see if technology can secure mark first and then allow individual feedback to student based on prior attainment and feedback. For 2016/17 resources are now available via Study Direct to support Academic Advisors in this.

2

Update on Institutional issues from 2013/14

Improve communication and workload planning for External Examiners (better communication regarding moderation timescales, consistent and clear evidence of a distinct marking and moderation, provision of statistical data as part of the moderation process, opportunities to meet students) - future roll out of E-Submission will enable external moderation at an earlier stage and will ensure evidence of separate marking and moderation processes are provided, along with statistical data. FAQs have been provided to External Examiners to help address some of the issues being raised.

To consider how best to use External Examiners – University held an induction session in January 2015 and February 2016 for new and continuing External Examiners to discuss their role and a provide an opportunity for them to meet with students and Faculty.

Organisation of the PAB (consideration of borderline strategy and use of condoned credit) – Some Schools consulted with board members in advance of the PAB to agree strategies for borderline and condoned credit application to reduce discussion regarding the regulations at the PAB. One School ran a pilot and held their PAB in two sittings with a core membership attending both. For 2016/17 a module failure may only be condoned where a mark has been achieved.

To maximise use of electronic means for detecting academic misconduct – University has rolled out the ESEF project at all levels. In 2016/17 Schools had the option to set level 7 assessments up as e-submission. This means more assessments at level 7 will go through Turnitin.2

3

2.1 Summary of Social Science Schools External Examiner

Area explored BMEc ESW1 Global Studies

LPS2 General comments if area not fully agreed by EE

no of reports received compared to the no expected

11/17 11/14 6/12 8/10

standards: appropriate 11/11 11/11 6/6 8/8marking criteria: appropriate/consistent

7/11 11/11 6/6 8/8 BMEC: …with International Financial Accounting II, there were no marks per sub-question and the marking scheme did not have detailed marks or guidance. (WARD)BMEC: Greater care might be taken by some markers to ensure that the use of language in feedback to students matches the marking criteria. For example, using words which appear under the definition for a mark of 70-80% should typically mean a mark in that range. (CONNOR)BMEC: … some concerns with the marking of the dissertations. I think there are some misalignments among the assessment criteria, the comments provided and the final marks. In other words, I see problems with the implementation of the marking criteria. (RABELLOTI)BMEC: In the past I suggested that, when relevant, marking criteria that indicate what is expected from a pass, good and excellent answers would have been useful. I would have liked to see such criteria for some, i.e. substantial (essay type) questions. This is still not available for some courses. (KLECUN)

performance: comparable

8/11 11/11 5/6 8/8 BMEC: There were a large cohort of students and many fails. Even so, I actually think that the marks are inflated and that more should perhaps have failed. Given the large influx of students I can only assume that this is a result of entrance requirements being too low (WARD)BMEC: …the standard of dissertations is on average less satisfying. As noted in my previous reports, the weaknesses of the dissertations can be due to a time factor, which has not yet been addressed (RABELLOTTI)BMEC: A few reports suffered from poor/very poor English (KLECUN)GLOBAL-IDS: I did consider that student performance dropped a little for the dissertation, which was disappointing (WRIGHT)

learning outcomes: 9/11 11/11 4/6 8/8 BMEC: I have only one point to raise. I believe that in the module ‘Accounting for Decision 1 Excluding PGCE2 Including CPE

4

positive comments Makers’, either the module outline be changed or the paper should have a computational financial accounting question to meet the current objectives of the module. (WARD)BMEC: ...concerns about the number of failures (34 out of 79 clear fails and additional 7 marginal fails). I noted that this might have been a particular poor cohort. (KLECUN)GLOBAL-IDS: surprised to note that there was no form of formative assessment/feedback for the other term papers/essays… potentially disadvantage students who are coming to University after a gap of several years (RAO)GLOBAL-IDS: The Poverty Policy and Programming module… learning outcome as being …students learn to “Collaborate effectively with international development professionals from diverse backgrounds …no group work in this module, so I don’t know how this learning outcome is supposed to be assessed. (POULTON)

marks awarded: appropriate

9/11 10/11 4/6 8/8 BMEC: There was one case where both markers awarded a distinction but concentrated on negative comments and used words like ‘good’, rather than the superlatives on the marking criteria. I had no quibble with the numerical mark awarded, but did flag this divergence between quantitative and qualitative feedback to the course convenors. (BALMER)BMEC: …dissertations. In that case, the marking criteria are excessively complicated and they are not clearly reflected in the evaluation and in the final mark (RABELLOTTI)ESW: I do think that some scripts on the border of pass and fail are given the “benefit of the doubt “ and passed. I have queried this a number of times and of course it is down to professional judgment but would encourage the team to ensure that papers which are borderline are failed if they don’t meet the standard (GRANT)GLOBAL-IDS: Of the two MA programmes I am external examiner for, the MA in Environment and Development has tended to use the mark range better than the MA in Social Development, which tends to cluster marks in the range of good (60-69), despite quite a lot of variation in the quality of the work (RAO)GLOBAL-IDS: MA programme in Poverty and Development. This is related to FHEQ criteria for level 7… the case of the mid-term assessment, I found that the marking was over-generous. (POULTON)

processes: sound, appropriate and fairly conducted

7/11 11/11 5/6 7/8 BMEC: some concerns that in some of the modules the marks awarded were a little high and some students who were on the borderline were awarded pass marks (WARD)BMEC: No evidence of plagiarism software being used (WARD)BMEC: I am concerned that exam boards are still in a position to vote for a particular approach at the beginning of a PAB and that this can lead to different outcomes (CONNOR)

5

BMEC: …are still different interpretations of the regulations during the BMEc PGT Progression and Award Boards. (DEKKERS)GLOBAL-IDS: surprised to learn that the written work is not put through source-matching software upon submission, such as Turnitin. I consider this an essential part of a robust and fair assessment (WRIGHT)LPS: There was some discussion at the beginning of the PAB about how we would deal with students in the discretionary zone. I would strongly recommend that a form of words is agreed in advance at a pre-meeting. (HAUGHTON)

teaching, learning and assessment: positive comments

11/11 11/11 6/6 8/8

T&L methods and opportunities: positive comments

11/11 11/11 6/6 8/8

materials: sufficient 10/11 9/11 4/6 7/8 BMEC: There have been instances this year where work was sent at short notice for submission of my views in under a week. (CONNOR)ESW: It would be helpful in the future to be able to access teaching materials in Blackboard site (GREER)ESW: …need to draw attention to the late receipt of sample work. (FERTIG)GLOBAL-IDS: One thing which is frustrating is that I don’t have any idea on whether specific questions are set for assessment or students choose their own topic. There is no information on assessment in the module outline. (RAO)GLOBAL-IDS: There was a period in the middle of the year when the communication of requirements and the sending of appropriate samples of scripts was extremely poor and unsatisfactory. Whole batches of unsampled scripts arrived with no explanation or supporting module outlines, marksheets and information (LEONARD)LPS: The time allowed for the June papers to be externalled was impossibly tight (HAMILTON)

6

2.2 General comments made by External Examiners which relates to individual SchoolsBMEC: The quality of teaching and learning methods and learning opportunities compares well with the general standard at British Business Schools and is better than many. BMEC: Care needs to be taken to ensure all markers are feeding back substantively in the dissertation module, but this is largely the case. Some second markers might usefully be encouraged to provide more feedback BMEC: Greater care might be taken by some markers to ensure that the use of language in feedback to students matches the marking criteria. For example, using words which appear under the definition for a mark of 70-80% should typically mean a mark in that range.BMEC: …some marks ending with ‘9’. I strongly suggest avoiding such marks (for a major piece of work)BMEC: There have been instances this year where work was sent at short notice for submission of my views in under a week.BMEC: I believe that in the module ‘Accounting for Decision Makers’, either the module outline be changed or the paper should have a computational financial accounting question to meet the current objectives of the module. BMEC: The standard of dissertations is on average less satisfyingBMEC: In Module 941N1 Accounting for Decision Makers performance was disappointing. There were a large cohort of students and many fails. Even so, I actually think that the marks are inflated and that more should perhaps have failed. Given the large influx of students I can only assume that this is a result of entrance requirements being too low. The exam paper was appropriate. BMEC: Should the name of the course be changed (‘Consumer Buyer Behaviour’?) to reflect the fact that the topic of Organisational Buyer Behaviours is not touched upon at all?BMEC: 30% of the final examination is based upon Multiple Choice questions. I think that the University of Sussex should take a view as to whether or not this format is appropriate at MSc level. UNIBMEC: A different organization of the term to offer a longer period for the elaboration of the dissertation. uniBMEC: My only issue was with International Financial Accounting II, there were no marks per sub-question and the marking scheme did not have detailed marks or guidance. BMEC: The reporting and referencing skills need continuous attentionESW: elsewhere in the University video or audio feedback is being trialled as possibilities as part of new e-marking practice, which the team might want to review and consider.ESW: Once again, we are reminded of the need to have in place quality of mentoring which really supports trainee learning. But we are also reminded that mentors themselves are facing a mulitiplicity of demands on their time. ESW: if possible, some fine tuning of the “met” and “not met” criteria in relation to learning outcomes. It strikes me that the value of this is limited unless some kind of qualitative assessment of the degree to which the criteria are either “met” or “not met”.ESW: I have not received any feedback from my EE Report last year. ESW: Building in FT MAE peer support time throughout the academic year to facilitate culturally specific, international-student specific issues are accommodated ESW: Ongoing review of curriculum to ensure "new" developments (eg Progress 8) are fully embedded

7

ESW: in relation to language support and academic literacy. In is not uncommon to find students struggling to develop an academic voice and I would support the call for additional support identified in relation to [these] particular studentsESW: It might be worth considering whether the PGCE team could agree consistent expectations around written English: for example, are contractions acceptable or not? Do all markers pick up on apostrophe errors and comma spliced sentences? And so on. It seemed that the Primary marking team may be more consistent in this respect than the Secondary team.GLOBAL: I was a bit surprised to note that there was no form of formative assessment/feedback for the other term papers/essays [other than the dissertation] GLOBAL: Many Universities are now moving towards multiple forms of assessment, so students whose first language is not English are not disadvantagedGLOBAL: Using the range of marks available, rather than clustering in the merit rangeGLOBAL: Secondly, two different mark sheets seem to be in use (although some markers don’t use them at all). These indicate overlapping but not identical items of feedback – one refers to structure of the argument, originality, literature and sources, presentation, writing skills (marked out of 5) while the other refers to argument, structure, use of literature, presentation (marked out of 4). Once again, a clarification of the procedure with the aim of achieving consistency of practice would be desirable. LPS: the absence of annotations on the hard copies of the exams and essays does make it a little more difficult for the student to pinpoint where, specifically, certain aspects might have been improved LPS: Some marks given for certain papers seemed to be very high (well above 80). Such marks would not have been given for similar works in institutions where I have worked.LPS: A model answer could be included as generic feedback, following each student’s personalised feedback LPS: Consideration of whether an assessed essay plan, or alternatively non-assessed formative feedback on an essay plan, might be provided across all modules to ensure consistency in terms of student expectations and overall learning experience.LPS: The module choice available to students is relatively limited in the second term. LPS: The feedback on the written work is ample, but could be more focused on instructing students on how to raise the work to a higher marking band.LPS: One essay on a module was referred to as ‘solid’ and ‘good’, but yet was awarded a distinction mark. For something deserving of a distinction mark words like ‘solid’ and ‘good’ seem rather incongruous.LPS: There was annotated discrepancy in the quality of the work for assessment submitted throughout the year, and the rather lower quality of the dissertation, which was regrettable.

8

2.3 Areas of good practice arising from External Examiner reports directed to individual Social Science Schools

Area Summary of contentAssessment and marks

Global: Moderation processes are through and scrutinise an appropriate sample of scripts.LPS: The feedback that was given by the markers was elaborate, clear, coherent and pin-pointed the elements to be strengthened by the students.ESW: I agreed with the grades awarded on all assignments and it was evident that there had been a careful moderation process.BMEC: There was a good balance of general comments which would be helpful for improving on future assessments, and specific comments that demonstrated arkers had engaged with the substance of the assessments.

Standards Global: This is a high level and well executed MA in a specialist field. It is widely recognised as suchLPS: The standards set by the University are in accordance with national benchmarks for the qualification being studied.ESW: The standards set by the School of Education and Social Work for the suite of programmes I have been external examining are all firmly rooted in the QAA Masters level standards and are equivalent to one another.BMEC: The standards set by the University are entirely appropriate for the qualification being studied

Processes Global: Assessment and examination processes and determination of awards were sound, appropriate and conducted fairly in line with academic regulations.LPS: the marks were spread across the board, which shows that there was a very good quality of teaching and learning methods employed.ESW: there has been a high degree of professionalism in this respectBMEC: As in previous years, I am fully satisfied with the process. The process is appropriate and transparent

Teaching, learning and assessment

Global: Excellent feedback is provided systematically, supporting the marks awarded and offering constructive and detailed commentsLPS: examination and essay questions were interesting and thought provokingESW: The curriculum is designed to provide students with a rich learning experienceBMEC: I received clearly moderated work from across the spectrum of grades, all with good, constructive feedback and appropriate marking

T&L methods and opportunities

Global: There is a good range of high quality teaching and learning methods and opportunitiesLPS: The programme offers a good blend of compulsory and optional elements taught in a variety of different ways according to the needs of the module.ESW: It is evident...that the course has equipped [students] well for their professional careers, and helped them develop professional autonomyBMEC: The quality of teaching and learning methods and learning opportunities compares well with the general standard at British Business Schools and is better than many.

Other BMEC: the “hands on” approach to the curriculum is exactly what students seeking an M.A. in economics should be exposed to ESW: The quality of feedback provided to students by tutors after classroom observation: this is given sensitively and constructively but always highlights the most salient points. This is a real strength.ESW: It is evident...that the course has equipped [students] well for their professional careers, and helped them develop professional

9

autonomy.Global: I was enthused by the use of research design / proposal exercises (assessed) as part of the dissertation. This is excellent practice. My only concern was that these were often graded higher than the final dissertation, which perhaps inevitably could not live up to the (usually) empirical expectations of the proposal

2.4 Institutional Issues Related to Social Science Schools

10

BMEC: In Management accounting and control N8951 the mean mark seems very low, lower than last year...I wonder if students are fit on entry level and equipped with sufficient ability to write coherently in English or are competent enough to do the course? BMEC: There are still different interpretations of the (discretionary) regulations during the BMEc PGT Progression and Award Boards. BMEC: condoning can still result in students receiving merit and distinction; this does not make sense BMEC: I find the moderation process in Sussex to be very hands off. Whilst there was evidence of moderation the evidence was limited. In some of the modules the moderator just put their mark on the overall feedback sheet for the whole group and said yes.ESW: Team review of effective ways to maintain the quality of marking feedback whilst maximising the new opportunities afforded by e-marking.ESW: I would like to focus upon the nature of feedback provided to students at the dissertation phase. The Assignment Feedback Proforma has a space for a Second Examiner, though the sample of dissertations I received had feedback from only one internal ExaminerESW: I have never had a formal written response to any of my external examiners reports. At my university this is a compulsory part of the process and the QA of programmes. I have raised this in all my reports but have never had a response ESW: I do need to draw attention to the late receipt of sample work…[this] meant that timely analysis and comments to the Course Team proved to be impractical. Global: I had requested last year that when there is substantial variation between the first and second marker, I would like to see both sets of comments, as the marks have often been adjusted to the higher one, rather than meeting midway. This has not happened. Global: I have not really had any feedback on comments made in earlier reportsGlobal: In some rare and exceptional cases there is some woefully brief feedback to students with very cursory and even brusque sentences -or even one word comments -for each of the various criteriaGlobal: While overall the process has been managed well, there have been a few glitches once again in terms of timing during the last academic year, related partly to mis/lack of appropriate communication LPS: It was not always clear whether the work had been double-marked; that could be indicated more explicitly LPS: The Exam Board meeting was well run. It would have been useful if conventions regarding the use of discretion in determining qualification had been discussed before the meeting so that a firm proposal regarding discretionary decision making could have been put to the board at the start of the meeting.

3.1 Summary of Science Schools External Examiner Reports

Area explored EngInfo Life Sciences

MPS Psychology General comments if area not fully agreed by EE

no of reports received compared to the no expected

6/6 3/3 2/6 5/7

standards: 6/6 3/3 2/2 5/5

11

appropriate

marking criteria: appropriate/consistent

6/6 3/3 2/2 5/5

performance: comparable

6/6 3/3 2/2 5/5

learning outcomes: positive comments

5/6 3/3 1/2 5/5 ENGINF: Of Greater concern to me are the low marks and high failure rates on some of the thermo-fluids modules. I believe the School should review this module, and others which have had consistently poor student performance over the last few years, to check that the examinations and coursework on these modules remain fit for purpose. (THOMPSON)MPS: However, coursework marks, which tend not to go through the same level of quality assurance, carry a rather high weight. (RAJANTIE)

marks awarded: appropriate

5/6 2/3 2/2 5/5 ENGINF: I noticed that the difference in marks between 1st and 2nd examiners of the dissertations that I sampled was, compared to my past experience, unusually small. It would be an improvement to have more detailed information/transparency on the independent marking process.LIFESCI: I think the correct marks were assigned but markers should be encouraged to ensure that their marks are consistent with the summary remarks. (BERRY)

processes: sound, appropriate and fairly conducted

6/6 2/3 2/2 5/5 LIFESCI: A difference in attainment between optional modules was evident from statistics provided that was not explained by differential intake, with poorer attainment in the genome stability option compared to others. The observation was discussed with the team with recommendation that this ought to be examined and action considered as the team may deem suitable (including adjusting expectations or additional support) (MITTNACHT)

teaching, learning and assessment: positive comments

6/6 3/3 2/2 5/5

T&L methods and 6/6 3/3 2/2 5/5

12

opportunities: positive commentsmaterials: sufficient 6/6 2/3 1/2 5/5 LIFESCI: I would also suggest that it would be useful to have individual module

spreadsheets detailing assignments with marks and breaking down exam marks by individual questions…….When work has been submitted late and subjected to a penalty, it would be useful to know whether the marks that we are given are before or after the penalty has been applied. (BERRY)MPS: I was given access to exam scripts of borderline students, but no question papers. It would be useful to see the questions as well, and also other students' scripts for comparison. (TUNARU)

13

3.2 General comments made by External Examiners which relates to individual Schools

EngInf: In one piece of coursework, on Strategic Management (519H3), it was not clear to me how the individual contributions to the coursework were assessed, even though one of the pieces…had 4 students contributing to it. EngInf: the [CDM] written reports are sometimes a little weaker than the project dissertations for the MSc in HCI…It might be useful to give the CDM students more training on how to write up research and technical projects.EngInf: While a CD submission allows for larger files to be submitted, there is a serious drawback in that accessing and viewing the work is made more difficult. I can only assume that some students might also experience difficulties writing files to a CD (if their modern laptop does not have a CD drive). EngInf: I did not receive responses to my comments on examination papers until I came to the PAB. It would have been better to receive the responses from examiners to my comments sooner in order to 'close the loop' on the exam paper review process. EngInf: It is difficult for an external examiner to get a good view of resources. EngInf: Arriving for one day to look at the material for two MSc programmes can be overwhelming. [it would be useful to receive] a simple and concise summary of the programme, the modules that students took (compulsory and optional)ENGINF: in a small number of examination questions the breakdown of marks is inadequate…it is very difficult for students to estimate the effort associated with the various components of the question.ENGINF: This year means marks between coursework and examination marks were closer than last year which again is a good outcome.LIFESCI: There were some minor issues with access to online submitted materials. LIFESCI: there were occasions when the summary comments used terminology that, in the marking criteria, were associated with a different mark range...I think the correct marks were assigned but markers should be encouraged to ensure that their marks are consistent with the summary remarks.PSYCH: During my meeting with students I was made aware that some students had experienced difficulties accessing suitable training cases whilst on placementPSYCH: There were no recordings of student presentations and although scripts were supplied, a more robust moderation of this element would be possible if these were present. PSYCH: I was able to talk with one of the students on the course. This provided a useful additional perspective on the course...It would be useful to meet with a group of students in advance of the next Progress and Award Board.PSYCH: With diagnosis occupying a central place in work to address personality disorder in particular, I felt students might be encouraged to present a stronger defence of their arguments in this respect. PSYCH: whilst some markers are quick to pick up when students have not clarified the use of pseudonyms this does not always happen and the centrality of the client’s confidentiality might be stressed. PSYCH: Translation tables used to convert raw scores for observed structured clinical examinations into percentages, as described in previous reports and discussed with the programme team.PSYCH: Some of the methods that students train in are advanced and reflect the state of the art in the fieldPSYCH: Yes, I could access any material as required and [was] provided in the first place with sufficient material.

14

PSYCH: the OSCE examination would deliver a clearer picture of students’ skills with additional time allowed and this is something that might be considered by Course staff.

3.3 Areas of good practice arising from External Examiner reports directed to individual Science Schools

Area Summary of contentAssessment and marks ENGINF: Mark sheet also gives clear description to justify marks given. Very good practice.

ENGINF: The level of formative and individualised feedback given was at a very high standard.MPS: Marking is fair in general.PSYCH: The feedback to students is detailed, encouraging, developmentally appropriate, and very clear as to how to improve the work.

Processes ENGINF: All processes were applied to a high standardENGINF: The courses adhere to the UK-SPEC Engineering benchmarks...and are uniformly of very high standard. LIFESCI: I am confident that the process was sound, appropriate and fairLIFESCI: I was given access to a considerable amount of material and excellent support from University staff. MPS: Borderline cases were discussed thoroughly at the PAB, and the decisions were fair and consistent.

15

PSYCH: This is an exceptionally well run courseTeaching, learning and assessment

ENGINF: The different modes available for individual modules provide a range of ways for students to present their skills and understanding of the materialsLIFESCI: This is a very good course with excellent and coherent design and expert teaching.LIFESCI: Standards of all assignments are high, with research relevant experience that should equip students well with the tools they need for a science career.LIFESCI: Assessments cover and are well suited to assess learning outcomes across these different components.MPS: Second marking and presentation of dissertation is laudable.PSYCH: I particularly commend the programme’s approach to self-reflective learningPSYCH: The course material and the learning outcomes clearly map onto each other with the assessments doing an excellent job of measuring the extent to which the students have achieved those outcomes.PSYCH: The standards exemplified in the course materials and the work supplied for moderation suggest a strong focus on quality

T&L methods and opportunities

ENGINF: the breadth of learning and teaching methods…is very good and students have plenty of opportunity to demonstrate their learning.LIFESCI: The course content is current and informed by experts in their fields.LIFESCI: Within and between modules there is a good range of practical and theoretical material delivered via lectures, workshops and practicals.PSYCH: the teaching and learning methods are very innovative

Other ENGINF: All materials needed were available...it was relatively straightforward to find all the information required. The administrative staff were available at short notice and very helpful in providing any information required.

3.4 Institutional Issues Related to Science Schools

EngInf: It would have been useful to see the list and distribution of marks for each student (all modules) and the entire programme (all modules, all students), in advance to make it easier to identify cases that warranted special attention, if any. EngInf: I did not receive a formal response to my external examiner's report last year and would like to receive one in future years if at all possible.ENGINF: I noticed that the difference in marks between 1st and 2nd examiners of the dissertations that I sampled was, compared to my past experience, unusually small. It would be an improvement to have more detailed information/transparency on the independent marking process.LIFESCI: I would also suggest that it would be useful to have individual module spreadsheets detailing assignments with marks and breaking down exam marks by individual questions…….When work has been submitted late and subjected to a penalty, it would be useful to know whether the marks that we are given are before or after the penalty has been applied. (BERRY)

16

LIFESCI: the existence of a pass/fail module (appropriate in the programme) did raise some possible anomalies in calculating outcomes under university regulations. I am keen to see this assessment retained and the module team should liaise with the university to see whether regulations might be modified at university level to address these issues.

4.1 Summary of Arts & Humanities Schools External Examiner Reports

Area explored English HAHP MFM SCLS General comments if area not fully agreed by EEno of reports received compared to the no expected

5/5 4/5 7/8 1/1

standards: appropriate

5/5 4/4 7/7 1/1

marking criteria: appropriate/consistent

5/5 3/4 6/7 1/1 HAHP: At the lower end…I wonder if more could be done to articulate quite how work 'at postgraduate level' is different to an equivalent range of marks at undergraduate level (JOHNS)MFM: More clarity is needed for marker (and external

17

examiners) about the appropriate weighting of practice and theoretical elements in practice projects (JARRETT)

performance: comparable

5/5 4/4 7/7 1/1

learning outcomes: positive comments

5/5 4/4 7/7 1/1

marks awarded: appropriate

4/5 4/4 7/7 0/1 ENGLISH: I’d recommend that the comments were less personalised and more student-friendly. Expressions like ‘you don’t seem to understand’ could be replaced with ‘there is a lack of understanding’ (GREGORIOU)SCLS: My only concern this year was that in the case of assignments which are deemed to have failed more direct feedback could be given in relation to the revisions that would need to be made in order for the paper to pass (BAKER)

processes: sound, appropriate and fairly conducted

5/5 4/4 7/7 1/1

teaching, learning and assessment: positive comments

5/5 4/4 7/7 1/1

T&L methods and opportunities: positive comments

4/5 4/4 7/7 1/1 ENGLISH: The critical review format was not well understood by students and perhaps developing that so that they are clear and staff members are clearer as to why it exists, rather than an essay, would be an enhancement. (LESLIE)

materials: sufficient 4/5 4/4 5/7 1/1 ENGLISH: What I have not received is an overview of the course as a whole – by which I mean the profiles of all graduating students at the point of the exam board. (LESLIE)MFM: ...projects were not delivered in time for review… (JARRETT)MFM: The turnaround time between receiving the dissertations and the boards taking place was very short this year – only a few days – but I understand that this was due to unforeseen special circumstances.(WILLIAMS)

18

4.2 General comments made by External Examiners which relates to individual Schools

English: I was surprised that I was not required to look at any dissertations.English: As I said in previous years, it'd be good if the students received their scripts back annotated with marker comments... I'd rather see a selection of essays annotated (and photocopied)...than the full set of essays submitted.ENGLISH: The need to closely, accurately and properly reference sources needs stressing.HAHP: Further (or firmer) advice about the dissertation topics may be advisable on occasionHAHP: The standards of presentation in many dissertations and essays remains quite low.HAHP: There is a certain 'sameness' about the underlying ideological approach of students whose work I have seenMFM: It would be prudent over the next few years to assess the outcomes and experiences of ab initio practice students against those with practice elements in their undergraduate qualifications to assess the viability and/or achievements of this pathway.MFM: I am hoping to hear from the team in due course, as recommended in my reports, as to a way forward with developing the level of academic writing in the commentaries that accompany the creative work.

19

MFM: It would be recommended to develop more robust pre-emptive moderation processes for modules in which there are multiple markers, such as final projects…establishing clear marking parameters prior to assignment evaluation can be more effective in achieving fair outcomes.SCLS: some inconsistency in the feedback given on writing style, assignment organisation and citation and referencing protocol.

4.3 Areas of good practice arising from External Examiner reports directed to individual Arts & Humanities Schools

Area Summary of contentStandards ENGLISH: Good standards are maintained by strong teaching, careful marking and detailed feedback

HAHP: The standards are very good. The programme is demanding...the expectations/requirements are clear. MFM: The complexity of this degree in combining high level theoretical engagement and practice offers students the requisite depth and breadth and opportunities to develop original, creative interrogations of concepts at the leading edge of the field.SCLS: The standards set were appropriate and as expected from this levelSCLS: The interface between theory and practice continues to be a strength of this programme.

Marks awarded ENGLISH: Markers were really attentive to the work, responding at generous length.HAHP: [I] was overwhelmingly impressed by the care taken by [examiners] to provide thorough feedback.MFM: The feedback given in this course to students is excellent and amongst the most comprehensive and useful I have seen in any universitySCLS: the marking and moderating appeared fair and to have been carried out following the established procedures.

Assessment and marksPerformance

Processes ENGLISH: assessment and the determination of awards has been sound, appropriate and conducted in good faith in an environment of fairness and professionalismHAHP: The application of procedures this year was excellent, and I have detected increasing rigour during [my tenure]

20

MFM: The University’s procedures for compensation are carefully applied at the exam board, with care taken to ensure that decisions in each case are referenced against similar decisions for other students.SCLS: I am satisfied that they have been sound, appropriate and fairly conducted

Teaching, learning and assessment

ENGLISH: The modules in general are challenging, requiring substantial knowledge & occasioning especially lively responsesENGLISH: An efficient and fair-minded PAB.HAHP: Curatorial placements bring a lot to the student experience and appear to be managed well.SCLS: The programme and module outlines and the work produced by the students would suggest high standards of teaching and learning

T&L methods and opportunities

ENGLISH: The degree provides ample scope for students to pursue learning opportunities through advanced independent study

4.4 Institutional Issues Related to Arts & Humanities Schools

English: A final grade can be achieved which differs significantly from that acquired in the dissertation. [It] should figure in the final award not just through its double weighting but also through its bearing on the final mark. English: I think that a selection of scripts (one for each band) could be sent from each module next time.HAHP: I remain unconvinced that it is wise to have a single set of feedback. As an external I would like to see comments from both internal markers disaggregated.MFM: Standard spaces for the kinds of practice undertaken in this course will need to move beyond facilities for desktop computing to include studio contexts for more haptic or embodied creative practice. Securing such facilities is important for the continued relevance of Digital Media degree courses and...[to] develop advanced digital practice in emerging fields.MFM: the final batch of work arrived a bit lateMFM: The provided samples were limited this year

21