Event notifications

John Durrant

May 2014

Event notifications - volume

43

38

54

39

33 35 34

30

37

23

34

30

43

0

10

20

30

40

50

60

70

Mar-13 Apr-13 May-13 Jun-13 Jul-13 Aug-13 Sep-13 Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14

Nu

mb

er

of

Ev

en

t N

oti

ficati

on

s r

eceiv

ed

Event notifications - type

What do we do with event notifications?

Process

– acknowledge

– record

– assign

– ask for more information?

Evaluate

– are we sufficiently assured AO is mitigating incident/adverse

effect?

– are we sufficiently assured AO is addressing root cause?

– one off or part of a trend?

– management of event

Escalate

– incident management, investigation or regulatory action?

Analysis

Analysis to

– inform us of areas of concern

– build our understanding of the types of event notifications reported

– inform understanding of systemic risk and areas for further

monitoring and thematic review

Entity risk assessment

– we also analyse event notifications in the AO risk profiles, taking

into consideration the size of an awarding organisation and where

we have had no event notifications

Reporting

– monthly trend reports to Chief Regulator and Ofqual Board

What is an Adverse Effect?

Use these questions to judge if your incident could adversely effect:

– students – does this incident mean your current or potential

learners will be prejudiced?

– development, delivery and awarding – will this incident mean you

are not meeting the Conditions?

– standards – is the standard of your qualification compromised or

likely to be compromised?

– public confidence – will this situation affect public confidence in

your qualification?

Scenarios

One scenario per table – there are five different ones

Discuss the scenarios and think about:

– what information do you need to collect and how?

– what actions you will take and when?

– what are the potential adverse effects in your scenario?

– who will you inform of your actions and when?

You will receive additional information during the task. This might

mean your decisions and actions change

Feedback – Scenario one

Maladministration/malpractice

– security breaches and invigilation failings

Incident could adversely affect

– AOs ability to deliver and award qualifications in accordance with

the conditions

– performance standards

– public confidence

More information needed?

– do both incidents relate to door supervisor tests?

– are failings isolated or systemic across centre?

Action

– notify regulator and other AOs

– replace question paper, observe invigilation

– Investigate and apply sanction

Feedback – Scenario two

Breach of data protection law

– sensitive personal data shared and maybe lost

Incident could adversely affect

– AOs ability to deliver and award qualifications in accordance with

the conditions

– public confidence

More information needed?

– did the EV delete the information from the missing memory stick

before sending it?

– how was the missing memory stick sent and can it be tracked?

Action

– notify regulator

– seek legal advice about data protection responsibilities

– change centre and standardisation procedures

Feedback – Scenario three

Question paper error

– part question error causing difficulties for students

Incident could

– prejudice students

– adversely affect AOs ability to deliver and award qualifications in

accordance with the conditions; standards and public confidence

More information needed?

– how many students failed exam by getting 1a) wrong and not

completing the paper?

– is the qualification graded?

Action

– notify regulators and centres

– apply special consideration and review outcomes at awarding

– root cause analysis

Feedback – Scenario four

Marking error

– assessor transcription error not spotted by moderator

Incident has

– has prejudiced students

– adversely affect AOs ability to deliver and award qualifications in

accordance with the conditions; standards and public confidence

More information needed?

– does the error impact on just one component/unit or the whole

qualification?

– is the error isolated to one moderator?

Action

– notify regulator, centres and students

– re-issue results

– root cause analysis and mitigation

Feedback – Scenario five

Maladministration or malpractice

– ghost candidates?

Incident could

– adversely affect AOs ability to deliver and award qualifications in

accordance with the conditions; standards and public confidence

More information needed?

– why did the centre lose its funding contract?

– is the identity of the students genuine?

Action

– notify regulator and AOs

– apply sanction?

– investigate and sample evidence of assessment

Table discussion

Would you have done anything differently if you had known the

outcome at the beginning?

Is there anything more proactive you could have done to make your

decisions quicker and more effective?

Is there anything that has surprised you about:

– the way other tables have approached either the same task as

you, or different tasks?

– the way Ofqual has responded to the task?

Is there anything you disagree with?

Do you now know what, when and how to report to Ofqual?

Next steps for us

Review of non-statutory guidance

– include more information on what we would expect to receive

Review of statutory B3 guidance

– revised examples of positive and negative indicators

Introduce severity categorisation

– to inform analysis and reporting