Portfolio Committee on Higher Education

Report on the Quality Assurance of NC (V) L2-4; NATED N2-3 and GETC:

ABET L4

3 February 2016Dr Mafu S Rakometsi

WHAT CAN WE LEARN FROM THE NSC RESULTS?

Introduction to context, principles, approaches and

processes Dr Mafu Rakometsi - CEO of Umalusi

Quality Assurance of theDHET, IEB & BENCHMARK November

2015 examination

Faith Ramotlhale: Senior Manager Quality Assurance of Assessments

Section17 of the GENFETQA Act(5) The Council must, with the concurrence of the Director-

General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has—

(i) conducted the assessment free from any irregularity that may

jeopardise the integrity of the assessment or its outcomes;(ii) complied with the requirements prescribed by the Council

for conducting assessments;(iii) applied the standards prescribed by the Council which a

learner is required to comply with in order to obtain a certificate; and

(iv) complied with every other condition determined by the Council.

Regulatory Framework

Quality Assurance of AssessmentNQF Act Section 27 (h) The QC must develop and implement policy and

criteria for assessment for the qualifications on its sub-framework.

Framework for Quality Assurance of Assessment

Evaluation and /or accreditation of assessment bodies; Periodic inspection of assessment systems; Ongoing monitoring of assessment systems; Quality assurance of external examinations through: Moderation of examination question papers, PATs, CATs and

ISATs Monitoring and moderation of SBA Monitoring of the conduct, administration and management of

assessment and examination processes Moderation of marking Management of concessions and examination irregularities Standardisation of assessment outcomes

Moderation of question papers

PURPOSE: To ensure that the question papers are of the required

standard; To ensure that the question papers are relatively: Fair Reliable representative of an adequate sample of the

curriculum representative of relevant conceptual domains representative of relevant levels of cognitive

challenge

ApproachDHET:NATED/NC(V)) DHET: GETC IEB: GETC BENCHMARK: GETCOffsite Centralised Offsite Offsite

Moderation Process:•Question paper development - examiners (assessment body)•Internal moderation - Internal moderators (assessment body)•External moderation - External moderators (Umalusi)•Subsequent moderations and approval

Moderation of the question papers

Technical criteria Internal moderation Content coverage Text selection, types

and quality of questions

Cognitive demands Marking

memorandum/ guidelines

Language and bias Predictability

Criteria:

Question Paper ApprovalAssessment body

No of papersmoderated

Approval of Papers

DHET: GETC 26/26 54% first moderation46% second and third moderation

IEB: GETC 8/8 88% first moderation12% second moderation

BAA: GETC 2/2 50% first moderation50% second moderation

DHET: NATED N2-N3

N2: 16/16 N3: 20/24

Deviation of 4 Business English papers- late submission and incorrect versions of papers submitted

DHET: NC (V) L2-4

L2: 25/25L3: 25/25L4: 84/84

(Back-up papers of 2014 used, new papers approved where curricula changed and for 4 new Level 4 subjects)

Moderation of the question papersAreas of Good Practice: No question paper leakages (GETC: IEB, BAA & DHET) Approved GETC QPs 98% (DHET) and 97% (IEB)

compliance ratings. Some improvement in the internal moderation of the

NATED subjects observed.

Directives for ComplianceAssessment bodies should: Improve the quality and standard of internal moderation -(DHET, IEB, BAA),Increase the percentage of question papers approved at first moderation -(DHET, IEB, BAA),Improve the quality of marking guidelines -(DHET),Improve the quality of the question papers (GETC) for TRVT4 and WHRT4 – consistently poor quality since November 2013,Review outdated and/or underspecified curricula for the (NATED) subjects.

Directives for Compliance The DHET is required to: develop assessment frameworks for the NATED N2 and N3 subjects to address amongst others, the cognitive demands and the weighting of topics;Review its tracking system for question paper moderation to ensure that correct versions of question papers, marking guidelines, internal moderator reports and assessment frameworks are submitted to Umalusi for external moderation, (NC(V) and NATED);Adhere to its management plan for the setting and submission of question papers for moderation.

Internal Assessment (SBA)Definition:Internal assessment refers to any assessment conducted by the provider , the outcome of which counts towards the achievement of the qualifications:

GETC: 50%; NC (V) fundamental subjects 25%; NC (V) vocational subjects 50%; and NATED subjects 40%;

Purpose:Ascertain the degree to which the assessment body is attempting to ensure standardisation across providers,Ascertain the standard and quality of the tasks,Establish the extent and quality of internal moderation and feedback, and

Verify the reliability and validity of the assessment outcomes.

Internal Moderation - GETC Assessment Body

CATs moderated and approved

SBA Portfolios

DHET 5 sets x 26 learning areas

On-site moderation:•251 portfolios•6/9 PEDs – 40 centres

IEB None On-site moderation: •4/8 Learning Areas•40 portfolios

Benchmark None On-site moderation:•2/2 LAs•20 portfolios

Site-Based Assessment: GETCAreas of Good Practice Year-on-year improvement regarding the administration of SBA, (DHET)Implemented self-evaluation instrument and visited centres to support and improve implementation of SBA tasks, (BAA).

Directives for compliance: GETC Assessors (educators) do not understand the use of

rubrics, and marking guidelines, (BAA, IEB & DHET); The DHET and IEB should develop a strategy to monitor

and support the implementation of the SBA tasks to improve the quality of teaching and learning;

The IEB should revise their User Guides and Facilitator Guides and present these to Umalusi for moderation and approval;

BAA must implement a strategy to ensure that training providers implement all SBA tasks.

Internal Assessment (NC (V)/NATED)ISAT (tasks) ISAT (conduct) ICASS

NATED 8/9 subjectsNC( V) 13/16 55/55 sites 55/55 subjects

Areas of good practice: ISATSome colleges had effective systems and procedures in place to ensure the smooth implementation of the tasks.

Directives for compliance: ISATDefective ISATs should be revised (DHET plan in place),ISATs should be distributed to colleges in time for effective procurement of resources and implementation,Enrolments must be in line with available resources to effectively assess candidates’ acquisition of the practical skills required in the workplace,To improve the quality and standard of marking and scoring,

Internal Assessment (NC (V)/NATED)Directives for compliance: ISAT (continued)The DHET must ensure that ISATs are implemented as per the specifications of the specific tasks and that effective moderation of the conduct of the tasks occurs at all levels.

Internal Assessment: NC (V)/NATEDDirectives for compliance/improvement: NC (V) ICASS

The DHET should provide training on the internalisation of SAGs to promote effective planning, facilitation and conduct of assessment.Lecturers require training in setting and moderation practices as well as the review of assessments and tasks.

Directives for compliance/improvement: NATED ICASS

The DHET is required to enforce the implementation of the NATED ICASS guidelines at all sites to ensure that appropriate assessment tasks are developed and marks are correctly converted and captured.

Monitoring of Examinations: GETC

Directives for compliance (DHET)The DHET should ensure that: There is adequate and secure storage rooms for the

storage of question papers, The safety and security of examination materials is

improved, as some centres keep answer scripts overnight,

All candidates submit proper identification during the writing of examinations.

Assessment body DHET IEB BAANo exam centres 45 15 6

Monitoring of Examination: GETCDirectives for compliance (BAA & IEB)

Improve the following instances of poor compliance with exam regulations:Inadequate storage facilities for QPs with poor security arrangements,Poor administration regarding exam regulations,Use of cellphones in the examination venues,Candidates allowed to write without proper identification

Monitoring of Examinations: TVET

Areas of good practice Improved systems in the conduct of examinations and

measures to ensure the security of examination materials, Effective training of chief invigilators and invigilators, Conducive environment for the writing of examinations.Directives for compliance The DHET should ensure that errata arrive on time at the

various examination centres The DHET should ensure that all examination centres have

adequate and reliable security for the storage of examination materials.

NC (V) and NATEDNo. exam centres 41

Monitoring of Marking Phase: GETC

Areas of Good Practice: Marking centres were generally well organised and suitable for the task, (DHET,IEB,BAA)Adequate and experienced security personnel at the gates and entrances to the venues, (DHET,IEB, BAA)Training of marking personnel was generally thorough and effective, (DHET,IEB,BAA)

DHET IEB BenchmarkNo. of marking centres

11 1 1

No. centres monitored

6 1 1

Monitoring of Marking Phase: GETCDirectives for compliance:The Assessment Bodies should ensure that there is compliance with the following examination regulations:Availability of the management plans (DHET, IEB & BAA)Inadequate security measures (DHET, IEB)Poor systems to manage and report irregularities (DHET)Administrative mistakes such as sending wrong scripts to centres (DHET),

Monitoring of Marking Phase: TVET

Areas of good practice:•Marking centres provided an environment that was conducive to effective marking and control.

NATED NC (V)No of marking centre monitored

7/10 1/3

Monitoring of Marking Phase: TVETDirectives for compliance:

The DHET must ensure that approved marking guidelines are received in good time at all marking centres,THE appointment of markers should be completed well in advance.

Purpose: Moderation of marking determines the standard and quality

of marking and ensures that marking is conducted in accordance with agreed practices

Umalusi engages with the following processes during the moderation of marking:

Centralised memorandum discussions Verification of marking (centralised and on-site)

Verification of Marking:

Areas of good practice:The panel discussions and the marking of exemplar scripts improved the quality of the marking guidelines (DHET, IEB, BAA).Directives for compliance:The DHET must ensure that it improves the quality of Afrikaans versions of most marking guidelines, The DHET should develop a system to ensure that the final memoranda are discussed and cascaded at provincial level,The IEB should ensure that the marking personnel come prepared at the marking centre,The BAA should prioritize the training of markers before the actual commencement of marking.

Memorandum Discussions - GETC

Marking guideline discussions and verification of marking TVET

Areas of good practice:More markers came prepared to the marking centreSample marking took place in most subjectsIn the majority of subjects there was good adherence to the marking guidelinesIn most subjects markers kept notes to enhance the quality of chief marker and internal moderation reports

NC (V) NATED N2 NATED N3Marking guideline discussions

38 subjects 5/16 subjects moderated

11/20 subjects moderated

Verification of marking

L2 8/25L3 6/25L4 55/76 subjects

12/16 19/20

Marking guideline discussions and verification of marking TVET (cont)

Directives for improvement/compliance:

The DHET should:ensure that adequate, suitably qualified and experienced marking staff is recruited and appointed timeously,enforce the attendance of the general training session and the marking guideline discussion by all marking staff,ensure that the correct versions of marking guidelines are distributed and used across the different marking centres,Improve the quality of internal moderation and comply with the 10% internal moderation of all scripts in all subjects, (NC(V)

Areas of Good Practice:Good internal moderation practices (DHET, IEB, BAA).Good quality and standard of marking, (DHET, IEB, BAA).

Verification of Marking: GETC

DHET IEB BenchmarkOn-site moderation

15/26 LA 945

scripts

4/8 LA 83 Scripts

2/2 LA 40 scripts

Directives for compliance:The DHET should ensure that all irregularities are reported by the Internal Moderators and Chief Markers, The DHET should put measures in place to avoid damaged and/or lost computer discs containing the practical work of candidates in KZN,The BAA should prioritise the training of markers before the commencement of marking.

Verification of Marking

Standardisation and Resulting Provision of GENFETQA Act, 2001 as amended in 2008-–

Council may adjust raw marks, International practice – large scale assessment systems, Standardisation – process used to mitigate the effect of

factors other than learners knowledge and aptitude on the learners performance,

Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions.

Verification of the Resulting Process Planned Status Subject structures Subject structures verifiedCandidate registration System verified during state

of readiness visitsGeneration of mark sheets MonitoredCapturing of Marks MonitoredStandardisation data & Booklets

Data sets received and verified

Capturing of adjustments VerifiedStatistical moderation & resulting

Verified

Statistical ModerationScope of standardisation GETC 2015:Decision Number of subjects

DHET IEB BMSubjects standardised

26 8 2

Raw marks accepted 13 8 2Adjusted mainly upwards

11 - -

Adjusted mainly downwards

2 - -

Statistical Moderation - GETCAreas of good practice:Booklets and data sets were submitted according to the prescribed timelines (DHET, IEB, BAA).

Standardisation and approval meetings went smoothly.

Statistical Moderation - GETCDirectives for improvement: The DHET should develop and implement strategies to

improve the poor performance of learners; high number of irregularities; and absenteeism to write the examinations,

The IEB should develop its internal quality assurance of standardisation booklets to ensure that accurate data on graphs and statistics is submitted,

The BAA should document the processes and procedures outlining standard operating procedures w.r.t. capturing and resulting.

Statistical Moderation TVETScope of standardisation 2015:

Decision Number of subjectsNATED NC (V)

Subjects standardised: Engineering Studies : Business Studies

594 252

Raw marks accepted: Engineering Studies : Business Studies

29 2

122

Adjusted mainly upwards: Engineering Studies : Business Studies

28 2

127

Adjusted mainly downwards 1 2No decision 1 1

Statistical Moderation: TVETAreas of good practice

KS norms replaced by Ogives (NATED)

Directives for improvement:

The DHET should ensure that: Standardisation booklets are submitted on time to avoid

delays in the standardisation of results, Accurate data sets and low capture rate is improved to avoid

delays in the approval of results, The consolidated irregularity report is submitted to Umalusi

24 hours prior to the standardisation of results,

Conclusion The quality assurance processes presented above were

conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations.

From the findings of the quality assurance processes some pockets of good practices were evident but on the other hand, there are still many challenges that need to be addressed.

Significant interventions are required to improve the curriculum, teaching and learning, exposure of learners to a compulsory practical component and assessment for the NATED programmes to make it a meaningful qualification.

Conclusion The outdated and underspecified curriculum and lack of

subject assessment guidelines for the NATED programmes creates problems for appropriate quality assurance to be conducted.

The DHET is commendable for the radical actions taken to curb paper leakages in the NATED examinations.

The development and implementation of the ogive method for the first time in 2015 is also worth commendations.

Umalusi takes this opportunity to express appreciation to the department of higher education and training for their concerted efforts in delivering credible examinations in 2015.

Thank you!