SQA’s Approach to Quality Assurance of AssessmentSQA’s Approach to Quality Assurance of Assessment

Matthew McCullaghQuality Manager

Welcome

欢迎

Why are we here?Why are we here?

To introduce you to SQA’s New Approach to Quality Assurance

To help you understand the key changes to Quality Assurance process

To help you become familiar with:– The new Quality Assurance Criteria– The new Confidence Statements– How visits are planned, conducted and

reported

Monday 10th September 2012Monday 10th September 2012

9.00 Coffee and Registration

9.40 Introduction

9.45 New Approach to QA Overview, Aims, Objectives and Overview of Event

10.30 Break

10.45 Systems Verification & Evidence

12.00 Lunch

1.30 Systems Verification & Evidence

2.45 Break

3.00 Qualification Verification & Evidence

4.15 Summary & Outline of Day 2

4.30 Close

SQA’s Approach to QASQA’s Approach to QA

SQA is moving to introducing a new approach to Quality Assurance of qualifications delivered in China

It is based on SQA’s

Principles of Quality Assurance:Open, fair and transparent Aimed at risk reductionProportionate Intelligence ledA shared responsibility Constantly reviewed and improved

What is the new approach to Quality Assurance

What is the new approach to Quality Assurance

SQA now operates a transparent, intelligence-led, risk based model for quality assurance of approved centres and potential centres delivering our HND qualifications in China.

Timeline of Implementation of New Approach in China

Timeline of Implementation of New Approach in China

September 2011

May 2012

September 2012

September 2012-2013

March 2012

New Quality CriteriaNew Quality Criteria

Five Categories: – Management of a centre– Resources– Candidate Support– Assessment & Verification– Records/Data Management

All signposted to four QA processes All pre-rated as High/Medium/Low impact All supported by possible sources of evidence

The New Approach to Quality Assurance: The Four Processes

The New Approach to Quality Assurance: The Four Processes

Systems Approval

Qualification Approval

Systems Verification

Qualification Verification

High/Medium/Low Impact CriteriaHigh/Medium/Low Impact Criteria

All criteria are important However, some are more important than others Those that are most important are rated as High

Impact criteria Those that are less important are rated as Low

Impact criteria In between these two are medium impact

criteria. Failure to meet a High Impact criteria will have a

greater impact on the outcome of the visit

Sufficiency of evidenceSufficiency of evidence

No Systems Verifier or Qualification Verifier can change the impact rating of a criteria.

For each criterion you verify, you must decide whether the centre has presented you with:– Sufficient evidence to meet the criterion– Some, but insufficient evidence to meet the

criterion– No evidence to meet the criterion

Good Practice and GuidanceGood Practice and Guidance

As part of the verification visit, you should identify good practice that is evident in the centre.

You should also give advice and guidance to help centre staff improve the delivery of our qualifications

This is a mandatory part of your role as a Qualification Verifier

Sufficiency of evidence Sufficiency of evidence

We will support you and help you to understand what sort of evidence centres can provide to meet each criterion

Once you have made your decision regarding the sufficiency of evidence, SQA staff will then calculate our level of confidence

This will depend on the impact level of the criteria and the sufficiency of the evidence presented

Introduction of Confidence StatementsIntroduction of Confidence StatementsIn relation to Qualification

VerificationConfidence Statement

In relation to Systems Verification

High level of confidence in the maintenance of SQA standards within this Verification Group

 High Level of Confidence

High level of confidence in the systems that support the maintenance of SQA standards within this centre

Broad confidence in the maintenance of SQA standards within this Verification Group

 Broad Confidence

Broad confidence in the systems that support the maintenance of SQA standards within this centre

Reasonable confidence in the maintenance of SQA standards within this Verification Group, although moderate risks exist within the following categories:(insert appropriate category)(insert appropriate category)

 Reasonable Confidence

 

Reasonable confidence in the systems that support the maintenance of SQA standards within this centre, although moderate risks exist within the following categories:(insert appropriate category)(insert appropriate category)

Minimal confidence in the maintenance of SQA standards within this Verification Group as significant risks exist within the following categories:(insert appropriate category)(insert appropriate category)

 Minimal Confidence

Minimal confidence in the systems that support the maintenance of SQA standards within this centre as significant risks exist within the following categories:(insert appropriate category)(insert appropriate category)

No confidence in the maintenance of SQA standards within this Verification Group as severe risks exist within the following categories:(insert appropriate category)(insert appropriate category)

 No Confidence

No confidence in the systems that support the maintenance of SQA standards within this centre as severe risks exist within the following categories:(insert appropriate category)(insert appropriate category)

How Confidence Statements are calculatedHow Confidence Statements are calculated

Confidence statements are given for the outcome of each Systems Verification and Qualification Verification visit for each of the categories of criteria verified

These are based on the sufficiency of the evidence provided to the you

A summary statement of confidence is also given, based on the confidence statements for each category

What if non-compliances are identified?

What if non-compliances are identified?

Where there is No evidence or insufficient evidence to meet a criterion, you must agree with centre staff, during the visit: – the action the centre needs to take in order to

become compliant. – The evidence they need to produce – Where to send this evidence– The date by which the action must be taken

What happens then?What happens then?

SQA will ask you to consider the evidence the centre sends us and ask you to consider whether it is Sufficient or insufficient.

Depending on the sufficiency of the evidence, the Confidence Statement will be re-calculated.

Sanctions Sanctions

If the outcome of a verification visit results in a Confidence Statement of Reasonable, Minimal or No Confidence, SQA staff may decide to place a sanction on a centre until our confidence increases to Broad or High.

Our confidence may decrease if a centre fails to meet action points agreed during a verification visit.

Sanctions will only be used where necessary.

SanctionsSanctionsIn relation to Qualification

VerificationConfidence Statement

In relation to Systems Verification

 High Level of Confidence

Entry in Action Plan  Broad Confidence

Entry in Action Plan

Suspension of specific qualification certification:by verification group by qualification  Suspension of approval application by verification group Suspension of direct certification claim status:by verification group by qualification

 Reasonable Confidence

Suspension of all existing qualification approval Suspension of centre certification Suspension of certification for all qualifications with assessed components Suspension of qualification approval application - whole centre

Suspension of specific qualification approval:by verification groupby qualification

 Minimal Confidence

Withdrawal of all existing specific qualification approval

Withdrawal of specific qualification approval:by verification groupby qualification

No Confidence Withdrawal of centre approval

Benefits of new approach for centres

Benefits of new approach for centres

Open and transparent to all Clear guidance available to verifiers Proportionate response where issues are

identified Focused on Good Practice and improvement as

well as compliance

QuestionsQuestions

Introducing the Quality Assurance Criteria

Introducing the Quality Assurance Criteria

Systems Verification Criteria

You have been provided with criteria from the two categories of the Systems Verification

Criteria (Management of a Centre and Resources). In addition to this you have been provided

with many examples of evidence.

Working in your groups please match the appropriate examples of evidence to the relevant

criteria.

Formulate a list of points on flip chart paper to help you explain the chosen evidence for each

criterion to your colleague verifiers. The summary should also include possible sources of

evidence.

Timings: 30 minutes discussion and research and flip –charting your points

30 minutes to feedback

Please appoint a spokesperson(s) to present the flip-charted points.

Each group will be assigned a facilitator to answer any questions/queries you may have.

New Approach to Quality AssuranceNew Approach to Quality Assurance

Welcome Back

欢迎回来

Introducing the Quality Assurance Criteria

Introducing the Quality Assurance Criteria

Systems Verification Criteria

You have been provided with criteria from the two categories of the Systems Verification

Criteria (Management of a Centre and Resources). In addition to this you have been provided

with many examples of evidence.

Working in your groups please match the appropriate examples of evidence to the relevant

criteria.

Formulate a list of points on flip chart paper to help you explain the chosen evidence for each

criterion to your colleague verifiers. The summary should also include possible sources of

evidence.

Timings: 30 minutes discussion and research and flip –charting your points

30 minutes to feedback

Please appoint a spokesperson(s) to present the flip-charted points.

Each group will be assigned a facilitator to answer any questions/queries you may have.

Introducing the Quality Assurance Criteria

Introducing the Quality Assurance Criteria

Qualification Verification Criteria

You have been provided with criteria from the two categories of the Systems Verification

Criteria (Management of a Centre and Resources). In addition to this you have been provided

with many examples of evidence.

Working in your groups please match the appropriate examples of evidence to the relevant

criteria.

Formulate a list of points on flip chart paper to help you explain the chosen evidence for each

criterion to your colleague verifiers. The summary should also include possible sources of

evidence.

Timings: 30 minutes discussion and research and flip –charting your points

30 minutes to feedback

Please appoint a spokesperson(s) to present the flip-charted points.

Each group will be assigned a facilitator to answer any questions/queries you may have.

Tuesday 11th September 2012Tuesday 11th September 2012

9.30 Welcome

9.35 Process: Preparing & Planning a Visit

10.50 Break

11.00 Process: Conducting a Visit

12.20 Lunch

1.50 Decision Making/Action Planning/Reporting

3.05 Break

3.20 Question & Answer

4.20 Close

QuestionsQuestions

Planning for systems/qualification Verification

Planning for systems/qualification Verification

The systems/qualification verifier role activities

Prepare for verification Plan verification Carry out verification Communicate verification decision Complete verification report

Verification planning – the centres’ involvement

Verification planning – the centres’ involvement

Each centre plays a vital role at the planning stages of a visit. They help ensure

An optimum time is selected for visits to take place

There is enough evidence for the verifier to sample (for the verifier and centre to gain value from the visit)

Visit Planning - Qualification Verification

Visit Planning - Qualification Verification

SQA (China Office) agrees optimum date (s) for visiting verification. This is an agreement between

SQA, each centre and each QV

SQA selects Units to be verified and communicates this to each centre (copying to

SQA China Office)

Qualification Verifier contacts centre to confirm visit date(s) and travel arrangements

SQA (China Office) confirms to each QV, the centre(s) they should visit, the date of each

visit, and the Units they should verify

Visit Planning - Systems Verification

Visit Planning - Systems Verification

Accept centre allocation from SQA

Make initial contact with each centre

Formulate Visit Plan

Request information from centre

Send visit planning documentation

Workshop 4: Identifying the preparations a centre would make.

Workshop 4: Identifying the preparations a centre would make.

Before the verification visit.

In your groups, identify the types of preparations a centre would usuallymake prior to receiving a verification visit. Write down the points you have identified on the post-its (one point perpost-it) provided. A facilitator will collect post-its from all of the groups to be used to support a plenary discussion.

In formulating your points, you may wish to refer to SQA’s New Approach to Quality Assurance: Guidance on Visiting/Systems Verification for HND Qualifications in China as your information source. Timings: 30 minutes discussion (and jotting down points)

15 minutes plenary discussion

Each group will be assigned a facilitator to answer any questions/queries you may have.

Visit Planning – Qualification Verification: Applying a sampling

frame

Visit Planning – Qualification Verification: Applying a sampling

frame SQA is responsible for systematically sampling on a percentre basis:

Selected Units (against selected cohorts) Assessment/internal verification decisions and practice Methods/Instruments of assessment Problematic/revised units Locations where assessment takes place Evidence of candidates work for the Units selected Documented evidence of policies/procedures (implementation)

Carry out systems/qualification verification

Carry out systems/qualification verification

The systems/qualification verifier role activities

Prepare for verification Plan verification Carry out verification Communicate verification decision Complete verification report

The role of the systems/qualification verifier in carrying out the visit

The role of the systems/qualification verifier in carrying out the visit

Agreeing the agenda and running order

Conducting sampling activities

Judging the sufficiency of evidence

Workshop 5: Identifying a centre’s involvement during a verification visit

Workshop 5: Identifying a centre’s involvement during a verification visit

During the verification visit

In your groups, using the grid provided, identify how centres might get involved/support each of the following stages when the verifier is conducting a verification visit.

In formulating your points, you may wish to refer to SQA’s New Approach to Quality Assurance: Guidance on Visiting/Systems Verification for HND Qualifications in China as your information source. Timings: 30 minutes discussion (and jotting down points)

15 minutes plenary discussion

Each group will be assigned a facilitator to answer any questions/queries you may have.

Agreeing the agenda and running order

Agreeing the agenda and running order

The centre’s role:

Discussing agenda/timings/order

Agreeing level of involvement when verifier is sampling of evidence

Being clear on arrangements for sampling/interviews

Communicating arrangements to relevant centre staff

Conducting sampling activities Conducting sampling activities The centre’s role:

Providing access to all required documentation (policies, procedures, candidate evidence, assessment records etc)

Making suitable accommodation available for sampling and interviews

Allowing the right personnel to be available at the relevant

times

Maintaining an absence if or when presence is not required

In new NAQA Guidance - Conducting sampling activities

In new NAQA Guidance - Conducting sampling activities

Complete Units (all Outcomes complete) Incomplete Units (some Outcomes complete) Evidence not yet internally verified Assessment/verification decisions Assessment and verification practice Interviews with assessors/verifiers/candidates Referencing of evidence to standards

Judging evidence in relation to each criterion

Judging evidence in relation to each criterion

Centre provides (through documentation and or

interviews) as much relevant information as

possible in support of each criterion.

Access Access

Increased understanding

More questions

Access to information

Carry out systems/qualification verification

Carry out systems/qualification verification

The systems/qualification verifier role activities

Prepare for verification Plan verification Carry out verification Communicate verification decision Complete verification report

Communicating verification decisions

Communicating verification decisions

The verifier’s roles involves:

Delivering balanced feedback Explaining their verification decisions

Communicating verification decisions

Communicating verification decisions

Balanced feedback – the verifier’s aim is to provide an objective reflection of each centre’s level of compliance in relation to each criterion.

Communicating verification decisions

Communicating verification decisions

Specifically, feedback will cover:

criteria that was met good practice recommendations agreed action points

Communicating verification decisions

Communicating verification decisions

SQA consider the explanation of verification

decisions to be an important part of the feedback

process.

Centres should be clear as to how decisions have

been made in relation to each criterion.

Clarifying verification decisions Clarifying verification decisions The use of questioning is key. If questions are carefullyworded, they can benefit both parties. Some examples:

Can you explain this point again, I am not sure if I have fully understood it? Did I mention that we also have evidence in file ……….? Therefore, we met this criterion because …

(paraphrase)? Can you give me a description of the type of evidence

you are looking for, so that I can fully meet this action point?

We proposed that this evidence…. covered the unit specification, was our thinking along the right lines?

Agreeing action points Agreeing action points

Specific: Centres should be clear in terms of the action they need to take to close-off the action point.

Measurable: How will the centre know that the action point has been met, what will be the measure of success?

Achievable: The centre must have sufficient time in which to achieve the agreed action (consider who is involved, the amount of work required).

Relevant: The action must directly relate to the criterion.

Time bound: Agreement of specific and realistic date(s) for the action to be completed by.

Workshop 6:After the verification visitWorkshop 6:After the verification visit

After the verification visit

Consider the activities a centre would undertake as a result of the outcomes ofa verification visit In particular consider :

• what should happen• who should be responsible• when it should happen • how you know activities have been completed successfully

Write your points on the post-its provided (one point per post-it).

In formulating your points, you may wish to refer to SQA’s New Approach to Quality Assurance: Guidance on Visiting/Systems Verification for HND Qualifications in China as your information source.

Timings: 30 minutes discussion (recording your points) 15 minutes plenary discussion

Each group will be assigned a facilitator to answer any questions/queries you may have.

After the verification visit: centre responsibilities

After the verification visit: centre responsibilities

Once verification has taken place, the centre is

responsible for following up on the outcomes of the

verification report. Those responsibilities would

include:

• Dissemination of the report• Allocating responsibility for actions• Tracking progress and achievement of actions

SQA’s Approach to Quality Assurance of AssessmentSQA’s Approach to Quality Assurance of Assessment

Welcome Back

欢迎回来

QIIPS:Quality, Integrity, Innovation, Partnership

& Service

QIIPS:Quality, Integrity, Innovation, Partnership

& Service

Sounds like:

“QUIPS” – a dictionary definition:

“A clever remark often prompted by the occasion”.

SQA Values SQA Values

Quality

Integrity

Innovation

Partnership

Service

SQA ValuesSQA Values

Quality

As custodians of the standards we verify, we

continually strive to maintain consistent quality in

their assessment.

SQA Values SQA Values

Integrity

We use open and honest communication with internal and external customers (centres), promoting transparency, and building trust with others.

We take a standardised approach to verification as laid down by SQA and agreed by consensus within our Qualification Verification team, even if our personal opinions may at times conflict.

SQA Values SQA Values

Innovation

We view unfamiliar approaches to assessment and verification with an open mind, whilst ensuring quality is maintained.

We believe that creativity should be seen not as a threat to quality assurance but as an opportunity for improving upon existing practice.

SQA Values SQA Values

Partnership

We understand that by working in partnership with SQA centres, we will achieve common goals of excellence and consistency in assessment.

SQA Values SQA Values

Service

We maintain a professional approach with SQA centres at all times, regardless of the challenges we might face in carrying out our work.

Workshop7

Workshop7

Values Exercise

SummarySummary

To introduce you to SQA’s New Approach to Quality Assurance

To help you understand the key changes to Quality Assurance process

To help you become familiar with:– The new Quality Assurance Criteria– The new Confidence Statements– How visits are planned, conducted and

reported

QuestionsQuestions