The Qualification Manager’s Handbook€¦ · • The Office of Qualifications and Examinations...

TheQualificationManager’sHandbook

2

CEO’s foreword

I am pleased to present the first edition of The Qualification Manager’s Handbook

from the Federation of Awarding Bodies (FAB).

This Qualification Manager’s Handbook provides practical information for people in

assessment and awarding organisations, whether this is new information for those just

starting their career or a useful reminder for more experienced members of staff.

The handbook includes an overview of the sector, an introduction to the key areas of the

qualification lifecycle and insight into the range of topics and activities, including validity,

in which a member of staff in an awarding organisation may engage.

The handbook is part of a wider strategy to define the body of knowledge required

to design and deliver high quality assessments and qualifications. This is an ongoing

programme, encouraging debate and research, ensuring that new ideas are shared

through FAB as well as informing our training programme and of course future

editions of the handbook. In an increasingly globalised qualification market, the

reputation, dynamism and skills in the UK awarding sector puts us in a strong

position to play a leading role by encouraging and sharing new ideas, building our

reputation as thought leaders.

Post-16 skills has suffered over many years because policy-makers have constantly

underestimated the expertise needed to develop robust and valid technical and

professional qualifications, often referencing inappropriate academic assessment

methods. A more clearly articulated body of knowledge will challenge this

misconception, helping to inform more effective policies that deliver the results

for learners and employers that we all want to see.

This handbook is a collaborative effort, but the main thanks go to Trisha Fettes who

wrote the book, sharing her skills and knowledge as well as presenting all of our ideas

in a coherent manner; also Prue Huddleston who chaired the editorial board as well as

all those people involved with the development, scrutiny and production, including our

friends at Policy Connect.

This handbook is just the beginning, it is a living document – and I’d like to invite you

to become part of the wider editorial team. We welcome your comments and feedback.

Let’s work together on to make this a comprehensive, practical and well-thumbed

resource for you and everyone in the sector.

Email your thoughts and comments to [email protected]

Stephen Wright

Chief Executive, Federation of Awarding Bodies

Acknowledgements

Particular thanks for contributions to this guide are due to:

Stephen Wright who commissioned the handbook

Trisha Fettes (lead writer)

Karen Daws from FAB

Barry Smith and Steven Harper (co-authors)

Professor Prue Huddleston, Chair of FAB’s Awarding Advisory Group (AAG)

Paul Newton, Ofqual

Norman Gealy

Denise Clarke

Gavin Busutti-Reynaud and colleagues from AlphaPlus Consultancy Ltd

Dee Arp

Thanks go to

Richard Chapman

Patrick Craven

Inga Fitzgerald

Debra Malpass

Cathy O’Shaughnessy

Charlotte Payne

Susan James Relly

Rob Ricketts

Lynn Senior

Debbie Tuckwood

Jonathan Tummons

Tracy Turville

Louise Young

4

Introduction 6

Purpose of this handbook and structure 7

Overview of the Awarding Sector 9

Chapter 1: So, You Want to Develop a Qualification 21

1.1 Overview of the qualification lifecycle 22

1.2 Managing your qualification’s lifecycle 25

Chapter 2: Understanding Validity And Assessment 41

2.1 What makes a qualification sufficiently valid 43

2.2 Planning for validation 50

2.3 Principles of assessment 53

2.4 Forms of assessment 60

Chapter 3: How To Design A Qualification 71

3.1 Conduct market research and consultations 73

3.2 Establish your qualification’s objective and target level 83

3.3 Specify what your qualification will measure 89

3.4 Identify your approach to assessment (strategy) 98

3.5 Specify how your qualification will be assessed 108

Chapter 4: How To Develop A Qualification 111

4.1 Shape the qualification and its progression 113

4.2 Develop the assessment apparatus, including support materials 121

4.3 Put it all together to inform delivery of your qualification 132

Chapter 5: How To Deliver A Qualification 137

5.1 Prepare for delivery of your qualification 139

5.2 Provide support for qualification personnel and learners 154

5.3 Understand what happens when candidates take assessments 163

5.4 Maintain controls and security, deal with malpractice 170

5.5 Evaluate candidates’ performance 180

5.6 Report results, deal with appeals, award and certificate 185

Chapter 6: Qualification Review And Evaluation 189

6.1 Prepare for review and evaluation 191

6.2 Review the performance of your qualification 193

6.3 Evaluate your qualification, including its validity 198

6.4 Use data and findings 207

Appendices 211

A) Assessment methods – key features 211

B) References and further information 221

Contents

6

Introduction

7Introduction

Purpose of this handbook and structure

Purpose of the handbook

This handbook has been commissioned by the Federation of Awarding Bodies (FAB).

FAB is a membership organisation which supports and represents the interests of UK

vocational Awarding Organisations.

The focus of the handbook is on regulated vocational qualifications. It does not set out

to cover the particular requirements of general qualifications, such as Advanced General

Certificates of Education (GCE A Levels) and degrees, or unregulated qualifications,

although some content may be relevant.

It has been designed primarily for use by Qualification Managers who are new to their

post. It may also be used as a handy source of reference for others with a specific interest

in the development and management of vocational qualifications.

How it is used will depend on your level of experience and the particular context(s) in

which you work. The size and structure of your Awarding Organisation will determine

the range of your particular responsibilities.

Whole books have been written on many of the subjects, for example, different forms of

assessment and theories that underpin them. It has not been possible, therefore, to cover

these in detail. The aim of the handbook is to provide insights into the range of topics and

activities in which Qualification Managers may engage. References are provided for you

to explore in more depth, topics of particular interest.

See Appendix B for further reading and FAB’s online library.

Structure of the handbook

Firstly, an overview of the UK Awarding Sector is offered to set the scene for subsequent

chapters which outline key tasks and requirements for managing the lifecycle of a

qualification.

Chapter 2 broadly discusses validation and assessment, the understanding of which

is central to developing and delivering qualifications. Subsequent chapters cover each

lifecycle stage: how to design, develop, deliver, review and evaluate a qualification.

Each chapter has a summary of key points and references to relevant support and

publications available through FAB. As qualification development is a creative and cyclical

process, at points in the text you may need to refer to other sections of the handbook.

Where this may be helpful, clear signposts are provided, including use of arrows.

8 Introduction

Appendix A includes different methods of assessment.

Appendix B provides references and details of key organisations in the UK concerned

with vocational qualifications including apprenticeships; and those supporting

international work.

The handbook reflects developments in the sector at the time of writing. However, this

working environment is continually moving in response, for example, to government

policy and market demands.

It is acknowledged that many of the behaviours of Awarding Organisations will be driven

by qualification regulations and funding. But these are subject to change. Rather than

replicate current regulations, this handbook tries to identify general principles and tasks

that are likely to have a longer shelf-life.

This means, however, that you will need to keep abreast of developments as the sector

grows and develops. You are encouraged to use the FAB website and weekly updates to

assist in this.

Access to updates via FAB

The FAB website has a repository for resources and literature of relevance to

the vocational Awarding Sector, including that linked to each chapter of this

handbook

The website also includes notice of training events, meetings and webinars:

http://www.awarding.org.uk/home

The FAB Weekly update is the newsletter for FAB members. To access the latest

issue, you need to login with your membership details. If you would like to join

the mailing list, please email: [email protected].

See Appendix B for web links to qualification regulators, government departments and

other organisations that provide updates.

9Introduction

Overview of the Awarding Sector

Awarding Organisations operate in a dynamic policy, regulatory and delivery landscape

that is subject to both market forces and changes in government policy. This can have

an impact on: the types and design of qualifications; what drives education and training

providers that deliver qualifications; and who does what in the system.

The Education and Skills sector, in which Awarding Organisations work, is diverse.

For example, the range of providers include:

• Schools, colleges of further education and universities

• Public sector and private training providers

• Adult and community learning providers

• Employers of different sizes (those with internal training departments and

those without).

The sector caters for learners of different ages, abilities, personal circumstances,

experiences, motivations and aspirations. It employs teachers and trainers with different

backgrounds, qualifications and varying experiences in industry and teaching expertise.

There is a diversity of potential learning environments beyond the traditional classroom.

Successive government reforms of vocational education and training have had

implications, not only for the qualifications being developed, but for organisations

responsible for their development, approval and quality assurance. For example,

some reforms have led to the closure, merger or repositioning of some key organisations,

and establishment of new bodies. In some cases, the competitive market has been

reduced through, for example, sole licensing arrangements. In other cases, markets

are opening up.

See page 14 for a discussion of some of the challenges faced by Awarding Organisations.

10 Introduction

Some key organisations concerned with qualifications

Examples of key organisations in England:

• The Department for Education (DfE) has overall responsibility for wholly,

or partly, publicly-funded education, apprenticeships and skills. This includes

approving regulated qualifications for students under the age of 19 (Section

96 of Learning and Skills Act, 20001)

• The DfE also sets the policy parameters within which the new Institute for

Apprenticeships operates. The Institute approves Apprenticeship Assessment

Organisations. From 2018 it will manage Level 2 and 3 technical certificates

and maintain a register of Level 4 and 5 technical qualifications which are

eligible for public subsidy through the Education and Skills Funding Agency

• The Education and Skills Funding Agency (ESFA), a DfE-sponsored executive

agency, is accountable for funding education and training for children, young

people and adults. It regulates academies, FE Colleges, employers and training

providers receiving public funding, and manages the Register of Approved

Apprentice Assessment Organisations2.

• The Office of Qualifications and Examinations Regulation (Ofqual)

recognises Awarding Organisations that are required, or wish, to have their

non-degree qualifications regulated.

1 Section 96 of the Learning and Skills Act 2000 gives the Secretary of State power to approve qualifications suitable for delivery to young people aged less than 19 years old according to the following age categories: pre-16; 16-18; 18+. Public funding is considered for qualifications which have been section 96 approved. Available at: https://www.gov.uk/guidance/section-96-qualifications

2 The Register of Apprenticeship Assessment Organisations (RoAAOs) is a list of organisations that have been assessed as being suitable to conduct end-point assessments of apprentices and be in receipt of public funds. Further information available at: https://www.gov.uk/government/collections/register-of-apprentice-assessment-organisations

11Introduction

Examples of key organisations in the devolved administrations:

• In Wales, the Welsh Government, through the Department for Education and

Skills, has overall responsibility for qualifications policy and for determining

qualifications priorities in Wales

• Qualifications Wales, established through the Qualifications Wales Act 2015,

regulates non-degree qualifications and the qualifications system in Wales.

It is sponsored by the Welsh Government and accountable to the National

Assembly for Wales, but is independent of government. A strategic authority

provides regulatory oversight of post-compulsory education

• In Northern Ireland, the Department of Education is responsible for the

curriculum and assessment in schools and 14-19 policy. The Department

for the Economy is responsible for skills and employment, including

apprenticeships and further education. The Council for the Curriculum,

Examinations and Assessment (CCEA) acts on their behalf on a day-to-day

basis, with its regulation arm (CCEA Regulation) regulating qualifications

• In Scotland, the Scottish Qualifications Authority (SQA) reports to the

Cabinet Secretary for Education and Skills who is also responsible for Skills

Development Scotland (SDA). SDA supports people and businesses to develop

and apply their skills. SQA Accreditation regulates qualifications other

than degrees, and approves and quality assures Awarding Bodies that offer

qualifications in Scotland.

SQA and CCEA both have awarding functions separate to their regulatory

responsibilities.

Sector Skills Councils and Sector Skills Bodies work with employers across the UK

to define skills needs and standards in their industry, although their activity has been

reducing as a result, for example, of reforms to apprenticeships in England.

12 Introduction

Awarding Organisations

Awarding Organisations design, develop and quality assure the delivery and assessment

of qualifications. They support users of their qualifications and issue certificates to

formally recognise individual achievements.

The functions of Awarding Organisations include:

• Developing high quality qualifications that meet the needs of employers,

other stakeholders such as HE, and learners

• Approving centres/third parties and working with them to ensure high quality

delivery of qualifications, or dealing directly with learners, e.g. professional

bodies (offering exams linked to career progression)

• Carrying out activity designed to assure the quality and validity of the

qualifications awarded

• Developing innovative products and services to support their centres/third

parties and learners.

Organisations that award vocational qualifications, vary in size and governance. They can

be Registered Charities, Chartered Institutes, commercial businesses, employers who

have established their own awarding body, small sector-specific or large cross-sectoral

bodies. An increasing number of qualifications are now designed by Professional Bodies.

In England, National Colleges3 may also award some Level 4 and 5 qualifications.

Not all qualifications are regulated. Awarding Organisations may seek to be regulated

because it is a requirement for their qualifications to be used in particular contexts, for

example schools or Colleges of Further Education. They may choose to be regulated

because it signals quality to users and offers a shop window to attract potential

customers.

For 2015-2016, there were over 150 Awarding Organisations in England, Wales and

Northern Ireland recognised by Ofqual (2017: 254).

3 National Colleges have been funded by government in England to deliver high-level technical training at levels 4-6 for the high speed rail, nuclear, on shore oil and gas, digital skills, creative and cultural industries

4 Ofqual (2017), Statistical Release. Annual Qualifications Market Report - England, Wales and Northern Ireland. Academic Year 2015/16. Available at: https://www.gov.uk/government/statistics/annual-qualifications-market-report-academic-year-2015-to-2016

13Introduction

Qualification frameworks

• The Regulated Qualifications Framework (RQF) is managed by the office of

Qualifications and Examinations Regulations (Ofqual) and shared with Northern

Ireland.

• The Credit and Qualifications Framework for Wales (CQFW) is managed by

a strategic operational partnership comprising the Welsh Government, Higher

Education Funding Council for Wales (HEFCW) and Qualifications Wales.

• The Scottish Credit and Qualifications Framework Partnership manages the

Scottish Credit and Qualifications Framework (SCQF). Partnership members include

the Quality Assurance Agency for Higher Education, College Development Network,

the Scottish Qualifications Authority and Universities Scotland.

Within these frameworks, regulated qualifications are classified according to type, level

and size. Level descriptors outline the general outcomes of learning. This allows broad

comparisons to be made between qualifications in the UK and those in Europe that are

classified according to the European Qualifications Framework (EQF)5.

Such qualification frameworks are shaped by:

…broader developments, including the internationalisation of education, changing models of regulation and accountability, demands for greater transparency in educational standards and outcomes, and advances in technology and digitalisation. (O’Connor, in Coles, 2016: Foreword6)

Quality Assurance

The regulatory authorities publish Conditions of Recognition (or directives in Scotland)

with which Awarding Organisations must comply if their qualifications are to be included

in the respective registers of regulated qualifications.

For many Awarding Organisations, recognition by one or more of the above Regulators

will provide the appropriate indicator of quality. For those interested in qualifications

offered within the Higher Education sector, quality may be indicated by the Quality

Assurance Agency (QAA) operating in its own right or, from 2018/19, as a co-regulator

under the direction of the Office for Students (OfS).

In England, there are four options for external quality assurance of Approved Assessment

Organisations conducting End-Point-Assessments (EPAs)7in apprenticeships. These

External Quality Assurance (EQA) bodies may be:

• An employer group with its own approach

• Ofqual

• A professional institution or

• The Institute for Apprenticeships (via a contracted partner).

The QAA/OFS regulates degree apprenticeships.

5 The EQF covers all types of qualifications from Level 1 to Level 8 in the RQF and Level 12 in the SCQF. Available at: http://www.accreditedqualifications.org.uk/european-qualifications-framework-eqf.html

6 Coles, M. (2016) National Qualifications Frameworks Reflections and Trajectories. Qualifications Policy Insights. Quality and Qualifications Ireland7 Each apprenticeship standard in England has an employer designed, independent End-Point Assessment (EPA). The EPA assesses the Knowledge, Skills and

Behaviours (KSBs) that have been developed throughout the apprenticeship. The nature and approach to assessment varies across standards, the detail being set out explicitly in each assessment plan

14 Introduction

Some challenges being faced by Awarding Organisations

There are various challenges that affect how Awarding Organisations operate and

what qualifications they offer, and where.

These include those relating to:

• Government policy developments

• Government funding

• Legal requirements

• Divergence between developments in England and those in

Wales, Northern Ireland and Scotland

• Opportunities for extending work into international markets.

Figure 1 will give you a flavour of some policy developments in England that are

influencing market behaviour at the time of writing.

Reform of technical education Two pathways (technical and academic)New technical education qualifications for 14-16 pupils and post-16 college learners/apprentices

New bodiesEducation and Skills Funding Agency (ESFA) (merger of SFA and EFA)Institute for Apprenticeships (IfA)Office for Students (OfS) (merger of HEFCE and OFFA from April 2018)

Performance tablesNew rules for inclusion of publicly-funded qualifications

Functional skills reform

New classification of occupations Framework for use in reformed apprenticeships and Technical Education - 15 groups of occupa-tions (routes)

Reform of apprenticeshipsNew funding arrangements (UK-wide employer levy)Replacement of frameworks with standards‘Basket’ of qualifications replaced by End-point assessments (EPAs) EPAs delivered by Approved Apprenticeship Assessment Organisations

Key PolicyDevelopments

Figure 1: Some examples of policy developments in England

15Introduction

You can explore these developments by accessing documents via the FAB website

(see chapter end) and/or using the links in Appendix B.

Just to pick out some issues relating to a few of these policy areas:

Performance tables

As a result of recommendations from the Wolf Review8of vocational education, the

number of qualifications attracting performance points in English schools and colleges

has been reduced.

To qualify, qualifications need a range of characteristics, for example, in terms of size and

synoptic assessment. They must be submitted for regulation by Ofqual and approved

by the DfE. This takes time and can lead to delays in informing centres about the

qualifications that will count, limiting the opportunity to plan provision.

Government funding

Learners, of course, can fund their own study towards a qualification. Some Awarding

Organisations operate in a context where government funding is not of relevance to

them, for example, professional bodies that embed their qualifications into career

profession routes.

However, where public funding is used to support particular government policies, with

certain age groups, this can impact on Awarding Organisations in a number of ways.

For example:

• Along with funding comes rules that education and training providers must satisfy to

offer a qualification within their particular context

• Funding can affect customer behaviour, their choice of qualifications and, ultimately,

the financial viability of the qualification offer

• Risks of malpractice may increase where release of funding depends upon learners

achieving a qualification. There may also be more appeals against results, which will

have cost implications for Awarding Organisations.

8 Department for Business, Innovation & Skills and Department for Education (3 March 2011) Review of vocational education – the Wolf report.

16 Introduction

Legal requirements

The legal requirements of Awarding Organisations and qualification regulation is a

specialist area of Law. There can be serious consequences for non-adherence, including

costs in arguing a case. It is important, therefore, that Awarding Organisation staff

understand how Laws relate to their work and how to protect the reputation and

interests of their Awarding Organisation.

UK laws99 of particular importance include those relating to the:

• Equalities Act 2010: Awarding Organisations must ensure accessibility in design and

monitor qualifications for any feature which could disadvantage a group of learners

who share a particular characteristic, such as: age; disability; gender reassignment;

race; religion or belief; sex and sexual orientation

• Data Protection Act 1998: Awarding Organisations (as data controllers) must ensure

that any processing of personal data for which they are responsible comply with this

Act

• General Data Protection Regulation (GDPR) will have direct effect in UK Law from 25

May 2018

• Competition Act 1998: Awarding Organisations are prohibited from engaging in anti-

competitive agreements with other businesses and abusing a dominant position in a

market.

Check for updates to these laws and regulations at: http://www.legislation.gov.uk/.

9 Equalities Act: http://www.legislation.gov.uk/ukpga/2010/15/contents, a Data Protection Act: https://www.gov.uk/data-protection/the-data-protection-act Currently, the UK government is calling for views on the GDPR: https://www.gov.uk/government/consultations/general-data-protection-regulation-call-for-views Competition Act: http://www.legislation.gov.uk/ukpga/1998/41/contents

17Introduction

Divergence between England and the devolved administrations

Scotland has long operated its own system and types of qualifications, but Wales and

Northern Ireland have historically shared some qualifications and their design features

with England. With the devolved administrations having more direct responsibilities for

qualifications and apprenticeships, there is more scope for divergence from how

things are run in England.

For example:

• Apprenticeships in England have been reformed, following the Richard Review

(2012)10. The framework approach (a ‘basket’ of qualifications) has been replaced by

an employer-led Standards approach, with End-point Assessments (EPAs)

• Although the Apprenticeship Levy11 is UK wide, Wales, Northern Ireland and

Scotland have retained use of National Occupational Standards (NOS)12 and

Frameworks. This raises issues for those operating across the UK such as employers

who may prefer one apprenticeship for their sector

• In England, a framework of 15 routes to skilled occupations is being used in the

reform of technical education qualifications13 and apprenticeships. This is instead of

using the Standard Occupational Classification (SOC)14 system that has historically

informed NOS. Within the technical pathway, one qualification will be approved from

one approved Awarding Organisation for each occupation, or cluster of occupations,

in each of these routes (see Figure 2).

Awarding Organisations may find they need to develop specific products for specific

markets in different parts of the UK. For example, Wales is developing its own approach

to regulation, prioritising qualifications for use in Wales and restricting the number in the

market through competitive tendering (initially for the Health & Social Care sector).

Different sets of criteria for similar products may be found such as those for key skills,

functional skills and essential skills15. Many Awarding Organisations are not in this

market. For those that are, such skills can represent quite a large market share.

10 Department for Business, Innovation & Skills (27 November 2012) The Richard Review of Apprenticeships11 Department for Education (October 2016) Apprenticeship Funding in England from May 2017: https://www.gov.uk/government/uploads/system/uploads/

attachment_data/file/562401/Apprenticeship_funding_from_May_2017.pdf12 For further information on National Occupational Standards: https://www.gov.uk/government/publications/national-occupational-standards13 For further information: Department for Business, Innovation & Skills/ Department for Education (July 2016) Post 16 Skills Plan.The Report of the

Independent Panel on Technical Education (April 2016)14 For further information on Standard Occupational Classification: https://www.ons.gov.uk/methodology/classificationsandstandards/

standardoccupationalclassificationsoc15 Wales, Northern Ireland and England have, in the past, shared specifications for Key Skills qualifications. Scotland has had its own Core Skills units (still

being used). In England, Key Skills have been largely replaced at Entry level to Level 2 by Functional skills qualifications in English, maths and ICT. Northern Ireland still offers Key Skills qualifications, but also has Essential skills qualifications in Literacy and Numeracy at Entry Level and Communication and Application of Number at Levels 1 and 2. Wales has a single suite of Essential Skills from Entry level 1 to Level 3

18 Introduction

Agriculture,Environmental and

Animal Care

Agriculture,Environmental and

Animal Care

Business andAdministrative

Business andAdministrative

Catering andHospitality

Catering andHospitality

Childcare andEducation

Childcare andEducation

454,726employed

2,204,478employed

568,998employed

1,060,804employed

Park Ranger

Conservationist

Agricultural Technician

Horticulturalist

Farmer

Office Manager

Human ResourcesOfficer

Administrative Officer

Housing Officer

Chef

Events Manager

Catering Manager

Butcher

Baker

Nursery Assistant

Early Years Officer

Teachning Assistant

Youth Worker

Protective ServicesProtective ServicesSales, Marketingand ProcurementSales, Marketingand Procurement

Social CareSocial Care Transport andLogistics

Transport andLogistics

398,400employed

957,185employed

865,941employed

589,509employed

Police Officer

Non-commissioned Officer

Maritime OperationsOfficer

Fire Service Officer

Buyer

Procurement Officer

Sales Account Manager

Market Research Analyst

Estate Agent

Careworker

Residential Warden

Welfare Counsellor

Probation Officer

Home Carer

Ship’s Officer

Railway SignallingTechnician

HGV Driver

ConstructionConstructionCreative &

DesignCreative &

DesignDigitalDigital

Engineering and Manufacturing

Engineering and Manufacturing

1,625,448employed

529,573employed

351,649employed

1,319,645employed

Bricklayer

Electrician

Building/Civil Engineering Technician

Construction Supervisor

Carpenter/Joiner

Arts Producer

Graphic Designer

Audio-visual technician

Upholsterer

Journalist

Programmer

Software Developer

Network Admin.

Web Designer

IT Technician

Vehicle Mechanic

Engineering Technician

Energy Plant Operative

Process Technician

Aircraft Fitter

Hair & BeautyHair & Beauty Health & ScienceHealth & ScienceLegal, Finance &

AccountingLegal, Finance &

Accounting

293,004employed

915,979employed

1,325,482employed

Dental Nurse

Laboratory Technician

Pharmaceutical Technician

Sports Therapist

Nursing Assistant

Hair Dresser

Beauty Therapist

Barber

Paralegal

Accounting Technician

Financial AccountManager

Legal Secretary

Payroll Manager

We expect these routes to be delivered

primarily throughapprenticeships

Key

Examples of occupationsto which each route

could lead

Number employed inoccupations within route,excluding graduate roles

Figure 2: Information from FAB webinar on Post-16 Skills Plan (information presented by DfE, 9 December 2016) http://www.awarding.org.uk/webinars/past-webinars/item/post-16-skills-plan

19Introduction

International markets

In 2013, education was identified as one of the key industries that the Department for

Business, Innovation and Skills (BIS) wished to work with in developing an investment

and growth strategy. There was a perceived capacity to grow the international business

in higher and further education, skills development, international schools and related

products and services, including qualifications.

Some Awarding Organisations have conducted a significant amount of business in

international markets and have, over a number of years, established a reputation for

their products and services. In recent years, it has become increasingly popular for other

Awarding Organisations to consider the opportunities that are available.

In emerging and developing economies, the ability to buy-in ready-built Technical and

Vocational Education (TVET) is often attractive. For others, a customised award approach

is found to be more appropriate. Currently, there is a focus on exporting Awarding

Organisations’ products and services in the Middle East and North Africa Area (MENA)

and China, but increased interest is coming from India and South East Asia.

However, although international work can be rewarding, competition is getting stiffer

and is not without risk. For example, there may be a tension when the client wants the

benchmarking or (regulatory) features of the qualifications, but also wants to make

significant changes. This may meet their needs, but call into question the status of the

changed product and the claims that can be made about it back in the UK. Be clear about

compliance obligations. Also, seek advice on what it is like on the ground when there is

likely to be in-country work. To succeed in this market, there is a need to be proactive,

agile, vigilant and connected.

Several British Government support agencies assist Awarding Organisations in

working overseas (See Appendix B).

20 Introduction

Further information

FAB offers events, training and webinars on topics of relevance to this section of the

handbook, including an introduction to the awarding sector and researching international

markets. Check the FAB website for forthcoming events.

Examples of information available from FAB library: www.awarding.org.uk

• Brown, D., Harris, M. and Fletcher, T. (2011) Reforming the Skills System: lessons learned the hard way, London/Leicester: Institute of Directors/CFE

• Coles, M. (2016) National Qualifications Frameworks. Reflections and Trajectories.

Qualifications Policy Insights. Quality and Qualifications, Ireland

• Department for Business, Innovation & Skills and Department for Education

(3 March 2011) Review of vocational education – the Wolf report

• Department for Education (February 2015) Review of vocational education, 2011.The Wolf Report: recommendations final progress report

• Department for Business, Innovation & Skills/ Department for Education (July

2016) Post 16 Skills Plan

• Department for Business, Innovation & Skills/ Department for Education (April

2016) The Report of the Independent Panel on Technical Education (known as the

Sainsbury Review)

• Department for Business, Innovation & Skills (27 November 2012) The Richard Review of Apprenticeships

• FAB International Guide (August 2015)

• Institute for Apprenticeships (February 2011) Driving the quality of apprenticeships in England

• Quality Assurance Agency (May 2017) Quality assuring higher education in apprenticeships, current approaches

For descriptions of key organisations, including those that support international work,

and further references and information, see Appendix B.

21

Chapter 1

So, you want to develop a qualification?

22So, you want to develop a qualification?

1.1 Overview of the qualification lifecycle

This chapter describes the typical lifecycle of a qualification and what is involved in

project managing the different stages of the cycle.

Key points from the chapter

• Qualification design and development is a creative, systematic and

collaborative process

• The qualification lifecycle has 4 main stages: design; development; delivery

and maintenance; review and evaluation

• But, revisiting stages is likely to be necessary to refine the qualification in

response to inputs and feedback from a variety of stakeholders involved in its

development and implementation

• Successful implementation of the cycle is dependent upon having the

necessary knowledge, skills and understanding, and effective project

management of both product development and the delivery service

• A systematic approach to the management of qualification design,

development and delivery helps to ensure: the objectives are clearly defined

and understood by all parties; a realistic timeline and budget; a logical

approach to planning; consistent means of monitoring and control for quality

and the management of risks; review and evaluation are built into the process

• Thought needs to be given from the start to the data to be collected on the

performance of the qualification, including evidence for validity, to inform

evaluation and quality improvements.

23 So, you want to develop a qualification?

The qualification lifecycle has four main stages:

1 Qualification design

2 Qualification development

3 Qualification delivery and maintenance

4 Review and evaluation.

Figure 3 offers an overview of key tasks relating to each stage of the lifecycle,

reflecting content of the chapters in this handbook.

However, it is acknowledged that qualification design and development is a creative

and collaborative process. It will be reliant on consultation with, and the input and

feedback from, a variety of stakeholders at different stages of the lifecycle. There may be

conflicting ideas and demands which will need to be resolved. This is likely to necessitate

the revisiting of earlier stages to refine, for example, product design, content and shape.

The qualification will require review and evaluation to inform quality improvements and

decisions on revisions – to keep it relevant and up-to-date, for example, in response to

new technology, changing markets, competition, rules and regulations. At some point, it

will be necessary to make decisions on withdrawal of the qualification.

Your role and the extent of your responsibilities will depend upon the size and structure

of your Awarding Organisation. Even if you are not directly involved in all aspects of

qualification lifecycle, it is still important to understand the whole process and the

constraints under which your Awarding Organisation operates. This includes being

familiar with regulatory and market requirements. Successful implementation of the

cycle is also dependent upon good project management, including quality control.


Figure 3: The Qualification Lifecycle16

16 Informed by the qualification lifecycle diagram produced by Ofqual (March 2015): https://www.gov.uk/government/publications/ofquals-regulatory-strategy-statement

Prepare for delivery of a qualificationAssure compliance and submit to qualification register/s

Launch and publicise the qualificationEnsure quality assurance/control systems & procedures are in place

Establish and communicate the business caseIdentify the need and support for developing/revising the qualification

Research current and potential marketsEstimate costs, risks and business benefits

Present business case for approvalCommunicate with and engage stakeholders

Project Management of the cycle, including quality control

Pre-project thinking; project initiation stage; project specification & planning; monitoring and control; test, review & refine stage

• Review the qualification’s performance

• Evaluate the qualification, including its validity

• Use data/findings and feedback to inform quality improvements

• Shape the qualification and its progression

• Develop the assessment apparatus, including support materials

• Put it all together to inform delivery of the qualification (qualification specification)

• Establish the qualification’s objective and target level

• Specify what the qualification will measure (proficiency specification)

• Design the qualification’s assessment

• Provide support for qualification personnel and learners

• Mark and standardise• Maintain controls and security, deal

with malpractice• Evaluate candidates’ performance• Report results, deal with appeals,

award and certificateDel

iver

& m

ain

tain

R

evie

w &

Eva

luat

e Design

a Qu

alificatio

nD

evelop

a qu

alificatio

n

See Chapter 3See Chapter 6

See Chapter 5 See Chapter 4


1.2 Managing your qualification’s lifecycle

1.2.1 What a Qualifications Manager needs to know and be able to do

In a dynamic landscape, it will be important to engage in continuing professional

development to keep your skills and knowledge up-to-date.

Key features of your role are likely to include:

• Representing the interests of your Awarding Organisation, in line with its mission,

values, vision and business objectives

• Abiding by its policies and quality systems

• Maintaining good internal relationships

• Developing and maintaining strong external relationships with stakeholders,

customers/centres and consultants/contractors.

In addition to expertise in the subject of the qualification, as a

Qualification Manager, you will need to have a good understanding of:

• The awarding sector landscape and market competition

• Qualification frameworks and regulation

• Legislation relating to qualifications, including Equalities Law

• Policy developments in the sector

• Government funding and how this can be applied

• The context(s) in which qualifications are delivered.

It is particularly important that you understand:

• What makes a qualification sufficiently valid

• The importance of validation planning

• The principles and ethics of assessment

• The different forms of assessment and their suitability

• Also, good practice in developing a qualification specification.


In executing your responsibilities, you will need to apply a range of skills and qualities,

for example:

• Personal and inter-personal qualities, customer handling skills

• Leadership and teamwork skills, e.g. the ability to put a team together,

persuade and motivate others, build relationships, delegate

• Strategic thinking, planning and organisational skills, with ability to manage

priorities and workloads, operate under pressure within tight deadlines

• Written and oral communication skills, e.g. active listening, presentation skills,

the ability to adapt communication to suit different audiences and settings,

write and disseminate reports

• Negotiation and influencing skills, e.g. to gain agreement and commitment

from others, to deal with any conflicting demands of stakeholders

• Information and communication technology skills, e.g. in creating and

maintaining documentation, and communicating with stakeholders

• Analytical, number and problem solving skills, e.g. in attending to detail,

providing solutions to complex technical problems, understanding/using

statistical data.

In addition to knowledge and skills relating specifically to developing qualifications,

you will need to be a strong project manager.

Good project management, defined as the: ‘application of processes, methods,

knowledge, skills and experience to achieve the project objectives’, will be key to

delivering a quality ‘product’ and ‘service’ (APM17).

To support the development and delivery of a qualification, there are a number of project

management software programs on the market. These enable the processing of large

amounts of data, and fast and accurate calculations, which allows for quicker responses

to changing circumstances. Such programs offer a structure to help you take a disciplined

approach – standardise methods for collecting and presenting information. It is relatively

easy to make back-ups to avoid the need for disaster recovery.

However, if not already established, there are resource implications, including time to put

such a system in place and train staff. While these programs can help, to make the system

work for you it is necessary to understand fully their functions and still be able to apply

knowledge of project management methods.

There is a well-established body of knowledge that underpins project management, and

its associated tools, skills and techniques.

For examples of project management resources and support for professional

development see Chapter 1 end.

17 Association for Project Management, The APM Body of Knowledge – Definitions. Available at: https://www.apm.org.uk/media/1605/final-proof-bok-6-definitions.pdf


1.2.2 Managing product design and development

A systematic approach to the management of qualification design and development helps to ensure:

• The objectives are clearly defined and understood by all parties

• A realistic timeline

• A logical approach to planning and estimating costs

• A consistent means of monitoring and control

• Review and evaluation is built into the cycle.

In your role as a project manager, you will typically engage in a process that

involves the following stages of qualification (‘product’) development.

Figure 4: Typical stages in project managing the development of a product (the qualification)

Pre-project

stage

Projectspecification

& planning

Monitoring &controlling

project stages

Test, review & refine stage

Project Initiation

stage


Figure 5: 5WH questions

Pre-project stageThe initiation stage of the project is most important to its success. However, it is

worthwhile spending time to do some preliminary thinking before initiating the project.

You may not be responsible for taking all the decisions, or providing all the necessary

information - seek clarification, if needed, as to the extent of your responsibilities and

accountabilities. But, the 5WH technique may help you to consider aspects of what is

involved in developing a qualification.

WIP

So you wantto develop a

qualification?

Why?

Where?

Who?

When?

What?

How?


do you want to develop a qualification?

There has to be a clear rationale and business case for the proposed qualification,

informed by market research that has identified both the need and likely demand for the

qualification, and the competition.

Reasonable estimates are required of the number of learners likely to take the

qualification. This is not only for satisfying the requirements of the qualification regulator

at a later stage, but to gain approval and the necessary resources from your Awarding

Organisation to go ahead with the project. This applies when proposing to re-develop an

existing qualification, as well as any new qualification. Resources should not be wasted on

a qualification that is not going to have a market.

will the qualification be made available?

Where (the context in which) the qualification is offered will have a bearing on the

regulations it will have to meet for its design.

Will the qualification be made available in England, and/or in one or more of the devolved

administrations of the UK? Is it the intention that the qualification will be for use in

compulsory education, post-compulsory education and training, professional and/or

other contexts? Perhaps, you want to develop a qualification for an international market.

will be involved with the qualification?

Key questions include:

• Who is the qualification for?

• Who needs to be consulted about its development?

• Who needs to be involved in developing the qualification?

Consider whether your qualification could be used by those under 19 years of age when

‘discharging their statutory duty to participate in education and training’, and thus

relevant for the purpose of the Education and Skills Act 2008. Will it be for 14-16 year

olds and/or young people aged 16-18 in education, those doing apprenticeships and/or

will it be for older learners in the workplace? Will the learners include those who need

the qualification to gain professional status or to operate in a safety-critical environment?

Evidence of support from employers and other potential users will be needed to confirm,

for example, that the qualification will provide a benefit to learners. Employers and other

stakeholders may also be directly involved in the design and development process.

You will also need to consider who to work with at different levels of your Awarding

Organisation. External people may need to be contracted, for example, to provide

technical expertise with regard to qualification content and design of its assessments.

There is likely to be a database of people who have been used previously to which you can

refer. FAB also has a list of consultants who may be able to help.

Why?

Where?

Who?


will the qualification be available?

When will your qualification go live?

Consider the time it may take to complete each stage of the design and development

process, and any issues that might delay hand-over of the product.

Take into account the time needed to test and refine measurement instruments, and

perhaps pilot your qualification, which will require feedback from stakeholders.

Submission to the relevant register/s of regulated qualifications could take longer than

you expect. For example, if your qualification is to be offered in Northern Ireland, then

it will need to go through CCEA Regulation’s accreditation process which increases the

timeline. Ofqual allows Awarding Organisations to upload qualifications to the RQF

without going through this process.

else needs to be considered?

What are the funding arrangements which will impact on your qualification’s take-up, for

example, where centres need to access public funding?

What are the regulations regarding the design of a qualification? What will be

the qualification’s level? If you are designing a qualification for inclusion on DfE’s

performance tables, you will need to take account of its guidance which specifies the

characteristics of such qualifications.

What are you expected to deliver? What will be the interim deliverables as the project

progresses, for example, draft specifications for consultation, reports? What quality

standards need to be met?

At this stage, you will not be able to identify exact costs and resource needs, but you may

have some idea of what budget and resources may be needed to produce the ‘product’

and support materials. What have similar qualifications cost to develop?

What are the internal and external constraints within which you have to work? What

are the risks associated with developing the qualification? It may be helpful to draw on

lessons learned from similar projects to think about what problems might arise and what

might be done to ameliorate their impact or likelihood.

How?

will the qualification be designed and developed?

How will the qualification be designed and developed – the phases of the project and

work tasks involved? How will you communicate your plan to interested parties? How will

you gain support and consult stakeholders? How will the product be reviewed, refined

and evaluated?

Check through the chapters on how to design and develop a qualification to inform your

thinking.

When?

What?

For a description of how to design a qualification, see Chapter 3.

For a description of how to develop a qualification, see Chapter 4.


If you have thought through the above, you will now be well-placed to move to the

project initiation stage.

Project initiation stage

The initiation stage of the project involves:

• Setting the objectives

• Defining the scope of the project

• Establishing the strategy.

If you have done some prior thinking, this stage should be relatively straightforward,

but do not be tempted to pay less attention to this stage as it will be critical to gaining

approval to design and develop a qualification - informing decisions about its viability.

Setting objectives

Project objectives should be stated clearly, and be measurable and achievable.

If you are to secure project approval and the support of your organisation, it will be

particularly important to ensure that the objectives of the project are shown to fit with

your Awarding Organisation business objectives. This may, however, require some

compromise.

In looking at the relationship between your project objectives and those of the business,

some degree of trade-off may be required, for example, in relation to time, cost and

quality.

If there are inconsistencies between project and business objectives, what are the

priorities?

Figure 6: Eternal triangle of project management

Objectives

QualityCost

Time


Defining the scope of the project

Use of the 5WH technique at the pre-project stage will have helped you to consider the

scope of the project, but you will need to reflect on your initial thinking and firm up on:

• The focus and objectives of the project

• The Awarding Organisation departments affected by the qualification design

and development work

• External people to be involved

• The project deliverables

• Phases of the work and timescale

• Likely resource and skill needs.

Now think about what might be outside the scope, and the constraints. Some

constraints will be external over which you have no control, for example, regulations that

qualifications must meet. Others will be set internally, and typically might have cost and

time limitations which will have an impact on the level of resourcing.

Establishing the strategy

Having defined the project’s objectives and scope, the next step is to agree on how to

take the project forward, including identifying levels of authority. ‘Who will decide what’,

is crucial to the smooth running of any project. Set out roles and responsibilities for both

project staff and any external people; who, for example, will provide information and/or

conduct specific pieces of the work?

Part of the strategy should be a consideration of risks that may limit, or prevent,

achievement of project objectives.

At this stage, this involves:

• Identifying the risks associated with the project

• Assessing the likelihood of these risks occurring

• Assessing impact on both your project and Awarding Orgaisation if these risks occur

• Identifying measures to help prevent them occurring

• Identifying contingency arrangements to ameliorate their effects.

Sources of further information about risk management methods and tools to assist in

this process, are provided at the end of this chapter.

Outcomes of the above activities will all help in establishing and communicating the

business case, and in informing how the project will be organised.


Project specification and planning stage

This stage involves:

• Defining the work breakdown structure (WBS) and outputs

• Planning the qualification design and development work

• Communicating the plan.

Defining the work breakdown structure (WBS)

Breaking down the project into manageable tasks is the key to you being able to plan,

monitor and control the qualification design and development work.

Your Awarding Organisation may have a WBS template to use, or examples that have

been used for previous qualifications, to use as a checklist. Typically, three or four levels

are sufficient.

Designing the Qualification

Establish your

qualification’s objective

and target level

Task

Task

Task

StatementProficiency

specification

Measurement

specification

Task

Task

Task

Task

Task

Task

Specify what your

qualification

will measure

Design your

qualification’s

assessment

Figure 7: Key work tasks for designing a qualification


You may need to involve other people who are familiar with, and have the detailed

knowledge of, what is involved, for example consult with experts in developing

assessment tasks.

The WBS will help in producing a project plan which will inform the allocation of work

and resources, as well as estimations of cost.

At this stage, you should begin writing a description of the end product (the qualification)

and skeleton descriptions of the component products to be refined as the qualification is

developed.

Figure 8: Key work tasks for developing a qualification

Task

Develop the Qualification

Components

Progression Map

Assessment tasks

Mark Scheme

QC/QA materials

Qualification

specification

Shape your

qualification

and its

progression

Develop the

assessment

apparatus

Put it all

together to

inform

delivery

Task

Task

Task

Task

Task

Task

Task

Task

Task


Planning the qualification design and development work

Planning involves focusing on results and how you will get there.

Designing and developing a qualification can be a fairly lengthy process, so it is useful

to define milestones as measures of progress towards the end product – a qualification

ready for launch and delivery. Ideally, this should be done in consultation with others to

be involved in the work.

It may be helpful to start with the end product and work backwards in identifying what

you need to achieve at each point before the next task can begin (the dependencies).

Recognise, however, that some tasks may need to run concurrently, or overlap, whereas

others need to be completed in a defined order.

By identifying the dependencies of preceding and succeeding tasks, it will be possible to:

• Determine the shortest time for completing qualification design and development

• Identify tasks which may present particular risks

• Pinpoint where slack may occur; and

• Establish a critical path through the project: the areas warranting most concentration

to help ensure the project is completed on time.

A good planning tool, such as a Gantt Chart, can then help you to schedule the project in

an appropriate format.

Much will still depend upon factors such as the skills, experience and availability of

personnel to undertake the tasks, time and other resource constraints. Success will also

depend upon effective management of a great deal of information and decisions on who

will need what information and when.

Outputs from detailed planning activities are likely to include:

• Project Plan - outlining the project’s objectives and how they will be achieved,

to provide the starting point for project control and decision-making steps to keep

the project on track

• Quality Plan - which sets out quality objectives, methods and procedures, and who

will be responsible for quality assurance and control; standards for deliverables

(project products), supervision and review; quality checkpoints

• Risk Management Plan - including risk identification, quantification and impact,

response development and risk control

• Communication Plan - the process to ensure proper collection and distribution of

project information and a schedule for meetings and progress reports.

It is also very important at this stage to plan for validation of your qualification.


Communicating the project plan

As a Qualifications Manager, you will need to develop lines of communication, and

maintain communication links, with managers and colleagues within your organisation

and external interested parties including stakeholders, contractors and customers. All

key people need to be involved to make clear, for example, who reports to whom.

During the course of the work, a variety of communications and tools will be used, but

you will need to record clearly all important agreements and instructions. The frequency

of reports should be agreed and document control exercised to ensure key documents

are sent out in a timely way to the right people. Some of the information may be very

sensitive and it will be up to you to determine who has access to it.

Monitoring and controlling the project stages

Control of your project is about authorising the work and monitoring progress in a

regular and disciplined way, to ensure that its objectives are met in terms of time, cost

and quality. It involves watching for changes, reviewing the situation and then taking

corrective action. It is about identifying problems, working out what needs to be done

and then doing it in a way that is helpful to project personnel.

Things can go wrong, but if you maintain a high degree of control you will be able to spot

potential problems early enough to do something about them. Project control starts with

the project plan and involves monitoring project performance against the plan.

It relies on capturing data on the progress and the status of the work activities being

carried out; although care needs to be taken to avoid information overload. It is not just a

matter of collecting information, you will need to analyse it to assess its implications and

inform decisions on any changes to be made.

Test, review and refine stage

A quality review to inform refinements can be initiated at any point. For example, you can

build feedback loops into the monitoring and control stage of the project to ensure, along

the way, that project deliverables meet the required quality standards.

There are various ways of expressing the quality improvement cycle. Figure 9 presents

one example.


In reviewing the project as a whole, checks will need to be made on:

• Performance against project objectives

• Performance against the project plan (including any revised versions)

• The effects of any internal and/or external events on the viability of the product –

the business case

• Lessons learned that can be applied to inform quality improvements in future.

Most products, including a qualification, need time in use before they can be fully

evaluated. Piloting your qualification, or at least its assessment components, can thus

be useful in testing out whether things are performing as they should, allowing time for

feedback from users and opportunity to resolve any problems before going live.

Figure 9: Quality improvement cycle

PlanEstablish objectives and

processes to achieve results

Stakeholderinputs

ProductRefined products

Stakeholderneeds/wants

Stakeholdersatisfaction

ContinuousQuality

Improvement

DoImplement plan, manage

resources and collect data

ReviewAnalyse data, draw out learning

and what needs to change

ActTake corrective action,

test for impact


1.2.3 Managing delivery of the qualification

While you will have had close involvement in the more discrete design and development

phases, the scale of the work in delivering your qualification means that management will

be reliant on a greater range of personnel. Many of these will be operating at a distance.

There will be others within your organisation who have responsibility for specific aspects

of delivery, their precise role(s) being dependent on your Awarding Organisation’s

size and structure.

The project management processes in which you engaged while managing your

qualification’s design and development, and associated techniques, are also relevant to

managing its delivery.

Use the 5WH method for thinking through delivery.

Why?

Keep upfront in your thinking your qualification’s objective and rationale.

Why learners and others might want to use/purchase it.

Where?

Refer back to your research on contexts in which you expect your

qualification to be used.

For example, to understand the opportunities and constraints under which deliverers

will be operating; where candidates will be working towards your qualification – in an

education or training context, in the workplace, through distance learning.

Who?

Who are the key people in your Awarding Organisation concerned with marketing,

compliance, administration and other functions?

Remind yourself about who the qualification is for, and think about their needs during

delivery.

• Who will be working directly with learners, assessing and evaluating their

performance?

• Who will be responsible for quality assurance?

Those to be involved will be influenced by whether your qualification is internally and/or

externally assessed and quality assurance requirements.

When?

You will need to be clear about the timeline for:

• Marketing, publicity and launch

• When the qualification will go live – operational start date –

and will need to be reviewed

• When candidates can take assessments – at set points in the year, on demand?

• Collecting feedback from users

• Publishing results and issuing certificates

• Providing the required information to the regulator/s.


What?

• What arrangements need to be made with centres, other third parties, publishers?

• What information and support will you need to provide for candidates, and for

personnel working directly with learners?

• What training and support will be required for your own staff, quality assurance

personnel, markers?

• What controls and security need to be put in place?

• What data will you need to provide for your qualification, e.g. for your Awarding

Organisation, the regulator/s, stakeholders?

• What systems will you need to collect and store these data to enable easy retrieval

and analysis?

• What are the risks associated with delivery of your qualification and what measures

will you take to prevent them occurring, or to ameliorate their effects?

How?

• How will your qualification be delivered – by your own professional body, through

centres, other parties – and quality assured?

• How will you communicate with all those involved?

• How will potential users be made aware of the availability of your qualification, gain

updates and relevant resources?

• How will you ensure equality of opportunity in terms of access to information and

resources? For example, will you put all presentations and training materials on your

Awarding Organisation’s website so that no-one is disadvantaged?

• How will you manage any conflicts of interest, appeals, and deal with malpractice or

maladministration?

• How will your qualification be reviewed and evaluated?

• How will you provide evidence of your qualification validity?

You do not have to have the answers to all these questions at this stage. The point of

them is to focus your mind on delivery. This will help to tease out the issues for attention

and identify where the pinch points may occur.

As a next step, it may be helpful to produce a process flow diagram for the

implementation and eventual awarding process for your qualification.

For fuller information on how to deliver a qualification, see Chapter 5.


Further information

FAB offers a range of professional development opportunities of relevance to

Qualification Managers, for example: group meetings and events; webinars; conferences;

training courses. Fab also has a list of consultants available to help on specific aspects of

your work.

Examples of information available from FAB: www.awarding.org.uk

Project management resources include links to the following:

• Resources, training and qualifications offered by The Association for Project

Management (APM), the Chartered body for the project profession

• Guidance and resources on projectmanagement.com, including a Gantt chart

template

• Online professional development courses, webinars and units for certification:

offered by The Project Management Institute (PMI)

• PRINCE2 (PRojects IN Controlled Environments) - a structured method and standard

for effective project management, e.g. available through APMG.

Guidance on risk management, including the following:

• Overview of definitions and risk management processes, and guidance on getting

started from the Institute of Risk Management (IRM).

Information on IT software project management systems, such as:

• Creatiogreen system, an Awarding Officer focused ‘workflow solution which

underpins governance, qualification development, centre management and learner

registration/certification processes’

• Quartz, an IT system that includes support for ’registrations, award, certification,

qualification development, centre approval and management. Compliance and quality

assurance, and automated work-flow management’

• Parnassus, a cloud-based management system for Awarding Organisations covering

all business functions from learner registrations to certification.

41

This chapter includes

9Discussion on what makes a qualification sufficiently valid

9 Advice on planning for validation

9Guiding principles for assessment

9 An overview of different forms of assessment.

Chapter 2

Understanding validity and assessment

42Understanding validity and assessment

There is a large body of research on validity and assessment, with concepts continually

evolving. Gaining an understanding for practical application purposes can thus take time.

Writings on the topic can use different terminology for the same (or virtually the same)

thing. Policy-makers, employers, the general public and learners may look at validity

differently (and have different purposes for doing so). Various constraints can also

impact on how concepts are applied in practice.

This chapter aims to steer a course through all of this in exploring some interpretations of

validity and principles for assessment to inform a pragmatic approach to the design and

development of your qualification.

To understand more fully the thinking underpinning concepts relating to validity and

assessment, it is recommended that you read the papers listed at chapter end,

available through FAB.


• A qualification should be fit for purpose. Its objective and content should meet the needs of its users

and results should be a trusted measure of what a person can do

• The key question is whether your qualification is sufficiently valid. Good assessment design involves

putting in place safeguards to avoid, as much as possible, loss of validity along the construction chain

(some loss is inevitable)

• Current thinking about validity that underpins qualification regulation is argument-based –

claims and inferences made about the qualification are tested throughout its lifecycle by

evaluating evidence to support the argument

• Validity is determined by all features and processes put in place to ensure results are as accurate and

useful as possible. Validity is about everything that supports good measurement

• Validation of a qualification needs to be built into the cycle and planned from the start, to identify

sources of evidence and the means to collect and analyse data to inform and justify the validity

argument

• To be fit for purpose, the qualification also needs to be: reliable (consistent in its measurement

and results); comparable (generate outcomes of comparable standards); manageable (feasible;

appropriate in demand on participants); minimise bias (adverse outcomes for learners sharing a

common attribute)

• The approach to assessment should be informed by research and consultation in relation to the

target proficiency. There should be a clear statement of its purpose, with level of demand being

consistent with this purpose. Account should be taken of the assessment’s practicability (in terms

of impact, administrative, time, resource and cost factors) and fairness (so that it is free from

unnecessary barriers to success and promotes equal opportunities)

• Some assessment methods are more appropriate to assessing cognitive domain, and some better for

assessing competence or skills-based domain; multiple indicators of performance tend to be used for

the latter

• Whether assessment is internal or external is a matter of who sets the task, who performs the

assessment and the nature of quality assurance and control.

43 Understanding validity and assessment

2.1 Whatmakesaqualificationsufficientlyvalid?

Qualifications are important because they are believed to:

… show clearly and publicly the knowledge, skills and attributes that an individual has gained, especially to inform prospective employers and future providers of education and training. The value of qualifications is therefore in very large part defined by how well they are understood by those to whom they may be presented as evidence. (Stasz, 2011: 8 18)

Results need to be a trusted measure of what a person knows and can do.

Trust in qualifications plays a crucial role for people. Qualification holders who have completed a programme and passed the required exams or assessments must inspire confidence that they have actually acquired the learning outcomes associated with the qualification. In this case their qualification has value and they can use it for employment, further education… (Cedefop, 2015: 1519)

Interpretations of validity

If you read academic articles, you will find that thinking about how best to conceptualise

validity has shifted over the years and between scholars.

For example, where it is understood as a unitary concept20, validity goes beyond a

concern about whether the qualification is ‘assessing what it is intended to assess’

to include a concern about ‘whether the inferences drawn from the results are well-

founded’. It is ‘an issue of professional responsibility rather than merely the concern of

test developers’ (Stobart and Gipps, 1997: 4221).

Current thinking underpinning regulatory exercises, designed to determine the extent

to which qualifications are valid, is argument-based. This means that there needs to be a

clear proposition that supports the proposed interpretation and intended use of results.

The interpretive argument is put to the test by evaluating the evidence collected at

various stages of the assessment process, to justify the validity argument.

The focus of attention on sources of evidence and analysis to support the validation

argument, is thought to better reflect the complexity and multi-faceted nature of the

concept than some other approaches to validity.

Perhaps more simply put, claims and inferences you want to be able to make for your

qualification’s assessments are tested out by scrutinising what happens to the robustness

of these claims at each stage of the assessment, development, implementation and

evaluation process. This is the understanding of validity adopted in this handbook. If you

wish to explore the various validity theories, and how they have developed over time,

some suggested reading is listed at chapter end.

18 Stasz, C. The Purposes and Validity of Vocational Qualifications. SKOPE Research Paper No. 105 November 2011, quoting Department for Children, Schools and Families (DCSF), 2010

19 Cedefop (2015). Ensuring the quality of certification in vocational education and training. Luxembourg: Publications Office. Cedefop research paper, No 5120 Validity as a unitary concept encapsulates different types of validity, for example: construct validity - the extent to which the assessment is likely to give an

adequate measure of the abilities(construct) being assessed; content validity - the extent to which the assessment covers the target proficiency or domain necessary to demonstrate good performance; predictive validity – the extent to which the assessment predicts accurately some future performance.

21 Stobart, G. and Gipps, C. (1997) Assessment. A teachers guide to the issues. 3rd Edition. London: Hodder & Stoughton


In combining the two arguments put forward by Kane (interpretation/ use argument –

IUA and validity argument), Newton (201322) describes Ofqual’s interpretation of the

validity of a particular qualification in figure 10.

Figure 10: What Ofqual means by validity (Newton, 2017: slide 723)

Validity is understood holistically in relation to the qualification’s measurement lifecycle

(see Figure 11).

The measurement lifecycle brings everything into play to ensure that results are as

accurate and useful as possible. The measurement construction chain adopts a step-by-

step approach: clarification; elicitation; evaluation; combination; interpretation. This

can serve as a framework to test out the interpretive argument for validity (what the

qualification purports to do) in order to build the evidence-informed validity argument

(what it actually does).

For example, it might be concluded that measurement interpretations will be both

accurate and useful:

• IF the target proficiency is properly specified, and

• IF evidence of that proficiency is faithfully elicited, and

• IF the elicited evidence is faithfully evaluated, and

• IF the appraisals are faithfully combined into results, and

• IF the results are faithfully interpreted.

(Newton, 2017)

22 Newton, P.E. (2013). Two kinds of argument? Journal of Educational Measurement, 50 (1), 105-109. Kane, M.T. e.g. (2001) Current concerns in validity theory. Journal of Educational Measurement, 38, 319-342

23 Newton, P. E. Validity -what it is and why it matters. Presentation at Ofqual Conference, 2017, The Vox Conference Centre, Marston Green, 28 February. Newton P.E. (2017) An approach to understanding validation argument, Ofqual

So this is what we mean by validity

■ The validity of a particular qualification is□ the degree to which it is possible to measure□ whatever that qualification needs to measure□ by implementing its assessment procedure.

■ The validity of a particular qualification is determined by□ ALL of the features and processes that are put in place to ensure that results are

as accurate as possible and as useful as possible.


Good assessment design involves establishing safeguards along the construction chain.

For example, by appointing staff with sufficient expertise, rigorous training, quality

assurance mechanisms for identifying and rectifying human error and so on. However,

there will always be some loss of validity.

For example, in moving from Step 1 to Step 2, the translation (or articulation) of

measurement objectives into a workable strategy and the design of the assessment tasks

are usually carried out by different people. This can pose a potential risk to validity in

terms of the extent to which faithfulness can be fully achieved due, for example, to lack

of clarity or misunderstanding of what is required.

Thekeyquestioniswhetheryourqualificationwillbesufficientlyvalid

The notion of sufficiency in validity acknowledges that trade-offs and compromises will

be required. For some steps, there may be small losses of validity which are reasonably

acceptable. But, it is particularly important not to lose a constant stream of validity along

the whole chain or have big losses in terms of qualification design and interpretation of

the result by the end-user.


Figure 11: The qualification lifecycle24 (as it relates to the qualification’s measurement)

24 Working diagram reproduced with permission from Paul Newton, Ofqual, 19 May 2017

ClarificationStep

ElicitationStep

EvaluationStep

CombinationStep

InterpretationStep

TargetProficiency

DesignOutputs

The measuring procedure is

designed

Specification ofmeasurement

situation, requirements

and aims

Specification of features/processes

which realise elicitation step

Micro-validation evidence, analysis

and argument

Macro-validation evidence, analysis

and argument

Evidence, analysis and argument related to

the wider acceptability of the measuring

procedure


which realise evaluation step


which realise combination step


which realise interpretation step

Assessment tasks

Task-specific QA/QC materials Task-specific

marking schemes

Session-specific standard setting/linking outcomes

DevelopmentOutputs

The apparatus for measuring candidates

are developed

Set of performances

Measurement result

Candidate certificateMeasurementinterpretation

USES AND IMPACTS

DeliveryOutputs

A measurement isdelivered for each

candidate

ReviewOutputs

The measuring procedure

is evaluated

Clarification Step

Measurement objectives are clarified

Elicitation Step

Multiple performances are elicited from each candidate (via assessment tasks) to provide a sample of evidence of proficiency

Evaluation Step

Each performance in the sample is evaluated in terms of what it implies about candidate proficiency

ConstructionChain

Key to steps in construction chain

Combination Step

The set of performance evaluations, for each candidate, is combined and transformed into an overall measurement result

Interpretation Step

Measurement results are interpreted by those for whom they have been provided

Set of evaluations


Reliability and comparability

In considering the validity of a qualification you will also need to look at the relationship

between validity and two other concepts: reliability and comparability.

Reliability is a necessary condition of validity. It is a measure of the consistency of the

results of qualifications and their assessments. It is generally held that an assessment

cannot be valid unless it is also reliable.

However, an assessment can be reliable without being valid. For example, a multiple-

choice test can be very reliable, but would not be valid if the aim of the assessment was

to test the ability to write continuous prose. To be valid, the assessment must test the

abilities it claims to test.

If a qualification has an examination, for example, reliability is about:

…making sure a candidate gets a fair result, irrespective of who marks the paper, what types of questions are used…, which topics are set or chosen to be answered on a particular year’s paper, and when the examination is taken. (Ofqual, May 2013: 125)

Of course, many qualifications are awarded on the basis of combining results from the

different assessments to give candidates the best chance to demonstrate what they

can do. This is where it can get more challenging, particularly where tasks are internally

assessed.

A degree of variability in results from one set of assessments to the next is inevitable - this is the measurement error, and distinct from operational errors (…which are not inevitable). [But] a composite score is not only related to the reliabilities of its components, it is also affected by the way the scores are combined and the extent to which the components themselves are correlated with each other. (Ofqual, May 2013:4)

For further discussion about the reliability of assessments, see Chapter end.

For further reading, see Appendix B.

Comparability also contributes to validity. It is concerned with generating assessment

outcomes that are comparable in standards, for example: between assessments within

a qualification; between similar qualifications; with other awarding organisations; over

time.

For the public, authentication of claims that the standards, in the ‘same’ qualification in different years, and in different qualifications which claim equivalence, is vital. (Oates, in Elliot, 2011:126 )

25 Ofqual (May 2013) Introduction to the concept of reliability26 Oates, T. In Elliott,G. (2011). A guide to comparability terminology and methods. Research Matters: A Cambridge Assessment Publication,

Special Issue 2,9–19.


There are public expectations that standards should be maintained over time. But,

there are various factors that can impact on the extent to which comparability can

be achieved. These factors include: time lapse between qualifications; differences in

assessment models or changes in qualification structure; availability of appropriate

measures of achievement; differences in size and demand of qualifications, content of

what is being taught and tested; the different purposes of qualifications. (summarised from Greatorex, 2001:327)

Ways in which comparability studies have been tackled include:

• Mode A where the objective is to investigate whether two qualifications are

equivalent (using measures of prior/concurrent/subsequent outcomes; comparing

candidates attempting both at the same time; expert judgement)

• Mode B where the objective is to describe similarities and differences between the

qualifications, not necessarily considered equivalent.

(Bell, in Greatorex, 2001).

Another method suggested for comparing qualifications is based on fitness for purpose

for progression into employment or further or higher education. It involves making

‘independent judgements of the relative value of qualifications and components, from the

perspective of those who use them in recruitment’ (Coles & Matthews, 199528).

There is not space in this handbook to go into great detail, but Elliot29 offers a useful

framework for understanding the arguments in the literature, including terminology

and methods. This may be of particular interest when it comes to evaluating your

qualification.

For further details of comparability methods, see Chapter 6.

Further considerations

For your qualification to be fit for purpose, alongside validity, reliability and

comparability, you will need to consider its manageability: the feasibility of carrying out

your qualification’s particular assessment processes. This will involve making a judgement

about the extent to which the demands are reasonable. You will need to balance the scale

of impact on the participants against the usefulness of the outcomes from the assessment

processes.

There will also be a need to minimise bias: ensure that your qualification’s assessment

does not produce unreasonably adverse outcomes for learners who share a common

attribute. This is related to fairness to all and statutory equality duties.

27 Greatorex, J. Can vocational A levels be meaningfully compared with other qualifications? A paper presented at the British Educational Research Association Conference, University of Leeds, UK, 13-15 September 2001

28 Coles, M. & Matthews, A. (December 1995) Fitness for purpose. A means of comparing qualifications. A report to Sir Ron Dearing29 Elliot, G. A guide to comparability terminology and methods. In Research Matters – Special Issue 2: Comparability, January 2011


Other concepts bigger than validity to consider include:

Acceptability – the strength of the argument in favour of providing a qualification

which focuses on ‘overall social responsibility’, and includes dimensions [such as]

‘legality, the availability of resources – financial, human, or otherwise; social and

educational policy alignment; moral fairness; public credibility and so on’.

(Busutti-Reynaud & Whitehouse, 14 November 201630)

Your qualification may have sufficient validity, with reliable assessments, but if its delivery

is too demanding, and/or costly, this will impact upon its take-up. If it does not align with

government policies, this may affect how potential customers view the relevance of the

qualification and could restrict access to public funding (if this was required).

30 Busutti-Reynaud, G. and Whitehouse, G. Basic Principles of Validity, presentation 14 November 2016, AlphaPlus Consultancy Ltd


2.2 Planning for validation

Awarding Organisations are responsible for the validity of their qualifications. You

should, therefore, be able to explain the systems and approaches you have used for

ensuring your qualification is sufficiently valid: whether it is testing the appropriate skills

and knowledge and is fit for its particular purpose(s). You should also be able to provide

evidence to justify your argument.

Validation is not only concerned with assessment results, but includes a focus on

assessment procedures and processes at each stage of the measurement lifecycle. It is

also reliant on the co-operation of stakeholders and trained staff to ensure the required

information is available for analysis.

This means that validation needs to be planned from the start.

There are two main approaches (at either end of a continuum) to looking at validation:

• Macro-validation: a broad, holistic evaluation relating to the overall validity claim,

i.e. that it is possible to measure what needs to be measured by implementing your

assessment procedure

• Micro-validation: a narrower, targeted evaluation relating to underpinning validity

claims, i.e. that each of the features and processes that comprise your assessment

procedure has been effectively designed to enable you to measure what needs to be

measured (Newton, 201631).

For further details of comparability methods, see chapter 6.

These approaches can generate different types of evidence to justify the validity of your

qualification and produce a robust validity argument.

For example:

• Evidence that you already hold, for example: qualification handbook; assessments

guide; documentation of quality assurance processes

• Evidence that you could collect relatively easily, for example: documentary evidence

of stakeholder support for assessment method; item-level data for externally-set and

marked tests; sample assessments and mark schemes; copies of visit/moderation/

verification reports

• Evidence that you may find more difficult or expensive to routinely collect,

for example: formal data to evidence marker reliability for internally marked

assessments; quantitative and qualitative evidence of how assessments and results

are perceived to be accurate and appropriate by stakeholders; evidence of how useful

results from assessments are in predicting future performance in the workplace.

(AlphaPlus, February 2014 :52)32

31 Newton, P.E. (2016) Macro and micro-validation: Beyond the ‘five sources’ framework for classifying validation evidence and analysis. Practical Assessment, Research and Evaluation, 21 (12)

32 AlphaPlus (February 2014) Validation of vocational qualifications. Commissioned by Ofqual


Some aspects of validation may require a specialist partner. At the planning stage, you will therefore need to

make a judgement about the extent to which you wish, and can afford, to modify current processes to routinely

gather additional validity evidence.

An important consideration concerns the collection, storage and analysis of data to inform your validation

argument. You will need processes, procedures, systems and trained staff to ensure that the required

information is collected. Having a set of questions for each stage of the lifecycle may help you in planning

what evidence to collect, and when (see Table 1).

Table 1: Examples of considerations and potential evidence (from FAB training event facilitated by AlphaPlus)

Qualificationdesignconsiderations

Questions

• Is the purpose/use of the qualification clear?

• Is the required standard or proficiency clear?

• Is the need for the qualification clear?

• Are all important learning outcomes included?

• Can the (mis) use of the qualification be managed?

• Is the communication of the purpose and use clear?

Potential validation evidence

• Mapping to agreed standards

• Details of key stakeholders (organisation and individual)

• Evidence of user consultation

• Follow up research on uses and understanding.

Assessment procedure design process considerations

Questions

• Are formal credentials and experience required of developers?

• Is qualification-specific training required/ provided?

• Is a clear specification for (assessment) task development provided?

• How is each task quality assured?

• Is evidence of previous task performance fed-back to provide a closed-loop system?


• Record of assessor qualifications and currency of experience

• Records of delivery, attendance and performance at training events

• Public and private specification documents including rationale for design choices

• Quality assurance /review documents

• Records of assessment task research and seminars with assessment designers.

Assessment procedure design outcome considerations

Questions

• Is the assessment task practical to deliver?

• Does the assessment task produce appropriate evidence against which to make competence judgements?

• Can the candidate understand the assessment task?

• Is the task free from bias?

• Is it clear that the task has met the required quality standards?

• Are the indicators of competence (and incompetence) clear?


• Feedback (ad-hoc and formally elicited) from centres or third parties

• Sampled candidate output mapped back to qualification specification and standards

• Records of candidate consultation or trials

• Quality assurance review records

• Findings from longitudinal studies or expert user trials.


Table 1 continued: Examples of considerations and potential evidence (from FAB training event facilitated by AlphaPlus)

Assessment procedure delivery process considerations

Questions

• Are the assessment administration requirements clear?

• Are the administration requirements met?

• Does delivery require trained staff? If so, are they trained?

• Does any aspect of administration lead to unfair performance variation?

• Is the impact of unavoidable variation between test administration compensated for in marking and/or awarding?


• Documentation produced to consistent standards

• Records of centre/third party staff training

• Follow-up research on centre/third party performance variation

• Records of Special Considerations

• Analysis of performance of groups with protected

• characteristics

• Candidate surveys and interviews.

Assessment procedure outcome reporting considerations

Questions

• Is any weighting of sources of evidence in final outcome clear?

• Are borderline cases reviewed holistically?

• What level of detail is provided to users of the qualification outcome?

• Is information provided on the level of certainty regarding outcomes?

• Are standards over time monitored and reported?

• Are checks made on consistency of outcomes with parallel measures?


• Aggregation model ideally includes: rationale for design choices; past scripts and other materials from candidates

• Records from awarding reviews

• Records of marking quality assurance

• Records of using statistical measures of reliability and item performance

• Records of using statistical measures of standards over time

• Findings from demand studies and qualitative judgements of comparator qualifications.

Qualificationreviewconsiderations

Questions

• How quickly does the field change?

• How frequently do the stakeholders think the qualification should be reviewed?

• Are there any context changes affecting the qualification?

• Has stakeholder use and interpretation of the qualification changed?

• Are the volume or other commercial considerations driving a change?


• Record of the review of standards

• Findings from stakeholder consultations and surveys

• Record of comparison with competitor qualifications

• Findings from time-based analysis of cohorts.


2.3 Principles of assessment

Assessment is a powerful tool: it can shape curriculum, teaching and learning; it can affect how learners come to see themselves both as learners and in a more general sense as competent or not; through labelling and sorting learners (certificating and selecting) it affects how learners are viewed by others; it controls access to further education and high status careers…Given this, a consideration of ethical issues and equity is highly pertinent (Gipps, 200433)

Assessment needs to be:

• Valid in terms of coverage and measurement of the target proficiency

• Reliable in providing repeatable outcomes at different times and places for

candidates with comparable characteristics

• At the right level of demand in terms of the degree of challenge presented to the

learner

• Fair: promote equal opportunities and be free from unnecessary barriers to success

• Practicable in terms of the time, resource and cost demands of administration.

The aim is to have practicable and fair assessments at the right level of demand, with an

adequate level of reliability, without compromising their validity.

Validity is one of the most important properties of assessments and needs to be present in sufficient degree if their quality and fitness for purpose is to be guaranteed by awarding organisations. (Curcin et al, February 2014: 634)

An assessment is valid when it:

• Is appropriate for its purpose

• Has been designed to allow candidates to show that they have the required

knowledge, understanding and skills to meet the standards of the qualification

• Allows all assessors to make reliable assessment decisions

• Allows the interpretation and inferences to be made which can be drawn from

assessment outcomes to be meaningful and justifiable.

(SQA, March 2015: 635)

Reliability goes hand-in-hand with validity. An adequate level of reliability is important

because it is concerned with an assessment’s stability, consistency and precision.

Reliability is also about providing repeatableoutcomes for candidates with comparable

characteristics who are assessed at different times and in different places. High reliability

would mean broadly the same outcomes would result if the assessment process was

repeated.

33 Gipps, C. (2004), Beyond Testing: Towards a theory of educational assessment, Routledge Falmer 2nd Edition 34 Curcin, M., Boyle, A., May, T. and Rahman, Z. (February 2014) A validation framework for work-based observational assessment in vocational qualifications.

London: The City and Guilds of London Institute (commissioned by Ofqual)35 SQA (March 2015) Guide to Assessment for Centres offering Ofqual Regulated Qualifications. Glasgow: SQA


Reliability may not, however, be perfect as there are a number of sources of potential

unreliability which may or may not be easy to address. For example, one ‘marker might

be more or less lenient on particular questions than the next…markers may make clerical

mistakes…’. An Awarding Organisation can address this by ‘trying to write very clear mark

schemes, training markers and checking their work (a process called standardisation)’.

However, little can be done by an Awarding Organisation about variability in candidate

performance due, perhaps, to how they are feeling on the day. (Ofqual, 16 May 201336)

It is worth the effort of trying to address any reliability issues since this helps to ensure

confidence in the assessment process. Part of this effort involves choosing reliability

measures which are appropriate for the forms of assessment in question.

The body of work is growing on measuring reliability in vocational qualifications (e.g.

studies conducted by individual Awarding Organisations such as City & Guilds; Ofqual’s

Reliability Programme). But, you will find that most studies of reliability have been

concerned with general qualifications where examinations have been the predominant

form of assessment.

For many years, estimating the reliability of tests, has involved use of statistical models,

the oldest being Classical Test Theory (also known as True Score Theory). Other, more

modern approaches, include Cronach’s Alpha, Item Response Theory (IRT)

and Generalisability Theory37.

In developing tests, definitions and measures of reliability have commonly included:

• Test-re-test reliability, based on gaining the same marks if the test is repeated

• Mark-remark reliability, which looks at the agreement between markers

• Parallel forms/split-half reliability, looking at whether similar tests give the same

results.

Vocational qualifications can, of course, include examinations and tests that are

amenable to statistical measures of reliability, But, they typically use a range of other

forms of assessment (see section 2.4), each of which presents a challenge in terms of

achieving reliability.

Generally speaking, a portfolio approach to assessment remains most prevalent in which

candidates are required to accumulate naturally-occurring evidence, such as work

products and evidence drawn from professional discussions and observations. Thus, the

exact nature of evidence can vary between candidates.

Reliability in vocational qualifications has largely been seen as related to the specificity

of the assessment criteria and requirement for gathering sufficient evidence to enable

assessors to make reliable and consistent judgements about the candidate’s achievement

of the learning outcomes. The reliability of decisions is about inter-rater agreement

(agreement between assessors which is checked by an internal verifier and externally

verified). Mark-remark measures of reliability can also be used.

36 Ofqual (16 May 2013) Introduction to the concept of reliability37 For an overview of these theories, see: Ofqual (16 May 2013), Conceptualising and interpreting reliability: Summary. Available at: https://www.gov.uk/

government/publications/reliability-of-assessment-compendium/conceptualising-and-interpreting-reliability-summary


In vocational qualifications, assessment often includes observation of performance.

Watson (1994) suggests that in observation-based assessment of complex competences38

‘reliability is dependent, not only on adequate quality control to ensure consistency

across assessors’, but upon:

… sensible decisions about the range and number of observations of performance that are required to make a reliable judgement about competence. Watson, 1994: 16139

In discussing the reliability of coursework and teacher assessment, Stobart and Gipps

(199740) suggest that the ‘fact that performance is assessed on a number of occasions

(i.e. use of repeated measures) can be more reliable than a one-off test’.

A further consideration is the extent to which evidence of achievement is is captured,

interpreted and recorded faithfully (what Wiliam, in re-casting reliability as a small part

of the dependability of an assessment, calls ‘the fidelity of assessment’41).

So, in summary: if the necessary quality control and quality assurance procedures have

been implemented to ensure assessment criteria are applied consistently, and evidence

is sufficient and recorded faithfully, comparisons can then be attempted to establish

reliability across and between assessment interventions.

Level of demand and fairness in assessment

Measures are necessary to ensure that the level of demand - the degree of challenge

that the assessment presents for the learner - remains the same from year to year.

Demand should be the same: across all assessment options which are made available to

the learner; as any previous assessments; as assessment support materials like specimen

assessments, published in support of the qualification.

The aim is also to ensure that assessment promotes fairness and equity while maintaining

the qualification’s integrity. It needs to be free from unnecessary barriers or impediments

to success or influences detracting from this central purpose.

You must take equality, diversity and inclusion (EDI) seriously in both assessment design

and in practice. This includes thinking about access requirements and Reasonable

Adjustments to lessen or remove the effects of a substantial disadvantage.

For information about these requirements, see Chapter 4.

This need to ensure assessment is ethical and fair is backed up by law.

38 Complex competences are defined by Watson as the integration of skills, knowledge and attitudes in constructing a holistic view of a problem or situation often associated with higher level, professional work

39 Watson, A. Strategies for the Assessment of Competence. The Vocational Aspect of Education, 46:2, 155-165, 199440 Stobart, G. and Gipps, C. (1997), ibid, p. 4241 Wiliam, D. (1992) Some technical issues in assessment: a user’s guide. In British Journal of Curriculum and Assessment, 2, 3, 11-20


Box 1: The Equality Act 2010

The Equality Act 2010

The Equality Act 2010 provides a legal framework to protect the rights of

individuals and to advance equality of opportunity for all.

It ensures individuals are treated fairly by: removing barriers which people may

face due to their differences; eliminating discrimination; offering the same access

to employment opportunities.

The Act supports diversity by recognising and valuing people’s differences,

promotes and looks to safeguard inclusion by making sure no-one is excluded

from fulfilling their potential because of any of the following nine protected

characteristics: age; disability; gender reassignment; marriage and civil

partnership; pregnancy and maternity; race; religion or belief; sex; sexual

orientation.

The Act also aims to prevent all forms of discrimination, including: direct

discrimination; direct by perception; direct by association; discrimination

arising from disability; indirect discrimination; harassment and victimisation.

(Drawn from the Equality Act 201042)

Assessment has to:

• Be free from gender, ethnic, political, cultural, other discrimination, or stereotyping

• Use content, resources and assessment materials that recognise the achievements

and contributions of different groups

• Where appropriate, provide a balance of assessment methods and permit alternative

approaches.

Use of a range of assessment tasks involving a variety of contexts, a range of modes

within the assessment, and response format and style, is most likely to offer learners

alternative opportunities to demonstrate achievement if they are disadvantaged by any

one particular assessment.

42 Equalities Act, 2010. Available at: http://www.legislation.gov.uk/ukpga/2010/15/pdfs/ukpga_20100015_en.pdf Also, check for any amendments at: http://www.legislation.gov.uk/ukpga/2010/15/contents


Other principles of assessment

These may be more procedural, but nonetheless should be kept in mind.

They relate to the need to ensure assessment evidence is:

• Authentic

i.e. the evidence presented for assessment is the candidate’s own work

• Current

i.e. the work has been generated and presented within a reasonable timeframe,

making it still a relevant representation of ability

• Sufficient

i.e. the work presented is enough to meet the requirements of the assessment.

In considering validity and reliability of assessments, level of demand and their fairness

and equity, once again you need to keep an eye on the practicalities. All assessments

have to be practicable: manageable and proportionate for both the learner and

assessment provider. Practicality is useful to keep in mind because it looks at the

assessment holistically and includes the amount of impact the assessment makes, and the

administrative, time, resource and cost factors, helping to ensure that these factors do

not outweigh the benefits.

Summaries of the key principles for assessment practice are provided in the following

tables43. They include those which are pertinent to each stage of a qualification’s lifecycle:

design and development; delivery; review and evaluation.

43 Drawing on the work of Oates, T. (January 2009) The Cambridge Approach. Principles for designing, administering ad evaluating assessment. Cambridge: Cambridge Assessment; and other sources, such as AlphaPlus (2006) Assessment – Aspects of Principles and Practice


Box 2: Guiding principles of assessment design and development

Guiding principles of assessment: design and development

• The assessment’s rationale should be informed by research and consultation

to determine the target proficiency – the skills, knowledge and understanding

- and processes that comprise effective performance in the ‘subject area’

• There should be a clear statement of the assessment’s purpose – what it is

intended to assess and also appropriate uses of information and outcomes

from the assessment. The approach to the assessment, its form, grading (if

any) and reporting arrangements should be matched to this purpose

• The precision of the assessment should be set at a level which is consistent

with its purpose and its uses, and take into account limitations on precision

when estimating, documenting and reporting on reliability

• The structure of the assessment should reflect the sequencing of activities

that are authentic to the vocational sector served by the qualification, with

marks and credit given to significant aspects of performance. The assessment

must guard against leaving out important aspects of the target proficiency,

but also avoid unnecessary repetition of the same skills, knowledge and

understanding

• Coverage of the vocational subject content and processes must be sufficiently

representative to enable reasonable claims to be made about: a) what the

candidate knows, understands and can do at the point of assessment; and

b) the extent to which the candidate should be able to reproduce the skills,

knowledge and understanding in contexts beyond the assessment

• The assessment should be designed with sensitivity to the needs of specific

groups to avoid bias: unwanted effects that may result in unfairness -

differences in terms of access to the assessment and outcomes

• Support materials should be developed in formats that are accessible to

assessment administrators and learners, setting out: the rationale and

purpose of each assessment; procedures; security arrangements; and

regulations, e.g. for special arrangements and special considerations.

Specimen and exemplary materials should be well-matched to ‘live materials’.


Box 3: Guiding principles of assessment: delivery

Guiding principles of assessment: delivery

• Assessments should be administered in accordance with procedures, security

arrangements and regulations set out by the Awarding Organisation, and

include contingency plans for dealing with any incidents - security breaches,

irregularities or disruptions to the assessment

• Assessment should take place in an environment and set of circumstances

which are managed in way that does not compromise candidates’ capacity to

perform or pose a threat to validity

• Regulations for dealing with loss, absence or late arrival of evidence from

assessment should be adhered to, and quality control and quality assurance

processes for marking procedures

• Results should be published according to the schedule, with due regard for

candidate confidentiality

• Review and appeals procedures should be published in media that are readily

accessible to administrators and candidates, and conducted in accordance

with the stated regulations; candidates’ outcomes from assessment should be

kept until results are published and to allow time for review and appeals

• Clear guidance should be provided on legitimate uses of information from

assessments.

Box 4: Guiding principles of assessment: review and evaluation

Guiding principles of assessment: review and evaluation

• Formative evaluation should be used, including feedback from the various

stakeholders, to inform assessment design and development

• Evaluation of initial cycles of the assessment should inform its refinement,

with data from assessment and other sources used to ensure a precise, initial

fix of standards

• Data should be collected on the characteristics of candidates taking the

assessments and documentation on the characteristics of the assessments

including documents developed for quality assurance, to inform reviews and

audits

• Validation studies should be conducted, where necessary, to find out whether

the assessment is performing as expected, and scores derived from the

assessment reasonably accord to those from other assessments relating to

target proficiency.


2.4 Forms of assessment

There are a number of ways that assessments are categorised to differentiate between

different modes, including for example:

• Formative versus summative assessments

• Continuous versus terminal

• Norm-referenced versus criterion-referenced

• Internal versus external assessment.

Mention is made of these distinctions for completeness, although their usefulness in

practice may be questionable. As such, they should not distract you from the central

purpose of assessment. Whatever form of assessment chosen, it needs to be fit for

purpose: do the job it is selected to do because it is an appropriate way to do it.

Formative versus summative, continuous versus terminal

Formative assessment can be described as focusing on specific aspects of a target

proficiency or learning aim to let the candidate know how well they are progressing and

what they need to do, using the performance to inform the next stages of their learning.

Assessment conditions are relaxed and informal.

Summative assessment can be described as assessing the qualification’s key content, to

allow clear measurement of attainment or performance against the target proficiency.

This may involve use of multiple attainment indicators, i.e. more than one form of

assessment. Assessment conditions are formal.

In practice, the distinctions are not clear cut. For example, summative assessment could

be an examination at the end of a course of study (terminal), but it could take the form of

an end of module/unit test during the course, as part of continuous assessment. Results

may be used at the time for formative purposes.

Outcomes from formative assessment may subsequently contribute to summative

assessment. Information from either or both modes of assessment may be used for

diagnostic purposes: to identify areas of strength, gaps in learning, where problems are

being experienced, to inform how best to support learning towards a qualification.

Recognition of prior learning (RPL44) may be part of the assessment process. This can be

quite time consuming if it is done on a case-by-case basis. It is therefore helpful when

designing a qualification, to be clear about what parts might be best suited to the RPL

process.

44 Recognition of prior Learning (RPL) is a process that considers whether an individual can demonstrate that they can meet assessment requirements for Accreditation of Prior Learning (APL). RPL is appropriate for use where an individual’s learning has not been formally recognised, drawing, for example, on learning in education, work-related, community or voluntary activities. Candidates are asked to generate evidence for knowledge and skills they have already acquired and do not need to develop through a course of learning towards the target qualification.


Norm-referencing versus criterion-referencing

In addressing how different levels of attainment can be differentiated, common measures

on which judgements about standards are made are:

• Norm-referenced which depends upon relative standard - grades, or the pass-fail cut

off, are assigned on the basis of a fixed proportion of candidates gaining them, rather

than on a set of marks

Or

• Criterion-referenced which depends upon an absolute standard – what needs to be

done is specified, so candidates who meet these requirements will pass (whether or

not others do too).

Norm-referencing is useful when needing to rank successful candidates for selection

purposes. However, it is ‘easy to relate an individual’s performance to that of a norm

group without knowing what, exactly, the assessment is representing’ (Wiliam, 199745).

In contrast, criterion-referencing provides information about what learners can actually

do, although there has been criticism that the trend towards providing more and more

levels of supporting detail to help ensure consistency of interpretation, has led to over-

complexity. This is especially the case where there is grading with further supporting

detail.

Wiliam suggests that neither criterion-referenced assessment nor norm-referenced

assessment provides an adequate theoretical underpinning for authentic assessment

of performance. He proposes alternative ‘construct-referenced assessments in which

outcomes are interpreted by reference to a shared construct among a community of

assessors’. He argues that the ‘agreement between raters (i.e. intersubjectivity) can, in

many cases, be sufficiently good even for high-stakes assessments, although “classical

indices” of reliability are inappropriate’ (Wiliam, 1997:26).

For example, teachers can be trained to use appropriate standards for assessing

portfolios by the use of ‘agreement trials’. These trials involve marking different pieces

of work, with feedback from an expert who has also marked the work, until the teachers

demonstrate convergence on the correct marking standard.

45 Wiliam, D. Construct-referenced assessment of authentic tasks: alternatives to norms and criteria. Paper presented at the 7th Conference of the European Association for Research in Learning and Instruction, Athens, Greece August 26-30, 1997


Internal versus external assessment

Table 2 illustrates some typical assessment methods grouped according to whether

assessment is internally or externally controlled.

However, here it is important to remember that most forms of assessment can be

administered in ways that make them perform like internal assessment or external

assessment. The key differentiating factors making the assessment more internal or

external are to do with:

• Who sets the assessment task

• Who performs the assessment or marks the assessment task

• The nature of the quality assurance and quality control procedures.

For example, an assessment task can be externally set and externally marked by an

Awarding Organisation, delivered either on paper or electronically. Examples include: a

‘traditional’ test employing one or more question types, such as structured/short-answer

questions, multiple-choice or other objective questions.

It is also possible for an assessment to be externally set by an Awarding Organisation

or internally set by a centre, internally assessed and then externally verified. Forms of

assessment may be various, for example: tests of knowledge, practical assessments,

assignments.

Table 2: Typical assessment methods – internally and externally controlled

Internally-controlled methods Externally-controlled methods

• Locally set and marked tests

• Professional discussion with the tutor

• Role play with the tutor

• Observation by the tutor

• Skills tests administered locally (in the delivery centre)

• Projects and assignments, internally set and marked

• Learner-generated work, e.g. work-based products

• Reflective journals

• Case studies.

• Formal examinations and tests

• Professional discussion with an independent assessor

• Role play with an independent assessor

• Observation by an independent assessor

• Skills tests independently assessed

• Projects and assignments, completed under controlled conditions and externally marked.


In controlled assessments, assignments must be taken within the Awarding

Organisation’s specified assessment window. Controlled conditions relate to task setting,

task taking (time, resources, supervision, authentication, collaboration) and task marking.

All controls for the assignment must be adhered to and are specific to the individual

assignment. The level of control required in supervising controlled assessment are

detailed in the relevant specification. Tasks can be formal: have a high level of control, like

examinations; or be less formal.

Quite often, there is a perception, for example by end-users and the general public, that

external assessment has more rigour than internal assessment. This is not the case where

quality control of procedures and quality assurance of internal assessments are in place

and strictly monitored.


Whatmethodsarebestforassessingdifferentaspects ofthetargetproficiency?

This is an important question when considering fitness for purpose of the assessment

method.

Table 3 sets out examples of various assessment methods used in vocational

qualifications, grouped according to what they are considered to assess best.

Table 3: Typical assessment methods—what they assess best

Best for assessing skills Best for assessing knowledge/understanding

• Learner-generated work (usually gathered in portfolios), e.g. work-based products produced

• Observation of performance in a work environment, or the conditions of the workplace

• Trade or skills tests

• Practical assignments.

• Examinations, tests, including multiple-choice assessments

• Presentations

• Written projects and assignments

• Reflective journals

• Interviews and professional discussions.

For further information about the range of different assessment methods and some

features, strengths and weaknesses of each, see Appendix A.

Examples are given below of methods used to assess the cognitive domain and an

assessment framed around naturally-occurring evidence. Synoptic assessment is

described to show how the integration and application of cognitive abilities and skills

can be assessed.

Methods for assessing cognitive domain

Where the domain is focused on knowledge and understanding and assessed through an

external examination:

• The level of attainment is determined by assessors who are independent (entirely

unconnected to the candidate or their learning to avoid conflict of interest)

• The sampling strategy is made more robust by having unseen questions which

means the candidate has to go into the exam having studied the entire domain

• The open-ended exam questions can invite complex and in-depth responses

showing depth of understanding.

A single method for assessing the cognitive domain can be sufficiently robust to

meet, for example, gatekeeper expectations who value high-stakes, external

examinations.


Assessment framed around naturally-occurring evidence

Methods for assessing evidence generated, for example, on-the-job and brought together

in a portfolio, can include a combination of:

• Observation

used to see the candidate perform tasks in a work-environment or simulated

environment; preparation, process and completion can be seen being performed.

What is seen (and not seen) can be used to inform the professional discussion

• Assessmentofwork-basedproducts

used to provide evidence of the ability to produce the desired outcomes (this is

often further supported by work log books; witness testimony and other forms of

evidence); again this can be used to inform the professional discussion

• Professional discussion

used to test candidate’s knowledge and to probe any areas seen as lacking through

the above methods.

In this illustration, very different assessment methods are used to target different

domains within the target proficiency and, between them, look to create a

complementary assessment strategy.

This strategy:

• Values the importance of realistic performance generated in the workplace

• Understands the need to assess the different domains to ensure full proficiency

(knowledge, skills and performance)

• Looks to avoid unnecessary duplication in what each method is contributing

• Acknowledges the importance of evidencing attainment across the domain (with

products/artefacts and work log showing depth).

In this type of scenario, the assessment expectations will be public and made explicit to

candidates, as a set of performance criteria. Where the target proficiency involves safety-

critical factors, the observation, rather than being in the workplace, may be based on a

simulated assessment task in a controlled environment.

By adding in witness testimony from an expert witness (someone highly familiar with

both the target proficiency and the candidate’s work), the assessor’s judgements can

also be substantiated. However, if this form of supporting evidence is to be used to

support assessment of performance, it is important to identify which learning outcomes

are to be addressed.


Box 5: Witness testimony

Witness testimony

A witness testimony is a statement from someone familiar with the candidate’s

work abilities and performance. The more reliable the witness the more valuable

the testimony. A credible witness would be someone with familiarity with the

assessment objectives and an expert themselves in the target proficiency. This

puts them in an advantageous position to comment.

If they are not connected with training or learning delivery, this is ideal because

there is no conflict of interest. So, in a workplace situation a witness testimony

from a competent operations team supervisor is potentially more credible than

someone from the training team responsible for training the candidate.

Witnesses can confirm competence or acceptable performance for parts of the

target proficiency that are hard to plan into assessment. This might be because

they happen too infrequently and involve performance and behaviour unseen or

hard to access or determine by the assessor.

Synoptic assessment

Synoptic Assessment requires candidates to combine elements of their learning from

different parts of a programme to show their ability to integrate and apply their skills,

knowledge and understanding with breadth and depth.


Box 6: Example of Synoptic Assessment

Example of synoptic assessment

A synoptic assessment can be used as an opportunity for candidates to be

examined as an advanced practitioner in their particular subject area through:

• A face-to-face oral examination where each candidate is examined by a

suitable panel of examiners. Questions are asked with or without visual aids

about different aspects of the subject. The aim to ensure that the candidate is

able to draw together all aspects of the designated subject area.

and/or

• A practical examination during which the candidate is expected to

demonstrate a logical approach, integration and application of knowledge

gained from across the designated area. Candidates may be asked to

demonstrate practical skills as appropriate for a synoptic exam.

Sometimes, synoptic assessment is positioned towards the end of a programme.

This means that it takes on a high-stakes role.

For example:

• In England, apprentices must undertake an independent end-point

assessment (EPA) which is a synoptic assessment of the knowledge, skills and

behaviours learnt throughout the apprenticeship. Its purpose is to make sure

the apprentice meets the standard set by employers and is fully competent

in the occupation. It is taken at the very end of the on-programme phase

of training when their employer (and in some cases their training provider)

is satisfied that they have met the “gateway” criteria to undertake the

assessment.

• An apprenticeship certificate is awarded only after the EPA is successfully

completed.

Alternatively, synoptic assessment can be placed in projects or project units

where qualification-wide or several units can be drawn upon and applied at any

point. The synoptic function can also be delivered by an external test or external

assessment unit which is clearly identified as synoptic.


E-assessment

The term e-assessment is generally applied to any type of assessment intervention

that has a technology component. This tends to involve one or more of the following:

e-testing; e-portfolios; and/or e-marking.

The term includes anything from minimal use of technology used in direct relation to the

assessment process, such as the transfer of data from optical mark reader (OMR) forms

and scanning of scripts, to much more technological assessment interventions.

For example, state-of-the-art e-portfolios can capture the learner’s information to:

…create a dataset that can be used to draw inferences about a learner’s progress and achievement (Mislevy et al. 2012). Such a dataset can have a qualitative nature, based on written reflections and narrative feedback, as well as a quantitative nature, for instance, the number of professional tasks carried out and performance scores received. Marieke van der Schaaf et al, November 2016:36146

Adaptive and assistive technology is an important facet of e-assessment’s potential.

Learners with additional support needs may find e-assessment preferable to traditional

assessments. There is a level of adaptability in how assessments can be presented visually

on-screen to better meet particular needs. Most e-assessment systems are capable of

supporting assistive technologies commonly used, such as screen readers.

The key benefits of e-assessment as used today

Reilly (August 2013)47 points to the importance when considering e-assessment of: ‘not

underestimating the institutional cultures and the social norms that have to change’.

Nevertheless, there is now a substantial volume of e-assessments being undertaken, and

an increasing track record of the robustness and reliability of the technology.

Some of the main benefits of successful and effective uses of assessment-related

technologies include the following:

• Theabilitytoprovideimmediatefeedback. Where tests are marked by computer,

the results are available immediately, and can provide information not only about the

overall result, but also give information on the nature of the performance across the

assessment

• Improved assessment validity. For example, on-screen assessments can be more

valid when they contain rich information (not just text, but sound and video) and

allow the candidate to interact with the information and present their information as

flexibly as they wish

• Increasedflexibility. Assessments can be provided at a more diverse range of

locations and times

• Moreefficientandenvironmentallyfriendlyadministration. E-assessment, and

the associated e-administration of candidates, reduces the administration burden on

centres and overhead cost (adapted from Winkley, 201048).

46 Marieke van der Schaaf, Jeroen Donkers, Bert Slof, Joyce Moonen-van Loon, Jan van Tartwijk, Eric Driessen, Atta Badii, Ovidiu Serban & Olle Ten Cate (November 2016) Improving workplace-based assessment and feedback by an E-portfolio enhanced with learning analytics Published online: https://link.springer.com/content/pdf/10.1007%2Fs11423-016-9496-8.pdf

47 Reilly, M. Further Education Learning Technology: A horizon scan for the UK Government. Foresight Horizon Scanning Centre, Ariel Research Services, August 2013

48 Adapted from Winkley, J. E-assessment and innovation, BECTA, 2010, p.20


Technology changes the medium for assessment practice, but it is important to remember that the central principles of assessment, the key considerations and concerns, still apply. Use of e-assessment should be seen as part and parcel of the ‘business as usual’ practices of qualification and assessment development. The same lifecycle challenges, quality processes, validity checks and concerns still apply.


Further Information

FAB offers training and webinars of relevance to this chapter.

Check the website for forthcoming events.


• AlphaPlus Consultancy (2014) Validation of vocational qualifications. Final Report

• Cambridge Assessment (January 2009, revised April 2017) The Cambridge Approach. Principles for designing, administering ad evaluating assessment

• Cedefop (2015). Ensuring the quality of certification in vocational education and training.

Luxembourg: Publications Office. Cedefop research paper; No 51

• Curcin, M., Boyle, A., May, T. and Rahman, Z. (February 2014) A validation framework for work-based observational assessment in vocational qualifications. London: The City

and Guilds of London Institute (commissioned by Ofqual)

• Elliott, G. (2011). A guide to comparability terminology and methods. Research Matters: A Cambridge Assessment Publication, Special Issue 2,9–19

• Harth, H. and Hemker,B.T. City & Guilds (January 2011) On the reliability of results in vocational assessment (Part of Ofqual’s Reliability Programme)

• Newton, P.E., Baird, J. Goldstein, H., Patrick, H. and Tymms, P. (Eds) (2007) Techniques for monitoring the comparability of examination standards. London: QCA

• Newton, P. E. Validity - what it is and why it matters. PowerPoint presentation at Ofqual

Conference, 2017, The Vox Conference Centre, Marston Green, 28 February

• Newton P.E. (2017) An approach to understanding validation arguments, Ofqual

• Ofqual (16 May 2013) Introduction to the concept of reliability

• Stasz, C. The Purposes and Validity of Vocational Qualifications. SKOPE Research Paper No. 105 November 2011

For further references, including those on assessment, see Appendix B.

71

Chapter 3

How to design a qualification

This chapter provides information and guidance on how to:

9 Conduct market research and consultations to inform the business case and design of your qualification

9 Establish your qualification’s objective and target level

9 Specify what your qualification will measure

9Design your qualification’s assessment.

72How to design a qualification

A prime concern will be to properly specify the target proficiency: the abilities that the

qualification is aiming to measure. This will provide the focus for assessment design, but

also the starting point for building your validity argument.

There will be two important outputs from the design stage of your qualification’s

lifecycle:

1 The proficiency specification which sets out in more detail the qualitative aspects of

the target proficiency (the abilities to be demonstrated) and the quantitative aspects

(what different levels of these abilities look like)

2 The measurement specification which clarifies assessment objectives and describes

all aspects of how candidates will be assessed (methods/assessment tasks).

All the features and processes which realise each step in the measurement construction

chain will need to be considered (see Chapter 2 for a reminder of this chain).

This is to ensure results from your qualification will be as accurate and useful as possible.


• Market research provides an insight into the characteristics, behaviours

and needs of a qualification’s target customers and stakeholders, to support

better business decisions

• Consultations with potential users, in particular employers, will help to

identify whether there is support for the qualification and their priorities for

its design

• For inclusion in registers of regulated qualifications, a level needs to be

identified for the qualification to indicate the difficulty and complexity of the

knowledge and skills associated with it. Where applicable, a level should also

be identified for any component for which a value of credit is assigned

• Abilities that candidates are required to demonstrate – the target proficiency

– need to be identified up-front and described in detail in the proficiency

specification

• The overall objective of the qualification should be such as to lead to a benefit

for the learners who have reached a specified level of attainment

• Assessment is an important initial design consideration, with decisions on

choice of assessment methods informed by the qualification’s objective,

proficiency specification and needs of end-users

• How decisions are arrived at, how the proposition is tested out, and eventual

justification for choice of methods will all be evidence of the processes to

ensure validity of the assessment

• The measurement specification should cover all features of the assessment

and take into account management, quality assurance and resource

implications.

73 How to design a qualification

3.1 Conduct market research and consultations

3.1.1 Market Research

Market research can be described as:

…the collection and analysis of information about markets, organisations and people to support better business decisions…Research that is properly structured can be used to deliver information that can provide insight into a target audience. [However]… data on its own is virtually useless; the value comes from interpreting the results and understanding their significance within a business context. (ISBA, 2011:3749)

Such research can provide information to:

• Inform the qualification’s business case

• Inform policy development, implementation and evaluation

• Help in understanding the behaviour and attitudes of target audiences, and find out

about their needs and preferences for the provision of products and services

• Identify the sizes, composition and profile of different markets

• Track changes and trends over time

• Enable a two-way dialogue with key stakeholders, and inform choice of methods for

communicating with them.

The analysis of findings from market research will help your Awarding Organisation to

keep pace with its target market, as well as test reactions to new products before they

are launched.

It should be noted that market research is different from marketing which is concerned

with the processes of bringing a product to market.

For discussion on marketing, see Chapter 5.

There are two main types of market research you can carry out:

3 Secondary research which involves collecting and analysing information and

statistical data that are already available, collected by someone else.

4 Primary research which is information you gather yourself through methods such as

customer surveys

You are likely to need findings from both types of research to decide whether your

qualification will be competitive in the market place.

49 ISBA (2011) A guide to understanding and working with market research agencies & consultancies. London: ISBA. Available at: https://www.mrs.org.uk/pdf/ISBA_Guide_to_Market_Research.pdf


Secondary research

Start by conducting some secondary research to get a feel of the market. This will help

you target your primary research more precisely. Look at similar qualifications, products

and services that are available.

• Can you learn from them?

• What do they do well?

• What do they do less well?

• How is your qualification and support package going to be better?

Understanding what your competitors are up to is critical. How are they working with

your target group/s? Where are the gaps in their offer? Are groups of potential users

being ignored?

Sources of information about competitors, products and services include:

• Registers of regulated qualifications, e.g. to allow searches of qualifications offered

by the different Awarding Organisations

• Websites of individual Awarding Organisations, e.g. to see what each offers in

terms of resources and support for customers, and annual reports on how they are

performing

• Statistical releases and reports from the regulators, e.g. to gain an overview of the

qualifications market and statistics on take up and the award of qualifications and

trends.

Consider how you might differentiate your qualification, for example, in terms of

approach to assessment, the use of e-learning and online assessments and so on.


Identify different segments of your market

Identify groups that you expect to be using your qualification: their size, composition and

profile.

This will not be a quick task, given the complexity of the education and training

marketplace. For example, the adult skills system is concerned with:

…all the resources, programmes, institutions and organisations through which employers and employees access training, support their career development, and seek certification for their skills and knowledge. (Brown et al, 2011: 850)

As a start, explore your Awarding Organisation’s customer data to identify which market

segments your existing customers fall into and where you might build market share.

Then explore other sources of information. Look at the characteristics of the learner

populations, demographic forecasts, the range of education and training providers, data

on student achievements and destinations; and the nature of the labour market.

Sources of such information include the following:

• The Office for National Statistics (ONS)51which produces official statistics on a

range of key economic, social and demographic topics; labour market statistics

• Government department databases/reports52, for example: the national pupil

database; reports on student destinations

• Education and Skills Funding Agency (ESFA)53which publishes National

Achievement Rates for Apprenticeships, and education and training data broken

down by: gender, ethnicity, age, learning difficulty/disability; sector subject area; type

of institution

• Inspectors of education and training54, which publish inspection reports on

providers, subject and thematic reports, and annual reports with analysis and

commentary on performance of education and training.

Developing a qualification is an expensive business and you cannot afford to waste effort

and resources on a qualification that will not be used. Many qualifications on the Register

of Regulated Qualifications (RQF) were not awarded during 2015/1655.

Data such as market size can help in deciding whether it is worth investing in the market,

but you will also need to know how the market works and key market trends.

50 Brown, D., Harris, M. and Fletcher, T. (2011) Reforming the skills system - lessons learned the hard way. London/Leicester: Institute of Directors/CFE51 The Office for National Statistics (ONS): https://www.ons.gov.uk/52 Pupil database for England: https://www.gov.uk/government/publications/national-pupil-database-user-guide-and-supporting-information Reports on

student destinations (England): https://www.gov.uk/government/collections/statistics-destinations53 Education and Skills Funding Agency: https://www.gov.uk/government/organisations/education-and-skills-funding-agency54 Ofsted (England): https://www.gov.uk/government/organisations/ofsted ; Estyn (|Wales): https://www.estyn.gov.wales/language ; ETI (Northern Ireland):

www.etini.gov.uk ; Education Scotland: https://education.gov.scot/what-we-do/inspection-and-review/About-inspections-and-reviews55 Ofqual (2017) Annual qualifications market report, England, Wales and Northern Ireland. Academic Year 2015/16 – less than half the qualifications on the RQF

had been awarded during 2015/16


Where they exist, Sector Skills Councils keep up-to date with what is happening in their

industry. They can be a good source of information offering, for example: insights into

trends and issues; industry profiles, including workforce characteristics and employer

skill needs; news items.

International Markets

Sources of information on opportunities to develop a qualification for the overseas

market include:

• Bid notification services offered by commissioning agencies

• Bid tracker tools; many are free, but the commercial ones tend to be more

sophisticated and also have better filtering

• Pipeline intelligence gathering through partnerships and networks via the

Department for International Trade.

There are several government departments, bodies and agencies that promote export

opportunities for British businesses, including Awarding Organisations. They provide

information on the overseas market and how to partner or position your Awarding

Organisation internationally. You can sign up to the newsletter of The Department for

International Trade which also offers webinars on topics of relevance to exporters.

For examples of such organisations, see Appendix B.

Primary research

In addition to obtaining information from the above sources, you may wish to conduct

some fresh research. This will enable you to gather more detailed information from

specific groups in your target population. Use of methods such as questionnaire surveys

and/or focus groups will help you to understand more precisely what is likely to appeal to

each group and meet their needs.

Your Awarding Organisation may be large enough to have its own research department,

with colleagues who have expertise in market research. If not, you could go it alone or,

alternatively, commission another organisation or consultant to do the research for you.

If this is the preferred option, the Institute of British Advertisers (ISBA) offers a guide

to understanding and working with market research agencies and consultancies. The

Market Research Society (MRS) also provides a list of agency services and advice on

specific areas of expertise and sectors. One place to start is the Research Buyer’s Guide

at: www.theresearchbuyersguide.com.

MRS is the largest professional association representing providers and users of market

research. Its work includes setting and upholding professional standards via its Code of

Conduct (see Box 7). This code can be used to inform your practice and make judgments

about the market research activities of others.


Box 7: MRS Code of Conduct

MRS Code of Conduct

The MRS Code is based upon ten key principles.

In summary, researchers shall:

1 Ensure that participation in their activities is based on voluntary informed

consent

2 Be straightforward and honest in all their professional and business

relationships

3 Be transparent as to the subject and purpose of data collection

4 Respect the confidentiality of information collected in their professional

activities

5 Respect the rights and well-being of all individuals

6 Ensure that respondents are not harmed or adversely affected by their

professional activities

7 Balance the needs of individuals, clients and their professional activities

8 Exercise independent professional judgement in the design, conduct and

reporting of their professional activities

9 Ensure that their professional activities are conducted by persons with

appropriate training, qualifications and experience

10 Protect the reputation and integrity of the profession.


3.1.2 Consultations

As with market research, a key question is whether you have the required expertise and

resources available to design and manage an effective consultation strategy.

Effective consultation is not always easy to achieve. Typical difficulties include: engaging the right cross-section of stakeholders, rather than a narrow or self-selecting group; interpreting the results, particularly where responses appear to conflict; balancing consultation feedback against other factors that affect decisions, such as practicalities (costs and resources) and statutory requirements. (FISSS, November 2014)56

Can you do it alone?

It may be more cost-effective and efficient to contract with an external supplier

to design and conduct the consultations across the breadth of development for

all qualifications in a particular sector.

Using specialists will have a cost, but offers significant advantages, such as:

• Confidence that consultation exercises will be properly designed

and carried out

• Access to trained interviewers where needed

• Expert analysis and presentation of results.

If you decide to use an external agency to plan and implement your consultation

strategy, allow time to: prepare a brief; tender; and select your working partner.

Federation for Industry Sector Skills and Standards (November 2014)

56 Federation for Industry Sector Skills and Standards (November 2014) Improving Apprenticeship standards through consultation. A guide and toolkit for employers


Initial consultation

Users of qualifications are people who have a legitimate interest in your qualification or

type of qualification being proposed.

They may include:

• Learners and learner representatives

• Centres or third parties that have been involved in delivering your Awarding

Organisation’s qualifications

• Others in schools, further and higher education establishments

• Employers and employer representatives

• Government departments and agencies, including funding agencies (to allow for

claims to be made for government funding)

• Professional bodies

• Others such as consumer organisations and regulatory authorities (e.g. the Food

Standards Agency, Competent Person Register).

Findings from your market research will have identified which users to consult on the

detail of your qualification.

The first task will then be to gather evidence of support for your proposed qualification

from these users. Their reasons for supporting it will vary, but a central concern should be

whether users think that it would provide a benefit to learners.

For discussion on benefits to learners, see Section 3.2.1.

The relevant regulator/s will need to be satisfied that you have evidence of this support,

for example, letters from employers confirming their support for qualification content

and explaining how learners with the abilities it covers could benefit their business.

Consultation with employers is an important part of establishing the relevance,

demand and support for your qualification. They will also have an important role in its

development.

Ideally, employers to be consulted should include SMEs and micro-businesses alongside

larger companies. The range and proportion of each will depend upon the sector to be

served by your qualification.


Employer views and inputs are widely sought. They have a business to run and other

Awarding Organisations will also be seeking their input. This means that it may take some

time to gain access to the employers you need. So, start with those who have already

worked with your Awarding Organisation who may be willing to voice their support and/

or advise on the best ways of engaging others. If you are starting from scratch or need

to broaden your range of employers by targeting individual employers, Chambers of

Commerce, trade associations and the Federation of Small Businesses (FSB) may be

able to help.

Employer views are important, but do not forget other users, especially

potential customers.

Ongoing consultation

In addition to initial consultations to gauge support, there needs to be a formal,

structured process by which stakeholders, including employers, can contribute to your

qualification’s design and development.

The Federation for Industry Sector Skills and Standards (FISSS) has published a toolkit for

designing and conducting consultations which focuses on apprenticeships. This could be

adapted for your purpose.

The toolkit57 suggests planning your consultation strategy and holding pre-

consultation meetings to help shape the purpose and agenda of the consultations

and encourage ‘buy in’.

Although evidence of support for your proposed qualification will be needed early on,

your strategy should take into account the need to plan for multiple consultation points.

This is to allow for the various stakeholders to engage in the different stages of the

development of your qualification. Potential issues can then be raised and addressed

to avoid costly modifications at a later stage. However, care needs to be taken to avoid

stakeholder ‘fatigue’ by asking too much of them, too often.

Consultation methods will need to be selected once you have planned:

• When to conduct consultations: key points in your qualification’s lifecycle where

stakeholder input would be useful for specific purposes

• With whom to consult: particular type/s of stakeholder or a more general population.

You will need to formulate carefully the questions which will elicit the information you

require. You will also need to ensure that consultees have the information necessary to

make informed and relevant responses.

57 Federation for Industry Sector Skills and Standards (November 2014) Improving Apprenticeship standards through consultation. A guide and toolkit for employers. FISSS.


Some methods may be more appropriate than others for consulting with your target

groups and the types of questions you want to ask (and mode of response required).

Consider comparative costs and advantages.

FISSS recommends using a multi-channel approach to consultation, combining web-

based tools with face-to-face and/or written approaches. This gives potential consultees

the opportunity to choose the way they would prefer to participate and provide you with

a more diverse range of responses (See Box 8).

When analysing responses, it is important to adopt a systematic and consistent approach.

It helps to collate results into an accessible form, such as a spreadsheet or database

organised, for example, by question, theme, type of respondent. Results can be displayed

using graphical forms such as charts and graphs.

Findings from the analysis of responses can then be used to inform decisions on the

design of your qualification, and to provide feedback to stakeholders.

For further discussion of consultation in relation to review and evaluation,

see Chapter 6.


Box 8: Examples of consultation methods

Examples of consultation methods

Connected mass deliberation: links large numbers of participants simultaneously for a one-off event

using a combination of face-to-face methods and technologies such as wireless voting, television and

networked computer systems to aid responses

Consultation document: documents are released to provide information and invite formal written

responses

Deliberative polling and dialogue: a randomly selected and demographically representative group

completes a questionnaire at the beginning of the process. Participants are provided with documents

on various proposals, including arguments for and against each. Participants debate the pros and cons

provided for each proposal and then complete a second questionnaire to assess changes from initial

perceptions and why those changes occurred

Employer panels: a consultative body of demographically-representative participants who respond to

proposals on a long-term basis

Online discussion groups: a group of subscribers communicate on a particular topic through a single

e-mail address

Web-forum: combines the basics of web-based asynchronous discussions threads on particular topics

with aspects of live chat, document sharing

Forums/strategy groups: structured and regular meetings with interest groups to provide information,

seek views and develop/endorse plans and strategies

Opinion polls, questionnaires, surveys: a process for collecting information and opinions through a

pre-set list of questions. Can be administered via mail (paper-based), online, or a combination of these

Seminars, conferences, focus groups: a formally organised, one-off meeting to exchange and gather

information on key issues. Usually follow a format of presentations followed by small group discussions,

ending in a large group discussion of key issues. May be hosted face-to-face or electronically using web-

based webinar/conference software

Social networking platforms such as LinkedIn, Facebook and Twitter are used to pose questions and

promote online discussions

Telephone interviews: pre-selected individuals are asked a series of questions to gather information on

a specific topic by trained interviewers

Telephone polling/voting lines: automated phone number(s) normally used to collect votes although

can also collect comments, or opinions. Platform can be voice or SMS.

FISS (November 2014)

For a discussion of the pros and cons of these methods,

see FISSS (November 2014: 31-35).


3.2 Establish your qualification’s objective and target level

3.2.1 Establish your qualification’s objective

It is important that you establish the overall objective of your qualification and consider

why someone would want to use it. To comply with the regulatory conditions of Ofqual,

CCEA Regulation and/or Qualifications Wales58, this overall objective must be such

as to lead to a benefit for learners who have reached a specified level of attainment.

If your qualification is to be regulated by another body, you will need to check on its

requirements.

Programmes of learning can benefit learners without the need for a qualification.

The question to be addressed is: in what ways is it intended that your qualification

adds value for the learner?

A qualification might be of benefit to the learner if it:

• Provides evidence that can be used to show and secure the confidence of others that

they have acquired, for example:

» The knowledge, understanding and skills within a particular subject domain at

a certain level to enable them to progress to a higher level qualification, e.g. to

satisfy any pre-requisites for that higher level

» Underpinning knowledge, understanding and skills, and insights into the world of

work, in relation to a broad occupational area or a specific occupational area, e.g.

to demonstrate work readiness

» Skills, knowledge and understanding which confirms their ability to perform

functions of an occupational role to the standards required, or capability

associated with a different or higher level role, or specialisation

• Enhances feelings of achievement, personal or professional self-worth.

There may be one or more purposes to which learners can put their qualification.

For example:

• To meet requirements in applying for, and gaining entry into, a job or a place in

Further or Higher Education

• To allow entry to a professional membership organisation

• To contribute towards meeting requirements for a license to practice

• To provide evidence of expertise, for example, to clients/customers or to their

employer to progress within their career.

58 It is always wise to check regulations. For example, at the time of writing, Qualifications Wales is conducting a review of its Standard Conditions of Recognition on areas such as clarity and levels of prescriptiveness, from Autumn 2017 to Autumn 2018. A revised version should be available in Spring 2019.


Other users have their own purposes for qualifications. For example:

• To secure funding, e.g. for a vocational education or training programme

• For selection purposes, e.g. to enable employers and education gatekeepers to

differentiate between applicants for a job or course

• For predicting likely success, e.g. to inform employer judgement on whether an

employee is ready for promotion or a different role

• For employers to provide evidence of staff expertise to clients/customers

• For comparing educational standards, e.g. as an accountability measure for

government to make judgements about school/college performance; for international

benchmarking; for parents/carers to see how a school is performing in relation to

others (see note on performance tables).

Such purposes can be ‘intertwined’, for example: ‘a qualification, as currency in the labour

market…and in the education and training system, can be used as a key performance

indicator for funding purposes’; but is also important to individuals… (see, for example,

Keep and James, 201059).

It is thus possible for qualifications to have more than one purpose, reflecting the

interests of different stakeholders.

However, problems can arise:

… if the purposes and functions of a single qualification become too numerous, diverse and contradictory; [they may, for example], impact negatively on perceptions of users to the validity of the qualification for their particular priorities. (Cedefop, 2010: 2060)

It is important to get your qualification’s objective right and describe it accurately.

This is because end-users’ perceptions of purpose can affect the qualification’s:

…currency in the labour market (whether employers want them) and their links with professional and career pathways (whether they are valued by professional bodies), which in turn can substantially affect take-up (use value). (Unwin et al, 200461)

If your qualification’s objective is related to use by young people aged under 19 years,

then you will need to check out the legal and regulatory requirements for offering

qualifications to this age group.

For example, Awarding Organisations have a legal obligation under the Education and

Skills Act, 2008, to decide whether or not young people under 19 years old would use

their regulated qualifications for the purpose of education or training. If a qualification is

relevant to the Act, the Ofqual criteria must be applied, including the assignment of hours

for guided learning (Ofqual, 2015)62.

59 Keep, E.J. and James, S. (2010) “Recruitment and selection – A review of extant research and some thoughts on its implications for education and training policy”, In: SKOPE Research Paper. Cardiff: Cardiff University, SKOPE

60 European Centre for the Development of Vocational Training (2010) Changing Qualifications: A Review of Qualifications Policies and Practices, Luxembourg: Publications Office of the European Union.

61 Unwin, L., Fuller, A., Turbin, J., & Young, M. (2004). What determines the impact of vocational qualifications? A literature review. London: Department for Education & Skills

62 Ofqual (2015) Criteria for Determining Whether a Qualification is Relevant for the Purposes of the Education and Skills Act 2008 For All Awarding Organisations https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/461247/criteria-for-determining-whether-a-qualification-is-relevant-for-the-purposes-of-the-education-and-skills-act-2008.pdf A ‘number of hours for guided learning’ in relation to a form of a qualification, means a number of notional hours representing an estimate of the amount of actual guided learning which could reasonably be expected to be required in order for persons to achieve the standard required to obtain that form of the qualification. See: www.gov.uk/government/publications/total-qualification-time-criteria


Box 9: A note on Performance Tables in England

A note on Performance Tables in England63

Performance tables include statistical information on how learners have

performed at their first attempt in public examinations at age 16 and 18 in

England.

The Department for Education determines what publicly funded qualifications

attract performance points and can be included, for the purpose of school

and college accountability, in these published tables. Performance points for

vocational qualifications are calculated on the basis of size, level and number of

possible (passing) grades.

DfE only approves qualifications for recognition in these Performance Tables

that support progression to learning at the next level or skilled employment. The

categories of vocational qualifications for the 2018 tables are: technical awards

for students aged 14-16; technical certificates, tech levels and applied general

qualifications for students aged 16-19.

3.2.1 Establish your qualification’s target level

For inclusion in the registers of regulated qualifications, you will need to identify a level

for your qualification.

Your previous market research and consultations may have indicated the level to target.

You can now use the level descriptors in the relevant qualification framework/s as a

guide to the difficulty and complexity of the knowledge and practical skills that may be

required. You only have to achieve a sufficient match to decide whether your qualification

is at one level rather than another. If your qualification is small, for example, it is unlikely

to cover everything. To get a feel of level, you can also look at similar qualifications at the

same level and other benchmark statements or agreed industry statements on levels.

The Regulated Qualifications Framework (RQF) shared by England and Northern

Ireland has the following levels: Entry 1 to 3; and Levels 1 to 8. The RQF maps to, but

does not include qualifications in, the Framework for Higher Education Qualifications (FHEQ)64.

The RQF level descriptors are outcomes-focused and set out the skills, knowledge

and understanding that you might typically expect of someone with a qualification at

that level (see table 4).

The Credit and Qualifications Framework for Wales (CQFW) also has 9 Levels, but

includes qualifications offered in higher education. Levels 4−8 are intended to be

consistent with the five levels within the FHEQ. It uses the same level descriptors

as the RQF.

63 Information on Performance Tables available at: https://www.gov.uk/government/publications/2018-performance-tables-technical-and-vocational-qualifications

64 Quality Assurance Agency (2008) Framework for Higher Education Qualifications: http://www.qaa.ac.uk/en/Publications/Documents/Framework-Higher-Education-Qualifications-08.pdf


The Scottish Qualifications Framework (SCQF) has 12 levels: levels 1, 2 and 3 broadly

correspond to Entry levels in the RQF/CQFW; SCQF level 4 corresponds to level 1,

SCQF level 5 corresponds to RQF/CQFW level 2 and so on.

Box 10: SCQF Level Descriptors

SCQF Level Descriptors65

The descriptors in the SCQF outline the general outcomes of learning at SCQF

levels 1-12 under five broad headings:

• Knowledge and understanding (mainly subject based);

• Practice (applied knowledge and understanding);

• Generic cognitive skills (e.g. evaluation, critical analysis);

• Communication, numeracy and IT skills; and

• Autonomy, accountability and working with others.

The descriptors are for guidance only; it is not expected that learners will cover

every aspect. Use of terms in the descriptors has been standardized.

All the Qualifications Frameworks map to the European Qualifications Framework

(EQF) which allows for comparison across frameworks from different countries.

See how qualifications in the different frameworks compare at: http://ccea.org.uk/sites/

default/files/docs/accreditation/guidance/Qualifications_can_cross_Boundaries.pdf

(published March 2017).

All of your qualification’s content and assessment may not need to be at the level the

learner is expected to reach on achieving it. For example, if it is likely to be delivered

over a long period of time, you may like the learner to progress from one level to

another within the qualification, with content and assessment taking into account their

increasing ability.

However, when assigning more than one level to a qualification, you must be satisfied

that the range of achievement is sufficient, and have a clear rationale for doing this.

You must take all reasonable steps to ensure that, where applicable, the level assigned

to each component is appropriate when considered against the level assigned to the

qualification as a whole.

65 Information on SCQF level descriptors available at: http://scqf.org.uk/wp-content/uploads/2014/03/SCQF-Level-Descriptors-WEB-Aug-2015.pdf


The Regulated Qualifications Framework66

Table 4: The Regulated Qualifications Framework

Level Knowledge descriptor (the holder…) Skills descriptor (the holder can…)

Entry 1

Progresses along a continuum that ranges from the

most elementary of achievements to beginning to

make use of knowledge and/or understanding that

relate to the subject or immediate environment.

Progress along a continuum that ranges from the

most elementary of achievements to beginning to

make use of skills that relate to the subject or the

immediate environment.

Entry 2

Has basic knowledge or understanding of a subject

and/or can carry out simple, familiar tasks; and

knows the steps needed to complete simple

activities.

Carry out simple, familiar tasks and activities. Follow

instructions or use rehearsed steps to complete

tasks and activities.

Entry 3

Has basic knowledge and understanding to carry out

structured tasks and activities in familiar contexts;

and knows and understands the steps needed to

complete structured tasks and activities in familiar

contexts.

Carry out structured tasks and activities in familiar

contexts.

Be aware of the consequences of actions for self

and others.

1

Has basic factual knowledge of a subject and/

or knowledge of facts, procedures and ideas to

complete well-defined routine tasks and address

simple problems; and is aware of aspects of

information relevant to the area of study or work.

Use basic cognitive and practical skills to complete

well-defined routine tasks and procedures. Select

and use relevant information.

Identify whether actions have been effective.

2

Has knowledge and understanding of facts,

procedures and ideas in an area of study or field of

work to complete well-defined tasks and address

straightforward problems.

Can interpret relevant information and ideas. Is

aware of a range of information that is relevant to

the area of study or work.

Select and use relevant cognitive and practical skills

to complete well defined, generally routine tasks

and address straightforward problems. Identify,

gather and use relevant information to inform

actions.Identify how effective actions have been.

3

Has factual, procedural and theoretical knowledge

and understanding of a subject or field of work to

complete tasks and address problems that while

well-defined, may be complex and non-routine. Can

interpret and evaluate relevant information and

ideas.

Is aware of the nature of the area of study or work.

Is aware of different perspectives or approaches

within the area of study or work.

Identify, select and use appropriate cognitive and

practical skills, methods and procedures to address

problems that while well defined, may be complex

and non-routine. Use appropriate investigation to

inform actions.

Review how effective methods and actions.

4

Has practical, theoretical or technological

knowledge and understanding of a subject or field

of work to address problems that are well-defined

but complex and non-routine. Can analyse, interpret

and evaluate relevant information and ideas. Is

aware of the nature of approximate scope of the

area of study or work. Has an informed awareness

of different perspectives or approaches within the

area of study orwork.

Identify, adapt and use appropriate cognitive

and practical skills to inform actions and

address problems that are complex and nonroutine

while normally fairly well-defined.

Review the effectiveness and appropriateness of

methods, actions, methodis and results.

66 Ofqual (2015) Qualification and component levels. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/461637/qualification-and-component-levels.pdf


Level Knowledge descriptor (the holder…) Skills descriptor (the holder can…)

5

Has practical, theoretical or technological


of work to find ways forward in broadly defined,

complex contexts. Can analyse, interpret and

evaluate relevant information, concepts and ideas.

Is aware of the nature and scope of the area of

study or work. Understands different perspectives,

approaches or schools of thought and the reasoning

behind them.

Determine, adapt and use appropriate methods,

cognitive and practical skills to address broadly

defined, complex problems. Use relevant research

or development to inform actions.

Evaluate actions, methods and results.

6

Has advanced practical, conceptual or technological


of work to create ways forward in contexts where

there are many interacting factors. Understands

different perspectives, approaches or schools of

thought and the theories that underpin them. Can

critically analyse, interpret and evaluate complex

information, concepts and ideas.

Determine, refine, adapt and use appropriate

methods and advanced cognitive and practical skills

to address problems that have limited definition

and involve many interacting factors. Use and,

where appropriate, design relevant research and

development to inform actions. Evaluate actions,

methods and results and their implications.

7

Reformulates and uses practical, conceptual or

technological knowledge and understanding of

a subject or field of work to create ways forward

in contexts where there are many interacting

factors. Critically analyses, interprets and evaluates

complex information, concepts and theories to

produce modified conceptions. Understands the

wider contexts in which the area of study or work

is located. Understands current developments in

the area of study or work. Understands different

theoretical and methodological perspectives and

how they affect the area of study or work.

Use specialised skills to conceptualise and

address problematic situations that involve many

interacting factors. Determine and use appropriate

methodologies and approaches. Design and

undertake research, development or strategic

activities to inform or produce change in the

area of work or study. Critically evaluate actions,

methods and results and their short- and long-term

implications.

8

Develops original practical, conceptual or

technological understanding to create ways forward

in contexts that lack definition and where there

are many complex, interacting factors. Critically

analyses, interprets and evaluates complex

information, concepts and theories to produce

new knowledge and theories. Understands and

reconceptualises the wider contexts in which the

field of knowledge or work is located. Extends

a field of knowledge or work by contributing

original knowledge and thinking. Exercises

critical understanding of different theoretical and

methodological perspectives and how they affect

the field of knowledge or work.

Use advanced and specialised skills and techniques

to conceptualise and address problematic situations

that involve many complex interacting factors.

Formulate and use appropriate methodologies and

approaches. Initiate, design and undertake research,

development or strategic activities that extend

or produce significant change in the field of work

or study. Critically evaluate actions, methods and

results and their short- and long-term implications

for the field of work or knowledge and its wider

context.


Target Proficiency

3.3 Specify what your qualification will measure

What does it mean to be proficient in the area that is the focus of your qualification? What should those who have achieved your qualification know, understand and be able to do?

The target proficiency is what needs to be measured in the ‘real world’. For example, in

a qualification focused on electrical installation, the target proficiency might be ‘safe

and competent electrical work’. The proficiency specification articulates what ‘safe and

competent electrical work’, actually means in practice.

The proficiency specification is an important point of reference for each step in designing,

developing and delivering your qualification. It will be crucial in building your validity

argument. As such, it must represent, as faithfully as possible, the target proficiency.

Not only that, it will need to communicate clearly and accurately what is to be measured

to all those who need to know and the scale of what is required (e.g. the different levels

of abilities).

What abilities should be considered?

You may find that there are various, and maybe contradictory, views to consider when

identifying the abilities to be measured by your qualification. Stakeholders may intend

using your qualification for different purposes leading to different priorities: some may

want the qualification to primarily focus on a specific set of skills and knowledge to be

confident that qualification achievers will be able to do a particular job; others may want

a more broadly-based set of abilities.

Keep firmly in mind the main objective of your qualification and work with interested

parties to try and take account of these different perspectives. Ultimately, though, you

have to decide what you are going to achieve with the qualification – you will not be

able to please everyone where there is diverging opinion on what the qualification

needs to do.

The range of abilities to be considered will certainly include two main groups:

cognitive abilities and skills.

There have been various systems for classifying different abilities, typically listed in

hierarchical order of complexity in which the higher, more complex abilities subsume the

lower. The most commonly known, perhaps, is Bloom’s Taxonomy (see Box 11).

Such taxonomies may be a useful checklist for discussing and firming up on the types of

abilities that are relevant to your qualification. You will see the thinking behind them

reflected in the design of many existing qualifications.


Box 11: Domains in Bloom’s Taxonomy of Educational Objectives

Domains in Bloom’s Taxonomy of Educational Objectives67

This is a classification framework which has three parts (or overlapping domains):

cognitive domain (knowledge; comprehension; application; analysis; synthesis;

evaluation); affective domain (feelings, emotions and behaviour, i.e. attitude, or

‘feel’); and psychomotor domain (manual and physical skills, i.e. skills, or ‘do’).

The cognitive domain is described as including:

• Recall of facts, e.g. ability to remember and repeat information which

has been learnt

• Comprehension, e.g. ability to understand what has been taught or presented,

to explain, state the reasons for something

• Application of knowledge, e.g. ability not only to recall and understand, but

apply what has been learnt to a problem or situation that is new to the learner

• Synthesis, e.g. ability to put together information from different parts of a

programme to produce something new, such as when designing a product

• Analysis, e.g. ability to breakdown something into its constituent parts, to

identify the organisational structure and relationships between the parts

• Evaluation, e.g. ability to judge the worth of a service or product.

The psychomotor domain is described as including:

• imitation, e.g. ability to copy action of another; observe and replicate

• Manipulation, e.g. ability to reproduce activity from instruction or memory

• Develop precision, e.g. ability to execute skill reliably, independent of help

• Articulation, e.g. ability to adapt and integrate expertise to satisfy a non-

standard objective

• Naturalization, e.g. automated, unconscious mastery of activity and related

skills at strategic level.

The affective domain includes abilities to: receive; respond; value; organise or

conceptualise values; internalise or characterise values.

67 Bloom, B. S. (1956) Taxonomy of Educational Objectives. London: Longman Group. There are other taxonomies which you may like to explore, e.g. Ebel (1979) based on examination questions; Coltham and Fines (1971), based on Bloom.


The distinctions between different abilities will be important when deciding on the

most valid methods of assessment. In many technical subjects, for example, learners will

be required to interpret drawings (requiring cognitive abilities) and produce drawings

(physically draw, or use IT) to communicate technical information (requiring skills).

More than one method may be needed for assessing these different abilities.

But Bloom’s taxonomy, like others, continually evolves as others amend and/or extend

it or produce alternative versions. For example, the psychomotor domain originally

addressed skills development relating to manual tasks and physical movement. This

has been updated to include other skills such as technical skills (e.g. those involved in

operating IT equipment) and communication skills.

The various lists should not, therefore, constrain you in specifying your qualification’s

proficiency. They represent only one way of looking at abilities and are subject to on-

going debate, for example, about the hierarchical ordering of abilities such as synthesis

and evaluation. The distinction between cognitive and skill-based abilities may not be as

clear cut, either, as that suggested by having separate lists.

For example, someone is unlikely to do something well if they do not have some

knowledge and understanding of the processes involved. Most skills are not devoid of

knowledge and different types can be used in applying them: ‘know that’; ‘know how’;

‘tacit knowledge’; ‘work process knowledge’68.

You will find many perspectives and nuances in definitions of what is meant by skill.

For example:

At the core of the term “skill” is the idea of competence or proficiency – the ability to do something well…to perform a task to a pre-defined standard of competence, but to many the word also connotes a dimension of increasing ability (i.e. a hierarchy of skill). …it also evokes images of expertise, mastery and excellence… (National Skills Task Force, 2000: 2169)

In this short quote alone, several concepts associated with skill are mentioned70. As

well as quality of performance, there is the notion of developmental progression (e.g.

along a continuum from novice to expert). The ability to ‘transfer’ or ‘adapt’ skill to new

settings, is often included in definitions of skill. The specifications for generic skills

(variously labelled as common, core, key, essential or transferable skills) are examples of

how cognitive abilities and skills have been combined for application within and across

different contexts, at different levels of complexity.

68 For example: Ryle (1949) made a distinction between ‘knowing that’ (propositional knowledge that underpins or enables action) and ‘knowing how’ (practical knowledge that is inherent in action). Polanyi (1967) coined the term ‘tacit knowledge’: knowledge that is important to skilled behaviour, but difficult to tell others. A study by Boreham and Fischer (2002) developed the concept of ‘work process knowledge’: knowledge constructed in practice at work when, for example, employees solve problems.

69 National Skills Task Force (2000) Research Report. DfEE70 To be competent, proficient and expert are stages described in the Dreyfus model of progression (Dreyfus and Dreyfus, 1986). Eraut describes competence

as the ability to ‘recognise features of practical situations and to discriminate between them, to carry out routine procedures under pressure and to plan ahead’. Proficiency ‘marks a different approach to the job: behaviour is… semi-automatic; situations are apprehended more deeply and the abnormal is quickly spotted and given attention’. Progression from ‘proficiency to expertise finally happens when the decision making as well as situational understanding becomes intuitive rather than analytic…and thus requires significantly more experience’. Eraut M. (1994) Developing Professional Knowledge and Competence. Falmer Press (125/6). The concept of ‘mastery’ has long underpinned vocational qualifications, with candidates having to meet all the assessment criteria to pass.


For some qualifications, what candidates need to know, understand and be able to do may already have been identified by others.

For example, National Occupational Standards (NOS) have historically informed what

National [and Scottish] Vocational Qualifications (N/SVQs) need to measure.

NOS describe standards of performance that individuals must achieve when

carrying out functions in the workplace, together with underpinning knowledge and

understanding. They are usually expressed as performance criteria, derived from

functional analysis of workplace tasks or occupations and developed in partnership

with employers and other stakeholders (usually through the relevant Sector Skills

Council or Standards Setting Organisation).

Functional analysis defines the nature of an occupational sector and the functions

performed within it. This is an essential process in defining occupational

competence71 and in setting boundaries between different occupations. Functions

are the activities a person is expected to do as part of their job. They must have a

clear purpose and outcome that are valuable to an employer. Once these functions

are identified, everything else that should go into the NOS can be agreed with

employers (UKCES, June 2011: 2472).

Table 5 gives an extract from one example of a NOS. You will see it is specified in

considerable detail, with performance and knowledge/ understanding specified

separately. In Wales, Northern Ireland and Scotland, NOS remain the basis for the vast

majority of vocational qualifications. However, in England, there is a move away from

NOS to the use of other standards in some contexts.

71 Occupational competence, as used in NVQs, is not only about whether someone can perform the functions as presented in a particular work role or job, taking account of all the organisational and interpersonal problems they have to cope with in their day-to-day work. It is also about whether they will be able to perform these functions in another company or organisation. This is why understanding the principles underpinning the activities and an awareness of variations they may meet in a different context, is also important (Jessop, 1990). In Bees, M. & Swords, M. (eds) National Vocational Qualifications and Further Education. London: Kogan Page/NCVQ: 21)

72 UK Commission for Employment and Skills (June 2011) NOS Strategy 2010-2020 – Revised Strategy June 2011. Available at: http://www.ukces.org.uk/assets/ukces/docs/supporting-docs/nos/nos-strategy-2011.pdf


Show understanding of customer service principles

Table 5: Extract from a National Occupational Standard for customer service73

Performance

criteria

Use accepted customer service language

You must be able to

P1 Communicate to customers your organisation’s service offer, how it balances organisational and customer needs and how it meets customer expectations

P2 Compare the service offers of commercial, public sector and third sector organisations and how they each meet customer expectations

P3 Discuss with colleagues steps that team members can take to deal with different customers and different customer service situations

Apply customer service principles in your customer service role

You must be able to

P4 Follow the key policies and procedures in your organisation for the delivery of services or products

P5 Show an awareness of how your organisation’s service approach and service offer fit within your own industry and differ from those of other industries

P6 Show an awareness of how your organisation’s service approach and service offer differ from your organisation’s competitors

P7 Discuss with colleagues how your organisation’s ethical and value base fits with organisational needs and customer expectations

Knowledge and understanding

You must be able to

K1 How your organisation’s service offer manages customer expectations within financial and other resource limitations

K2 How customers form their expectations of services or products

K3 How customer service can contribute to best value in a public sector or third sector organisation

K4 How customer service can provide a competitive advantage for a commercial organisation

K5 How you deal with different customer behaviours and personalities to achieve customer satisfaction

K6 The importance of effective communication and teamwork for the delivery of excellent customer service

K7 The importance of continuous improvement within customer service

K8 The key policies and procedures in your organisation for the delivery of services or products and why it is important to follow them

K9 How your industry’s approach to customer service differs from the approach of one other industry

K10 The service offer of competitors of your organisation or any organisations offering similar services or products

K11 The features and benefits of your organisation’s services or products that influence customer service delivery and customer satisfaction

K12 How your organisation balances its needs and goals with customer expectations and needs

K13 The ethical and value base of your organisation’s approach to customer service

73 http://www.ukstandards.org.uk/PublishedNos/CFACSF3.pdf


Awarding Organisations that are an Approved Apprenticeship Assessment Organisation,

will have Apprenticeship Standards to describe what is to be measured in End-Point

Assessments. In England, these Standards replace apprenticeship frameworks (commonly

understood as a ‘basket of qualifications’ and other features necessary for certification74).

Box 12: Apprenticeship Standards

Apprenticeship Standards are designed by employers to meet their needs, the

needs of their sector and the economy more widely. They should be ‘short, easy

to understand documents that describe the Knowledge, Skills and behaviour

(KSBs) required to undertake a specific occupation well, and to operate

confidently within a sector. Standards focus on how an apprentice should

demonstrate mastery of an occupation, and meet professional registration

requirements in sectors where they exist (for example, in engineering, science

and accountancy).

See Table 6 for an extract from an Apprenticeship Standard. Note that ‘behaviours’ are

included, as well as knowledge/understanding and skills.

For qualifications to be used in England, the Institute for Apprenticeships convenes

convenes employer panels for each route to advise on the required knowledge, skills

and behaviours.

With some qualifications, of course, there is not a ready-made description of what

should be measured. Abilities to be measured in a new syllabus-based qualification,

for example, will be informed by the topics to be studied. But, these may need up-dating

and/or may not have the required detail. Direct consultations with subject specialists,

and working groups that include employers and professionals from the respective field,

may be required to agree what candidates actually need to know and/or do in relation

to each topic.

74 A qualification can now only be specified in the Apprenticeship Standard if it is: a mandatory requirement set by the regulator in the occupational area to which the Standard relates; required for professional registration; used as a hard sift when applying for jobs in the occupation relating to the Standard and, without it, an apprentice would be at a significant disadvantage as they progress in their career. Mandatory qualifications have to be regulated. If a qualification is not specifically referenced in the Standard, then any study towards it is deemed to be a choice by the provider about how to deliver the on-programme part of the apprenticeship.


All supervisors must have all the generic skills, knowledge and behaviour (set out below)

in addition to one operational area in line with their specialist function.

Table 6: Extract from Hospitality supervisor apprenticeship standard (Core generic skills, knowledge and behaviour)75

Knowledge and Understanding (Know it)

Skills

(Show it)

Behaviours

(Live it)

Business

People

Customers Understand the

importance of customer

profiles, how to build

them and understand

how this enables the

business to meet their

needs profitably and in

line with business / brand

standards

Coordinate the team

to deliver to customers

according to their needs in

line with business/brand

standards, enhancing

their experience where

appropriate

Proactively encourage a

customer-centric culture

Know the marketing

and sales activities of

the business and how to

support them to achieve

the desired outcome

Implement sales and

marketing strategies in

own area, ensuring the

team are fully supported

to deliver them. Make

suggestions for future

sales and marketing

activities within area of

responsibility

Be proactive in supporting

sales and marketing

activities

Understand the

requirements of the

product and brand

standards of the business

Actively promote the

brand and product; and

consistently maintain the

highest standards

Demonstrate a belief in

the brand and product the

business offers

Leadership

75 Further details available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/411921/HOSPITALITY_-_Hospitality_Supervisor.pdf


Communicating what is to be measured

There have been strong traditions, and often tight design rules, for specifying and

communicating what is to be measured by particular types of qualification. For

example, criterion-referencing has been a common approach in unit-based vocational

qualifications. Pre-determined, explicit performance criteria, against which candidate

performance is assessed, are used to communicate to the learner and assessor detailed

information about what has to be demonstrated and achieved.

However, with the replacement of the Qualifications and Credit Framework (QCF) with

the RQF, a new flexibility has been introduced by Ofqual76. Awarding Organisations

can now decide on their own approach (and Ofqual will make a judgement on its

appropriateness).

You will still see qualifications in the system that have used the QCF design rules (if they

are shown to be sufficiently valid). But, there is now an opportunity to look again at how

proficiency might be specified.

Forgetting any rules and starting with a blank piece of paper, how do you think your qualification’s proficiency can be best specified and communicated?

This is not a simple process. This is because there are different ways of thinking about

proficiency specifications: different theories representing different traditions and modes

of analysis. It will not be your decision alone. There will need to be wider discussion

within your Awarding Organisation, across its qualification offer, and with others, to

determine how to proceed.

At the time of writing, Newton (2017) is exploring possibilities for developing an

over-arching analysis (meta-analysis) drawing upon these different theories, to help

in developing proficiency specifications (see Figure 12 for an overview of his four-

dimensional model).

Each dimension in his model (presented in sequential logic) poses particular questions.

But, in order to develop the proficiency specification, an ‘integrated and iterative process

involving stakeholders is required in establishing a shared understanding of the essential

characteristics of the qualification’s proficiency target’. Furthermore, people involved

at different points in a qualification’s lifecycle may benefit from information being

presented in different ways, for example, graphically, rather than in writing.

76 Ofqual was finding that the QCF rules were placing too much attention on structure for the design of vocational qualifications in England and not enough on validity. Also, it was felt that the rules were not flexible enough to meet the various needs of those using vocational qualifications.


ProficiencySpecification

Qualitative and QuantitativeRepresentation of the

Target Proficiency

Delineate the target proficiency

Socio-politics

Communicate its (essential) characteristics

Semiotics

IntegratedProcess

SequencialLogic

Identify its (particular) features

Psychology

Conceptualise its (general) structure

Philosophy

Figure 12: Proficiency specification: four-dimensional model (Newton77)

For further explanation, see Newton’s presentation notes

(available through Ofqual or FAB library)

77 Newton, P. E. Specifying the proficiency that a qualification needs to measure. JVET Conference, 7-9th July 2017, Worcester College, Oxford. Ofqual. Check for up-dates and future research papers on Ofqual’s website if you wish to further explore these ideas.

The four-dimensional model suggests that we need answers to questions like these whenever we construct a proficiency specification.

Socio-political mode What does the qualification need to do? And therefore what proficiency does the qualification need to measure?Philosophical mode How should we conceptualise ‘possessing’ the proficiency? How should we conceptualise components of the proficiency construct? How should we conceptualise points on the proficiency scale?Psychological mode What are the ‘contents’ of the proficiency construct and scale?Semiotic mode How should we communicate the proficiency construct and scale?


3.4 Identify your approach to assessment (strategy)

Assessment is essentially a procedure for measuring what each qualification candidate

knows, understands and can do. The procedure comprises the specifications that make

explicit the features and processes that will remain constant from one occasion to the

next. In addition to having a proficiency specification (derived from the target proficiency)

which sets out the abilities to be measured, it is also important to specify the proficiency

scales – the features that characterise the different levels of the target proficiency.

Assessment should be seen as an important initial design consideration rather than an

afterthought. Work through the construction chain steps (repeated from chapter 2 in

figure 13) to identify the key features and processes that will establish confidence in the

claims you want to make for your qualification and inferences about its attainment.

Your qualification’s objective, target proficiency and the needs of employers and other end-users of the assessment should be central to your thinking.

Although the construction chain is presented in linear format, you will find that

reconciling demands when thinking through assessment possibilities is an iterative

process. For example, in making design decisions at step 1 (clarification), some of these

decisions will be dependent on feedback from those who will be interpreting the results

(at the interpretation step), such as candidates, employers, university gatekeepers and/or

funding agency.

Along the intervening steps of the construction chain, those involved -candidate,

assessor, aggregator - all need to be clear about the measurement objectives and

methods and processes to be used.

It is recommended that you read through how to develop and deliver a qualification

to inform your thinking. See Chapters 4 & 5.


Examples of things to think about

Identify features and processes to enable multiple performances to be elicited from each candidate and to provide a sample of evidence of proficiency.

Identify features and processes to evaluate performances in the sample in terms of what it implies about candidate proficiency

Clarify measurement objectives

Clarification Step

Elicitation Step

Evaluation Step

Identify features and processes to enable measurement results to be interpeted by those for whom they have been provided.

Interpretation Step

What exactly do you hope to achieve through measuring candidates’ performances?

Whose needs must be satisfied?

What type/s of assessment methods, and how many, will you specify?

What aspects of the target proficiency will each assessment task cover?

What will be the processes and conditions required in administrating the tasks?

Who will be involved to ensure the sample evidence of proficiency is produced?

What criteria will be used to evaluate candidates’ performances and/or system of marking (if any)?

Will assessment components be weighted?

What quality assurance and control procedures need to be in place to ensure the evaluation reflects the true quality of each candidate’s performance?

Identify features and processes to enable the set of performance evaluations for each candidate to be combined and transformed into an overall measurement result

Combination Step

What will be the rules for aggregating results from the individual evaluations?

How will the overall pass/fail and any higher classes of pass (grades) be decided (if not predetermined, e.g. in a rating scale)?

Will there be any compensation allowed?

What procedures, if any, will be adopted for borderline/special cases?

How will results be presented – in what form?

How will they be made available, to whom and when?

What information will be provided to help in understanding the results?

Figure 13: Steps in the measurement construction chain to inform design of assessment


Think through: the types of assessment methods required; the processes involved in

assessing the performances of a candidate who will take your qualification and the

conditions under which assessment takes place; the method by which the overall

assessment result will be arrived at; how the result will be communicated to facilitate

interpretation.

See Box 13 for some key features to consider when developing an assessment

methodology.

Box 13: Key features of an assessment methodology

Key features of an assessment methodology

• Allows the generation of evidence that is consistent with the objective

of the qualification and targets only abilities drawn from the proficiency

specification

• Makes it possible for candidates to show (and you to measure) the extent to

which they have the required abilities

• Pitches the assessment at the level of demand consistent with the

qualification objective and level

• Creates the opportunity to elicit sufficient evidence to demonstrate

attainment, and where differentiated attainment in the assessment is

recognised (grading), sufficient opportunity to demonstrate higher degrees of

attainment within the level of demand

• Enables differentiated levels of performance in a clear and consistent way

based solely on the content of the proficiency specification

• Invites candidates to make connections and apply their learning in

demonstrating attainment

• Enables the full range of learners to take the qualification and access

assessment which is free of any unnecessary barriers to attainment

• Encourages a positive approach to assessment, making sure the amount of

assessment isn’t unnecessarily burdensome

• Ensures that any choice of routes through the qualification are assessed in

ways that are comparable in terms of the levels of demand

• Details exactly how the assessment will be monitored and kept under review

• Identifies the quality control and quality assurance needs

• Identifies the key capability and capacity needed to support the lifecycle

processes

• Has assessment objectives and expectations that are transparent, clear of

ambiguity and accessible, to help encourage a focus on the higher order

knowledge, skills and understanding central to the qualification objective,

rather than an over-focus on tasks.


The processes that sit behind each of the features in Box 13 are all evidence of how you

can ensure your assessments are, and remain, valid: whether they are assessing the right

thing, in the right way, to produce accurate and useful assessment results’ (Newton,

2017).

In building confidence in the results of assessment, potential conflict of interest should

not be forgotten. For example, achieving good qualification results has implications for

some centres in terms of performance ratings and funding. Think through the steps that

will need to be taken to ensure compliance with quality assurance requirements for the

assessments, for example, that teachers do not assess and internally verify their own

work.

It will be your insight, experience and expertise as a qualification manager that will inform

the development of a viable, initial assessment proposition which can then be tested

and refined. How the decision was arrived at, the methods used to consult and test out

the proposition and the eventual justification for the choice, will all be evidence of the

processes employed to ensure the validity of the chosen assessment methodology.

How do I select the most appropriate assessment method/s for my qualification?

You will recall from chapter 2 that:

• Some methods are considered to be more appropriate to skills-based assessment:

observation of performance in a work environment; simulations; trade or skills tests;

practical assignments

• Other methods better suit the assessment of cognitive abilities: examinations, tests

and multiple-choice assessments; presentations; written projects and assignments;

reflective journals; interviewing, oral and written questions

• Many assessment interventions for qualifications use more than one method of

assessment to ensure that the full extent of the target proficiency of the qualification

is assessed.

Refer to your proficiency specification to consider the range of different abilities included

and then decide what methods could be suitable and whether you need more than one.

For more detailed description of different methods, their pros and cons,

see Appendix A.


A table can be used to assist this process. For example:

In column 1 In column 2 In column 3

List the different types of abilities to be assessed

List the method/s that could be used to assess these abilities; and check the validity of each method – that it is capable of assessing the particular ability in question

Rate the reliability of those methods judged to be valid, using a scale (e.g. from poor to very good) – the extent to which it is capable of providing consistency in results

You could add a further column to note factors that might influence choice of method

relating, for example to: how your qualification users and funders might perceive it;

manageability and development costs.

At the same time, bear in mind Reasonable Adjustments that could be made for assessing

candidates with disabilities.

Reasonable Adjustments are adjustments made to an assessment for a qualification

so as to enable a disabled learner to demonstrate his or her knowledge, skills and

understanding to the levels of attainment required by the specification for that

qualification.

These adjustments must not affect the validity and reliability of assessment, but the

needs of all potential candidates should be considered when designing and developing

the assessment. Think through the implications of written assessment methods, health &

safety for practical assessments and how materials for different types of assessment may

be presented, for example: how language could be modified and materials presented in a

way that they can be used with assistive technology such as screen reading software, in

Braille, audio format and British Sign Language.

How many methods?

You may find that one assessment method can assess more than one ability, for example,

both recall and comprehension of information related specifically to your qualification’s

subject matter. In other cases, you will need more than one method to assess different

types of abilities. You will also need to take into account, how many times a candidate has

to demonstrate a particular ability to be confident in the results of the assessment: the

same ability, or its application in a different way or situation.

The more methods that are used, the greater the number of assessment hurdles presented to candidates. It is preferable, therefore, to limit assessments to just those that are required to do the job.

Where you plan to use different assessment methods, the assessment strategy should

involve looking at how these work together.


Influence of qualification users on choice of methods

In deciding on assessment methods, you may experience pressure from some user groups,

and strong reasons, with regard to the approach to assessment. For example, some may

want:

• High stakes examination made up of unseen exam questions, set and assessed by

an independent third party, because the users want the level of reassurance and

confidence they view these conditions as having

• On-the-job, live observation of action being carried out to demonstrate competence.

Be aware of what the gatekeepers will expect to see: those who control progression

routes that could be accessed by achievers of your qualification. What expectations do

they have about how it should have been assessed?

Some may want different weighting78 to be given to a specific type of assessment (or

learning outcome and/or topic) to reflect what they feel is particularly important in their

sector.

If you want to claim that your qualification is accepted by the user groups, these

expectations will need to be taken into account. You may need to engage with these

groups to explain your strategy and argue your case.

When it comes to an apprenticeship in England, some of the end-point assessment

decisions will have been taken by the employers developing the Apprenticeship Standard

and Assessment Plan (see Box 14, for expectations with regard to the content of

assessment plans). In this case, you will need to work with this plan and those who have

developed it.

78 Weighting: the percentage of the total marks in an assessment scheme allocated to a particular part of that assessment, such as: 50% practical assignments; 30% examination; 20% synoptic assessment or particular learning outcomes


Box 14: Assessment plans for apprenticeships in England

Assessment Plans for Apprenticeships in England

Assessment Plans are designed by the trailblazer group to help reassure employers that apprentices completing a Standard will have a comparable set of skills, and that those skills will be transferable between employers.

The ‘What’ section of an Assessment Plan should contain sufficient information to help determine:

• What each assessment method needs to assess - employers are asked to be

explicit about which knowledge, skills and/or behaviours in the published

standard each assessment method is testing

• Who sets the parameters for each assessment method – will this be individual

Assessment Organisations or are the parameters set in the EPA plan itself?

The ‘How’ section should contain sufficient information to help determine:

• How the ‘what’ is to be assessed - what methods will be used, over what time

span the EPA will take place, and the duration of each assessment method. For

example, will it be assessment over one day or one week?

• What assessment tools will be used

• What the apprentice will have to do exactly

• Where the assessment will take place

• How assessment will be undertaken by all employers, regardless of size.

The ‘Who’ section should contain sufficient information to determine:

• Who will carry out the assessment, and how they will demonstrate

independence from those conducting the training

• Whether several assessors are needed, such as in a panel arrangement, and

how they will work together and who has the casting vote

• Any minimum requirements for assessors, such as sector knowledge, assessor

qualification, experience.


External or internal assessment. How do I choose?

The differentiators that really determine whether assessment is internal or external are

more to do with the design of the assessment task, implementation and the conditions of

assessment, than the method itself. For example:

• External assessments are normally written and set by Awarding Organisations,

administered under controlled conditions and externally marked or assessed by a

third party not connected to the candidate or the teacher (normally the Awarding

Organisation). If the assessment counts towards certification, occurs infrequently

with the questions only seen during the assessment, and involves third parties, this

will feel quite high-stakes to the learner

• Internal assessments are normally written, set and marked by assessors in the

centre delivering the qualification. Locally-devised and assessed tasks are then

quality assured using the ‘assessor - internal verifier - external verifier’ style of

approach which normally starts with a centre approval process run by the Awarding

Organisation.

Many qualifications actually use a mix of internal and external assessments. To decide

what type or how much of each should be used, look at the claims you want candidates to

be able to make on successfully achieving the qualification and the views of other users.

Gatekeepers may express more confidence in the claims and inferences made for a

qualification, and feel more secure about its integrity, if it has a high level of external

assessment. For example, it may be felt that the risks will be reduced of any conflicts of

interest entering into the assessment process.

There are, however, ways of introducing some of the characteristics of external

assessment into internal assessment. For example, you might want to control possible

variations in assessments by supplying centres with assessment tasks or allowing them

to draw down assessments, such as assignments, from centrally-held resources. That way

you are aware of the quality of the assessments that are made available to candidates.

You may decide to offer a pre-implementation check on centre-devised assessments;

again this introduces some quality assurance and external control.


What about grading?

The availability of grading within a qualification essentially signals the intention to

acknowledge and reward different levels of performance or achievement above a pass.

If the grading is to have meaning and purpose, it must have a clear relationship with the

qualification’s objective and target proficiency. All users of the qualification must be able

to interpret the grades in the same way79.

Not all vocational qualifications are graded above a pass. Typically, when assessing work-

based performances, all the assessment criteria must be met for each learning outcome.

Assessments are carried out when the candidate is ready and if successful the overall

result (combined from each assessment) is usually recorded as a passing grade80.

There is no compensation rule81.

However, many more vocational qualifications are now being designed to be graded.

Where criterion referencing is used for assessing work-based performances, grading may

be pre-determined by using additional criteria that have to be met for a particular grade

such as merit or distinction.

Where the qualification includes an assessment component such as an examination,

grading is likely to involve an awarding committee established by the Awarding

Organisation to identify cut-off scores (grade boundaries) after the assessment results

have been aggregated.

If your qualification is to be graded, you will need to develop a valid grading methodology

which should include robust quality processes and procedures to ensure grading is

applied consistently and reliably.

For examples of grading and awarding processes, see chapters 4 & 5.

79 Seeley, M (1991) The Mismatch between Assessment and Grading, Education Leadership 52 (2) 4-6. In Guskey, T and Jung, L (2013), Answers to Essential Questions about Standards, Assessment and Grading, & Reporting, Sage Publications, London/Corwin, California

80 This approach is based on a non-compensatory aggregation model.81 Compensation is a marking process that allows for marginal error in a limited number of assessment areas on the basis that overall performance remains

sufficient to merit the award of the qualification. A compensation rule allows a candidate who is just below the pass level in one component to compensate for this shortcoming by a correspondingly above-minimum mark in another component.


Once you have decided on the most appropriate assessment methods and degree of

externality (if any), it will be possible to identify:

• The conditions under which these will be applied, including quality control and quality

assurance procedures

• Who will need to be involved in the administration of the assessments and quality

assurance at Awarding Organisation level and within the delivery centre

• The relevant supporting documentation required such as marking schemes,

grading criteria, guidance on the aggregation of results, sufficient sampling of

internal assessments and/or examination scripts, authentication of candidate’s

work and so on.

In developing your assessment strategy, you will have engaged in a cyclical process. You

may have found yourself being pulled away from initial choices and having to compromise

on certain things. This is because at the end of the day, assessment has to be manageable,

scalable and affordable as well.

However, having done all this thinking, you will now be well-placed to develop your

measurement specification.


3.5 Specify how your qualification will be assessed

Use the outcomes from developing your assessment strategy to produce your

measurement specification. This will help to ensure the validity and reliability of the

assessment and that coverage of the target proficiency is:

• Well-balanced and thought through

• Related to the objective of your qualification

• Consistent from year to year82.

It offers a further opportunity to check that the abilities in the proficiency specification

can be assessed in the way that you thought.

Box 15: Key features of a measurement specification

Key features of a measurement specification

The overall specification should:

» Set the assessment objectives in context

» Describe how abilities and topics are to be assessed

» Specify how to use the assessment method/s, supported by a rationale

» Detail the nature of the assessment conditions, frequency of assessments and

resit procedures (if needed)

» Detail how performance can be differentiated and grading (if used); and how

to apply marking schemes (if any)

» Detail how the assessments are to be administered, dealing with the practical

issues

» State the measures to be taken to ensure evidence is authentic, current, and

sufficient

» Describe how record-keeping and quality assurance and control processes

should be applied and why

» Set out any Reasonable Adjustments and Special Considerations that are

possible

» Identify the necessary stimulus and support materials needed, along with any

training requirements for assessors

» Identify the review points for the assessment process and practice.

82 Ward, C. (1980) Designing a scheme of assessment. Cheltenham: Stanley Thornes (Publishers) Ltd


The nature of the chosen assessment method will determine more exactly the level

of detail to be provided in the specification. For example, you may need to determine

the weighting for each of the main topics and abilities to be assessed and the forms of

response required, e.g. written answers, drawings, calculations. The specification will

help to ensure that weightings are appropriate in relation, for example, to the balance

between understanding and application, and also consistent from year to year.

Where you are using more than one assessment method, you may find it useful,

if feasible, to have a specification for each method.

For key design features and factors to consider for different assessment methods,

see Appendix A.

The form of the individual specifications will be dependent upon the type of assessment

method. For example, a fairly detailed specification is possible for an objective test

(e.g. stating number of questions, how they are allocated to different abilities/topics;

instructions on how to answer the questions; time allowed; how the questions will be

marked and so on), whereas a specification for a project may be limited to stating the

assessment objectives, supported by a clear brief and guidance to ensure consistency

in marking. Each method will raise different issues, for example, potential security risks

which will need to be taken into account.

In developing the overall measurement specification, these individual specifications

should be compared to check that each assessment has a distinct or complementary

role. This will help to avoid duplication or unnecessary overlap which would add to the

assessment workload. You will need to specify how the result or grade from each method

of assessment is to be aggregated.

After a period of using the measurement specification, some amendment may be needed

to it, for example, to remove any ambiguities. But, your aim should be to produce a

specification that can remain unchanged for the life of your qualification.

This supports the reliability of assessment and helps in testing the validity of your

qualification by clearly documenting its assessment build and implementation process.

It can also contribute to business continuity, helping to develop a corporate ‘memory’ for

your qualification, and with managing risks associated with transition in personnel.


Further information

FAB offers training and webinars of relevance to this chapter. For example, webinars on:

researching international markets; the review of employer use of NOS; different forms of

assessment - the benefits and limitations; making qualifications inclusive.

Examples of resources and publications available in the FAB library:

• ISBA (2011) A guide to understanding and working with market research agencies & consultancies. London: ISBA.

• Federation for Industry Sector Skills and Standards (November 2014) Improving Apprenticeship standards through consultation. A guide and toolkit for employers. FISSS

and Employer-defined Apprenticeship standards. A toolkit for enablers and facilitators of Trailblazer groups. FISSS

• European Centre for the Development of Vocational Training (2010) Changing Qualifications: A Review of Qualifications Policies and Practices, Luxembourg:

Publications Office of the European Union.

• Ofqual (2015) Qualification and component levels.

• Newton, P. E. Specifying the proficiency that a qualification needs to measure.

JVET Conference, 7-9th July, Worcester College, Oxford.

Further references and information of relevance to this chapter can be found

in Appendix B.

111 How to develop a qualification

Chapter 4

How to develop a qualification

This chapter offers information and guidance on how to:

9 Shape your qualification and its progression

9Develop the ‘apparatus’ for measuring candidates’ performance, including qualification support materials

9 Put it all together to inform delivery of your qualification.

112How to develop a qualification

If you have engaged in the processes described in the previous chapter, you will have:

• A proficiency specification that sets out clearly what achievers of your qualification

should know, understand and be able to do

• A measurement specification which clarifies assessment objectives and describes all

aspects of how candidates will be assessed.

These provide the starting point for moving on to the development stage of your

qualification’s lifecycle.

Exactly what you need to produce will largely depend on the method/s of assessment you

have chosen to use.

For an insight into how your qualification might be delivered, to provide the context

for its development, see Chapter 5. It is recommended that you read Chapter 5 before

working through this chapter.


• Qualifications need to have a clear structure for the assessment of skills, knowledge and understanding and take into account progression

• Progression can be vertical - from lower to higher levels in the same subject area/suite; or lateral, for example, by combining components with those of a complementary subject at the same level to gain more breadth

• Benchmarking the qualification to skills, knowledge and understanding required by employers, the professions and higher education, will make clear how the qualification may be used for career development

• The assessment tasks developed to elicit performances from each candidate will depend on the abilities to be assessed and the nature of the work/subject; they may be set internally and/or externally

• The type of marking scheme, if any, will depend on the nature of the assessment task and the required responses

• In developing assessment tasks, it is important to keep in mind their future use and security arrangements; analyses of how they perform informs on-going development

• Technical reports on all aspects of the development process will inform evaluation of the qualification and provide evidence to support the validity argument

• Assessment tasks must have written documentation to support their proper use

• A range of support material will provide further detail and examples to aid understanding of what is required; exemplar responses to the tasks are particularly helpful to those taking and delivering the qualification

• An accurate and clear qualification specification will be central to the successful delivery of the qualification, together with other support and resources made available to candidates and key personnel.


4.1 Shape the qualification and its progression

Remind yourself of your qualification’s overall objective and what users of your

qualification are looking for.

Draw on findings from your market research and consultations with stakeholders, to

ensure you have a clear understanding of your qualification’s market and the purpose/s to

which it may be put.

4.1.1 Qualification structure

Firstly, check on funding rules and with the respective regulator/s for any conditions

relating to the structure of qualifications.

For example

In Scotland, Awarding Bodies can choose to offer a qualification based on approved

qualification products83, using the approved qualification structure. Some have

multiple pathways from which Awarding Bodies can choose. These can be covered by

a single qualification or split into separate qualifications.

Where there are no approved products, Awarding Bodies can design their own

qualification structures (providing they provide SQA Accreditation with a rationale

and evidence that the qualification is fit for purpose in Scotland)84.

In the previous chapter, mention was made of new flexibilities for the design of regulated

qualifications that have resulted from the QCF85 being replaced by the Regulated

Qualifications Framework (RQF).

Qualifications in the QCF were required to use the QCF unit pro forma. This had

five main components: title, learning outcomes, assessment criteria, level and credit

value. It allowed no design features to be added to this standard format, although

there could be a choice of assessment methods/tasks.

Learning outcomes specified the abilities (knowledge, understanding and/or practical

ability) that were to be assumed of learners after the Awarding Organisation

awarded credit. Assessment criteria specified what it was that had to be checked -

what the learners could do - before the Awarding Organisation awarded credit.

Size was communicated by the assignment of credit value and entitling a qualification

as an Award (the smallest), Certificate or Diploma. It was possible to offer an Award

comprising one unit or module, but the larger qualifications could have

83 ‘Qualification products’ are the documents developed by a Sector Skills Organisation and approved by SQA Accreditation. These products must provide: a qualification structure that includes: qualification title; number and combination of units; any prescribed pathways for sub-functions; an assessment strategy, with guidance for those who will deliver, assess and quality assure it (assessment principles). They may also include: an SCQF credit rating statement: SCQF level and credit value for the qualification and component units; a signposting document indicating opportunities for assessment of Workplace Core Skills in the assessment of the qualification. SQA Accreditation (March 2016), Developing Scottish Qualification Products for Approval: A Guide for Standard Setting Organisations. Available at: http://accreditation.sqa.org.uk/accreditation/About_Us/External_Stakeholders/Developing_Scottish_Qualification_Products_for_Approval_Final_March_2016.pd

84 SQA Accreditation (February 2017) Developing Qualifications for Accreditation. A Guide for Awarding Bodies. Available at: http://accreditation.sqa.org.uk/accreditation/accreditationfiles/Quals/Developing_Qualifications_for_Accreditation__Final_.pdf

85 The rules for the Qualifications Credit Framework (QCF) were withdrawn on 30 September 2015. QCA (2008) Regulatory Arrangements for the QCF


several components, offering scope for a greater range of options towards their

achievement. Rules of combination dictated which units/modules were mandatory

and which were optional.

If you search the registers of regulated qualifications, you will see that for some

qualifications, Awarding Organisations have continued to use the QCF structure.

This is because it suits the qualification’s objective, proficiency target and the needs

of their sector.

Table 7 illustrates one qualification on the RQF that is structured in units based on

professional standards for teachers, tutors and trainers in education and training.

Table 7: Extracts from Level 4 Certificate in Education and Training (RQF) offered by several Awarding Organisations

Extract from Mandatory Unit: Assessing Learners in Education and Training (Level 4, 6 credits)

Learning Outcome (The Learner will):

Assessment Criterion (The Learner can):

1 Be able to use types and methods of assessment to meet the needs of individual learners

1.1 Explain the purpose of types of assessment used in education and training

1.2 Analyse the effectiveness of assessment methods in relation to meeting the individual needs of learners

1.3 Use types and methods of assessment to meet the individual needs of learners

1.4 Use peer and self-assessment to promote learners’ involvement and personal responsibility in the assessment for, and of, their learning

1.5 Use questioning and feedback to contribute to the assessment process.

2 Be able to carry out assessments in accordance with internal and external requirements

2.1 Identify the internal and external assessment requirements and related procedures for learning programmes

2.2 Use assessment types and methods to enable learners to produce assessment evidence that is valid, reliable, sufficient, authentic and current

2.3 Conduct assessments in line with internal and external requirements.

Extract from Mandatory Unit: Understanding Roles, Responsibilities and Relationships in Education and Training (Level 3, 3 credits)

Learning Outcome (The learner will):

Assessment Criterion (The learner can):

1 Understand the teaching role and responsibilities in education and training

1.1 Explain the teaching role and responsibilities in education and training

1.2 Summarise key aspects of legislation, regulatory requirements and codes of practice relating to own role and responsibilities

1.3 Explain ways to promote equality and value diversity

1.4 Explain why it is important to identify and meet individual needs.

2 Understand ways to maintain a safe and supportive environment

2.1 Explain ways to maintain a safe and supportive learning environment

2.2 Explain why it is important to promote appropriate behaviour and respect for others.


Each unit focuses on particular areas of skills and/or knowledge and understanding,

with learning outcomes stated and associated sets of assessment criteria against which

candidates will be assessed. Both mandatory units (all at Level 4) and optional units (with

some at Level 3) are included in this qualification.

Technical qualifications in England

Technical qualifications are a form of vocational qualification specifically for use

in England with 14-16 year olds or post-16 learners studying in colleges or on

apprenticeships. They have particular features and rules for their structure.

In shaping a technical qualification, you will need to refer to the target proficiency which

has been informed by the Standard, and take account of the features described in Box 16.

But, it is still possible to be creative.

For example

City & Guilds present the learning content for its Technical qualifications in ‘units’

structured in terms of learning outcomes which are supported by learning topics (not

as units of assessment).

This structure is designed to encourage a focus on teaching rather than assessment

of “everything”. The candidates will be expected to study and develop the knowledge

and skills required to achieve the learning outcomes, so that at the point of

assessment they are able to deal with whatever the assessment requires of them.

The Assessment Objectives are described in a generic way, but contextualised to the

content of the respective qualification when it comes to its assessment through an

examination and assignment.

(City & Guilds, April 2016: 4-6 4)

86

86 City & Guilds -Teaching, Learning and Assessment. Version 1.0. April 2016


Box 16: Features of Technical Certificates and Tech Levels in England87

Technical Certificates and Tech Levels for school/college performance tables5

A Purpose Statement for a Tech Level or a Technical Certificate must explicitly

state:

• The industry, occupation or technical role for which they prepare the student

• Subjects which might complement the qualification

• Whether the qualification is part of a suite and why a student would

choose this qualification instead of others in the suite; and also, explain other

sizes of qualification that are available, and describe the ‘added value’ of

larger sized qualifications. For example, a larger qualification may prepare

the student for different job roles, or a license to practice, that a smaller

qualification does not

• Who supports the qualification, drawing from the letters of support provided

as evidence for the ‘Recognition characteristic’.

The qualification’s size must relate to: its purpose, level of study and skills and

knowledge to be developed; and to its value in terms of hours of guided learning

(rather than the Total Qualification Time).Technical Certificates must be at least

150 Guided Learning Hours (GLH) to enable completion within a one-year study

programme before progressing to a Tech Level, apprenticeship or employment;

Tech Levels must be at least 300 GLH, to provide sufficient time for meaningful

skills development that will equip a student to follow a specific occupation or

technical role.

A qualification specification must state the specific content that students must

pass to achieve it. Mandatory content and associated contribution to the overall

grade must make up at least: 40% of a Technical Certificate; 40% of a Tech Level.

Technical qualifications must include synoptic content that helps develop a

student’s appreciation and understanding of connections between different

elements of a body of knowledge and skills, relating to a subject or occupation/

occupational group.

The proportion of a qualification’s content that is externally assessed, and the

associated contribution to the overall grade, should reflect the purpose and

content of the qualification and must be at least: 25% of a Technical Certificate;

30% of a Tech Level.

A distinction, merit, pass, fail structure or a more detailed grading scale must

be applied to the overall qualification and reflect attainment across its content,

with difference between each grade detailed and clear. The degree of stretch and

amount of assessment contributing to overall grade must be appropriate for the

qualification’s level and size.

87 Department for Education Guidance for Awarding Organisations submitting technical and applied qualifications for school and college performance tables: https://www.gov.uk/government/publications/14-to-19-technical-and-applied-qualifications-technical-guidance


Box 17: Technical qualifications: Progression

Progression in technical qualifications (England)

There are three levels of technical qualifications: Technical Awards

(level 1 and 2); Technical certificates (Level 2); Tech Levels (Level 3)

• Technical Awards: broad Level 1 and 2 qualifications for those aged 14-16

focusing on applied study of an industry sector or occupational group, or

acquisition of practical or technical skills. They are designed to fulfil entry

requirements for both academic and vocational study post-16, alongside

GCSEs

• Technical Certificates: Intermediate (Level 2) qualifications for post-

16 students wishing to specialise in a specific industry, occupation or

occupational group. They equip students with specialist knowledge and skills,

enabling entry to an Apprenticeship, employment in occupations where

employers recognise entry at Level 2, or where needed to progress to a Tech

Level

• Tech Levels: Advanced (Level 3) technical qualifications, on a par with A levels

and recognised by employers, for post-16 students wishing to specialise in a

specific industry, occupation or occupational group. They equip a student with

specialist knowledge and skills, enabling entry to an Apprenticeship or other

employment, or progression to a related higher education course.

• Technical Certificates and Tech Levels provide a ‘license to practice’ or

exemption from professional exams.

At the time of writing, technical qualifications are in the early stages of development, so

please check for up-dates from the Department for Education and Ofqual.

If you have a free-hand in terms of how you structure your vocational qualification,

keep in mind the needs of learners and the sector it serves. What structure would best

communicate what needs to be covered and achieved? Have a look at other qualifications

that have recently been included in the registers, to see examples of alternative

structures.


4.1.2 Progression

Progression should be a key factor when considering the shape of your qualification and

clearly communicated to its users.

Consider how your qualification can:

• Build upon a smaller or lower level qualification in the same subject area/suite

• Prepare learners for the next size up or higher level of qualification in the same

subject area/suite

• Link to other qualifications in the same subject or complementary area

• Offer candidates an opportunity to progressively build confidence, skills and/or

knowledge and understanding whilst working towards it, for example by including

components at different levels

• Lead to specific jobs within respective occupation/s and, where relevant, membership

of a professional body and admission to higher education.

Progression is not always vertical. It may be lateral. Learners may wish to explore

a particular aspect of the subject area in more depth, for example by taking a

component from another qualification in the same subject area, but at a higher level.

Alternatively, they could decide to broaden their learning by combining components

from a qualification with another at the same level which focuses on a different, but

complementary subject.

Progression into and within employment/the professions

Consultation with employers to elicit support for your qualification will have provided an

indication of types and levels of job for which it may provide entry, but you may need to

do more research.

Box 18: Example of how one professional body communicates progression88

The Chartered Institute of Professional Development Profession Map

To support individuals in their career development, CIPD has developed a

map that defines the building blocks of effective people management practice

across professional specialisms from entry-level roles to leadership roles. There

are 10 professional areas within the Map, including two core areas: Insights,

Strategy and Solutions; and Leading HR. For each area, the Map describes what

people need to do (the activities) and what they need to know at four bands of

professional competence, as well as outlining the predominant behaviours that

underpin good practice regardless of specialism.

The map clearly describes what is needed for professional membership.

88 The CIPD Profession Map. Available at: https://www.cipd.co.uk/learn/career/profession-map


Progression to higher education

If your qualification could be used for admissions to higher education, revisit outcomes

from your consultation with stakeholders to identify to its acceptability for entry into HE

course/s.

Benchmarking your qualification to a higher education framework will assist those

learners who wish to progress to higher education, by identifying skills, knowledge and

understanding that may also be covered in a degree or post-graduate programme.

For example:

The qualifications descriptors of the FHEQ89 set out outcomes for the main UK

HE qualifications at each level and demonstrate the nature of change between

levels. The qualification descriptors include a statement of intended outcomes,

achievement of which are assessed and which a student should be able to

demonstrate for the award of the qualification. They also provide a statement of

wider abilities (in terms of higher level skills and other attributes) that the typical

student could be expected to have developed. (QAA, August 2008: 10)

Figure 14: Overview of the progression landscape from Level 3 to 7 (England)90

89 QAA (August 2008) Framework for Higher Education Qualifications. Available at: http://www.qaa.ac.k/en/Publications/Documents/Academic-Credit-Framework.pdf

90 Based on UCAS Diagram, UCAS (June 2017) Progression Pathways 2017: Pathways through higher education: 11. Available at: https://www.ucas.com/progression-pathways-2017

Level

4Level

5Level

6Level

7Level

3

Going up the levels

Masters LevelApprenticeships

Bachelor's Degree (hons)

Foundation Degree

Higher NationalCertificate

Higher NationalDiploma

Top up to Bachelor’s Degree (Hons)

Top up to Bachelor’sDegree (Hons)

Foundation Year

Higher apprenticeship linked to an HE courseEmployer-fundedTop-up provision

Degree Apprenticeship

Higher apprenticeshipEmployer-funded Top-up provision


Broadly speaking: Level 3 prepares learners for entry into HE; Level 4 supports

progression to the second year of an undergraduate programme; Level 5, to the final year;

and Level 6 supports progression to a Master’s Degree.

UCAS offers a range of resources that can assist in considering progression into and

through HE, including a search facility for course entry requirements and publications.

For example:

• The UCAS Tariff 9 provides information about qualifications deemed to be

appropriate for admissions to higher education and a way of comparing your

qualification with others

• The Qualification Information Profiles (QIPs) have contextual information about

qualifications, such as structure and assessment approach, in a common format.

If your qualification is included in the DfE Performance Tables in England (it is a Tech

Level, for example), it will automatically be added to the Tariff. Otherwise, it will need to

be nominated, for example, by a school or college; you can also make UCAS aware of any

new qualification. 91

91 UCAS Tariff Points can be used by Higher Education to communicate their entry requirements to prospective students (not all do so). Using information from the UK regulators, (i.e. qualification specifications), the number of Tariff points is calculated by multiplying qualification size (4 bands based on guided or notional learning hours) by grade calculation (12 grade bands). The new Tariff can be applied to the full range of Level 3/SCQF Level 6 qualifications. This means that many more vocational qualifications can now be included, supporting widening access and participation. Further information available at: https://www.ucas.com/corporate/about-us/our-work-policy-community/new-ucas-tariff-proposal-process-notify-us-new-qualifications


4.2 Develop the assessment apparatus, including support materials

4.2.1 Develop the assessment tasks

Assessment tasks are used to elicit multiple performances from each candidate to provide a sample of evidence of proficiency.

This means that there needs to be sufficient tasks available to enable candidates to

demonstrate the abilities required by the target proficiency.

All assessments for your qualification must have written documentation that sets out the

criteria for assessing the required knowledge, skills, understandings and behaviours (the

proficiency construct).

Documentation should also describe, where appropriate, what different levels of those

abilities look like (the proficiency scale) and the quality assurance procedures to ensure

fit for purpose assessments.

Assessment tasks that elicit naturally-occurring evidence

If your qualification involves internally assessing naturally occurring evidence, gathered

together in a portfolio, the tasks will be varied. They will be dependent upon the nature

of the work and the contexts in which opportunities occur for assessing the candidate’s

abilities.

Ensuring sufficient and appropriate tasks are available will be the responsibility of

those delivering the qualification. It may be necessary for them to create additional

opportunities for assessment where the candidate’s normal work does not allow for

all the requirements of the specification to be met.

Quality control and quality assurance

Assessors, internal and external verifiers/moderators must have appropriate expertise

and experience, training and support, to fulfil their roles and responsibilities.

For examples of support for assessment and quality assurance personnel,

see Chapter 5.

Quality control and quality assurance relating to specific assessment tasks will depend on

the nature of the tasks. Some examples could include:

• Providing clear briefs for the assessment of performance in relation to the target

proficiency and controls to secure the reliability of assessment decisions

• Providing detail of what to look for in observations; agreeing the form of recording;

training of observers, to minimise their influence on what is being observed/candidate

behaviour; avoiding overly-staged performance


• Defining the purpose and amount of structure required for tasks that involve oral

questioning, and how answers are to be recorded, to ensure consistency in approach.

For example, providing set questions and follow-up probes for interviews, training in

techniques for professional discussions and on how to avoid bias/leading questions,

to support consistent assessor behaviour

• Defining the range of suitable products/artefacts from different contexts, the amount

of autonomy and support allowed; agreeing on tools, materials and equipment to be

available for use (these must reflect up-to-date practice in the respective sector); and

any time limits.

Other types of assessment tasks

It has been suggested, previously, that vocational qualifications may not solely rely on

assessment tasks that elicit naturally occurring evidence.

They can include tasks specifically-designed for assessing particular abilities outlined in

the proficiency specification. These may include assessment tasks set by the centre/third

party and/or the Awarding Organisation.

There is not space to go into the detail of every type of assessment task. So, look at the

assessment tasks, and associated materials, for other qualifications offered by your

Awarding Organisation and see Appendix A for some of the factors to be taken into

account for developing a range of different assessment tasks.

In this section of the handbook, some of the factors to consider will be related to:

assignments; written examinations/tests.

In the example shown in Box 19, assessment tasks include an examination and a practical

assignment in addition to a unit involving the assessment of naturally-occurring evidence.

92Box 19: Example of a qualification with a mix of assessment tasks

Example of qualification with a mixed task approach

A qualification structured in 3 mandatory units has:

• Two units internally-assessed, with one using a practical assignment set

by the Awarding Organisation. Conditions under which the assignment is

undertaken (within a real or realistic working environment11) are specified in

the unit

• One knowledge-based unit externally assessed by an examination marked by

the Awarding Organisation. The exam consists of different types of questions:

multiple choice; short-answer; one question inviting an extended response

Centre assessors mark the practical assignment using marking criteria and

guidance provided by the Awarding Organisation.

92 Broadly speaking, a Realistic Working Environment (RWE) is one that is managed as a real work situation, with assessment carried out under realistic work pressures. Tasks should be completed in a way and timescale that would be acceptable in the work situation represented, and meet expectations for productivity. The range of services, products, tools, materials and equipment should be up-to-date and available. Account must be taken of any legislation relating to the type of work carried out. Candidates must be given workplace responsibilities to enable them to meet the requirements


Assignments

An assignment can be practical, theoretical or a mix of the two (see Box 19 for a practical

example). It could be externally-set by your Awarding Organisation, or internally-set by

the centre/third party.

When an assignment is internally-set, you will need to produce guidance for centres/third

parties on how it should be structured (a template could be provided), how it should be

marked and, where applicable, graded.

When externally-set, you should specify what it covers and requires in relation to your

qualification’s target proficiency: the abilities to be assessed; the topics/activities to be

covered; and any relevant conditions.

Conditions may relate to: the time frame for completion; resources to be made available;

degree of supervision; whether the assignment needs to be completed in a real or

realistic working environment.

Box 20: Example of an approach to a practical assignment

Example of an approach to a practical assignment

A practical assignment could be designed to focus on the application of practical

skills, and knowledge and understanding, in producing a product relating to a

work situation. It could have three sections: planning; development; evaluation.

Candidates could be provided with a brief, or choice of briefs, and expected to

demonstrate that they can:

• Interpret the brief

• Gather information to aid understanding of the brief

• Make decisions on the type of product to produce which will meet the brief

• Select and manage resources

• Produce the product

• Evaluate how they have tackled the assignment.

Evidence of how candidates have demonstrated required abilities includes:

• A plan

• A product

• Documentation of processes underpinning the practical activities

• A written or oral evaluation report.

The assignment developer would specify: the title; objectives of the assignment;

qualification topic/s covered and the abilities to be assessed; instructions for

doing the assignment. Permissible resources and support may be stated. The

structure of the plan and evaluation report, and what these should cover, could

also be specified (and possibly word lengths).

An assignment would typically include a cover sheet which the candidate signs to

confirm identity and that it is all their own work.


A marking scheme will need to be developed, including documentation for recording

marks.

At a simple level, an assignment’s marking scheme could specify the total marks

available for each assessment objective or section of the assignment. These marks

might be allocated evenly across the assessment objectives, or sections of the

assignment, or they might be weighted differently to reflect, for example, the

respective complexity and/or scale of the activities.

Further details about different forms of marking and weighting can be found below.

Quality assurance and quality control

Before an assignment is used, the quality assurance process should ensure that it is fit for

purpose. For example, that it:

• Is designed only to assess abilities in the proficiency specification (do not require

additional cognitive abilities in understanding what is required)

• Is pitched at the appropriate level of demand

• Takes account of the accessibility needs of intended candidates with regard to format

and use of language

• Is capable of being used to assess the selected abilities under the specified conditions.

The level of quality control will relate to whether the assignment is externally or

internally set. For example, there will be a high level of control where it is externally-set

and the conditions under which it is taken are externally-defined. Internally-devised

assignments could be subject to pre-checking by the Awarding Organisation to ensure

it is appropriate. Where an assignment is internally marked by centres/third parties,

the internal and external verification/moderation process will apply.

Written examinations/tests

Typically, these include objective questions and/or constructed questions.

• Objective questions are those for which there is only one correct (and

predetermined) answer from a given list. Multiple choice is the most common type of

objective question: a question or incomplete statement is followed by 4-5 answers of

which only one is correct. Other types include multiple response which offer a greater

range of possible answers and those that require the ranking of given information

• Constructed questions are those for which candidates have to compose an answer.

These include short-answer and structured questions which can be quite precisely

worded and elicit reasonably consistent responses, and essay questions where the

range of responses given is likely to vary considerably.

Developing questions for written examinations/tests is a specialised activity.

Typically, experts in the qualification’s subject and in writing questions will be engaged

to write the questions.

To produce valid and reliable objective questions, you will need enough writers (a

minimum of 3 is recommended) and at least one other person (preferably a team) familiar

with the subject to vet and edit the questions: check on the balance of questions and any

overlap; and identify those questions which are acceptable and those that are not.


If you are not experienced in this area of qualification development, you may like to

undertake some training so as to fully understand the process and issues that need to be

considered.

FAB offers webinars and training events on relevant topics (see chapter end).

Further reading is detailed in Appendix B.

93Box 21: Writing objective questions

Writing objective questions: multiple choice 12

Write questions for one topic at a time and then re-arrange. Questions should be

worthwhile, suited to the abilities to be tested, at a reasonable level of difficulty

and not overlong. Include information candidates will need.

A multiple choice item has 3 distinct parts:

• Stem – the question, statement or problem being set

• Key – the correct answer

• Distractors – the incorrect alternatives.

1 The stem should pose the question or statement (avoid negative stems).

It should be clear and as concise as possible

2 Each correct answer – the key – must be technically right and clearly

preferable to others in the list of possible options

3 Each of the possible options should be in the same form (homogeneous) and

not subject to becoming out-of-date or local variation in practice. The

alternative answers (distractors) must be wrong, but plausible

4 There should be no clues which lead the candidate to the correct answer

(even if they don’t know)

5 A logical and consistent order should be used (with no overlap or

correct answers placed in the same position)

It is recommended that the assessment task is tested* before it goes live, to:

9 Check on the quality of the item, whether it is performing as it should

9 Test the administrative system for producing, running, marking and reporting

9 Obtain information on its validity and reliability.

*A Beta test can be used for this purpose, one method of which involves 3 distinct control groups: Group A – those with little or no experience of the subject matter; Group B - a representative sample of the target group for the assessment; Group C comprising subject experts. Comparing results from the 3 groups can help to determine whether the task has been pitched at the correct level and to check on where the pass mark has been set.

93 Drawn from FAB training event (May 2016): Multiple Choice Question Design and Development (tutor Peter Bennett)


In presenting a written examination or test paper:

• The heading should include title of the assessment task, date, time allowed, number

of questions to be answered

• Instructions for answering the questions must be very clear, preferably with an

example of how to record the selected answer

• A test booklet or separate answer sheet will be needed (if answers are not to be given

online)

• Questions must be easy to read, e.g. in two columns, sufficiently spaced

• There should be consistency in labelling, e.g. when using letters/numbers

• Easiest questions may be presented first, the most difficult at the end.

Questions to ask in reviewing task presentation include:

• Is the task presented in a way that is free from bias, clear and accessible, error -free

and unambiguous?

• Is there any pre-requisite cultural knowledge needed to access the intended meaning

of the task?

• Has the age of the learner who may reasonable be expected to take the qualification

been taken into account?

• Is the language usage and vocabulary level used in the assessment task appropriate

for the level of demand of the assessment?

• Is any advantage or disadvantage likely to be experienced by any candidate?

On-going development

Development of questions for written examinations/tests does not stop after the first

paper is produced. You will need to think ahead to future uses of these questions. This

includes reviewing and evaluating how questions have performed.

Pre-testing may be useful before the paper goes live, for example, to test out whether the

questions are suitable and the time allowed for answering is appropriate.

For objective questions, it is possible to conduct statistical analysis of the results to find

out how individual questions have performed.

Common methods include those used to work out:

• Facility values – what percentage of total candidates chose particular answers

• Discrimination values – whether a particular question discriminated well between

good and poor performances of candidates

• Reliability values – how likely it will be that a candidate would get the same result if

the same assessment task was taken again in the same circumstances.

Analysis of statistics available after the assessment can inform its on-going development

and decisions about banking94 questions.

94 Banking is the term used to describe the process of keeping secure examination questions, tests or other assessment materials so they can be used in future assessment tasks. Banking is most commonly used for objective questions, individually or in whole tests, but can be used for other assessments, such as short-answer and structured questions (with a good marking scheme), practical examinations/tests and assignments


Banking

Once the questions have been analysed, those judged to be suitable for further use can be

banked. Banking allows for good questions to be retained for re-use, making the most of

initial capital outlay and effort in developing them. It enables results from different years

to be compared on common questions taken by each set of candidates. More frequent

assessment can be supported, for example, by speeding up the process of producing

further assessment tasks.

The Bank can be organised to follow your qualification’s specification, for example,

by grouping together those questions on the same topic or those relating to the same

abilities (where computerised banks are used, the structure may vary). Each set of

questions must have its associated marking scheme.

It is vital that once a standard is set, this is maintained. This can be achieved through

using a ‘control set of questions’ which will appear in every sitting of the assessment. If,

for example, two groups obtain the same marks on the ‘control questions’, but different

marks on the rest of the assessment, then it may be concluded that one set of questions

was harder than the other.

The greater the percentage of the control questions, the more reliable the equating of

scores, but greater use will increase the likelihood of their security being compromised.

Particular care needs to be taken with regard to the security of items in the bank.

An item bank is a significant piece of intellectual property with considerable monetary value. The elements that make it valuable are the content of the items, the organization of those items into meaningful categories that can be externally related, and the documentation that supports the validity of the included tests and items for real-world inferences…There are three threats to the security of an item bank: theft, disappearance, and exposure… The primary methods of prevention are access control and [for computerised versions] encryption of material. Vale (2004): 27695

Quality assurance and quality control

Written examinations/tests tend to be high stakes assessments and will need robust

quality assurance and quality control of the whole process of developing, producing and

making them available.

For example:

• Ensuring question writers have the necessary expertise and experience is crucial to

the quality assurance process. For new question writers, a guide may be developed

for this purpose, together with ‘hands-on’ training to provide instruction, and

opportunities to practise and gain feedback from expert writers

• The validity of the final assessment result also relies on quality control of the

process involved in putting together the collection of questions in the examination/

test papers, to ensure accuracy. Each paper should be written against a detailed

specification and the drafts reviewed and edited by someone who has specialised

editorial skills in this area. Papers should be proof-read at every stage in their

development and at the point of publication.

95 Vale, D.C. Computerized item banking in Downing et al (2006). Vale offers a working definition of an item: most basically, an item must include a stimulus. That stimulus can be a simple question. Or it can be a question followed by several alternative answers. It may be part of a larger structure, consisting of other stimuli, such as a passage of text or some other form of reference material. And it may occur in concert with other items, either optionally or as a requirement


Security issues are a particular concern when the papers are being printed or packaged

for computerised administration. This is because it will be the final version and a wider

range of people will have access to copies. Errors in printing or formatting ‘can seriously

reduce the validity evidence for a test and create an aura of distrust and anxiety

concerning many other aspects of the test, its construction, and scoring’ (Downing, 2004:

15). Proof-reading is essential.

Security standards and procedures should be developed and monitored, including those

for electronic transmission of test items. All examination/ test materials must be securely

kept when not being directly worked on by those with authority to do so.

Marking schemes

Task-specific mark schemes96 form part of the assessment apparatus if appropriate to the

type of assessment. They take two main forms, although a mix of approach can be used.

Mark schemes in which the answers are identified and given a specific mark allocation

and then totalled at the end, are commonly described as analytic. This type of scheme

is most appropriate for assessment tasks using short, highly structured questions.

Those in which the assessor makes a general judgement about quality of work, rather

than marks responses point-by-point, are described as impression marking. Multiple

marking may take place with two or three assessors; their judgements are then

averaged out. This form of marking is most suitable for essays, but can be used for

projects.

In developing your mark scheme/s, factors that need to be taken into account, include:

the nature of the subject matter and abilities being assessed; and the purpose to which

the results will be put.

The marking of objective questions will be quick as the correct answers are pre-

determined and not subject to the judgement of markers.

A well-defined marking scheme is possible with short-answer and structured

questions, but reliability will not be as high as with objective questions. It will be

necessary to try and ensure all markers follow the same rules and mark to the

same standards. It becomes more difficult to achieve consistency in marking essay

questions as no two answers will be alike; more judgement is needed in marking

responses.

Where assessment tasks require brief responses - a few words or one or two paragraphs,

a points-based mark scheme can be developed.

This type of scheme can also cover responses in graphical form such as diagrams. It is

largely prescriptive: pre-determines the salient (or the most credit-worthy) points to

be looked for and usually has a one-to-one correspondence between salient points and

marks. For example: one mark for naming…; one mark for describing…; one mark for

explaining…; one mark for giving an example of… Markers are left to locate the relevant

points and identify all variations that deserve credit.

96 Information about marking schemes drawn from Ward, C. (1980) Designing a scheme of assessment. Cheltenham: Stanley Thornes (Publishers) Ltd: 101-105


If your assessment task requires longer answers, ranging from one or two paragraphs to

extended responses or multi-page essays, you may choose to develop a level-based mark

scheme.97

For this type of scheme, you will need to describe the number of levels of response, each

of which is associated with a band of one or more marks. Each Mark Band descriptor

covers all the content that you think is relevant for the respective learning outcome.

Assessors are asked to apply a principle of best fit when deciding the mark for a response.

Table 20: An example of what part of a levels-based mark scheme might look like

Mark Band 1 Mark Band 2 Mark Band 3

Learning outcome 1:

1-2 marks 3-4 marks 5-6 marks

Provides a limited explanation of…, including a few techniques … and demonstrates a basic understanding of…

Provides a clear explanation of…., including some techniques … and demonstrates a reasoned understanding of…

Provides a comprehensive explanation of…., including a full range of techniques … and demonstrates a thorough understanding of…

1-2 marks 3-4 marks 5-6 marks

Describes limited features of…

Describes some features of…

Describes in detail features of…

Weighting

One important consideration in developing mark schemes is that of weighting.

Decisions should have been made in producing your measurement specification about

relative weightings, if any, within the total assessment for your qualification. Weightings

usually take the form of allocating additional points/marks or percentages.

If your qualification includes more than one assessment task (component), you could

weight each to reflect, for example, the relative importance placed on these by employers

or other users of your qualification, such as 50% practical unit assessments; 30% exam;

and 20% synoptic assessment.

Alternatively, particular learning outcomes in a unit or module could be weighted

differently by allocating more marks to those that are particularly valued, for example,

those that require candidates to demonstrate skills or focus on application of knowledge

and understanding of a particular topic.

97 Definitions of points-based and level-based mark schemes informed by Massey, A.J. & Raikes, N. Item level examiner agreement. Paper presented at the 2006 Annual Conference of the British Educational Research Association, 6-9 September 2006, University of Warwick, UK


4.2.2 Develop qualification support materials

In developing the assessments, technical reports should be produced that fully document

all aspects of the development process.

The precise content of each technical report will depend on the type of assessment, but

may typically include:

• Overview and objectives of the assessment

• Description of the assessment

• Processes and procedures used in developing it

• Technical features of the assessment such as: statistical analyses of items; rules and

procedures for setting standards, sampling, the aggregation and reporting of results

• Quality control and security procedures

• Rationale to support the interpretation of results.

Technical reports will inform the review and evaluation of your qualification, and provide

evidence to support your validity argument.

They can also be used to inform the qualification specification which will be the primary

source material to guide centre/third party assessment practice. This specification should

be developed in consultation with employers and other stakeholders (see section 4.3 for

what it should cover).

Alongside the qualification specification, a range of support materials can be made

available to provide further detail and examples to assist candidates, assessment

professionals, teachers and trainers in understanding and delivering what is required. You

may wish to develop a pack of materials for each of these groups.

For example:

An assessor pack to supplement your qualification specification might be developed

to include (as appropriate to how your qualification is to be assessed), information and

guidance on:

• How to ensure assessment requirements can be interpreted consistently by

candidates and teachers/trainers

• How to ensure naturally occurring and internally-designed assessment tasks cover

the abilities set out in the proficiency specification

• The type and nature of acceptable evidence that may be presented in a portfolio

(direct and indirect ), including how to check on its authenticity

• How to mark any internally-set assessment tasks and/or administer externally-set

tasks, including security arrangements

• The limits to which candidates can be given assistance with work to be assessed and

allowed to re-draft work

• How to keep track of candidates’ progress

• Circumstances in which to apply Reasonable Adjustments, Special Considerations

• The records and materials to be retained for verification of assessment, for example

records of questioning and responses to oral questions.

Forms and templates for assessment activities could also be included.


A candidate pack might be developed to include, where appropriate:

• The qualification’s content, structure and progression routes

• Teaching and learning, support and review arrangements

• How to make claims for the Accreditation of Prior Learning (if applicable)

• What to expect during the assessment process (nature of the assessment tasks, how

they will be administered, and when)

• An explanation of portfolios, including their purpose, how to organise and present

evidence for assessment (indexing, summarising, labelling)

• Type and nature of acceptable evidence for a portfolio, such as written, audio-visual

and computer-generated evidence, artefacts and products

• The route to certification and how results will be reported

• Data protection, complaints procedure and appeals process.

Support materials to be developed for verifiers/moderators might include information

and guidance on:

• What is to be expected of them in terms of ensuring the validity of internal

assessments and the reliability of assessment decisions, including standardisation98

• How to conduct the verification/moderation process, including sampling

requirements

• Action to be taken if there are any issues with regard to centre/third party

assessments or other activities relating to the qualification

• The records that need to be kept of activities and reports required.

Teachers/tutors should have access to the qualification specification and additional

information and guidance on how to help learners acquire the abilities required to

achieve your qualification and prepare them for the assessments.

Exemplar material will be particularly useful in communicating messages about the

sufficiency and quality of responses required by assessment task:

… carefully chosen exemplars or actual samples of student work can illustrate the standards. They can provide concrete examples of what the features or expectations of performance look like. Further, a range of exemplars can show students the different ways of satisfying the requirements of the stated standards. (Klenowski and Wyatt-Smith, 2014:11199)

Having exemplars available, with and without expert assessor commentary, can help

those taking and assessing the qualification to get a feel of the standard required.

Using actual candidate responses to assessments can be the most effective way to convey

key messages for multiple potential uses. For example:

• To train and standardise markers - to establish consistency and agreement over time

in marking

• To provide insights for teachers and tutors - to help them to understand better the

levels of expectation and marking process

• To enable candidates to get a clear sense of the requirements by seeing work

completed by others in response to similar assessment tasks; they too can use this to

better understand how they will be assessed

• To clarify the difference between threshold attainment and exemplary performance,

again a message that can be reflected on by all concerned in the assessment process.

98 Standardisation is an on-going process involving monitoring standards to ensure assessment decisions across all assessors are consistent and reliable99 Klenowski, V. and Wyatt-Smith, C., 2014, Assessment for Education: Standards, Judgement and Moderation, Sage Publications Ltd


Of course, with a new qualification, it will take time to build up these exemplars. Piloting

can generate exemplar materials while a qualification is being rolled out and bedded

down. But, it should be remembered that these early responses and initial standards

should be replaced with candidate material generated during a mature running stage of

the qualification.

It may be possible to offer sample examination questions with marking instructions

to enable users to mark anonymised candidate responses and then compare their

marks with marks and feedback by the examiner. These examples could be provided in

interactive, online format.

Any specimen materials to be provided must mirror the level of demand and the

relationship with the target proficiency demonstrated by the actual assessment

materials.

4.3 Put it all together to inform delivery of your qualification

Before your qualification is made available, you will need to put together a full

specification. This must be clear and accurate, and communicate to a reasonably informed

person what the qualification requires of candidates, and how the qualification is fit for

purpose.

While developing the qualification specification, you should engage with potential

users (for example, through employer forums or external focus groups) to review

your qualification’s content, structure, purpose and accessibility. Feedback from users

will inform the ongoing review of your specification and the development of future

specifications.

If your qualification is put on the Regulated Qualifications Register (RQF), a web link to

the specification will be required. This means it is essentially a public document. Potential

customers may form their views on whether the qualification is suitable for their purpose

from what the qualification specification is like, for example, its clarity and the extent to

which your qualification appears to be administratively straightforward. So, take this into

account when developing the specification.

Before you start to put things together, make sure you are familiar with your Awarding

Organisation’s house style for such documents and its quality assurance and sign off

arrangements. Most importantly, check on what the relevant regulator expects you

to include. For example, depending on the regulator, you may be required to specify

the: Total Qualification Time (TQT) and/or Guided Learning Hours (GLH) assigned to

your qualification (see Box 22); and any value for credit assigned to it and any of its

components.

In estimating TQT and/or GLHs, pay due regard to any relevant and reasonably available

information, including that from users of similar qualifications and any organisations with

relevant expertise such as employer organisations, funding agencies, inspectorates and

statutory bodies.


Box 22: A note on Total Qualification Time (TQT) and Guided Learning Hours (GLHs)

A note on TQTs and GLHs

Qualifications regulated by Ofqual, Qualifications Wales and CCEA Regulation must:

• Be assigned a level in line with the level descriptors

• Have a size, expressed in terms of Total Qualification Time (TQT); and, where

appropriate, Guided Learning Hours (GLHs).

Total Qualification Time is the number of notional hours which represents: an estimate of the total amount of time that could reasonably be expected to be required in order for a learner to achieve and demonstrate achievement of the level of attainment necessary for the award of a qualification.

TQT comprises the following two elements:

• The number of hours which an Awarding Organisation has assigned to a

qualification for Guided Learning; and

• An estimate of the number of hours a learner will reasonably be likely to

spend in preparation, study or any other form of participation in education

or training, including assessment, which takes place as directed by – but not

under the immediate guidance or supervision of – a lecturer, supervisor, tutor

or other appropriate provider of education or training.

If you decide to assign credit to your qualification or unit(s), credit must be calculated as TQT divided by 10.

Guided Learning is the activity of a learner in being taught or instructed by – or otherwise participating in education or training under the immediate guidance or supervision of – a lecturer, supervisor, tutor or other appropriate provider of education or training. For these purposes, the activity of participating in education or training shall be treated as including the activity of being assessed if the assessment takes place under the immediate guidance or supervision of an appropriate provider of education or training.

GLHs can be useful for administrative purposes: for people who are funding qualifications, planning curricula, or performance tables; or for employers who want to have an idea of how long their employees are going to be away from the workplace for study purposes.

No learner is required to take a certain amount of time; some may take less and some may take more time. Qualifications can sit at different levels, but require similar or different amounts of study and assessment time.

For details of the TQT Criteria, see:

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/597612/total-qualification-time-criteria.pdf


In Scotland, SQA does not require awarding bodies to specify the TQT, but does apply the

concept of notional learning hours. See: http://scqf.org.uk/wp-content/uploads/2014/05/

SCQF_handbook_WEB_FINAL_2015.pdf

Scotland has its own regulatory principles for the accreditation of qualifications, with

credit-rating being a separate activity.

You may have had a working title for your qualification, but now is the time to confirm

the titling you will use for regulation and publication purposes. Generally, this means

including information on: the name of your awarding organisation; the level of your

qualification; and a concise indication of its content. If your qualification has a Type then

this must also be included, for example, Technical Level.

It will be important to ensure that the title is not misleading to your qualification’s users

and has a clear relationship to other qualifications or units that are similar to yours. The

title should reflect the knowledge, skills and understanding that will be assessed.

If you are offering a qualification based on approved qualification products in

Scotland, you must use the exact title and unit titles from the approved structure. No

SVQ qualification titles may be adapted to include the name of your awarding body,

but this is permissible for other qualifications. The title of a qualification or unit must

be unique and reflect its function and content, and where appropriate, the context.

If you are designing your own qualification, you should consider the relationship

between qualifications when allocating titles. Decide whether it is sufficient to have

different levels of qualifications with the same title; whether the title will clearly

indicate differences between content and complexity. The word ‘level’ must only be

used when referring to an SCQF level100.

100 SQA Developing Scottish Products for Approval (March 2016) http://accreditation.sqa.org.uk/accreditation/About_Us/External_Stakeholders/Developing_Scottish_Qualification_Products_for_Approval_Final_March_2016.pdf


Box 23: A note on Total Qualification Time (TQT) and Guided Learning Hours (GLHs)

So, what exactly do I need to include in my qualification specification?

The precise detail of what you need to include will be set out in the regulator’s

conditions for recognition.

In summary, a specification typically sets out:

• The qualification’s title

• The qualification’s objective and how it is fit for purpose

• The level or levels assigned to it and any of its components

• Any other qualification which a learner must have completed before taking

the qualification

• Any units/modules that must have been completed before the qualification

will be awarded and any optional routes

• Any prior knowledge, skills or understanding which the learner is required to

have before taking the qualification, or other requirements

• The knowledge, skills and understanding which will be assessed as part of the

qualification (with a clear indication of coverage and depth)

• The method(s) of assessment and any associated requirements

• The criteria against which learners’ levels of attainment will be measured

(such as assessment criteria or exemplars)

• Any specimen assessment materials

• Any specified levels of attainment

• Any TQT/GLH, and credit (if applicable)

• Clear contact information where any queries about the specification can be

raised.

You may wish to add other information to enhance your qualification’s offer

to potential users.


Further information

FAB offers a range of training events, resources and webinars to support this chapter.

For example, webinars on:

• Engaging with UCAS and Qualification Information Profiles (QIPs)

• How to write high-quality test items

• E-assessment and grading vocational qualifications.

For example, training events such as:

• Introductory training on how to develop vocational qualifications

• Training on Multiple Choice Question Design and Development


Links to reference material cited in this chapter, include:

• Downing, S.M. and Haladyna, T.M. (2006) Handbook of Test Development. London:

Lawrence-Erlbaum Associates (including chapter on Standard Setting by Gregory J

Cizek; chapter on Computerized Item Banking by David C Vale)

• A second edition is available for purchase: Lane, S, Raymond, M.R. & Haladyna T.M.

(2016) Handbook of Test Development. Routledge

• Klenowski, V. and Wyatt-Smith, C., 2014, Assessment for Education: Standards, Judgement and Moderation, Sage Publications Ltd

• Massey, A.J. & Raikes, N. Item level examiner agreement. Paper presented at the 2006

Annual Conference of the British Educational Research Association, 6-9 September

2006, University of Warwick, UK

For further references and information of relevance to this chapter, see Appendix B.

137

Chapter 5

How to deliver a qualification

This chapter describes what is involved in delivering a qualification. It covers:

9How to prepare for the delivery of your qualification

9 Support for qualification personnel and learners

9What happens when candidates take assessments

9 Controls and security and how to deal with malpractice

9How to evaluate candidates’ performances (produce an overall result)

9How to report results, deal with appeals, award and certificate.

138How to deliver a qualification


• Marketing means putting the right Product in the right Place, at the right Price and at the right time

for Promotion, but also managing People, Process and Physical environment in providing a service

for its delivery

• Effective guidance and information must be provided for purchasers and users, and arrangements

made (a written, enforceable agreement) with centres/third parties to ensure delivery meets

regulatory requirements

• Systems, personnel and procedures need to be in place to support deliverers and learners,

quality assure delivery, gather assessment evidence and deal with enquiries, requests,

complaints and appeals

• It is important to understand what happens when candidates take assessments, including

the roles and responsibilities of key personnel (assessors, examiners, internal and external

verifiers/moderators)

• Maintaining controls includes quality assuring both assessment decisions and outcomes

• Maintaining security is crucial during the processes involved in handling the evidence, including

checking its authenticity, assuring confidentiality and integrity, and being alert to conflicts of interest

• Every Awarding Organisation will face some level of risk of malpractice or maladministration and

will need to take all reasonable steps to prevent this, and apply sanctions where they occur

• Evaluating candidates’ performance requires reliability in marking and standardisation to achieve

a high level of consistency and comparability across assessments and confidence in the final

assessment decision

• Results must be reported clearly, accurately and in a timely way, and have a clearly set out

appeals process

• The awarding of the final certificate needs to be timely and efficient, with certificates meeting

the required quality criteria.

In developing your qualification, you will have established its structure and progression,

and developed its assessment apparatus, with associated support materials.

In putting it all together, you will have a full qualification specification and additional

materials to help candidates, and those involved in the delivery of your qualification,

understand what is required.

The expected outputs of the delivery stage of your qualification will include for each

candidate:

• A set of performances from engaging in assessment tasks

• A set of evaluations which make their proficiency clear

• An overall measurement result from combining and transforming the set of

evaluations

• A certificate.

Appropriate quality assurance and quality control procedures will have ensured that

users of the qualification can have confidence that the qualification reflects the true

proficiency of those who have achieved it.

139 How to deliver a qualification

5.1 Prepare for delivery of your qualification

Regardless of whether, or not, you will be directly responsible for delivering your

qualification, you will need to ensure that it is in a prepared state for launch and delivery.

Furthermore, you will need to collect relevant data during delivery to inform the review

of your qualification and evaluation of how it performs.

5.1.1 Check on requirements for regulation of your qualification

Before any qualification can be included on a register of regulated qualifications, your

Awarding Organisation, and the qualification, must satisfy the regulator’s requirements

(for example, conditions of recognition).

It is important that you check on the relevant regulator’s requirements to ensure your qualification complies101. As these may change in response, for example, to government

policy developments, new funding rules and so on, what follows in this section, is a

general indication of what might be required.

For your Awarding Organisation to have its qualifications regulated, it is likely to have to

satisfy criteria that relate to:

• Its governance and financial viability

• The availability of adequate resources and arrangements to develop, deliver and

award qualifications

• The ability to identify and manage risks, manage incidents and prevent, investigate

and manage malpractice and maladministration

• Willingness and ability to provide information as necessary to the regulator

• Any guidelines or principles of good practice relating to its behaviour

• Arrangements it makes with third parties, centres and publishers.

Usually, there needs to be a named person (sometimes called a Responsible Officer) who

is the authoritative contact for all matters of interest to the regulator. Consult with this

person (if not you) when checking on regulatory requirements.

SQA Regulation has requirements which relate specifically to qualifications offered in

Scotland.

101 Web links to qualification regulators are provided in Appendix B


For qualifications regulated by Ofqual, Qualifications Wales and CCEA Regulation,

requirements typically relate to:

• General requirements with regard, for example, to: the qualification’s fitness for

purpose and accessibility; procedures for review; making information available to

help meet teachers’ needs; responding to enquiries and complaints; management of

its withdrawal

• Design and development of a qualification, for example, the requirement to: have

an objective and support for it; meet requirements on titling, level, time and, if

applicable, credit; publish a specification; ensure its assessment is fit for purpose and

can be delivered; recognise prior learning

• The provision of information on a qualification to purchasers, including information

on fees and its features

• The setting and delivering of the assessment relating, for example, to: use of

language and stimulus materials; maintaining confidentiality of assessment materials;

registration of learners; arrangements for Reasonable Adjustments and Special

Consideration; completion of the assessment under required conditions; effective

and efficient delivery

• The marking and issuing of results, including: arrangements and options for marking

and moderation procedures; monitoring specified levels of attainment; checking

the sufficiency of evidence to support results; the issue of results and timescales for

publishing these

• Appeals and certificates, including: having in place an appeals and complaints

process; meeting the requirements for certificate design and content; the timely issue

of certificates.

You will also need to check on the process for putting your qualification on a register of

regulated qualifications.

For example, Ofqual generally allows Awarding Organisations to upload a qualification

on to the Register. It will be on the Register the next day (unless there is an accreditation

requirement on the Awarding Organisation or on the category of qualification).

At the time of writing, CCEA Regulation has a universal accreditation requirement. So,

if you wish to offer your qualification in Northern Ireland it will need to be accredited

before it can be listed (and this can take time). Qualifications Wales uses Qualifications in

Wales (QiW) which is a separate system.

In addition, other stakeholders may require your qualification to have particular

characteristics, for example: Department for Education requirements for qualifications to

be included in school/college performance tables in England.

The next step is to ensure everything is accurately communicated in published

materials and available in good time for take-up of your qualification, and it is marketed

appropriately.

In marketing it will be particularly important to communicate, in publications and on

your website, the fact that your qualification is regulated (as distinct from those that are

unregulated) and by what body.


5.1.2 Consider the marketing of your qualification

Marketing includes: putting the right Product in the right Place, at the right Price, and

at the right time for Promotion. These four factors (the 4Ps) make up the basis of the

marketing mix.

Market research will have enabled you to identify your target market and the benefits

customers are expecting from using your qualification, as well as provided data to inform

the business case.

Customers buy a product to meet a need. This is why it was important at an early stage

to be clear about what will appeal to them and whether, or not, there will be a market for

your product. Your qualification also has to be competitively priced and stand-out from

the competition. However, it will always be important to balance what the market wants

with your central concern to produce a high quality and valid qualification.

Some questions to consider include:

Product • What does the customer want from your product?

• What features does it have to meet these needs?

• How and where will customers use it?

• How is it differentiated versus your competitors?

Place • Where will customers look for your product?

• Where will they access information to learn more about it?

Price • Are there established price points for products in this area?

• How will price compare with your competitors?

Promotion • How will your Awarding Organisation reach your target customers,

advertise your product?

• When will be the best time to promote it and make it available so that those

delivering it have sufficient lead-in time to prepare?


Your qualification will be part of your Awarding Organisation’s product portfolio and, like

any product, it will have a number of stages in its working life: launch; growth; maturity;

and decline. As a regulated qualification, it will also have a shelf life. Your Awarding

Organisation will be concerned with sustaining sales beyond the launch.

It will be important for your qualification to have a strong brand image and unique selling

point (USP). This is where packaging of your product comes in (what comes with the

qualification specification to make it an attractive product), although regulators may

insist it is made clear to purchasers that it is also available as a stand-alone product.

Where others plan to publish materials designed to support the preparation of learners

for assessments, such as study guides, you will need to make clear and publish the criteria

for informing decisions about whether or not to endorse them. You will be concerned,

for example, to check the credentials of the publisher and its authors, and quality control.

Care must be taken to ensure that these publications accurately reflect what your

qualification requires and do not have an adverse effect on those taking it (mislead

them as to requirements).

Of course, you are not only providing a product, your Awarding Organisation will be

offering a service. You may have got the product right, but if its delivery is not fully

supported this will impact on take-up102.

Customers will be concerned about what they are getting for their money, but also the

training, support and resources available to enable them to achieve successful outcomes

for their learners. This will be a critical factor for those whose funding relies on achieved

qualifications.

Managing the service of your qualification

Managing a service can be more complicated than managing product development as it

can be affected by factors outside of your control, making standardisation more difficult.

In marketing a service, three additional factors apply concerning: People; Process; and

Physical Environment (see Figure 12).

Figure 12: The Extended Marketing Mix

102 There are various quality standards for organisations that want to ensure that their products and services consistently meet customer’s requirements, and that quality is consistently improved. One example is the ISO 9000 family of Standards. Information is available at: https://www.iso.org/iso-9001-quality-management.html

Place

Price

Product

The Seven

Ps

PromotionPeople

Process

Physical Environment


People Customers will make judgements about service provision and delivery based on

the people representing your Awarding Organisation. Staff require the appropriate

interpersonal skills, aptitude, service knowledge and training in order to deliver a quality

service. Best practices in people management and development can be recognised

through accreditation schemes such as Investors in People103.

Importance of good customer service

Good customer service should be considered a priority. A customer service policy should

be in place, with appropriate procedures and practices to respond quickly to requests for

information. You should have a clear statement of the service you aim to provide

and ensure that staff have appropriate customer service information and training.

Some Awarding Organisations publish and monitor their service level standards, for

example, the time to be taken for responding to enquiries, complaints or appeals.

During delivery, you will be interested in what your customers think about your product

and service. An annual customer survey is one of the means of checking on customer

service performance, as well as monitoring response times and maintaining and reviewing

customer service records.

Records of queries should be kept and included in the review of provision. Typical

information requested by assessors and tutors could be included in a Frequently Asked

Questions (FAQ) section of your Awarding Organisation’s website, if appropriate.

Although tutors and assessors may initially raise queries with a line manager, or person

specifically responsible for quality assurance within their organisation, they may like to

have contact details of a person representing your Awarding Organisation, for example, if

they cannot find the information they need on the website.

Process This aspect of the marketing mix is concerned with the systems used to deliver your

service efficiently. All services need to be underpinned by clearly defined processes.

Essentially, this means everybody knows what to do and has the means by which to

provide the service.

Physical Environment

Your customers may not visit your Awarding Organisation’s premises, but they will visit

its website and make assumptions about the quality of service they are likely to receive

from their experience of this environment. For example, ease of access to the information

they need, range of resources available, clarity and tone of the communication.

103 Investors in People is an example of a Standard for people management, offering accreditation to organisations that adhere to this Standard. The standard describes what it takes to lead, support and manage people for continuous improvement. Information is available at: https://www.investorsinpeople.com/


5.1.3 Provide information for potential purchasers and users

In working with centres/third parties to support the delivery of regulated vocational

qualifications, Awarding Organisations need to provide effective guidance and any

information reasonably required for the purpose of ensuring delivery is in line with the

regulations.

When included in a register of regulated qualifications, the title of your qualification

has to be used consistently in advertising and communications with users. You will

have checked on the regulator’s requirements on titles in developing your qualification

specification.

It is important to know that your Awarding Organisation must not advertise or promote

its qualifications in a manner that is likely to be misleading to users of qualifications and

must take all reasonable steps to ensure that any person connected with it does not do so

either.

Fees

Where your Awarding Organisation produces a list of standard fees for its qualifications

and for any associated services, it must ensure that the list is accessible to potential

purchasers. As a minimum, they should be given a copy of the list or have access to

it on your website. If you do not have a list, information on fees must be provided on

request, or at least a reasonable estimate of the prices. Information on fees should be

accompanied by that on the main features of the qualification and associated services.

Fee information should be given early enough to enable purchasers to plan for the

delivery of your qualification.

Your Awarding Organisation may wish to offer other products and/or services to support

the delivery of your qualification, to make it an attractive option to potential purchasers.

In this case, it will be important that purchasers know in advance the cost of these ‘add-

ons’ and that it is also possible to buy the qualification as a self-standing product 4. 104

5.1.4 Make arrangements with centres/third parties

The arrangements you need to make for the delivery of your qualification will depend

on how and where it is to be delivered. For example, centres may be in the UK or other

countries; they may have satellite sites affiliated to them. Assessment services may be

sub-contracted to third parties.

If you work for a Professional Body, learners who wish to take your qualification will

most likely have to register directly with your organisation by completing an

application form with the required registration fee. They may learn online rather

than physically attend a teaching/study centre. On registration, membership may

be granted to allow access to support, such as that provided through a Continuing

Professional Development (CPD)scheme.

When the time comes, learners will be responsible for booking their examination

with an approved examination centre (which may, or may not, also be an approved

study centre) and paying the examination fee.

104 Drawn from Ofqual (June 2016), General Conditions of Recognition


Where any third party undertakes any part of the delivery of your qualification, your

awarding organisation must:

• Ensure that arrangements accord with its conditions of recognition

• Monitor and, where appropriate, enforce such arrangements; and

• Take all reasonable steps to avoid unnecessary or unduly burdensome requirements

on the third parties.

Effective communications systems will need to be in place with third parties and any

subcontractors, to keep them up-to-date with the requirements of your Awarding

Organisation and the Regulator/s.

Where a centre undertakes any part of delivering your regulated qualification,

arrangements should include a written and enforceable agreement.

Typically, this agreement sets out all the requirements with which the centre must

comply. For example, each centre should have available sufficient managerial and other

resources, and retain a workforce of appropriate size and competence, to effectively and

efficiently undertake the delivery of the qualification as required.

In addition, the agreement should include requirements for each centre to:

• Take all reasonable steps to ensure that your Awarding Organisation is able to comply

with the regulator’s Conditions of Recognition (or similar)

• Take all reasonable steps to comply with requests for information or documents made

by your Awarding Organisation or the relevant regulator as soon as practicable

• Assist your Awarding Organisation in carrying out any reasonable monitoring

activities and assist the relevant regulator in any investigations made for the

purposes of performing its functions

• Undertake the delivery of the qualification required by your Awarding Organisation

in accordance with Equalities Law

• Operate a complaint-handling procedure or appeals process for the benefit of

learners105.

Any moderation processes to be undertaken by your Awarding Organisation, or on its

behalf, should be set out. A sanctions policy should be established, to be applied in the

event that centres fail to comply with these requirements. In the event that a centre

withdraws from its role in delivering your qualification, the withdrawal process should

also be made clear and be followed. Your Awarding Organisation must take all reasonable

steps to protect the interests of learners.

It is likely that your Awarding Organisation will already have an agreement in place with

centres. This will need to be checked against regulatory requirements, updated for your

qualification and reissued to centres; they will need to be made aware of any changes and

when they take effect.

105 Extracts from Ofqual (June 2016) General Conditions of Recognition. Check on full requirements with the relevant regulator


If you do not currently have a written enforceable agreement in place, you are advised to:

• Identify who can authorise the agreements within your Awarding

Organisation, for example the Responsible Officer, a Senior Officer

• Consider obtaining a legal opinion on your centre agreements

• Identify ways that you will enforce the centre agreements

• Identify who can authorise these agreements on behalf of your centres, for example

the Head of Centre or Principal

• Develop a clear plan for how the agreements will be issued (electronically online, hard

copy, soft copy etc.), including the duration of the agreements and how often they will

be reissued to centres to confirm their on-going agreement

• Identify resources to be allocated to support the issue of the agreements to centres

and a monitoring process to ensure the agreement is in place with all centres

• Develop a policy in relation to the action you will take with any centres who refuse to

sign the agreements. Having an agreement in place is a regulatory requirement and

so continuing to work with centres without an agreement would be deemed to be

non-compliance.

By signing the enforceable agreement, your Awarding Organisation will be confirming

that it agrees to set out all requirements for the continued approval to deliver your

qualification. This will provide the centres with confidence that they have been provided

with all relevant requirements with which they must comply.

By signing the agreement, the centre will be confirming that it will ensure that it

continues to meet these requirements. This will provide you with a reference to refer

back to if the centre fails to do something required.

For some qualifications, there may be additional requirements in delivery, such as

meaningful activities involving employers (see Box 24).


Box 24: Employer involvement in Tech Levels and Certificates.

Meaningful activities involving employers (DfE, August 2016: 331106)

The Department for Education requires all Awarding Organisations offering Tech

Levels and/or Tech Certificates for performance tables in England to: ‘confirm

that providers have secured employer involvement for every learner’ taking an

approved Tech Level qualification on the 2018 and later lists, and for Technical

Certificates on the 2019 and later lists.

All students aged 16-19 working towards one of these qualifications must

undertake ‘meaningful activity’ involving employers from a list of eligible

activities reproduced below. Whilst the DfE has allowed for flexibility in the

way that providers and employers work together to support students learning,

the contribution that meaningful activities make to the qualification must be

significant.

Activities and their contribution to the students’ learning are set out in each

qualification specification. They relate to one or more elements of the mandatory

content of the qualification as a minimum.

Examples of ‘meaningful activity’ involving employers include:

• Structured work-experience or work-placements that develop skills and

knowledge relevant to the qualification

• Project(s), exercises(s) and/or assessments/examination(s) set with input from

industry practitioner(s)

• One or more units delivered or co-delivered by an industry practitioner(s).

This could take the form of master classes or guest lectures

• Industry practitioners operating as ‘expert witnesses’ that contribute to

the assessment of a learner’s work or practice, operating within a specified

assessment framework. This may be specific project(s), exercise(s) or

examination(s), or all assessments for a qualification.

106 Department for Education (August 2016) Technical and applied qualifications for 14 to 19 year olds Key stage 4 and 16 to 19 performance tables from 2019: technical guidance for awarding organisations. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/546027/14-19_Qualifications_Technical_Guidance.pdf


5.1.5 Ensure systems, procedures and personnel are in place

Having enforceable agreements with centres/third parties is part of your Awarding

Organisation’s overall responsibility for ensuring systems, procedures and personnel are

in place for managing your qualification’s delivery and quality assurance in line with the

regulations107.

Centres/third parties must be prepared for what they are required to do to deliver

your qualification, or any parts of the delivery. For example, they will need to know the

arrangements for the security of assessments and for managing assessments such as

examination timetabling and processes.

Systems and procedures

You will need to have the systems and procedures for managing: enquiries, requests

and complaints; conflicts of interest; appeals; and for investigating malpractice and

maladministration. FAB offers training which covers basic tools, procedures and

templates (see chapter end).

A written complaints procedure should be established, maintained, published and at all

times complied with. This should include procedures and timescales for responding to

complaints and dealing with their subject matter.

In responding to enquiries from users of your qualification, your Awarding Organisation

will be expected to answer accurately, fully and within a reasonable time any reasonable

enquiries received by it (without breaching a duty of confidentiality).

Your Awarding Organisation must take all reasonable steps to manage the risk of conflict

of interest, including having a specific policy.

A robust conflict of interest policy should set out:

• Evidence of the contractual obligation for assessors to disclose conflicts (include your

conflicts of interest declaration form as supporting evidence)

• Detail of the processes to record conflicts and how to actively manage the process to

resolve conflicts, including where employer or provider sites are used for assessment.

The appeals108 process should be clearly set out and readily available to your customers.

The process is quite costly because it involves remarking or reassessing candidate

work. So, to mitigate this risk, quality assurance processes need to build confidence and

consistency in the assessment process so customers feel confident that decisions made

are correct.

For details on dealing with appeals, see section 5.6.1.

107 In October 2017, Ofqual launched the Ofqual Handbook – an online resource that includes rules or guidance for all awarding organisations and for all regulated qualifications This will make it more straightforward to check what is required

108 Appeals are a process through which an awarding body or centre may be challenged on the outcome of a decision


It is especially important to ensure there are robust procedures for preventing and

investigating incidents of malpractice or maladministration (see box 23 for a summary of

procedures relating to malpractice and maladministration).

Maladministration is any activity, neglect, default or other practice that results in

the centre or candidate not complying with the specified requirements for delivery of

the qualifications as set out in the relevant codes of practice, where applicable.

Malpractice is any deliberate activity, neglect, default or other practice that

compromises the integrity of the assessment process and /or validity of certification.

For dealing with malpractice in practice, see section 5.4.3.

Every awarding organisation will face some level of risk of malpractice or

maladministration occurring in relation to the qualifications that they award (whether

these qualifications are regulated or not). The level of risk will vary depending on a

number of characteristics including: the subject/nature of the qualifications, the level of

funding the qualifications attract and the providers that deliver the qualifications.

Incidents of malpractice/maladministration can potentially lead to learners being disadvantaged and can require the conduct of costly and time-consuming investigations. They may also cause reputational damage to an awarding body, their qualifications and the wider qualifications sector.

It is, therefore, in the interests of awarding bodies, their centres and their learners to prevent malpractice or maladministration from occurring, wherever possible. Where it is not possible to prevent this, it is in everyone’s interest to ensure that all cases of suspected or actual malpractice/maladministration are dealt with quickly, thoroughly and effectively.

(FAB, September 2013109)

Thus, it is very important that procedures for dealing with such incidents are up-to-date

and communicated across centres, and any satellite sites, sub-contractors and third

parties involved with your qualification.

109 FAB Malpractice and Maladministration Guide, September 2013


Box 25: Procedures relating to malpractice and maladministration

Summary of procedures relating to malpractice and maladministration*

Awarding Organisations must establish and maintain, and at all times comply

with, up to date written procedures for the investigation of suspected or alleged

malpractice or maladministration; and ensure that such investigations are carried

out rigorously, effectively, and by persons of appropriate competence who have

no personal interest in their outcomes.

Examples of ‘positive indicators’ that would suggest an Awarding Organisation is

likely to comply include:

• Policies, practices and/or procedures in place that reduce the risk of

malpractice and maladministration such as plagiarism, collusion, tampering,

breach of confidentiality of assessment materials

• Use and regular review of a standardised investigations policy and approach

that sets out:

» Who investigates concerns

» How an investigation is undertaken

» How whistleblowers will be treated, and in such a way that individuals will

not be prejudiced

» When and how interested parties will be notified lawfully

» How any interviews should be conducted

» How facts will be gathered and evidence found, collated and stored

» How documentary evidence will be verified

» How confidentiality of investigation materials is assured

» How records of its investigations will be presented and their accuracy

assured

» How and when any visit to centres will be announced and undertaken

» Which principles will be followed when it undertakes an investigation with

other bodies.

*Drawn from Ofqual’s Guidance to the General Conditions of Recognition, July 2016 (Condition A8). Check with the relevant regulator for full details.


A further consideration are the arrangements you will need for making Reasonable Adjustments to the assessment for your qualification so as to enable disabled learners

to demonstrate their knowledge, skills and understanding to the levels of attainment

required.

G6.2 An awarding organisation must, in accordance with Equalities Law, have in place clear arrangements for making reasonable adjustments in relation to qualifications which it makes available.

G6.3 An awarding organisation must publish details of its arrangements for making Reasonable Adjustments, which must include details as to – a. How a Learner qualifies for a Reasonable Adjustment, and b. What Reasonable Adjustment will be made.

(Ofqual, June 2016:67)

Reasonable Adjustments may be permitted: at the discretion of the centre; in

consultation with the external verifier; on application to your Awarding Organisation.

FAB has a guide110 which includes forms that may be appropriate to use for centres to

gain permission to make Reasonable Adjustments, record Adjustments and report back

to you; and list of sources of further information.

To prepare for situations when candidates are put at a disadvantage, but not eligible

for Reasonable Adjustments, your Awarding Organisation should have in place clear

arrangements for Special Consideration to be given. It must publish details of: a) how a

learner qualifies; and (b) what Special Consideration will be given.

Special Consideration may be given to a Learner who has temporarily experienced: (a)

an illness or injury, or (b) some other event outside of the Learner’s control, which has

had, or is reasonably likely to have had, a material effect on that Learner’s ability to

take an assessment or demonstrate his or her level of attainment in an assessment.

(Ofqual, June 2016: 68)

Personnel involved in delivering a regulated qualification

Personnel involved with your qualification will depend on whether it is internally and/ or

externally assessed and how it is quality assured.

Some will be working in centres and directly employed by them: tutors/teachers;

assessors; internal verifiers/moderators . Others will be contracted by your Awarding

Organisation: external verifiers/moderators; examiners. Third parties typically contract

with personnel to conduct assessments; some third parties may operate as study centres

and have tutors.

Within centres, personnel will include teachers (widely defined):

A teacher is a person who prepares any learner, or any person likely to become a learner, for assessment for a qualification and who does so: a) as a lecturer, supervisor, tutor or other appropriate provider of education or training, or b) in circumstances in which that preparation takes place primarily at home. (Ofqual, June 2016: 88)

110 FAB (January 2012) Application of Reasonable Adjustments and Special Consideration in vocational qualifications


In the delivery of regulated vocational qualifications, the tutor’s/teacher’s role is to help

learners to acquire the knowledge, understanding, skills and behaviours required to

achieve the qualification. Depending on how the qualification is delivered and assessed,

tutors/teachers can also be involved in one or more parts of the assessment process

(initial assessment; formative assessment; summative assessment).

Several tutors/teachers may be involved in delivering a qualification in order to cover its

breadth and to address the particular needs of learners.

Where assessment focuses on naturally-occurring evidence and/or internally-set

assessment tasks such as assignments, a system will need to be in place to provide

assessors and internal and external verifiers/moderators.

Assessors could be teachers and/or internal verifiers (but they are not permitted to verify

their own assessments). Some assessors may specialise in certain assessment methods,

such as observation of performance, or specialist topics.

Internal verifiers/moderators111 are responsible for ensuring the validity of internal

assessments and reliability of assessor judgements.

This includes:

• Organising the internal verification process

• Supporting assessors

• Checking assessment tasks to ensure validity

• Arranging standardisation exercises

• Observing assessments

• Sampling assessment decisions

• Ensuring internal verification/moderation decisions are consistent

• Preparing for external verification/moderation

• Maintaining verification/moderation records.

Standardisation is an important part of their responsibilities and contributes to

continuous quality improvement.

External verifiers will also be concerned with how internal standardisation is conducted

within their centres.

For what happens in Internal standardisation, see 5.4.1.

111 The titles verifier and moderator reflect particular traditions, but essentially these personnel fulfil the same role in the quality assurance system; the terms tend to be used interchangeably


An external verifier/moderator is typically required to:

• Participate in verification/moderation events organised centrally

• Carry out visits to centres to conduct verification/moderation, and/or conduct online

or postal verification, as required

• Conduct prior verification of centres’ internally-devised assessment tasks

• Participate in approval, post-approval and development visits

• Assist internal assessors and internal verifiers/moderators with policy and subject-

specific enquiries

• Deliver and participate in external verifier/moderator training events

• Identify any indications of malpractice and/or maladministration

• Maintain records and report to their Awarding Organisation.

Where your qualification includes externally-set assessments, such as an examination

(as the sole assessment task, or combined with other types), personnel will include

examiners. Examiners are individuals with vocational subject expertise who are required

to use their professional judgement in marking candidates’ responses to questions.

Not all Awarding Organisations use examiners, but where they do, these are usually

directly contracted with and then trained to mark candidates’ examination scripts and/or

externally-set assignments (not internally marked).

The number of examiners should be kept to the minimum, but will depend on: the number

of candidates; the nature of your qualification; the experience of the examiners (which

will affect the amount of marking they can do within the specified period of time).

Examiners are likely to be supported by: Chair of Examiners who has responsibility,

overall, for maintaining standards across different specifications in a vocational subject

area; Chief Examiner who is responsible for your particular specification; Principal

Examiner for each component of your qualification.


5.2 Provide support for qualification personnel and learners

5.2.1 Provide support for tutors/teachers

Regulation (by Ofqual, CCEA Regulation and Qualifications Wales) relating to support

for tutors/teachers delivering qualifications is specifically concerned with making

information available that they may reasonably require for the purpose of preparing

learners for assessments.

‘Positive indicators’ that would suggest an Awarding Organisation is likely to comply with this regulation include:

• Publishing on its website a sample of past assessments and other information that may reasonably be required for teachers to prepare learners for assessment of its qualifications, clearly linked from the pages containing the relevant qualification specifications;

• Responding promptly to requests for information from teachers, where they need the information to prepare Learners for assessment and the information is not otherwise available’.

(Ofqual, July 2016: 53112)

Tutors/teachers will expect to have access to your qualification specification with

guidance and, where possible, a sample of past assessments with commentaries. This is to

help them to develop a clear understanding of the knowledge, understanding, skills and

behaviours expected of candidates so they can adequately prepare them for summative

assessment.

They will also appreciate up-to-date information being available quickly, either from your

Awarding Organisation’s website or by communicating with someone online, by email,

phone or other appropriate way.

Centres are responsible for having competent staff, but training relating specifically

to your qualification could be offered by your Awarding Organisation (see Box 26 for

examples of typical events).

112 Ofqual (July 2016) Guidance to the General Conditions of Recognition for Awarding Organisations. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/538339/guidance-to-the-general-conditions-of-recognition-July_2016.pdf


Box 26: Examples of training for tutors/teachers

Examples of training events offered to tutors/teachers

Examples of training, provided face-to-face and/or online, include:

• Introductory training for tutors/eachers who are new to the Awarding

Organisation and its qualifications

• Events to launch/ get to know a new qualification

• Events which offer opportunities to gain more detailed information on the

qualification’s teaching content, and resources to support its delivery

• Workshops that focus on specific topics such as those on assessment and use

of technology, with opportunities to learn through practical activities

• Continuing professional development courses to update skills, knowledge and

understanding.

Tutors/teachers in the Further Education and Skills sector are not required to hold any

particular qualifications, but the need to develop dual professionals – those who can

combine teaching and occupational expertise - is becoming increasingly recognised as

important.

Vocational teaching, learning and assessment is a sophisticated professional occupation and demands, therefore, robust initial and continuous development of expertise… The best vocational teachers…have dual identities, as occupational specialists and pedagogical experts…[but] it takes time for teachers and trainers to realise their dual identities. Dual professionals are not born; they need support to develop. (CAVTL 2013: 20113)

If they do not already hold one, tutors/teachers may appreciate support in working

towards an appropriate qualification114 with opportunities to further develop and have

recognised their skills in teaching and assessment.

Continuing Professional Development (CPD) activities, provided by your Awarding

Organisation and/or others will be important115. For example, opportunities for updating

occupational knowledge and skills to ensure they remain current and keep pace with

technological developments.

Over time, there will be opportunities for drawing on the experiences of tutors/teachers

in using your qualification to provide a range of resources in a variety of formats such

as: initial assessment tools; schemes of work; assessment plans; teaching and learning

resources; pro forma to track and record progress and achievement; resources to use

with learners such as activity planning and review sheets, and observation records.

113 Commission on Adult Vocational Teaching and Learning (2013) It’s all about work…Excellent adult vocational teaching and learning. London: LSIS114 There are different levels of teaching qualifications approved for use within the sector, but it is considered best practice for teachers to hold a full

qualification, i.e. a level 5 Diploma in Education & Training [DET]. If the qualification is taken at a university or FE college it may be called a Post Graduate Certificate of Education (PGCE), a Post Compulsory Education and Training (PCET) or a Cert Ed. There are specialist qualifications for teachers who work with students with Special Educational Needs and Disability (SEND)

115 For example, the Education and Training Foundation (ETF) has developed Professional Standards for Teachers and Trainers in Education and Training. It is described as an ‘aspirational’ framework and includes “How to” guides. Available at: http://www.et-foundation.co.uk/wp-content/uploads/2014/05/4991-Profstandards-A4_4-2.pdf ETF has also developed modules with the Career College Trust to help improve the digital skills of practitioners, including one on online assessment. Available at: http://www.et-foundation.co.uk/supporting/support-practitioners/learningtechnology-support/


Downloadable teaching ideas and resources from an online resource bank could be

provided. However, before publishing materials developed by practitioners, any copyright

issues will need to be resolved.

In summary, Box 27 offers some guiding principles in providing support for teachers.

Box 27: Principles of effective practice in supporting tutors/teachers

Principles of effective practice in supporting tutors/teachers

An Awarding Organisation should:

• Encourage teachers to share resources and information and create win-win

situations. Awarding Organisations can facilitate this via teacher/practitioner

forums and other IT networking opportunities

• Aim to keep administration and paperwork requirements to a minimum and

allow centres/third parties to develop and use their own paperwork, where

possible

• Encourage teachers and other staff involved in delivering the qualification

to register for relevant Awarding Organisation updates, e-bulletins

and newsletters, and contribute to them. There may also be Awarding

Organisation Facebook pages, Twitter feeds, blogs etc. that they can access

• Provide good quality training to help teachers and other staff deliver the

qualification, for example, passive or interactive online training such as

videos, webinars, skype, or open workshops and seminars, customised in-

house training and consultancy. Training may be free, or included as part of a

registration fee; there may be a fixed or negotiated fee

• Encourage teachers to take relevant qualifications and keep their skills and

knowledge up-to-date through continuing professional development

Some Awarding Organisations create Awards for centres and tutors/ teachers

in recognition of good practice.

5.2.2 Provide support for assessors and quality assurance personnel

All those involved in assessing/examining and quality assurance associated with your

qualification must have clear and sufficient detailed information, and training and support

they reasonably need, to meet their roles and responsibilities (see, for example, chapter 4,

for suggested content of an assessor pack).

It is particularly important that all assessors/examiners and quality assurance personnel understand the importance, purpose and features of the standardisation process (see Box31).

Inexperienced personnel can benefit from exercises during which they may individually

assess/mark the same piece of work or evidence, discuss their assessment decisions and

are then given constructive feedback from an experienced verifier/moderator.


Supporting assessors is part of the responsibilities of internal verifiers/moderators in

ensuring validity of internal assessments and reliability of assessor judgements.

This support should include:

• Providing an induction for new assessors

• Regular observation of assessors carrying out their roles

• Providing constructive feedback on their practice

• Ensuring assessors participate in continuing professional development.

You will have set the parameters for internal assessments that define the scale and

nature of acceptable tasks. Guidance and training should also be available on those

comprising your qualification, for example, on how to: design and mark assignments;

conduct observations of performance; conduct a professional discussion. Also, to explain

and provide examples of recording documentation, such as that to be used in observing

performances and in presenting and verifying evidence.

It is considered good practice for Awarding Organisations to specify that assessors

and quality assurers of their qualifications have appropriate training and hold (or be

working towards) a relevant qualification. This is to ensure they are both competent and

confident, and to support standardisation of assessment decisions and practice across

assessors and centres.

You will need to check the availability of suitable qualifications on the respective registers

of regulated qualifications as, at the time of writing, some are currently being withdrawn/

revised. For guidance on pre-existing qualifications which could be used to inform the

content of training programmes see:

Guidance produced by The Education and Training Foundation (ETF), in association

with City & Guilds (2016, with 2017 update), on qualifications for assessors, internal

and external quality assurance personnel, qualification providers and awarding

institutions. This guidance is available to all Awarding Organisations at: https://www.

feadvice.org.uk/i-am-ite-provider-etf-guidance-documents/assessor-and-quality-

assurance-qualifications

For examiners, an example of an appropriate qualification is the Postgraduate

Certificate in Educational Assessment and Examinations (in association with Cambridge

Assessment). This Level 7 qualification includes principles of assessment (purposes,

validity and quality of forms of assessment, standard setting, consistency and reliability);

validation of assessment evidence in public examinations, including external examining

and awarding.

The Institute of Assessors and Internal Verifiers is a membership organisation that offers

regular professional up-dating, including free lectures and a best practice magazine, plus

The Licentiate grade membership. http://iavltd.co.uk/


Although centres are responsible for ensuring they have competent internal verifiers/

moderators, your Awarding Organisation might offer training on:

• How to conduct observations of, and provide feedback on, assessment practice

• How to conduct internal standardisation; and what to do if assessor practice does not

meet requirements.

External verifiers could deliver this training in the respective centre and/or during

centrally-organised events (face-to-face or online), with the necessary support and

materials.

It will be your Awarding Organisation’s direct responsibility to train external verifiers/

moderators and (where used) external examiners it has contracted with to provide

quality assurance or marking services relating to your qualification. In particular, external

verifiers will require training in how to spot examples of malpractice or maladministration,

and understand the procedures to follow and your Awarding Organisation’s sanctions

policy (see Box 25 for a summary of procedures and section 5.4.3 for process).

If examiners and external verifiers are new to your Awarding Organisation, they will need

an induction programme and to become familiar with the relevant codes of practice. For

inexperienced personnel, it is helpful to offer mentoring from an experienced verifier/

examiner during their early verifying/examining experiences.

All verifiers and examiners will need opportunities to gain advice and up-to-date

information on at least an annual basis.

Again, over time, you will be able to make available exemplars and other resources

drawn from practice to support the training of assessors/examiners and external

verifiers/moderators.


5.2.3 Provide support for learners

Awarding Organisations offering vocational qualifications have an important role to play

in ensuring learners get appropriate information, guidance and support (see chapter 4)

for suggested content for a candidates’ pack). This includes those registered through

centres such as colleges, training providers and employers, as well as those engaging

directly with an Awarding Organisation.

Your Awarding Organisation will have to ensure that the rights of individual learners to

access qualifications and assessments are met in a wide range of circumstances. Where

minimum entry requirements have been set, these must be appropriate and not create

discriminatory barriers.

Learning providers should provide all learners with a learning agreement or equivalent

document which explains what they are entitled to and what is expected of them. It is also

appropriate for learners to be informed of the appeals/ complaint procedure and what

would happen in the event of the qualification being withdrawn.

Some learners, particularly those working towards higher level qualifications, may be

taking a distance and online route to a qualification and have or need little or no face-to-

face teaching or support from a tutor/teacher. Increasingly, qualification teaching and

learning and assessment resources are available online in a variety of formats.

An online resource bank could include:

• Reading lists

• Self-assessment materials and tools

• Diaries, learning logs, review/ progress records

• Activity work sheets, video clips, webinars

• Past exam papers, practice test papers (if appropriate)

• Links to useful websites and other sources of information.

You should be aware of what should happen in your centres (see Table 19 for questions

regarding learner support). For example, when a learner starts their qualification at a

college, training provider or workplace they should go through induction and complete an

initial assessment process which will help to identify any specific learning and/or support

needs.


Learners will have a range of abilities and needs, and include those with learning

difficulties and/or disabilities. For example, learners with:

• Communication and Interaction needs - difficulties with reading or writing which

may be reduced through use of a reader, scribe, BSL/English Interpreter, screen

reading or voice activated software

• Cognition and learning needs - learning difficulties/comprehension difficulties which

may be assisted by extra time in examinations and support for reading and writing

• Sensory and physical needs - a hearing impairment and/or visual impairment which

may need to have assessment material modified and a BSL/English interpreter,

practical assistant, reader and scribe made available. In addition, learners may

benefit from the use of assistive technology and extra time to complete assessments

• Behavioural, emotional and social needs – those who may benefit from supervised

rest breaks and separate accommodation, either within the centre or at an alternative

venue. Those with attention difficulties may need the use of a prompter

• English as an additional language needs – those who may benefit from extra time

during assessments that are time constrained and/or the use of bilingual dictionaries.

(Drawn from FAB Guide, January 2012: 16)

There is a code of practice which provides statutory guidance for organisations that work

with young people (aged up to 25) who have special educational needs or disabilities116 .

Some candidates may be eligible for Reasonable Adjustments to assessments if their

disability or difficulty places them at a disadvantage in the assessment situation, in

comparison to a person who is not disabled. Every candidate who is disabled will not

necessarily be entitled to, or need, an adjustment to assessment. For example, they may

have developed coping mechanisms which minimise or remove the need for Reasonable

Adjustment. The demands of the qualification should be taken into account.

Awarding Organisations and centres/third parties have a responsibility to ensure that

the process of assessment is robust and fair, and allows the candidate to show what they

know and can do without circumventing the assessment criteria.

Examples of types of adjustment may include:

• Changes to assessment conditions

• Use of mechanical, electronic and technological aids

• Modifications to presentation of assessment material

• Alternative ways of presenting responses

• Use of an access facilitator.

However, adjustments to assessments: should not invalidate the assessment

requirements of the qualification; should not give the candidate an unfair advantage.

Adjustments should reflect the candidate’s normal way of working and be based on the

individual needs of the candidate.

For those who are not eligible for Reasonable Adjustments, but could be at a

disadvantage, your Awarding Organisation may be able to make adjustments under the

regulation for Special Consideration.

116 Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_data/fil e/398815/SEND_Code_of_Practice_January_2015.pdf


Table 19: Questions that might be asked by an Awarding Organisation representative when monitoring centre practices in relation to learners and examples of evidence

Support & Monitoring of Learners Appropriate Evidence may include:

a) What sort of induction programme is provided for learners?

• Information about the qualification for learners, e.g. hand-

outs/booklets/PowerPoint presentations

• Documents from induction programmes, dated where possible

• Minutes from meetings where information for learners and

induction programme are discussed

• Lesson plans

b) How are learners’ learning and development needs met?

• Minutes from meetings where learners’ learning and

development needs are discussed

• Documents which demonstrate communication between staff

regarding learners’ learning and development needs

• Records of discussions with learners regarding their learning

and development needs

• Baseline results from initial assessment

• Individual Learning Plans (ILP)

• Evidence of the process of tracking learners’ learning, such as

an action plan or Independent Learning Plan (ILP)

• Evidence of the differentiation of work

c) What arrangements are provided for review and on-going support?

• Regular review schedule

• Monitoring/mentoring programme

• Minutes from meetings where on-going support for learners is

discussed


regarding arrangements for review and ongoing support of

learners

• Records of discussions with learners to review their learning

and on-going support

d) What records are used to monitor learners’ progress throughout qualifications?

• A signed and dated document, such as a table in MS Word or

in Excel, showing all the learners and what work they have

completed

• Other on-line tracking of assessment/internal moderation

• Wall charts, record sheets


Support & Monitoring of Learners Appropriate Evidence may include:

e) What support is provided for learners with special educational needs and disabilities?

• SEND policy, referral documents

• Adapted materials for those with special needs and disabilities

used in the teaching of qualifications

• Minutes from meetings where support for learners with

special needs is discussed

• Specialised Teaching Assistant timetable


regarding providing support for learners with special needs

• Records of discussions with learners with special needs to

ascertain whether their needs are being met

• Individual Education Plans (IEPs)

f) What evidence is there of a centre data protection policy and is it being applied to ensure all learner details are kept confidential?

• Data protection policy and user instructions

• Management systems in place to protect candidate data,

such a computer software and security considerations

• Signed declarations from learners

• Staff handbook

g) What arrangements are in place to support a learner who wants to make a complaint or appeal?

• Centre appeals policy and complaints policy

• Information about the complaint process and appeals process

located in an area accessible to learners

• Documents given out to learners containing information about

these processes

• Minutes from a meeting where support for learners who might

want to go through the appeals process is discussed

• Information about administrative systems in place for learners

going through the appeals process

h) How is access and equality of opportunity ensured and fair assessment for all learners guaranteed?

• The centre’s equal opportunities/inclusion policy and selection

procedures

• Statistics demonstrating that the centre monitors access to

the Awarding Organisation’s qualifications

• Minutes from meetings, and documents which demonstrate

action has been taken to ensure equality of opportunity for

all candidates


Special Considerations

This regulatory condition ensures that candidates who have experienced an illness, injury

or other event outside of their control, which will affect their ability to demonstrate their

knowledge, skills and understanding, are not unfairly advantaged or disadvantaged (as far

as is possible).

Special consideration could include, but is not limited to:

• An adjustment to the arrangements for accessing an assessment for a candidate who

is not disabled, but whose ability to access the assessment has been affected by an

injury or illness

• An adjustment to a learner’s mark for a learner whose performance in an assessment

has been affected

• Awarding additional marks where a learner has been able to take an exam, but where

their performance has been affected by an illness, injury or other exceptional event

outside of their control

• Making arrangements to allow a learner to access an exam, such as providing

assistance to write in a written exam for a learner who has suffered an injury which

prevents them from writing

• Allowing an alternative assessment opportunity for the learner at a later date

• Carrying over the fee for an assessment that the learner was unable to take to a later

assessment opportunity117.

Take into account your qualification’s objective when deciding on the types of Special

Consideration, if any, that should be available.

Do not provide special consideration where this would alter or prevent your qualification

from providing a reliable indication of the knowledge, skills and understanding being

measured by the assessment objectives.

5.3 Understand what happens when candidates take assessments

For all assessments, it is helpful to think them through from the perspective of the

candidates: what they need to know and understand about the assessment to be in a

position to put in their best performance. Also, to consider arrangements for conducting

the assessments and the associated materials, procedures and communications required.

You will then be in a better position to ensure that centres/third parties can be given all

the necessary information to deliver your qualification.

The previous section explained about Reasonable Adjustments and Special

Consideration for particular candidates, when granted in advance.

117 List drawn from Ofqual (July 2016) Guidance to the General Conditions of Recognition, page 91-94


However, a candidate who is fully prepared and present for a scheduled assessment may

still be eligible for Special Consideration if, for example:

• Performance in the assessment is affected by circumstances beyond the control

of the candidate such as recent personal illness, accident, bereavement or serious

disturbance during the assessment

• Alternative assessment arrangements which were agreed in advance of the

assessment prove inappropriate or inadequate

• Part of an assessment is missed due to circumstances beyond the control of the

candidate

• There is a sufficient difference between the part of the assessment to which

Special Consideration is applied and other parts of the qualification that have been

achieved to infer that the candidate could have performed more successfully in the

assessment. (FAB, January 2012118)

Each Awarding Organisation will have its own arrangements for applications for Special

Consideration. But, you will require evidence to be supplied by the centre/third party that

the candidate has been affected at the time of the assessment by a particular condition.

Note that a candidate will not be eligible if any part of the assessment is missed due to:

personal arrangements including holidays or unauthorised absence; preparation for a

component is affected by difficulties during the course, such as disturbances through

building work, lack of proper facilities, changes in or shortages of staff, or industrial

disputes.

The candidate’s result must reflect their achievement in the assessment and not

necessarily their potential ability. Special consideration, if successful, may result in a small

post-assessment adjustment to the mark of the candidate. The size of the adjustment, if

any, will depend on the circumstances and reflect the difficulty faced by the candidate.

A note on contingency plans

What schools and colleges should do if exams or other assessments are seriously

disrupted is outlined in the UK Government’s Contingency Plan for England, Wales

and Northern Ireland. This may be downloaded at: https://www.gov.uk/government/

publications/exam-system-contingency-planengland-wales-and-northern-ireland

The following few pages offer examples of what might happen during particular types of

assessment: internal assessment using portfolios; written examinations.

They are for illustrative purposes only to inform your thinking about how, for example,

your Awarding Organisation will monitor centre/third party practice.

In section 5.4 consideration is given to maintaining security with these and some other

types of assessments.

118 Taken from FAB Guide on the Application of Reasonable Adjustments and Special Consideration in vocational qualifications, January 2012


Example of internal assessment using portfolios

Figure 13 illustrates a process for building, presenting and verifying portfolios.

Figure 13 Building, presenting and verifying a portfolio

Portfolio building introduced to the candidate

The candidate is introduced to portfolio-based assessment

and what they need to do

Portfolio building starts

The candidate starts to compile evidence from different sources, for example, work that

they have produced and evidence provided by others such as that from observations

and discussions. Progress is reviewed at regular intervals with an appropriate person

(supervisor, tutor, assessor)

Portfolio completed and assessed

When evidence gathering is complete, the candidate organises their evidence in a

logical way and as advised, for example: lists evidence under unit headings; inserts page

numbers; completes Awarding Organisation mandatory documentation as far as they

can; signs and dates evidence records/ assessment checklists. Candidate passes on

their portfolio to the assessor who checks the evidence against the assessment criteria

for validity, sufficiency and reliability. If considered acceptable, the assessor signs a

statement to that effect and completes relevant documentation

Portfolio internally verified

Assessor passes on the portfolio to the centre’s Internal Verifier who is responsible for

checking assessment decisions and providing feedback to the assessor. If the portfolio is

judged to be a pass (not needing to be referred back to the candidate), the centre notifies

the Awarding Organisation

Portfolio externally verified

The completed portfolio is made available for sampling by the External Verifier. If

selected for sampling, the candidate may be interviewed to allow for feedback. if

acceptable, achievement is recorded and passed on to the Awarding Organisation.

Portfolios are returned to the centre

Awarding Organisation checks and records achievement

When any other requirements are met, such as successful outcomes from tests or

examinations, the awarding organisation issues a certificate


In monitoring centres, your Awarding Organisation will be concerned that centres ensure

key staff (assessors, internal verifiers, tutors/teachers, supervisors) understand their

roles and responsibilities in relation to portfolio-based assessment associated with your

qualification. Assessors should be able to judge when portfolio evidence is authenticated

and of appropriate form and sufficiency for meeting the assessment criteria, and be able

to provide guidance and feedback to candidates.

Centres should also have available information and documentation for others who

may be appropriate to contribute to candidates’ portfolio evidence, such as workplace

supervisors, line managers or customers. For example, an observation checklist could be

provided for line managers to complete, with guidance which includes a request to record

specific examples of what the candidate actually said or did.

In advance of assessments, candidates should have all the information they need to

understand the purpose of a portfolio, what they need to do in compiling it and who will

be involved in assessing and verifying their work.

Candidates should be made aware of the variety of evidence that could be presented, as

relevant to the nature of the qualification and the particular context in which they work.

Examples might include:

• Written evidence they have produced such as reports, assignments, notes, extracts

from reflective journals, logbooks and work diaries, letters/emails, action plans,

printouts of screen displays

• Evidence that others have provided (suitably referenced) such as observation

records, records of discussions, questioning and witness statements

• Graphics such as sketches, diagrams, charts, technical drawings, design work

• Audio-visual evidence such as photographs, sound and moving image recordings,

accompanied by explanatory statements

• Artefacts and products or assessor records from examining these and a set of

photographs.

Box 28: Information for candidates about assessment via observations

Observations

Where observations are being used for assessment, the candidate needs to be

clear about the criteria for assessment and what the observer hopes to see, for

example, the demonstration of specific skills, knowledge, behaviour. They will

also need to know if there is anything to do in advance of the observation or have

with them. They could have access to a copy of the observation checklist to be

used.

The candidate should be provided with opportunities to practise for summative

assessment and get feedback on their performance so they are aware of

improvements that need to be made. This can help to increase candidate

confidence and reduce anxiety. They should be reassured that if there is

insufficient evidence from the observation there will be other assessment

opportunities. The location and timing of the observation should be appropriate

and fair to the candidate. Candidates with particular needs may require special

arrangements.


Methods for organising the evidence and recording documents should be fully explained.

Candidates need to understand that quality is more important than quantity. They should

be advised on how to keep their portfolio evidence safe, for example, by backing up

electronic evidence, keeping hard copies in a safe place.

Those with disabilities should be made aware of any special provisions that are available

to them. Where a candidate’s work is annotated or transcribed to aid legibility, a covering

statement/letter should be included to confirm that it is the candidate’s own, unaided,

work and represents the candidate’s own words.

During the process of building their portfolio, keeping an evidence log, to date and briefly

describe each item of evidence, can be helpful. Candidates should have regular reviews

of their progress and guidance, for example, in checking that their evidence covers the

relevant assessment criteria (that it is sufficient, valid evidence) and in planning further

opportunities for gathering evidence. Portfolio evidence should be stored securely.

When structuring and presenting evidence in portfolios, candidates should meet the

requirements set out by your Awarding Organisation. This is likely to include completion

of evidence summary sheets which index evidence relating to assessment criteria for

each component of the qualification. Cross-referencing against assessment criteria for

more than one component helps candidates to make maximum use of their evidence.

Following assessment and checking by the internal verifier, the portfolio may be included

in the sample selected for external verification.

Box 29: Sampling of portfolios

Sampling of portfolios

All portfolios should be available, so that the external verifier may request others

to look if there are concerns about the initial sample. They may include paper-

based or electronic portfolios.

The number of portfolios sampled will depend on the number of entries for

the qualification. If it is the first time the centre has assessed the particular

qualification, then there would be an expectation of a larger sample.

The sample is likely to cover:

• All assessors for the qualification, including those who may work in satellite

sites, and those with different levels of experience (more portfolios would be

sampled for new assessors than experienced ones)

• All internal verifiers (if more than one) responsible for the qualification

• All types of candidates, for example, different groups, full-time and part-time

learners, those working on different sites

• All assessment methods used.

Awarding Organisations may specify that candidates should be interviewed with

their portfolios.


Candidates should be given opportunities to resubmit a portfolio if evidence is judged not

to meet the required standard first time around. They should also be aware of their right

to appeal and know what the procedure involves.

Example of external assessment - written examinations

Where centres are responsible for examinations, your Awarding Organisation will need

to make sure they have appointed and trained invigilators to conduct the examination

with candidates.

Centres should have an appropriate room (quiet, suitably heated and ventilated), and

consider candidates with a disability in terms of access and emergency evacuation

procedures. Plans for dealing with any disruption should be in place and communicated

to all relevant staff.

Figure 14: A process for written examinations

In advance of the examination

Candidates should have information about: the date, time, place of the

examination; the nature of the examination; anything they need to bring/have in

order to complete the assessment; expected behaviour during the examination.

On the day of the examination

It is the responsibility of the examination invigilator to ensure:

• Examination room walls are cleared of anything that may help candidates

• Seating arrangements are such as to allow all candidates to have a desk or

table big enough to hold exam papers, to face the same direction and prevent

them from overlooking others’ work

• There is a reliable clock, clearly visible to all candidates and ‘examination in

progress’ notices are on the door/s

• Essential information is on prominent display, e.g. centre number, subject /

Unit title and paper number; actual starting and finishing times of the exam.,

emergency evacuation procedures, centre appeal procedure.

Exam papers and blank scripts are taken to the exam room by the examination

officer.


During the examination

The invigilator:

• Checks all candidates have the required identity documents; completes the

relevant section of the Candidate Information Sheet (where used)

• Advises candidates of emergency/evacuation procedure, examination

regulations to follow, including leaving unauthorised materials, mobiles

outside of the room

• Opens the packets of examination papers, in the examination room in front of

the candidates, distributes these and checks that candidates have the correct

paper

• Specifies the length, start and finish time of the examination, and the earliest

time that candidates can leave the room

• Reads the Instructions on the front of response sheet to candidates and

demonstrates how to complete the sheet and make changes to answers.

It should be announced clearly when candidates may begin the exam. Invigilators

should do this in accordance with Awarding Organisation and regulatory

requirements in order to:

• Ensure that all candidates have an equal opportunity to demonstrate their

abilities

• Assure the security of the examination papers and completed response

sheets, before, during and after the examination

• Prevent possible candidate malpractice

• Prevent possible administrative failures.

The invigilator must be present in the examination room at all times; questions

papers should never be left unattended. They must supervise and give complete

attention to candidates throughout the whole time that an examination is in

progress, moving around the room at frequent intervals and giving regular time

checks. They must be alert to cheating, malpractice and situations that may occur

(e.g. a candidate feeling unwell, needing to be accompanied to the toilet); record

any suspicion or incident.

At the end of the examination the invigilator should tell the candidates to stop

working and remind them that they are still under examination conditions;

collect all examination papers and examination response sheets before

candidates are allowed to leave.

After the examination

Exam papers, completed scripts, and any blank scripts, are then collected by the

exam officer and stored securely until collected by the courier who takes them to

the Awarding Organisation for marking. Scripts to be marked online are scanned

and kept securely by the Awarding Organisation. Those to be marked in hard

copy are sent to individual examiners.


5.4 Maintain controls and security, deal with malpractice

5.4.1 Maintaining controls

Your Awarding Organisation will be operating within a system in which the regulators

exercise ‘controls over entry into the regulated market’ and set ‘regulatory requirements’

to ensure the provision of ‘high-quality, sufficiently valid qualifications’. The regulators

check their requirements are being met, for example, ‘by auditing Awarding

Organisations and by evaluating the actual performance of qualifications’ (Ofqual, March

2015: 5-6119).

In turn, your Awarding organisation is expected to exercise controls on those who deliver

its qualifications, for example, by having policies, systems and procedures in place which

set out the requirements to be an approved centre, or a contracted third party, and

monitoring practice and compliance with these requirements.

Centre approval is a process during which a centre is judged to be capable, or not, of

delivering the qualification. It is a check to ensure that a key level of responsibility for

assessment can be assigned to the centre. Ongoing checks of continued fitness for

purpose is conducted by using external verifiers or audit processes. As described in

section 5.1.4 written, enforceable agreements should be in place.

For some experienced centres, direct claims status may be appropriate (see Box 30).

Box 30: Direct claims status

Direct Claims Status (DCS)

A centre with direct claims status is judged to have a skilled and knowledgeable

internal verifier/moderator and robust quality systems. These enable it to make

the right decisions about awards to candidates, without constant checks by the

Awarding Organisation.

A centre with DCS can directly claim certificates for candidates, thus reducing

the need for an external verifier/moderator to visit so often to monitor the

centre’s status. Much will depend on the size of the centre.

If your qualification has external assessment, then your Awarding Organisation appoints

someone to carry out that role: an external assessor or examiner. Your Awarding

Organisation is responsible for recruiting staff for external roles and for training and

supporting them. After examination scripts are marked by the examiner, rigorous checks

need to be made (usually by senior examiners) to ensure: marking is consistent with your

mark scheme; that, where applicable, conversions from raw marks to UMS scores have

been calculated accurately and the application of grade boundaries is correct.

119 Ofqual (March 2015) Regulatory Strategy Statement


If your qualification has internal assessment delivered through an approved centre,

this is likely to involve a two stage process usually referred to as ‘internal and external

verification’. The approved centre has a responsibility to quality assure assessment

decisions within its centre and your Awarding Organisation appoints someone such as

an external verifier to check the centre’s quality assurance practice. The roles of internal

and external verifiers are described in section 5.1.5.

For details of quality control and quality assurance for some internally and externally

assessed tasks, revisit Chapter 4, 4.2.1.

Part of quality control is to ensure that centres have robust internal quality assurance

processes, supported and monitored by your Awarding Organisation using, for example,

external verifiers.

Features of effective internal quality assurance include:

• Policies, procedures, systems in place to support good practice and reduce the risks of

malpractice and maladministration

• Senior manager support and supervision

• Competent staff with relevant qualifications and experience

• Scheduled meetings and regular communications between assessors/ verifiers,

including qualification review meetings, and records of decisions

• Observations of assessment practice

• Staff development activities

• Regular communication between assessors and verifiers

• Positive feedback from external quality assurance personnel.

Standardisation is key feature of effective quality assurance (see Box 31).


Box 31. Key features of internal standardisation

Internal standardisation

The purpose of internal standardisation is to monitor standards and to ensure

decisions across all assessors in a centre, or a third party, are consistent and

reliable.

The process should be overseen by a nominated quality assurance person,

such as an internal verifier/moderator, who would be expected to: coordinate

assessment activities; provide advice and support to assessors, including advice

on interpretation of standards; monitor and observe assessment practice.

Internal standardisation should involve: a review of assessment tasks, such as

assignments which are internally-set; standardisation of assessment decisions;

sampling of assessment evidence; reviews of assessment practice.

• All assessors should be involved, including any working on different sites

• All ranges of assessment decisions should be reviewed

• The amount of work sampled should be proportionate to qualification entry

numbers for the particular cohort and characteristics of this cohort

• All assessment methods used should be covered.

External verifiers, working on behalf of the Awarding Organisation, will expect

to see evidence of this process taking place on an ongoing basis. Good records

should be kept of the whole process, decisions and action taken to address, for

example, inconsistences. Where necessary, work should be reassessed. Feedback

should be given to the assessor/s.


5.4.2 Maintaining security

Security is a major concern throughout the whole process from developing the

assessment tasks to certification. The key is to look at fit for purpose methods of

protecting the assessment process from any risks to authenticity, security and any

breaches of confidentiality.

Confidentiality

Protection and building confidence in the processes surrounding assessment involves:

• The implementation of confidentiality policies and processes for staff and

contractors, making sure their application is actively monitored

• Working to clear terms of reference for investigations undertaken into suspected or

alleged breaches of confidentiality

• Using a log to record of all allegations of breaches of confidentiality, including those

that are not investigated

• Keeping risk registers and conflict of interest declarations that assessors, contractors

and associates have signed.

Where there are suspected or alleged breaches of confidentiality (including through the

loss or theft of confidential assessment materials), your Awarding Organisation will be

expected to:

• Investigate that breach

• Ensure that any investigation is carried out rigorously, effectively and by appropriate

people who have no personal interest in its outcome; and

• Establish whether a breach of such confidentiality has occurred, so far as possible.

Authenticity

This relates to the evidence being genuine: it is the candidate’s own work. Candidates

should be aware of the consequences of copying and passing off other peoples’ work as

their own; they should be advised on how to acknowledge sources of information.

The ways to mitigate risk to authenticity will depend on the nature of the assessment

methods used. For example, one set of arrangements would involve independent

invigilators checking identity and presiding over an examination, and the secure transfer

and storage of candidates’ work.

Security

The processes around gathering, storing and handling assessments have to be secure and

the integrity of the evidence protected and assured at all stages. See Box 32 for what

should happen for on-screen tests.


Box 32. Security considerations for on-screen tests taken within a centre

Security considerations for on-screen tests taken within a centre

The Awarding Organisation sets the minimum technical specifications for

on-screen tests, and authorises centres to use them that have the necessary

expertise and delivery support system, as well as the relevant software installed

to administer them. There must be appropriate policies and procedures in place

with regard to securities.

The centres, sufficient work stations should be available, including at least one

replacement computer, and printers, where required. The equipment must be

fit for purpose, and checked by a competent person before use. Centres may

consider it useful to use one workstation as a ‘control centre’ monitored by an

invigilator or IT support team member.

Management of the secure test environment must be robust: the network and

the security of the hardware and software to deliver the tests, and candidate

responses, must be protected. Centres must ensure security arrangements

are in place to prevent unauthorised access to the test materials and external

communication with other computer users; whether or not it is intentional, this

may constitute malpractice.

When taking the test, the candidate is responsible for inputting their ID and

password, and ensuring that the name of the test and their details are correct.

Requirements for supervision of candidates between tests may vary. For

example, where the test is restricted to a set examination window, confidential

material (including discarded print-outs) must be kept secure throughout the

range of dates within which the test can be taken. Where the test is made

available on demand, once the candidate has completed it, he or she may leave

the examination room.

The invigilator must: make sure candidates are familiar with the instructions,

procedures and regulations for the test, particularly how to navigate and

respond; check that candidates know how to: request technical assistance; check

that all candidates have logged on successfully, or have been logged on by the

centre; ensure that technical support is available throughout the on-screen test

in relation to malfunctioning of equipment, software or test itself.

At the end of the examination, the invigilator must: ensure that software is closed

as necessary (some do it automatically); check that any necessary back-ups have

been made and stored securely; collect copies of candidates’ work and additional

print-outs before candidates leave the examination room; ensure that candidates

are supervised at all times, if they are required to print work off outside the time

allowed for the test.


5.4.3 Dealing with malpractice

Sources of potential malpractice include:

• The candidate – examples of malpractice include:

plagiarism; identity fraud; collusion with others; inclusion of inappropriate, offensive,

discriminatory or obscene material in assessment evidence (this includes vulgarity

and swearing that is outside of the context of the assessment, or any material of a

discriminatory nature, including racism, sexism and homophobia); inappropriate

behaviour during an internal assessment that causes disruption to others, including

shouting and/or aggressive behaviour or language and having an unauthorised

electronic device that causes a disturbance during the assessment; producing

frivolous content unrelated to the assessment in scripts or coursework; use of

unauthorised aids and possession of unauthorised materials (including mobile

phones, notes and so on)

• The centre - examples of malpractice include:

insecure storage of assessment instruments and marking guidance; misuse of

assessments, including inappropriate adjustments to assessment decisions; failure

to comply with requirements for accurate and safe retention of candidate evidence,

assessment and internal verification records; failure to comply with your Awarding

Organisation’s procedures for managing and transferring accurate candidate data;

deliberate falsification of records in order to claim Reasonable Adjustments or

Special Consideration, or gain certificates

• Member of centre staff or contractor - examples of malpractice include:

a breach of security such as failure to keep exam material secure, tampering with

coursework; deception like manufacturing evidence of competence, fabricating

assessment or internal verification records; the provision of improper assistance to

candidates such as permitting the use of a Reasonable Adjustment over and above

the extent permitted by your Awarding Organisation’s policy; prompting candidates

in assessments by means of signs or verbal or written prompts; excessive direction

from assessors to candidates on how to meet the standards.

You should also be aware of the potential malpractice in your own organisation. For

example: insecure storage of assessment tasks and marking guidance; misuse of

assessments, including inappropriate adjustments to assessment decisions; deliberate

falsification of records in order to support the issuing of certificates; knowingly failing to

adhere to regulations.


Your Awarding Organisation will be required to:

• Take all reasonable steps to prevent the occurrence and recurrence of any

malpractice or maladministration

• Establish whether malpractice has occurred and promptly take all reasonable steps to

prevent or mitigate any ‘Adverse Effects’

» An ‘adverse effect’ is any act, omission, event, incident or circumstance that: prejudices learners; affects public confidence in qualifications; affects the standards of qualifications which the Awarding Organisation makes available; or, affects its ability of the awarding body to undertake development, delivery or award of qualifications in accordance with the regulatory conditions

• Have, and comply with, up-to-date written procedures for the investigation of

suspected or alleged malpractice or maladministration, and a sanctions policy

• Ensure that all investigations are carried out rigorously by appropriate people and

take appropriate action (sanctions) against those responsible

• Review how each centre/third party deals with, or intends to prevent and investigate

any cases of malpractice or maladministration

• Provide (upon request) guidance to centres/third parties on how to best to prevent,

investigate and deal with malpractice and maladministration.

An analysis should be undertaken of the level of risk your Awarding Organisation faces.

In conducting this analysis, it may be useful to consider some of the causes of malpractice,

such as the funding linked to certain qualifications (including apprenticeship funding)

to help to identify areas that may be at a greater risk of malpractice. For example, if a

qualification attracts high rates of funding for a centre/provider then it may present a

higher risk than non-funded qualifications.

There may of course be a wide range of causes of malpractice, and it is important to

identify what these might be in relation to your qualification and appropriate action to

prevent and deal with any malpractice that arises.

However, even when robust steps are taken to prevent malpractice it is possible that

your Awarding Organisation may still face cases of actual or suspected malpractice. It is

important that these cases are dealt with promptly and in line with the requirements of

the regulators and your Awarding Organisation’s procedures.

Dealing with malpractice allegations may involve the following phases set out

in Figure 15.

Notify ing the regulator/s

Investigation


Figure 15: Phases in dealing with malpractice

Identification

Awarding Organisation response

Notifying the regulators

Investigation

Report

Decision

Sanctions, penalties

Appeals against decision

Maintaining Records

Alerting other Awarding Organisations


Identification

There will be a number of ways to identify malpractice, for example: at centre level

through on-going internal quality assurance activity and monitoring, feedback from

centre staff or learners; at Awarding Organisation level, through external quality

assurance activity, complaints, information from other Awarding Organisations; at

regulator level via feedback from stakeholders and others.

Awarding Organisation response

This may include:

• Taking no further action

• Bringing the matter to the attention of the Head of Centre or their nominee asking

them to investigate the alleged malpractice and to produce a written report on the

outcome

• Investigating the matter directly, or by a nominated third party, especially where

there is an allegation of fraud or serious threat to the integrity of certification or

where a centre does not have the capacity to conduct an unbiased investigation

• Considering whether the Regulator/s should be notified of the matter.

Notifying the regulators

Incidents have to be reported to the relevant regulator/s, should it be deemed to cause or

likely to cause an Adverse Effect.

Investigation

In conducting the investigation due regard should be paid to: confidentiality; the rights of

individuals; retention and storage of evidence and records; the need to draw conclusions

and make decisions based on evidence, and agree an action plan; the need for any

sanctions applied to be commensurate with the level of non-compliance identified (and

evidenced).

Report

Where the investigation into the alleged malpractice has been carried out by the centre,

the Head of Centre should submit a written report to your Awarding Organisation. If

your Awarding Organisation has carried it out, the responsible member of staff should

provide the report.

Decision

Your Awarding Organisation should follow its specific procedures for arriving at a final

decision on the malpractice investigation. These procedures might include a malpractice

panel composed of senior members of staff who are experienced in the quality assurance

of assessment and able to determine the outcome in cases of suspected malpractice.


Sanctions and penalties

Sanctions and penalties are imposed on centres where malpractice has been proven

in order to: minimise the risk to the integrity of the qualifications, both now and in the

future; ensure that only those candidates who have reached the required standard are

awarded the qualification; maintain the confidence of the public in the delivery and

awarding of qualifications; deter others from doing likewise. The sanction to be applied,

for example on centres, centre staff or candidates, will depend on the nature and scale of

malpractice and your Awarding Organisation’s policy.

Appeals against decisions

If a centre disagrees with the decision, it can appeal. An appeal against a malpractice

decision will usually be required, in writing, from the Head of Centre requesting a review

of the decision and why it is felt your Awarding Organisation’s decision is wrong. Any

evidence submitted to support this claim must be relevant to the case being made. Your

Awarding Organisation should respond to all appeals in line with its published appeals

policy and procedure.

Maintaining records

Records should include:

• A report containing a statement of the facts, a detailed account of the circumstances

of alleged malpractice, and details of any investigations carried out by the centre into

the suspected case of candidate malpractice

• Written statements from the centre staff and candidates involved

• Any work of the candidate and internal assessment or verification records relevant to

the investigation

• Details of any remedial action to ensure the integrity of certification now and in the

future.

Alerting other awarding bodies

You must notify other Awarding Organisations where these cases are likely to impact

on them. The challenges here include: knowing which other Awarding Organisations to

notify (you may need to request this information from the centre, your own staff, other

Awarding Organisations); knowing who to notify within another Awarding Organisation,

especially where they are large or operate complex structures; knowing how much

information to provide and the legal implications of this.

For further details of dealing with malpractice, see the FAB Guide at:

http://www.awarding.org.uk/images/Guidance/FAB_Malpractice_and_Maladmi

nistration_Guidance_September_2013.pdf


5.5 Evaluate candidates’ performance

This is the step in your qualification’s construction chain (see the measurement lifecycle

in chapter 2 for a reminder) that requires the set of performances for each candidate to

be combined and transformed into an overall measurement result.

This might be as simple as totalling the marks awarded for each response. Typically, though, raw marks will need to be transformed into a reporting metric. This might require a standard setting panel, to determine a cut-off point between passing and failing, according to which the candidate’s raw mark can be classified (as pass or fail). (Ofqual 2017:21120)

It is the task of the aggregator to represent the true value of the assessor’s evaluation

profile in the overall result. The nature of the evaluation profile will depend on the types

of assessment comprising the qualification. For example, it could include the subset

of marks on a test, with the passing mark determined by a standard setting panel, and

the subset of pass/fail judgements on observations of performance against assessment

criteria. To achieve a pass grade, overall, would require a pass for each subset.

A key question is how much of the abilities being tested is enough for candidates to pass? Where other grades are applied, a further question is: what do these actually mean in relation to the target proficiency?

The extent to which the final result is a true representation of the target proficiency

(what has to be measured as expressed in the proficiency specification), will be dependent

on both the quality assurance processes – the consistency in approach to the process

of assessment by all those making the assessment decisions – and the quality control

processes which focus more on the outputs of assessments and how they are judged in a

comparable way.

Standard setting121 is ‘arguably one of the most important tasks in the test development,

administration and reporting process’ Although it does not recognise the full complexity

of setting standards, it can be understood as a ‘process to establish “cut scores” during

which a prescribed procedure must be followed. A number is assigned to create one ‘cut

score’ (to differentiate between two degrees of performance such as pass/fail, allow/deny

a license or award/withhold a credential) or, in some cases, more than two categories’.

(Cizek, 2004).

The Angoff method122 is a one method that has been commonly used for standard setting

in relation to tests.

120 Ofqual (2017) An approach to understanding Validation arguments121 The term ‘standard’ in this instance is referring to the ‘passing scores’, but the term can be used in different ways: standards of content (what needs to

be covered/demonstrated); standards of demand (the assessment challenge); as in standards over time (when comparing assessments, for example, over different years).

122 This method uses subject experts to review a test to predict how many minimally qualified (threshold) learners would answer questions correctly. The average based on the subject experts’ predictions for a test question becomes its predicted difficulty. The sum of each judge’s predicted difficulty values across the test and then averaged across the different subject matter experts, is calculated to give the Angoff cut score. The pass mark is then positioned in relation to this score. Angoff, W.H. (1971). Scales, norms and equivalent scores. Educational Measurements. Washington, DC: American Council on Education. Note that variations to this method are also used


Ward (1980: 104-5), describes two main approaches to determining standards from the

assessment/marking process: norm-referencing; criterion-referencing.

Norm referencing approach

For written examinations/tests, results are usually determined by norm-referencing. In

norm-referencing, the standards set are the same from year to year. Any differences in

the level of marks awarded are due to the paper (its level of difficulty) or its marking. The

same percentage of candidates pass each year. Grading above pass also takes place after

the marking is complete through a process of setting grade boundaries.123 This process is

conducted by an awarding committee appointed by your Awarding Organisation.

(Summarised from Ward, 1980: 105)

Box 33. An approach to determining grades using an awarding panel or committee

An approach to determining grades using an awarding panel or committee 22

Grading takes place as a separate activity to marking and can use a combination of expert judgement and

statistical data.

An awarding panel or committee can be convened which includes: Chair of Examiners who is responsible

overall for maintaining standards across qualifications in a particular vocational subject area; Chief

Examiner for the particular qualification; one or more Principal Examiners (depending on number of

papers, or units). These members need to be carefully selected and trained for the task.

Usually, the awarding session is also attended by an officer from the Awarding Organisation who is

responsible for ensuring correct procedures are followed and a colleague who can provide information

and statistical data, for example: technical information such as mark distributions for the papers; marking

scheme; data on any changes in entry patterns; centre grade estimates; evidence from the regulatory

authority.

During the process of making recommendations on grade boundaries for each paper or unit, panel/

committee members look at examples of candidates’ work. For example, they could start at the top of

the mark range in which the grade boundary is to be expected, and make a judgement about whether the

work is worthy of that grade and then look at the bottom of the mark range to decide whether the work

is not worthy. This usually results in a range of marks known as the ‘zone of uncertainty’. Other methods

used include statistically-produced grade boundaries. The panel/committee is provided with scripts

at the recommended boundary and a mark above and below; they then decide whether the boundary

is acceptable. Grade descriptions can be developed by others in advance, or during the course of the

standard-setting procedure, to illustrate what performance looks like at each grade.

When taken in previous years, archive papers that were on the boundary marks for key grades can be

used to help make a judgement of any difference in demand between the current and past papers.

Information, such as statistical data,is usually fed into the process for members to review and help them

to generate meaningful, realistic judgements. Once panel/committee members have made their individual

decisions on each grade boundary, their judgements are then collated for discussion. Documentation

and other evidence, such as participants’ confidence in the process and results, are assembled to inform

evaluation of the validity of resulting performance standards.

123 Grade boundaries are the marks needed to achieve a particular grade for a qualification. Boundaries are based on the assessment’s level of demand and the performance of all candidates that took the assessment. A qualification will have separate grade boundary decisions made for it each year to ensure that, across the same level, the same amount of effort is required to achieve a particular grade.


The increase in on-screen marking makes it easier for Awarding Organisations to ‘collect details about performance on individual questions and provide this information to awarding committees. Additionally, on-screen marking technologies enable scripts to be scanned before they are marked, and some scripts are now provided on-screen rather than on paper’. (Greatorex, 2008: 4)

In unitised qualifications, when learners may take units across several series where

different grade boundaries have been set, you will need to have a system for comparing

performance on the same unit in different series, and between different units. A Uniform

Mark Scale124 (UMS) can be used for this purpose.

Criterion-referencing approach

In contrast to norm-referencing, in criterion-referencing, candidates pass if they meet

pre-determined criteria. However, standards will only be maintained from year to year if

the criteria are well-designed.

In vocational qualifications which rely on assessment of naturally-occurring evidence

against pre-determined criteria, each performance of the candidate is evaluated in

terms of what it implies about the candidate’s proficiency. The evaluations for each

candidate (with evidence put together in a portfolio) are looked at to determine whether

the candidate has passed the qualification – has met all the assessment criteria and

requirements (which are presented upfront).

In an assignment, grading decisions above a pass can be made by markers by following

instructions from the Awarding Organisation to transform marks into grades.

For example, these instructions could ask markers of an assignment to:

• Add the marks together from each stage of the assignment to give a total mark

• Turn the total mark into a percentage

• Convert the overall percentage mark for each candidate into a grade and band

(using a table provided by the Awarding Organisation)

• Check against grade descriptions.

See Table 20 for what grading descriptions could look like.

124 A uniform mark scale is a mechanism for reporting, recording and aggregating candidates' unit test performances. Each qualification is allocated a total uniform mark, split between the units in proportion to their importance (weighting). Candidates’ UMS for each unit is calculated from their raw mark and then converted using a table, or online calculator. Candidates who achieve the same standard will have the same uniform mark, irrespective of when the unit was taken.


Table 20: What grading descriptions could look like for a practical assignment

Pass Merit Distinction

Evidence as a whole shows:



An acceptable interpretation of the assignment brief

An accurate and fairly innovative interpretation of the assignment brief

An accurate and innovative interpretation of the assignment brief

Adequate quality product-related and documented evidence for the planning, developing and evaluating stages

Good quality product-related and documented evidence for the planning, developing and evaluating stages

High quality product-related and documented evidence for the planning, developing and evaluating stages

Adequate application of integrated knowledge, understanding and skills, but some lack of continuity and consistency

Fairly effective and reasonably consistent application of integrated knowledge, understanding and skills

Effective and consistent application of integrated knowledge, understanding and skills

Relevant and reasonably well-structured plan and evaluation report, with an adequate level shown of subject/occupational expertise in producing the product

Relevant and well-structured plan and evaluation report, with a good level shown of subject/occupational expertise in producing the product

Relevant and tightly structured plan and evaluation report, with a high level shown of subject/ occupational expertise in producing the product

In figure 16, the qualification comprises three different types of component. A points

system is used in assessing or marking each component, points are then totalled and

transformed into an overall grade.

Figure 16: Grade calculated from results for more than one component

Points from

mandatory units

Points from a synpotic

assessment

Qualification

grade calculated

Points from an

external test

Total qualification

points


When there is grading available above a pass (for example, merit and distinction),

this creates greater complexity and risk and requires more grade boundaries to be

determined. It is essential to have good moderation processes for ensuring the reliability

of marking and accurate results overall.

Moderation processes can range from statistical approaches which adjust score

distributions - to ensure comparability of grading between assessment centres like

colleges and/or training providers (an example of quality control) - to group-based

moderation processes using discussion and judgemental review (an example of a quality

assurance approach).

The small margins that govern major statements about achievement need to be carefully

dealt with and re-checked to ensure the candidate receives the level and type of award

their performance deserves.

Borderline-checking is the process where scripts on the borders of grades are re-marked. It needs to happen when the mark boundaries are set and stable and candidates in the ‘risk’ areas can be identified. The areas of achievement that end up in this borderline territory (for example, near passes, not quite a merit, almost a distinction) are key areas in any moderation and quality assurance work for assessors.

The level of risk in the marking process, which adds pressure to boundary decisions,

is influenced by the levels of clarity in the specifications, the quality of the assessors’

understanding and training and the consistency and comparability you can build up in

performance across assessor teams.


5.6 Report results, deal with appeals, award and certificate

This is the stage in your qualification’s lifecycle (step 5 of the construction chain:

measurement interpretation) where you will be providing the measurement result that

will be interpreted by different users of your qualification and interested parties.

‘This happens in the real world, so there may be many interpretations drawn for each

candidate depending on how many times the result is used’ (Ofqual, 2017). For example,

a passing grade indicated on the certificate might be interpreted as meaning that the

candidate is safe and competent to practice the function stated in the qualification title

such as electrical installation.

5.6.1 Report results and deal with appeals

In offering regulated qualifications, your Awarding Organisation has a responsibility to:

• Issue results for all units and qualifications

• Publish expected dates or timescales for the issue of those results

• Issue results which are clear and readily capable of being understood by users of

qualifications

• Issue results which accurately and completely reflect marking of assessments

(including outcome of any moderation, other quality assurance process)

• Ensure that the issue of results is timely

• Take all reasonable steps to meet any date or timescale it has published for

the issue of results.


Generally, the appeals process must cover all qualifications your Awarding Organisation

makes available and must allow for appeals against:

• The results of assessments

• Reasonable Adjustments and Special Consideration decisions and any action to be

taken against a candidate or a centre following an investigation into malpractice or

maladministration.

You will need to ensure that the appeals process is free from risk of any conflict of

interest. This means that it must not be conducted by individuals who may be seen to

have personal interest in the decision being appealed. As an extension of this approach,

the process must involve at least one decision-maker who is not a fellow employee or

connected to your Awarding Organisation. It is also important that the appeals process is

clearly documented, published and available with timelines for the outcome of an appeal

shown.

Should an appeal lead you to discover a failure in your own assessment process, you will

be expected to take all reasonable steps to:

• Establish the scope and impact of the failure by identifying any other candidate who

has been affected by the failure

• Correct or, where it cannot be corrected, mitigate as far as possible the effect of the

failure; and

• Ensure that the failure does not recur in the future.


5.6.2 Award and certificate

Award and certificate is the final stage of what should have been a well-choreographed

process to enable your customers to have a clear sense of the end-to-end assessment

and awarding process. This should include timelines set out, either as a service level

agreement or as a service commitment.

Your systems should be set up to ensure that you deliver on this because it is a stage of

the process that can disproportionally disrupt or impact on candidates and centres/third

parties delivering your qualification.

Early indication of success may help take some of the pressure off waiting for the

final certificate to arrive. There may be key reasons why candidates need proof of

achievement, for example, they may require certain grades to gain admissions to higher

education. However, award and certification can only take place once your quality

procedures have run their course and you are entirely confident in the statement about

achievement you are about to make.

There are different processes that may impact on the ability of your Awarding

Organisation to award and issue certificates in a timely manner. For example: quality

control processes that influence the grades or success rates of cohorts; where final

issuing of the certificate is the ultimate responsibility of another organisation such as the

Institute for Apprenticeships in the case of end-point assessment and certification for

apprenticeships.

In the first case, you should know in advance the time needed to run the quality control

processes. This should form part of managing customer expectations and be built into

your service level agreement (if applicable).

You have a responsibility to ensure that the result of each assessment taken by

a candidate in relation one of your qualifications reflects the level of attainment

demonstrated by them. To do this, you will need to make sure that you take into account

all admissible evidence generated by the candidate as part of those assessments.

Where you specify the amount or type of evidence to be generated by candidates and

what is admissible in an assessment, you should ensure that this is consistency applied

(unless there are any Reasonable Adjustments or Special Consideration that can require

it to be altered).

Similarly, when you establish a rule about how the final mark for a qualification will be

calculated from marks for different assessments, a clear and accessible message needs

to be sent out to those who need to understand it. This rule must be applied to all those

taking the qualification (unless there has been some reasonable adjustments or special

consideration granted that require it to be altered).


There are some basic quality criteria when it comes to the certificates issued. For

example, your Awarding Organisation should:

• Publish the expected dates or timescales for the issue of certificates

• Ensure that the issue of certificates is timely

• Issue only certificates which are clear and readily capable of being understood by

users of qualifications

• Issue only certificates which are accurate and complete and which reflect accurate

and complete results

• Maintain a record of all certificates and replacement certificates issued

• Not include a qualification which is not a regulated qualification on a certificate which

contains regulated qualifications.


Ofqual, for example, requires that all reasonable steps are taken, including having

procedures in place, to ensure that:

• A certificate and any replacement certificate can be issued to any learner who has a

valid entitlement to that certificate or replacement certificate

• Any certificate is not issued to a learner who does not have a valid entitlement to that

certificate,

• Any certificate can be revoked if the result on the certificate is false because of

malpractice, maladministration, or is revealed to be inaccurate because of an appeals

process.

Generally, certificates need to:

• Identify both the learner and the certificate itself

• Clearly display the title of the qualification as it appears on the relevant Register (and

any Endorsement known after the qualification is submitted to the Register)

• Ensure that replacement certificates are identifiable as such.

• The certificate must also indicate if assessment was conducted in another language

(such as not English in England; not Welsh and/or English in Wales)

Different regulators may have slightly different requirements with regard to the content of a certificate, and use of their logos. So, it is wise to check.


Further information

FAB offers a range of training events, guidance, resources and webinars to support this

chapter. For example:

Webinars

9 Legal issues when selling qualifications overseas

9 How the British Council can help awarding organisations with overseas markets

9 The USA vocational qualifications market

9 A practical guide to selling qualifications to employers

9 Ensuring a fit-for-purpose complaints process

9 Ensure exam integrity with Online Proctoring/Invigilation

Guidance

9 FAB Guide to Centre Arrangements

9 FAB Malpractice and Maladministration Guide

9 The Application of Reasonable Adjustments and Special Consideration in Vocational

Qualifications

Training

9 Handling complaints, appeals and enquiries

9 An introduction to malpractice investigation management (Day 1)

9 Malpractice investigation interviewing management (Day 2)


Documents and/or links to reference material cited in this chapter, include:

• Commission on Adult Vocational Teaching and Learning (2013) It’s all about work…Excellent adult vocational teaching and learning. London: LSIS

• Cizek, G. J (2004) Standard Setting. In Downing, S.M. and Haladyna, T.M. (2006)

Handbook of Test Development. London: Lawrence-Erlbaum Associates

• Greatorex, J., Novakovac, N. and Suto I., What attracts judges’ attention – a comparison of grading methods. Annual Conference of International Assessors for Education

Assessors, September 2008

• Robinson, C. Awarding examination grades: current processes and their evolution,

in Newton et al (Eds) (2007) Techniques for monitoring the comparability of examination standards

For further references and information of relevance to this chapter, see Appendix B.

189

Chapter 6

Qualification review and evaluation

This chapter is intended to help you to:

9 Prepare for review and evaluation

9 Review the performance of your qualification

9 Evaluate your qualification, including its validity

9Use data and findings to inform decisions about your qualification.

190Qualification review and evaluation


• Findings from review and evaluation should enable robust justification of a qualification’s validity and

conclusions to be drawn, informed by evidence, that may be used to drive continual improvement and

maintain high standards

• A priority task is to review how assessments have been performing, taking into account outputs

at each stage of the measurement construction chain and feedback from those working with the

assessments

• How users view the qualification and interpret the results will be an important indication of whether,

or not, assessments are performing as they should

• Evaluation is about assessing, with a degree of rigour, the validity of the qualification and drawing

conclusions from the evidence on the extent to which it is achieving its objective

• The range of potential evidence could be extensive and so a systematic approach is needed in

identifying, organising and recording where it can be found

• All evaluators need to consider representativeness when sampling the target population and size of

sample, to allow for looking at different variables and some degree of generalisation when making

claims about the qualification

• Whatever evaluation methods are chosen, it is important to follow an ethical code of practice and

necessary to meet data protection requirements

• In evaluating the validity of the qualification, macro-validation and micro-validation research can be

used to investigate outcome-related and process-related questions

• Type/s and sources of evidence will depend on the qualification’s assessment model: external,

internal or a mix of the two

• Analysis is about looking critically at what the data are saying, to identify patterns and relationships

and draw meaning from them, to find out, for example, the extent to which the validity argument is

justified

• In presenting findings, the needs of different target audiences, and purpose of reporting, should be

considered, with clear reference to evaluation methodology.

This chapter is relevant to reviewing and evaluating a new qualification you have

developed or one that pre-existed and needs to be revised or considered for withdrawal.

Review and evaluation are part of your project management responsibilities. You will

want to assess the effectiveness of your project management to identify lessons learned

to share within your Awarding Organisation and inform future practice.

But, this chapter focuses specifically on reviewing and evaluating the performance of

your qualification. This is when you really find out whether, or not, your data collection,

storage and retrieval system is efficient and meets your needs.

Earlier, the importance was stressed of planning and building in opportunities for review

and evaluation from the start of the qualification development process, with systems

put in place to support data management. You will recall, for example, the importance of

having a validation plan (see chapter 2).

At this stage, it will be worthwhile spending some time reflecting on the content of

previous chapters (see Figure 21) and using the 5WH question technique to help

prepare for your review and evaluation activities.

191 Qualification review and evaluation

6.1 Prepare for review and evaluation

Figure 21: Summaries of chapter content to aid reflection

In Chapter 1, readers were encouraged to plan: how their qualification would be reviewed and evaluated; how they could provide evidence of its validity. It was advised that:

• Quality reviews to inform refinements could be undertaken at various points in a qualification’s lifecycle, with feedback loops to ensure, along the way, that deliverables met the required standard

• Reviews should consider the qualification’s performance against its objectives, the effects of any internal and/or external events on its validity, and lessons learnt that could inform quality improvements.

In Chapter 2 on understanding validity and assessment, information and guidance were given on:

• What makes a qualification sufficiently valid

• How to plan for validation

• Guiding principles for assessment

• Different forms of assessment.

Planning for the validation of a qualification at an early stage can help when it comes to evaluating it, for example, by ensuring relevant data are collected.

See Table 1 for potential sources of validation evidence during a qualification’s lifecycle.

In Chapter 3 on how to design a qualification, information and guidance were given on how to:

• Use market research and consultation with stakeholders to help understand the characteristics, needs and preferences of potential qualification users

• Establish a qualification’s target objective and level

• Specify the abilities a qualification will measure (target proficiency) and what this means in terms of construct and scale (proficiency specification)

• Specify how a qualification could be assessed

(measurement specification).

The processes and outcomes from these activities can all provide evidence to support the validity argument when evaluating a qualification.

In Chapter 4 on how to develop a qualification, information and guidance were given on how to:

• Shape a qualification and its progression

• Develop the ‘assessment apparatus’ (assessment tasks, marking schemes)

• Develop qualification support materials

• Put it all together with a qualification specification to

inform delivery.

Working through the various stages of identifying, challenging and testing thinking about assessment, can help in building an evaluative framework.

In Chapter 5 on delivery of a qualification, information and guidance were provided on how to:

• Prepare for the delivery of a qualification

• Support qualification personnel and learners

• Operate controls and security, and deal with malpractice

• Evaluate candidates’ performances (produce overall results)

• Report results, deal with appeals, award and

certificate

Looking at what happens when candidates take assessments can provide insights into how assessments perform and indicate the types of information that might be available to inform evaluation, for example, policies, assessment and quality assurance records and feedback from customer service activities.

If you have worked through this

handbook, you may find it helpful to

spend time in reflecting on your learning.

Revisiting the content of previous

chapters could serve as a reminder about

processes and outcomes that could be

used to inform the review of

your qualification.

If you are only focusing on the topics of

this chapter, the summary boxes here

will give you an idea of what has been

previously covered.


5WH questions to think through review and evaluation

Why?

Why do you need to review and evaluate your qualification?

There may be several reasons why you need to do this, for example:

• To assess the extent to which your qualification is valid -performing as it should and meeting its objective

• To find out whether, or not, it is meeting the expectations and needs of stakeholders

• To comply with the requirements of the regulator and funding agency

• To satisfy your Awarding Organisation’s requirements

• To secure public confidence in your qualification

• To share lessons learned within your Awarding Organisation and with others

• To inform quality improvements

• To inform decisions on revisions or withdrawal.

Where?

Where will you find evidence of how your qualification is performing?

If you planned for validating your qualification, you will have identified types and potential sources of evidence at each of the lifecycle stages, for example: technical reports on assessment tasks produced at the design stage; records of external verifier checks on centres’ assessment practice during delivery of your qualification. Revisit this plan in checking out what happened in practice at each lifecycle stage.

Who?

Who will you involve in your review and evaluation?

These are likely to include: Awarding Organisation colleagues; subject and assessment experts and stakeholders who have contributed to the development of your qualification; candidates, tutors/teaching staff, assessors and quality assurance personnel; qualification end-users.

When?

When are the outcomes of this process needed?

If your qualification is included on a register of regulated qualifications, it will need a specified review date.

Your Awarding Organisation will specify dates by which it needs information about how your qualification has been performing, for example, to report to its governors, the regulator/s a funding agency, and to include in an annual report.

What?

What data and information will you need?

Different types of data and information will have been collected during the development and delivery of your qualification. If you have been systematic about this, it should now be relatively straightforward to pull out what you need, for example, findings from examiner and external verifier reports and surveys of stakeholders.

But, you will also need to access information, such as data provided by the regulator and funding agency, to put the performance of your qualification into a wider context and enable comparisons against other qualifications.


How?

How will you use data and information?

You will have generated a large quantity of rich data, both quantitative and qualitative.

But, these data will need to be analysed to provide insights into the performance of

your qualification and to inform decisions about any changes to be made.

A lot of your hard work in collecting information will be wasted if results are not

presented effectively (clearly and in an appropriate form) to your target audiences.

You will also need to be aware of how (and why) others will be using the data and

information you present. Guidance would help them to accurately interpret and use

findings.

6.2 Review the performance of your qualification

Regulators of qualifications typically stipulate that an Awarding Organisation must keep

under review its approach to the development, delivery and award of qualifications. This

is for the Awarding Organisation to assure itself that its approach remains appropriate.

It is most likely that you will need to routinely conduct:

• An annual review of the performance of your qualification; and

• More in-depth reviews, for example, every three to five years.

In addition to scheduled reviews, a review may be instigated if a systemic problem is

found during an investigation, for example, into an event that has resulted in an Adverse

Effect or through received complaints.

If your qualification has been operational for some time, your Awarding Organisation

may have already included it in a review process. If it has a large number of qualifications,

this review will most likely have focused on a sample that reflects the diversity of its

qualification offer. For example, to reflect the characteristics of its qualifications in terms

of their: level/type; volume – high and low; funding source; stakeholder engagement.

Characteristics may be weighted according to priorities and risk, for example, those

which are high stakes.

If your qualification has not been subject to review, you can develop your own framework,

to consider and record systematically:

• Types of evidence

• Who is responsible for producing this evidence

• Where and how the evidence is held

• Nature of the evidence

• Use of evidence

• Process and systems support

• Justification of your approach.

(AlphaPlus Consultancy FAB training event, January 2017)


It is worthwhile spending time doing this as outcomes from this review will be helpful

later in defining your evaluation’s objectives and scope, and when justifying validity

claims for your qualification.

What can I expect centres/third parties to contribute to the review?

The Regulator/s and written agreements with your Awarding Organisation will have

required centres/third parties to assist you in carrying out any reasonable monitoring

activities.

So, you can expect centres/third parties will:

• Maintain and make available all Learner records and details of achievement in an

accurate, timely and secure manner in line with the requirements of your Awarding

Organisation and Data Protection Legislation

• Take all reasonable steps to comply with requests for information, data or documents

required by your Awarding Organisation or by the regulator/s.

• Retain complete and accurate records from completion of all qualifications and make

these available to your Awarding Organisation upon request. The records required

may include assessment and verification records, certificate claims, candidate data

for each qualification and so on.

As people work with the assessments, you will have feedback from a range of personnel.

For example, a group of markers can offer a great source of feedback on the extent to

which a mark scheme is clear, useable and fit for purpose. If this is the case, it will be

useful to review the extent to which there is a level of agreement about how it performs.

Maintaining assessment evidence and materials

Part of the responsibility of an Awarding Organisation is to gather and maintain

assessment evidence and materials to inform, for example, the monitoring of standards in

assessment across the range of users of the assessments, and over time if they are used

continually.

If test items are fairly secure and sufficient in number, and the marking standard remains

the same, it is possible to accurately compare candidates’ results over time in tests or

examinations that use objective questions (items) by re-using well-written items of

known difficulty from a bank.

It is less easy to maintain consistency over time for internally assessed work, especially

where assessment tasks are designed by the centre. The expectation here is that centres/

third parties will retain samples of candidates’ work which have been internally quality

assured for monitoring by an external quality assurer.

These samples can be used in reviewing your qualification. However, it should be noted

that such work is the property of the respective candidates. Permission must therefore

be obtained to allow for copies (not originals) to be used.


A priority task is to review how assessments have been performing

Following implementation of assessments, monitoring their performance should

have given you an indication of how they were shaping up against requirements, both

individually and in combination.

Starting with the definition of the target proficiency, go through the construction chain

to check what should have happened at each step – who should have done what. See

Figure 22 for a quick overview and some questions to consider in reviewing the validity of

assessments.

Figure 22: Steps in the construction chain (Newton, 2017)124

124 Newton, P.E. Validity – what it is and why it matters. Presentation at Ofqual Conference 2017, Vox Conference Centre, 28 February

InterpretationStep

Qual. User

An interpretationof the result foreach candidate

Aggregator

An overall result for eachcandidate

Assessor

A set of evaluationsfor each candidate

Candidate

A set of task performances for each candidate

Qual. Designer

The proficiencyspecification

CombinationStep

EvaluationStep

ElicitationStep

ClarificationStep

TargetProficiency

Did you establish the credentials (expertise/integrity) of key agents and their facilitators before appointing them?

Have those key agents and their facilitators been given the right training, guidance and supervision?

Have safeguards been put in place to prevent human error and deliberate subversion?

Have you satisfactorily ensured the accuracy and security of the data thatyou capture and store?

And so on…

And across all five steps

Four examples of many potential validity threats


The outputs available for review from this construction chain will include:

• The proficiency specification - setting out the proficiency construct (the abilities

that the qualification is intended to represent) and proficiency scale (what different

levels of those abilities might look like)

• Measurement specification – setting out assessment tasks (such as an assignment,

a practical task, a multiple-choice test) to elicit performances from each candidate,

together with task-specific materials such as a marking scheme, observation checklist

and quality control/assurance procedures to ensure results reflect the candidate’s

true proficiency

• Sets of task performances and evaluations – samples of how each candidate has

performed and been evaluated on each of the assessment tasks in terms of what it

says about their proficiency

• Overall result – samples of how evaluations have been combined and transformed

into an overall result (through, for example, session-specific standard-setting

activities).

For fuller details of the construction chain, see Chapter 2.

If you took the advice in Chapter 2, you will have planned what evidence you could

collect for validation purposes, and where and when this evidence may be found in

your qualification’s lifecycle. It will now be useful to review this plan and identify what

evidence is actually available for you to review.

Types and sources of evidence that you will have, will be largely dependent upon your

chosen assessment model: whether it is based exclusively on external assessment or

internal assessment, or a mix of the two.

For example:

…if your qualification is awarded on the ‘basis of performance on a single 40-item

multiple choice test which all candidates sit, almost everything will be standardized’.

This makes it ‘quite straightforward to generate many of the traditional examples of

validation evidence and analysis’.

However, if it is based exclusively upon assessment that is devolved to assessors

who use a variety of approaches, ‘many critical assessment processes will not be

standardized at all. Validity will be heavily dependent upon the effectiveness of higher

order features or processes that are designed into the assessment procedure to

ensure that all assessors have sufficient expertise, integrity and understanding of the

qualification standard…’

Critical validation evidence, here, might include ‘assessor credentials, documentation

of assessment strategy approval mechanisms, training and exemplification materials,

documentation of moderation processes, moderation quality control metrics, and so

on’.

(Newton, December 2016: 10/11)

For examples of where you might find potential validation evidence,

see Chapter 2, Table 1.


It is recommended that you [re-]visit Chapter 2, and associated reading, to ensure you

have an understanding of basic concepts and terminology before reviewing evidence to

support your validity arguement.

Part of deciding whether, or not, assessments have been performing as they should, will be how users, including candidates, education and training providers and gatekeepers to employment and higher education, view your qualification and how they interpret the results.

Some indication of how users are perceiving your qualification can be obtained by

reviewing in-house records, for example:

• Customer service records, for example, those held on enquiries and complaints, to

identify any common issues or those that are specific to particular types of customer

• Responses to customer satisfaction surveys, for example, to gain an overview of how

your qualification is being received in different contexts, for example, in a school,

college or workplace

• Records on requests for Reasonable Adjustments, for example, to identify any

barriers resulting from the design of your qualification that could be removed during

revisions

• Records from the appeals process, for example, to identify any issues with particular

assessment tasks and marking procedures

• Documented feedback, for example, from candidates during visits to centres,

participants in training events, stakeholder events

• Statistics kept on uptake/sales, drop-out/growth rates.

It can also be useful to consult with internal staff who were not involved in developing,

delivering or awarding your qualification, to gain their perspectives on assessments and

results.


6.3 Evaluate your qualification, including its validity

6.3.1 An overview of evaluation

What is evaluation about?

Evaluation is about analysing how your qualification is performing and drawing

conclusions about the extent to which it is achieving its objective.

It involves:

• Analysing data and information from the review of your qualification, to gain insights

into your qualification’s strengths and weaknesses

• Supplementing this information by researching the wider context, to draw

conclusions and inform quality improvements.

Validity is of central importance in focusing on assessment and whether, or not, your

qualification accurately measures its target proficiency.

You will be interested in the effectiveness of approaches and efficiency of systems

for ensuring your qualification is sufficiently valid, that is whether, or not, it is testing

appropriate abilities and meeting the required standards: has relevant content (standard

of content); is at the right level of demand (standard of demand); produces results that

accurately reflect candidates’ attainment (standard of attainment).

But, evaluation is broader than a consideration of the validity of your qualification

(important as this aspect is). It also means looking at wide-ranging evidence to make a

judgement about:

• The acceptability of your qualification in terms, for example, of its manageability

within available resources, policy alignment and public credibility

• Any revisions that will need to be made to your qualification in response to changes in

the contexts in which your qualification is used.

The consequences of assessment results will also be of interest, such as the extent to

which your qualification predicts future performance and affects candidates’ life chances,

and impacts on centres, for example, in terms of their access to funding and teaching and

learning.

Of course, what you can do in terms of evaluation will be dependent on available

resources. So, it will be important to capitalise on common evaluation processes

operating across your Awarding Organisation and to decide on priorities in discussion

with colleagues.

One priority is likely to be ensuring you have the statistical data and other information

readily available that is required by the regulator of your qualification. For example, to

enable your Awarding Organisation’s Governing Body to be confident that the annual

statement of compliance is accurate (and documented evidence can be supplied, if

needed) and that any events which could lead to an Adverse Effect have been identified,

notified and addressed.


Evaluation methods

Your evaluation could involve:

• Desk-based methods, for example, scrutiny of reports and other documents, such

as examiner/external verifier reports, the qualification specification and support

materials

• Quantitative methods, for example, customer surveys to find out how many thought

this and that; statistical methods for analysing performance of assessments

• Qualitative methods, for example, interviews and/or focus groups with stakeholders

to elicit their views and experiences of using your qualification, case studies to

illustrate centre practice.

Most evaluations employ a mix of these types of methods. This can help to ensure an

evaluation is robust.

So, how do I make a start on evaluating my qualification?

Firstly, re-visit your starting point for designing your qualification:

• Why it was needed – where the demand was coming from

• Stakeholder interests

• Expected (average) candidate profile

• Predicted numbers

• Target proficiency.

Such information, some of which would have informed your business case, can then be

used as a baseline for evaluating the extent to which your qualification is delivering what

it set out to do.

The next task will be to address questions such as:

• Has your qualification’s objective been found well-defined and supported by a clear

rationale?

• Has your qualification’s structure proved sufficiently clear and appropriate?

• Has progression been clearly communicated?

• Has the measurement specification faithfully reflected the target proficiency and

required standards?

• Have the assessment tasks faithfully reflected the measurement specification?

• Have assessment results been deemed accurate and appropriate?

Some answers will be found in revisiting documentation produced during the design

and development stages of your qualification’s lifecycle, for example, the measurement

specification and technical reports on assessment tasks. Others will require finding out

what your qualification’s users think.

An important question in determining whether, or not, your qualification has met its

objective will be: to what extent has it been of benefit to those achieving it and others

who use it. This will not be quick to answer as it will rely on data being collected from

users some time hence after your qualification has been awarded.


Look at previous evaluations conducted by your Awarding Organisation, and other

bodies, and the methods they have used. For example, Pye Tait’s research commissioned

by Ofqual which was designed to:

• Understand and assess employers’ overall perceptions of, and confidence in, specific

qualifications and assessments

• Determine employers’ use of such qualifications and assessments when recruiting

new staff; and

• Determine employers’ use of such qualifications and assessments as part of training

programmes for staff.

(Pye Tait Consulting, July 2017:16125)

For a reminder of methods suggested for consultation purposes (some of these could also

be used in your evaluation), see Chapter 3.

So, how do I find out what others think about my qualification?

It helps to have a good overview of the different contexts in which your qualification is

used in order to identify those groups whose views you will need seek.

From researching the sector to gauge support for your qualification, you are likely to

have a good feel of its make-up in terms of the mix of large, small and medium enterprises

(SMEs) and micro-businesses, as well as the different types of work they do. You will

already have information on candidates provided by your centres/third parties.

You now need to consider who to involve directly in the evaluation of your qualification.

For example:

• Candidates who reflect the target population

• Employers from the sector for which your qualification was designed

• Users of your qualification, such as employers and professional bodies with an

interest in your qualification who may provide evidence of whether, or not, it is

achieving its objective

• Other external stakeholders that rely on the qualification, for example, gatekeepers

from institutions of Further and Higher Education.

It is likely that you have engaged with most of these groups during your initial and

ongoing consultations to inform the design and development of your qualification.

Start off with candidates and those who were involved in developing your qualification.

Then think about how to reach a wider range of potential respondents and how to select

your sample (the smaller sub-set which is representative of your target population as a

whole).

All evaluators need to consider the representativeness of potential respondents: the

extent to which individuals, groups and situations are typical of the target population as a

whole and contexts in which your qualification is used. This is important because it allows

you to generalise when it comes to making claims about your qualification.

125 Pye Tait Consulting (July 2017). Employer qualification perception survey. Final Report


Judgements will need to be made on: the sample size; the representativeness of the

sample; and access to the sample. Your approach to sampling can be as simple or as

complicated as you like. Much will depend on the finance, resources and time you have

available.

What will be important, though, is to make clear your approach to selecting your sample

when reporting findings and making claims about your qualification. There are various

sampling methods which you can explore. Some are mentioned below in relation to

particular methods for collecting data.

What if I want to use a questionnaire survey?

The steps in developing a survey using a questionnaire are set out in Figure 23.

Where you primarily wish to collect quantitative data using a questionnaire survey – find

out who and how many thought this and that, there are two standard procedures to

consider for selecting your sample:

• Random sampling: whereby potential respondents are selected at random from a list

(the sampling frame). For example, in its more systematic form, to achieve a sample

of 100 from a list of 2,000 candidates who achieved your qualification in a particular

year, every twentieth person could be selected. This may not provide an equal gender

balance, but this requirement can be put into the frame.

• Stratified sampling: a more sophisticated approach whereby the target population is

broken down into groups, each of which contains people with similar characteristics

such as all male, all female, all students taking the qualification in schools. A random

selection is then made from each group of interest to you.

The size of the sample will depend upon the purpose of the survey and the nature of the

target population. A sample size of 30 is suggested by many to be the minimum number

for basic statistical analysis of data.

But, sample size will also depend on what sorts of relationships (variables) you will want

to explore within the subsets of your sample. You need to think about this in advance.

Each subset will need to be big enough to represent the population under survey. For

example, if you are interested in finding out whether there are any differences in how

employer recruiters and HEI admissions officers perceive your qualification, you will need

to have enough of both in your sub-sets to represent their respective population.


Take into account the resources and time you

have available

Decide on the sample taking into account the variables you will want

to explore, and how you intend to analyse the data

Design the questionnaire and

associated instructions

Structure and draft the questions

All closed/some open?

Choose methods for processing and

analysing data

Define your survey objectives

Decide on the information needed and survey method/s, e.g. online, postal, telephone

Pilot survey, taking into account research ethics and protocols, including data protection requirements

Amend questionnaire where necessary and prepare to go live

Re-visit information from the review process to

feed into the evaluation

Figure 23: Planning a survey by questionnaire


What if I intend to use qualitative methods such as focus groups/other interviews?

Surveys by questionnaire will enable you to access larger target populations, with greater

choice in terms of representativeness, than qualitative research methods. Nevertheless,

it will still be important when planning focus groups or individual interviews, to attempt

some acceptable level of representativeness in your sample by including informants with

typical characteristics of their larger group.

Box 34: Spotlight on focus groups

Focus groups

Focus groups can be used to gather qualitative data from qualification users in

a short space of time, to explore their experiences and views. They are a form

of group interview, but differ in that participants are encouraged to interact, to

share and comment on each other’s points of view and experiences, to explain

why they think a particular way about a qualification.

The design of focus group research will vary, based on the nature of the research

questions. But, there are some general principles:

• Standardisation of questions: focus groups can vary in the extent to which

they follow a structured schedule or allow discussion to emerge

• Number of focus groups conducted: will depend on the 'segmentation'

or different stratifications (e.g. age, sex, size of company, type of provider)

identified as important to the topic. Several may be conducted, for example, in

different parts of the country or online. Focus groups for each of the different

sub-groups could be conducted on the same day in the same place, and then

allowed to come together to identify where their views and experiences are

common, and where they differ

• Number of participants per group: 6-10 homogeneous people is typical, but

there may be reasons to have smaller or slightly larger groups

• Level of moderator involvement: extent to which structured questions are

asked and group dynamics are actively managed; this can vary from a high to

low control.

A key consideration is how data are captured, for example: digitally record the

discussion (with participants’ permission) ensuring each speaker at any one point

can be identified; have more than one person to take notes; ask participants to

record their points on flip-charts.

Careful facilitation is needed to help ensure all participants have an opportunity

to make their views known – no one individual dominates the discussion.


Interviews may be conducted face-to-face, by telephone or online. They can range

from ‘fully structured’ (where the same questions are asked in the same order to all

interviewees) to ‘quite open’ (those that have a general framework with a starting point

and objective, but no set questions). ‘Semi-structured’ interviews are inbetween (they can

use both set questions and offer more open opportunities to discuss the topic).

Whatever method/s you choose to employ in eliciting the views of people, you will need

to adhere to an ethical code of practice, including the protocols for conducting surveys

and interviews, confidentiality and data protection requirements.

Your Awarding Organisation is likely to have its own code, but if not, look at examples

provided by others such as that offered by the UK Research Integrity Office (UKRIO).

Available at: http://ukrio.org/publications/code-of-practice-for-research/

What about all these documents I have collected?

In addition to data from the above activities, you will have a range of different documents

which will need to be easily retrieved for analysis. It will also be important to attempt

some kind of classification of these.

Document summary forms will be useful to include, for example: title; type; importance;

who produced it and when; and brief summary of content.

Documentary evidence will include that from reviewing reports and information

provided by other organisations to give, for example, an indication of the value others

place on your qualification. Such evidence may include: inclusion in performance tables

or list of qualifications that are eligible for funding/learner loans; recognition in the UCAS

Tariff; mention in admissions and recruitment literature.

Part of this exploration will be consideration of any internal and/or external events that

may have impacted on your qualification. These may be economic, political or social in

nature.

To come to an understanding about the performance of your qualification in relation to

those of others, firstly look to data held by your Awarding Organisation on its full range of

vocational qualifications.

You can then explore wider sources that provide data on, and analyses of, qualifications.

See, for example research methods used by Coles and Matthews (195126) that looks at

fitness for purpose of qualifications being compared in relation to enabling progression

into employment or further or higher education.

Some examples of sources of information, include:

• Research and reports published by the regulators and relevant government

departments and bodies

• UCAS end of cycle reports on university trends

• Evaluation reports on the implementation of qualifications, from University and other

research centres

• Reports by the Education and training inspectors.

126 Coles, M. and Matthews, A. (1995) Fitness for purpose. A means of comparing qualifications. A report to Sir Ron Dearing (to be considered as part of his review of 16-19 education)


For instance, along with research on topics of direct relevance to developing

qualifications (and archive material), Ofqual publishes an annual qualifications market

report (see Box 35). Although it may not include qualification-specific data, it does offer

an overview on how the market is operating and trends.

References to other sources and weblinks are provided in Appendix B.

Box 35: Examples of data available from Ofqaul

Examples of data available from Ofqual

Ofqual publishes an Annual qualifications market report which now covers both

general and vocational qualifications, plus quarterly updates of the VQ market

which contains headline statistics, along with data, that highlight key trends.

These statistics include:

• Number of qualifications and certificates by level, type and sector subject area

• Number of certificates awarded in the past 5 years

• Number of certificates by second-tier sector subject area

• 50 qualifications with the highest number of certificates

• 50 awarding organisations with the highest number of certificates.

Separate headline statistics are published for England, Wales and Northern

Ireland. The statistics of the quarter are compared to relevant historical data.

The updated VQ quarterly has National Statistics status subject to ongoing

conversations with the Office for Statistics Regulation (the regulatory arm of

the UKSA). The release of data and headline statistics by Ofqual aligns with

UK Statistics Authority’s Business plan 2016-2020 which aims to focus “on

the release of data, metadata and key headlines, moving away from traditional

‘statistical bulletins’ and static publications whenever appropriate, informed by

users.”

In conducting your own evaluation, you will develop a good understanding of the issues

which will also help in making judgements about the robustness of findings and statistics

reported by others.


Evaluating your qualification’s validity

Although you will be evaluating your qualification more broadly, establishing that it has

sufficient validity (and justifying the validity argument), is an important responsibility of

your Awarding Organisation.

In Chapter 2, two main approaches to looking at validation(at either end of a continuum’)

were described: micro validation and macro-validation. Examples of different types

of evidence were given to produce a robust validity argument, together with a set of

questions to consider at each stage of your qualification’s lifecycle. Section 6.2 of this

chapter also refers to types of evidence in relation to different types of assessment.

…macro-validation research tends to investigate outcome-related, or product-related questions (akin to the customer’s perspective); whereas micro-validation research tends to investigate input-related, or process-related questions (akin to the engineer’s perspective)… two sides of the same coin, providing complementary perspectives within a comprehensive validation program… (11) [or a ‘fuzzy continuum’] (7). (Newton, December 2016127: 11/7)

You may find, as Newton suggests, that you have available at an earlier stage more

micro-validation evidence than macro-validation. This is because evidence and analysis

of micro-validation will have arisen, in part, as ‘a natural by-product of designing and

developing assessments’.

If you are evaluating a well-established qualification, you will have an archive of material

on which to draw, such as sampled candidate work that can be mapped against the

qualification specification and standards, and year-on-year records from assessment and

quality assurance activities. For a new qualification, of course, there will be limitations

with regard to the range of evidence available for evaluation purposes.

For macro-validation purposes, you should have evidence drawn from surveys of your

qualification users, maybe from a series of surveys. You may also have findings from

longitudinal studies of candidates which will enable you to make judgements on the

extent to which your qualification is meeting their progression expectations.

127 Newton, P. E. (Ofqual) Macro- and Micro-Validation: Beyond the ‘Five Sources’ Framework for Classifying Validation Evidence and Analysis. In Practical Assessment, Research and Evaluation, Volume 21, Number 12, December 2016.


6.4 Use data and findings

Findings from your evaluation should enable you to provide a robust justification for

your qualification’s validity, supported by stakeholders, and put your qualification into

a wider context. Conclusions drawn from analysis will provide an informed basis for

decisions about your qualification and contribute to driving continual improvement and

maintaining high standards.

Analysis of data

Analysis is about looking critically for what the data are telling you, to identify patterns,

relationships, common threads and consistencies between them, and exceptions to the

rule, and to draw meaning from them that can be justified.

Before reaching the formal stage of analysis, you may have engaged in some analysis

of data during the course of collecting these. This will provide you with some ideas for

classifying and coding the data and for follow-up questions. The coding process will help

to clarify what you will finally claim about your qualification and the grounds for these

claims, to move from description to explanation.

So, your first task will be to structure and organise the data - sort them into manageable

‘chunks’ so it is easier to derive meaning from them and identify their significance. For

example, you could use summary sheets for recording questionnaire responses and

collating interview responses under sub-headings, such as themes. You could display

selected data in graphical forms, such as graphs, charts, tables; use descriptions such as

profiles of participants, quotes from interviews.

IT programs can assist in this process, for example in storage and retrieval, especially

where you are handling large amounts of data. Some programs can do more than this,

but your informed insights will be just as, if not more than, important.

Using outcomes from the analysis

Analysing data collected using different methods (a form of triangulation128) will

strengthen the validity of your findings. Having data of different types from different

sources will offer you an expanded picture of how your qualification is performing.

You will be aware, of course, that it is easy to collect so much data, of such variety, that it

becomes impossible to analyse it all in time to inform decisions about any changes to your

qualification. For example, there may be some urgency to act on feedback received from

regulator/s.

But, particular care needs to be taken that candidates are not unfairly disadvantaged by

making changes too soon. You should also consider whether issues identified may apply

more widely, to other qualifications and/or Awarding Organisation practice. In these

cases, action may need to be agreed with others.

Keep in mind the broader picture, but remember your concern is to make judgements

about the validity of your qualification and to inform decisions about how it can be

improved to better suit its objective.

128 In addition to method triangulation which involves use of more than one method for collecting information, other forms of triangulation used in research include: data triangulation which involves data collected over a period of time from more than one location and more than one person; researcher triangulation which involves, for example, the use of more than one observer for the same performance


You will need to decide whether findings from your evaluation mean that your

qualification:

• Would benefit from refreshment, for example, to boost take-up

• Needs rather more revision, for example to reflect a particular development in the

sector it serves

• Needs to be withdrawn or replaced with a new version, for example, due to changes in

regulations such as requirements for the design of qualifications.

If you have been reviewing and evaluating a pre-existing qualification, findings may

suggest that the time has come to consider its withdrawal. It may be that you are obliged

to do this by the regulator/s as your qualification has come to the end of its shelf-life or is

no longer fit for purpose, or qualification reforms are such that it is not possible to make

revisions to fully comply with new design or other regulatory requirements.

There may be other factors that mean that withdrawal, rather than revision, may be

required.

For example:

• Changes in the specific technical or professional area for your qualification since it

became operational, for example, technological developments requiring new work

tasks and different abilities

• Changes in stakeholder needs and expectations

Wider context changes that may affect your qualification, for example, reforms in

education and training, new regulations.

There may be commercial reasons to do this, such as a marked drop-off in sales, candidate

registrations, certifications; and/or substantial changes in the sector in which the

qualification is offered.

If your qualification is regulated by Ofqual, CCEA Regulation and/or Qualifications Wales,

a condition of recognition will apply to managing withdrawal of your qualification.

SQA Accreditation also has procedures for withdrawing qualifications.

Withdrawal will be at the point when your Awarding Organisation first:

(a) ceases to register learners for the qualification, (b) ceases to deliver or award that qualification to learners, (c) surrenders its recognition in respect of that qualification, or (d) has its recognition withdrawn by the regulator in respect of that qualification’

(CCEA Regulation, May 2017).

Regardless of regulator, you will need to plan for withdrawal, with dates for: final

registrations for new candidates and assessments and retake opportunities for existing

learners; and deadlines for completing actions. You will need to keep your plan under

review, seeking and analysing feedback about how the withdrawal is progressing; and

make changes to the plan, as necessary.

Your Awarding Organisation will be concerned to keep its customers, for example by

informing them of alternative or replacement qualifications that may suit their purpose.


Presenting and sharing evaluation findings

You will want to share findings from your evaluation. But in doing this, you will need to

decide what you will share, how you will do this and with whom you will share (internally

and externally).

There will be a variety of people who will be interested, for example:

• Colleagues within your Awarding

Organisation

• Centres and third parties that have

been delivering your qualification

• Other potential customers

• Competitors

• Stakeholders

• The regulators

• Funding agency

• Policy makers

• Researchers

• Careers guidance personnel

• The general public.

These target audiences are likely to require findings to be presented in different ways and

at different levels of detail to meet their needs.

Some organisations to which you are required to report, may ask for information to be

structured in a certain way and available in a particular format.

In all cases, however, information should be included that enables readers to make

judgements on the robustness of the research, so they can be confident in the results.

Some may appreciate full details of methodology, others the basics, including survey

numbers and response rates.

209


Further information

FAB offers training and webinars of relevance to this chapter such as those on planning

for validation.

• Check the website for forthcoming events.

• Examples of information available from: FAB library: www.awarding.org.uk

References in FAB’s library include:

9 Coles, M. and Matthews, A. (1995) Fitness for purpose. A means of comparing qualifications. A report to Sir Ron Dearing

9 Newton, P.E. Validity – what it is and why it matters. Presentation at Ofqual Conference

2017, Vox Conference Centre, 28 February

9 Newton, P. E. (Ofqual) Macro- and Micro-Validation: Beyond the ‘Five Sources’

Framework for Classifying Validation Evidence and Analysis. In Practical Assessment, Research and Evaluation, Volume 21, Number 12, December 2016

9 Pye Tait Consulting (July 2017). Employer qualification perception survey. Final Report

For further references of relevance to this Chapter, see Appendix B.

211

A good assessment method is valid (in terms of measurement of abilities appropriate

to the target proficiency); reliable (in being able to provide repeatable outcomes

at different times and places for candidates with comparable characteristics); and

practicable (in terms of the time, resource and cost demands of administration). A further

consideration is the extent to which Reasonable Adjustments can be made to support fair

assessment. Very often there is a need for main and supporting assessment methods, or

complementary methods, to secure a proper understanding of a candidate’s level

of attainment.

Portfolios are not included in the tables as a separate method as they involve organising

and presenting a variety of evidence of abilities which have been assessed using different

methods such as observation, questioning, assignments. A key concern is the authenticity

of the evidence presented.

Appendix ASome examples of assessment methods–key features

212Appendix A

Multiple-choice test

Key design requirements • Decision needs to be made on coverage of target proficiency – type/s of abilities amenable to this type of assessment

• Objective questions need to be designed to have: items with an appropriate stem and key; pre-determined, correct answer/s to be selected from a given list of possibilities (distractors)

• When assembling items, the order needs careful thought, for example, to provide no clues which lead candidates to the correct answer

What it assesses well • Cognitive abilities: retention of knowledge; factual recall, Time management

What it assesses less well • Inappropriate for assessing practical skills and behaviours

• Not good for assessing higher-order cognitive abilities such as synthesis

Quality control/ assurance • Use of subject experts trained and skilled in item writing; robust editing/vetting of questions; testing and statistical analysis of how items perform

• Externally-set and marked by the Awarding Organisation either on paper or on computer

• Security of items and tests has to be protected at all stages (production, application, transportation, storage)

• Taken under controlled conditions, strictly supervised

Other considerations • Need for sufficient writers (a minimum of 3), plus at least one other person (preferably a team) to vet and edit questions

• Item bank needs to be large enough to adequately cover subject matter and number of assessment occasions

• Innovative items can be expensive to design (but getting cheaper)

• It is also possible to have externally-set, but internally marked tests (with marking scheme and guidance) or internally-set and marked tests (with external moderation. Can also be subject to external controls on design)

Written examination

Key design requirements • Can include one or more question types. See above for objective questions

• Constructed questions can be designed to require short-answers and/or extended answers/essay-type responses. These need to be carefully written bearing in mind: coverage of the target proficiency; need for grading and extent of compensation, if any

• Clear, unambiguous instructions needed on how to answer the questions - candidates may be offered a choice of questions

What it assesses well • The extended-answer/essay questions are good for assessing higher-order cognitive abilities such as analysis, argument, synthesis of complex issues. Can be used to assess ability to handle/interpret data

• Time management

What it assesses less well • Inappropriate for assessing practical skills and behaviours

Quality control/ assurance • Papers externally-set and rigorously checked for quality, including presentation of paper, e.g. for clarity, appropriate use of language, any bias or pre-requisite cultural knowledge required

• Externally marked by trained markers using a clear marking scheme; standardisation of marking

• Security of papers and tests has to be protected at all stages (production, application, transportation, storage)

• Taken under controlled conditions, strictly supervised

Other considerations • Reliant on having a sufficient number of trained markers

• More difficult to achieve consistency in marking for extended answers and essays (rely more on subjective judgement of marker) than, say, for responses to multiple choice questions

• Challenging for some candidates, for example, those who require Reasonable Adjustments

• Can create stress/anxiety when high stakes which may impact on performance of candidates

213 Appendix A

‘Open-book’ examination

Key design requirements • Is a variation to above in that it allows candidates to take certain reference material into the examination

• Consideration needs to be given to the types of material commonly used in courses leading to the qualification and relationship to target proficiency

What it assesses well • Understanding of facts (but less reliance on recall of facts)

• Good for assessing ability to apply knowledge to particular problems, assimilation of information and logical and evaluative responses

What it assesses less well • Practical skills

Quality control/ assurance • Parameters of the assessment and nature of the ‘openness’ clearly defined and monitored

• Clear controls and adjustments for fair assessment

Other considerations • Variation and scope of the ‘openness’ in terms of the material used increases the challenge of designing the marking scheme and standardisation of marking

• Access to relevant materials needs to be considered in terms of fair assessment

Oral examination/test

Key design requirements • Design should consider the purpose of the assessment and structure/ordering of questions

• Keep in mind overall objective of the qualification and how oral exam/test complements other assessment methods in covering the target proficiency

• Thought needs to be given to how to record answers and criteria for assessment, any differentiation/grading

• Guidance needed to help candidates prepare for the examination/test

What it assesses well • Oral examinations/tests are good at assessing knowledge and understanding


• Large groups

Quality control/ assurance • Trained, skilled question writers; rating scale for marking, with clear guidance on acceptable responses

• Conducted by external examiner under controlled conditions (time, place)

• Possible to make recordings so that consistency of marking can be assessed

• Security arrangements in place for candidate responses

Other considerations • Resource and time intensive

• Need to consider scope for Reasonable Adjustments for those with speech/language difficulties

214Appendix A

Interviews/ oral questioning

Key design requirements • Define purpose and coverage of target proficiency in relation to other assessment methods used

• Decide on degree of structure for questions and use language appropriate to candidates’ ability level

• Consider scope for Reasonable Adjustments for those with speech/language difficulties

• Decide on how responses will be recorded, criteria for assessment

What it assesses well • Good for assessing: decision-making process; understanding of procedures; underpinning knowledge in support of performance-based assessments; filling gaps in evidence relating to particular aspects of the target proficiency

What it assesses less well • Practical skills and competences

Quality control/ assurance • Set questions with follow-up probes bring a measure of consistency

• Code of practice and training for interviewers/questioners to minimise bias, put candidate at ease, ask questions/behave in an appropriate manner

• Possible to make recordings so that consistency of marking can be assessed


Other considerations • Resource and time intensive

• The relative formality or informality of the assessment intervention needs to be factored in and consistently applied

• Can be used as an alternative to written responses (where these are recorded) opening up access for those with a visual disability or motor control problems affecting clarity and/or speed of writing

Professional Discussions

Key design requirements • Clarity of purpose needed. Defined coverage of target proficiency and identification of required outcomes

• Contribution of the professional discussion to an overall grade (if applicable) needs to be thought through at the design stage

• Questions/discussion topics need to be designed to elicit responses of an appropriate depth

• Method of recording the discussion decided – paper-based, video, audio

• Guidance needed for candidates on what to expect: format; timing; areas/points to be covered; any supporting documentation required such as a portfolio

What it assesses well • A holistic approach to assessing knowledge and understanding and in establishing how a candidate is performing

• Analytical and decision-making abilities, problem solving in response to challenge from the assessor

What it assesses less well • Practical abilities

• Less articulate candidates

Quality control/ assurance • Skills training for assessor in professional discussion techniques to keep the discussion focused, avoiding leading questions

• Code of practice for conducting the discussion

• Can be recorded so that consistency in marking can be assessed

• Clear referencing of evidence from the discussion to standards/assessment criteria to enable verification

Other considerations • A good way of testing the validity of a candidate’s evidence, confirming any grading decisions

• Goes beyond a question and answer session

• Needs to be conducted in a supportive environment

• Feedback to candidate is helpful following assessment of discussion

215 Appendix A

Assignments

Key design requirements • Clearly constructed brief, or choice of brief, required and marking scheme.

• Consideration needs to be given to any requirements with regard to conditions under which it is to be taken, such as a real or realistic working environment, permissible resources and support

• As with other methods, the relationship this assessment has with others within the overall approach needs to be worked through at the design stage

What it assesses well • Can be designed to assess cognitive abilities and/or practical competences

• Problem-solving and analytical skills (if scenario/brief, or resources are supplied for interpretation)

• Research skills, planning and task/time management

What it assesses less well • Can be challenging to write and mark if target proficiency is capable of being demonstrated in multiple contexts

Quality control/ assurance • The level of quality control will depend on whether it is externally or internally set and who does the marking. For example, high control is possible if externally-set and marked. For internally-devised assignments, guidance can be provided on their design requirements; they can be pre-checked by the Awarding Organisation before use

• Can be taken under controlled conditions, such as a specified assessment window with timing and place specified

• Reliability in marking is a prime concern when internally marked. Requires standardisation of marking and external moderation of assessment decisions

Other considerations • Can be time-consuming and resource intensive to produce

Observation

Key design requirements • Purpose of observation and abilities to be assessed need to be clarified in relation to target proficiency – and whether observation is to supplement evidence of the same, or gather evidence of a different, performance. In the latter case, type of supporting evidence should be identified

• The method of recording what is observed needs to be decided, for example, use of observation schedule/scale and logistics of its use

• Guidance produced for candidates on what to expect

What it assesses well • Competence, skills and behaviour; application of knowledge

• Opportunity for holistic assessment (substantial assessment task)

What it assesses less well • Underpinning knowledge and understanding (so needs to be assessed using a complementary method such as questioning)

Quality control/ assurance • Training for observer on how to minimise their influence and possible bias; how to use the recording method

• Code of practice for conducting the observation

• Details of what is to be assessed, and assessment decisions across assessors, are standardised

• Borderline performance or unseen aspects of performance followed up via questioning, professional discussion and/or further observation

Other considerations • Practical considerations such as: where opportunities suitable for observation will be available; agreement on time and place; permissions for site access

• Care needs to be taken about overly ‘staged’ performance

216Appendix A

Aural examination

Key design requirements • Precisely target abilities to be assessed, with mark scheme and grading criteria (where applicable). Identify potential Reasonable Adjustments

• Develop practice activities to prepare candidates, with precise use of words to be used by examiner in delivering the exam

• Specify timing, technology and conditions for the examination

What it assesses well • Listening skills

• Comprehension

What it assesses less well • Inappropriate for most aspects of target proficiencies

Quality control/ assurance • Quality control of stimulus (recorded or live) to ensure clarity, appropriate pace

• Conducted by External examiner under controlled conditions, with clear protocols and code of practice to protect integrity of the examination

• Background information repeated on several occasions to avoid undue reliance on memorisation


Other considerations • Limited application in vocational qualifications

Product or artefact production

Key design requirements • The range of suitable products from different contexts and over a set time period all need to be identified.

• Extent to which production of the artefact addresses the target proficiency needs to be clarified

• The amount of autonomy and limits to support in producing the product/ artefact need to be set out in guidance

• Procedures for authenticating evidence need to be considered

What it assesses well • Naturally occurring evidence

• Quality of the product/ artefact

• Application of knowledge and skills

• Competence, hand skills, production skills (depending on artefacts)

What it assesses less well • Process of artefact production, underpinning knowledge and the reasons for decisions taken – supplementary evidence required

Quality control/ assurance • Acceptable and unacceptable evidence, as well as what constitutes sufficient evidence, are defined

• Sufficiency in both what the product/ artefact should demonstrate and quality expected are clearly communicated to candidates, assessors

• Clear criteria agreed for rewarding additional levels of performance

• Checks on authenticity of evidence

• Standardisation of assessment decisions across assessors

Other considerations • Authenticity of the product/artefact as evidence of the candidate’s work is a key issue

• The extent to which the product/ artefact addresses all the assessment criteria or whether supporting assessment methods are required

• Products or artefacts can be engineered products, software programming, media products produced, documentation production, video evidence of performing a service etc

217 Appendix A

Skills or Trade Test

Key design requirements • Purpose of test and relationship of the simulated conditions need to be identified in relation to the target proficiency

• Number of different tests required to cover competence and relationship with other methods needs to be established

• Assessment control conditions need to be set, together with levels of questioning and support, tools and techniques that can be used

• Guidance for candidates on what to expect

• Decisions made on specificity of marking scheme and recording method

What it assesses well • Skills and certain competences

What it assesses less well • Understanding of the wider knowledge-base beyond the focus of the test

Quality control/ assurance • Training of assessors, including how to use marking scheme/recording method

• Set control conditions for conducting the test and supervision

• Use of authentic materials, tools and equipment to reflect up-to-date practice in the workplace

• Standardisation of assessment decisions across assessors, especially if different assessment sites are used

• External verification

Other considerations • Time and resource intensive

• How best to support the skills/trade test with complementary forms of assessment like questioning, witness testimony or work logs

Simulation

Key design requirements • Design to replicate as closely as possible real-life conditions relating to the target proficiency The closer to the actual work-place task the better able the simulation is to address the target proficiency

• Produce clear assessment instructions and marking schemes

• Guidance needed for candidates on what to expect

What it assesses well • Large elements of a practical target proficiency

• Application of skills, knowledge and understanding, behaviours

What it assesses less well • Knowledge

Quality control/ assurance • Training for assessors, including that on how to use marking scheme and recording method and to ensure they have a clear sense of the impact of any variations in simulated environment or methods across different contexts

• Use of authentic materials, tools and equipment to reflect up-to-date practice in the workplace.

• Standardisation of assessment decisions across assessors, especially if different assessment sites are used

• External verification

Other considerations • Useful in assessing critical, but infrequent activities

• Cost needs to be considered in setting up the simulation and weighed against the importance of the evidence being collected in this way

• Realistic work environment might be a training salon (hairdressing) or training restaurant. There may be site access requirements for assessors

• Ability to scale and replicate the simulated environment or conditions needs to be considered, as well as any access and conflict of interest issues

218Appendix A

Coursework

Key design requirements • This would tend to be set and marked locally, but exemplars and sample assessment tasks provided to centres may help to provide an indication of the level and nature of the assessment tasks that should be set. Clarity of which aspects can be evidenced this way is needed

• Where coursework is in response to nationally set assessment tasks, tasks need to be clearly defined and controls set for resources and timing. Marking scheme or checklist of assessment objectives need to be produced (as applicable)

What it assesses well • Knowledge and understanding; skills (depending on the nature of the coursework); time management; self-expression; problem solving and self-evaluation skills

What it assesses less well • Summative assessment leading to final certification

Quality control/ assurance • Training and quality focus needs to be targeted at delivery end to make sure coursework is useable

• Local assessment will need moderation processes to quality assure outputs. For objective-related assessments, candidate work can be sampled to provide evidence of attainment. For unstructured coursework assessments, an analytical mark scheme can be used

• Centrally marked work needs to ensure validation and authentication processes are in place to confirm the work is the candidate’s own and carried out in accordance with requirements

Other considerations • Can be resource and time intensive both locally and centrally

• May be difficult to be certain the work is the candidate’s own

• Where coursework is unstructured (all exercises are taken during classwork, as homework), no account is taken of variations in the level of difficulty of the exercises between groups within a centre and between centres

• Where coursework is objective related - systematic and drawn up before the start of course - there is more emphasis on the candidate’s attainment at the end (less during the course). Good coverage, and balance, of the syllabus; can be used year on year

Project

Key design requirements • Project briefs need to be carefully created and the relationship with the assessment objectives and other assessment methods made clear

• Depending on the amount of local or centrally devised setting and assessment, different guidance material will be needed to ensure consistency in approach

What it assesses well • Application of knowledge; work skills; practical skills

• Able to assess large parts of the programme in a holistic way

• Time management and self-directed learning

What it assesses less well • Underpinning knowledge. Process. (unless development work is present)

Quality control/ assurance • Local assessment needs quality assurance processes in place to ensure reliable marking and comparability of standards across assessors

• External markers should have training to ensure consistency across a range of permitted contexts

Other considerations • Time consuming to prepare, carry out and assess, but encourages innovation and self-expression

• Numerous possible topics and approaches

• Group projects are good for assessing teamwork, but more difficult to assess each candidate’s contribution (which impacts on reliability)

219 Appendix A

Dissertation

Key design requirements • A brief or proposal needs to be designed which is candidate generated, with approval process controlled by the assessor

• The nature of the formative assessment and any on-programme assessment needs to be made clear. This is to avoid candidates departing too far from its focus or required standard over what can be a protracted period for generating assessment evidence

What it assesses well • High-order analytical skills

• Original thought and self-expression

• Research skills

• Complex ideas and extended argument

What it assesses less well • Practical skills (other than those involved in research)

Quality control/ assurance • Quality control is needed in shaping the process so the candidate keeps on track with regard to the target proficiency (without undue influence)

• Matching of candidate subject with assessor interest is important

• Corroboration of marking is needed

Other considerations • Time and resource heavy because of the marking involved and the support needed before submission

• Normally needs to be supported by a Viva or professional discussion

Case studies

Key design requirements • The role of the case study in relation to the overall assessment requirement needs to be defined

• Often used to allow a candidate to demonstrate application of knowledge and skills by showing understanding of events in a specific context. This means that given the potential variety of context, the purpose of the case study and its relation to the key qualification aim needs to be clear. Clarity of purpose will then help underpin consistency in the marking process. This method is likely to be accompanied by other methods and the relationship and interaction between each method needs to be clear and purposeful

What it assesses well • Problem solving and analytical skills

• Application and transfer of knowledge and understanding

• Initiative, responses to complex ideas

• Synoptic assessment of target proficiency


Quality control/ assurance • The resources and related tasks need to be aligned with the higher order qualification aims and assessment objectives

• Clear marking criteria are needed

• Marking and standardisation is important given the risk of variation in responses to the assessment task

Other considerations • Has development resource costs

• Useful in establishing real life scenarios and problems without the real time issues

220Appendix A

Reflective logs, journals or workbooks

Key design requirements • Production of best practice guidance. Areas of the programme able to be evidenced this way needs to be communicated, with guidance on levels of permissible support. Relationship with other assessment methods needs to be clear and explained

What it assesses well • Process and decision making, understanding; problem solving, self-evaluation skills

• Useful in supporting other assessment methods


Quality control/ assurance • Assessor training needed to help support the range of potential formats and contexts

Other considerations • Time consuming when assessing this form of evidence

• Introduces an element of flexibility and the ability to innovate in how candidates use these methods

Personal Statements, e.g. diaries, blogs, vlogs etc.

Key design requirements • Needs to have a measure of structure and direction to ensure evidence is useful and useable

What it assesses well • Evidencing development of knowledge and skills over time

• Offers some insight into process

What it assesses less well • Skills, competence (unless directly related to the format or communication skills being used, such as IT and/or communication skills in a self-posted blog)

Quality control/ assurance • Checks on authenticity and levels of independence in producing evidence may be needed in some forms of this evidence if the focus is on outcome quality and not process. For example, if the personal statement is assessed as a self-appraisal or product, whether it is assessed as evidence of process and development

Other considerations • Best used as supporting evidence and self-selected as a form of evidence

221

Appendix B

222Appendix B

B1 Key organisations of relevance to the awarding sector

B1.1 Government departments and other relevant organisations

UK or England only

Department for Education (DfE)

Has overall responsibility for wholly, or partly, publicly-funded education, apprenticeships and skills

• www.gov.uk/government/organisations/department-for-education

Education and Skills Funding Agency (ESFA)

An executive agency sponsored by the DfE, accountable for funding education and training for children, young people and adults. It regulates academies, FE Colleges, employers and training providers receiving public funding and manages the Register of Approved Apprentice Assessment Organisations (RoAAOs) for England

• www.gov.uk/government/organisations/education-and-skills-funding-agency

• www.gov.uk/government/collections/register-of-apprentice-assessment-organisations

Institute for Apprenticeships (IfA)

Has its policy parameters set by the DfE and approves Apprenticeship Assessment Organisations for England. From 2018, it manages Level 2 and Level 3 technical certificates and maintains a register of Level 4 and 5 technical qualifications which are eligible for public subsidy through the ESFA

Overseeing external quality assurance of apprenticeships (Blog) How to Guide for Trailblazers

• https://www.gov.uk/government/organisations/institute-for-apprenticeships

• https://apprenticeships.blog.gov.uk/2017/08/25/overseeing-external-quality-assurance-of-apprenticeships/

• https://www.gov.uk/government/publications/how-to-develop-an-apprenticeship-standard-guide-for-trailblazers

Office for National Statistics (ONS)

Collects, analyses and disseminates statistics about the UK’s economy, society and population

• www.ons.gov.uk/

Office for Standards in Education, Children’s Services and Skills (Ofsted)

Inspects and regulates services that care for children and young people, and services providing education and skills for learners of all ages

• www.gov.uk/government/organisations/ofsted

Quality Assurance Agency for Higher Education (QAA)

Works across the UK to raise standards and quality of higher education

• www.qaa.ac.uk/en

Sector Skills Councils and Sector Skills Bodies

Work with employers across the UK to define skills needs and skills standards in their industry. Information available through The Federation for Industry Sector Skills and Standards (FISSS) which represents, promotes and supports SSCs

• http://fisss.org/sector-skills-council-body/

Wales

Department for Education and Skills

The Welsh Government, through the Department for Education, has overall responsibility for qualifications policy and for determining qualifications priorities in Wales

Announcement of a new strategic authority to provide regulatory oversight of post-compulsory education

• http://gov.wales/?lang=en

• http://gov.wales/newsroom/educationandskills/2017/new-authority-to-oversee-skills-and-the-higher-and-further-education-sectors-in-wales/?lang=en

223 Appendix B

Estyn

Independent of, but funded by, the National Assembly for Wales to inspect quality and standards in education and training in Wales (and learners in England funded by Wales)

• www.estyn.gov.wales/language

Higher Education Funding Council for Wales (HEFCW)

Regulates fee levels at universities, ensures a framework is in place for assessing the quality of higher education and scrutinises the performance of universities and other designated providers

• https://www.hefcw.ac.uk/home/home.aspx

Northern Ireland

Department of Education

Has responsibility for the curriculum and assessment in schools and 14-19 policy in Northern Ireland

• www.education-ni.gov.uk/

Department for the Economy

Is responsible for skills and employment, including apprenticeships and further education in Northern Ireland. Also funds universities in Northern Ireland

• www.economy-ni.gov.uk/

Council for the Curriculum, Examinations and Assessment (CCEA)

Acts on behalf of the above departments on a day-to-day basis. Brings together the three areas of curriculum, examinations and assessment. Has a Regulation arm (see below) and also operates separately as an Awarding Organisation

• ccea.org.uk/

Education and Training Inspectorate (ETI)

Part of the Department of Education providing independent inspection services and policy advice to the DE and other departments such as the Department for the Economy

• www.etini.gov.uk

Scotland

The Cabinet Secretary for Education and Skills

Has responsibility for Skills Development Scotland and is reported to by the Scottish Qualifications Authority (SQA) which has a regulatory arm (see below) and operates, separately, as an Awarding Organisation

• https://beta.gov.scot/about/who-runs-government/cabinet-and-ministers/cabinet-secretary-education-skills/

• www.sqa.org.uk/sqa/70972.html

Skills Development Scotland (SDA)

Supports people and businesses in Scotland to develop and apply their skills

• www.skillsdevelopmentscotland.co.uk/

Scottish Funding Council (SFC)

Funds colleges and universities in Scotland. Also supports university research and creation of innovation centres

• www.sfc.ac.uk/

The Scottish Credit and Qualifications Framework Partnership

Manages the Scottish Credit and Qualifications Framework (SCQF) – see below. Partnership members include the Quality Assurance Agency for Higher Education, College Development Network, the Scottish Qualifications Authority and Universities Scotland.

• scqf.org.uk/about-us/background-to-scqf/

Education Scotland

A Scottish Government executive agency charged with supporting quality and improvement in Scottish education, including inspecting schools

• education.gov.scot/what-we-do/inspection-and-review/About-inspections-and-reviews

224Appendix B

B1.2 Qualification regulators and related information

UK or England only

The Office of Qualifications and Examinations Regulation (Ofqual)

Recognises Awarding Organisations that are required, or wish, to have their non-degree qualifications regulated in England. Operates the Portal, shared with Northern Ireland, that hosts the Register of Regulated Qualifications

• www.gov.uk/government/organisations/ofqual

• register.ofqual.gov.uk/

Regulated Qualifications Framework (RQF)

Maps to, but does not include qualifications in, the Framework for Higher Education Qualifications (FHEQ). Has Entry levels (1-3) and levels 1-8

• https://www.gov.uk/government/publications/regulated-qualifications-framework-a-postcard

Ofqual (September 2017) Guidance - Awarding Organisations: understanding our regulatory requirements.

Provides access to key documents: General Conditions of Recognition; Guidance to the General Conditions of Recognition; Additional Certificate requirements; Logo requirements; qualification and component levels; Total Qualification Time (TQT)

• https://www.gov.uk/guidance/awarding-organisations-understanding-our-regulatory-requirements

The Office for Students (OfS)

Will be formally established from April 2018, to be the single regulator for the higher education sector [Result of merging Higher Education Funding Council for England (HEFCE) and the Office for Fair Access (OFFA)]

• https://www.gov.uk/government/publications/office-for-students-business-case

• www.hefce.ac.uk/

• www.offa.org.uk/

Wales

Qualification Wales (QW)

Regulates non-degree qualifications and the qualifications system in Wales. It is sponsored by the Welsh Government and accountable to the National Assembly for Wales, but operates independently of government

• http://qualificationswales.org/?lang=en

Credit and Qualifications Framework for Wales (CQFW)

Managed by QW as part of a strategic operational partnership with the Welsh Government and Higher Education Funding Council for Wales (HEFCW). Has 9 levels (Entry; 1-8) and includes qualifications offered in higher education. Levels 4-8 are consistent with the five levels in the Framework of Higher Education Qualifications (FHEQ)

• http://gov.wales/topics/educationandskills/ qualificationsinwales/creditqualificationsframework/?lang=en

Qualifications in Wales (QiW)

A new and comprehensive source of information on approved and designated qualifications awarded in Wales

• https://www.qiw.wales

QW Regulatory Document List (May 2017)

Descriptions, including Standard Conditions of Recognition (March 2017)

• http://gov.wales/topics/educationandskills/qualificationsinwales/creditqualificationsframework/?lang=en

225 Appendix B

Northern Ireland

Council for the Curriculum, Examinations and Assessment (CCEA) Regulation

Monitors Awarding Organisations and regulates qualifications in Northern Ireland, including Professional and Technical (vocational) qualifications.

Shares RQF and the Register with Ofqual

• ccea.org.uk/regulation/

• register.ofqual.gov.uk/

General Conditions of Recognition (May 2017)

Regulatory policies and procedures

• http://ccea.org.uk/news/2016/general_conditions_recognition_february_2016

Scotland

Scottish Qualifications Authority (SQA) Accreditation

Focuses on accreditation of qualifications (other than degrees), regulation and research and statistics

• accreditation.sqa.org.uk/accreditation/home

Scottish Credit and Qualifications Framework (SCQF)

Has 12 levels (level 4 corresponds to level 2 in the RQF and CQFW)

SCQF level descriptors

• scqf.org.uk/

• http://scqf.org.uk/wp-content/uploads/2014/03/SCQF-Level-Descriptors-WEB-Aug-2015.pdf

Regulated Principles and Regulatory Directives

Define the standards Awarding Bodies should meet in order to be compliant

• accreditation.sqa.org.uk/accreditation/Regulation/Regulatory_Requirements

Accreditation of Qualifications • http://accreditation.sqa.org.uk/accreditation/Qualifications/Accreditation_of_Qualifications

B2 Key organisations that support international work

Organisation Links

British Chambers of Commerce (BCC)

Network of Accredited Chambers of Commerce across the UK that helps businesses by providing representation, services, information and guidance to its members

• www.britishchambers.org.uk

British Council

Creates international opportunities for the people of the UK and other countries. They are on the ground in six continents and over 100 countries. It is also possible to sign up to the British Council’s Services for International Education Marketing (SIEM) system which provides access to a range of information including opportunities, contact details of staff in-country and education events around the world

• www.britishcouncil.org/

British Exporters Association (BExA)

International trade association which represents all sectors of the export community

• www.bexa.co.uk

226Appendix B

Department for International Trade (DTI)

International economic department, responsible for: bringing together policy, promotion and financial expertise to break down barriers to trade and investment, and help businesses succeed; delivering a new trade policy framework for the UK as we leave the EU; promoting British trade and investment across the world; building the global appetite for British goods and services

• www.gov.uk/government/organisations/department-for-international-trade

Exports Credit Guarantee Department

Provides useful information about financial matters and insurance

• www.ecgd.gov.uk

Foreign and Commonwealth Office (FCO)

A ministerial department, supported by 12 agencies and public bodies. The FCO promotes British interests overseas, supporting citizens and businesses around the globe. The website provides a range of useful information including travel advice by country, passport information and an overview of what the government is doing in overseas countries

• www.gov.uk/government/organisations/foreign-commonwealth-office

Institute of Export (IOE)

Provides a range of information on exporting and importing and also provides a range of training courses

• www.export.org.uk

Institute of Translation & Interpreting (ITI)

The UK’s only dedicated association for practicing translation and interpreting professionals. The ITI has over 3000 interpreter and translator members who specialise in more than 100 languages and dialects from around the world. ITI also maintains a directory of qualified professional translators and interpreters, who have been assessed according to ITI’s admission criteria

• www.iti.org.uk/

International Chambers of Commerce (ICC)

Aims to further members’ interests by making it easier to trade internationally. They do this by working with governments and businesses worldwide to open up markets, establish a level playing field for enterprise, and promote sensible regulation

• www.international-chamber.co.uk/

Technical and Vocational Education and Training (TVET UK)

Dedicated to exporting the capabilities of the technical and vocational education sector. Their members include UK education providers and suppliers

• www.tvetuk.org/

UK Trade & Investment (UKTI) Education

Focuses on high value commercial education and training exports that may require a consortium approach. The team aims to identify and communicate opportunities to UK providers, bring together UK consortia and helps providers to prepare and promote bids. It also aims to provide a comprehensive, coordinated service for overseas governments and large companies wanting to engage UK partners in their large-scale education projects. The team offers general support to UK companies and organisations working in the education and skills sector to win export business. This is provided through support for trade fairs, seminars and inward or outward trade missions. Produces a newsletter for exporters

• https://www.gov.uk/government/organisations/ukti-education

227 Appendix B

B3 References and sources of further information

B3.1 General references, including government reports

• Bees, M. & Swords, M. (Eds) National Vocational Qualifications and Further Education. London: Kogan Page/NCVQ

• Brown, D., Harris, M. and Fletcher, T. (2011) Reforming the Skills System: lessons learned the hard way, London/Leicester: Institute of Directors/CF

• Commission on Adult Vocational Teaching and Learning (2013) It’s all about work…Excellent adult vocational teaching and learning. London: LSI

• Coles, M. (2016) National Qualifications Frameworks. Reflections and Trajectories. Qualifications Policy Insights. Quality and Qualifications, Ireland

• Department for Business, Innovation & Skills and Department for Education (3 March 2011) Review of vocational education – the Wolf report

• Department of Education (February 2015) Review of vocational education, 2011.The Wolf Report: recommendations final progress report

• Department for Education (October 2016) Apprenticeship Funding in England from May 2017: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/562401/Apprenticeship_funding_from_May_2017.pdf

• Department for Business, Innovation & Skills/ Department for Education (July 2016) Post 16 Skills Plan

• Department for Business, Innovation & Skills/ Department for Education (April 2016) The Report of the Independent Panel on Technical Education (known as the Sainsbury Review)

• Department for Business, Innovation & Skills (27 November 2012) The Richard Review of Apprenticeships

• European Centre for the Development of Vocational Training (2010) Changing Qualifications: A Review of Qualifications Policies and Practices, Luxembourg: Publications Office of the European Union

• Institute for Apprenticeships (February 2011) Driving the quality of apprenticeships in England

• Keep, E.J. and James, S. (2010) Recruitment and selection – A review of extant research and some thoughts on its implications for education and training policy. In: SKOPE Research Paper, No.88 Cardiff: Cardiff University, SKOPE

• National Skills Task Force (2000) Research Report. DfEE

• Ofqual (March 2015) Regulatory Strategy Statement: https://www.gov.uk/government/publications/ofquals-regulatory-strategy-statement

• Ofqual (2015) Criteria for Determining Whether a Qualification is Relevant for the Purposes of the Education and Skills Act 2008 For All Awarding Organisations https://www.gov.uk/government/publications/criteria-for-determining-whether-a-qualification-is-relevant-for-the-purposes-of-the-education-and-skills-act-2008

• Quality Assurance Agency (May 2017) Quality assuring higher education in apprenticeships, current approaches

• Unwin, L., Fuller, A., Turbin, J., & Young, M. (2004). What determines the impact of vocational qualifications? A literature review. London: Department for Education & Skills

Sources of information on:

• European Qualifications Framework: http://www.accreditedqualifications.org.uk/european-qualifications-framework-eqf.html

• Framework for Higher Education Qualifications: http://www.qaa.ac.uk/en/Publications/Documents/Framework-Higher-Education-Qualifications-08.pdf

228Appendix B

• National Occupational Standards: http://www.ukstandards.org.uk/Pages/index.aspx

• Performance Tables (England): https://www.gov.uk/school-performance-tables

• Standard Occupational Classification: https://www.ons.gov.uk/methodology/classificationsandstandards/standardoccupationalclassificationsoc

B3.2 Project management resources

Project management resources include those offered by:

• The Association for Project Management (APM), including The APM Body of Knowledge – Definitions. Available at: https://www.apm.org.uk/sites/default/files/bok%205%20definitions.pdf

• Projectmanagement.com: https://www.projectmanagement.com/default.cfm

• The Project Management Institute (PMI): https://www.pmi.org/

• APMG International, for example: PRINCE2 (PRojects IN Controlled Environments) - a structured method and standard for effective project management. Available at: https://apmg-international.com/product/prince2

Guidance on risk management:

• Institute of Risk Management (IRM), overview of definitions and risk management processes, and guidance on getting started: https://www.theirm.org/events/special-interest-groups/charities/

• Available free to download: https://www.theirm.org/knowledge-and-resources/thought-leadership/charities-and-voluntary-organisations.aspx Risk Management for charities: getting started supplementary guidance https://www.theirm.org/media/1238690/CharitiesGuidanceV6FINAL.pdf

IT software project management systems, such as:

• Creatiogreen system, an Awarding Organisation focused ‘workflow solution which underpins governance, qualification development, centre management and learner registration/certification processes’: http://www.creatio.org.uk/awardingorgs

• Quartz, an IT system that includes support for ’registrations, award, certification, qualification development, centre approval and management. Compliance and quality assurance, and automated work-flow management’: http://www.quartz-system.com

• Parnassus, a cloud-based management system for Awarding Organisations covering all business functions from learner registrations to certification: http://www.parnassusonline.com/

B3.3 Assessment

• Ahmed, A. and Pollitt, A. (June 2010) Alternatives to test accommodation: New models for pupils with speech and language difficulties, the Support model. Ofqual

• AlphaPlus Consultancy (2006) Assessment – Aspects of Principles and Practice

• Cambridge Assessment Research Matters (free biannual publication to share assessment research). Available at: http://www.cambridgeassessment.org.uk/news/our-publications/research-matters/

• Cambridge Assessment (January 2009, revised April 2017) The Cambridge Approach. Principles for designing, administering and evaluating assessment

• Downing, S.M. and Haladyna, T.M. (2006) Handbook of Test Development. London: Lawrence-Erlbaum Associates. Available at: https://www.scribd.com/document/348965232/Downing-S-M-Haladyna-T-M-2006-Handbook-of-test-development-Lawrence-Erlbaum-Associates-pdf (Accessed 21 September 2017)

• Gipps, C. (2004), Beyond Testing: Towards a theory of educational assessment, Routledge Falmer 2nd Edition

229 Appendix B

• Lane, S, Raymond, M.R. & Haladyna T.M. (2016) Handbook of Test Development. Routledge (Second edition)

• Marieke van der Schaaf, Jeroen Donkers, Bert Slof, Joyce Moonen-van Loon, Jan van Tartwijk, Eric Driessen, Atta Badii, Ovidiu Serban & Olle Ten Cate (November 2016) Improving workplace-based assessment and feedback by an E-portfolio enhanced with learning analytics. Published online: https://link.springer.com/content/pdf/10.1007%2Fs11423-016-9496-8.pdf

• Scottish Qualification Authority (March 2015) Guide to Assessment for Centres offering Ofqual Regulated Qualifications. Glasgow: SQA

• Reilly, M. Further Education Learning Technology: A horizon scan for the UK Government. Foresight Horizon Scanning Centre, Ariel Research Services, August 2013

• Stobart, G. and Gipps, C. (1997) Assessment. A teachers guide to the issues. 3rd Edition. London: Hodder & Stoughton

• Ward, C. (1980) Designing a scheme of assessment. Cheltenham: Stanley Thornes (Publishers) Ltd

• Watson, A. Strategies for the Assessment of Competence. In the Vocational Aspect of Education, 46:2, 155-165, 1994

• Wiliam, D. (1992) Some technical issues in assessment: a user’s guide. In British Journal of Curriculum and Assessment, 2, 3, 11-20

• Wiliam, D. Construct-referenced assessment of authentic tasks: alternatives to norms and criteria. Paper presented at the 7th Conference of the European Association for Research in Learning and Instruction, Athens, Greece, August 26-30, 1997

• Winkley, J. (2010) E-assessment and innovation, BECTA. Available at: http://archive.teachfind.com/becta/emergingtechnologies.becta.org.uk/upload-dir/downloads/page_documents/research/emerging_technologies/e-assessment_and_innovation.pdf

B3.4 Validity

• Validity and validation research (commissioned by Ofqual and QCA): https://www.gov.uk/government/publications/validity-and-validation-research Including: AlphaPlus Consultancy (2014) Validation of vocational qualifications. Final Report Newton (2007) Evaluating Assessment systems. Ofqual

• Curcin, M., Boyle, A., May, T. and Rahman, Z. (February 2014) A validation framework for work-based observational assessment in vocational qualifications. London: The City and Guilds of London Institute (commissioned by Ofqual)

• Busutti-Reynaud, G. and Whitehouse, G. Basic Principles of Validity, presentation 14 November 2016, AlphaPlus Consultancy Ltd

• Kane, M.T. (2001) Current concerns in validity theory. Journal of Educational Measurement, 38, 319–342

• Newton P.E. (2017) An approach to understanding validation arguments. Ofqual

• Newton, P. E. Validity–what it is and why it matters. PowerPoint presentation at Ofqual Conference, 2017, The Vox Conference Centre, Marston Green, 28 February

• Newton, P.E. (2016). Macro- and Micro-Validation: Beyond the ‘five sources’ framework for classifying validation evidence and analysis. Practical Assessment, Research & Evaluation, 21 (12). Available at: http://pareonline.net/getvn.asp?v=21&n=12

• Stasz, C. The Purposes and Validity of Vocational Qualifications. SKOPE Research Paper No. 105 November 2011

230Appendix B

B3.5 Reliability

• Ofqual (16 May 2013) Reliability of assessment: compendium: https://www.gov.uk/government/publications/reliability-of-assessment-compendium. Including: The Reliability Programme: Final Report Introduction to the concept of reliability The reliability of results in vocational assessment: the case of work-based certifications: Full reportConceptualising and interpreting reliability: Full reportNo news is good news? Talking to the public about the reliability of assessment: Full report

• Harth, H. and Hemker, B.T. City & Guilds (January 2011) On the reliability of results in vocational assessment (Part of Ofqual’s Reliability Programme). Available at: http://www.voced.edu.au/content/ngv:64401

B3.6 Comparability of qualifications

• Coles, M. and Matthews, A. (1995) Fitness for purpose. A means of comparing qualifications. A report to Sir Ron Dearing

• Elliott, G. (2011). A guide to comparability terminology and methods. Research Matters: A Cambridge Assessment Publication, Special Issue 2,9– 19

• Greatorex, J. Can vocational A levels be meaningfully compared with other qualifications? A paper presented at the British Educational Research Association Conference, University of Leeds, UK, 13-15 September 2001

• Newton, P.E., Baird, J., Goldstein, H., Patrick, H. & Tymms, P. (2007). (Eds.). Techniques for monitoring the comparability of examination standards. London: Qualifications and Curriculum Authority. Available at: https://www.gov.uk/government/publications/techniques-for-monitoring-the-comparability-of-examination-standards

B3.7 Qualification design and development

• Bloom, B. S. (1956) Taxonomy of Educational Objectives. London: Longman Group

• Department for Education (August 2016) Guidance for Awarding Organisations submitting technical and applied qualifications for school and college performance tables: https://www.gov.uk/government/publications/14-to-19-technical-and-applied-qualifications-technical-guidance

• Davis-Becker, S, and Buckendahl, C. W. (Eds) (2017) Testing in the Professions Credentialing Policies and Practice. Routledge

• Eraut M. (1994) Developing Professional Knowledge and Competence. Falmer Press

• Federation for Industry Sector Skills and Standards (November 2014) Improving Apprenticeship standards through consultation. A guide and toolkit for employers. And Employer-defined Apprenticeship standards. A toolkit for enablers and facilitators of Trailblazer groups

• Newton, P. E. Specifying the proficiency that a qualification needs to measure. JVET Conference, 7-9th July 2017, Worcester College, Oxford

• Newton, P.E. (2017). There is more to educational measurement than measuring: The importance of embracing purpose pluralism. Educational Measurement: Issues and Practice, 36 (2), 5-15

• Ofqual (2015) Qualification and component levels. Available at: https://www.gov.uk/government/publications/qualification-and-component-levels

• SQA Accreditation (March 2016), Developing Scottish Qualification Products for Approval: A Guide for Standard Setting Organisations. Available at: https://www.sqa.org.uk/sqa/64144.html

231 Appendix B

• SQA Accreditation (February 2017) Developing Qualifications for Accreditation. A Guide for Awarding Bodies. Available at: http://accreditation.sqa.org.uk/accreditation/accreditationfiles/Quals/Developing_Qualifications_for_Accreditation__Final_.pdf

• UCAS (June 2017) Progression Pathways 2017: Pathways through higher education: Available at: https://www.ucas.com/progression-pathways-2017

• UCAS Tariff Points–proposal process and notification: https://www.ucas.com/corporate/about-us/our-work-policy-community/new-ucas-tariff-proposal-process-notify-us-new-qualifications

• UK Commission for Employment and Skills (June 2011) NOS Strategy 2010-2020 – Revised Strategy June 2011. Available at: https://www.gov.uk/government/publications/national-occupational-standards

• Vale, D.C. Computerized item banking. In Downing, S.M. and Haladyna, T.M. (2006) Handbook of Test Development. London: Lawrence-Erlbaum Associates

B3.8 Qualification delivery

• Department for Education and Department of Health (January 2015) Special Educational Needs and Disability code of practice: 0-25 years. Statutory guidance for organisations that work with young people (aged up to 25) who have special educational needs or disabilities. Available at: https://www.gov.uk/government/publications/send-code-of-practice-0-to-25

• Gravells, A. Principles and Practices of Assessment (January 2016) and Principles and Practices of Quality Assurance (September 2016). Available at: http://www.anngravells.com/

• Underwood, J. (December 2006) Digital Technologies and dishonesty in examinations and tests. QCA. Available at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.506.7321&rep=rep1&type=pdf

Laws and regulations - updates: http://www.legislation.gov.uk/

• Competition Act: http://www.legislation.gov.uk/ukpga/1998/41/contents

• Data Protection Act: https://www.gov.uk/data-protection/the-data-protection-act

• Department for Digital, Culture Media & Sport (7 August 2017) A New Data Protection Bill: Our Planned Reforms: https://www.gov.uk/government/collections/data-protection-bill-2017

• Equalities Act: http://www.legislation.gov.uk/ukpga/2010/15/contents

B3.9 Standard setting/awarding

• Cedefop (2015) Ensuring the quality of certification in vocational education and training. Luxembourg: Publications Office. Cedefop research paper, No 51

• Cizek, G. J (2004) Standard Setting. In Downing, S.M. and Haladyna, T.M. (2006) Handbook of Test Development. London: Lawrence-Erlbaum Associates

• Greatorex, J., Novakovac, N. and Suto I. What attracts judges’ attention – a comparison of grading methods. Annual Conference of International Assessors for Education Assessors, September 2008

• Klenowski, V. and Wyatt-Smith, C., 2014, Assessment for Education: Standards, Judgement and Moderation, Sage Publications Ltd

• Massey, A.J. & Raikes, N. Item level examiner agreement. Paper presented at the 2006 Annual Conference of the British Educational Research Association, 6-9 September 2006, University of Warwick, UK

• Robinson, C. Awarding examination grades: current processes and their evolution, in Newton et al (Eds) (2007) Techniques for monitoring the comparability of examination standards

232Appendix B

• Zieky, M. & Perie, M. (2006) A Primer on Setting Cut Scores on Tests of Educational Achievement, including Excerpts from Livingston S & Zieky M. Passing Scores: A Manual for Setting Standards of Performance on Educational and Occupational Tests. Educational Testing Service. Available at: https://www.ets.org/Media/Research/pdf/Cut_Scores_Primer.pdf (Accessed 4 August 2017)

B3.10 Review and evaluation

• Cohen, L., Manion, L. and Morison, K. (2011) Research Methods in Education, 7th Edition, Routledge. Companion website: http://cw.routledge.com/textbooks/cohen7e/

• Ofqual Vocational qualifications dataset (updated 14 September 2017): https://www.gov.uk/government/collections/statistics-vocational-qualifications

• Ofqual (14 Septmber 2017) Vocational and other qualifications quarterly, April-June2017: https://www.gov.uk/government/statistics/vocational-and-other-qualifications-quarterly-oct-todec-2016

• Ofqual (2017) Statistical Release. Annual Qualifications Market Report - England, Wales and Northern Ireland. Academic Year 2015/16. Available at: https://www.gov.uk/government/statistics/annual-qualifications-market-report-academic-year-2015-to-2016

• Pye Tait Consulting (July 2017). Employer qualification perception survey. Final Report. Available at: http://www.pyetait.com/latest-headlines/employer-perceptions-of-qualifications/

233

© 2017 Federation of Awarding Bodies

IBSN: 978-1-5272-1520-7

All rights reserved. For permission requests,

write to the publisher, addressed

“Attention: FAB Handbook,” at the address below.

Federation of Awarding Bodies

11th Floor, Beaufort House

15 St Botolph Street

London, EC3A 7BB

Date post:	10-May-2020
Category:	Documents
Upload:	others
View:	5 times
Download:	0 times