Collaborative provision audit - Open University - MAY 2005 · Collaborative provision audits are...

Open University

MAY 2005

Preface The Quality Assurance Agency for Higher Education (QAA) exists to safeguard the public interest insound standards of higher education (HE) qualifications and to encourage continuous improvementin the management of the quality of HE.

To do this QAA carries out reviews of individual HE institutions (universities and colleges of HE). InEngland and Northern Ireland this process is known as institutional audit. QAA operates similar butseparate processes in Scotland and Wales. For institutions that have large and complex provisionoffered through partnerships, QAA conducts collaborative provision audits in addition toinstitutional audits.

The purpose of collaborative provision audit

Collaborative provision audit shares the aims of institutional audit: to meet the public interest inknowing that universities and colleges are:

providing HE, awards and qualifications of an acceptable quality and an appropriate academicstandard, and

exercising their legal powers to award degrees in a proper manner.

Judgements

Collaborative provision audit results in judgements about the institutions being reviewed.Judgements are made about:

the confidence that can reasonably be placed in the soundness of the institution's present andlikely future management of the quality of the academic standards of its awards made throughcollaborative arrangements

the confidence that can reasonably be placed in the present and likely future capacity of theawarding institution to satisfy itself that the learning opportunities offered to students throughits collaborative arrangements are managed effectively and meet its requirements; and

the reliance that can reasonably be placed on the accuracy, integrity, completeness andfrankness of the information that the institution publishes, (or authorises to be published)about the quality of its programmes offered through collaborative provision that lead to itsawards and the standards of those awards.

These judgements are expressed as either broad confidence, limited confidence or no confidenceand are accompanied by examples of good practice and recommendations for improvement.

Nationally agreed standards

Collaborative provision audit uses a set of nationally agreed reference points, known as the'Academic Infrastructure', to consider an institution's standards and quality. These are published byQAA and consist of:

The framework for higher education qualifications in England, Wales and Northern Ireland (FHEQ),which includes descriptions of different HE qualifications

The Code of practice for the assurance of academic quality and standards in higher education

subject benchmark statements, which describe the characteristics of degrees in different subjects

guidelines for preparing programme specifications, which are descriptions of the what is onoffer to students in individual programmes of study. They outline the intended knowledge,skills, understanding and attributes of a student completing that programme. They also givedetails of teaching and assessment methods and link the programme to the FHEQ.

The audit process

Collaborative provision audits are carried out by teams of academics who review the way in whichinstitutions oversee their academic quality and standards. Because they are evaluating their equals,the process is called 'peer review'.

The main elements of collaborative provision audit are:

a preliminary visit by QAA to the institution nine months before the audit visit

a self-evaluation document submitted by the institution four months before the audit visit

a written submission by the student representative body, if they have chosen to do so, fourmonths before the audit visit

a detailed briefing visit to the institution by the audit team six weeks before the audit visit

visits to up to six partner institutions by members of the audit team

the audit visit, which lasts five days

the publication of a report on the audit team's judgements and findings 22 weeks after theaudit visit.

The evidence for the audit

In order to obtain the evidence for its judgement, the audit team carries out a number of activities,including:

reviewing the institution's own internal procedures and documents, such as regulations, policystatements, codes of practice, recruitment publications and minutes of relevant meetings, aswell as the self-evaluation document itself

reviewing the written submission from students

asking questions of relevant staff from the institution and from partners

talking to students from partner institutions about their experiences

exploring how the institution uses the Academic Infrastructure.

The audit team also gathers evidence by focusing on examples of the institution's internal qualityassurance processes at work through visits to partners. In addition, the audit team may focus on aparticular theme that runs throughout the institution's management of its standards and quality.This is known as a 'thematic enquiry'.

From 2004, institutions will be required to publish information about the quality and standards oftheir programmes and awards in a format recommended in document 03/51, Information on qualityand standards in higher education: Final guidance, published by the Higher Education FundingCouncil for England. The audit team reviews how institutions are working towards this requirement.

ISBN 1 84482 369 5

© Quality Assurance Agency for Higher Education 2005

All QAA's publications are available on our website www.qaa.ac.uk

Printed copies are available from:Linney DirectAdamswayMansfieldNG18 4FN

Tel 01623 450788Fax 01623 450629Email [email protected]

Summary 1

Introduction 1

Outcome of the collaborative provision audit 1

Features of good practice 1

Recommendations for action 1

National reference points 2

Main report 4

Section 1: Introduction 4

The institution and its mission as it relates to collaborative provision 4

Background information 5

The collaborative provision audit process 6

Developments since the institutional audit of the awarding institution 7

Section 2: The collaborative audit investigations: the awarding institution's processes for quality management in collaborative provision 7

The awarding institution's strategicapproach to collaborative provision 7

The awarding institution's framework for managing the quality of the students' experience and academicstandards in collaborative provision 8

The framework for direct and collaborative teaching provision 9

The framework for accredited and validated provision 10

The awarding institution's intentions for enhancing the management of itscollaborative provision 12

The awarding institution's internal approval, monitoring and reviewarrangements for collaborative provision leading to its awards 12

Direct and collaborative teaching partnerships 12

Summary 14

Accredited partnerships and validated programmes 14

External participation in internal reviewprocesses for collaborative provision 20

External examiners and their reports in collaborative provision 20

Direct and collaborative teachingpartnerships 20


The use made of external reference points in collaborative provision 23

Review and accreditation by externalagencies of programmes leading to the awarding institution's awards offered through collaborative provision 24

Student representation in collaborative provision 25

Feedback from students, graduates and employers 26

Student admission, progression, completion and assessment information for collaborative provision 28

Assurance of the quality of teaching staff in collaborative provision; appointment, appraisal, support and development 28

Direct and collaborative teachingpartnerships 28


Assurance of the quality of distributed and distance methods delivered through an arrangement with a partner 29

Learning support resources for students in collaborative provision 30

Academic guidance and personal support for students in collaborative provision 31

Section 3: The audit investigations:published information 31

The experience of students in collaborative provision of the publishedinformation available to them 31

Contents

Reliability, accuracy and completeness ofpublished information on collaborativeprovision leading to the awarding institution's awards 33

Findings of the collaborative provision audit 36

The effectiveness of the implementation of the awarding institution's approach tomanaging its collaborative provision 36

The effectiveness of the awarding institution's procedures for assuring thequality of educational provision in itscollaborative provision 37

The effectiveness of the awarding institution's procedures for safeguarding the standards of its awards gained through collaborative provision 39

The awarding institution's use of theAcademic Infrastructure in the context of its collaborative provision 41

The utility of the CPSED as an illustration of the awarding institution's capacityto reflect upon its own strengths andlimitations in collaborative provision, and to act on these to enhance quality and safeguard academic standards 41

Commentary on the institution's intentions for the enhancement of itsmanagement of quality and academicstandards in its collaborative provision 42

Reliability of information provided by the awarding institution on itscollaborative provision 42

Features of good practice 43

Recommendations 43

Appendix 44

The Open University's response to thecollaborative provision audit 44

Summary

Introduction

A team of auditors from the Quality AssuranceAgency for Higher Education (QAA) visited theOpen University (the University, or OU) from 9to 13 May 2005 to carry out an audit of thecollaborative provision offered by the University.The purpose of the audit was to provide publicinformation on the quality of the programmesof study offered by the University througharrangements with collaborative partners, andon the discharge of the University'sresponsibility as an awarding body in assuringthe academic standard of its awards madethrough collaborative arrangements.

To arrive at its conclusions the team spoke tomembers of staff of the University, and read awide range of documents relating to the waythe University manages the academic aspects ofits collaborative provision. As part of the auditprocess, the audit team visited four of theUniversity's collaborative partners where itspoke to students on the University'scollaborative programmes and to members ofstaff of the partner institution.

The words 'academic standards' are used todescribe the level of achievement that a studenthas to reach to gain an award (for example, adegree). It should be at a similar level acrossthe UK.

Academic quality is a way of describing howwell the learning opportunities available tostudents help them to achieve their award. It isabout making sure that appropriate teaching,support, assessment and learning opportunitiesare provided for them.

The term 'collaborative provision' is taken tomean 'educational provision leading to anaward, or to specific credit toward an award, ofan awarding institution delivered and/orsupported and/or assessed through anarrangement with a partner organisation' (Codeof practice for the assurance of academic qualityand standards in higher education: Collaborativeprovision and flexible and distributed learning(including e-learning), 2004, paragraph 13),published by QAA.

In an audit of collaborative provision bothacademic standards and academic quality arereviewed.

Outcome of the collaborativeprovision audit

As a result of its investigations the audit team'sview of the University is that:

only limited confidence can reasonably beplaced in the soundness of the University'spresent and likely future management ofthe academic standards of its awardsmade through collaborative arrangements,and that

only limited confidence can reasonably beplaced in the present and likely futurecapacity of the University to satisfy itselfthat the learning opportunities offered tostudents through its collaborativearrangements are managed effectively andmeet its requirements.

Features of good practice

The audit team identified the following area asbeing good practice:

the opportunities for enhancementprovided though the curriculumconferences and the OU ValidationServices annual conference.

Recommendations for action

The audit team also recommends that theUniversity should consider further action in anumber of areas to ensure that the academicquality of programmes and standards of theawards it offers through collaborativearrangements are maintained. The teamconsiders it essential that the University:

strengthens the means by which itestablishes confidence in the security andcomparability of the academic standardsof its awards provided through accreditedpartners, and

implements valid and reliable mechanismsto enable it to have confidence that thequality of learning opportunities andstudent support provided through all its

Collaborative provision audit: summary

page 1

accredited partners are managed effectivelyand meet the University's requirements

and considers it advisable that the University:

ensures that all local external examinersappointed for courses offered overseas areappropriately independent and haveexperience of undertaking the deliveryand assessment of UK higher education.

National reference points

To provide further evidence to support itsfindings, the audit team also investigated theuse made by the University of the AcademicInfrastructure which QAA has developed onbehalf of the whole of UK higher education. TheAcademic Infrastructure is a set of nationallyagreed reference points that help to define bothgood practice and academic standards. Theaudit found that the University was makingeffective use of the Academic Infrastructure inthe context of its collaborative provision.

In due course, the audit process will include acheck on the reliability of the teaching qualityinformation (TQI) published by institutions inthe format recommended in the HigherEducation Funding Council for England'sdocument 03/51, Information on quality andstandards in higher education: Final guidance.The audit team was satisfied that theinformation that the University and its partnersare publishing currently about the quality of itscollaborative programmes and the standards ofits awards is reliable, and that the University ismaking adequate progress to providing TQIdata for its collaborative provision.

Open University

page 2

Main report

Main report 1 An audit of the collaborative provisionoffered by the Open University (the University, orOU) was undertaken during the period 9 to 13May 2005. The purpose of the audit was toprovide public information on the quality of theprogrammes of study offered by the Universitythrough arrangements with collaborativepartners, and on the discharge of the University'sresponsibility as an awarding body in assuringthe academic standard of its awards madethrough collaborative arrangements.

2 Collaborative provision audit supplementsthe institutional audit of the University's ownprovision. The process of collaborative provisionaudit has been developed by the QualityAssurance Agency for Higher Education (QAA) inpartnership with higher education institutions(HEIs) in England. It provides a means forscrutinising the collaborative provision of an HEIwith degree-awarding powers (awardinginstitution) where the collaborative provision wastoo large or complex to have been included inthe institutional audit of the awarding institution.The term 'collaborative provision' is taken tomean 'educational provision leading to an award,or to specific credit toward an award, of anawarding institution delivered and/or supportedand/or assessed through an arrangement with apartner organisation' (Code of practice for theassurance of academic quality and standards inhigher education (Code of practice), Section 2:Collaborative provision and flexible and distributedlearning (including e-learning), published by QAA2004, paragraph 13).

3 The collaborative provision audit checkedthe effectiveness of the University's proceduresfor establishing and maintaining the standardsof academic awards through collaborativearrangements; for reviewing and enhancing thequality of the programmes of study offeredthrough collaborative arrangements that leadto those awards; for publishing reliableinformation about its collaborative provision;and for the discharge of its responsibility as anawarding body. As part of the collaborativeaudit process, the audit team visited four of theUniversity's collaborative partners.

Section 1: Introduction

The institution and its mission as itrelates to collaborative provision

4 The Open University was founded byRoyal Charter in 1969 and began teachingstudents in 1971. It was established specificallyand distinctively to provide supported openlearning opportunities for students to studywith the University from a distance on a part-time basis while continuing with othercommitments. Its administrative headquartersare at Walton Hall in Milton Keynes,Buckinghamshire, and it has 13 regional centreslocated throughout the United Kingdom. It has55 collaborative partners, 19 of whom areoverseas, operating throughout the EuropeanUnion, Eastern Europe, Switzerland, Sloveniaand extending into the Middle and Far East.

5 The University has full degree-awardingpowers. Its undergraduate and most of itspostgraduate taught study is fully modular, andstudents register for credit-rated courses(modules), and accumulate credit viacompletion of courses toward qualifications.About 600 courses, many of which areinterdisciplinary, are available across a widerange of subjects. There is an open admissionpolicy for undergraduate taught programmes,and students, the majority of whom are matureand in employment, enter the University with awide range of study goals and previouseducational and general experience. There is anactive research community, and the Universityalso offers full-time master's degrees in researchmethods, and full and part-time externalresearch degrees. An institutional audit of theUniversity by QAA took place in March 2004.The current audit focuses on the University'sdischarge of its responsibility as an awardingbody for awards made in its name througharrangements with collaborative partners.

6 The collaborative provision self-evaluationdocument (CPSED) prepared for the audit bythe University explained that the Universitycategorises its collaborative provision which maylead to an award of the University as follows:

Open University

page 4

direct distance teaching

collaborative distance teaching

accreditation and validation

research sponsoring (for research degrees).

The last of these was included in theinstitutional audit of the University in 2004. Thecurrent audit is concerned with the University'smanagement of accreditation and validation ofprogrammes of study leading to the University'sawards, and with direct and collaborativedistance teaching.

7 The University has some 27 per cent of theUK's part-time higher education students whenexpressed in full-time equivalents (FTEs), and 35per cent of UK part-time undergraduate levelstudents. In 2003-04 about 15,400 studentswere registered for validated awards of theUniversity at accredited institutions, and nearly24,000 students studied outside the UK throughdirect and collaborative teaching partnerships.

8 Open University Validation Services (OUVS)was set up in1992 and provides an externalaccreditation and validation service, including thevalidation of programmes in subject areas outsidethe University's portfolio. OU Worldwide Ltd(OUW) is a separately incorporated companyestablished in 1997 as the internationalcommercial division of the University. It managesthe contractual and financial aspects of makingthe University's learning materials available on aninternational scale, working with the University'sfaculties and schools who are responsible formanaging the quality and academic standards ofthis type of provision.

9 At the time of the audit, the University wasengaged in a review of academic governance, inparticular of the Senate, Academic Board and itssubstructure. The CPSED anticipated that thisreview would allow some rationalisation of themanagement of quality and academic standardsthrough the governance structure.

Mission statement10 The University, following a major review ofits mission and strategic priorities, adopted itscurrent Mission Statement in March 2004:

'The Open University is open to people, places,methods and ideas.

It promotes educational opportunity and socialjustice by providing high-quality universityeducation to all who wish to realise theirambitions and fulfil their potential.

Through academic research, pedagogicinnovation and collaborative partnership itseeks to be a world leader in the design,content and delivery of supported open anddistance learning'.

11 This Statement continues to draw on thephilosophy and values enunciated at the timeof the inauguration of the University in 1969. Italso specifically identifies collaborativepartnership as a key means of achieving itsinstitutional objectives. It is supported by a setof core values and strategic priorities, detailedin the document OU Futures 2004-08.

Background information

12 The published information available forthis collaborative provision audit included:

statistical data provided by the HigherEducation Statistics Agency (HESA), theHigher Education Funding Council forEngland (HEFCE), Universities and CollegesAdmissions Service, Higher EducationResearch Opportunities and the Universitythe information on the University's website, including its degree prospectuses andcourse cataloguesthe report of the continuation auditundertaken by QAA in May 1998,published in January 1999the report of the institutional auditundertaken by QAA in March 2004reports of reviews by QAA of provision atsubject level, carried out in the University'spartner institutions and published sincethe 1998 continuation auditthe report of the collaborative provisionaudit by the Higher Education QualityCouncil in February 1996, published inJuly 1996

Collaborative provision audit: main report

page 5

reports of reviews by QAA of theUniversity's overseas collaborativeprovision, published since the 1998continuation audit

report on the University's FoundationDegree provision, published in June 2003.

13 The University and its partners providedQAA with:

an institutional CPSED

prospectuses and course catalogues

documentation associated with each ofthe four partner institutions visited as partof the audit

documentation associated with each ofthe six partner institutions which were thesubject of desk studies as part of the audit

sample professional, statutory andregulatory body (PSRB) reports relating topartner provision

the University's Corporate Plan, Plans ForChange: the University's Strategic Plan for2002 to 2012

statistical information, which included ademographic analysis of the collaborativestudent population, student progress andachievement at course and award level.

14 During the briefing and audit visits, theaudit team was given ready access to a range ofthe University's documents in hard copy and ona CD-ROM accompanying the CPSED. TheUniversity also provided the team with a rangeof documentation relating to partner institutionsselected by the audit team for visits and fordesk-based studies. The partner institutionsvisited provided further documentation. Theteam also had access to, and was given copiesof, internal documents and committee minutesrelating to partner institutions.

The collaborative provision auditprocess

15 Following a preliminary meeting at theUniversity in July 2004 between a QAA assistantdirector and representatives of the Universityand students, QAA confirmed in September2004 that four partner visits would be

conducted between the briefing and auditvisits. The University provided its CPSED inDecember 2004. On the basis of the CPSEDand other published information, the auditteam confirmed the four partner institutionsthat it would visit. The University provided QAAwith briefing documentation in March 2005 foreach of these partner institutions.

16 The OU Students' Association (OUSA) didnot consider that it was in a position to representthe views of students studying throughcollaborative partners of the University. It wasagreed that student opinion should be canvassedthrough the partner visits instead.

17 The audit team visited the University from4 to 6 April 2005 for the purposes of exploringwith senior members of staff of the University,members of staff actively involved incollaborative partnerships, studentrepresentatives and representatives from partnerinstitutions, matters relating to the managementof quality and academic standards incollaborative provision raised by the University'sCPSED and other documentation, and ensuringthat the team had a clear understanding of theUniversity's approach to collaborativearrangements. At the close of this briefing visit,a programme of meetings for the audit wasagreed with the University.

18 The audit team confirmed that in additionto the four partner visits, a further six links werefollowed through desk-based studies. Theserepresented the University's partnershipsoverseas as well as another UK partner.

19 Visits to partner institutions followed thebriefing visit and took place in the period 7 Aprilto 4 May 2005. During these visits members ofthe audit team met senior staff, teaching staffand student representatives of the partnerinstitutions. The team is grateful to the staff andstudents of those partner institutions for theirhelp in understanding the University'sarrangements for collaborative provision.

20 The audit visit took place at the Universityfrom 9 to13 May 2005. Meetings were held withgroups of staff of the University associated withits collaborative provision and the five desk-based

Open University

page 6

studies were carried out. The team is grateful toall those staff and students, both of the Universityand its partners, who participated in meetings.

21 The audit team comprised Dr M Edmunds,Dr R Gadsden, Miss R Pelik, Mr N Pratt, Dr MWing, auditors, and Dr K Hodgson, auditsecretary. The audit was coordinated for QAAby Dr D J Buckingham, Assistant Director.

Developments since the institutionalaudit of the awarding institution

22 The previous institutional audit took place inMarch 2004. While none of the features of goodpractice or points for further considerationrelating to the 2004 audit refers directly tocollaborative provision, some have an implicitbearing and are addressed later in this report.The report of the 2004 audit noted as features ofgood practice the University's monitoring of thesecurity of its academic standards; its systematicand comprehensive collection and use offeedback from students; its arrangements forappointing, monitoring and supporting associatelecturers; its proactive stance in giving guidanceand support to students; and the third-partymonitoring system for research students.

23 The 2004 report identified a number ofpoints for further consideration. These includedusing the governance review as a means ofarticulating more clearly the University'sapproach to assuring and enhancing the qualityof provision; reviewing the effectiveness of itsprocedures for determining whether intendedprogramme learning outcomes will be metthrough pathways leading to named awards;systematically including external subjectexpertise in the procedures for approvingprogrammes leading to named awards;considering the effectiveness of its presentsystem for gaining a university-level overview ofannual review activity at course and programmelevels; and reflecting on the consequences ofusing different versions of programmespecifications for different audiences. Inaddition, the report identified that it wasdesirable for the University to consider ensuringthat research students have clear informationabout their financial entitlements.

Section 2: The collaborativeaudit investigations: theawarding institution's processesfor quality management incollaborative provision

The awarding institution's strategicapproach to collaborative provision

24 The CPSED confirmed that the University'sMission identifies collaborative partnership as akey means of achieving its objectives in relationto its own curriculum and its delivery, and drewattention to the University's core values andstrategic priorities, set out in the document OUFutures 2004-08. It stated that the Universityaspires to be 'a world-class university with aninternational presence', and plans to use'accreditation of institutions and other types ofpartnership as a means of expanding globalreach', but only where the University's criteriafor partnership are fully met. The University'spriorities are then interpreted by 'middle-tier'academic, pedagogic and enabling plans. Theseinclude two strategy documents: Size & Shape:The Curriculum Strategy 2003-2008, and arecently approved international strategy.

25 This audit is concerned with the models ofcollaboration that the University categorisesunder the headings of:

direct distance teaching

collaborative distance teaching

accreditation and validation.

26 In direct and collaborative distanceteaching arrangements, students are registeredas students of the University, on Universitycourses, studying within the University'scurriculum for the University's awards. Directand collaborative teaching partnerships aretherefore sometimes also referred to collectivelyas 'curriculum partnerships'. The Universityretains quality assurance responsibility in itsentirety in such arrangements, this responsibilitybeing discharged through the relevant centralacademic units of the University.


page 7

27 In direct teaching partnerships, theUniversity, not the partner, is responsible fordelivery of academic aspects of the programme,while in the collaborative teaching modelpartners are partly responsible for aspects oftuition and assessment. The CPSED explainedthat in the case of direct distance teaching'tutorial support and continuous assessment…arecarried out by OU staff', while in collaborativeteaching partners have 'delegated responsibilityfor the appointment and management ofassociate lecturers'. The choice of whetherteaching partnership is direct or collaborativedepends on the University's evaluation of thecapability of the proposed partner institution todeliver University courses and programmes.

28 The CPSED identified two further'common variants within the collaborativeteaching model'. The first, courses intranslation, is where students register and studyfor a University award, but the course materialis provided and teaching takes place in thelocal language. The second, delegation ofresponsibility for a particular set of services, iswhere the partner institution takes someresponsibility for aspects of teaching andlearning, and assessment. Quality assuranceresponsibilities, however, remain with theUniversity. While these models do not generallyapply to validated provision, nonetheless someor all elements of it may be involved,particularly where the partner institution drawson the University's curriculum content in thedevelopment of its programmes.

29 In the accreditation and validation model ofcollaborative provision, the University'sresponsibility for the management and oversightof processes to assure the quality and academicstandards of provision is discharged throughOUVS. Validation is the process whereby theUniversity assures itself that 'the proposed highereducation programme is of appropriate quality,the intended learning outcomes are at anappropriate level and the standard set iscomparable to similar awards offered throughoutUK higher education'. Each programme must beseparately validated. Only accredited institutionscan offer validated programmes of study.Accreditation is a status achieved by a partner

institution where the University has confirmed toitself that 'the partner has a suitable environmentand infrastructure for the development, deliveryand quality assurance of validated awards'.Students are registered with the accreditedinstitution and follow programmes of studyowned and delivered by the accreditedinstitution but validated by the University.Students are registered with the University bythe accredited institution only for the purpose ofreceiving an award of the University.

The awarding institution's frameworkfor managing the quality of thestudents' experience and academicstandards in collaborative provision

The institutional-level framework30 The CPSED confirmed that the University 'isresponsible for the academic standards of all itsawards and the quality of programmes leadingto them', and that 'its framework for managingquality and standards is designed to ensure thatstandards are appropriately set and achieved'.While formal responsibility for the academicstandards of all awards lies with the University'sSenate, the Academic Board and its committeesand subcommittees take forward theseresponsibilities. In terms of collaborativeprovision the key committee of the AcademicBoard is the Curriculum and Awards Board (CAB)to which the Validating Committee (ValC), theCurriculum Partnerships Committee (CuPC) andAwards Committee report. The Quality andStandards Board (QSB) also receives somerelevant information from these committees andreports to the Academic Board on those items.

31 The Curriculum and Awards Board (CAB)takes strategic oversight of the University'scurricula and awards, and formulates relevantacademic policy. It has responsibility foroverseeing the review of collaborativeprovision, confirming review procedures havebeen followed and for identifying any majorissues requiring a university-level response. TheCPSED explained that 'major issues whichrequire response at University level' arereported by CAB to the QSB, and through it tothe Academic Board.

Open University

page 8

32 The Quality and Standards Board (QSB)oversees the University's quality assuranceframework. The framework for review of coursesand awards is set by the Internal ReviewCommittee (a subcommittee of QSB), but not forthe review of collaborative partnerships which isoverseen by CAB who reports on it to QSB.

33 Validating Committee (ValC) exercisesauthority on behalf of the University 'forapproving and for reviewing accreditedinstitutions, and validating and revalidatingprogrammes'. The CPSED explained that ValC'is responsible through the Curriculum andAwards Board to the Academic Board for alldecisions concerning accreditation andvalidation'. ValC is charged with responsibilityfor ensuring that academic standards arecomparable 'with awards granted andconferred throughout higher education in theUK', and for the 'approval and monitoring ofexternal examiners for taught courses in respectof accredited institutions'. It is responsible foroverseeing the effectiveness of the University'sprocedures for accreditation and validation. TheCommittee has substantial externalmembership, as does CAB. Two keysubcommittees of ValC are the AccreditationWorking Group, which provides initial scrutinyof proposals for accreditation, and the workinggroup which scrutinises the annual monitoringreports from accredited partners.

34 Curriculum Partnerships Committee(CuPC) is responsible to CAB for 'all regulatoryand procedural matters relating to curriculumpartnerships' and the institutional managementof quality and standards relating to partnershipsother than those operating under accreditationand validation. Its key relevant functionsinclude the approval of direct and collaborativeteaching partnerships and the evaluation andreview of established partnerships.

35 All awards derived from the University'scurriculum and which may be taught throughdirect or collaborative teaching partnerships'are considered and recommended for approvalby the Awards Committee', which is required to'take cognisance of the University's validatedprogrammes and awards'.

36 Most responsibilities with regard tocollaborative provision reside with the Pro-Vice-Chancellor (PVC) (Curriculum and Awards) whohas responsibility for the quality and standards ofcollaborative provision, strategic planning andoperational matters. The PVC (Curriculum andAwards) chairs CAB, ValC and CuPC as well asoverseeing the work of OUVS. The Director ofOUVS is responsible for the 'effective operation ofthe Validating Committee and working groups',relationships with accredited institutions and themanagement of the budget of OUVS. OUWBoard has oversight of the financial, contractualand commercial aspects of international directand collaborative teaching partnerships.

37 The CPSED provided a brief update on theongoing review of governance, and indiscussions with senior staff the audit team wasinformed that the review was now complete.The main relevant changes are a clearerarticulation of CAB's responsibilities for settingand monitoring academic standards for allawards and a redefinition of the role of QSB asa quality assurance and enhancementcommittee. CAB is to be renamed the'Curriculum, Awards and Validation Committee'which 'would make explicit and reflect theimportance of validation activity within theUniversity', with ValC continuing to report to it.The University was of the view that the resultsof the governance review would have no realimpact on the present arrangements formanagement of academic standards andquality assurance of collaborative provision, aview with which the audit team concurred.

The framework for direct andcollaborative teaching provision

38 OUW acts as a first point of contact forenquiries from overseas about the University'scourses, materials and methodology. TheUniversity's learning materials are commerciallyavailable to both individuals and establishmentsacross the world. Many of these relationshipsonly involve the sale or licensing of theUniversity's materials for use by otherinstitutions for their own awards, but wherethey lead to an award of the University, OUWmanages some partnership aspects of overseas


page 9

collaborative provision. In all cases, however, thequality and academic standards of provision inthese relationships is managed by theUniversity's faculties and schools, with thequality assurance arrangements operating underthe standard procedures and structures forquality assurance that apply to the University'sown provision. The CPSED explained that OUWpartnership managers 'liaise closely withmembers of academic staff on issues relating tothe academic provision'. The audit team formedthe view that there is a strong workingrelationship between the 'home' academic unitand the OUW partnership manager in respect ofthis type of collaborative arrangement.

39 With direct and collaborative teachingpartnerships the University retains responsibilityfor quality assurance 'in its entirety'. The CPSEDstated that 'all students who take OU coursesshould experience substantially the sameteaching materials, learning resources andassessment'. Quality assurance procedures,therefore, are set out in the University'sstandard documentation, with any specificvariations being documented in the operationshandbook for each partnership.

40 In direct teaching partnerships studentsare taught and assessed by University staff. Thepartner has responsibility formarketing/promotion, registration, fees and thedistribution of University course material. All theUniversity's standard monitoring processesapply, including those for gathering andevaluating student feedback and for monitoringperformance and achievement. The Universityis confident that its framework for monitoringthe quality of students' learning opportunitiesin direct teaching is effective, a view with whichthe audit team would concur.

41 In the case of direct and collaborativeteaching partnerships, the University has alesser role in the direct management andoversight of the students' learningopportunities. Some responsibility for deliveryand assessment is delegated to the partner, andstaff may be recruited locally, but the overallstudent experience is still overseen according tostandard University procedures. Students are

advised that their first point of contact is thelocal institution where they are studying.

42 In both direct and collaborative teachingpartnerships the CPSED explained that students'are subject to the sameexamination/assessment as all other studentsregistered for those courses'. Assessment followsthe same process as 'home' OU courses, as setout in the University's Code of Practice forStudent Assessment, and in course assessmenthandbooks. Where responsibility for any aspectsof assessment is delegated, it is recorded in thepartnership contract and operations handbook.In the case of courses delivered and assessed inlanguages other than English, local externalexaminers are appointed who verify aspects ofthe translation of material. Assessment in thelocal language is verified by a sample oftranslated scripts seen by a second UK marker.The effectiveness of procedures and consistencyin teaching and assessment is monitored in thesame way as all University provision. Distinctcourse codes are used so that examination andassessment boards are able to monitor studentperformance and standards in collaborativeprovision, but the administration of the processis otherwise handled by the University'sAssessment, Credit and Awards Office 'exactlythe same as for all other courses'.

43 The framework for managing the qualityof learning opportunities and security of theacademic standards of awards for direct andcollaborative teaching partnerships is, therefore,largely built around the University's standardprocesses, supplemented by additionalmechanisms to enable collaborative provisionto be distinguished and managed effectively.

The framework for accredited andvalidated provision

44 OUVS is responsible, on behalf of theUniversity, for the accreditation of institutionsand for the validation of accredited partners'programmes of study. Quality assurance ofvalidated programmes is seen primarily as theresponsibility of the accredited partnerinstitution, the ability of the partner toundertake the required level of responsibility

Open University

page 10

having been assessed in the process ofaccreditation. In some cases, the University willhave had experience of working with thepartner under the direct and collaborativeteaching partnership model and will, therefore,already have some measure of the partner'scapacity to accept the delegated authority ofaccreditation. The requirements for achievingaccreditation are set out in the Handbook forValidated Awards (the Handbook). While theUniversity retains ultimate authority foracademic standards, there is a very high degreeof delegation to accredited partners withprogrammes being seen by the University as'owned and delivered' by accredited partners.This level of delegation means that theUniversity expects its accredited partners totake action when issues arise without waitingfor it to require or initiate action, although theUniversity has, in the last two or three years,become more proactive in these matters.

45 The framework for accreditation,validation and annual monitoring is set out inthe Handbook. The extent to which thelearning opportunities available to the studentare satisfactory is established both throughaccreditation of the partner and throughvalidation of the individual programmes. Anecessary condition set for accreditation is thatpartners are able to provide an appropriatelearning environment and 'accept collectiveresponsibility for the quality of students'learning experiences'. Accredited institutionsare given 'full responsibility for teaching,assessment and further development' ofvalidated programmes, and are regarded by theUniversity as being responsible for the quality oftheir students' experience. The students'primary relationship is seen to be with thepartner. This delegated responsibility isoverseen by the University through annualmonitoring and formally reconfirmed ininstitutional review and revalidation.

46 The University has no direct role in theassessment of student achievement on itsvalidated programmes, the accredited partnerhaving full delegated responsibility for theassessment of students. Partners' academic

regulations are considered by the University ataccreditation/institutional review, and anyprogramme specific regulations atvalidation/revalidation. The Handbook specifiesthe matters which must be covered within theacademic regulations. The CPSED explainedthat 'the Board of Examiners…or equivalentacademic authority of an accredited institutionis authorised to recommend the conferment ofa validated award'.

47 The primary means used by the Universityfor the monitoring of student achievement inrelation to the academic standard that itrequires of its awards is the reports of theexternal examiners of the validatedprogrammes. The CPSED, the Handbook, andsenior staff who met the audit team confirmedthat the University regards external examinersof validated programmes as belonging to theaccredited partners, not to the University. Priorto 2001, external examiners of validatedprogrammes only reported directly to theaccredited partner, and their reports wereavailable to the University, in due course, as partof annual monitoring. Since 2001, externalexaminers have also reported directly to theUniversity through OUVS. Instruction andassessment in languages other than English arepermitted, subject to a set of stipulatedrequirements. In such cases the Universityrequires an appropriate pool of bilingual persons.The University's framework for managingacademic standards in its collaborative provisiondoes not include a mechanism for comparativeoverview by the University of studentachievement, or for providing comparative datato partners that might aid them inbenchmarking student achievement againstsimilar provision in the sector

48 The University has recognised a need tostrengthen its oversight of assessment ofstudent achievement in validated provision. TheCPSED stated that 'a closer overview andunderstanding by the University of assessmentpractices in accredited institutions is one of theareas identified for the first period ofaccreditation for new institutions', and that,from September 2004, an OUVS partnership


page 11

manager will 'observe a sample of assessmentboards in newly accredited institutions'.

49 The framework for managing the qualityof the learning opportunities available tostudents and academic standards of awards inaccredited partnerships and validatedprogrammes is built upon a substantial degreeof authority delegated to the partner. Althoughthe University has recently sought to strengthenthis framework, and the plans it has identifiedto further this, the audit team was notpersuaded that the University's framework is yetsufficiently robust and rigorous to enable theUniversity to have full confidence in itseffectiveness for assuring the quality ofprovision and the academic standards of itsawards in validated programmes.

The awarding institution's intentionsfor enhancing the management of itscollaborative provision

50 At the time of the audit visit the Universitywas about to publish a revised (2005) editionof the Handbook, and was continuing toevaluate and enhance the effectiveness of CuPCand ValC and their subgroups. It wasimplementing procedural improvements inareas such as the development of templates toimprove the standard of reporting frompartners, and had recognised the need toencourage accredited partners to make betteruse of quantitative data. The audit teamconsidered that the procedural improvements,including the use of the subgroups, werepositive moves towards enhancing theUniversity's capacity for the effectivemanagement of its collaborative provision.

51 The University provides a variety of formaland informal enhancement opportunities for itspartners. CuPC and ValC work to identifyappropriate areas for enhancement which arethen addressed through the OUVS AnnualConference, curriculum conferences andseminars organised by OUVS and by theUniversity's Centre for Research in HigherEducation (CHERI). The network of OUVSaccredited institutions provides opportunitiesfor individual staff development through staff

visiting one another's institutions in a variety ofcapacities, for example, as members orobservers of accreditation/validation panels.

52 At the time of the audit visit, ValC hadrecently initiated a student feedback project withthe intention of identifying good practice inaccredited partners in student evaluation ofvalidated provision. The audit team consideredthat while this is a laudable project, goodpractice in these matters could have been readilyidentified through greater attention to thesystematic use of the annual monitoring process.

53 The audit team learnt from its discussionsand visits to partners that the revisedaccreditation, review and validation processeshave been welcomed by accredited partners.While the University offers some appropriateopportunities for enhancement to partners, ofwhich the curriculum conferences and OUVSannual conference are good examples, itappeared to the team that the extent to whichthe partners avail themselves of these, or other,opportunities is left largely to their discretion.

The awarding institution's internalapproval, monitoring and reviewarrangements for collaborativeprovision leading to its awards

Direct and collaborative teachingpartnerships

Approval54 Partners in direct and collaborativeteaching arrangements are approved through abusiness appraisal. Direct and collaborativeteaching partnerships have to be approved byboth the Board of the relevant central academicunit of the University and by CuPC, whichrecommends approval to CAB. Withinternational partnerships the OUW Board isalso involved to assess the financial andcontractual issues. University staff considering adirect and collaborative teaching partnershiphave recourse to advice from the University'sCurriculum Partnerships Advisory Service (CPAS)and from OUW, and can draw on a series ofToolkits, available on the web and in hard copy,which provide information about each stage of

Open University

page 12

the process and each aspect of partnership. Areview of collaborative provision resulted in theestablishment of CPAS at the end of 2002, theproduction of guidance materials and a moresystematic approach to the development of thisarea of the University's activity.

55 The teaching materials used within directand collaborative partnerships are mainly theUniversity's standard course materialsdeveloped through its normal internalprocedures, with some material being adaptedfor specific collaborative partners to meet localand cultural needs. Some programmes areoffered in translation, in which case theaccuracy of the translation is checked throughthe use of external examiners local to thepartner, confirmed as necessary by theUniversity (see also below, paragraph 85).

Annual monitoring56 Annual monitoring for University coursesdelivered through direct and collaborativeteaching partnerships was introduced in 2003(for courses delivered in 2002), and the CPSEDexplained that the process was designed tofocus on the 'health' of the partnership. Thismonitoring has enabled CuPC to take anoverview across direct and collaborativeteaching partnerships for the first time. Anumber of queries were raised by this overview,and a report following up on these items wasreceived by CuPC, enabling it to reviewprogress. The first operation of this procedurewas evaluated in detail by CuPC. Theimprovements made for the following cycleincluded more direct involvement of partners(the 2003 pro formas had been completedsolely by partnership managers), and the taskof detailed scrutiny was delegated to asubgroup set up for the purpose, reporting toCuPC. The process is seen by CuPC as iterative,and 'designed to inform and add value…toenhance relationship management throughobtaining and responding to comments frompartners'. The audit team heard that furtherchanges are planned which will seek to alignthis process with other processes, and to makemore explicit any distinctions between thecollaborative versions of courses and

mainstream delivery within standard annualcourse reports and in the reports of externalexaminers. The use of distinguishing coursecodes gives the University the ability to conductextensive comparative analysis of studentachievement between different providers.

57 The audit team concluded that the annualreview of curriculum partnerships and the followup of issues reported to CuPC allowed theUniversity to exercise oversight and to identifyissues for wider consideration. The process hasbeen established only recently, but had beencritically reviewed. Changes have been madewhich are designed to seek constructivealignment with linked monitoring activities and,in future, further to improve the timing andanalysis and briefing of those involved.

Periodic review58 The review process for individual coursesoffered through direct and collaborativeteaching partnerships is the same reviewprocess that is applied to courses offered by theUniversity itself. The CPSED noted thatsufficient distinctions are made to enable theUniversity to identify any significant differencesin the academic standards achieved or in thestudent experience, and that this ability fordifferentiation is being further developed with aseparate report form for courses presented inpartnership. The periodic review ofprogrammes was introduced by the Universityfor the first time in 2003. All programmes inbroad subject groups are reviewed on a six-yearcycle. The CPSED explained that this process'does not specifically or separately addressdirect and collaborative teaching but doesaddress the effectiveness of the reviewarrangements as a whole within the area ofprovision'. Partnerships are based on a fixed-term contract. The audit team was informedthat there is no formal review procedure for thepartnerships themselves, but that the future ofpartnerships is discussed by CuPC, OUW andthe relevant central academic unit for individualpartnerships depending on the partner and thenature of the link. External involvement in suchdiscussions is limited to external membership ofCuPC. As a result of such reviews of


page 13

partnerships some might be encouraged toconsider moving from direct or collaborativeteaching to accreditation and validation, and/ormight move out of direct teaching to licensingof the University's materials.

Summary

59 Overall, the audit team concluded that theUniversity's procedures for the approval,monitoring and review of the quality andacademic standards of collaborative provisionleading to its awards offered through direct andcollaborative teaching arrangements areappropriate. While the team formed the viewthat the University could make better use of theinformation available to it from thesepartnerships for the purposes of taking a widerinstitutional-level view of the health anddevelopment of such arrangements, itconsidered that the University's currentprocedures and the measures being taken tofurther improve current procedures provided asound basis for the management of thisdimension of its collaborative provision.

Accredited partnerships andvalidated programmes

Approval and re-approval of accreditedpartners60 The CPSED defined accreditation as theprocess by which partners seeking to offervalidated programmes are 'judged by theUniversity, deploying a panel of peers, to meetestablished principles confirming it can providea satisfactory environment for the conduct andquality assurance of HE programmes leading toOU validated awards'. It stated that the process,which focuses at institutional level, confirms'that the partner has a suitable environmentand infrastructure for the development, deliveryand quality assurance of validated awards' asset out in its Principles for Institutional Approvalsection of the Handbook, and that the partnerhas 'the capacity and ability to undertake thosequality assurance functions which will bedelegated to them by the University'.

61 The accreditation process involves anextended period of preparation, including

advice, guidance and preliminary visits (theadvisory visit and the facilitation visit). TheCPSED explained that the review of procedureswhich has taken place since 2001 is intended tomake 'the initial and subsequent approvalprocess more iterative and developmental'.Accreditation itself involves the production of asubmission document and an accreditation visitby a panel. Guidance is provided to prospectiveinstitutions and to panel members in theHandbook. The panel is normally chaired by amember of ValC (who may be a member of theUniversity's staff or may be an externalmember) and who is, thus, able to speak to thereport at ValC which has delegated authorityfrom the University to make decisions aboutaccreditation on behalf of the University.

62 Accreditation is based upon a very highdegree of delegated responsibility to the partnerinstitution which, according to the CPSED,includes 'full responsibility for teaching,assessment and further development' of validatedprogrammes. Partners have responsibility for themonitoring, evaluation and quality enhancementof programmes; the establishment, managementand conduct of boards of examiners; theengagement, briefing and training of externalexaminers; the appointment, appraisal anddevelopment of staff and for the developmentof the curriculum. Further delegated authoritymay be given where ValC has 'confidence in [apartner's] internal quality assurance procedures',although there remains a reporting requirementto OUVS. Additional delegated authority maybe granted for the organisation and conduct ofprogramme validation and revalidation, theappraisal of nominations for external examinersand for major amendments to validatedprogrammes. At the time of the audit visit, sixpartners had delegated authority for validationand revalidation, and two partners haddelegated authority for the scrutiny of externalexaminer nominations.

63 Accreditation of an institution is formallyre-approved through institutional review. ValCdiscusses and endorses the outcomes ofinstitutional reviews. The audit team found thata number of the discussions at ValC recordedongoing concerns with aspects of the

Open University

page 14

management of quality at partner institutions,and it was clear to the team that the Universitywas now being more forceful in its response tosuch shortcomings with, for example, reviewsresulting in re-approval of accreditation forperiods of less than five years.

64 The CPSED stated that, in revising theHandbook, the institutional review process hadbeen significantly changed to enable theUniversity 'to evaluate more explicitly theeffective discharge of key quality assurancefunctions' by its accredited partners, and thatreviews take place at intervals of now not morethan five years. The University has also nowmade it clear that removal of delegatedauthority, or even of accredited status, can bean outcome of review, and that the 'formaloutcome of successful institutional review is re-accreditation and confirmation that delegatedauthority is being exercised effectively'. TheCPSED explained that a rolling programme ofinstitutional reviews under the revisedprocedures will provide the University with'more robust evidence and assurance that thesignificant demands of accreditation are beingmet', and will 'bring all accredited institutions,new and long-standing, to a clearunderstanding of the expectations ofaccreditation' which it believes to be necessary.

65 The constitution of accreditation andinstitutional review panels was a matter ofconcern to the audit team. The team noted thatduring the previous two years, one panel forinstitutional accreditation consisted of only threemembers; three panels for institutional reviewcomprised only three members and one panelhad only two members. In considering theappropriateness of the composition of panels tomake judgements on behalf of the University,the team noted panels with no current membersof the University, panels that drew heavily onstaff of accredited institutions, and an instance ofa panel having majority representation frominstitutions who are themselves accreditedpartners. While the team recognised thedevelopmental value of engaging members ofaccredited partner institutions in the processes ofaccreditation and review, it questioned the

soundness of such significant deployment ofmembers of institutions who are themselvesaccredited by the University in panels that aremaking accreditation judgements on behalf ofthe University.

66 The extent to which the wider academiccommunity within the University is willing to beinvolved in accreditation and validation wasunclear to the team. ValC noted in 2002 thatthe proposal for the 'vast majority of OUVSevents to be chaired by an OU colleague' didnot receive its full support. Concern wasexpressed at ValC about the 'lack oftransparency about how panels are convened',and it was clear from the minutes of meetingsthat other suggestions for obtaining greaterownership by academic staff of the work ofOUVS were not progressed. Overall, the teamquestioned whether the composition ofapproval and review panels was consistentlyappropriate for making the judgement, onbehalf of the University, to approve or re-approve institutional accreditation.

67 The CPSED acknowledged that a limitationof the University's model of accreditationrelated to institutions seeking 'closer academicpartnership', and the audit team heard fromrepresentatives of accredited partners that theywould look to other validating partners wherethey wanted greater academic and subject levelsupport than currently provided by theUniversity's accreditation model. The teamnoted that the 2003 QAA report on theUniversity's partnership with KoldingKøbmandsskole expressed concerns about thelimited discipline-level contact between thepartner and a relevant academic unit within theUniversity, and recommended consideration of'a greater level of subject-based liaison'. TheUniversity specifically rejected this, reporting toValC in October 2004 that 'there is no intentionto increase academic oversight by subjectexperts at the OU…as this is inconsistent withaccreditation'. The team also noted that matterof subject level engagement was raised byOUVS in its Unit Plan, April 2004 where it statesthat the OUVS is 'severely impeded' in its abilityto respond to demands for the validation of


page 15

Foundation Degrees by 'assumptions…that bestpractice in higher education partnerships withfurther education is characterised by very closeacademic partnership and close qualityassurance oversight by the awarding institution'and that its work is likely 'soon to bemarginalised'. OUVS observes that 'it can onlymake a strong contribution if the Universitydecides to move' towards 'a different approachto working with the further education sector'.

Programme approval68 Programmes offered through anaccredited institution that lead to an award ofthe University are approved through validation.Validation is defined in the CPSED as theprocess 'by which an accredited institutionproposes a programme of study for approval',and culminates in a 'judgement by a panel ofpeers about whether or not a proposedprogramme of study for a validated awardmeets the requirements of that award'. TheHandbook provides detailed information on theprocess to both accredited institutions and tomembers of validation panels. It explains that avalidation panel 'makes decisions on behalf ofValidating Committee, which are reported tothe Committee at its next meeting', and 'willtherefore include advisers, external to OUVSand the institution'. There is no requirement fora panel to include a member of academic staffof the University. Partnership managersassemble panels drawing on their database ofcontacts and, although they try to involve amember of the University's academic staff, theaudit team was told that the links with facultiesof the University were limited.

69 The decisions of validation panels arereported to ValC. OUVS arranges for a formalletter of approval to be issued. Partnershipmanagers are responsible for ensuring that anyconditions for approval have been met, and willseek the guidance of the validation panelbefore conditions are signed off.Recommendations are followed up as part ofannual monitoring. Programme validations andrevalidations are reported to ValC. The auditteam was told that ValC would pick up mattersarising from validation reports but did not seek

a systematic overview of such matters. Theteam studied minutes of meetings of ValC, andnoted that the Committee had identified thedifficulty of evaluating an institution's progresswith recommendations in annual monitoring,and its reliance on the partnership managers toidentify issues that should be followed up. Theteam considered that a process of formalreporting to ValC that an institution had met allrecommendations was lacking.

Annual monitoring of validated programmes70 The University's annual monitoringrequirements of validated programmes consistsof an institutional commentary, an executivesummary, the annual programme monitoringreport (AMR) pro formas and copies of theexternal examiners' reports and the responsesprovided to them. This information is reviewedby the partnership manager who completes apro forma summarising key points andidentifying any matters from the previous yearwhich have not been addressed. The subgroupof ValC undertaking the scrutiny of annualmonitoring reports considers this informationfrom all partners during a dedicated whole-dayevent. This outcome of this event is thefinalisation of the individual partnershipmanagers' summary pro formas and an overallsummary of themes and observations that issubmitted to ValC and sent, together with theindividual pro formas, to accredited partners.This process was operated for the first time in2003. Prior to that only a selection of annualmonitoring reports was scrutinised by ValC.

71 The cycle for programmes operating tothe standard academic year is for annualmonitoring to have passed through a partners'own quality assurance procedures in time forthe report pro forma, as required by theUniversity, to be submitted to it for 15December. Reports are then reviewed by thepartnership manager in OUVS and consideredby the scrutiny subgroup of ValC in February.The report of that group is considered by ValCin March, after which individual institutionsreceive their partnership managers' pro formasand feedback. It was reported to the audit teamthat this feedback was inevitably quite long in

Open University

page 16

coming, thus limiting the ability of partners totake timely action. The University explainedthat any changes to annual monitoring policyfor the coming cycle are usually notified inAugust. This timing of notification of proceduralchange was reported by partners who met theteam to be a source of frustration to them, andto hinder their ability to operate the qualityassurance requirements of accreditation.

72 The 2004 overview commentary resultingfrom scrutiny by the subgroup of ValC notedimportant sections missing in a number ofreports, variability in quality, limited reportingon student feedback with limited criticalreflection and resultant plans. Nevertheless, thesubgroup felt able to express 'overall confidencethat standards were upheld'. The 2005 overviewcommentary noted that 'all reports met thestandards required' and 'showed animprovement' over the previous year. Even so,some reports were noted as still having sectionsmissing, with variability in the reporting ofstudent feedback, wide variation in the statisticalsections and overall variation within institutionson a programme-by-programme basis. It wasnoted that the pro forma 'did not at present askinstitutions to set out how it had responded topoints raised in the previous review…toensure...the quality loop…could be completed'.The commentary again draws attention to lateand inadequate external examiners' reports and,while the University was 'urged to continue tosupport partner institutions' in obtainingsatisfactory reports, it is clear that the problem isseen to be primarily that of the partner and notof the University. The subgroup, however, againconfirmed 'overall confidence that standardswere upheld'. While the audit team was told thatpartnership managers are responsible foridentifying and chasing actions not taken, and,reportedly, 'just keep chasing', there is no formalreporting of the completion of recommended oroutstanding action to ValC, or a mechanismthrough which the University maintains oversightof the progress of actions and the closure ofquality loops. It appeared to the team thatbecause accreditation delegates such substantialresponsibility for quality assurance to partners,the University has limited sanctions available to it

beyond recourse to the triggering of arevalidation or institutional review.

73 The CPSED recorded that 'the generalstandard of annual monitoring by accreditedinstitutions, the organisation of thearrangements for their scrutiny, and theeffectiveness of Validating Committeeinvolvement have all improved in 2003-04'. Theview was expressed to the audit team by seniorstaff of the University that the University hadcome to accept that the nature of the evidencethat it was relying upon was not sufficientlyrobust and secure, and that it had, therefore,moved to make it more robust.

74 The CPSED stated that accreditation'delegates to accredited institutions the qualityassurance associated with monitoringprogrammes of study', and that the role of theUniversity is to ensure that partners 'arecarrying out these activities with positiveresults, and to identify from their reports anygood practice or quality issues which mighthave wider relevance…to accredited partners'.The CPSED acknowledged that its review ofannual monitoring in 2001-02 concluded thatthe 'activity was not fully effective'. In duecourse the University produced the revisedHandbook. While the audit team consideredthat the changes introduced in this revisionwere helpful, it questioned the effectiveness ofthe action taken by the University on someserious shortcomings it had identified,shortcomings which were reiterated in 2002 bya QAA overseas audit of the University'spartnership link with an accredited partner. TheUniversity informed the team that 'the annualmonitoring process was revised immediately aspart of the action on the QAA overseas audit',and 'a letter and guidance material was sent toall accredited institutions advising them of thenew process and requirements'. Nevertheless,the team noted the slow response to action bysome accredited partners, and the apparentinability of the University to ensure urgentaction by partners in such cases. For example,the team noted many instances in partnershipmanagers' reports on annual monitoring whereissues were recorded as having been raised in


page 17

previous annual reports, and instances where itwas observed that no action has been reportedin the programme pro forma or institutionalsummary about serious issues. Theidentification of the same matters in successiveannual reports does not appear to result inaction by the University, and failure to takeaction does not appear to have consequencesfor the partner. Whether this reflects adislocation between the actual monitoringbeing undertaken by a partner and therecording of it on the University's pro forma ora failure by the partner to monitor, it isnevertheless unsatisfactory, and does notreliably enable the University to assure itselfthat academic standards are being maintainedand that the quality of students' learningopportunities continue to meet its expectationsfor the validated programme.

75 In 2004, CAB received for the first time areport combining commentary on the annualmonitoring of accredited partnerships and thatcarried out for direct and collaborative teachingpartnerships. This report describes processesand comments on the organisation of them,thus confirming administrative process andprocedural matters. It does not, however,attempt to draw out themes or to comment onthe overall effectiveness of the processes inenabling the University to assure itself aboutthe academic standards and the quality of thelearning opportunities in its collaborativeprovision. The audit team considered thecombined report to be an interestingdevelopment that has yet to find its fullpotential for allowing the University to take anannual overview of the health of itscollaborative activity as a whole.

Periodic programme review76 Periodic review of programmes is theprocess of revalidation by which programmesoffered by accredited partners are re-approved.The process is structured to review the designand delivery of the programme, and involvesthe production of a critical reflection and revisedfull programme documentation, and culminatesin a visit and a judgment by a panel of externalpeers. Re-approval may be for the full term of

five years or for a lesser period, and may be withor without conditions and recommendations. Inexceptional cases approval may be withdrawn.A recently revised process, which is beingpiloted with a number of partners, placesgreater emphasis on development over timerather than on a single event.

77 In initial approval of a validatedprogramme, the University is able to withholdapproval until conditions are met. Once avalidated programme is established, however, itis less easy to suspend registration of newstudents, and it was clear to the audit teamthat conditions for re-approval have not alwaysmet in a timely fashion. The team's study of theminutes of ValC showed that failure to meet re-approval conditions is not routinely reported tothat Committee.

Summary78 In its CPSED the University expressed itsconfidence that the accreditation process wasrobust and rigorous, and provided it with thenecessary information to be confident in itsaccredited partners' ability to meet and sustainthe requirements of accreditation. This viewwas reiterated to the audit team in meetingswith representatives of the University and ofaccredited partners during the visits. TheUniversity has, nonetheless, sought tostrengthen its procedures and to make a seriesof changes, some of which were yet to beimplemented or still under development at thetime of the audit visit. In so doing, theUniversity has gone some way towardsrecognising that partners are likely to require ahigher level of monitoring and supportfollowing accreditation than has hitherto beenthe case. The team considered that this rancounter to the stated principle of accreditationas a process which confirms a partner's abilityfully to meet the requirements of accreditationand the substantial level of delegated authoritythat accreditation carries. The team formed theview that the University's model ofaccreditation has not enabled the Universityreliably to ensure that its partners can sustainthe requirements of accreditation. The teamrecognised that the University has made a series

Open University

page 18

of changes so as to increase the level ofinformation it receives to enable it to monitorits partnerships more actively and effectivelythan once it did, but, in raising the extent ofpost-accreditation monitoring, is in effectmoving away from the underlying principlethat its model of accreditation provides it withthe necessary information to be confident in itsaccredited partners' ability to meet and sustainthe requirements of accreditation.

79 In illustrating the move toward 'a closeroverview and understanding by the University ofassessment practices in accredited institutions,the CPSED noted that the University is'managing two cases of accredited institutionsoutside the UK context, where the gap betweenour expectations and their practice has raisedconcern'. While it is important that such gapsare identified, their emergence raises doubtabout the robustness of the initial accreditationjudgement and the confidence placed in thatjudgment by the University. These casessuggested to the audit team that either therenever was a shared understanding of theexpectations of the University by the partners -in which case accreditation should not havebeen approved - or that gaps between theUniversity's expectations and a partner's practicecan develop after the criteria for accreditationhave been fully satisfied.

80 Notwithstanding the procedural changesthat have been made to the University's modelof accreditation since its establishment in 1992,there appeared to the audit team to be a tensionbetween the underlying principle of the modeland the reference in the CPSED to 'evidencefrom our own processes that different kinds ofaccredited partners require different kinds ofsupport and oversight'. The team also consideredthere to be some tension between delegated'authority for taking action to tackle issues raisedthrough' monitoring, given by the University toits accredited partners, and the information thatthe University legitimately requires to assure itselfabout quality of provision and academicstandards of awards made in its name. The teamwould encourage the University to consider if it,too, sees these tensions.

81 Accreditation is built upon peer review.While the engagement of independent externalexperts in matters of authority or delegatedauthority to assure quality and academicstandards is a cornerstone of UK highereducation, the audit team was not persuadedthat the way in which peer review is beingutilised in the University's approach toaccreditation is fully appropriate. Consideringthe significance of the decisions made by anaccreditation panel on behalf of the University,it is critically important that panels represent anappropriate level of current and independentexperience of UK higher education, and have agood understanding of the way in which thebroader higher education agenda is developingand safeguarding the quality and academicstandards of higher education provision in theUK. The team considered that the University'sown academic staff are under-represented onpanels, many of which drew heavily on staff ofaccredited institutions for their membership.The University may wish to reflect upon themerit of making a distinction between theengagement of members of partner institutionsin a developmental role in the accreditationprocess and their engagement on accreditationpanels as independent external experts andadvisers to the University.

82 The audit team acknowledged the extent towhich the University had sought to rectify theserious shortcomings in accreditation andvalidation that had been identified by theUniversity itself and by external audit and reviewevents. It saw the evidence that the University hasbegun to be more robust in its response toconcerns arising from accreditation and validationprocedures, and is exercising its right to initiateearly review and suspend delegated authoritywhen necessary. The team formed the view,however, that in giving a very high level ofdelegated authority to its partners, the Universityhas not always been able to discharge its ownresponsibilities for securing the academicstandards of its awards and for assuring thequality of the learning opportunities available tostudents through validated programmesdelivered by its accredited partners. The report ofthe QAA review of the partnership with Kolding


page 19

Købmandsskole noted strong reservations aboutmonitoring and oversight of the provision by theUniversity, and made a judgement of 'only alimited confidence in the University's stewardshipof the quality and standards of its validatedawards'. The current audit team was notpersuaded that changes made by the Universitysince that judgement are sufficient to justify ahigher level of confidence.

External participation in internalreview processes for collaborativeprovision

83 In direct and collaborative teachingarrangements an external member of CuPC isinvolved in both partner approval and review,while partner and programme approval andreview with accredited partners are predicatedon peer review. The use of external input isextensive and a fundamental part of theapproach of the University to accreditation. TheUniversity is prepared to validate programmesin academic areas it does not itself provide and,in such cases, relies substantially upon externalpeers to provide the subject expertise tounderpin the validation decision. Externalmembers of panels are drawn from current andformer members of UK HEIs and fromaccredited partners of the University. The auditteam was not persuaded, however, that allexternal members of panels were fully 'external'or sufficiently independent in view of theirprevious involvement with the University. Theteam's view of the use made by the Universityof external participation in its approval,monitoring and review processes in themanagement of its collaborative activity hasbeen set out in paragraphs 54 to 82 above.

External examiners and their reportsin collaborative provision


84 For courses within the University'scurriculum delivered by direct and collaborativeteaching, the role of external examiners isdefined in the document, The Role of ExternalExaminers at the Open University, and is the

same as for the courses delivered to studentswho are studying for the University's awardsdelivered through the supported open learningmode. The external examining processes forcourses delivered through direct andcollaborative teaching partnerships are thesame as those delivered by open supportedlearning. The report of QAA's 2004 institutionalaudit found these to be effective. The currentaudit team found from its examination ofdocumentation relating to a sample of directand collaborative teaching partnerships that theexternal examining process was operating inaccordance with the processes laid down, andwas able to endorse the finding of the 2004audit that the process was effective.

85 Where courses are delivered in translation,a local external examiner is appointed whoundertakes duties on behalf of the Universityand advises the UK Examinations andAssessment Board (EAB). The role of the localexternal examiner is laid down in the OUWOperations Handbook which is customised foreach partner. Their duties include the reviewingof borderline pass/fail scripts and reading andconfirming to the UK EAB that the translatedcourse materials and assessments are meetingthe same standards as in the UK. Local externalexaminers are nominated by the partnerinstitution and approved by the relevant schoolboard of the University and by the PVC(Curriculum and Awards). The nominee shouldhave academic expertise and experienceappropriate to the level of the course beingexamined and be appropriately fluentlybilingual. While the University's externalexaminers are required to have appropriateexpertise and experience to maintain academicstandards in the context of UK highereducation, local external examiners' experienceof UK higher education can be satisfied solelythrough the possession of a UK awardcompleted overseas; they are not necessarilyrequired to have experience of delivering andassessing in UK higher education. The auditteam also noted that, in some cases, localexternal examiners might not be appropriatelyindependent, as it found that some localexaminers had been employed by the University

Open University

page 20

as associate lecturers. While the externalexamining process for direct and collaborativeteaching partnerships is generally robust, theUniversity is advised to ensure that all localexternal examiners appointed for coursesoffered overseas are appropriately independentand have experience of undertaking the deliveryand assessment of UK higher education.


86 In the case of accreditation and validation,the accredited partner nominates the externalexaminers for the validated programmes whomust meet the rigorous criteria set out in theHandbook. The University approves theaccredited partner's nominees after seeking theadvice of three expert referees, and thenominees are then appointed by the academicboard of the partner institution. The audit teamfound evidence in a limited number of cases ofaccredited institutions' external examinernominations being rejected by the University,and formed the view that the process by whichthe University approved the appointment ofexternal examiners for validated programmeswas rigorous. Where an accredited institution hasfurther delegated powers for the appointment ofexternal examiners, the Handbook stipulates that'nominations are then sent to three externalappraisers who have been agreed with OUVS.The majority agreement of the three appraisers isrequired, following which the OUVS is informedin writing…and OUVS then approves thenominees on behalf of the University'.

87 It was made clear to the audit team bysenior staff of both the University and ofaccredited institutions that the externalexaminers are seen as operating on behalf of theaccredited institution rather than on behalf ofthe University. The 2004 version of theHandbook specifies that external examiners arerequired to 'make a report annually to theAcademic Board of the accredited institution, tobe copied simultaneously to the Open University,on the conduct of the assessments justconcluded and on the standards being set andachieved'. It also stipulates that externalexaminers of validated programmes should

report to the University on any matters of seriousconcern arising from the assessments which putat risk the standards of the University's award.

88 The CPSED explained that accreditedinstitutions were expected to brief externalexaminers on their role and responsibilities assoon as possible after taking office, preferablyby visiting the accredited institution to meetstaff and students. The Handbook definesclearly the areas on which the externalexaminer must be briefed by accredited partnerinstitutions. The CPSED noted that following anevent for representatives from accreditedinstitutions on external examining, OUVS wasconsidering offering a generic inductionprogramme and briefing materials. TheUniversity is moving toward explicitlyapproving the briefing materials of accreditedpartner institutions for external examiners,noting that for new partners this would takeplace during the early period of accreditationand for existing partners at institutional review.

89 External examiners' reports are expectedto conform to a template whose format andcontent is defined in the Handbook. Thetemplate requires examiners to comment onthe structure, organisation, design, markingand fairness of all assessments. Reports areaddressed to the academic board of theaccredited institution and, since 2001, also tothe Vice-Chancellor of the University.Partnership managers read all externalexaminers' reports within their portfolio. TheHandbook specifies that accredited institutionsare expected to ensure that reports of externalexaminers are received, are formally considered,that appropriate action is taken, and thatexternal examiners receive a response settingout action taken following receipt of reports.

90 External examiners' reports are consideredby the accredited institution's academic boardas part of its annual monitoring procedures toensure that the validated programme meets itsstated objectives. Annual monitoring starts atthe course report level, which includes asummary of the issues in the externalexaminers' report and the programme team'sresponses. Partner institutions are required to


page 21

produce an AMR for the University whichincludes, as appendices, the external examiners'reports together with the accredited partner'sresponses. The partnership manager produces apro forma response to the AMR, and both theAMR and the pro forma response arescrutinised by the subgroup of ValC to ensurethat issues raised by external examiners havebeen resolved. Where serious issues are raisedby the report of an external examiner, thepartnership manager is expected immediatelyto engage in discussion with the partnerinstitution as to how the issue is being resolved.

91 The June 2003 report of QAA's audit of theUniversity's partnership link with KoldingKøbmandsskole concluded that 'whilerecognising that the University had recentlytaken steps to strengthen the external examiner'srole at the College, the [audit] team could notbe confident that the examiners' work hadfacilitated a careful monitoring of standards overthe period of the validation thus far'. The CPSEDexplained that the University had taken actionon many areas identified in the report, but itrobustly defended its accreditation model thatassumes a direct relationship between theexternal examiner and the accredited institution.The June 2004 report of QAA's enhancement-ledinstitutional review (ELIR) of an accreditedpartner also identified a number of weaknessesrelating to the external examining process,including the way in which course teamsrespond to serious issues raised by externalexaminers, the need to provide a more reliableappraisal of external examiner reports, and theneed to ensure that the college receivessatisfactory and timely external examiner reports.

92 The University's annual monitoring ofvalidated programmes reviews externalexaminers' reports and the accreditedinstitution's responses on an individual basis,and the subgroup of ValC comments on generalissues which emerge from the review. In theannual monitoring for 2003-04 the subgroup'sreport noted that a limited number of externalexaminers' reports had not been received intime to be included in the annual review, andthat for inadequate reports the University wouldsupport the partner in requesting that the

report be resubmitted. The University does notproduce an overview of the external examiners'reports for validated programmes.

93 The audit team considered at lengthwhether the use of external examiners and theirreports for validated programmes wassufficiently strong and scrupulous and reflectedthe precepts of the Code of practice, publishedby QAA, the team's concern being whether theUniversity had retained sufficient responsibilityfor the functions of external examiners ofvalidated programmes by delegating much ofthe responsibility to the accredited institutions.The extent of the delegation was confirmed bythe University and the partner institutionsvisited by the team who all stated that externalexaminers of validated programmes areexaminers of the accredited partner, not of theUniversity. The team noted that the Universityhad strengthened a number of processes in itsframework for the management of academicstandards in the recent past to allow it to havemore control of the external examining processfor validated programmes. The team foundevidence that immediate action had been takenby the University to ensure the properresolution of a significant issue relating tostandards which was raised by externals at anexamination board and in their subsequentreports, and was able to conclude that in aserious situation, where standards are perceivedas being at risk, the University has the power torespond immediately to the concerns ofexternal examiners. On balance, the teamconcluded that the University had strengthenedits link with external examiners of validatedprogrammes sufficiently to retain the ultimateresponsibility for their functions, and thatexternal examiners and their reportscontributed sufficiently to the University'scapacity to be confident of the security of theacademic standard of its awards achievedthrough validated programmes. The Universityis strongly encouraged, however, to considerhow it might further strengthen its direct linkswith external examiners, for example, byensuring that all external examiners' reports areproduced in a timely manner and to anappropriate standard, and by taking an

Open University

page 22

overview of external examiners' reports toidentify common issues and to assist the workof the scrutiny subgroup of ValC.

The use made of external referencepoints in collaborative provision

94 The Handbook makes clear that accreditedpartners are expected to demonstrateknowledge and understanding of the AcademicInfrastructure in their institutional qualityassurance arrangements, programmesubmissions and delivery. The University'saccreditation process requires partnerinstitutions to demonstrate that they arebenchmarking their quality assurancearrangements to accepted good practice in UKhigher education, including the AcademicInfrastructure. The audit team found that somepartner institutions took responsibility forengagement with the Academic Infrastructurefor themselves while others looked to theUniversity to provide guidance. Workshops andseminars have been offered for partners onaspects of the Academic Infrastructure.

95 The University has mapped its institutionalpolicies and procedures against each section ofthe Code of practice, published by QAA, andstatements about the alignment demonstratedby this mapping have been reported to QSB.The CPSED explained that, while the Code isused 'to inform institutional processes for theapproval and review of collaborative provision',the precepts of the Code are embedded inUniversity policies and processes, and are notnecessarily drawn to the attention ofinstitutions in direct and collaborative teachingpartnership arrangements. All accreditedinstitutions were required to submit a mappingof their own policies and procedures againstthe Code by December 2004. At the time of theaudit visit the analysis of the outcomes of thismapping by partner institutions had not beencompleted, but the team was informed that theoutcomes would be reviewed in detail throughinstitutional reviews over the next few years.

96 The University has its own levels and awardsframework that sets out the various awards of theUniversity, from undergraduate certificates

through to higher doctorates, including singlecredit-bearing courses. The 2004 institutionalaudit of the University verified that University'sframework is consistent with The framework forhigher education qualifications in England, Walesand Northern Ireland (FHEQ), published by QAA,and the Credit Guidelines for HE qualificationsjointly prepared by four regional creditassociations and consortia. Accredited institutionspropose programmes for validation for an awardfrom the University's framework and, whereappropriate, ValC considers proposals for newawards and makes recommendations to CAB fortheir entry onto the framework.

97 Programme specifications became a formalrequirement for validation and revalidation from2003. The University's programme specificationtemplate, which requires reference to subjectbenchmark statements, must be used byaccredited institutions when submittingprogrammes for validation. While programmespecifications have been required for new orreviewed programmes since 2003, theUniversity has not made it a requirement forprogramme specifications to be implementedbefore the programme is reviewed. At the timeof the audit visit a sample check by OUVSindicated that nine out of 250 programmespecifications had not yet been produced, andthe last one was not due to be completed untilthe programme review in the academic year2006-07. The Handbook defines the contents ofprogramme specifications in some detail, and anexemplar is provided for guidance. TheHandbook states that the programmespecification should be included in theprogramme handbook, but it must be capableof being separately presented to readers who donot receive the programme handbook. Externalexaminers are required to comment in theirreports on the quality of knowledge and skillsdemonstrated by students in the light of therelevant subject benchmark statement, and ofthe FHEQ (or the Scottish Credit andQualifications Framework where appropriate).

98 The audit team's discussions with the staffof the accredited partner institutions that itvisited indicated a high awareness of theAcademic Infrastructure and a willingness to


page 23

ensure that proper account is taken of itsrequirements and guidance. OUVS hasrecognised that small institutions, those with asmall proportion of their work in highereducation, and overseas institutions, havespecial problems in using the AcademicInfrastructure in an effective way, and has madesignificant efforts to provide support in suchcases. The University is reviewing the outcomesof a mapping exercise that accreditedinstitutions carried out against the Code ofpractice, published by QAA. The team wouldencourage the University to review partners'engagement with the Academic Infrastructureon a more regular basis, perhaps as part of theannual monitoring process. Overall, the teamwas able to conclude that the AcademicInfrastructure is being properly addressed bythe University and its accredited institutions.

Review and accreditation by externalagencies of programmes leading to theawarding institution's awards offeredthrough collaborative provision

99 A wide range of the University'scollaborative provision has been subject to reviewby QAA, including the quality audit of overseascollaborative links, reviews at subject level and ofFoundation Degree provision, and the ELIR of anaccredited partner. The audit team was told thatthe University considers institutional-level reportson its collaborative partners, and seeks tomonitor its accredited partner's response to QAAthrough annual monitoring at subject level; butthe team found the evidence for this was sparse.It was explained that it is important for OUVS toreceive copies of action plans arising from subjectreviews of validated provision and of accreditedpartners reporting on all their engagements withQAA. This seemed to the team to be a matter ofrecord-keeping rather than of active oversightand monitoring by the University of partner'saction plans resulting from these external reviews.The University does not seek to take an overviewof themes and issues emerging from QAAengagements with its accredited partners,arguing that there is no benefit to be gainedfrom looking across accredited provision aspartners are all so different.

100 At institutional level, QAA reports inrespect of accredited partners of the Universityhave included judgements of limited confidencein academic standards, in one case in respect of'the University's stewardship of the quality andstandards of its validated awards', and inanother in respect of the partner institution's'procedures for the present and likely futuremanagement of the academic standards of itsprovision'. In the former case, the response bythe University to the recommendations of thereport was considered in detail by ValC and byQSB, CAB and the Academic Board, with ValCnoting from its discussions that 'the key issueemerging from the report is the perceivedinsufficient evidence that OUVS requires' fromits accredited partners. The OUVS Unit Plan for2004-05 noted that the report had given 'addedimpetus to reviews and changes that werealready in train'. The latter case was, however,only 'noted' by ValC. The audit team was toldthat the response was made at the level ofOUVS and, indeed, found no response to theimplications of the recommendations of thereport in the proceedings of university-levelcommittees, or any comment made upon theaction plan prepared by the accredited partnerinstitution. The University explained that incases of QAA's direct engagement withaccredited institutions, ValC has only 'noted' theinstitutions' action plans because although theUniversity keeps an overview on such mattersthe accreditation model places the responsibilityfor action on the accredited institution. Theteam was told that the Academic Board knowswhen serious issues arise from externalengagements, and it appears that this case didnot fall into that category, despite earlierevidence of shortcomings at subject level. Theteam noted that, in several cases, concerns thatformed the basis of 'limited confidence' or 'noconfidence' judgements in QAA reports hadbeen identified by the University in earlierannual monitoring reports, but the Universityappeared to have no effective mechanisms toenable it to ensure that timely remedial actionwas taken by partners to address or rectifyidentified shortcomings. In one case of ajudgement of limited confidence the report

Open University

page 24

identified repeated concerns, and observed that'the follow-up mechanisms in place (for theValidating Committee and OUVS) to addressany action planning for improvement were lessthan effective'. The current audit team wouldconcur with this observation.

101 At subject level, QAA reports have includedjudgements of 'no confidence' in validatedprogrammes in Law (March 2002) and inBuilding and Surveying (December 2003) and'limited confidence' in Sociology andAnthropology (January 2004). The qualitymanagement and enhancement (QME) aspect ofa subject review reports has been graded as'significant improvement could be made' invalidated programmes for Nursing (May 2000)and Biosciences and Subjects Allied to Medicine(May 2000). A number of subject review reportsof validated programmes identify shortcomingsin the provision of learning resources, staffingissues and other concerns likely to impact uponthe quality of the learning opportunities availableto students. The University emphasised to theaudit team that the majority of external reviewsat subject level have resulted in positivejudgements, pointing out that, in one accreditedpartner, one of the 'no confidence' and one ofthe 'limited confidence' judgements noted aboveshould be seen in the context of eight'confidence' judgements in the same period.While recognising that the majority of externalreviews of validated programmes have hadpositive outcomes, the team considered thatthe evidence of negative outcomes indicatedthat the University's process of accreditationdoes not reliably enable the University to assureitself that its partner institutions are able todeliver the level of responsibility delegated tothem. It also indicated to the team that theprocesses of validation, revalidation and annualmonitoring do not reliably enable the Universityto assure itself that programmes continue to beof an appropriate academic standard aftervalidation, or that the quality of the learningopportunities offered by validated programmesis reliably maintained.

102 The CPSED provided only limitedinformation on how the University engageswith the outcomes of PSRB accreditation and

review. It noted that some PSRB activity takesplace, and that 'account is taken of theseactivities in subsequent OUVS engagementswith the accredited institution'. The audit teamwas informed that the outcomes of PSRBactivity with accredited partners was notreviewed by the University, and that theUniversity did not seek to draw any lessonsfrom, or take an overview of, suchengagements by its partners.

103 The University does not maintain asystematic overview of the outcomes of reviewand accreditation of validated programmes byexternal agencies. It does not analyse or takeaction on the reports of such events in a waythat suggests that it takes responsibility forresponding to the critical comments made byexternal agencies that bear upon its stewardshipof the academic standards of its awards madethrough validated programmes, or on itsultimate responsibility for assuring the quality oflearning opportunities offered by thoseprogrammes. The audit team concluded that thelevel of engagement of the University withreview and accreditation by external agencies ofits validated programmes did not give theUniversity the capacity to ensure that suchreviews would make a positive contribution to itsmanagement of the quality and academicstandards of its validated provision.

Student representation incollaborative provision

104 The University's Student Charter applies toall students on direct and collaborative teachingarrangements but not to students on validatedprogrammes of study in accredited institutions.There is, however, an expectation expressed inthe Handbook that accredited institutions havearrangements in place to ensure that studentshave an opportunity to comment on theirexperience, and that accredited partners have aculture that ensures that students will be listenedto and given an appropriate response. TheUniversity allows for variation in representativearrangements in both its direct and collaborativeteaching partnerships and accredited institutionsin order to accommodate local practices andprocedures. Arrangements for representation in


page 25

direct and collaborative teaching partnerships arespecified and agreed at the time the partnershipis set up. In the case of accredited institutions,the arrangements for representation is consideredinitially in the accreditation process. TheHandbook states that an institutional reviewpanel will normally meet student representativesto discuss, inter alia, 'student representationwithin the committee structures andopportunities to feedback to staff'. The teamnoted with interest that one accredited institutionhad invited its students to make a writtensubmission to its institutional review panel.

105 Complaints and appeals are initially dealtwith by the relevant accredited partner orcurriculum partner, with students onlyaccessing the University's procedures when thepartner's procedures have been exhausted. TheUniversity has acknowledged that accreditedinstitutions are dealing with an increasingnumber of complaints, and that there is a needto provide them with guidance on this matter.Students are informed of their rights in thisrespect through their accredited institution'sprogramme handbook, and through the OUVSStudent Handbook (or the OUW variant) that isprovided to all students.

106 The CPSED did not take a view on theeffectiveness of the arrangements for managingstudent representation, but the view generallyexpressed by University staff and staff of partnerinstitutions who met the audit which team wasthat this was a problematic area had not beenfully resolved by previous efforts and initiatives.Students who met the team expressed generalsatisfaction with the way they were consulted bytheir accredited institutions, and with theirinstitutions' responses to this consultation.Generally, however, students had little knowledgeas to how their views were made known beyondtheir accredited institution to the University, or ofany University response to these views.

107 The University's policy of allowing for localvariation is an appropriate acknowledgementthat different partners can secure appropriatearrangements for student representation indifferent ways in accordance with localpractices and traditions. The audit team

considered, however, from its study of theavailable documentation, that the extent towhich the suitability of these arrangements isexamined during the accreditation, annualmonitoring and institutional review wasvariable. While there are several examples ofgood practice by partner institutions in studentrepresentation, the University does not have theinformation to allow it to be sure that there areadequate arrangements for studentrepresentation in all cases. The University mightconsider the merit of giving more specificguidance on its expectations for studentrepresentation to support consistency in theapproach taken by panels and partnershipmanagers in approval and review processes.

Feedback from students, graduatesand employers

108 The arrangements for gathering and usingfeedback from students studying through directand collaborative teaching partnerships areagreed at the time the partnership is set up, withthe local partner in some instances beingresponsible for gathering and responding tofeedback. These arrangements, however, are notdefined in either the formal agreements oroperations handbooks. The CPSED explainedthat the mechanism for feedback may be astandard survey, delivered and analysed by theUniversity's Student Research Centre, or may beinitiated and analysed by the partner institution,particularly where students are taught andassessed in a language other than English. Ineither case, the findings are referred back to therelevant academic unit of the University forincorporation in normal review processes. TheCPSED noted that in 2003, for the first time, theannual review of curriculum partnershipsincluded a specific question on the use ofstudent feedback, which 'enabled us to gain abetter oversight at institutional level of the rangeof practice in place and the key findings'.

109 In respect of accredited institutions, theHandbook makes clear that the University expectsthese institutions to be responsible for gathering,analysing and responding to student feedback,and in order to become accredited a partner

Open University

page 26

institution must be able to demonstrate that ithas an appropriate system for carrying this out.The details of these arrangements vary from oneaccredited partner to another, accommodatingdifferences in local practices and processes, as aresult of which there is a wide range ofinterpretations by partner institutions of themeaning and nature of student feedback; thelevel and coverage of its analysis and reflection,and the ways in which it is used to bring aboutimprovements in the provision. The Universitydoes not define a minimum expectation. Asuggested student feedback questionnaire wascirculated to accredited institutions some timeago, but the audit team was given to understandthat its use was variable. The use made byaccredited institutions of their feedbackarrangements is monitored through the annualmonitoring process.

110 The CPSED made little reference tofeedback from graduates and employers, otherthan to point out that an accreditation panel'would expect to see evidence including theaccount taken of teacher, student, graduate,employer, external examiner and external peerfeedback' to support a claim that a partnerinstitution has in place systems for regularmonitoring and review. The audit teamrecognised that many of the students engagedin collaborative arrangements would bestudying part-time, and that informal feedbackcould be expected to come from theiremployers. The University might, nevertheless,wish to reflect on how more formal feedbackfrom employers, and feedback from graduatedstudents on the extent to which theirexperience of might have impacted upon theiremployment and employability, could be usedmore effectively to enhance the provision.

111 The audit team discussed aspects ofstudent feedback with staff and students of theaccredited institutions that it visited. All of theseinstitutions had mechanisms in place forgathering student feedback. The students whomet the team were generally satisfied with theways in which they were invited to give feedbackand with the mechanisms for taking action on

their feedback. In each case, the feedback andresponse cycle was contained within theaccredited institution, and engaged with theUniversity only at the point of reportingoutcomes into the annual monitoring process.

112 The CPSED noted that the reporting of theoutcomes from student feedback into annualmonitoring 'has tended to be weak', and overthe past two years the University has 'includeda focus on this area of reporting' in its feedbackto accredited institutions. The effective use ofstudent feedback in the management of thequality of provision has been an area ofconcern to the University for some time, anddespite attempts to improve the position,including repeated encouragement bypartnership managers in the commentaries onannual monitoring to their accreditedinstitutions, is still very variable. The need toimprove the quality of student feedback hasbeen signalled by ValC, and a project topromote more consistently effective use ofstudent feedback has recently begun with arequest to partners to identify and report goodpractice. This is not the first attempt by theUniversity to initiate improvements in the use ofstudent feedback. The audit team formed theview that if the current initiative is to succeedwhere others have failed, the University mayneed to do more than offer guidance orworkshops and seminars on expectations andgood practice to partner institutions to take up,or not, as they choose. Analysis, and reflectionupon the analysis, of student feedback areimportant inputs to the management of thequality of provision, but the University does notreliably have good management informationfrom its accredited institutions derived fromstudent feedback. The team would encouragethe University, in its student feedback project,not only to address matters of good practice infeedback but also take the opportunity tostrengthen the ways in which it is able to makeeffective and consistent use of the outcomes ofstudent feedback in satisfying itself of thequality of programmes that lead to its awards.


page 27

Student admission, progression,completion and assessmentinformation for collaborative provision

113 For students who are registered on coursesdelivered through direct or collaborative teachingpartnerships, data on students are collected fromthe partner institution. The information requiredis the same as for other directly registered OUstudents, and the University's student recordssystem is used to record registrations, assignmentgrades and achievement. This use of theUniversity's student records system enablescomparisons of results to be made across all ofthe University's directly registered students,including those studying through direct andcollaborative teaching partnerships.

114 For accredited provision, students areregistered with the accredited institution, whichin turn registers its students for the awards of theUniversity, and provides the University with dataon student registrations, continuations andcompletions at programme level. Anadministrative audit process has been piloted foraccredited institutions, including an assessmentof admissions and records procedures andarrangements. It is intended that theadministrative audit will take place alongsideeach new accreditation process and eachinstitutional review from 2004-05 onwards.

115 Progression and retention is monitored andevaluated by the accredited institution andreported to the University through the annualmonitoring process. The Handbook givesguidance on the statistics to be provided inannual programme reports, covering targetnumbers, applications, enrolments, entryqualifications, progression and retention, resitsand repeats, classifications and destinations. Inaddition, an annual letter is sent out beforeannual monitoring commences giving anyadditional guidance. For the session 2003-04 thisadditional guidance included the need to gathercomplete sets of statistics and to critically analysethese data in the context of the institution. Inthe annual report, programme teams arerequired to comment on their statistics, with thesuggestions given of comparisons with previous

years, whether there is room for improvement,and how they fit with the institution's mission.Annual monitoring reports are read by thepartnership managers who produce acommentary on their particular institutions usinga pro forma that includes a section on thestatistical analysis. The audit team noted thatnumber of the pro forma commentaries reporteda lack of complete statistics and an insufficientlycritical analysis of the available data.

116 The audit team concluded that the analysisof completion and progression statistics fordirect and collaborative teaching arrangementswas effective at the course level, and enabledthe University to make comparisons to be madewith students engaged with its own provision.For accredited institutions, however, despitespecific attention being paid to this area byOUVS, annual student statistics were in somecases not complete and were not alwayscritically analysed. The University does notproduce any overview of the progression andcompletion statistics provided by its accreditedinstitutions. The team formed the view thatvariation in the completeness and analysis ofthese data was denying the University usefulinformation on the health of its validatedprogrammes, and that the lack of an overviewof these data across the range of validatedprogrammes inhibited the University's ability topromote good practice.

Assurance of the quality of teachingstaff in collaborative provision;appointment, appraisal, support anddevelopment


117 In the case of direct teaching partnerships,the University ordinarily provides all tuitionsupport, and tutors are directly employed bythe University. Arrangements for appointment,monitoring, induction, support, training anddevelopment are similar to those for staff whoteach the University's own courses. Thesearrangements were previously reviewed as partof the 2004 institutional audit of the University.

Open University

page 28

118 In the case of collaborative partnerships, thepartner nominates associate lecturers for approvalby the University according to normal Universitycriteria. The University is involved directly in themonitoring, induction and support of these staff.Partners in more mature collaborative teachingarrangements may appoint and monitor staff. Insuch cases, the audit team was informed that theUniversity provides a full job specification forpartner staff. The University maintains oversightof the quality of staff through a variety of meansincluding the monitoring of staff assessment, andthe analysis of student cohort performanceprofiles at award boards. Staff of the partnerinstitutions are invited to attend curriculumconferences and other briefings. The teamconsidered that staff appointment, monitoringand support in the case of direct andcollaborative partnerships is soundly managed.


119 The CPSED noted that accreditedinstitutions are expected 'to demonstrate theircapacity to take delegated authority for…theappointment, appraisal and development ofstaff who teach on validated programmes' as aprerequisite for institutional accreditation, andthat 'their ability to take this responsibility is thesubject of scrutiny at accreditation andinstitutional review'. Additionally, the Handbookstipulates that during the validation andrevalidation of programmes, strategies forteaching and learning support, theappropriateness of staff, staffing levels and staffappraisal mechanisms are examined. The CPSEDnoted that all staff 'must be appropriatelyqualified and experienced' and that they must'demonstrate a shared understanding of thelearning outcomes of a programme'. The teamformed the view that these processes aregenerally effective in assuring the suitability ofstaff and staffing levels, and the appropriatenessof partner policies and procedures associatedwith staff appointment, appraisal, developmentand support.

120 As part of the annual monitoring processaccredited institutions review staffing and staffdevelopment matters. This review is informed by

the comments of external examiners. Accreditedinstitutions are also expected to provide OUVSwith the curricula vitae (CVs) of new staff. Theaudit team found that the CVs were, in general,provided to the University for new staff, butrelated commentary on staff turnover was notalways apparent in the annual monitoringdocumentation. Although review of the CVs didnot appear to be a requirement, the team heardthat, in practice, partnership managers willexamine CVs and raise any associated concernsthrough the annual monitoring process or atprogramme revalidation. Where evidence wasavailable within annual monitoring reports, theteam found that partnership managers generallyaddressed unresolved staffing issues within theircommentary. The team noted that a number ofrecent QAA audits or reviews of accreditedinstitutions and validated programmes hadcommented on staffing issues.

121 A particular strength of the University is itsprovision of staff development opportunities foraccredited partners. Partners receive regularupdates on significant national academicquality developments, including changes to theCode of practice, published by QAA, and theevolving teaching quality information (TQI)requirements. An OUVS website is available tostaff of partner institutions and, although theuse made of this varied from partner to partner,several staff of partner institutions who met theaudit team commented that they found thiswebsite a useful resource. Seminars andworkshops are generally provided either byOUVS or CHERI, and the team noted inparticular the opportunities for enhancementprovided though the curriculum conferencesand the OUVS annual conference. The teamlearnt that there is good attendance by staff ofpartner institutions at seminars and conferences,and noted that these events were seen by themas valuable staff development opportunities.

Assurance of the quality of distributedand distance methods deliveredthrough an arrangement with a partner

122 Under the direct and collaborative teachingmodels, the University delivers programmesthrough distance methods. In these models of


page 29

collaboration, the University takes responsibilityfor validating and developing the programme,provides all teaching materials, and managesassessment and examination arrangements. Thepartner undertakes local activities includingmarketing programmes, and providing studentadvice, support and guidance. Additionally, in thecase of collaborative distance-learningarrangements, the partner also employs tutor orworkplace assessors directly to deliver tutorialsand mark continuous assessment, and may alsooffer University materials in translation. Theefficacy of these arrangements has beendiscussed elsewhere within this report.

123 In a few instances, programmes validatedto accredited partners are delivered by distancemethods. In such cases, partners assumeresponsibility for developing, delivering andassessing provision. The Handbook has recentlybeen supplemented with guidance foraccredited institutions wishing to offer this kindof programme. Given that validated distance-learning programmes offered by accreditedpartners are a recent development, this auditdid not examine such provision in any detail,and offers no comment on the appropriatenessof the quality assurance arrangements.

Learning support resources forstudents in collaborative provision

124 Students studying in direct andcollaborative teaching arrangements areregistered as students of the University and, assuch, have access to the University's electronicresources and course materials. In the case ofstudents following validated programmes, theUniversity expects the accredited institutions tobe responsible for the provision of learningsupport resources, and the accreditationprocess includes a check that the partnerinstitution can provide an appropriate learningenvironment for study at higher educationlevel. Each accreditation, institutional reviewand validation event should include a meetingor tour to review resources and facilities, atwhich the adequacy of the facilities to supportprogrammes should be tested in accordancewith guidance given in the Handbook for

Validated Awards. The level and continuingappropriateness of the resources is monitoredthrough the annual monitoring process and bythe partnership managers.

125 The accreditation, institutional review andvalidation reports seen by the audit teamgenerally had a section confirming that the panel'toured' the learning resources and found theresources to be satisfactory. Some members ofValC who met the audit team expressedreservations about the effectiveness of thismethod of evaluating the adequacy of learningresources, and considered that there was a needto check with students that suitable resources are,and have been, available to them on acontinuing basis. In the light of these comments,the University might wish to reflect upon whetherits approach to the initial approval of learningsupport resources is sufficiently systematic andgives appropriate attention to forward planning.Students who met the team during the visits toaccredited partners were generally satisfied withthe level of resources provided for them.

126 The annual monitoring pro forma does notrequire a direct commentary on learning supportresources. The University expects partners toreport on changes on such resources, includinginadequacies in provision, as a part of theirgeneral reflections in the annual monitoringprocess, but few had done so in the reports seenby the audit team. Partnership managers andexternal examiners are also expected to reporton any problems arising with learning resources,but unless their attention is brought to suchmatters by the partners' staff or students it isdifficult to see how they would become aware ofdifficulties in this area. The audit teamconsidered that the University's capacity tomaintain an effective oversight of its validatedprogrammes would benefit from a moresystematic process for the interim monitoring ofthe resources available to support learningbetween review and revalidation events, and amore structured process for evaluating thesuitability of learning support resources at initialapproval and validation. In cases where aspectsof the provision are considered not to besatisfactory, there appear to be no formal

Open University

page 30

sanctions that the University can apply short ofearly review or withdrawal of approval, althoughthe University explained that 'where appropriateOUVS makes a judgement about applyingsanctions in proportion to the problem'. Theteam formed the view that a defined scale ofmeasures to be taken when problems areidentified would complement a more systematicmonitoring process, and would be of help topartnership managers in ensuring that there istimely corrective action.

Academic guidance and personalsupport for students in collaborativeprovision

127 Academic guidance and personal supportfor students in direct teaching arrangementsare provided by University staff. Guidance andsupport in collaborative teaching arrangementsare frequently provided by staff of the partnerinstitution, and where programmes aredelivered in a language other than English thisis always the case. These staff are recruited bythe partner to an agreed person specification.Their performance in matters of academicguidance is monitored by the relevant centralacademic unit of the University through theUniversity's normal statistical monitoring ofstudents' performance, in the same way thatthe University monitors the performance of staffsupporting its own courses. The adequacy ofpersonal support to students is monitoredthrough the annual monitoring process. Fromits study of the available documentation, theaudit team found the monitoring by theUniversity of academic guidance in direct andcollaborative teaching arrangements to beeffective. The monitoring of the personalsupport, however, appeared to be given ratherless attention, and was dependant on thevariable quality of reporting in the annualmonitoring process for curriculum partners.

128 Accredited institutions are charged withresponsibility for the academic support andguidance of students on validated programmes.As part of the accreditation process, the partnerinstitution must demonstrate that it has anappropriate system for student guidance and

support covering arrangements for studentinduction, the role of the personal tutor, careersguidance at programme and institutional level,counselling and student welfare, support forstudents with disabilities, financial advice andsupport, and support for students undertakingplacements. The exact form of these arrangementsvaries between partners due to different localtraditions and expectations. They are tested inreview and validation processes, and monitoredthrough the annual monitoring process.

129 The reports of institutional review andrevalidation events indicated that matters ofacademic guidance and personal support werefrequently dealt with by ensuring the studentswere satisfied with the provision. The audit teamfound little evidence that systems or practicesrelating to academic guidance and support wereexplored with the staff of the accreditedinstitution. The lack of reporting on the quality ofacademic guidance and personal support as partof the annual monitoring exercise has not beenthe subject of comment by the scrutiny subgroupof ValC. The University emphasised to the teamthat, in its model of accreditation, it holds theaccredited institution accountable for theseaspects of the student experience, andresponsible for taking timely corrective action asnecessary. The University appears to rely on theinfluence of partnership managers to identify anyunsatisfactory situations and deal with them asappropriate. The team considered that theUniversity's capacity to maintain an effectiveoversight of its validated programmes wouldbenefit from a more systematic process forinterim monitoring between review andrevalidation events.

Section 3: The auditinvestigations: publishedinformation

The experience of students incollaborative provision of the publishedinformation available to them

130 In the case of direct and collaborativeteaching partnerships, the methods used by the


page 31

University for assuring the quality of materialsprovided to students are largely the same as forthose programmes provided directly by theUniversity. These processes were found to beeffective in QAA's 2004 institutional audit. Avariety of materials is provided, including thoserelated to promotion and marketing ofprogrammes, learning materials and studentguidance. All students have access to the on-linestudent handbook. The CPSED explained thatlarge and cross-faculty partnerships provide theirown handbooks which are checked by theUniversity before publication. Programmespecifications are required for programmesdelivered through direct or collaborativeteaching partnerships. Student transcripts andcertificates are issued by the University in a formconsistent with those produced for its ownawards gained through its own courses.

131 Where programmes are offered intranslation, the collaborative partner translates avariety of materials including learning materialsand the student handbooks. The locallyappointed external examiner has a role inreporting on the accuracy of translations. Theaudit team noted from the minutes of CuPCthat the University is introducing a processwhereby it can confirm that locally producedpromotional material is accurately translated.The team heard that programme specificationsare not provided in translation, and suggeststhat doing so would be of benefit to students inrelevant collaborative teaching arrangements.

132 In accreditation and validation, theUniversity has a number of means whereby itseeks to ensure that accredited partners provideappropriate and reliable published information onits relationship with the University, and on thevalidated programmes that it offers. TheUniversity provides partners with a standard textthat describes accreditation and validationintended for use in promotional material. Anyvariation to this form of words is subject toapproval by the University through OUVS.Publicity materials are scrutinised by the panelduring programme validation and revalidationevents, and the audit team was told bypartnership managers that they check websiteson an ad hoc basis to confirm the accuracy of

marketing material. To address the task ofassuring the veracity of material produced inlanguages other than English, the University hasrecruited staff with the necessary language skillsto review such materials. The recently introducedadministrative audit (see above, paragraph 114),to be undertaken prior to accreditation andreview, encompasses a review of marketing andpromotional materials. Students who met theteam during the visits to accredited partnersexpressed the view that, overall, the publicitymaterial that they had seen was fair and accurate.

133 The University provides the Student'sGuide To Studying On A Programme ValidatedBy The Open University to accredited partnersfor distribution to students on validatedprogrammes. The audit team considered this tobe a helpful guide to the nature of therelationship between the University and thepartner, and noted that the guide clearly laidout the opportunities for students to appealdirectly the University. Most students who metthe team during the visits to accreditedpartners had a general awareness of the role ofthe University in the design, development andquality assurance of their programme of study.

134 The University expects programmehandbooks to be available to all students onvalidated programmes. The Handbook forValidated Awards specifies that programmehandbooks should be provided for the panel atvalidation or revalidation events, and that adefinitive version of those handbooks should belodged with OUVS. The University provides aprogramme handbook specification for the useof programme teams preparing for validation orrevalidation. Regulations relating to theprogramme of study are approved by theUniversity during accreditation and areadditionally made available to students.Students who met the audit team during thevisits to accredited partners reported that theprogramme handbook and associated materialwere found to be helpful and informative, andthat regulations had been made available.

135 The CPSED stated that the University hasthe expectation that all validated programmeswill provide programme specifications, and

Open University

page 32

provides partners with a template for thispurpose. Programme specifications are beingproduced on a rolling basis, and allprogrammes validated or revalidated since2003 are required to provide specifications aspart of the validation or revalidation process.Although not all programmes had programmespecifications at the time of the audit visit, inthe light of the clear requirement to providespecifications for all programmes and theimminence of scheduled reviews for validatedprogrammes currently without specifications,the audit team considered that the Universitywas taking appropriate steps to ensureprovision of programme specifications forvalidated programmes.

136 All certificates are issued by the University.These comply with the guidance given in theCode of practice, published by QAA, relating tocollaborative provision. There is an obligationon partners to issue transcripts to each studentat a number of prescribed stages throughouttheir studies. The transcripts are provided underarrangements agreed at institutional approvalor review, and the University provides a full andhelpful template for transcripts in theHandbook. The provision by partners ofaccurate and complete transcripts is one of theaspects of collaborative arrangements that is tobe confirmed as part of the recently introducedadministrative audits.

137 On the basis of its study of the University'ssystems for assuring the quality of informationsupplied to students, of samplingdocumentation provided to students, and ofhearing the views of students, the audit teamconcluded that the University has appropriatemechanisms to assure the provision of accurate,clear and useful information to students onvalidated programmes.

Reliability, accuracy and completenessof published information oncollaborative provision leading to theawarding institution's awards

138 The audit process included a check onprogress towards meeting the requirements ofHEFCE's document 03/51, Information on quality

and standards in higher education: Final guidance.The policy for the provision of data andinformation by accredited partners was approvedat University level by ValC, which has determinedthat in the case of accredited partners, qualitativeTQI information for all partners should beprovided. All accredited partners are nowrequired to submit summaries of externalexaminers' reports, and the University will providesummaries of programme revalidation reports.These summaries were in the process of beingplaced on the TQI website at the time of theaudit visit. Partners with their own area of theTQI website are responsible for maintaining theassociated TQI, but are required to submit anymaterial to posted on the web to the Universityfor prior approval. In the case of quantitative TQIdata, any data provided is that as submitted aspart of the University's HESA data set. The dataformat is the same as that for the University'sown courses, provision of which was found to besatisfactorily addressed in QAA's 2004 institutionalaudit of the University. Based on the evidenceseen during the current audit, the audit teamformed the view that the University was makingappropriate progress towards providing TQI dataas specified in HEFCE 03/51.


page 33

Findings

Findings of the collaborativeprovision audit 139 An audit of the collaborative provisionoffered by the Open University (the University orOU) was undertaken during the period 9 to13May 2005. The purpose of the audit was toprovide public information on the quality of theprogrammes of study offered by the Universitythrough arrangements with collaborativepartners, and on the discharge of the University'sresponsibility as an awarding body in assuring theacademic standard of its awards made throughcollaborative arrangements. As part of thecollaborative audit process, the audit team visitedfour of the University's collaborative partners. Thissection of the report summarises the findings ofthe audit. It concludes by identifying features ofgood practice that emerged during the audit,and making recommendations to the Universityfor action to enhance current practice in itscollaborative arrangements.

The effectiveness of theimplementation of the awardinginstitution's approach to managingits collaborative provision

140 The University aspires to be a 'world-classuniversity with an international presence', and itidentifies collaborative partnership as a keymeans of achieving its mission. Two 'middle-tier'strategies of the University, its curriculumstrategy, 2003 to 2008, and a recently approvedinternational strategy, include strategic aspectsof collaborative activity, but the University doesnot articulate a separate specific strategy forcollaborative provision. Rather, it responds toapproaches from potential collaborative partnersand assesses their capabilities for engagingsuccessfully with the University through one ofits partnership models.

141 The University defines three models ofcollaborative partnership which are the subjectof this audit: accreditation and validation; directdistance teaching; and collaborative distanceteaching. In direct teaching partnershipsstudents are taught at the partner institution byUniversity staff, while in collaborative teaching

partnerships partners are wholly or partlyresponsible for aspects of tuition andassessment of the University's curriculum. Inboth cases the students are registered asstudents of the University on University coursesfor named awards of the University.

142 Accreditation is the process by which theUniversity approves a partner institution ashaving a 'suitable environment andinfrastructure for the development, delivery andquality assurance' of validated programmesleading to awards of the University. Validation isthe process of confirming that a programmeoffered by an accredited institution is 'ofappropriate quality, the intended learningoutcomes are at an appropriate level, and thestandards set is comparable with similar awardsoffered throughout UK higher education'.Students on validated programmes areregistered with the accredited partnerinstitution, and are registered with theUniversity by the partner only for the purposeof receiving an award of the University.

143 The University's approach to managing itscollaborative provision therefore differssignificantly between its models of partnership.The direct and collaborative teachingpartnership models are characterised by alimited amount of delegated authority to thepartners, a close alignment between thelearning opportunities offered through thecollaborative provision and those offeredthrough the University's own courses, and astrong working relationship between thepartner and University's own academic units inthe management of the quality and academicstandards of the collaborative provision. Theaccreditation and validation model ischaracterised by a high level of authoritydelegated to the accredited partner for thedelivery, support and assessment of thepartner's own programmes leading to an awardof the University. Management of the qualityand academic standards of validated provisionis seen principally as the responsibility of thepartner institution, the capacity of the partnerto undertake these responsibilities beingformally assessed by the University at initial

Open University

page 36

accreditation and formally re-confirmed atinstitutional review. The University expects itsaccredited partners to identify issues relating tothe management of quality and academicstandards and take appropriate action withoutthe University requiring or initiating action.

144 Overall, the audit team found that theUniversity was managing the direct andcollaborative teaching partnership models ofcollaborative provision effectively. Theaccreditation and validation model is built upona very high level of authority delegated to theaccredited partner institution. Although theUniversity has recently sought to strengthen theextent to which it monitors validatedprogrammes, and has identified plans todevelop this further, the team found that theimplementation of the University's approach tothis model of collaborative provision was notyet sufficiently robust and rigorous to enablethe University be fully confident of effectivemanagement of the quality of provision offered byaccredited partners, or of the academic standardsof its awards gained through that provision.

The effectiveness of the awardinginstitution's procedures for assuringthe quality of educational provisionin its collaborative provision

145 The University retains responsibility forquality assurance in its direct and collaborativeteaching partnership provision, using itsstandard procedures in the same way as it doesfor its own provision. The 2004 institutionalaudit found these procedures worked effectivelyfor the University's own provision, and thefindings of this collaborative provision auditsuggest that the University's procedures forassuring the quality of collaborative provisionthat is provided through direct andcollaborative teaching arrangements are equallyeffective. The audit team's concerns about theUniversity's procedures for assuring the qualityof educational provision focus entirely onaccreditation and validation arrangements.

146 Accreditation is based on the partnerinstitution being able to demonstrate, throughthe accreditation process, that it meets the

University's criteria for providing 'a satisfactoryenvironment for the conduct and qualityassurance' of programmes leading to awards ofthe University. If accreditation is achieved, thepartner institution is given delegated authorityand 'full responsibility for teaching, assessment,and further development' of validatedprogrammes. The audit team considered thatthis model for assigning substantial delegatedauthority was not entirely consistent with theview expressed by the University that 'differentkinds of partners require different levels ofsupport and oversight'.

147 The University explained that it was takingaction in the management of two accreditedpartnerships 'where the gap between ourexpectations and their [assessment] practice hasraised concerns'. While acknowledging that theUniversity has identified these concerns and isacting on them, the audit team considered thatthe identification of a post-accreditation 'gap'cast further doubt on the robustness of theinitial accreditation judgement, and on theconfidence placed in that judgment by theUniversity. There is a significant body ofevidence to indicate that the University'saccreditation model for assigning a substantiallevel of managerial and operational authority bythe University to a partner institution is notsufficiently robust in practice reliably to ensurethat accredited partners meet, and continue tomeet, the high level of responsibility for qualitymanagement with which they are charged. TheUniversity argued that the body of positiveevidence indicated the robustness of the model,but the team took the view that a model that isable to assure quality in many, but not all, casesis not a robust quality assurance model.

148 Accreditation is built upon peer review. Inview of the significance of the decisions madeby an accreditation panel on behalf of theUniversity, it is critically important that panelsrepresent an appropriate level of current andindependent experience of UK highereducation, and have a good understanding ofthe way in which the broader higher educationagenda is developing and safeguarding thequality and academic standards of highereducation provision in the UK. The team

Collaborative provision audit: findings

page 37

considered that the University's own academicstaff are under-represented on accreditationpanels, which draw heavily upon representationfrom institutions who are themselves accreditedpartners. The University may wish to reflectupon the merit of making a distinction betweenthe engagement of members of partnerinstitutions in a developmental role in theaccreditation process and their engagement onaccreditation panels as independent externalexperts and advisers to the University.

149 The University is implementing proceduralchanges in accreditation and validation that theaudit team considered to be positive movestowards enhancing the University's capacity forthe effective management of its collaborativeprovision. In many of the procedural changesthe University has sought to increase the level ofinformation it receives in monitoring the qualityand academic standards of validatedprogrammes. In doing so, it was the team's viewthat the University is implicitly recognising thatthe substantial responsibility assigned to partnersby its model of accreditation does not sit entirelycomfortably with its own responsibility as theawarding body. In recent measures taken by theUniversity to strengthen its annual monitoring ofvalidated programmes offered by accreditedpartners, the University has gone some waytowards recognising that partners are likely torequire a higher level of monitoring and supportfollowing accreditation than has hitherto beenthe case, but this runs counter to the model ofaccreditation as a process that confirms apartner's ability to accept fully the substantiallevel of delegated authority that accreditationcarries. The team saw some tensions within theUniversity's model of accreditation, and wouldencourage the University to consider if it, too,sees these tensions.

150 Senior staff of the University expressed tothe audit team the view that the Universityaccepted that the nature of the evidence that itwas relying upon in monitoring accreditedpartners' management of the quality ofvalidated provision was not sufficiently robustand secure, and that it had, therefore, moved tomake it more robust. The team acknowledgedthe extent to which the University had sought

to rectify shortcomings in accreditation andvalidation that had been identified by theUniversity itself and by external audit and reviewevents. The team remained concerned,however, by the apparent inability of theUniversity to ensure urgent action by partners tosome serious shortcomings that the Universityhad itself identified, and which were variouslyreflected in the reports of QAA audits andreviews at institutional and subject levels. Theteam noted many instances in partnershipmanagers' reports on annual monitoring whereissues were recorded as having been raised inprevious annual reports, and instances where itwas observed that no action has been reportedon serious matters relating to quality andstandards of the provision. The identification ofthe same matters in successive annual reportsdoes not appear to result in action by theUniversity, and failure to take action does notappear to have consequences for the partner.Because accreditation delegates such substantialresponsibility for quality assurance to partners,the University has limited sanctions available toit beyond recourse to the triggering of arevalidation or institutional review, and the teamconsidered that the University might see meritin defining a scale of measures short of suchmajor sanctions for partnership managers toapply in a timely manner when problems areidentified. Whatever the underlying cause,failure to take corrective action on problemsdespite their continuing identification isunsatisfactory, and does not give confidence inthe effectiveness of the University's proceduresfor assuring the quality of educational provisionoffered through the validated programmesprovided by accredited partners.

151 The report of QAA's 2002 audit of theUniversity's partnership with KoldingKøbmandsskole noted that 'the audit team wasnot entirely convinced that monitoringmechanisms currently in place permit theUniversity to have sufficient level of confidencethat the quality of learning opportunities andstudent support provided in the partnerinstitution is satisfactory', and that the Universitymay not have the necessary structures andprocedures to 'provide it with a firm grasp on

Open University

page 38

information relating to the nature and quality ofthe provision in accredited institutions'. Aconclusion of that report was that the University'sarrangements at the time of the audit were notsufficiently robust to enable the University 'tohave the necessary level of confidence that ifproblems arose with quality or standards it wouldquickly be made aware of them'. A recurringquestion that the current audit team posed foritself was the extent to which developments inthe University's procedures for assuring thequality of provision in accreditation andvalidation arrangements would make thecomments and conclusions of the 2002 audit of apartnership link no longer valid. While noting theprocedural changes in the processes ofaccreditation, validation and monitoring that theUniversity has made since 2002, the team wasnot convinced that these were yet sufficient toguarantee that the findings of an audit of theUniversity's link with an accredited partner wouldnot now reach conclusions similar to those of the2002 audit. As noted above (paragraph 145), theteam found the University's procedures forassuring the quality of direct and collaborativeteaching arrangements to be effective, and itemphasises that its concerns about theUniversity's procedures for assuring the quality ofeducational provision focus on the accreditationand validation arrangements. Nevertheless, anaudit of collaborative provision requires an overalljudgement on the capacity of the University tomanage its collaborative arrangements effectively,without distinguishing between the type ornature of its portfolio of collaborativearrangements. The team therefore concludedthat only limited confidence can reasonably beplaced in the present and likely future capacity ofthe University to satisfy itself that the learningopportunities offered to students through itscollaborative arrangements are managedeffectively and meet its requirements.

The effectiveness of the awardinginstitution's procedures for safeguardingthe standards of its awards gainedthrough collaborative provision

152 In the case of courses delivered throughdirect and collaborative teaching partnerships,

the processes of the assessment of studentachievement, and the monitoring of thatassessment by external examiners, are the sameas those operated for the University's own opensupported learning. Statistics on progressionand achievement for students studying throughdirect or collaborative teaching partnerships arecollected and analysed in the same way as forother registered students of the University. Arange of statistical data is produced at courselevel, and comparisons of results are made withthe University's own students as part of theannual course monitoring process. The auditteam considered that for direct andcollaborative partnership the process for theevaluation of student statistics provided aneffective mechanism for maintaining theacademic standards of courses.

153 Where courses are taught and assessed intranslation, a local bilingual external examiner isappointed. The audit team noted that it waspossible for the only UK higher educationexperience of the local external examiner to be asa student on a UK award that might have beenstudied outside the UK, and for OU associatelecturers to be regarded as suitable for such anappointment. The team questioned whether theUniversity's criteria for the appointment of localexaminers for courses in a language other thanEnglish were sufficient to ensure that all localexternal examiners are appropriatelyindependent and have an experience ofundertaking the delivery and assessment of UKhigher education that allows them to make anexpert judgement on the standards of materialsin translation. Apart from this, the team wasable to conclude that the University haseffective procedures for safeguarding thestandards of its awards gained through directand collaborative teaching arrangements.

154 The audit team's concerns about theUniversity's procedures for assuring the quality ofeducational provision focus principally onaccreditation and validation arrangements.Accredited partners nominate the externalexaminers for their validated programmes. TheUniversity approves the accredited partner'snominees after seeking the advice of threeexpert referees, and the nominees are then


page 39

appointed by the academic board of the partnerinstitution. Some accredited institutions havebeen given further delegated powers by theUniversity for the appointment of externalexaminers without the need to seek theUniversity's approval of nominees, in which casethe University requires evidence that thenomination has been scrutinised by threeindependent assessors. The University and itsaccredited institutions regard the externalexaminers as operating on behalf of theaccredited institutions rather than on behalf ofthe University. External examiners are required toreport back to the accredited institution and tothe Vice-Chancellor of the University on studentperformance and the academic standards thatthey achieve. The Handbook for ValidatedAwards also stipulates that external examiners ofvalidated programmes should report to theUniversity on any matters of serious concernarising from the assessments which put at riskthe standards of the University's award.

155 External examiners' reports and theresponses to them form part of the annualprogramme monitoring process. Partnershipmanagers are able to take immediate action onany issues relating to academic standardswithout waiting for these issues to be processedthrough formal annual monitoring procedure,but the University does not take an overview ofexternal examiners' reports on validatedprogrammes. The audit team concluded, onbalance, that the University had strengthenedits link with external examiners sufficiently todischarge its responsibility as the awardinginstitution for ensuring that external examinerscontribute effectively to the maintenance ofacademic standards of validated programmes.However, the University is encouraged toconsider how it might further strengthen itsdirect links with external examiners, forexample, by ensuring that all externalexaminers' reports for validated programmes areproduced in a timely manner and to a suitablestandard, and by taking an overview of mattersraised in external examiners' reports in order toidentify common issues that the University couldusefully address at institutional level to informthe enhancement of its collaborative provision.

156 A wide range of the University'scollaborative provision has been subject toaudit and review by QAA, including overseascollaborative links, reviews of validatedprogrammes at subject level, reviews ofFoundation Degree provision and theenhancement-led institutional review of anaccredited partner institution. The Universitydoes not seek to take an overview of themesand issues in respect of the academic standardsof its awards emerging from QAA engagementswith its accredited partners.

157 QAA subject-level reviews at a particularaccredited partner identified weaknesses in theprocedures for external examining in reports inthree successive years from 2002. In the 2004external review of that accredited partner, thereview team expressed 'serious reservations aboutthe implementation of the procedures forexternal examining', and came to a judgement oflimited confidence in the partner's procedures forthe management of the academic standards of itsprovision. Despite these clear messages aboutpotential weaknesses in the security of academicawards made in its name, the University hasapplied no sanctions to ensure that the delegatedauthority conferred by accreditation is soundlyheld, and, indeed, continues to recognise theadditional delegated powers that it awarded tothat accredited partner in 2002 to operate itsown validation events.

158 The audit team considered that the bodyof evidence indicated that the University'sprocedures for safeguarding the standards of itsawards gained through its accreditation andvalidation model of collaborative arrangementsare not fully effective. As noted above(paragraph 153), the team was able toconclude that the University has effectiveprocedures for safeguarding the standards of itsawards gained through direct and collaborativeteaching arrangements. Nevertheless, an auditof collaborative provision requires an overalljudgement on the capacity of the University tomanage its collaborative arrangementseffectively, without distinguishing between typeor nature of its portfolio of collaborativearrangements. The team therefore concludedthat only limited confidence can reasonably be

Open University

page 40

placed in the soundness of the University'spresent and likely future management of theacademic standards of its awards made throughcollaborative arrangements.

The awarding institution's use of theAcademic Infrastructure in thecontext of its collaborative provision

159 Accredited institutions are expected tohave knowledge and understanding of theAcademic Infrastructure, and the accreditationprocess requires them to demonstrate that theyare benchmarking their quality assurancearrangements to the Academic Infrastructureand other accepted good practice in UK highereducation. The University has mapped its owninstitutional policies and procedures againsteach section of the Code of practice for theassurance of academic quality and standards inhigher education (Code of practice), published byQAA, and requested its accredited institutionsto submit a mapping of their own policies andprocedures against the Code by December2004. At the time of the audit visit the analysisof the outcomes of this mapping by partnerinstitutions had not been completed, but theteam was informed that the outcomes wouldbe reviewed in detail through institutionalreviews over the next few years. The audit teamconcluded that the University had given allsections of the Code proper consideration, andwas in the process of ensuring that partnerinstitutions had also done so.

160 The University has its own levels andawards framework, and accredited institutionspropose programmes for validation for anaward from that framework. The 2004institutional audit of the University found thatthe University's levels and awards frameworkwas consistent with The framework for highereducation qualifications in England, Wales andNorthern Ireland (FHEQ), published by QAA.The programme validation process is used tocheck that validated programmes conform tothe University's levels and awards framework,and the audit team concluded that the processwas sufficiently robust to do so effectively.

161 Programme specifications for validationand revalidation of programmes have beenrequired since 2003. The programmespecification template requires reference tosubject benchmark statements. Externalexaminers are expected to comment in theirreports on the quality of knowledge and skillsdemonstrated by students in the light of therelevant subject benchmark statement, and ofthe FHEQ (or the Scottish Credit andQualifications Framework where appropriate).

162 In its discussions with the staff of theaccredited institutions the audit team found agood understanding of the AcademicInfrastructure and a willingness to ensure thatproper account is taken of its requirements andguidance. The Open University ValidationServices (OUVS) has recognised that somepartner institutions were relatively unfamiliarwith the use of Academic Infrastructure, and hasmade significant efforts to provide them withadditional support and guidance. Overall, theaudit team found that the University is makingeffective use of the Academic Infrastructure inthe context of its collaborative provision.

The utility of the CPSED as anillustration of the awardinginstitution's capacity to reflect uponits own strengths and limitations incollaborative provision, and to act onthese to enhance quality andsafeguard academic standards

163 The audit team found the collaborativeprovision self-evaluation document (CPSED) tobe a clear descriptive document that went tosome length to explain carefully the variousmodels of collaborative provision supported bythe University. It made a significantcontribution to the team's understanding of thedifferent models of collaboration, and of thearrangements that the University has put in placefor the maintenance and enhancement of qualityand standards of its collaborative provision. Whileit provided some illustration of reflection onoperational matters, there was a lack of reflectionat a more strategic and institutional level. Overall,


page 41

the team found the CPSED to be a fair illustrationof the University's capacity to reflect on its ownstrengths and limitations in its approach tocollaborative provision.

Commentary on the institution'sintentions for the enhancement of itsmanagement of quality and academicstandards in its collaborativeprovision

164 The University reviews its practices andprocesses through the Curriculum PartnershipsCommittee and through the ValidatingCommittee. At the time of the audit visit theUniversity was piloting a 'staged' accreditationprocess designed to provide greater support andguidance to institutions wishing to achieveaccredited status. It was implementingprocedural changes to bring aboutimprovements in such matters as the standard ofreporting from partners, and the use ofquantitative data, and was in the process ofcapturing these in a revised (2005) edition of theHandbook for Validated Awards. The audit teamconsidered that the procedural improvements,including the use of scrutiny subgroups, werepositive moves towards enhancing theUniversity's capacity for the effectivemanagement of its collaborative provision.

165 The University provides a variety ofenhancement opportunities for its partnerinstitutions such as the OUVS AnnualConference, curriculum conferences, seminarsorganised by OUVS and by the University'sCentre for Research in Higher Education(CHERI), and more informal opportunitiesbased on the OUVS Network. Partners are freeto avail themselves of these and otheropportunities as they deem necessary to fulfiltheir responsibility to enhance their provision.Opportunities for individual staff developmentare available through staff visiting one another'spartner institutions in a variety of capacities, forexample, as members of accreditation orvalidation panels. The University expectsenhancement activity to be reported andidentified through the annual monitoringprocess. The audit team saw few examples of

such reporting in papers coming forward to therelevant University committees, although itdiscussed with staff of accredited partnersinteresting examples of enhancement whichhad not been identified by the annualmonitoring process. The team formed the viewthat the University provided good opportunitiesfor enhancing practice, but the extent to whichthe partners avail themselves of theseopportunities is left largely to their discretion.Overall, the team considered that theenhancement of the quality of the provisionoffered through collaborative arrangementswould benefit from a more proactive approachfrom the University.

Reliability of information provided bythe awarding institution on itscollaborative provision

166 A wide range of published materialsassociated with the University's collaborativeprogrammes is provided, including marketingand publicity materials, learning resources,student guidance and student handbooks.Some of this material is provided directly by theUniversity; other material is provided bypartners. In the latter case, where appropriate,the University provides guidance on the style,format and content of the materials, and hassystematic approval processes in place toconfirm the reliability of information that isprovided. Where material is provided intranslation, the University has put in placesuitable mechanisms to confirm the accuracy ofthe material.

167 The University has well-developed plans inplace to provide the teaching quality information(TQI) set defined in HEFCE's document 03/51,Information on quality and standards in highereducation: Final guidance, and has mappedcurrent collaborative provision against theserequirements. The audit team was satisfied thatthe information that the University and itspartners are publishing currently about thequality of its programmes and the standards of itsawards is reliable, and that the University ismaking adequate progress to providing TQI datafor its collaborative provision.

Open University

page 42

Features of good practice

168 Of the features of good practice noted inthe course of the collaborative provision audit,the audit team noted in particular:

i the opportunities for enhancementprovided though the curriculumconferences and the OUVS annualconference (paragraphs 53, 121).

Recommendations

169 It is essential that the University:

i strengthens the means by which itestablishes confidence in the security andcomparability of the academic standardsof its awards provided through accreditedpartners (paragraphs 65, 66, 71, 74, 81,82, 92, 93, 100, 103, 116)

ii implements valid and reliable mechanismsto enable it to have confidence that thequality of learning opportunities andstudent support provided through all itsaccredited partners are managedeffectively and meet the University'srequirements (paragraphs 65, 66, 67, 69,71, 72, 74, 79, 81, 82, 103, 107, 125,126, 129)

and advisable that the University:

iii ensures that all local external examinersappointed for courses offered overseas areappropriately independent and haveexperience of undertaking the deliveryand assessment of UK higher education(paragraph 85).


page 43

Appendix

The Open University's response to the collaborative provision audit reportWe are pleased that our arrangements for direct and collaborative teaching have been deemedeffective, but very disappointed that arrangements for accredited and validated provision, despitethe progress acknowledged in the QAA report, are still seen as needing further development to gaina verdict of full confidence. We endorse the positive statements made about our accredited andvalidated provision managed through OUVS, in particular: that our external examiner approvalprocess is rigorous (paragraph 86), the strength of our staff development systems (paragraph 121),that QAA framework academic infrastructure issues have been properly addressed (paragraph 98),and that the series of changes we have made has resulted in an information flow which enables usto monitor partnerships more actively and effectively (paragraph 78).

As part of our programme of continuous improvement to OUVS services we have already takenfurther action since the audit visit. This includes developing the academic audit role through whichfaculty staff will increase their involvement in QA-related processes on validated programmes. Theacademic governance review, now nearing completion, has involved us in revising the terms ofreference of key committees and ensuring that there is greater oversight of performance across theUniversity's validated provision and more systematic oversight of progress on review outcomes andconditions. We already have a first draft of an action plan, as required by QAA, which will respondto all other issues raised in the report. We note and endorse the next stage whereby QAA approvesour action plan, since it is clearly important both for the OU and more widely that confidence inthe OU's collaborative and validated provision is maintained.

Open University

page 44

RG

167 10/05

Date post:	06-Oct-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Collaborative provision audit - Open University - MAY 2005 · Collaborative provision audits are...

Documents