The External Review Report as The External Review Report as

Ground for ENQA Board DecisionGround for ENQA Board Decision

Tibor Szanto

Vice-President, ENQA

Barcelona, 17 March 2009

Some statistics*

Externally reviewed agencies: 23Membership: 40 full + 8 candidate

Review reports on ENQA website: 21(ACQUIN, NQAI missing)

Year of report• 2005: 3• 2006: 4• 2007: 7• 2008: 7

Lenght of reports: pp.10 (HSV) – 64 (NVAO)

* Source: ENQA website (www.enqa.eu, 14-15 March 2009)

17.03.2009, Barcelona Tibor Szanto 2

Type and co-ordination of review

Type A (ENQA memb. crit. / ESG)Type B (also other aspects)

• statutory functions• ECA Code of Good Practice• special context (e.g. bi-national character, NVAO)• follow-up on previous review

Co-ordination • national• ENQA (+ other non-national)

17.03.2009, Barcelona Tibor Szanto 3

National (internat.) context in reports

Various approaches:• brief history of agency only (in many cases)• description of national context• description of international context (too)

To be considered:• background information needed• relevance of national (internat) context for

judging compliance

17.03.2009, Barcelona Tibor Szanto 4

Coverage – ESG in reports

Part I – mentioned in some cases, in relation to agency activities/standards

Part II• discussed mostly under Standard 3.1

(Part III, Use of external QA processes)

• discussed separately in some cases• in some other cases not discussed at all

(first reports)

Part III – discussed in all reports

17.03.2009, Barcelona Tibor Szanto 5

ENQA additional criteria1

Not really discussed in the reports.

ENQA Criterion 3 – Resources (ESG 3.4)Agencies should have adequate and proportional

resources, both human and financial, to enable them to organise and run their external quality assurance process(es) in an effective and efficient manner, with appropriate provision for the development of their processes, procedures and staff.

17.03.2009, Barcelona Tibor Szanto 6

ENQA additional criteria2

ENQA criterion 8 - Miscellaneousi. The agency pays careful attention to its declared principles at

all times, and ensures both that its requirements and processes are managed professionally and that its judgements and decisions are reached in a consistent manner, even if the judgments are formed by different groups

ii. If the agency makes formal quality assurance decisions, or conclusions which have formal consequences, it should have an appeals procedure. The nature and form of the appeals procedure should be determined in the light of the constitution of the agency.

iii. The agency is willing to contribute actively to the aims of ENQA

17.03.2009, Barcelona Tibor Szanto 7

Presentation of findings in reports

Diverse approaches• evidence and statements• evidence, some reflections, conclusions• evidence, analysis, conclusions• evidence, analysis, conclusions,

recommendations

Length, level of elaboration varies.

Something is missing in many cases…

17.03.2009, Barcelona Tibor Szanto 8

Presentation of findings – the ideal case

Best practice (what the Board makes happy:)

• ESG Parts II and III fully covered• Each standard discussed separately• Structure (under each standard):

a) evidence („description”, „facts and data”)b) analysis („consideration”)c) conclusion (judgement on compliance)d) recommendation (if any)

Don’t forget about ENQA additional criteria!

17.03.2009, Barcelona Tibor Szanto 9

Judgements in the reportsCases:

• separate judgement on each criterion• overall judgement too

Some examples as to wording:• „the agency complies with…”• „the criterion has been met”• „the criterion has been complied with”• „sufficiently complies”• „substantially complies”• „criteria are being largely met”• „full compliance has not been achieved”• „not yet fully compliant”

17.03.2009, Barcelona Tibor Szanto 10

Judgements expected

The Agency is with the criterion/ESG standard• fully compliant• substantially compliant• partially compliant• non-compliant

Overall judgement not required (but can be made if the panel wishes to do so)

17.03.2009, Barcelona Tibor Szanto 11

Recommendations expected

Best practice:• under the relevant criterion

PLUS / OR• summarised either in the Executive Summary

or at the end of the report in a concluding chapter

Should stand out, well structured!

SHOULD BE! Reviews: accountability + improvement

17.03.2009, Barcelona Tibor Szanto 12

The Board’s headache1

„The European Standards and Guidelines were not written as a set of binding regulations or requirements and are not a set of compliance criteria to be checked off.”(NOKUT Report, 2008. 11.p)

The ENQA membership criteria ARE compliance criteria to be checked off … or not?

The compliance paradox?

17.03.2009, Barcelona Tibor Szanto 13

The Board’s headache2

Are there various levels of compliance?

Linguistically-logically: definitely not.

(„Slightly pregnant”)

Socio-logically: yes

The substantial (social pregnancy) paradox?

17.03.2009, Barcelona Tibor Szanto 14

The solution?

Wisdom,advice,

and good reportsneeded!

17.03.2009, Barcelona Tibor Szanto 15

Thank you for your

attention!