ENQA seminar: “First external evaluations of quality assurance

agencies – lessons learned”

Panel discussion: Practicalities and challenges of self and

external reviews: perspective of the expert team

Michael Gerhard Kraft (secretary of the review of AQU Catalunya and

academic staff member FH Council (FHR))

2

Review Process (flow chart)

ToR SE-report written evidence+

ENQA briefing package

Telephone conference

RT draws uplines of inquiryand timetable

Site visit: further evidence, verifying

written evidence

Report: template / draft / amendments by the agency

Final reportand state-

ment of theAgency sub-

mitted to the ENQA Board

3

Lessons learned

Terms of reference should state clearly the assignment of the review team (RT) (sole vs. multiple purpose review)

Focus should be on the ESG, when additional areas for evaluation are being asked for, one should bear in mind that they might bind/require additional resources and a certain composition of the RT (e.g. teaching staff and research assessment of the Spanish agencies)

Conclusions of the RT should distinguish between the different areas of evaluation but should take the consistency of the whole report into consideration

Heavy workload particularly for the Chair and the Secretary

4

Email correspondenceand some of my handwriting

Lessons learned

5

Lessons learned

Quality and comprehensiveness of the self evaluation report is important for the preparation of the RT (principles of self-criticism, objectiveness and openness / not window-dressing!)

The agency presented a comprehensive evidence list on the internet, from where the documents were well accessible • but it is important to prioritise: maybe the agency should

state, which documents it considers particularly important as evidence for its findings in the self-evaluation report.

Good overview on the higher education system in which the agency under review operates is important in order to appreciate national peculiarities

6

Lessons learned

Take your time! • Timetable of the site-visit should provide for sufficient

time to reflect and discuss findings of the interviews and to work on the draft report

It’s all about communication• useful to have a preparatory telephone conference, if

it is not possible to meet in advance• ENQA telephone conference was helpful in addition to

written statements of the ENQA Board (clarifying issues)

Drafting the report• Methodological structure of “description, analysis and

conclusion” is very useful

7

Lessons learned

Findings and experiences of RT should be made available to other expert panels and be taken into account for improvement of the ENQA briefing package

• ENQA has already reworked the briefing package (especially on how to interpret the ESG)

One should keep in mind that although the whole undertaking is about ENQA membership, it should be aimed at improvement in character and not control

• guiding “principle of trust” Important to distinguish between aspects which are

under the control of the agency and those which are due to national legislation

• should be clearly distinguished in the review report

8

ESG, their nature and interpretation

Scope of the ESG for the review• “The membership provisions of ENQA correspond to part 3 of

the ESG incorporating also part 2, in Criterion 1 of the membership provisions.” (ENQA Briefing Package)

How to interpret the ESG?• Principle vs. rules based approach• Respecting the diversity of national contexts• Generic principle vs. specific requirement• One has to be cautious in adapting them:

- ESG 3.7: student involvement (“as appropriate”)- ESG 3.6: “Independence”: has implications for the areas of: financing,

composition of the agency, conflict of interest, QA outcomes, reports, etc.- Publication policies

Clear interpretation of the ESG by the ENQA Board should be communicated to the RT• however, a certain scope for interpretation will always remain

9

ESG, their interpretation and improvement

Question of compliance?• Provide the RT with a notion of what

- compliance- substantial compliance- partial compliancemeans.“Any areas of total non-compliance were unacceptable, and that some

element of compliance would be necessary against all criteria in order to provide an overall judgment based on ‘substantial’ compliance. Each case would be examined on its merits and at a local basis.” (ENQA Board)

Reflexive handling of the ESG Self critical, evaluative approach also necessary for the

development and improvement of the ESG as well as their interpretation

Thank you very much for your attention!