György BAZSA President, HAC ENQA

György BAZSA President, HAC

ENQADebriefing/brainstorming meeting for the

training of experts for agency reviews

Barcelona, 17 March 2009

A possible option: grading scheme for judging

compliance with criteria

Basic quantitative requirements - of a fine wine:

ethanol 9 – 13 %sugar 0 – 100 g/lpH 3 – 4

- of a nice hostess:bust size 80 – 99 cmhip size 60 – 65 cmwaistline 75 – 90 cm

- of an excellent scientist:publications minimum 50citations minimum 200impact factor minimum 40

But still there is a need for the evaluation of its/her/his quality by experts!

Now, what about a good quality assurance agency?

i. Membership

REGULATIONS OF ENQA(as from 26 September 2008)

TITLE III. INVOLVEMENT IN ENQA

a) Full Membership

Full Membership of ENQA is open to quality assurance agencies that meet the relevant criteria described in Annex I Membership Provisions document.

b) Candidate Membership

Candidate Membership is available to quality assurance agencies that meet the criteria described in Annex I Membership Provisions document.

ANNEX 1 – Membership Provisions

CHAPTER I. CRITERIA FOR FULL MEMBERSHIP

a) Full Membership of ENQA is open to quality assurance agencies in the field of higher education from EHEA member states that have been operating and conducting actual evaluation activities for at least two years.

Before being accepted as a Full Member, an applicant agency must satisfy the Board that it meets the eight criteria, listed below. The applicant agency will thereby also meet the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). - (8 or 15 criteria?)

b) An applicant agency may apply for Candidate Membership rather than Full Membership in the first instance. The Board will grant such member-ship if it believes

- that the applicant demonstrates, through its application, substantial compliance with a majority of the criteria and further believes

- that the applicant will be able to demonstrate, through an acceptable external review submitted within no more than two years Candidate Membership, full compliance with the criteria for Full Membership.

QAAdocuments QAAexp. report

Candidate m. Full memberFull member

ENQA

ENQA

non non

non non

ii. Evaluation of the compliances by the expert panel

Dictionary and stages of compliance

Compliant: in accordance with a specified body or rules (usually used in combination):

Fully (compliant): entirely or wholly

Substantially (compliant): of ample or considerable amount, quantity, size, etc.

Partially or partly (compliant): in part, to some extent or degree, not wholly; being such in part only, not total or general, incomplete

Non-compliant: failure or refusal to comply, as with a law, regulation, or term of a contract.

ENQA National reviews (quide):

summarised conclusion stating whether the agency is

- fully or substantially compliant;

- partially compliant; or

- non-compliant.

ESU

The study visit team believes that objectivity is best served by using a three level marking system.

- Compliance, where QA implements the ESG in an effective manner;

- Partial compliance, where QA practices are based on the correct interpretation of the ESG, positive aspects have been identified but the manner of implementation is not effective enough.

- No compliance, where ARACIS practices are based on misinterpretations of the ESG …;

EUA

- fully compliant

- substantially compliant

- partly compliant

- (non-compliant)

Summary (in the case of ARACIS): substantially compliant

March 2009!: Guidelines for external reviews of ENQA member and applicant agencies- fully compliant: the Agency adheres to the letter

(literal interpretation) and spirit (intent) of thecriterion/ESG standard;

- substantially compliant: if the criterion/ESGstandard is not fulfilled to the letter, the Agency might still be considered substantially compliant if the principle/spirit of the stipulated criterion/ESG standard is followed in practice;

- partially compliant: the Agency does not fully or substantially comply with the criterion/ESG standard;

- non-compliant: the Agency does not comply with the criterion/ESG standard.

The way in which the membership criteria/ESG are to be interpreted by the review panel and the Board

A possible option: grading scheme for judging compliance with different criteria

EvaluationEvaluation GradingGrading

FullyFully compliant compliant + + ++ + +

SubstantiallySubstantially compliant compliant + + / -+ + / -

PartiallyPartially compliant compliant + / - - + / - -

NonNon-compliant-compliant / / - - -- - -

The grading is proposed by the review panel

„The Review panel will need to exercise its judgement, in the light of all the evidence before it, and the intentions of the ENQA membership criteria / ESG, as to whether substantial compliance is being achieved, taking account of the circumstances in which the Agency is required to operate (see section 3.6.2). There is little point in adopting a ‘hard line’ position in respect of compliance with the ESG if, by doing so, trustworthy and credible agencies are prevented from gaining Full membership of ENQA through no fault of their own.”

„There may, as a result, be occasions where a strict reading of the ENQA membership criteria / ESG would indicate substantial rather than full compliance with one or more standards, but the panel may come to the conclusion that this should not jeopardise an overall judgement of compliance. This may occur when national legislation requires a different approach, or where an alternative way of achieving an objective is used, or because of other justifiable reasons.”

„Where an agency is found to be either partially compliant (+ - -) or non-compliant (- - -) with a criterion, the reason for this should be explained. Full or substantial compliance may be impossible for some agencies, owing to restrictions placed on them by the very nature of their work and/or legislation in place in their country(ies) of operation. When considering such cases, the ENQA Board will take mitigating circumstances such as these into account.”

A possible option for 15 criteria: grading scale for granting ENQA membership

Decided by the BoardProposed by the review panel

PointsPoints LimitLimit Overall Overall grading?grading?

Membership Membership grantedgranted

45 – 4145 – 411 partly c.1 partly c.

0 non-c.0 non-c.Fully ?Fully ? Full memberFull member ++





45 – 4145 – 411 partly c.1 partly c.


40-39-3840-39-38 Board decidesBoard decides





45 – 4145 – 411 partly c.1 partly c.



37-36-3537-36-352 partly c.2 partly c.

1 non-c.1 non-c.Substan-Substan-

tially ?tially ?Candidate Candidate membermember ++





45 – 4145 – 411 partly c.1 partly c.



37-36-3537-36-352 partly c.2 partly c.








45 – 4145 – 411 partly c.1 partly c.



37-36-3537-36-352 partly c.2 partly c.




31 31 →→ ------ Non Non Non-memberNon-member ++

PointFully 3+

Subst 2+

Partly 1

Non 0

Sum

Full member

41

12 2 1 0 41

11 4 0 0 41

41

13 1 0 1 !!! 41 13 0 2 !!! 0 41

Full or candidate member

403938

11 1 3 0 38

10 3 2 0 38

9 5 1 0 38

8 7 0 0 38

Candidate member

373635

11 1 1 2 36

10 2 2 1 36

9 4 1 1 36

8 5 2 0 36

7 8 !!! 0 0 37

PointFully 3+

Subst 2+

Partly 1

Non 0

Sum

Candidate or non-member

343332

10 0 2 3 !!! 32

9 1 3 2 32

8 3 2 2 32

8 2 4 1 32

7 5 1 2 32

7 3 5 0 32

6 6 2 0 32

Non-member31

8 3 1 ! 3 ! 31

8 2 3 ! 2 ! 31

8 1 5 ! 1 ! 31

iii. Further proposals

- renumbering of ENQA criteria

- public list of data

- publicity of ENQA evaluation

- QA vs. QP

1 1 Executive summary 2+ - Glossary3 2 Introduction (giving contextual information: reason(s) for the commis-

sioning of the review; the place of the agency in the quality assurance structure of its jurisdiction; the main functions; the engagement of the agency with the ESG);

4 3 Findings (each membership criterion should be discussed separately)4.1 3.1 ENQA criterion 11 1: Activities / ESG 3.1, 3.34.2 3.2 ENQA criterion 11 2: External qual. assurance proc./ESG 2 4.3 3.3 ENQA criterion 22 3: Official status / ESG 3.24.4 3.4 ENQA criterion 33 4: Resources / ESG 3.4 4.5 3.5 ENQA criterion 44 5: Mission statement / ESG 3.5 4.6 3.6 ENQA criterion 55 6: Independence / ESG 3.6

4.7 3.7ENQA criterion 66 7: External quality assurance criteria and processes used by the members / ESG 3.7

4.8 3.8 ENQA Criterion 77 8: Accountability procedures / ESG 3.8 5+ 4 Any sections relating to additional Terms of Reference of the review6 5 Conclusion and recommendations (?)7 6 Annexes (Key pieces of evidence – i.e. extracts from legislation,

policies and procedures etc. – may be added as appendices)

Further proposals (out of scope of the title):

ENQA should prepare a list of compulsory data and information about QAA to be posted permanently on member agency’s homepage (partly in English).

Application, self-evaluation report, date of site visit and composition of visiting panel should be posted on the QAA’s homepage like on ENQA homepage.

QA vs. QP: quality assurance is crucial, but quality of the „product” (learning outcomes etc.) is the main aim. Should we move to QP direction?

Thank you for your kind attention!

0.1 System of documents - ENQA Review of ARACIS 07.01.2009

date pages

1. Common documents of ENQA reviews

1.1 Briefing pack current 13

1.2 Regulations of ENQA (incl. membership provisions)* 26.09.2008 16

1.3 ESG version 03 by ENQA* 2007 38

1.4 ENQA organised external reviews 22.11.2005 3

1.5 Report of the panel: Introduction … current 2

1.6 Financial guidelines (ENQA coord. reviews) current 2

1.7 Scrutiny, external reviews for full membership current 12

1.8 National reviews guidelines 21.09.2006 8

2. Special ENQA-ARACIS documents of this review

2.1 Letter of ENQA to ARACIS 07.07.2008 1

2.2 Letter of ARACIS to ENQA 10.07.2008 1

2.3 ENQA nomination letter to Prof. Bazsa 18.12.2008 2

2.4 Agenda ARACIS – telephone briefing 2008 2

2.5 ARACIS review team CV-s 2008 11

2.6 Mihai Floroiu's visiting card 2008 1

2.7 Letter ENQA to ARACIS 100507 10.05.2007 2

2.8 Contract ARACIS ENQA 27.11.2008 3

2.9 Terms of reference ARACIS ENQA 27.11.2008 6

3. Romanian HE background documents (constraints)

3.1 A1 Ordinance 75 and Law 87 2005-2006 14

3.2 A9 Romanian education system 2008 4

3.3 Tables 1, 2, 3 on Romanian HE 2008 4

3.4 A2 CNEEA activity 1993-2004, statistics 1993-2004 1

4. ARACIS own documents for ENQA evaluation

4.1 Self evaluation report (SER) of ARACIS 08.12.2008 53

4.2 A3 Mission statement current 4

4.3 A4 Methology 2006 72

Evaluation Guides for accreditation of -

4.4 A5 Guide I Study programmes DD.11.2006 108

4.5 A5 Guide II Institutions DD.12.2006 65

4.6 A5 Guide III HEIs and study programmes DD.12.2006 39

4.7 A5 Guide IV Teaching staff current 44

4.8 A5 Guide V Distance education current 29

4.9 A6 Programmes evaluated, statistics 18.12.2008 1

4.10 A7 Action plan 2007-2009 for ARACIS internal QA current 15

4.11 A10 Code of ethics current 13

4.12 Specific aspects to be adressed by ARACIS current 8

4.13 ARACIS certificat 2007 1

5. Reports of earlier external reviews of ARACIS

5.1 A8 Monitoring Report of ARACIS 2007 ? 50

5.2 A11 EUA evaluation report - 2008 27.08.2008 36

5.3 A12 ESU evaluation report - 2208 DD.09.2008 33

5. Reports of earlier external reviews of ARACIS

5.1 A8 Monitoring Report of ARACIS 2007 ? 50

5.2 A11 EUA evaluation report - 2008 27.08.2008 36

5.3 A12 ESU evaluation report - 2208 DD.09.2008 33

6. Miscellaneous

6.1 Quality Assurance in Romania – ppt 28.06.2007 2

6.3Press releases of ARACIS reviews on Internet 2008 4

6.2 EDU-CER letter to EUA - ENQA 14.09.2008 5

6.4 ENQA letter to EDU-CER 24.09.2008 1

6.5 Informtaions about AIEQA and AEACE 2009 2

6.6 ARACIS letter on EDUCER 2009 3

6,7 English and Roman AECEA no activities 2009 1

6.8 AECEIS website 2009 1

6.9 GA ENQA former associates 2008 2

7. Panel's own documents

7.1 ENQA visit final programme 2009 1

7.2 Minutes telephone ARACIS review 2009 4

7.3 Mapping grid 2009 12

7.4 PhD in Romania 2004 11

7.5 Questions for ARACIS interviews 2009 1

7.6 Remarks JW 2009 3

7.7 further questions GB 2009 2

total pages: 809

ENQA Board decisions: (see Scrutiny form)

1. OAA’s first application for membership:

a) candidate membership granted for a maximum of 2 years (documentation or external report based ) or

b) full membership granted for 5 years (based an independent external review report only).


2. Candidate member’s application for a full membership – based on a full evaluation procedure:

- full membership for 5 years provided or

- neither full nor prolonged candidate membership granted, QAA debarred, but remains on the mailing list and has the possibility of a new membership application only after 2 years.


3. Full members’ mandatory cyclical review in each 5 year:

- full membership re-confirmed for another 5 years or

- candidate membership granted in order to conform with full membership criteria in 2 years or

- neither full, nor candidate membership is granted?

CHAPTER II. APPLICATION FOR MEMBERSHIP

Application form and documentation

Applications for membership of ENQA shall be in the form specified by the Board (details obtainable from the Secretary General). Applications for membership are considered and decided upon by the Board on the basis either of submitted documentation alone, or of submitted documentation and a visit to the applicant body.

Applications for Full Membership will only be consi-dered where an independent external review report on the agency’s conformity with the membership criteria, carried out in a manner and to a standard acceptable to the Board, is received.

If, after consideration of the review report by the Board, the Candidate Member is not, in the opinion of the Board, in full compliance with all the criteria, the application will lapse and the applicant will, by the decision of the Board, not be allowed to reapply for membership until a further period of two years has elapsed. During this period the agency will not be a Candidate Member but will remain on the ENQA mailing list to ensure information dissemination on the activities of ENQA.

ENQA co-ordinated reviews: Briefing pack for review panel members

The level of conformity with the ESG that is expected is ‘substantial compliance’, not rigid adherence.

There may, as a result, be occasions where a strict reading of the ESG would indicate substantial rather than total compliance on one or more standards, but the panel may come to the conclusion that this should not jeopardise an overall judgement of compliance. …

This does not mean that a clear failure to meet the standard or its intention will be condoned, and it is unlikely that more than one or two examples of ‘substantial’, rather than ‘total’ compliance will be acceptable to a review panel.

Date post:	21-Jan-2016
Category:	Documents
Upload:	bijan
View:	39 times
Download:	0 times

György BAZSA President, HAC ENQA

Documents