32
Thomas Blanc de la Carrere & Mark Frederiks JOQAR Observation Missions: Evaluation Report
Thomas Blanc de la Carrere & Mark Frederiks
JOQAR Observation Missions:
Evaluation Report
!
Copyright!©!2013!by!the!European Consortium for Accreditation in higher education!
ECA OCCASIONAL PAPER ISBN/EAN: 978-94-90815-08-0
Authors: Thomas!Blanc!de!la!Carrere!&!Mark!Frederiks!
!
All!rights!reserved.!This!information!may!be!used!freely!and!copied!for!!
non9commercial!purposes,!provided!that!the!source!!is!acknowledged!!
(©!European!Consortium!for!Accreditation).!!
Additional!copies!of!this!publication!are!available!via!www.ecahe.eu!
2
JOQAR Observation Missions:
Evaluation Report October 2013
1
4
Table of contents
1. Introduction .............................................................................................................. 6
2. Lessons learned ....................................................................................................... 10
2.1 Introduction ......................................................................................................... 10
2.2 Relevance of the observation missions’ framework ............................................ 11
2.3 Observation report ............................................................................................... 11
2.4 Observation process ............................................................................................. 12
2.5 Interactions .......................................................................................................... 13
2.6 Content of MULTRA and admission procedure.................................................... 14
3. Conclusions and recommendations .......................................................................... 16
Annex 1. MULTRA text .................................................................................................. 20
Annex 2. MULTRA Admission procedure........................................................................ 23
Annex 3. Outline Observation Report ............................................................................ 24
Annex 4. Evaluation forms ............................................................................................ 28
JOQAR observation missions – Evaluation form for agencies .......................................... 28
JOQAR observation mission – Evaluation form for Observers ......................................... 29
5
6
1. Introduction
ince the late 1990’s there has been a substantial increase in the number of higher
education programmes, which are being offered jointly by consortia of higher
education institutions in different European countries. In 2009 the number of joint
programmes was estimated to be around 2,5001. At a recent meeting the European
Commission gave the estimate that there are 4,000 joint programmes in 2013. While both
the educational and political importance of joint programmes in the Bologna process is
indisputable, the implementation of joint programmes is still hampered by serious
problems, especially in the domain of recognition and quality assurance (QA) of such
programmes.
The European Consortium of Accreditation in higher education (ECA)2 has been working in
recent years on issues concerning the QA and recognition of joint programmes in the
TEAM II project (2008-2010)3 and the JOQAR project (2010-2013). JOQAR is the acronym
for Joint programmes: Quality Assurance and Recognition of degrees awarded.4 Before
these projects started the ECA members agreed in June 2007 on principles for
accreditation procedures regarding joint programmes5. The TEAM II and JOQAR projects
were funded by the European Commission and involved not only QA agencies but also
ENIC-NARICs as project partners. This report is an outcome of the JOQAR project and will
particularly focus on the mutual recognition of accreditation of joint programmes. The
recognition of joint degrees is of course also an important matter, and for the results we
refer to two other JOQAR reports6.
1 http://www.ehea.info/Uploads/Documents/Stocktaking_report_2009_FINAL.pdf 2 http://www.ecahe.eu 3 Transparent European Accreditation decisions & Mutual recognition agreements part 2 (TEAM II), see:
http://ecahe.eu/w/index.php/TEAM_II 4 For a description and outcomes of the JOQAR project see: http://ecahe.eu/w/index.php/JOQAR 5 http://ecahe.eu/home/about/eca-documents/main-documents-for-mutual-recognition/ 6 In collaboration with ENIC-NARICs a ECA report has been published dealing with questions like: How to award
a joint degree in such a way that it is recognised? How can the design of the joint degree and the Diploma Supplement facilitate access to the labour market? See Axel Aerden & Hanna Reczulska, “Guidelines for
S
7
Since its foundation ECA has been aiming at mutual recognition of accreditation and QA
results and decisions. A first step towards reaching this goal was the signing of twelve
bilateral mutual recognition agreements between ECA members in 2007. Broadening the
scope of mutual recognition is especially important for joint programmes, which normally
need to undergo several national accreditation procedures. With a Multilateral Agreement
on the Mutual Recognition of Accreditation Results regarding Joint Programmes
(MULTRA7) in force, however, these multiple procedures can be replaced by one single
procedure. This MULTRA was first signed on 14 December 2010 by six agencies. The
purpose of the MULTRA is to simplify the accreditation of joint programmes and to provide
an efficient way to expand mutual recognition to more countries.
The core of the MULTRA agreement is that:
“The signing accreditation organisations agree to apply the ECA principles for accreditation procedures regarding joint programmes; and confirm that within their competences they accept the results of the accreditation procedures of the other signing accreditation organisations when accrediting joint programmes”.
For the full text of the MULTRA we refer to Annex 1. Agencies wishing to join the MULTRA
have to fulfil the requirements of the admission procedure (see Annex 2). The
Management Group of ECA acts as the coordination point for the admission procedure. It
checks whether the applicant agency fulfils the admission requirements (particularly
whether an external review against the ESG criteria, the ECA Code of Good Practice or
equivalent international standards has been carried out), and selects two observers (who
are members of a MULTRA signatory agency) to carry out an on-site-observation. The
observation of the procedure results in an observation report written by the observers
according to the agreed format8. The Management Group than reads the observation
Good Practice for Awarding Joint Degrees”, 2012, ECA, The Hague. Available at: http://ecahe.eu/home/services/publications/
7 http://ecahe.eu/home/about/eca-documents/mutual-recognition-agreements/ 8 Please refer to Annex 3
8
report and proposes whether the applicant agency should be admitted to MULTRA.
Consensus of all signatory agencies is needed for admission of new members.
The MULTRA stands for a high level of trust between accreditation agencies. This trust
between the partners is based on evidence gained through observations of procedures
amongst the MULTRA agencies. The observation is not meant to repeat the external
evaluation of an agency, but aims to gain mutual trust through observing accreditation
practice. The observation should provide evidence that the accreditation procedures and
standards are free of significant differences from those of MULTRA agencies and that the
results of accreditation procedures of joint programmes can thus be accepted by MULTRA
agencies.
All seven JOQAR partners who are QA agencies and not yet included in MULTRA were given
the opportunity to undergo an observation mission. Five of these seven agencies indeed
decided for an observation mission. Three agencies (EVA, SINAES and ZEvA) also indicated
their willingness to join MULTRA and be subject of an observation. Five agencies had
completed an external review against the ESG (and were ENQA full member and listed in
EQAR). Two agencies (CNA and SINAES) had an external review against the INQAAHE
Guidelines for Good Practice (INQAAHE GGP) whilst SQAA decided to undergo a ESG
external review set up by ECA in conjunction with an observation of a procedure. The
following observation missions took place from 2011 to 2013:
x CNA (Colombia)
x SINAES (Costa Rica)
x EVA (Denmark)
x AQAS (Germany)
x FIBAA (Germany)
x ZEvA (Germany)
x SQAA (Slovenia)
x AQU Catalunya (Spain).
9
The observation reports are publicly available on the ECA website9. In all eight cases the
observations have led to a positive result and six agencies signed the MULTRA. SQAA will
formally sign MULTRA in December 2013 and SINAES will sign in February 2014.
The observations were evaluated through carrying out a survey among the agencies and
the observers having participated in the observation missions. For the questionnaires we
refer to Annex 4. Based on this valuable input and the practical experience gained during
the missions, we will critically assess and formulate recommendations regarding the:
x Observation framework and procedure
x Relevance of the observation report template
x Functioning of the process of the observation procedure, including the site-visit
x Interactions between the agencies, the observers and the panels
x Content of MULTRA
x Admission procedure to join MULTRA
9 http://ecahe.eu/w/index.php/JOQAR - Achievements
10
2. Lessons learned
2.1 Introduction
he observation missions constituted the basis for decision-making on admitting
new agencies to the Multilateral Agreement on the Mutual Recognition of
Accreditation Results regarding Joint Programmes (MULTRA). The intention was
to observe an accreditation procedure of a joint programme but this could not happen in
all cases as there were not always joint programme accreditation procedures occurring
during the time of observation. In those cases a regular programme accreditation was
observed as this was a “proxy” to how an accreditation of a joint programme would be
conducted by the agency.
Eight observation missions were carried out, each by two persons associated with two
agencies that had previously joined the MULTRA. An observation format was drafted and
used for carrying out the observation. The format relates to standards for good practices in
accreditation (the ECA Code of Good Practice) and includes a comparison of accreditation
frameworks on essential points, as well as information on the procedure, site visit and the
how the achievement of learning outcomes is assessed. Finally, the observers were asked
to give their conclusions in the observation report on the question whether the observed
agency could be admitted to MULTRA.
Thus, the observation missions consisted of: the observation format, the observation of
the procedure (site visit), and a short observation report based on the format which was
finalised by the observers after comments from the observed agency. The observation
reports were discussed by the ECA Management Group and after a positive
recommendation forwarded to the MULTRA signatories for decision-making on admittance
of the observed agency to the MULTRA.
In the following sections of this evaluation report, we are going to examine the functioning
of these elements and the relevance of the supporting documents in the light of the
T
11
experience gained during the observation missions and the feedback collected through an
ad hoc anonymous survey among the participating agencies and observers. In the next
chapter we will formulate some recommendations.
2.2 Relevance of the observation missions’ framework
he observers agree unanimously that the observation missions’ framework is
appropriate to define whether an agency can join MULTRA or not. The "on site"
observation also showed to be the appropriate format. One observer suggested
to devote more time to the site visit to gather more in-depth information. Although this
was not a problem now, in borderline cases the time available could be too short.
According to the observers, the standards of the ECA Code of Good Practice and the other
criteria in the observation report template are sufficient to provide evidence that the
accreditation procedures and standards of the observed agencies are free of significant
differences from those of MULTRA agencies. It is suggested however to introduce some
adjustments in order to compare procedures and methodologies whilst evaluating
agencies from outside the European Framework. In those cases the standards should
consider not only the compliance with the ESG, but also the reliability of the agency and its
potential to meet the ESG. A better documentation on the national context of the applying
agency and on the notion of “joint programme” would be useful.
2.3 Observation report
he observers agree that the observation template report is a valuable tool for the
report writing, but also for the preparatory work prior to the site visit. The
observers recommended no improvements.
According to the observed agencies the observation report added value to internal
discussions of the agencies regarding the development and quality of their procedures.
Some examples of these discussions in the agencies that took place after the observations:
Agency 1. "Regarding the initial accreditation of joint programmes the observation
mission helped to highlight the additional aspects to be considered while
assessing a joint programme vs. assessing a “regular” programme."
T
T
12
Agency 2. " it would be important for [us] to include more international peers during
the visits. This is one of our main tasks this year. A second element is
linked to the way learning outcomes are evaluated."
Agency 3. "The lack of students and site visits in accreditation of new programmes
were discussed."
Agency 4. "Extend publication of accreditation results from decisions only to full
evaluation reports. Strengthen dialogue and advisory activities with the
stakeholders. Shorten the period of re-accreditation to 5 years. Combine
institutional and programme evaluation."
2.4 Observation process
f the analysis of the observed agency procedure was not considered as problematic by
the observers, the understanding of the local system prior to the analysis was in some
cases an issue.
Observers expressed a need for more thorough explanations of general terms, methods
and procedures followed by the agency to be observed. This could be supported with an
online collaborative platform to share documents (methodology, agendas, practicalities,
contacts, drafts reports…).
To provide background information, some of the observed agencies used pre-existing
documents written for different procedures, e.g. a self-evaluation report for the external
review by ENQA. An observer argued that this background information should be purpose
driven for the MULTRA membership. The aim should solely be to provide information for
the observers on the concrete procedure and the framework in which this takes place.
Since two observers participate in each mission, an observer wished for a division of tasks
between the two observers (report writing, specific observation area ...).
As a general rule the observers had no difficulty to formulate the final recommendation as
to admitting the agency to MULTRA.
The observed agencies mentioned the following desired improvements of the observation
process:
- Start the organisation of the observation more in advance;
- Allow observers more time to get familiar with local frameworks and procedures;
I
13
- Be more strict regarding the fluency in English of the observers and, if possible,
make sure that the observers understand the local language.
2.5 Interactions
n the whole all interactions proved to be mutually fruitful and went smoothly in
a climate of mutual trust and constructive exchange.
Interaction Observers / Panels: In the perception of the observers, the panels were not
“disturbed” by their presence and were following well-established procedures that were
in no way modified to “please” the observers. The assessments rules of the observed
agencies were clear to the observers and overall were applied appropriately by the panel.
Interaction Observers / Agency: Overall the communication with the agencies was
efficient and the observers received quick answers. An observer particularly appreciated
the fact that he did not have to use a third language (English) and the conversations could
take place in the language of the observed agency. Here we see an implicit suggestion to
select, if possible, observers mastering the language of the country to reduce risks of
communication difficulties.
The agencies were in general impressed by the knowledge and the deep understanding of
their procedures that the observers demonstrated. The agencies appreciated the
opportunity to share with the observers their experiences and points of view about the
quality of higher education. Although no doubt existed - in the perception of the agencies -
as to the overall competence, professionalism and the expertise of the observers, the
language barrier could be an issue. Communication was not easy in one case, where none
of the participants was a fluent speaker of English. This agency representative reported
that the observers did not seem to be familiar enough with the agency's tasks and
functions despite background information provided in advance in oral and written form.
Interaction Observed Agency / Coordination Point: The observed agencies reported they
had received valuable support and explanations from the Coordination Point (a role
performed by the ECA Secretariat). The fact that some of the observed agencies are
O
14
involved in different ECA activities also facilitated communication and the understanding
of the philosophical underpinnings of the tasks to be carried out. One agency reported
some difficulties due to insufficient internal communication within the agency and another
agency indicated that the response time to their queries could be improved.
2.6 Content of MULTRA and admission procedure
he text of the MULTRA itself, including the admission procedure, was not
evaluated through the survey as this is not so much of concern to observers or
observed agencies but it is of relevance to the MULTRA signatory agencies. In
Working Group 1 of ECA (in which several MULTRA agencies participate), therefore, these
topics were briefly discussed.
In addition, the following two statements within the MULTRA need to receive attention
from the MULTRA signatory agencies:
“on the condition that the signing agencies continue to exchange information about their accreditation systems on a regular basis. Substantial changes of the accreditation systems should be communicated without delay.”
In some cases substantial changes in accreditation systems have happened but these were
neither widely nor formally shared between the MULTRA agencies. As the effectiveness of
any agreement ultimately depends on the trust between the partners it would be
recommendable to devise a procedure for regular (e.g. annual) updating by agencies filling
in a small questionnaire if changes in the system have occurred.
“The agreement is valid for three years. After a re-evaluation of the terms of the agreement it can be extended by consent of all parties.”
As the MULTRA was originally signed on 14 December 2010 this implies that a re-
evaluation of the terms and consensus on the extension of the agreement should take
place before the end of 2013. Therefore, a survey will be held among MULTRA agencies to
carry out this evaluation and seek agreement on the extension of MULTRA. Through this
T
15
survey a revision of the admission procedure could also be discussed. As the JOQAR project
funding comes to an end, future observations will have to be paid for differently. The
admission procedure requires two observers and states that “as a rule the costs of the
observation (travel and accommodation) are covered by the interested agency”. This
means that joining MULTRA will lead to significant costs, which could in particular be a
problem for agencies outside of Europe, where travel costs are also higher. A solution
could be to continue with two observers (as subjectivity could be high with only one
observer) but to reduce costs by letting only one observer go to the site visit. The other
observer stays at home, concentrating on reading the framework and other materials, and
providing feedback to the other observer, particularly about the observation report.
16
3. Conclusions and recommendations
s a consequence of the TEAM II project a multilateral mutual recognition
agreement regarding the accreditation results of joint programmes (MULTRA)
was set up as part of the JOQAR project. The essence of MULTRA is that each of
the signatory agencies accepts the accreditation results regarding joint programmes of
other signatory agencies. As a consequence, only one accreditation procedure for a joint
programme is needed and other agencies can base their own national decisions on the
results of this procedure without setting up their own national procedures. So where in
the pilots procedures each time a consultation with all involved agencies had to take place
to maximize chances of accepting the results of the single accreditation procedures,
MULTRA does not require this anymore as the signatory agencies have already agreed to
accept the results of a procedure by one of the other agencies.
The MULTRA was first signed on 14 December 2010 and has been expanded with new
agencies ever since. In the JOQAR project eight new agencies, including two in Latin
America, indicated their willingness to join MULTRA. These agencies were all reviewed
externally, as is the first admission requirement for MULTRA. The external review mostly
happened against the Standards and Guidelines for Quality Assurance in the European
Higher Education Area (ESG). However, an external review against criteria which are
comparable to the ESG is also possible, e.g. the two Latin American agencies had been
reviewed against the Guidelines of Good Practice of the global network of QA agencies
INQAAHE10.
The second admission requirement for MULTRA is that the applicant agency has to
undergo an observation. This means that two observers from two agencies that have
already signed MULTRA observe a procedure of the applicant agency and write an
observation report. This observation report forms the basis for the decision-making by the
10 http://www.inqaahe.org/main/professional-development/guidelines-of-good-practice-51
A
17
MULTRA signatories on the admittance of the new agency. This procedure ensures that not
only formal requirements, as in the ESG, are fulfilled but that there is also evidence on how
the agency conducts a procedure in practice. The likelihood of acceptance of decisions of
other agencies is very much increased because the MULTRA procedure actively involves
agencies in expressing their trust in other agencies. The observations were evaluated
positively and led to some minor recommendations on improving the observation
procedure:
x Some adjustments in the observation report format in order to compare
procedures and methodologies whilst evaluating agencies from outside the
European Framework
x Allocate more time for preparing the observation and especially for observers to
get familiar with local frameworks and procedures (including e.g. documentation
on online platform)
x Be more strict regarding the fluency in English of the observers and, if possible,
make sure that the observers understand the local language.
As an re-evaluation and agreement on extension of MULTRA is due before the end of 2013
it is recommended to organise a survey of the MULTRA signatory agencies. Through this
survey the experiences of the agencies with MULTRA, and also the admission procedure
(measures to reduce the costs of observations) and the information exchange in case of
major changes in the system could be discussed.
As a result of the JOQAR project the MULTRA consists of agencies in Austria, Colombia,
Costa Rica11, Denmark, France, Germany, the Netherlands and Flanders, Poland, Slovenia12,
and Spain. There is an interest from agencies in other countries to join MULTRA. It would
indeed be desirable if more agencies join MULTRA; this makes the acceptance of the
results of accreditation procedures of other agencies a lot easier. Signing the MULTRA is in
line with the aim of the Ministers in the Bucharest Communiqué “to recognise quality
assurance decisions of EQAR-registered agencies on joint and double degree
11 The signing will take place in February 2014. 12 The signing will take place in December 2013.
18
programmes”13. MULTRA even enhances the ambition level of the Ministers because it is
also open for non-European agencies that are not listed in EQAR but have been reviewed
against international QA standards comparable to the ESG. As an observation of a real life
procedure is a requirement for admittance to MULTRA, it obtains the trust among agencies
that is needed for realising the recognition of results or decisions made by other agencies.
By giving their support to MULTRA the Ministers can reach their aims regarding the QA and
accreditation of joint programmes without intruding on national sovereignty. At the same
time MULTRA opens the windows to countries outside of Europe which is an aim of the
Bologna process. This international cooperation is also essential for joint programmes
which do not stop at European borders.
13 EHEA Ministerial Conference “Making the Most of Our Potential: Consolidating the European Higher
Education Area. Bucharest Communiqué”, Bucharest 2012: p. 2.
19
20
Annex 1. MULTRA text
Multilateral Agreement on the Mutual Recognition of Accreditation Results
regarding Joint Programmes (MULTRA)
Introduction
The European Consortium for Accreditation in higher education (ECA) was founded in 2003
and consisted originally of 12 accreditation organisations from 8 countries. Since then, ECA
was enlarged to 17 organisations from 11 European countries.
ECA acts as a project organisation aiming at mutual recognition of each other’s
accreditation and quality assurance results. A first step towards reaching this goal was the
signing of twelve bilateral mutual recognition agreements between ECA members in 2007.
Broadening the scope of mutual recognition is especially important for joint programmes
which normally need to undergo several national accreditation procedures. With a
multilateral agreement on mutual recognition (MULTRA) in force, however, these multiple
procedures can be replaced by one single procedure.
The MULTRA stands for a high level of trust between accreditation agencies. This trust
between the partners is based on evidence gained through intense cooperation and
observations of procedures amongst the MULTRA agencies.
Agencies that wish to join the MULTRA have to fulfil the requirements of the admission
procedure (Annex 1). The Management Group of ECA shall coordinate the admission
procedure, organise the observations according to the agreed format (Annex 2), and
provide a recommendation on admission to the agencies that are already part of the
MULTRA. The agencies that have signed the MULTRA decide on the admission. A new
agency is only admitted if all agencies that have signed the MULTRA approve the
admission.
21
Preamble
Aiming to contribute to the internationalisation of higher education and to enhance the
mobility of students and graduates in Europe;
with the purpose to simplify the accreditation and recognition of joint programmes and
degrees awarded and to provide an efficient way to expand mutual recognition to more
European Higher Education Area countries;
with the additional purpose to strengthen the commitment of agencies to collaborate and
to reach up to high quality standards
convinced that the multilateral recognition agreement between accreditation
organisations enhance transparency in the European Higher Education Area and provide
important information on the quality of foreign institutions and programmes for students,
higher education institutions and the labour market;
building on the cooperation with ENIC-NARIC centres regarding the recognition of
qualifications awarded by joint programmes;
relying on the work of ECA since 2003 to build up mutual trust between the member
organisations;
taking into account the already signed bilateral mutual recognition agreements or the
positive external evaluation results of the signing agencies;
acknowledging the variety of national higher education systems and accepting the
existence of different legal prerequisites for accreditation [and quality assurance]
procedures, standards and decisions.
22
Agreement
The signing accreditation organisations agree to apply the ECA principles for accreditation
procedures regarding joint programmes;
and confirm that within their competences they accept the results of the accreditation
procedures of the other signing accreditation organisations when accrediting joint
programmes;
on the condition that the signing agencies continue to exchange information about their
accreditation systems on a regular basis. Substantial changes of the accreditation systems
should be communicated without delay.
The signing accreditation organisations agree to give each other access to all relevant
documents relating to the accreditation results. All documents must be treated
confidentially until they are published.
The terms of the Agreement may be modified at any time upon the consent of all parties.
Consensus of all signatory parties is needed for admission of new members.
Any of the signatory parties may denounce this agreement by written notification to the
other parties at any time. The written notification must include the reasons for and the
date of the termination of this agreement.
The agreement is valid for three years. After a re-evaluation of the terms of the agreement
it can be extended by consent of all parties.
As a prerequisite to signing this agreement the interested agency must have a valid
bilateral mutual recognition agreement with at least one of the signatory parties or must
undergo the admission procedure described in Annex 1 based on an observation format
(Annex 2).
Signatures
The following undersigning ECA member organisations join this Multilateral Agreement on
the Mutual Recognition of Accreditation Results regarding Joint Programmes:
23
Annex 2. MULTRA Admission procedure
The admission procedure can be further developed and amended during the JOQAR
project that will test and evaluate the steps of the procedure.
The ECA Management Group (MG) should function as “coordination point” for the
admission procedure.
The requirement to sign the agreement is:
a) The interested agency must have a valid bilateral recognition agreement
or
b) The interested agency must have a recent (not older than 5 years) external
evaluation against the ESG criteria, the ECA Code of Good Practice or against a set
of standards that can be considered as equivalent.
The ECA MG decides on the fulfillment of this criterion by checking the submitted
documents. If this requirement is met then the ECA MG nominates two observers (who are
members of a MULTRA agency) to carry out an on-site-observation. One observer should
join a site-visit of a programme accreditation procedure, the other one should get evidence
of the accreditation practice through discussions with the agency’s representatives. The
observers use an observation format (Annex 2) and should conclude their report with a
recommendation. The MG formally checks the observation report and forwards the report
including a decision proposal to the MULTRA members.
As a rule the costs of the observation (travel and accommodation) are covered by the
interested agency.
Consensus of all signatory parties is needed for admission of new members.
24
Annex 3. Outline Observation Report
As taken from the Observation report template.
Framework of the procedure
In which framework did the procedure take place? This section should address the elements listed below on order to provide an in-depth understanding of the accreditation practice of the observed agency. 1. Structure of accreditation framework (including relevant documents, e.g. legislation,
...) 2. Accreditation standards 3. Additional requirements for the assessment of joint programmes 4. Focus of the accreditation procedure (e.g. input factors, internal quality assurance,
... ) 5. Assessment of achievement of learning outcomes 6. Enhancement strategies for institutions 7. Responsibility for accreditation procedures 8. Steps in the accreditation procedure 9. Assessment rules and decision scale when accreditation is granted (e.g. excellent,
insufficient; conditions, ...) 10. Decision-making process of the agency (rules and responsibility) 11. Period of accreditation 12. Appeal system 13. Publication policy 14. Average number of procedures per year
<Your observations regarding the framework of the procedure>
25
Site visit
1. The expert panel
This section of the observation report is based on the ECA Principles for the Selection of Experts. Here the observations concerning the selection of the expert panel are presented.
Number of panel members: <Total amount>
Gender balance <Comment on the gender balance>
EXPERTISE INCLUDED IN THE PANEL OF THE OBSERVED PROCEDURE
EXPERTISE INCLUDED x experience in quality assurance in higher education <Yes/No> x appropriate academic qualifications and scientific or professional
reputation in the relevant area(s)
x relevant international experience that provides a basis for making international comparisons
x knowledge on teaching and learning methods x expertise in development, design, provision and evaluation of higher
education programmes
x knowledge of the country-specific system of higher education, institutions and applicable legislation
x student representatives in the respective area(s) x representatives from the labour market x a significant proportion of panel members from outside the country
<Your observations regarding the panel composition>
2. The procedure
This section of the observation report is mainly based on the ECA Code of Good Practice. Here the observations concerning the ECA standards relating to the accreditation procedure and standards are presented.
<Your observations regarding the procedure>
26
3. Learning Outcomes
How and by what means is the assessment of achieved learning outcomes taken into account during the procedure?
<Your observations regarding the learning outcomes>
4. ECA Code of Good Practice: standard 14
Standard The accreditation procedures must include self-documentation/-evaluation by the
higher education institution and external review (as a rule on site)
Question - How is the accreditation procedure structured?
Reference points - Self-documentation/-evaluation and external review are part of the accreditation procedure
- External reviews encompass on site visits at the higher education institutions - The external review team is instructed clearly about its tasks - The accreditation organisation provides specific regulations in case of ex ante-
accreditations
<Your observations regarding compliance with this standard>
5. ECA Code of Good Practice: standard 15
Standard The accreditation procedures must guarantee the independence and competence of
the external panels or teams
Question - How is the independence of external panels guaranteed? - Are selection criteria for expert panels set up?
Reference points - Selection criteria for external panels/expert committees are set up and published by the accreditation organisation.
- Selection criteria assure competence and independence of external experts - Independence of the experts is assured by a written statement - The decision about the composition of the expert team is made by the
accreditation organisation in a transparent way
<Your observations regarding compliance with this standard>
6. ECA Code of Good Practice: standard 16
Standard The accreditation procedures must be geared at enhancement of quality
Question - Which elements and mechanisms within the accreditation process are used to enhance quality at the higher education institution?
27
Reference points - The accreditation process contains elements that promote quality development and improvement of the higher education institution
- The accreditation process should respect autonomy, identity and integrity of the higher education institutions
<Your observations regarding compliance with this standard>
7. ECA Code of Good Practice: standard 17
Standard The accreditation standards must be made public and comply with European
practices taking into account the development of agreed sets of quality
standards
Questions - Which are the quality standards and criteria used for accreditation procedures? - Do they meet international standards?
Reference points - The quality standards and criteria used in the accreditation procedures correspond to European good practices
- The quality standards and criteria are made public - The process of formulation of the quality standards and criteria is transparent
and involves all important stakeholders
<Your observations regarding compliance with this standard>
28
Annex 4. Evaluation forms
JOQAR observation missions
– Evaluation form for agencies –
Please insert your answers in the corresponding frames below
1. Have you received adequate support / explanations from the JOQAR project?
2. What is your experience of communication with the observers? In your perception, did the observers have sufficient knowledge and experience with regard to the understanding of your agency procedures?
3. Did the observation report add any value to your internal discussions about the development and quality of your procedures? Can you give examples?
4. Did you receive any feedback from the panel? If yes, please indicate.
5. Would you deem any other elements helpful to prepare the observation missions? Or do you have any other suggestions for improvement of the procedure?
29
JOQAR observation mission
– Evaluation form for Observers –
Please insert your answers in the corresponding frames below
1. Is the observation missions’ framework appropriate to define whether an agency can join MULTRA?
2. Are there any missing criteria or standards in the framework? Do we need to include additional elements to provide evidence that the accreditation procedures and standards are free of significant differences from those of MULTRA agencies?
3. Was the observation report template useful for the report writing?
4. Did the observers have the impression the observed panel was in any way “prepared” by the agency for the observation procedure?
5. Was it in any way problematic to perform comparative analysis of the agency procedures?
6. Were the agency assessments rules clear and used appropriately by the observed panel?
30
7. Was it difficult for you in any way to formulate the final recommendation as to admitting the agency to MULTRA?
8. Did you find the communication with the agency efficient?
9. Do you have any suggestions for other improvements of the procedure?
www.ecahe.eu
