ENQA AGENCY REVIEW THE SWEDISH HIGHER EDUCATION … · 2021. 1. 12. · OTHER SOURCES USED BY THE...

ENQA AGENCY REVIEW

THE SWEDISH HIGHER EDUCATION AUTHORITY (UKÄ)

ENQA AGENCY REVIEW 2020

TUE VINTHER-JØRGENSEN, PATRICK VAN DEN BOSCH, MELITA KOVACEVIC,

GOHAR HOVHANNISYAN10 DECEMBER 2020

1/51

CONTENTSEXECUTIVE SUMMARY ........................................................................................................................................... 3

INTRODUCTION ..................................................................................................................................................... 4

BACKGROUND OF THE REVIEW AND OUTLINE OF THE REVIEW PROCESS ..................................... 4

Background of the review .................................................................................................................................. 4

Review process .................................................................................................................................................... 4

HIGHER EDUCATION AND QUALITY ASSURANCE SYSTEM OF THE AGENCY ................................. 6

Higher education system .................................................................................................................................... 6

Quality assurance ................................................................................................................................................ 7

UKÄ ......................................................................................................................................................................... 8

UKÄ’s organisation/structure............................................................................................................................. 8

UKÄ’s functions, activities, procedures ............................................................................................................ 8

UKÄ’s funding .................................................................................................................................................... 11

FINDINGS: COMPLIANCE OF UKÄ WITH THE STANDARDS AND GUIDELINES FOR QUALITY

ASSURANCE IN THE EUROPEAN HIGHER EDUCATION AREA (ESG) ............................................... 12

ESG PART 3: QUALITY ASSURANCE AGENCIES ..................................................................................... 12

ESG 3.1 Activities, policy, and processes for quality assurance.................................................................. 12

ESG 3.2 Official status ...................................................................................................................................... 14

ESG 3.3 Independence ..................................................................................................................................... 15

ESG 3.4 Thematic analysis ............................................................................................................................... 17

ESG 3.5 Resources ............................................................................................................................................ 18

ESG 3.6 Internal quality assurance and professional conduct .................................................................... 20

ESG 3.7 Cyclical external review of agencies ............................................................................................... 21

ESG PART 2: EXTERNAL QUALITY ASSURANCE .................................................................................... 22

ESG 2.1 Consideration of internal quality assurance................................................................................... 22

ESG 2.2 Designing methodologies fit for purpose ...................................................................................... 25

ESG 2.3 Implementing processes ................................................................................................................... 28

ESG 2.4 Peer-review experts .......................................................................................................................... 30

ESG 2.5 Criteria for outcomes ....................................................................................................................... 31

ESG 2.6 Reporting............................................................................................................................................. 33

ESG 2.7 Complaints and appeals .................................................................................................................... 34

ADDITIONAL OBSERVATION ........................................................................................................................... 36

CONCLUSION ......................................................................................................................................................... 37

SUMMARY OF COMMENDATIONS .............................................................................................................. 37

OVERVIEW OF JUDGEMENTS AND RECOMMENDATIONS ................................................................ 37

2/51

SUGGESTIONS FOR FURTHER DEVELOPMENT ..................................................................................... 38

ANNEXES .................................................................................................................................................................. 39

ANNEX 1: PROGRAMME OF THE SITE VISIT .......................................................................................... 39

ANNEX 2: TERMS OF REFERENCE OF THE REVIEW ........................................................................... 44

ANNEX 3: GLOSSARY ..................................................................................................................................... 49

ANNEX 4. DOCUMENTS TO SUPPORT THE REVIEW ......................................................................... 50

Documents provided by UKÄ ........................................................................................................................ 50

Other sources used by the review panel ...................................................................................................... 51

3/51

EXECUTIVE SUMMARY

This report analyses the compliance of the Swedish Higher Education Authority

(Universitetskanslersämbetet, UKÄ) with the Standards and Guidelines for Quality Assurance in the

European Higher Education Area (ESG). It is based on an external review conducted from January -

December 2020.

The online site visit took place from 29 September till 1 October 2020. The analysis of the self-

assessment report, the additional documents provided by UKÄ and the agency website, together with

interviews held during the site visit, provided evidence as to which extent UKÄ meets the ESG 2015

standards.

The following activities of UKÄ are addressed in the external review:

- Institutional reviews of the HEIs' quality assurance processes (audit), including legal

supervision of HEIs as a preparatory step

- Programme evaluations

- Appraisal of applications for degree-awarding powers (initial accreditation)

- Thematic evaluations

Thematic evaluations were not to be assessed in the external review according to the terms of

reference. However, the panel realised during the site visit that thematic evaluations are indeed a

separate external quality assurance activity within the scope of the ESG, as described in UKÄ’s self-

assessment report. Therefore, the panel decided to include this activity in the external review.

After a period of distrust and turbulence in connection to Swedish external quality assurance, UKÄ

has now developed a robust quality assurance framework, which is broadly recognised and respected

by institutions and stakeholders in the higher education system of Sweden. The latter, including

students, feel strongly involved in the different quality assurance activities of UKÄ as well as in the

governance of the agency. UKÄ has established a solid internal quality assurance policy with very well

described processes, which support the employees in their daily work and ensure consistency.

UKÄ to some extent suffers from bureaucratic and complex processes and can take various steps to

improve this in order to provide even more clarity and reduce the workload of the HEIs. UKÄ has

not yet established a complaints procedure, but this issue should be possible to tackle in the near

future. In addition, a follow-up mechanism for the appraisals of degree awarding powers is not present

in the current system.

The panel found UKÄ’s level of alignment with the ESG standards to be the following:

- Fully compliant with the following ESG standards– 3.1, 3.2, 3.4, 3.5, 3.6, 3.7, 2,1 2.4, 2.5,

2.6

- Substantially compliant with the following ESG standards – 3.3, 2.2, 2.3

- Partially compliant with the following ESG standard –2.7

4/51

INTRODUCTION

This report analyses the compliance of the Swedish Higher Education Authority

(Universitetskanslersämbetet, UKÄ) with the Standards and Guidelines for Quality Assurance in the

European Higher Education Area (ESG). It is based on an external review conducted in January -

December 2020.

BACKGROUND OF THE REVIEW AND OUTLINE OF THE REVIEW PROCESS

BACKGROUND OF THE REVIEW ENQA’s regulations require all member agencies to undergo an external cyclical review, at least once

every five years, in order to verify that they act in substantial compliance with the ESG as adopted at

the Yerevan Ministerial Conference of the Bologna Process in 2015.

UKÄ’s predecessor was a full member of ENQA between 2000 and 2012. UKÄ has been an affiliate

of ENQA since October 2014 and is applying for ENQA membership. For the first time, UKÄ is also

applying for EQAR registration.

As the agency has undergone many changes since it lost the ENQA membership in 2012, this external

review can be considered as UKÄ’s first external review against the ESG. Therefore, the panel paid

particular attention to the policies, procedures, and criteria in place.

REVIEW PROCESS The 2020 external review of UKÄ was conducted in line with the process described in the Guidelines

for ENQA Agency Reviews. The panel for the external review of UKÄ was appointed by ENQA and

composed of the following members:

• Tue Vinther-Jørgensen, Chief Consultant, Ministry of Higher Education and Science, Denmark

- Chair, quality assurance professional (ENQA nominee)

• Patrick Van den Bosch, Quality Assurance advisor, Flemish Higher Education Council - Quality

Assurance, Belgium - Secretary, quality assurance professional (ENQA nominee)

• Melita Kovacevic, Full Professor at the Department for Speech and Language Pathology,

Faculty for Education and Rehabilitation Sciences, University of Zagreb, Croatia - Panel

member, academic (EUA nominee)

• Gohar Hovhannisyan, Executive Committee member of the European Students’ Union; PhD

student at the Brusov State University of Languages and Social Sciences., Armenia - Panel

member, student (ESU nominee, member of the European Students’ Union Quality Assurance

Student Experts Pool)

From February 2020 onwards, the secretary had regular e-mail exchanges with the chair and panel

members, the ENQA Review Coordinator and the UKÄ review coordinator. On 13 February 2020,

the panel received the self- assessment report. The site visit was originally planned for 11 -13 May

2020.

On 18 March 2020, the panel members, in consultation with ENQA, decided that the site visit should

be postponed due to the COVID-19 outbreak. It was expected that the situation would not change

by May. This did not mean that the review was put on hold. In fact, the panel decided to continue with

studying all material and planned the site visit for mid-September.

5/51

During spring and early summer, panel members requested additional documents from UKÄ. These

were provided promptly by the UKÄ coordinator. Meanwhile it was decided to hold a site visit from

29 September – 1 October 2020 at the UKÄ premises in Stockholm.

On 28 August 2020, one month prior to the site visit, the panel, in consultation with the ENQA

Review Coordinator, decided that the site visit would take place online as it was not possible for all

members to travel freely between Sweden and their home country due to the COVID 19-outbreak.

The schedule for the site visit was adjusted to fit better an online visit mode. This was done in full

compliance with the provisions for online ENQA agency reviews’ visits as set out in the Protocol for

Online Site visits approved by the ENQA Board on 18 June 2020.

SELF-ASSESSMENT REPORT

The panel learned from the SAR that as soon as the UKÄ Director General had decided that UKÄ

should undergo an ENQA agency review and made a formal request to ENQA, an internal project

was set up. The head of the Department of Quality Assurance was assigned project owner, and a

project manager was appointed together with a core project team of three staff members. The

development of the SAR has taken place in close connection with the development and launch of

UKÄ’s new management system and with an ambition to have a transparent and inclusive methodology.

The project management has had regular meetings and checks with the project owner with the

Director General and UKÄ’s top management team. A project reference network with

representatives from all departments has supported the project team with documents and served as

an informal sounding board providing feedback. For stakeholders, a brief background, relevant links

and information about the application process, including a timeline, was made available on the agency

website.1

All departments and units or working groups participated in mini-workshops conducting a SWOT-

analysis focusing on the agency’s performance in activities relevant to this external review. Similar

activities were also conducted with stakeholder groups, UKÄ’s Advisory Board, and the top

management team. Results were sorted thematically and summarised in the SWOT-analysis. For staff

at the Department of Quality Assurance, the project team hosted two workshops focusing on ESG

2.2-2.5 (November 2019 and January 2020).

In the later stages of the process, two international readers were invited to review parts of the draft,

for suggestions on structure and comprehension. At the final stage, the head and deputy heads of the

Department of Quality Assurance, and a limited number of staff who had not been part of the project

group, commented on the draft report.

The panel found the SAR very open and self-reflective, informative and to the point. Together with

many supporting documents, the SAR provided reliable information about UKÄ’s level of compliance

with the ESG.

SITE VISIT The online site visit took place from 29 September till 1 October 2020. On 23 September 2020, the

panel had an online kick off meeting to prepare the site visit. On 28 September 2020, a pre-visit

meeting with the agency contact person and Heads of groups of the Department of Quality Assurance

took place to clarify elements related to the overall system and context.

On the first day of the site visit itself, the panel had a meeting with the Director General, the Head of

the Department of Quality Assurance and the Heads of group of the Department of Quality

1 https://english.uka.se/about-us/news--events/nyheter/2020-02-26-uka-submits-self-assessment-report-to-enqa.html

https://english.uka.se/about-us/news--events/nyheter/2020-02-26-uka-submits-self-assessment-report-to-enqa.html

https://english.uka.se/about-us/news--events/nyheter/2020-02-26-uka-submits-self-assessment-report-to-enqa.html

6/51

Assurance; a meeting with the team responsible for preparation of the self-assessment report; two

meetings with representatives from the Department of Quality Assurance; a meeting with

representatives from Departments of Higher Education Analysis, Legal Affairs and Administration and

IT and a meeting with the Advisory Board.

On the second day of the site visit, the following meetings were scheduled: a meeting with heads of

some reviewed HEIs/HEI representatives; a meeting with ministry representatives; a meeting with

quality assurance officers of HEIs; a meeting with assessors and a meeting with stakeholders, such as

representatives from stakeholder reference groups and advisory groups.

On the last day of the online site visit, there was a meeting with representatives from the Swedish

National Union of Students; a meeting with members of the appeals committee; a meeting with the

Director General and the Head of the Department of Quality Assurance to clarify any pending issues

and a final de-briefing meeting with staff and Board members of the agency to inform about the panel’s

preliminary findings.

The panel appreciated the open and in-depth discussions it had with the various stakeholders. Although

the panel believes that an online site visit can never generate the same non-verbal interaction between

all people present, the site visit provided an opportunity to get to know the work of UKÄ in depth.

During the site visit, the panel had various internal consensus-forming discussions. At the end of the

site visit, the panel held an internal meeting during which it agreed on the preliminary conclusions

related to the level of compliance of UKÄ on each of the standards. The secretary of the panel then

drafted the report in cooperation with the other panel members. A final version of the draft was sent

to the ENQA Review Coordinator. The draft report was submitted to UKÄ for factual verification in

November 2020, where UKÄ was given one week to comment on the report. The final report was

sent to the ENQA Board for consideration in December 2020.

HIGHER EDUCATION AND QUALITY ASSURANCE SYSTEM OF THE AGENCY

HIGHER EDUCATION SYSTEM Traditionally, Swedish higher education does not only involve educating youth after secondary

education. It also includes possibilities for continuing development for professionals. Students of all

ages also take courses without planning to graduate, for instance as part of lifelong learning, and it is

common to return to higher education after previous studies. This results in a higher average age of

students than in some other countries.

Higher education in Sweden is organised according to the cycles of the Bologna process. Students may

either follow a programme or take freestanding courses. Courses at the first- and second-cycle levels

can be assembled into a degree programme. One academic year corresponds to 60 higher education

credits (equal to ECTS credits) for full-time studies. There are three types of qualifications in higher

education: general qualifications; qualifications in the fine, applied and performing arts; and professional

qualifications.

All first- and second-cycle education consists of courses that may be combined into programmes, but

there are several large programmes leading to a professional degree that are more strictly regulated.

In addition to programmes that lead to the award of qualifications, higher education offers a wide

range of freestanding courses, many of them offered through distance learning, partly or entirely

online. Students may select their own combination of courses, and if these combinations meet

stipulated requirements, a qualification may be awarded.

The majority of higher education is provided by public higher education institutions (HEIs), comprising

around 90% of the registered students.

7/51

Sweden has a uniform system for higher education and the same legislation applies in most aspects to

all higher education, although independent higher education providers are also regulated by specific

legislation.2. Both public-sector HEIs and independent education providers are required to participate

in the national evaluations of higher education. The primary difference between a university and a

university college is that universities enjoy general entitlement to award degrees at the master and

doctoral levels, while university colleges must apply for permission to award 2-year Master degrees

(second cycle) and third-cycle degrees in specific areas. Most independent higher education providers

are small and only have programmes in one or a few areas. The government determines what

qualifications may be offered and requirements for them in the form of scope and objectives. The right

to issue qualifications may be designated in part by legislation, and in part by special authorisation

decisions from UKÄ or the government.

Higher education is free of charge, except for students from countries outside the EU/EEC area or

Switzerland. These students must pay both application fees and tuition fees for first and second cycle

education.

QUALITY ASSURANCE The Higher Education Act (1992:1434) regulates, among other things, that the operations of higher

education institutions shall be arranged to ensure that high standards are attained in courses and study

programmes and in research.

For each public HEI and for UKÄ, there are also government instructions and public service

agreements defined by the government, which direct the agencies’ operations through funding and

specific commissions. The government instruction for UKÄ states that the agency is responsible for

quality assurance of HEIs’ operations through its external quality assurance activities. In addition, it

states that these matters under UKÄ’s responsibilities concern both public and independent higher

education providers (independent HEIs).

For independent HEIs, the Award of Certain Degrees Licensing Act (1993:792) provides regulations

that they are required to follow to be allowed to award specific degrees. This includes that the

independent HEIs must follow the requirements of the Higher Education Act, and that they are

required to participate in external follow-ups and evaluations of their programmes. Independent HEIs

must therefore align with the same national regulations on quality assurance as public HEIs, and as

such also fall under the remit of UKÄ’s external quality assurance activities.

The national system of external quality assurance consists of four types of activities, which are all

conducted by UKÄ:

- Institutional reviews of the HEIs' quality assurance processes (audit)


- Appraisal of applications for degree-awarding powers (accreditation leading to

permission to award degrees in specific areas)

- Thematic evaluations.

2 law: Lag om tillstånd att utfärda vissa examina (SFS 1993:792), https://www.riksdagen.se/sv/dokument-lagar/dokument/svensk-forfattningssamling/lag-1993792-om-tillstand-att-utfarda-vissa_sfs-1993-792

8/51

UKÄUKÄ was established in 2013 when three agencies (the National Agency for Higher Education

Services; the National Agency for Higher Education and the International Programme Office)

were merged into two new agencies: UKÄ and the Swedish Council for Higher Education3.

UKÄ’S ORGANISATION/STRUCTURE As for all other government agencies, the Swedish Parliament (Riksdag) decides on the allocation

of resources to UKÄ. The operations of UKÄ are laid down in the instructions issued by the

government. These instructions define its areas of responsibility and the tasks to be undertaken.

Each year the government issues a public service agreement that specifies the targets and the

funding for these operations. The government can also assign additional tasks during the year.

The Director General of UKÄ is the head of the agency and solely responsible for decision-

making but may delegate decision-making according to UKÄ’s line of delegation. As a rule,

decisions taken by the Director General are prepared in advance by UKÄ staff, and in the case

of external quality assurance, based on assessments made by peer-review experts. Further, the

agency has an Advisory Board, whose members are appointed by the government. An important

task for the Advisory Board is to provide democratic insight into the work of the agency.

The various activities of the agency are conducted by the departments as follows:

• The Department of Quality Assurance is responsible for the external quality assurance

activities within the national quality assurance system.

• The Department of Higher Education Analysis is responsible for the official statistics on higher

education. They analyse the trends and developments in the Higher Education Sector and

monitor how efficiently HEIs make use of their resources..

• The Department of Legal Affairs is responsible for the legal supervision of HE and monitoring

of students’ rights, and provide administrative support and legal expertise to the Higher

Education Appeals Board and the Higher Education Expulsions Board.

• The Department of Administration and IT is responsible for ensuring internal governance and

control, coordinating budget, planning, follow-up and annual report, activities to develop HEI

leadership, coordinating, monitoring and improving systems for records management, finance,

procurement, facilities management, human resources and IT, and UKÄ’s communication and

information activities.

UKÄ’S FUNCTIONS, ACTIVITIES, PROCEDURES The national system for external quality assurance comprises four separate but interlinked activities.

The quality assurance activities of UKÄ comprise institutional reviews of the HEIs' quality assurance

processes, programme evaluations, appraisal of applications for degree-awarding powers, and thematic

evaluations. Although they differ in scope and focus, the activities are conducted within the national

quality assurance system established in 2016. The system has been developed with the aim to be in

accordance with the ESG, the Swedish Higher Education Act, and the Swedish Higher Education

Ordinance.

The ‘Terms of Reference’ (annex 2) of this external review states that the following activities of UKÄ

have to be addressed in the external review:

3 The Swedish Council for Higher Education (UHR) is a government agency tasked with providing support to the education sector through a number of various activities such as evaluation of foreign qualifications and promoting international cooperation and mobility.

9/51

- Institutional reviews of the HEIs' quality assurance processes (audit), including legal

supervision of HEIs as a preparatory step


- Appraisal of applications for degree-awarding powers (accreditation).

UKÄ considers thematic evaluation as a separate external QA activity within the scope of the ESG. In

this way, UKÄ has also considered thematic evaluations in its SAR. Based on a dialogue between EQAR

and UKÄ, EQAR considered that thematic evaluations are part of the external quality assurance

system, but not a separate external QA activity within the scope of the ESG. Therefore, it has not

been included in the terms of reference.

During the site visit, it quickly became clear to the panel that the thematic evaluations are a separate

activity. The methodology used in the thematic evaluations is much more than system-wide analyses

as the process contains an evaluation of each institution by a peer expert team, including a student

member. The methodology applied to thematic evaluations is developed and adapted to the relevant

theme, but it should follow the methods used for the other external quality assurance activities

whenever possible. A follow up procedure and a possibility to appeal are also in place.

Institutional reviews

Institutional reviews are the cyclical element in the Swedish quality assurance system covering all

institutions every six years. The purpose of institutional reviews is to confirm that HEIs’ quality

assurance processes ensure high quality education and support their quality enhancement. The reviews

focus on how well HEIs’ quality assurance processes, including follow-up, measures and feedback

procedures, aim to help to systematically ensure and enhance the quality of the courses and

programmes at all levels. They also aim to contribute to improving HEIs’ quality, since the assessors

in their reports identify both examples of good practice and areas in need of improvement. The

reviews aim to verify that HEIs ensure that courses and programmes at all levels comply with applicable

laws, ordinances and the ESG.

The basis for the reviews is a self-evaluation report written by the HEI, a student report, interviews,

site visits, audit trails, and other information. All assessment material for the review is considered in

the overall assessment.

Programme evaluations

The purpose of programme evaluations is to monitor the outcomes of first-, second- and third cycle

programmes, and to contribute to the HEIs’ own quality improvements for the reviewed programmes.

The programme evaluations assess the actual conditions and results, e.g. that a programme meets the

requirements in applicable laws and ordinances, ensure that students have opportunities to achieve

the national qualitative targets and also achieve these targets.

In addition, programme evaluations assess how well the follow-up, measures and feedback processes

systematically contribute to ensuring and enhancing quality in the evaluated programmes. Failure to

meet quality standards may result in the revoking of degree awarding powers. The evaluations aim to

enhancing HEIs’ quality since UKÄ’s assessors provide feedback in their reports on both identified

examples of good practice and areas in need of improvement.

UKÄ may either evaluate all programmes at selected HEIs or programmes leading to a specific degree

at all HEIs in order to provide a national overview of the quality of a particular programme. The

programmes to be evaluated by UKÄ in the current six-year cycle were selected according to the

following criteria:

10/51

• A selection of programmes that were not covered by the 2011– 2014 external quality

assurance system, including third-cycle programmes.

• A national overview of the quality of some professional qualifications, identified as important

to the sector, primarily concerning regulated professional qualifications.

• A programme of which there are indications that it risks not fulfilling the threshold level for

quality of programmes.

• If an HEI’s internal quality assurance processes do not meet the criteria in UKÄ’s institutional

reviews, a number of programmes may be selected for additional evaluation.

The assessment material consists of the HEI’s self-evaluation report, degree projects (1st and 2nd

cycles), general study plan, individual study plans and list of publications (3rd cycle), data on teaching

staff, interviews with students or doctoral students and representatives of the reviewed programmes,

and other material that UKÄ produces.

Appraisal of applications for degree-awarding powers

The purpose of appraisal of applications for degree-awarding powers is to examine whether an HEI

meets the necessary prerequisites for students to achieve the objectives defined in the Higher

Education Act, and the Higher Education Ordinance. For third-cycle degrees, the assessment also

includes the positions assumed in a government bill on doctoral programmes.

UKÄ is charged by the government to determine whether HEIs shall be given degree-awarding powers.

The exemptions are independent HEIs, the Swedish Defence University, and the Swedish University

of Agricultural Sciences. For these HEIs, the decision of degree-awarding powers lies with the

government, although the applications are generally referred to UKÄ for assessment and appraisal,

before the government decides.

Most public HEIs in Sweden have independent powers to determine which degrees they can issue.

Although all HEIs must apply for degree awarding powers for the professional degrees and degrees in

the fine arts regulated in the Higher Education Ordinance, there are additional limitations as to what

degrees each type of HEI may issue.

Thematic evaluations

Thematic evaluations assess how HEIs fulfil tasks of importance for quality in higher education assigned

to them by legislation. The primary purpose of the thematic evaluations is quality enhancement, and

they do not imply any sanctions for the HEIs. The first thematic evaluation was completed in 2017 and

investigated HEIs’ work with promotion of sustainable development in higher education. A second

thematic evaluation about widening participation in higher education is being conducted in 2020-2022.

The methodology applied to thematic evaluations is developed and adapted to the relevant theme, but

it should follow the methods used for the other external quality assurance activities whenever possible.

Activities outside the scope of the ESG

In addition to quality assurance of higher education and research, UKÄ is tasked with the legal

supervision of HEIs, and with monitoring developments and trends in the higher education sector.

UKÄ has also a responsibility in complaints about HEIs. The agency is also responsible for statistics in

the higher education sector. These activities can also contribute to the national quality assurance

system.

11/51

International activities

UKÄ’s current focus in international engagements is on collaborations within the Nordic Network of

Quality Assurance (NOQA), The European Association for Quality Assurance in Higher Education

(ENQA) and The European Consortium for Accreditation (ECA).

Representatives of UKÄ regularly attend meetings organised within the ENQA cooperation, and a staff

member of UKÄ was until recently a board member of ECA. UKÄ is also a member of The

International Network for Quality Assurance Agencies in Higher Education (INQAAHE), and the

Quality Audit Network (QAN).

Within NOQA, Nordic quality assurance agencies have worked together on several projects since the

early 1990s. The projects have been significant in terms of learning from each other and enhancing

cooperation between the organisations. On the European level, between 2016 and 2019, UKÄ

coordinated an EU-funded project aimed at quality assuring European human rights education:

Modernisation, Education and Human Rights (MEHR). In 2018, UKÄ participated in the Bologna Peer

Support Group on quality assurance in higher education and research. During spring 2020, UKÄ

participated in staff exchange with other European quality assurance agencies. Another example is the

EUniQ project, where UKÄ is one of eight participating quality assurance agencies.

UKÄ regularly involves peer-review experts from the Nordic countries. The external quality assurance

activities may also be conducted in English, e.g. when it is difficult to find peer-review experts in Sweden

with no conflicts of interest. To date, two programme evaluations and one institutional review have

been conducted in English.

UKÄ’S FUNDING The targets and funding of UKÄ’s operations are specified in a yearly public service agreement. The

funding framework for UKÄ in 2020 is SEK 156,376,000 (appr. EUR 15,283,000). The budget for the

Department of Quality Assurance is SEK 48,647,000 (appr. EUR 4,754,000). The agency is also given

government assignments as on-going tasks during the year. UKÄ has structural funding to take up

yearly government assignments. If needed, UKÄ receives additional funding for some of these

government assignments. The panel elaborates on UKÄ’s funding under ESG 3.3. and ESG 3.5.

12/51

FINDINGS: COMPLIANCE OF UKÄ WITH THE

STANDARDS AND GUIDELINES FOR QUALITY

ASSURANCE IN THE EUROPEAN HIGHER

EDUCATION AREA (ESG)

ESG PART 3: QUALITY ASSURANCE AGENCIES

ESG 3.1 ACTIVITIES, POLICY, AND PROCESSES FOR QUALITY ASSURANCE

Standard:

Agencies should undertake external quality assurance activities as defined in Part 2 of the ESG on a

regular basis. They should have clear and explicit goals and objectives that are part of their publicly

available mission statement. These should translate into the daily work of the agency. Agencies should

ensure the involvement of stakeholders in their governance and work.

Evidence

In 2012, the Swedish government issued ‘The government Act for the Swedish Higher Education

Authority (2012:810)’. This act defines the broad scope within which UKÄ can operate. UKÄ has a

quality policy for the overall direction for all activities of UKÄ. The document contains UKÄ’s vision

and overall goals for UKÄ’s activities and is available on the agency’s website. UKÄ summarises its

vision as ‘Development Monitoring – Assuring Sweden’s Status as a Knowledge Society’. The goals and

objectives of the different external quality assurance activities are described in various documents such

as the different guidelines for evaluations of each of the quality assurance activties.

UKÄ has several different tasks granted by the government as described in the introduction. There is

no hierarchy between them. The four quality assurance activities are executed by the Quality

Assurance Department following an extensive timetable for the current six-years cycle. According to

an overview of comleted and ongoing external quality assurance assessments in the SAR, the current

2015-2020 cycle includes 24 institutional reviews, 314 programme evaluations, 392 follow-ups of

programme evaluations, 88 appraisals of applications for degree-awarding powers and 2 thematic

evaluations. Of course appraisal of applications for degree-awarding powers are initiated on the

individual demand of an institution. Once this power is achieved, there is not an expiry date. Therefore,

this activity is not cyclical. Thematic evaluations are a recurring activity without a cyclical element.

The staff from the different departments informed the panel that there is a good cooperation between

different departments. For example, the Quality Assurance Department uses the data such as the

report that comes from the legal supervision of HEI. The legal department also handles complaints of

students. When this is related to a programme, this can be taken up with the quality assurance

department. The latter can decide in some cases to conduct a programme evaluation of the

programme students complain about.

UKÄ has an Advisory Board. This board is appointed by the Swedish government to advice the

Director General of UKÄ in different matters. It is common by Swedish law for official agencies to

have an advisory board. An advisory board cannot take decisions. The Director General must be the

one who decides on matters which are of a principled nature or of greater importance or which

concern regulations according to the Government Agencies Ordinance (Myndighetsförordning, SFS

2007:515). UKÄ’s management informed the panel that the Advisory Board is treated as a formal

board. Nevertheless, the Director General of UKÄ remains the sole responsible for decision making.

13/51

The Advisory Board holds four meetings a year. The members of the board met by the ENQA panel

mention that they can have open discussions about different aspects of UKÄ’s tasks. They are confident

they can have a real influence on the evaluation of the quality assurance system. Currently the Advisory

Board is reflecting on developing a research evaluation methodology, which is to be incorporated into

the institutional reviews.

In the Advisory Board there are 8 representatives from the HEIs, from the students, and from the

professional field, as well as an international member. The students in the board are nominated by the

Swedish National Union of Students. All members informed the panel that they are appointed in their

personal capacity and not as representative of an organisation. Although they are appointed by the

government, they are not in contact with the government about UKÄ, nor do they represent the

government.

UKÄ created also three reference groups with stakeholders. They were set up to discuss and

comment on the overarching principles for the quality assurance system and its external activities.

These groups include representatives from various organisations representing:

• the HEIs, students, and teachers

• the representatives from the professional field

• international quality assurance experts

The latter were appointed to provide additional international perspectives.

Next to this, UKÄ also regularly participates in dialogue meetings organised by stakeholders such as

the Association of Swedish Higher Education Institutions (SUHF), The Swedish National Union of

Students (SFS), and HEIs quality officers.

Analysis4

The panel can confirm that UKÄ has a clear and explicit, publicly available mission statement including

a vision and goals. These are without any doubt translated into its daily work with the four types of

external quality assurance activities, which indeed are conducted on a regular basis . The stakeholders

are very well involved in the governance of the organisation. The Advisory Board members are

appointed by the government but must function independently from the government and their

institutions or organisations. The appointment of the Advisory Board can thus be considered as a

formality, as there are no ties between the members of this board and the government. The panel

suggests enhancing the international composition of the Advisory Board. The board could benefit from

a more international perspective which is now only provided by one Norwegian member. The panel

believes that broadening UKÄ’s scope outside the Scandinavian area, could provide innovative insights.

The panel found it remarkable that neither the Advisory Board nor any other committee is responsible

or accountable for decision-making: it is the sole responsibility of the Director General. In theory, the

Director General could make autonomous decisions without considering the findings of the staff, the

panels or the Advisory Board. However, the panel was assured by Director General and by UKÄ staff

that all decisions in relation to external quality assurance until now have been in line with the

conclusions of the expert panels. The internal agreements and unwritten rules make it unlikely that in

the Director General should deviate from this practice, although the Director General in principle

could choose to do so.

The panel finds that there is a clear distinction between the agencies’ activities within and outside the

scope of the ESG. Those within the scope of the ESG are carried out by a formally separate

4 The panel’s analysis of every ESG is valid for all four quality assurance activities of the agency, unless stated otherwise.

14/51

department, i.e. the quality assurance department, which nevertheless makes use of information from

other departments. UKÄ’s activities in the other departments clearly falls outside the scope of the

ESG, for instance statistics and legal supervision. UKÄ does not offer consultancy services.

Panel commendations

The panel witnessed a great involvement of students. From its meeting with students and all other

stakeholders, the panel learned that students participate actively in the Advisory Board, reference

groups and play an active role in nominating students for panels.

Panel suggestions for further improvement

The panel suggests enhancing the international composition of the Advisory Board, as the panel

believes that broadening UKÄ’s scope outside the Scandinavian area, could provide innovative insights.

Panel conclusion: fully compliant

ESG 3.2 OFFICIAL STATUS

Standard:

Agencies should have an established legal basis and should be formally recognised as quality assurance

agencies by competent public authorities.

Evidence

UKÄ is established based on the government Instruction for the Swedish Higher Education Authority

(2012:810). As stated in this instruction, UKÄ has to operate (independently) under the Ministry of

Education and Research and receives its funding through allocations directly from the state budget.

UKÄ’s activities in general are regulated by the government instruction and the yearly public service

agreement.

The SAR states that the agency is officially responsible for conducting quality assurance of HEIs’

operations in Sweden through the four defined external quality assurance activities. Every year UKÄ

is to present a report to the government that its quality assurance activities have contributed to quality

enhancement and high quality in the operations of HEIs in Sweden.

All Swedish HEIs are obliged to undergo the external quality assurance activities carried out by UKÄ,

and the outcomes of the activites have formal implications for the HEIs, for instance when it comes

to degree awarding powers etc. These activities serve both regulatory and quality enhancement

purposes. The representatives from the Swedish National Union of Students and evaluated Swedish

HEIs met by the panel consider it as an obvious fact that UKÄ is the official agency for the external

quality assurance activities in Swedish higher education.

Since 2014 UKÄ operates as an affiliate member of ENQA. The panel learned from the SAR that UKÄ

is also a recognised partner in NOQA, QAN, ECA, INQAAHE and the Bologna Peer Support Group.

Analysis

The panel finds it well documented that UKÄ has an established legal basis for its operations and is

formally recognised as a quality assurance agency by the Swedish state. The decisions made by UKÄ

as a result of external quality assurance activities are recognised by HEIs, the Swedish government and

other stakeholders.


15/51

ESG 3.3 INDEPENDENCE

Standard:

Agencies should be independent and act autonomously. They should have full responsibility for their

operations and the outcomes of those operations without third party influence.

Evidence

Organisational Independence

The level of independence in the relationship between the Swedish national agency for higher

education (UKÄ’s predecessor) and the government has earlier been under debate. The SAR states

that there is a general agreement that the government was too deeply involved in the process leading

up to the launching of the earlier quality assurance system in 2011. During the cycle of programme

evaluations carried out in the period of 2011-2014, the commission from the government contained

several details regarding the method for performing the evaluations including the use of specific

assessment materials. For instance, it was predefined that the methodology should have a strong

outcome focus, and that assessments of a large amount of students theses from the programmes

should be a key element in the reviews. This prevented the quality assurance agency from designing

its methods independently and making them fit for purpose.

The panel learned from its discussion with the UKÄ management and the representatives of the

ministry that the loss of the ENQA membership has led to changes in the relationship between UKÄ

and the government. The panel examined the ‘The Constitution of Sweden: The Fundamental Laws

and the Riksdag Act, 2016’. The panel learned that the government gives a broad instruction of what

task UKÄ has to undertake without intervening in the organisational and operational process. As an

example, the representatives of the ministry pointed out that the government cannot steer UKÄ in

what they need to do operationally.

In order to carry out its tasks, UKÄ is funded by the government. This government regularly gives

additional government tasks to UKÄ for which UKÄ can be additionally funded. Most of these

additional tasks do not consider the Department of Quality Assurance, but can for instance relate to

larger developmental tasks such as finding models to secure the level of competence within the Health

Sector. However, the Department of Quality Assurance has in July 2017 been commissioned with the

task of developing a methodology for external quality assurance of research. UKÄ has structural

funding to take up these additional government assignments. If needed, UKÄ receives additional

funding for some of these government assignments. The panel elaborates on UKÄ’s funding under ESG

3.3. and ESG 3.5.

In their meetings with the panel both the UKÄ management and the representatives of the ministry

stressed the open dialogue between UKÄ and the Ministry of Education, respecting each other’s tasks

and responsibilities. The UKÄ Director General and the State Secretary meet every six weeks. In the

meetings both parties inform each other about their current activities. Topics for new assignments for

UKÄ are always discussed in these meetings before being officially announced.

The Swedish law stipulates that in its independent agencies such as UKÄ the Director General is

appointed by the government as is the Advisory Board. The panel examined these legal provisions as

well as the documents leading to the appointment of the Director General and the Advisory Board.

In Sweden, Director Generals (generaldirektör) of public authorities are as a rule recruited in an open

process and are subsequently appointed by the government.

Advisory Boards (insynsråd) are appointed by the government to advise the Director General and

guarantee democratic transparency and influence in the governing of the authority, as in the case of

16/51

UKÄ. The appointment of the Advisory Board is de facto endorsed by the government, after

consultation with UKÄ on suitable competences to be represented. The students on the Advisory

Board are nominated to UKÄ after selection by the Swedish National Union of Students. The members

of the Advisory Board mentioned in their meeting with the panel that they perceive their appointment

by the government as a formality. They identify themselves as part of UKÄ organisational structure

and do not view themselves as representing the government.

Operational Independence

The independence of all government agencies, including UKÄ, is safeguarded in Swedish legislation.

The government establishes the general principles for UKÄ in the government instruction, allocates

resources, and follows up the activities, among them quality assurance. However, neither the Riksdag

nor the government or other authorities have the right to interfere with UKÄ’s decision-making

according to the Swedish Constitution chapter 12 section 2 (SFS 1974:152).

The Riksdag has cancelled the former detailed regulations as to the procedures and methodology of

the external quality assurance activites. The present regulations – especially UKÄ’s instruction from

the government – only describe the overall activities in the national system for external quality

assurance of higher education. The panel learned that UKÄ decided to focus on professional

programmes in its programme evalutions, although the government had also expressed a need to

regularly evaluate certain programmes leading to a professional degree, in particular a number of

programmes leading to regulated professions, e.g. teacher degrees (Government brief to the Riksdag,

2015). UKÄ decides on relevant themes for the thematic evaluations. However, thematic evaluations

may also be conducted as assignments from the government, as in the example of the ongoing

evaluation on widening participation. It was stated in the meeting with the representatives from the

ministry, that these decisions were made after consultation with and on request of UKÄ, as the

decision would gain more authourity if formally taken by the ministry.

UKÄ is led by the Director General. He acts on behalf of UKÄ, is solely responsible for its activities,

and responds directly to the government. The Advisory Board is appointed by the government but

has no decision-making powers. UKÄ is thus responsible to hire its own staff, in defining its own

procedures and methodologies. The recruitment, nomination and appointment of panel experts is the

responsibility of UKÄ, although UKÄ asks the HEI, student unions and stakeholder organisations to

propose candidates for the panels. Panel members have to sign a declaration of independence stating

any conflicts of interest with the institution(s) under review.

UKÄ is expected to do additional government assignments every year. For some bigger governmental

assignments UKÄ receives an additional funding to do these assignments. Last summer UKÄ got a new

government assignment about follow up of the consequences of COVID-19 pandemic on higher

education. The methodology and procedures for these assignments are determined by UKÄ.

Independence of formal outcomes

The Director General is the sole responsible for decision-making. His decision making of every

external quality assurance activity is primarily based on the reports with judgement of the expert

groups. The staff of UKÄ present the judgement of the expert groups to the Director General and

can comment on it. Until now, the Director General always followed the advice of each expert group.

The fact that the Director General himself has to take the decisions, is enshrined in the law.

In the case of appraisal of degree awarding powers, it is the government that takes the final decision

in case of independent higher education providers, as well as the Swedish University of Agricultural

Sciences and the Swedish Defence University, based on the decision made by UKÄ’s Director General.

17/51

Analysis

Since ENQA’s review in 2012, the relationship between UKÄ and the government has undergone

changes regarding the way in which UKÄ’s quality assurance activities are set up. The current

instruction from the government is rather brief and held in general terms concerning UKÄ’s quality

assurance activities.

The panel is convinced that UKÄ is currently able to work independently. Based on the events of the

past there is a strong awareness that its independence has to be safeguarded. There is a potential risk

in the fact that the Director General and the Advisory Board are appointed by the government.

Especially since the Director General is solely responsible for decision-making at UKÄ.

The panel has understood that this method of organisation is enshrined in law for all agencies with a

director general and an advisory board and that UKÄ itself has little opportunity to change this.

However, UKÄ could still take initiatives to further safeguard its independence. UKÄ could for

instance to this end also request changes in the agency’s government instruction, formal statutes etc.

so that independence can be further guaranteed.

Panel recommendation

The panel recommends that UKÄ should take measures to further safeguard its capacity to

independently design its methodologies in external quality assurance.

Panel conclusion: substantially compliant

ESG 3.4 THEMATIC ANALYSIS

Standard:

Agencies should regularly publish reports that describe and analyse the general findings of their

external quality assurance activities.

Evidence

UKÄ conducts external quality assurance activities in regular rounds of reviews and evaluations.

Analyses of results are made after completed evaluation rounds. The aim of these analyses is to

develop and improve the processes of upcoming quality assurance activities and to spread insights and

good practices in the sector. The panel had the opportunity to read these analyses, for instance reports

gathering experiences and reflections from the rounds of programme evaluations of initial teacher

training programmes and of ph.d.-programmes. The analyses have also been published on the website.5

The UKÄ website6 shows that they publish many studies and analyses based on the work UKÄ does

both within and outside the scope of the ESG. The analysis that UKÄ considers to be 'thematic analysis'

building on results from the agency's external quality assurance activities, are de facto mainly, almost

exclusively, system wide analyses. The number of these thematic analyses is limited to one or two per

year.

5 Några reflektioner och erfarenheter efter lärosätesgranskningarna i omgång ett, UKÄ 2019; Utbildningsutvärderingarnas effekter – En genomgång av effekterna av det nationella utvärderingssystemet 2011–2014, UKÄ Report 2015:21; Granskning för utveckling. 95 utvärderade utbildningar på forskarnivå 2017–2018, UKÄ Report 2019:1; Små forskarutbildningsmiljöer. Utmaningar och framgångsfaktorer 6www.uka.se

http://www.uka.se/

18/51

The quality assurance activity 'thematic evaluations' can also be seen as contributing to thematic

analyses as they present a national picture of the situation regarding a particular theme in Swedish

higher education, for instance HEI’s work with sustainable development.

Analysis

The panel appreciates the thematic analyses made by UKÄ. These are sharp and comprehensive

analyses of their contribution to the quality assurance in Swedish higher education and of their own

quality assurance processes. There is a clear regularity in carrying out these thematic analyses (at least

after each evaluation round).

In addition, in each thematic evaluation, UKÄ also provides a picture of how all HEIs perform on the

theme under investigation. This part of thematic evaluations can therefore be considered as a thematic

analysis.

The panel further believes that the information UKÄ collects with thematic evaluations can, together

with the information that UKÄ holds from its other tasks, result in even more and exciting thematic

analyses. The prerequisite for this, however, is that the Quality Assurance Department that carries

out the evaluations, and the other departments that carry out other activities, sit together and

thoroughly reflect on the added value of new thematic analyses that the departments can generate

together.


The panel believes that based on all the information collected by UKÄ, more thematic analyses can be

carried out. The panel believes that this could be stimulated by better cooperation between the various

departments, each of which has valuable information at its disposal.


ESG 3.5 RESOURCES

Standard:

Agencies should have adequate and appropriate resources, both human and financial, to carry out

their work.

Evidence

The panel learned from its meetings with the UKÄ management and representatives of the ministry

that for all government agencies, the Riksdag (which is the Parliament) decides on the allocation of

resources. This also applies to UKÄ.

According to the information in the SAR, the funding framework for UKÄ in 2020 is SEK 156,376,000

(appr. EUR 15,283,000). This is an increase of SEK 7,341,000 compared to 2019. The funding saved

from 2019 was used for investments in operational development. In 2020, UKÄ has a maximum

available loan from the National Debt Office of SEK 13,500,000 for investments in fixed assets for

administrative purposes. For 2020, this limit is expected to need an increase of SEK 6,000,000 due to

the agency’s move to new premises in autumn 2020. The budget for the Department of Quality

Assurance is SEK 48,647,000 (appr. EUR 4,754,000).

Based on a public service agreement, the government assigns UKÄ with additional tasks. The panel

learned this is usually taken care of by the UKÄ Department of Analysis. As a result, this usually has

19/51

fewer implications for the work of the Department of Quality Assurance. UKÄ always considers room

for its staff to carry out its annual government assignments. In case of larger assignments, UKÄ and

the government will jointly consider how much additional funding is required to carry out the

assignment.

The ENQA review panel had meetings with the staff of all departments. As pointed out in UKÄ's

SWOT analysis, high staff turnover and a high workload are challenges that deserve attention. The

UKÄ staff agrees that these are not very easy challenges to solve. The staff emphasises that the high

staff turnover is partially caused by the heavy workload whereas the management argues that in the

past the right profiles were not always recruited to carry out the tasks.

The number of employees has been relatively stable until 2019, and since then the number of positions

for project managers, analysts and legal advisors was increased. This investment was made due to an

increasing number of assignments from the government regarding the quality assurance of research,

as well as the increased resources required to conduct the external quality assurance activities. The

Department of Quality Assurance is UKÄ’s largest department and has been so for a long time with

on average 38 employees out of 99 employees of UKÄ.

The panel believes that using resources efficiently, without too high staff turnover and workload are

topical issues for UKÄ. These issues have recently been addressed through the already mentioned

reallocation of resources and hiring of additional staff, but also by prioritising and postponing the start

of certain external quality assurance projects and other activities. The ENQA review panel learned

from its meetings with the staff that although UKÄ normally manages to staff external quality assurance

projects as planned and deliver expected results on time, this is not without putting a strain on already

stretched human resources. As an example, the programme evaluations of teacher programmes

caused too much work pressure on the staff. The staff mentioned to the ENQA panel that UKÄ

learned from this period and they will use less labour-intensive methods and digital support when

assessing all Swedish nursing programmes.

In their 2020 operational plan, UKÄ shows awareness of the listed challenges. Further digitisation and

technical platforms that are fit for purpose, e.g. case management systems, are prioritised measures

taken to streamline administration and promote efficiency.

The ENQA review panel learned from its meeting with the Department of Quality Assurance that the

latter has developed an extensive introduction for new employees, which includes mentorships and a

series of seminars. For all staff there are many subject specific meetings, but other ways of

professionalisation and competence development are not very common at UKÄ. Staff can be granted

time to participate in conferences when they ask for it on their own initiative.

Analysis

The panel finds that UKÄ is appropriately funded by the government. For its quality assurance activities

UKÄ has a fixed grant from the Riksdag. In case UKÄ carries out additional activities, they also receive,

when necessary, an additional payment from the government for this purpose. It is the panel’s opinion

that this practice appears to work well and that UKÄ has sufficient financial resources to carry out its

activities within the scope of the ESG.

However, the ENQA review panel is of the opinion that the workload of the staff is quite high, with

some people resigning from UKÄ for this reason. This area of attention is not unknown to UKÄ. This

has clearly been discussed internally on several occasions and various measures have been taken to

make the problem of work pressure less dominant. Although the pressure seems to have eased during

the last 1-2 years, it remains a working point for UKÄ to prepare for appropriate reactions should

staff again report a high workload or should staff turnover rise again.

20/51


The panel suggests preparing for appropriate reactions should staff again report a high workload or

should staff turnover rise again.


ESG 3.6 INTERNAL QUALITY ASSURANCE AND PROFESSIONAL CONDUCT

Standard:

Agencies should have in place processes for internal quality assurance related to defining, assuring

and enhancing the quality and integrity of their activities.

Evidence

”UKÄ’s Quality Policy” states the overall direction for all activities of UKÄ. It clarifies UKÄ’s vision

and overall goals for the agency’s activities. The quality policy is available on UKÄ’s external website

as well as on the agency’s intranet.7

UKÄ has established processes for internal quality assurance for the whole organisation, and these are

illustrated in UKÄ’s management system. The management system is a tool for the management at all

levels to make activities carried out by UKÄ efficient and of high quality. In addition, it is within the

frame of the management system that UKÄ’s management provides professional and ethical

foundations for the employees. The internal quality assurance procedures regarding UKÄ’s external

quality assurance activities are thus integrated in the overall management system.

The various guidelines for UKÄ’s external quality assurance activities and the project manager’s manual

are important tools in the daily operations of the department and provide support for employees.8

The importance of these guidelines was stressed in interviews with both UKÄ management, staff, and

peer experts. There are bi-weekly meetings for the whole department which focus on planning,

monitoring and issues relating to the working environment.

There are also seminars where reports, drafts, and methods are discussed. Twice a year the whole

department engage in full-day workshops. Examples of themes for such activities in recent years has

been how to enhance our quality culture, project-based work. The whole staff is also involved in giving

the Head of Department input to the operational plan. The panel learned from its meetings with the

UKÄ staff that groups engaged in the different external quality assurance activities meet weekly to

share more detailed progress-reports and reflections on feedback from assessors and HEIs. In addition,

the Advisory Board and the reference groups also reflect on the working methods used at UKÄ.

HEIs informed the panel that they are asked to give feedback to UKÄ in a questionnaire after

undergoing an external quality assurance activity. The same is the case for assessors enganged in UKÄ’s

activities.

7 https://english.uka.se/about-us/what-we-do/ukas-quality-policy.html 8 Guidelines for applications for degree-awarding powers; Guidelines for evaluation of third-cycle programmes; Guidelines for handling of discrimination and sexual harassment; Guidelines for re-assessment of decisions relating to the quality assurance of higher education; Guidelines for recruitment; Guidelines for reviewing the HEIs´ quality assurance processes; Guidelines for the assessment panel’s report (institutional reviews and programme evaluations); Guidelines for the evaluation of first- and second-cycle programmes; Guidelines for the evaluation of third-cycle programmes; Guidelines for the reassessment of decisions within external quality assurance of higher education (in Swedish); Guidelines regarding expenses and gifts.

21/51

UKÄ does not subcontract any of its external quality assurance work to other parties.

Analysis

The ENQA review panel has received and studied numerous documents (see annex 4) supporting

UKÄ’s internal quality assurance and professional conduct. On this basis, the panel finds UKÄ’s

approach to be solid and well-functioning. Also, on the UKÄ website, the panel could easily find a lot

of relevant information on this subject.9 In drafting these documents, UKÄ clearly uses both the staff's

own experience and the input of a diverse group of stakeholders. The panel notices that stakeholders

are regularly consulted, and their opinions are taken into account in future planning of activities.

In the view of the panel, UKÄ has thus a strong commitment to internal quality assurance. This was

confirmed in various interviews during the site visit. The panel found the professional attitude of UKÄ’s

staff laudable. The panel was convinced by the consistency with which interviewees described the

activities and procedures of UKÄ. All documents consulted are detailed and informative and thus

enable good quality of reviews. For instance, the guidelines for the staff achieve the level of detail

which the panel finds to enable any new staff member to quickly take up his or her responsibilities.

Panel commendation

The panel commends the richness of observed IQA documents as well as applauds their thorough

preparation for supporting UKÄ's internal quality assurance and professional conduct.


ESG 3.7 CYCLICAL EXTERNAL REVIEW OF AGENCIES

Standard:

Agencies should undergo an external review at least once every five years in order to demonstrate

their compliance with the ESG.

Evidence

UKÄ is undergoing its first external review for ENQA membership. The panel learned UKÄ intends

to undergo this external review at least once every five years. UKÄ also explicitly states that they

want to become an EQAR-registered agency.

In the past, UKÄ’s predecessor underwent an external review. After structural reforms of the agency

and a six-year ENQA affiliate membership, UKÄ let the panel know that the agency intends to the

request to undergo cyclical external reviews every five years in the future.

Analysis

Considering UKÄ's intention to be an active member of the ENQA community and to opt for EQAR

registration, the panel is confident that UKÄ will keep undergoing an external review at least once

every five years in order to demonstrate their compliance with the ESG.


9 https://english.uka.se/about-us/what-we-do.html

22/51

ESG PART 2: EXTERNAL QUALITY ASSURANCE

ESG 2.1 CONSIDERATION OF INTERNAL QUALITY ASSURANCE

Standard:

External quality assurance should address the effectiveness of the internal quality assurance processes

described in Part 1 of the ESG.

Evidence

The SAR states that the national system for quality assurance of higher education is designed so that

the separate external quality assurance activities jointly address all standards of the ESG Part 1. The

unified approach of UKÄ’s four interlinked quality assurance activities aims to align with both national

legislation and the ESG, in order to support and enhance the internal quality assurance processes of

HEIs.

Because of the differences in the objective and approach of the different external quality assurance

activities, as well as their historical evolution, not every standard is addressed in each activity. For

example, appraisals of degree-awarding powers focus on the preconditions and the HEI’s capacity to

offer a certain study programme, while programme evaluations focus on the outcomes and quality of

a specific programme. The panel learned that the purpose of the institutional reviews is to make a

broader assessment of the overall internal quality assurance process of the HEI. The external quality

assurance activities of UKÄ should therefore be viewed as complementing each other. The institutional

reviews in itself touch upon all the ESGs part 1. This quality assurance activity can be considered as

the most important one in the Swedish system as all institutions have to undergo an institutional

review.

For each of the quality assurance activities UKÄ has a set of developed guidelines. The criteria used in

the evaluations do not match Part 1 of the ESG one on one. Nevertheless, UKÄ clearly shows in its

SAR how its criteria match the ESG for each activity. The panel also examined the reports resulting

from the practice of the four quality assurance activities. The reports show that the UKÄ criteria are

clearly reflected in the reports.

Thematic evaluations play a special role in the four interlinked quality assurance activities. In these

evaluations, there is no formal decision leading to possible sanctions after the evaluation. More than

in the other activities, the focus is on enhancement and the detection of good examples. As a result,

the method is somewhat different than in the other quality assurance activities. How the standards

are addressed in thematic evaluations depends on the theme and scope the evaluations conducted.

As presented in the table below, the agency, mapped its external QA activities and the ESG part 1.

ESG Criteria in UKÄ’s external quality assurance activities addressing the

ESG

1.1 Policy for

quality

assurance

Institutional reviews:

• 1.2: The HEI’s quality assurance policy

1.2 Design and

approval of

programmes


• 2.4: Students’ ability to complete their studies in time

• 3.1: Design, development, establishment, and closure of programmes

• 3.2: Students’ active role in learning processes

23/51

• 3.4: Connection between national and local goals, teaching activities and

examinations.

• 5.1: Student influence

Appraisals of degree-awarding powers:

• Subject area/main field of study/professional qualification

• The degree within the scope of the degree ordinance/System of qualifications

• The scope and definition of the subject area/main field of study

• The public interest of the qualification from a national perspective

• Students’ achievement of degree targets

Programme evaluations:

• The scientific/artistic, and professional, environment for the programme,

including the connection between research and education


1.3 Student

centered

learning,

teaching and

assessment


• 1.4 Participation, engagement and responsibility

• 3.2. Students active role in learning processes

• 3.5. Improvement and development of courses and programmes



• Student influence



1.4 Student

admission,

progression,

recognition

and

certification


• 3.7. Admissions, credit transfers and awarding degrees; student appeals


• Follow-up, measures and feedback

• Students are able to complete their studies in time

1.5 Teaching

staff


• 2.1. Staff skills and the needs of the educational operations

• 2.2. Supportive environment for the development of skills and conditions for

efficiency

• 3.3. Close connection between research and education


• Teachers’ numbers and expertise



24/51

1.6 Learning

resources and

student

support


• 2.3. Infrastructure, student support and teaching resources


• There is stable and appropriate infrastructure

• Resources are used effectively



• The scientific/artistic, and professional, environment for the programme;

including the connection between research and education

1.7

Information

management


• 1.5. The implementation of measures in strategic governance, quality work,

and development of the quality system.




• Students’ preparedness to face changes in working life

• Collaboration with surrounding society


• Feedback to relevant stakeholders


• Students’ preparedness to face changes in working life

• Collaboration with surrounding society

1.8 Public

information


• 1.6. Communicating results generated by the quality system

• 3.6. Communicating planned and implemented measures.

1.9 On-going

monitoring

and periodic

review of

programmes


• 1.5. The implementation of measures in strategic governance, quality

work,and development of the quality system

• 1.6. Communicating results generated by the quality system

• 2.3. Infrastructure, student support and teaching resources

• 2.4. Students’ ability to complete their studies in time.

• 3.3. Close connection between research and education


• 3.6. Communicating planned and implemented measures

• 6.1. Students’ preparedness to face changes in working life




25/51

• Systematic follow-up of programme content, design, implementation and

examinations

• Translation of follow-ups into measures for quality improvement

• Feedback to relevant stakeholders

1.10 Cyclical

external

quality

assurance


• 1.1. The HEI’s quality system ensures the quality of the programmes and is

connected to the overarching goals and strategies which the HEI has

established for its educational offerings

Analysis

As the panel examined UKÄ’s criteria for its four quality assurance activities and checked the presence

of the criteria in the evaluation reports on the UKÄ website, the panellists are in no doubt that the

standards of the ESG Part 1 are effectively translated into UKA’s evaluation criteria. The standards are

all to a large extent covered by the criteria in the institutional reviews. They are also clearly present

in the programme evaluations and the appraisal of degree-awarding powers, but somewhatless visible

in the thematic evaluations. However, the last referred finding is related to the fact that the criteria

for thematic evaluations depend on the theme to be evaluated (see also the ESG 2.2 and 2.5).


ESG 2.2 DESIGNING METHODOLOGIES FIT FOR PURPOSE

Standard:

External quality assurance should be defined and designed specifically to ensure its fitness to achieve

the aims and objectives set for it, while taking into account relevant regulations. Stakeholders should

be involved in its design and continuous improvement.

Evidence

As indicated earlier (see chapter on the ESG 3.1.), UKÄ involves its stakeholders in its work in several

ways. The ENQA panel met representants of most of the stakeholders involved in the Advisory Board

and the reference groups. The stakeholders were very satisfied with their level of involvement in the

design of quality assurance procedures, and in the ongoing dialogue about how the activities are

conducted in practice.

The agency held several dialogue meetings in 2016 to discuss and receive comments on the proposal

for a new quality assurance system. The meetings targeted vice-chancellors, quality officers and teacher

representatives, students, and employer and labour market organisations.

All stakeholders stress that UKÄ has listened to their comments and acted when designing the current

quality assurance procedures. The procedures are documented in the guidelines for each quality

assurance activity. However, the representatives of the HEIs report in their meeting with the panel

that they have the impression that the solutions provided by UKÄ are sometimes administrative

solutions rather than sustainable solutions. An example was given of a criterion that caused ambiguity

in practice. Instead of resolving the ambiguity, the criterion was split into several criteria. This was not

as such wrong, according to the representatives of the HEIs, but resulted in an additional complexity

and administrative burden without creating added value in terms of content.

26/51

The management and quality assurance officers from selected HEIs indicated in the meetings with the

panel that UKÄ’s quality assurance activities have become an integral part in the quality assurance

processes of each HEI. UKÄ’s quality assurance activities are, according to them, always an opportunity

to look deep into the HEI’s educational system. They consider such activities as an external

reinforcement for processes the organisations must do anyway. The agency’s activities are strategically

used by the HEIs to improve such processes, according to the representatives of the HEIs.

The HEIs therefore consider especially the institutional reviews to be useful and to enhance quality.

They do, however, indicate that they see the process as a heavy workload. In addition, there is an

overlap in the information they provide for the procedures of the institutional review and the legal

supervision of HEIs, which functions as a preparatory step. The two activities are carried out by

different departments within UKÄ. On the basis of the interviews held by the ENQA review panel, it

was not clear to what extent the UKÄ panels make use of the information from the 'legal supervision'

in institutional reviews and to what extent the UKÄ panels would benefit from other information that

could be provided by the legal department of UKÄ.

UKÄ intends its four quality assurance activities to be complementary in a trust-based matter.

Therefore, the panel learned, the institutional reviews are to be seen as the central element in the

system involving all HEIs. The number of programme evaluations is more limited taking into

consideration the total number of programmes in Swedish higher education. In most cases, when UKÄ

conducts a programme evaluation, they evaluate the same programme in all institutions. During the

last years, the teacher training programmes have been evaluated. Now UKÄ is conducting 200 3rd

cycle evaluations and is planning the upcoming evaluations of the nursing programmes. The ENQA

review panel learned from its meeting with the representatives from the ministry that if the

government sees that some kind of programmes do not have the right quality, the government

requests for a broader view on the HE landscape and instructs UKÄ to carry out such an evaluation.

In spring 2020 UKÄ decided to evaluate one programme of the specific institution as the agency had

received several complaints from students of that programme. UKÄ’s Department of Legal Affairs

conducted a legal investigation. After this investigation, UKÄ decided the Department of Quality

Assurance would do a programme evaluation.

HEIs indicated in the meetings with the ENQA panel that programme evaluation is taking a lot of time

and can be considered as double work with internal quality assurance activities of the HEIs. In UKÄ’s

perspective, the scope is different in these two quality assurance activities. As an example, UKÄ states

that the ‘student perspective’ is evaluated in the institutional review on an overarching level, witha

panel focusing on the processes, whereas programme evaluations focuss on how ‘student perspective’

works in practice in the programme.

The activity ‘appraisal of degree awarding powers’ is organised since the 1990’s. The process is still

the same but UKÄ adjusted the assessment criteria to fit better its other quality assurance activities.

They have, for example, added students and representatives from the professional field in these

evaluations.

The HEIs feel that thematic evaluations generate an additional administrative burden. Integrating

thematic evaluations into the institutional reviews was raised several times as a wish in the discussions

the ENQA review panel had with representatives of the HEIs.

The representatives from UKÄ admit that the quality assurance system as a whole can become a heavy

burden for HEIs, and it is crucial for both UKÄ and HEIs to make sure that evaluations and reviews

are managed at a reasonable cost as regards time and other resources. The panel learned that UKÄ

has taken into careful consideration the feedback from HEIs, e.g. by reducing the number of assessment

criteria.

27/51

HEIs, students’ unions, and labour market organisations are closely involved in the evaluation

processes in UKÄ’s assessment panels, where they contribute actively to self-reflection and

improvement of the quality assurance methods and processes. Within UKÄ and in dialogue with

different stakeholders, there is a continuous effort to make the procedures more efficient. UKÄ has

institutionalised an onngoing dialogue with the stakeholders about the evaluations practices by

establishing three reference groups. UKÄ states in its SAR that it is the agency’s general impression

from surveys as well as feedback conferences and dialogue meetings that the different external quality

assurance activities are considered useful and contribute to quality enhancement in HEIs.

In order to further strengthen effort to avoid administrative burden, the 2020 operational plan for the

Department of Quality Assurance focuses on how UKÄ’s external quality assurance activities best

create results that are beneficial for the development of the higher education and research sectors,

and how the agency’s quality assurance processes can be made more efficient and sustainable, for UKÄ

as well as for the HEIs. UKÄ’s management stated that the pandemic has given UKÄ an extra push in

this direction, which will lead to a much more stable organisation.

To secure efficient administration and documentation, UKÄ has developed a case management system

for the external quality assurance, with working spaces for UKÄ’s project managers, assessment panels

and HEIs’ administrators. This tool thus supports a collaborative environment for each individual

external quality assurance project.

Analysis

Based on the above presented evidence, the panel is convinced that the external quality assurance

activities are well-designed by the agency to fulfil their purpose. These activities are also documented

in the guidelines. The panel examined these guidelines and finds that the documents are well

understandable though somewhat complicated in their wording.

UKÄ is aware that its quality assurance activities create a heavy workload for the HEIs. UKÄ listens

to its stakeholders and also tries to find solutions. All stakeholders are very positive about the fact

that the dialogue with UKÄ has increased significantly over the past years. The stakeholders feel well

involved and listened to in the organisation of EQA. UKÄ reflects on quality assurance through a

continuous dialogue between UKÄ and the HEIs and other stakeholders. However, the stakeholders

also indicate that there is still room for improvement regarding the efficient implementation of

stakeholder suggestions. The message the panel got from the discussions with stakeholders is that

UKÄ’s procedures are somewhat slow and bureaucratic, but that this is also what makes them robust.

Nevertheless, the panel suggest UKÄ must stay reflective about this issue and try to reduce the

administrative burden its activities place on HEIs.

UKÄ could look at all its activities together and always ask itself what value each part of each quality

assurance activity provides for ensuring the quality of Swedish higher education. The panel suggests

first and foremost, an even sharper profiling of institutional reviews versus programme evaluations

leading to a slimmer concept for the latter. The panel believes that it would be valuable for UKÄ to

have a debate about the overall relevance of external programme evaluations in the current form and

about the possibility of integrating thematic evaluations into the institutional reviews.

Panel commendations

The panel commends on the involvement of stakeholders in UKÄ’s search to ensure fitness for

purpose in its quality assurance activities.

28/51

Panel recommendations

The panel recommends UKÄ to go further in reducing the HEIs’ workload in its quality assurance

activities. In doing so, UKÄ should consider to which extent the different activities can be integrated

or complement each other better.


ESG 2.3 IMPLEMENTING PROCESSES

Standard:

External quality assurance processes should be reliable, useful, pre-defined, implemented consistently

and published. They include:

- a self-assessment or equivalent

- an external assessment normally including a site visit

- a report resulting from the external assessment

- a consistent follow-up

Evidence

The evaluation process is described in detail in the guidelines for each of the four quality assurance

activities. These guidelines are publicly available on the website of UKÄ.10 For each of the EQA

activities, a self-assessment report based on the guidelines is required. Moreover, for appraisal of

applications for degree-awarding powers, the self-assessment report is the actual application

document. Many HEIs state in the surveys that the preparation of the self-evaluation report is the

most important part of the assessment process. The panel learned from its meetings with the UKÄ

staff that a topical issue is how to instruct HEIs to provide evidence of their procedures without

submitting too much documentation, which is a burden to the HEIs as well as to the assessment panels.

Interviews are an integral part of each external quality assurance activity, except for the one thematic

evaluation conducted until now. Interviews may be conducted through regular site visits, as in the case

of UKÄ’s institutional reviews, or online. UKÄ staff states that the latter form of interviews can reduce

the workload for the agency and the panels. UKÄ has developed the concept of a site visit introducing

online interviews. One of the added values is that in case of several campuses, the panel can remain

on one location, which is a saving of time and financial resources. All interviewees involved in online

site visits stated they are able to fulfil the same task as in a regular site visit.

For all activities, the findings of the assessment are summarised in a report written by the group of

external assessors. The assessors’ report forms the basis for UKÄ’s decision-making. The panel

learned from the meetings with the representatives from HEIs that the assessors’ reports provide

clear guidance for institutional action and quality enhancement, stating how the assessment areas and

assessment criteria have been evaluated, and highlighting strengths and examples of good practice, as

well as challenges and areas for development. For all activities, the decision and the report are

published together on UKÄ’s website.

In the case of appraisal of applications for degree-awarding powers, the assessment panel’s report

makes a recommendation to UKÄ as to whether the application should be met or refused. Except for

10 Guidelines for applications for degree-awarding powers; Guidelines for evaluation of third-cycle programmes; Guidelines for reviewing the HEIs´ quality assurance processes, Guidelines for the evaluation of first- and second-cycle programmes.

29/51

independent higher education providers and programmes within defence and agricultural science, UKÄ

is charged by the government to determine whether HEIs shall be given degree-awarding powers.

As for thematic evaluations, the report is the main outcome, as these evaluations do not result in any

decisions on formal sanctions. However, only one thematic evaluation has been carried out so far, and

the precise form for reporting results may differ in the planned second thematic evaluation.

Except for appraisals of degree awarding powers, UKÄ has follow-up processes for supporting the

actions taken by an institution or programme under review. Clear feedback from the initial assessors’

report and UKÄ’s feedback conferences provide support for HEIs in their efforts to enhance the

quality of their programmes or quality assurance procedures.

In institutional reviews, HEIs that have had their quality assurance processes approved are followed

up through dialogue meetings, surveys, conferences and in other ways. HEIs with “quality assurance

processes approved with reservations” are followed up in the assessment criteria judged as unfulfilled,

for the assessment areas that are deemed as not satisfactory. The HEI is to present the measures it

has taken no later than two years after the decision. UKÄ appoints an assessment panel that follows

up the measures. Additional material and online interviews are included in the follow-up if needed.

For HEIs with ’Quality assurance processes under review’, a new, complete review of the HEI's quality

assurance processes is planned to be carried out in which all assessment areas deemed as not

satisfactory will be followed up. UKÄ and the HEI will in dialogue decide on the time for the follow-

up review, and when the HEI will present the measures it has taken.

In programme evaluations, HEIs with programmes under review will have one year to address

deficiencies and submit an action report to UKÄ. To review the HEI’s report of measures taken, UKÄ

will appoint a panel of assessors which, if necessary, can request additional documentation and require

an interview if they consider that the material does not suffice to make an assessment. When the panel

considers that it has adequate documentation, the taken actions are evaluated and a report is then

submitted to UKÄ. Supported by the panel’s report, UKÄ will decide if the programme maintains high

quality or if the degree-awarding powers are to be revoked. The forms of follow-up for programmes

that have been judged high quality, may vary and could range from surveys and telephone interviews

to feedback conferences and other types of follow-up activities.

For the appraisals of degree awarding powers, there is no formally organised follow-up by UKÄ.

Institutions that have not gotten their applications granted can follow-up and reapply at a later stage.

In the one thematic evaluation conducted until now, UKÄ arranged a national follow-up conference in

cooperation with the Association of Swedish Higher Education Institutions (SUHF).

Analysis

The panel finds that UKÄ has established clear guidelines for its four quality assurance activities. These

guidelines have been clearly set out in terms of the procedures to be followed. The different steps:

SAR, a site visit/interviews and a report resulting from the external assessment are present in the

UKÄ guidelines. Only in the one thematic evaluations conducted so far no interviews have been

included in the procedure.

For its institutional reviews and programme evaluations UKÄ has a clear follow-up procedure. This is

strict for programmes and institutions that have received a positive assessment 'with reservation’. For

those institutions and programmes that have received a positive assessment, the follow-up is rather

soft and aiming at supporting rather than assessing the HEIs’ follow-up messures. It takes the form of

surveys, telephone interviews, feedback conferences and sometimes other types of follow-up activities.

Although it was clear to the panel that follow-up activities do take place, it was not completely clear

after what criteria UKÄ decides the kind of follow-up activities to be used. In the one thematic

30/51

evaluation a national follow-up conference has been held giving the HEIs the possibility of common

reflections and discussion.

No follow-up activities are organised for the appraisals of degree awarding powers activity. The value

of this external quality assurance activity could be enhanced by organising a follow up for this activity.

This could be e.g. a progress report after 2 years in the cases where HEIs have gained degree awarding

powers in a new field.


The panel recommends UKÄ to establish a pre-defined follow-up mechanism for programmes with a

positive assessment in the appraisals of degree awarding powers aiming at supporting these

programmes in maintaining high quality. The panel also recommends UKÄ to more clearly define the

follow-up procedures for those institutions and programmes that have received a positive assessment

in the institutional reviews and in the programme evaluations.


ESG 2.4 PEER-REVIEW EXPERTS

Standard:

External quality assurance should be carried out by groups of external experts that include (a)

student member(s).

Evidence

All the external quality assurance activities are conducted by assessment panels with peer-review

experts, including students, recruited by UKÄ. Collectively, each panel is to have sufficiently broad and

extensive expertise to assess all assessment areas and criteria of the quality assurance activity in

question. The panel elaborates on this in more detail later on in this analysis.

For eacht quality assurance activity, each panel includes at least a number of experts from Swedish or

foreign HEIs, a student representative, and a representative of the labour market. Because of the

difference in focus and scope of the activities, the number of assessors in each panel and the skills and

experience required of the assessors varies. UKÄ takes care that a panel represents a geographical

spread and is gender-balanced.

The ENQA review panel learned that UKÄ has internal guidelines to support the selection of

assessors. The panels are recruited through a nomination process where all Swedish HEIs, Swedish

National Union of Students (SFS), and a number of labour market organisations are invited to nominate

assessors based on a specified profile. For instances where nominations are not enough in terms of

numbers or sufficiently qualified nominees, UKÄ uses its internal experience and national and

international networks to gather additional nominations. As a quality assurance measure before the

panels are officially appointed by UKÄ, HEIs have the opportunity to comment on the composition of

the panels, e.g. point out potential conflicts of interest. UKÄ also checks if the nominees fulfil the

requirements of independence. Panel members must sign a declaration of independence. The panel

learned that stakeholders are very positive that they can nominate panel members as this strengthens

the ownership of the evaluation.

Upon request, the panel received a list with numbers of foreign assessors in UKÄ panels. Currently

the number of international experts is limited. All external stakeholders met by the ENQA review

panel, including the representatives of the HEI’s are in favour of more evaluations conducted in English.

The HEIs that have already experienced English evaluations are very much in favour of them. With

31/51

international experts, the language used can be a concern, so UKÄ has primarily recruited assessors

from the Nordic countries or with a working knowledge of Swedish. However, programme evaluations

have also been conducted in English and one institutional review has been conducted in English at the

request of the HEI. Since the need for conducting external quality assurance activities in English will

most likely increase, the panel believes UKÄ must be ready to adapt its processes for recruitment of

assessors, while taking into account the responsibilities of public bodies laid down in the Swedish

Language Act. The latter causes an additional administrative burden as many documents have to be

translated.

Training of assessors is integrated in UKÄ’s quality assurance activities. UKÄ staff responsible for

managing a review or an evaluation are responsible for providing the necessary support to the

assessors during the entire process. Initial training is ensured through introductory meetings, and UKÄ

provides assessors with the “Introduction for assessors in UKÄ’s reviews”, the published guidelines

for the external quality assurance activity in question, and background information on the HEI or the

main areas of study to be evaluated. All panel members are trained in the same way. Sometimes special

training sessions for students focusing on the student-perspective in panels are organised. The Swedish

National Union of Students (SFS) also arranges sometimes its own training sessions for students who

want to be part of the pool of student assessors. The panel assessors met by the ENQA review panel

stated that the introductory meetings of the trainees with the agency focus on discussion and reflection

about education and quality assurance.

Analysis

The panel found that all quality assurance activities of UKÄ are carried out by a group of external

assessors that include a student member. The composition of the panels is diverse and tailored to the

type of intended evaluation. However, UKÄ does not always include international assessors and is

often limited to recruiting Nordic assessors. The panel finds that the input of more international

profiles could further strengthen the panels.

To carry out their activities, panel members are thoroughly trained by UKÄ. The ENQA review panel

notes that the manual for panel members contains all relevant information about UKÄ, Swedish higher

education, the panels’ assignment and task as well as practical information. All meetings the ENQA

panel had have shown that the selection of panels, as well as the work carried out by these panels, is

appreciated by all involved. The thorough selection procedure enables UKÄ to avoid any conflicts of

interest.


The panel suggests selecting more international assessors from outside the Nordic region to be

involved in the panels.


ESG 2.5 CRITERIA FOR OUTCOMES

Standard:

Any outcomes or judgements made as the result of external quality assurance should be based on

explicit and published criteria that are applied consistently, irrespective of whether the process leads

to a formal decision.

32/51

Evidence

The outcomes of the agency’s EQA activities are all based on the explicit criteria – i.e. those for the

institutional review, programme evaluations and appraisal for degree-awarding powers. The

explanation of the assessment criteria, assessment areas, and possible overall assessments in each

external quality assurance activity is described in the guidelines for each activity, published on UKÄ’s

website and communicated to HEIs and assessors.

UKÄ uses a two-point scale for its programme evaluations and appraisels for degree awarding powers:

satisfactory (high quality) or not satisfactory (Under review, follow-up after 1 year). If the outcome in

case of a re-assessment is still “not satisfactory”, it means that degree-awarding powers are revoked.

For institutional reviews UKÄ uses a three-point-scale: Approved quality assurance processes,

Approved quality assurance processes with reservations (follow up no later than two years after

decision), and Quality assurance processes under review (timeline for follow-up is decided in dialogue

with UKÄ).

For a HEIs internal quality assurance processes to be approved, and for a programme to be evaluated

as being of high quality, each criterion must be deemed satisfactory. The same applies for an application

for degree-awarding powers to be approved. However, not all sub-criteria need to be assessed

satisfactory in order for the criterion as a whole to be assessed satisfactorily. Both the number of

unfulfilled sub-criteria, and the scope and severity of the deficiencies, are considered by the assessment

panel when determining whether criterion as a whole is satisfactory or not. For the final overall

assessment, the panel must make the same considerations as when regarding the criteria.

In institutional reviews in case if one or more assessment areas are deemed not satisfactory, the panel

must make the consideration whether a HEI is approved with reservations, or put under review. The

assessment in this case cannot be based solely on the number of unsatisfactory assessment criteria,

but rather on an overall assessment of the HEI’s quality assurance processes, and on the severity of

the shortcomings related to the assessment criteria. The assessment panel can for instance consider

if it will be possible for the HEI to follow-up on and overcome the shortcomings within a reasonable

timeframe (2 years).

Regardless of the criteria or whether the panel has reached a positive or negative assessment, the

panels must be able to justify their findings based on the assessment material. In the reports, they must

clearly explain the assessment of each criterion and how they have reached their final overall

assessment. After the panel finished its report, the UKÄ Director General takes a decision based on

the panel's conclusions.

For thematic evaluations there are no predefined criteria that apply to all thematic evaluations. The

themes are decided in cooperation with the government, and the methodology is developed and

adapted to the relevant theme. The criteria are drawn up in function of a particular thematic evaluation.

The guidelines contain pre-defined criteria for the evaluation related to each of the sub-themes.

However, these criteria are then published and consistently applied in the evaluation of all HEIs. As a

thematic evaluation is primarily intended to be an activity to enhance the quality, it is an evaluation

without a strict pass or fail. The HEIs with which the panel met, indicate that the criteria in the first

thematic evaluation were clear to them.

Analysis

The panel found that the outcomes for the institutional reviews, programme evaluations and appraisals

for degree-awarding powers are all based on the explicit criteria. For thematic evaluations there are

no predefined criteria that apply to all thematic analyses, but the criteria are drawn up in function of

a particular theme of the thematic analysis. The panel notes that this method of working is clear and

transparent but the potential for the HEIs to enhance their activities and procedures with inspiration

33/51

from the criteria are limited due to the fact that the criteria are only formulated in connection to the

preparation of each thematic evaluation. It would be possible to incorporate themes from current

thematic evaluation in the institutional reviews, e.g. as new criteria.

Panel suggestion for further improvement

The panel suggests striving for a set of generic criteria for the thematic evaluations, for example about

the HEI’s policy, governance, implementation and results in relation to the theme under review, that

can be applied to several themes. The agency could consider integrating the thematic evaluations into

the institutional reviews.


ESG 2.6 REPORTING

Standard:

Full reports by the experts should be published, clear and accessible to the academic community,

external partners and other interested individuals. If the agency takes any formal decision based on

the reports, the decision should be published together with the report.

Evidence

UKÄ’s reports are drafted by the panel members. The UKÄ staff is not considered to be a part of the

agency’s panels and has a role of project managers. The agency staff informed the ENQA panel that

the project managers assure that all reports follow the guidelines, and additionally check that the

language is clear for any external reader. The staff also assures that a panel operates in accordance

with the guidelines, while at the same time not intervening in the evaluation process.

UKÄ panel members find the agency’s criteria on EQA activities a clear basis for drafting a report,

although UKÄ staff indicates that it takes a lot of effort to make sure that for example the texts of

reports of different programme evaluations are comparable. Before a report is then sent to the HEI

under review, the report is discussed in a meeting involving all project managers.

The assessment panel’s draft report is sent to the HEI for commenting before UKÄ makes its final

decision. The purpose of this step is to give HEIs the opportunity to familiarise themselves with the

content and comment on any factual errors in the report. HEI admit they would like to have input on

other aspects of the reporting as well. A HEI’s written response is included in the panel’s report when

published. Additionally, UKÄ’s decision is published in the same place. The HEIs interviewed by the

ENQA panel during the site visit confirm that UKÄ’s reports are clear and informative.

UKÄ makes information on completed external quality assurance activities available to HEIs as well as

to other stakeholders, mainly through an open database called “Högskolekollen” accessible via the

agency’s website. An assessment panel’s report includes assessments of the HEI’s fulfilment of each

assessment criteria and sub-criteria, and an overall assessment. During the site visit, HEI management

stated that they read the other HEIs’ evaluations to learn from other evaluations.

In the international perspective of higher education and quality assurance, UKÄ has recognised an

increasing need to make decisions and reports from the agency’s external quality assurance activities

available in English. Nevertheless, at present, only reports from a few programme evaluations

conducted in English have been published.

Finally, the panel has learned that UKÄ has identified a need to improve the dissemination and use of

assessment reports and to enhance the functionality and user-friendliness of the agency’s web-based

34/51

services. To this end, UKÄ’s has decided to develop a new layout for the external website in the

course of 2020.

Analysis

The panel noted that all reports of the agency have a context description, a description of the individual

procedure, including the assessors involved, evidence, analysis and findings, conclusions, sharing of

some elements of good practice and recommendations for follow-up actions. All reports are made

accessible on the UKÄ website.

The panel believes the assessors’ reports are easily readable. The comments on a report by the HEI

are an integral part of this report (in a form of an annex), which makes the report transparent. The

panel additionally found that the final UKÄ decision is published in the same place as the report. The

website, however, could benefit from a better structure for disseminating the reports.


The panel suggests that UKÄ works on the agency’s website to make the reports even more easily

accessible by the stakeholders.


ESG 2.7 COMPLAINTS AND APPEALS

Standard:

Complaints and appeals processes should be clearly defined as part of the design of external quality

assurance processes and communicated to the institutions.

Evidence

UKÄ has no formal complaints policy in place. However, the panel was not notified by any of the

stakeholders that they should have difficulties in filing a complaint or comment on an ongoing

evaluation process, should this be needed, they can easily contact UKÄ’s staff and management.

Importantly, the panel learned that stakeholders feel deeply involved in the UKÄ’s quality assurance

activities and that the overall idea among the stakeholdes seems to be that if anyone has a complaint,

they would be able to approach someone within UKÄ about it.

As from 2017, UKÄ has introduced separate guidelines for the reassessment of decisions relating to

external quality assurance activities. Following the introduction of such new guidelines, institutions can

request that a decision is reviewed. A special appeals committee with three external experts and three

substitutes has been appointed for a period of two years. The ”Guidelines for Appeals in the Quality

Assurance of Higher Education” is published on the UKÄ website. UKÄ aims to address such requests

within three months from receiving the request.

The appeals committee started its work in early 2019. The committee’s work is not to double check

the evaluation, but to check if the procedures of the agency have been followed the way they should.

Currently, the committee can only assess whether an error has been made and whether this error

could have affected the assessment outcome.

The appeals committee makes an assessment and delivers it to UKÄ. Based on Swedish law, the

assessment of the appeals committee does not in itself has any legal consequence but must be accepted

by the Director General before leading to a follow-up. The Director General can therefore in principle

choose to ignore the assessment of the appeals committee, but in practical life he will reconfirm the

35/51

assessment and then decide if and how the error should be corrected. This can include summoning

the whole panel of assessors – or a part of the panel – for reassessment of the issue at stake.

Until now, the appeals committee received only one appeal about a programme evaluation of a

doctoral programme. The committee had the opportunity to look at the appeal and factual comments

from UKÄ. Then, the committee held an online as well as a physical meeting in order to reach the

following conclusion: an error had been made by UKÄ and such an error could have affected the

assessment of the panel and hence also UKÄ’s final decision. The decision was sent to the Director

General. He then decided to organise a re-assessment of the programme.

Analysis

Although the panel is confident, based on discussions with the various stakeholders, that complaints

are likely to be taken up by UKÄ, this cannot be guaranteed. Given the structure of UKÄ with clear

decision-making structures, it is indispensable for UKÄ to develop a formal complaints procedure.

This procedure should allow an institution to state its dissatisfaction about the conduct of the process

or of those carrying it out. A clear reference to the complaints procedure should be published by

UKÄ so that anyone who wishes to make use of it, will be aware of its existence.

The appeals procedure developed by UKÄ is clear and known to all concerned and can be applied to

all four types of external quality assurance activities conducted by UKÄ. The appeals committee takes

its task to heart and has built up some initial experience with the one appeal that has been filed so far.

It is striking to the panel that efforts have been made to assure that the appeals committee carries out

its work in complete independence and objectivity, but that then such decisions do not have to be

followed by UKÄ. The panel believes this risks to undermine the authority of the appeals committee.

The panel is aware of the fact that the cause of this situation is the legal provisions and therefore not

a choice of UKÄ itself.

This stated, the panel understands from the discussions held with the agency management that UKÄ

plans to always follow the decisions of the appeals committee. However, the panel is of a strong

oppinion that the appeals committee’s powers can be extended by authorising the body to make

recommendations as to if and how UKÄ should correct a potential error in judgements.


The panel recommends the agency to establish a complaints procedure. The procedure should be

made known to all concerned parties.

The panel recommends extending the powers of the appeals committee to make recommendations

to UKÄ on how to correct errors in quality assurance procedures that potentially can have affected

the assessment outcome.

Panel conclusion: partially compliant

36/51

ADDITIONAL OBSERVATION

Evaluation of research by the agency

As reported to the panel through the submitted SAR, in the summer of 2017 the Government gave

UKÄ a new task to extend the current national quality assurance system to include reviewing of higher

education institutions’ quality assurance processes for research. The first steps have already been

taken by the agency: A new method has been developed, and a pilot testing has been taking place from

September 2019 to November 2020. The plan of UKÄ is to incorporate this additional evaluation

aspect into the institutional reviews.

Since research evalautions do not fall within the scope of the ESG, this new quality assurance activity

of the agency has not been further unfolded neither in the SAR nor in the panel’s interviews during

the site visit. However, on one or two occasions, stakeholders mentioned this activity expressing their

concerns, i.e. that introducing evaluation of research as part of the institutional reviews might not be

a good idea as it could have a negative impact on the workload for HEIs. The stakeholders also

expressed a concern that such a coupling of external quality assurance activities could lead to an

unproductive fragmentation of quality processes related to research. The panel understands that UKÄ

intends to focus only on general aspects of quality assurance for research provided by HEIs, and not

to evaluate research itself. Still, bearing in mind the already heavy workload of the HEIs, the added

value of this new activity should be well considered before making the final decision about integrating

it into the institutional reviews. The panel is also wondering if this new task will be an additional burden

for the UKÄ staff as well as to what extend the staff will need additional training to perform the task.

On the other side, it still remains to be seen whether this potential new aspect of the quality assurance

process in the Swedish higher education will prove to be beneficial for the research performance of

the HEIs in Sweden.

37/51

CONCLUSION

SUMMARY OF COMMENDATIONS ESG 3.1 The panel witnessed a great involvement of students. From its meeting with students and all

other stakeholders, the panel learned that students participate actively in the Advisory Board,

reference groups and play an active role in nominating students for panels.

ESG 3.6 The panel commends the richness of observed IQA documents as well as applauds their

thorough preparation for supporting UKÄ's internal quality assurance and professional conduct.

ESG 2.2 The panel commends on the involvement of stakeholders in UKÄ’s search to ensure fitness

for purpose in its quality assurance activities.

OVERVIEW OF JUDGEMENTS AND RECOMMENDATIONS ESG 3.1 Fully compliant

ESG 3.2 Fully compliant

ESG 3.3 Substantially compliant

The panel recommends that UKÄ should take measures to further safeguard its capacity to

independently design its methodologies in external quality assurance.







The panel recommends UKÄ to go further in reducing the HEIs’ workload in its quality assurance

activities. In doing so, UKÄ should consider to which extent the different activities can be integrated

or complement each other better.


The panel recommends UKÄ to establish a pre-defined follow-up mechanism for programmes with a

positive assessment in the appraisals of degree awarding powers aiming at supporting these

programmes in maintaining high quality. The panel also recommends UKÄ to more clearly define the

follow-up procedures for those institutions and programmes that have received a positive assessment

in the institutional reviews and in the programme evaluations.




ESG 2.7 Partially compliant

38/51

The panel recommends the agency to establish a complaints procedure. The procedure should be

made known to all concerned parties.

The panel recommends to extend the powers of the appeals committee to make recommendations

to UKÄ on how to correct errors in quality assurance procedures that potentially can have affected

the assessment outcome.

In light of the documentary and oral evidence considered by it, the review panel is satisfied that, in the

performance of its functions, UKÄ is in compliance with the ESG.

SUGGESTIONS FOR FURTHER DEVELOPMENT ESG 3.1 The panel suggests enhancing the international composition of the Advisory Board, as the

panel believes that broadening UKÄ’s scope outside the Scandinavian area, could provide innovative

insights.

ESG 3.4 The panel believes that based on all the information collected by UKÄ, more thematic

analyses can be carried out. The panel believes that this could be stimulated by better cooperation

between the various departments, each of which has valuable information at its disposal.

ESG 3.5 The panel suggests preparing for appropriate reactions should staff again report a high

workload or should staff turnover rise again.

ESG 2.4 The panel suggests selecting more international assessors from outside the Nordic region

to be involved in the panels.

ESG 2.5. The panel suggests striving for a set of generic criteria for the thematic evaluations, for

example about the HEI’s policy, governance, implementation and results in relation to the theme under

review, that can be applied to several themes. The agency could consider integrating the thematic

evaluations into the institutional reviews.

ESG 2.6 The panel suggests that UKÄ works on the agency’s website to make the reports even more

easily accessible by the stakeholders.

39/51

ANNEXES

ANNEX 1: PROGRAMME OF THE SITE VISIT

23 SEPTEMBER

TIMING TOPIC PERSONS FOR INTERVIEW

12.45 –

13.00

Checking the stability of

internet connection

13.00 –

16.00

Review panel’s kick-off

meeting and preparations

for day I

28 September


15.45 –

16.00

Checking the stability of

internet connection

16.00 –

17.00

A pre-visit meeting with the

agency contact person to

clarify elements related to

the overall system and

context.

Ulf Hedbjörk, project manager of UKÄ ENQA application/agency

contact person, Department of Quality Assurance

Viveka Persson, Head of Institutional reviews/Thematic evaluations

group, Department of Quality Assurance

Lisa Jämtsved Lundmark, Head of Programme evaluations/Appraisal of

applications for degree-awarding powers group, Department of

Quality Assurance

Per Westman, Head of Strategic Planning, Department of

administration and IT

29 September


8.45 – 9.00 Connection set-up

9.00 – 9.30 Review panel’s private

meeting

9.30 – 9.45 Break / Connection set-up

9.45 –

10.30

Meeting with the Director

General, the Head of the

Department of Quality

Assurance and the Heads of

group of the Department of

Quality Assurance

Anders Söderholm, Director General

Karin Järplid Linde, Head of Department of Quality Assurance





Quality Assurance

10.30 –

10.45

Review panel’s private

discussion / Connection set-up

10.45 –

11.15

Meeting with the team

responsible for preparation

of the self-assessment report

Ulf Hedbjörk, project manager and coordinator of SAR, Department

of Quality Assurance

40/51

Charlotte Elam, senior project manager, Department of Quality

Assurance

Erik Kyhlberg, project manager, Department of Quality Assurance

Agnes Ers, Business Strategist, Director General’s Office, Department

of administration and IT

11.15 –

11.30



11.30 –

12.15

Meeting with representatives

from the Department of

Quality Assurance

1. Programme evaluations +

Appraisal of applications for

degree-awarding powers

Anna-Karin Malla, senior project manager

Ulrika Thafvelin, senior project manager

Emma Wimmerstedt , senior project manager

Charlotte Ejsing, project manager

Nils Olsson, project manager

Sebastian Steele, project manager

Ellinor Alvesson, project manager

- all Department of Quality Assurance

12.30 –

12.45


discussion

12.30 –

13.30

Lunch break

13.30 –

14.00



14.00 –

14.45


from the Department of

Quality Assurance

2. Institutional reviews +

Thematic evaluations

Kristina Tegler Jerselius, senior project manager

Charlotte Elam, senior project manager

Stella Annani, project manager

Carin Dänsel, project manager

Anna Rudebeck, project manager

Henrik Holmquist, project manager

Erik Kyhlberg, project manager

Bo Sandberg, project manager

- all Department of Quality Assurance

14.45 –

15.00

Break / Connection set-up

15.00 –

15.45


from Departments of Higher

Education Analysis, Legal

Affairs, and Administration

and IT.

Annika Pontén, Head of Department of Higher Education

Analysis/Deputy Head of Authority (UKÄ)

Marie Kahlroth, senior analyst, Department of Higher Education

Analysis

Christian Sjöstrand, Head of Department of Legal Affairs

Jessica Levin, legal advisor, Department of Legal Affairs

Peter Wikhagen, Head of Department of Administration and IT

Agnes Ers, Business Strategist, Department of Administration and IT

Martin Wincent, Head of Communications group, Department of

Administration and IT

15.45 –

16.00



16.00 -

16.45

Meeting with the advisory

Board

Sigbritt Karlsson, President, KTH Royal Institute of Technology

Per Abrahamsson, Deputy University Director, Uppsala University

41/51

Nicoline Frølich, Head of Research, Nordic Institute for Studies in

Innovation, Research and Education

Samuel Engblom, Policy Director, Swedish Confederation of

Professional Employees

Anton Jägare, student, Swedish National Union of Students

Pil Maria Saugmann, doctoral student, Swedish National Union of

Students

16.45 –

17.30

Wrap-up meeting among

panel members and

preparations for day II

30 September



9.00 – 9.45 Review panel private

meeting


10.00 – 10.45 Meeting with heads of

some reviewed HEIs/HEI

representatives

Stefan Bengtsson, Vice-Chancellor, Chalmers University of

Technology, Vice Chair of The Association of Swedish Higher

Education Institutions

Hans Adolfsson, Vice-Chancellor, Umeå University

Maria Knutson Wedel, Vice-Chancellor, Swedish University of

Agricultural Sciences

Kerstin Tham, Vice-Chancellor, Malmö University

Martin Norsell, Vice-Chancellor, Dalarna University

Robert Egnell, Vice-Chancellor, Swedish Defence University

Maria Lantz, Vice-Chancellor, University College of Arts, Crafts and

Design

Johanna Adami, Vice-Chancellor, Sophiahemmet University


discussion


11.20 – 12.00 Meeting with ministry

representatives

Stefan Engström, Secretary of State to the Minister for Higher

Education and Research

Sara Bringle, Senior Advisor, Unit for Higher Education, Ministry of

Education and Research

12.00 – 13.00 Lunch break


discussion / Connection set-

up

13.30 – 14.15 Meeting with quality

assurance officers of HEIs

Malin Östling, Quality Coordinator, University of Gothenburg

Erwin Van Rijswoud, Quality Coordinator, Örebro University

Helena Loborg, Faculty Programme Director Educational Sciences,

Linköping University

Silja Marit Zetterqvist, Quality Coordinator, Head of department,

Ersta Sköndal Bräcke University College

42/51

Helena Eklund Snäll, Analyst, Division of Planning, Swedish University

of Agricultural Sciences

Åsa Ekberg, Head of Office, Quality and Evaluation, Lund University

Thomas Nilsson, Professor, Pro Dean, Karlstad University



up

14.30 – 15.15 Meeting with assessors Anette Sandberg, Professor, Mälardalen University

Helena Holmström Olsson, Professor, Malmö University

Bo-Anders Jönsson, Pro Vice-Chancellor, Lund University

Håkan Wiklund, Pro Vice-Chancellor, Mid Sweden University

Christina Nygren-Landgärds, Professor, University of Agder

(Norway)

Björn Brorström, former Vice-Chancellor, University of Borås

Kristina Westlund, Senior Lecturer, Malmö municipality


discussion


16.00 – 16.45 Meeting with

stakeholders, such as

representatives from

stakeholder reference

groups and advisory

groups

Reference group HEIs/HE sector:

Åsa Kettis, Head of Division for Quality Enhancement, Uppsala

University/Association of Swedish Higher Education Institutions

Tom Edoff , Swedish National Union of Students

Karin Åmossa, Head of Research and International Affairs, Swedish

Association of University Teachers and Researchers

Reference group labour market/working life:

Eva Marie Rigné, Expert, Swedish Association of Local Authorities

and Regions

International Advisory Committee:

Fiona Crozier, Independent Consultant (previously QAA UK)

Achim Hopbach, Independent Consultant (previously QA Austria)

16.45 – 17.30 Wrap-up meeting among

panel members:

preparation for day III and

provisional conclusions

1 Oktober




meeting


9.45 -10.30 Meeting with

representatives from the

Simon Edström, Chairperson

Linn Svärd, Vice-Chairperson

43/51

Swedish National Union of

Students.

Karl Kilbo Edlund, Representative Quality Assurance

Simon Eriksson, student representative

Frida Sundqvist, policy secretary

Sebastian Lagunas Rosén, policy secretary



up

10.45 – 11.15 Meeting with the appeals

committee

Jonas Tosteby, Deputy Vice-Chancellor, Dalarna University

Rune Heiberg Hansen, Director, Damvad Analytics

11.15-12.30 Meeting among panel

members to agree on final

issues to clarify

12.30 – 13.30 Lunch break

13.30 – 14.00 Meeting with the Director

General and the head of

the department of Quality

Assurance to clarify any

pending issues



14.00 – 15.30 Private meeting among

panel members to agree

on the main findings

15.30 Final de-briefing meeting

with staff and

Council/Board members

of the agency to inform

about preliminary findings



Christian Sjöstrand, Head of Department of Legal Affairs





Quality Assurance

Per Westman, Head of Strategic Planning, Department of

administration and IT

Ulf Hedbjörk, project manager of UKÄ ENQA application/agency

contact person, Department of Quality Assurance

44/51

ANNEX 2: TERMS OF REFERENCE OF THE REVIEW

External review of the Swedish Higher Education Authority (UKÄ)

by the European Association for Quality Assurance in Higher Education (ENQA)

TERMS OF REFERENCE

January 2020

1. Background and context

The Swedish Higher Education Authority (Universitetskanslersämbetet, UKÄ) was established in 2013,

partly replacing the former agency for quality assurance the Swedish National Agency for Higher

Education (Högskoleverket) established in 1995. UKÄ is an independent government agency and its

responsibilities and operations are regulated through instructions. The targets and funding of UKÄ

operations are specified in a yearly public service agreement. The agency is also given government

assignments as on-going tasks during the year. UKÄ is led by the Director General, who reports

directly to the government, and is comprised of four departments. The agency also has an Advisory

Board that aids the Director General.

Quality assurance in higher education in Sweden is a shared responsibility between the HEIs and UKÄ.

UKÄ is the official quality assurance agency for higher education and research in Sweden. The

objectives of UKÄ’s quality assurance activities are both to review the performance of higher

education, and to contribute to the HEIs’ quality improvement.

The quality assurance activities of UKÄ comprise institutional reviews of the HEIs' quality assurance

processes, programme evaluations, appraisal of applications for degree-awarding powers, and thematic

evaluations. Although they differ in scope and focus, the activities are conducted within the national

quality assurance system established in 2016. The system has been developed in accordance with the

ESG, the Swedish Higher Education Act, and the Swedish Higher Education Ordinance.

In addition to quality assurance of higher education and research, UKÄ is tasked with the legal

supervision of HEIs, and with monitoring developments and trends in the higher education sector. The

agency is also responsible for statistics in the higher education sector. These activities also contribute

to the national quality assurance system.

UKÄ has been an affiliate of ENQA since October 2014 and is applying for ENQA membership. The

agency was a full member of ENQA between 2000 and 2012.

UKÄ is applying for EQAR registration for the first time.

2. Purpose and scope of the evaluation

This review will evaluate the way in which and to what extent UKÄ fulfils the requirements of the

Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). Consequently,

the review will provide information to the Board of ENQA to aid its consideration of whether

membership of UKÄ should be granted and to EQAR to support UKÄ application to the register.

2.1 Activities of UKÄ within the scope of the ESG

45/51

In order for UKÄ to apply for ENQA membership and for registration in EQAR, this review will

analyse all activities of UKÄ that are within the scope of the ESG, i.e. reviews, audits, evaluations or

accreditation of higher education institutions or programmes that relate to teaching and learning (and

their relevant links to research and innovation). This is independent of whether the activities are

carried out within or outside the EHEA and whether they are obligatory or voluntary.

The following activities of UKÄ have to be addressed in the external review:

- Institutional reviews of the HEIs' quality assurance processes (audit), including legal supervision of

HEIs as a preparatory step


- Appraisal of applications for degree-awarding powers (accreditation).

3. The Review process

The review will be conducted following the methodology of ENQA Agency Reviews. The process is

designed in line with the Guidelines for ENQA Agency Reviews and the requirements of the EQAR

Procedures for Applications.

The evaluation procedure consists of the following steps:

- Formulation of the draft Terms of Reference for the review;

- Finalising the Terms of Reference for the review following EQAR’s Eligibility Confirmation (if

relevant);

- Nomination and appointment of the review panel;

- Self-assessment by UKÄ including the preparation and publication of a self-assessment report;

- A site visit by the review panel to UKÄ;

- Preparation and completion of the final evaluation report by the review panel;

- Scrutiny of the final evaluation report by the ENQA Review Committee;

- Analysis of the scrutiny by the Board of ENQA and their decision regarding ENQA membership;

- Follow-up of the panel’s and/or the Board’s recommendations by the agency, including a voluntary

progress visit.

3.1 Nomination and appointment of the review team members

The review panel consists of four members: one or two quality assurance experts (at least one of

which is currently employed by an ENQA member agency), an academic employed by a higher

education institution, a student member, and eventually a labour market representative (if requested).

One of the members will serve as the chair of the review panel, and another member as a review

secretary. For ENQA Agency Reviews at least one of the reviewers is an ENQA nominee (most often

the QA professional[s]). At least one of the reviewers is appointed from the nominees of either the

European University Association (EUA) or the European Association of Institutions in Higher

Education (EURASHE), and the student member is always selected from among the ESU-nominated

reviewers. If requested, the labour market representative may come from the Business Europe

nominees or from ENQA. An additional panel member may be included in the panel at the request of

the agency under review. In this case, an additional fee to cover the reviewer’s fee and travel expenses

is applied.

The panel will be supported by the ENQA Secretariat review coordinator who will monitor the

integrity of the process and ensure that ENQA’s requirements are met throughout the process. The

ENQA staff member will not be the secretary of the review and will not participate in the discussions

during the site visit interviews.

Current members of the Board of ENQA are not eligible to serve as reviewers.

46/51

ENQA will provide UKÄ with the list of suggested experts and their respective curricula vitarum to

establish that there are no known conflicts of interest. The experts will have to sign a non-conflict of

interest statement as regards the UKÄ review.

3.2 Self-assessment by UKÄ, including the preparation of a self-assessment report

UKÄ is responsible for the execution and organisation of its own self-assessment process and shall

take into account the following guidance:

- Self-assessment is organised as a project with a clearly defined schedule and includes all relevant

internal and external stakeholders;

- The self-assessment report is broken down by the topics of the evaluation and is expected to

contain, among others: a brief description of the national HE and QA system; background

description of the current situation of the Agency; an analysis and appraisal of the current situation;

proposals for improvement and measures already planned; a SWOT analysis; each criterion (ESG

part II and III) addressed individually. All agency’s QA activities (whether within their national

jurisdiction or outside of it, and whether obligatory or voluntary) will be described and their

compliance with the ESG analysed.

- The report is well-structured, concise and comprehensively prepared. It clearly demonstrates the

extent to which UKÄ fulfils its tasks of external quality assurance and meets the ESG and thus the

requirements of ENQA membership.

- The self-assessment report is submitted to the ENQA Secretariat which has four weeks to pre-

scrutinise it before forwarding the report to the panel of experts. The purpose of the pre-scrutiny

is to ensure that the self-assessment report is satisfactory for the consideration of the panel. The

Secretariat will not judge the content of information itself but whether the necessary information,

as stated in the Guidelines for ENQA Agency Reviews, is present. For the second and subsequent

reviews, the agency is expected to enlist the recommendations provided in the previous review

and to outline actions taken to meet these recommendations. In case the self-assessment report

does not contain the necessary information and fails to respect the requested form and content,

the ENQA Secretariat reserves the right to reject the report and ask for a revised version within

two weeks. In such cases, an additional fee of 1000 EUR will be charged to the agency.

- The report is submitted to the review panel a minimum of six weeks prior to the site visit.

3.3 A site visit by the review panel

The review panel will draft a proposal of the site visit schedule which shall be submitted to the agency

at least two months before the planned dates of the visit. The schedule is to include an indicative

timetable of the meetings and other exercises to be undertaken by the review panel during the site

visit, the duration of which is usually 2,5 days. The approved schedule shall be given to UKÄ at least

one month before the site visit, in order to properly organise the requested interviews.

The review panel will be assisted by UKÄ in arriving in Stockholm, Sweden.

The site visit will close with a final de-briefing meeting outlining the panel’s overall impressions but not

its judgement on the ESG compliance of the agency or the granting or reconfirmation of ENQA

membership.

3.4 Preparation and completion of the final evaluation report

On the basis of the review panel’s findings, the review secretary will draft the report in consultation

with the review panel. The report will take into account the purpose and scope of the evaluation as

defined under articles 2 and 2.1. It will also provide a clear rationale for its findings concerning each

47/51

ESG. A draft will first be submitted to the ENQA review coordinator who will check the report for

consistency, clarity and language, and it will be then submitted to UKÄ usually within 10 weeks of the

site visit for comment on factual accuracy. If UKÄ chooses to provide a statement in reference to the

draft report, it will be submitted to the chair of the review panel within two weeks after the receipt

of the draft report. Thereafter, the review panel will take into account the statement by UKÄ and

finalise and submit the document to ENQA.

The report is to be finalised within three months of the site visit and will not exceed 40 pages in length.

When preparing the report, the review panel should also bear in mind the EQAR Policy on the Use and

Interpretation of the ESG to ensure that the report will contain sufficient information for the Register

Committee for application to EQAR.

For the purpose of applying for ENQA membership, UKÄ is also requested to provide a letter

addressed to the Board of ENQA outlining its motivation for applying for membership and the ways

in which UKÄ expects to contribute to the work and objectives of ENQA during its membership. This

letter will be taken into consideration by the Board of ENQA together with the final evaluation report

when deciding on the agency’s membership.

4. Follow-up process and publication of the report

UKÄ will consider the expert panel’s report and will publish it on its website once the Board of ENQA

has made its decision. The report will also be published on the ENQA website, regardless of the

review outcome and decision by the Board. UKÄ commits to preparing a follow-up plan in which it

addresses the recommendations of the review panel and to submitting a follow-up report to the Board

of ENQA within the timeframe indicated in the Board’s decision on membership. The follow-up report

will be published on the ENQA website, in addition to the full review report and the Board’s decision.

The follow-up report will be complemented by a small-scale progress visit to the agency performed

by two members of the original panel (whenever possible). This visit will be used to discuss issues,

based on the ESG, considered to be of particular importance or a challenge to UKÄ. Its purpose is

entirely developmental and has no impact on the judgement of membership and/or judgment of

compliance of the agency with the ESG. Should the agency not wish to take advantage of this

opportunity, it may opt out by informing the ENQA Review Coordinator about this.

5. Use of the report

ENQA shall retain ownership of the report. The intellectual property of all works created by the

expert panel in connection with the review contract, including specifically any written reports, shall

be vested in ENQA.

The review report is used by the Board of ENQA for the purpose of reaching a conclusion on whether

UKÄ is in compliance with the ESG and can thus be admitted/reconfirmed as a member of ENQA.

The report can also be used for registration on EQAR, and is designed to serve these two purposes.

However, the review report is to be considered final only after being approved by the Board. Once

submitted to ENQA and until it is approved by the Board, the report may not be used or relied upon

by UKÄ, the panel, or any third party and may not be disclosed without the prior written consent of

ENQA. Since UKÄ is obliged to hand out the report to any third party in accordance with the Freedom

of the Press Act, the report will not be handed to the agency until the report is approved by the

Board. The approval of the report is independent of the decision of the ENQA Board on membership.

48/51

6. Budget

UKÄ shall pay the review related fees as specified in the contract between ENQA and UKÄ.

It is understood that the fee of the progress visit is included in the overall cost of the review and will

not be reimbursed in case the agency does not wish to benefit from it.

In the event of a second site visit required by the board of ENQA and aiming at completing the

assessment of compliance, and should the agency accept a second visit, an additional fee of 500 EUR

per expert, as well as the travel and subsistence costs related to the second site visit will be charged

to the agency.

7. Indicative Schedule of the Review

Agreement on terms of reference January 2020

Appointment of review panel members January 2020

Self-assessment completed 14 February 2020

Pre-screening of SAR by ENQA coordinator 29 February 2020

Preparation of site visit schedule and indicative timetable March 2020

Briefing of review panel members April 2020

Review panel site visit May 2020

Draft of evaluation report and submitting it to ENQA coordinator

for pre-screening

Early-July 2020

Draft of evaluation report to UKÄ End-July 2020

Statement of UKÄ to review panel if necessary End-August 2020

Submission of final report to ENQA September 2020

Consideration of the report by Board of ENQA October 2020

Publication of report October 2020

49/51

ANNEX 3: GLOSSARY

ENQA European Association for Quality Assurance in Higher Education

ESG Standards and Guidelines for Quality Assurance in the European Higher Education Area,

2015

EQAR European Quality Assurance Register for HigherEducation

HE higher education

HEI higher education institution

INQAAHE International Network for Quality Assurance Agencies in Higher Education

NOQA Nordic Quality Assurance Network in Higher Education

QA quality assurance

QAN Quality Audit Network

SAR self-assessment report

SUHF Association of Swedish Higher Education Institutions

SFS Swedish National Union of Students

UKÄ Swedish Higher Education Authority (Universitetskanslersämbetet)

50/51

ANNEX 4. DOCUMENTS TO SUPPORT THE REVIEW

DOCUMENTS PROVIDED BY UKÄ

• Annual Report for the Swedish Higher Education Authority 2019 (end of February 2020)

• Appointment of Anders Söderholm as Director General of UKÄ (government decision

2017-04-27)

• Appointment of the current Advisory Board (government decision 2019-04-25)

• Appraisal of applications for degree-awarding powers - example

• Assessors' competence profiles

• Award of Certain Degrees Licensing Act (1993:792)

• Communications policy (in Swedish)

• Conflict of interest form

• Framework for self-initiated follow-ups and analyses (internal memo, Department of Quality

Assurance)

• Granskning för utveckling. 95 utvärderade utbildningar på forskarnivå 2017–2018, UKÄ

Report 2019:1 (in Swedish)

• Guidelines for applications for degree-awarding powers

• Guidelines for evaluation of third-cycle programmes

• Guidelines for handling of discrimination and sexual harassment

• Guidelines for re-assessment of decisions relating to the quality assurance of higher

education

• Guidelines for recruitment

• Guidelines for reviewing the HEIs´ quality assurance processes

• Guidelines for the assessment panel’s report (institutional reviews and programme

evaluations)

• Guidelines for the evaluation of first- and second-cycle programmes

• Guidelines for the reassessment of decisions within external quality assurance of higher

education (in Swedish)

• Guidelines regarding expenses and gifts

• Guidelines regarding handling of conflicts of interest

• Higher Education Institutions in Sweden – 2019 status report

• Institutional review - example

• International experts in UKÄ review panems

• Introduction for assessors in UKÄ’s reviews

• List of feedback conferences held during the current cycle

• Meetings of the Department of Quality Assurance: autumn 2019 – spring 2020

• Minutes ’department day’ 29 April 2020.

• Minutes from the last four UKÄ’s Advisory Board meetings

• Minutes from UKÄ’s three QA reference groups

• Några reflektioner och erfarenheter efter lärosätesgranskningarna i omgång ett, UKÄ 2019.

In English: Some reflections and lessons following the first round of higher education

institutional reviews

• National system for quality assurance of higher education. Presentation of a government

assignment, UKÄ Report 2016:15

• Operational Plan for the Department of Quality Assurance 2020

• Operational Plan for the Swedish Higher Education Authority 2020

• Outcomes of UKÄ’s external QA activities 2015-2020

51/51

• Plan for maintaining and developing qualifications and competence of employees individually

and as a collective

• Programme evaluation - example

• Project Manager's Manual

• Public Service Agreement for 2020 for the Swedish Higher Education Authority

• Quality assurance and quality enhancement 2019 - Report to the government early July

(2020-07-08)

• Regulation (2012:810) with instructions for the Swedish Higher Education Authority

• Små forskarutbildningsmiljöer. Utmaningar och framgångsfaktorer. UKÄ Report 2019:17. In

English: Small third-cycle programme environments

• Some reflections and lessons following the first round of higher education institutional

reviews

• The Administrative Procedure Act (2017:900)

• The constitution of Sweden 28/08/16

• The Department of Quality Assurance’s 2020 Operational Plan

• The effects of programme evaluations – An analysis of the effects of the national evaluation

system 2011–2014, UKÄ report 2015:21,

• The government Instruction for the Swedish Higher Education Authority (2012:810)

• The Higher Education Ordinance (1993:100)

• The Instrument of government (1974:152)

• The Swedish Higher Education Act (1992:1434)

• The Swedish Higher Education Authority’s guidelines for managing conflicts of interest,

reg.nr 222-506-13

• Timetable for UKÄ assessments planned 2017-2022

• UKÄ Annual Report 2019 - ongoing translation

• UKÄ decision re-assessment KTH 2 june 2020

• UKÄ’s annual financial report 2019

• Utbildningsutvärderingarnas effekter – En genomgång av effekterna av det nationella

utvärderingssystemet 2011–2014, UKÄ Report 2015:21. In English: The effects of

programme evaluations – An analysis of the effects of the national evaluation system 2011–

2014

OTHER SOURCES USED BY THE REVIEW PANEL

• https://english.uka.se/

https://english.uka.se/

ENQA AGENCY REVIEW 2020

THIS REPORT presents findings of the ENQA Agency Review

of the Swedish Higher Education Authority (UKÄ),

undertaken in 2020.

Date post:	18-Jan-2021
Category:	Documents
Upload:	others
View:	6 times
Download:	0 times