HA Leader: Maria M. Hofmarcher-Holzhacker

Health System Intelligence / Medical University of Vienna, Centre of Public Health, Department of Health Economics e-mail: maria.hofmarcher@healthsystemintelligence

This project is funded by the Health Programme of the European Union

HA6. Priority setting methods in health information

Technical report Final Version April, 13 2017

This paper is part of the BRIDGE Health Project, which is funded by the Health Programme of the European Union (grant 664691). The views expressed herein are those of the authors and do not necessarily reflect the views of the European Commission.

This working paper is circulated for discussion and comment purposes. It has not been peer-reviewed or been subject to the review by the BRIDGE Health consortium.

© 2017 by Maria M. Hofmarcher, Nataša Perić, and Judit Simon. All rights reserved. Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission provided that full credit, including © notice, is given to the source.

Acknowledgements The following experts are part of this working group:

AUTHOR: Maria M. Hofmarcher-Holzhacker (maria.hofmarcher@healthsystemintelligence or maria.hofmarcher-holzhacker@meduniwien.ac.at), Health System Intelligence / Medical University of Vienna, Centre of Public Health, Department of Health Economics, Austria

AUTHOR: Judit Simon (judit.simon@meduniwien.ac.at), Medical University of Vienna, Centre of Public Health, Department of Health Economics, Austria

AUTHOR: Nataša Perić (natasa.peric@meduniwien.ac.at), Medical University of Vienna, Centre of Public Health, Department of Health Economics, Austria

Reinhard Busse (rbusse@tu-berlin.de), Technische Universität Berlin, Fachgebiet Management im Gesundheitswesen, Germany

Zeynep Or (or@irdes.fr), Institut de Recherche et Documentation en Economie de la Santé – IRDES, France

Peter Smith (peter.smith@imperial.ac.uk), Imperial College, UK

Herman Van Oyen (herman.vanoyen@wiv-isp.be), Scientific Institute of Public Health ISP-WIV, Belgium

Peter Achterberg (peter.achterberg@rivm.nl), Rijksinstituut voor Volksgezondheid en Milieu (RIVM), Netherlands

Angela Fehr (FehrA@rki.de), Robert Koch Institute, Germany

Sabrina Hense (HenseS@rki.de), Robert Koch Institute, Germany

Thomas Ziese (ZieseT@rki.de), Robert Koch Institute, Germany

Suggested citation: Hofmarcher, M.; Perić, N.; Simon, J. (2017) Priority setting methods in health information, BRIDGE Health Technical Report 04/2017 on Recommendations of priority setting methods for an European Research Infrastructure Consortium on Health Information for Research and Evidence-based Policy (HIREP-ERIC), HS&I and Medical University of Vienna.

Table of Contents

I. Introduction and background .................................................................... 1

II. Aims ................................................................................................. 2

III. Approach ........................................................................................ 2

IV. Results ........................................................................................... 3

Overview of priority setting methods ....................................................... 3

Methods for generating and ranking priorities ................................................ 4

Box 1: Examples of ranking generated priorities ............................................ 12

Barriers and facilitators to priority setting ................................................... 14

Priority setting in the BRIDGE Health partner short survey ................................ 14

Health targets supporting priority setting ................................................. 15

Box 2: Germany – gesundheitsziele.de ........................................................ 15

Box 3: Austria – Rahmengesundheitsziele .................................................... 16

Priority setting on supra-national level .................................................... 17

Box 4: Horizon 2020 and European Health Programme ..................................... 17

Box 5: Joint Programming Initiative (JPI) – More Years, Better Lives (MYBL) ........... 18

V. Implications and limitations .................................................................... 19

VI. List of Abbreviations .......................................................................... 20

VII. List of Tables and Figures .................................................................... 20

VIII. References ..................................................................................... 21

IX. Appendix ........................................................................................ 25

Appendix 1: Overview of definitions used for working definition ........................... 25

Appendix 2: Search strategy ...................................................................... 28

Appendix 3: Questionnaire to BRIDGE Health partners........................................ 29

Appendix 4: Viergever’s Nine Common Themes of Good Practice’ conceptual framework for priority setting in health research ........................................................... 30

Appendix 5: IRIS prioritization tool used by ECDC ............................................. 31

I. Introduction and background

Health Information (HI) is all data, evidence and knowledge that describe health and health service performance at individual or population level to facilitate research, promotion, prevention, care and support policy-making [1]. On the basis of this characterization, the BRIDGE Health concept paper proposes the establishment of a European Research Infrastructure Consortium on Health Information for Research and Evidence-based Policy (HIREP-ERIC) which will be tasked to prioritize its research activities [1, 2].

Priority setting (PS) is a challenge at all levels and contexts in health systems because demand for health care usually exceeds available resources [3]. At the same time priority setting differs across countries and within health service areas, e.g. ambulatory care, hospital care etc. Both taxpayers/patients and funders/payers are demanding greater accountability for how resources are spent and how research and health system goals are met [3]. The key challenge in setting (investment) priorities is to find the right balance between different approaches to health research including [4] epidemiological, health policy and systems research, as well as improving existing and developing of new health interventions. In summarizing specific concepts and approaches1 we define priority setting in health information as a:

Systematic, explicit and transparent decision-making process to prioritize research in population health, in health services and health systems research (i.e. health research). Expected benefits to society include greater adhesion and interest of policymakers and stakeholders in the research process and results, reduced research duplication and enhanced collaboration across disciplines.

Thus, the focus of the paper is on priority setting in health research, and not on health interventions or health care provision [5-13]. In particular, the paper does not evaluate rationing approaches such as multi-criteria decision analysis (MCDA) [5, 13], or Value-of-Information Analysis (VOI) to inform clinical decisions [14-16]. Due to the breadth and scope of literature in this area, a broader analysis would not have been feasible. Implications of this are discussed in the limitations section.

European research and innovation in health helps to tackle the societal challenge of health, demographic change and well-being [17]. It is therefore essential to have a systematic, explicit and transparent approach established that guides investment decisions. For example, Ettelt et al. (2011) identified in their overview of the infrastructure and capacity of health services research (HSR) in Europe that “in most countries, no structured process of identifying priorities for HSR was reported” [18]. Priority-setting procedures “are often indirect or ex post, for example, through the selection, review and approval of research proposals after scientific review by members of the research community [18]”. Moreover, stakeholders, e.g. health directors were found to use subjective criteria more often than objective criteria even though objective criteria is regarded as more important [19]. Hence, having a systematic prioritization process in place still does not guarantee that the establishment of priorities will not be based on the preferences of those at the table [19]. This also reflects the large number of possible competing research ideas. In addition, the

1 See Table 6 in Appendix 1

1

outcome of research is inherently uncertain, and the impact of research is difficult to predict and measure [4, 20].

Increasingly, the development of health goals is used in many countries to address emerging health needs [21, 22] and to guide investment decisions to respond to social values, e.g. equity in access, while supporting the advancement of health information. However, there is virtually no consensus regarding which, or whose, values should guide decisions and how these values should inform priority setting [23]. While key health system goals like access, efficiency and quality are broadly shared across countries they involve trade-offs and complex decision making.

Even though there is no gold standard, approaches to design and to implement health research priority setting processes exist for a variety of contexts. In some cases they are well-analysed and well-documented [20, 24-26].

II. Aims

First, this paper seeks to identify methods to inform priority setting at European level to facilitate improved co-ordination among researchers and other stakeholders.

Second, we aim at identifying approaches and methods which are transferable to the priority setting process in health information. This is crucially important as a core task of a European Research Infrastructure Consortium on Health Information for Research and Evidence-based Policy (HIREP-ERIC) is to develop methodologies for establishing health information priorities to reduce health information inequality2 [1, 2].

The expected outcomes of this paper are to present:

An overview of common approaches, tools and methods used to prioritize health research.

Common decision criteria that facilitate the process of priority setting. Recommendations on how the various actors in the proposed governance structure

of an HIREP-ERIC could facilitate evidence-based, systematic and transparent priority setting health and health system research.

III. Approach

A. First, we conducted a structured literature search of peer-reviewed literature in June and July 2016. The search was carried out using the databases PubMED and EMBASE (see Appendix 2 for search strings). Complementary to this, we performed a targeted search of internet publications using Google Search. The reference lists of identified literature were scanned manually for relevant additional sources. Also, literature/initiatives recommended by experts and considered important or relevant was included where appropriate.

2 Health information inequality refers to the “the unequal capacity among countries to monitor and evaluate population health and health system performance at the national level using routinely collected data” (see technical report on HA2).

2

B. Second, an online questionnaire was sent to 23 BRIDGE Health partners to survey individual expertise for all seven horizontal activities [1] including experiences with priority setting activities and current national practice in health target setting. The survey was conducted during the months November and December 2015 with Limesurvey, a web-based software program. The survey contained eight windows: one on personal information and one for each horizontal activity with an average set of four questions (see Appendix 3). Preliminary results were presented at the 2nd BRIDGE Health Steering Committee in February 2016 in Brussels.

C. Finally, results from the european Health System Indicator (euHS_I) survey will be used to showcase expert assessment in defining levels of indicators and identifying data gaps in relevant health information areas that may also translate into specific research needs [27]. The euHS_I survey contains 361 consolidated indicators out of a list of 2148 used or proposed health and health system indicators that we identified in 46 relevant international and national health information initiatives of which 45 percent comes from EU MS. The survey went live in June 2016 asking BRIDGE Health and other experts coming from the European Commission, OECD and WHO to map selected indicators to performance domains, and to assess the importance of their information content. First results are expected in late fall and will be presented at the conference of the European Public Health Association in November 2016.

IV. Results

Overview of priority setting methods

In this section the results from the literature search are used to give an overview of priority setting methods in health research.

Given the absence of uniform standards to develop priorities for health research [24, 25], Figure 1 shows a generic nine-item checklist to guide the steps before, during and after the priority setting process [28].

3

Figure 1: Checklist for health research priority setting

Source: [28]

To date, published evaluations of applied priority setting exercises are generally lacking [24]. Bryant et al. (2014) emphasized this limitation in a narrative review of health research priority setting methods, models and frameworks used in high-income countries. Bryant and colleagues found that, among 11 different priority setting exercises identified, none had been evaluated to assess the employed process or the extent to which the exercise had achieved its goals [24, 25]. The lack of evaluation in priority setting approaches was also identified in low- and middle-income countries [24].

Methods for generating and ranking priorities

Nonetheless a variety of approaches and practical strategies to guide and evaluate research priority setting exists and is well-documented [4, 20, 25, 26, 28-31]. Nasser et al. (2013) from the Cochrane Priority Setting Methods Group describe priority setting as technical (e.g. using economic analysis) or interpretive (e.g. with stakeholder involvement) [26]. Such research priority setting approaches can be further categorized as those using and compiling “existing data sets like burden of disease and other that focus on what questions would be important for future research (foresight approaches) such as Delphi method, or horizon scanning to explore novel and unexpected issues along with persistent problems and trends in health care” [26, 32]. The identification and methods of involvement of relevant stakeholders both play an important role when establishing priorities.

Table 1 and Table 2 Table 2 give an overview of the identified methods and processes for generating priorities.

Preparation

•Articulate the contextual factors that underpin the process. •Decide who should be involved. •Choose what information should be gathered. •Establish plans for translating research priorities into projects. •Assess necessity for use of a comprehensive approach.

Decision-making

•Select relevant criteria to focus discussion. •Choose a method for deciding on priorities.

Follow-up

•Define when and how evaluation of process and outcome will occur. •Communicate the approach that was used transparently.

4

Table 1: Methods for identifying priorities

Method Description Reference

Workshops, focus groups or round tables

- events that bring key stakeholders together - group-based techniques encourage respondents to think in

terms of the common good and their interests as members of society

- different formats: i) mixed groups (merge of representatives of multiple stakeholder groups), ii) separate groups (e.g. research funders, researchers)

Advantage: increases the likelihood that different views can be openly debated.

Disadvantage: some individuals may have greater dominance in a group situation leading to views or concerns of individuals being neglected.

[25], [33]

Stakeholder surveys or questionnaires

- eliciting individual opinions/preferences through ranking, Lickert scales, voting, using an ‘important, not important, not sure’ process etc.

- respondents are more likely to consider their own private interest

Advantage: potential to reach a large number and wide range of stakeholders.

Disadvantages: - Challenges with designing surveys that are appropriate for

stakeholders of various backgrounds/expertise. - Interpretation may be required to collate responses if open-

ended questions are asked.

[25], [20, 33, 34]

Calls for submission or comment

- seek to utilise stakeholders’ personal and/or professional perspectives and expertise in forms such as national call for briefs

Advantage: enable a wide range of stakeholders to be reached.

Disadvantage: requires stakeholders to have a level of written expertise in order to respond.

[25], [20]

Nominal group technique

- a structured group information gathering process that aims to combine idea generation and consensus building into a single meeting.

Advantages: - Facilitates equal participation of all group members. - Reduces the domination of the discussion by a single person or

group of people. - Results in a set of prioritised solutions or recommendations

that are agreed democratically by the majority of group members.

Disadvantages: - Structured process can minimise discussion and reduce

opportunities for the development and refinement of ideas.

[25], [24]

5

Method Description Reference

Delphi technique

- a systematic, interactive forecasting method relying on a panel of experts and questionnaires.

- Participants eligible with related backgrounds and experiences concerning target issue are selected through a nomination process.

- Participants answer a questionnaire, then the results (usually a statistical representation of the group response including reasons for judgements) are circulated to all participants. Participants are encouraged to revise their original responses in light of the responses of other participants, allowing sharing of information and reasoning among participants.

- Generally, two to four rounds are conducted, with the answers of participants converging towards consensus.

- Rate or ranking of replies AND consensus building.

Advantages: - Does not require face-to-face meetings and therefore is

relatively free of social pressure, dominance of individuals or groups, and is inexpensive.

- Multiple iterations and feedback process. - Flexible to change. - Anonymity of respondents.

Disadvantages: - Numerous rounds of questionnaires can be time consuming and

requires commitment from individuals over a period of time to avoid low response rate.

- Vulnerable to differential response rates and can have high rates of attrition between rounds.

- May force a middle-of-the-road consensus, militating independent judgements.

- Potential for investigators and facilitators to bias opinions. - Does not provide methodology for identifying participants. - Lack of criteria transparency.

[25], [20], [35]

Public input session

- a public meeting to seek input from stakeholders

Advantages: - Promotes public awareness of the topics being addressed. - Allows for a wide range of stakeholders to contribute.

Disadvantages: - Public setting may inhibit expression of ideas which could

draw criticism or debate. - Public setting may disadvantage/discourage non-expert

stakeholders from contributing alongside experts. - Practical/time constraints in receiving input from large

numbers of participants.

[25]

Desk top approach

- refers to secondary research as literature searching and review

- Evidence Gap Map (EGM) • are thematic evidence collections covering a

particular issue for evidence-informed policies and strategic research prioritisation

[34]

6

Method Description Reference

• identify evidence from systematic reviews and impact evaluations

• provide a graphical display of areas with strong, weak or non-existent evidence on the effects of development programmes and initiatives.

Advantages: - ease of access - low cost to acquire

Disadvantages: - quality of the research - incomplete information - not timely

7

Table 2: Frequently used priority setting processes that combine different methods and ranking approaches (reproduced from [20])

ENHR* CAM* James Lind Alliance COHRED* CHNRI*

Overall process National research priority setting focusing on inclusiveness of participation, broad-based consultations at different levels, both quantitative & qualitative info used, and stewardship by small working group.

Brings together economic and institutional dimensions for reducing burden of disease.

Brings together patients, carers and health professionals. Mix of data gathering, quantitative and qualitative analysis for priority setting in areas of treatment uncertainty.

Uses a management process for national level exercises

Addresses gaps in the existing research priority methods. Assist decision making and consensus development.

Identification/involvement of stakeholders

Through small representative working group: researchers, decision makers, health service providers and communities.

Individual, household and community; health ministry and other health institutions; other sectors apart from health; and macroeconomic level actors.

Through Priority Setting Partnerships (PSP) which brings patients, carers and clinicians equally together and agree through consensus.

Refers to “Step 2: setting the scene”: 1) develop the focus and scope of cycle, 2) define ethical standards, 3) formalise engagement of partners.

Participants identified by management team based on their expertise (e.g. number of publications, experience in implementation research, etc.).

Participants include stakeholders who might not have the technical expertise but have a view on the health topic of concern.

* ENHR = Essential National Health Research, CAM = Combined Approach Matrix, COHRED = Council on Health Research for Development, CHNRI = Child Health and Nutrition Research Initiative

8

Table 3: Frequently used priority setting processes that combine different methods and ranking approaches (reproduced from [20]), cont.

ENHR* CAM* James Lind Alliance COHRED* CHNRI* Identification of research ideas

1) evidence based situation analysis (health status, health care system, health research system). 2) ideas from nomination process. 3) consensus: brainstorming, multi-voting, nominal group technique, round-table.

1) measure disease burden, 2) analyse determinants, 3) get present level of knowledge, 4) evaluate cost and effectiveness, 5) present resource flow. For each theme information is provided via workshops and brainstorming. Information is populated into matrix which reveals gaps and top priorities.

1) Recommendations through PSP or literature review. 2) verify through systematic reviews. 3) extract reported confidence intervals that are below effect line 4) consensus: virtual interim priority ranking, final workshop to agree upon top 10 priorities

Through compound approaches (ENHR, CAM, Burden of Disease) or foresight techniques (Visioning, Delphi). A method are recommended.

Based on current evidence. Each participant is asked via an online survey to provide max of three research questions. Consolidated by management team.

Scoring criteria YES • Appropriate to the

level of action • Detailed in definition • Independent of each

other • Contain information

base • Reflect equity

promotion and development

• Manageable number Expressed in a common language

n/A No clear criteria are identified which to use.

Various ranking techniques: direct and indirect valuation.

YES, 5 standard criteria used

• Answerability • Equity • Impact on burden • Deliverability • Effectiveness (Optional: low cost, sustainability, acceptability, feasibility, innovation and originality)

* ENHR = Essential National Health Research, CAM = Combined Approach Matrix, COHRED = Council on Health Research for Development, CHNRI = Child Health and Nutrition Research Initiative

9

Table 4: Frequently used priority setting processes that combine different methods and ranking approaches (reproduced from [20]), cont.

ENHR* CAM* James Lind Alliance COHRED* CHNRI* Scoring options Each criteria is scored:

point score to each criteria OR number of score choices to each criteria.

n/A Ranked AND qualitative consensus

Ranked Each criteria is scored: point score to each criteria in the scale of 0, 0.5 and 1 or in the scale of 1 to 100.

Advantages • Broad based inclusion and participation of different stakeholders

• Multidisciplinary and cross-sectorial approach

• Partnership development

• Transparent process • Systematic analyses of

health needs • To be used if the

country does not have many reliable data [32]

• Creates framework of information

• Identifies gaps in knowledge

• Facilitates comparisons between sectors

• Broad inclusion of actors

• 3D-CAMP includes equity

• Takes into account underrepresented groups

• Applicable to small scale prioritisation (e.g. hospital)

• Mixture of methods

• Overview approach providing steps

• Discusses wide range of options

• Flexible to contexts and needs

• Simple, inclusive and replicable and thus systematic and transparent process.

• Independent ranking of experts

• Less costly

Disadvantages • Vague criteria and just few guidelines for their application

• Needs stronger representation of other stakeholders (e.g. private sector, international agencies)

• Does not provide method for identifying participants

• Difficult and time-consuming as involves multi-stake discussion

• No instruction on how to score; not repeatable/systematic

• Does not provide method for identifying participants

• Time consuming to identify and verify treatment uncertainties

• Selection of criteria not clear

• Not suitable for global level

• Very clinically oriented

• Too general and unspecific

• Lack of criteria transparency

• Potentially represent collective opinion of the limited group of people included in the process

• Scoring affected by currently on-going research

* ENHR = Essential National Health Research, CAM = Combined Approach Matrix, COHRED = Council on Health Research for Development, CHNRI = Child Health and Nutrition Research Initiative

10

Figure 2 shows the distribution of methods used and summarizes recent mapping [20]. It clearly shows that the priority setting approach developed and first applied by the Child Health and Nutrition Research Initiative (CHNRI) and the Delphi method appear the two most common strategies. The review covers research priority setting exercises performed at different levels, ranging from macro (health system) to meso (individual hospitals) as outlined in Table 2 [20]. These were identified in health research published in PubMed indexed journals since 2011. While the Delphi approach is popular in setting priorities it is expected that methods with a well-defined and encompassing structure – such as the CHNRI method – will eventually replace the Delphi method, as they offer more transparency and are better repeatable. Yoshida [20] reported that among the 165 identified studies that set health research priorities, few either had a vaguely described processes of group decision making or did not provide a method description at all.

Figure 2: Distribution of methods, tools and approaches used for setting health research priorities (source: PubMed, 2001 to 2014

Source: reproduced with permission from [20]

Bryant et al. (2014) describe prioritisation as a process where individuals or groups rank identified research priorities in terms of their importance or significance [26]. Specific criteria are normally provided to support this process. Techniques include [26]:

• subjective ranking based on perception of social and scientific merit; • simple counting of the number of times a priority area was mentioned with the most

frequently mentioned ranked first; • ranking based on sophisticated criteria including

11

o data on prevalence, mortality, morbidity, cost and variability, utility of area for decision making, information gaps, variability in care, and gaps in translation); and

• ranking using five point scale designed to capture the need for the research (likely benefit of research to the organisation and patient care, relevance to policy initiatives, burden of disease, costs of the service and to patients, and practice variation) and research potential (feasibility, degree of management commitment to the issue, study design, and participants).

Box 1: Examples of ranking generated priorities

The European Centre for Disease Prevention and Control (ECDC) has developed a specific tool for scoring scientific priorities, the so called “IRIS”. Topics are ranked by the Advisory Board on a 3 point scale (-1, 0, 1) against 12 weighted indicators (see Appendix 5) from four priority categories [36, 37]:

Impartiality

1. Affects/involves majority of MS 2. Strengthens capacity building and/or networking in MS 3. Reducing inequities or variations in public health practice among MS

Resources

4. Saves MS resources when coordinated/performed at the EU level 5. Aims to provide more cost-effective intervention than existing ones 6. Benefits relative to investments

Impact

7. Applicability 8. Preventive potential to decrease disease, disability, or death 9. Addresses knowledge and/or methodological gap

Significance

10. Emerging or escalating public health issue on EU level 11. Burden of disease/relevance to PH 12. High demand as expressed by stakeholders/high risk perception

The Advisory Board includes public health experts from all Member States, the European Commission, WHO, learned societies and patient organizations [36, 37].

Also, the Child Health and Nutrition Research Initiative (CHNRI) developed a sophisticated approach for identifying and deciding on research priorities as outlined in detail in Table 2. Table 6 presents criteria of the CHNRI method that we propose to be applied also in priority setting exercises performed in a future HIREP-ERIC.

12

Table 3: Examples of possible criteria which can be used for setting priorities in health research investments

Criterion Question

Acceptability / Public opinion

How likely will the proposed research be approved, taking into account any possible resistance based on ethical or political grounds and public opinion?

Affordability How likely will the results improve affordability of existing policies and programs?

Answerability How likely will the objectives be met given the current state of science and the size of the gap in knowledge?

Applicability How likely will the results be immediately applicable for guiding policies and programs?

Deliverability How likely is that the results will improve the delivery of existing policies and programs?

Equity How likely is that the proposed research will benefit those who are most vulnerable?

Feasibility How likely is that the cost of the proposed research will be a feasible investment?

Potential effect on disease burden

How likely is the proposed research to lead to significant improvement in disease burden reduction?

Sustainability / Effectiveness

How likely is that the results will improve sustainability of existing policies and programs?

Usefulness Given the quality of existing evidence, how likely is it that the proposed research will fill a critical gap in knowledge?

Existing research capacity

How likely is it that the objectives will be met given existing research capacity?

Alignment with other policies

How well are the objectives aligned with other existing policies in society?

Generation of commercial products / Novelty & potential for translation

How likely is that the proposed research will lead to patents and generate commercial products?

Competitiveness and publication impact / Attractiveness

How likely is that the results of the research will be seen as competitive against other ongoing work and be accepted for publication the journals with the highest impact factor?

Source: Adapted from [38, 39]

13

Barriers and facilitators to priority setting

Research also points towards barriers and facilitators of good practice priority setting. This is particularly relevant when designing a prioritisation process as it is envisaged by a HIREP ERIC. Table 4 summarises key limiting and facilitating factors with respect to setting priorities in health care and health research [8, 25, 40].

Table 5: Barriers and facilitators for explicit priority setting

Barriers Facilitators

- lack of trust between stakeholders - unbalanced representation of

expertise and stakeholders - conflict of interest between

experts - advisory panel lacking health

economic knowledge and/or allocation experience

- politics preventing (program) process evaluation

- discontinuity of personnel - too many administrative demands

leaving priority setting as a low priority activity

- multi-component approaches require considerable time and resources

- difficulties conceptualising the process and intended outcomes and generating initial priorities, especially within the advisory group

- difficulties making decision within the advisory group

- ideas not being suggested by researchers out of fear that idea would be appropriated by others

- senior level managerial and scientific champions

- fairness o stakeholders understand the

process o stakeholders feel engaged

- strong leadership - culture to learn and change - integrated budgets - resources earmarked for the process

itself and follow-up on recommendations

- built in incentives for appropriate and efficient spending

- structured techniques perceived as useful in facilitating agreement about priorities

- piloting of questionnaire proved useful

- separate consultation exercises for ‘non-professional’ consumers or special groups as different methods can be applied

Source: Adapted from [8, 25, 40]

Priority setting in the BRIDGE Health partner short survey

A survey was sent out to 23 BRIDGE Health partners with a broad variety of expertise in health research to review their expertise or their involvement in priority setting, see Appendix 3. In total, we received 18 responses from representatives of all 12 BRIDGE Health work packages. Results show that 50% (N=9) of surveyed BRIDGE Health partners have been involved in priority setting methods/activities such as:

• Providing expert opinion on burden of disease issues.

14

• Selection of health measurements for national Health Examination Surveys (availability, reliability, feasibility, interesting, (ethical) acceptability, costs, public health importance).

• Indicator development. • Fixing and hierarchizing priorities through a common conceptual framework. • Determining most appropriate study population (vulnerable groups) and selecting

most promising chemicals (aimed at covering both, long-term and short-term exposure).

Survey participants indicated that priority setting is often opportunistic and driven by availability of data echoing findings from the literature. Respondents engaged in priority setting activities have experience with both, consensus based (Delphi method, group discussions, focus groups, workshops) and metrics based (burden of disease, availability of register data and data linkage with personal data) approaches.

Health targets supporting priority setting

The relevance of health target as a prioritisation tool is echoed by responses from BRIDGE Health survey participants. While knowledge about global developments in this area appears limited (N=4) BRIDGE Health partner indicated that the German and Austrian approach in health target setting is a familiar model to priority setting3.

The growing and sustained interest in establishing a process of health target setting among European governments was documented in various mapping exercises [21, 22, 41]. Earlier research shows that ten countries in Europe had adopted or drafted policies which included health targets [21, 22]. According to the most recent WHO Health 2020 policy indicator framework 40% (N=12) of WHO Member States indicated that they had established a national or subnational target-setting process in 2010. In 2013 this rate increased to 50% involving 15 Member States. [42].

Overall health targets are a recognized tool used in health policy for improving health system performance [41]. “Setting health targets is a common approach of health systems to coordinate and steer activities of stakeholders, to increase accountability and to support the development and prioritization of health policies and strategies” [43]. Literature suggests that the setting of targets starts with (1) principles and values and is followed by (2) goals; (3) objectives; (4) qualitative targets; (5) quantitative targets and ends with the development of (6) indicators and a monitoring system [21, 44, 45].

Defining and implementing health targets requires in most cases “a fresh approach to health intelligence” by revealing gaps in health information [22]. Along those lines, Box 2 and Box 3 summarize the processes of health target setting in Germany and Austria.

Box 2: Germany – gesundheitsziele.de

In Germany, a national health targets process has been in place since the year 2000. More than 120 actors in the health system, including the Robert Koch Institut (RKI), the German Statutory Pension Insurance Scheme and the National Association of Statutory Health

3 Further mentioned by BRIDGE Health partners was prioritisation schemes for human biomonitoring (HBM) chemicals existing in Germany, France, the USA and Canada (see 1st WP6 deliverable)

15

Insurance Physicians, are involved in the process. The health targets process is based on an action cycle which includes the selection of national targets based on 13 criteria (see Table 6), the development of targets, their implementation, their evaluation based on a relevant set of indicators, and, if necessary, their revision. The process is based on the WHO Health for All-Initiative, and in Germany, it was indeed triggered at the regional level by the federal states. Today, health targets processes are coordinated by the Health Targets Network (Kooperationsverbund gesundheitsziele.de) and exist at local, regional and national levels [43, 46]. (RKI, BRIDGE Health partner survey)

Table 6: List of criteria developed by the Health Targets Network for the standardized evaluation of potential health targets

Criterion Definition Severity in terms of mortality

The health problem causes a high mortality.

Severity in terms of morbidity

The health problem causes a high burden of disease.

Prevalence The health problems and its risk factors are highly prevalent in the population.

Potential for improvement

The health problem can be adequately addressed.

Economic relevance The health problem is associated with considerable direct and indirect costs, which can be addressed through appropriate measures.

Ethical aspects The health target is of high ethical relevance and not associated with ethical concerns.

Equal opportunities The health target contributes towards mitigating social and health disparities.

Importance as perceived by the population

The health problem is perceived by the population and by politicians to be of high priority.

Measurability The achievement of the health target is measurable. Feasibility in terms of measures and instruments

Measures and instruments necessary for the implementation of the health target are available.

Feasibility in terms of stakeholders

Stakeholders are willing to implement measures aiming towards the implementation of the health target.

Opportunities for the participation of the population

The population and particularly those affected by the health problem are able to participate in the implementation of the health target.

Legal framework The legal basis for measures necessary to implement the health target is available.

Source: [43]

Box 3: Austria – Rahmengesundheitsziele

Being part of the current government program, the Austrian health target setting process that started in 2010 is based on a broad participatory approach. It actively involved around 40 relevant political and social stakeholders that constitute the guiding body, the so called “plenum” [47]. Accompanied by a continuous monitoring and reporting mechanism that is overseen by the Austrian Public Health Institute, Austrian health targets were developed in three phases, being i) preparation, ii) operationalization, and iii) realization. By means of an online consultation during the preparation phase, everyone interested in this topic had the opportunity to submit their views and opinions [47]. Broad participation of senior level policy and societal actors has been regarded as one of the key success factors of the process

16

to generate joint ownership [48]. For each of the 10 adopted health targets, inter-sectoral working groups met regularly in order to formulate sub-targets, concrete actions and indicators, and to discuss the implementation of all measures. Since 2012, packages of measures of 5 working groups are available and being implemented.

Health targets are based on a number of guiding principles. These include ‘focus on health determinants’, ‘health-in-all-policies approach’ and ‘promoting health equity’ [47]. Health equity and Health In All Policies (HIAP) are key for the process and are also embedded in specific health targets [47].

Priority setting on supra-national level

Besides the national health target setting approaches, based on expert recommendations we identified further examples on how priorities are being set within the context of the European Commission focusing on health and health system topics that frame Europe’s strategy. Also member states driven Joint Programming Initiatives have set activities in these fields.

Box 4: Horizon 2020 and European Health Programme

Within the current RTD4 Framework Programme Horizon 20205, funding opportunities for prioritised topics for a given year are set out in a multiannual work programme (WP) [49].

The consultation of stakeholders ranging from industry and research to representatives of civil society is an integral part of the programming process [50]. The representatives of these stakeholder groups take up a consultative role in the form of Advisory Groups (AG). Experts are selected based on specific criteria such as proven competence and experience [51]. As in the former 7th EU Framework Programme (FP7), members and activity reports of advisory group meetings are made public [51, 52]. For example, the AG for the Societal Challenge 1, “Health, Demographic Change and Well-being”, used small focus groups to facilitate the compilation of chapters for the latest 2016 advice report towards defining potential priorities in the WP for 2018-2020 [53]. In addition, open and/or targeted consultation activities are conducted to gather further views and inputs by both European level associations and national level organisations. The final decision on the final text of the WPs remains with the Programme Committees composed of delegates and experts of national governments [54].

The official documents studied imply, that concerns about transparency and stakeholder involvement were raised in the ex-post evaluation report of FP7. These were taken up by Horizon 2020 [52, 55]. Nevertheless, it still remains a rather top-down approach in funding research [52, 56, 57].

In the long run, the definition of priorities and objectives post Horizon 2020 [58] for research and innovation in biomedical and health research is being supported by the Scientific Panel

4 Research and Technological Development 5 Horizon 2020 reflects the policy priorities of the Europe 2020 strategy and addresses major concerns shared by citizens in Europe and elsewhere.

17

for Health (SPH) [59], a science-led expert group6. The objective of the SPH is to identify concrete challenges and propose solutions for the hurdles and barriers to innovation [60].

Similar to the process described above, the 3rd Health Programme for the period 2014-2020 [61] results from an extensive consultation process with Member States’ representatives, National Focal Points designated by Member States’ authorities, the Council working party on Public health at Senior Level and the informal Health Council [62]. The EU Health Policy Forum, health professionals and patients associations provided additional expert advice.

To implement the 3rd Health Programme, the European Commission prepares and adopts the annual Work Programme by defining the priorities in close consultation with Member States health authorities serving on the Programme Committee [63]. Implementation of the Programme is entrusted to the Consumers, health, Agriculture and Food Executive Agency (CHAFEA), which organizes the calls and supervises the evaluation procedures leading to the award of co-funding to the best actions [64].

Box 5: Joint Programming Initiative (JPI) – More Years, Better Lives (MYBL)

Being a member-states driven activity, the Joint Programming Initiative (JPI) “More Years, Better Lives (MYBL)– The Potential and Challenges of Demographic Change” aims at enhancing more coordination and collaboration between European and national research programmes related to demographic change [65]. The JPI is a collaborative project between participating countries, not a European funding programme. It intends to help to “shape the funding priorities of national and European agencies”. In order to promote such synergies, a Strategic Research Agenda (SRA) defining priorities for research to inform policymaking was produced through an iterative process of discussions and feedback involving the representatives of [66]:

• the participating states (General Assembly), • the national research bodies (Scientific Advisory Board), • five transnational working groups of scientific experts (Scientific Advisory Board),

and • representatives of European stakeholder groups (Societal Advisory Board).

During the conceptualisation stage a survey on social relevance of research themes was conducted addressing all JPI members [67]. In addition, each working group prepared a report which provided the basis for the development of the agenda. After a round of consultations, a draft version was circulated to all JPI member states for comment. These comments were considered in a joint meeting of the Scientific and Societal Advisory Boards. The draft was then revised and the final agenda was approved by the General Assembly constituting the main decision making body.

6 Coming from a broad community of stakeholders in biomedical & health research, professionals in health care, biotech and industry, and patients.

18

Following criteria provided guidance in the decision making process for prioritizing research topics [68]:

• “What works”: programme must inform policy making and address important questions from a future perspective

• “Where demographic change is a central component” • “Where a comparative and transnational perspective will add value”

In summary, a survey instrument, face to face approaches, such as meetings and workshops, as well as literature reviews were applied during the JPI agenda setting process. Also, the Societal Advisory Board played an important role in engaging the general public as part of its public engagement approach. While being precisely and transparently documented, the studied documentation of the SRA setting process [65, 66, 68] lacks information on how the scientific experts were selected (just nomination by countries). We neither found evidence for the use of scoring or voting methods including specific criteria for selecting the research themes.

V. Implications and limitations

In their review Bryant et al. (2014) conclude that it is “not possible to provide strong evidence-based recommendation about optimum methods to set research priorities” [25]. The selection of the appropriate method should be based upon the context of the priority setting process.

In line with the analysed literature, we recommend that priority setting processes in an HIREP-ERIC should [25, 69]:

• be inclusive by adopting a comprehensive concept of priority setting as developed in this paper

• be overseen by a well-managed and resourced multi-disciplinary advisory group, • involve a broad representation of stakeholders, • utilise objective and clearly defined criteria for generating and ranking priorities, • be systematic and transparently documented, and • be evaluated.

An Advisory Group provides credibility to the process of determining research priorities, and ensures the developed priorities are relevant and feasible [25]. Therefore, the adoption of a governance structure of an HIREP-ERIC requires to ensure boards capability for developing and implementing priorities. The relevant body should elect a member to chair the group. Also, a process should be put in place to manage any potential conflicts of interest.

It is crucially important that decision makers and policy setters can make informed and justified decisions by understanding points of convergence and divergence between participating stakeholders. Such approaches help promoting a sense of inclusion as well as a sense of ownership of the problem by participating stakeholders. Inclusion and ownership are important parts of resilient, adapted and coordinated actions [70].

19

This paper excludes models and strategies in health care priority setting referring to rationing of health care services and interventions [5-11]. This limitation may have led to the omission of some relevant studies. Also, this paper did not discuss the evaluation of the health target setting process.

But literature provides a useful checklist to set up a conceptual framework for priority setting in health research (see Table 9, Appendix 4) which appears suitable for an HIREP-ERIC. At the same time, criteria as developed in the CHNRI (see Table 6) should be embedded in this process to facilitate transparent decision making.

VI. List of Abbreviations

AG Advisory Group

CAM Combined Approach Matrix

CHAFEA Consumers, health, Agriculture and Food Executive Agency

CHNRI Child Health and Nutrition Research Initiative

COHRED Council on Health Research for Development

ENHR Essential National Health Research

FP7 7th EU Framework Programme

HI Health information

HIAP Health In All Policies

HIREP-ERIC European Research Infrastructure Consortium on Health Information for Research and Evidence-based Policy

HSR Health Services Research

JPI-MYBL Joint Programming Initiative - More Years, Better Lives

MCDA Multi-criteria decision analysis

MS Member State

NICE National Institute for Health and Care Excellence

PS Priority setting

SRA Strategic Research Agenda

VOI Value-of-Information Analysis

WP Work Programme

VII. List of Tables and Figures

Table 1: Methods for identifying priorities ......................................................... 5

20

Table 2: Frequently used priority setting processes that combine different methods and ranking approaches (reproduced from [20]) ........................................................ 8 Table 3: Frequently used priority setting processes that combine different methods and ranking approaches (reproduced from [20]), cont. ................................................ 9 Table 4: Frequently used priority setting processes that combine different methods and ranking approaches (reproduced from [20]), cont. ............................................... 10 Table 5: Barriers and facilitators for explicit priority setting .................................. 14 Table 6: List of criteria developed by the Health Targets Network for the standardized evaluation of potential health targets ............................................................. 16 Table 7: Overview of definitions used for working definition .................................. 25 Table 8: Search strategy ............................................................................. 28 Table 9: Nine Common Themes of Good Practice’ conceptual framework and its adaption for the evaluation of health target setting process adapted from [24] ....................... 30

Figure 1: Checklist for health research priority setting .......................................... 4 Figure 2: Distribution of methods, tools and approaches used for setting health research priorities (source: PubMed, 2001 to 2014 .......................................................... 11

VIII. References

1. Van Oyen, H. and P. Bogaert, Bridge Health: Concept Paper - Technical Report BRIDGE Health 2016: Brussels.

2. Rosenkötter, N., et al., Key features of an EU health information system: a concept mapping study. The European Journal of Public Health, 2016. 26(1): p. 65-70.

3. Savedoff, W.D., What should a country spend on health care? Health Aff (Millwood), 2007. 26(4): p. 962-70.

4. Rudan, I., et al., Setting Priorities in Global Child Health Research Investments: Assessment of Principles and Practice. Croatian medical journal, 2007. 48(5): p. 595-604.

5. Baltussen, R., Louis, Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Effectiveness and Resource Allocation, 2006. 4(1): p. 1-9.

6. Neiger, B.L.T., R.Fagen, M. C., Basic priority rating model 2.0: current applications for priority setting in health promotion practice. Health promotion practice, 2011. 12(2): p. 166-171.

7. Guindo, L.A.W., M.Baltussen, R.Rindress, D.Van Til, J.Kind, P.Goetghebeur, M., Which criteria are used in health care decisionmaking and priority setting? A literature review for an international survey of decisionmakers. Value in Health, 2011. 14(3): p. A30.

8. Gibson, J.L.M., Douglas K.Singer, Peter A., Setting priorities in health care organizations: criteria, processes, and parameters of success. BMC Health Services Research, 2004. 4: p. 25-25.

9. Sabik, L.M.L., Reidar K., Priority setting in health care: Lessons from the experiences of eight countries. International Journal for Equity in Health, 2008. 7: p. 4-4.

10. Hauck, K., P.C. Smith, and M. Goddard, The economics of priority setting for health care: a literature review. Washington: World Bank, 2004.

11. Goddard, M., et al., Priority setting in health–a political economy perspective. Health Economics, Policy and Law, 2006. 1(01): p. 79-90.

21

12. Noorani, H.Z.H., D. R.Boudreau, R.Skidmore, B., Priority setting for health technology assessments: A systematic review of current practical approaches. International Journal of Technology Assessment in Health Care, 2007. 23(3): p. 310-315.

13. Thokala, P., et al., Multiple Criteria Decision Analysis for Health Care Decision Making—An Introduction: Report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value in Health, 2016. 19(1): p. 1-13.

14. Minelli, C. and G. Baio, Value of Information: A Tool to Improve Research Prioritization and Reduce Waste. PLoS Med, 2015. 12(9): p. e1001882.

15. Claxton, K.P. and M.J. Sculpher, Using value of information analysis to prioritise health research. Pharmacoeconomics, 2006. 24(11): p. 1055-1068.

16. Claxton, K. and J. Posnett, An economic approach to clinical trial design and research priority-setting. Health economics, 1996. 5(6): p. 513-524.

17. European Commission. Health. HORIZON 2020 - The EU Framework Programme for Research and Innovation 2016 [cited 2016 August 31]; Available from: https://ec.europa.eu/programmes/horizon2020/en/area/health.

18. Ettelt, S., Nicholas, Health services research in Europe and its use for informing policy. Journal of Health Services Research & Policy, 2011. 16(suppl 2): p. 48-60.

19. Earle-Richardson, G.S., M.Wyckoff, L.Strogatz, D.May, J.Jenkins, P., Community views and public health priority setting: how do health department priorities, community views, and health indicator data compare? Health Promot Pract, 2015. 16(1): p. 36-45.

20. Yoshida, S., Approaches, tools and methods used for setting priorities in health research in the 21(st) century. Journal of Global Health, 2016. 6(1): p. 010507.

21. Hofmarcher, M., M. Kraus, and M. Riedel, Gesundheitsziele und –indikatoren als Steuerungsinstrument der sozialen Krankenversicherung. 2004, Institute for Advanced Studies (IHS).

22. Wismar, M., et al., Health targets in Europe. Learning from experience. 2008: European Observatory on Health Systems and Policies.

23. Kaplan, W., et al., Priority Medicines for Europe and the World 2013 Update, in WHO. 2013, World Health Organization.

24. Mador, R.L.K., K.Simard, A.Haroun, V., Using the Nine Common Themes of Good Practice checklist as a tool for evaluating the research priority setting process of a provincial research and program evaluation program. Health Res Policy Syst, 2016. 14: p. 22.

25. Bryant, J.-F., RobWalsh, JustinStewart, Jessica, Health research priority setting in selected high income countries: a narrative review of methods used and recommendations for future practice. Cost Effectiveness and Resource Allocation, 2014. 12(1): p. 1-11.

26. Nasser, M., VivianUeffing, ErinCrowe, SallyOliver, SandyCarlo, Reshma, Evidence in agenda setting: new directions for the Cochrane Collaboration. Journal of Clinical Epidemiology, 2013. 66(5): p. 469-471.

27. Hofmarcher-Holzhacker, M., N. Peric, and J. Simon. european Health System_Indicators survey (euHS_I survey). 2016 [cited 2016 August 31]; Available from: http://survey.euhs-i.eu/questionnaire.

28. Viergever, R.F.O., SylvieGhaffar, AbdulTerry, Robert F., A checklist for health research priority setting: nine common themes of good practice. Health Research Policy and Systems, 2010. 8(1): p. 1-9.

29. Tomlinson, M., et al., A review of selected research priority setting processes at national level in low and middle income countries: towards fair and legitimate priority setting. Health Research Policy and Systems, 2011. 9: p. 19-19.

30. McGregor, S., K.J. Henderson, and J.M. Kaldor, How Are Health Research Priorities Set in Low and Middle Income Countries? A Systematic Review of Published Reports. PLoS ONE, 2014. 9(10): p. e108787.

22

31. Chalmers, I., et al., How to increase value and reduce waste when research priorities are set. Lancet, 2014. 383(9912): p. 156-65.

32. Montorzi, G., S. De Haan, and C. Ijsselmuiden, Priority Setting for Research for Health: A Management Process for Countries. 2010.

33. Hasman, A., Eliciting reasons: empirical methods in priority setting. Health Care Anal, 2003. 11(1): p. 41-58.

34. Cochrane Methods Priority Setting Group. Top Tips for Research Priority Setting (from Cochrane Vienna 2015 Workshop). 2015 [cited 2016 August 26]; Available from: http://methods.cochrane.org/prioritysetting/top-tips-research-priority-setting-cochrane-vienna-2015-workshop.

35. Duffield, C., The Delphi technique. The Australian journal of advanced nursing: a quarterly publication of the Royal Australian Nursing Federation, 1988. 6(2): p. 41.

36. de Carvalho Gomes H, et al., IRIS—a tool for prioritising guidance development at the European Centre for Disease Prevention and Control (ECDC), in Better Knowledge for Better Health, Abstracts of the 21st Cochrane Colloquium; 2013 19-23 Sep. 2013, John Wiley & Sons: Québec City, Canada.

37. Kramarz, P., et al., Vaccine-preventable diseases: the role of the European Centre for Disease Prevention and Control. Clinical Microbiology and Infection, 2014. 20, Supplement 5: p. 2-6.

38. Rudan, I., et al., Setting priorities in global child health research investments: Universal challenges and conceptual framework. Croatian Medical Journal, 2008. 49(3): p. 307-317.

39. Rudan, I., et al., Setting Priorities in Global Child Health Research Investments: Guidelines for Implementation of the CHNRI Method. Croatian Medical Journal, 2008. 49(6): p. 720-733.

40. Mitton, C. and C. Donaldson, Health care priority setting: principles, practice and challenges. Cost effectiveness and resource allocation : C/E, 2004. 2: p. 3-3.

41. Smith, P.C. and R. Busse, Targets and performance measurement, in Performance Measurement for Health System Improvement: Experiences, Challenges and Prospects. 2010, Cambridge University Press. p. 509-536.

42. World Health Organization, Qualitative indicators for monitoring Health 2020 policy targets. 2014, World Health Organization Regional Office for Europe: Copenhagen.

43. Brzoska, P.E., U.Kimil, A.Razum, O.Sass, A. C.Salman, R.Zeeb, H., Reviewing the topic of migration and health as a new national health target for Germany. International journal of public health, 2015. 60(1): p. 13-20.

44. Van Herten, L.M.G.-S., L. J., Targets as a tool in health policy. Part II: Guidelines for application. Health Policy, 2000. 53(1): p. 13-23.

45. Czypionka, T., et al., Schwerpunktthema Gesundheitsziele: weit verbreitet oder ein spärlicher Einzelfall in Europa? Health System Watch, 2005. 4-05.

46. Robert Koch-Institut, Welche Bedeutung kommt Gesundheitszielen im Gesundheitswesen zu?, in Gesundheit in Deutschland 2015, A.-C. Saß, et al., Editors. 2015, Robert Koch-Institut. p. 394-406.

47. BMGF. Austrian health targets. 2013 [cited 2016 August 26]; Available from: http://www.gesundheitsziele-oesterreich.at/austrian-health-targets/.

48. Braunegger-Kallinger, G., How policy and societal actors develop health targets – A success story. EuroHealthNet Magazine, 2016. 7.

49. European Commission. What is a Work Programme ? HORIZON 2020 - The EU Framework Programme for Research and Innovation 2016 [cited 2016 August 31]; Available from: https://ec.europa.eu/programmes/horizon2020/en/what-work-programme.

50. European Commission. SC 1 Health, demographic change and wellbeing - Work Programme 2016-2017. HORIZON 2020 - The EU Framework Programme for Research and Innovation 2016 [cited 2016 August 31]; Available

23

from: https://ec.europa.eu/programmes/horizon2020/en/sc-1-health-demographic-change-and-wellbeing-work-programme-2016-2017-preparation.

51. European Commission. Horizon 2020 Advisory group for Health, demographic change and wellbeing (E02942) Register of Commission Expert Groups and Other Similar Entities 2016 07 Jul 2016 [cited 2016 August 31]; Available from: http://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetail&groupID=2942.

52. Fresco, L.O., et al. Commitement and Coherence - Ex-Post-Evaluation of the 7th EU Framework Programme (2007-2013). 2015 [cited 2016 August 30]; Available from: http://ec.europa.eu/research/evaluations/pdf/fp7_final_evaluation_expert_group_report.pdf#view=fit&pagemode=none.

53. European Commission - Advisory Group for Societal Challenge 1. Advice for 2018–2020 of the Horizon 2020 Advisory Group for Societal Challenge 1, "Health, Demographic Change and Well-being". 2016 [cited 2016 August 31]; Available from: http://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetailDoc&id=23709&no=1.

54. ERA Portal Austria. Horizon 2020 Programme Committees. [cited 2016 August 31]; Available from: https://era.gv.at/directory/165.

55. European Commission. Commission presents its evaluation of the 7th Framework Programme for Research. 2016 [cited 2016 August 31]; Available from: http://europa.eu/rapid/press-release_MEMO-16-146_en.htm.

56. Commission, E. Decision-making. European Comission at work 2016 [cited 2016 August 31]; Available from: http://ec.europa.eu/atwork/decision-making/index_en.htm.

57. Lelieveldt, H. and S. Princen, The Politics of the European Union. 2nd ed. 2015. 58. European Commission - Scientific Panel for Health. Scientific Panel for Health: 3rd

Plenary meeting 30 October 2015, Brussels, Executive summary. 2016 [cited 2016 August 30]; Available from: https://ec.europa.eu/programmes/horizon2020/sites/horizon2020/files/SPH_3rd-plenmeeting_exec-summary.pdf

59. European Commission. Scientific Panel for Health (SPH). HORIZON 2020 - The EU Framework Programme for Research and Innovation 2016 [cited 2016 August 31]; Available from: https://ec.europa.eu/programmes/horizon2020/en/h2020-section/scientific-panel-health-sph.

60. Health, E.C.-S.P.f. Better Research for Better Health - A vision for health and biomedical research from the Scientific Panel for Health. 2016 [cited 2016 August 31]; Available from: https://ec.europa.eu/programmes/horizon2020/sites/horizon2020/files/SPH_VisionPaper_02062016.pdf

61. REGULATION (EU) No 282/2014 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 11 March 2014 on the establishment of a third Programme for the Union's action in the field of health (2014-2020) and repealing Decision No 1350/2007/EC. 21.3.2014.

62. Communication of the European Commission. Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL on establishing a Health for Growth Programme, the third multi-annual programme of EU action in the field of health for the period 2014-2020. COM(2011) 709 final 2011 [cited 2016 August 31]; Available from: http://ec.europa.eu/health/programme/docs/prop_prog2014_en.pdf.

63. European Commission. ANNEX I Health Programme - Work Programme for 2016. 2016 [cited 2016 August 31]; Available from: http://ec.europa.eu/health/programme/docs/wp2016_annex_en.pdf.

64. European Commission. Ex-post evaluation of the 2nd Health programme 2008-2013 under Decision No 1350/2007/EC establishing a second programme of Community

24

action in the field of health (2008-2013). COM(2016) 243 final 2016 [cited 2016 August 31]; Available from: http://ec.europa.eu/health/programme/docs/ex-post_2nd-hp-2008-13_comm-report_en.pdf.

65. Joint Programming Initiative (JPI). The JPI - MYBL. 2016 [cited 2016 August 26]; Available from: http://www.jp-demographic.eu/.

66. Joint Programming Initiative (JPI), More Years, Better Lives - Strategic Research Agenda on Demographic Change, in Joint Programming Initiative (JPI) “More Years, Better Lives The Potential and Challenges of Demographic Change”. 2014: Brussels.

67. Pieper, R., et al. An internal JPI-MYBL survey on the relevance of research themes by the Societal Advisory Board (SOAB) and the General Assembly of the Joint Programming Initiative “More Years – Better Lives” (JPI-MYBL). 2013 [cited 2016 August 26]; Available from: http://www.jp-demographic.eu/wp-content/uploads/2015/08/SOAB-Survey-report-August-2013.pdf.

68. Joint Programming Initiative (JPI), The JPI - MYBL Process, in Exploration Alignment Implementation Public Engagement, Supplement to the Strategic Research Agenda. 2015: Brussels.

69. Oxman, A.D.S., Holger J.Fretheim, Atle, Improving the use of research evidence in guideline development: 2. Priority setting. Health Research Policy and Systems, 2006. 4(1): p. 1-7.

70. Hongoh, V.M., P.Gosselin, P.Samoura, K.Ravel, A.Campagna, C.Cissé, H. D.Waaub, J. P., Multi-stakeholder decision aid for improved prioritization of the public health impact of climate sensitive infectious diseases. International Journal of Environmental Research and Public Health, 2016. 13(4).

71. Robinson, S., et al., Setting priorities in health. Birmingham: Health Services Management Centre and Nuffield Trust, 2011.

72. Jo, H.S.K., D. I.Oh, M. K., National Priority Setting of Clinical Practice Guidelines Development for Chronic Disease Management. J Korean Med Sci, 2015. 30(12): p. 1733-42.

73. HSR-Europe, Health Services Research into European Policy and Practice. Final report of the HSREPP project. 2011: Utrecht.

74. Kinding, D., What Is Population Health?, in Improving Population Health - Policy. Practice. Research., D. Kinding, Editor. n.d.

75. Kindig, D. and G. Stoddart, What Is Population Health? American Journal of Public Health, 2003. 93(3): p. 380-383.

76. Kindig, D., What Are We Talking About When We Talk About Population Health?, in Health Affairs Blog. 2015.

IX. Appendix

Appendix 1: Overview of definitions used for working definition

Table 7: Overview of definitions used for working definition

Term Definition Source Priority setting “Who gets what at whose expense” [38]

“Priority setting is a ‘science’ intending to serve the needs of a community or a society at a specific point in time, within given policy, context, time limit, and financial constraints. It is value-driven and there are many interest

[38]

25

stakeholders who will necessarily promote a diverse set of opinions and values.”

Priority setting in health care

“Priority setting is a more or less systematic approach to distributing the available resources among demands to fashion the best health care system possible, given the constraints.”

[10]

“Processes by which decisions about the allocation of scarce health care resources are taken”

[71]

“Prioritization is a systematic approach to allocating resources for creating the ‘best’ health-care system, subject to a variety of demands and limited resources. [40]”

[72]

Distribution of limited resources among competing programmes or people.

Own definition

Health research priority setting

“Health research priority setting seeks to select priorities that will have the largest benefit to the health of populations, reduce duplication of effort and promote collaboration.”

[24]

“The primary aim of research priority setting is to gain consensus about areas where increased research effort including collaboration, coordination and investment will have wide benefits to society.”

“Priority-driven research has a clearly defined purpose, with an emphasis on answering questions of key importance that are likely to have a significant impact on knowledge or practice in the short to medium term. The use of systematic, explicit and transparent process of setting health research priorities ensures that research is funded that has the greatest potential public health benefit, that research funding and outputs are aligned with the needs of decision makers, and that there is efficient and equitable use of limited resources, with less duplication of research effort. Priority setting should be as evidence-based as possible, while also incorporating the views of a wide range of stakeholders.”

[25]

Health information Health information is all data, evidence and knowledge that determines health and health service performance at individual or population level to facilitate research, promotion, prevention, care and support policy-making.

[1]

Health research “Health research can take many forms, from clinical trials of drugs through to qualitative research studies.”

http://www.hra.nhs.uk/patients-and-

26

the-public-2/types-of-study/

“Health research is defined as ‘any activity that is undertake to generate presently non-existing knowledge that will eventually be used to reduce the existing disease burden (or other health-related problem) in human population.” “The tree most fundamental and mutually exclusive health research domains: i) health research to assess burden of health problem (disease) and its determinants, ii) health research to improve performance of existing capacities to reduce the burden, iii) health research to develop new capacities to reduce the burden.”

[38]

Medical research “It is the basic research (also called bench science or bench research),[1] applied research, or translational research conducted to aid and support the development of knowledge in the field of medicine.”

https://en.wikipedia.org/wiki/Medical_research

Health Services Research (HSR)

“HSR is the multidisciplinary field of scientific investigation that studies how social factors, financial systems, organisational structures and processes, health technologies and personal behaviours affect access to health care, the quality and cost of health care and, ultimately, the health and wellbeing of citizens (Lohr and Steinwachs, 2002; AcademyHealth, 2007).”

[73]

Population health Population health is defined as the health outcomes of a group of individuals, including the distribution of such outcomes within the group.

[74-76]

27

Appendix 2: Search strategy

The following search strings were used:

• “health information” OR

• “(priority setting OR setting priorities OR target setting OR setting targets OR health goal* OR health target*)” AND

• followed by respective terms for the different aspects and levels of health information: “research”, “health system”, “public health”, “prevention”, “promotion” and “policy making”

Table 8: Search strategy

Setting/Planning PRIORITY HEALTH / RESEARCH

METHODS

EM Health care planning

ME health priorities

EM Health EM methodology

ME Health planning

EM research priority

EM research EM conceptual framework

EM Health care policy

ti/ab priori* : priorities / prioritize / prioritization / prioritizing

INFORMATION

ti/ab methods / methodology

EM Resource allocation

ti/ab agenda EM Information processing

ti/ab concept / conceptual

EM Decision making

ti/ab target ti/ab expected value of information

ti/ab framework

ME Decision making

ti/ab Goal

ti/ab criteria of/for

EM Health care

ti/ab policy ti/ab recommendation*

ME Policy making

REVIEW

ti/ab guide / guidelines

EM Group process

EM Review ti/ab tool*

28

EM Consensus ME review (publication type)

ti/ab setting / planning / rationing / allocating / making

EM…Embase; ME…Medline/Pubmed; ti/ab…title/abstract

Appendix 3: Questionnaire to BRIDGE Health partners

1. In your area, have you been engaged so far in tasks of priority setting in general and health information in particular?

a) Please specify in one or two sentences. 2. What methods have you used? What are the priority setting methods you are

familiar with and/or have experience with? (e.g. consensus based approach or a metrics based approach (pooling individual rankings), or a combination)

a) Please specify. b) In your opinion, what are common challenges when deciding on priorities in

health policy areas in general? 3. How are priorities in health information identified in your country at national,

regional and local level? What methods/systematic approaches are being used? 4. Are you aware of best practices / case studies for priority setting methods? If yes,

which ones? 5. Are you aware of global initiatives and/or initiatives in your country as a means of

priority setting which develop and define health goals?

29

Appendix 4: Viergever’s Nine Common Themes of Good Practice’ conceptual framework for priority setting in health research

Table 9: Nine Common Themes of Good Practice’ conceptual framework and its adaption for the evaluation of health target setting process adapted from [24]

Theme Description as outline in the checklist [28]

Evaluation questions adapted from the checklist Viergever, 2010)

Context Articulating the contextual factors that underpin the process

1. Will the established goals, underlying values and principles continue to be relevant the next time the program facilitates priority setting?

2. Are there changes to the number of resources available for the next priority setting cycle?

Inclusiveness Deciding who should be involved in setting research priorities

3. Did appropriate stakeholders participate in the most recent priority setting cycle and was there balanced representation?

Information gathering

Choosing what information should be gathered to inform the process

4. Was the most recent priority setting exercise appropriately informed?

5. Did the provided information sources support decision making?

Planning for implementation

Establishing plans for translating research priorities into projects

6. In previous cycles, were there challenges to translating the research priorities into research?

Criteria Selecting relevant criteria to focus discussion

7. In the most recent priority setting cycle, were the criteria effective for decision making, and will the criteria continue to be relevant for the next cycle?

Methods for deciding on priorities

Choosing a method for deciding on priorities

8. In the most recent priority setting cycle, were the methods for deciding on priorities appropriate and effective for decision making?

Use of a comprehensive approach

Assessing whether a comprehensive approach is necessary or if a tailored process and methods are required

9. Are there elements in comprehensive approaches and priority setting methods which are transferable to the priority setting process? Specifically, the Listening Model, COHRED, CHNRI, ENHR, CAM and Delphi technique.

Transparency Communicating the approach that was used to set priorities

10. Did all stakeholders receive information about the process and outcomes of the most recent priority setting process?

Evaluation Defining when and how evaluation of process and outcome will occur

11. Are further evaluation activities required to assess the delivery and outcomes of the priority setting process?

30

Appendix 5: IRIS prioritization tool used by ECDC

Source: [36]

31