GUIDELINE Disinvestment - CCATESccates.org.br/content/_pdf/en_PUB_1437692942.pdf · 2...

Faculdade de Farmácia - UFMG

Dep. de Farmácia Social

www.ccates.org.br

GUIDELINE

Disinvestment

Belo Horizonte - MG

April - 2015

METHODOLOGICAL GUIDELINE Disinvestment

This work was developed as part of cooperation agreement between the Department

of Management and Incorporation of Health Technologies of the Ministry of Health of

Brazil, the Pan American Health Organization and the SUS Collaborating Centre for

Technology Assessment and Excellence in Health.

Elaboration:

Lívia Lovato Pires de Lemos (CCATES/UFMG)

Juliana Alvares (CCATES/UFMG)

Daniel Resende Faleiros (CCATES/UFMG)

Renata Cristina Rezende Macedo do Nascimento (CCATES/UFMG)

Ana Luísa Caires de Souza (SES-MG)

Augusto Guerra Afonso Júnior – coordination (CCATES/UFMG)

Contributors:

Alessandra Maciel Almeida (CCATES/UFMG)

Cristina Mariano Ruas Brandão (CCATES/UFMG)

Eli Iola Gurgel Andrade (FM/UFMG)

Juliana de Oliveira Costa (CCATES/UFMG)

Mariângela Leal Cherchiglia (FM/UFMG)

Rosângela Maria Gomes (CCATES/UFMG)

Vânia Eloísa Araújo (CCATES/UFMG)

Expert Review:

Brian Goodman (Karolinska Institute, Stockholm, Sweden and Strathclyde Institute of

Pharmacy and Biomedical Sciences, Strathclyde University, Glasgow, UK)

Technical review:

Francisco de Assis Acurcio (CCATES/UFMG)


SUMMARY

PRESENTATION ................................................................................................................. 2

1. INTRODUTION ....................................................................................................... 3

2. STEPS OF DIVESTMENT PROCESS ......................................................................... 6

3. PERMANENT PROGRAM OF PERFORMANCE EVALUATION OF TECHNOLOGIES .. 8

4. IDENTIFICATION OF POTENTIAL TECHNOLOGIES FOR DISINVESTMENT .............. 9

4.1. Active search ....................................................................................................... 11

4.2. Demands from society ........................................................................................ 13

4.2.1. Conformity analysis ............................................................................................ 14

5. PRIORITIZATION .................................................................................................. 15

6. HEALTH TECHNOLOGY REASSESSMENT (HTR) ................................................... 19

6.1. Disinvestiment modalities .................................................................................. 19

6.1.1. Price renegotiation for public health systems .................................................... 22

6.2. Health Technology Reassessment process ......................................................... 22

6.3. Disinvestment recommendation ........................................................................ 24

6.3.1. Transition Period ................................................................................................. 25

7. IMPLEMENTATION OF DISINVESTMENT DECISION ............................................ 27

7.1. Organizational implications ................................................................................ 27

7.2. Protocols and compendiums update .................................................................. 27

7.3. Regulatory mechanisms and incentives ............................................................. 28

7.4. Dissemination ..................................................................................................... 28

7.4.1. Dissemination and adhesion strategies .............................................................. 29

7.5. Monitoring the disinvestment ............................................................................ 30

8. FINAL REMARKS .................................................................................................. 31

REFERENCES .................................................................................................................... 33

2


PRESENTATION

In recent years, significant advances have been seen in the field of health technologies

incorporation. Standardized analysis of the efficacy, safety, effectiveness and efficiency

have contributed in a significant way to the new technologies incorporation process –

or to validate technologies already incorporated – in health systems. However, many

challenges still permeate the yet unexplored process of disinvestment of health

technologies.

Health technologies disinvestment can be understood as the removal of health

resources from existing health care technologies or practices that now provide little or

no health gain for their additional costs. Consequently, it aims to maximize the health

gain from available resources. As a result, the technologies evaluated are those

incorporated into health systems that, in some way, no longer present an acceptable

cost-effectiveness ratio, or those that present safety problems or even those that have

become obsolete with the passage of time.

Thus if the evidence is confirmed, the technology should no longer be financed by the

health system. The resources normally used for its provision can, then, be allocated to

other technologies that provide real benefit to the population, or to subgroups of

patients – the ones that present the best investment return.

This document aims to aid managers and researchers in developing recommendations

for the divestment of health technologies, in particular medicines, improving the

quality use of health care resources.

3


1. INTRODUTION

According to the World Health Organization (WHO), health technology “refers to the

application of organized knowledge and skills in the form of devices, medicines,

vaccines, procedures and systems developed to solve a health problem and improve

quality of lives" (OMS, 2015). In a simplified way, health technologies are those that

promote health, prevent and treat diseases and rehabilitate people.

New health technologies incorporation processes – already well established in

European countries – have been used in Brazil, and other American countries to

rationalize the increasing expenditure on health. These processes help to combat

technological speculation, understood as replacing the use of traditional technologies

by other, often without clearly defined utility, increasing the cost of treatment,

without actually improving results. However, unlike other sectors, incorporation of

health technologies often is not substitutive or resource saving (SILVA JUNIOR E ALVES,

2007).

In health systems where resources are limited and demands are endless, opportunity

costs – understood as the cost of a particular technology measured by the benefits lost

by stop investing in other technology (LAPORTE, 2001) – suggests disinvestment as a

rationalization strategy of the use of health care resources.

Many terms have been used for disinvestment (MACKEAN et al., 2013). However,

disinvestment is generally understood to be the partial or complete withdrawal of a

particular technology from the list of technologies provided to the population and has

as its main objectives the exclusion of a technology with safety issue or with a

unfavorable cost-effectiveness ratio. As a result, an explicit view of re-allocating

resources to higher value technologies (ELSHAUG et al., 2007; CADTH, 2009).

However, several challenges related to disinvestment have to be highlighted. These

include:

4


Identification of a candidate technology: In most cases the obsolescence is not

easily identified. In addition, there may be obsolescence gradients such as

patient groups that benefit. However, may be difficult identifying,

characterizing and enclosing this sub-group. In addition, local specificities such

as greater or lower use of technology.

The need for greater theoretical basis compared to technology incorporation

process. This is because it represents the interruption or modification of a

service, managers, health professionals and patients and they should be

"convinced" that the interruption reflects benefit. Quality, accuracy and validity

specifications of technical and scientific studies will be more complex and

robust. Furthermore, possible divestment processes or technologies will not be

encouraged or requested by the commercial organizations, meaning that the

production of such studies are more likely to be funded exclusively by the

health system. This fact is relevant when there is a need for conducting primary

studies, often expensive and long.

Social and political difficulty of removing a technology from clinical practice:

This difficulty occurs especially when there is no replacement by other

technologies considered "better". One way to minimize this challenge is to

make it clear to every stakeholder the difference between pure resource saving

and the will to improve health care and health outcomes efficiently (HENSHALL

et al., 2012).

Not to waste resources already invested: For health systems that purchase,

stock and provide health technologies, such as medicines and medical devices,

it is not enough just to identify the technology and stop providing to citizens.

Managers should consider the need for consumption of stocks, as well as

reverse logistics issues that may present themselves as complex, requiring joint

efforts of managers at different levels.

For the disinvestment to occur, it takes the active involvement of diverse social actors

in all stages of the process. Evaluations and decisions should be transparent and

5


implementation should include the transfer of knowledge to the actors (HENSHALL et

al., 2012). Users/consumers of health systems should be encouraged to recognize their

role as financiers of technologies, not only those that directly impact them, but also of

all others. It is important that they understand the strategies adopted for the

sustainability of the health system.

Disinvestment is an important part of the process that establishes the use of

technologies in health systems. It is a process that demands permanent monitoring of

the effectiveness of technologies used. The process requires planning of the legal and

organizational mechanisms to allow and facilitate both the investment and

disinvestment of technologies, enabling excellence in service to users and rational use

of scarce health care resources.

6


2. STEPS OF DIVESTMENT PROCESS

The health technology disinvestment process itself may be more complex than the

health technology incorporation (investment). However, it basically follows the same

steps, as shown in Figure 1.

The demands can originate from within the health system with the prospective

identification/ active search, or originate from society. Conformity with social demands

should be assessed, while the originating from health system will already be made

appropriately and in accordance with previously established parameters. Following

this, a priority setting analysis is undertaken.

After the prioritization, the health technology disinvestment assessment begins. If the

result leads to disinvestment recommendation but there are a subgroup of patients

that benefit from the technology, or, in other words, it has an acceptable cost-

effectiveness ratio for a sub-group of patients but not for everyone, it is suitable to

implement restrictions to use and/or to renegotiate prices.

If there is a technology, identified in the disinvestment analysis that could replace the

one listed, substitution can occur. If substitution is not possible, full withdrawal or

retraction and/or price renegotiation is recommended. The final conclusion to not

disinvest, or to totally or partially disinvest, is accompanied by a complete report to

the principal manager of the health system, who will decide to maintain the

technology in all or some patients or to disinvest in all or sub-groups of patients.

After the decision is publicized and, if the manager choses to disinvest, the execution

of the disinvestment process is initiated considering the specific aspects of the

technology in question identified during the analysis.

7


Figure 1. Stages of health technology disinvestment process.

8


3. PERMANENT PROGRAM OF PERFORMANCE EVALUATION OF TECHNOLOGIES

The health system should establish a team to undertake such analysis - the ‘Permanent

Program of Performance Evaluation of Technologies’. This team has the responsibility

for carrying out the whole process of disinvestment of health technology as well as to

actively search candidate technologies to disinvest.

Patient associations, medical societies, and research centers should be considered

partners of the Program. The latter two may be invited to participate in both the

prospect of candidate technologies, and in the re-evaluation of the candidate

technologies.

The team has the responsibility to continually monitor the effectiveness of the

technologies used in the health system, as well as those offered in the market. It is also

part of the activities of the Program, to plan legal and organizational mechanisms to

enable and facilitate the technology disinvestment, enabling excellence in service to

users and rational use of health care resources.

The Program team is also responsible for developing and implementing strategies to

promote the active involvement of social actors in all stages of the disinvestment

process, promoting transparency of their actions and the transfer of knowledge. It is

also responsibility of the team to promote awareness of the role that the

users/consumers play in the disinvestment process, providing them with relevant

information.

9


4. IDENTIFICATION OF POTENTIAL TECHNOLOGIES FOR DISINVESTMENT

Health technology indication for disinvestment analysis can arise from within health

systems or from society. Pertinent technologies should be considered when the

technology provided by the health system is completely or partially inadequate in

meeting essential requirements for its use as specified in detail below:

Safety

o There is unacceptable potential safety risk for users, society or the

environment related to the use of technology;

o There is evidence demonstrating that the technology generates

unacceptable risk-benefit concerns;

o There is evidence showing that new technologies get the same results,

but record significantly higher levels of patient safety.

Effectiveness

o Absence or low evidence of efficacy and effectiveness of the

technology;

o There is evidence showing inefficiency and ineffectiveness of the

technology in one or more indications;

o There is evidence showing that new/other technologies, with the same

results, have better cost-effectiveness ratio;

o There is evidence showing that new technologies, with same price,

present much better results of efficacy/effectiveness.

Cost

o There is evidence demonstrating that the monetary value of the

technology is not translated in the proposed benefit;

o There is evidence showing that there are other technologies, that get

the same results with lower prices/ costs;

10


o There is no reduction in the deflated price of the technology over time,

showing that its price is not compatible to the price of other

technologies available in the market that provide the same results.

Cost-effectiveness

o There is evidence proving that a competing technology presents a

better cost-effectiveness ratio.

Disuse

o There is evidence showing that the technology is no longer being used

in the health system.

Inappropriate use

o There is evidence showing that the technology is being improperly used.

For example, in excess, or when it is used by groups for which the

technology is not intended to.

Logistics

o There is evidence showing that a competing technology needs a

simplified logistics plan, with proven lower costs.

Availability

o Unavailability in the market.

Acceptability

o There is evidence showing that the technology produces important

discomfort/pain;

o There is evidence showing that the technology produces negative

repercussions because it is considered "invasive";

o There is evidence showing that the technology leads to poor quality of

life;

o There is evidence showing concerns with patient compliance/

adherence with a given technology.

11


Adequacy

o There is evidence showing that the technology is not relevant for clinical

practice.

Contraindications

The lack of alternative therapy can counter indicate the technology to the divestment

process, with exceptions in cases of safety problems or significant negative balance

between benefits and risks.

4.1. Active search

Active searching for candidate technologies for disinvestment should be part of the

evaluation process of technologies incorporated into any health care system. The

"Permanent Program of Performance Evaluation of Technologies" can perform a

periodic review of technologies provided at all levels of care.

An important strategy is to search and review all supplies, procedures and medicines

used by a medical specialty. Thus, all items identified can be reassessed in order to

optimize the services of this health field and prevent unmanaged replacement or even

overuse other technologies, phenomena that can happen if a technology is randomly

selected and disinvested. The medical societies and patient associations can be

collaborators in this identification.

Considering the limited resources and the growing need for investment in new

technologies, special attention should be given to technologies that represent higher

unit costs or additional financial resources. Research methods on the use of drugs such

as suggested by the World Health Organization, are recommended for the

identification of candidate technologies to divestment (Drug and Therapeutics

Committees: A Practical Guide

http://apps.who.int/medicinedocs/pdf/s4882e/s4882e.pdf).

12


It is also important to assess the unnecessary use of technologies that consume

resource without effective results, e.g. mouthwash with fluoride associated with

fluoride toothpaste, that has no additional health gain to brushing with fluoridated

toothpaste alone (MARINHO et al., 2004).

The use of clinical protocols and their periodic update are relevant tools for the

rational use of technologies and associated costs. In practice, the evaluation for the

incorporation of a new technology could lead to the identification of those candidates

for disinvestment. The active search for reports and alerts, e.g. adverse reactions,

contraindications and restrictions of use, from regulatory agencies and international

organizations and health technology assessment institutes, also helps with the

identification of technologies for the disinvestment process.

Non-drug technologies, often including procedures, can be more difficult to identify

during the review of the list of included technologies. This problem can be overcome

by systematic reviewing the literature at fixed intervals. Elshaug et al. (2012) used

specific terms combined in an electronic databases search strategy to identify non-

drug technologies candidates for disinvestment (Chart 1).

13


Chart 1. Bibliographic search strategies for monitoring non-drug technologies candidates for divestment

Item Terms

Safety (unsaf*) OR (danger*) OR (adverse event) OR (poor outcome) OR (low quality) OR (harm*) OR (containdicat*)

OR

Effectiveness

(ineffect*) OR (supersede*) OR (irrelevant*) OR (outdated) OR (new evidence) OR (overuse*) OR (unproven) OR (inappropriate*) OR (equivoc*) OR (uncertain*) OR (obsolete) OR (inferiority) OR (superiority)

OR

Political solutions

(disinvest*) OR (coverage with evidence development) OR (CED) OR (access with evidence development) OR (AED) OR (access with evidence generation) OR (reallocat*) OR (resource release) OR (reinvest*)

NOT

Pharmaceutical exclusion

(drug therapy [mh]) OR (drug industry [mh]) OR (pharmaceutical services [mh]) OR (pharmaceutical preparations [mh]) OR (pharmacogenetics [mh]) OR (pharmacoepidemiology [mh]) OR (technology, pharmaceutical [mh])

* : Truncation character; [mh]: Medical subject heading

4.2. Demands from society

Just as it happens for requests for inclusion, potential exclusion of health technologies

can be requested by any social actor - citizens, health professionals, associations,

industries, among others.

The applicant must submit to the disinvestment manager in the health system

documentation containing, when applicable, the following information about the

technology:

Name of technology (avoid brand name);

Indication for which the disinvestment is proposed;

Rationale justifying the evaluation for disinvestment (based on scientific

evidence and/or safety alerts);

Indication of disinvestment modality (price renegotiation; full delisting;

restriction; retraction; or substitution);

Existing and available alternative to the specified technology

Suggestion for relocation of possible funds raised in the case of divestment.

14


4.2.1. Conformity analysis

The social demands for disinvestment must pass a document verification analysis that

all required documents are present. At this stage, a merits assessment can be

undertaken and the demand for a potential disinvestment analysis can be judged as

valid or invalid according to the following criteria:

The technology is already ongoing analysis for disinvestment;

The technology is not listed in the health system;

The documentation does not address the requirements for evaluation.

15


5. PRIORITIZATION

When different technologies are identified and there is no possibility to analyze them

concurrently, it may be necessary to consider their priority to perform disinvestment

analysis. In such cases it is recommended to use “Multicriteria Decision Analysis” using

Value Measurement Models (using the weighted sum, Analytic Hierarchy Process ou

Programme Budgeting and Marginal Analysis). In this method, criteria and respective

weights are established. The weights must reflect the importance of each criterion for

decision making (DEPARTMENT FOR COMMUNITIES AND LOCAL GOVERNMENT, 2009;

THOKALA and DUENAS, 2012).

For divestment some minimum criteria, adapted from the criteria proposed by Elshaug

et al. (2009) for the divestment and from the criteria used by the Canadian Agency for

Drugs and Technologies in Health (CADTH) for prioritizing technologies for evaluation

in the context of merger / financing (HUSEREAU et al., 2010), are suggested Different

criteria may be set according to the type of technology, e.g. medicines, devices,

surgical techniques, etc., or groups of diseases.

16


The steps to be followed in the demands prioritization process are shown in Figure 2

and explained in detail below:

Figure 2. Prioritization process of technologies for Health Technology Reassessment

•Among the identified technologies, the ones related to health risks should be prioritized

Safety issue

•High cost per procedure, high cost due to the volume, or an aggregate measure of both

Cost of service

•Related to health: eg., rough estimate of quality of life adjusted years lost

•Related to costs: eg., gross estimated savings per patient; release of additional resources, etc.

•Overall assessment of the maintenance of equity in care if the financing of health technology is modified (eg., access for subgroups of patients)

Probable impacts

•Priority should be given to technologies for which there are cheaper alternative with equivalent or better results

Cost-effective alternative

•Conditions associated with low disability or morbidity and low mortality rates (excluding orphan diseases) can influence the prioritization differently health conditions with high disability / morbidity or mortality. Low burden conditions may reduce the potential for dispute; high burden diseases can pose a greater scope for reinvestment / reallocation of resources

Burden of disease

•Rigorous assessment requires robust evidence. Typically the evidence is not 100% conclusive, but they must be suitable to be useful in decision making

Sufficient evidence available

•Time and budget possibility of conducting a study to support decision making when there is little evidence available

Possibility to generate evidence for decision-

making

•An intervention that probably does not result in "significant survival" or benefit can be prioritized

Futility

17


Step 1: Preparation of a previously structured executive summary for each of

technology, so that it is feasible to compare the main features of the various demands.

This summary should emphasize absolute numbers that use comparable features with

same level of importance and use the same scale of measure (whenever possible).

Strategic indicators must be included, e.g.: burden of disease; level of complexity of

care, e.g. primary care, high dependency care, intensive care, financial values and the

population covered.

Step 2: Formation of an expert panel for the evaluation of the demands. It is advisable

to collect declaration of conflict of interest from all members.

Step 3: Send to the panelists the executive summaries of the technologies under

consideration and the prioritization matrix.

Step 4: Analysis of the findings with each individual completing the priority matrix.

Each participant assigns weights 1 to 5 for the technologies under consideration. At

this stage, technologies are compared to each other with respect to each criterion, and

there may be a tie. When there are subcriteria these can be translated into numbers,

that is, equivalent to 5 "very serious" (safety), or "no" (clinical impact) and 1 equals

"no" (safety) or too high (clinical impact) (Chart 2).

Chart 2. Example of prioritization matrix for Health Technology Disinvestment Assessment

Technologies

Criteria

Safety Issue

Costs Probable Impacts

Alternatives Burden of

disease Sufficient evidence

Possibility to

generate evidence

Futility

Technology A 1-5 1-5 1-5 1-5 1-5 1-5 1-5

Technology B 1-5 1-5 1-5 1-5 1-5 1-5 1-5

... ... ... ... ... ... ... ...

Technology Z 1-5 1-5 1-5 1-5 1-5 1-5 1-5

18


After completing the matrix, the final score is given by:

Final Score = W1 S1 + W2 S2 + W3 S3 + W4 S4 + W5 S5 + W6 S6 + W7 S7 +...+ Wi Si

In which:

“W” is the criterion weight

"S" is the score attributed to the technology concerning each criterion

Step 5: Submit the completed matrix to the convened panel, which lists the

technologies in descending order of score, i.e., from the priority one for non-

priority one. In the case of disagreement, or in the case of tie, a meeting can be

called to establish consensus.

19


6. HEALTH TECHNOLOGY REASSESSMENT (HTR)

Disinvestment in health is indicated when a particular technology no longer presents a

good cost-effectiveness ratio, or poor safety, adhesion, acceptance and adequacy

registries. The purpose of the HTR is to provide clear and complete information to the

health care system manager to aid in the decision making process.

The RTS report should be succinct and recommendation must be clear and objective.

These aspects are very relevant, since scientific information should be understandable

and capable of guiding and to support the decisions of managers.

6.1. Disinvestiment modalities

The disinvestment in a health technology may occur in the following forms, not

mutually exclusive: Full delisting/full withdrawal; restriction (limitation of access);

retraction (reduction of the number of times a technology is supplied); and

substitution in all or a defined sub-group of patients. Each of these modalities has

distinct advantages and implications for clinical practice and management (Chart 3). In

addition to restriction or retraction, renegotiation of prices is advisable.

Full withdrawal/full delisting refers to the total withdrawal of the supply of certain

technology by the health system.

Restriction is applicable when the technology is now restricted to groups or subgroups

of users who strictly meet the criteria for the use of the product/service.

Retraction refers to the reduction in the number of times a product/services is offered

to the user/ patient. It should occur when the decrease in the number of times the

technology is offered does not affect the health outcomes expected. For example,

increasing the time period between a screening test.

20


Substitution is the form of disinvestment adopted when it is found that there is an

alternative available, with the same results of the technology provided by the health

system, proven to be more cost-effective.

21


Chart 3. Meanings, advantages and disadvantages of each disinvestment modality Modalities Meaning Advantages Disadvantages

Full delisting/Full withdrawal

The removal by the health system of a particular technology

It offers greater and faster rationalization of the use of health care resources

It can help prevent ambiguity about the availability of services and products to groups or subgroups of users

In most times it is unpopular

Political difficulties to implement (principle of continuity of public service)

Potential increase in the demand of alternatives (unmanaged substitution), including judicial

It can take a long time to implement

Restriction The provision of technology by the health system will be restricted to groups or subgroups of users, with strictly the criteria for its use

It offers rationalization of the use of health care resources

It can be reversed or enlarged as required

It can promote safer services

Politically unpopular

It may seem discriminatory

Potential increase in demand for alternatives (unmanaged substitution)

Difficult to monitor and maintain

Retraction The supply of the technology by the health system will be reduced in number, e.g. 1 instead of 2 mamographies a year)

It offers fast rationalization of the use of health care resources

Potentially more acceptable than full withdrawal

Potential increase in demand for alternatives (unmanaged substitution)

Can be difficult to monitor and maintain in practice

Lower financial benefits than full withdrawal

Substitution The currently offered technology will be replaced by another that with the same of with higher performance and know to be cost-effective

It ensures that an equivalent treatment/service is available

Potentially more acceptable than full withdrawal, restriction and retraction

It can help prevent unmanaged substitution

Guarantee of the clinical efficacy of the substitute technology

It may be unpopular

Lower financial benefits than full withdrawal

Adapted from Daniels et al. (2013)

22


6.1.1. Price renegotiation for public health systems

When the Health Technology Reassessment (HTR) indicates the restriction or

retraction of a given technology, it is essential that managers undertake a comparative

costs analysis of the treatment or of the technology use, so as to negotiate with

suppliers prices compatible with the market and with the new consumption scenario.

This practice intends to price a technology according to its performance (value-based

pricing).

For the price renegotiation, the manager has important tools: HTR results, with data

on safety and cost-effectiveness; purchasing power due to the high volume and

centralization of acquisition of higher added value items; and international price

parameters.

The introduction of a new item, replacing other previously listed ones, should also be

preceded by price negotiations, with the possibility of immediate or stepped cost

reductions.

It is noteworthy that the price renegotiation can be used as an alternative in order to

maintain a technology on the list. If the price decrease converts a technology

considered not cost-effective to cost-effective status, the maintenance of this

technology option may be preferable to disinvestment.

6.2. Health Technology Reassessment process

The process of Health Technology Reassessment (HTR) resembles the one for Health

Technology Assessment (HTA). The main difference lies in the fact that for

disinvestment, additional data on access and on organizational and logistical aspects

are analyzed.

After the steps of technologies performance evaluation, search for candidate

technologies to disinvestment and prioritizing, HTDA steps are shown below:

23


Understanding the problem to be evaluated

Investigation of the treatment pathways of which the technology is part

Investigation of the annual requests and effective use of the technology

Analysis of the technology taking into account safety data (phase I and phase II

clinical trials); efficacy (phase III clinical trials); effectiveness (observational

studies and clinical trials); efficiency among competing technologies (cost

effectiveness and cost utility analysis); feasibility between competing

technologies (budget impact analysis), considering:

a. Retrieval of the available evidence

b. Generation of new primary data (if necessary)

i. Non-inferiority clinical trial

ii. Registry

c. Synthesis of the evidence

d. Interpretation of the available evidence

When applicable, a description of the logistics stages of programming,

purchase, distribution, storage and supply

When applicable, evaluation of patient and professional adherence to

technology

Calculation of the financial savings resulting from the possible rationalization

of the use of health resources resulting from the disinvestment of a given

technology

Preparation of the report

a. Minimal content:

i. Executive summary

ii. Introduction

1. Specification of the problem to be evaluated

2. Explanation of the rationale for the evaluation

3. Description of the treatment pathways of which the

technology is part

24


4. Description of the annual requests and effective use of

the technology

iii. Search/source of evidence description

iv. Synthesis of the evidence

v. Recommendations

1. Elaboration of scenarios

2. Indication of duration of the Transition Period

It is noteworthy that, in cases where the technology is ineffective or present security

problem in studies with active comparator, cost-effectiveness studies may not be

performed, still it is necessary to conduct cost analysis.

6.3. Disinvestment recommendation

The recommendation to disinvest in a health technology should point out that

treatment alternatives are available, especially those available to users of the health

system, because, even if there are problems with safety, or the technology in question

does not present favorable cost-effectiveness ratio, the absence of alternative therapy

may contraindicate the disinvestment.

It is important that, whenever possible, the recommendation briefly present possible

scenarios - from maintaining the technology in its current format until disinvestment in

each indicated modality; with their clinical and social consequences; necessary

measures; and estimated amount to be made available for relocation for each

scenario.

An experiment with the removal technology from some health services can also be

suggested. The results of this study may be able to confirm the evidence of

ineffectiveness described in the RTS and then guide the final decision of the manager.

This suggestion, however, should already be accompanied by indications of partner

institutions for management and the study design.

25


The formation of panels with representatives of health professionals, patient

organizations, general population and managers help to legitimize the divestment

process. Participants should be chosen by their peers, considering their technical

participation, in addition to their degree of interest in the results of the assessment. It

is important that the result of HTR is subject to public consultation after its

completion.

6.3.1. Transition Period

The report sent to the health system manager, with the results of the analysis and the

recommendation for disinvestment, should provide for a transitional period, taking

into account the time required to implement the decision, as well as for organizational

aspects and the required dissemination strategies.

The transition period may assume three time horizons from the decision of the health

system chief manager, and should be agreed with the actors responsible for

implementing decision, according to the following description:

Short term: up to six months

Medium term: one year

Long-term: two to three years

In the case of a health system that buys and stocks the technologies (products and

services), the duration of the transitional period should consider the time required for

the consumption of inventories and the finalization of purchase contracts, and, if

applicable, the period to organize the logistics for acquisition, storage and distribution

of the substitute product (Chart 4).

26


Chart 4. Strategies for completion of product inventories and services according disinvestment modality

Technology use Modality of disinvestment

Full delisting Restriction Substitution

Continuous Patients using will continue to receive up to consumption of inventories.

New demands will not be attended

Justification the requirement for the prescription

New demands will be attended in accordance with the restrictions established.

Patients using will continue to receive up to consumption of inventories.

New demands will be attended with the new technology.

Sporadic New demands will be attended until the consumption of inventories.

Justification requirement for the prescription

Patients will receive the technology in stock (the disinvested one or the new one)

In the retraction disinvestment modality, as the technology will continue to be offered,

but in smaller quantities, the duration of the transition period should take into account

the dissemination of the decision.

When the decision for disinvestment is based on safety-related issues, it is

recommended that the transition period be kept to a minimum, considering the time

required for dissemination of the decision and the necessary logistics.

27


7. IMPLEMENTATION OF DISINVESTMENT DECISION

7.1. Organizational implications

When it is decided to disinvest in a technology, the managers must be alert to a series

of aspects that will have to be carefully evaluated and modified. With regard to human

resources, it may be necessary to redistribute, hire and train staff to put together an

assistance network for the dissemination and, when needed, to deal with the

substitute technology.

It will be necessary to implement logistics measures to collect the remaining products,

and any publicity material that informs patients and physicians that the disinvested

technology is provided by the health system.

It is suggested to develop Standard Operating Procedures with sufficient

information/complexity to deal with problems related to the interruption in the supply

of any given technology to the population. For example, what to do, or what to inform

a citizen that presents themselves to a healthcare provider with a document informing

them that the technology is provided to users (in the previous standards).

7.2. Protocols and compendiums update

Clinical protocols and drug lists should be created or updated to include the

disinvestment. For health systems in which the supply of a technology demands

protocol compliance, inclusion and exclusion criteria should be (re)defined to include

disinvestment in the modalities restriction and retraction. In health systems in which

the primary care provider is the first contact between patient and care, clinical

protocols should cover recommendations for referral to specialist care.

It is advisable to include in the clinical protocols "what not to do", such as the pattern

adopted by the National Institute for Health and Care Excellence (NICE), to reduce the

number of unnecessary interventions or of doubtful value in an individual level.

28


It is advisable to develop simplified guides with key messages in different versions for

health professionals, managers and patients, which can be physically or electronically

distributed along with the clinical protocols.

To keep up with media trends, it is also suggested to develop explanatory videos about

the diagnosis and treatment of the disease, and for the dissemination of the new

protocol. These videos can be made available on the internet and physically distributed

in partnership with professional councils and patient associations.

7.3. Regulatory mechanisms and incentives

For the supply of medicines and rehabilitation procedures guideline compliance

assessment can be performed to ensure that the established health care decision tree

is being followed. “Pay for performance” can also be used, in which the health system

better remunerates the provider for following the guidelines.

7.4. Dissemination

The disinvestment dissemination strategies focus on the main social actors involved in

the provision of technology: managers, health professionals and users, their

associations and other players in this Government. The dissemination documents

should aim at transparently explaining the decision. There should be clearly

explanations of the causes and consequences of the disinvestment decision so as to

assure the full understanding of those who provide and use the technology, as well as

to prevent the occurrence of unmanaged substitution and judicial demands.

The managers of all sectors involved should be considered partners in the execution of

the decision and its implementation and so all must have access to complete

information on the exclusion/inclusion of technology. The dissemination can occur

through official documents and meetings with the participation of representatives of

health system users.

29


Health professional councils should also be considered as partners in the

implementation of any disinvestment decisions, as they can provide important tools

for dissemination such as magazines and journals, and the dissemination of

information through e-mail or newsletter to their affiliates.

Patients associations are also recognized key stakeholders in the implementation of

the divestment as their support is critical to the success of the intent.

7.4.1. Dissemination and adhesion strategies

For prescribers, Academic Detailing is suggested as a key tactic for the dissemination

of disinvestment. In this strategy, inspired by the Pharmaceutical Industry, prescribers

are visited for the dissemination of inclusions and divestments, and carefully explained

on the scientific evidence that led to them. On this occasion, the prescriber will also be

informed about the transition time and mechanism required to justify future

prescriptions. In the case of restrictions or retraction disinvestment modalities. In

addition during the visits, advertising materials (bulletin, flayers etc) directed to

patients will be delivered, so that each prescriber becomes a partner in the

dissemination of information about the disinvestment.

This approach can be adapted for visits to Government stakeholders such as public

defenders and judges, focusing on clear explanations for the disinvestment. It is

necessary to inform what the health system will continue, or will provide for the

treatment of the disease that was affected by the divestment and how will be the

access to these technologies.

The dissemination to users is especially important in the case technologies used

continuously and can occur through the distribution of a bulletin, in simplified

language, explaining the disinvestment of a particular technology. This bulletin can be

distributed to users by the pertinent health team. The important thing is that the

health system user obtain reliable information in a clear and complete manner, of the

30


reasons for disinvestment, its consequences and, when applicable, information about

the substitute technology

The bulletin must answer questions such as:

Will I stop receiving my medicine? - Why? – Until when will I receive my

medicine? - My health is at risk because I have used this medication until today?

- Should I stop using this medicine today? – Does my doctor know this?

The users should be instructed to seek their doctor for more information and

treatment adjustments. Mass dissemination channels such as radio, television,

newspapers and the internet can also be used when needed.

When possible, dissemination to users should occur after the dissemination to health

professionals and the Judiciary Power, since these should be prepared to answer

questions and requests from users regarding the disinvestment. It is also strongly

advised the provision of a direct communication channel, via telephone, for example,

so that users can ask questions related to the disinvestment to trained professionals.

7.5. Monitoring the disinvestment

Monitoring the use of technology is required to validate or change the decisions taken

based on HTR. It also allows for evaluation of the implementation process and to track

the success of the dissemination strategies and regulatory mechanisms.

It is very important to monitor health outcomes, since with divestment we seek to

maintain or improve results with the optimization of resources. Possible strategies

include building a registry with clinical and demographic data from electronic data

collection systems. Partner institutions can be selected to conduct the study which

must be approved by the Research Ethics Committee.

31


8. FINAL REMARKS

The investment /divestment in technologies should be focused on the best risk-

benefit ratio for the population, looking forward to providing more cost-effective

treatments and services and optimizing the use of scarce resources.

Health technology disinvestment process may present itself as more complex than the

incorporation and may face complicating factors such as a lack of scientific evidence,

publication bias, and political, ethical and social issues, as it can be understood,

erroneously, as "loss of an acquired right”.

The strategy to be adopted for the implementation of the disinvestment may vary

considerably from technology to technology, especially in terms of their degree of

consolidation in clinical practice and society. Thus, disinvestment process should be

fully transparent and participative, always based on the best evidence available and

supported by effective dissemination of information.

The full delisting of a technology should never be the main focus of the analysis.

Probably, few technologies are candidates for this type of disinvestment. Even if the

analysis results indicate evidence of low effectiveness or cost-effectiveness, the

perceptions of users and how they deal with the technology are points of great

relevance to the legitimate the decision-making process of health systems managers.

Whenever possible, social participation, especially of representatives of the users

directly affected by the decision, is relevant.

Another point of great importance concerns the identification of subgroups of users

who benefit from the technology candidate for divestment. It is vital that strict criteria

are established to ensure that those users are not harmed by the decision to disinvest.

Also the commitment to reinvest the funds raised with the divestment in activities and

services related to the affected illness or disease group can enhance the acceptability

of both professionals and patients.

32


Most health systems are major buyers of technologies, whether services, equipment or

direct consumer products. Thus, is important to establish conducts, throughout the

disinvestment analysis process, in order not to create market speculations. It is well

known that buying relationships are governed by agreements between the parties, but

the mere expectation of maintenance or simple information of disinvestment on a

technology by the health systems can generate large speculations in the market.

Finally, we highlight the merits of the decision. It is extremely important that the

manager makes a conscious decision, within the principles established by the health

system, and it is strongly based on ethical, technical and legal parameters.

33


REFERENCES

ASSESSMENT APPRAISAL DECISION: (Good) Practice examples and recommendations (Structured abstract). Health Technology Assessment Database. BASTIAN, H. et al. Choosing health technology assessment and systematic review topics: the development of priority-setting criteria for patients' and consumers' interests. Int J Technol Assess Health Care, v. 27, n. 4, p. 348-56, Oct ISSN 1471-6348 (Electronic) 0266-4623 (Linking). BRASIL. Constituição da República Federativa do Brasil de 05 de outubro de 1988. Diário Oficial da República Federativa do Brasil. Brasília (DF): Senado; 1988. Título VIII. Da Ordem Social. Capítulo II. Seção II. Da Saúde. Arts 196-200 BRASIL. Diário Oficial da União. Lei nº 8080/90. Dispõe sobre as condições para promoção, proteção e recuperação da saúde, a organização e o financiamento dos serviços correspondentes e da outras providências. Brasília DF, 19 de setembro de 1990. BRASIL. Diário Oficial da União. Lei 8142/90. Dispõe sobre a participação da comunidade na gestão do Sistema Único de Saúde (SUS) e sobre as transferências intergovernamentais de recursos financeiros na área da saúde e dá outras providências. Brasília DF, 28 de dezembro de 1990. BRASIL. GABINETE DO PRESIDENTE. Lei nº 12.401, de 28 de abril de 2011. Altera a Lei no 8.080, de 19 de setembro de 1990, para dispor sobre a assistência terapêutica e a incorporação de tecnologia em saúde no âmbito do Sistema Único de Saúde - SUS. Diário Oficial [da] República Federativa do Brasil, Poder Executivo, Brasília, DF, 19 abr 2011a. Seção 1, p. 1-2. DANIELS N. Accountability for reasonableness : Establishing a fair process for priority setting is

easier than agreeing on principles. BMJ : British Medical Journal. 2000; 321(7272): 1300-1301.

Disponível em <http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1119050/pdf/1300.pdf>.

Acesso em 29 jun 2015.

DANIELS, T.; WILLIAMS, I.; ROBINSON S.; SPENCE, K. "Tackling disinvestment in health care

services", Journal of Health Organization and Management, (2013), Vol. 27 Iss 6 pp. 762 - 780

DRUMMOND, M. F. et al. Key principles for the improved conduct of health technology assessments for resource allocation decisions. Int J Technol Assess Health Care, v. 24, n. 3, p. 244-58; discussion 362-8, Summer 2008. ISSN 0266-4623 (Print) 0266-4623 (Linking). ELSHAUG, A. G. et al. Identifying existing health care services that do not provide value for money. Med J Aust, v. 190, n. 5, p. 269-73, Mar 2 2009. ISSN 0025-729X (Print) 0025-729X (Linking). ELSHAUG, A. G.; HILLER, J. E.; MOSS, J. R. Exploring policy-makers' perspectives on disinvestment from ineffective healthcare practices. Int J Technol Assess Health Care, v. 24, n. 1, p. 1-9, Winter 2008. ISSN 0266-4623 (Print) 0266-4623 (Linking).

34


ELSHAUG, A. G.; HILLER, J. E.; TUNIS, S. R; MOSS, J. R. Challenges in Australian policy processes for disinvestment from existing, ineffective health care practices. Australian and New Zealand Health Policy, 2007;4:23.

ELSHAUG, A. G.; WATT, A. M.; MUNDY, L.; WILLIS, C. D. Over 150 potentially low-value health care practices: an Australian study. MJA 197 (10) · 19 November 2012.

EMENGO, H. Review of approaches and techniques to identify health interventions suitable for disinvestment (Project record). Health Technology Assessment Database. EUR-ASSESS. Report from the EUR-ASSESS Project. Int J Technol Assess Health Care. 1997;

13(2)

FRONSDAL, K. B. et al. Health technology assessment to optimize health technology utilization: using implementation initiatives and monitoring processes. Int J Technol Assess Health Care, v. 26, n. 3, p. 309-16, Jul ISSN 1471-6348 (Electronic) 0266-4623 (Linking).

GALLEGO, G.; HAAS, M.; HALL, J.; VINEY, R. Reducing the use of ineffective health care interventions : a rapid review. CHERE Centre for Health Economics Research and Evaluation.

GALNARES CORDERO, L. Map on shared resources in HTA and Health Services (Structured abstract). Health Technology Assessment Database 2009. GARCIA-ALTES, A. [The introduction of technologies in health care systems: from saying to doing]. Gac Sanit, v. 18, n. 5, p. 398-405, Sep-Oct 2004. ISSN 1578-1283 (Electronic) 0213-9111 (Linking).

GARNER, S.; LITTLEJOHNS, P. Do NICE’s recommendations for disinvestment add up? BMJ | 13 AUGUST 2011 | VOLUME 343

GERDVILAITE, J.; NACHTNEBEL, A. Disinvestment. Overview of disinvestment experiences and challenges in selected countries. Health Technology Assessment Database. HTA- Projektbericht. 2011; Nr. 57. Wien: Ludwig Boltzmann Institut fur HealthTechnology Assessment.

GODMAN, B.; WETTERMARK, B.; HOFFMANN, M.; ANDERSSON, K.; HAYCOX, A.; GUSTAFSSON, L. L. Multifaceted national and regional drug reforms and initiatives in ambulatory care in Sweden: global relevance. Expert Rev. Pharmacoeconomics Outcomes Res. 9(1), 65–83 (2009).

GREEN, C.; GERARD, K. Exploring the social value of health-care interventions: a stated preference discrete choice experiment. Health Econ, v. 18, n. 8, p. 951-76, Aug 2009. ISSN 1099-1050 (Electronic) 1057-9230 (Linking). GRUTTERS, J. P. et al. Bridging trial and decision: a checklist to frame health technology assessments for resource allocation decisions. Value Health, v. 14, n. 5, p. 777-84, Jul-Aug ISSN 1524-4733 (Electronic) 1098-3015 (Linking). Haas, M. et al. Breaking up is hard to do: why disinvestment in medical technology is harder

35


than investment. Aust Health Rev, v. 36, n. 2, p. 148-52, May ISSN 0156-5788 (Print) 0156-5788 (Linking). HAINES, T. et al. A novel research design can aid disinvestment from existing health technologies with uncertain effectiveness, cost-effectiveness, and/or safety. J Clin Epidemiol, v. 67, n. 2, p. 144-51, Feb ISSN 1878-5921 (Electronic) 0895-4356 (Linking). HENSHALL, C.; SCHULLER, T.; MARDHANI-BAYNE, L. Using health technology assessment to support optimal use of technologies in current practice: the challenge of "disinvestment". Int J Technol Assess Health Care, v. 28, n. 3, p. 203-10, Jul ISSN 1471-6348 (Electronic) 0266-4623 (Linking). 2012

HODGETTS, K.; HILLER, J. E.; STREET, J. M.; CARTER, D.; BRAUNACK-MAYER, A. J.; WATT, A. M.; MOSS, J. R.; ELSHAUG, A. G.; THE ASTUTE HEALTH STUDY GROUP. Disinvestment policy and the public funding of assisted reproductive technologies: outcomes of deliberative engagements with three key stakeholder groups. BMC Health Services Research 2014, 14:204.

HOLLINGWORTH, W.; CHAMBERLAIN, C. Nice recommendations for disinvestment. BMJ 2011;343:d5772 DOI: 10.1136/bmj.d5772

HOLLINGWORTH, W. et al., Apr to identify and prioritise opportunities for disinvestment in health care: a cross-sectional study, systematic reviews and qualitative study IBARGOYEN-ROTETA, N.; GUTIERREZ-IBARLUZEA, I.; ASUA, J. Guiding the process of health technology disinvestment. Health Policy, v. 98, n. 2-3, p. 218-26, Dec ISSN 1872-6054 (Electronic) 0168-8510 (Linking). International Journal of Technology Assessment in Health Care, 24:3 (2008), 244-258

JUNIOR, S.; GOMES, A.; ALVES, C. A. Modelos Assistenciais em Saúde: desafios e perspectivas. In: Márcia Valéria G.C. Morosini e Anamaria D.Andrea Corbo (org). Modelos de atenção e a saúde da família. Rio de Janeiro: EPSJV/Fiocruz, 2007. p 27-41. ISBN: 978-85-98768-24-3.

JUSTO, L. P. et al. Revisão sistemática, metanálise e medicina baseada em evidências: considerações conceituais. J Bras Psiquiatr. 2005; 54 (3).

LEGGETT, L. et al. Health technology reassessment of non-drug technologies: current practices. Int J Technol Assess Health Care, v. 28, n. 3, p. 220-7, Jul ISSN 1471-6348 (Electronic) 0266-4623 (Linking). MACKEAN, G. et al. Health technology reassessment: the art of the possible. Int J Technol Assess Health Care, v. 29, n. 4, p. 418-23, Oct ISSN 1471-6348 (Electronic) 0266-4623 (Linking).

MACKEAN, G.; NOSEWORTHY, T.; ELSHAUG A. G.; LEGGETT, L.; LITTLEJOHNS, P.; BEREZANSKI, J. et al. Health technology reassessment: the art of the possible. International Journal of Technology Assessment in Health Care. 2013;29(4):418-23.

36


MARINHO, V. C. C.; HIGGINS, J. P. T.; SHEIHAM, A.; LOGAN, S. Combinations of topical fluoride (toothpastes, mouthrinses, gels, varnishes) versus single topical fluoride for preventing dental caries in children and adolescents. Cochrane Database of Systematic Reviews 2004, Issue 1. Art. No.: CD002781. DOI: 10.1002/14651858.CD002781.pub2.

MITTON, C.; DIONNE, F.; DONALDSON, C. Managing healthcare budgets in times of austerity: the role of program budgeting and marginal analysis. Appl Health Econ Health Policy, v. 12, n. 2, p. 95-102, Apr ISSN 1179-1896 (Electronic) 1175-5652 (Linking).

MITTON, C.; DONALDSON, C. Health care priority setting: principles, practice and challenges. Cost Effectiveness and Resource Allocation, (2004), 2, 3 e10.

MITTON, C.; DONALDSON, C.; WALDNER, H.; EAGLE, C. The evolution of PBMA: towards a macro-level priority setting framework for health regions. Health Care Management Science, (2003), 6, 263e269.

NIELSEN, C. P.; FUNCH, T. M.; KRISTENSEN, F. B. Health technology assessment: research trends and future priorities in Europe. J Health Serv Res Policy, v. 16 Suppl 2, p. 6-15, Jul ISSN 1758-1060 (Electronic) 1355-8196 (Linking). OSINOWO, A.; SIMPSON, S. Out with the old, in with the new: identifying health technologies for disinvestment - can early awareness and alert systems play a part? [abstract]. HTAi 7th Annual Meeting: Maximising the Value of HTA; 2010 Jun 6-9; Dublin, Ireland: 88 p. OXMAN, A. D.; SCHUNEMANN, H. J.; FRETHEIM, A. Improving the use of research evidence in guideline development: 2. Priority setting. Health Research Policy & Systems: 14 p. 2006. PEIRO, S. [In healthcare, more or less, less is more: notes on healthcare disinvestment]. Rev Esp Sanid Penit, v. 16, n. 3, p. 68-74, ISSN 1575-0620 (Print) 1575-0620 (Linking).

PICHETTI, S. ; SORASITH, C. ; SERMET, C. Analysis of the impact of removing mucolytics and expectorants from the list of reimbursable drugs on prescription rates: A time-series analysis for France 1998–2010. Health Policy 102 (2011) 159– 169.

POLISENA, J. et al. Case studies that illustrate disinvestment and resource allocation decision-making processes in health care: a systematic review. Int J Technol Assess Health Care, v. 29, n. 2, p. 174-84, Apr. ISSN 1471-6348 (Electronic) 0266-4623 (Linking).

ROBINSON, S.; DICKINSON, H.; FREEMAN, T.; WILLIAMS, L. Disinvestment in health— the challenges facing general practitioner (GP) commissioners. Public Money & Management, 31:2, 145-148, DOI:10.1080/09540962.2011.560714.

SACKETT, D. L. et al. Evidence-based medicine: Whai it is and what it isn’t. BJM, 312: 71-2,

1996.

SCHOEN, C. et al. Toward higherperformance health systems: adults' health care experiences

in seven countries, 2007. Health Aff (Millwood)2007;26(6):w717-w734.

37


THORNHILL, M. H.; DAYER, M. J.; FORDE, J. M.; COREY, G. R.; CHU, V. H.; COUPER, D. J.; LOCKHART, P. B. Impact of the NICE guideline recommending cessation of antibiotic prophylaxis for prevention of infective endocarditis: before and after study. BMJ 2011;342:d2392 doi:10.1136/bmj.d2392.

VIEIRA, F.S. Ações judiciais e direito à saúde. Rev Saúde Pública 2008;42(2):365-9.

VIEIRA, F.S. Gasto do Ministério da Saúde com medicamentos: tendência dos programas de

2002 a 2007. Rev Saude Publica. 2009; 43(4):674-81. DOI: 10.1590/S0034-

89102009005000041

WATT, A. M. et al. The ASTUTE Health study protocol: deliberative stakeholder engagements

to inform implementation approaches to healthcare disinvestment. Implement Sci, v. 7, p.

101, ISSN 1748-5908 (Electronic) 1748-5908 (Linking)

WETTERMARK, B.; GODMAN, B.; ANDERSSON, K.; GUSTAFSSON, L. L.; HAYCOX, A.; BERTELE, V. Pharmacoeconomics 2008; 26 (7): 537-550.

WETTERMARK, B.; GODMAN, B.; NEOVIUSA, M.; HEDBERGE, N.; MELLGREND, T. O.; KAHANF, T. Initial effects of a reimbursement restriction to improve the cost-effectiveness of antihypertensive treatment. Health Policy 94 (2010) 221–229.

WHO. World Health Organization. Health Technology. Acessado em: 10/07/2015555.

Disponível em: http://www.who.int/topics/technology_medical/en/

Date post:	23-Mar-2018
Category:	Documents
Upload:	truongtruc
View:	229 times
Download:	1 times

GUIDELINE Disinvestment - CCATESccates.org.br/content/_pdf/en_PUB_1437692942.pdf · 2...

Documents