QUALITY IMPROVEMENT TO ASSESS AND AUDIT COMPLEXITY …

The Pennsylvania State University

The Graduate School

College of Engineering

QUALITY IMPROVEMENT TO ASSESS AND AUDIT COMPLEXITY IN

TRANSLATIONAL RESEARCH

A Dissertation in

Industrial Engineering and Operations Research

by

David A. Munoz Soto

2015 David A. Munoz Soto

Submitted in Partial Fulfillment

of the Requirements

for the Degree of

Doctor of Philosophy

August 2015

ii

This dissertation of David A. Munoz Soto was reviewed and approved* by the following:

Harriet Black Nembhard

Professor and Interim Department Head of Industrial and Manufacturing Engineering

Dissertation Adviser

Chair of Committee

Paul Griffin

Virginia C. and Joseph C. Mello Chair and Professor of Industrial and Systems Engineering

H. Milton Stewart School of Industrial & Systems Engineering

Georgia Institute of Technology

Deirdre McCaughey

Associate Professor of Health Policy and Administration

Ling Rothrock

Associate Professor of Industrial and Manufacturing Engineering

Conrad S. Tucker

Assistant Professor of Industrial and Manufacturing Engineering

Assistant Professor of Engineering Design

* Signatures are on file in the Graduate School.

iii

ABSTRACT

The large gap between proven clinical knowledge and its implementation in clinical practice is a pressing

challenge faced by the health community. It has been estimated that adults in the U.S. receive only about

half of their recommended care. This is in part, due to the complexities and current inability of translating

knowledge to effectively impact health outcomes. Moreover, the lack of understanding of the complexities

involved in translational research have resulted in a poor allocation of resources. As an effort to accelerate

the rate at which new discoveries become clinical practice, the National Institutes of Health (NIH) explicitly

made translational research a central priority and has invested heavily in developing an infrastructure

through the Clinical and Translational Science Awards (CTSAs). The arc of this dissertation is in alignment

with this priority.

Translational research experts have argued that the existing models in translational research have not been

able to fully capture the complexities, dynamisms, and fragmentations of this long process. In response,

data-driven tools and robust frameworks are expected to help analyzing, and hence, accelerating this

knowledge translation. These frameworks are needed for assuring an efficient and effective decision-

making process that support the tactical and strategic allocation of healthcare resources.

Although Quality Improvement (QI) approaches have been found to be promising to solve a wide variety

of problems in healthcare, their implementation in translational research has not been fully explored.

Moreover, in healthcare fields, QI has been mostly associated with Lean and Six Sigma techniques.

However, in order for QI techniques to address translational research challenges, a wider QI scope is

needed. In response to these challenges, a comprehensive QI research approach is used in this dissertation

to provide frameworks that inform healthcare decision makers, and hence, have a positive impact on

translational research. The frameworks presented are applied to different case studies that use them to

generate evidence for professional applications.

The main body of this dissertation is divided into three parts. The first part proposes a combined Quality

Function Deployment (QFD) and Analytic Hierarchy Process (AHP) framework for assessing the

complexity of translational research. Specifically, this framework is used to identify and quantify the

importance of the different operational steps and corresponding technical requirements along the

translational research process. This framework was applied to a case study of a primary care-based weight

iv

control intervention. The second part proposes a Social Network Analysis (SNA) approach for evaluating

collaboration and multidisciplinarity networks. The evaluation includes the identification of collaboration

patterns, leaders, influencers, bridgers of knowledge, and research clusters. A case study that analyzes

collaboration on obesity research at the intra-institutional level is presented to illustrate the potential

benefits and applicability of this framework. Finally, a goal programming (GP) model and a cost-

effectiveness analysis (CEA) approach is proposed to guide the proposal selection problem and estimate

the potential impact of healthcare interventions respectively. Specifically, a GP model was developed for

the proposal selection of a CTSA’s hub from a strategic perspective. Additionally, a model for rapid

estimation of impact is applied to an early detection of intervention of Parkinson’s disease. Lastly, a

combination of these two techniques is modeled to incorporate cost-effectiveness measures into the

proposal selection problem.

These studies cover relevant topics that aim to support the understanding of translational research and offer

pathways for a more efficient translation of new discoveries into clinical practice through QI research

approaches.

v

Table of Contents

List of Tables .............................................................................................................................................. ix

List of Figures .............................................................................................................................................. xi

List of Symbols and Abbreviations ............................................................................................................. xii

Important Terms and Definitions ............................................................................................................... xiv

Acknowledgements ................................................................................................................................... xvi

Chapter 1 ................................................................................................................................................... 1

INTRODUCTION

1.1 Statement of the Problem .......................................................................................................... 2

1.2 Gaps that Need to be Filled and Problems that Need to be Solved ........................................... 3

1.3 Motivation and Challenges ........................................................................................................ 3

1.4 Research Objective and Main Contributions ............................................................................. 4

1.4.1 Research objectives ............................................................................................................. 4

1.4.2 What will be added to the field of knowledge .................................................................... 4

1.5 Methodology Overview, Tools, and Questions to be Addressed .............................................. 5

1.5.1 Overall approach ................................................................................................................. 5

1.5.2 Tools description ................................................................................................................. 7

1.6 Summary of Main Gaps, Motivation, and Contributions ........................................................ 10

Chapter 2 ................................................................................................................................................. 14

LITERATURE REVIEW

2.1 Overview of Systems Engineering Tools in Healthcare .......................................................... 15

2.1.1 Tools for system design .................................................................................................... 15

2.1.2 Tools for system analysis .................................................................................................. 16

2.1.3 Tools for system monitoring and control .......................................................................... 16

2.2 Tools to Capture Customer Needs and Technical Requirements ............................................ 17

2.2.1 Quality function deployment ............................................................................................ 17

2.2.2 House of quality ................................................................................................................ 17

2.2.3 Analytic hierarchy process ................................................................................................ 19

2.2.3.1 AHP procedure for obtaining weights .............................................................................. 19

2.2.3.2 AHP in healthcare applications ........................................................................................ 22

2.2.3.3 AHP and translational research ....................................................................................... 22

2.2.4 Integration of QFD and AHP ............................................................................................ 23

2.3 Social Network Analysis ......................................................................................................... 24

2.3.1 Network representation ..................................................................................................... 24

2.3.2 SNA metrics ...................................................................................................................... 25

2.3.3 Applications of SNA ......................................................................................................... 27

vi

2.3.4 SNA in healthcare ............................................................................................................. 28

2.3.5 SNA to assess collaboration networks .............................................................................. 28

2.4 Multiple-Criteria Optimization ................................................................................................ 29

2.4.1 Goal programming ............................................................................................................ 30

2.4.1.1 Weighted goal programming ............................................................................................ 30

2.4.1.2 Preemptive goal programming ......................................................................................... 31

2.4.1.3 Tchebycheff goal programming ........................................................................................ 32

2.4.1.4 Obtaining weights ............................................................................................................. 32

2.4.1.5 Scaling and normalizing goal constraints parameters ..................................................... 33

2.4.2 Goal programming in healthcare ...................................................................................... 33

2.5 Proposal selection methods ..................................................................................................... 34

2.6 Cost-Effectiveness Analysis .................................................................................................... 36

2.6.1 Impact of healthcare interventions and the use of QALY ................................................ 38

2.6.2 Estimating QALY ............................................................................................................. 40

2.6.3 Instruments to estimate QALY ......................................................................................... 42

2.6.3.1 EQ-5D ............................................................................................................................... 42

2.6.3.2 SF-36 ................................................................................................................................. 46

2.6.3.3 SF-12 ................................................................................................................................. 48

2.6.3.4 SF-6D ................................................................................................................................ 51

2.6.3.5 QWB-SA ............................................................................................................................ 53

2.6.3.6 Comparison between instruments ..................................................................................... 54

2.6.3.7 Discussion and limitations ................................................................................................ 54

Chapter 3 ................................................................................................................................................. 56

QUANTIFYING COMPLEXITY IN TRANSLATIONAL RESEARCH: AN INTEGRATED

QUALITY FUNCTION DEPLOYMENT – ANALYTIC HIERARCHY PROCESS APPROACH

3.1 Introduction ............................................................................................................................. 56

3.2 Methodology ........................................................................................................................... 59

3.2.1 Identification of markers and technical requirements ....................................................... 60

3.2.2 Determining marker weights for each translational research phase .................................. 61

3.2.2.1 Pairwise comparison matrix ............................................................................................. 61

3.2.3 Building the house of quality ............................................................................................ 62

3.2.3.1 Correlation between technical requirements .................................................................... 62

3.2.3.2 Relationship matrix between technical requirements and markers .................................. 63

3.2.3.3 Technical requirement weights ......................................................................................... 63

3.3 Case Study: a Primary Care-based Weight Control Intervention ............................................ 64

3.4 Results ..................................................................................................................................... 65

3.4.1 Identification of process markers and technical requirements .......................................... 65

3.4.2 Pairwise comparison matrices, consistency and weights .................................................. 66

3.4.3 Correlation among TRs and relationship among TR-marker pairs ................................... 69

3.4.4 Determining the importance of each technical requirement in translational research ...... 72

3.5 Discussion ............................................................................................................................... 73

3.6 Conclusion ............................................................................................................................... 74

vii

Chapter 4 ................................................................................................................................................. 76

EVALUATING COLLABORATION AND MULTI-DISCIPLINARITY AND THEIR IMPACT

ON TRANSLATIONAL RESEARCH

4.1 Introduction ............................................................................................................................. 76

4.2 Methodology ........................................................................................................................... 77

4.3 Case Study: Collaboration in Obesity Research ...................................................................... 78

4.3.1 Identification of obesity researchers ................................................................................. 79

4.3.2 Classification of expertise ................................................................................................. 79

4.3.3 Social network analysis for obesity researchers ............................................................... 80

4.3.4 Cross-institutional collaboration ....................................................................................... 80

4.4 Results ..................................................................................................................................... 81

4.4.1 Intra-institutional collaboration networks ......................................................................... 82

4.4.2 Interdisciplinary collaboration by affiliation .................................................................... 85

4.4.3 Cross-Institutional collaboration networks ....................................................................... 87

4.5 Discussion ............................................................................................................................... 88

4.6 Conclusions ............................................................................................................................. 89

Chapter 5 ................................................................................................................................................. 91

GUIDING THE STRATEGY AND RESOURCE ALLOCATION OF HEALTHCARE

ORGANIZATIONS BASED ON IMPACT OF HEALTH INTERVENTIONS

5.1 Introduction ............................................................................................................................. 91

5.2 Methodology: Goal Programming Model for Proposal Selection ........................................... 94

5.2.1 Model overview ................................................................................................................ 94

5.2.2 Generic model ................................................................................................................... 97

5.2.2.1 Phase 1: Understanding the strategy ................................................................................ 97

5.2.2.2 Phase 2: Understanding the constraints ........................................................................... 97

5.2.2.3 Phase 3: Formulating the model....................................................................................... 98

5.2.2.4 Phase 4: Solving and validating ..................................................................................... 108

5.3 Case Study: Proposal Selection in a CTSA Hub ................................................................... 109

5.3.1 Identifying goals and constraints .................................................................................... 109

5.3.2 Obtaining goal weights ................................................................................................... 110

5.3.3 Formulating goal constraints........................................................................................... 110

5.3.4 Objective function ........................................................................................................... 114

5.3.5 Set of system constraints ................................................................................................. 114

5.4 Results of Proposal Selection ................................................................................................ 117

5.5 Discussion of Proposal Selection .......................................................................................... 118

5.6 Methodology: A Rapid Impact Estimation of Healthcare Interventions ............................... 120

5.7 Case Study: Impact Estimation for Early Detection of Parkinson’s Disease ........................ 124

5.7.1 Case study overview ....................................................................................................... 124

5.7.2 Parkinson’s disease background ..................................................................................... 126

5.8 Results of Rapid Impact Estimation ...................................................................................... 128

5.8.1 Potential QALYs gained ................................................................................................. 128

viii

5.8.2 Cost per QALY ............................................................................................................... 130

5.8.3 Overall impact on society ............................................................................................... 130

5.8.4 Sensitivity analysis ......................................................................................................... 131

5.9 Discussion of Rapid Impact Estimation ................................................................................ 134

5.10 Incorporating Economic Evaluation into the Proposal Selection Problem ........................... 136

5.11 Conclusions ........................................................................................................................... 137

Chapter 6 ............................................................................................................................................... 139

CONTRIBUTIONS AND FUTURE WORK

6.1 Identifying Key Drivers and Prioritizing Efforts ................................................................... 140

6.2 Closing Existing Gaps ........................................................................................................... 140

6.3 Engaging the Participation of Health Professionals .............................................................. 141

6.4 Guiding the Strategy of Healthcare Organizations ................................................................ 142

6.5 Future Work .......................................................................................................................... 142

REFERENCES ......................................................................................................................................... 145

Appendix A. House of Quality ................................................................................................................. 160

Appendix B. List of Proposals and Characteristics ................................................................................... 161

Appendix C. Proposals’ coefficients ......................................................................................................... 162

Appendix D. PBi coefficients ................................................................................................................... 162

Appendix E. Distribution of enrollment in graduate school (For illustration purposes only)................... 163

Appendix F. LINDO Code ........................................................................................................................ 163

Appendix G. MOS SF-36 (RAND 36-Items version). Obtained from www.rand.org ............................. 166

ix

List of Tables

Table 1-1. Research questions .................................................................................................................... 10

Table 1-2. Gaps, approach, and main contributions of Chapter 3 ............................................................... 11



Table 2-1. AHP scale definition .................................................................................................................. 20

Table 2-2. Random consistency index ........................................................................................................ 21

Table 2-3. EQ-5D self-reported questionnaire ............................................................................................ 43

Table 2-4. Coefficients for TTO tariffs (modified from Dolan et al., 1995) .............................................. 44

Table 2-5. EQ-5D - D1 Valuation model .................................................................................................... 45

Table 2-6. SF-36 Health status and interpretation ...................................................................................... 47

Table 2-7. SF-36 - Physical functioning dimension.................................................................................... 48

Table 2-8. SF-12 Health dimensions and summary of content ................................................................... 49

Table 2-9. SF-12 Regression coefficients ................................................................................................... 50

Table 2-10. SF-6D Health dimensions and levels ....................................................................................... 52

Table 2-11. SF-6D Models with interaction effects .................................................................................... 53

Table 3-1. AHP scale definition .................................................................................................................. 61

Table 3-2. Correlation intensity .................................................................................................................. 63

Table 3-3. Markers for the obesity peer-led intervention ........................................................................... 65

Table 3-4. Technical requirements for the obesity peer-led intervention ................................................... 66

Table 3-5. T1 – Pairwise comparison matrix .............................................................................................. 66



Table 3-8. Consistency analysis values ....................................................................................................... 68

Table 3-9. Marker weights .......................................................................................................................... 68

Table 3-10. Relationship matrix for T1 ...................................................................................................... 70



Table 3-13. Technical requirements relative weights ................................................................................. 72

Table 4-1. Expertise classification criteria .................................................................................................. 79

Table 4-2. 15 Most frequently-used journals .............................................................................................. 82

Table 5-1. Rating method to obtain goal weights ..................................................................................... 108

x

Table 5-2. Expert's scores and goal priorities ........................................................................................... 110

Table 5-3. Baseline for comparison and relevant parameters by type of intervention .............................. 125

Table 5-4. Hoehn and Yahr stages and characteristics ............................................................................. 127

Table 5-5. HRQoL of treated vs untreated PD patients by HY stage ....................................................... 129

Table 5-6. Data for estimating overall impact on society ......................................................................... 131

xi

List of Figures

Figure 1-1. Expanded QI research toolkit ..................................................................................................... 5

Figure 1-2. Methodology diagram ................................................................................................................ 6

Figure 2-1. House of quality diagram ......................................................................................................... 18

Figure 2-2. Directed and undirected graphs ................................................................................................ 25

Figure 2-3. Types of health interventions (adapted from Jamison et al. (2006)) ........................................ 37

Figure 2-4. Histogram of distribution of HRQoL ranges for US and UK .................................................. 46

Figure 3-1. Comparison among the four major translational research models (modified from Trochim et

al., 2011) ............................................................................................................................................. 58

Figure 3-2. QFD-AHP Methodology Diagram ........................................................................................... 64

Figure 3-3. Extended process marker model .............................................................................................. 69

Figure 3-4. Technical requirements correlation .......................................................................................... 70

Figure 3-5. Relative importance for technical requirements on each "T" phase ......................................... 73

Figure 4-1. Overview of methodology to assess intra-institutional collaboration ...................................... 78

Figure 4-2. Number of publications and average number of citations per publication ............................... 81

Figure 4-3. General collaboration network ................................................................................................. 83

Figure 4-4. Collaboration network with at least two publications between researchers ............................. 83

Figure 4-5. Expert sub-clusters collaboration networks ............................................................................. 84

Figure 4-6. Collaboration network per affiliation ....................................................................................... 86

Figure 4-7. Cross-institutional collaboration for obesity experts ................................................................ 87

Figure 5-1. Sections distribution of Chapter 5 ............................................................................................ 93

Figure 5-2. Overview of methodology for the proposal selection problem ................................................ 96

Figure 5-3. Rapid high-level impact estimation (RHIE) framework ........................................................ 122

Figure 5-4. QALYs gained by currently undiagnosed individuals ........................................................... 129

Figure 5-5. Cost-effectiveness sensitivity for QALYs gained and cost per diagnosed case ..................... 133

Figure 5-6. Cost-effectiveness tornado sensitivity for relevant parameters .............................................. 134

xii

List of symbols and abbreviations

A&F Audit and Feedback

AHP Analytic Hierarchy Process

AHRQ Agency for Healthcare Research & Quality

CQI Continuous Quality Improvement

CEA Cost-Effectiveness Analysis

CUA Cost-Utility Analysis

CTSA Clinical and Translational Science Award

CTSI Clinical and Translational Sciences Institute

D&I Dissemination and Implementation

DALY Disability-Adjusted Life Year

EBM Evidence-based Medicine

EBP Evidence-based Practice

EHR Electronic Health Record

FMEA Failure Mode and Effects Analysis

GDP Gross Domestic Product

GP Goal Programming

HA Hazard Analysis

HOQ House of Quality

HRQoL Health-Related Quality of Life

ICER Incremental Cost-Effectiveness Ratio

IOM Institute of Medicine

MCDM Multiple Criteria Decision Making

NAE National Academy of Engineering

NIH National Institute of Health

PD Parkinson’s Disease

PSU Pennsylvania State University

QALY Quality-Adjusted Life Year

QFD Quality Function Deployment

QI Quality Improvement

RCA Root Cause Analysis

RHIE Rapid High-level Impact Estimation

xiii

SNA Social Network Analysis

SG Standard Gamble

T1 Phase 1 in Translational Research




TQM Total Quality Management

TTO Time Trade-Off

TR Technical Requirements

VAS Visual Analogue Scale

VOC Voice of the Customer

WHO World Health Organization

WoK Web of Knowledge

xiv

Important Terms and Definitions

Audit and Feedback In the healthcare field, Audit and Feedback can be defined as “any

summary of clinical performance of health care over a specified period of

time aimed at providing information to health professionals to allow them

to assess and adjust their performance.” (Jamtvedt et al., 2006).

Cost-Effectiveness In the healthcare context, cost-effectiveness analysis “helps identify

neglected opportunities by highlighting interventions that are relatively

inexpensive, yet have the potential to reduce the disease burden

substantially… helps identifying ways to redirect resources to achieve

more.” (Jamison et al., 2006).

Dissemination “Dissemination is the targeted distribution of information and intervention

materials to a specific public health or clinical practice audience. The

intent is to spread knowledge and the associated evidence-based

interventions.” (NIH, 2007)

Evidence-Based Medicine “Evidence based medicine is the conscientious, explicit, and judicious use

of current best evidence in making decisions about the care of individual

patients. The practice of evidence based medicine means integrating

individual clinical expertise with the best available external clinical

evidence from systematic research.” (Sackett et al., 1996).

Implementation “Implementation is the use of strategies to adopt evidence-based health

interventions and change practice patterns within specific settings.” (NIH,

2007).

Intervention According to the AHRQ, an intervention is “any type of treatment,

preventive care, or test that a person could take or undergo to improve

health or to help with a particular problem. Health care interventions

include drugs (either prescription drugs or drugs that can be bought

without a prescription, food, supplements (such as vitamins), vaccinations,

screening tools (to rule out a certain disease), exercises (to improve

fitness), hospital treatment, and certain kinds of care (such as physical

therapy).”

xv

Quality Improvement Quality Improvement is a systematic approach to analyze the performance

of a system and combine efforts to improve it. In healthcare settings, it can

be defined “as the combined and unceasing efforts of everyone—

healthcare professionals, patients and their families, researchers, payers,

planners and educators—to make the changes that will lead to better

patient outcomes (health), better system performance (care) and better

professional development.” (Batalden and Davidoff, 2007).

Translational Research “Translational research fosters the multidirectional integration of basic

research, patient-oriented research, and population-based research, with

the long-term aim of improving the health of the public.” (Rubio et al.,

2010).

xvi

Acknowledgements

I would like to thank all of those from which I received support during this long academic journey. During

these last four years, I had the opportunity to meet amazing people in different situations and from very

different backgrounds. From all of them, I have learned to see the world differently and develop a very

open mindset.

I would like to express my sincere gratitude to my advisor and mentor, Dr. Harriet Black Nembhard. Her

guidance and mentorship during these years were essential for keeping myself focused and motivated in

researching relevant topics that could potentially have a positive impact on society. I would also like to

thank each one of the members of my dissertation committee — Dr. Paul Griffin, Dr. Deidre McCaughey,

Dr. Ling Rothrock, and Dr. Conrad Tucker — for their advice and helpful insights in developing this

dissertation. There are many others that directly or indirectly helped me to finish this dissertation. I

gratefully acknowledge Dr. Jennifer Kraschnewski, from the College of Medicine at Penn State, for being

willing and excited to exploring new engineering-focused methodologies and their application to

translational research. I would also like to thank Dr. Xuemei Huang for her receptiveness and enthusiasm

about using frameworks to estimate the impact of healthcare interventions. I also want to thank Kate

Camargo for her support, commitment, and availability to help on various topics related to this dissertation.

During the past four years, I had the opportunity to work on several projects with a diverse group of people.

I would like to thank all of those that collaborated with me during my years at Penn State. I am grateful to

the Pediatric Intensive Care Unit at the Penn State Hershey Medical Center, especially to Dr. Gary Ceneviva

and Dr. Robert Tamburro. I am also grateful to Windy Alonso, Dr. Judith Hupcey, and Alison Walsh, from

the College of Nursing at Penn State.

I also want to thank the students, professors, and staff of the Harold and Inge Marcus Department of

Industrial and Manufacturing Engineering at Penn State. Especially, I would like to thank Dr. Jeya Chandra,

Dr. Catherine Harmonosky, Dr. Ravi Ravindran, Dr. Christopher Saldana, Erin Ammerman, and Olga

Covasa. I also want to thank the Center for Integrated Healthcare Delivery Systems (CIHDS) and the Center

for Health Organization Transformation (CHOT) for giving me the opportunity to work on challenging

projects and meet gorgeous people.

I am grateful to the “Comisión Nacional de Investigación Científica y Tecnológica de Chile” (CONICYT)

which provided financial support through the “Becas Chile” scholarship. I also want to thank to the Penn

State Clinical and Translational Sciences Institute for their financial support through a graduate

xvii

assistantship. Part of this work was supported by the National Center for Research Resources and the

National Center for Advancing Translational Sciences, National Institutes of Health, through Grant UL1

TR000127.

I also want to thank my lab-mates for making these years more exciting and enjoyable. Especially, I want

to thank Hyojung Kang and Nate Bastian. I am also thankful to all my friends of Happy Valley, especially,

to Los Tikis, Patada FC, and LAGRASA for being our substitute families and for making winters warmer.

Finally, but most importantly, I would like to thank my wife, Carolina, and my lovely sons, Javier, and

Diego, for their support and immeasurable love during our years at Penn State. Additionally, I would like

to thank my parents, siblings, and God. Without their help and love, this would not have been possible.

1

Chapter 1

INTRODUCTION

Despite the uncountable achievements in medical discoveries in the U.S., there is still a large gap between

proven knowledge and its impact on people’s health (IOM, 2001, Green et al., 2009, Glasgow et al., 2012).

Some have argued that a main problem in the U.S. healthcare system is its weakness of applying what has

been learned through research (IOM, 2001, Berwick, 2003). It is estimated that adults in the U.S. receive

only about half of their recommended processes involved in care (McGlynn et al., 2003). As a consequence

of this inability of using the evidence generated, poor results have been obtained not only with respect to

impacting health outcomes, but also in terms of costs (Woolf, 2007, Grimshaw et al., 2012a).

In order to close the existing gaps, new designs and solutions must be explored to ensure that patients

receive their recommended healthcare (Reid et al., 2005, McHugh and Barlow, 2010, Davies et al., 2010,

Glasgow et al., 2012). Therefore, a substantial redesign using new tools and approaches is needed to

improve safety and quality, while reducing, or at least maintaining costs (Selker et al., 2011). In this sense,

Quality Improvement (QI) methodologies have been considered to be one of the main strategies for

addressing pressures for change and improvement in healthcare (Grol and Grimshaw, 2003). However, in

healthcare fields, QI methodologies have been mostly limited to the applicability of lean and six sigma

techniques (Ting et al., 2009). As a result of this limited scope, some have argued that these basic

techniques, although useful in several applications, cannot solve the complex problems faced by the

healthcare systems, including translational research efforts. In response, integrating more complex

research-oriented tools into the basic QI toolkit has been argued to be necessary to strengthen the benefits

2

and scope of QI in healthcare (Baldwin et al., 2012). Additionally, the integration of a more research-

oriented QI methodology could serve not only to addressing tactical and operational challenges, but also

facing problems at the strategic levels. Therefore, QI research can be used to bridge the gap between what

we know and what we do with this knowledge (Ting et al., 2009) by speeding the translation of effective

interventions into clinical practice and supporting the strategies to implement those interventions at the

point of care (Baldwin et al., 2012).

Auditing change through data-driven techniques is one of the key needs to understand why the efforts

invested in basic research are not producing an optimal effect on people’s health. Although the discussion

of the potential benefits of QI approaches in translational research has already been initiated (Baldwin et

al., 2012, Schmittdiel et al., 2010, Schweikhart and Dembe, 2009, Feldstein and Glasgow, 2008), more

research is needed to clarify their scope and generate frameworks that could be efficiently actionable into

practice. While many systems engineering methodologies, including QI, have been successfully applied to

solve different challenges in healthcare systems, there is still a lack of research focusing on translational

research. Moreover, data-driven tools are needed for assuring the efficiency and effectiveness of the

decision making process, especially, for an optimal allocation of healthcare resources. Certainly, it creates

a motivation to investigate how QI research approaches could be applied to accelerate the pace at which

new discoveries are integrated into clinical practice.

1.1 Statement of the Problem

Various irrefutable facts about the U.S. healthcare system, indicate that new approaches are needed to

improve the quality of care and ensure that the best-known practices are disseminated and implemented

adequately. Although the term translational research has become popular during the past decade, there is

still a lack of agreement on how to measure and monitor its complex, dynamic, and large-scale nature.

These factors have resulted in disagreement and lack of robust tools to address its complexity (IOM, 2001).

Understanding large-scale systems and their dynamics is a prime opportunity for multi-disciplinary

collaboration to fill existing gaps. In order to overcome these challenges, funding agencies from health and

engineering sectors have called for a stronger healthcare-engineering enterprise collaboration effort (IOM,

NAE, AHRQ) (Reid et al., 2005). A major contribution expected from this partnership is the development

of more robust, generalizable and sustainable frameworks to assess and audit the complexity of knowledge

transference (Glasgow et al., 2012). These frameworks should contribute to a wiser distribution of resources

and achievement of the maximum value given those resources, and therefore, reduce the costs of poor

quality.

3

1.2 Gaps that Need to be Filled and Problems that Need to be Solved

Although translational research has been studied for more than 30 years (Wolf, 1974), discrepancies in its

meaning and scope have led to the generation of different conceptual models to understand its continuum

nature. However, those models have failed to provide metrics to evaluate the long process of translating

research. It is estimated that on average, it takes 17 years from basic medical discoveries to be included as

regular healthcare practice. Additionally, according to Antes et al. (1999) the time lag between scientifically

proven knowledge and the introduction into medical routine takes on average between 8 and 10 years.

Measuring the effects and complexity of translational research is far from being a trivial task. The inability

of current methods to assess translation has caused that many of the proven discoveries to be lost in

transition (Butler, 2008). Consequently, most of the resources allocated to improve people’s health have

not been distributed based on evidence but mostly on pure intuition. Therefore, despite the investments that

have been made in translational research, public health benefits remain still far from optimal. In response

to this, more robust and data-driven frameworks should be provided not only to demonstrate success in the

integration of new discoveries into health policy and practice, but also to provide feedback and inform

decision makers in the biomedical research enterprise (Glasgow et al., 2012).

1.3 Motivation and Challenges

Systems engineering techniques, including QI, have a vast history of success in evaluating and auditing

complex, dynamic, large-scale systems. The main motivation for this research is to utilize the QI knowledge

gained in other industries to provide robust frameworks to assess and inform translational research at the

strategic level. Although QI has been considered to provide conceptual strategies to address current gaps

between ideal and actual care, its use has remained limited (Ting et al., 2009, Shojania et al., 2004).

Generalizable and sustainable research-oriented frameworks are needed in healthcare to truly accelerate the

journey of new discoveries to become regular clinical practice based on evidence. Although tremendous

advances have been made in generating interventions based on evidence, poor results have been obtained

in terms of implementing the best-known treatments and practices. One of the reasons for this to happen is

the lack of robust frameworks to evaluate and monitor multiple dimensions of translational research and

inform decision makers. The research-oriented QI framework proposed will not only be used to evaluate

complexity, but also to identify interesting interactions among different elements of the healthcare systems,

and guide the allocation of efforts by providing data-driven tools to different key healthcare stakeholders

and decision makers. Hence, resources can be spent wisely to maximize the impact on people’s health. In

summary, the final goal of the QI research approach presented in this dissertation is to provide a better

4

understanding of the existing gap in translational research, measuring it, and providing robust tools for

closing it.

1.4 Research Objective and Main Contributions

1.4.1 Research objectives

The objective of this dissertation is to investigate how QI research methodologies can be implemented to

assess the complexity in translational research and support a better strategic allocation of resources. This

broad objective is split into five sub-objectives, as follows:

• Provide models to map and quantify complexity in translational research. These models will

generate evidence that can be used at the strategic level to guide and inform key decision makers

on the allocation of resources along the translational research process.

• Identify the key elements involved in translational research and their impact on the pace at which

new discoveries become regular clinical practices.

• Investigate methods and meaningful metrics to assess collaboration networks and multi-

disciplinarity as well as their effects on an efficient translation of knowledge.

• Provide models to prioritize health interventions based on their value, the potential impact on

people’s health, and multiple-criteria fit on the healthcare organizations’ strategy.

• Develop a robust framework to guide the understanding of healthcare interventions’ impact based

on cost-effectiveness measures.

1.4.2 What will be added to the field of knowledge

This dissertation shows that QI research methodologies can contribute to the understanding, evaluation, and

monitoring of complexity in translational research. Moreover, it must be noted that the scope of QI used in

this dissertation is broader than the one typically used in healthcare fields. Thus, a more research-oriented

QI approach is used by integrating more comprehensive tools into the basic QI toolkit. This will help

addressing complex issues in translational research that require more inclusive approaches. Consequently,

the existing gap between the proven knowledge and its impact on health outcomes can be understood,

measured, and closed. The key contribution of the proposed approach is to provide robust frameworks that

generate evidence to inform healthcare stakeholders and guide the allocation of resources and efforts at

5

both tactical and strategic levels. A non-exhaustive illustration of the expansion of the basic QI toolkit and

its typical scope of action to a more comprehensive QI research toolkit is presented in Figure 1-1.

Figure 1-1. Expanded QI research toolkit

1.5 Methodology Overview, Tools, and Questions to be Addressed

1.5.1 Overall approach

A general QI research framework is proposed to address the main stated objective of this dissertation. Each

one of the sub-objectives is address separately in the body chapters provided in this dissertation. However,

a clear connection between the covered topics will be observed among the chapter to address the continuum

nature of translational research. Another relevant aspect of this dissertation is that case studies on different

translational research-related topics are used as proof-of-concept. These case studies serve to provide

guidance on the use of the frameworks and generate evidence for professional applications (Zucker, 2009).

Therefore, in this dissertation, these cases help to demonstrate the feasibility and principles of using an

expanded QI research toolkit to address complex issues of translational research.

Low High

Complexity

Op

erat

ion

alT

acti

cal

Str

ateg

ic

Dec

isio

n lev

el

Imp

act

Sh

ort

-ter

mL

ong-t

erm

Quality function

deploymentHouse of

quality

Root cause

analysis

Failure modes and

effects analysis

5 WHYs

Design of

experiments

Simulation

Mathematical

Programming

System Dynamics

Knowledge

Discovery

Queuing Theory

Cost effectiveness

analysis

Statistical

quality control

Value stream

mapping

To

p-d

ow

n i

nfl

uen

ce

Traditional (Basic) QI toolkit Expanded QI research toolkit

6

This dissertation is organized into six chapters. In Chapter 1, an introduction to motivate the topics covered

and objectives of this dissertation are presented. Chapter 2 presents a literature review to establish the

foundations under which this dissertation is built upon. Chapters 3, 4, and 5 represent the body of this

dissertation. More specifically, Chapter 3 assesses and quantifies complexity in translational research,

Chapter 4 evaluates collaboration and multidisciplinarity in translational research, and Chapter 5 provides

guidelines for multiple-criteria resource allocation and cost-effectiveness of healthcare interventions.

Finally, in Chapter 6, a summary of the main contributions and future research lines is presented.

Figure 1-2 presents the summary diagram of the proposed methodology and tools to be used. The bottom

part of the figure utilizes a translational research model based on 4 phases, T1, T2, T3, and T4, which

represent the process from basic research (T1) to health outcomes (T4). A more detailed explanation about

the approach and their potential benefits can be found in their respective chapter.

Figure 1-2. Methodology diagram

T1PHASE T2 T3 T4

• MCDM

• GP

• QALY

• CEA

TOOLS

• SNA

• Bibliometric

• Survey

• Data Mining

• QFD

• AHP

• HOQ

• Brainstorming

QUALITY IMPROVEMENT TO ASSESS AND AUDIT

COMPLEXITY IN TRANSLATIONAL RESEARCH

CH 4

CH 3

CH 5

Quantifying Complexity in Translational Research: An

Integrated Quality Function Deployment – Analytic Hierarchy

Process Approach

Evaluating Collaboration and Multi-disciplinarity and their

Impact on Translational Research

Guiding the Strategy and Resource

Allocation of Healthcare Organizations

Based on Estimated Impact

7

1.5.2 Tools description

As illustrated in Figure 1-2, various tools, most of them widely classified into the QI toolkit, will be used

in the proposed research. As previously mentioned, the basic QI toolkit will be expanded by introducing

other complementary systems engineering and social science tools to maximize the potential benefits of QI

research approaches. A general explanation of the tools used and others proposed for future research is

given below:

AHP “The Analytic Hierarchy Process is a decision making model that aids us in making

decisions in a complex world. It is a three part process which includes identifying

and organizing decision objectives, criteria, constraints and alternatives into a

hierarchy; evaluating pairwise comparisons between the relevant elements at each

level of the hierarchy; and the synthesis using the solution algorithm of the results

of the pairwise comparisons over all the levels. Further, the algorithm result gives

the relative importance of alternative courses of action.” (Saaty, 1977).

Benchmarking “Benchmarking is the process of measuring and improving products, services and

practices in comparison to the toughest competitors or those organizations that are

recognized as industry leaders. Benchmarking is about searching for industry best

practices that lead to superior performance and analyzing and learning from those

practices.” (Dixon and Pearce, 2011).

Bibliometrics Bibliometrics is a set of mathematical and statistical approaches to analyze large

amounts of academic literature. Usually, data related to citations and keywords are

used to identify interesting publication patterns.

Brainstorming “Brainstorming is a way of collecting the maximum number of ideas on a subject

from members of a team without considering the validity or practicality of the

ideas. The purpose of brainstorming is to generate a list of ideas when a team would

benefit from having as broad a range of ideas or alternatives as possible.” (Dixon

and Pearce, 2011).

CEA Cost-effectiveness analysis (CEA) is an economic technique that seeks to

understand and calculate outcomes with respect to costs. In practice, CEA is

widely used to compare alternatives based on their cost-effectiveness metrics. In

healthcare, the most common manner to express CEA is based on quality-adjusted

life years or other units of health gains per units of cost.

8

Control Charts “A control chart is a run chart with statistically determined upper and lower process

limits, called control limits, which indicate the range of variation that exists in a

process. Control limits are not the same as specification limits or thresholds for

action. Rather, control limits are intended to prevent attributing observed variation

in a process to a special cause when it is due to a common cause and vice versa.

Control charts are useful for determining the stability and capability of a process.

A control chart consists of three lines: The centre line represents the overall

average value of the sample statistic. The upper and lower lines, the control limits,

are set by establishing the confidence intervals for the sample statistic.” (Dixon

and Pearce, 2011).

Data Mining Data mining is a computational technique to discover interesting patterns in large

data sets. It extracts information from large, usually unstructured sets of data to

summarize useful information.

GP Goal programming (GP) is a technique to solve multiple criteria problems. The

objective function of GP models seeks to minimize deviations of different criteria

with respect to satisfying target levels. There are various variants in the GP

formulation including preemptive, weighted, and Tchebysheff, among others.

HOQ House of Quality (HOQ) is one of the main tools used in QFD to capture customer

requirements and identify the technical factors that fulfill those requirements. This

methodology was proposed by Hauser and Clausing (1988) as a way to improve

product quality based on a structured methodology to translate customer needs into

measurable technical requirements. Thus, HOQ can be seen as a conceptual map

for quality improvement.

MCDM Multiple-criteria decision making (MCDM) methods seek to solve complex

problems that involve multiple conflicting criteria. The solution is typically based

on an objective function in which the different criteria are weighted according to

the decision maker’s preference.

QFD QFD is a quality tool that offers a structured framework to transform customer’s

requirements into characteristics of either a new product/service or an old system.

SNA “Social network analysis (SNA) is a set of theories, tools, and processes for

understanding the relationships and structures of a network. The “nodes” of a

9

network are the people and the “links” are the relationships between people. Nodes

are also used to represent events, ideas, objects, or other things. SNA practitioners

collect network data, analyze the data (e.g., with special purpose SNA software),

and often produce maps or pictures that display the patterns of connections

between the nodes of the network.” (Hoppe and Reinelt, 2010).

Survey “A survey is the systematic collection of information by means of self-completed

questionnaires, interviews or observations from a large number of people, events,

records, literature or other data sources. The purpose of a survey usually is to

identify trends or patterns.” (Dixon and Pearce, 2011).

10

1.6 Summary of Main Gaps, Motivation, and Contributions

This chapter introduced the main objectives included in this dissertation. In Table 1-1, a list of research

questions that were covered or could be covered by future research aligned to the proposed topics is

presented. Additionally, a summary of current research gaps that need to be filled, motivation, and main

contributions of this dissertation is presented in Tables 1-2, 1-3, and 1-4.

Table 1-1. Research questions

Chapter Questions

Chapter 3:

Assessing and

quantifying

complexity in

translational

research

What are the most important operational steps in translational research?

Could some operational steps be generalizable to other translational research

efforts?

What are the critical technical requirements (TR) in translational research?

What is the impact of those TR on each translational phase?

How complex and dynamic is translational research?

Is the funding structure supporting an accelerated knowledge translation?

What TR should be prioritized to assure success in moving new discoveries into

practice?

Could an agreement be generated on the important operational steps in

translational research?

Is the allocation of resources being properly conducted?

Chapter 4:

Evaluating

collaboration and

its impact on

translational

research

What are the current opportunities to improve collaboration in translational

research?

Are collaboration networks strong enough to accelerate translational research?

Who are the leaders of opinion or influencer in the collaboration network?

Is there any structural hole that needs to be filled to accelerate translational

research?

Would a facilitator be needed to eliminate structural holes in collaboration?

Are there any interesting collaboration patterns in the collaboration network?

What meaningful network metrics must be considered to assess collaboration and

multidisciplinary efforts?

Is the organization adequately supporting the collaboration to meet its strategic

goals?

What collaborative initiatives could be implemented to achieve the organization’s

goals?

Chapter 5:

Guiding the

strategy and

resource allocation

of healthcare

organizations

How could strategy be modeled to guide resource allocation?

What translational research projects provide the most value through the

organization’s eyes?

How could strategy and long-term goals be characterized and operationalized to

support healthcare decision makers?

What is the relative importance of the different long-term goals?

How could multiple-criteria support the selection of an optimal mix of proposals?

How could a researcher rapidly estimate the impact of a healthcare intervention?

11

Table 1-2. Gaps, approach, and main contributions of Chapter 3

Chapter 3 Gaps that need to be filled and Motivation

Disagreements about the meaning and scope of translational research have led to a lack of

robust frameworks to evaluate complexity of translation (Woolf, 2008). This has caused

inefficiencies in the allocation of resources having as a final consequence a huge gap between

evidence-based interventions and clinical practice (IOM, 2001, Green et al., 2009, Glasgow et

al., 2012). In response to this, major funding agencies have asked for frameworks in which

systems engineering tools and quality improvement efforts can provide data-driven solutions

by informing and supporting the decision making processes (Reid et al., 2005).

Approach and Main Contributions

The proposed QFD-AHP framework contributes to a better understanding of complexity in

translational research. The framework helps to identify and quantify the impact of various

operational steps and technical requirements on translating new discoveries into practice.

These results create evidence on a strategic level to inform key stakeholders about how efforts

and resources should be allocated to optimally move clinical innovations to impact people’s

health.

As claimed by Woolf (2008), discrepancies in the meaning of translational research have led

to an unclear definition of its scope. In response to this, the proposed methodology can quantify

complexity and generate evidence independently of the model or definition adopted by key

stakeholders in translational research. Therefore, the QFD-AHP framework is robust and

flexible enough to be applied consistently across various health disciplines.

A case study is shown to illustrate the usability of the framework. Future research work

includes a more detailed explanation of how this tool can be used to generate agreement on the

most important elements involved in translating knowledge, identification of benchmark

within and between health disciplines, and mechanisms to identify similar translational

research projects in which best practices can be formalized and standardized.

12



The CTSA has emphasized that enhancing collaboration is one of its core objectives (RFA-

TR-14-009). Even though collaboration and multi-disciplinary are seen as critical components

to accelerate translational research (Barrett et al., 2008, Marincola, 2003), lack of metrics and

methodologies to assess collaborative efforts has limited our ability to investigate how to

improve and design collaborative networks.


The proposed SNA methodology provides visualization of collaboration networks and

identifies meaningful metrics to assess collaboration. In practice, SNA contributes to

identifying leaders, clusters, and patterns at the individual, as well as the organizational level.

Additionally, SNA is capable of assessing structural collaboration holes, identifying

multidisciplinarity patterns, and determining whether the organization is providing the

infrastructure to accelerate translational research. Future research work includes a more

detailed analysis to identify differences between current and optimal collaboration networks

to accelerate the translation of knowledge. In addition, the impact of programs such as the

CTSI, could be characterized by auditing and monitoring the changes in the collaborative

structures and patterns over time.

13



The healthcare system in the U.S. has been progressively shifting to a paradigm in which data-

driven support is considered to provide guidance to understand value (Kaye et al., 2014).

However, there is still a gap in terms of understanding the drivers of value and how its multiple

factors can be balanced to provide strategic guidance for an effective healthcare service and

management. In this sense, data-driven decision making tools are need to aim for a better

distribution of resources based on anticipated impacts that different healthcare interventions

can have on the population (Patrick and Erickson, 1993).


The use of GP based on strategic goals of a healthcare organization allowed for a more

informed and sustainable allocation of resources. In particular, the strategy of the CTSI can be

operationalized and formalized to guide the selection of a mix of proposals that provides the

best value for the organization based on multiple, typically conflicting, criteria. The GP

framework was found to provide good guidance to understanding and formalizing the goals of

the organization, identifying and formalizing the constraints, using historical data to provide

feedback, and selecting an optimal mix of proposals that fit into the strategy of the

organization. Another main contribution of this research is the provision of a rapid impact

estimation framework to guide the researcher through relevant questions to estimate the impact

of a healthcare intervention. Moreover, this multiple-criteria optimization model can be

complemented by incorporating cost-effectiveness analysis. The contributions presented in

this chapter are highly aligned with current needs expressed by the NCATS Advisory Council

Working Group and the recommendations of the External Board Advisory group to strengthen

the impact of the CTSI PSU hub.

Chapter 2

LITERATURE REVIEW

This chapter explores literature that helps to build the knowledge in which this dissertation is based upon.

This literature review covers pertinent topics to expand and support a more research-oriented quality

improvement approach to be applied in healthcare fields. Hence, this review explores relevant technical

aspects as well as some areas of applications of current literature in quality improvement and systems

engineering linked to healthcare fields. The tools and methodologies to be reviewed provide a better

understanding of the different complexities in the translational research process, as well as some elements

of the healthcare system. Moreover, these tools can be used to align and design decisions that led to more

informed quality improvement efforts, and hence, improve the performance of healthcare systems.

This literature review chapter is structured according to the main body chapters mentioned in Chapter 1. In

section 2.1 an overview of systems engineering tools that can be applied in the healthcare domain is

presented. Section 2.2 reviews a set of tools to translate customers’ needs into technical requirements.

Section 2.3 provides a review of social network techniques. Finally, in sections 2.4, 2.5, and 2.6 topics

related to multiple-criteria optimization, proposal selection, and cost-effectiveness analysis are reviewed.

15

2.1 Overview of Systems Engineering Tools in Healthcare

Systems engineering tools and techniques offer tremendous aid to address complex challenges in healthcare

and improve its performance (Grossman, 2008, Reid et al., 2005, McDonough et al., 2004). Healthcare

systems are by nature dynamic and complex. Therefore, the analysis of such systems requires techniques

that support the understanding of their elements and interactions from a systemic perspective. However,

although systems engineering tools have been successfully applied in various complex industries, their use

in healthcare has remained relatively low. In recognition of this fact, the IOM and NAE have encouraged

the use of system engineering tools as they promise to have a significant impact on quality and effectiveness

of healthcare systems (Reid et al., 2005). The systems engineering tools can be broadly classified into three

groups: 1) tools for system design; 2) tools for system analysis; and 3) tools for system control and

monitoring. The first group of tools aims to developing and designing new and better healthcare systems

and processes. The second group aims to analyzing existing healthcare systems based on a better

understanding of their complexity and performance. The third group of tools seeks to control the

performance of a healthcare system, and thus, recommend corrective or preventing actions to achieve the

targeted levels of performance.

2.1.1 Tools for system design

This set of tools is used to design new healthcare systems assuring that their characteristics meet the

requirements of different stakeholders involved in the supply chain of healthcare provision. This set

includes techniques such as Design for Six Sigma (DFSS) (Breyfogle III, 2003, Yang and El-Haik, 2003),

concurrent engineering (Prasad, 1996), Human Factors (HF) engineering (Sanders and McCormick, 1987,

Lehto and Landry, 2012), and Quality Improvement (QI) for failure analysis tools (Breyfogle III, 2003). In

DFSS, a system is designed based on the expected outcomes of a product or service. An important element

to consider when designing a system is to capture the voice of the customer or key stakeholders. This leads

to the identification of the main customer needs that guide the development of engineering systems

parameters to meet those requirements (Goffin et al., 2012). Similarly, concurrent engineering also seeks

to develop new systems that meet needs and customers’ aspirations. Some of the specific tools used for this

purpose are: quality function deployment (QFD) (Akao et al., 1990), house of quality (HOQ) (Hauser and

Clausing, 1988), design for X (Huang, 1996), and design of experiments (DOE) (Fisher, 1992, Fisher, 1935,

Kuehl and Kuehl, 2000), among others. Two of these techniques; QFD and HOQ, are described in more

details in section 2.2. Human factors engineering tools are also typically used for designing systems. One

of the main considerations of this field is to understand the interaction among different human elements in

a system to simplify its complexity. Some of the HF’s main areas of design include the understanding of

16

physical, cognitive, and organizational ergonomics. Finally, QI methods are useful for characterizing a

system as a way to diminish or remove potential causes of error. In this sense, the use of tools such as the

failure mode and effects analysis (FMEA) (Stamatis, 2003) and root cause analysis (RCA) (Wilson, 1993)

can provide an adequate understanding of the system and avoidable defects to guide an enhanced design.

2.1.2 Tools for system analysis

These tools serve to identifying the key elements of a system, understanding their behavior, interactions,

and performance of the system. The main aim of these tools is to identify areas of opportunity for improving

a system. The range of tools that can be used for system analysis is broad. It includes mathematical and

statistical analysis, simulation, management, financial, and knowledge discovery tools. Mathematical and

statistical tools include linear programming, dynamic programming, queuing theory, and multiple-criteria

programming, among others (Ravindran, 2007). A deeper description of multiple-criteria programming is

given in section 2.4. Simulation techniques are typically used to analyze the behavior and performance of

a system and respond to various “what-if” questions or scenarios to improve the system’s performance. In

practice, discrete-event systems (Banks et al., 2000) and Montecarlo simulation techniques (Rubinstein and

Kroese, 2011) have been successfully used in healthcare fields. Systems management techniques are used

to analyze the systems across its different elements. Some of the tools composing this group are: supply

chain management, game theory, and systems dynamic models. Financial engineering tools provide a better

understanding of causal relationships among different system variables. These group includes tools such as

econometrics (Wooldridge, 2012), data-envelopment analysis (Cooper et al., 2007, Coelli et al., 2005), and

risk analysis (Kaplan and Garrick, 1981), among others. Finally, knowledge discovery tools aim to extract

useful knowledge from large databases. Some of the techniques included in this group are: data mining

(Tan et al., 2006), principal component analysis (Jolliffe, 2002), and social network analysis (Borgatti et

al., 2009, Scott, 2012). This last tool is described into more detail in section 2.3 and applied in Chapter 4.

2.1.3 Tools for system monitoring and control

This group of systems engineering tools is used to monitor and control the processes and performance of a

system. The main aim of these tools is to maintain the system performance operating under specified

parameters which are considered to achieve expectations. The most relevant tools of this group are

statistical process control techniques and other QI techniques to complement the monitoring and control of

healthcare processes (Breyfogle III, 2003).

17

2.2 Tools to Capture Customer Needs and Technical Requirements

2.2.1 Quality function deployment

QFD is a quality tool that offers a structured framework to transform customer requirements into

characteristics of either a new product/service or an existing system. This methodology was first proposed

in Japan during the late 60’s and formalized in 1972 (Akao, 1972). The main aim of this tool is to translate

customer desires into product design or specific characteristics. Understanding these elements can reduce

the product development time by half and start-up engineering costs by about 30% (Hauser and Clausing,

1988). Although this technique was initially developed to support product design, it has been also

implemented to design new services (Jeong and Oh, 1998, Trappey et al., 1996, Ermer and Kniper, 1998,

Jacques et al., 2009). In practice, QFD is typically supported by a technique called the House of Quality

(Govers, 1996). More details about the HOQ are given in section 2.2.2.

QFD techniques have been used as decision support tools in various industries (Zare Mehrjerdi, 2010, Chan

and Wu, 2002). Main functional fields of QFD include product development, quality management,

customer needs analysis, design, planning, decision-making engineering, management, teamwork, timing,

and costing. Hence, QFD is considered to be a very versatile and flexible tool for a wide number of

applications (Chan and Wu, 2002). Although the applications of QFD in healthcare have remained low,

they are expected to increase (Gremyr and Raharjo, 2013). In healthcare, QFD has been applied to improve

rehabilitation services (Einspruch, 1996), improve quality of services in healthcare systems (Radharamanan

and Godoy, 1996), transferring residents’ expectations into improvements in a residential nursing home

(Chang, 2006), radiation safety management (Moores, 2006), and incorporating customer requirement to

redesign and renewal of healthcare organizations (Dijkstra and van der Bij, 2002), among few other

applications. Chaplin and Akao (2003) propose a comprehensive methodology for using QFD within the

healthcare domain. Their methodology is based on five steps; the voice of the customer, the voice of the

organization, the voice of the process, the voice of the staff, and actions of staff.

2.2.2 House of quality

The HOQ is one of the main tools used in QFD to capture customer requirements and identify the technical

factors that fulfill those requirements. This methodology was proposed by Hauser and Clausing (1988) as

a way to improve product quality based on a structured methodology to translate customer needs into

measurable technical requirements. Thus, HOQ can be seen as a conceptual map for quality improvement.

Usually, seven elements are needed to build a HOQ framework; Customer Requirements (What), Technical

Requirements (How), Customer Requirements Weights (relative importance of “What”), Relationship

18

matrix (relations between “What” and “How”), Correlation matrix (inner dependency among “How”),

Impact of Technical Requirements, and Competitive Benchmarking. Figure 2-1 shows a traditional diagram

for the HOQ methodology.

Figure 2-1. House of quality diagram

1. Customer Requirements Matrix: This section contains a list of customer requirements (VOC), also

called customer attributes (CAs).

2. Weights of Customer Requirements: This section contains the relative importance of the customer

requirements previously identified. This is based on the premise that through the customers’ eyes,

some attributes are more important than others, and therefore, they should be somehow prioritized.

3. Technical Requirements Matrix: Contains a list of technical requirements or drivers to fulfill the

customer requirements. The technical requirements are also known as engineering characteristics

(ECs).

4. Correlation Matrix: It establishes the relationship intensity among the technical requirements.

5. Relationship Matrix: It is the body of the HOQ. Its function is to establish connection between

customer’s needs and technical requirements designed to improve the product or system.

6. Impact of Technical Requirements Matrix: It determines the absolute, relative and order priority for

each technical requirement. This section serves as a roadmap for engineers or designers to prioritize

technical requirements according to their importance in addressing the customer needs.

Relationship MatrixCompetitive

Benchmarking

Customer

Requirements

(What)

Impact of Technical

Requirements

Technical

Requirements

Correl. Matrix

CR

W

eig

hts

1 2

3

4

5

6

7

19

7. Competitive Benchmarking: Its purpose is to compare how well customer needs are met compared

to the competitors. This section provides a self-evaluation that could be used to generate

competitive advantages and a better understanding of the competition.

2.2.3 Analytic hierarchy process

AHP was first proposed by Saaty (1980) as a mechanism to determine the importance of different criterion

and be able to compare alternatives based on multiple objectives. AHP is a powerful decision making tool

capable to deal with complex, non-linear and multiple-criteria problems, such as those presented in

healthcare systems. The range of applications of AHP is wide; it includes but is not limited to resource

allocation, evaluating alternatives, and conflict resolution. A main advantage of this tool is that it is suitable

for both tangible and intangible factors. AHP allows for quantifying complex qualitative factors in a

structured way. Another clear advantage of AHP is its flexibility since it has a tolerance for judgment

inconsistency.

An extensive survey of the areas of application of AHP is presented in Vaidya and Kumar (2006). In the

article, eight main categories in the use of AHP were identified; selection, evaluation, benefit-cost analysis,

allocations, planning and development, priority and ranking, decision making, and forecasting. Areas of

application of AHP are wide; they include but are not limited to business, logistics, manufacturing,

education, military and also healthcare.

2.2.3.1 AHP procedure for obtaining weights

One of the main uses of AHP is to obtain weights that can serve to rank different criteria according to their

relative importance. To obtain these weights, a pairwise comparison between all the criteria is conducted.

From this step, a pairwise comparison matrix A is developed by determining the relative importance of one

criteria over the other. Typically, a 9-point scale is used to complete the matrix (Saaty, 1980). An

explanation of this scale and its appropriateness for AHP is given by Saaty (2001). A description of the

intensities of importance using a 9-point scale is presented in Table 2-1.

The pairwise comparison matrix A is constructed as follows:

𝐴 =

[

1 𝑎1,2 𝑎1,3

𝑎2,1 1 𝑎2,3

𝑎3,1 𝑎3,2 1

… … 𝑎1,𝑀

… … 𝑎2,𝑀

… … …… … …… … …

𝑎𝑀,1 𝑎𝑀,2 …

. . . … …… . . . …… … 1 ]

20

Where a1,2 represents how much more important is criterion 1 with respect to criterion 2. It must be noted

that a2,1 is the reciprocal of a1,2. It is also intuitive the fact that the main diagonal is populated by 1s since a

criterion compared to itself is equally important. The pairwise comparison matrix is an 𝑀x𝑀 matrix. The

number of pairwise comparison questions needed to generate the matrix is 𝑀(𝑀 − 1)/2.

Table 2-1. AHP scale definition

Intensity of

Importance Definition Explanation

1 Equal importance Two activities contribute equally to the objective

3 Moderate importance According to experience a criterion is slightly more

important than the other

5 Strong importance According to experience a criterion is strongly more


7 Very strong or

demonstrated importance

According to experience a criterion is favored very

strongly over the other

9 Extreme importance Evidence shows that a criterion is absolutely more


2,4,6,8 Intermediate values

After building the pairwise comparison matrix, a normalization procedure is needed to obtain the relative

weights for each criterion. A normalized pairwise comparison matrix N is constructed by dividing each cell

value by the sum of its corresponding column represented by Sj. Finally, the weight or relative importance

of the criterion is obtained by averaging the cell values across the correspondent raw. The formulas and

matrix structure used are as follows:

𝑆𝑗 = 1 + ∑ 𝑎𝑞,𝑗 , 𝑗 = 1,… ,𝑀

𝑀

𝑞=1,𝑞≠𝑗

Eq. 21

𝑁 =

[ 𝑛1,1 𝑛1,2 𝑛1,3

𝑛2,1 𝑛2,2 𝑎2,3

𝑛3,1 𝑛3,2 𝑛3,2

… … 𝑛1,𝑀

… … 𝑛2,𝑀

… … …… … …… … …

𝑛𝑀,1 𝑛𝑀,2 …

… … …… … …… … 𝑛𝑀,𝑀

]

21

𝑁 =

[

1/𝑆1 𝑎1,2/𝑆2 𝑎1,3/𝑆3

𝑎2,1/𝑆1 1/𝑆2 𝑎2,3/𝑆3

𝑎3,1/𝑆1 𝑎3,2/𝑆2 1/𝑆3

… … 𝑎1,𝑀/𝑆𝑀

… … 𝑎2,𝑀/𝑆𝑀

… … …… … …… … …

𝑎𝑀,1/𝑆1 𝑎𝑀,2/𝑆2 …

. . . … …… . . . …… … 1/𝑆𝑀 ]

Thus, the weights for each criteria are given by the following column vector w:

𝑤 =

[ 𝑊1

𝑊2

⋮⋮

𝑊𝑀

]

=1

𝑀

[ ∑ 𝑛1,𝑗

𝑀

𝑗=1

∑ 𝑛2,𝑗

𝑀

𝑗=1

⋮⋮

∑ 𝑛𝑀,𝑗

𝑀

𝑗=1

]

Since the values were normalized, the weights will sum to 1. These values could be used to prioritize criteria

and conduct a comparison among different alternatives based on their scores for each criterion.

In order to assure the consistency of the responses, a consistency procedure should be used to minimize

errors and make valid inferences at the end of the procedure. For these purposes, a consistency index (CI)

and consistency ratio (CR) can be used and calculated as follows:

𝐶𝐼 =𝜆𝑚𝑎𝑥 − 𝑛

𝑛 − 1 Eq. 22

Where 𝜆𝑚𝑎𝑥 is the average of the maximum eigenvalues and can be obtained by solving:

𝐴𝑤 = 𝜆𝑚𝑎𝑥𝑤 Eq. 23

Then, the consistency index can be calculated as follows:

𝐶𝑅 =𝐶𝐼

𝑅𝐼 Eq. 24

Where RI is the random consistency index depending on n, and can be obtained from Table 2-2.

Table 2-2. Random consistency index

n 1 2 3 4 5 6 7 8 9

RI 0 0 0.58 0.9 1.12 1.24 1.32 1.41 1.45

22

Usually, a threshold of 0.1 is used to determine if the consistency is acceptable. In case of achieving a

consistency ratio greater than 0.1, the evaluator is asked to revise his/her pairwise judgments to reduce

inconsistency and be able to make credible inferences from this technique (Saaty, 1977).

2.2.3.2 AHP in healthcare applications

Liberatore and Nydick (2008) reviewed AHP use in medical and healthcare decision making. According to

their article, AHP appears to be a promising support tool that can be used in almost every healthcare

process/area. Evaluation and selection of the best treatment or therapies for a certain population can be

determined as well as healthcare technologies and policies. From the 50 articles reviewed by Liberatore

and Nydick (2008), 29 of them were classified into management and administration, while 21 of them were

categorized as patient care applications.

Lai (2010) proposed an AHP methodology to evaluate the sustainability of knowledge-based communities

in the healthcare industry. Pecchia et al. (2011) used AHP to assess the relative importance of risk factors

for preventing falls in the elderly population. They identified and prioritized 35 risk factors organized into

general and clinical categories. Interesting differences in opinion from the various physicians involved in

the process were identified. Generating agreement was found to be critical to facilitate the implementation

and diffusion of evidence-based programs to allocate the resources in a wise manner. A novel application

of Monte Carlo-AHP to rank quality attributes in dental services was proposed by Hsu and Pan (2009). The

article presents a two phases methodology in which the authors examine the structure of dental services

using AHP. Then, they use Monte Carlo simulation to determine the priorities of the attributes identified.

Health technology assessment using AHP was studied by Danner et al. (2011), in which they include both,

patients and healthcare professionals to elicit patient preferences.

2.2.3.3 AHP and translational research

Although the popularity in the use of AHP in the health domains has been growing consistently, few

researchers have explicitly investigated the use of AHP to help assess and quantify translational research.

Cheever et al. (2009) presented one of the first articles published explaining that the use of AHP can be

very helpful to address the lack of speed in the translational research process. They used AHP for a pilot

project to prioritize cancer antigens. By the time of the publication, no cancer vaccine was yet approved.

Their methodology proposed four modules: 1) Identification of participants, criteria to be evaluated and

alternatives; 2) Essential criteria identification, categorization and comparison of these criteria. This phase

also included the weight of the criteria; 3) Evaluation of alternatives; and 4) Report and analysis of

information to help the decision making process. This study was able to rank 75 selected antigens. Even

23

though the AHP study conducted did not make a decision to launch a new vaccine, it provided a structured

way to evaluate different alternatives. To date, this article has been cited more than 400 times capturing the

attention of various researchers. It is expected that the number of related work will increase over the

following years.

Fernandez et al. (2010) agree with Cheever et al. (2009) that AHP is a very suitable tool to be used in

translational research in various settings including cancer clinical decisions. On the other hand Lang et al.

(2009) believe that AHP is not well described for translational research since it has many sources of bias.

They criticized the use of AHP and stated that besides its popularity in other areas, many potential

drawbacks can be found once it is used to prioritize cancer antigens. They argue that lack of transparency

in the AHP approach could lead to inaccurate results in the aim of prioritizing the antigens. However, by

properly selecting a group of evaluators, “lack of transparency” is reduced as well as potential bias.

Therefore, accurate results can be obtained to make valid inferences from the AHP methodology and its

application in translational research.

2.2.4 Integration of QFD and AHP

Many successful attempts of integrating QFD and AHP can be found in the literature (Kwong and Bai,

2002, Kwong and Bai, 2003, Karsak et al., 2003). These tools have been demonstrated to complement each

other, and therefore, significant benefits can be extracted by combining both methods. Ho (2008) conducted

a literature review for the integrated analytic hierarchy process and its applications. According to his article,

the integration QFD-AHP is one of the most suitable approaches for various areas since they complement

each other. AHP is able to overcome inconsistency to evaluate the relative importance for the attributes

described by the customers. Then, those weights can be used by QFD tools.

A handful of studies in healthcare have attempted to use the QFD-AHP integration with promising results.

Chang (2006) applied these concepts to enhance nursing home service quality. QFD was used to capture

resident expectations and transfer them to improvements. A Fuzzy AHP helped to prioritize and calculate

quality based on the client requirements. Although QFD-AHP has been found to be extremely helpful in

many areas, there is still a lack of investigation in its use in healthcare, especially in translational research.

Our main hypothesis is that the use of an integrated QFD-AHP could help to better understand complexity

and the factors that impact the acceleration of discovery into health outcomes. Thus, it can provide a robust

framework to understand and quantify complexity in translational research, which could be later used as

evidence for strategic decision making tool. In Chapter 3, a QFD-AHP approach is presented and applied

to translational research.

24

2.3 Social Network Analysis

Network-based analysis is a rapidly growing field as it is considered to be effective for extracting useful

information from large data and various applications. In addition, trends of use of network-based techniques

have been supported and facilitated by the development of automated computing analysis (Gündüz-

Öğüdücü and Etaner-Uyar, 2014, Scott, 2012). In this regard, Social Network Analysis (SNA) has been

widely used to understand complex interconnections within a network. A social network can be defined as

a set of entities or nodes connected by their relations (Scott and Carrington, 2011). In order to study the

different patterns and communalities among different groups of nodes or entities, SNA – a branch of graph

theory – is a widely used approach as it provides both structural and mathematical analysis (Gündüz-

Öğüdücü and Etaner-Uyar, 2014). Traditionally, SNA has been used as a visual technique to develop

network graphs, where nodes represent the entities of a network (individuals or groups) and the edges

represent the relationship between two entities. Furthermore, SNA provides metrics that can be used to

assess parts of a network or the network as a whole. Unlike other traditional evaluation methods in which

simple averaging and outcomes are compared, SNA provides a rich structure to evaluate the

interrelationship among individuals or institutions.

2.3.1 Network representation

The mathematical representation of a network, G, can be written as follows:

G = (V, E)

Where V is a set of entities or nodes, and E is a set of edges (pair of nodes).

A graph can be either directed or undirected depending on the characteristics of connection between the

nodes. A graph is said to be directed if the relationship among nodes has some sense of directionality. For

instance, leadership networks are usually treated as directed as some nodes influence other nodes in the

network but in some cases the opposite direction is inexistent. In contrast, undirected networks only denote

relationship but no directionality. A graphical representation of graph and its directionality is shown in

Figure 2-2. The sets of the directed graph can be written as V={1,2,3,4,5,6} and E={(1,2) ,(1,3), (2,1), (2,4),

(3,2), (3,4),(4,3), (4,5), (5,3), (5,6), (6,5)}. The undirected graph can be written as V={1,2,3,4,5,6} and

E={(1,2) ,(1,3), (2,3), (2,4), (3,4), (3,5), (4,5), (5,6)}.

25

Figure 2-2. Directed and undirected graphs

Another useful representation of these set of nodes and edges is called adjacency matrix. This matrix is a

square matrix of size nxn in which n represents the number of nodes in the network. The cells of the matrix

represent the presence of an edge between two nodes. For instance, a non-zero entry at position (i,j)

represents that there is an edge between nodes i and j. Additionally, this entry can represent the strength of

the corresponding edge. For undirected graphs, the adjacency matrix is symmetric.

2.3.2 SNA metrics

The study of SNA involves the graphical visualization of networks and the measurement of relevant metrics

that help understanding and characterizing the networks’ structure. These metrics seek to provide useful

information to understand the characteristics of the nodes, cluster, and network as a whole (Borgatti et al.,

2009, Hansen et al., 2010, Scott and Carrington, 2011). Network or cluster metrics are those which are

calculated for the whole network or subgroups of it. This group includes metrics such as density and

geodesic distance. On the other hand, vertex-related metrics are those which are calculated for each node

of a network. This group includes centrality metrics such as degree centrality, betweenness centrality, and

eigenvector centrality.

Density

One of the most used graph metrics is the density of the network. The density represents the number of

links among all the nodes as a proportion of all the potential links, indicating the connectedness of the entire

network. Density values range from 0 to 1, being 1 the value representing a completely connected network

(i.e. every entity is directly connected to each network member). This metric can be calculated for the

network as a whole or for different groups or clusters within the network. The maximum number of edges

in an undirected semantic network is given by |V|*(|V|-1)/2. The density of the network can be defined as:

26

𝐷𝑒𝑛𝑠𝑖𝑡𝑦 =2|𝐸|

|𝑉|(|𝑉| − 1) Eq. 25

Where |V| is the size of the network, represented by the number of nodes in the network, and |E| is the

number of edges in the network.

In a social network context, including healthcare organizations, denser networks are better for dissemination

of knowledge. However, from a collaborative perspective, denser networks can also indicate lack of

specialization, and potentially, decrease the rate of innovation.

Geodesic distance

The geodesic distance is defined as the shortest path or route between two nodes. In non-weighted edges

networks, the geodesic distance between two nodes is the minimum number of edges connecting them. This

metric indicates how reachable a particular node is for the other nodes. Typically, this metric is used to

evaluate the cohesion of a network. In order to characterize networks or clusters, the maximum and average

geodesic distances are used. In a social network context, the geodesic distance indicates the level of

reachability between the individuals composing the network.

Degree centrality

While the density and geodesic distance metrics are related to the whole network or clusters, the degree

centrality is a vertex-related network metric. In an undirected network, it measures the number of direct

connections of a particular node. Consequently, the degree centrality can be used as an indicator of the

importance of a node. In directed graphs, this metric is split into two; indegree and outdegree centrality

which represent the number of edges towards or from a node respectively. The degree centrality of a node

v is usually written as Cd(v) = deg(v). In a social network context, nodes with a relatively high degree

centrality are considered to be central for the network as they have a higher probability or receiving and

transmitting the information that flows within the network (Abraham et al., 2009). Hence, the centrality is

used as the main indicator to identify leaders. Therefore, it can be seen as a metric of influence (Freeman,

1979). A mathematical theoretical perspective on centrality metrics can be found in Borgatti and Everett

(2006).

27

Betweenness centrality

Betweenness centrality is another widely used vertex-related metric in SNA. This metric indicates the

relative level of centrality of a node with respect to other groups in the network. Technically, it quantifies

the number of times that a node serves as a bridge along the shortest path between other pairs of nodes.

Mathematically, the betweenness centrality of a node v can be written as:

𝐶𝑏(𝑣) = ∑𝜎𝑠𝑡(𝑣)

𝜎𝑠𝑡𝑠≠𝑣≠𝑡∈𝑉

Eq. 26

where σst is the total number of shortest paths between node s and node t, and σst(v) is the number of those

shortest paths that pass through node v.

In a social network context, individuals with high betweenness centrality are known as “bridgers” since

they support connectivity among different clusters. Bridgers play an important role for emerging healthcare

fields that require collaborative efforts from multiple disciplines. In other words, bridgers serve as

facilitators for multi-disciplines to advance at a faster rate.

Eigenvector centrality

Another relevant centrality measure of a node is the eigenvector centrality (Estrada and Rodriguez-

Velazquez, 2005). This metric is typically used to quantify the influence of a given node in a network.

Those nodes with a high eigenvector centrality are well connected to other nodes which are also well

connected. Hence, individuals with relatively high eigenvector centrality are also perceived as influencers

and are considered to have a higher degree of popularity within the network (Abraham et al., 2009).

2.3.3 Applications of SNA

The range of applications of SNA is wide. It has been applied to social sciences (Ma et al., 2014, Hatala,

2006), communication (Luo and Zhong, 2015, Liao et al., 2014, Hancock and Raeside, 2010), management

(Cross et al., 2002, ZHAO et al., 2014, Aubke et al., 2014), politics (Vasquez et al., 2011, Leifeld, 2013,

Matti and Sandström, 2013, Crooks et al., 2014, Khan et al., 2014), defense and crime (Duijn et al., 2014,

Nash et al., 2013, Sparrow, 1991, Ting and Tsang, 2014, Lu et al., 2010, Malm and Bichler, 2011), and

healthcare, among others.

28

2.3.4 SNA in healthcare

Applications of SNA in health related areas have been increasing during the past years. SNA has been used

in a wide range of health applications including collaboration, behavior change, and management, among

others. Collaboration applications include the assessment of physician collaboration (Uddin et al., 2013),

assessment of multidisciplinary health sciences collaboration (Weng et al., 2008), and evaluation of medical

culture (Lurie et al., 2009). Behavior change and disease spread applications of SNA include the assessment

of the spread of obesity in large social networks (Christakis and Fowler, 2007), the visualization of the

network phenomena in smoking cessation (Christakis and Fowler, 2008), the interaction patterns that

predict weight loss (Chomutare et al., 2014), and the description of the outbreak dynamics of tuberculosis

(Gardy et al., 2011). SNA has also been used to support management decision making. Other applications

include the design of dissemination strategies and innovations (West et al., 1999), support the introduction

of new vaccines (Wonodi et al., 2012), validation of infrastructure for inter-institutional translational

research (Hunt et al., 2012), evaluation of political feasibility of healthcare reforms (Wang, 2012), and

identification of next generation of trainers in nurse development programs (Benton and Fernández, 2014).

A review of other applications of SNA in healthcare can be found in Chambers et al. (2012).

Recently, SNA has gained major attention in translational research fields. The CTSA has called for a

stronger multidisciplinary collaboration across the multiple CTSA hubs. In response, some authors have

argued that using SNA could bring many advantages for analyzing collaboration patterns ((Long et al.,

2014, Hunt et al., 2012, FalkKrzesinski et al., 2010, Bian et al., 2014). Therefore, SNA is seen as a

promising technique for understanding mechanisms in which research collaboration can be enhanced to

develop a stronger multidisciplinary network (Bian et al., 2014).

2.3.5 SNA to assess collaboration networks

From a managerial and organizational perspective, SNA has been extensively validated as a tool to

investigate collaboration patterns to inform management about current opportunities in organizational

networks (Tichy et al., 1979, Brass, 1995, Borgatti et al., 2009). In this sense, understating the social capital

of an organization becomes relevant for management (Tsai and Ghoshal, 1998, Borgatti and Foster, 2003).

It has been argued that social capital facilitates the creation of value at both the dyadic and business unit

levels (Tsai and Ghoshal, 1998). Practical applications such as identifying flows of information, generating

teams, and understanding the impact of informal networks are just some examples of how SNA can be

implemented at the organizational level to assess collaboration. This last application has been found to be

relevant for supporting the strategy of organizations. Studies have demonstrated the importance of informal

29

networks and their impact on employee job satisfaction and performance (Cross et al., 2002). Hence, this

provides an opportunity for management to improve organizational efficiency through the use of

collaborative networks. Benton and Fernández (2014) use SNA to identify the influencers or leader of a

nurse network as a way to identify future generations of trainers.

Due to the importance of collaboration in research fields, SNA techniques have been used for analyzing

research collaboration networks (Li et al., 2013, Wang et al., 2012, Bornmann and Leydesdorff, 2015, Bian

et al., 2014, Zare-Farashbandi et al., 2014, Zhang et al., 2013, Abbasi et al., 2012, Zhai et al., 2014).

According to a recent study, more than half of the publications in various areas are co-authored (Bozeman

et al., 2013). The majority of the studies of research collaboration relies on bibliometric data. According to

Bozeman et al. (2013), this type of data has the advantages of being verifiable, stable, and easy to obtain.

In Chapter 4, a SNA bibliometric-based approach is used to evaluate collaboration networks in obesity

research. The study includes insights about the identification of leader or influencer, bridgers of knowledge,

clusters of research, and different multidisciplinary collaborative patterns.

2.4 Multiple-Criteria Optimization

In real life applications, decision makers are often exposed to problems in which multiple criteria must be

considered to select the best course of action among different alternatives. These decisions often exhibit

three main characteristics: the presence of multiple criteria, criteria that are conflicting, and the need for

making compromises or trade-offs among those criteria to achieve a balanced decision (Ravindran, 2007).

These problems are often classified into the field of multiple criteria decision making (MCDM) problem.

Usually, the main aim of MCDM techniques is to identify the best alternative given various conflicting

criteria.

In healthcare, for example, various criteria are considered when allocating resources, prioritizing patients,

scheduling patients, etc. (Hans et al., 2012). Unfortunately, given the complexity of these settings, usually

conflicting criteria are present and compromises among them are required. Therefore, in practical situations,

methods should be used to address those criteria based on their importance to reach a given objective.

A generic multiple criteria mathematical problem can be written as:

𝑀𝑖𝑛 𝑍 = (𝑓1(𝑥), 𝑓2(𝑥),… , 𝑓𝑘(𝑥))

Subject to: 𝑥 ∈ 𝐹

30

Where x is a set of m decision variables over which the decision maker has control. The set F is a feasible

subset of the solution space X. Therefore, F is defined by a set of constraints as function of x.

One of the most used techniques to solve these types of problems is Goal Programming (GP). In GP, all the

objectives are assigned target levels for desired achievement and also a relative priority to meet those

targets. In other words, GP treats those targets as aspirational levels and not as absolute constraints as in

other mathematical approaches.

2.4.1 Goal programming

Goal programming has been extensively used to help multiple criteria decision making (MCDC) since its

introduction in mid-1950s (Charnes et al., 1955). A more formal structure and theory was given by Charnes

and Cooper (1961). Further development of this technique was presented by Ijiri (1965), Lee (1972), and

Ignizio (1976). Since mid-1970 and due to its wide range of applications, GP became one of the most

popular techniques in MCDM field. There are four main philosophies under which GP is framed:

satisficing, optimizing, ordering or ranking, and balancing (Jones and Tamiz, 2010). Satisficing refers to

the situation in which the decision maker is satisfied with a level of aspiration that is typically less than

optimal. This is clearly incorporated into GP models as they contain a set of goals to be reached. Optimizing

refers to selecting the best option or alternative given a set of possible decisions. Ordering and ranking is

present in all GP models as decision makers may have some preferences to achieve certain goals. Finally,

balancing seeks to find a trade-off between the achievements of the goals.

In GP, the main objective is to minimize undesired deviations from pre-specified targets. For example, a

goal that is not fully achieved has an under-achievement, which is associated to a negative deviation. On

the other hand, some goals can be over-achieved, then, a positive deviation is associated to that goal.

In goal programming there are two types of constraints; systems constraints and goal constraints. Systems

constraints are also called hard constraints as they cannot be violated. In other words, these set of constraints

are more restrictive and have to be satisfied before the goal constraints. Goal constraints are called soft

constraints. There are different variants of GP formulations to minimize unwanted deviations; weighted,

preemptive, and Tchebycheff, among others. The selection of a specific formulation depends on the

problem that is being considered.

2.4.1.1 Weighted goal programming

In the weighted GP, also called non-preemptive GP, it is assumed that the decision maker believes that

goals are not equally important. Hence, they are assigned different weights. In such cases, the objective of

31

the model seeks to minimize the total weighted deviations from the targets. In general, a GP model can be

written as follows:

𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑍 = ∑(𝑤𝑖+𝑑𝑖

+ + 𝑤𝑖−𝑑𝑖

−) Eq. 27

𝑘

𝑖=1

𝑆𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑓𝑖(𝑥) + 𝑑𝑖− − 𝑑𝑖

+ = 𝑏𝑖 for 𝑖 = 1,… , 𝑘

𝑔𝑗(𝑥) ≤ 0 for 𝑗 = 1,… ,𝑚

𝑥𝑗, 𝑑𝑖−, 𝑑𝑖

+ ≥ 0 for all 𝑖 and 𝑗

This model seeks to minimize the weighted sum of deviational variables in which the variables

𝑑𝑖− and 𝑑𝑖

+ represent the under achievement and over achievement of the i-th goal with respect to its target

bi. In this approach, the decision maker is asked to specify an acceptable level of achievement (bi) for each

criterion fi and a positive weight wi representing the relative importance of the criterion associated with the

deviation between fi and bi. This approach for GP model is also referred as pre-specified or cardinal. A

different approach is required when absolute priorities exists. In such cases, preemptive GP is used.

2.4.1.2 Preemptive goal programming

Given the fact that in practical applications it is sometimes difficult to assign relative priorities for meeting

different criteria, ordinal order or absolute priorities can be used instead. In such a case, the different criteria

and their goals are being met in a pre-specified order according to their preemptive priority. In other words,

preemptive goal programming is a sequential optimization process in which low priority goals are

considered only after high priority goals are achieved. The preemptive GP can be formulated as follows:

𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑍 = ∑𝑃𝑝

𝑝

∑(𝑤𝑖𝑝+𝑑𝑖

+ + 𝑤𝑖𝑝−𝑑𝑖

−) Eq. 28

𝑖

𝑆𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 𝑓𝑖(𝑥) + 𝑑𝑖− − 𝑑𝑖

+ = 𝑏𝑖 for 𝑖 = 1,… , 𝑘

𝑔𝑗(𝑥) ≤ 0 for 𝑗 = 1,… ,𝑚

𝑥𝑗, 𝑑𝑖−, 𝑑𝑖

+ ≥ 0 for all 𝑖 and 𝑗

where Pp represents priority k with the underlying assumption that Pp is much larger than Pp+1. The ith

deviational variable at priority k are represented by the weights 𝑤𝑖𝑝+ and 𝑤𝑖𝑝

− . In essence, this approach

32

consists of sequential single objective optimizations by successive optimizing the goals with higher priority.

Then, in order to optimize the goal associated to Pp+1, the goal Pp must be optimized previously.

2.4.1.3 Tchebycheff goal programming

A third variant of GP is the Tchebycheff GP also known as MinMax GP. This variant seeks to minimize

the maximum deviation from the stated goals. In other words, instead of minimizing the sum of all

deviations from the targets, the model minimizes the maximal deviation from any target. The use of this

approach is practical when the decision maker tries to achieve a balance of achievement for the set of goals

instead of prioritizing some goals over the others.

The objective function of this model becomes:

𝑀𝑖𝑛 𝑀𝑎𝑥 (𝑑1−, 𝑑2

−)

This can be reformulated as a linear objective by setting

𝑀𝑎𝑥 (𝑑1−, 𝑑1

−) = 𝑀 ≥ 0

This formulation is equivalent to the following:

𝑀𝑖𝑛 𝑍 = 𝑀

𝑆𝑢𝑏𝑒𝑐𝑡 𝑡𝑜 𝑀 ≥ 𝑑1−

𝑀 ≥ 𝑑2−

2.4.1.4 Obtaining weights

Typically, not all criteria can be considered equally important. In this case, the GP variant that should be

used is the weighted GP. In such cases, it is important to weight the criteria based on the decision maker’s

preferences. Assigning weights has two main purposes: accounting for changes in the variation range for

each evaluation measure, and accounting for the degrees of importance to these ranges of variation

(Kirkwood, 1997).

There are various techniques that can be used to obtain the weights or relative importance of the different

criteria. One of the most used methods consists of using a rating scale. In this technique the decision maker

is asked to rate the importance of each criteria in a scale from 1 to 10, being 1 the least important and 10

the most important. An alternative to this approach is the ranking method, in which Borda Count can be

used to rank criteria accordingly (Black, 1976). Another widely used method is the Analytic Hierarchy

33

Process (Saaty, 1980). In this method the decision maker is asked to conduct a pairwise comparison among

all the criteria. The respondent quantifies how important one criteria is compared with the others in a scale

from 1 to 9 (Vaidya and Kumar, 2006).

2.4.1.5 Scaling and normalizing goal constraints parameters

As in goal programming multiple criteria are incorporated into a single objective function, goal constraints

must be scaled or normalized. This reduces the bias due to the magnitude of the parameters used. The main

aim of normalizing procedures is to put those goal constraints or deviational weight variables into a

comparable basis. Although the interpretation of the results becomes harder, they are more accurate in

reflecting the decision makers’ preferences. The selection of a specific scaling method or a group of them

will depend on the nature of the problem under analysis and decision makers’ preference. Various scaling

methods have been studied, including simple scaling, ideal value, linear normalization, and vector scaling,

among others (Masud and Ravindran, 2008). Velazquez et al. (2010) study different combinations of

weighting and scaling methods, and how their selection may yield to considerably different solutions.

2.4.2 Goal programming in healthcare

Giving the nature of the healthcare system in which multiple criteria must be considered by decision makers,

GP has become a popular approach for determining an optimal allocation of resources. As stated by Lee

and Kwak (1999), the problem of resource allocation in healthcare is a significant and integral part of

strategic planning to provide effective healthcare service and management. Wacht and Whitford (1976)

were pioneers in using GP in healthcare. They proposed a model for expenditure allocation in non-profit

hospitals. Some of the goals included in their paper are: facilitate teaching and research, create a center of

excellence in healthcare, maintain skilled and motivated healthcare workforce, among others. Tingley and

Liebman (1984) developed a model for public health resource allocation. Specifically, they used integer

GP to support resource allocation of a state program aiming to provide nutritious food supplements and

nutrition education to low income groups. Lee et al. (1999) used an AHP-based integer GP model to identify

the best healthcare information resource planning. In their approach, they considered four main goals:

budget allocation, project implementation, network construction, and human resource allocation. A two-

phase approach using simulation and GP for healthcare planning is proposed by Oddoye et al. (2009). In

the first phase of their approach, they estimate key metrics using simulation. In the second phase they use

GP for trade-off analysis. The objectives used were patient queue lengths for nurses, patient queue lengths

for doctors, total length of queues in the system, waiting time within the queues, and number of beds. Blake

and Carter (2002) use GP to strategic resource allocation in acute care hospitals. The main objective of the

proposed model is to provide a right case mix while accounting for case cost. Thus, the objective function

34

of this problem ensures that the hospital is able to generate enough revenue, ensures that physicians are able

to generate their expected income level, right level of capacity use, and allow physicians to perform a case

mix preferred by them. A model for human resource allocation is presented by Kwak and Lee (1997) to

assign personnel to different shifts considering the minimization of total payroll while accounting for

patient satisfaction. Kwak and Lee (2002) used GP to support resource allocation with strategic planning

for business process infrastructure in healthcare systems. For their purposes, four main goals were

considered: financial budget process, information management process, operational process, and personnel

process. These goals were incorporated to allow the healthcare system to respond to new innovations and

competitiveness. A model for planning resource requirements in healthcare organizations is proposed by

Bretthauer and Cǒté (1998). They present two specific applications: a blood bank and a health maintenance

organization.

Other applications of GP in healthcare include nurse scheduling (Moz and Pato, 2005, Arthur and

Ravindran, 1981, Azaiez and Al Sharif, 2005, Musa and Saxena, 1984, Ozkarahan and Bailey, 1988,

Ozkarahan, 1991), allocation of surgeries to operating rooms (Ozkarahan, 2000), supply and demand

planning considering hospital location and service allocation (Chu and Chu, 2000), and healthcare waste

management (Chaerul et al., 2008), among others.

In summary, GP models have been applied in healthcare to facilitate decision-making planning process and

managerial policy. This technique provides a rich structure to overcome the challenges of balancing

different criteria to achieve the “best” solution for different MCDC problems. Applications of GP in

healthcare have been proposed for both tactical and strategic levels to optimize the allocation of resources.

Although GP has been extensively studied in healthcare applications, the proposal selection problem has

not been addressed in the literature. Given the nature of this problem, in which multiple strategic goals

should be taken into account, the authors envision that GP models can bring its various advantages to

address the proposal selection problem.

2.5 Proposal selection methods

Decision makers in various industries, including healthcare, deal with complex resource allocation

dilemmas. From a utilitarian perspective, decision makers aim to distribute the resources and select a certain

mix of projects that maximizes the value given the available resources. However, most of the time, assessing

the potential value of those projects is on its own a very challenging task. This fact is especially relevant in

scientific resource allocation decisions. In this case, scientific organizations providing internal or external

funding require different proposal selection methods to assess the quality and broadly estimate the potential

impact of those proposals. Most of the existing and currently used proposal selection methods to assess the

35

quality of a proposal include: overall impact, significance, innovation, approach, investigator’s credentials,

and broad impact of the proposal, among other indicators that could serve as predictors of success. These

elements are typically used as estimators of value. However, there is no general agreement on what

scientific excellence means (van Arensbergen and van den Besselaar, 2012). In addition, most scoring

methods for proposal selection are used as screening tool as some other considerations are difficult to

quantify in isolation, and their value is better assessed in terms of mix of selected proposals. Hence, the

final selection of proposals is typically conducted by expert evaluators considering only those proposals

that meet some minimum quality criteria.

One of the main challenges in proposal selection is to remove subjectivity. Bias has been found to be

impossible to eliminate completely from the review process (Lamont, 2009). Therefore, the same proposal

could be given significantly different scores depending on the composition of the reviewing committee. For

example Sandström and Hällsten (2008) found that the relationship between the reviewer and the applicant

influences the rating. Additionally, studies have concluded that the grant allocation is apparently determined

about half by the characteristics of the proposal and the applicant, and half by chance (Cole and Simon,

1981). In this sense, more robust proposal selection methods are needed to optimally allocate resources

while considering multiple relevant criteria in the decision making process. Moreover, these methods are

needed for improving transparency, quality, and legitimacy of grant allocation practice and policy (van

Arensbergen and van den Besselaar, 2012).

Although proposal selection methods have been proposed in fields of decision analysis, operations research,

economic models and others (Tian et al., 2005, Yager, 1993), it has been found that, in general, these

elaborated knowledge-based models and methods are not being used, and therefore, their impact has been

limited in practice (Liberatore and Stylianou, 1995). One of the key considerations that has been recognized

to be important in proposal selection is that the selected projects should have a fit into the overall

programmatic agenda of the organization (Yager, 1993). However, there is still a lack of tools and

understanding for organizations to guide the proposal selection to support their long-term goals, vision, and

strategy. In response to these needs, multiple-criteria optimization models provide a god fit for

incorporating multiple objectives into a single optimization function.

36

2.6 Cost-Effectiveness Analysis

Cost-utility analysis (CUA) and cost-effectiveness analysis (CEA) are two of the most used tools to guide

health policy making in resource allocation. Cost-effectiveness research integrates 1) measurements of

change of health post-intervention, 2) duration of life expectancy, and 3) cost of treatment or intervention

(Lubowitz and Appleby, 2011). The integration of these three elements provides a support for guiding

choices about public health policies based on effectiveness and costs of health interventions. Therefore,

opportunities of a better allocation of scarce resources could be achieved while improving health outcomes.

Consequently, the use of CEA in health and medicine has been recommended to provide an appropriate

baseline for comparisons in resource allocation decisions (Weinstein et al., 1996). One of the key

characteristics of CEA is that it standardizes the gains in health relative to the cost of different interventions

(Jamison et al., 2006). Although it is not the only criteria that is considered in resource allocation decisions,

it is an important one as it integrates financial and scientific aspects.

In practice, CEA is mostly used to compare different health interventions based on cost-effectiveness

measures such as cost per quality-adjusted life years (QALYs) and incremental cost-effectiveness ratio

(ICER). To conduct a CEA, researchers need to understand and specify the different impacts of the health

intervention (Jamison et al., 2006). Therefore, the researcher should anticipate what aspects of the health

will change due to the intervention. These aspects could include the risk, duration, or severity of a health

problem. The potential impact will mostly depend on the characteristics of the intervention. The types of

interventions can be categorized into three groups; primary prevention, secondary prevention, and case

management or tertiary interventions. The first group includes those health interventions that aim to

decrease the risk of an adverse health event. The secondary prevention group seeks to prevent an adverse

health event to occur again. Finally, the case management group includes cures, acute care, rehabilitation,

and other interventions that occur after an adverse health event (Figure 2-3).

As in any other methods to provide a fair estimation, an adequate level of detail is required. In this sense,

to conduct a CEA, the intervention needs to be fully characterized and defined at its different levels of care.

In addition, the costs involved in the implementation and delivery of the health intervention should be

understood. The cost structure selected for conducting the CEA depends on the type of intervention and

researcher focus. Hence, the estimation could include direct and/or indirect costs. For comparison of

interventions, however, the cost structure used for the estimation must be the same to avoid bias due to

inconsistent bases. For consistency across studies, including only direct costs is generally recommended

(Jamison et al., 2006).

37

3

4 Figure 2-3. Types of health interventions (adapted from Jamison et al. (2006))

5

Cost-effectiveness analysis generally uses QALY to guide and distribute healthcare resources more

efficiently (Shiroiwa et al., 2010). Typically, cost-effectiveness thresholds are used determine whether an

intervention achieve an acceptable value given the costs involved (Torrance et al., 1996). However, even

though the concept of threshold is used by healthcare decision makers in practice, explicitly setting them is

politically sensitive (Zwart-van Rijkom et al., 2000). Moreover, not using explicit thresholds can be

considered attractive by decision makers as it gives them room for other considerations (Eichler et al.,

2004). Nevertheless, the thresholds can be inferred from past allocation decisions. In U.K., for example, an

ICER of £20,000 - £30,000 per QALY (approximately between US$ 30,000 – 50,000) is typically used

(Devlin and Parkin, 2004, McCabe et al., 2008), while in the U.S., the threshold is US$50,000 –

US$100,000 per QALY. A justification for these thresholds can be found in Shiroiwa et al. (2010). In

practice, most decision makers in the U.S. agree that interventions that cost less than US$50,000 – 60,000

per QALY gained provide good value (Owens et al., 2011). A different method to evaluate whether an

intervention is or not cost-effective was presented by the WHO. They say that an intervention that costs

less than three times the per capita GDP per DALY is cost-effective. Additionally, an intervention is very-

cost effective when its cost is less than the per capita GDP per DALY.

As claimed by the WHO, the estimation of disease burden metrics through CEA can be useful for informing

decision makers about the overall impact of different health interventions (World Health Organization,

2009). Economic burden studies also require incorporating metrics on the effectiveness of interventions. In

recognition of the importance of this last point, the WHO called for a much coherent use of conceptual

foundations to guide economic impact studies. This aims to enhance consistency, comparability, and

coherence of economic impact among different health studies. Therefore, a more informed health policy

38

discussion can be held to improve people’s health and close existing disparities while accounting for cost-

effectiveness factors.

Assessing the benefits, harms, and costs of an intervention is important to truly understand whether it

provides a good value for the money (Owens et al., 2011). Consequently, systematic approaches to support

the resource allocation should be used rather than intuition of the health policy makers (Eichler et al., 2004).

According to Neumann et al. (2005), the use of CEA could help Medicare to spend its resources more

efficiently. Thus, CEA must be used as a comprehensive strategy to change the incentives at different levels.

Finally, CEA must not be used as strict guideline for resource allocation. There may be other ethical

considerations to implement interventions that do not achieve the typically used cost-effectiveness

threshold (Owens et al., 2011).

2.6.1 Impact of healthcare interventions and the use of QALY

Typically, most of the journal articles describe interventions with results that are statistically significant,

however, most of those articles fail to provide guidance and metrics to evaluate their potential health impact.

As argued by Fielding and Teutsch (2013), articles concerning clinical or population-health interventions

should provide structured information to quantify their potential impacts, and thus, justify their

implementation.

Although it must be recognized that investing on different types of interventions is important to fairly

distributing resources, there should also be considered that investments that lead to the greatest health

impact should have priority when competing for scarce resources. In order to examine the benefits and

harms of an interventions that is expected to be implemented in practice, Fielding and Teutsch (2013)

propose the use of quantitative factors; burden of disease, preventable burden, and economic value.

To account for these quantitative factors, the use of QALYs appears to be an attractive solution to normalize

the comparison among different interventions. The main principle behind QALY is to correct or calibrate

people’s expectancy of life based on their quality of life or health status. This measure has been widely

used to guide the allocation of resources in healthcare settings (Weinstein et al., 2009). The main purpose

or practical reason for using QALY is the assumption that decision makers in healthcare seek to maximize

health improvement across the population given limited resources to be allocated. An alternative metric

based on similar concepts is the disability-adjusted life years (DALYs). The DALY is a measure that

combines the time lost due to premature death and the time lived in a health state lower than the optimal

health state, also referred as disability (World Health Organization, 2013).

39

Challenges and need for coherent methods to compare interventions

Recently, the CTSA called for more coordination and collaboration within the CTSA hubs and CTSA

networks with the overall objective of improving the Nation’s capacity to address and eliminate health

disparities. According to their recent RFA (RFA-TR-14-0009), the vision of the program has to be

supported by an emphasis on three main themes: workforce diversity, track record in translational and

clinical research, and integration of healthcare and research. This last theme becomes very relevant to

ensure the synergies between the health care delivery system and the translational research enterprise.

Therefore, assuring the implementation of research advances into clinical care settings and prioritize

translational research interventions. In addition, the NCATS Advisory Council Working Group developed

some guidelines to guide strategies to strengthen the CTSA program. One of the specific strategic goals is

to train researchers about how to properly conduct studies that anticipate the impact of their proposals.

While a lot of effort has been put in demonstrating the statistically significance of health interventions,

there are still gaps in assessing the potential impact that those interventions can have on people’s health.

Most of the reported interventions have not quantified their impact on reducing health disparities, size of

the population to be targeted, potential to improve quality of life, potential to reduce the risk of a certain

disease, etc. In order to optimally allocate the resources to impact people’s health, metrics that estimate this

impact are needed. Calls for proposals should encourage a formal mechanism to estimate impact, and thus,

improve their ability to impact society.

Another key factor for encouraging health researchers to pay more attention to the estimation of the real

impact of their interventions is that it can result in more credibility, and therefore, more willingness from

funding agencies to provide funding to those initiatives. This is not something new, Winslow (1951)

collected and disseminated evidence about the potential economic impact associated to various health

interventions as a way to encourage and persuade governments to invest more money into public health.

Additional challenges might also be addressed to properly estimate the impact of a certain public health

intervention on “intangibles”. Future research should be oriented to developing frameworks that transform

those intangibles to metrics that can be measured and valued. As argued by Fielding and Teutsch (2013),

presenting this type of information can be somehow technically challenging, however, it is even more

challenging for researchers to find new ways to think about the value of their work. In order to support the

comparison of interventions based on their impact, coherent metrics such as QALYs should be used. This

standard metric would add coherence and consistency when comparing different interventions from a cost-

effectiveness perspective.

40

According to Weinstein et al. (2009) there are six principles and assumptions that underlay the conventional

QALY approach used in resource allocation decisions.

1. A resource allocation has to be made.

2. The consequences of the resources allocated might have an impact on health states and the

duration of those health states.

3. Resources are limited.

4. The main objective of the decision maker is to maximize the population’s health subject to

resource constraints.

5. Health is defined as a weighted function of value over a relevant time horizon

6. Value can be measured in terms of preferences and desirability

Although using metrics such as QALY and DALY do not solve all the challenges in the resource allocation

process, it certainty helps when comparing interventions based on a coherent baseline that accounts not

only for people’s health but also can be incorporated as cost-effectiveness metrics that inform the economic

evaluation of interventions.

2.6.2 Estimating QALY

As previously mentioned, QALY is one of the most used metrics to assess the impact of health

interventions. This metric has been used consistently for about four decades. Zeckhauser and Shepard

(1976) used the term QALY for the first time to propose a metric that combines duration and quality of life.

A few years later, Pliskin et al. (1980) demonstrated that QALY maximization based on the utility theory

is justifiable under two conditions; utility independence between health status and life of years, and risk

neutrality with respect to life of years. Technically, the QALY incorporates the different health related

quality of life status (HRQoL) that can range from 0, being dead, and 1 which represents a maximum or

perfect health status.

Currently, QALYs are used in most economic assessments conducted by agencies that encourage the cost-

effectiveness factors as fundamental component of their decision-making processes (Sassi, 2006).

Mathematically, the number of QALYs lived by a person can be expressed as follows:

𝑄𝐴𝐿𝑌 𝑖𝑛 𝑜𝑛𝑒 𝑦𝑒𝑎𝑟 = 1 ∗ 𝑄, 0 ≤ Q ≤ 1 Eq. 29

where Q is the health status or HRQoL weight lived during the year under calculation. Then, the expected

quality-adjusted life or quality-adjusted life expectancy (QALE) at a certain age a of disease is defined as:

41

𝑄𝐴𝐿𝐸 = ∑ 𝑄𝑡

𝑎+𝐿

𝑡=𝑎 Eq. 210

where L is the residual life expectancy of the individual at age a, and t is the number of years that the

individual is expected to be attached to the corresponding HRQoL. Typically, discounting factors are used

to calibrate the utility of QALY. In other words, translating future QALYs into a present value. Then, in

this case we can calculate a discounted QALE.

𝐷𝑖𝑠𝑐𝑜𝑢𝑛𝑡𝑒𝑑 𝑄𝐴𝐿𝐸 = ∑𝑄𝑡

(1 + 𝑟)𝑡−𝑎

𝑎+𝐿

𝑡=𝑎 Eq. 211

where r is the discount rate or normalization factor to evaluate health using present value. Typically a

discount rate between 3 – 5% is in line with the Global Burden of Disease (GBD) and practical guidelines

(Brouwer et al., 2000). In order to compare the impact of health interventions, pre-intervention QALY and

post-intervention QALY must be compared. This metric will give an estimate of the QALYs gained as a

result of the health intervention.

𝑄𝐴𝐿𝑌𝑠 gained = ∑𝑄𝑡

𝑖

(1 + 𝑟)𝑡−𝑎

𝑎+𝐿𝑖

𝑡=𝑎− ∑

𝑄𝑡

(1 + 𝑟)𝑡−𝑎

𝑎+𝐿

𝑡=𝑎 Eq. 212

where Qi is the vector related to the health status quality of life weights predicted after the health

intervention for each time step t.

Although these calculations appear to be easy, the big challenge is to accurately obtain an estimate of the

parameter Q. Most of the methods to estimate Q are based on general public’s opinion. Usually, individuals

are asked to imagine themselves in different health states and then to think about the trade-off of sacrificing

years of life or what risk in death (percentage) they would be willing to take in order to achieve a full health

state (Dolan, 2008). These methods are known as the time trade-off method (TTO) and the standard gamble

method (SG). Typically, these methods utilize two ways to obtain the data; patients and general public.

When asking patients, they can relate their values of health based on current experiences. On the other hand

when asking the general public, they should imagine themselves being in different health states. Out of

these two data sources, many health economists prefer the valuations from the general public. However,

some others have argued about the suitability of using preference-based methods of the general public

(Dolan, 2008).

42

2.6.3 Instruments to estimate QALY

In this section, a brief description of different instruments utilized to calculate “the Q of QALY” is given.

Typically, there are three main methods to estimate the Q or HRQoL; visual analogue scale (VAS), time

trade-off, and standard gamble (Bravo Vergel and Sculpher, 2008). From these methods, various health

valuation instruments have been proposed in the literature including the EQ-5D, SF-36, SF-12, SF-6D, and

WBQ, among others.

2.6.3.1 EQ-5D

In order to facilitate the calculation of QALYs, Williams (1995) developed the EuroQol instrument to

measure the HRQoL or Q. The purpose of this instrument is to provide a single index score for each health

state. The instrument relies on a questionnaire and a rating scale used as a “thermometer” of health state.

According to this, the three main contributions of EuroQol to measure HRQoL are:

1. Simple way to generate descriptive data

2. Simple way to utilize people’s perception to evaluate their own health status

3. Provide a preference-based generic index of health benefits

The EQ-5D instrument consists of a self-reported questionnaire that includes five main dimensions: 1)

mobility, 2) self-care, 3) usual activities, 4) pain/discomfort, and 5) anxiety/depression. The key question

asked to guide the respondent is “which statements best describe your own health state today?”, only one

of the levels (alternatives) is selected per dimension. In order to generate the value sets for the EQ-5D,

TTO and VAS are typically used. The VAS consists of a vertical diagram, typically 20-cms long, whose

scale ranges from 0 (worst imaginable health state) to 100 (best imaginable health state). In the VAS, the

respondent is asked to indicate how well or bad his/her own health is today. The EQ-5D survey is presented

in Table 2-3.

Under the EQ-5D model, the health state of an individual can be represented by a string of category levels.

In this sense, the value of the health state is a function of the categories already presented.

𝑄 = 𝑓(𝑀𝑖, 𝑆𝑖, 𝑈𝑖 , 𝑃𝑖 , 𝐴𝑖)

Where Mi, Si, Ui, Pi, and Ai represent the level (1,2, or 3) of the dimensions mobility, self-care, usual

activity, pain / discomfort, and anxiety / depression, respectively. As each one of the five categories contains

three levels, there are 243 different health states (35), in addition, two other health states are added; “dead”

and “unconscious”, giving a total of 245 health states. An individual’s health state can be represented as a

string containing the levels for the different dimensions. For example, an individual with a health status of

43

12132 indicates that the individual has no problems in walking about, has some problems with washing or

dressing him/herself, has no issue with performing usual activities, has extreme pain or discomfort, and is

moderately anxious or depressed.

Table 2-3. EQ-5D self-reported questionnaire

Dimension Statement Mark

Mobility I have no problems in walking about

I have some problems in walking about

I am confined to bed

Self-care I have no problems with self-care

I have some problems with washing or dressing myself

I am unable to wash or dress myself

Usual

activities

I have no problems with performing my usual activities (e.g.,

work, study, housework, family or leisure activities)

I have some problems with performing my usual activities

I am unable to perform my usual activities

Pain /

Discomfort

I have no pain or discomfort

I have moderate pain or discomfort

I have extreme pain or discomfort

Anxiety /

Depression

I am not anxious or depressed

I am moderately anxious or depressed

I am extremely anxious or depressed

According to Rabin and de Charro (2001), the EQ-5D instrument is being used in a variety of ways that

includes: monitoring health status of a patient over time, assessing seriousness of particular conditions in a

patient at different moments in time, providing evidence about medical effectiveness of drugs and

treatments, guiding resource allocation, and establishing health status baseline for populations of interest.

The EQ-5D framework has been used in a wide range of health issues such as rheumatoid arthritis (Hurst

et al., 1997, Marra et al., 2005), mental health (Lamers et al., 2006), diabetes (Clarke et al., 2002), and

Parkinson’s disease (Schrag et al., 2000), among others.

Valuation and models

In order to provide valuation for the different health states, a common practice is to select a set of health

states with a mixture of severity levels to be directly assessed or valued (Szende et al., 2007). The first

attempt to formalize health state values was proposed in the mid-1990s. Dolan et al. (1995) propose a model

to estimate the value of life based on different health states using the EQ-5D instrument. This model is

known as the social tariff (health state) TTO framework. As in other TTO approaches, the evaluator asks

44

the respondent to quantify the trade-off between sacrificing quantity of life (years) in order to improve their

quality of life. This EQ-5D tariff framework is based on empirical results obtained from a large survey

study in the U.K. in which a set of HRQoL valuations were obtained for 45 health states. The scores are

explained by 12 parameters including a constant associated with any distance from the full health status

(11111), two components within each dimension (10 components in total), and a term referred as N3 that

is an additional coefficient included when any dimension is at level 3. The coefficients for these 12

components are presented in Table 2-4.

Table 2-4. Coefficients for TTO tariffs (modified from Dolan et al., 1995)

Dimension (Component) Coefficient

Constant (a) -0.081

Mobility Level 2 -0.069

Level 3 -0.314

Self-care Level 2 -0.104

Level 3 -0.214

Usual activity Level 2 -0.036

Level 3 -0.094

Pain / Discomfort Level 2 -0.123

Level 3 -0.386

Anxiety / Depression Level 2 -0.071

Level 3 -0.236

N3 -0.269

Hence, the health state index of this model is given by:

𝑄 = 1 − 0.081 (if any deviation from full health) − 0.069𝑀2 − 0.314𝑀3 − 0.104𝑆2 − 0.214𝑆3

− 0. .036𝑈2 − 0.094𝑈3 − 0.123𝑃2 − 0.386𝑃3 − 0.071𝐴2 − 0.236𝐴3

− 0.269𝑁3 Eq. 213

The variables used are binary, in addition only one level can be 1 for each dimension. The value of health

for the example previously shown (12132) is 0.124 (1 – (0.081 + 0 + 0.069 + 0 + 0.386 + 0.071 + 0.269)).

As it can be seen from the model, the two dimensions in level 3 that impact the HRQoL the most are pain

/ discomfort and mobility. On the other hand, the dimension of usual activity presents the lowest impact for

level 2 (-0.036) and level 3 (-0.094). Although the Pearson coefficient (r2) for this valuation model was not

very high (0.46), given the type of data analyzed for the generation of the model, it is considered very good

(Dolan et al., 1995).

45

As previously mentioned, the model proposed by Dolan et al. (1995) is based on a valuation survey

conducted in U.K. including 3,395 respondents. Similar studies have been conducted around the world,

including the U.S. Shaw et al. (2005) uses a TTO approach to generate a model based on a sample of 4048

respondents using 45 health states. This model is called D1 valuation model and its coefficients are shown

in Table 2-5.

Table 2-5. EQ-5D - D1 Valuation model

Dimension (Component) Coefficient

Mobility Level 2 -0.146

Level 3 -0.558

Self-care Level 2 -0.175

Level 3 -0.471

Usual activity Level 2 -0.140

Level 3 -0.374

Pain / Discomfort Level 2 -0.173

Level 3 -0.537

Anxiety / Depression Level 2 -0.156

Level 3 -0.450

D1 +0.140

I2-square -0.011

I3 +0.122

I3-square +0.015

where:

D1 : ordinal variable that represents the number of movements away from full health state (11111)

beyond the first

I2 : ordinal variable that represents the number of dimensions at level 2 beyond the first

I3 : ordinal variable that represents the number of dimensions at level 3 beyond the first

Then the health state index is given by:

𝑄 = 1 − 0.146𝑀2 − 0.558𝑀3 − 0.175𝑆2 − 0.471𝑆3 − 0.140𝑈2 − 0.374𝑈3 − 0.173𝑃2 − 0.537𝑃3

− 0.156𝐴2 − 0.450𝐴3 + 0.140𝐷1 − 0.011𝐼22 + 0.122𝐼3 + 0.015𝐼3

2 Eq. 214

Returning to the previous example, in this case the individual with health state 12132 would have a value

of health of 0.401 (1- 0 – 0 – 0.175 – 0 – 0 – 0 – 0 – 0.537 – 0.156 – 0 + 2*0.140 – 0.011 + 0 + 0). This

model is considered to be fairly accurate as in only 7 health state valuations prediction (out of the 45 health

states included) the errors exceeded 0.05 in absolute magnitude. In terms of consistency, as in the previous

46

model, mobility and pain / discomfort are the most impactful dimensions in level 3. Similarly, the least

impactful dimension in this model is also usual activity for level 2 (-0.140) and level 3 (-0.374).

As argued by Shaw et al. (2005), the D1 valuation model is more appropriate for the U.S. population. A

comparison of the distributions of HRQoL obtained for the models proposed by (Dolan et al., 1995) for the

U.K. population and (Shaw et al., 2005) for the U.S. population is shown in Figure 2-4. Other studies from

different countries can be found in Szende et al. (2007).

Figure 2-4. Histogram of distribution of HRQoL ranges for US and UK

From the histogram of the HRQoL valuations for the U.S. and U.K. population-based models we can say

that for the same health states, the U.S.-based valuations are higher than the U.K.-based valuations. One of

the similarities of the models is that both of them present a bi-modal distribution.

As recommended by the Panel on Cost-Effectiveness in Health and Medicine (Weinstein et al., 1996, Gold,

1996), these approaches in which population-based preference weights are considered should be used to

inform decision makers in charge of allocating resources in the U.S.

2.6.3.2 SF-36

The 36-item short-form (SF-36) health survey is a self-reported measure of health status. The SF-36 was

developed as part of the Medical Outcomes Study (MOS) and formalized by the RAND Corporation (Ware

and Sherbourne, 1992). This instrument is defined as a “set of generic, coherent, and easily administered

quality-of-life-measures”. This form assesses eight health dimensions: 1) limitations in physical activities;

0

5

10

15

20

25

30

35

40

45

50

]-1 ; -0.5 ] ]-0.5 ; -0.4 ] ]-0.4 ; -0.3 ] ]-0.3 ; -0.2 ] ]-0.2 ; -0.1 ] ]-0.1 ; 0 ] ]0 ; 0.1 ] ]0.1 ; 0.2 ] ]0.2 ; 0.3 ] ]0.3 ; 0.4 ] ]0.4 ; 0.5 ] ]0.5 ; 0.6 ] ]0.6 ; 0.7 ] ]0.7 ; 0.8 ] ]0.8 ; 0.9 ] ]0.9 ; 1 ]

Fre

qu

ency

HRQoL

US UK

47

2) limitations in social activities; 3) limitations in usual role activities given physical problems; 4) bodily

pain; 5) general mental health; 6) limitation in usual role activities given emotional problems; 7) vitality;

and 8) general health perception. The SF-36 health states scales and the interpretation of its extreme scores

are presented in Table 2-6.

Table 2-6. SF-36 Health status and interpretation

Dimension

No of

items

No of

levels

Meaning of scores

Low High

Physical

functioning

10 21 Limited a lot in performing

physical activities including

bathing or dressing

Performs all types of physical

activities including the most

vigorous without limitations due to

health

Role limitation

– physical

4 5 Problems with work or other

daily activities as a result of

physical health

No problems with work or other


physical health, past 4 weeks

Social

functioning

2 9 Extreme and frequent

interference with normal

activities due to physical and

emotional problems

Performs normal social activities

without interference due to

physical or emotional problems,

past 4 weeks

Bodily pain 2 11 Very sever and extremely

limiting pain

No pain or limitations due to pain,

past 4 weeks

General mental

health

5 26 Feelings of nervousness and

depression all of the time

Feels peaceful, happy, and calm all

of the time, past 4 weeks

Role limitation

– emotional

3 4 Problems with work or other


emotional problems

No problems with work or other

daily activities due to emotional

problems, past 4 weeks

Vitality 4 21 Feels tire and worn our all of

the time

Feels full of pep and energy all of

the time, past 4 weeks

General health

perceptions

5 21 Believes personal health is

poor and likely to get worse

Believes personal health is

excellent

Several studies using this instrument have demonstrated its clinical validity in a wide range of diseases such

as gastrointestinal dysfunction (Bensoussan et al., 2001), asthma (Bousquet et al., 1994), HIV (Shahriar et

al., 2003, Hsiung et al., 2005, Riley et al., 2003), breast cancer (Mosconi et al., 2001, Bower et al., 2000),

migraine (Solomon et al., 1995, Osterhaus et al., 1994, Monzon and Lainez, 1998, Hung et al., 2013), and

digestive disorders (Hahn et al., 1999, Whitehead et al., 1996, Gralnek et al., 2000), among others.

The scores assigned depend on each dimension, for example the first category physical functioning

(limitation in physical activities) is composed of responses coded 1 (limited a lot), 2 (limited a little), and

3 (not limited at all). The physical functioning dimension is composed by 10 items or questions. These

items are shown in Table 2-7. In order to score the health index for this dimension, it should be noticed that

the scale of values ranges from 10 to 30. To transform this scale into a 0 to 100 scale, it can be said that

48

each point over 10 (baseline) in the direct SF-36 scale worth 5 points in a 100-points scale. For instance,

an individual that scores 3,3,2,1,2,3,3,3,2,1 for questions from 3 to 12 respectively, has a score of 23 for

the physical dimension. This is translated into a physical functioning score of 65 in a 100-points scale.

Table 2-7. SF-36 - Physical functioning dimension

Physical functioning item

Yes,

Limited

a lot

Yes,

Limited a

little

No, Not

limited at

all

[3] Vigorous activities, such as running, lifting heavy objects,

participating in strenuous sports

(1) (2) (3)

[4] Moderate activities, such as moving a table, pushing a

vacuum cleaner, bowling, or playing golf

(1) (2) (3)

[5] Lifting or carrying groceries (1) (2) (3)

[6] Climbing several flights of stairs (1) (2) (3)

[7] Climbing one flight of stairs (1) (2) (3)

[8] Bending, kneeling, or stooping (1) (2) (3)

[9] Walking more than a mile (1) (2) (3)

[10] Walking several blocks (1) (2) (3)

[11] Walking one block (1) (2) (3)

[12] Bathing or dressing yourself (1) (2) (3)

A complete version of the SF-36 Survey can be found in the Appendix G.

One of the main limitations that has been discussed in the literature is that SF-36 does not consider

preferences. It means that moving from one level to another is considered to be equally important (moving

from “limited a little” to “no limited” has the same weight than moving from “limited a lot” to “limited a

little”). Brazier et al. (2002) proposes a standard gambling approach to take into consideration preference-

based measures.

In order to make the SF-36 survey shorter, some other instruments have been proposed in the literature.

Instruments such as the SF-12 and SF-6D are two short versions of the SF-36 that have been proved to have

a very minimal loss of information. In addition to reducing the number of items included in the survey,

these versions have simplified wording and improved ambiguity. More details about these instruments are

presented in subsequent sections.

2.6.3.3 SF-12

The SF-36 instrument was modified to improve its brevity and psychometric performance. The form was

reduced to 12 items (SF-12) with very minimal information loss (Ware and Sherbourne, 1992). The SF-12

49

includes 8 dimensions; physical functioning, role limitations – physical, bodily pain, general health, vitality,

social functioning, role limitations – emotional, and mental health. The structure of the SF-12 is shown in

Table 2-8. A complete version of the SF-12 survey can be found in the Appendix H.

Table 2-8. SF-12 Health dimensions and summary of content

Dimension No.

Items

Summary of Content No.

response

choices

Range of responses

Physical

functioning

2 Moderate activities, climbing several

flights of stairs

3 “Yes limited a lot” to

“no limited at all”

Role limitations –

physical

4 Accomplishing less than wanted,

limitations in the kind of activities

2 Yes / No

Bodily pain 2 Effect of pain on normal work, both

inside and outside the home

6 “Not at all” to

“extremely”

General health 1 Personal evaluation of health 5 “Poor” to “excellent”

Vitality 1 Feeling full of life 6 “All of the time” to

“none of the time”

Social functioning 2 Extent to which physical health or

emotional problems interfere with

normal social activities

5 “All of the time” to


Role limitations –

emotional

2 Accomplishing less, not working as

carefully as usual

2 Yes / No

Mental health 2 Downhearted and blue, calm peaceful 6 “All of the time” to


One of the advantages of the SF-12 is that it requires only one third of the time for completion if compared

with the SF-36 (Lacson et al., 2010). Even though this short version uses only 12 out of the 36 items

included in the original version of the SF-36, studies have shown that the main component summaries

(physical and mental) of the SF-12 and SD-36 are correlated (Gandek et al., 1998, Lacson et al., 2010).


In order to provide a single index of HRQoL, Lundberg et al. (1999) proposes a regression model based on

the SF-12 instrument. The authors fitted the models for the general population using the VAS and TTO

methods. These models were able to explain 50% and 25% of the variance of the health state utilities

respectively. In addition to including the 12 items of the survey, the model also incorporates two

explanatory variables: age and gender. For the variable age, seven dummy variable categories were used.

The coefficients obtained from the regression analysis are shown in Table 2-9.

50

Table 2-9. SF-12 Regression coefficients

SD-12 Items VAS TTO

Constant 0.3473 0.5714

Age 20-29 years

30-39 years 0.0062 0.0056

40-49 years -0.0116 -0.0075

50-59 years -0.0175 -0.0117

60-69 years -0.0292 -0.0421

70-79 years -0.0551 -0.1410

80 - years -0.0835 -0.1920

Gender Women 0.0157 0.0014

Physical function [2a] Moderate activities Limited a little 0.0249 0..0051

Not limited at all 0.0821 0.0406

[2b] Climbing Limited a little 0.0223 0.0744

Not limited at all 0.0633 0.0974

Role functioning

- physical

[3a, 3b]Physical health

interfered with work

Did not accomplish less 0.0171 0.0218

Was not limited in the kind of work 0.0170 0.0032

Bodily pain [5] Pain interfered with

work

Quite a bit 0.0545 0.0237

Moderately 0.0949 0.0056

A little bit 0.1220 0.0291

Not at all 0.1390 0.0326

Vitality [6b] Energy A little of the time 0.0209 0.0252

Some of the time 0.0446 0.0417

A good bit of the time 0.0500 0.0362

Most of the time 0.0762 0.0357

All of the time 0.1010 0.0478

Social Function [7] Physical and emotional

health interfered with work



A little of the time 0.0832 0.0552

None of the time 0.0977 0.0805

Role function –

emotional

[4a, 4b] Emotional health

interfered with work

Did not accomplish less -0.0016 0.0129

Did work as carefully as usual 0.0021 -0.0075

Mental Health [6a] Calm and peaceful A little of the time 0.0089 0.0206

Some of the time -0.0056 -0.0019



All of the time 0.0317 0.0133

[6c] Downhearted and blue Most of the time -0.0282 0.0160



A little bit of the time 0.0559 0.0716

None of the time 0.0584 0.0711

51

2.6.3.4 SF-6D

Another reduced version of the SF-36 is the SF-6D which contains 6 main dimensions; physical

functioning, role limitations, social functioning, pain, mental health, and vitality (Brazier et al., 2002). Each

one of the dimensions (δ) has between two and six levels (λ). Under this model, a total of 18,000 health

states can be defined. The health dimensions and levels of the SF-6D are shown in Table 2-10.

In the SF-6D, the full health state is defined by 111111 and the worst health state possible is defined by

645655. For example an individual whose health state is 233322 means that the individual’s health limits

his/her a little in vigorous activities, accomplish less than he/she would like as a result of emotional

problems, individual’s health limits his/her social activities some of the time, have pain that interferes with

his/her normal work a little bit, he/she feels tense or downhearted and low a little of the time, and he/she

has a lot of energy most of the time.


Brazier et al. (2002) present a preference-based measure of health for the SF-6D applied to a representative

sample of the U.K. The valuation survey used a version of the standard gamble method. The models were

based on responses from 836 individuals that were asked to rank, and then value six different health states.

An orthogonal design procedure was used to generate 49 out of the 18,000 health states. The coefficients

found for these models are presented in Table 2-11.

This model incorporates two interaction effects; MOST and LEAST. The variable MOST assumes a value

of 1 if any of the five dimensions is at the most severe level. The variable LEAST assumes a value of 1 if

any of the five dimensions is at the least severe level. The best SF-6D mean model achieved an adjusted R2

of 0.58 which makes it very relevant for its use in cost-utility analysis (Brazier et al., 2002). Under the

different four models provided, for example, the value health for an individual with health state 233322

would be 0.223 (RE), 0.272 (Mean), 0.341 (RE-constant forced to unity), and 0.366 (Mean-constant forced

to unity). Out of these models, the Mean-constant forced to unity model is the one that is recommended to

be used in cost-utility analyses (Brazier et al., 2002).

52

Table 2-10. SF-6D Health dimensions and levels

Level Physical functioning Role limitation Social functioning Pain Mental health Vitality

1 Your health does not

limit you in vigorous

activities

You have no problem with

your work or other activities

as a result of your physical

health or any emotional

problem

Your health limits

your social activities

none of the time

You have no pain You feel tense or

downhearted and

low none of the time

You have a lot of

energy all of the time

2 Your health limits you a

little in vigorous

activities

You are limited in the kind

of work or other activities as

a result of your physical

health

Your health limits


a little of the time

You have pain but it

does not interfere

with your normal

work

You feel tense or

downhearted and

low a little of the

time

You have a lot of

energy most of the

time


little in moderate

activities

You accomplish less than

you would like as a result of

emotional problems

Your health limits


some of the time

You have pain that

interferes with your

normal work a little

bit

You feel tense or

downhearted and

low some of the

time

You have a lot of

energy some of the

time


lot in moderate

activities

You are limited in the kind

of work or other activities as

a result of your physical

health and accomplish less

than you would like as a

result of emotional

problems

Your health limits


most of the time

You have pain that


normal work

moderately

You feel tense or

downhearted and

low most of the time

You have a lot of

energy little of the

time


little in bathing and

dressing

Your health limits


all of the time

You have pain that


normal work quite a

bit

You feel tense or

downhearted and

low all of the time

You have a lot of

energy none of the

time


lot in bathing and

dressing

You have pain that


normal work

extremely

53

Table 2-11. SF-6D Models with interaction effects

Coefficients RE Mean

Constant forced to

unity

RE Mean

c 0.799 0.788 1 1

PF2 -0.023 -0.015 -0.05 -0.053

PF3 -0.021 0.011 -0.038 -0.011

PF4 -0.054 -0.018 -0.069 -0.04

PF5 -0.035 -0.034 -0.046 -0.054

PF6 -0.119 -0.084 -0.145 -0.111

RL2 -0.03 -0.021 -0.051 -0.053

RL3 -0.042 -0.03 -0.058 -0.055

RL4 -0.041 -0.024 -0.063 -0.05

SF2 -0.03 -0.023 -0.054 -0.055

SF3 -0.012 -0.04 -0.032 -0.067

SF4 -0.025 -0.042 -0.044 -0.07

SF5 -0.071 -0.058 -0.096 -0.087

PAIN2 -0.005 0.005 -0.037 -0.047

PAIN3 -0.013 0.004 -0.034 -0.025

PAIN4 -0.02 -0.025 -0.04 -0.056

PAIN5 -0.055 -0.049 -0.081 -0.091

PAIN6 -0.141 -0.136 -0.167 -0.167

MH2 -0.022 -0.03 -0.036 -0.049

MH3 -0.028 -0.019 -0.045 -0.042

MH4 -0.085 -0.089 -0.099 -0.109

MH5 -0.098 -0.109 -0.115 -0.128

VIT2 -0.006 -0.044 -0.032 -0.086

VIT3 -0.002 -0.031 -0.019 -0.061

VIT4 -0.001 -0.019 -0.022 -0.054

VIT5 -0.054 -0.064 -0.073 -0.091

MOST -0.052 -0.041 -0.084 -0.07

LEAST 0.049 0.048

2.6.3.5 QWB-SA

Another approach to estimate the HRQoL is the Quality of Well Being Self-Administered (QWB-SA) scale.

This instrument was developed in 1970’s with the objective of measure HRQoL (Kaplan et al., 1976). The

QWB-SA includes three main sections; presence or absence of chronic symptoms, acute physical symptoms

and mental symptoms, and levels of functioning. The first section is composed of 19 questions including

blindness and hearing problems. The second section includes 25 acute physical problems and 14 mental

54

symptoms. Finally, the third section contains a scale for self-reported levels of functioning including

mobility, physical activity, and social activity. In order to obtain preference weights for the different

elements included in the QWB-SA form, a sample that included 239 females and 191 males with ages in

the range of 18 to 85 was used (Seiber et al., 2008). The lowest score obtained for a living individual in the

QWB-SA is 0.09. As argued by Kaplan et al. (1993), although the QWB-SA model has been used in several

studies, its length and difficulty in administration have limited its implementation.

2.6.3.6 Comparison between instruments

The instruments that were described in this section represent just a small group of the most used generic

instruments that have been proposed in the literature. The selection of one instrument over the others will

depend mostly on the objective of the evaluation (including the level of detail and accuracy required),

patient population, and types of treatments involved. Out of the various generic multi-attribute utility

instruments available, the EQ-5D appears to be the most used one (Richardson and Manca, 2004, Wisløff

et al., 2014). However, several researchers have proposed the use of more than one instrument in parallel

as a way to avoid biasing the results due to the use of a specific instrument to estimate the HRQoL.

Moock and Kohlmann (2008) provides a comparison among different preference-based quality-of-life

measures. In their study, EQ-5, 15D, HUI2, HUI3, SF-6D, and QWB-SA were included for comparison

using results from rehabilitation patients with musculoskeletal, cardiovascular, or psychosomatic disorders.

Other studies comparing different valuation methods can be found in the literature (Brazier et al., 2004,

Walters and Brazier, 2005, Marra et al., 2005, Longworth and Bryan, 2003, Sach et al., 2006, Petrou and

Hockley, 2005, Bharmal and Thomas, 2006, Lamers et al., 2006, van Stel and Buskens, 2006, Grieve et al.,

2009, Stavem et al., 2005, Barton et al., 2008, Hatoum et al., 2004, Tsuchiya et al., 2006, Mutebi et al.,

2011, Kontodimopoulos et al., 2012, Gaujoux-Viala et al., 2011).

2.6.3.7 Discussion and limitations

The use of metrics such as QALYs has been found to be helpful for estimating the impact of health

interventions. In order to properly approximate the real impact that an intervention can have in an individual

and society, the estimation of the Q or HRQoL becomes relevant. Generic and disease-specific instruments

have been proposed to help calculating the HRQoL of various health states. The main advantage of using

these instruments is that the same baseline is maintained for different interventions. Therefore, the

comparison among interventions is coherently facilitated. As argued in the literature, calculating HRQoL

can be used in several ways that include the monitoring of a patient’s health status over time, assessing the

seriousness of a particular health condition and its progression over time, providing evidence regarding the

55

impact or medical effectiveness of different treatments, establishing health status baseline for different

populations of interest, and guiding resource allocation, among others (Rabin and de Charro, 2001). This

last point is critical for achieving an optimal impact on the population’s health given the scarce resources.

Moreover, studies have estimated that in the U.S., the number of life years saved could be doubled if

resources were properly allocated to those interventions which are relatively more cost-effective. Then, it

becomes substantially relevant to consider the cost-effectiveness of interventions as key factors for health

policy making in resource allocation decisions (Jamison et al., 2006).

Limitations of QALY

Although the QALYs have been extensively used for guiding health policy, there are still some challenges

and limitations that must be considered. In this sense, QALYs have been criticized not only on technical,

but also ethical grounds (Prieto and Sacristán, 2003). While traditional health economists have proposed

the maximization of health gains using QALYs, others have argued that fairness and equity in the

distribution of public resources are also important (Schwappach, 2002).

One of the technical limitations that has been discussed with respect to the use of QALY is that it does not

consider distributive factors such as relative priority in terms of age of life expectancy. For example, one

QALY weight the same for a child versus an older person. In order to overcome this, studies have been

presented to weight priorities accordingly. However, according to Schwappach (2002), it is still moderate

evidence that the general population tends to favor young over elderly in healthcare resource allocation.

Others have preferred to use DALYs as they account for particular age ranges. Another limitation of the

traditional use of QALY is that it does not discriminate with respect to the ranges in which a certain

treatment can act. For example, for QALYs, it is the same to improve Q from 0.2 to 0.4 than from 0.7 to

0.9. However, there is evidence that support that the general public tends to give priority to worst health

conditions (Ryynänen et al., 1999). Schwappach (2002) also argues that the assumption of proportionality

between the duration of the health improvement and social value can be misleading. In this sense, according

to QALY a health intervention that can improve life by 20 years is numerically twice as beneficial as

improving life by 10 years assuming the same benefits in the quality of life.

Even though the discussion about the proper use of QALYs is still not fully under agreement, it provides a

pragmatic and necessary guidance to improve complex decisions related to resource allocation (Garrison

Jr, 2009). Moreover, it has been stressed that formal models are necessary for informing the resource

allocation processes. However, formal evaluations should not be used as final answers, but as inputs for

further deliberations and fair decision processes (Nord et al., 2009).

56

Chapter 3

QUANTIFYING COMPLEXITY IN TRANSLATIONAL

RESEARCH: AN INTEGRATED QUALITY FUNCTION

DEPLOYMENT – ANALYTIC HIERARCHY PROCESS

APPROACH

3.1 Introduction

The foundation of knowledge in healthcare and health-related sciences has been established through a

tremendous investment of financial support by governmental and private agencies. Yet, there is a large gap

between the knowledge discovery and routine practice. To address this gap, a science has emerged to

“systematically study how a specific set of activities and designated strategies are used to successfully

integrate an evidence-based public health intervention within specific settings” (CDC, 2007). Work in this

domain goes by many titles, including: translation research, translational research, knowledge translation,

knowledge exchange, technology transfer, implementation research, and dissemination and implementation

(D&I) research (Brownson et al., 2012).

The National Institutes of Health (NIH) explicitly made translational research a central priority on their

roadmap for medical research (Zerhouni, 2003). As a way to support and help the acceleration of

translational research, the NIH launched the Clinical and Translational Science Award (CTSA) program in

2006. In 2013, the CTSA Consortium was comprised of 62 medical research institutions across the nation

(CTSA, 2011). The National Center for Advancing Translational Sciences (NCATS), a part of the NIH,

requested a FY 2013 budget of $639 million to support the CTSA initiative (DHHS, 2013). The funding

opportunity announcement (RFA-TR-12-006) clearly states the mission:

“Under NCATS, the goal of the CTSA program remains focused on integrated academic homes for

the clinical and translational sciences that increase the quality, safety, efficiency and speed of

clinical and translational research, particularly for NIH supported research.”

57

Some of the earliest work in translational research can be traced back to over 30 years ago (Wolf, 1974). In

the early 2000s translational research became a more widely used term and studied concept. Since then, a

number of publications and programs have been initiated to better understand and evaluate the importance

of translational research and its impact on healthcare outcomes (Woolf, 2008, Nathan, 2002, Drolet and

Lorenzi, 2011, Zerhouni and Alving, 2006).

Although the CTSA hubs supported over 5,000 publications during 2010 in diverse domains across the

translational research spectrum, there is still a lack of agreement on how to measure their impact on

healthcare outcomes. Evaluation methodologies must be proposed to ensure that the efforts placed on

translating basic research to clinical practice are effective and, ultimately, will have an impact on people’s

health. This lack of evaluation and tracking of translational research stems from the fact that it takes too

long to move from basic research to clinical practice. According to Westfall et al. (2007) it takes an average

of 17 years for new discoveries to become regular clinical practice. In addition, just 14% of those new

discoveries enter day-to-day clinical practice.

Discrepancies in the meaning of translational research have led to the generation of various models and

definitions. Although translational research can be defined in several different ways, the majority agree that

it is important for improving health (Woolf, 2008). The most popular models proposed to understand the

continuum of translational research are based on “T” phases or “translational blocks”. Sung et al. (2003)

described translational process in two phases. The first phase (T1) includes the knowledge gained from

laboratory testing to the development of new diagnosis and treatment tools. The second phase (T2)

translates those clinical studies to clinical practice. Due to the unclear scope of T2, a three-phase model

was proposed by Westfall et al. (2007) and Dougherty and Conway (2008). The third translational block

(T3) proposed accounts for the process necessary to implement knowledge into practice. This phase is also

known as the practice-based research block. Despite this additional phase added, some researchers argued

that the model was still incomplete and that the implementation of knowledge was made mostly through

physicians’ eyes not including other key practitioners in the translation and implementation of new

discoveries (Woolf, 2008). A new phase (T4) was included in the model offered by Khoury et al. (2007),

the phase of moving from health practice to healthcare outcome impact. The ultimate success of the T4

phase would be the improvement of public health and decreased cost of interventions (Kon, 2008). A

diagram to compare the four major translational research models was prepared by Trochim et al. (2011)

and is presented in Figure 3-1.

Currently, most of the research investments are made in the T1 phase. According to Moses et al. (2005),

more than half of the NIH budget is spent in basic research. However, no study has been proposed to

58

demonstrate that this budget allocation results in a maximum impact on people’s health. Generating and

using evidence to wisely prioritize the use of resources can accelerate the translation of knowledge into

policy and practice (Glasgow et al., 2013). Therefore, evidence is needed to understand how to optimally

allocate resources to the different translational research phases to achieve better healthcare outcomes. In

order to maximize the impact made by the investments in T1, it is essential to provide an adequate

investment in T2 and beyond. Thus, even though some people regard translational research as strongly

associated only with the T1 phase, more effort is needed in the remaining phases to accelerate the movement

of new discoveries into practice (Woolf, 2008). The number of publications related to each translational

phase also denotes the disproportion of attention for the T2 phase and beyond. It is estimated that only 3%

of published researches are mainly focused on T2, T3, and T4 (Khoury et al., 2007). It creates imbalances

that could have negative consequences in health and economics if not properly considered (Woolf, 2007).

Consequently, it is imperative to understand that translational research is more than just discovering new

drugs or treatments. It has to be seen as a continuum from bench to bedside. This idea has been advanced

by many authors including Drolet and Lorenzi (2011), Khoury et al. (2007), Trochim et al. (2011) and

Woolf (2008).

Research to Practice Continuum

T1

Gene Discovery /

Health Application

T2

Health Application /

Evidence-based Guideline

T3

Guideline /

Health Practice

T4

Practice /

Health Impact

T1

Basic Biomedical Science /

Clinical Efficacy Knowledge

T2

Clinical Efficacy Knowledge /

Clinical Effectiveness Knowledge

T3

Clinical Effectiveness Knowledge /

Improved Health Care Quality and Value ad Population Health

T1

Bench /

Bedside

T2

Bedside /

Practice-Based Research

T3

Practice-based Research /

Practice

T1

Basic Biomedical Research /

Clinical Science and Knowledge

T2

Clinical Science and Knowledge /

Improved Health

Basic

Research

Clinical

Research

Research

Syntheses

Practice-Based

Research

Health

Impact

Khoury et al., 2007

Doughert & Conway, 2008

Westfall et al., 2007

Sung et al., 2000

Figure 3-1. Comparison among the four major translational research models (modified from Trochim et

al., 2011)

59

It is essential to start generating agreement in translational research and its evaluation. In order to address

the lack of agreement in the various “T” models proposed, a general framework based on a process marker

model was presented by Trochim et al. (2011). This methodology consists of a series of process markers or

operational steps which are defined as a set of observable and measurable points specific to the study along

the translational research process. A main advantage of the process marker methodology is that it can be

used either by itself or under the “T” phases models previously mentioned. Even though the process marker

model provides a very clear framework to understand translational research and its steps, there is still a lack

of research determining the impact or contribution of each process marker to the overall goal of improving

and accelerating translation from basic to clinical practice.

In this chapter, we present an integrated Quality Function Deployment (QFD) – Analytic Hierarchy Process

(AHP) methodology to assess and quantify complexity in translational research. The first part of this

methodology serves as an extension of the process marker model since it provides a way to quantify the

importance of each operational step to accelerate translational research. In addition, the proposed

methodology captures the dynamic impact of different drivers along the translation from bench to bedside.

Finally, valuable insights are obtained to generate guidelines to better allocate resources and efforts when

moving from new discoveries to health outcomes.

3.2 Methodology

In this section, we introduce and explain in detail the three-phase methodology proposed to quantify

complexity in translational research. In the first phase, the objective is to identify the markers for each one

of the phases of translational research as well as the technical requirements or drivers for those markers.

The results obtained from this phase are used to present an extended process marker model. The second

phase will use the AHP methodology to determine the absolute and relative importance of the markers for

each “T” phase with respect to their ability to produce an impact on translational research. Finally, in the

third phase, an HOQ model will be developed as a way to find correlations between the technical

requirements and also the importance of those technical requirements for each translational research phase.

60

The variables and notation that will be used in the methodology are as follows:

𝑀𝑖,𝑗: 𝑀𝑎𝑟𝑘𝑒𝑟 𝑗 𝑖𝑛 𝑝ℎ𝑎𝑠𝑒 𝑖,

𝑊𝑖,𝑗: 𝑟𝑒𝑙𝑎𝑡𝑖𝑣𝑒 𝑤𝑒𝑖𝑔ℎ𝑡 𝑓𝑜𝑟 𝑚𝑎𝑟𝑘𝑒𝑟 𝑗 𝑖𝑛 𝑝ℎ𝑎𝑠𝑒 𝑖

𝑅𝑘: 𝑡𝑒𝑐ℎ𝑛𝑖𝑐𝑎𝑙 𝑟𝑒𝑞𝑢𝑖𝑟𝑒𝑚𝑒𝑛𝑡 𝑘

𝑟𝑘,𝑙: 𝑐𝑜𝑟𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑡𝑒𝑐ℎ𝑛𝑖𝑐𝑎𝑙 𝑟𝑒𝑞𝑢𝑖𝑟𝑒𝑚𝑒𝑛𝑡 𝑘 𝑎𝑛𝑑 𝑡𝑒𝑐ℎ𝑛𝑖𝑐𝑎𝑙 𝑟𝑒𝑞𝑢𝑖𝑒𝑚𝑒𝑛𝑡 𝑙

𝑃𝑖,𝑘: 𝑎𝑏𝑠𝑜𝑙𝑢𝑡𝑒 𝑤𝑒𝑖𝑔ℎ𝑡 𝑜𝑓 𝑡𝑒𝑐ℎ𝑛𝑖𝑐𝑎𝑙 𝑟𝑒𝑞𝑢𝑖𝑟𝑒𝑚𝑒𝑛𝑡 𝑘 𝑖𝑛 𝑝ℎ𝑎𝑠𝑒 𝑖

𝑝𝑖,𝑘: 𝑟𝑒𝑙𝑎𝑡𝑖𝑣𝑒 𝑤𝑒𝑖𝑔ℎ𝑡 𝑜𝑓 𝑡𝑒𝑐ℎ𝑛𝑖𝑐𝑎𝑙 𝑟𝑒𝑞𝑢𝑖𝑟𝑒𝑚𝑒𝑛𝑡 𝑘 𝑖𝑛 𝑝ℎ𝑎𝑠𝑒 𝑖

𝑐𝑗,𝑘: 𝐼𝑚𝑝𝑎𝑐𝑡 𝑜𝑓 𝑡𝑒𝑐ℎ𝑛𝑖𝑐𝑎𝑙 𝑟𝑒𝑞𝑢𝑖𝑟𝑒𝑚𝑒𝑛𝑡 𝑘 𝑖𝑛 𝑚𝑎𝑟𝑘𝑒𝑟 𝑗

where i = 1 … I, representing the i-th “T” phase, j = 1 … J, representing the j-th marker, k = 1 … K and l

= 1 … K, representing the k-th and l-th technical requirement respectively.

3.2.1 Identification of markers and technical requirements

As stated previously, the main objective of this phase is to identify a list of markers for each phase in the

translational research process and the technical requirements affecting those markers or operational steps.

Brainstorming will be the primary tool to generate a list of process markers and technical requirements.

The recommended brainstorming group size is at least 5 and at most 10 participants (Osborn, 1963). This

group size is adequate to identify the most important elements from various participants’ opinion.

In order to reduce the risk of cognitive burden in the use of AHP, the maximum number of markers per “T”

phase and technical requirements should be limited to 9. This number is known as Miller’s law which

determines that 7 ± 2 elements is the upper limit of human cognitive capacity to process information and

make inferences in a reliable and accurate manner (Miller, 1956). Additionally, this number was validated

by Saaty and Ozdemir (2003) as a way to reduce inconsistencies in judgment. Therefore, valid inferences

can be made from the results obtained from the pairwise comparison.

In the case of having a list of markers or technical requirements exceeding 9 elements, Borda count

methodology could be used to determine the 9 most important elements. In the Borda count methodology,

each evaluator ranks a list of n elements on a descending order of importance. Then, n points are assigned

to the first element in the list, the second element receives n-1 points and so on. The last element in the list

receives 1 point. Finally, the points are summed up for each element and the 9 elements with most points

are selected to be part of the final list.

61

3.2.2 Determining marker weights for each translational research phase

In the second phase of the proposed methodology, marker weights are determined for each translational

research phase using the AHP approach. From the marker list (i.e., operational steps in the translational

research process) obtained in phase 1, the relative importance of each marker is calculated using a pairwise

comparison among them. In this phase, it is recommended to check for consistency to reduce judgment

inconsistencies and assure valid results.

3.2.2.1 Pairwise comparison matrix

A pairwise comparison matrix needs to be built to further calculate the weights of each marker. Each pair

of markers is compared in terms of its importance and contribution to the final goal of producing impact on

its corresponding translational research phase, and therefore, on people’s health.

The evaluator is asked in a scale from 1 (equally important) to 9 (extremely more important) to determine

the importance of one marker over the other. Table 3-1, contains an explanation of the intensity importance

used for the pairwise comparison between the markers.

Table 3-1. AHP scale definition

Intensity of

Importance Definition Explanation

1 Equal importance Two activities contribute equally to the objective

3 Moderate importance According to experience an activity is slightly more

important than other

5 Strong importance According to experience an activity is strongly more

important than other

7 Very strong or

demonstrated importance

According to experience an activity is favored very

strongly over the other,

9 Extreme importance Evidence shows that an activity is absolutely more


2,4,6,8 Intermediate values

62

From the results obtained, a pairwise comparison matrix A is constructed. Where a1,2 represents how much

more important is marker 1 with respect to marker 2 in terms of their impact on translation.

𝐴 =

[

1 𝑎1,2 𝑎1,3

𝑎2,1 1 𝑎2,3

𝑎3,1 𝑎3,2 1

… … 𝑎1,𝑀

… … 𝑎2,𝑀

… … …… … …… … …

𝑎𝑀,1 𝑎𝑀,2 …

. . . … …… . . . …… … 1 ]

This matrix is used to calculate the weights or relative importance of the markers, and to check the level of

consistency of the evaluator. These procedures were explained in section 2.2.3.

3.2.3 Building the house of quality

The last phase of the proposed methodology is to “build the house of quality”. The main goal in this section

is to use the HOQ tool to find the correlation among technical requirements, contribution of each technical

requirement and its impact on the markers, and finally, calculate the importance of the technical

requirements for each translational research phase. This section will be divided into three subsections:

identifying correlation; identifying the impact of technical requirements; and calculation of the importance

of the technical requirements.

3.2.3.1 Correlation between technical requirements

From phase 1 in our methodology, we have already identified a list of technical requirements. In this sub-

section, the objective is to quantify the correlation among those technical requirements. This information

is recorded in the “roof” of the HOQ. The evaluators have to decide whether two technical requirements

are strongly positively correlated, positively correlated, non-correlated, negatively correlated or strongly

negatively correlated. The procedure should be repeated for each pair of technical requirements. It is

recommended to use the numbers provided in Table 3-2 to complete the pairwise comparisons. For instance,

let’s assume that ‘Administrative Support’ and ‘Regulations and Standards’ were identified as two of the

technical requirements. If the evaluator believes that those drivers are strongly positively correlated, then

the correspondent cell should be filled with a ‘9’.

63

Table 3-2. Correlation intensity

Correlation Number Symbol

Strong Positive 9

Positive 3

Negative -1

Strong Negative -3

3.2.3.2 Relationship matrix between technical requirements and markers

In this sub-section the objective is to obtain the relationship between the technical requirements and the

markers, in other words, it will quantify the impact of the drivers on the markers. The evaluators respond

whether the relationship between each TR-Marker pair is Strong (9), Medium (3), Weak (1), or No

relationship (0). This information is recorded in the relationship matrix which represents the body of the

HOQ. The information generated in this step is then used to calculate the importance of the technical

requirements on each translational research phase.

3.2.3.3 Technical requirement weights

This sub-section shows how to calculate the relative importance of the technical requirements for each

translational research phase. The calculated weights are located at the bottom part of the HOQ model. After

obtaining the absolute weights (𝑃𝑖,𝑘) and relative weights (𝑝𝑖,𝑘), rankings for each technical requirement

can be easily obtained by arranging them in descending order with respect to their weights. The formulas

to calculate absolute and relative importance are the following:

𝑃𝑖,𝑘 = ∑𝑐𝑗,𝑘 ∗ 𝑊𝑖,𝑗 ∀𝑖 ∀𝑘

𝑀

𝑗=1

Eq. 31

𝑝𝑖,𝑘 =1

∑ 𝑃𝑖,𝑙𝐾𝑙=1

∑𝑐𝑗,𝑘 ∗ 𝑊𝑖,𝑗 ∀𝑖 ∀𝑘 Eq. 32

𝑀

𝑗=1

From this analysis, valuable insights can be obtained about the relative importance of the technical

requirements for each translational phase. This can serve as evidence-based guidelines for allocation of

64

resources and efforts. In other words, priorities in investments can be determined to achieve a faster impact

on health outcomes. An illustrative summary of the methodology proposed is shown in Figure 3-2.

Figure 3-2. QFD-AHP Methodology Diagram

3.3 Case Study: a Primary Care-based Weight Control Intervention

A case study to illustrate the usability of the proposed QFD-AHP methodology is shown in this section.

The case study used is based on a pilot randomized controlled trial conducted by Dr. Jennifer Kraschnewski

(JK), a physician in the Penn State College of Medicine, to evaluate the impact of a volunteer peer-led

intervention for weight control in primary care. The main objective of the research is to determine the short-

term efficacy of a primary care-based weight control intervention in which successful volunteer peers

deliver a group-based program.

In order to facilitate the usability of the proposed methodology, a completely automated excel-based

template was created to obtain and collect the results. For illustrative purposes only, a 3T model was applied

by a single evaluator (JK) and the preference results are based on her opinion. More instances will be

required in a future to avoid the bias, reduce inconsistencies, and provide robust results to make valid

inferences.

65

3.4 Results

3.4.1 Identification of process markers and technical requirements

The process markers and technical requirements identified for the obesity program are shown in Table 3-3

and Table 3-4 respectively. Most of the markers and technical requirements are generalizable and could be

applied consistently in various initiatives and translational research efforts. Avoiding the lack of generality

in the identification of process markers and technical requirements will serve, in a future, to compare

interventions across different fields and identify appropriate benchmarks.

Table 3-3. Markers for the obesity peer-led intervention

Phase Code Process Marker Description

T1

M1,1 Pilot proposal

M1,2 Pilot funded

M1,3 Study proposal

M1,4 Study proposal funded

M1,5 Lab intervention

M1,6 Result analysis

M1,7 Guidelines for clinical trial

T2

M2,1 Develop obesity program and select target

M2,2 Submit IRB

M2,3 Recruit volunteers

M2,4 Training volunteers

M2,5 Program implementation

M2,6 Measure efficacy in sample population

M2,7 Larger sample and validity

M2,8 Patenting program

M2,9 Publish results

T3

M3,1 Pressing for public health reform

M3,2 Implementing research

M3,3 Study dissemination

M3,4 Population study and measure effectiveness on different populations

M3,5 Dissemination and best practices included in health policy

M3,6 Healthcare outcomes

66

Table 3-4. Technical requirements for the obesity peer-led intervention

Code Name

R1 Collaboration networks

R2 Administrative support

R3 Funding availability

R4 Community engagement

R5 Information technology

R6 Regulation and standards

R7 Equipment availability

R8 Organizational leadership

R9 Multidisciplinary team capacity

3.4.2 Pairwise comparison matrices, consistency and weights

A pairwise comparison matrix was constructed to compare the markers for each translational research

phase. Matrices for T1, T2 and T3 are shown in Table 3-5, 3-6, and 3-7 respectively.

Table 3-5. T1 – Pairwise comparison matrix

T1 M1,1 M1,2 M1,3 M1,4 M1,5 M1,6 M1,7

M1,1 1 3 3 5 3 3 3

M1,2 1/3 1 1 5 3 3 3

M1,3 1/3 1 1 3 5 5 5

M1,4 1/5 1/5 1/3 1 3 3 3

M1,5 1/3 1/3 1/5 1/3 1 1 1

M1,6 1/3 1/3 1/5 1/3 1 1 1

M1,7 1/3 1/3 1/5 1/3 1 1 1

From Table 3-5, it can be said, for instance, that Pilot proposal (M1,1) is moderately more important than

Study proposal (M1,3) and that Lab interventions (M1,5) is equally important than Result analysis (M1,6.). A

similar procedure to compare the importance of one marker over the other can be made for the following

phases.

67


T2 M 2,1 M 2,2 M 2,3 M 2,4 M 2,5 M 2,6 M 2,7 M 2,8 M 2,9

M2,1 1 1 5 3 3 5 7 1 7

M2,2 1 1 5 3 3 5 7 1 7

M2,3 1/5 1/5 1 1/3 3 3 5 1 7

M2,4 1/3 1/3 3 1 1/3 3 3 1 5

M2,5 1/3 1/3 1/3 3 1 3 3 1 5

M2,6 1/5 1/5 1/3 1/3 1/3 1 1 1 5

M2,7 1/7 1/7 1/5 1/3 1/3 1 1 1/3 1

M2,8 1 1 1 1 1 1 3 1 5

M2,9 1/7 1/7 1/7 1/5 1/5 1/5 1 1/5 1


T3 M3,1 M3,2 M3,3 M3,4 M3,5 M3,6

M3,1 1 3 5 5 7 7

M3,2 1/3 1 3 3 5 5

M3,3 1/5 1/3 1 3 3 3

M3,4 1/5 1/3 1/3 1 1 1

M3,5 1/7 1/5 1/3 1 1 1

M3,6 1/7 1/5 1/3 1 1 1

From the consistency analysis, it can be said that all three matrices are consistent. Consistency ratio values

for each pairwise comparison matrix are lower than 0.1, and therefore, the consistency of the evaluator is

acceptable. These values are shown in Table 3-8.

68

Table 3-8. Consistency analysis values

Translational

Phase

T1 T2 T3

n 7 9 6

CI 0.111 0.139 0.039

RI 1.320 1.450 1.240

CR 0.084 0.096 0.032

After checking consistency, the weights for each marker were calculated and are shown in Table 3-9.

Table 3-9. Marker weights

T1 Weight T2 Weight T3 Weight

M1,1 0.314 M2,1 0.223 M3,1 0.454

M1,2 0.187 M2,2 0.223 M3,2 0.239

M1,3 0.218 M2,3 0.113 M3,3 0.135

M1,4 0.108 M2,4 0.103 M3,4 0.063

M1,5 0.057 M2,5 0.109 M3,5 0.054

M1,6 0.057 M2,6 0.055 M3,6 0.054

M1,7 0.057 M2,7 0.031

M2,8 0.120

M2,9 0.022

According to the weights obtained for T1, it can be said that for the evaluator, marker M1,1 Pilot proposal is

the operational step with the highest relative importance, having a weight of 0.314. For T2, according to

the evaluator’s opinion, M2,1 Develop obesity program & select target and M2,2 Submit IRB are the most

important operational steps with a weight of 0.223 each. Finally, M3,1 Pressing for public health reform, was

the marker with the highest relative importance with a weight of 0.454. From the relative weights obtained,

many inferences can be made about what are the critical operational steps in this obesity peer-led

intervention program. The allocation of efforts can be conducted in a manner that is consistent with the

data.

Figure 3-3 shows an extended process marker model in which the height of the bars represents the relative

importance of the markers. It must be noticed that the relative importance is graphically valid for markers

within its corresponding translational phase. Therefore, we are not trying to evaluate how important one

69

translational phase compared to the others but rather how important the different operational steps are

within each translational phase. However, if the relative importance of each phase over the others is

available (using AHP, for example), the weights could have been normalized to graphically reflect the

importance of the markers not only in terms of its own phase, but in terms of its impact on the overall

process of accelerating translational research from bench to bedside.

Figure 3-3. Extended process marker model

3.4.3 Correlation among TRs and relationship among TR-marker pairs

In Figure 3-4, the correlation between each pair of technical requirements is shown. It can be seen, for

instance, that according to the evaluator, R1 Collaboration networks is strongly correlated to R5 Information

technology. On the other hand, there is no correlation between R6 Regulations and standards and R7

Equipment availability. The relationship matrixes for T1, T2 and T3 are shown in Table 3-10, Table 3-11,

and Table 3-12 respectively.

70

R1 R2 R3 R4 R5 R6 R7 R8 R9

Co

llab

ora

tion n

etw

ork

s

Adm

inistra

tive s

up

port

Fu

ndin

g a

vaila

bility

Co

mm

un

ity e

ngag

em

en

t

Info

rmatio

n te

chn

olo

gy

Regu

latio

n a

nd

sta

nd

ards

Eq

uip

men

t availa

bility

Org

aniz

atio

nal le

ad

ersh

ip

Mu

ltidisc

iplin

ary te

am

cap

acity

Figure 3-4. Technical requirements correlation

Table 3-10. Relationship matrix for T1

Marker Weight R1 R

2 R

3 R

4 R

5 R

6 R

7 R

8 R

9

M1,1

0.314 3 1 3 1 1 1 1 1 3

M1,2

0.187 3 3 9 1 1 1 3 3 3

M1,3

0.218 3 1 1 1 1 1 3 3 3

M1,4

0.108 3 3 9 1 1 1 1 3 3

M1,5

0.057 3 1 3 1 1 3 9 3 3

M1,6

0.057 3 1 3 1 3 1 1 1 3

M1,7

0.057 3 1 3 1 1 1 1 3 3

71


Marker Weight R1 R

2 R

3 R

4 R

5 R

6 R

7 R

8 R

9

M2,1

0.223 3 1 3 1 1 1 1 3 3

M2,2

0.223 1 3 1 1 1 3 1 1 1

M2,3

0.113 3 1 3 3 1 1 1 3 3

M2,4

0.103 3 1 3 3 1 1 1 1 3

M2,5

0.109 3 3 9 3 1 3 3 3 3

M2,6

0.055 3 3 9 3 1 3 3 1 3

M2,7

0.031 3 3 9 9 1 3 3 3 3

M2,8

0.120 3 3 9 3 1 1 1 3 3

M2,9

0.022 3 3 3 1 1 1 1 3 3


Marker Weight R1 R

2 R

3 R

4 R

5 R

6 R

7 R

8 R

9

M3,1

0.454 3 1 3 3 1 1 1 3 9

M3,2

0.239 3 3 9 9 1 1 3 3 9

M3,3

0.135 9 3 9 9 1 1 1 3 9

M3,4

0.063 9 3 9 3 1 3 3 3 9

M3,5

0.054 9 3 9 9 1 1 1 3 9

M3,6

0.054 3 1 3 3 1 1 1 3 3

According to the evaluator, the relationship between M1,5 Lab intervention and R7 Equipment availability

is strong (9), while the relationship between M1,5 Lab intervention and R2 Administrative support is weak

(1). From the relationship matrix, valuable inferences can be made about the most and least important

technical requirements required dynamically by each marker on each translational phase. This will also

allow understanding that translational research is dynamic and its needs change over time. The impact of

the technical requirements on the markers is not static; it will vary dynamically to fulfill current needs. For

72

instance, according to the evaluator’s opinion, R7 Equipment availability is strongly related to M1,5 Lab

intervention. On the other hand, it is weakly related with M1,1 Pilot proposal. This type of information

indicates that coordination of technical requirements is highly necessary to help the acceleration of

knowledge translation.

3.4.4 Determining the importance of each technical requirement in translational research

With the information previously collected, the importance or impact of the technical requirements on each

one of the phases in translational research can be quantified. The relative weights obtained are shown in

Table 3-13.

Table 3-13. Technical requirements relative weights

R1 R

2 R

3 R

4 R

5 R

6 R

7 R

8 R

9

Phase

T1 0.15 0.08 0.22 0.05 0.06 0.06 0.12 0.11 0.15

T2 0.13 0.10 0.22 0.11 0.05 0.09 0.07 0.11 0.13

T3 0.14 0.06 0.18 0.17 0.03 0.03 0.05 0.09 0.26

As expected, R3 Funding availability was found to be one of the most important technical requirements for

the markers to succeed on each T phase. According to the evaluator, R1 Collaboration network and R9

Multidisciplinary team capacity are crucial for T1 and T2. For T3, it is also important to consider R4

Community engagement to accelerate the translation of new discoveries to practice. The relative importance

or impact of the technical requirements on each “T” phase is shown in Figure 3-5.

From these results, useful guidelines of what and when technical requirements are critical for each phase

can be easily obtained. Thus, resources can be spent wisely throughout the large-scale, complex, and

dynamic translational research process.

73

A completed version of the HOQ developed is shown in Appendix A.

Figure 3-5. Relative importance for technical requirements on each "T" phase

3.5 Discussion

Mapping and evaluating translational research is essential to accelerate the implementation of new

discoveries into practice. Current translational research models provide little to no clues on how to evaluate

and quantify complexity of the long process of moving from basic science to health outcomes. Trochim et

al. (2011) address the discrepancies in the meaning of translational research by providing a process marker

model to map its continuum nature. Their framework aims to identify a set of clearly definable and

measurable steps. An advantage of the process marker model is that it can be either used independently –

i.e., just using operational steps – or under the “T” phases models used in translational research. Although,

their framework helps identifying the different operational steps along the translational research process,

there exists a need for a robust methodology to quantify this complexity and evaluate the importance or

impact of each one of those operational steps on the acceleration of discoveries to practice. In the first part

of the proposed methodology, a robust framework is presented to map translational research and determine

the relative importance of the operational steps within the process. Therefore, this approach can be seen as

an extended process marker model.

Additionally, the proposed QFD-AHP methodology includes many features that have not been previously

explored in translational research. For example, it allows for quantifying both the tangible and intangible

in a structured way. By understanding the importance and impact of each marker and the drivers that make

them succeed, resource allocation can be conducted in a smart manner based on evidence. Thus, a more

accurate strategic guideline to spend funds, effort, time and resources in the different phases of translational

0

0.05

0.1

0.15

0.2

0.25

0.3

R1 R2 R3 R4 R5 R6 R7 R8 R9

T1

T2

T3

74

research can be generated. As a consequence, we anticipate that limited resources could be wisely used to

support an accelerated, but smoother, journey from basic research to improving public health.

Although the main objective of the proposed methodology is to evaluate and determine the importance of

the operational steps and their drivers in translational research, some other useful features are worthy of

mention. One example is the possibility of identifying similar projects based on the values obtained for the

HOQ model. Usually, similar operational steps and technical requirements are present in various types of

researches and fields. Further analysis could be conducted to identify similar projects based on similarity

metrics. This approach could facilitate the sharing of best practices for similar studies, both between and

within a field. Additionally, collaboration of intra and cross-disciplinary studies could be promoted based

on evidence. Another advantage that can arise from identifying similar projects based on the QFD-AHP

methodology is the generation of benchmarks for each operational step. Although we did not provide

specific metrics to evaluate markers, similar best practices of procedures could be shared and adapted from

different projects and compared against the identified benchmarks.

To date, most of the engineering-based tools that have been applied successfully to solve healthcare-related

problems have been framed under an operational or tactical vision. Naturally, since translating research

from bench to bedside takes a very long time, operational and tactical tools could not be used to understand

and cover this scope under a systemic view. The QFD-AHP approach will allow understanding and

quantifying complexity in the large-scale, dynamic translational research process from a more strategic

perspective. Finally, future research is needed to incorporate more case studies that allow the use of this

framework in practice, and therefore, take advantage of all the potential benefits and applications already

mentioned.

3.6 Conclusion

The proposed QFD-AHP methodology can effectively quantify the complexity in translational research.

This well-structured methodology is robust and can be generalizable for various translational research

programs. From the results obtained, the dynamic impact of technical requirements on the process markers

can be identified. Additionally, it provides insights to coordinate those technical requirements to fulfill the

needs at different “T” phases. The evidence obtained from the QFD-AHP methodology could be used to

generate guidelines to assure that the proven best-known clinical procedures are transferred into health

policy and impacting health outcomes.

75

In summary, the proposed methodology could contribute significantly to generate agreement on the

important markers and technical requirements to be included when mapping and quantifying complexity in

translational research, allocate resources wisely, identify benchmarks within and between disciplines,

identify similar research projects to promote collaboration and share best practices. All the potential benefits

mentioned will aim to move from basic research to day-to-day clinical practice in a more effective and

efficient manner.

76

Chapter 4

EVALUATING COLLABORATION AND MULTI-

DISCIPLINARITY AND THEIR IMPACT ON

TRANSLATIONAL RESEARCH

4.1 Introduction

Collaboration is a key element in translational research (Marincola, 2003, Long et al., 2012, Bennett et al.,

2010). The National Institutes of Health (NIH) has explicitly made translational research a central priority

on their roadmap for medical research (Zerhouni, 2003). One of the initiatives to support team science and

collaboration efforts is the Clinical and Translational Sciences Award (CTSA) with a strategic goal to

enhance collaboration across the consortium. In order to achieve this goal, strong intra-institutional

collaboration networks are required and must be properly supported by the institutions.

Translational research networks aim to provide conditions under which research can be efficiently initiated

and spread. In addition, these networks can strategically bridge the gap between basic research and clinical

practice through interdisciplinary collaboration (Long et al., 2012). Further, given that disease etiologies

are multifactorial, collaboration across multiple disciplines is necessary to integrate medical research into

primary care (Carey et al., 2005, Calhoun et al., 2012). As a consequence, the study of multidisciplinary

teams has become increasingly investigated in the past years (Calhoun et al., 2012). The key hypothesis

behind this argument is that valuable synergies can be obtained by merging the expertise gained from

various fields (Choi and Pak, 2006, Knoben and Oerlemans, 2006). In consideration of these facts, the

CTSA consortium is encouraging its affiliated academic and medical institutions to adopt frameworks for

assessing collaboration.

77

Although understanding collaboration networks is critical for developing institutional structures that

support and enhance collaboration, its complexity has resulted in a lack of truly meaningful metrics for its

evaluation. In this Chapter, Social Network Analysis (SNA) principles are used to visualize collaboration

networks and provide different mathematical-based metrics. These metrics may be used to strategically

identify areas of opportunity in the network to accelerate collaboration, and therefore, impact the pace at

which new discoveries become clinical practice.

A profile research networking platform was initially developed at Harvard to speed the process of finding

potential collaborators with specific areas of expertise. Currently, most of the affiliated institutions have

implemented profile platforms that contain relevant information about their associated researchers. These

tools are expected to serve as drivers to accelerate the formation of collaborative teams, and ultimately,

accelerate translational research (Gewin, 2010).

In this chapter, a SNA-based methodology is presented for assessing collaboration. In order to illustrate its

use, a case study on obesity research is presented.

4.2 Methodology

This section proposes a methodology to understand, assess, and discovery interesting patterns among

collaborators in a health field or health area of research. The proposed methodology includes four phases:

1) Identification of researchers in the field; 2) Classification of Expertise; 3) Publication and Authorship

metrics; and 4) SNA to understand inter-disciplinary collaboration based on visualization of networks and

analysis of interesting, meaningful metrics. An overview of the proposed methodology is illustrated in

Figure 4-1.

The methodology starts with the category named “Identification of Researchers”. It includes the selection

of the health field in which collaboration should be assessed. This search could also include a multi-field

searching in case of interest in an emerging multi-disciplinary health field. Keywords related to the health

field under analysis must be gathered to guide the identification of researchers. This identification could be

conducted directly through the institutional profile directory. Currently, most of the CTSA hubs have

implemented profile tools to enhance collaboration among their affiliated researchers. Typically, those

searching platforms contain directory information such as affiliation, department, academic rank, and areas

of interest.

78

Identification of

Researchers

Selection of Health

Field

Keywords related to the

Health Field

Identify Researchers

through Institution

Profile Directory

Search Engine

Search keywords in a

Search Engine

Refine List of

Researchers

Retrieve publications

database and clean data

Expertise Classification

Categorize researchers

according to expertise

Data Preparation

Create collaboration

matrix ad database

structure

Social Network Analysis

Intra-institutional

collaboration networks

Interesting Metrics and

Clustering

Selection of SNA

software

Figure 4-1. Overview of methodology to assess intra-institutional collaboration

The second phase is named “Search Engine”. Its main objective is to refine the list of researchers and

retrieve their publications to further bibliometric analysis. Various search engines such as PubMed or the

Web of Sciences can be used to easily conduct the advanced searching and retrieve publication information.

A database is generated for the identified researchers and their publications related to the health field under

analysis.

Based on the information retrieved, researchers can be classified into groups according to their expertise.

This will be helpful to analyze collaboration patterns among different types of researchers. In addition, it

will serve to generate an overall understanding of potential leaders in the field. Due to the difficulty of

defining expertise, some broad assumptions or rules are needed. These rules could incorporate the number

of publications, average citations, date of the first publication in the field, etc. In parallel, a collaboration

matrix must be created to provide a structured database of co-authorships in an adjacency matrix format.

This is a symmetric matrix that contains pairwise publications based on co-authorship.

Finally, a SNA is conducted to identify collaboration patterns and visualize collaboration networks.

Currently, various SNA software are available, most of them free such as Pajek and NodeXL. Such software

will graph the collaboration networks, identify different collaboration patterns, and calculate interesting

metrics such as degree centrality, betweenness centrality, and density, among others.

4.3 Case Study: Collaboration in Obesity Research

Obesity is a major health issue in the United States. Its multi-factorial character makes multi-disciplinary

collaboration a relevant driver to truly understand the causes and potential interventions to support obesity

79

research. In this section, an illustration of the proposed approach is presented to assess intra-institutional

collaboration networks in obesity research.

4.3.1 Identification of obesity researchers

The first step was to identify researchers whose main area of research is obesity or related areas. The initial

list of obesity researchers was obtained through the Penn State Profiles webpage, available on the PSU

Clinical and Translational Sciences Institute (CTSI) website. Typical directory information such as

affiliation, department and academic rank were obtained in addition to a list of collaborators and

recommended similar researchers based on areas of interests, affiliations, and co-authorships. Since the

Penn State Profiles application was recently launched, when it was acceded, in October (2013), it only

contained the faculty members from the Penn State Hershey campus. However, faculty members as well as

select staff scientists from other campuses will be added soon to the system. The publications database used

by the Penn State Profile is updated monthly directly from PubMed.

After obtaining the initial list of obesity researchers, a more refined list was elaborated contrasting the

results against a list referred from the Web of Knowledge (WoK) search platform. The search conducted

included keyword in obesity fields and was restricted to the participation of at least one author affiliated to

PSU with no date restriction. Finally, a list of publications was retrieved, cleaned, and saved as the starting

database for further analysis.

4.3.2 Classification of expertise

The refined list of obesity researchers was used to categorize them into three groups based on their level of

expertise, defined as knowledge and skills (Todd, 2012). Due to the difficulty and lack of widely accepted

techniques to measure level of expertise, three main factors are proposed to categorize the researchers:

number of publications, average number of citations per publication, and year of first publication related to

obesity. The classification criteria mentioned are shown in Table 4-1.

Table 4-1. Expertise classification criteria

Group #

Publications

Average

citations

Year of 1st

publication

Expert >= 10 >=10 < 2005

80

Semi-Expert >= 5 >=5 < 2010

Young

Investigators Otherwise

The proposed classification is valuable to gain understanding and assess if different collaboration patterns

are obtained according to the level of expertise of the researchers. After classifying the authors according

to their level of expertise, different basic metrics were obtained to assess collaboration. Metrics such as co-

authorship, citation trends, frequently-used journals to disseminate, and average number of citations per

publication were calculated.

4.3.3 Social network analysis for obesity researchers

Bibliometric - a set of quantitative methods to analyze academic literature - was used to generate a database

that contains relevant information about the authors including affiliations, department, academic rank, and

publications. The data entries were obtained from multiple sources such as Penn State Profile, WoK and

Google Scholar. This database was cleaned, checked for integrity and analyzed using Microsoft Excel

2010® to create the collaboration matrixes based on co-authorship. NodeXL – a SNA software based on

Excel - was used to generate the collaboration network structure for individual researchers as well as for

affiliations, ranks, and interesting clusters.

Networks were graphed and analyzed to understand the dynamics of the collaboration patterns between

researchers and also among department affiliations. As usual in SNA, the vertexes represent the entities

(i.e., researchers or departments) under analysis and the edges represent the connection between two

different entities. In general, the algorithm used to generate the network graph was based on the Harel-

Koren Fast Multiscale algorithm (Harel and Koren, 2001). Cluster algorithms were also used to facilitate

the interpretation of the networks and to make valuable inferences from them. In most of the graphs

presented, the size of the vertexes was drawn proportionally to the number of publications. Consequently,

the width of the edges represents the intensity of the collaboration between two entities. Interesting metrics

such as betweenness centrality, in-degree centrality, and density are calculated and interpreted once

pertinent.

4.3.4 Cross-institutional collaboration

81

The group of the obesity researchers categorized as experts in the field was used to investigate their cross-

institutional collaboration networks in more detail. A new database was created to contain the new list of

obesity experts and their external collaborators including intra-institutional and cross-institutional

individuals. The database created was then transformed into collaboration matrixes for their evaluation.

Different interesting metrics were calculated and network graphs were developed to understand patterns

among cross-institutional collaboration using NodeXL.

4.4 Results

A list of 152 obesity researchers was retrieved from Penn State Profiles after querying key terms related to

obesity. The list of 152 obesity researchers was refined based on results obtained from the WoK. Finally, a

list of 114 researchers was generated, out of which 44 of them have published more than five articles. Figure

4-2, shows the researcher identification code, number of publications, and average number of citations per

publication for those obesity researchers with more than 5 publications related to obesity. According to the

search conducted, 779 papers were published by the 114 authors in obesity related areas. Those have been

cited a total of 28,412 times, with an average number of citations per item of 36.5, and an h-index of 91 (ie,

91 of the articles have been cited 91 times). Based on the stated categorization rules for expertise, 12

researchers were defined as experts, 22 as semi-experts and 80 as new investigators.

Figure 4-2. Number of publications and average number of citations per publication

The articles authored by the obesity experts group have been cited an average of 61 times, those authored

by obesity semi-experts have been cited an average of 28 times and those authored by obesity new-

investigators are on average cited 16 times.

82

Out of the 779 articles analyzed, 264 were published in the 15 most popular journals shown in Table 4-2,

which additionally defines the journal’s impact factor and total number and percent of manuscripts

published of the 779 articles.

Table 4-2. 15 Most frequently-used journals

Journal Title IF Count %

Am J Clin Nutr 6.5 35 13.3%

J Am Diet Assoc 3.8 33 12.5%

FASEB J 5.7 20 7.6%

Int J Obesity 5.2 20 7.6%

Obesity 3.9 18 6.8%

Appetite 2.5 17 6.4%

J Clin Endocrinol Metab 6.4 17 6.4%

J Nutr 4.2 17 6.4%

Sleep 5.1 17 6.4%

Physiol Behav 3.2 15 5.7%

Am J Physiol Endocrinol Metab 4.5 14 5.3%

Diabetes 7.9 11 4.2%

Obes Res 3.9 11 4.2%

Pediatrics 5.1 11 4.2%

Obes Surg 3.1 8 3.0%

4.4.1 Intra-institutional collaboration networks

The general collaboration network for obesity researchers was composed of 114 vertexes and 170 edges

representing the pairwise collaboration between the authors. An initial network graph was generated using

the Harel-Koren Fast Multiscale algorithm (shown in Figure 4-3).

The publication volume (i.e., number of publications) can be visualized by the size of the vertex and the

intensity of collaboration is illustrated by the width of the edges between two researchers. For illustration

purposes and to decrease the highly dense network obtained initially, Figure 4-4, shows only the edges or

connection among authors which have at least two co-authored publications. In addition, the code names

of the researchers categorized as experts are shown. This network graph could serve to visualize continuous

83

or more stable collaboration between two researchers and also the natural clusters or grouping patterns led

by the obesity experts.

Figure 4-3. General collaboration network

Figure 4-4. Collaboration network with at least two publications between researchers

In Figure 4-4, the different vertex shapes and colors represent the expertise of the researchers (experts:

black circle, semi-experts: blue diamond, and young investigators: green triangle). Clear patterns and

clusters among different levels of expertise can be easily determined from the visualization. A more in-

84

depth study of this network graph would be valuable to identify structural collaboration holes and discover

opportunities to accelerate knowledge flow and collaboration. For the experts group only, three

collaboration sub-clusters are encountered. See Figure 4-5.

Figure 4-5. Expert sub-clusters collaboration networks

On average, the experts group has a betweenness centrality of 194. On the other hand, the average

betweenness centrality for the semi-experts cluster and the new-investigators cluster is 41 and 34

respectively. This indicates that experts are more reachable for the other types of researchers and in most

cases serve as the main connection points for the network. For example, the expert researcher with the most

connections is directly linked to 13 researchers, including 5 experts. If we consider only the cluster of

experts, a density graph of 0.242 is achieved; that is, 24.2% of the total possible connections among each

pair of researchers is achieved in the network. In other words, 16 connections out of 66 maximum

connections are present in the experts’ network. The density for the expert group is higher than the

calculated densities for the semi-experts group (0.076) and new-investigators group (0.039). Naturally,

graph density can imply the connectedness of leaders and if proper collaboration networks are set at

different expertise groups to support mentoring.

The maximum degree of separation between two experts is two on each sub-cluster. In other words, the

separation among two experts on each sub-cluster is at most 1 intermediate researcher. This finding

reaffirms that the collaboration network structure among the experts group can be considered to be strong

and that experts are reachable to each other within their sub-group. However, it must be also be noted that

the overall experts collaboration network is disconnected, meaning that not all the researchers are directly

or indirectly connected to any other expert. Usually, this indicates that different sub-fields in obesity are

led by different groups of experts. For instance, the expert cluster composed of E1, E2 and E8 is mostly

associated with nutritional sciences. The cluster composed of E6, E7, and E9 has a main focus on cellular

85

and molecular physiology. The remaining cluster is mostly integrated by members associated with the field

of psychiatry.

Additionally, the metrics generated from the network graphs can be used to identify bridgers and central

researchers. For example, in the network there are 18 researchers with relatively high betweenness

centrality. This group was composed of 7 experts, 3 semi-experts, and 8 new investigators. The members

of this group serve as facilitators among different clusters or groups in the network. On the other hand,

leaders or central researchers to the overall network were identified by their in-degree centrality. Within

the network, there were 9 researchers with more than 7 connections. This group was composed of 4 experts,

2 semi-experts, and 4 new investigators. This type of insight is relevant for decision makers in charge of

the design of strategies to better disseminate, identify mentors, and provide optimal collaboration structures.

4.4.2 Interdisciplinary collaboration by affiliation

Interdisciplinarity at the researcher level

Interesting patterns of multidisciplinary collaboration are found within the obesity research network. When

considering all the publications at the researcher level, 201 connections or co-authorships occur within the

same health discipline (department) and 234 connections occur between researchers from different health

discipline. This gives an overall cross-inter discipline ratio of 1.164, which indicates that the within and

between disciplines collaboration is slightly favored to multidisciplinary collaboration. The cross-inter

discipline ratio for the semi-experts group (2.286) is higher than the experts (0.993) and young-investigators

groups (1.433). This might indicate that semi-experts tend to be more develop more multidisciplinary

collaborations. On the other hand, expert researchers have a collaboration pattern that slightly favored the

collaboration within their own disciplines.

Interdisciplinarity at the departmental level

In the previous section, SNA was used to visualize collaboration among individuals. In this section, network

graphs are shown to illustrate collaboration among the different health sciences. Thus, the multidisciplinary

capacity at PSU can be better understood for obesity research. In the following graphs, vertexes represent

the different department affiliations and edges represent the collaboration between two departments based

on co-authorship publications.

For this analysis, just those departments with at least 5 obesity related publications were considered. Of the

14 departments analyzed, 36 edges or collaboration connections were found. The network graph

representing the interdepartmental collaboration can be seen in Figure 4-6.

86

Figure 4-6. Collaboration network per affiliation

Collaboration clusters were identified using the Wakita-Tsurumi algorithm and the network layout was

created with the Harel-Koren Fast Multiscale algorithm. Three groups or clusters were identified according

to their collaboration patterns. The first departmental group is composed of Cellular and Molecular

Physiology (CMP), Neural and Behavioral Sciences (NBS), Pediatrics (PED), Psychiatry (PSY) and

Surgery (SUR), the second group includes Health Policy and Administration (HPA), Medicine (MED),

Obstetrics and Gynecology (O&G), Pathology (PATH) and Public Health Sciences (PHS), and the third

group is composed of Biobehavioral Health (BH), Human Development and Family Studies (HDFS) and

Nutritional Sciences (NS). From the collaboration network graph, it can be seen that there is a strong

collaboration between PHS and PSY, PHS and MED, PHS and O&G, MED and O&G, PED and HDFS,

HDFS and NS, and finally between HDFS and BH.

The departments with higher betweenness centrality are PHS, SUR, and MED with values of 26.633,

10.992, and 9.858 respectively. These values indicate that this set of health sciences serve as central points

to connect other disciplines in the field of obesity. The department with the most direct collaborative

connections is PHS, with 11 links with other departments. It is also interesting that the maximum separation

of the health departments with PHS is 2. This implies that PHS is an essential node in the health sciences

collaboration networks. It also indicates that PHS reaches a wider range of disciplines. However, it must be

noted that the main function of PHS is to provide statistical expertise services to other health science

departments. In this type of graph, it is also meaningful to calculate the network density. For the affiliation

network, an overall density of 0.396 was calculated. If PHS is removed from the network to properly assess

the connectedness of the other health sciences, we obtain a density of 0.321. The densities for clusters 1

87

(PED, PSY, SUR, NBS, and CMP), 2 (HPA, KIN, PHS, MED, O&G, and PATH), and 3 (HDFS, NS, and

BH) are 0.867, 0.700, and 1.000 respectively. These values may represent the level of integration of health

sciences in the field of obesity research.

The maximum degree of separation between two health sciences is three, which means that at most two

intermediate connections are needed to reach any other health science. The average distance between the

entities is 1.571.

4.4.3 Cross-Institutional collaboration networks

Although the main objective of this study was to assess intra-institutional collaboration among the different

obesity researchers at PSU, a cross-institutional collaboration network is presented for the researchers

categorized as experts. Based on the co-authorship analysis conducted, the group of experts was connected

with 588 individuals. The weighted average of collaborators per paper for the obesity experts group is 4.6

co-authors. In Figure 4-7 is presented the collaboration network which includes individuals from different

fields and institutions.

Figure 4-7. Cross-institutional collaboration for obesity experts

From this collaboration network, different clusters were identified by the Wakita-Tsurumi algorithm,

represented by different colors. For visualization purposes, contrary to the previous graphs that were shown,

the edges only represent the connection but not the intensity of the collaboration. An interesting observation

88

from the graph is that there is a cluster – in dark green - containing five of the researchers classified as

obesity experts. On the other hand, there is one cluster containing just one obesity expert.

As seen, SNA is a helpful methodology to understand the dynamics of collaboration networks, not only in

an intra-institutional setting, but also in cross-institutional collaboration. Most of the institutions awarded

by the CTSA have adopted SNA to assess collaboration and provide adequate internal structures that

facilitate interdisciplinary collaboration. However, a broader and more detailed analysis is needed to

evaluate cross-institutional collaboration and its effect on the acceleration of new discoveries to become

health policy.

4.5 Discussion

Building strong collaborative networks is required to accelerate translational research and provide effective

implementation of best practices. SNA techniques are helpful to assess collaboration and its changes over

time. It may be helpful to identify the factors of the complex setups to better collaborate and advance the

transfer of knowledge through efficient dissemination and implementation methodologies. As explicitly

stated by the CTSA consortium, generating multidisciplinary teams and enhancing cross-institutional

collaboration are required to ensure that the efforts of this grant will impact the public’s health.

In this chapter, SNA was applied at the health field level to identify leaders, identify clusters, quantify

network connectedness, identify bridgers or facilitators, and visualize interesting collaboration patterns.

The network graphs and interesting metrics provided will help to inform management about areas in which

networks can be improved, and therefore, impact dissemination, implementation and collaboration growth.

Decision makers in charge of developing collaborative programs among different health sciences can easily

visualize the structural holes that need to be filled to integrate the network. Additionally, by identifying the

leaders for each cluster, integration efforts can be conducted efficiently by using leaders as pillars to

enhance collaboration and communication channels to persuade other members of the networks. Moreover,

it has been shown that active dissemination methods, such as those involving leaders, are much more

effective than passive dissemination methods such as journals or other types of publications (Grimshaw et

al., 2001, Grimshaw et al., 2004, Grimshaw et al., 2012b). Then, researchers who are considered central or

bridger should be the first contact points to better disseminate interventions, policies, or knowledge

translation activities. Despite having different characteristics, all of these roles are essential for institutional

managers to assure the efficient knowledge transference.

89

Multidisciplinarity holds the promise of accelerating translational research, and hence, organizations should

generate mechanisms to promote multidisciplinary collaboration. In the obesity network analyzed, it was

found that between discipline ties or collaboration are slightly favored over within discipline ties. This was

especially notorious for the semi-experts group in which for each within discipline tie there were more than

two between discipline ties. The focus on multidisciplinary research was also observed when accounting

for the health science departments as the agents or nodes of the network. This analysis served to identify

the departments that serve as the bridgers between other departments, and hence, serve as multidisciplinary

facilitators. Again, this information is useful to better design collaborative efforts that aim to improve the

integration among health disciplines.

Some of the limitations of the first SNA approach are the inability of capturing all collaborative connections

using co-authorship analysis only, collaboration patterns could be hidden due to the lag to publish results,

and various other non-tangible collaborations cannot be captured by analyzing co-authorship. More detailed

and robust analysis could be achieved by mixing different data sources including surveys, questionnaires

or direct observations could complement the framework provided.

Finally, SNA should not be seen merely as technique to create network graphs, but as a technique to identify

teams and mentoring patterns, discover opportunities to accelerate knowledge flow and collaboration, raise

awareness and enhance networks, leverage peer support, identify structural holes, discover hidden patterns,

and assess collaborative efforts over time among others. Benefits of using SNA to understand collaboration

patterns are huge. It represents a suitable tool to evaluate, identify opportunities and enhance consortium-

wide collaboration across the nation.

4.6 Conclusions

Multi-disciplinary collaboration is seen as a critical element to succeeding in translational research, not

only for the development of pharmacological and non-pharmacological interventions, but also to implement

and disseminate those innovations (Hall et al., 2008). Although collaboration is known to be an important

component in translational research, evaluating it is complex. This research aimed to take advantage of

SNA to advance collaboration by understanding its complexity. As pointed out by Green et al. (2009), SNA

has become a widely used tool in health communication and diffusion theory. In this sense, diffusion is

essential to advance in dissemination and implementation sciences, and therefore, various potential benefits

can be obtained from SNA if properly used.

90

For the case presented, SNA was found to be a valuable technique to understand and visualize the intra-

institutional collaboration in obesity-related areas at the Penn State Hershey Medical Center. The use of a

recently launched tool – Penn State Profiles – and the WoK platform were highly helpful to identify obesity

researchers and generate collaboration matrixes. From the analysis conducted, different collaboration

patterns were presented and clusters were identified. The insights obtained are highly valuable to provide

advice and create an institutional structure to support an effective collaboration among different health

sciences and institutions as recommended by the CTSA consortium. This is critical to successfully move

new discoveries into clinical practice. In addition, leaders play a crucial role in social networking as they,

usually, have a considerable influence over the other members of the network. Therefore, promoting

collaboration networks is an important aspect not only to develop and testing better treatments but also to

accelerate their dissemination and implementation. In public health, there is a growing consensus with

respect to the ability of social networking to help developing more effective health programs. In part, this

is due to the increasing acceptability of system thinking and ecological approaches (Green et al., 2009, Best

et al., 2003). Moreover, the CTSA has argued that team science techniques should be adopted by every

CTSA hub as a way to better understand how to generate teams and generate long-term collaborations.

Certainly, SNA can also provide support to understand and monitor the performance metrics and structures

of team collaboration.

91

Chapter 5

GUIDING THE STRATEGY AND RESOURCE

ALLOCATION OF HEALTHCARE ORGANIZATIONS

BASED ON IMPACT OF HEALTH INTERVENTIONS

5.1 Introduction

Distributing resources in healthcare is a complex process as they are scarce relative to needs. Consequently,

understanding the drivers of value is relevant for an optimal resource allocation. The healthcare system in

the U.S. has been progressively shifting to a paradigm that focuses on value-based data-driven care (Kaye

et al., 2014). In order to accomplish a high quality care delivery at lower costs, audit and feedback efforts

are essential components to formalize the use of data and lessons learned, and therefore, the wise

prioritization of efforts. A key objective of data-driven healthcare is to maximize the value that scarce

resources can have on society. In this sense, using decision making resource allocation techniques is

important to support the effective use of those limited resources. As stated by Lee and Kwak (1999), the

problem of resource allocation in healthcare must be considered as a significant and integral part of s-

trategic planning to provide effective healthcare service and management. However, given the complexity

of the healthcare systems, allocating resources to different interventions is typically a difficult multiple-

criteria problem in which conflicting objectives are present (Guindo et al., 2012). This fact becomes

especially challenging and pressing for healthcare organizations and funding agencies to decide what mix

of interventions might be funded to maximize the allocation value. In addition, defining value on its own is

already challenging for decision makers. From a health policy point of view, policy makers should aim to

distribute resources among different competing interventions and populations based on their anticipated

benefits (Patrick and Erickson, 1993).

92

In order for organizations to define value, its vision must be well-articulated and its strategic planning must

be well-understood. Thus, resource allocation decisions are made not based solely on traditional criteria

such as quality and economic aspects, but also incorporating the fit of different courses of action into the

strategy of the organization or specific program. This alignment with the strategic plan motivates and guides

the investments to support the achievement of long-term objectives (Venkatraman, 1989). Hence, allocating

resources properly plays an important role in this aim. As a result, a good allocation of resources will lead

to a mix of projects and interventions that generate the greatest value through the organization’s eyes. For

example, the NIH through its CTSA program has stated the need of promoting diversity and

multidisciplinarity to take advantage of the synergies generated by these different groups of knowledge. In

addition, the CTSA encourages the development of a vision that promotes teamwork, facilitates

multidisciplinary translational teams, and enhances collaborations. Giving these general directions, it is

expected that CTSI awardee institutions allocate their resources considering the CTSA’s general

alignments. Hence, these multiple objectives should be taken into account to support health policy

guidelines, and thus, justify the implementation of health interventions that result in the greatest population

benefit.

By nature, data is needed to wisely conduct the resource allocation process and prioritization of health

interventions. In order for organizations such as governmental funding agencies, universities, and research

institutes to be able to evaluate and prioritize proposals, researchers should provide structured information

to quantify the potential impact of the proposed intervention, and thus, justify their implementation

(Fielding and Teutsch, 2013). This structured information should be coherent and comparable among

different types of interventions. Consequently, metrics concerning burden of disease, preventable burden,

and economic value are needed. In this sense, cost-effectiveness measures provide a clear and coherent

framework for intervention evaluation and comparison among them. Such metrics incorporate not only the

impact in terms of health but also the intervention’s cost, which is relevant for decision makers to effectively

allocate resources (Jamison et al., 2006). Nevertheless, it must be also recognized that cost-effectiveness

results should not be used as a strict guideline for resource allocation. There may be other considerations

(e.g., ethical issues, closing health disparities among different population, strategic focus on certain

diseases, etc.) to implement interventions that do not achieve the typically used cost-effectiveness

guidelines (Owens et al., 2011). Additionally, as previously argued, organizations should also consider the

prioritization of interventions that better fit their vision and long-term strategy. In order to properly account

for these multiple elements involved in the resource allocation decision making problem, multiple-criteria

techniques such as goal programming (GP) are suitable approaches for addressing this complexity, and also

serve to operationalize the organization’s strategy and drivers of value.

93

This chapter covers two relevant topics related to the resource allocation process: 1) mathematical

programming for the proposal selection problem; and 2) cost-effectiveness analysis. Finally, a combination

of these two main topics is briefly presented to guide the proposal selection incorporating cost-

effectiveness. The methodology for the proposal selection problem is presented in Section 5.2. The

framework proposed uses a multiple-criteria decision making technique to select a mix of interventions that

incorporates multiple objectives and the fit of these set into the organization’s strategy. Specifically, a GP

model is proposed for generating an optimal solution based on the value that a certain mix of interventions

provides (Section 5.3). In addition, a framework is presented to guide researchers through a rapid high-

level estimation of the impact of an intervention based on cost-effectiveness measures (Section 5.6). A case

study based on the rapid high-level impact estimation (RHIE) framework is presented in Section 5.7. These

topics potentially provide valuable insights and benefits for three main healthcare stakeholders:

organizations that provide funding; healthcare researchers; and people. Moreover, in Section 5.10 different

formulations are proposed to integrate these two topics. Summarizing, the first part of the chapter focuses

on providing models for organizations to guide an informed allocation of resources based on value. The

second part provides insights for healthcare researchers to think about health interventions in terms of their

potential impact. Finally, these two parts aim to have a positive impact on people’s health as a result of a

better understanding of drivers of value to a wise allocation of health resources. Hence, the sections of this

chapter are distributed according to Figure 5-1.

Figure 5-1. Sections distribution of Chapter 5

Proposal Selection

Method: Goal

Programming

Impact Estimation

Method: Cost-

Effectiveness Analysis

Strategic proposal selection incorporating cost-

effectiveness goals

Sections 5.2 – 5.7 Sections 5.6 – 5.9

Section 5.10

94

5.2 Methodology: Goal Programming Model for Proposal Selection

Proposal selection is a challenging organizational decision-making task. Organizations such as government

funding agencies, universities, and research institutes have to deal with complex decisions to select a

relatively reduced number of projects or proposals to be funded and implemented. In this sense, two of the

main reasons behind this challenging task are: 1) the difficulty of predicting the future success and impact

of the proposed projects, and 2) the decision involves a multi-evaluator system through multiple stages

(Tian et al., 2005). Hence, determining which of the relatively large set of the received proposals would

provide the greatest value, and consequently, are worthy of funding is a difficult task (Burstein and

Holsapple, 2008). In addition, decision making problems involving multiple experts become even more

complicated when multiple criteria must be considered (Yager, 1993). An additional challenge in resource

allocation, specifically in project or proposal selection, is to select a mix of them such that a course of action

is oriented to the achievement of the programmatic agenda or strategy of the organization. In order to

overcome these challenges, a multiple criteria-based framework is presented. This framework uses GP to

support decision makers in the proposal selection problem. A case study based on the Clinical and

Translational Sciences Institute at The Pennsylvania State University is used for illustration.

5.2.1 Model overview

In this section, a description of the proposed approach is provided to guide the proposal selection process.

This approach is divided into four phases: 1) Understanding the strategy, 2) Understanding the constraints,

3) Formulating the model, and 4) Solving and validating. The diagram of this approach is presented in

Figure 5-2.

Understanding the strategy: This is the initial phase of the approach and aims to understand the intent of

the organizational strategy, and how different goals could help to achieve its vision. This phase helps to

properly define the frame in which the organization wants to be not only in the short-term, but also in the

medium-long term guided by its strategic plan. For these reasons, the involvement of key leaders of the

organization is important to guide the identification of relevant organization’s goals. Additionally, we argue

that the strategy of an organization should be considered “alive”, as it must be able to take different shapes

to adapt to internal and external factors. In this model, the vision and mission of the organization receive

feedback and gain knowledge from historical and other data sources as a way to calibrate its strategy. This

calibration procedure is important to overcome current challenges to address population’s health in a more

dynamic manner. From this phase a list of goals and respective targets is identified. Finally, as some of

these goals might have different levels of relative importance, goal priorities must be defined.

95

Understanding the constraints: The second phase aims to identify current constraints that strictly define the

region of feasible solutions (restrict courses of action). These constraints can be related to compliance with

guidelines, regulations, and policy, and/or resource availability including people, equipment, and budget.

Although most of these constraints are internal, some others, especially those related to regulations can

have external components. As clearly developed in the theory of constraints principles (Goldratt, 1990), it

is fundamental to understand the constraints as they establish the limits of performance for any system, in

this case, resource allocation. In this phase, experts are asked to identify the set of constraints as well as

defining the limits of those constraints.

Formulating the model: Goal and system constraints have already been identified. In this section the aim is

to develop a mathematical model to solve the proposal selection model that incorporates an objective

function (minimize deviations from targets) subject to the set of constraints. One of the key challenges of

this phase is to properly incorporate meaningful data into the model. In this sense, extracting information

from historical data might be challenging in practice. Tools such as regression and decision trees could be

helpful to extract insights from this data. On the other hand, some of the coefficients used in the model will

require the collection of data from different sources such as researchers’ profile, scientific search engines,

and proposals’ characteristics among others. Finally, the analysis of historical data can also support the

calibration or modification of current strategic lines of the healthcare organizations.

Solving and validating: The last phase of the proposed approach aims to solve the model and validate its

results. There are various multipurpose and specialized software available in the market to solve multiple-

criteria problems (e.g., LINDO, LINGO, GAMS, and Microsoft Excel). Once the results have been

obtained, the experts or leaders of the organization must validate that those results are reflecting the

preferences and constraints that they wanted to incorporate in the proposal selection decision. In case that

the results are unreasonable from the experts’ perspective, the model might be calibrated by revisiting the

strategy (Phase 1) and the constraints (Phase 2). Otherwise, the selected proposals are announced.

96

Figure 5-2. Overview of methodology for the proposal selection problem

Identify Goals

Define Goal

priorities

Set targets for

those goals

Identify hard

constraints

Define

coefficients

Define limits

Understanding

strategy

Understanding

constraints

Formulating

the model

Formulate goal

constraints

Formulate

objective func.

Formulate syst.

constraints

Solving and

validating

Solve the

model

Reaso

nable

Assign funds

YES

NOCA

LIB

RA

TE

PHASE

DATA

SOURCE PROPOSED STEPS

POTENTIAL

TOOLS

• Vision

• Mission

• CTSA

• Guidelines

• Regulations

• Resource

availability

Historical

Data and

Data

Preparation

Steps

• Experts group

• Needs

assessment

• Ranking /

Rating / AHP

methods

• Experts group

• Budget analysis

• Mathematical

bound

approaches

• Regression

• Decision tree

analysis

• Experts’ opinion

• Bibliographic

search

• Software (Excel,

LINDO,

GAMS)

• Experts’ opinion

• Statistical

testing

FE

ED

BA

CK

97

5.2.2 Generic model

A generic model is presented to illustrate how this model can be implemented in an organization such as

the CTSI at PSU.

5.2.2.1 Phase 1: Understanding the strategy

The CTSI at PSU has five main goals that might be considered when selecting proposals.

1. Strategic planning goals: The selected proposals should be aligned with the strategy and vision of

the center.

2. Risk associated goals: The selected proposals should be aligned with current willingness or levels

of risks accepted by the center.

3. Potential benefits: One of the goals of the center is to fund proposals that will potentially lead to

benefits for the center’s prestige.

4. Training the new generation of researchers. The center is highly committed to fund proposals that

include new investigators.

5. Catalyze multidisciplinarity. One of the objectives of the center is to support proposals in which

multiple disciplines collaborate to advance knowledge.

In order to set targets for these goals, the metrics to operationalize them must be identified. Finally, different

procedures (rating, ranking, AHP) could be used to estimate the relative importance of each goal. A detailed

explanation is given in subsequent sections.

5.2.2.2 Phase 2: Understanding the constraints

The CTSI at PSU identifies the main constraints that must be considered for proposal selection

1. Budget constraints: The center cannot exceed the maximum budget to be allocated.

2. Quality constraints: The center cannot fund proposals whose quality is lower than a certain

threshold.

3. Mutually exclusiveness constraints: The center consider that two proposals that are very similar

cannot be simultaneously selected.

4. Minimum number of proposals selected constraints: The center is strict in the minimum number of

proposals to be selected.

5. Balancing fund constraints: The center is strictly committed to providing a fair balance of fund

across the departments.

98

Each one of these constraints is tied to a pre-defined limit that cannot be exceeded.

5.2.2.3 Phase 3: Formulating the model

After identifying the goals and constraints to be included, the GP model must be formulated based on its

corresponding parameters/coefficients.

Decision variable

The GP model proposed uses a binary approach in which the proposal i is either selected to receive fund or

not. Then the decision variable xi can take values 0 (non-selected) or 1 (selected).

Goal Constraints

1. Strategic planning

The strategic planning goal will account for a selection of projects that incorporates the vision of the center.

The strategy goals could be related to the study of a specific type of disease, use of a specific methodology

or framework, theme of health research, etc.

2. Risk associated

The risk associated to the project depends on various factors including PI’s experience, project team size,

health field, years of project, etc. Then, a risk factor is included on each project to account for the

dimensions mentioned.

𝑟𝑖𝑥𝑖 + 𝑑𝑟𝑖+ − 𝑑𝑟𝑖

− = 𝑅𝐼𝑆𝐾max _𝑡𝑎𝑟𝑔𝑒𝑡 for all 𝑖 Eq. 51

where:

ri Risk score associated to project i

RISKmax_target Maximum overall risk accepted by the center

𝑑𝑟𝑖− Negative deviational variable associated to the risk of proposal i

𝑑𝑟𝑖+ Positive deviational variable associated to the risk of proposal i

3. Potential Benefit

Some of the projects, if funded, might have the capacity for generating potential benefits for the center. For

instance bringing external funding to extend the projects or take them to a deeper level of research or/and

99

resulting in journal publications could be considered as beneficial for the center’s prestige. In order to

account for this, historical information might be used to estimate the potential benefit coefficient. For

example, the coefficient PBi could incorporate a score or probability of project i to eventually lead to

external funding (e.g., NIH, NSF, etc). This coefficient could also incorporate the academic history of the

investigators and their ability to submit proposals to external agencies, quality of the proposal, theme of

research or other relevant attributes. Usually, these types of goals are treated as consolidated targets and

not at the proposal level.

∑ 𝑃𝐵𝑖𝑥𝑖 + 𝑑𝑃𝐵+ − 𝑑𝑃𝐵

− = 𝑃𝐵min _𝑡𝑎𝑟𝑔𝑒𝑡 Eq. 52𝑛

𝑖=1

where:

PBi Potential benefit score associated to project i

PBmin_target Minimum overall potential benefit required by the center

𝑑𝑃𝐵− Negative deviational variable associated to the risk of the proposals

𝑑𝑃𝐵+ Positive deviational variable associated to the risk of the proposals

4. Training the new generation of researchers

One of the key strategic goals of the center is to train the new generation of health researchers and

investigators. In order to account for this goal, new investigators (young investigators) should be prioritized

or at least secured a minimum number of positions. For example, a coefficient represented by the number

of new investigators included on each proposal could be used. These types of goals are usually treated at

the group level. In this sense, the right hand side (RHS) value could be the minimum number of new

investigators expected by the center.

∑ 𝑁𝐼𝑖𝑥𝑖 + 𝑑𝑁𝐼+ − 𝑑𝑁𝐼

− = 𝑁𝐼min_𝑡𝑎𝑟𝑔𝑒𝑡 Eq. 53𝑛

𝑖=1

where:

NIi Training new investigators score associated to project i

NImin_target Minimum overall score for training new investigators required by the center

𝑑𝑁𝐼− Negative deviational variable associated to the training new investigators of the proposals

𝑑𝑁𝐼+ Positive deviational variable associated to the training new investigators of the proposals

100

5. Catalyze interdisciplinarity

Multidisciplinary research is seen as a positive aspect in proposal selection (Lowe and Phillipson, 2006).

Promoting multidisciplinarity by enhancing investigators and professionals to cross the health field

boundaries between disciplines is part of the vision of the center. This goal is aligned to the NIH’s stated

need to promote diversity and multidisciplinarity to take advantage of the synergies generated by these

different groups (RFA-TR-14-009). Recently, the CTSA lunched a RFA in which they formally encourage

the development of a vision that provides guidelines to incentive teamwork, to facilitate multidisciplinary

translational teams, and promote collaborations.

The coefficient MDi represents how well multi-disciplines are integrated in project i. This coefficient could

include factors such as number of researchers from different fields, number of institutions participating of

the project, integration of non-healthcare fields, and/or cross-campus collaboration, among others.

𝑀𝐷𝑖𝑥𝑖 + 𝑑𝑀𝐷𝑖+ − 𝑑𝑀𝐷𝑖

− = 𝑀𝐷min _𝑡𝑎𝑟𝑔𝑒𝑡 for all 𝑖 Eq. 54

where:

MDi Multidisciplinarity score associated to project i

MDmin Minimum multidisciplinarity score required by the center

𝑑𝑀𝐷𝑖− Negative deviational variable associated to multidisciplinarity of proposal i

𝑑𝑀𝐷𝑖+ Positive deviational variable associated to multidisciplinarity of proposal i

Some other goals could have been included if pertinent are:

Fund proposals that are more likely to increase the “value” or impact society. In other words, fund

those proposals whose expected research outcomes will have an important impact on people’s

health, and consequently, center prestige.

Account for a balance between the budget allocations among different health fields or departments.

Parameters such as the size of the departments or colleges could be used for this formulation.

Fund projects in which synergies can be obtained. For example, an intervention of informing and

diagnosing could provide much more value if they are implemented together.

Fund proposals that despite their good quality have little chances of receiving funds from external

agencies. This is especially notorious in under investigated diseases.

Fund proposals that are focused on contingent issues or others that have been stated by recognized

institutions such as the NIH as pressing issues that have not been explored enough.

101

Prioritize and balance proposals that can provide “quick wins” and others that are seen as critical

to supporting the strategy and vision of the center in the long-term.

System Constraints

1. Budget constraints

There is a fixed budget that cannot be exceeded.

∑ 𝐵𝑈𝐷𝑖𝑥𝑖 ≤ 𝐵𝑈𝐷𝑚𝑎𝑥

𝑛

𝑖=1 Eq. 55

where:

BUDi Budget required by proposal i

BUDmax Maximum budget allowed by the center to be distributed among the proposals

In this constraint we could have also incorporated a minimal amount (𝐵𝑈𝐷𝑖𝑚𝑖𝑛) to be allocated which the

decision maker believes that the project can receive without affecting the quality nor outcomes of the

project.

2. Quality constraints

A first filter of the center is to remove from the analysis those proposals that do not comply with the

minimum quality threshold established (Qmin). This constraint could be either used as an initial filter or

incorporated into the model.

(𝑄𝑖 − 𝑄min)𝑥𝑖 ≥ 0 for all 𝑖 Eq. 56

where:

Qi Quality score of proposal i

Qmin Minimum quality score for a proposal to be considered for funding

3. Mutually exclusiveness constraints

There are constraints related to mutually exclusiveness. In this case, depending on the characteristics of the

proposals, they cannot be selected simultaneously. For example, let us say that project 2 (x2) and project 4

(x4) are mutually exclusive.

102

Then this constraint can be modeled as follows:

𝑥2 + 𝑥4 ≤ 1

4. Fund at least a minimum number or proposals

At least a minimum number of proposals must be funded.

∑ 𝑥𝑖 ≥ 𝑁𝑃𝑅𝑂𝑃𝑚𝑖𝑛

𝑛

𝑖=1 Eq. 57

where:

NPROPmin Minimum number of proposals to be funded

5. Balancing funds constraints

These constraints account for providing a fair distribution of resources. Thus, strict rules are placed to avoid

that one department receives significantly more funding than the others. In this example, let us balance the

funding with respect to the proportion of graduate students in each department. Let us assume that proposals

2, 8, 19, and 20 come from the same department. Additionally, not more than a certain proportional amount

can be given to the department.

𝐵𝑈𝐷2𝑥2 + 𝐵𝑈𝐷8𝑥8 + 𝐵𝑈𝐷19𝑥19 + 𝐵𝑈𝐷20𝑥20

𝐵𝑈𝐷𝑚𝑎𝑥≤ (1 + 𝐵𝐹𝑚𝑎𝑥) ∗

𝑁𝑈𝑀𝑠𝑡𝑢_𝑙

𝑇𝑂𝑇𝐴𝐿𝑠𝑡𝑢 Eq. 58

where:

BFmax Maximum proportional amount that a department can receive

NUMstu_l Number of graduate students enrolled in department l

TOTALstu Total number of students enrolled

6. Non-negativity and binary constraint

The GP modeled is based on a 0 – 1, therefore, either the project is selected (1) or not (0). Additionally,

deviational variables are non-negative.

𝑥𝑖 is binary for all 𝑖

𝑑𝑟𝑖+ , 𝑑𝑟𝑖

− , 𝑑𝑀𝐷𝑖+ , 𝑑𝑀𝐷𝑖

+ ≥ 0 for all 𝑖 Eq. 59

103

Objective Function

The objective function in GP formulations aims to minimize unwanted deviations from targets. Hence, the

objective function of this GP model is:

𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑍 = 𝑤𝑟 ∑ 𝑑𝑟𝑖+

𝑛

𝑖=1+ 𝑤𝑃𝐵𝑑𝑃𝐵

+ + 𝑤𝑁𝐼𝑑𝑁𝐼+ + 𝑤𝑀𝐷 ∑ 𝑑𝑀𝐷𝑖

+𝑛

𝑖=1 Eq. 510

where:

wr is the relative importance of the risk-associated goal

wPB is the relative importance of the potential benefits-associated goal

wNI is the relative importance of the new investigators-associated goal

wMD is the relative importance of the multidisciplinarity-associated goal

It must be noticed that a goal was not included for a specific funding strategy. They will mostly depend on

specific short or long-term objectives (e.g., promoting community-based programs, use of big-data tools,

integration of technology, etc.).

Summary of Decision Variables and Coefficients

xi Decision variable indicating whether the proposal i is selected (1) or not (0)

ri Risk score associated to project i

RISKmax_target Maximum overall risk accepted by the center

𝑑𝑟𝑖− Negative deviational variable associated to the risk of proposal i

𝑑𝑟𝑖+ Positive deviational variable associated to the risk of proposal i

PBi Potential benefit score associated to project i

PBmin Minimum overall potential benefit required by the center

𝑑𝑃𝐵− Negative deviational variable associated to the risk of the proposals

𝑑𝑃𝐵+ Positive deviational variable associated to the risk of the proposals

NIi Training new investigators score associated to project i

NImin_target Minimum overall score for training new investigators required by the center

𝑑𝑁𝐼− Negative deviational variable associated to the training new investigators of the proposals

𝑑𝑁𝐼+ Positive deviational variable associated to the training new investigators of the proposals

MDi Multidisciplinarity score associated to project i

MDmin Minimum overall multidisciplinarity required by the center

104

𝑑𝑀𝐷𝑖− Negative deviational variable associated to multidisciplinarity of proposal i

𝑑𝑀𝐷𝑖+ Positive deviational variable associated to multidisciplinarity of proposal i

BUDi Budget required by proposal i

BUDmax Maximum budget allowed by the center to be distributed among the proposals

BFmax Maximum proportional value to account for balanced allocation of resources

Qi Quality score of proposal i

Qmin Minimum quality score for a proposal to be considered for funding

NPROPmin Minimum number of proposals to be funded

NUMstu_l Number of students enrolled in department l

TOTALstu Total number of students enrolled

wr Relative importance of the risk-associated goal

wPB Relative importance of the potential benefits-associated goal

wNI Relative importance of the new investigators-associated goal

wMD Relative importance of the multidisciplinary-associated goal

Model Coefficients

One of the key factors of success of GP models is to properly select and obtain the coefficients to be used

in both goal and system constraints. In this section, the description of different coefficients that could be

used in this generic model is presented. Some of the coefficients proposed seek to gain advantage from the

knowledge built through past experiences of funded proposals. In this regard, the CTSA has been actively

pushing for a better use and formalization of best practices within the CTSA’s awardees. Therefore, using

historical data as a way to formalize and operationalize best practices to be used in proposal selection is

required to advance the capabilities of the CTSA’s managers to better allocate resources based on evidence.

Risk Associated Score

In the proposed GP, one goal constraint accounts for the maximum amount of risk allowed by the center.

As previously proposed, the risk factor could include the years of experience of the PI, the average years

of experience of the PI, the health field or disease, the coverage of expertise required.

An alternative way to generate this score could be using historical data with different proposal attributes

such as those mentioned above and ask experts to classify those with respect to their perceived success. In

this case, according to the attributes, a risk score could be predicted for the new proposals under

105

consideration. This alternative is in practice difficult to implement as data points are currently limited.

However, as encouraged by the CTSA, sharing histories of success between the CTSA hubs in a repository

could tremendously increase the data available to make this type of analysis. Therefore, CTSA hubs could

learn not only from internal experiences, but also from other CTSA hubs making the use of its resources

much more efficient.

For this model, let us say that the PI’ expertise, including experience and intensity of publication, were

considered as the drivers of proposals’ risk. As previously mentioned, other factors such as team size and

disease investigated could also be incorporated as components of the risk goal.

PI expertise coefficient

In scientific research fields, the number of publications of a researcher can be seen as a measure of expertise

(Rodriguez and Bollen, 2005). In addition, publication data is usually seen as a suitable way to be used for

obtaining expertise (Cameron et al., 2007, Cameron et al., 2010). Another dimension of expertise is

experience (McEnrue, 1988). This experience can be typically measured as the length of time of a person

within a profession.

In this study, we considered that expertise is achieved after 8 years of experience. This information could

be estimated using the year of the first publication of the author in the given research field. Additionally

the number of papers could be extracted by querying databases such as PubMed. We will consider that

having published five or more scientific articles is a fair estimator of expertise in the area. A normalization

procedure can be used to express expertise in a range from 0 to 1. Thus, expertise score for a proposal can

be expressed as follows:

𝑃𝐼𝑒𝑥𝑝 =[𝑚𝑖𝑛 {(

𝑌𝑒𝑥𝑝 − 88 ) , 0} + 1] + [𝑚𝑖𝑛 {(

𝑁𝑝𝑎𝑝𝑒𝑟 − 55

) , 0} + 1]

2 Eq. 511

where:

PIexp: is the expertise score assigned to the PI

Yexp: is the years of experience of the PI

Npaper: is the number of papers of the PI

106

It must be noticed that, in this case, PIexp was obtained as the simple average between the two factors

mentioned. However, if the decision maker has some other insights or preferences in terms of the

importance of these factors, a weighted average can be used instead.

In order to simplify the analysis, the evaluator could use only one of the proposed metrics to determine the

level of expertise. However, using the two metrics gives a fairly good and complete balance in the

estimation of expertise. Solely using metrics of time or frequency could conduct to errors. In this regard,

studies have demonstrated that experience is a necessary, but not sufficient dimension for expertise

(Ericsson et al., 2007). In the healthcare domain, there are some examples in which experience has been

found to be significantly positively correlated to expertise (McHugh and Lake, 2010).

A more elaborated approach to quantifying topic expertise of a researcher based on bibliographic data from

scientific publications can be found in Cameron et al. (2010). However, for our purposes this level of detail

was not considered as value-added.

Potential Benefits Score

In order to support the proposed formulation, a Potential Benefit score (PBi) is required. This score can be

composed by likelihood of receiving external funding and potential to result in article publications. In this

case, a characterization of the type of research including type of disease and method might be used. A

prediction based on past proposals funded can be used for these purposes. Obtaining data about external

funding received and publications from already funded proposals is relatively easy. However, it must be

realized that it might be difficult to identify the right attributes to categorize papers.

The most straightforward metrics to use tangibles to quantify potential benefits are the number of papers

and capabilities of bringing external fund. In this case study, it will be assumed that based on historical

data, the likelihood of a proposal to result in an ideal number of papers and/or bringing external fund depend

on the PI’s experience, multidisciplinary score, and quality score.

If more historical data is available, metrics such as impact on people’s health, ability to reduce health

disparities, and size of potentially impacted population, among others could be incorporated into this

analysis. However, these metrics are not easily obtained or estimated especially due to the time that it takes

to get reliable estimations and the reduced size of historical data.

Other analysis including decision tree techniques could also be used to generate clusters or categories based

on significant attributes. Multiple-regression analysis techniques can also be explored to estimate the

potential benefits of a proposal given certain significant attributes based on past experiences.

107

Training the New Generation of Researchers Score

A key objective of the CTSI is to train the new generation of investigators. This goal can be set in terms of

proportion of new investigators involved in the proposals or simply using the number of new investigators

(NIi) to which a target of minimum expected total number of your investigators in the pool of selected

proposals is set. This last option is used in the proposed model. The number of young investigators can be

easily obtained directly from the proposal.

Catalyze Multidisciplinarity

A score to account for multidisciplinarity can also be easily obtained from the proposal information. One

of the metrics that can be used is the number of different health fields participating of the proposal or a

more elaborated score to account for different levels of multidisciplinarity.

Multidisciplinarity score

In this case, multidisciplinarity score of proposal i (MDi) is a normalized value of the number of disciplines

participating of the proposal. For illustration purposes, let us consider that the integration of three or more

disciplines is considered to constitute a fairly multidisciplinary team. Thus, it can be formulated as follows:

𝑀𝐷𝑖 = 𝑚𝑖𝑛 {(𝑁𝑈𝑀𝑚𝑑_𝑖 − 3

3) , 0} + 1 Eq. 512

where:

NUMmd_i is the number of disciplines participating of proposal i.

Goal Weights

In order to account for the relative importance of each one of the goals incorporated in the model, a

weighting method must be used. In this case, a rating method is used in which the decision maker is asked

to rate the importance of each goal using a 1 – 9 scale. Table 5-1 can be used to guide the respondent.

108

Table 5-1. Rating method to obtain goal weights

Rate Importance definition

1 This goal is not significant for the success of the CTSI

2 This goal has a low impact on the success of the CTSI

3 This goal is slightly important for the success of the CTSI

4 This goal is somewhat important for the success of the CTSI

5 This goal is neutrally important for the success of the CTSI

6 This goal is moderately important for the success of the CTSI

7 This goal is significantly important for the success of the CTSI

8 This goal is very important for the success of the CTSI

9 This goal is extremely important for the success of the CTSI

Let Wij be the rate assigned to goal i by decision maker j. Then the composite weight of the i-th goal is given

by:

𝑤𝑖 =∑ 𝑊𝑖𝑗

𝑚𝑗=1

∑ ∑ 𝑊𝑖𝑗𝑚𝑗=1

𝑛𝑖=1

, for 𝑖 = 1,… , 𝑛 and 𝑗 = 1,… , 𝑚 Eq. 513

Other more elaborated procedures to obtain the weights such as AHP could have been used.

5.2.2.4 Phase 4: Solving and validating

Goal programming models can be easily solved using linear programming software such as LINGO,

LINDO, and GAMS. The solution must be validated by the experts of the organization. This validation

process could include non-statistical and/or statistical validation techniques. Face validity is one of the

techniques in which experts’ opinion can be easily gathered. In such cases, the experts analyze whether the

obtained solution is in reality giving coherent results according to the different criteria previously defined.

In cases in which the solution is not seen as a good representation of the different goals and constraints,

Phases 1 and 2 must be reviewed for consistency. Additionally, some of the key pitfalls to avoid in goal

programming formulations could be checked as well. According to Jones and Tamiz (2010) some rules for

avoiding pitfalls are:

1. Always include both deviational variables in the formulation

2. Do not combine ideal target levels with lexicographic variant

3. Do not use an excessive number of priorities

109

4. Regard the weights, priority schemes, and target levels as initial estimates and not as set in stone

values

5. Always use an appropriate normalization scheme with the weighted variant

6. Tchebysheff variant and priority levels within the lexicographic variant need normalizing too if

they are not commensurable

7. Always perform validation of each normalization constant in the context of the decision situation

8. Always check the solution for Pareto efficiency and use a restoration scheme if required

9. Only penalize unwanted deviations

10. Always make an informed choice of variant based on the nature of the problem

5.3 Case Study: Proposal Selection in a CTSA Hub

In this section, a case study is presented to illustrate how the proposed approach can work in practice. The

coefficients representing the relative importance of the goals were obtained through a survey to the key

function leaders of the Pilot Projects key function area and some members of the CTSI’s executive

committee. On the other hand, the project’s information, including coefficients and scores presented here

are intended to illustrate the framework and do not necessarily reflect the proposals received by the CTSI

at PSU. A table with the list of proposals and its characteristics is shown in Appendix B.

5.3.1 Identifying goals and constraints

A list of four main goals were identified for the PSU CTSI.

1. Risk goal

2. Potential benefits goal

3. Training the new generation of researchers goal

4. Catalyze interdisciplinarity

Additionally, six hard constraints were identified

1. Budget constraint

2. Quality constraint

3. Mutually exclusiveness constrains

4. Fund at least a minimum number of proposals

5. Proportion of funded proposals constraints

6. Cross campus collaboration constraint

110

5.3.2 Obtaining goal weights

After identifying the goals and hard constraints, a group of six experts participated of a survey to assign

scores to the different goals with respect to their importance. Thus, the weights (relative importance) of

these goals were calculated. The opinions of the six experts regarding the four goals considered are shown

in Table 5-2. The experts assigned a score to each goal according to their level of importance using a scale

from 1 to 9.

Table 5-2. Expert's scores and goal priorities

Goal

Experts’ Score ∑𝑊𝑖𝑗

𝑚

𝑗=1

𝑤𝑖 E1 E2 E3 E4 E5 E6

Risk 3 6 9 6 8 2 34 0.193

Potential benefits 7 7 9 9 9 6 47 0.267

Training 6 6 9 9 8 6 44 0.250

Multidisciplinarity 9 8 9 9 8 8 51 0.290

From this analysis, it can be said that the most important goal for the CTSI leaders is multidisciplinarity

with a relative importance of 0.290. The least important goal is risk with a relative importance of 0.193.

5.3.3 Formulating goal constraints

In this section, goal constraints are formulated. Additionally, normalization procedures are used as needed

to reduce the bias given by the magnitude of the coefficients used in the formulation.

Risk Goal

In order to be consistent with the PIexp score calculated, the target for this goal is 1. Then the following

equations are to be included as goal constraints. The years of experience and number of papers for PIs are

shown in Appendix B and ri scores are shown in Appendix C.

(𝑟𝑖 − 1)𝑥𝑖 + 𝑑𝑟𝑖+ − 𝑑𝑟𝑖

− = 0 ∀𝑖 Eq. 514

−0.513𝑥1+ 𝑑𝑟1+ − 𝑑𝑟1

− = 0

−0.575𝑥2+ 𝑑𝑟2+ − 𝑑𝑟2

− = 0

−0.250𝑥4+ 𝑑𝑟4+ − 𝑑𝑟4

− = 0

−0.638𝑥7+ 𝑑𝑟7+ − 𝑑𝑟7

− = 0

111

−0.425𝑥13+ 𝑑𝑟13+ − 𝑑𝑟13

− = 0

−0.488𝑥14+ 𝑑𝑟14+ − 𝑑𝑟14

− = 0

−0.010𝑥17+ 𝑑𝑟17+ − 𝑑𝑟17

− = 0

The normalized contribution of this goal to the objective function is given by:

𝑤𝑟

⌈𝐵𝑈𝐷𝑚𝑎𝑥

𝐵𝑈𝐷𝑎𝑣𝑔|0.1⁄ ⌉∗ ∑ 𝑑𝑟𝑖

+𝑛

𝑖=1 Eq. 515

where:

𝐵𝑈𝐷𝑎𝑣𝑔|0.1 represents the average budget requested by all the proposals except the 10% lowest

and highest. In this case, as 20 proposals are being considered, this expression

calculates the average budget excluding the two proposals with the lowest

requested budgets (12 and 15), and two proposals with the highest requested

budgets (3 and 4).


𝐵𝑈𝐷𝑎𝑣𝑔|0.1⁄ ⌉ is used for normalizing the deviational variables to properly account for the goal

weight previously assigned. The resulting value could be explained as the expected

number of proposals to be funded assuming that the maximum budget allocated

for the program is used.

0.193

⌈600,00072,000⁄ ⌉

∗ ∑ 𝑑𝑟𝑖+

20

𝑖=1=

0.193

9∗ ∑ 𝑑𝑟𝑖

+20

𝑖=1

Potential Benefits Goal

Let us say that the center’s experts require a 40% achievement of this goal considering the consolidated

score. The scores are shown in Appendix C and were obtained from Appendix D. This score is based on

the PI’s experience, multidisciplinarity score, and quality of the proposal.

Then, this goal can be written as:


− = 𝑃𝐵𝑚𝑖𝑛

𝑛

𝑖=1 Eq. 516

112

Let us say that as the number of proposals is not known in advance, then an estimate is used.


− = 0.4 ∗ ⌈𝐵𝑈𝐷𝑚𝑎𝑥

𝐵𝑈𝐷𝑎𝑣𝑔|0.1⁄ ⌉𝑛

𝑖=1

For this case study, the equation is:

0.300𝑥1 + 0.200𝑥2 + 0.675𝑥3 + 0.300𝑥4 + 0.7003𝑥5 + 0.455𝑥6 + 0.130𝑥7 + 0.800𝑥8 + 0.195𝑥9

+ 0.800𝑥10 + 0.700𝑥11 + 0.700𝑥12 + 0.450𝑥13 + 0.450𝑥14 + 0.525𝑥15 + 0.900𝑥16

+ 0.800𝑥17 + 0.900𝑥18 + 0.675𝑥19 + 0.900𝑥20 + 𝑑𝑃𝐵+ − 𝑑𝑃𝐵

− = 0.4 ∗ 9

The contribution of this goal to the objective function is given by:

𝑤𝑃𝐵

⌈600,00072,000⁄ ⌉

∗ 𝑑𝑃𝐵+ =

0.267

9∗ 𝑑𝑃𝐵

+ Eq. 517

Training the New Generation of Investigators Goal

One of the center’s goals is to fund proposals in which new investigators play an essential role. This will

allow the center to train the new generation of researchers and provide substantial sustainability for keeping

the interest of health professionals to focus on scientific research. For this case, the center aims to have a

total pool of new investigators of at least ten. This can be written as follows:

∑ 𝑁𝐼𝑖𝑥𝑖 + 𝑑𝑁𝐼+ − 𝑑𝑁𝐼

− = 𝑁𝐼𝑚𝑖𝑛 Eq. 518𝑛

𝑖=1

Where NIi is the number of new investigators in proposal i (Appendix B). Thus, the equation for this case

is:

1𝑥1 + 2𝑥2 + 1𝑥3 + 0𝑥4 + 3𝑥5 + 1𝑥6 + 0𝑥7 + 1𝑥8 + 1𝑥9 + 2𝑥10 + 2𝑥11 + 1𝑥12 + 2𝑥13 + 0𝑥14 + 1𝑥15

+ 2𝑥16 + 0𝑥17 + 1𝑥18 + 2𝑥19 + 2𝑥20 + 𝑑𝑁𝐼+ − 𝑑𝑁𝐼

− = 10

The contribution of this goal to the objective function is:

𝑤𝑁𝐼

𝑁𝐼𝑚𝑖𝑛∗ 𝑑𝑁𝐼

+ =0.250

10∗ 𝑑𝑁𝐼

+

113

Catalyzing Multidisciplinarity Goal

The center is highly compromised with promoting scientific research that integrates different health fields.

In order to account for this goal, an interdisciplinarity score is proposed to evaluate how multidisciplinarity

the proposals are (Appendix C).

Thus, the goal constraints associated to multidisciplinarity are:

(𝑀𝐷𝑖 − 1)𝑥𝑖 + 𝑑𝑀𝐷𝑖+ − 𝑑𝑀𝐷𝑖

− = 0 ∀𝑖 Eq. 519

−0.333𝑥1+ 𝑑𝑀𝐷1+ − 𝑑𝑀𝐷1

− = 0

−0.333𝑥3+ 𝑑𝑀𝐷3+ − 𝑑𝑀𝐷3

− = 0

−0.333𝑥4+ 𝑑𝑀𝐷4+ − 𝑑𝑀𝐷4

− = 0

−0.667𝑥6+ 𝑑𝑀𝐷6+ − 𝑑𝑀𝐷6

− = 0

−0.667𝑥7+ 𝑑𝑀𝐷7+ − 𝑑𝑀𝐷7

− = 0

−0.667𝑥9+ 𝑑𝑀𝐷9+ − 𝑑𝑀𝐷9

− = 0

−0.333𝑥13+ 𝑑𝑀𝐷13+ − 𝑑𝑀𝐷13

− = 0

−0.333𝑥14+ 𝑑𝑀𝐷14+ − 𝑑𝑀𝐷14

− = 0

−0.333𝑥15+ 𝑑𝑀𝐷15+ − 𝑑𝑀𝐷15

− = 0

−0.333𝑥19+ 𝑑𝑀𝐷19+ − 𝑑𝑀𝐷19

− = 0

The contribution of this goal to the objective function is given by:

𝑤𝑀𝐷


𝐵𝑈𝐷𝑎𝑣𝑔|0.1⁄ ⌉∗ ∑ 𝑑𝑀𝐷𝑖

+𝑛

𝑖=1 Eq. 520

0.290

⌈600,00072,000⁄ ⌉

∗ ∑ 𝑑𝑀𝐷𝑖+

20

𝑖=1=

0.290

9∗ ∑ 𝑑𝑀𝐷𝑖

+20

𝑖=1

114

5.3.4 Objective function

The objective function for this goal programming model is:

𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝑍 =0.193

⌈600,00072,000⁄ ⌉

∗ ∑ 𝑑𝑟𝑖+

20

𝑖=1+

0.267

⌈600,00072,000⁄ ⌉

∗ 𝑑𝑃𝐵+ +

0.250

10∗ 𝑑𝑁𝐼

+

+ 0.290

⌈600,00072,000⁄ ⌉

∗ ∑ 𝑑𝑀𝐷𝑖+

20

𝑖=1

5.3.5 Set of system constraints

Budget Constraint

The allocated budget cannot exceed the maximum available budget of 600,000.

75,000𝑥1 + 64,000𝑥2 + 120,000𝑥3 + 120,000𝑥4 + 94,000𝑥5 + 45,000𝑥6 + 59,000𝑥7 + 68,000𝑥8

+ 80,000𝑥9 + 92,000𝑥10 + 55,000𝑥11 + 39,000𝑥12 + 55,000𝑥13 + 46,000𝑥14

+ 39,000𝑥15 + 46,000𝑥16 + 78,000𝑥17 + 79,000𝑥18 + 101,000𝑥19 + 115,000𝑥20

≤ 600,000

Quality Constraints

(𝑄𝑖 − 𝑄min)𝑥𝑖 ≥ 0 for all 𝑖

Qi values are obtained using the current scoring sheet used to evaluate proposals (Appendix C). Let us also

assume that the minimum quality required is 65.

Mutually exclusive proposals

There are some projects that due to their similarity or other specific attributes, cannot be simultaneously

selected to be funded. In this case, for example, the center could limit the number of proposals that address

diabetes as their primary area of research. According to the list of proposals (Appendix B), there are three

of them whose main interest is diabetes. However, according to the center’s strategy, at most two proposals

on diabetes can be funded. The mathematical formulation of this is given as follows:

𝑥1 + 𝑥5 + 𝑥12 ≤ 2

115

Similarly, the center limited the number of asthma proposals to be funded to 1. This constraint can be

written as:

𝑥3 + 𝑥8 ≤ 1

Additionally the center limited the number of liver related proposals to be funded to 1. This constraint can

be written as:

𝑥13 + 𝑥17 ≤ 1

Proportion of funded proposals

One of the strict constraints imposed by the leaders of the center is to consider a balance in the funding

given to the health departments. In this case, the amount of funding given to any department cannot exceed

three times its representativeness ratio. This ratio can be obtained from the graduate school enrollment data

by department or program. For example, PIs of proposals 2, 8, 19 and 20 are part of the same health

department (Medicine). Then, this constraint can be expressed as follows:

𝐵𝑈𝐷2𝑥2 + 𝐵𝑈𝐷8𝑥8 + 𝐵𝑈𝐷19𝑥19 + 𝐵𝑈𝐷20𝑥20

𝐵𝑈𝐷𝑚𝑎𝑥≤ 3 ∗

𝑁𝑈𝑀𝑠𝑡𝑢_𝑙

𝑇𝑂𝑇𝐴𝐿𝑠𝑡𝑢

where:

NUMstu_l is the number of students in department l

TOTALstu is the total number of students

According to the table shown in Appendix E, the following constrains must be considered:

Department of Biochemistry and Molecular Biology:

75,000𝑥1 + 120,000𝑥4

600,000≤ 3 ∗

30

264

Department of Cellular and Molecular Physiology:

59,000𝑥7

600,000≤ 3 ∗

9

264

116

Department of Comparative Medicine:

94,000𝑥5

600,000≤ 3 ∗

7

264

Department of Humanities:

55,000𝑥11

600,000≤ 3 ∗

15

264

Department of Medicine:

64,000𝑥2 + 68,000𝑥8 + 101,000𝑥19 + 115,000𝑥20

600,000≤ 3 ∗

75

264

Department of Microbiology and Immunology:

120,000𝑥3 + 92,000𝑥10

600,000≤ 3 ∗

22

264

Department of Neural and Behavioral Sciences:

79,000𝑥18

600,000≤ 3 ∗

12

264

Department of Obstetrics and Gynecology:

39,000𝑥15

600,000≤ 3 ∗

8

264

Department of Pediatrics:

45,000𝑥6 + 55,000𝑥13

600,000≤ 3 ∗

16

264

Department of Psychiatry:

80,000𝑥9 + 46,000𝑥14

600,000≤ 3 ∗

12

264

Department of Public Health Sciences:

39,000𝑥12 + 46,000𝑥16

600,000≤ 3 ∗

45

264

117

Department of Radiology:

78,000𝑥17

600,000≤ 3 ∗

13

264

Minimum number of funded proposals

At least nine proposals are expected to be funded by the center.

∑ 𝑥𝑖 ≥ 9 for 𝑖 = 1,… ,20𝑛

𝑖=1

Cross-campus required

In order to support cross-campus collaboration, the center is highly committed to have at least two proposals

that incorporate departments from different campuses.

𝑥1 + 𝑥5 + 𝑥7 + 𝑥8 + 𝑥13 + 𝑥16 + 𝑥17 + 𝑥19 + 𝑥20 ≥ 2

5.4 Results of Proposal Selection

The GP model was solved using the software LINDO 6.1 (See Appendix F for the formulation). The

solution was obtained after 235 iterations with an objective function of 0.26931E-01. The mix of proposals

that provides the maximum value (minimize deviations from the target goals) is composed of x2, x8, x10, x11,

x12, x15, x16, x18, and x20. All the remaining proposals were not selected as part of the optimal mix

(corresponding xi equals to zero). The solution includes three proposals from the departments of medicine,

two from the department of public health sciences, and one from the departments of microbiology and

immunology, humanities, obstetrics and gynecology, and neural and behavioral sciences. Additionally, the

selected mix of proposals is heterogeneous as it does not have duplicate categories (Heart failure, asthma,

allergy, Alzheimer, diabetes, endocrinology, dementia, drug addiction, and female infertility). The amount

of budget allocated is $597,000.

Two out of the four goals were completely satisfied. None of the undesirable deviational variables for the

goals of potential benefits and training the new generation of investigators were present. The potential

benefits were exceeded in total by 2.825, which represents 0.314 (2.825/9) over the minimum target of 0.4

required. The set of proposals selected includes fourteen new investigators, which is over the ten that was

118

set as the minimum target. The remaining goals had a least one non-zero unwanted deviational variable.

For the risk goal, the deviational variables 𝑑2+ and 𝑑8

+ have values of 0.575 and 0.200 respectively. It means

that although proposals 2 and 8 were selected, they do not fully satisfy the target that was set for the risk

goal. For the goal of catalyzing multidisciplinarity, one deviational variable was non-zero. In this case,

𝑑𝑀𝐷15+ had a value of 0.333. It means that proposal 15 was selected although it does not fully satisfy the

target that was set for the catalyzing multidisciplinarity goal.

The data presented in this case study was simulated to illustrate the usability of the proposed framework.

However, the validation of the model was conducted using real data obtained from a call for proposals

issued by the Penn State CTSI in September 2014. Twenty-four projects were submitted in response to the

call for proposals, and nine were ultimately funded. The results obtained from the model matched seven

out of the nine selected proposals from the pool of 24, indicating that the model serves as an accurate

representation of the final proposal selection criteria.

From a practitioners’ perspective, the GP model is an effective way to quantify the proposals’ merit and

provides an additional source of information to guide the selection process based on their fit to the

organizational goals. The current CTSI proposal selection process involves an informal weighing of the

criteria listed in the GP model, but it lacks the model’s scientific rigor and preciseness. The GP model

provides a defensible, data-driven method to compare proposals and rank them based on specific criteria

related to the CTSI’s strategic mission. Another positive aspect of the GP model is its flexibility; as the

CTSI’s priorities evolve, the GP model can be altered by changing the criteria weighting, or by adding or

removing criteria. Finally, the GP model could be disseminated within the CTSA consortium and improved

based on different institutions’ selection criteria.

5.5 Discussion of Proposal Selection

The budget allocation problem is an essential component to support the strategic goals, and therefore, the

sustainability of the organizations. Accordingly, funding agencies, universities, research institutes and

others in charge of distributing resources to different proposals, should utilize methods to determine a mix

of projects that lead to a maximum value. These methods should be based on evidence generated in both

contexts internal and external.

The GP model presented in this section was found to be a suitable approach for selecting an optimal mix of

proposals based on goals and constraints of the center. As included in the framework, before providing the

119

funding, these results should be validated with the organization’s experts. If the results are unreasonable,

the parameters used should be calibrated to properly model what the center is looking for. Under the current

assumptions, for instance, the department of comparative medicine has only seven graduate students, and

therefore, the proportionality ratio constraints in this case will make it infeasible for their proposals to be

selected. In case the experts’ board could model different exceptions to soften constraints for small

departments, or even make them stricter for large departments as a way of distributing resources more

heterogeneously.

In practical terms, one of the key challenges of this approach is to collect reliable data, especially those

metrics that uses past information to make inferences and formalize lessons learned. In this sense,

mechanisms to monitor funded proposals are needed to fully capture what factors have been successful and

replicate them in future proposal selection processes. In addition, a collaborative effort to share best

practices and relevant proposal’s outcome data between CTSA hubs could generate an important historical

baseline for formalizing best practices, and therefore, optimize the resources of the CTSA consortium. This

is relevant since collecting performance data from proposals is a slow process. Typically, capturing the

impact that a proposal or intervention can have, requires considerable delays. Therefore, it becomes

necessary to monitor the impact of those proposals even after their funded time frame is completed.

Supporting this latent need, a recent NIH request for application (RFA-TR-14-009) encourages tracking

subsequent activity data for at least 10 years as a way to capture interesting data, that could result in a better

allocation of resources in the future.

From the case study presented, it was interesting to observe the fairly robust agreement between the experts’

opinions. Most of the experts agreed that the goal of promoting multidisciplinarity was the relatively most

important. Additionally, the goal associated with the risk of the proposals was consistently scored as the

least important. In addition to revealing and quantifying the relative importance of the four goals under

consideration, in the future it would be relevant to include other goals that the experts consider as important

for the center’s success. In this case for example, according to the experts’ opinion it would be important

to fund proposals that consider a different range of translation efforts (i.e., incorporate basic science as well

as clinical practice and community engagement initiatives). Other goals such as the alignment of the

proposals with the current NCATS and CTSA priorities, potential to build on existing strengths, exploring

emerging research areas, utilization of current CTSI tools and resources, and innovative proposals were

also mentioned as important goals to be considered in the proposal selection process.

The proposed methodology is intended to complement the existing methods to evaluate proposals. The

main contribution of goal programming is the incorporation of strategic goals that support the vision and

120

objectives of centers such as the PSU CTSI. Currently, most proposal evaluation methods do not consider

the fit that those proposals have on the high level objectives of the institutions. Even though proposals can

be evaluated with respect to their research quality, there is still a need for methodologies that connect impact

measures and fit into the institutional strategy.

5.6 Methodology: A Rapid Impact Estimation of Healthcare Interventions

One of the main challenges that researchers face when preparing and presenting a proposal, is to provide a

high-level estimation of the impact of the proposed interventions. Typically, researchers struggle to

properly identify what elements they should consider in order to estimate relevant impact metrics. These

metrics should be clear, coherent, and transparent in order for the evaluator to compare the impact of the

different proposals, and therefore, make informed decisions and resource allocation. To rapidly provide a

high-level estimation of impact, we propose a framework (Rapid high-level impact estimation, RHIE) that

guides the researchers through different relevant questions about the intervention, propose the collection of

relevant information, and recommend cost-effectiveness metrics to be reported. The framework is

composed of five phases; 1) baseline for comparison, 2) understanding impact, 3) QoL data collection, 4)

Intervention characteristics and costs, and 5) robustness and sensitivity. An illustration of the framework is

presented in

In order to properly use the framework, the type of intervention being assessed needs to be understood.

These interventions can be categorized as diagnosis, therapeutic, managing, informing, or/and preventing.

Diagnosis is the identification of the nature of a health disease through the examination of its symptoms.

There are various elements that should be considered in this type of intervention such as potential early

detection, recognizable latent or early symptomatic health states, whether there is an acceptable treatment

for the health disease, existence of other screening or diagnosis tools and their sensitivity and specificity,

determine if current diagnosis tools are acceptable for the population of interest, and costs of potential case

findings, among others.

Therapeutic interventions aim to relieve a health disorder. These treatments are typically associated to

psychological problems in which a patient receives counseling from a therapist.

Managing interventions include those interventions that seek to provide a cure, decrease the short-term

severity of an acute health event (acute management), decrease the severity of a chronic health condition

121

(chronic management), full or partial restoration of physical, mental or social functioning, or reduce pain

and suffering from an incurable health condition (Jamison et al., 2006).

Informing can also be named as health teaching which is the communication of facts, ideas, and skills that

can have an impact on changing knowledge, attitudes, beliefs, behaviors, and practices of the individuals.

Some basic steps included in these interventions are the assessment of the population’s beliefs and

knowledge about a health disease and its risks; personalize risk based behavior; understand message,

channels of communication and training; clarify positive effects to be expected; identify and elicit barriers;

promote awareness and provide reminders; and evaluate progress, among others.

Preventing interventions seek to protect people from developing a disease, health condition, or

experiencing an injury. Typically, a preventing intervention involves regular education, legislation,

screening, and immunization. These interventions can be present at different levels, primary prevention,

secondary prevention, or case management (tertiary prevention).

122

Figure 5-3. Rapid high-level impact estimation (RHIE) framework

123

Baseline for comparison: This phase is necessary to understand and define what the different populations

and treatments under comparison are. For example, in the case of a diagnosis intervention, there are three

main relevant patient populations; those who have not been diagnosed yet, those who are receiving

treatment (typically the best known treatment), and general population. Other sub-categories of populations

could also be included such as best surgical treatment, best drug-based treatment, etc. At the end of this

phase, the patient populations to be compared are identified.

Understanding the impact: An intervention might have different types of impact on people’s health or cost

structure. The objective of this phase is to identify the different impacts that an intervention could

potentially have. The key questions that guide the understanding of the impact are: Is there any difference

in the HRQoL for the two patient populations being compared?, Is there any difference in the progression

of the disease between the two patient populations being compared?, and does the intervention have an

impact on the cost structure or intervention characteristics?. These three questions address the three main

components of cost-effectiveness analysis; mortality, morbidity, and costs. Considering the combination of

responses for the questions, different paths are possible. In cases in which the intervention does not have

an impact on any of the listed components, the intervention should not be considered for further evaluation

(unless there is another good reason to do so). The pathways will result on different sets of information that

should be collected for a high-level impact estimation analysis.

QoL data collection: For those interventions that will impact mortality or morbidity, different sets of

parameters should be collected with the final aim of calculating total QALYs and QALYs gained. There

are two main groups of parameters that can be collected or estimated; those related to the progression of

the disease, and those related to the quality of life. The first group could include parameters such as

mortality, stages of the disease, duration of the stages, progression rate, life expectancy, onset, and other

significant levels of granularity such as age, gender, ethnicity, etc. The second group includes relevant

parameters to obtain the HRQoL. In order to estimate this information, tools such as EQ-5D, SF-36, and

SF-6D could be used (See section 2.5.3). Sources of input to use the HRQoL valuation methods are

literature, small sample size (for rapid estimation), or expert’s opinion. Additionally, a hybrid source can

be used for a more accurate and calibrated estimation. Furthermore, in this section, the evaluator could

decide the time horizon of evaluation as well as the discount rate as required. Typically, most interventions

are estimated using a period of 10 years and a discount rate within the range of 3 – 5%.

Intervention characteristics and costs: In this phase, different cost-effectiveness metrics such as

cost/QALY, ICER, and total QALYs gained by the society, are estimated. Three main groups of parameters

124

are needed to be either collected or estimated. The first group includes information about the population

target. Some of the metrics that could be relevant are: population size under target, percentage of the

population suffering from a disease under different levels of granularity (phase, age, gender, ethnicity, etc.).

The second group includes metrics which are relevant to describe the intervention. For example, a

diagnosis-related intervention could include the coverage intended, access to the diagnosis mechanism,

accuracy of detection of disease, specificity, and sensitivity, among others. The third group of metrics serves

to identify the cost structure behind the health intervention. Based on this information, different cost-

effectiveness metrics can be calculated.

Robustness and sensitivity: This approach is intended to provide a rapid high-level estimation of impact of

a certain health intervention. Naturally, some of the parameters used will present a considerable amount of

uncertainty. In order to provide insights about the robustness of the estimation, sensitivity analyses on those

parameters are recommended. These analyses could be conducted to understand the different behavior of

costs or QALYs under different values of critical parameters. In addition, extreme values for those

parameters could be tested to see how they impact the cost-effectiveness metrics.

Before using this framework it must be noticed that its intention is to provide a quick guidance for high-

level estimation. However, deeper levels of granularity could be explored based on a similar logic. Some

of the most used comparisons between populations, intervention characteristics and costs of interventions

are shown in Table 5-3.

5.7 Case Study: Impact Estimation for Early Detection of Parkinson’s

Disease

In this section, a case study is presented to illustrate how QALYs can be calculated to guide cost-

effectiveness analysis that could later on be incorporated into the GP proposal selection problem as a goal.

The case study used in this section is based on a proposal that seek to use a non-invasive sensor device for

predicting early stage neurological disease progression.

5.7.1 Case study overview

The main hypothesis being tested in the proposal is that patient gait and movement data can inform

prediction models for an early stage detection of neurological anomalies in patients’ wellness. The

investigators propose the use of a non-invasive sensor to capture patients’ gait and movement. The proposed

prediction models are based on data mining techniques. Unsupervised and supervised machine learning

125

algorithms will be used to uncover hidden patterns. In order to study the impact of the intervention presented

in the proposal, we focus only on patients suffering from the Parkinson’s disease.

Table 5-3. Baseline for comparison and relevant parameters by type of intervention

Type of

intervention

Baseline for

Comparison

Intervention characteristics Cost structure

Diagnostic Undiagnosed

population

General population

Population with

existing treatment

Access and coverage

Accuracy of diagnosis

Sensitivity and specificity

Cost per diagnosis

tool

Specialist costs

Maintenance cost

Informing General population

Population at risk

Population with

existing treatment

Access to information and

education sources

Coverage of information and

education sources

Persuasiveness rate

Adoption and maintenance of

recommended habits

Cost of information

and education

mechanism

Reinforcement costs

Incentive for

adoption costs

Preventing General population

Population at risk

Untreated population

Access to preventing and

education programs

Coverage of preventing and

education sources

Adherence rates

Adoption and maintenance of

recommended habits

Cost of informing,

preventing, and

educating

Reinforcement costs

Incentive for

adoption costs

Therapist Untreated population

Population with the

disease or health

condition

Successful rates of the

therapy

Access to therapy

Coverage of therapy

Adherence rates

Maintenance recommended

habits

Cost of therapy

Reinforcement costs

Managing Untreated population

(when no treatment

exists)

Population with

existing treatment

Population with

proposed treatment

Medical/Surgical/Alternative

treatment successful rates

Access to case management

intervention

Eligibility for the treatment

Successful and adherence

rates

Cost of treatment

(medicine, surgical,

others)

Follow-up and

monitoring costs

126

5.7.2 Parkinson’s disease background

Parkinson’s disease is a progressive neurodegenerative disorder that typically affects the elderly population.

Only in the U.S., nearly one million people live with PD. Additionally, each year between 50,000 to 60,000

new cases are diagnosed (Parkinson's Disease Foundation, 2015). According to the National Parkinson

Foundation, about 6,400 people with PD die each year due to poor care (National Parkinson Foundation,

2014). This is also supported by studies that have shown that about 60% of individuals suffering from PD

do not get the expert care that they need (Landro, 2014). Some of the key characteristics of PD patients are

tremor, rigidity, bradykinesia, gait disturbances, and postural instability (Bakheit, 1995, Gelb et al., 1999).

As argued in the literature, using standardized screening procedures with acquisition of early motor

symptoms could help detecting individuals with high risk of PD (Gaenslen and Berg, 2010). A recent study

with a sample of 115 PD patients showed that about 99% of the patients consulted a physician because of

motor symptoms. In the pre-diagnosis phase (≤ 2 years) mild motor signs including asymmetric

bradykinesia and rest tremor were significant by the PD patients (Walter et al., 2013). Moreover, motor

signs such as asymmetry have been found to have a high sensitivity (88%), specificity (54%), and positive

predictive values (85%) for the diagnosis of PD (Busse et al., 2012).

Studies have estimated that when PD is diagnosed according to clinical criteria, more than 50% of the

dopaminergic cells have been degenerated (Fearnley and Lees, 1991). Unfortunately, the early detection of

PD is usually difficult (Bakheit, 1995). In addition, the accuracy of the diagnosis of PD has remained limited

(Gaenslen and Berg, 2010). According to Schrag et al. (2002) about 15% of patients that are diagnosed with

PD do not fulfil the clinical criteria for this disease. Moreover, about 20% of patients with PD that have

already visited medical attention have not been diagnosed.

Stages of Parkinson’s disease

There are different rating scale tools to determine the stage of PD. Most of these tools combine the severity

of movement symptoms and the impact of the disease on the individual’s daily activities. The two most

commonly used scales are the Hoehn and Yahr (Hoehn and Yahr, 1998) and the United Parkinson’s disease

rating scale (UPDRS). The Hoehn and Yahr scale has been widely used to classify PD patients into five

different stages depending on the severity of the effects of the PD on the patient. The UPDRS is a

questionnaire composed of 42 multiple-option questions categorized into four groups; mentation, behavior

and mood; activities of daily living; motor examination; and complications of therapy (Movement Disorder

Society Task Force on Rating Scales for Parkinson's Disease, 2003). Some of the main characteristics of

the Hoehn and Yahr stages are presented in Table 5-4.

127

Onset and Mortality

The average onset age for individuals suffering from PD is around 65 years, however some patients suffer

from a much earlier onset (Ishihara et al., 2007, Forsaa et al., 2010). In terms of mortality, studies have

shown that PD is associated to increased mortality compared to the general population (Bennett et al., 1996).

There are some estimations that patients with PD have a 2-fold to 5-fold more risk of mortality compared

to non-demented elderly people (Louis et al., 1997). An early detection and prevention of motor progression

is expected to be a promising strategy to increase life expectancy of patients with PD (Forsaa et al., 2010),

however, studies have not been able to demonstrate this.

According to Ishihara et al. (2007), the life expectancy of PD patients whose onset is between 25 and 39

years old, is 38 years. This is low if compared with the general population for which life expectancy is 49.

For onset between 40 and 64 years old the life expectancy of a PD patient is 21. On the other hand, the life

expectancy of the general population is 31. Similar conclusions were also provided for the age at time of

death. Life expectancy and age at the time of death were significantly worse for patients suffering from PD.

However, others have argued that given the pharmacological advances, most PD patients are not associated

to shortened life expectancy if managed well by specialists (Greener, 2009).

Table 5-4. Hoehn and Yahr stages and characteristics

Hoehn and

Yahr stage

Characteristics

Stage 1 1. Mild movement symptoms on one side of the body

2. Symptoms may be inconvenient but they do not affect the individual’s daily activities

3. Usually presents with tremor of one limb

4. Friends and relatives may notice changes in the individual’s posture, walking, or facial

expressions

Stage 2 1. Symptoms begin to affect both sides of the body

2. Minimal disability

3. Posture and gait affected

Stage 3 1. Body movements are slower

2. Problems with balancing and coordination may develop

3. Freezing episodes of some parts of the body may occur

4. Generalized moderately severe dysfunction

Stage 4 1. Severe symptoms

2. Walking is very limited

3. Inability to live alone

4. Tremor may be less than earlier stages

5. Rigidity and bradykinesia

Stage 5 1. Most of the time confined to bed or wheelchair

2. Inability to stand or walk

3. Cognitive problems may appear. They include hallucinations and delusions

4. Invalidism complete

128

Existing treatments and their cost-effectiveness

Although there is no cure for PD, treatments can decrease its symptoms. For example the use of carbidopa

and levodopa are found to significantly reduce the symptoms for one to five years in 25% of the patients.

In addition to carbidopa and levodopa, several other drug treatments have been developed in the past two

decades (Dams et al., 2011). Furthermore, surgical interventions have also been used for treating PD. The

deep brain stimulation (DBS) surgery has been found to be very promising. Studies estimated that DBS

provides additional 0.72 QALYs at an additional cost of $35,000 compared with the best medical

management known. Therefore, the ICER is $49,000 (Tomaszewski and Holloway, 2001). A detailed

review of cost-effectiveness analysis of treatments for Parkinson’s disease can be found in Dams et al.

(2011). This review includes evaluations of drug treatments, surgical options, and diagnosis.

5.8 Results of Rapid Impact Estimation

After understanding the characteristics of the disease, the RHIE framework can be used to estimate the

impact of the proposed intervention. According to the characteristics of this disease, there are treatments

that can have an impact on the quality of life of treated patients with respect to untreated/undiagnosed

individuals. However, the existing treatments do not have an impact on the progression of the disease.

Therefore, according to the RHIE framework, data is needed with respect to the quality of life of

undiagnosed/untreated patient and diagnosed/treated patients according to the required level of granularity.

In this case, the best grouping or level of granularity identified was to categorize PD patients according to

their HY stage. However, other more detailed categories or levels such as age, gender, and ethnicity could

have been used. Nevertheless, for a rapid estimation this data was not easily available.

5.8.1 Potential QALYs gained

Zhao et al. (2010) estimate the progression of PD by analyzing the transit time from one stage to another

using the HY scale. Their results indicate that the median time to transit from stage 1 to 2, from 2 to 2.5,

and from 2.5 to 3 are 20, 62, and 25 months respectively. Additionally, the transit times in more advanced

stages were 24 and 26 months to move from stage 3 to 4 and 4 to 5 respectively. Therefore, the overall

mean time from disease onset to phase 5 is about 13 years. This value is aligned to what has been found by

other studies. According to Hoehn and Yahr (1998) the median delays before reaching stage 4 and stage 5

are 9 and 14 years respectively. However, there is a high heterogeneity among patients. For example, about

one third of the patients with disease duration over 10 years are still in phases 1 and 2. In addition, it must

129

be noticed that, typically, treatments are not offered in stage 1. In order to estimate the HRQoL for each

stage, a literature review approach was used (Table 5-5). As also recommended in this framework, this

information could have been obtained from experts’ opinion or estimated from a small population sample.

Table 5-5. HRQoL of treated vs untreated PD patients by HY stage

Hoehn and

Yahr stage

Median

duration1

Cumulative

duration

Treated EQ-

5D SI2

Untreated EQ-

5D SI5

Stage 1 24 24 0.9 0.9

Stage 2 62 86 0.753 0.6

Stage 2.5 25 111 0.6 0.4

Stage 3 24 135 0.3 0.25

Stage 4 25 160 0.2 0.2

Stage 5 86 168 04 0 1 Zhao et al. (2010) 2 (Schrag et al., 2000) 3 Estimated 4 at the end of the stage 5 Estimated based on

symptoms 6 Estimated based on life expectancy of 79 years and onset at age 64.9.

From the data it can be said that on average, patients that were diagnosed in phase 2 could have received

treatment to increase their quality of life during 43.5 months. Additionally, those patients diagnosed in

phase 3, were untreated during all phase 2 (87 months) plus 12 months in phase 3. The potential QALYs

gained by individuals suffering from PD that are detected in stages 2 and 3 are shown in Figure 5-4.

Figure 5-4. QALYs gained by currently undiagnosed individuals

130

From the analysis, patients that were diagnosed in stages 2 and 3 lost 0.18125 (2.175months) and 0.8875

(10.65 months) QALYs respectively. An early detection of those patients can therefore add QALYs into

the society. It must be noted that this estimations did not include a discount rate to estimate QALYs in

present value.

5.8.2 Cost per QALY

The cost of this diagnosis intervention will mostly depend on the promotion mechanism to make the

diagnosis available for the population. For instance, at the most granular level, this device could be available

at the household level, in which the average household size in the U.S. is 2.58 (Census, 2010). However,

most families do not have access to this device (multi-sensor). Another, more realistic implementation

setting might be to install devices in places that reach fairly large groups of people (hospitals, commercial

centers, schools, universities, etc.). In such cases, the number of devices needed to reach potential PD

patients decreases substantially.

For the base case, it was estimated that the cost to detect one case is USD 10,000. In addition, the cost per

installed device was estimated to be USD 300. Accordingly, 33 devices are needed to cover a group of 82

individuals each to detect one case of PD in stages 2 or above. In other words, the number of screened

individuals to detect a PD individual in stages 2 or above is 2,722. These estimates include the assumptions

of sensitivity (90%), coverage (20%), undiagnosed population in stages 2 and 3 (65.2%). These estimates

lead to a cost-effectiveness of the intervention of 29,370 USD/QALY. Under these parameters and typically

used cost-effectiveness thresholds, this intervention is said to be cost-effective.

5.8.3 Overall impact on society

In order to estimate the overall impact on population of this health intervention, data with respect to the

target population, diagnosis parameters, and stages in which the disease is currently being diagnosed are

needed. This data is presented in Table 5-6. It must be noted that the estimation will include only the PD

population which is currently undiagnosed and not the new cases. In order to include the yearly new cases,

a time horizon of ten years would be recommended.

In the U.S. it has been estimated that approximately 200,000 individuals suffering from PD have not been

diagnosed (Huse et al., 2005). From the information presented in Table 5-6, it can be estimated that about

101,000 out of the 200,000 undiagnosed PDs are in stage 2. On the other hand, approximately 29,400

undiagnosed PDs are in stage 3. The diagnosis intervention presented in the proposal intends to target 20%

of the entire population and have an accuracy of detection of 90%. Therefore, it is estimated that the

potential new PD cases diagnosed in stages 2 and 3 are 18,180 and 5,292 respectively. In terms of QALYs,

131

undiagnosed PDs in stage 2 could gain 0.18125 QALYs. In other words 3,295 QALYs in total. For

undiagnosed PDs in stage 3, the society could gain 4,697 QALYs in total (0.8875 QALYs/patient). In

conclusion, 7,992 QALYs can be added into the society.

Table 5-6. Data for estimating overall impact on society

Population

Parameter Value Reference

Total U.S. population 319,436,548

inhabitants

http://www.census.gov/popclock/

Total undiagnosed PD 200,000 inhabitants Huse et al. (2005)

Annual PD diagnoses 60,000 (Parkinson's Disease Foundation,

2015)

% PD diagnosed in stage 2 50.5%1 Muslimović et al. (2007)

% PD diagnosed in stage 3 14.7% Muslimović et al. (2007)

Diagnosis mechanism parameters

Percentage of population to target with

the diagnosis tool

20%

Accuracy - Sensitivity 90% Previous analysis

Relevant calculated parameters (population scope)

Undiagnosed PD in stage 2 101,000

Undiagnosed PD in stage 3 29,400

Potential new cases diagnosed in stage 2 18,180

Potential new cases diagnosed in stage 3 5,292

Shifted population to new diagnosis

mechanism

12,000

1 Include patients in Stages 2 and 2.5

5.8.4 Sensitivity analysis

One of the main drawbacks of using high-level estimations is that some of the parameters used are subject

to significant uncertainty. In this regard, sensitivity analysis on those parameters provides a clear vision of

how robust the results in terms of cost-effectiveness are. While economic models are useful to approximate

the estimation of impact under given conditions, uncertainty cannot be removed. One-way sensitivity

analyses are intended to provide an assessment of the impact of changes in the parameters and how those

132

changes can have an impact on the model’s conclusions. Additionally, sensitivity analysis can help the

researcher to identify what are the key drivers of the model’s results.

One-way sensitivity

According to the estimations, undiagnosed individuals in stages 2 and 3, could gain 0.1813 and 0.8875

QALYs respectively. In order to facilitate the interpretation of the sensitivity analysis, a weighted average

of QALYs gain will be used. Therefore, on average an undiagnosed PD individual could gain 0.3405

QALYs if properly treated. A sensitivity analysis was conducted to account for the uncertainty that could

be present when estimating QALYs (Figure 5-5). The base case considers a cost per case of USD 10,000,

which under the current estimations is translated to USD 29,370 per QALY. This value is just over the

threshold for interventions considered to be very cost-effective (USD 25,000). It can be said that an

intervention with a cost per case of USD 10,000 is very cost-effective for values of QALYs gained of 0.4

and above, reaches the typical cost-effectiveness for ranges of QALYS gained between 0.2 and 0.4, and

becomes ineffective for values of QALYS gained of 0.1 or below. From these results, it can be said that the

intervention is robustly cost-effective under the uncertainty or errors of estimation of QALYs gained. The

same figure includes the sensitivity for three levels of cost per case. If the cost per case were double the

initial estimate (i.e., USD 20,000 per case), then in order for the intervention to be very cost-effective, the

QALYs gained must be 0.8 or above, to be cost-effective under the typical threshold the QALYs gained

must be between 0.4 and 0.8, and finally, the intervention is ineffective if the QALYs gained are 0.2 or

below. Finally, for an extreme cost per case of USD 30,000, the intervention will be very cost-effective if

the QALYs gained are 1.2 or above, typically cost effective for a range of QALYs gained between 0.6 and

1.2, and ineffective for values of QALYs gained below 0.3. These results also show that the intervention is

robust in terms of cost-effectiveness. Even though the cost per case is three times the initial estimate, the

intervention will still be cost-effective under the base case estimate of QALYs gained (0.34048). Under the

estimates for the QALYs gained, the cost per case could be increase up to USD 34,048 to maintain the cost

per QALY above the maximum cost-effectiveness threshold of 100,000 USD/QALY.

133

Figure 5-5. Cost-effectiveness sensitivity for QALYs gained and cost per diagnosed case

A tornado chart is presented in Figure 5-6 to show how sensitive the cost-effectiveness of the intervention

is given a ± 20% change on various relevant parameters including cost per case, cost per installed device,

device coverage, QALYs gained, accuracy, and sensitivity of the diagnosis tool. Out of these list, the last

two parameters have the same impact on the cost-effectiveness of the intervention. If one of these

parameters decreases by 20% (from 0.90 to 0.72), the cost per QALY increases to 45,891, which represents

an increment of 56%. On the other hand if one of these parameters increases up to 1 (approximately 11%),

the cost per QALY decreases to 22,028. A sensitivity for the QALYs gained parameter showed that if it

increases 20% (0.409), the cost-effectiveness becomes 24,475 USD/QALY. Additionally, if the QALYs

gained is decreased by 20% (0.272), the cost-effectiveness becomes 36,713 USD/QALY. Finally, the

parameters cost per case, cost per installed device, and coverage of the diagnosis device have the same

impact. If one of these parameters decreases by 20%, the cost per QALY decreases to 23,496. On the other

hand if one of these parameters increases by 20%, the cost per QALY increases to 35,244. From these

analysis, it can also be said that the intervention is fairly robust. Under a ± 20% one-way sensitivity for the

parameters shown, the intervention remains cost effective (< 50,000 USD/QALY).

134

Figure 5-6. Cost-effectiveness tornado sensitivity for relevant parameters

The selection of the implementation design plays an important role on the cost-effectiveness of this

diagnosis intervention. The base case assumed a medium reachability for each device (82 individuals per

device), however, other designs such as household level, in which the reachability per device is on average

2.58 individuals, are non-cost-effective. In this scenario, the cost per case becomes USD 319,691, which

lead to a cost of USD 938,940 per QALY. On the other hand, if the implementation design is capable of

reaching large groups per device (e.g., hospitals, supermarkets, drug stores, etc.), the intervention becomes

very cost-effective. For example, if the reachability per device can be increased to 200 individuals, the cost

per case decreases to USD 4,140, which is equivalent to a cost of 12,112 USD/QALY.

5.9 Discussion of Rapid Impact Estimation

The RHIE Framework for cost-effectiveness estimation of healthcare interventions was used to rapidly

estimate the potential impact of the proposal presented. One of the main challenges of the proposals is to

quantify their impact on people’s health. In this sense, cost-effectiveness analysis encourages researchers

to think about their proposed interventions in terms of the value that can be added into the society. It is

important, however, that the estimations isolate the impact of the proposed intervention to avoid double

counting the benefits of other existing factors. For example, the case study proposed a diagnosis

intervention, and therefore, only the benefits given by this intervention should be counted towards the

estimation of its impact and not the benefits of existing interventions to treat newly diagnosed patients.

135

It is also important to understand that some interventions might be highly benefited from other

interventions. For instance, a diagnosis intervention could potentially increase its benefits if a promoting or

informing intervention is also put in place. In such cases, synergies might also be estimated using a cost-

effectiveness analysis approach to demonstrate the value of investing in complementary interventions

instead of on a particular effort. In this regard, one of the main challenges of the rapid high-level impact

estimation was to properly estimate the coverage of each installed diagnosis device.

Although in the case study presented, the most representative level of granularity was the stage of the

disease, other levels such as age, gender, and ethnicity might be more relevant in other health diseases.

Therefore, it is encouraged to identify a level of granularity that provides reliable results for a high-level

estimation. It should also be noticed that the main aim of the rapid estimation framework was to guide the

estimation of impact with an emphasis on initial phases of investigation such as research proposals.

Nevertheless, the framework can be easily adjusted to incorporate a deeper level of detail and accuracy in

the estimation of impact.

Sensitivity analysis on relevant parameters and different scenarios is highly encouraged. Adding sensitivity

analysis to cost-effectiveness analyses are useful to evaluate the robustness of the estimations presented

and key drivers of cost-effectiveness of interventions. Additionally, it adds credibility in terms of providing

the range of conditions under which an intervention will still be justifiable from a cost versus value

perspective. Testing scenarios is also encouraged as a way to anticipate potential changes in the future. For

instance, promising drugs that shift the progression curve of PD are being tested. Therefore, having a

diagnosis mechanism such as the presented in the case study incorporates preparedness and could have a

larger impact in an scenario in which a treatment to impact PD progression is developed.

From a practitioner’s perspective, the proposed RHIE framework is expected to formalize and demonstrate

some unproven believes in health sciences. For instance, it is believed that the specialists in neurological

disorders might provide a more accurate treatment for PD patients than general practitioners. However,

proving this statement is typically difficult for practitioners. In response, the RHIE could provide a fair

quantification of the differences of impact between the treatments proposed by each group of physicians.

Therefore, such analysis could justify or not, the efforts to increase the capacity of specialists on certain

diseases. Another practical instance of the potential benefits of using the RHIE framework is to evaluate

the impact of complementary interventions or treatments. According to Dr. Xuemei Huang, (Director of

the Hershey Brain Analysis Research Laboratory for Neurodegenerative Disorders at the Penn State

Hershey Medical Center) it would be very interesting to evaluate the complementary benefits of multiple

136

treatments for PD patients such as physical therapy in addition to a drug-based treatments. The RHIE tool

provides a suitable framework for such evaluations.

5.10 Incorporating Economic Evaluation into the Proposal Selection Problem

As previously argued, CEA is especially useful when comparing interventions that address different types

of health diseases or risks (Jamison et al., 2006). Consequently, scarce resources can be allocated in such a

way that if they are distributed using cost-effectiveness principles, more health improvements are generated.

Incorporating strategic, health, and financial metrics in the proposal selection provides a robust mechanisms

to select a mix of interventions that is balanced around the different goals of an organization. In this regard,

cost-effectiveness metrics can be incorporated as an additional goal into the GP model previously

formulated and discussed.

For instance, the organization can have as a goal to select interventions that meet the typical cost-

effectiveness thresholds. This can be formulated as:

𝐶𝐸𝑖𝑥𝑖 + 𝑑𝐶𝐸𝑖+ − 𝑑𝐶𝐸𝑖

− = 𝐶𝐸𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 ∀𝑖 Eq. 521

where:

CEi Cost-effectiveness (USD/QAL) associated to project i

CEthreshold Threshold used in CEA. Typically 50,000 USD/QALY

𝑑𝐶𝐸𝑖− Negative deviational variable associated to cost-effectiveness of proposal i

𝑑𝐶𝐸𝑖+ Positive deviational variable associated to cost-effectiveness of proposal i

In this case, contribution of this goal could be expressed as:

𝑤𝐶𝐸


𝐵𝑈𝐷𝑎𝑣𝑔|0.1⁄ ⌉𝐶𝐸𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑

∗ ∑ 𝑑𝐶𝐸𝑖− Eq. 522

𝑛

𝑖=1

where:

wCE weight of cost-effectiveness goal

137

Some other alternatives to incorporate cost-effectiveness as a goal could be, for instance: 1) Limiting the

number of proposals that are over the threshold, 2) Targeting for the average cost-effectiveness of the mix

selected, and 3) Selecting interventions that are very cost-effective, among others. These alternatives can

be formulated as:

1) Limiting the number of proposals that are over the threshold

𝑥𝑖 + 𝑑𝐶𝐸+ − 𝑑𝐶𝐸

− = 𝑁𝐶𝐸𝑚𝑎𝑥 ∀𝑖 𝑖𝑛 𝑤ℎ𝑖𝑐ℎ 𝐶𝐸𝑖 > 𝐶𝐸𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 Eq. 523

where:

𝑁𝐶𝐸𝑚𝑎𝑥 Maximum number of proposals that are allowed to be over the cost-effectiveness threshold

𝑑𝐶𝐸− Negative deviational variable associated to the cost-effectiveness goal

𝑑𝐶𝐸+ Positive deviational variable associated to the cost-effectiveness goal

2) Targeting for the average cost-effectiveness of the mix selected

∑ 𝐶𝐸𝑖𝑛𝑖=1 𝑥𝑖


𝐵𝑈𝐷𝑎𝑣𝑔|0.1⁄ ⌉+ 𝑑𝐶𝐸

+ − 𝑑𝐶𝐸− = 𝐶𝐸𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 ∀𝑖 Eq. 524

3) Selecting interventions that are very cost-effective

𝑥𝑖 + 𝑑𝐶𝐸+ − 𝑑𝐶𝐸

− = 𝑁𝐶𝐸𝑣𝑒𝑟𝑦𝐶𝐸 ∀𝑖 𝑖𝑛 𝑤ℎ𝑖𝑐ℎ 𝐶𝐸𝑖 < 𝐶𝐸𝑚𝑖𝑛𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 Eq. 525

𝑁𝐶𝐸𝑣𝑒𝑟𝑦𝐶𝐸 Minimum number of proposals that are considered to be very cost-effectiveness.

5.11 Conclusions

The models proposed in this chapter seek to provide guidance to decision makers and researchers in

healthcare fields. Healthcare decision makers in charge of the resource allocation process face various

challenges regarding the trade-offs between conflicting objectives. In order to overcome some of these

challenges, a GP formulation is offered as a guidance to incorporate the different goals into a single model.

Additionally, the model is able to incorporate the strategy of the organization as a way of selecting the best

courses of action to aim the achievement of long-term goals. In particular, the GP model can be used in the

138

proposal selection problem to select the best mix of healthcare interventions. On the other hand, this chapter

offers a framework that can be used by researchers to rapidly estimate the impact of healthcare interventions

under a cost-effectiveness analysis umbrella. As argued by the NIH and the WHO, cost-effectiveness

analysis are useful to provide a coherent baseline for comparison among different interventions that seek to

improve people’s health. Accordingly, the provision of structured information to quantify the potential

impact of an intervention is highly needed to justify its implementation. Hence, cost-effectiveness metrics

are helpful for healthcare decision makers to evaluate and prioritize funding effectively.

The models and frameworks that were presented relay on data-driven information. Consequently,

organizations are encouraged to monitor and track data that can be used as a way to formalize and

operationalize lessons learned. For instance, the CTSA program has encouraged its CTSA hubs to track

information for at least 10 years. Additionally, sharing information of lessons learned among CTSA hubs

is a key priority to guide the use of the resources in a wiser manner. This aims to generate a better capacity

of impacting people’s health. Future work along these lines include the use of more sophisticated techniques

such as big data and machine learning to extract information from big data sources.

139

Chapter 6

CONTRIBUTIONS AND FUTURE WORK

This chapter summarizes the main contributions of this dissertation as well as future research lines that can

be based upon the work that was presented to expand the quality improvement research approaches. The

areas of contribution were categorized into four groups: 1) identifying key drivers and prioritizing efforts;

2) closing existing gaps; 3) engaging the participation of health professionals; and 4) guiding the strategy

of healthcare organizations.

It has been extensively reported that data-driven decision tools are needed for assuring the efficiency and

effectiveness of the decision making process. This is especially important for healthcare organizations

which are typically characterized by their fragmentation, dynamism, and complexity (Reid et al., 2005). In

order to face these challenges, this dissertation explores the potential contribution of traditional and non-

traditional quality improvement tools to support the understanding and analysis of complexity of healthcare

systems, in particular, the translational research process. Consequently, a more comprehensive and

inclusive research-oriented quality improvement approach can be developed.

One of the commonalities of the different methods that were explored in this dissertation is that all of them

are data-driven and can guide the allocation of efforts. This supports the aims of the NIH, IOM, and NAE

that seek to promote and enhance the use of data-driven approaches to optimize the use of scarce healthcare

resources. Consequently, healthcare decision makers have access to more robust tools and guidance for an

effective allocation of efforts and resources.

140

6.1 Identifying Key Drivers and Prioritizing Efforts

Generating frameworks to identify key drivers of improvement and prioritize efforts is one of the key

contributions of this dissertation. Quality improvement research frameworks were provided to be used by

decision makers and researchers at various levels in a healthcare organization. Given the relatively scarce

resources in healthcare fields, in order to maximize value, efforts must be prioritized according to their

contribution (Ockene et al., 2007). In Chapter 3, the combined QFD-AHP approaches served to identify the

key drivers in translational research as well as quantifying the importance of each operational step along

this process. From a health policy perspective, the results provide guidance for decision makers to

understand the dynamic of translational research and how different technical requirements are needed to be

prioritized depending on the translational phase. From these results, multi-disciplinary collaboration was

found to be one of the most important requirements for translational research efforts to succeed. In response,

Chapter 4 provided a comprehensive tool for evaluating collaboration. The insights obtained from this

chapter inform decision makers about the current status of collaboration and help understanding its gaps.

Additionally, depending on the type of network intervention, the SNA results aid to identify how to impact

the network efficiently. For instance, dissemination of knowledge can be conducted through the leaders or

influencers to accelerate the rate of information flow. Hence, this chapter provided another clear example

of how to prioritize efforts. Finally, in Chapter 5 a multiple criteria model was presented for the proposal

selection problem. This model aimed to select the best mix of proposals that maximize the whole value

while meeting strategic objectives of the organization. Moreover, the cost-effectiveness analysis presented

is helpful for guiding the resource allocation in a more effective manner. All three chapters provided tools

to prioritize efforts that result in a wiser distribution of resources.

6.2 Closing Existing Gaps

This dissertation contributed to the identification of current gaps and understanding courses of action that

could close them. Additionally, this dissertation expanded the literature that seek to build a partnership

between engineering and healthcare enterprises through an expanded quality improvement research toolkit.

These efforts respond to the three main aims expressed by the NAE and IOM (Reid et al., 2005):

“1) identify engineering applications that could contribute significantly to improvements in

healthcare delivery in the short, medium, and long terms; 2) assess factors that would facilitate or

impede the deployment of these applications; and 3) identify areas of research in engineering and

other fields that could contribute to rapid improvements in performance.”

141

Chapter 3 provided a robust framework for identifying and quantifying the importance of the different

operational steps and technical requirements needed. Having a clear mapping of the translational research

process will, certainly, help to build a stronger coordination between the different translational phases.

Consequently, the agreements generated can build a strong capacity to accelerate the pace at which new

discoveries become clinical practice, which is one of the main gaps in translational research. The framework

provided in Chapter 4 contributed to a better identification of current collaboration gaps. The visual and

mathematical-based metrics are highly valuable for designing actions to close those gaps. Finally, in

Chapter 5, mechanisms to close the gaps related to the understating of impact of healthcare interventions

and quantifying organizational strategic goals are provided with the ultimate aim of a better allocation of

resources.

These topics covered are aligned to the NAE-IOM collaboration effort that attempts to bridge the

knowledge and awareness that divides healthcare professionals from potential partners in systems

engineering. Moreover, the contributions made in this dissertation are aligned to what has been requested

by the CTSA to each one of the CTSA hubs in terms of a better coordination, collaboration, and use of

data-driven techniques. Those challenges are considered to be main gaps in translational research.

6.3 Engaging the Participation of Health Professionals

It has been widely studied that one of the main elements contributing to the inefficiency in healthcare

systems is the lack of a proper implementation phase. One of the main barriers concerning implementation

is the practically null collaboration between health professionals, health managers, and engineers. In

response to this, all three body chapters of this dissertation include the participation of health professionals.

This participation encompassed mechanisms for capturing the “voice” of different healthcare stakeholders,

quantifying opinions, and engaging them to use the frameworks presented in this dissertation. These points

are critical for the development of trust, shared understanding, and the generation of short and long-term

partnerships between engineering and healthcare. In Chapter 3, for instance, healthcare stakeholder’s

opinion was incorporated to identify key elements in translational research and quantify their importance.

In Chapter 4, healthcare stakeholders were integrated to provide meaning to the collaboration network

developed. Finally, in Chapter 5, healthcare managers’ opinions were used for developing the multiple-

criteria decision making model based on the current goals of the organization that were used as case study.

Moreover, the RHIE framework was presented to health specialists to develop a shared understanding of

the impact of healthcare interventions, and also, to encourage this approach for future quantifications of

impact.

142

6.4 Guiding the Strategy of Healthcare Organizations

Currently, most of the engineering tools that have been used in healthcare fields seek to solve problems at

the tactical or operational level. However, incorporating tools to guide the strategy should also be

considered as critical as they provide guidance for actions and policy to achieve a major goal. In order to

respond to this gap, all three main chapters of this dissertation provide quality improvement research

frameworks that can be used for informing healthcare managers and policy makers at the strategic level.

The framework provided in Chapter 3 could be used for a better coordination and translation between the

translational research phases. The overall roadmap for translation was included to guide a strategic

allocation of resources. In Chapter 4, strategic goals with respect to multi-disciplinary collaboration can be

achieved by having a better understanding of the network structures. Finally, in Chapter 5, the proposal

selection method provided seek to incorporate the organizational vision into the resource allocation problem

of the healthcare organization.

All the contributions are aligned with the objective of moving basic research to clinical practice in a more

effective and efficient manner. And thus, materialize promising innovative developments made in early

research phases.

6.5 Future Work

Opportunities for system and quality engineering researchers are wide in supporting translational research.

From this dissertation, several branches of future work in expanding quality improvement research can be

envisioned.

Future research lines based upon the work presented in Chapter 3 include: 1) Identification of translational

research benchmarks, 2) Generation of agreement about process markers and technical requirements, 3)

Identification of similar translational research efforts in which lessons of translation can be implemented,

and 4) Investigation and understanding of barriers in translational research. The QFD-AHP framework

provides a robust structure to quantify the importance of processes markers along the translational research

processes. Future work could include the operationalization of those process markers by identifying key

metric that could be then used to identify benchmarks between different translational research initiatives.

Moreover, the QFD-AHP framework could be extended and used as a mechanism to quantify the

importance of the different process markers and technical requirements in different translational research

143

efforts. This can serve to set the baseline and agreement in terms of where the focus should be when

translating knowledge from basic to practice. Similarly, the proposed framework could be extended to

evaluate similarities among different translational research projects. Hence, best practices and lessons

learned can be shared between and within translational research areas. Finally, the quantification of the

relative importance of the technical requirements in translational research could be used as a roadmap for

a more in depth analysis of barriers along the translational research process. Thus, other quality

improvement tools such as RCA and FMEA can provide support to elicit those barriers. Consequently, a

less fragmented journey from basic research to clinical practice can be achieved.

From Chapter 4, some of the future research lines envisioned are: 1) Understanding of “optimal”

collaboration networks, 2) Evaluation of program effectiveness, and 3) Evaluation of multi-disciplinary and

its evolution. SNA provides rich structure in terms of visualization and mathematical background to identify

and understand patterns of collaboration within a network. Future research envisions the use of SNA to

support the design of collaborative structures as a way to find and shape “optimal” networks. This concept

of optimal network will depend on the objective of the collaboration, it can be related to the acceleration of

the transmission of knowledge within the network, stronger clustering structures, stronger cohesion,

stronger multidisciplinarity, etc. In this sense, network modeling techniques, such as dynamic network

modeling, could be used to test the impact that different interventions can have on the network. In addition,

SNA can be used as a monitoring tool for program evaluation. This line of research becomes relevant to

address the key collaboration objectives aimed by the CTSA. In alignment to this, SNA could also be used

to evaluate how health research has become a multidisciplinary field and what aspects of the curriculum

should be modified to respond to the current needs. As it has been emphasized, multidisciplinarity plays a

key role in the understanding and acceleration of translational research.

Future work concerning the topics covered in Chapter 5 include: 1) Define a standardized structure to share

proposals’ outcomes and best practices, 2) Investigate mechanisms to formalize data capturing and

extraction of information, and 3) Use the RHIE framework in practical comparisons between healthcare

interventions. The CTSA called for a stronger collaboration and data sharing across the CTSA hubs. This

becomes relevant for the generation of data-driven approaches that seek to inform healthcare management.

The GP framework could help incorporating standardized proposals’ information with the aim of improving

resource allocation practices. Aligned to this idea, formal mechanisms for information extraction need to

be investigated. In this sense, data mining approaches could be appropriate for identifying patterns that

drive the success or failure of a healthcare intervention. Finally, in the short-term, the RHIE can be used

for rapid comparison between different healthcare interventions. In particular, the case of PD presented in

this dissertation will be extended to evaluate different treatments for PD patients. The main aim of this

144

future research is to evaluate whether there is a significant difference in terms of cost-effectiveness and

quality of life of patients receiving the treatment recommended by general practitioners, drug-based

treatment recommended by a specialist, and complementary alternative treatment recommended by a

specialist.

All the above mentioned future work lines aim to integrate systems engineering and quality improvement

research to support data-driven mechanisms and inform health decision making processes. And thus,

helping to close the current gaps in translational research.

145

REFERENCES

ABBASI, A., HOSSAIN, L. & LEYDESDORFF, L. 2012. Betweenness centrality as a driver of preferential attachment in the evolution of research collaboration networks. Journal of Informetrics, 6, 403-412.

ABRAHAM, A., HASSANIEN, A.-E. & SNÁŠEL, V. 2009. Computational social network analysis: Trends, tools and research advances, Springer Science & Business Media.

AKAO, Y. 1972. New product development and quality assurance–quality deployment system. Standardization and Quality Control, 25, 7-14.

AKAO, Y., MAZUR, G. H. & KING, B. 1990. Quality function deployment: integrating customer requirements into product design, Productivity Press Cambridge, MA.

ANTES, G., GALANDI, D. & BOUILLON, B. 1999. What is evidence-based medicine? Langenbeck's Archives of Surgery, 384, 409-416.

ARTHUR, J. L. & RAVINDRAN, A. 1981. A multiple objective nurse scheduling model. AIIE transactions, 13, 55-60.

AUBKE, F., WÖBER, K., SCOTT, N. & BAGGIO, R. 2014. Knowledge sharing in revenue management teams: Antecedents and consequences of group cohesion. International Journal of Hospitality Management, 41, 149-157.

AZAIEZ, M. N. & AL SHARIF, S. 2005. A 0-1 goal programming model for nurse scheduling. Computers & Operations Research, 32, 491-507.

BAKHEIT, A. 1995. Early diagnosis of Parkinson's disease. Postgraduate medical journal, 71, 151-152. BALDWIN, L.-M., KEPPEL, G. A., DAVIS, A., GUIRGUIS-BLAKE, J., FORCE, R. W. & BERG, A. O. 2012.

Developing a Practice-Based Research Network by Integrating Quality Improvement: Challenges and Ingredients for Success. Clinical and Translational Science, 5, 351-355.

BANKS, J., CARSON, J. & NELSON, B. 2000. DM Nicol, Discrete-Event System Simulation, Prentice hall Englewood Cliffs, NJ, USA.

BARRETT, J. S., FOSSLER, M. J., CADIEU, K. D. & GASTONGUAY, M. R. 2008. Pharmacometrics: a multidisciplinary field to facilitate critical thinking in drug development and translational research settings. The Journal of Clinical Pharmacology, 48, 632-649.

BARTON, G. R., SACH, T. H., AVERY, A. J., JENKINSON, C., DOHERTY, M., WHYNES, D. K. & MUIR, K. R. 2008. A comparison of the performance of the EQ‐5D and SF‐6D for individuals aged≥ 45 years. Health economics, 17, 815-832.

BATALDEN, P. B. & DAVIDOFF, F. 2007. What is “quality improvement” and how can it transform healthcare? Quality and Safety in Health Care, 16, 2-3.

BENNETT, D. A., BECKETT, L. A., MURRAY, A. M., SHANNON, K. M., GOETZ, C. G., PILGRIM, D. M. & EVANS, D. A. 1996. Prevalence of parkinsonian signs and associated mortality in a community population of older people. New England Journal of Medicine, 334, 71-76.

BENNETT, L. M., GADLIN, H. & LEVINE-FINLEY, S. 2010. Collaboration \& Team Science: A Field Guide, NIH Office of the Ombudsman, Center for Cooperative Resolution.

BENSOUSSAN, A., CHANG, S., MENZIES, R. & TALLEY, N. 2001. Application of the general health status questionnaire SF36 to patients with gastrointestinal dysfunction: initial validation and validation as a measure of change. Australian and New Zealand journal of public health, 25, 71-77.

BENTON, D. C. & FERNÁNDEZ, M. P. F. 2014. Social network analysis: A tool for the identification of next generation trainers. Collegian, 21, 311-318.

BERWICK, D. M. 2003. Disseminating innovations in health care. JAMA: the journal of the American Medical Association, 289, 1969--1975.

146

BEST, A., STOKOLS, D., GREEN, L. W., LEISCHOW, S., HOLMES, B. & BUCHHOLZ, K. 2003. An integrative framework for community partnering to translate theory into effective health promotion strategy. American Journal of Health Promotion, 18, 168-176.

BHARMAL, M. & THOMAS, J. 2006. Comparing the EQ‐5D and the SF‐6D Descriptive Systems to Assess Their Ceiling Effects in the US General Population. Value in Health, 9, 262-271.

BIAN, J., XIE, M., TOPALOGLU, U., HUDSON, T., ESWARAN, H. & HOGAN, W. 2014. Social network analysis of biomedical research collaboration networks in a CTSA institution. Journal of biomedical informatics, 52, 130-140.

BLACK, D. 1976. Partial justification of the Borda count. Public Choice, 28, 1-15. BLAKE, J. T. & CARTER, M. W. 2002. A goal programming approach to strategic resource allocation in acute

care hospitals. European Journal of Operational Research, 140, 541-561. BORGATTI, S. P. & EVERETT, M. G. 2006. A graph-theoretic perspective on centrality. Social networks, 28,

466-484. BORGATTI, S. P. & FOSTER, P. C. 2003. The network paradigm in organizational research: A review and

typology. Journal of management, 29, 991-1013. BORGATTI, S. P., MEHRA, A., BRASS, D. J. & LABIANCA, G. 2009. Network analysis in the social sciences.

science, 323, 892-895. BORNMANN, L. & LEYDESDORFF, L. 2015. Topical connections between the institutions within an

organisation (institutional co-authorships, direct citation links and co-citations). Scientometrics, 102, 455-463.

BOUSQUET, J., KNANI, J., DHIVERT, H., RICHARD, A., CHICOYE, A., WARE JR, J. E. & MICHEL, F. 1994. Quality of life in asthma. I. Internal consistency and validity of the SF-36 questionnaire. American journal of respiratory and critical care medicine, 149, 371-375.

BOWER, J. E., GANZ, P. A., DESMOND, K. A., ROWLAND, J. H., MEYEROWITZ, B. E. & BELIN, T. R. 2000. Fatigue in breast cancer survivors: occurrence, correlates, and impact on quality of life. Journal of Clinical Oncology, 18, 743-743.

BOZEMAN, B., FAY, D. & SLADE, C. P. 2013. Research collaboration in universities and academic entrepreneurship: the-state-of-the-art. The Journal of Technology Transfer, 38, 1-67.

BRASS, D. J. 1995. A social network perspective on human resources management. Research in personnel and human resources management, 13, 39-79.

BRAVO VERGEL, Y. & SCULPHER, M. 2008. Quality-adjusted life years. Practical neurology, 8, 175-182. BRAZIER, J., ROBERTS, J. & DEVERILL, M. 2002. The estimation of a preference-based measure of health

from the SF-36. Journal of health economics, 21, 271-292. BRAZIER, J., ROBERTS, J., TSUCHIYA, A. & BUSSCHBACH, J. 2004. A comparison of the EQ‐5D and SF‐6D

across seven patient groups. Health economics, 13, 873-884. BRETTHAUER, K. M. & CǑTÉ, M. J. 1998. A model for planning resource requirements in health care

organizations. Decision Sciences, 29, 243-270. BREYFOGLE III, F. W. 2003. Implementing six sigma: smarter solutions using statistical methods, John Wiley

& Sons. BROUWER, W., VAN HOUT, B. & RUTTEN, F. 2000. A fair representation of future effects: taking a societal

perspective. Journal of Health Services Research and Policy, 5, 1-5. BROWNSON, R. C., COLDITZ, G. A. & PROCTOR, E. K. 2012. Dissemination and implementation research in

health: translating science to practice, Oxford University Press. BURSTEIN, F. & HOLSAPPLE, C. 2008. Handbook on decision support systems 1, New York, Springer-Verlag. BUSSE, K., HEILMANN, R., KLEINSCHMIDT, S., ABU-MUGHEISIB, M., HÖPPNER, J., WUNDERLICH, C.,

GEMENDE, I., KAULITZ, L., WOLTERS, A. & BENECKE, R. 2012. Value of combined midbrain sonography, olfactory and motor function assessment in the differential diagnosis of early Parkinson's disease. Journal of Neurology, Neurosurgery & Psychiatry.

147

BUTLER, D. 2008. Translational research: crossing the valley of death. Nature News, 453, 840-842. CALHOUN, W. J., WOOTEN, K., BHAVNANI, S., ANDERSON, K. E., FREEMAN, J. & BRASIER, A. R. 2012. The

CTSA as an Exemplar Framework for Developing Multidisciplinary Translational Teams. Clinical and translational science.

CAMERON, D., ALEMAN-MEZA, B., ARPINAR, I. B., DECKER, S. L. & SHETH, A. P. A Taxonomy-based model for expertise extrapolation. Semantic Computing (ICSC), 2010 IEEE Fourth International Conference on, 2010. 333-340.

CAMERON, D. H. L., ALEMAN-MEZA, B., DECKER, S. & ARPINAR, I. B. 2007. SEMEF: A Taxonomy-Based Discovery of Experts, Expertise and Collaboration Networks. Citeseer.

CAREY, T. S., HOWARD, D. L., GOLDMON, M., ROBERSON, J. T., GODLEY, P. A. & AMMERMAN, A. 2005. Developing effective interuniversity partnerships and community-based research to address health disparities. Academic medicine: journal of the Association of American Medical Colleges, 80, 1039.

CHAERUL, M., TANAKA, M. & SHEKDAR, A. V. 2008. Resolving complexities in healthcare waste management: a goal programming approach. Waste Management & Research, 26, 217-232.

CHAMBERS, D., WILSON, P., THOMPSON, C. & HARDEN, M. 2012. Social network analysis in healthcare settings: a systematic scoping review. PLoS One, 7, e41911.

CHAN, L.-K. & WU, M.-L. 2002. Quality function deployment: A literature review. European Journal of Operational Research, 143, 463-497.

CHANG, C.-L. 2006. Application of quality function deployment launches to enhancing nursing home service quality. Total Quality Management & Business Excellence, 17, 287-302.

CHAPLIN, E. & AKAO, Y. 2003. Translate know-what into know-how. Quality progress, 36, 56-61. CHARNES, A. & COOPER, W. 1961. Management Models and Industrial Applications of Linear

Programming, Vol. I. Wiley, New York. CHARNES, A., COOPER, W. W. & FERGUSON, R. O. 1955. Optimal estimation of executive compensation

by linear programming. Management science, 1, 138-151. CHEEVER, M. A., ALLISON, J. P., FERRIS, A. S., FINN, O. J., HASTINGS, B. M., HECHT, T. T., MELLMAN, I.,

PRINDIVILLE, S. A., VINER, J. L. & WEINER, L. M. 2009. The prioritization of cancer antigens: a national cancer institute pilot project for the acceleration of translational research. Clinical Cancer Research, 15, 5323-5337.

CHOI, B. & PAK, A. 2006. Multidisciplinarity, interdisciplinarity and transdisciplinarity in health research, services, education and policy: 1. Definitions, objectives, and evidence of effectiveness. Clinical and investigative medicine. Medecine clinique et experimentale, 29, 351-364.

CHOMUTARE, T., XU, A. & IYENGAR, M. S. Social Network Analysis to Delineate Interaction Patterns That Predict Weight Loss Performance. Computer-Based Medical Systems (CBMS), 2014 IEEE 27th International Symposium on, 2014. IEEE, 271-276.

CHRISTAKIS, N. A. & FOWLER, J. H. 2007. The spread of obesity in a large social network over 32 years. New England journal of medicine, 357, 370-379.

CHRISTAKIS, N. A. & FOWLER, J. H. 2008. The collective dynamics of smoking in a large social network. New England journal of medicine, 358, 2249-2258.

CHU, S. C. & CHU, L. 2000. A modeling framework for hospital location and service allocation. International transactions in operational research, 7, 539-568.

CLARKE, P., GRAY, A. & HOLMAN, R. 2002. Estimating utility values for health states of type 2 diabetic patients using the EQ-5D (UKPDS 62). Medical Decision Making, 22, 340-349.

COELLI, T. J., PRASADA RAO, D., O’DONNELL, C. J. & BATTESE, G. E. 2005. Data Envelopment Analysis. An Introduction to Efficiency and Productivity Analysis, 161-181.

COLE, S. & SIMON, G. A. 1981. Chance and consensus in peer review. Science, 214, 881-886.

148

COOPER, W. W., SEIFORD, L. M. & TONE, K. 2007. Data envelopment analysis: a comprehensive text with models, applications, references and DEA-solver software, Springer Science & Business Media.

CROOKS, A., MASAD, D., CROITORU, A., COTNOIR, A., STEFANIDIS, A. & RADZIKOWSKI, J. 2014. International Relations State-Driven and Citizen-Driven Networks. Social Science Computer Review, 32, 205-220.

CROSS, R., BORGATTI, S. P. & PARKER, A. 2002. Making invisible work visible: Using social network analysis to support strategic collaboration. California management review, 44, 25-46.

DAMS, J., BORNSCHEIN, B., REESE, J. P., CONRADS-FRANK, A., OERTEL, W. H., SIEBERT, U. & DODEL, R. 2011. Modelling the Cost Effectiveness of Treatments for Parkinson’s Disease. Pharmacoeconomics, 29, 1025-1049.

DANNER, M., HUMMEL, J. M., VOLZ, F., VAN MANEN, J. G., WIEGARD, B., DINTSIOS, C.-M., BASTIAN, H., GERBER, A. & IJZERMAN, M. J. 2011. Integrating patients' views into health technology assessment: Analytic hierarchy process (AHP) as a method to elicit patient preferences. International journal of technology assessment in health care, 27, 369-375.

DAVIES, P., WALKER, A. E. & GRIMSHAW, J. M. 2010. A systematic review of the use of theory in the design of guideline dissemination and implementation strategies and interpretation of the results of rigorous evaluations. Implement Sci, 5, 14.

DEVLIN, N. & PARKIN, D. 2004. Does NICE have a cost‐effectiveness threshold and what other factors influence its decisions? A binary choice analysis. Health economics, 13, 437-452.

DIJKSTRA, L. & VAN DER BIJ, H. 2002. Quality function deployment in healthcare: Methods for meeting customer requirements in redesign and renewal. International Journal of Quality & Reliability Management, 19, 67-89.

DIXON, N. & PEARCE, M. 2011. Guide to Using Quality Improvement Tools to Drive Clinical Audits. DOLAN, P. 2008. Developing methods that really do value the ‘Q’in the QALY. Health economics, policy

and law, 3, 69-77. DOLAN, P., GUDEX, C., KIND, P. & WILLIAMS, A. 1995. A social tariff for EuroQol: results from a UK general

population survey, Centre for Health Economics University of York, UK. DOUGHERTY, D. & CONWAY, P. H. 2008. The “3T’s” road map to transform US health care: the “how” of

high-quality care. Jama, 299, 2319-2321. DROLET, B. C. & LORENZI, N. M. 2011. Translational research: understanding the continuum from bench

to bedside. Translational Research, 157, 1-5. DUIJN, P. A., KASHIRIN, V. & SLOOT, P. M. 2014. The relative ineffectiveness of criminal network

disruption. Scientific reports, 4. EICHLER, H. G., KONG, S. X., GERTH, W. C., MAVROS, P. & JÖNSSON, B. 2004. Use of Cost‐Effectiveness

Analysis in Health‐Care Resource Allocation Decision‐Making: How Are Cost‐Effectiveness Thresholds Expected to Emerge? Value in health, 7, 518-528.

EINSPRUCH, E. M. 1996. Quality function deployment (QFD): application to rehabilitation services. International Journal of Health Care Quality Assurance, 9, 41-46.

ERICSSON, K. A., WHYTE IV, J. & WARD, P. 2007. Expert performance in nursing: Reviewing research on expertise in nursing within the framework of the expert-performance approach. Advances in Nursing Science, 30, E58-E71.

ERMER, D. S. & KNIPER, M. K. 1998. Delighting the customer: Quality function deployment for quality service design. Total Quality Management, 9, 86-91.

ESTRADA, E. & RODRIGUEZ-VELAZQUEZ, J. A. 2005. Subgraph centrality in complex networks. Physical Review E, 71, 056103.

FALK‐KRZESINSKI, H. J., BÖRNER, K., CONTRACTOR, N., FIORE, S. M., HALL, K. L., KEYTON, J., SPRING, B., STOKOLS, D., TROCHIM, W. & UZZI, B. 2010. Advancing the science of team science. Clinical and translational science, 3, 263-266.

149

FEARNLEY, J. M. & LEES, A. J. 1991. Ageing and Parkinson's disease: substantia nigra regional selectivity. Brain, 114, 2283-2301.

FELDSTEIN, A. C. & GLASGOW, R. E. 2008. A practical, robust implementation and sustainability model (PRISM). Joint Commission Journal on Quality and Patient Safety, 34, 228-243.

FERNANDEZ, L. E., GABRI, M. R., GUTHMANN, M. D., GOMEZ, R. E., GOLD, S., FAINBOIM, L., GOMEZ, D. E. & ALONSO, D. F. 2010. NGcGM3 ganglioside: a privileged target for cancer vaccines. Clinical and Developmental Immunology, 2010.

FIELDING, J. E. & TEUTSCH, S. M. 2013. So what? A framework for assessing the potential impact of intervention research. Preventing chronic disease, 10.

FISHER, R. A. 1935. The design of experiments, Edinburgh, UK, Oliver and Boyd. FISHER, R. A. 1992. The arrangement of field experiments. Breakthroughs in Statistics. Springer. FORSAA, E., LARSEN, J., WENTZEL-LARSEN, T. & ALVES, G. 2010. What predicts mortality in Parkinson

disease? A prospective population-based long-term study. Neurology, 75, 1270-1276. FREEMAN, L. C. 1979. Centrality in social networks conceptual clarification. Social networks, 1, 215-239. GAENSLEN, A. & BERG, D. 2010. Early diagnosis of Parkinson’s disease. International review of

neurobiology, 90, 81-92. GANDEK, B., WARE, J. E., AARONSON, N. K., APOLONE, G., BJORNER, J. B., BRAZIER, J. E., BULLINGER, M.,

KAASA, S., LEPLEGE, A. & PRIETO, L. 1998. Cross-validation of item selection and scoring for the SF-12 Health Survey in nine countries: results from the IQOLA Project. Journal of clinical epidemiology, 51, 1171-1178.

GARDY, J. L., JOHNSTON, J. C., SUI, S. J. H., COOK, V. J., SHAH, L., BRODKIN, E., REMPEL, S., MOORE, R., ZHAO, Y., HOLT, R. & OTHERS 2011. Whole-genome sequencing and social-network analysis of a tuberculosis outbreak. New England Journal of Medicine, 364, 730-739.

GARRISON JR, L. P. 2009. Editorial: on the benefits of modeling using QALYs for societal resource allocation: the model is the message. Value in Health, 12, S36-S37.

GAUJOUX-VIALA, C., RAT, A.-C., GUILLEMIN, F., FLIPO, R.-M., FARDELLONE, P., BOURGEOIS, P. & FAUTREL, B. 2011. Comparison of the EQ-5D and the SF-6D utility measures in 813 patients with early arthritis: results from the ESPOIR cohort. The Journal of rheumatology, 38, 1576-1584.

GELB, D. J., OLIVER, E. & GILMAN, S. 1999. Diagnostic criteria for Parkinson disease. Archives of neurology, 56, 33-39.

GEWIN, V. 2010. Collaboration: Social networking seeks critical mass. Nature, 468, 993-994. GLASGOW, R. E., BROWNSON, R. C. & KESSLER, R. S. 2013. Thinking about Health-Related Outcomes: What

Do We Need Evidence about? Clinical and translational science, 6, 286-291. GLASGOW, R. E., VINSON, C., CHAMBERS, D., KHOURY, M. J., KAPLAN, R. M. & HUNTER, C. 2012. National

Institutes of Health approaches to dissemination and implementation science: current and future directions. American Journal of Public Health, 102, 1274-1281.

GOFFIN, K., VARNES, C. J., VAN DER HOVEN, C. & KONERS, U. 2012. Beyond the voice of the customer: Ethnographic market research. Research-Technology Management, 55, 45-54.

GOLD, M. R. 1996. Cost-Effectiveness in Health and Medicine, Oxford University Press. GOLDRATT, E. M. 1990. Theory of constraints, North River Croton-on-Hudson, NY. GOVERS, C. P. 1996. What and how about quality function deployment (QFD). International journal of

production economics, 46, 575-585. GRALNEK, I. M., HAYS, R. D., KILBOURNE, A., NALIBOFF, B. & MAYER, E. A. 2000. The impact of irritable

bowel syndrome on health-related quality of life. Gastroenterology, 119, 654-660. GREEN, L. W., OTTOSON, J., GARCÍA, C. & ROBERT, H. 2009. Diffusion theory and knowledge dissemination,

utilization, and integration in public health. Annual review of public health, 30, 151. GREENER, M. 2009. Does Parkinson's disease increase mortality? Progress in Neurology and Psychiatry,

13, 6-8.

150

GREMYR, I. & RAHARJO, H. 2013. Quality function deployment in healthcare: a literature review and case study. International journal of health care quality assurance, 26, 135-146.

GRIEVE, R., GRISHCHENKO, M. & CAIRNS, J. 2009. SF-6D versus EQ-5D: reasons for differences in utility scores and impact on reported cost-utility. The European Journal of Health Economics, 10, 15-23.

GRIMSHAW, J., THOMAS, R., MACLENNAN, G., FRASER, C., RAMSAY, C., VALE, L., WHITTY, P., ECCLES, M., MATOWE, L., SHIRRAN, L. & OTHERS 2004. Effectiveness and efficiency of guideline dissemination and implementation strategies.

GRIMSHAW, J. M., ECCLES, M. P., LAVIS, J. N., HILL, S. J., SQUIRES, J. E. & OTHERS 2012a. Knowledge translation of research findings. Implement Sci, 7, 50.

GRIMSHAW, J. M., SCHÜNEMANN, H. J., BURGERS, J., CRUZ, A. A., HEFFNER, J., METERSKY, M. & COOK, D. Disseminating and Implementing Guidelines. Proceedings of the American Thoracic Society, Vol. 9, No. 5, 2012b. 298-303.

GRIMSHAW, J. M., SHIRRAN, L., THOMAS, R., MOWATT, G., FRASER, C., BERO, L., GRILLI, R., HARVEY, E., OXMAN, A. & O'BRIEN, M. A. 2001. Changing provider behavior: an overview of systematic reviews of interventions. Medical care, II2--II45.

GROL, R. & GRIMSHAW, J. 2003. From best evidence to best practice: effective implementation of change in patients' care. The lancet, 362, 1225-1230.

GROSSMAN, J. H. 2008. Disruptive innovation in health care: Challenges for engineering. BRIDGE-WASHINGTON-NATIONAL ACADEMY OF ENGINEERING-, 38, 10.

GUINDO, L. A., WAGNER, M., BALTUSSEN, R., RINDRESS, D., VAN TIL, J., KIND, P. & GOETGHEBEUR, M. M. 2012. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc, 10.

GÜNDÜZ-ÖĞÜDÜCÜ, Ş. & ETANER-UYAR, A. Ş. 2014. Social Networks: Analysis and Case Studies, Springer. HAHN, B. A., YAN, S. & STRASSELS, S. 1999. Impact of irritable bowel syndrome on quality of life and

resource use in the United States and United Kingdom. Digestion, 60, 77-81. HALL, K. L., FENG, A. X., MOSER, R. P., STOKOLS, D. & TAYLOR, B. K. 2008. Moving the science of team

science forward: collaboration and creativity. American journal of preventive medicine, 35, S243--S249.

HANCOCK, P. & RAESIDE, R. 2010. Analysing communication in a complex service process: an application of social network analysis in the Scottish Prison Service. Journal of the Operational Research Society, 61, 265-274.

HANS, E. W., VAN HOUDENHOVEN, M. & HULSHOF, P. J. 2012. A framework for healthcare planning and control. Handbook of healthcare system scheduling. Springer.

HANSEN, D., SHNEIDERMAN, B. & SMITH, M. A. 2010. Analyzing social media networks with NodeXL: Insights from a connected world, Morgan Kaufmann.

HAREL, D. & KOREN, Y. A fast multi-scale method for drawing large graphs. Graph drawing, 2001. Springer, 183-196.

HATALA, J.-P. 2006. Social network analysis in human resource development: A new methodology. Human Resource Development Review, 5, 45-71.

HATOUM, H. T., BRAZIER, J. E. & AKHRAS, K. S. 2004. Comparison of the HUI3 with the SF‐36 Preference Based SF‐6D in a Clinical Trial Setting. Value in Health, 7, 602-609.

HAUSER, J. R. & CLAUSING, D. 1988. The house of quality. Harvard Business Review, 66, 63-73. HO, W. 2008. Integrated analytic hierarchy process and its applications–A literature review. European

Journal of operational research, 186, 211-228. HOEHN, M. M. & YAHR, M. D. 1998. Parkinsonism: onset, progression, and mortality. Neurology, 50, 318-

318. HSIUNG, P.-C., FANG, C.-T., CHANG, Y.-Y., CHEN, M.-Y. & WANG, J.-D. 2005. Comparison of WHOQOL-BREF

and SF-36 in patients with HIV infection. Quality of Life Research, 14, 141-150.

151

HSU, T.-H. & PAN, F. F. 2009. Application of Monte Carlo AHP in ranking dental quality attributes. Expert Systems with Applications, 36, 2310-2316.

HUANG, G. Q. 1996. Design for X: concurrent engineering imperatives, Springer Science & Business Media. HUNG, C.-I., LIU, C.-Y. & WANG, S.-J. 2013. Migraine predicts physical and pain symptoms among

psychiatric outpatients. The journal of headache and pain, 14, 1-8. HUNT, J. D., WHIPPLE, E. C. & MCGOWAN, J. J. 2012. Use of social network analysis tools to validate a

resources infrastructure for interinstitutional translational research: a case study. Journal of the Medical Library Association: JMLA, 100, 48.

HURST, N., KIND, P., RUTA, D., HUNTER, M. & STUBBINGS, A. 1997. Measuring health-related quality of life in rheumatoid arthritis: validity, responsiveness and reliability of EuroQol (EQ-5D). Rheumatology, 36, 551-559.

HUSE, D. M., SCHULMAN, K., ORSINI, L., CASTELLI‐HALEY, J., KENNEDY, S. & LENHART, G. 2005. Burden of illness in Parkinson's disease. Movement disorders, 20, 1449-1454.

IGNIZIO, J. P. 1976. Goal programming and extensions, Lexington Books Lexington, MA. IJIRI, Y. 1965. Management goals and accounting for control, North Holland Pub. Co. IOM 2001. Crossing the quality chasm: A new health system for the 21st century, Institute of Medicine,

National Academies Press. ISHIHARA, L. S., CHEESBROUGH, A., BRAYNE, C. & SCHRAG, A. 2007. Estimated life expectancy of

Parkinson’s patients compared with the UK population. Journal of Neurology, Neurosurgery & Psychiatry, 78, 1304-1309.

JACQUES, M., BACCARANI, C., ANDRONIKIDIS, A., GEORGIOU, A. C., GOTZAMANI, K. & KAMVYSI, K. 2009. The application of quality function deployment in service quality management. The TQM journal, 21, 319-333.

JAMISON, D. T., BREMAN, J. G., MEASHAM, A. R., ALLEYNE, G., CLAESON, M., EVANS, D. B., JHA, P., MILLS, A. & MUSGROVE, P. 2006. Disease control priorities in developing countries, World Bank Publications.

JAMTVEDT, G., YOUNG, J., KRISTOFFERSEN, D., O'BRIEN, M. & OXMAN, A. 2006. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev, 2.

JEONG, M. & OH, H. 1998. Quality function deployment: An extended framework for service quality and customer satisfaction in the hospitality industry. International Journal of Hospitality Management, 17, 375-390.

JOLLIFFE, I. 2002. Principal component analysis, Wiley Online Library. JONES, D. & TAMIZ, M. 2010. Practical goal programming, Springer. KAPLAN, R. M., ANDERSON, J. P. & GANIATS, T. G. 1993. The quality of well-being scale: rationale for a

single quality of life index. Quality of life assessment: key issues in the 1990s. Springer. KAPLAN, R. M., BUSH, J. W. & BERRY, C. C. 1976. Health status: types of validity and the index of well-

being. Health services research, 11, 478. KAPLAN, S. & GARRICK, B. J. 1981. On the quantitative definition of risk. Risk analysis, 1, 11-27. KARSAK, E. E., SOZER, S. & ALPTEKIN, S. E. 2003. Product planning in quality function deployment using a

combined analytic network process and goal programming approach. Computers & industrial engineering, 44, 171-190.

KAYE, A. D., OKANLAWON, O. J. & URMAN, R. D. 2014. Clinical performance feedback and quality improvement opportunities for perioperative physicians. Advances in medical education and practice, 5, 115.

KHAN, G. F., YOON, H. Y. & PARK, H. W. 2014. Social media communication strategies of government agencies: Twitter use in Korea and the USA. Asian Journal of Communication, 24, 60-78.

KHOURY, M. J., GWINN, M., YOON, P. W., DOWLING, N., MOORE, C. A. & BRADLEY, L. 2007. The continuum of translation research in genomic medicine: how can we accelerate the appropriate integration

152

of human genome discoveries into health care and disease prevention? Genetics in Medicine, 9, 665-674.

KIRKWOOD, C. W. 1997. Strategic decision making. Multi-objective Decision Analysis with Spreadsheets, Wadsworth.

KNOBEN, J. & OERLEMANS, L. A. 2006. Proximity and inter-organizational collaboration: A literature review. International Journal of Management Reviews, 8, 71-89.

KON, A. A. 2008. The Clinical and Translational Science Award (CTSA) Consortium and the translational research model. The American Journal of Bioethics, 8, 58-60.

KONTODIMOPOULOS, N., PAPPA, E., CHADJIAPOSTOLOU, Z., ARVANITAKI, E., PAPADOPOULOS, A. A. & NIAKAS, D. 2012. Comparing the sensitivity of EQ-5D, SF-6D and 15D utilities to the specific effect of diabetic complications. The European Journal of Health Economics, 13, 111-120.

KUEHL, R. O. & KUEHL, R. 2000. Design of experiments: statistical principles of research design and analysis, Duxbury/Thomson Learning Pacific Grove, CA:.

KWAK, N. & LEE, C. 1997. A linear goal programming model for human resource allocation in a health-care organization. Journal of Medical Systems, 21, 129-140.

KWAK, N. & LEE, C. W. 2002. Business process reengineering for health-care system using multicriteria mathematical programming. European Journal of Operational Research, 140, 447-458.

KWONG, C. & BAI, H. 2002. A fuzzy AHP approach to the determination of importance weights of customer requirements in quality function deployment. Journal of intelligent manufacturing, 13, 367-377.

KWONG, C. & BAI, H. 2003. Determining the importance weights for the customer requirements in QFD using a fuzzy AHP with an extent analysis approach. Iie Transactions, 35, 619-626.

LACSON, E., XU, J., LIN, S.-F., DEAN, S. G., LAZARUS, J. M. & HAKIM, R. M. 2010. A comparison of SF-36 and SF-12 composite scores and subsequent hospitalization and mortality risks in long-term dialysis patients. Clinical Journal of the American Society of Nephrology, 5, 252-260.

LAI, H. F. Applying fuzzy AHP to evaluate the sustainability of knowledge-based virtual communities in healthcare industry. Service Systems and Service Management (ICSSSM), 2010 7th International Conference on, 2010. IEEE, 1-6.

LAMERS, L., BOUWMANS, C., VAN STRATEN, A., DONKER, M. & HAKKAART, L. 2006. Comparison of EQ‐5D and SF‐6D utilities in mental health patients. Health economics, 15, 1229-1236.

LAMONT, M. 2009. How Professors Think: Inside the Curious World of Academic Judgment, Harvard University Press.

LANDRO, L. 2014. Specialized Care Urged for Treating Parkinson's Disease [Online]. http://www.wsj.com/articles/SB10001424052702304479704579580020917565860: The Wall Street Journal. [Accessed January 27, 2015 2015].

LANG, J. M., ANDREI, A.-C. & MCNEEL, D. G. 2009. Prioritization of cancer antigens: keeping the target in sight.

LEE, C. & KWAK, N. 1999. Information resource planning for a health-care system using an AHP-based goal programming method. Journal of the operational research society, 50, 1191-1198.

LEE, C., KWAK, N. & OTHERS 1999. Information resource planning for a health-care system using an AHP-based goal programming method. Journal of the operational research society, 50, 1191-1198.

LEE, S. M. 1972. Goal programming for decision analysis, Auerbach Philadelphia. LEHTO, M. R. & LANDRY, S. J. 2012. Introduction to human factors and ergonomics for engineers, Crc Press. LEIFELD, P. 2013. Reconceptualizing major policy change in the advocacy coalition framework: A discourse

network analysis of German Pension politics. Policy Studies Journal, 41, 169-198. LI, E. Y., LIAO, C. H. & YEN, H. R. 2013. Co-authorship networks and research impact: A social capital

perspective. Research Policy, 42, 1515-1530. LIAO, P.-C., LEI, G., FANG, D. & LIU, W. 2014. The relationship between communication and construction

safety climate in China. KSCE Journal of Civil Engineering, 18, 887-897.

http://www.wsj.com/articles/SB10001424052702304479704579580020917565860:

153

LIBERATORE, M. J. & NYDICK, R. L. 2008. The analytic hierarchy process in medical and health care decision making: A literature review. European Journal of Operational Research, 189, 194-207.

LIBERATORE, M. J. & STYLIANOU, A. C. 1995. Expert support systems for new product development decision making: a modeling framework and applications. Management science, 41, 1296-1316.

LONG, J. C., CUNNINGHAM, F. C. & BRAITHWAITE, J. 2012. Network structure and the role of key players in a translational cancer research network: a study protocol. BMJ open, 2.

LONG, J. C., CUNNINGHAM, F. C., CARSWELL, P. & BRAITHWAITE, J. 2014. Patterns of collaboration in complex networks: the example of a translational research network. BMC Health Services Research, 14, 225.

LONGWORTH, L. & BRYAN, S. 2003. An empirical comparison of EQ‐5D and SF‐6D in liver transplant patients. Health economics, 12, 1061-1067.

LOUIS, E. D., MARDER, K., COTE, L., TANG, M. & MAYEUX, R. 1997. Mortality from Parkinson disease. Archives of neurology, 54, 260-264.

LOWE, P. & PHILLIPSON, J. 2006. Reflexive interdisciplinary research: the making of a research programme on the rural economy and land use. Journal of Agricultural Economics, 57, 165-184.

LU, Y., POLGAR, M., LUO, X. & CAO, Y. 2010. Social network analysis of a criminal hacker community. Journal of Computer Information Systems, 51, 31-41.

LUBOWITZ, J. H. & APPLEBY, D. 2011. Cost-effectiveness analysis of the most common orthopaedic surgery procedures: knee arthroscopy and knee anterior cruciate ligament reconstruction. Arthroscopy: The Journal of Arthroscopic & Related Surgery, 27, 1317-1322.

LUNDBERG, L., JOHANNESSON, M., ISACSON, D. G. & BORGQUIST, L. 1999. The relationship between health-state utilities and the SF-12 in a general population. Medical Decision Making, 19, 128-140.

LUO, Q. & ZHONG, D. 2015. Using social network analysis to explain communication characteristics of travel-related electronic word-of-mouth on social networking sites. Tourism Management, 46, 274-282.

LURIE, S., FOGG, T. & DOZIER, A. 2009. Social network analysis as a method of assessing medical-center culture; three case studies. Academic medicine: journal of the Association of American Medical Colleges, 84, 1029.

MA, F., LI, Y. & CHEN, B. 2014. Study of the collaboration in the field of the Chinese humanities and social sciences. Scientometrics, 100, 439-458.

MALM, A. & BICHLER, G. 2011. Networks of collaborating criminals: Assessing the structural vulnerability of drug markets. Journal of Research in Crime and Delinquency, 48, 271-297.

MARINCOLA, F. M. 2003. Translational medicine: a two-way road. J Transl Med, 1, 1. MARRA, C. A., WOOLCOTT, J. C., KOPEC, J. A., SHOJANIA, K., OFFER, R., BRAZIER, J. E., ESDAILE, J. M. &

ANIS, A. H. 2005. A comparison of generic, indirect utility measures (the HUI2, HUI3, SF-6D, and the EQ-5D) and disease-specific instruments (the RAQoL and the HAQ) in rheumatoid arthritis. Social science & medicine, 60, 1571-1582.

MASUD, A. S. & RAVINDRAN, A. 2008. Multiple criteria decision making. In: RAVINDRAN, A. (ed.) Operations Research and Management Science Handbook. Taylor & Francis Group, LLC.

MATTI, S. & SANDSTRÖM, A. 2013. The defining elements of advocacy coalitions: Continuing the search for explanations for coordination and coalition structures. Review of Policy Research, 30, 240-257.

MCCABE, C., CLAXTON, K. & CULYER, A. J. 2008. The NICE cost-effectiveness threshold. Pharmacoeconomics, 26, 733-744.

MCDONOUGH, J., SOLOMON, R. & PETOSA, L. 2004. Quality improvement and proactive hazard analysis models: deciphering a new Tower of Babel. Aspden P, Corrigan JM, Wolcott J et al, 471-508.

MCENRUE, M. P. 1988. Length of experience and the performance of managers in the establishment phase of their careers. Academy of Management Journal, 31, 175-185.

154

MCGLYNN, E. A., ASCH, S. M., ADAMS, J., KEESEY, J., HICKS, J., DECRISTOFARO, A. & KERR, E. A. 2003. The quality of health care delivered to adults in the United States. New England journal of medicine, 348, 2635-2645.

MCHUGH, M. D. & LAKE, E. T. 2010. Understanding clinical expertise: nurse education, experience, and the hospital context. Research in nursing & health, 33, 276-287.

MCHUGH, R. K. & BARLOW, D. H. 2010. The dissemination and implementation of evidence-based psychological treatments: a review of current efforts. American Psychologist, 65, 73.

MILLER, G. A. 1956. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological review, 63, 81.

MONZON, M. & LAINEZ, M. 1998. Quality of life in migraine and chronic daily headache patients. Cephalalgia, 18, 638-643.

MOOCK, J. & KOHLMANN, T. 2008. Comparing preference-based quality-of-life measures: results from rehabilitation patients with musculoskeletal, cardiovascular, or psychosomatic disorders. Quality of Life Research, 17, 485-495.

MOORES, B. 2006. Radiation safety management in health care–The application of Quality Function Deployment. Radiography, 12, 291-304.

MOSCONI, P., APOLONE, G., BARNI, S., SECONDINO, S., SBANOTTO, A. & FILIBERTI, A. 2001. Quality of life in breast and colon cancer long-term survivors: an assessment with the EORTC QLQ-C30 and SF-36 questionnaires. Tumori, 88, 110-116.

MOSES, H., DORSEY, E. R., MATHESON, D. H. & THIER, S. O. 2005. Financial anatomy of biomedical research. Jama, 294, 1333-1342.

MOVEMENT DISORDER SOCIETY TASK FORCE ON RATING SCALES FOR PARKINSON'S DISEASE 2003. The Unified Parkinson's Disease Rating Scale (UPDRS): status and recommendations. Movement disorders: official journal of the Movement Disorder Society, 18, 738.

MOZ, M. & PATO, M. A bi-objective network flow approach for nurse rerostering. Proceedings of the INOC, 2005. 817-824.

MUSA, A. & SAXENA, U. 1984. Scheduling nurses using goal-programming techniques. IIE transactions, 16, 216-221.

MUSLIMOVIĆ, D., POST, B., SPEELMAN, J. & SCHMAND, B. 2007. Motor procedural learning in Parkinson's disease. Brain, 130, 2887-2897.

MUTEBI, A., BRAZIER, J. E. & WALTERS, S. J. 2011. A comparison of the discriminative and evaluative properties of the SF-36 and the SF-6D index. Quality of Life Research, 20, 1477-1486.

NASH, R., BOUCHARD, M. & MALM, A. 2013. Investing in people: The role of social networks in the diffusion of a large-scale fraud. Social Networks, 35, 686-698.

NATHAN, D. G. 2002. Careers in translational clinical research—historical perspectives, future challenges. JAMA: the journal of the American Medical Association, 287, 2424-2427.

NATIONAL PARKINSON FOUNDATION. 2014. Do You Have Early Warning Signs of Parkinson’s Disease? [Online]. http://www.parkinson.org/About-Us/Press-Room/NPF-In-The-News/2014/June/Do-You-Have-Early-Warning-Signs-of-Parkinsons-Dis. [Accessed January 27, 2015 2015].

NEUMANN, P. J., ROSEN, A. B. & WEINSTEIN, M. C. 2005. Medicare and cost-effectiveness analysis. New England Journal of Medicine, 353, 1516.

NIH, N. I. O. H. 2007. Building the Science of Dissemination and Implementation in the Service of Public Health. Available: http://www.obssr.od.nih.gov/di2007/about.html.

NORD, E., DANIELS, N. & KAMLET, M. 2009. QALYs: some challenges. Value in Health, 12, S10-S15. OCKENE, J. K., EDGERTON, E. A., TEUTSCH, S. M., MARION, L. N., MILLER, T., GENEVRO, J. L., LOVELAND-

CHERRY, C. J., FIELDING, J. E. & BRISS, P. A. 2007. Integrating evidence-based clinical and community strategies to improve health. American journal of preventive medicine, 32, 244-252.

http://www.parkinson.org/About-Us/Press-Room/NPF-In-The-News/2014/June/Do-You-Have-Early-Warning-Signs-of-Parkinsons-Dis

http://www.parkinson.org/About-Us/Press-Room/NPF-In-The-News/2014/June/Do-You-Have-Early-Warning-Signs-of-Parkinsons-Dis

http://www.obssr.od.nih.gov/di2007/about.html

155

ODDOYE, J. P., JONES, D. F., TAMIZ, M. & SCHMIDT, P. 2009. Combining simulation and goal programming for healthcare planning in a medical assessment unit. European Journal of Operational Research, 193, 250-261.

OSBORN, A. F. 1963. Applied Imagination; Principles and Procedures of Creative Problem-solving: Principles and Procedures of Creative Problem-solving, Scribner.

OSTERHAUS, J. T., TOWNSEND, R. J., GANDEK, B. & WARE, J. E. 1994. Measuring the functional status and well‐being of patients with migraine headache. Headache: The Journal of Head and Face Pain, 34, 337-343.

OWENS, D. K., QASEEM, A., CHOU, R. & SHEKELLE, P. 2011. High-value, cost-conscious health care: concepts for clinicians to evaluate the benefits, harms, and costs of medical interventions. Annals of internal medicine, 154, 174-180.

OZKARAHAN, I. 1991. A disaggregation model of a flexible nurse scheduling support system. Socio-economic planning sciences, 25, 9-26.

OZKARAHAN, I. 2000. Allocation of surgeries to operating rooms by goal programing. Journal of Medical Systems, 24, 339-378.

OZKARAHAN, I. & BAILEY, J. E. 1988. Goal programming model subsystem of a flexible nurse scheduling support system. IIE transactions, 20, 306-316.

PARKINSON'S DISEASE FOUNDATION. 2015. Statistics on Parkinson's [Online]. http://www.pdf.org/en/parkinson_statistics. [Accessed January 27, 2015 2015].

PATRICK, D. L. & ERICKSON, P. 1993. Health status and health policy. Quality of life in health care evaluation and resource.

PECCHIA, L., BATH, P., PENDLETON, N. & BRACALE, M. 2011. Analytic hierarchy process (AHP) for examining healthcare professionals’ assessments of risk factors. Methods Inf Med, 50, 435-444.

PETROU, S. & HOCKLEY, C. 2005. An investigation into the empirical validity of the EQ‐5D and SF‐6D based on hypothetical preferences in a general population. Health economics, 14, 1169-1189.

PLISKIN, J. S., SHEPARD, D. S. & WEINSTEIN, M. C. 1980. Utility functions for life years and health status. Operations research, 28, 206-224.

PRASAD, B. 1996. Concurrent engineering fundamentals, Prentice Hall Englewood Cliffs, NJ. PRIETO, L. & SACRISTÁN, J. A. 2003. Problems and solutions in calculating quality-adjusted life years

(QALYs). Health and quality of life outcomes, 1, 1-8. RABIN, R. & DE CHARRO, F. 2001. EQ-SD: a measure of health status from the EuroQol Group. Annals of

medicine, 33, 337-343. RADHARAMANAN, R. & GODOY, L. P. 1996. Quality function deployment as applied to a health care

system. Computers & industrial engineering, 31, 443-446. RAVINDRAN, A. R. 2007. Operations research and management science handbook, Crc Press. REID, P. P., COMPTON, W. D., GROSSMAN, J. H. & FANJIANG, G. 2005. Building a Better Delivery System:

A New Engineering/Health Care Partnership, National Academies Press. RICHARDSON, G. & MANCA, A. 2004. Calculation of quality adjusted life years in the published literature:

a review of methodology and transparency. Health economics, 13, 1203-1210. RILEY, E., BANGSBERG, D., PERRY, S., CLARK, R., MOSS, A. & WU, A. 2003. Reliability and validity of the SF-

36 in HIV-infected homeless and marginally housed individuals. Quality of Life Research, 12, 1051-1058.

RUBINSTEIN, R. Y. & KROESE, D. P. 2011. Simulation and the Monte Carlo method, John Wiley & Sons. RUBIO, D. M., SCHOENBAUM, E. E., LEE, L. S., SCHTEINGART, D. E., MARANTZ, P. R., ANDERSON, K. E.,

PLATT, L. D., BAEZ, A. & ESPOSITO, K. 2010. Defining translational research: implications for training. Academic medicine: journal of the Association of American Medical Colleges, 85, 470.

http://www.pdf.org/en/parkinson_statistics

156

RYYNÄNEN, O.-P., MYLLYKANGAS, M., KINNUNEN, J. & TAKALA, J. 1999. Attitudes to health care prioritisation methods and criteria among nurses, doctors, politicians and the general public. Social science \& medicine, 49, 1529-1539.

SAATY, T. L. 1977. A scaling method for priorities in hierarchical structures. Journal of mathematical psychology, 15, 234-281.

SAATY, T. L. 1980. The analytical hierarchical process. J Wiley, New York. SAATY, T. L. 2001. Deriving the AHP 1-9 scale from first principles. ISAHP 2001 proceedings, Bern,

Switzerland. SAATY, T. L. & OZDEMIR, M. S. 2003. Why the magic number seven plus or minus two. Mathematical and

Computer Modelling, 38, 233-244. SACH, T., BARTON, G., DOHERTY, M., MUIR, K., JENKINSON, C. & AVERY, A. 2006. The relationship between

body mass index and health-related quality of life: comparing the EQ-5D, EuroQol VAS and SF-6D. International Journal of Obesity, 31, 189-196.

SACKETT, D. L., ROSENBERG, W. M., GRAY, J., HAYNES, R. B. & RICHARDSON, W. S. 1996. Evidence based medicine: what it is and what it isn't. BMJ: British Medical Journal, 312, 71.

SANDERS, M. S. & MCCORMICK, E. J. 1987. Human factors in engineering and design, McGRAW-HILL book company.

SANDSTRÖM, U. & HÄLLSTEN, M. 2008. Persistent nepotism in peer-review. Scientometrics, 74, 175-189. SASSI, F. 2006. Calculating QALYs, comparing QALY and DALY calculations. Health Policy and Planning, 21,

402-408. SCHMITTDIEL, J. A., GRUMBACH, K. & SELBY, J. V. 2010. System-based participatory research in health

care: an approach for sustainable translational research and quality improvement. The Annals of Family Medicine, 8, 256-259.

SCHRAG, A., BEN-SHLOMO, Y. & QUINN, N. 2002. How valid is the clinical diagnosis of Parkinson's disease in the community? Journal of Neurology, Neurosurgery & Psychiatry, 73, 529-534.

SCHRAG, A., SELAI, C., JAHANSHAHI, M. & QUINN, N. P. 2000. The EQ-5D—a generic quality of life measure—is a useful instrument to measure quality of life in patients with Parkinson's disease. Journal of Neurology, Neurosurgery & Psychiatry, 69, 67-73.

SCHWAPPACH, D. L. 2002. Resource allocation, social values and the QALY: a review of the debate and empirical evidence. Health Expectations, 5, 210-222.

SCHWEIKHART, S. A. & DEMBE, A. E. 2009. The applicability of Lean and Six Sigma techniques to clinical and translational research. Journal of investigative medicine: the official publication of the American Federation for Clinical Research, 57, 748.

SCOTT, J. 2012. Social network analysis, Sage. SCOTT, J. & CARRINGTON, P. J. 2011. The SAGE handbook of social network analysis, SAGE publications. SEIBER, W. J., GROESSL, E. J., DAVID, K. M., GANIATS, T. G. & KAPLAN, R. M. 2008. Quality of well being

self-administered (QWB-SA) scale. San Diego: Health Services Research Center, University of California.

SELKER, H., GROSSMAN, C., ADAMS, A., GOLDMANN, D., DEZII, C., MEYER, G., ROGER, V., SAVITZ, L. & PLATT, R. 2011. The common rule and continuous improvement in health care: A learning health system perspective. Institute of Medicine Discussion Paper, October, 1.

SHAHRIAR, J., DELATE, T., HAYS, R. D. & COONS, S. J. 2003. Commentary on using the SF-36 or MOS-HIV in studies of persons with HIV disease. Health Qual Life Outcomes, 1, 25.

SHAW, J. W., JOHNSON, J. A. & COONS, S. J. 2005. US valuation of the EQ-5D health states: development and testing of the D1 valuation model. Medical care, 43, 203-220.

SHIROIWA, T., SUNG, Y.-K., FUKUDA, T., LANG, H.-C., BAE, S.-C. & TSUTANI, K. 2010. International survey on willingness-to-pay (WTP) for one additional QALY gained: what is the threshold of cost effectiveness? Health economics, 19, 422-437.

157

SHOJANIA, K. G., MCDONALD, K. M., WACHTER, R. & OWENS, D. K. 2004. Closing the quality gap: a critical analysis of quality improvement strategies, Agency for Healthcare Research and Quality Rockville, MD:.

SOLOMON, G. D., SKOBIERANDA, F. G. & GENZEN, J. R. 1995. Quality of life assessment among migraine patients treated with sumatriptan. Headache: The Journal of Head and Face Pain, 35, 449-454.

SPARROW, M. K. 1991. The application of network analysis to criminal intelligence: An assessment of the prospects. Social networks, 13, 251-274.

STAMATIS, D. H. 2003. Failure mode and effect analysis: FMEA from theory to execution, ASQ Quality Press.

STAVEM, K., FRØLAND, S. S. & HELLUM, K. B. 2005. Comparison of preference-based utilities of the 15D, EQ-5D and SF-6D in patients with HIV/AIDS. Quality of Life Research, 14, 971-980.

SUNG, N. S., CROWLEY JR, W. F., GENEL, M., SALBER, P., SANDY, L., SHERWOOD, L. M., JOHNSON, S. B., CATANESE, V., TILSON, H. & GETZ, K. 2003. Central challenges facing the national clinical research enterprise. Jama, 289, 1278-1287.

SZENDE, A., OPPE, M. & DEVLIN, N. J. 2007. EQ-5D value sets: inventory, comparative review and user guide.

TAN, P.-N., STEINBACH, M. & KUMAR, V. 2006. Introduction to data mining, Pearson Addison Wesley Boston.

TIAN, Q., MA, J., LIANG, J., KWOK, R. C. & LIU, O. 2005. An organizational decision support system for effective R&D project selection. Decision Support Systems, 39, 403-413.

TICHY, N. M., TUSHMAN, M. L. & FOMBRUN, C. 1979. Social network analysis for organizations. Academy of management review, 4, 507-519.

TING, H. H., SHOJANIA, K. G., MONTORI, V. M. & BRADLEY, E. H. 2009. Quality improvement science and action. Circulation, 119, 1962-1974.

TING, S. & TSANG, A. H. 2014. Using social network analysis to combat counterfeiting. International Journal of Production Research, 52, 4456-4468.

TINGLEY, K. M. & LIEBMAN, J. S. 1984. A goal programming example in public health resource allocation. Management Science, 30, 279-289.

TODD, M. K. 2012. Physician Integration & Alignment: IPA, PHO, ACOs, and Beyond, CRC Press. TOMASZEWSKI, K. J. & HOLLOWAY, R. G. 2001. Deep brain stimulation in the treatment of Parkinson’s

disease a cost-effectiveness analysis. Neurology, 57, 663-671. TORRANCE, G. W., SIEGEL, J., LUCE, B., GOLD, M., RUSSELL, L. & WEINSTEIN, M. 1996. Framing and

designing the cost-effectiveness analysis. Cost-effectiveness in Health and Medicine, 54-81. TRAPPEY, C. V., TRAPPEY, A. J. & HWANG, S.-J. 1996. A computerized quality function deployment

approach for retail services. Computers & Industrial Engineering, 30, 611-622. TROCHIM, W., KANE, C., GRAHAM, M. J. & PINCUS, H. A. 2011. Evaluating translational research: a process

marker model. Clinical and translational science, 4, 153-162. TSAI, W. & GHOSHAL, S. 1998. Social capital and value creation: The role of intrafirm networks. Academy

of management Journal, 41, 464-476. TSUCHIYA, A., BRAZIER, J. & ROBERTS, J. 2006. Comparison of valuation methods used to generate the

EQ-5D and the SF-6D value sets. Journal of health economics, 25, 334-346. UDDIN, S., HOSSAIN, L., HAMRA, J. & ALAM, A. 2013. A study of physician collaborations through social

network and exponential random graph. BMC health services research, 13, 234. VAIDYA, O. S. & KUMAR, S. 2006. Analytic hierarchy process: An overview of applications. European

Journal of operational research, 169, 1-29. VAN ARENSBERGEN, P. & VAN DEN BESSELAAR, P. 2012. The selection of scientific talent in the allocation

of research grants. Higher Education Policy, 25, 381-405.

158

VAN STEL, H. F. & BUSKENS, E. 2006. Comparison of the SF-6D and the EQ-5D in patients with coronary heart disease. Health and Quality of Life Outcomes, 4, 20.

VASQUEZ, J. A., DIEHL, P. F., FLINT, C., SCHEFFRAN, J., CHI, S. H. & RIDER, T. J. 2011. The ConflictSpace of cataclysm: The international system and the spread of war 1914–1917. Foreign Policy Analysis, 7, 143-168.

VELAZQUEZ, M. A., CLAUDIO, D. & RAVINDRAN, A. R. 2010. Experiments in multiple criteria selection problems with multiple decision makers. International Journal of Operational Research, 7, 413-428.

VENKATRAMAN, N. 1989. Strategic orientation of business enterprises: the construct, dimensionality, and measurement. Management science, 35, 942-962.

WACHT, R. F. & WHITFORD, D. T. 1976. A goal programming model for capital investment analysis in nonprofit hospitals. Financial Management, 37-47.

WALTER, U., KLEINSCHMIDT, S., RIMMELE, F., WUNDERLICH, C., GEMENDE, I., BENECKE, R. & BUSSE, K. 2013. Potential impact of self-perceived prodromal symptoms on the early diagnosis of Parkinson’s disease. Journal of neurology, 260, 3077-3085.

WALTERS, S. J. & BRAZIER, J. E. 2005. Comparison of the minimally important difference for two health state utility measures: EQ-5D and SF-6D. Quality of Life Research, 14, 1523-1532.

WANG, G.-X. 2012. A network approach for researching political feasibility of healthcare reform: The case of universal healthcare system in Taiwan. Social Science \& Medicine.

WANG, T., ZHANG, Q., LIU, Z., LIU, W. & WEN, D. 2012. On social computing research collaboration patterns: a social network perspective. Frontiers of Computer Science, 6, 122-130.

WARE, J. E. & SHERBOURNE, C. D. 1992. The MOS 36-item short-form health survey (SF-36): I. Conceptual framework and item selection. Medical care, 473-483.

WEINSTEIN, M. C., SIEGEL, J. E., GOLD, M. R., KAMLET, M. S. & RUSSELL, L. B. 1996. Recommendations of the Panel on Cost-effectiveness in Health and Medicine. Jama, 276, 1253-1258.

WEINSTEIN, M. C., TORRANCE, G. & MCGUIRE, A. 2009. QALYs: the basics. Value in health, 12, S5-S9. WENG, C., GALLAGHER, D., BALES, M. E., BAKKEN, S. & GINSBERG, H. N. Understanding interdisciplinary

health sciences collaborations: a campus-wide survey of obesity experts. AMIA Annual Symposium Proceedings, 2008. 798.

WEST, E., BARRON, D. N., DOWSETT, J. & NEWTON, J. N. 1999. Hierarchies and cliques in the social networks of health care professionals: implications for the design of dissemination strategies. Social science \& medicine, 48, 633-646.

WESTFALL, J. M., MOLD, J. & FAGNAN, L. 2007. Practice-based research—“Blue Highways” on the NIH roadmap. Jama, 297, 403-406.

WHITEHEAD, W. E., BURNETT, C. K., COOK III, E. W. & TAUB, E. 1996. Impact of irritable bowel syndrome on quality of life. Digestive diseases and sciences, 41, 2248-2253.

WILLIAMS, A. 1995. The role of the EuroQol instrument in QALY calculations, Publication Secretary, Centre for Health Economics, University of York.

WILSON, P. F. 1993. Root cause analysis: A tool for total quality management, ASQ Quality Press. WINSLOW, C. A. 1951. The cost of sickness and the price of health. . Bulletin of the World Health

Organization, 84, 153. WISLØFF, T., HAGEN, G., HAMIDI, V., MOVIK, E., KLEMP, M. & OLSEN, J. A. 2014. Estimating QALY gains in

applied studies: a review of cost-utility analyses published in 2010. PharmacoEconomics, 32, 367-375.

WOLF, S. 1974. Editorial: The real gap between bench and bedside. The New England journal of medicine, 290, 802.

159

WONODI, C., PRIVOR-DUMM, L., AINA, M., PATE, A., REIS, R., GADHOKE, P. & LEVINE, O. 2012. Using social network analysis to examine the decision-making process on new vaccine introduction in Nigeria. Health policy and planning, 27, ii27--ii38.

WOOLDRIDGE, J. 2012. Introductory econometrics: A modern approach, Cengage Learning. WOOLF, S. H. 2007. Potential health and economic consequences of misplaced priorities. JAMA: the

journal of the American Medical Association, 297, 523-526. WOOLF, S. H. 2008. The meaning of translational research and why it matters. JAMA: the journal of the

American Medical Association, 299, 211-213. WORLD HEALTH ORGANIZATION 2009. WHO guide to identifying the economic consequences of disease

and injury. WORLD HEALTH ORGANIZATION 2013. WHO methods and data sources for global burden of disease

estimates 2000-2011. Global Health Estimates Technical Paper WHO/HIS/HSI/GHE. YAGER, R. R. 1993. Non-numeric multi-criteria multi-person decision making. Group Decision and

Negotiation, 2, 81-93. YANG, K. & EL-HAIK, B. S. 2003. Design for six sigma, McGraw-Hill New York. ZARE-FARASHBANDI, F., GERAEI, E. & SIAMAKI, S. 2014. Study of co-authorship network of papers in the

Journal of Research in Medical Sciences using social network analysis. Journal of research in medical sciences: the official journal of Isfahan University of Medical Sciences, 19, 41.

ZARE MEHRJERDI, Y. 2010. Quality function deployment and its extensions. International Journal of Quality & Reliability Management, 27, 616-640.

ZECKHAUSER, R. & SHEPARD, D. 1976. Where now for saving lives? Law and contemporary problems, 5-45.

ZERHOUNI, E. 2003. The NIH roadmap. Science, 302, 63-72. ZERHOUNI, E. A. & ALVING, B. 2006. Clinical and translational science awards: a framework for a national

research agenda. Translational Research, 148, 4-5. ZHAI, L., YAN, X., SHIBCHURN, J. & SONG, X. 2014. Evolutionary analysis of international collaboration

network of Chinese scholars in management research. Scientometrics, 98, 1435-1454. ZHANG, C., YU, Q., FAN, Q. & DUAN, Z. 2013. Research collaboration in health management research

communities. BMC medical informatics and decision making, 13, 52. ZHAO, G.-H., ZHANG, D.-W., CHENG, Y. & LI, W. 2014. A Study on the Job Satisfaction of Employees based

on Social Network Analysis. ZHAO, Y. J., WEE, H. L., CHAN, Y. H., SEAH, S. H., AU, W. L., LAU, P. N., PICA, E. C., LI, S. C., LUO, N. & TAN,

L. 2010. Progression of Parkinson's disease as evaluated by Hoehn and Yahr stage transition times. Movement Disorders, 25, 710-716.

ZUCKER, D. M. 2009. How to do case study research. School of Nursing Faculty Publication Series. Amherst, MA: University of Massachusetts - Amherst.

ZWART-VAN RIJKOM, J. E., LEUFKENS, H. G., BUSSCHBACH, J. J., BROEKMANS, A. W. & RUTTEN, F. F. 2000. Differences in attitudes, knowledge and use of economic evaluations in decision-making in the Netherlands. Pharmacoeconomics, 18, 149-160.

160

Appendix A. House of Quality

161

Appendix B. List of Proposals and Characteristics

Proposal

Proposal Classification Proposal and Team Characteristics PI expertise

Department Categories Disciplines

involved

Cross-

campus?

(1. Yes / 0

No)

NI

(number)

Team

Size

Fund req

(BUDi)

PI

experience

#

papers

Proposal 1 Biochemistry and Molecular

Biology Diabetes 2 1 1 4 75,000 3 3

Proposal 2 Medicine Heart failure 3 0 2 5 64,000 2 3

Proposal 3 Microbiology and Immunology Asthma 2 0 1 3 120,000 14 8

Proposal 4 Biochemistry and Molecular

Biology Breast Cancer 2 0 0 5 120,000 4 5

Proposal 5 Comparative Medicine Diabetes 4 1 3 7 94,000 10 13

Proposal 6 Pediatrics Acne 1 0 1 4 45,000 19 9

Proposal 7 Cellular and Molecular

Physiology Kidney disease 1 1 0 2 59,000 1 3

Proposal 8 Medicine Asthma 3 1 1 5 68,000 9 3

Proposal 9 Psychiatry Amnesia 1 0 1 3 80,000 3 2

Proposal 10 Microbiology and Immunology Allergy 4 0 2 7 92,000 9 7

Proposal 11 Humanities Alzheimer 3 0 2 6 55,000 10 12

Proposal 12 Public Health Sciences Diabetes 3 0 1 5 39,000 12 7

Proposal 13 Pediatrics Liver disease 2 1 2 7 55,000 6 2

Proposal 14 Psychiatry Sleep disorders 2 0 0 3 46,000 5 2

Proposal 15 Obstetrics and Gynecology Endocrinology 2 0 1 4 39,000 14 8

Proposal 16 Public Health Sciences Dementia 4 1 2 8 46,000 18 6

Proposal 17 Radiology Liver disorders 3 1 0 9 78,000 10 4

Proposal 18 Neural and behavioral Sciences Drug addiction 3 0 1 5 79,000 12 10

Proposal 19 Medicine Hepatic Failure 2 1 2 7 101,000 16 13

Proposal 20 Medicine Female infertility 4 1 2 7 115,000 19 15

162

Appendix C. Proposals’ coefficients

Proposal Proporti

on of YI PIexpi MDi Qi PBi

Proposal 1 0.250 0.488 0.667 88 0.300

Proposal 2 0.400 0.425 1.000 98 0.200

Proposal 3 0.333 1.000 0.667 97 0.675

Proposal 4 0.000 0.750 0.667 80 0.300

Proposal 5 0.429 1.000 1.000 61 0.700

Proposal 6 0.250 1.000 0.333 59 0.455

Proposal 7 0.000 0.363 0.333 78 0.130

Proposal 8 0.200 0.800 1.000 87 0.800

Proposal 9 0.333 0.388 0.333 66 0.195

Proposal 10 0.286 1.000 1.000 81 0.800

Proposal 11 0.333 1.000 1.000 66 0.700

Proposal 12 0.200 1.000 1.000 67 0.700

Proposal 13 0.286 0.575 0.667 80 0.450

Proposal 14 0.000 0.513 0.667 88 0.450

Proposal 15 0.250 1.000 0.667 67 0.525

Proposal 16 0.250 1.000 1.000 82 0.900

Proposal 17 0.000 0.900 1.000 87 0.800

Proposal 18 0.200 1.000 1.000 99 0.900

Proposal 19 0.286 1.000 0.667 77 0.675

Proposal 20 0.286 1.000 1.000 95 0.900

Appendix D. PBi coefficients

PI Experience

(Years)

Multidisciplinarity Score (MD)

0 - 0.25 0.25 - 0.50 0.50 - 0.75 0.75 - 1.00

Quality of the Proposal (Q)

0 - 25 25 - 50

50 - 75

75 - 100

0 - 25

25 - 50

50 - 75

75 - 100

0 - 25

25 - 50

50 - 75

75 - 100

0 - 25

25 - 50

50 - 75

75 - 100

0 - 2 0.00 0.00 0.05 0.10 0.00 0.00 0.07 0.13 0.00 0.00 0.08 0.15 0.00 0.00 0.10 0.20

2 - 4 0.00 0.05 0.15 0.20 0.00 0.07 0.20 0.26 0.00 0.08 0.23 0.30 0.00 0.10 0.30 0.40

4 - 6 0.05 0.10 0.20 0.30 0.07 0.13 0.26 0.39 0.08 0.15 0.30 0.45 0.10 0.20 0.40 0.60

6 - 8 0.05 0.15 0.35 0.40 0.07 0.20 0.46 0.52 0.08 0.23 0.53 0.60 0.10 0.30 0.70 0.80

More than 8 0.05 0.15 0.35 0.45 0.07 0.20 0.46 0.59 0.08 0.23 0.53 0.68 0.10 0.30 0.70 0.90

163

Appendix E. Distribution of enrollment in graduate school (For illustration purposes only)

Department Enrollment

Biochemistry and Molecular Biology 30

Cellular and Molecular Physiology 9

Comparative Medicine 7

Humanities 15

Medicine 75

Microbiology and Immunology 22

Neural and behavioral Sciences 12

Obstetrics and Gynecology 8

Pediatrics 16

Psychiatry 12

Public Health Sciences 45

Radiology 13

Total 264

Appendix F. LINDO Code

!OBJECTIVE

Min

0.021dR1P+0.021dR2P+0.021dR4P+0.021dR7P+0.021dR8P+0.021dR9P+0.021dR13P+0.021dR14P+0.021dR17P

+0.030dPBP+0.025dNIP+0.032dMD1P+0.032dMD3P+0.032dMD4P+0.032dMD6P+0.032dMD7P+0.032dMD9P+

0.032dMD13P+0.032dMD14P+0.032dMD15P+0.032dMD19P

Subject to

!GOAL CONSTRAINTS

!Risk Goals

-0.513x1 + dR1P - dR1N = 0

-0.575x2 + dR2P - dR2N = 0

-0.250x4 + dR4P - dR4N = 0

-0.638x7 + dR7P - dR7N = 0

-0.200x8 + dR8P - dR8N = 0

-0.613x9 + dR9P - dR9N = 0

-0.425x13 + dR13P - dR13N = 0

-0.488x14 + dR14P - dR14N = 0

-0.100x17 + dR17P - dR17N = 0

164

!Potential Benefits

0.300x1+0.200x2+0.675x3+0.300x4+0.700x5+0.455x6+0.130x7+0.800x8+0.195x9+0.800x10+0.700x11+0.700x12

+0.450x13+0.450x14+0.525x15+0.900x16+0.800x17+0.900x18+0.675x19+0.900x20+dPBP-dPBN=3.6

!Training New Generation

x1+2x2+x3+3x5+x6+x8+x9+2x10+2x11+x12+2x13+x15+2x16+x18+2x19+2x20+dNIP-dNIN=10

!Catalyzing multidisciplinary

-0.333x1 + dMD1P - dMD1N=0

-0.333x3 + dMD3P - dMD3N=0

-0.333x4 + dMD4P - dMD4N=0

-0.667x6 + dMD6P - dMD6N=0

-0.667x7 + dMD7P - dMD7N=0

-0.667x9 + dMD9P - dMD9N=0

-0.333x13 + dMD13P - dMD13N=0

-0.333x14 + dMD14P - dMD14N=0

-0.333x15 + dMD15P - dMD15N=0

-0.333x19 + dMD19P - dMD19N=0

!SYSTEM CONSTRAINTS

!Budget

75000x1+64000x2+120000x3+120000x4+94000x5+45000x6+59000x7+68000x8+80000x9+92000x10+55000x11+

39000x12+55000x13+46000x14+39000x15+46000x16+78000x17+79000x18+101000x19+115000x20 <= 600000

!Mutually Exclusiveness

x1+x5+x12 <=2

x3+x8 <=1

x13+x17 <=1

!Proportion

75000x1+120000x4<=204545

59000x7 <=61364

94000x5 <=47727

55000x11 <=102273

64000x2+68000x8+101000x19+115000x20 <=511364

120000x3+92000x10 <=150000

79000x18 <=81818

39000x15 <=54545

45000x6+55000x13 <=109091

80000x9+46000x14 <=81818

39000x12+46000x16 <=306818

78000x17 <=88636

!Minimum number of proposals

x1+x2+x3+x4+x5+x6+x7+x8+x9+x10+x11+x12+x13+x14+x15+x16+x17+x18+x19+x20 >=9

!Cross campus

x1+x5+x7+x8+x13+x16+x17+x19+x20 >= 2

!Non-negativity

dR1P >=0

dR2P >=0

165

dR4P >=0

dR7P >=0

dR13P >=0

dR14P >=0

dR17P >=0

dR1N >=0

dR2N >=0

dR4N >=0

dR7N >=0

dR13N >=0

dR14N >=0

dR17N >=0

dPBP >=0

dPBN >=0

dNIP >=0

dNIN >=0

dMD1P >=0

dMD1N >=0

dMD3P >=0

dMD3N >=0

dMD4P >=0

dMD4N >=0

dMD6P >=0

dMD6N >=0

dMD7P >=0

dMD7N >=0

dMD9P >=0

dMD9N >=0

dMD13P >=0

dMD13N >=0

dMD14P >=0

dMD14N >=0

dMD15P >=0

dMD15N >=0

dMD19P >=0

dMD19N >=0

END

int x1

int x2

int x3

int x4

int x5

int x6

int x7

int x8

int x9

int x10

166

int x11

int x12

int x13

int x14

int x15

int x16

int x17

int x18

int x19

int x20

Appendix G. MOS SF-36 (RAND 36-Items version). Obtained from www.rand.org

[1] In general, would you say your health is:

Excellent (1), Very good (2), Good (3), Fair (4), Poor (5)

[2] Compared to one year ago, how would you rate your health in general now:

Much better now than one year ago (1)

Somewhat better now than one year ago (2)

About the same (3)

Somewhat worse now than one year ago (4)

Much worse now than one year ago (5)

The following items are about activities you might do during a typical day. Does your health now limit you in these

activities? If so, how much? (Circle one number on each line)

Yes, Limited a lot

Yes, Limited a little

No, Not limited at all

[3] Vigorous activities, such as running, lifting heavy objects, participating in strenuous sports

(1) (2) (3)

[4] Moderate activities, such as moving a table, pushing a vacuum cleaner, bowling, or playing golf

(1) (2) (3)

[5] Lifting or carrying groceries (1) (2) (3)

[6] Climbing several flights of stairs (1) (2) (3)

[7] Climbing one flight of stairs (1) (2) (3)

[8] Bending, kneeling, or stooping (1) (2) (3)

[9] Walking more than a mile (1) (2) (3)

[10] Walking several blocks (1) (2) (3)

[11] Walking one block (1) (2) (3)

[12] Bathing or dressing yourself (1) (2) (3)

167

During the past 4 weeks, have you had any of the following problems with your work or other regular daily activities as

a result of your physical health? (Circle one number on each line)

Yes No

[13] Cut down the amount of time you spent on work or other activities (1) (2)

[14] Accomplished less than you would like (1) (2)

[15] Were limited in the kind of work or other activities (1) (2)

[16] Had difficulty performing the work or other activities (for example, it took extra effort) (1) (2)

During the past 4 weeks, have you had any of the following problems with your work or other regular daily activities as

a result of any emotional problems (such as feeling depressed or anxious)? (Circle one number on each line)

Yes No

[17] Cut down the amount of time you spent on work or other activities (1) (2)

[18] Accomplished less than you would like (1) (2)

[19] Didn't do work or other activities as carefully as usual (1) (2)

[20] During the past 4 weeks, to what extent has your physical health or emotional problems interfered with your normal

social activities with family, friends, neighbors, or groups? (Circle one number)

Not at all (1), Slightly (2), Moderately (3), Quite a bit (4), Extremely (5)

[21] How much bodily pain have you had during the past 4 weeks? (Circle one number)

None (1), Very mild (2), Mild (3), Moderate (4), Severe (5), Very severe (6)

[22] During the past 4 weeks, how much did pain interfere with your normal work (including both work outside the home

and housework)?

Not at all (1), A little bit (2), Moderately (3), Quite a bit (4), Extremely (5)

168

These questions are about how you feel and how things have been with you during the past 4 weeks. For each question,

please give the one answer that comes closest to the way you have been feeling. How much of your time during the

past 4 weeks…

All of the Time

Most of the Time

A Good Bit of the Time

Some of the Time

A Little of the Time

None of the Time

[23] Did you feel full of pep? (1) (2) (3) (4) (5) (6)

[24] Have you been a very nervous person? (1) (2) (3) (4) (5) (6)

[25] Have you felt so down in the dumps that nothing could cheer you up?

(1) (2) (3) (4) (5) (6)

[26] Have you felt calm and peaceful? (1) (2) (3) (4) (5) (6)

[27] Did you have a lot of energy? (1) (2) (3) (4) (5) (6)

[28] Have you felt downhearted and blue? (1) (2) (3) (4) (5) (6)

[29] Did you feel worn out? (1) (2) (3) (4) (5) (6)

[30] Have you been a happy person? (1) (2) (3) (4) (5) (6)

[31] Did you feel tired? (1) (2) (3) (4) (5) (6)

[32] During the past 4 weeks, how much of the time has your physical health or emotional problems interfered with

your social activities (like visiting with friends, relatives, etc.)?

All of the time (1), Most of the time (2), Some of the time (3), A little of the time (4),

None of the time (5)

[33] How TRUE or FALSE is each of the following statements for you.

Definitely True

Mostly True

Don't Know

Mostly False

Definitely False

33. I seem to get sick a little easier than other people (1) (2) (3) (4) (5)

34. I am as healthy as anybody I know (1) (2) (3) (4) (5)

35. I expect my health to get worse (1) (2) (3) (4) (5)

36. My health is excellent (1) (2) (3) (4) (5)

169

Appendix H. SF-12 Health Survey (http://www.sf-36.org/demos/SF-12.html)

[1] In general, would you say your health is:

Excellent (1), Very good (2), Good (3), Fair (4), Poor (5)

[2] The following questions are about activities you might do during a typical day. Does your health now limit you in

these activities? If so, how much?

Yes, Limited a lot

Yes, Limited a little

No, Not limited at all

[a] Moderate activities, such as moving a table, pushing a vacuum cleaner, bowling, or playing golf

(1) (2) (3)

[b] Climbing several flights of stairs (1) (2) (3)

[3] During the past 4 weeks, have you had any of the following problems with your work or other regular daily activities

as a result of your physical health?

Yes No

[a] Accomplished less than you would like (1) (2)

[b] Were limited in the kind of work or other activities (1) (2)

[4] During the past 4 weeks, have you had any of the following problems with your work or other regular daily

activities as a result of any emotional problems (such as feeling depressed or anxious)?

Yes No

[a] Accomplished less than you would like (1) (2)

[b] Did work or other activities less carefully than usual (1) (2)

[5] During the past 4 weeks, how much did pain interfere with your normal work (including both work outside the

home and housework)?

Not at all (1), A little bit (2), Moderately (3), Quite a bit (4), Extremely (5)

[6] These questions are about how you feel and how things have been with you during the past 4 weeks. For each

question, please give the one answer that comes closest to the way you have been feeling. How much of the time

during the past 4 weeks...

All of the Time

Most of the Time

A Good Bit of the Time

Some of the Time

A Little of the Time

None of the Time

[a] Have you felt calm and peaceful? (1) (2) (3) (4) (5) (6)

[b] Did you have a lot of energy? (1) (2) (3) (4) (5) (6)

[c] Have you felt downhearted and blue? (1) (2) (3) (4) (5) (6)

[7] During the past 4 weeks, how much of the time has your physical health or emotional problems interfered with your

social activities (like visiting friends, relatives, etc.)?

All of the time (1), Most of the time (2), Some of the time (3), A little of the time (4),

None of the time (5)

Vita David A. Munoz

EDUCATION

Ph.D. Dual degree in Industrial Engineering and Operations Research Aug 15

Master of Engineering in Industrial Engineering Dec 13

Department of Industrial & Manufacturing Engineering GPA: 3.86

The Pennsylvania State University, University Park, United States

Industrial Engineering Professional Title May 09

Bachelor Degree in Engineering Sciences Sept 08

Department of Industrial Engineering

Universidad del Bío-Bío, Concepción, Chile

ACADEMIC EXPERIENCE

SCHOLAR (Global Engineering Leadership Program) Sep 14 – Aug 15

SCHOLAR (Center for Health Organization Transformation, CHOT, NSF) Aug 13 – Aug 15

SCHOLAR (Center for Integrated Healthcare Delivery Systems, CIHDS) Sep 11 – Aug 15

RESEARCH ASSISTANT (Clinical and Translational Science Institute, CTSI, NIH) Jan 13 – Aug 15

INSTRUCTOR - SIX SIGMA METHODOLOGY (IE436) Jan 14 – May 14

GRADER – STATISTICAL QUALITY CONTROL (IE434) Aug 12 – Dec 12

GRADUATE RESEARCHER (Supported by the Defense Advanced Research Project Agency, DARPA) Jan 12 – Dec 12

Department of Industrial & Manufacturing Engineering

The Pennsylvania State University, University Park, PA, USA

RESEARCHER (Thesis Research) Sep 08 – Apr 09

Modeling & Simulation Army Center, Chilean Army, Santiago, Chile

TEACHING ASSISTANT Mar 05 – Aug 08

Differential and Integral Calculus, Multivariable Calculus, Thermodynamics, Engineering Economy and Simulation.

Department of Industrial Engineering, Universidad del Bío-Bío, Concepción, Chile

PUBLICATIONS

Published papers

Munoz, D., Nembhard, H., and Kraschnewski, J., (2014). “Quantifying complexity in translational research: An integrated

approach.” International Journal of Health Care Quality Assurance, Vol. 27 Iss: 8

Munoz, D., Queupil, J., and Fraser, P., (Forthcoming). “Assessing Collaboration Networks in Educational Research: A Social

Network Analysis Approach.” International Journal of Educational Management

Munoz, D., and Bastian, N. (Forthcoming). “Estimating Cross-training Call Center Capacity through Simulation.” Journal of

Systems Science and Systems Engineering.

Munoz, D. (Forthcoming). “Assessing the research efficiency of higher education institutions in Chile –a data envelopment

analysis approach.” International Journal of Educational Management.

Papers under Journal Review

Munoz, D., Nembhard, H., and Kraschnewski, J., (Submitted). “Social Network Analysis to Evaluate Intra-Institutional

Collaboration Capacity – A Case Study on Obesity.”

Munoz, D., Bastian, N., and Ventura, M., (Submitted). “A Mixed-Methods Approach for Healthcare Process Improvement: A

Case Study.”

Munoz, D., and Tucker, C., (Submitted). “A Semantic Network for Modeling the Structure of Textually-Derived Information

and its Impact on Receivers' Response States.”

Conference Proceedings

Munoz, D., and Kang, H. (2015), “A Dynamic Network Analysis Approach for Evaluating Knowledge Dissemination in a Multi-

Disciplinary Collaboration Network in Obesity Research.” Winter Simulation Conference (WSC). Huntington Beach, CA, USA.

Munoz, D., and Tucker, C., (2014) “Assessing Students’ Emotional States: An Approach to Identify Lectures that Provide an

Enhanced Learning Experience.” In Proceedings of The International Design and Engineering Technical Conferences (ISERC).

Buffalo, New York, USA.

Munoz, D., Bastian, N., and Ventura, M., (2014) “A Workflow Assessment for a Pediatric Intensive Care Unit: A Mixed-

Methods Approach.” In Proceedings of The ISERC. Montreal, Canada. (Best Paper Award of the Healthcare Systems Track)

Munoz, D., Alonso, W., and Nembhard, H., (2014) “A Social Network Analysis-based Approach to Evaluate Workflow and

Quality in an Intensive Care Unit.” In Proceedings of The ISERC. Montreal, Canada.

Gillam, P., Nembhard, H., and Muñoz, D., (2014) “The Role of Quality Improvement Methods in Translational Research.” In

Proceedings of The ISERC. Montreal, Canada. 2014.

Munoz, D., and Brutus, M., (2013) “Understanding the Trade-offs in a Call Center.” In Proceedings of WSC. Washington D.C.

Date post:	25-Mar-2022
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

QUALITY IMPROVEMENT TO ASSESS AND AUDIT COMPLEXITY …

Documents