Measuring Interdisciplinarity: A Unique Comparison Between the Researcher and

Research Proposal

Asha BalakrishnanVanessa Peña

Bhavya Lal (blal@ida.org), Task Leader

November 5, 2011American Evaluation Association, 2011

Relevant Task Goal• Federal agency wished to know if their set-

aside potential transformative research (PTR) program was funding interdisciplinary research (IDR)

11/5/2011

Associated Study Questions

• How interdisciplinary is the set-aside program’s research [as compared with traditional programs]? – How interdisciplinary is the body of knowledge on

which the awarded proposal draws?• How interdisciplinary is the funded PI [as

compared with PIs of traditional programs]?– How interdisciplinary is the PI’s prior publication

history?

11/5/2011

Definition of Interdisciplinarity

• Interdisciplinary approaches integrate separate disciplinary data, methods, tools, concepts, and theories in order to create a holistic view or common understanding of a complex issue, question, or problem.

• The critical indicators of interdisciplinarity in research include evidence that the integrative synthesis is different from, and greater than, the sum of its parts.

• Process of knowledge integration is important11/5/2011

C.S. Wagner et al. / Journal of Informetrics 165 (2011) 14–26

Operationalizing Interdisciplinarity – Concept of Diversity Score

• What is the metric?– The “I-score” measures the number of disparate scientific

fields connected by a researcher’s body of work. (Porter, 2007)• How is it measured?

– By using the subject categories of the cited references. A co-citation analysis of all journals has been measured empirically.

• What does it intend to represent?– The I-score intends to measure the interdisciplinarity of the

researcher’s work. – Low I-scores indicate that the researcher draws primarily from

a specialized body of work.

11/5/2011

Evaluative Approach

I-Score metric was employed to test if differences existed between test program awards and comparison awards to determine if:a) Test program is attracting more

interdisciplinary PIsb) Test program is soliciting more

interdisciplinary proposals

11/5/2011

Comparison Group Selection

Selection of Comparison Awards:To assess the difference between the proposals funded by test program and proposals funded through traditional mechanisms, we considered many options and then selected a set of funded proposals from programs as the comparison group.

Selection Methodology:

Identified a set of comparison awards from funded active awards that were:• On the order of total funding of test

program• On the order of total duration of test

program

Randomly selected 44 “comparison” awards

44 “test program” awards

11/5/2011

PROPOSAL INTERDISCIPLINARITY

11/5/2011

Measuring Interdisciplinarity of Proposals

• Activity:– Gathered the cited references from the Treatment

and Comparison Group proposals– Extracted journal titles from references and

matched to Thomson ISI subject categories– Used techniques and tools described in Porter

et.al., 2010* to compute integration scores

*Porter, et al. Practical research proposal and publication profiling. Research Evaluation 19(1): 29-44 (2010).

11/5/2011

Caveats

• We assume that the references cited in the awarded proposal represent the interdisciplinarity of the proposed ideas

• Not all references cited in a proposal were indexed in ISI’s Web of Knowledge database (ranged from 25% to 90% coverage)

• Subject category assignment of journals is a proxy for “discipline”

11/5/2011

Treatment Proposal Integration Scores are Higher than the Comparison Group Proposal Integration Scores

Less than 0.35 0.351 to .45 0.451 to 0.55 0.551 to 0.65 0.651 to 0.75 0.751 to 0.85 Above 0.85

Test 1 1 5 12 21 4 0

Comparison 2 6 14 15 4 3 0

2.5

7.5

12.5

17.5

22.5

Coun

t of P

ropo

sals

11/5/2011

Visual Representation of Average Test and Comparison Group Proposal Integration

ScoreTest Program FY2007-FY2009

Average I-Score = 0.64Comparison Group FY2007-FY2009

Average I-Score = 0.55

Env Sci & Tech

Bus & Mngt

Econ. Polit. & Geo Comp Sci

Eng

Clin Med

Biomed

Cog Sci

Matl Sci

Chem

Inf Dis

Geo

Agri Sci

Hlth & Soc

Ecol Sci

Clin Med

Biomed

Inf Dis

Matl Sci

Geo

Bus & Mngt

Econ. Polit. & Geo Comp Sci

Eng

PhysPhys

Psych

Ecol Sci

Psych

11/5/2011

Distribution of Disciplines – Project Level

I-Score = 0.64

I-Score = 0.68

I-Score = 0.6511/5/2011

RESEARCHER INTERDISCIPLINARITY

11/5/2011

Measuring Principal Investigator Interdisciplinarity

• Activity:– Searched and obtain each of the Treatment and

Comparison Group Principal Investigator’s publication histories (from Jan 1980 to June 2010) in the ISI Web of KnowledgeSM database

– Employed integration score technique based on Porter et.al* to determine integration score of each PI (44 test PIs and 44 Comparison group PIs)

– For PIs with a low number of publications, sensitivity analyses were conducted to ensure conclusions still held

*Porter, et al. Measuring researcher interdisciplinarity. Scientometrics 72(1): 117-147 (2007).11/5/2011

Caveats

• Only journal publications available in ISI’s Web of Knowledge database included

• Subject category assignment of journals is a proxy for “discipline”

• Only measured the PI’s interdisciplinarity, not those of the Co-PI’s or the team

11/5/2011

No Difference Between Treatment PI Integration Scores and Comparison Group PI Integration Scores

Less than 0.35

0.351 to .45 0.451 to 0.55

0.551 to 0.65

0.651 to 0.75

Above 0.75

Test 4 16 11 8 5 0

Compar-ison

4 11 11 13 5 0

1

3

5

7

9

11

13

15

17

Coun

t of P

ropo

sals

11/5/2011

Average Test and Comparison Group PI Integration Score

Comparison Group FY2007-FY2009Average PI I-Score = 0.48

Test Program FY2007-FY2009Average PI I-Score = 0.50

Env Sci & Tech

Bus & Mngt

Econ. Polit. & Geo Comp Sci

Eng

Clin Med

Biomed

Cog Sci

Matl Sci

Comp Sci

Eng

Clin Med

Biomed

Cog Sci

Hlth & Soc

Inf Dis

Phys

Chem

Geo

Env Sci & Tech

Matl Sci

Chem

Econ. Polit. & Geo

Inf Dis

Geo

11/5/2011

Summary of Findings

• I-scores for test group PIs and comparison group PIs are not different

• I-scores from test groups proposal cited references are higher than I-scores from comparison group proposal cited references

Set-aside PTR program is not attracting PIs with a more interdisciplinary background than other programs. However, program is successful attracting researchers to submit more interdisciplinary ideas!

11/5/2011

Methodological Finding: I-score does not tell the complete story

ExamplePI Maps

ExampleProposal Maps

I-Score = 0.41I-Score = 0.64 I-Score = 0.73

I-Score = 0.42I-Score = 0.72 I-Score = 0.54

11/5/2011

Caveats: Study Based on Many Assumptions

• I-score is a good measure of interdisciplinarity– Little is known about how subject categories are

generated and how static they are over time– Journal coverage in databases is incomplete

• Proposal cited references reflect knowledge integration– Rhetorical vs. reward citations

11/5/2011

Future Questions of Interest

• How interdisciplinary was the team (integrate all PI publications into one set and determine i-score)– Are younger PIs more interdisciplinary?

• Testing on larger population of awards – What is the relationship between PI and proposal I-scores?

• Exploratory look found none in the test group, and significantly positive for comparison group

– What is the relationship between Proposal Integration Score and PTR Score?• Exploratory look found none

11/5/2011