Date post: | 15-Nov-2014 |
Category: |
Science |
Upload: | kevin-boyack |
View: | 68 times |
Download: | 1 times |
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Indicators of Innovative Research2014 Science & Technology Indicator Conference
Leiden UniversityThe Netherlands
Sept 3, 2014
Richard Klavans (presenter)Kevin W. BoyackHenry SmallAaron A. SorensenJohn P.A. Iaonnidis
SciTech Strategies Berwyn, PA.SciTech Strategies, Albuquerque, NMSciTech Strategies, Bala Cynwd, PA.
Temple University School of Medicine, Philadelphia, PA.Stanford University School of Medicine, Stanford, CA.
2
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
“What gets us into trouble is not what we don't know. It's what we know for sure that just ain't so.”
Mark Twain
Motivation
3
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Motivation
Ioannidis et al [Nature, 2013] assumes that highly cited papers are more innovative
4
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Motivation
We hope that the ‘distance’ betweena citing paper & its references will correlate with innovativeness
Ioannidis et al [Nature, 2013] assumes that highly cited papers are more innovative
5
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Motivation
Uzzi et al. [Science, 2013]argue that one needs a combination oftypical & atypicalknowledge relationships in order to be innovative. He also assumes thatinnovative papers arehighly cited.
We hope that the ‘distance’ betweena citing paper & its references will correlate with innovativeness
Ioannidis et al [Nature, 2013] assumes that highly cited papers are more innovative
6
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Is this so?
Invited 400 biomedical researchers to participate in a survey» Invitation came from Ioannidis (Stanford University)» Researchers had over 25,000 citations and H index > 75» Participation rate was 30.8% (123 out of 400)
Researchers were asked to rate their top 10 (most cited) papers» Papers had to be published between 2005-2008 (& in the Scopus database)» Ratings were along six dimensions of impact
Survey results will be compared with bibliometric indicators» Traditional Indicators» Indicators suggested by K&B [2013]» Indicators suggested by Uzzi et al. [2013]
7
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Survey Design
8
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Survey Analysis
Assigned Papers to Impact Categories
9
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Bibliometric Data*
Common Sense 1. Review Paper?2. # of References3. Team Science (# of Authors)
K&B [2013]4. Distance between citing paper and cited references
USMJ [2013] 5. Typical Knowledge Relationships6. Atypical Knowledge Relationships
* Scopus Data
10
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Finding #1:
Most High Impact Papers represent ‘Normal Science’
Type of Impact # of Papers % of Sample
Progress 335.4 28.8%
Synthesis 262.9 21.3%
Broader Interest 220.0 17.8%
Innovation 195.9 15.9%
Surprise 99.3 8.0%
Difficulty 41.6 3.4%
Unassigned 58.0 4.7%
68.9%
27.3%
11
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Finding #2:
Traditional indicators work (but ‘team science’ is associated with the less innovative
papers!) Type of Impact % Review # Ref #Authors
Progress 12.5 51.7 17.8
Synthesis 26.1 71.0 16.2
Broader Interest 17.3 59.8 15.5
Innovation 6.3 42.9 12.3
Surprise 4.5 41.7 11.4
Difficulty 12.7 51.1 9.8
Unassigned 12.1 55.0 14.9
F-stat 13.8 9.63 1.90
Probability .0000 .0000 .0773
12
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Finding #3:
‘Distance’ does not identify innovative papers
Type of Impact Distance
Progress 2.05
Synthesis 2.01
Broader Interest 2.13
Innovation 2.05
Surprise 2.04
Difficulty 1.98
Unassigned 2.04
F-Stat .42
Probability .867
13
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Finding #4:
Only atypical knowledge relationships works
Type of Impact Atypical Typical
Progress .491 .511
Synthesis .506 .480
Broader Interest .574 .475
Innovation .636 .467
Surprise .608 .500
Difficulty .620 .451
Unassigned .672 .500
F-stat 4.61 .41
Probability .0001 .875
14
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Finding #5
And the Overall Explanatory Value is very low (R2~.033)
Regression Equation Dependent Innovation or Surprise (0,1 variable) Variable:
t P>|t| Independent Review Paper (0,1) -5.14 .000 Variables: References Log(# ref) -2.08 .037
Team Log(# authors) -1.81 .071Atypical (0,1) 3.87 .000
15
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
highly cited papers are not necessarily innovative.
Implications
‘distance’ didn’t work.
Atypical knowledge relationships works….
but why?
16
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Implications
And we still don’t know how to identify innovative papers
17
Physics Chemistry Engineering Biology Disease Medicine Brain Health SocialComputer HumanitiesEarth
Thank-you for your attention!