Date post: | 17-Jan-2016 |
Category: |
Documents |
Upload: | stephanie-booth |
View: | 213 times |
Download: | 0 times |
What Might a Theory-Based Roadmap for Prospective Evaluation
and Developing Innovation Policy Look Like?
Presented atAmerican Evaluation Association Conference
November 2009 Gretchen Jordan, Sandia National Laboratories
Portions of the work presented here were completed for the U.S. DOE Office of Science by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the author.
SAND Number: 2009-7359C
G. Jordan AEA November 2009 2
• Prospective evaluation in context
• National interest (SoSP, SciSIP)
• Models of what is known about innovation, what we need to know
• Theories and an example
• Conclusions
Outline
G. Jordan AEA November 2009 3
Evaluation in the Policy Cycle
Foresight Technology Roadmapping
TechnologyAssessment
Wolfgang Polt30-10-2007
G. Jordan AEA November 2009 4
National Interest: SoSP and SciSIP
• The science of science policy (SoSP) is an emerging field of interdisciplinary research, the goal of which is to provide a scientifically rigorous, quantitative basis from which policy makers and researchers can assess the impacts of the Nation’s scientific and engineering enterprise, improve their understanding of its dynamics, and assess the likely outcomes.
• A National Science and Technology Council (NSTC) Interagency Task Group (ITG)• The Science of Science & Innovation Policy (SciSIP) program was established at NSF in 2005.
John MarburgerApril 2005
G. Jordan AEA November 2009 5
SoSP Workshop in December 2008
Primary Conclusion of SoSP Roadmap: “Expert judgment” remains the best available
decision support tool for science policy makers, but
a nascent community of practice is emerging in the science policy arena that holds enormous potential to provide rigorous and quantitative decision support tools in the near future. ”
The White House SoSP Interagency Task Group should take the lead to set the Federal agency research agenda.
G. Jordan AEA November 2009 6
White House S&T Priorities for the FY 2011 Budget
Agencies should describe in their budget submission how they are• prioritizing activities toward four challenges and strengthening four
cross-cutting areas (which include productivity of research institutions)
• Expecting outcomes of research in above areas, providing quantitative metrics where possible
• Building capacity to rigorously evaluate programs, and how assessments have been used to eliminate or reduce programs
• Operating in the open innovation model and supporting long term high-risk, high payoff research
Agencies will:• Develop outcome oriented goals for S&T, target investment toward
high performers, develop ‘science of science policy” tools that can improve management and assessment of impact
-Peter Orszag, John Holdren, August 4, 2009
G. Jordan AEA November 2009 7
The SoSP Roadmap 10 Science Questions
1. What Are The Behavioral Foundations Of Innovation?
2. What Explains Technology Development, Adoption And Diffusion?
3. How And Why Do Communities Of Science And Innovation Form And Evolve?
4. What Is The Value Of The Nation’s Public Investment In Science?
5. Is It Possible To “Predict Discovery”?
6. Is It Possible To Describe The Impact Of Discovery On Innovation?7. What Are The Determinants Of Investment Effectiveness?8. What Impact Does Science Have On Innovation And Competitiveness?9. How Competitive Is The U.S. Scientific Workforce?10. What Is The Relative Importance Of Different Policy Instruments In Science Policy?
Theme 1: Understanding Science
and Innovation
Theme 2: Investing in Science
and Innovation
Theme 3: Using the Science of Science Policy to
Address National Priorities
The National Imperative
Science Questions
Findings
Recommendations
Source: J. Lane, April 2009
G. Jordan AEA November 2009 8
http://www.cs.unibo.it/schools/AC2005/docs/Bertinoro.ppt#266,11,The Blind Men and the Elephant
Parts are studied and understood better than the whole!
Source: Bhavya Lal, STPI, at AEA 2006
G. Jordan AEA November 2009 9
A Science of Science and Innovation Policy must build a theory that connects levels
ResearchTeam
ResearchOrganization
The Sector’sIdea Innovation
Network
The Sector’sNational and
Global Context
micro meso macro
G. Jordan AEA November 2009 10
Anticipate effects of Scientific discovery
Anticipate effects of
science on R&D
Understand technology
development & diffusion
Understand behavioral
foundations
Understand network
behaviors
Science Workforce
competitiveness
Impacts on competitiveness,
etc.
Assess real time value of new knowledge
Relative importance of
policy instruments
Determinants of investment effectiveness
7
2
31
810
9
65 4
SoSP Roadmap Questions Rearranged into a Three Level Logic Model
Draft by G. Jordan 12/12/2008
Investment, incentives, Use
People & organizational inputs & incentives
The (non-linear) S&T, R&D process
Understanding a multi-level eco-system
Macro
Micro
Meso
G. Jordan AEA November 2009 11
The science must explain relationships among institutions
Demand
Consumers (final demand)Producers (intermediate demand)
Industrial system Education and research system
Political system
Government
Governance
RTD Policies
Professional educationand training
Higher educationand research
Public sectorresearch
Large companies
Mature small/ mediumenterprises ( SMEs)
New, technology -based firms
Infrastructure
Intermediaries
ResearchinstitutesBrokers
Banking, venture capital
IPR and information
Innovation andbusiness support
Standardsand norms
Framework conditionsFinancial environment; taxation and incentives; propensity to innovation
and entrepreneurship; mobility
A National Innovation System Model
The potential reachof public policies …
Demand
Consumers (final demand)Producers (intermediate demand)
Industrial system Education and research system
Political system
Government
Governance
RTD Policies
Professional educationand training
Higher educationand research
Public sectorresearch
Large companies
Mature small/ mediumenterprises ( SMEs)
New, technology -based firms
Infrastructure
Intermediaries
ResearchinstitutesBrokers
Banking, venture capital
IPR and information
Innovation andbusiness support
Standardsand norms
Framework conditionsFinancial environment; taxation and incentives; propensity to innovation
and entrepreneurship; mobility
Source: Arnold and Kuhlman, 2001
A National Innovation System Model
The potential reachof public policies …
G. Jordan AEA November 2009 12
Source: G. Jordan, 2007. Modified from R. Cooper/ Exxon’s Stage Gate, Hage & Hollingsworth’s Idea Innovation Network
Marketing R&D, Quality R&D
Diffusion and use
Engineering & manufacturing R&D
7
8
6
Connectivity and Throughput
Production, Refinement
Micro, meso, macro
impacts9
10
The science must explain connections among arenas of research and development
G. Jordan AEA November 2009 13
ConfirmationAwareness Persuasion Decision Implementation
Feedback
Continued adoptionLater adoption
DiscontinuanceContinued rejection
Adoption
Rejection
Product Characteristics
• Relative advantage
• Compatibility
• Complexity
• Trialability
• Observability
Characteristics of the decision-making unit
• Adopter type
• Personality type
• Communication behavior
• Socio-economic status
Socio-cultural/market environment
• Market structure
• Market segments
• Prior practice
• Culture and norms
• Innovativeness
Communication field• Broadcast
• Contagion
Source: Everett Rogers 1994 as modified by Innovologie, LLC. 2005
The science must understand Diffusion and relate it to R&D
G. Jordan AEA November 2009 14
All this information is useful to predict where and how policy makers can intervene to achieve desired goals
Marketing R&D, Quality R&D
Diffusion and use
Engineering & manufacturing R&D
78
6
Connectivity and Throughput
Production, Refinement
Micro, meso, macro
impacts910
Marketing R&D, Quality R&D
Diffusion and use
Engineering & manufacturing R&D
78
6
Connectivity and Throughput
Production, Refinement
Micro, meso, macro
impacts910
Socio-cultural/market environment
• Market structure
• Market segments
• Prior practice
• Culture and norms
• Innovativeness
Socio-cultural/market environment
• Market structure
• Market segments
• Prior practice
• Culture and norms
• Innovativeness
ConfirmationAwareness Persuasion Decision Implementation
Feedback
Continued adoptionLater adoption
DiscontinuanceContinued rejection
Adoption
Rejection
Product Characteristics
• Relative advantage
• Compatibility
• Complexity
• Trialability
• Observability
Characteristics of the decision-making unit
• Adopter type
• Personality type
• Communication behavior
• Socio-economic status
Communication field• Broadcast
• Contagion
ConfirmationAwareness Persuasion Decision Implementation
Feedback
Continued adoptionLater adoption
DiscontinuanceContinued rejection
Adoption
Rejection
Product Characteristics
• Relative advantage
• Compatibility
• Complexity
• Trialability
• Observability
Characteristics of the decision-making unit
• Adopter type
• Personality type
• Communication behavior
• Socio-economic status
Communication field• Broadcast
• Contagion
Interventions at micro, meso, and/or macro
levels?
G. Jordan AEA November 2009 15
Theories that could be integrated to understand how we can drive innovation
Research Team– Management of innovation literature, learning theory
Research Organization– Organizational innovation theories– Research Profiles theory
Science/technological Sector– Idea Innovation Network on S&T/R&D process– Network theories– Diffusion theory– Sector economic models
National and global context– Modes of coordination theories – Institutional and institutional change theory– Policy decision making –theories of
G. Jordan AEA November 2009 16
One possible decision tool to identify bottlenecks, policy objectives & effectiveness
Socio economic outcomes
Technical progress
Network connectedness
Organizational profiles – do
attributes match the profile?
RTD arenas – are there sufficient funds
Portfolios -need more/ less radical, large scope?
Modes of coordination –
effective?
Capabilities –Level, mix, availability
High riskcapital –
available where
Basic research
Manufacturingresearch
Applied research
Development research
Quality research
Commercialization research
Macro- Institutional Rules as they affect the sector
Micro - funds allocation by arena and profile
INNOVATION
Meso - Performance byTech sector and arena
Policy Objectives-Structural-Technical
Source: Jordan, Hage, and Mote, 2006, 2007, 2008
G. Jordan AEA November 2009 17
Conclusion
• Innovation occurs within a multi-level, complex, dynamic eco-system
• Prospective evaluation predicts
• Prediction requires understanding, characterization, theory
• There are theories that can be used now
• Synthesis of existing theories and building new theories are needed going forward.
G. Jordan AEA November 2009 18
Selected ReferencesArnold, E. (2004). Evaluating research and innovation policy: A systems world needs systems evaluations. Research Evaluation, 13(1), 3-17
Hage, Jerry, G.B. Jordan and J. Mote (2007). A Theories-Based Innovation Systems Framework for Evaluating Diverse Portfolios of Research: Part Two - Macro Indicators and Policy Interventions. Science and Public Policy, 34(10): 731-741.
Jordan, G. B., Hage, J., & Mote, J. 2008. A theories-based systemic framework for evaluating diverse portfolios of scientific work, part 1: Micro and meso indicators. In C.L.S. Coryn & Michael Scriven (Eds.), Reforming the evaluation of research. New Directions for Evaluation, 118, 7–24.
Jordan, G.B. 2006. Factors Influencing Advances in Basic and Applied Research: Variation Due to Diversity in Research Profiles. In Innovation, Science, and Institutional Change: A Handbook of Research, J. Hage and M. Meeus (eds). Oxford University Press: Oxford, 173-195.
Mote, J., Y. Whitestone, G. Jordan and J. Hage. 2008. Innovation, Networks and the Research Environment: Examining the Linkages. International Journal of Foresight and Innovation Policy 4(3): 246-264.
Reed, John H, G. Jordan, Using Systems Theory and Logic Models to Define Integrated Outcomes and Performance Measures in Multi-program Settings, in Research Evaluation, Volume 16 Number 3 September 2007.