The Impact of Research on Development Policy and Practice:
This much we know.A Literature Review and the Implications for ICT4D
Dr. Roger HarrisVisiting Professor
Institute of Social Informatics and Technological [email protected]://www.rogharris.org/
@HarrisrwhRoger
INTRODUCTION
Higher Education Minister Datuk Seri Mohamed Khaled
Nordin
“Universities should live up to their roles
within the local community while
making an impact in the international
arena.” What does he mean by impact?
Universities must make an impact
What is the impact of research?
Academic Research
Research ImpactsAcademic Impact
TrainingPublications Citations
Socio-economic Impact
Capacity building
Socio-economic benefits
Public Policy
Research Impacts
Academic Research
Research Impacts
Academic Impact
TrainingPublications Citations
Socio-economic Impact
Capacity building
Socio-economic benefits
Public Policy
My question is, are we making any impact?
Achieving either type of impact does not automatically lead to achieving
the other.
There are considerable differences between the processes that lead to
each.
High quality research is always the starting point for both.
Non-academic impacts require more attention than academic.
Two communities
Researchers
• puzzle why there is resistance to policy change despite clear and convincing evidence;
• generally do not see policy engagement as part of their role;
• face strong organizational disincentives that dissuade them from engaging in outreach beyond the scientific community;
• face structural barriers to engaging in knowledge translation activities.
Practitioners and Policy-makers
• lament that researchers do not speak with one voice;
• bemoan the inability of researchers to make their findings accessible and digestible in time for policy decisions;
• lack confidence in their researchers;• regard research-based evidence as a
minor factor when policies for development are formulated and practices shaped.
Researchers, practitioners, and policy makers live in
parallel universes…
…with different values, languages, time-frames,
reward systems and professional ties
Two ‘impacts’
Researchers’ impact
• usually reflected by the impact factor of the journal in which the research report is published;
• impact factor is a measure of the average number of citations that have been made to recently published papers;
• frequently used as a proxy for the relative importance of a journal within its field.
Policy and practitioners’
impact
• for research to have any impact, the results must inform and shape policies and programmes, and be adopted into practice;
• use or consideration of research findings for actual outcomes of social change;
• emphasizes the need for economic and societal impacts.
The implications of two ‘impacts’
Academic ‘impact’
misrepresents what impact is
Research institutions discourage impact-
oriented research by prioritizing the
number and frequency of
publications in peer-reviewed
For policy research, academic peer
review and citation counting are
regarded as too limited for effective
evaluation
The activities that make up the work of knowledge transfer
pose risks to an academic career as they are not widely
accepted as legitimate forms of
scholarship.
Academic performance indicators have given rise to an identity representation of academics as performers;• Researchers are
assessed on the basis of their ‘hits’ instead of on the substance of their work.
• Performativity stifles innovation; promotes superficiality and conformity
THEMES IN THE LITERATURE
Intent
Researchers need to have the intent to
influence policy and/or practice
Intent to influence must be expressly
included among the research objectives
Communication
The most cited factor; repeatedly
stressed
Should start early in the research;
designed into the plan
Communication for development
(C4D) is a sub-field of international
development
Crucial for researchers to communicate effectively in a language that policymakers understand
Evidence requires ‘translating’ the
technical language to make it
comprehensible for policy-makers
Plain language communication
with the public is not widely
accepted as a legitimate form of
scholarship
Information and Communication Technologies
ICTs are blurring the line dividing academia and
professional and amateur writers; e.g.,
op-ed writers, bloggers
Participatory communication processes in development use
ICTs for rapid, multi-sited, multimedia and participant-
driven production and communication of research, as
it unfolds
The ‘echo-chamber’ effect of social media;
@odi_development’s 50 biggest followers have a combined reach of 4.3
million people
Many UK academics are reluctant to adopt web 2.0 tools for their work.
There is a lack of institutional incentives for using web 2.0 tools or for publishing online
UK researchers are discouraged from publishing online by the policy of having international
peer-reviewed journal citations, rather than online
citations, count towards academic promotion
Important to understand the online behaviour of the target
audiences
Open sharing and joint activity is at odds with the nature of the research process, where
the tradition is for solo teams of researchers to prepare their
findings privately
IntermediariesCultural differences,
systemic inadequacies in incentive structures
and the apparent weaknesses in
academic communication skills
and processes, imply a need for
intermediaries
Knowledge intermediary organisations synthesise,
interpret and communicate research, enable access and
make information edible, create demand for information, enable
marginalised voices, create alternative framings, connect
spheres of action, enable accountability, informing,
linking, matchmaking, facilitate collaboration and build sustainable institutions
Developing countries often lack
intermediary institutions that carry
research to policy - mechanisms of policy influence are missing
Policy Entrepreneurs
Critical skills ; being able to;
understand politics
identify key
players
synthesise research by simple
compelling stories
network effectively
build programmes
that bring all these factors
together
A policy entrepreneur is an individual who invests time and resources to
advance a position or policy
It is a role for researchers wishing to influence policy.
Networks
Although research may not have direct
influence on specific policies, the production
of research may still exert a powerful indirect influence
through introducing new terms and shaping
the policy discourse
Research findings and concepts
circulate and are gradually filtered through various policy networks
Various types of networks, such as
policy streams, policy
communities, epistemic
communities, think tank networks and advocacy coalitions
Internet networks facilitate the role
of policy entrepreneur
that is played by intermediary organisations
National, regional and global
networks are playing an
increasing role in development
policy
Incentives
Incentives to produce outputs
that reach a broader section of society are so low
that if engaged in at all, this occurs as an after-thought once
results are published
Incentives are needed for
researchers to engage effectively
with users of their research
Incentives are needed for
officials to engage more closely with
researchers
A determining factor for whether research-based and other forms of evidence are likely to be adopted by policy makers and practitioners
Research is more likely to contribute to policy if
the evidence fits within the political and institutional
limits and pressures of policy makers, and if it
resonates with their assumptions
Researchers must know and understand the key stakeholders in the policymaking process, and understand the way in which the door can be opened to politicians and public interest
Knowledge producers
need to be more self-aware of the political nature of their
engagement in policy processes
Political Context
Demand
Understand the demand for research among policy-makers and
practitioners, by, for example, mapping the existing information-
demand and information-use environment.
The effectiveness and impact of research will be driven by
continuous interactions between supply-drivers and
demand-drivers
Researchers are advised to participate in activities that
would stimulate demand for their outputs,
We know more about how to improve the supply of
evidence than we do about how to improve the
demand for it
Models of knowledge transfer
Science-push or
knowledge-driven
A unidirectional and logical flow of information
from researchers to policy makers.
Demand-pull or problem-
solving
Commissioning information from
researchers by policy makers with
the intent of addressing a well-
defined policy problem.
Interactive
A reciprocal and mutual activity
involving researchers and users in the development,
conduct, interpretation, and
application of research and
research-based knowledge.
EngagementNeed for closer relationships
between researchers and research users, especially
policy-makers. Public engagement processes draw on
a range of methods and approaches
Individuals are moving from simply accessing or consuming
the content and services offered by an online platform to becoming more involved in recommending or promoting it
and actively co-creating the content.
Not all researchers see policy engagement as part of their
role; they accord it a low priority. It is more suited to
those who see themselves as issue advocates
Fewer than 5% of academics regard engagement with the
media as an outlet for scientific findings as having any
consequence for measuring scientific performance at their
institutions.
IMPLICATIONS
Implications for ICT4D research
Identifying the particular contribution of ICTs to
specific development goals has proven to be extremely
difficult
Whilst the contribution in terms of technology diffusion and use –
especially of mobile phones – is easy to detect, the focus has only
recently shifted towards the question of development impact
The poor quality of ICT impact assessment to date derives from its
lack of conceptual foundations. ICT4D researchers need to produce high-quality research if they wish to
influence policy and practice.
Few researchers in ICT4D are drawn from the development studies
discipline, resulting in the use of an impoverished understanding of
development . There is little evidence of any impact of ICT4D research on
development policy or practice.
LESSONS
Lessons for research and academic institutions
Research & academic
institutions
Restructure institutional
incentives to take into account actual
impact.
Create incentives to invest in
dissemination and an expanded range of research
products.
Raise awareness and encourage social change
agents, knowledge brokers and linkage
mechanisms.
In hiring, balance consideration of
publication record with capabilities such as
originality, creativity, commitment, depth of
field experience and impact orientation.
Lessons for researchers
Interact with stakeholders to ensure relevance of research
questions and outputs.
Identify uptake pathways as part of
project design.
Design projects to meet end users’ needs and
aspirations.
Share and publish experiences of how research
results have been ‘translated’ or used for a non-scientific audience.
Lessons for journal editors and publishing organizations
Challenge researchers to propose ways to evaluate
the real impact of their work.
Provide incentives to researchers to publish practitioner-oriented
results of relevance to civil society.
Break the language barrier by publishing ‘mirror’
papers; translations of the complete paper into the language of where the
research was undertaken.
Recognize that sustainable change is a long-term process. Support longer term project time frames (4–10 years) in which sufficient dialogue occurs at the initiation of projects.
Expand proposal requirements to
include the sharing of relevant research
results in an accessible format to appropriate
audiences.
Ensure proposals designate sufficient funds for translation, printing, mailing costs and communication.
Acknowledge that originality often occurs at the fringes. Identify and support small
but innovative, locally driven initiatives.
Lessons for Donors
Examples
Research to Action
Caters for the strategic and
practical needs of people trying to improve the
uptake of development research; in
particular those funded by DFID
Science into Policy
Publication of the UK’s National Environment
Research Council, which helps scientists to;
recognise the relevance of science to policy-makers;
identify available opportunities, routes and best practice to influence
policy-making; and communicate science in
an appropriate and accessible way, to the right policy-makers,
showing how it fits their policy needs.
ResearchImpact
Connects university research with research users
across Canada to ensure that research helps to
inform decision-making. Knowledge Mobilization
Units work to match researchers with key policy-
makers in government, health, and social service agencies to ensure that
academic research is employed by policy-makers and community groups to develop more effective, efficient, and responsive public policies and social
programs.
Conclusions
ICT4D research has been assessed overall as lacking
Little if any evidence of any impact on practice
and policy-making
Need major cultural and
institutional shifts
Some of the changes that are necessary can be initiated
relatively easily by promoting the transitions that ICT4D researchers
will have to make in order to increase the relevance of their
work to wider audiences
Framework for the impact of research on policy and practice
Policy and
practice
IntentComm-
unication
ICTs
Interm-ediaries
Policy Entrep-reneursNetworks
Incentives
Political Context
Demand
Engage-ment
References• Adolph, B., Jones, S.H., and Proctor, F., Learning Lessons on Research Communication and Uptake, Triple Line Consulting Ltd for DFID, 2010. • Brown, C., Are Southern Academics Virtually connected? GDNet, August 2012. http://depot.gdnet.org/cms/files//GDNet_study_of_adoption_of_web_2_tools_v2.pdf accessed 28 March 2013• Carden, F., Knowledge To Policy: Making The Most Of Development Research, SAGE Publications Inc/IDRC, 2009. • Court, J., and Maxwell, S., Policy Entrepreneurship for Poverty Reduction: Bridging Research and Policy in International Development, Journal of International Development, vol. 17, pp 713–725, 2005. • Court, J. and Young, J., Bridging Research and Policy in International Development: An Analytical and Practical Framework, Development in Practice, Volume 16, Number 1, February 2006. • Datta, A., Deliberation, Dialogue and Debate: Why Researchers need to Engage with others to Address Complex Issues, IDS Bulletin, Volume 43, Number 5, September 2012. • de Vibe, M., Hovland I., and Young, J., Bridging Research and Policy: An Annotated Bibliography, ODI Working Paper 174, Sep 2002. • DFID, Social Media Engagement; A Report of the Activities on the R4D Project, • http://r4d.dfid.gov.uk/pdf/outputs/Communication/R4D%20Social%20Media%20Engagement%20Report_HR.pdf, accessed 28 March 2013.• Fisher E. and Holland, J.D., Social Development as Knowledge Building: Research as a Sphere of Policy Influence, Journal of International Development, vol. 15, pp 911–924, 2003. • Gendron, Y., Constituting the Academic Performer: The Spectre of Superficiality and Stagnation in Academia, European Accounting Review, Vol.17 No.2, pp 97-127, 2008. • Gregory, N.F., Should Aid Donors Support Economic and Social Research? Journal of International Development, Vol. 4, No. 2, pp 233-242, 1992. • Greijn, H., Linking Research-Based Evidence to Policy and Practice, Research, Policy and Practice. Capacity, Issue 35, December 2008. • Harvey, B., Lewin, T. and Fisher, C., Is Development Research Communication Coming of Age? IDS Bulletin, Volume 43, Number 5, September 2012. • Heeks, R., Do Information and Communication Technologies (ICTs) Contribute to Development? Journal of International Development, vol. 22, pp 625–640, 2010. • Hovland, I., Communication of Research for Poverty Reduction: A Literature Review. Overseas Development Institute, Working Paper 227, October 2003. • Hovland, I., Making a difference: M&E of policy research, Overseas Development Institute, July 2007. • Jacobson, N., Butterill, D. and Goering, P, Organizational Factors that Influence University-Based Researchers’ Engagement in Knowledge Transfer Activities, Science Communication, Vol. 25 No. 3, pp 246-259, March 2004.
Jones, H., Jones, N., Shaxson, L. and Walker, D., Knowledge, Policy and Power in International Development: A Practical Framework for Improving Policy, ODI Background Note, January 2013. • Kleine, D., ICT4WHAT? Using the Choice Framework to Operationalise the Capability Approach to Development, Journal of International Development, Volume 22, Issue 5, pp 674–692, 2010. • Langou, G.D., Developing Capacities for Policy Influence, Research, Policy and Practice, Capacity, Issue 35, December 2008. • Lewin, T. and Patterson, Z., Approaches to Development Research Communication. IDS Bulletin, Volume 43, Number 5, September 2012. • Masset, E., Mulmi, R. and Sumner, A., Does Research Reduce Poverty? Assessing the Welfare Impacts of Policy-Oriented Research in Agriculture, Institute of Development Studies, the University of Sussex Brighton, UK, March
2011. • Newman, K., Fisher, C. and Shaxson, L., Stimulating Demand for Research Evidence: What Role for Capacity-building? IDS Bulletin, Volume 43, Number 5, September 2012. • O’Neil, M., What Determines the Influence that Research has on Policy-Making? Journal of International Development, vol. 17, pp 761–764, 2005. • OECD, Opportunities, Challenges and Good Practices in International Research Cooperation between Developed and Developing Countries, April 2011. • Ryan, J.G and Garrett, J.L., The Impact of Economic Policy Research: Lessons on Attribution and Evaluation for IFPRI, International Food Policy Research Institute, Oct 2003. • Saxena, N.C., Bridging Research and Policy in India, Journal of International Development, vol. 17, pp 737–746, 2005. • Sen, K., Rates of Return to Research: A Literature Review and Critique, DFID, 2005. • Shanley, P. and López, C., Out of the Loop: Why Research Rarely Reaches Policy Makers and the Public and What can be Done, BIOTROPICA 41(5), pp 535–544, 2009. • Shaxson, L., Improving the Impact of Development Research through Better Research Communications and Uptake, Report of the AusAID, DFID and UKCDS Funded Workshop: London, November 29th and 30th 2010, • Stone, D. RAPID Knowledge: Bridging Research and Policy at the Overseas Development Institute, Public Administration and Development, v29, pp 303–315, 2009. • Sumner, A., Ishmael-Perkins, N. and Lindstrom, J., Making Science of Influencing: Assessing the Impact of Development Research, IDS Working Paper 335, Sep 2009. • Taylor, M., Bridging Research and Policy: A UK Perspective, Journal of International Development, vol. 17, pp 747–757, 2005. • Weiss, C. Research for Policy’s Sake: The Enlightenment Function of Social Research, Research, Policy Analysis, 3:4 p.531, Fall, 1977. • Wheeler, J., Creating Spaces for Engagement: Understanding Research and Social Change, Development Research Centre on Citizenship, Participation and Accountability, 2007. • Williams, G., The Disciplining Effects of Impact Evaluation Practices: Negotiating the Pressures of Impact Within an ESRC–DFID Project, Transactions of the Institute of British Geographers, No. 37 pp 489–495, 2012. • Young, J., Research, Policy and Practice: Why Developing Countries are Different, Journal of International Development, vol. 17 pp 727–734, 2005. • Young, J., Impact of Research on Policy and Practice, Research, Policy and Practice, Capacity, Issue 35, December 2008.