Post on 12-Aug-2019
transcript
2011
PARP‟s evaluation activity
on the innovation area
Warsaw, 5th March 2014
Jacek Pokorski
Evaluation Unit, Enterprise and Innovation Department
2014
PARP carries out evaluations of programmes under implementation from the beginning of its activity (2000).
Evaluations cover:
Programmes financed from domestic funds:
“Innovation vouchers” programme, subsidies for information and counselling services, acquisition of certificates, financial services (loans and guarantees), other domestic programmes.
European programmes in the 2004-2006 perspective
SOP ICE – direct support to enterprises and institutions providing services for companies stimulating their competitiveness
SOP HRD – support for training of employees of enterprises in the development of their competences and consequently the competitiveness of enterprises
European programmes in the 2007-2013 perspective
OP IE – various development-oriented support instruments for enterprises and business environment institutions
HC OP – various support instruments for development of their employees’ competences and consequently the competitiveness of enterprises
OP DEP – support for socio-economic cohesion of 5 poorest regions of Poland
Evaluation in PARP
National Evaluation Unit
European Commission
Managing Authorities
1st level Intermediate Bodies
2007-2013
OP IE
Priorities
Measures
2nd level Intermediate Bodies
HC OP
OP DEP OP
OP
ROP x 16
PARP in the European Funds’ evaluation system
In the evaluation processes, PARP cooperates
with competent Intermediate Bodies and
Managing Authorities of Programmes
and socio-economic partners under
Evaluation Steering Groups
Schemes / Sub-measures
Projects
EVALUATION PROCESS IN PARP
Contracting out to
Research institutes
Free-lancing evaluators
Consulting companies
Direct subsidies for enterprises,
R&D, HR development,
Infrastructure investments…
Partners
Universities
Public administration
Stakeholders
Business organizations
The way we do evaluation INTERNAL
EXTERNAL
Main subjects of
the research
Evaluation criteria
relevance, effectiveness,
efficiency, utility, sustainability,
coherence
Evaluation types and methodology
Depending on programme logic and information needs, PARP carries out:
ex-ante evaluation (prior to mobilisation of support)
on-going evaluation (during the programme - focus on direct outcomes and programme management)
ex-post evaluation (after conclusion of support provision – focus on outcomes and impacts)
ad-hoc evaluation (at various programme levels; STTS expert evaluations focused on specific problems and areas), including:
evaluations of a programme / priority or scheme / measure / project…
horizontal / thematic evaluations
We use various study and analysis techniques – quantitative and qualitative – each time adapting their capabilities to project's needs. We use innovative counterfactual impact evaluation techniques (among others Propensity Score Matching)
Evaluation process participants
PARP’s evaluations are participated primarily by addressees and recipients
of provided support:
entrepreneurs (natural persons establishing their business, SMEs,
large companies, strategic state-owned companies)
business suport organisations – BSOs (financing, counselling,
information institutions, scientific institutions and research and
development units, science and technology parks and other)
cooperative relations of enterprises, science and research units and
non-governmental organisation (clusters)
public institutions – self-government authorities, non-governmental
organisations, commerce chambers and various socio-economic
partners
natural persons – employees of enterprises (using the support in the
development of competences), persons using the socio-economic
infrastructure
media
Current PARP‟s evaluations
of pro-innovative programmes (i.a.)
“The Innovation Barometer” – on going evaluation of 9 Measures
of The Innovative Economy OP, 2007-2013, addressed to Polish enterprises.
a systematic approach to tracking the situation of beneficiaries after
receiving the support - from programme‟s theory, outcomes indicators,
tools for research to multi-evaluation scheme implementation
(project duration: 2011-15)
evidence- based impact evaluations of certain Measures IE OP
(net effect measurement)
Evaluation on demand for support in new perspective 2014-2020
in the following areas: enterprise research & development activity
electronic business
enterprise internationalisation
enterprise design activity
„THE INNOVATION BAROMETER
PROJECT‟
- a systematic approach to evaluation
the pro-innovative public programmes
in Poland
The concept of on-going evaluation
Determination of effects of taken measures
Collection of data on a regular basis, systematically (on-going)
Measurement of indicators established for the needs of the Programme
and assessment of their values
quantification of changes resulting from the conducted intervention
access to knowledge on the extent to which a given programme turns out
to be effective (i.e. implements the set objectives)
Key to take on-going decisions
in the implementation process!
MEASURING DIRECT OUTCOMES
ON-GOING EVALUATION SYSTEM FOR OP IE
„THE INNOVATION BAROMETER PROJECT‟ (2011-2015):
multidimensional and cyclic evaluation study that enables to measure and asses
on-going outcomes of the Programme (IE OP) in the field of innovativeness,
research tools and indicators developed by PARP and group of external
experts (2008-2010),
measurement of direct results presented as indicators
(i.e. changes related to beneficiaries as a result of programme intervention)
assessment of the instrument’s impact on the area it was addressed for
model of the on-going evaluation in OP IE based on the 2004-2006 period
experience (SOP ICE).
„INNOVATIVENESS BAROMETER”
- THE ON-GOING EVALUATION
OF SELECTED OP IE MEASURES
Evaluation framework:
PARP Internet Evaluation System based on 8 CAWI questionnaires
for enterprises
Two analyses of enterprises: during the projects’ implementation
and 24 months after their settlement.
Indicators are coherent with the Central Statistical Office during the research on
economic condition and innovativeness of enterprises
Metodology enable to calibration of the projects’ quality
and support efficiency measurement
4.4 New investments of
high innovative potential
1.4-4.1 Support for
goal-oriented projects and
Support for imple-
mentation of results of
R&D works
4.2 Stimulation of R&D activity
of enterprises and support
within the scope of industrial
design
6.1 Passport to export
3.3.2 Development of
system facilitating
investing in SME’s
8.1 Support for economic
activity as regards
electronic economy
8.2 Support for imple-
mentation of eletronic
business – B2B
5.4.1 Intellectual
property management
CAWI ‘s
questionnaire CAWI ‘s
questionnaire
CAWI ‘s
questionnaire CAWI ‘s
questionnaire
CAWI ‘s
questionnaire
CAWI ‘s
questionnaire
CAWI ‘s
questionnaire
CAWI ‘s
questionnaire
PARP’s Internet Evaluation System
for BAROMETR
• Safety and simplicity to
manage the data
• The possibility of the
current control and easy
management of the
evaluation process
Starting research,
monitoring response rate
Implemen-
tation Help Desk for
responders
Data
analysis
Comparability of the data
to the public satistics
Raports and
presentations
of the results
Meeting the information’s needs
of PARP, Ministries and public
opinion.
Design of the research tools
Scope of the CAWI„s questionnaires
(3) Block of the
specific questions for
each Measure
• Profile of beneficiaries and projects,
business models
• Measurement of indicators established
(2) Block of the core
questions (common to
all Measures)
• Sources of information on co-financing
• Motives the use of public support
• Models of work on a formal application
and manage projects,
• The cost of rental companies specialized
in obtaining grants
• The problems during projects'
implementation
Co
mp
arab
ilit
y b
etw
een
the
Mea
sure
s
(1) Indicators form
• Data collected according to the Central
Statistical Office (GUS)/ Eurostat
methodology
• Data on the economic situation
of enterprises (income, employment,
investment in innovation activities, etc.)
• Innovativeness of enterprises
Co
mp
arab
ilit
y i
n
the
pu
bli
c st
atis
tic
The logic of measurement - the project‟s level
Initial measurement P≤ A + 1/3*[C-B] Final measurement K= D + 24months
Date of signing the
financing agreement
Estimated date
for start of the project
A C B
Date of settlement of the
project
D
Estimated date for
completion of the project P Period of projects'
sustainability
K
The introduction of two perspective of measurement allows to:
1) examine two important moments in the implementation of projects:
- the beginning of the project (with the possibility to ask about the period of grant
application preparation) and
- the first two years of sustainability of the project
2) Collect data for all years between initial and final measurement and test current economic
trends observed among beneficiaries OPIE
R&D activity
R&D activity carried out by the beneficiaries Change (%) between the period before
and after participation in the program
r r+2 r+1
Product innovations
n=799
43%
45% 44%
49%
60% 63%
0%
10%
20%
30%
40%
50%
60%
70%
before grant from application to 24 months after settlement of the project
products service product innovation (total)
11%
4%
5%
New or substancially developed products
Process innovations
44%
53%
24%
38% 35%
46%
0%
10%
20%
30%
40%
50%
60%
before grant from application to 24 months after settlement of the project
production logistic supporting processes in the company
New or substantially developed methods n=799
60%
32%
19%
r r+2 r+1
Financial indicators (total sales)
z-2 z-1 z z+1 z+2
z-2 z-3 z-1 z+1 z z+2
Employment indicators
n=851
n=1094
Using the results of the OP IE
on-going evaluation
On-going management of the implementation process based on
information on quantified support results
Comparison of tendencies in terms of effectiveness of the support
(in relation to pojects selected in the following tenders, in various economic
contexts, etc.)
Reporting, information, and promotional functions
The results will provide knowledge on the new perspective (effectiveness
and efficiency of the proinnovative instruments for enterprises and BSOs
support)
The results will be used in the ex-post evaluation of the Programme
Rigorous impact evaluation
- to improve measurement and conclusions
2014: PARP is going to launch 'counterfactual impact evaluation’
approach in Innovation Barometer Project.
We do not know to what extent the Programme itself changed
the economic condition of the beneficiaries (programme’s effects).
We can not rely on information about the Programme’s effects declared
by the beneficiaries, that is why we plan more rigorous analyses
relating reliable counterfactual situation.
publications on the evaluation
methodology and results:
www.parp.gov.pl/index/more/14819
www.parp.gov.pl/index/more/22858
www.parp.gov.pl/index/more/24238
publications, reports
and presentations of the evaluation
results available at: www.parp.gov.pl
More information on evaluation in PARP
periodic evaluation conferences:
www.konferencja-ewaluacja.pl
Thank you for your attention
Polish Agency for Enterprise Development
81/83 Pańska Street
00-834 Warsaw, Poland
www.parp.gov.pl
E-mail: jacek_pokorski@parp.gov.pl