1

1

Assessing the impact of combined

data collection strategies and of

response rate on the measurement of

innovation activity in Norway

Lars Wilhelmsen, Statistics Norway

Nordisk Statistikermøte, Bergen 14-17. August 2013

2

Background for the project

• Norway scores highly on several core economic indicators:

– GDP per capita

– Trade balance

– Unemployment

• But below the EU average on certain R&D and innovation comparisons

based on the CIS

3

Internationally: Enterprises with product and/or

process innovation activity, CIS 2010

0

10

20

30

40

50

60

Latv

ia

Ungarn

Rom

ania

Pole

n

Slo

vaki

a

Bulg

aria

Lita

uen

Malta

Spania

Norg

e

EU

27

Nederland

Kro

atia

Fra

nkr

ike

Slo

venia

Tsje

kkia

Italia

Danm

ark

Irla

nd

Øste

rrik

e

Sverige

Kypro

s

Fin

land

Luxem

burg

Belg

ia

Estla

nd

Port

ugal

Tyskl

and

Prosent avalle foretak

0

10

20

30

40

50

60

70

Perc

en

t o

f p

op

ula

tio

n

4

Background for the project

• Norway scores highly on several core economic indicators:

– GDP per capita

– Trade balance

– Unemployment

• But below the EU average on certain R&D and innovation comparisons

based on the CIS

• And our share of innovators have been declining

5

Norway: Innovation surveys, 2004-2010

0

5

10

15

20

25

30

35

2004 2005 2006 2007 2008 2009 2010

Perc

en

t o

f en

terp

rices i

n t

he p

op

ula

tio

n

Product and/or process (PP-) innovators

PP-innovators with in-house R&D

6

Background for the project

• Norway scores highly on several core economic indicators:

– GDP per capita & GDP per capita growth

– Trade balance

– Unemployment

• But below the EU average on certain R&D and innovation comparisons

based on the CIS

• And our share of innovators have been declining

• This may be seen as a paradox if we expect a positive relationship

between innovation scores and economic success

• It has been questioned by policy actors, businesses and media whether

the Norwegian scores on common innovation indicators are too low and

does not reflect reality

About the Norwegian Innovation survey

• Modeled closely on the standardized Community Innovation

Survey (CIS) questionnaire developed by Eurostat

• The CIS and the Business Enterprise R&D surveys have

been combined in Norway since 2001

– First part: Questions on R&D

– Second part: Questions on innovation

• Most countries have separate surveys

– (incl. Sweden, Finland and Iceland)

– while some have combined surveys (incl. Denmark)

• Mandatory survey with enforcement

– High response rate (>95%)

Combined vs. separate surveys in general

• Timeliness of results

• NSO survey cost

• Response burden for the enterprises

• Data linkages and data coherency

• Weighing different user needs

– International benchmarking

– Researchers

– Policymakers

– Policy implementing agencies

– National accounts

9

Methodological exercise: A separate CIS

• We wanted to explore two separate (but related) issues

– Combined R&D and innovation survey vs. a separate CIS Does the presence of detailed R&D questions impact the reporting of innovation?

– The importance of response rate; a mandatory vs. a voluntary survey Overall, response rates are on the rise (most countries in the 75-85% range)

But there are countries where they are much lower (down to ~25%)

Other countries have seen a drop in innovation rates after making their survey

mandatory

There is some doubt about the comparability of data with and without non-

response adjustment

• Thus, we sent an innovation-only survey to two distinct extra samples

– One with and one without a mandatory response requirement (and enforcement)

– In addition to the regular survey, so three samples from the same population

• Final response rates: – 96% (combined, mandatory), 95% (separate, mandatory), 41% (separate, voluntary)

10

Results: Product and/or process innovation,

combined vs. separate survey

0 %

5 %

10 %

15 %

20 %

25 %

30 %

35 %

40 %

Combined Survey Mandatory Innovation Survey

Sample

Pe

rce

nt

of

po

pu

lati

on

11

Possible explanations for the findings

• Different type of respondents? – We know surveys that ask for lots of ”hard numbers” are more likely to be

answered by accounting or similar.

– We know some respondents see the CIS portion as an appendix to the R&D survey when they are combined

• Lower threshold for reporting an activity as innovative? – Asking for a too detailed breakdown of certain R&D activities may

suppress reporting of other R&D activity that does not fit the available categories Interestingly, a significantly higher share of enterprises report having

performed R&D in the separate innovation survey.

– A similar mechanism may be also in play for non-technological innovation when removing the R&D part of the survey

– R&D-focused respondents may answer innovation questions based on R&D outcomes in the enterprise rather than from an overall perspective.

• Respondents finding the survey more relevant and applicable to them, and thus answering more diligently

12

Product and/or process innovation,

mandatory vs. voluntary survey

0 %

5 %

10 %

15 %

20 %

25 %

30 %

35 %

40 %

45 %

50 %

Mandatory Innovation Survey Voluntary Innovation Survey

Sample

Perc

en

t o

f p

op

ula

tio

n

13

Possible explanations for the findings

• Readily apparent self-selection in the voluntary sample

• We did not have the possibility of doing a non-response

survey

• Internationally (some) countries with a very low response

rate and a high share of innovators, further increase this

after performing non response adjustments

– Hypothesis: Non response adjustment on our voluntary sample

would yield higher results than our mandatory sample Simplification, different respondents and the possibility of interacting

with the interviewer may further strengthen the trends observed when

removing the detailed R&D questions

In which case we would get deceptively high innovation rates with

deceptively low sampling errors

Follow-up interviews with respondents

• Organized as semi-structured qualitative interviews

– Thoughts on response burden Separate surveys perceived as a higher response burden

– But only if receiving both

Most government mandated reporting seen as an unnecessary chore – R&D perceived as more difficult and less “fun” to answer

– Some respondents eager to report on innovation

On reflection, little opposition to separating the surveys

– Factors influencing the results The concept of R&D is confusing

– Seen as a somewhat alien “white lab coat” activity

– Also limits the understanding of what constitutes a reportable innovation

Chance that different people would respond to the surveys if separated – Several respondents thinks this will lead to better reporting

The shared questionnaire is seen as being too long and convoluted – “Invites” cases of “substantial none-response”, just to get it over with

– Questions on innovation, being located after R&D, may suffer the most

14

15

Lessons learned

• Context matters when attempting to measure “hard to quantify social phenomena” – A combined survey seems to work well for industries where a traditional

linear innovation model with R&D-driven innovations is applicable.

– A separate survey may give more correct results for industries, in particular in services, with low R&D intensity, or where innovations are developed in an incremental/iterative process

• Respondents do NOT have a clear and accurate understanding about what constitutes an innovation

• Norwegian innovation rates are probably somewhat underreported compared to many other CIS-implementing countries – But we don’t know exactly by how much

– And we don’t know how these findings would impact a comparison against any particular country

• Does not say anything about the observed declining trend in the Norwegian innovation rates

• It is still unclear which results are “better” or most useful

16

So, now what?

• Doing a full scale separate innovation survey for 2013

– Gauging the “correct” level of innovation compared to the regular

combined CIS 2012

• Will revisit the shared vs. combined debate with core

users once these results are available

• There are both benefits and drawbacks with combined

surveys

– Impossible to fulfill all users’ needs all the time

• Some compromises will have to be made either way

– Limited room to increase total cost

– Unwillingness to increase overall response burden

Thank you for your attention!

17