HAL Id: halshs-01133963https://halshs.archives-ouvertes.fr/halshs-01133963

Submitted on 24 Mar 2015

HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.

A multi-criteria and multi-actor perspective for theevaluation of sustainability services

Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

To cite this version:Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen. A multi-criteria and multi-actor perspective for theevaluation of sustainability services. 5th International Conference on Applied Human Factors andErgonomics AHFE 2014, Jul 2014, Krakow, Poland. �halshs-01133963�

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

1

A multi-criteria and multi-actor perspective

for the evaluation of sustainability services

Kirsi Hyytinen1, Faïz Gallouj2 and Marja Toivonen1

1 Technical research Centre of Finland, PL 1000, 02044 VTT, Finland

2 Faculté des Sciences Économiques et Sociales, Université de Lille 1, 59655 Villeneuve d’Ascq. France

ABSTRACT

The purpose of this paper is to study the challenge of the evaluation in the context of the services. Because of their

specific nature the traditional evaluation methods and measures are not able to capture neither the diversity of the

innovations nor the multifaceted dimensions of performance. This paper aims to contribute to the need for a more

diverse evaluation approach. We focus on the context of service innovation in the environmental sector, in which

we develop further multi-criteria and multi-actor perspectives. The multi-criteria framework describes the impacts of

sustainability services and the multi-actor framework aims at analyzing the impacts from the perspective of various

actors involved. Thus, our study provides a two dimensional approach to assess the impacts of services. The focus is

on understanding the dynamics of service creation in the environmental sector, and on using new evaluation

methods and indicators in that sector.

Keywords: Sustainable services, service innovation, multi-criteria evaluation, multi-agent framework

INTRODUCTION

Systematic evaluation is an activity that has been an essential part of strategic thinking for decades. It has been

linked both to the general development of organizations and to more specific functions. It has played a particularly

important role in the context of innovation and R&D. In public interventions – often carried out in the form of policy

programs – evaluation has been an embedded practice whose purpose has been to support decision making (Rip,

2003). Since the early days of evaluation, impact assessment has been a typical way to implement evaluation, which

has meant that evaluation has been understood in terms of performance-related steering and monitoring (Chelimsky,

1997; Rip, 2003). The primary idea has been to produce indicator based information to prove accountability and to

legitimate the role and existence of individual organizations and policy instruments.

This kind of an approach to evaluation includes several problems. First, it is backward looking and does not work

well as a guiding instrument in the current society that is characterized by rapid changes (Todd and Wolpin, 2010;

Weijermars and Wesemann, 2013). Second, the approach is “atomistic”: it focuses on individual organizations or

policies and overlooks the fact that impacts are co-produced by several actors and emerge in a broader context (Rip,

2003). Third, the indicator based assessment simplifies phenomena that emerge in cyclic, complex and long-term

processes. Indicators assume a simple causal relationship between intervention and impacts, which is incompatible

with the modern view about the emergence of innovations (Hansson, 2006; Van der Knaap, 2006; Cozzens and

Melkers, 1997). Summarizing, the linear input-output-outcome -thinking included in the traditional evaluation does

not correspond to the complex development processes and the multiple relationships between the contributing actors

(Arnold, 2004; Patton, 2011).

As an answer to these problems, researchers have suggested a more versatile view: “plurality of methods”

(Dyehouse et al., 2009; Williams and Imam, 2007). Systems thinking have often been applied hand in hand with this

view. It means highlighting interrelationships and feedbacks between the whole, its parts and the environment they

are interacting with (Cabrera et al., 2008).

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

2

Recently, the increasing “servitization” of society has put pressure to develop a more advanced approach to

evaluation. Here, an additional problem linked to the traditional impact assessment has become apparent: the

indicators have been one-sidedly technologically focused (Toivonen, 2010; Rubalcaba et al., 2013). These kinds of

indicators do not capture the immaterial and interactive aspects that are central characteristics of services. Today

services are also increasingly forming systemic wholes – especially the most urgent problems in the present society

cannot be solved via the development of individual services, but service systems. The issues of sustainability are an

illustrative example of this development. The current evaluation methods are unable to tackle the challenge of the

systemic nature of innovation and service creation (Smith, 2000; Edqvist, 2005; Ahrweiler, 2010).

This paper aims to contribute to the above described need for a more diverse evaluation approach. We focus on the

context of service innovation, in which we develop further a multi-criteria and multi-actor perspective for

evaluation. Our starting points are the multi-criteria framework developed by Djellal and Gallouj (2010), and the

multi-actor framework developed by Windrum and Garcia-Goñi (2008). The former strives for better visibility of

the multi-faceted dimensions of service innovation and highlights the importance of quality and social value. The

impacts of innovation are analyzed using the concept of performance which is also interpreted broadly. The purpose

is to diminish the “double innovation and performance gap” that derives from a narrow view on these concepts and

has resulted in a difference between the reality and the ability of the traditional measures to perceive it (Djellal and

Gallouj, 2010, 2013). The latter framework – the multi-actor analysis – is built on the characteristics-based view on

service innovation (Gallouj and Weinstein, 1997). It focuses on understanding the dynamics and impacts of service

innovation from the perspective of the different actor groups participating in the service development.

In the conceptual part of our paper we open the “innovation and performance gap” in the service context and review

the opportunities for new kinds of evaluation approaches. Then we discuss and concretize the multi-criteria and

multi-actor frameworks by applying them in services that are emerging to support the strivings for sustainability.

More specifically, we analyze service innovations and their assessment in the context of sustainable energy

production. The multi-criteria framework that we develop further utilizes the categorization of different “worlds of

services” – equating to different sets of justificatory criteria (Gadrey, 2005; Djellal and Gallouj, 2012). Empirical

data has been collected in Finland in the context of public-private development programs in the area of

environmental sustainability. The following research questions are guiding our work:

- How and by what means should the outcomes and impacts of environmental services be evaluated in order

to take into account the complex and integrative nature that characterizes innovation in these services?

- What are the impacts of environmental service innovations from the perspectives of different societal fields

and of different actors?

The results of our study provide a two dimensional approach to assess the impacts of services by combining the

multi-criteria and multi-actor frameworks. The focus is on understanding the dynamics of service creation in the

environmental sector and the use of evaluation methods and indicators in that sector. At a more detailed level, our

study provides analytical material about the complementarities and contradictions between different actors and

indicators in service development. Because the links between services and environmental issues have been scarcely

studied, our study also contributes to narrowing the gap between these two fields – a point whose importance has

been emphasized recently (Rubalcaba et al., 2012; Djellal and Gallouj, 2010; Gadrey, 2010).

The paper is divided into six sections. The second section after this introduction is based on literature and discusses

the challenges of measurement and evaluation, particularly in the context of services. The third section presents the

two main frameworks that we apply in our study to evaluate the outputs and impacts of services: the multi-criteria

and multi-agent frameworks. In the fourth section, our case-study context and methodology are described. The fifth

section analyzes the results of the application of the frameworks in the assessment of environmental services.

Concluding remarks ends up the paper.

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

3

MEASUREMENT CHALLENGE IN SERVICE INNOVATION AND

PERFORMANCE

The importance of invisible innovation and performance

For two decades, service studies and specifically the studies on service innovation have argued that the traditional

tools, indicators and measures do not capture the performance, innovativeness and impacts in services (Sundbo,

1998; Metcalfe & Miles 2000). A central background reason for the existence of the gap is the “assimilation”

perspective adopted in the early service research (Coombs and Miles 2000; Gallouj, 1994). This perspective

analyzed services innovation as an imitation of technological and manufacturing innovations, i.e. it did not

recognize independent innovation possibilities in services. The perspective was based on the traditional definition of

innovation as an invention which results from an R&D project (Howells, 2004). The linear, stage-gate model of an

innovation process, which was raised to the position of a norm and marketed as a prerequisite for success (e.g.

Cooper and de Brentani, 1991), increased the bias.

These arguments and perspectives have been developed in the context of service innovations. However, they are

strongly rooted in general innovation theories basing on the Schumpeter‟s definition of innovation that has

afterwards been regenerated to “the neo-Schumpeterian” theory of innovation or “the broad view of innovation”

(Lundvall, 2007). These traditions are interlinked by several common aspects that affect both the definition of

innovations and evaluation of their effects and impacts. Important cornerstones are complexity and uncertainty of

innovation process, intangible nature of innovation (focus in new solutions and processes) and systemic view of

innovation encompassing multiple sources and actors taking part into the innovation process. (Kline and Rosenberg,

1986; Dosi,1988; Lundvall, 1992; Freeman, 1991; Nelson and Rosenberg 1993). Also from the perspective of

general innovation literature conclusion has been parallel: current practices in defining innovation and evaluating

them do still follow the mainstream linear innovation thinking which simplifies too much the innovation process as

well as the complex dynamics between actors contributing innovation (e.g. Arnold 2004, Patton 2011). Thus, it does

not take into account the systemic view of innovation (e.g. Smith, 2000; Edqvist, 2005; Ahrweiler, 2010) and does

not capture the multifaceted and diversified performance of innovations (e.g. dealing with societal issues such as

health, education, sustainable development) (see e.g. Freeman & Soete, 2009).

A broader view on innovation has increasingly gained ground, but the assimilationist origin of the perspective on

innovation and services is still influential. In particular, the measurement and indicators of innovation and

performance are still largely based on the industrialist and technologist definitions (e.g. Smith, 2000; Edqvist, 2005;

Smits and Kuhlmann, 2004; Djellal & Gallouj, 2010, 2013). In addition to the narrow view on innovation, the

dominating view on performance is also mechanical and narrow. It is usually linked to the concept of productivity

which refers to the linear and mechanistic input-output function (e.g. Djellal and Gallouj, 2010, 2013; Kellog

foundation, 2004; Patton 2011). Its‟ traditional definition is unable to recognize the “hidden performance”

concerning the societal aspects of services and innovations like equality, ecological sustainability and societal well-

being. It also often excludes the aspect of social innovation. (e.g. Rubalcaba et al., 2012)

Djellal and Gallouj (2010) have described the interaction between performance and innovation by referring to the

visible vs. invisible nature of these phenomena. Whereas technology-based innovations are visible, non-

technological innovations are invisible. As regards performance, the authors link the visible-invisible dichotomy to

short-term vs. long-term influences. Both in scientific and the managerial discussions, short-term influences of

performance are often analyzed in terms of productivity and growth. Longer-term influences are increasingly

analyzed in terms of environmental or social sustainability. There are four possibilities in the relationship between

innovation and performance as Figure 1 illustrates (ibid., 668). The most apparent relation is between visible

innovation and visible performance, but visible innovation may also lead to invisible performance by promoting the

long term ecological sustainability or societal well-being. Correspondingly, invisible innovation may be a source of

visible performance, i.e. growth and productivity, or promote sustainability.

Figure 1. Innovation and performance gap in measurement of services (source Djellal and Gallouj 2010)

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

4

Opening the specific characteristics of service innovation and performance

After the realization of the narrowness of the assimilation view, two alternatives have gained ground. The first is the

differentiation perspective – also known as the demarcation perspective (Coombs and Miles, 2000; Gallouj 1994,

1998). It focuses on the specific characteristics of service innovation and has highlighted the difficulty of

recognizing “newness” and its creator in the service context (Preissl, 2000). The second alternative is an integrative

or synthesis perspective (Cooms and Miles, 2000; Gallouj 2002) which has become increasingly relevant due to the

blurring lines between goods and services. It highlights the production and consumption that focus on integrated

solutions and systems.

Recently, these approaches have also been applied when searching better indicators for innovation and performance

in services. The peculiar characteristics of services that specifically have been pointed out in this context are

intangibility and the central role of interaction; the latter refers to the central role of co-production between the

provider, customers and partners. An important implication of intangibility is the difficulty of defining the “unit of

output” and differentiating the product from the process. These aspects challenge the definition of innovation and

quality improvements in it (ibid.). Interactivity increases the complexity of the development of services innovations

(Gallouj et al. 2013). Ignoring these complex and dynamic relationships (cf. Arnold 2004) is often linked to the

traditional technologic measures and the linear innovation model (Smith, 2000; Edqvist, 2005; Ahrweiler, 2010). It

may lead to the oversimplification of the reality and to the biased understanding – not only of the impacts of services

and innovations – but also of their drivers and dynamics (Arnold, 2004).

New aspects in the analysis are inclusion of the time factor and the social nature of services (Djellal and Gallouj,

2013). Time highlights the dynamic nature of services, focusing to their evolution over time. Considering time as a

dynamic factor leads to the differentiation of short-term outputs from medium and long-term effects. The social

nature of services derives from the fact that the value and benefit of services is always defined by users. This means

that different actors have different values, and it is just this multiplicity of values which makes it necessary to include

various criteria in evaluation. In addition to immediate users, it is important to take into account the multiple values

of indirect users as well as those of different actors participating in the development, (Djellal and Gallouj, 2010,

2013). Thus, in service innovation boundary lines are blurring – not only between products and processes – but also

organizations often change simultaneously (Preissl, 2000). These kinds of combinatory innovations and their

broader socio-economic impacts cannot be tackled on the basis of the traditional linear and industrialist models.

The double gap has been noticed to cause significant implications to public policies, which are still very technologist

oriented and do not take into account the innovation and performance gap included. Thus, the invisible innovation

and performance remain invisible in the policy making, causing problems – not only in the performance

measurement as such – but also in target setting, and in steering and policy planning. In order to improve the

situation, both the visible and invisible aspects in innovation and performance has to be included in an integrative

way (Djellal and Gallouj, 2010).

However, the analysis above does not mean that all existing innovation and performance measures and indicators

should be abandoned. Instead of that, the need for a more diversified, multi-criteria framework can be identified.

The framework should also be flexible, and besides the traditional economic measures (such as productivity and

performance), focus on analyzing the aspects relating to social innovation and use value linked to the performance.

This kind of an approach also prefers the concept “evaluation” over the concept of “measurement” (Djellal and

Gallouj, 2010, 2013).

A NEW FRAMEWORK FOR EVALUATION

Multi-criteria approach to diversify the perspectives of evaluation

These above described challenges in measurement and evaluation of services intend to assess the performance and

impacts on the basis of a multidimensional approach which takes into account the special characteristics of services

as well as their aspects of quality and social value (Djellal and Gallouj, 2010, 2013; Rubalcaba et al., 2013). Like

Djellal and Gallouj (2010, 2013) have argued one potential and diversified way of analyzing the various purposes

and outcomes of services is to link to them to the idea of different “worlds”, which are the industrial and

technological word, the market and financial world, the relational and domestic world, the civic world, the world of

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

5

innovation and the word of reputation. Referring to the initial description of worlds (deriving for the economic

tradition; Boltanski and Thévenot, 1991; Gallouj, 2002) aim is in giving an insight to different justificatory aspects

in the society considering their different principles and values.

In that model the outcomes are evaluated from the perspective of different goals or target areas, encompassing both

the traditional measures arising from the industrial and technical as well as market and financial world and the

modern evaluation criteria taking into account the needs of knowledge society (e.g. relational aspects such as trust

and aspects relating to the responsibility and societal wellbeing). In addition to the different target areas the model

focuses on the long time-scale in the generation of impacts by dividing outcomes into the direct, short-term outputs

and indirect, long term-outcomes.

We have made some modifications to the original model. Our changes concern especially the aspect of “the civic

world” that originally was focused to social relations characterized by the ethical issues such as equal treatment and

fairness. In this analysis we suggest to integrate the concept of societal value and responsibility into the idea of civic

world and also rename it “responsible world”. Our aim is to better take into the account the aspects of social

innovations, environmental sustainability and societal well-being (see Rubalcaba et al., 2012, Djellal and Gallouj,

2013). In the original model also the word of innovation is differentiated. Because in study focuses is on innovations

the perspective of innovation world is excluded as a vertical dimension. Instead we clarify the different elements of

innovation horizontally by analyzing the core aspect of innovation in the light of each world.

In our study innovations are analyzed based on the following different worlds given the specific justification criteria

related to the each of the worlds: Industrial and technical world (Volumes, flows and technical operations); Market

and financial world (Value and monetary and financial transactions); Relational world (Interpersonal and

organizational relations, trust, quality of relationship); Responsible world (Values like sustainable development,

responsibility, equal treatment and justice), Reputational world (Brand and image). The different impact criteria

described in the analysis are interlinked and dynamic in their nature, thus they me reinforcing and contradicting each

other.

Multi-actor perspective to analyze the impacts of multiple actors involved in services creation

The interactive and subjective nature of services (and their implications to the evaluation) leads to emphasize the

multi-actor nature of services. In the recent literature the multi-actor perspective has been proposed as an analytical

framework to better understand the complex interaction between policy makers (society), public and private service

providers as well as users of the services and to help to evaluate the benefits of the services from the different

perspectives (Gallouj et al., 2013). By integrating the actors‟ perspective to the evaluation we aim to emphasize the

subjective nature of evaluation; evaluation criteria are always dependent on actors‟ specific values and interests in a

specific situation. In addition we aim to pay attention to the interaction of various actors and their values creating an

evaluation situation (cf. Giddens, 1987).

Interaction may be operationalized based on the multi-agent framework that has been developed specifically to the

context of public and private innovation networks to develop innovations and services (Windrum, 2013).The central

idea in the framework is to study and understand the complex interactions between multiple actors in concerning

service innovations. Framework is based on the former work of (Windrum and Garcia-Goñi, 2008; Gallouj and

Weinstein, 1997). When the primary work focuses to analyze services based on their characteristics in the later work

the core is in actors. Particularly important are the characteristics and preferences of the actors. One of the values of

this dimension is that it puts particular emphasis on the public sector‟s role in the innovation process. Typically

innovation research has been focused on the private sector innovation without putting too much effort to understand

the role of public sector in creation of innovations. However, according to Windrum (2013), both political

organizations and non-governmental organizations play an important role in innovation generation and

dissemination and needs to be considered. Compared to the former model the value added in the recent one is in the

recognition of public sector actors and thus the interaction between economic, social and political spheres (ibid.).

In our study we are analyzing the impacts from the perspective of multiple different actors taking part to the service

and innovation development. In the analysis the preferences and thus the benefits are emphasized from the

viewpoint of society (represented by policy makers), public sector service providers, private sector service providers

and end users.

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

6

CASE CONTEXT AND METHODOLOGY

In our empirical analysis the focus is in the complex combinatory services in the area of environmental analysis and

measurement. Paper studies outputs and outcomes from the viewpoint of different worlds and different actors

focusing to the Environmental data platform that aims to be a comprehensive platform to support continuous data

gathering and real-time environmental monitoring, analysis and reporting. Innovation has been developed in

research and development program built on public-private innovation network that aim to promote systemic change

in the field of environment in Finland.

We have applied the case study methodology and qualitative approach in our study. The main method in data

gathering in our study was face-to-face interviews (30 in total). The main topics focused on the manifestations of

new innovative solutions within the programs, factors that promote or slow down their generalization, impacts of the

innovations and their evaluation. The interviews were gathered between February and June 2013. The final sample

represented actors in the area of environment in a versatile way. It consisted of representatives of private companies

(e.g. companies developing environmental and industrial measurement), and universities and other public research

organizations.

To analyze and interpret empirical data we applied the modified multi-criteria and multi-actor frameworks. To start

we analyzed what are the impacts of innovation in a short and long term from the perspective of different worlds. In

the analysis we reflect impacts both from the traditional and modern perspectives. From the industrialist viewpoint

typical evaluation criteria are for example technical operations and volumes. Similarly market world uses criteria

like value (in financial terms), market share and business transaction. Relational and responsible worlds aim to

broaden the focus to take into account the societal value as well as intangible, interactive, social and systemic nature

of services. Moreover, world of reputation analyzes innovation from the perspective of brand and image effect.

Finally we illustrate in very general level how the impacts vary from the perspective of different actors participating

into the innovation creation. In the analysis the role of policymakers, public service providers, private service

providers and end-users are taken into account.

RESEARCH RESULTS

Multi-criteria perspective for the analysis of new sustainable services

Environmental data platform (table 1) is from an industrial perspective a prototype platform to gather and share

environmental data. As a short term output it integrates real-time data sources (e.g. measured data of water quality

and satellite data concerning environment and atmosphere) and provides visualized maps based on the data. In a

long-term goal is to integrate multiple data sources in it and provide “cloud-based comprehensive solution” to

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

7

Table 1. A multi-criteria framework to analyse the impacts of environmental data platform

Industrial and technological Market and financial Relational Responsible Reputation

Innovation/Input A prototype platform for

environmental data gathering and

sharing (integrating real-time data

sources & data storage)

Free access to public sector data Connectivity via one mediator

to multiple data sources and

end-users

Open and equal access to public

sector data

Image as an innovation to

enhance sustainability

Output (direct, short

term)

Varied measured and satellite data concerning environment and

atmosphere (e.g Weather radar

visualised on the map)

Economical of free access to multiple environmental data

sources

New connections and actor networks created (via new

solution)

Transparency of public data; Easy access

Short term image

Performance related to

output

Ability to process increasing amount of environmental data more quickly

and more effectively

Reduced cost of sharing data

Reduced cost of providing data

Increased connectivity between multiple actors

Increased transparency of public data

Increased availability and better use of public data

Improvement/change in reputation and image

Outcome (indirect,

longterm)

“Cloud-based comprehensive

solution” for producing and sharing environmental data

Architecture for real time environmental monitoring, analysing

and reporting, bid data

New end-user applications created

New start-ups crated; new jobs

New market opening based on

environmental monitoring

New opening export possibilities

Integrating citizens and

citizens‟ requirements to the data provision

“Users as a data providers” (Social innovation)

Personalised environmental data

Trust

Equality in sharing information

(data)

Quality controlled data

Platforms applied e.g. in

education: environmental

education and awareness

Long-term reputation and

image

Performance related to

outcome

Improvements in weather forecasts,

environmental information and warnings

Increased quality and reliability of environmental data

Increasing number of business start-ups based on environmental data;

Increasing number of new jobs

Improvements in the generation

of various types of revenues based on environmental data

Increased usability of

environmental data /knowledge

Increased trust

Increased awareness of

environmental issues

Increased responsibility in

decision making (individual, firm and policy level)

Long-term

improvement/change of reputation and image

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

8

produce and share environmental data. Cloud-based solution would make the access to big data possible. In addition

the architecture for real-time monitoring, analyzing and reporting will be created to improve the quality and

reliability of environmental information, weather forecasts and warnings. New architecture helps in developing new

end-user applications and thus accelerates business start-ups generation. Our respondents highlighted that services

are in the central role in the development of comprehensive solution and new information architecture.

In the market world the main characteristic of the innovation is free access to the multiple data sources and

especially to public sectors data. According to our interviewees free access to data is considered to be an important

social goal and as a starting point for the development of data platform. In a long term goal is in opening new

markets based on environmental analysis and in creation of new export possibilities.

Preferences and valuation principles in the relational world highlight the interaction between multiple actors and a

role of end users is the development and use of service. Central innovative aspect here is the connectivity via one

mediator to multiple data sources. As output goal trust in the public-private innovation networks is enhanced. From

the performance viewpoint, that enhances actors‟ connectivity and consolidates the networks. Long-term goal is to

integrate citizens to the data provision, to better take into account the specific users‟ needs and requirements and

hence provide personalized environmental data. In a long term users will be integrated part of the environmental

data generation. That development removes the clear distinction between the production and use of data (citizens are

becoming data producers), which has been considered as a prerequisite for a systemic change in the sector.

From the perspective of responsible world overall sustainability is highlighted as a central value in development of

new solution. The other value based aspects highlighted are transparency and the citizens‟ equal rights to participate

to the data provision. Innovative aspect relates to the open and equal access to public sector data. As a short term

goal platform enhances transparency, availability and multifaceted us of public data. As a long-term goal

respondents have considered the possibilities to apply new end-user applications for example to the game and school

worlds to support the environmental education and thus increase the awareness of environmental issues from the

early age. From the performance viewpoint that may increase responsibility as a value in decision making in

individual, firm and policy level.

Finally in the world of reputation innovation is analyzed from the perspective of brand and image effect. From a

short and long term the aim is in analyzing how the innovation impact to the attractiveness of different actors. The

aspects concerning the image and brand needs to be analyzed from the perspective of specific actors and are thus

analyzed in details in the following sub-chapter.

To summarize the central findings from the perspective of different type of evaluation criteria following aspects

have been recognized. According to our interviews services have important role in renewing the environment sector.

To solve the most urgent societal problems, like sustainability, there is need to develop technologies and services as

systemic wholes instead of individual technologies or services. The emergence of comprehensive service solutions

in renewing the environment and energy technologies can be perceived by traditional measures. However, the focus

from the perspective of industrial and financial world is in technical characteristics of the concepts and solutions, not

so much in their social and systemic nature.

To capture the interactive and social nature of service development as well as its‟ interlinks to the social goals and

system level changes the other criteria are needed. Like our example show, relational and responsible worlds are

particularly important to make these aspects visible. Like our respondents highlighted the changing roles of data

users and producers (“from user to producer”) plays a central role in the renewal and a system level change in a

sector. That can be perceived only from the perspective of relational world. In addition our example shows that

impacts generated in the different worlds are often interdependent and complementary to each other. Some factors in

relational and responsible worlds can be seen as a prerequisite to effects generated from the viewpoint of technical

and financial worlds. For example for the development of environmental data platform the transparency and open

access to public sector data is seen as a precondition.

Multi-actor perspective in the analysis of new sustainable services

In the following discussion and table (2) we exemplify the impacts from the perspective of multiple actors. Central

actors taking part into the service innovation development are society (policy makers setting the societal goals and

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

9

Table 2. A multi-actor framework to analyse the impacts from the perspective of different actors

Industrial and technical Market and financial Relational Responsible Reputation

Private service provider Knowledge transfer: Better use of academic knowledge as a

business opportunity

Use of multiple data sources to

create new business

Variety of commercial

applications

New revenues

Access to international markets

Export incomes

Mutual trust and co-creation between multiple

partners

Opportunity for new roles

and responsibilities

(intermediator)

Sustainable business

Awareness: corporate social

responsibility in firm level decision making

Predictive information of environmental changes

(managing risks)

Image as a successful and international start-up

company

Socially responsible company

(as a competitive advantage);

corporate social responsibility

Public service provider Access to the big data

Knowledge transfer between

multiple partners

New profound and multifaceted

knowledge of environment and atmosphere

New public services

New (research)markets based on multifaceted environmental

data

Mutual trust and co-creation between multiple

partners

Opportunity for new roles

and responsibilities

(enabler, intermediator)

Predictive information of environmental changes

Respond to the social needs

Safety, security and health

benefits

Internationally high level environmental knowledge

(publications)

Responsible public

organisation

End user Access of to the multiple data sources and big data

Receive and provide real-time environmental data (e.g. via smart

phone)

Better (measures) health

Job opportunities

Free access to public data

Expenditures (applications, new

devices)

Change in role: users as data providers

Influence; ability to participate

Personalised environmental

information based on your

location)

Usability of environmental information

Awareness and sensitivity to ecological issues

Empowered citizen

Better health (experienced)

Role as an active and empowered citizen

Influential citizen

Society (Policy makers) New environmental knowledge

New start-ups created

New jobs created; employment

rate

Growth of services,

“servitization” , service society

New business ecosystems created

Tax revenues (new companies,

new taxpayers)

Investments; public

expenditures (e.g. renewed

educational infra)

Export share of GDP

Market share of services (%)

Economical growth and productivity

Trust in innovative

networks consolidates the

innovation networks

Creation of innovative,

creative and entrepreneurial

culture

Environmental education

Transparency in society

Social impacts e.g. health, living conditions

Quality of societal services

Responsibility in decision

making

Innovation system: enabler

for creativity, services and

innovations

Democratic society

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

10

preferences for the development), public service providers, private service providers and end-users. By integrating

the actors‟ perspective to the evaluation we aim to emphasize the subjective nature of evaluation; evaluation criteria

are always dependent on actors‟ specific values and preferences in a specific situation. In the analysis we zoom

environmental data platform from the perspective of longer term outcome to exemplify the diversity of the impacts

based on the values and preferences of different actors.

Based on the preferences and values of the private service providers it is natural the benefits of the innovation

relates to the knowledge transfer, the development of new commercial applications and thus in the creation of new

business opportunities. From the financial perspective, that brings new revenues and opens up the access to the

international markets. New evaluation criteria enrich the analysis by highlighting the importance of mutual trust

between multiple partners, which has been highlighted as a core aspect in development of new innovations and

services. In addition it relational world makes visible the opportunity for the new and changing roles and

responsibilities when developing and offering new services. Moreover sustainability as a value and as a competitive

advantage is has also been highlighted from the perspective of private service providers: a reputation as a

responsible company is a competitive advantage for the contemporary companies which may boost their success in

international market.

A long term outcome for the public service providers relates primarily to the access and new opening opportunities

based on big data. That creates opportunity for the profound and multifaceted knowledge production which

enhances more quality environmental data production and based on that creation of new public services. From the

perspective of public sector service providers the opening the new markets and research opportunities were

highlighted. And like the private sector companies they also found that knowing new actors and creating the mutual

trust between the multiple partners for deepening the collaboration are in the part of the main benefits from the

process. In the other words, trust can be characterized as an enabling factor in the co-creation of innovation. From

the responsibility viewpoint provision of predictive information of weather and environmental changes helps to

manage potential environmental risks. Because of the specific role of public sector service providers this perspective

is highlighted here. Capacity to provide environmental (e.g. safety environment) and societal impact (e.g. citizens

health and good living conditions) belongs to the roles and functions of public service providers.

From the viewpoint of end-users receiving personalized real-time environmental data is the most self-evident benefit

of the innovation. The data personalized to your own needs and received via you own mobile phone based on your

location may – for example real-time pollen reports for the allergic and asthmatic person – helps people to predict

the pollen risks and thus have important health impacts. In a longer term the citizens‟ ability to provide data and act

as an active and equal data provider and empowered citizen was examined as an important impact. From the

relational viewpoint the change in end-users role was recognized as an important impact from individual but also

societal point of view: their active and equal role as an environmental data provider creates preconditions for the

new innovative and service oriented culture as well as demonstrates the potential for the systemic change in the

sector.

Society is serving the multiple actors thus the impacts described here is combination of everything we have

represented above. In enabling the creation of new business ecosystems society‟s preference is to enhance the

emergence of start-ups, help them to internationalize (export share of GDP) and to create the jobs. From the

financial viewpoint that brings new revenues for the society and helps in developing societal services (which in turn

causes impact in health and living conditions). In addition society has a role to promote and consider more soft and

social aspects in the society. For example create new „programs‟ (meaning devices and educational programs) for

the environmental education and thus raising the awareness and responsibility of citizens are the most important

impacts from the viewpoint of responsible world. In addition transparency of the society relating to the openness of

public data is a self-evident goal for the society.

To summarize, like our analysis show experience of impacts and benefits are strongly related to the preferences and

values of actors. Both their goal and motivation to take part into co-creation is different and thus are the experienced

impacts. From that perspective it is evident that there are contradictions regarding the benefits of different actors.

However, different goals and preferences are in the most cases seen as complementaries which enhances the co-

creation. When considering the impacts crosswise from the perspective of multiple actors and different worlds we

create more diversified and analytical picture of the generated or potential impacts. In addition analysis makes

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

11

visible that in many cases the impacts are strongly interlinked and are systemic in nature.

CONCLUDING REMARKS

The purpose of this paper is to study the challenge of the evaluation in the context of the services. Because of their

specific nature the traditional evaluation methods and measures are not able to capture neither the diversity of the

innovations nor the multifaceted dimensions of performance in the sector. This paper aims to contribute to the need

for a more diverse evaluation approach. We focus on the context of service innovation in the sector on environment

and energy, in which we develop further multi-criteria and multi-actor perspectives. Multi-criteria framework

describes the impacts of new sustainable services and multi-actor framework aims in analyzing the impacts from the

perspective of multiple actors taking part into the service and innovation development.

As a result we identified that services have an important role in renewing the energy and environment sector. The

emergence of comprehensive service solutions in renewing the environmental sector can be perceived partially the

by traditional measures. However the focus from that angle is in technical and measurable characteristics of the

concepts and solutions. From the traditional perspective their interactive nature and multifaceted performance

(including the social and value based aspects) cannot be captured. To perceive and make visible the societal goals of

the service innovations as well as their interactive nature relational and responsible worlds are particularly

important. Moreover, our analysis also shows that impacts generated in the different worlds are often interdependent

and complementary to each other. Some factors in relational and responsible worlds can be seen as a prerequisite to

effects generated from the viewpoint of technical and financial worlds. In other words relational and responsible

aspects are needed to generate long-term technological and economic impacts.

From the multi-actors perspective impacts and benefits are strongly related to the preferences and values of actors.

Analysis make visible that there are evident (value and motivation based) contradictions between different actors.

However, when there is a mutual trust these are seen as complementaries which enhances the co-creation. When

considering the impacts crosswise from the perspective of multiple actors and different worlds we create more

diversified and analytical picture of the generated or potential impacts. In addition analysis makes visible that in

many cases the impacts are strongly interlinked and are systemic in nature.

Regarding to the further studies more studies would be useful to test the generalizability of our results. Furthermore

to go deeper to the dynamic relations (systemic nature) of the different actors and impacts the application of

systemic modelling methods could be the next step. In addition generating indicators based on these findings would

be both interesting and useful.

REFERENCES

Ahrweiler, P. (ed.) (2010), “Innovation in Complex Social Systems”. Routledge, New York

Arnold, E. (2004), “Evaluating research and innovation policy: a systems world needs systems evaluations”. Research

Evaluation, Volume 13 No.1, pp. 3–17.

Cabrera, D., Colosi, L. & Lobdell, C. (2008), “Systems thinking”. Evaluation and Program Planning, Volume 31 No. 2, pp. 299-

310

Chelimsky, E. (1997), “The Coming Transformation in Evaluation”. In E. Chelimsky & W. Shadish, (eds.) Evaluation for 21st

century. A handbook. Thousand Oaks. Sage Publications, pp. 1-26.

Coombs, R. and Miles, I. (2000), “Innovation, measurement and services: the new problematique”. In J.S. Metcalfe and I. Miles

(eds), Innovation Systems in the Service Economy: Measurement and Case Study Analysis, London: Kluwer Academic

Publishing, pp.85–104.

Cooper R.G. & De Brentani U. (1991), “New industrial financial services: what distinguishes winners”, Journal of Product

Innovation Management,8 (2), pp. 75-90.

Cozzens, S. & Melkers J. (1997), “Use and Usefulness of Performance Measurement in State Science and Technology Programs"

Policy Studies Journal, Volume 25 No. 3, pp. 425-435

Djellal, F. & Gallouj, F. (2010), “The Innovation gap and the performance gap in the service economies: a problem for public

policy”, in F. Gallouj & F. Djellal, (eds.), The Handbook of Innovation in Services. A Multi-disciplinary Perspective,

Edward Elgar, Cheltenham, UK, pp. 653-673

Djellal, F. & Gallouj, F. (2013), “The Productivity in services: measurement and strategic perspectives.” The Service Industries

Journal Volume 33 No 3-4 pp. 282-299

Djellal, F. & Gallouj, F. (2013b), “How public-private innovation networks in services (ServPPINs) differ from other innovation

5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen

12

networks: what lesson for theory?”, in F. Gallouj, & L. Rubalcaba, P. Windrum, (eds.) Public-private innovation networks

in services, Edward Elgar, Cheltenham, UK.

Dosi, G. (1999), “Some notes on national systems of innovation and production, and their implications for economic analysis”, in

D. Archibugi & J. Howells and J. Michie, (eds.): Innovation Policy in a Global Economy, Cambridge: Cambridge

University Press.

Dyehouse, M., Bennett, D., Harbor, J., Childress, A. & Dark, M. (2009), “A comparison of linear and systems thinking

approaches for program evaluation illustrated using Indiana Interdisciplinary GK-12”. Evaluation and program planning

32(2009) 187.196.

Edquist, C. (2005), “Reflections on the systems of innovation approach”. Science and public policy, Volume 31 No. 6, 485-489.

Freeman C. (1991), “Networks of innovators: A synthesis of research issues”, Research Policy, Volume 20 No. 5, 499-514.

Freeman, C. & Soete, L. (2009), “Developing science, technologic and innovation indicators. What can we learn from past”.

Research Policy, Volume 38, pp. 583-589

Gadrey, J. (2010), “The environmental crisis and the economics of services: the need for revolution. in: The Handbook of

Innovation in F. Gallouj & F. Djellal (eds.), Services. A Multi-disciplinary Perspective, Edward Elgar, Cheltenham, UK,

pp. 653-673

Gallouj, F. & Weinstein, O. (1997), “Innovation in services”. Research Policy, Volume 26 No. 4/5, 537-556.

Gallouj, F. (1994): “Economie de l'innovation dans les services”, Editions L'Harmattan, Logiques. Économiques, Paris.

Gallouj, F. (2002),” Innovation in the service economy: The new wealth of nations”, Edward Elgar.

Gallouj, F., Rubalcaba, L., Windrum, P. (2013), “Public-private innovation networks in services”. Edward Elgar.

Giddens, A. (1987). “Social Theory and Modern Sociology”. Cambridge: Polity Press.

Hansson, F. (2006), “Organisational use of evaluation”. Evaluation Volume 12 No. 2, pp.159-178.

Howells, J. (2004), „Innovation, consumption and services: encapsulation and the combinatorial role of services‟, The Service

Industries Journal, Volume 24 No. 1, pp. 19-36.

Kellogs Foundation, 2004. “Using logic models to bring together planning, evaluation & action logic model development guide”.

Battle Creek, Michigan. http:www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

Kline, S.J. and Rosenberg, N. (1986), “An Overview of Innovation”, in R. Landau & N. Rosenberg (eds.): The Positive Sum

Strategy – Harnessing Technology for Economic Growth, Washington D.C.: National Academy Press.

Lundvall, B-Å. (1992), “National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning”, London and

New York: Pinter Publishers.

Metcalfe, J.S. and Miles, I. (eds.) (2000), ”Innovation Systems in the Service Economy – Measurement and Case Study

Analysis”. Boston, Dordrecht and London: Kluwer Academic Publishers.

Nelson, R.R. and Rosenberg, N. (1993), “Technical Innovation and National Systems”, in R.R. Nelson (ed.): National Innovation

Systems – A Comparative Analysis, New York and Oxford: Oxford University Press.

Patton, M. Q. (2011). “Developmental evaluation: Applying complexity concepts to enhance innovation and use”. New York,

NY: Guilford.

Preissl, B. (2000): “Service Innovation: What makes it different? Empirical Evidence from Germany”, in J.S. Metcalfe, & I.

Miles (eds.): Innovation Systems in the Service Economy – Measurement and Case Study Analysis. Boston, Dordrecht and

London: Kluwer Academic Publishers.

Rip, A. (2003), “Societal Challenges for R&D Evaluation”, in P. Shapira and S. Kuhlmann (eds.), Learning from Science and

Technology Policy Evaluation: Experiences from the United States and Europe. Cheltenham: Edward Elgar, pp. 35-59.

Rubalcaba, L. & Michel, S. & Sundbo, J. & Brown, S.W. & Reynoso, J. (2012), Shaping, Organizing and Rethinking Service

innovation: A multidimensional Framework. Journal of Service Management, Volume 23, Issue 5, pp. 696 – 715, ISSN

1757-5818,

Smith K.R. (2000), “Innovation as a systemic phenomenon: Rethinking the role of policy”. Enterprise and Innovation

Management Studies Volume 1 No. 1, pp. 73-102.

Smits, R. & Kuhlmann, S. (2004), “The rose of systemic instruments in innovation policy”. Foresight and Innovation Policy,

Volume 172

Sundbo, J. (1998), “The theory of innovation: entrepreneurs, technology and strategy”. Edward Elgar, Cheltenham, UK.

Toivonen, M. (2010), “Different types of innovation processes in services and their organizational implications”, in F. Gallouj &

F. Djellal (eds.), The Handbook of Innovation in Services. A Multi-disciplinary Perspective, Edward Elgar, Cheltenham,

UK, pp. 653-673

Van der Knaap, P. (2006), “Performance Evaluation and Performance management. Owercoming the Dowensides of Policy

Objectives and Performance Indicators.” Evaluation Volume 12 No 3, pp. 278-293

Williams, B. & Imam, I. (eds.) (2007), “Systems Concepts in Evaluation: An Expert Anthology”. Point Reyes: EdgePress of

Inverness.

Windrum, P. & García-Goñi, M. (2008), A neo-Schumpeterian model of health services innovation, Research Policy, Volume 37

No. 4, pp.649-672.

Windrum, P. (2013), “Multi-agent framework for understanding the success and failure of ServPPINs, in Public-private

innovation networks in services, Gallouj, Faîz, Rubalcaba, Luis, Windrum, Paul (Eds.) Edward Elgar, Cheltenhamn, UK