Situated EvaluationSituated Evaluation

Tony ColomaTony ColomaMME 207MME 207

Situated Evaluation:• Conceives technology users as active creators, rather than

as “passive recipients of technological products and scientific knowledge (Eglash, 2004).

• Emphasizes the unique characteristics of each situation in which the innovation is used.

• The object of interest is not the idealized form in the developer’s specs, but rather, the realization through use.

What is innovation?What is innovation?• No standard definitionNo standard definition• Core elements: Core elements:

NewNewCreates valueCreates valueWidely adoptedWidely adopted

• ““The process of making changes to something The process of making changes to something established by introducing something new established by introducing something new whether it is an idea, method or device” whether it is an idea, method or device” (Von (Von Hippel)Hippel)

Why is Innovation a critical issue now?Why is Innovation a critical issue now?• New processes have made it possible to create mass New processes have made it possible to create mass

markets for products that were previously the markets for products that were previously the preserve of elitespreserve of elites

• New products have allowed people to achieve higher New products have allowed people to achieve higher living standards. Innovative companies are more living standards. Innovative companies are more likely to grow and prosperlikely to grow and prosper

Types of Innovation Program Types of Innovation Program EvaluationEvaluationEx-ante evaluationEx-ante evaluation, , conducted before the implementation conducted before the implementation

of a program, and focusing on what its objectives are and of a program, and focusing on what its objectives are and how (and how far) they should be achieved.how (and how far) they should be achieved.

Intermediate evaluationIntermediate evaluation, , reviewing the progress of the reviewing the progress of the program, or its achievements at some point into the program program, or its achievements at some point into the program – usually reserved for long-lasting programs.– usually reserved for long-lasting programs.

Real-time evaluationReal-time evaluation, , following the program in detail following the program in detail through its operation.through its operation.

Ex-post evaluationEx-post evaluation, , examining the results of the program examining the results of the program after it has been completed (and possibly more than once).after it has been completed (and possibly more than once).

Types of EvaluationTypes of Evaluation

IdealizatioIdealizationnThe elements of the innovation as intended by The elements of the innovation as intended by developersdevelopers

RealizationRealizationThe ways the innovation was used, The ways the innovation was used, modified, and recreated by users modified, and recreated by users in situ.in situ.

arithmetic

Molecule

Aspects of Situated Evaluation1.The Idealization of the Innovation • Looking into the idealization of a technical system

or program• Targets how participants in the setting perceived

the innovation• It is also an index of the intentions of the

developers• Innovation is not seen as an agent that acts upon

the users or the setting, but rather as one or more element added to the complex and dynamic system.

2.Setting in which the innovation appears

• Cultural• Institutional• Pedagogical

3.Realizations of the Innovation : Realization of the innovation is studied in different settings• Analyzing how the idealization was

consonant or dissonant with existing social practices.• How innovation’s use led to new social

organization• Examine changes in the design of the

innovation brought by its use and the ways these changes relate to new practices.

  Formative Summative SituatedFocus Innovation Effects of the

InnovationSocial Practices

Audience Developer User User and Developer

Purpose Improve the innovation

Decide whether to adopt innovation

Learn how the innovation is used

Variability of Settings

Minimized to high-light technology

Controlled by balance design or random sampling

Needed for contrastive analysis

Measurement Tools

Observation/Interview/Survey

Experiment Observation/Interview

Time of Assessment

During development

After initial development

During and after development

Results List of changes to the technology

Table of measures contrasting groups

Ethnography

Evaluation should:Evaluation should:• Help determine whether an investment to generate Help determine whether an investment to generate

innovation was useful, and whether the type of innovation was useful, and whether the type of innovation was appropriate.innovation was appropriate.

• Ensure that the design of the prospective Innovation Ensure that the design of the prospective Innovation for Development is rigorous enough to avoid “spin”.for Development is rigorous enough to avoid “spin”.

What criteria should be used for What criteria should be used for the evaluation?the evaluation?Effects on the firm or other participantEffects on the firm or other participantEffects on industry more broadlyEffects on industry more broadlyCollaboration effectsCollaboration effectsEffects on the broader portfolio of innovation support Effects on the broader portfolio of innovation support

projects/programsprojects/programsSynergy effectsSynergy effectsConsistency and sustainability effectsConsistency and sustainability effects

IKED (IKED (International Organization for Knowledge Economy and Enterprise Development)

The ultimate goal is toThe ultimate goal is tomeasure the effectiveness of the measure the effectiveness of the intervention in the context of its real intervention in the context of its real impacts.impacts.AppropriatenessAppropriateness: do the programs address an objective that : do the programs address an objective that can be clearly related to policy rationale in terms of market, can be clearly related to policy rationale in terms of market, policy, system or government failure?policy, system or government failure?

SuperioritySuperiority: Are the programs more effective than other sets : Are the programs more effective than other sets of instruments that might have achieved the same goals?of instruments that might have achieved the same goals?

Systemic effectivenessSystemic effectiveness: How does the policy interact : How does the policy interact with other policies? Is there efficiency or sub-with other policies? Is there efficiency or sub-optimality in the broader setup?optimality in the broader setup?

EfficiencyEfficiency: Are the programs cost-effective in : Are the programs cost-effective in achieving their specific objectives?achieving their specific objectives?

Adaptive EffectivenessAdaptive Effectiveness: To what extent do the results : To what extent do the results from evaluations feed back into policy design and from evaluations feed back into policy design and implementation? Does policy design ensure a implementation? Does policy design ensure a sufficient degree of flexibility enabling it to respond sufficient degree of flexibility enabling it to respond to the need for change?to the need for change?

REFERENCES:REFERENCES:SMART INNOVATION: A Practical Guide to Evaluating Innovation ProgrammesSMART INNOVATION: A Practical Guide to Evaluating Innovation Programmes

Twindale, M., Randall, D., and Bentley, R. Situated evaluation for cooperative systemsTwindale, M., Randall, D., and Bentley, R. Situated evaluation for cooperative systems

Bruce, B., Rubin, A., Bruce, B., Rubin, A., Junghyun, A. Situated Evaluation of Socio-Technical SystemsJunghyun, A. Situated Evaluation of Socio-Technical Systems

Owston, R. Owston, R. Models and Methods for EvaluationModels and Methods for Evaluation

Wallace, E. , Wallnau, K. A Situated Evaluation of the Object Management Group’s (OMG) Object Wallace, E. , Wallnau, K. A Situated Evaluation of the Object Management Group’s (OMG) Object Management Management Architecture (OMA)Architecture (OMA)