8/14/2019 Tom Koontz Friday

1/18

11

Evaluating the Performance of

Collaborative Environmental

Governance

Tomas M. KoontzTomas M. Koontz

School of Environment & Natural ResourcesSchool of Environment & Natural Resources

Ohio State UniversityOhio State University

Craig W. ThomasCraig W. Thomas

Daniel J. Evans School of Public AffairsDaniel J. Evans School of Public Affairs

University of WashingtonUniversity of Washington

8/14/2019 Tom Koontz Friday

2/18

22

The Problem We FaceThe Problem We Face

Collaboration has increasingly replaced otherCollaboration has increasingly replaced otherforms of environmental governanceforms of environmental governance

Such as centralized planning andSuch as centralized planning and

regulationregulation This may be a good thingThis may be a good thing

But we have very little knowledge about theBut we have very little knowledge about the

environmental outcomes of collaborationenvironmental outcomes of collaboration Collaboration appears to be good for socialCollaboration appears to be good for social

outcomes (like social capital)outcomes (like social capital)

But it may be bad for the environmentBut it may be bad for the environment

8/14/2019 Tom Koontz Friday

3/18

33

What is Collaborative Governance?What is Collaborative Governance?

Multiple governmental andMultiple governmental and

nongovernmental stakeholdersnongovernmental stakeholders

working together to develop andworking together to develop and

advance a shared visionadvance a shared vision

8/14/2019 Tom Koontz Friday

4/18

44

Research QuestionsResearch Questions

Is collaborative governance good for theIs collaborative governance good for the

environment?environment?

Does collaborative governance produceDoes collaborative governance producebetter environmental outcomes than otherbetter environmental outcomes than other

forms of governance (such as centralizedforms of governance (such as centralized

planning or regulation)?planning or regulation)?

8/14/2019 Tom Koontz Friday

5/18

55

How Collaboration Might AffectHow Collaboration Might Affect

the Environmentthe Environment

Negatively: Lower common-denominatorNegatively: Lower common-denominator

solutions during policy adoptionsolutions during policy adoption

Positively: More durable solutions duringPositively: More durable solutions during

implementationimplementation

8/14/2019 Tom Koontz Friday

6/18

66

Inputs

(e.g., human,

financial,

and technical

resources)

End

outcomes

(social and

environmental)

Processes

(e.g., collabor-

ation)

Intermediate

outputs

(e.g., plans)Intermediate

outcomes

(e.g. human

behavior)

All other

non-programmatic

variables

End

outputs

(e.g., projects)

Basic Logic Model

8/14/2019 Tom Koontz Friday

7/18

77

State of the Literature onState of the Literature on

Collaborative EnvironmentalCollaborative Environmental

GovernanceGovernance

Empirical research has focused on:Empirical research has focused on:

Inputs (e.g., human, technical, financial)Inputs (e.g., human, technical, financial) Processes (e.g., consensus, ADR)Processes (e.g., consensus, ADR)

Outputs (e.g., plans, permits, projects)Outputs (e.g., plans, permits, projects)

Social outcomes (e.g., trust, social capital)Social outcomes (e.g., trust, social capital)

Very little research on environmental outcomesVery little research on environmental outcomes

8/14/2019 Tom Koontz Friday

8/18

88

Recent Empirical Claims aboutRecent Empirical Claims about

Environmental OutcomesEnvironmental OutcomesLeach and Sabatier (2005) measureLeach and Sabatier (2005) measure

participantparticipant perceptionsperceptions of environmentalof environmental

changechange

Others use outputs as proxies for outcomesOthers use outputs as proxies for outcomes

(either intentionally or carelessly)(either intentionally or carelessly)- Salmon recovery plans; farmland preservation;- Salmon recovery plans; farmland preservation;

Superfund projects; choicesSuperfund projects; choices

- Beware: outputs may not be good proxies for- Beware: outputs may not be good proxies for

outcomes, due to intervening variablesoutcomes, due to intervening variables

8/14/2019 Tom Koontz Friday

9/18

99

Possible Research DesignsPossible Research Designs

ExperimentalExperimental

Quasi-experimentalQuasi-experimental

StatisticalStatistical

Case StudiesCase Studies

8/14/2019 Tom Koontz Friday

10/18

1010

Experimental MethodsExperimental Methods

WhatWhat

Randomized trialsRandomized trials

With focus on collaboration as aWith focus on collaboration as a

management interventionmanagement intervention

FeasibilityFeasibility

Very lowVery low Primarily due to long time horizons forPrimarily due to long time horizons for

environmental changeenvironmental change

8/14/2019 Tom Koontz Friday

11/18

1111

Quasi-Experimental MethodsQuasi-Experimental Methods

WhatWhat

Also focus on collaboration as a managementAlso focus on collaboration as a management

interventionintervention

But w/o randomized controls or other featuresBut w/o randomized controls or other featuresof classic experimental designof classic experimental design

FeasibilityFeasibility

Easy to find natural experimentsEasy to find natural experiments

Harder to find matched-case controls (usingHarder to find matched-case controls (using

most-similar comparative methods)most-similar comparative methods)

Harder to find valid longitudinal dataHarder to find valid longitudinal data

8/14/2019 Tom Koontz Friday

12/18

1212

Statistical MethodsStatistical Methods

WhatWhat Analyzes correlations among variablesAnalyzes correlations among variables

to test probability of causal effects (not causalto test probability of causal effects (not causal

mechanisms)mechanisms)

FeasibilityFeasibility

High for existing data setsHigh for existing data sets e.g., PART (with caveat)e.g., PART (with caveat)

Harder to find valid longitudinal dataHarder to find valid longitudinal data

8/14/2019 Tom Koontz Friday

13/18

1313

Inputs

(e.g., human,

financial,

and technical

resources)

End

outcomes

(social andenvironmental)

Processes

(e.g., collabor-

ation)

Intermediate

outputs

(e.g., plans)Intermediate

outcomes

(e.g. human

behavior)

All other non-programmatic

(exogenous) variables

End

outputs

(e.g., projects)

Experimental, Quasi-Experimental,and Statistical Methods

8/14/2019 Tom Koontz Friday

14/18

1414

Case Study MethodsCase Study Methods

WhatWhat Emphasizes process tracing in causalEmphasizes process tracing in causal

sequencessequences

Searches for causal mechanisms, not justSearches for causal mechanisms, not justcausal effectscausal effects

FeasibilityFeasibility

HighHigh But requires careful use of counterfactuals,But requires careful use of counterfactuals,

congruence with theory, and controlledcongruence with theory, and controlledcomparisons (holding other variables constant)comparisons (holding other variables constant)

8/14/2019 Tom Koontz Friday

15/18

1515

Inputs

(e.g., human,

financial,

and technical

resources)

End

outcomes

(social andenvironmental)

Processes

(e.g., collabor-

ation)

Intermediate

outputs

(e.g., plans)Intermediate

outcomes

(e.g. human

behavior)

All other non-programmatic

(exogenous) variables

End

outputs

(e.g., projects)

Case Studies

8/14/2019 Tom Koontz Friday

16/18

1616

Accountability for PerformanceAccountability for Performance

Collaboration should be held to the sameCollaboration should be held to the same

performance standards as other forms ofperformance standards as other forms of

governancegovernance

Collaboration may be quite complementaryCollaboration may be quite complementaryto developing performance mgt systemsto developing performance mgt systems

Incentives for partners to monitor, gatherIncentives for partners to monitor, gather

information, develop metrics for successinformation, develop metrics for success

But how can we do it from a methodologicalBut how can we do it from a methodological

perspective?perspective?

8/14/2019 Tom Koontz Friday

17/18

1717

Our Research AgendaOur Research Agenda

In processIn process Case studies of public sector collaborative effortsCase studies of public sector collaborative efforts

To show links from processes through to outcomesTo show links from processes through to outcomes

Coding PART reportsCoding PART reports

Matched case comparisons for statistical analysis:Matched case comparisons for statistical analysis:

collaborative v. non-collaborative cases within programscollaborative v. non-collaborative cases within programs

But many outputs/outcomes are miscodedBut many outputs/outcomes are miscoded

Link to theory for generalizabilityLink to theory for generalizability

Future ResearchFuture Research Quasi-experimental methodQuasi-experimental method

Using remote sensing data, water quality reports, etc.Using remote sensing data, water quality reports, etc.

Ideas for comparative cases, anyone?Ideas for comparative cases, anyone?

How do perf. mgt. systems get started?How do perf. mgt. systems get started?

8/14/2019 Tom Koontz Friday

18/18

1818

ConclusionConclusion Is collaborative governance good for theIs collaborative governance good for the

environment?environment?

Does collaborative governance produceDoes collaborative governance producebetter environmental outcomes than otherbetter environmental outcomes than other

forms of governance (such as centralizedforms of governance (such as centralizedplanning or regulation)?planning or regulation)?

These are very important (and too oftenThese are very important (and too often

unasked) questionsunasked) questions

But they are difficult to answerBut they are difficult to answer

And they will keep us busy for a long timeAnd they will keep us busy for a long time