Post on 01-Nov-2014
description
transcript
CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS
Page 1
CESAR WORKING DOCUMENT SERIES Working document no.2
Erreichbarkeitsatlas as Planning Support System Testing the performance in supporting strategy making
M. te Brömmelstroet 26 January 2011
This working document series is a joint initiative of the University of Amsterdam, Utrecht University, Wageningen University and
Research centre and TNO
The research that is presented in this series is financed by the NWO program on Sustainable Accessibility of the Randstad: http://www.nwo.nl/nwohome.nsf/pages/nwoa_79vlym_eng
CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS
Page 2
CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS
Page 3
TABLE OF CONTENT
1. INTRODUCTION ............................................................................................................... 4
2. ASSESSING THE PERFORMANCE OF A PSS ....................................................................... 5
2.1 Dual goals of PSS .................................................................................................................. 5 2.2 Operationalizing PSS goals ................................................................................................... 5
3. SETUP OF THE EXPERIMENT ............................................................................................ 7
3.1 Intervention: The Erreichbarkeitsatlas ................................................................................. 7 3.2 Strategy making trial with students ...................................................................................... 8 3.3 Testing the performance ...................................................................................................... 8
APPENDIX 1: PROCESS QUESTIONNAIRE .............................................................................. 14
APPENDIX 2: OUTCOME PERFORMANCE QUESTIONNAIRE .................................................. 17
APPENDIX 3: INSTRUMENT USABILITY QUESTIONNAIRE ...................................................... 18
CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS
Page 4
1. INTRODUCTION
There are a large number of computer based systems that aim to support integrated land use and transport planning; more than 100 in the Netherlands alone (Al 2005). These so-‐called Planning Support Systems (PSS) have been developed since the 1970s and are still continuously improved. In a recent survey among land use and transport planners in the Netherlands we found that, just as in almost all planning fields, these instrument fail to support an important phase of planning where land use and transport should come together; the strategic planning phase (Te Brömmelstroet 2010). Figure 1 lists the reasons that were found to block the widespread use of these tools in daily practice of integrated land use and transport strategy making. Figure 1 Bottlenecks for PSS in integrated land use and transport strategy making (% of respondents (124)) that think it is a (highly) problematic.
Figure 1 shows that it is mainly a set of soft characteristics that hamper widespread use of PSS. Transparency, low communication value, user friendliness and interactiveness are seen as (highly) problematic by more than half of the respondents. There is no shortage of ideas to bridge what has been coined the implementation gap. Some of these focus on improving PSS software by adding new functionality: PSS that are more integrated (i.e. What If developed by Klosterman 1999), more interactive (i.e. Urban Strategy developed by TNO 2011) or more user-‐friendly (i.e. UrbanSim developed by Waddell 2002, 2011). Others focus more on the hardware, such as Maptables (see Vonk & Ligtenberg 2009) and other visual gadgets. Then there is the process-‐oriented line that focuses on bridging the human gap between the potential end-‐users and the PSS developers with more participative, iterative PSS development structures (i.e. Lee 1973; 1994; Te Brömmelstroet & Schrijnen 2010; Vonk 2006). One of the instruments that aim to follow the strategy of improving the software is the Accessibility Atlas (Erreichbarkeits Atlas) that has recently been developed by the Technical University of Munich. Here, we report on a randomized clinical trial that was set up in 2011 to test the performance of this innovative instrument in supporting strategy making processes. To do so, I first define how the performance of a PSS can be measured (section 2). Then, I describe the methodological choices and set up of the experiment (section 3). In section 4, the results are presented and this report closes by discussing the findings and reflecting on its meaning.
CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS
Page 5
2. ASSESSING THE PERFORMANCE OF A PSS
2.1 Dual goals of PSS Recently we have proposed an integrated performance framework that allows us to structurally test if PSS, or specific improvement strategies for them, are effective in supporting strategy making processes (Te Brömmelstroet, 2012). This framework is based on the concept of dual goals of PSS. First, many PSS explicitly aim to improve planning processes, e.g. by structuring them better and/or making them more interactive, integrative and participatory. Next to that PSS aim to improve the outcomes of these processes (e.g. strategies, plans and projects), e.g. by providing relevant content (knowledge, information) and facilitating design-‐analyse loops. In strategic planning this link is particularly problematic. Following Couclelis, who addressed land use planning, we can assert that strategic planning
“is a hopelessly complex human endeavour. It involves actions taken by some to affect the use of land controlled by others, following decisions taken by third parties based on values not shared by all concerned, regarding issues no one fully comprehends, in an attempt to guide events and processes that very likely will not unfold in the time, place, and manner anticipated” (Couclelis, 2005; p. 1355).
Following this, it is often said that strategy making problems are ‘wicked’ (Rittel and Webber, 1984; Christensen, 1985); problems on which there is no consensus and for which there are no clear-‐cut answers or solutions. Pelzer (2012) refers to this problem as a double complex one: the subject of planning is complex, as is the process of planning itself. In such situations, a rigid protocol of planning steps is hardly helpful. It is the development of the capacity to deal with these problems, rather than final solutions that is the general aim of strategic planning (Healey, 2006). Planners here (should) aim to become aware and learn to cope with complexity and the “unknown unknowns” (Taleb, 2007) instead of collecting knowledge to reduce it/them.
2.2 Operationalizing PSS goals Both goals of PSS (i.e. improving process and improving outcomes) have been operationalized into a multidimensional framework. Based on academic literature on ideational output (e.g. Dean et al. 2006) the quality of a planning outcome can be rated on four dimensions; • novelty (originality, paradigm relatedness), • workability (implementability, acceptability), • relevance (applicability effectiveness), and • specificity (completeness, Implicational explicitness, clarity). For a planning outcome to be of high quality, it has to score on all these dimensions. Based on academic work on Group Model Building (Rouwette et al. 2002) we operationalized the quality of the process in nine dimensions; • reactions (enthusiasm, satisfaction, credibility), • insight (in problems, in others’ assumptions), • commitment, • behavioural change, • communication, • shared language, • consensus (on problem, goal, strategies), • cohesion and • efficiency.
CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS
Page 6
Again, this means that for a planning process to be of high quality it needs to score high on all these dimensions. Table 1 Dimensions of outcome Table 2 Dimensions of process Dimension Operationalization
1. Novelty 1a. Originality
1b. Paradigm relatedness
2. Workability 2a. Implementability
2b. Acceptability
3. Relevance 3a. Applicability
3b. Effectiveness
4. Specificity 4a. Completeness
4b. Implicational explicitness
4c. Clarity
Dimension Operationalization
5. Reaction 5a. Enthusiasm
5b. Satisfaction
5c. Credibility
6. Insight 6a. Insight in problem
6b. Insight in assumptions
7. Commitment
8. Behaviour
9. Communication
10. Shared language
11. Consensus 11a. On problem
11b. On goals
11c. On strategies
12. Cohesion
13. Efficiency gains
CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS
Page 7
3. SETUP OF THE EXPERIMENT
3.1 Intervention: The Erreichbarkeitsatlas In 2011, the Faculty for Urban form and mobility of the Technical University of Munich developed an accessibility atlas for the Munich Metropolitan region. This instrument allows users to explore the opportunities and treats for this growing region in terms of potential activities to be reached within different travel times by different modes of transport. Examples of maps that can be explored, and that can be interactively developed by the user via an online platform, are the number of people that can reach intercity train stations within 30 minutes or mapping the commuting flows compared to the accessibility quality of these connections. The Erreichbarkeitsatlas is an attempt to transferuse the potential of accessibility as a professional language for integrated land use and mobility planning issues to the realm of urban and regional planning. Accessibility concepts are increasingly acknowledged as fundamental to understand the functioning of cities and urban regions. In particular, accessibility instruments are able to provide a framework for understanding the reciprocal relationships between land use and mobility.
Figure 2 A map from the Erreichbarkeitsatlas: commuting flows vs. accessibility quality
A collection of these maps is used in the experiment in printed form. In the next section, I will describe the experiment in more detail.
CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS
Page 8
3.2 Strategy making trial with students To test the performance of this instrument in supporting strategy making a randomized controlled trial with students was set up. A total of 34 voluntary students took part in the trial, from the Master Environmental Engineering and from the Master Transport Engineering. These students were randomly divided in a control group (that received only a set of empty maps of the region) and a treatment group (that received the maps of the Erreichbarkeitsatlas, introduction to these maps and support by one of the developers). These groups were then again divided in random groups of three students that then engaged in a strategy making process (of 60 minutes) for the Munich Metropolitan region. The planning problem that all groups received on paper is presented in the box below. In the near future, the Europäische Metropolregion München is expecting a strong growth in both economic and demographic terms. In the coming 20 years, the region faces considerable spatial and mobility challenges: within the region, a total of 60.000 new working places (offices and industry), 60.000 new houses and supporting leisure and shopping areas have to be allocated. The region aspires that these new developments are primarily located in places with high public transport accessibility. There, sustainable mobility with limited negative impacts can be guaranteed. You will form a strategic planning team together with two colleagues. This team is asked to develop a rough spatial strategy for this allocation challenge. You will have a map of the region and an empty sheet to your disposal on which you can formulate a strategy in text and in geographical drawings (please remember to include a legend). In total you will have a maximum of 60 minutes to perform this. In this competition you are asked to find good locations (on the level of municipalities) and use the complete region as an “empty sheet” where all locations are potentially possible. Don’t worry about the size, form of these locations. Your strategy will be judged by external experts on the quality of the final result. This quality will be judged in terms of: • Novelty; • workability; • relevance and; • specificity.
3.3 Testing the performance After 60 minutes all groups had to hand in their strategy. For this, they could use a map with an attached empty sheet to write down their ideas and argumentation. These strategies were then our input for the performance test. This test was done in three steps. First, all participants were asked to fill in a questionnaire related to the quality of the planning process. This questionnaire consisted of 32 statements that each related to the dimensions of table 1. For each of these statements, the participant was asked to rate his/her agreement on a 7-‐point Likert scale. This questionnaire is attached in appendix 1. The outcomes of these ratings were grouped and this allowed us to compared the process dimensions of the control group with these of the treatment group. The resulting strategies were rated on their outcome dimensions as listed in table 2. This was extended with one statement on the amount of accessibility logic followed. Again, the same method of (28) 7-‐point Likert scale statements was used. The questionnaire is listed in appendix 2.
CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS
Page 9
Each strategy was rated by two external raters. For this, two PhD students that are specialized in integrated land use and transport from the University of Amsterdam were asked. The third and last measurement questionnaire was focused on the perceived quality of the Erreichbarkeitsatlas. For that, again statements were offered, but now only to the participants of the treatment group. They were asked to respond to several dimensions of the instrument that are generally related to the usability of such instruments (see e.g. figure 1). This questionnaire is attached in appendix 3.
CESAR Working Document Series no. 2 Erreichbarkeitsatlas as PSS
Page 10
4. RESULTS OF THE EXPERIMENT
4.1 Effects on the planning process
The average scores on the planning process dimensions are listed in table 3. The table distinguishes the groups that did not receive planning support from the Erreichbarkeitsatlas from the groups that did. On the bottom, the differences between the two are presented with the level of significance (deviation from the zero hypothesis that there is no effect). Although both groups score above average (4) on all dimensions, all dimensions but cohesion (no effect) show a negative effect of the support of the Erreichbarkeitsatlas. These negative effects are significant on a 0.05 level for the dimensions (bold for grouped dimensions) satisfaction, reaction, insight in the problem, insight in assumptions, insight, communication, shared language, consensus on problem, consensus on goals, consensus and efficiency.
4.2 Effects on the planning outcome
The average scores on the planning outcome dimensions are listed in table 4. The table is set up similar to table 3. Here, most dimensions score below or just above average (4). Statements on two of them (paradigm relatedness and Implementability) were not filled in by one of the external raters due to a lack of insight in them. These should therefore not be considered as valuable scores. The effects of the Erreichbarkeitsatlas are more mixed here. Small positive scores are found for bold for grouped dimensions) accessibility logic (0.39), applicability (0.19), completeness and specificity (both 0.13). None of these effects are significant. The other dimensions all score negatively, but also not significantly so.
4.3 Perceived quality of the instrument
The usability characteristics of the Erreichbarkeitsatlas are very positively rated by the participants that received support from it, see table 5. Especially, the participants found the instrument easy to understand (5.32). They rated the ability of the instrument to support creating (5.22) and evaluating (5.17) strategic ideas as very positive as well. Important factors as transparency and user-‐friendliness also received a score over 5 (on a scale from 1to 7). The lowest score is for the understandability of the indicators used (still 4.58).