WP8 – Impact Assessment & Simulations · 2017-04-25 · WP8 – Impact Assessment & Simulations ....

FP7-SMARTCITIES-2013

STREETLIFE Steering towards Green and Perceptive Mobility of the Future

WP8 – Impact Assessment & Simulations

D8.1.1 – Evaluation Plan (initial)

Due date: 30.09.2014 Delivery Date: 02.10.2014

Author(s): Philipp Gilka (DLR), Davide Frigeri (CAIRE), Alberto Merigo (CAIRE), Francesco Avesani (CAIRE), Mika Vuorio (CGI)

Partner(s): CAIRE, CGI, DLR

Editor: Philipp Gilka (DLR) Lead Beneficiary of Deliverable: DLR

Dissemination level: Public Nature of the Deliverable: Report

Internal Reviewers: Vesa Meskanen (CGI), Cristian Roverato (City of Rovereto)

Ref. Ares(2014)3660293 - 04/11/2014

FP7 - 608991 - STREETLIFE D8.1.1 – Evaluation Plan (initial)

WP8 – Evaluation Plan (initial) STREETLIFE Consortium Page 2 of 93



EXECUTIVE SUMMARY

The ‘Evaluation Plan’ aims for preparing and structuring the whole evaluation process of the STREETLIFE project to successfully run the pilots at the three sites. Since all sites have a different focus and differ in their research objectives a methodology had to be established which covers these prerequisites. Additionally, the methodology had to take into account the overall project objectives as stated in the STREETLIFE Description of Work.

As a starting point a literature review of existing evaluation frameworks or projects with similar contexts has been conducted. Based on European research projects, such as FESTA, Amitran and Quartet Plus as well as the CIVITAS, evaluation methodologies and evaluation frameworks have been considered as basis for the STREETLIFE evaluation plan.

In close cooperation with the pilot evaluation manager the pre-conditions of the pilot sites were then considered to elaborate and define the pilot research objectives. Based on these objectives, site-specific pre-conditions of the planned evaluation were defined by setting up research questions, hypotheses and success criteria. Aiming for answering these research questions, relevant indicators (performance indicators) have been selected and were defined in detail. In order to acquire and analyse the required data methods and tools have been described. All that has been concluded in site specific evaluation matrixes to support the local evaluation manager during the whole process chain of evaluation.

At the end use DISCLAIMER below:

Disclaimer: This project has received funding from the European Union’s Seventh Framework Programme for research; technological development and demonstration under grant agreement no 608991. The information and views set out in this publication are those of the author(s) and do not necessarily reflect the official opinion of the European Communities. Neither the European Union institutions and bodies nor any person acting on their behalf may be held responsible for the use which may be made of the information contained therein. © Copyright in this document remains vested with the STREETLIFE Partners



D8.1.1 – Evaluation Plan

Table of Contents

EXECUTIVE SUMMARY ................................................................................................................................... 3

ABBREVIATIONS ............................................................................................................................................... 6

EXPLANATIONS FOR FRONTPAGE .............................................................................................................. 8

PARTNER ............................................................................................................................................................. 8 LIST OF FIGURES .............................................................................................................................................. 9

LIST OF TABLES ................................................................................................................................................ 9

1. INTRODUCTION ........................................................................................................................................... 10 1.1. INTENDED AUDIENCE.................................................................................................................................. 10 1.2. DOCUMENT STRUCTURE ............................................................................................................................. 10 1.3. CONCEPT AND DELIVERABLE OBJECTIVES .................................................................................................. 10

1.3.1. Objectives of the Impact Assessment ................................................................................................. 10 1.4. BACKGROUND ............................................................................................................................................ 11

1.4.1. Existing Evaluation Frameworks ...................................................................................................... 11 1.4.2. Conclusion ......................................................................................................................................... 23

2. EVALUATION PLAN .................................................................................................................................... 24 2.1. METHODOLOGY ......................................................................................................................................... 24 2.2. DEFINITION OF EVALUATION AND IMPACT ASSESSMENT PHASES .............................................................. 25 2.3. THEORETICAL BACKGROUND ..................................................................................................................... 27

2.3.1. Research Objectives .......................................................................................................................... 27 2.3.2. Research Questions ........................................................................................................................... 27 2.3.3. Hypothesis ......................................................................................................................................... 27 2.3.4. Indicators - Performance Indicators ................................................................................................. 28

2.4. IMPACT CATEGORIES.................................................................................................................................. 29 2.5. MEASURES AND METHODS USED ................................................................................................................ 30

2.5.1. Success Criteria ................................................................................................................................. 31 2.5.2. Evaluation Schema ............................................................................................................................ 32

2.6. USER GROUPS ............................................................................................................................................. 33 2.6.1. Sample size ........................................................................................................................................ 33

2.7. TYPES OF TEST TRIALS ................................................................................................................................ 33 2.7.1. Initial Survey – usability test ............................................................................................................. 33 2.7.2. Field Trials ........................................................................................................................................ 34

2.8. DATA ANALYSIS PROCESS .......................................................................................................................... 35 2.8.1. Synchronisation with interviews and other subjective sensors .......................................................... 37

3. DEFINITION PHASE – PILOT SITE BERLIN .......................................................................................... 38 3.1. OBJECTIVES ................................................................................................................................................ 38 3.2. RESEARCH QUESTIONS ............................................................................................................................... 38 3.3. HYPOTHESIS ............................................................................................................................................... 39 3.4. TRIALS AND SURVEYS ................................................................................................................................ 45

4. DEFINITION PHASE – PILOT SITE ROVERETO .................................................................................. 46 4.1. OBJECTIVES ................................................................................................................................................ 46 4.2. RESEARCH QUESTIONS ............................................................................................................................... 46



4.3. HYPOTHESIS ............................................................................................................................................... 48 4.4. TRIALS AND SURVEYS ................................................................................................................................ 67

5. DEFINITION PHASE – PILOT SITE TAMPERE ..................................................................................... 68 5.1. OBJECTIVES ................................................................................................................................................ 68 5.2. RESEARCH QUESTIONS ............................................................................................................................... 68 5.3. HYPOTHESIS ............................................................................................................................................... 69 5.4. TRIALS AND SURVEYS ................................................................................................................................ 83

6. CONCLUSION................................................................................................................................................ 84 APPENDIX A: IMPACT CATEGORIES AND INDICATORS .................................................................... 85

PILOT SITE SPECIFIC HYPOTHESIS .......................................................................................................... 87

PILOT SITE SPECIFIC HYPOTHESIS .......................................................................................................... 91

APPENDIX C: LITERATURE ......................................................................................................................... 93



ABBREVIATIONS

BaU Business-as-Usual

CO Confidential, only for members of the Consortium (including the Commission Services)

BER STREETLIFE Berlin-Pilot

CBA Cost-benefit-analysis

D Deliverable

DoW Description of Work

EXP Experiment

FP7 Seventh Framework Programme

FLOSS Free/Libre Open Source Software

FOT Field Operational Test

GUI Graphical User Interface

(HY) Hypothesis

IPR Intellectual Property Rights

IRTE Integrated Road Transport Environment

ITS Intelligent Transport Systems

MGT Management

MS Milestone

OS Open Source

OSS Open Source Software

O Other

P Prototype

P&R Park and Ride

PI Performance Indicators

PU Public

PM Person Month

R Report



ROV STREETLIFE Rovereto-Pilot

RQ Research Question

RTD Research and Development

TRE STREETLIFE Tampere-Pilot

UC Use Case

UI User Interface

WP Work Package

Y1 Year 1



EXPLANATIONS FOR FRONTPAGE

Author(s): Name(s) of the person(s) having generated the Foreground respectively having written the content of the report/document. In case the report is a summary of Foreground generated by other individuals, the latter have to be indicated by name and partner whose employees he/she is. List them alphabetically.

Partner(s): Name of the partner(s) whose employee(s) the author(s) are. List them alphabetically.

Editor: Only one. As formal editorial name only one main author as responsible quality manager in case of written reports: Name the person and the name of the partner whose employee the Editor is. For the avoidance of doubt, editing only does not qualify for generating Foreground; however, an individual may be an Author – if he has generated the Foreground - as well as an Editor – if he also edits the report on its own Foreground.

Lead Beneficiary of Deliverable: Only one. Identifies name of the partner that is responsible for the Deliverable according to the STREETLIFE DOW. The lead beneficiary partner should be listed on the frontpage as Authors and Partner. If not, that would require an explanation.

Internal Reviewers: These should be a minimum of two persons. They should not belong to the authors. They should be any employees of the remaining partners of the consortium, not directly involved in that deliverable, but should be competent in reviewing the content of the deliverable. Typically this review includes: Identifying typos, Identifying syntax & other grammatical errors, Altering content, Adding or deleting content.

PARTNER

Fraunhofer Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.

FBK Fondazione Bruno Kessler

SIEMENS Siemens AG

DFKI Deutsches Forschungszentrum für Künstliche Intelligenz GmbH

AALTO Aalto University

DLR Deutsches Zentrum für Luft- und Raumfahrt

CAIRE Cooperativa Architetti e Ingegneri - Urbanistica

Rovereto Comune di Rovereto

TSB Berlin Partner for Business and Technology

Tampere City of Tampere

Logica CGI Suomi Oy

VMZ VMZ Berlin Betreibergesellschaft mbH



LIST OF FIGURES

Figure 1: FESTA-V Model ...................................................................................................... 13

Figure 2: Amitran-Methodology .............................................................................................. 14

Figure 3: Three levels of assessment (QUARTET PLUS) ...................................................... 18

Figure 4: CIVITAS Evaluation Framework ............................................................................. 21

Figure 5: Evaluation and Impact Assessment Methodology .................................................... 25

Figure 6: Phases of Impact Assessment ................................................................................... 26

Figure 7: Timeline Impact assessment ..................................................................................... 27

LIST OF TABLES

Table 1: Research questions generation ................................................................................... 27

Table 2: Hypothesis generation ................................................................................................ 28

Table 3: Impact Categories ...................................................................................................... 30

Table 4: Measures and Methods used ...................................................................................... 31

Table 5: Example of the evaluation matrix .............................................................................. 32

Table 6: BER Evaluation Matrix .............................................................................................. 39

Table 7: ROVERETO PILOT SITE SPECIFIC HYPOTHESIS ............................................. 48

Table 8: ROV- Evaluation matrix ............................................................................................ 50

Table 9: TRE – Evaluation matrix ........................................................................................... 70

Table 10: Hypotheses on safety ............................................................................................... 90

Table 11: Hypotheses on user acceptance ................................................................................ 92

Table 12: Hypotheses on user compliance, for the user behaviour category ........................... 92



1. INTRODUCTION

1.1. Intended audience

The present initial ‘Evaluation Plan’ prepares and structures the whole evaluation process of the STREETLIFE project. In close cooperation with the pilot evaluation manager the pre-conditions of the pilot sites were considered. In this regard objectives of the pilots are elaborated and defined. Based on these, research questions, hypotheses and success criteria have been derived. With the aim of answering these research questions, relevant indicators (performance indicators) have been selected and were defined in detail. Methods and tools to receive the required data have been discussed. The local evaluation manager will be set into the position to have a clear understanding of the evaluation process.

1.2. Document structure

Chapter 1 provides within the ‘Introduction’ the overview of the various number of STREETLIFE scientific and technological objectives, the impact categories affected through the STREETLIFE system. Additionally, we present a literature review of existing evaluation frameworks or projects (Chapter 1.4.1), which have been adopted within similar contexts. Therefore the European projects, such as FESTA, Amitran and Quartet Plus as well as the CIVITAS, specifically with regard to their evaluation methodologies and evaluation frameworks, have been considered as basis for the STREETLIFE evaluation plan.

In chapter 2 the ‘Evaluation Plan’ is described. In this regard an evaluation and impact assessment process chain (Chapter 2.1.1) was set up to describe the different steps of the planned evaluation activities. Also, the ‘Theoretical background’ (Chapter 2.1.2) is established to guide the pilot cities by defining and deriving the pre-conditions of the planned evaluation.

Chapter 3, 4 and 5 describe the ‘Definition Phase’ of the STREETLIFE Evaluation plan. All pilots compile their site-specific pre-conditions of the planned evaluation.

Chapter 6 then concludes the initial ‘Evaluation Plan’.

1.3. Concept and deliverable objectives

1.3.1. Objectives of the Impact Assessment

STREETLIFE follows a number of scientific and technological objectives. First, there are overall STREETLIFE objectives which are addressing different topics, such as:

• Mobility data collection

• Monitoring and decision making

• End-user engagement



These objectives will be explained in more detail in Chapter 2. The core objectives STREETLIFE aims for are:

- Reducing carbon emissions through sustainable urban mobility solutions based on Information and Communication Technologies (ICT)

- Personalised multimodal mobility information on their smartphone will motivate citizens to select sustainable transport means for their travel.

Above these project level objectives, the pilot sites have a number of specific objectives. These addressing different impact categories and to draw conclusion about the effects STREETLIFE will have (see Chapter 3, 4 and 5).

The present Evaluation Plan will provide the basis for evaluation and impact assessment to be able to (answering the following):

• Demonstrate the potential impact of STREETLIFE on user behaviour, the traffic system and carbon emissions

• Examine benefit-cost ratios and implementation issues

• How can STREETLIFE sustainably change the travel behaviour of citizens into a more eco-friendly behaviour?

• Which impact does STREETLIFE have on transport efficiency of a city?

• To what extent STREETLIFE can decrease the carbon footprint of individuals and the carbon emissions within a city caused by land-based transport?

• Develop a framework and guidelines to be applied for validation and assessment of the STREETLIFE system at different pilot sites

• Identify and define common indicators for performance measurement at each individual level and the whole system;

• Describe methods for data collection and analysis as well as impact assessment

• Set up a validation plan for each test site

1.4. Background

1.4.1. Existing Evaluation Frameworks

There is not a single way of conducting successful field trial studies. The review of a selection of field trial studies and especially their methodologies reveals that many different approaches are possible and can be taken into account to plan, run and analyse field trial studies. The selection is mainly based on their relevance with regard to STREETLIFE. First, the FESTA



methodology is highly relevant as it is specifically developed to plan, conduct and analyse field operational tests (FOT) of ITS in road transport. Although STREETLFIE is not an FOT the methodology developed can be adopted to the STREETLIFE purposes. Secondly, the Amitran methodology was selected as it is a standardised methodology to evaluate ITS with regard to their effects on CO2 emissions and transport efficiency. That makes it very relevant to consider for STREETLIFE. Thirdly, the Quartet Plus approach was considered to be useful as is was a use case for a developed evaluation methodology were certain methods have been used. The evaluation activities in QUARTET PLUS focused on a “single site, single application” assessment in six cities, to a “multi-site, single application” comparative assessment. Here, certain parallels to STREETLIFE could be identified. Finally, CIVITAS was considered to bring added value to the literature review as the focus was on the nature and the extent of the impacts made by the measures introduced in different cities.

1.4.1.1. FESTA – Methodology [1]

1) What is the framework about? FESTA is defined as methodology to plan, conduct and analyse field operational tests (FOT) for ITS in road transport. Intelligent Transport systems in this regard focuses mainly on in-vehicle applications and systems dedicated to car and truck transport, which can be assistant systems such as lane keeping functionality or information systems i.e. recommended speed. In fact, the FESTA-methodology provides a very good guideline of the complex design and execution of an FOT. The guidelines provided by FESTA consider the entire process of planning and running a field trial, starting from functionality description and defining relevant hypothesis to performance indicators and experimental procedures.

2) What and how evaluation is recommended to conduct? The main objective of the FESTA methodology is to give insights in the system performance. FESTA is designed as V-model (Figure 1), starting with identifying systems and functions as well as describing its intended benefits and the specific limitations. Following these, reasonable use cases can be derived. Use cases are means to describe the boundary conditions under which a function is intended to be analysed.

Secondly, specific research questions have to be identified. Research questions are general questions to be answered by compiling and testing related specific hypotheses. The goal is to define statistically testable hypotheses and find measurable indicators to test the hypotheses.



Figure 1: FESTA-V Model

Finally, hypotheses can only be tested by means of reasonable indicators. Appropriate performance indicators (PIs) have to be chosen that allow answering the hypotheses. They can be divided into qualitative and quantitative indicators and can be derived from one or several measures. The PI selection is very close related to the budget and other limitations of the project.

Additionally, two “scaling up” steps – Impact Assessment, and socioeconomic Cost-Benefit-Analysis were proposed by the FESTA methodology.

3) How the guidelines can be used for STREETLIFE?

The methodology proposed by FESTA is very useful to consider for the STREETLIFE activities. STREETLIFE impact assessment aims for analysing and assessing the impact of ITS with regard to green transport. The formalization of necessary steps as described in the V-model can be adapted to this purpose and provide therefore benefit also for other than large scale FOT.

1.4.1.2. AMITRAN [2]

1) What is the framework about?



Amitran – (Assessment methodologies for ICT in multi-modal transport from User Behaviour to CO2 reduction), is a standardised methodology to evaluate ITS with regard to their effects on CO2 emissions and transport efficiency. Amitran aims for contributing to the development of ITS solutions that allow more efficient multi-modal transport of goods and passenger mobility.

The methodology is seen in the context of other assessment tools. The opportunities emerging from the assessment approach should be exploited not only with respect to CO2 effects, but also, for instance, regarding indicators reflecting the traffic quality. Different data needs and the particularities of different model types have to be considered. The limitations of existing models and the combination of them is therefore a major challenge. In order to make Amitran a useful and successful methodology, the relevant ITS and the transport processes leading to changes in CO2 emissions that have to be identified, described and assessed.

2) What and how evaluation is recommended to conduct?

The definition of the methodology is the general outline of the Amitran methodology including its main steps. The figure below illustrates the chain from ITS systems to CO2 emissions. A logical overview of how ITS systems can have impact on CO2 emissions has been made available. The main elements of this chain are (1) system categorisation, (2) factors and parameters influenced by ITS, (3) transport system, (4) parameters relevant for CO2 emissions and (5) scaling up.

Figure 2: Amitran-Methodology Starting from an ITS system, the system can have a direct or indirect influence on driver or traveller behaviour and on vehicle conditions. These mechanisms are distinguished into four groups:

(1) ITS system and service

The Amitran methodology focuses on Intelligent Transport Systems and Services for which the user wants to carry out a CO2 impact assessment. Amitran takes into account ITS in various fields: systems related to passengers and freight used in road, rail, and inland waterway traffic. To make sure that all relevant systems are covered,



different categories of ITS are taken as a starting point, and systems belonging to these categories are identified. Six system categories are distinguished, following the classification from the ECOSTAND project [3]:

- Navigation and Travel Information

- Traffic Management and Control

- Demand and Access Management

- Driver Behaviour and Eco-Driving

- Logistics and Fleet Management

- Safety and Emergency Systems

The classification is suitable for Amitran with respect to the factors and parameters that the systems have influence on.

(2) Factors and parameters influenced by ITS

Within the second block specification of factors and parameters influenced by ITS are made because ITS impacts CO2 emissions in transport through multiple pathways. They have an effect on the type of modelling. In the figure the chronology of decisions is given from left to right. A distinction between long term and short term pre-trip decisions is made. In addition, four levels of factors and parameters influenced by ITS can be identified:

- Parameters describing traffic demand

- Parameters describing driving behaviour and vehicle condition

- Indirect factors influencing traffic demand and driving behaviour

- Long term effects of ITS

A more extensive explanation of Figure 3 can be found in Deliverable 3.1 of the Amitran project.

(3) Transport system

The third block connects the factors and parameters that are influenced by ITS to parameters relevant for CO2 emissions. Changes in the factors and parameters influenced by ITS will in turn affect the transport system. Some ITS systems can affect more than one factor or parameter, and there can be links between.

(4) Relevant parameters for CO2 emissions

The fourth block contains the relevant parameters for determining CO2 emissions. They can be extracted from the previous blocks. From the ITS measures to be evaluated, as described in D3.1, certain aspects must be known for a complete evaluation. Intelligent Transport Systems



influence the relevant parameters for CO2 emissions of a certain trip or collection of trips. There are several parameters such as driving behaviour, vehicle kilometres, vehicle type, and weight of the vehicle and infrastructure characteristics, influencing CO2 emissions. Furthermore, some circumstances of surroundings are known to affect the outcome. These circumstances are called situational variables as described in section 2.3. A situational variable represents an aspect of the surroundings made up of distinguishable levels.

(5) Scaling up

The last step in the Amitran framework as depicted in Figure 2 is the scaling up of the CO2 emissions from local level to a higher level. As the steps before in the framework delivered results at a local scale, policy and business decisions are taken based on expected effects at a more global level, scaling up the results becomes even more important. Amitran proposes two types of scaling up. The first method of scaling up is through modelling, using a macroscopic multimodal traffic model on EU-27 level (or another level that is chosen). The second method is a direct method, using a knowledge base with statistical information. Within Amitran a knowledge base was developed by providing a central point of reference, where links to all relevant information for scale up and database can be found. This information might also be relevant for a macroscopic approach (e.g. to predict penetration rate or acceptance). The choice between scaling up through modelling or statistical information depends on issues like the availability of models and the type of effect.


The Amitran methodology aims to determine the effect of ITS on CO2 emissions, i.e. the change in CO2 emissions with regard to the baseline scenario (where no ITS is used). The accuracy of the absolute value of the baseline and future scenario are therefore of less importance than the difference between the two scenarios. For factors that will not change in the future scenario (e.g. cold versus hot engine), the inclusion is therefore of less relevance.

The Amitran framework is developed in such a way that it is open to new kinds of systems and system categories, and it can be easily extended in the future if needed.

The framework presents the first important step towards a systematic and standardised assessment methodology for ITS. It enables appraisals of a wide range of systems from specialised systems in various fields: systems related to passengers and freight used in road, rail, and inland waterway transport. Hence, Amitran does not provide a toolset or data base to enable the user to calculate the CO2 savings directly. Microscopic demand models have to be in place and transport network simulations need to be conducted. However, the WIKI prepared within Amitran (http://amitran.teamnet.ro) will guide the user through the necessary steps and data required.

http://amitran.teamnet.ro/



1.4.1.3. QUARTET PLUS


The QUARTET-PLUS project was designed to complete and follow the work started with the QUARTET project in the European Transport Telematics Program. The urban sites of Athens, Birmingham, Stuttgart and Turin were assigned the task of designing, developing and implementing the architecture and components of an Integrated Road Transport Environment (IRTE). The project demonstrated that an urban IRTE built on the principles of co-operative monitoring and co-operative control was feasible and had good potential for improving traffic and transport conditions, but in order to prove that the QUARTET approach was the right one for the assigned task was conceived the QUARTET PLUS project.

The major objective of QUARTET PLUS was to demonstrate that IRTE could provide a suitable platform for implementing transport telematics applications in a way that can produce optimal benefits for operators, travellers and also for industry.


The general goal of the evaluation activities in QUARTET PLUS was to proceed from a “single site, single application” assessment in the six cities, to a “multi-site, single application” comparative assessment. The main demonstration activities focus on three areas:

• the impact of cooperative control strategies and coordination between actors in IRTE

• the impact of multimodal information implemented using the IRTE platform

• the impact of public transport management implemented using the IRTE platform and examination of the new opportunities the platform offers for enhance public transport management

Validation was achieved first at the pilot-cities level. Then, within the assessment added views and dimensions were needed to provide a European dimension. In parallel with the trials and analyses carried out at each site, a strong coordination of the evaluation tasks was performed at the project management level to ensure that the approach to evaluation at each site was as uniform as possible.

The QUARTET PLUS Evaluation Group coordinated the activities for the comparative evaluation, provided guidelines and a common framework. QUARTET PLUS involved six cities and a great variety of applications. Therefore it was extremely important for the project to clarify the different levels of assessment:

• At local level, the lowest, the local IRTE applications aimed at providing benefits in the area of multimodal information of public transport. The applications in the six cities could be proved in their local context, but there was the need to validate these systems also on the open market. In other words, there was a need to demonstrate that these applications were transferable and comparable.



• The second level is the comparative one: the different applications were located in a matrix of commonalities which included seven categories:

1) Strategies on line (short term)

2) Strategies off-line (long term)

3) Info Internet

4) Public Transport info

5) VMS Info

6) PT Priority and Regularity

7) Electronic Wallet

The comparative assessment was performed in each category, and a set of common indicators were agreed and a common framework was drawn up for the validation tasks. This second level allowed different applications dealing with the same goal to be compared.

• The third level is the integration level. The demonstration that a set of tools was available in an open market took place at the end of the second level. With the third level, the QUARTET PLUS project wanted to show that all these systems were different pieces of a puzzle which took on their whole significance when put together in a general structure. The pieces of the puzzle could be put together as if they were present in one virtual site if those pieces were able to fulfil two prerequisites: one is that the whole concept of IRTE architecture is valid, and the other is that the IRTE concept matched with the requirements of the market.

Figure 3: Three levels of assessment (QUARTET PLUS)



Concerning the question how to evaluate in QUARTET PLUS a series of experiments were carried out to test the potential of telematics in optimizing mobility management. For comparison purposes, the travel times for public and private transport were measured along major city roads under different regulation conditions. Different technologies and different scenarios were evaluated in three of the QUARTET PLUS sites.

For evaluating the benefit for private traffic, tests were carried out with floating cars driving along selected routes ranging from 4 to 6 km length, and travel times were recorded for public transport vehicles taking into account whole service lines.

Additional field trials were run also to verify the influence of telematics on individual transport demand: a series of O/D experiments were carried out to measure all the components of the travel time for a sample of trips. Floating vehicles and passengers who had to make journeys in different parts of the city on different data were extracted at random. Comparisons were made between scenarios where the system wasn’t operating and travellers were choosing the best route by themselves using a map, and scenarios in which all the functions of the system were activated and the travellers were following the routes recommended by the system. These experiments aimed to assess the benefits due to the integrated effect of controlled management and information given to travellers.

Together with direct experiments of the effect of IRTE on traffic management and demand, in the QUARTET PLUS project took place also opinion surveys and user panels to make an indirect evaluation of the effect on travel demand and on the overall service quality.


Regarding the QUARTET PLUS evaluation framework, there are some parts that already are integrated into the evaluation framework of STREETLIFE, but there also parts that won’t appear in the STREETLIFE evaluation framework.

Surveys will be part of the STREETLIFE evaluation, especially regarding the evaluation of the user’s satisfaction and the level of functionality reached by the STREETLIFE app. The first iteration of the STREETLIFE project will focus on evaluating the pilot-level dimension, while the main aim of the second iteration will be impact assessment at project-level and EU-level, and for this second phase QUARTET PLUS will be an important reference.

1.4.1.4. CIVITAS


The CIVITAS initiative brings together cities that are introducing sustainable urban transport policy measures. Regarding the evaluation phase in all the CIVITAS projects, the main goal is understand the nature and the extent of the impacts made by the measures introduced in the cities.

In CIVITAS the evaluation task has been divided into impact evaluation and process evaluation:

• Impact evaluation includes the evaluation of a wide range of technical, social, economic and other impacts of the measures resulted from the implementation of the project by the cities



• Process evaluation is about the measurement of indicators that deal with implementation and operation of measures, including information, communication and participation

Impact evaluation deals with understanding the practical/technical effects of measures within the city while process evaluation is concerned with understanding more clearly the reason behind success or failure of the measures implemented.

The evaluation within the CIVITAS projects runs on 4 different levels: single measure, package level (groups of measures that complement each other or that regard the same type of indicator); city level and CIVITAS level. The objectives and the scope of the evaluation at each level are direct and inter-related at the same time. European cross-site evaluation is a distinct exercise as compared to the evaluation undertaken by the individual projects and cities. The EC needs to evaluate the results of the measures implemented by the CIVITAS cities. That provides insights on how the measures perform. The comparison between different cities and projects provide knowledge on the effectiveness of specific measures and packages of measures and help in this regard, to identify good practices and transferability.

The evaluation framework considers all the elements and forms a basis for a common understanding of the objectives of the evaluation for all the groups involved.

There are some assumptions that need to be fulfilled about methodologies used for evaluation in each CIVITAS site:

• The general approach for evaluation must be consistent across the CIVITAS cities

• The indicators used for measuring the impacts must be consistent across the CIVITAS cities. However, this does not prevent cities from having their own local indicators for assessment at the local level

• The methods of measurement must be consistent across the CIVITAS cities or at least produce comparable results.

• Transferability of the results must be assessed in order to draw conclusions at the European level



Figure 4: CIVITAS Evaluation Framework


Impact evaluation describes quantifiable effects of a measure, based on measurable indicators collected before, during and after the measure’s implementation. Impact evaluation is an assessment or estimate of the impacts or effects of a measure, i.e. concerning safety, environmental conditions or transport efficiency on the particular target groups that are affected. For this purpose, relevant indicators need to be determined which describe important characteristics of the situation and which can be measured or estimated both, before and after the implementation of the measure. An appropriate comparison can be made of any changes or with any alternatives.

In the case of impact evaluation, the choice about which impacts were going to be included into the evaluation process was taken considering all the measures and the objectives through these questions:

• What impacts does the CIVITAS measure have?

• Do the impacts have influence on achieving the CIVITAS objectives?

• Are the impacts direct or indirect?



• Will it be feasible to measure the impacts within the lifetime of the CIVITAS evaluation process?

For cross-site evaluation it was necessary to define a set of common indicators and to determine those indicators in a comparable way at different sites. Two basic requirements are used to select the proper indicators: they have to identify clearly the related performance and they have to be capable of reliable assessment using the measurement methods chosen.

In the process evaluation task, the main concept was to analyse the implementation process activities for CIVITAS measures within individual cities and projects. In order to achieve this goal in the project, an electronic database was implemented in order to collect developments of the implementation process. The main goal of the procedure was to develop new findings about factors of success and strategies to overcome possible obstacles by analysing all relevant information. A questionnaire was developed to collect data with questions about the objectives of the measure, description of its implementation, evaluation of the performance of the implementation process and lessons learned. The data were collected in 3 waves: at the beginning, in the middle and in the end of CIVITAS.

The methodical approach is the same both for impact evaluation and for process evaluation, and is based on “before –and-after” comparisons. The Baseline, Business-as-usual and After- situations provide a common structure to conduct surveys and other measurements needed to provide such consistent comparisons.

Baseline surveys are necessary to enable subsequent changes resulting from CIVITAS measures to be determined by the cities and have to be carried out prior to the introduction of CIVITAS measures. The baseline measurements should be trustworthy enough to achieve two goals: judge statistically expected changes in an accurate way and have data necessary for the impact predictions of the business-as-usual scenarios.

The Business-as-Usual (BaU) scenario is used to predict what would have happened at the end of the project if the CIVITAS measures were not introduced. The objective of the BaU is determining the impacts of the measures between scenarios with and without the measures. Possible ways to estimate the business-as-usual situation include forecasting from historical data, modelling, or monitoring a site with the similar features where the measures haven’t been applied. At the end of the project, it may be necessary to update the BaU scenarios because some of the factors involved may have changed during the course of the project.

The After situation provides a final set of measurements for evaluation which can be compared with baseline and BaU situations to assess the effectiveness of the measures implemented.


The CIVITAS guidelines can be useful for the STREETLIFE project in many ways. The first idea that the STREETLIFE consortium considers is the separation between impact evaluation and process evaluation. Probably in STREETLIFE the separation is not as easy and clear as it is in CIVITAS, but the concept of having a clear separation of indicators connected to the mere execution of pilot and indicators about the impact of the project on sustainable mobility and CO2 emissions is something we have to ponder about.



The evaluation method in CIVITAS is similar to the one we’re using in the STREETLIFE project. We have already planned to measure the baseline for our indicators. The Business as Usual scenarios are very interesting, but their implementation in the STREETLIFE project is related to the measurability of these types of scenarios, and at the moment it has to be checked and controlled if we have enough data to run the modelling required having those type of scenarios.

1.4.2. Conclusion

The methodologies presented in this section are intended to provide a formalized and practical framework rather than a cook book. The methodologies will necessarily have to be adapted to the specific case and questions set up in STREETLIFE in order to answer the research questions and to increase the efficiency of the approach.



2. EVALUATION PLAN

2.1. Methodology

Based on the analysis of existing evaluation frameworks the STREETLIFE evaluation and impact assessment methodology [Figure 5] is mainly based on the core scientific and technological objectives of the project (see 1.3) which are the following: Mobility data collection Obj-1 Deliver efficient real-time techniques for the collection, integration, correlation, and analysis of multi-provider and multi-technology mobility-related data sources exploiting future internet and cloud computing technologies. Obj-2 Deliver advanced techniques for collecting real-time traffic information from crowdsourcing and floating data. Monitoring and decision making Obj-3 Deliver advanced techniques and tools for real-time monitoring and analysis of mobility resources and carbon emissions. Obj-4 Deliver proactive techniques and tools supporting the simulation and adaptation of mobility resources and policies. End-user engagement Obj-5 Deliver techniques and tools for multi-modal, accurate, effective, and personalized route planning and real-time travel assistance. Obj-6 Deliver techniques and tools to drive the participation and engagement of end-users and to promote and encourage green behaviours. System scalability, security, and reuse Obj-7 Define the STREETLIFE global and pilot-specific architecture, taking into account the end-users requirements, the system extensibility, and security aspects. System evaluation Obj-8 Customize, deploy, operate and evaluate the proposed Urban Mobility System on three city pilots. Obj-9 Deliver results on impact assessment of the proposed solutions, in terms of traffic, end-users behaviour, and reduction of carbon emissions, and derive useful guidelines on mobility strategies for smart cities of the future. A common blue-print architecture [D2.2.1 – STREETLIFE Blueprint Architecture, Security Architecture, and site-specific architectures (initial)] is developed to define the software and service for the STREETLIFE project. This architecture is generic enough to be used for different cities and shall reduce the risk of fragmentation. Each city pilot possesses different needs, ambitions and goals but also different infrastructures and legacy systems. Moreover, pilot site specific research objectives have been established to focus specifically on the local needs. To support the research activities, hypotheses are defined. Based on the objectives, different system functionalities are developed and related uses cases created. The role of the use cases is to describe situations where STREETLIFE would bring additional benefit to the user. Being able to answer the defined research questions, performance



indicators (PI) are established. For the different pilots two iterations are planned for testing. During the test trials, several measures will be realised to collect relevant data. Within the data analysis, performance indicators will be derived by mathematical calculation out of these data. A statistical evaluation of the derived performance indicator will provide information to the developed functionalities and the impact categories. In accordance to that, the impact assessment answers the established research question with the special focus on user behaviour, impacts on the transport system and environment. Based on these findings future mobility strategies for small, medium and large cities can be discussed.

Figure 5: Evaluation and Impact Assessment Methodology

2.2. Definition of Evaluation and Impact Assessment Phases

In order to reduce the complexity of the evaluation and impact assessment methodology a 4 phases approach was selected:



1. Definition Phase

The Definition Phase is a process that establishes the objectives of the pilot site based on the user needs, defines hypotheses, use cases and finally indicators as well as methods to measure the impact.

2. Implementation Phase

Within the Implementation phase the different trials will be conducted. Two iterations are foreseen to improve and enhance the system. Different kinds of trials allow also getting feedback from the user regarding the acceptance, compliance and usefulness. Additionally, further user requirements can be acquired and methods can be tested.

3. Consolidation Phase

The Consolidation Phase checks and analyses the available data. A comparison of expectations and experiences and/or with-and without will be conducted. In addition a cross-sectional analysis provides additional benefit.

4. Assessment & Proof of Concept

The Assessment and Proof of Concept Phase will answer the research question and provide findings regarding the project results. A cost-benefit-analysis (CBA) then provides a cost-benefit-ratio. A comparison of expectations and experiences and/or with-and without will be conducted. In addition a cross-sectional analysis provides additional benefit.

Figure 6: Phases of Impact Assessment

According to the time schedule of the two test trial iterations as planned in WP6 a timeline for the different phases of Impact assessment is shown below:

Definition Phase ConsolidationPhase

Assessment & Proof of Concept

ImplementationPhase

Identification / Definition of Research questions

Define Hypotheses

Define Indicators

Define Success Criteria

Define Use Cases

Answer Research Questions

Findings

Initial Survey

CBA

User Requirements

Pre-HMI Analysis

Test Hypotheses

Data check and analysis

Comparison of expectations and experiences / with-and-without

Cross-Sectional analysis

Define situational variables



Figure 7: Timeline Impact assessment

2.3. Theoretical Background

2.3.1. Research Objectives

The set up and definition of objectives is a major task in preparation of the envisaged research. It must be clear why to put effort and resources in this work. Based on the set objectives research questions shall be derived.

2.3.2. Research Questions

A research question (RQ) is a statement of what the researcher wants to discover or prove. Table 1: Research questions generation Best practices • Should be not too broad or not too narrow. In this case

each RQ will be addressed to expected impacts and inefficiencies

• A guiding syntax could be: Will <application or system> that gives <feature> lead to <effect> on <something or somebody>?, where: − <application or system>: any object of development in

the project. − <feature>: specific characteristic of application/system − <effect>: In this kind of projects they are normally

formulated as positive effects and related to main objective categories (environment, mobility, etc.).

− <something or somebody>: the target of the benefit

Example Does the use of the STREETLIFE system reduce CO2 emissions in urban environments?

2.3.3. Hypothesis

The hypothesis (HY) is formulated as an answer to a RQ and contains a specific and testable prediction (which can be confirmed or rejected) about the relationship between two variables.

1st Iteration 2nd Iteration

EvaluationEvaluation

M16 - 18M1-12 M13 - 16

Definition Phase ImplementationPhase

ConsolidationPhase

M30 - 36M12-24 M24 - 30

Definition Phase ImplementationPhase

ConsolidationPhase



Table 2 below represents the approach of hypothesis generation. This approach was used to define a selection of hypotheses affecting the different impact categories. They can be consulted in Appendix B: Hypothesis

Table 2: Hypothesis generation

Best practices • A well thought out and focused RQ leads directly to a (number of) HY.

• For approving the hypothesis a related performance indicator is needed. Thus it is recommended to define the performance indicator(s) directly when defining the hypotheses.

• The identified indicators should be feasible to obtain (in terms of what to measure, at reasonable costs, etc...).

• The hypothesis is tested with a success criterion (SC), through which the validation of the hypotheses is examined. Preferably, the success criterion is expressed in a quantitative way (e.g. a certain percentage change)

Example STREETLIFE leads to a shift in mode choice from car to cycle on routes up to 6 km by Z%.

2.3.4. Indicators - Performance Indicators

The overall aim of performance indicators (PI) is to measure - under different impact categories - whether the application/system success criteria are achieved and to be able to answer the defined hypothesis. In the context of the STREETLIFE objectives a list of indicators has been established (see Appendix) which cover the different impact categories. There are quantitative or qualitative measurements expressed as a percentage, index, rate or other values and can be obtained directly from measures or derived, from in car data or from external sensors or questionnaires.

Based on this list the pilots have selected a number of indicators which cover their objectives best. Another criterion for selection was also the ability to measure the relevant data.



2.4. Impact Categories

The scientific and technical objectives and in addition the pilot site specific objectives affect different impact categories. STREETLIFE will have effects on the category environment, mobility and user behaviour. A short description of these categories is presented in the following.

Environment

The impact category environment addresses in terms of transportation: noise, energy consumption and exhaust emissions. Exhaust emissions include many different substances like: CO2, NOx, PM, HC, CO, SO2 and others. A selection of environmental criteria generally dependant on the location with its sensitivity to which they apply and the focus of the implemented system. In STREETLIFE, the focus is on potential savings of CO2 emissions.

User behaviour

The impact category user behaviour addresses the impact of the STREETLIFE system on the user by considering the sub-categories: user acceptance, user compliance and safety. The impact category user behaviour is about how the user is going to react with the recommendations of the service.

The sub-category user acceptance covers aspects such as ease of use and usefulness. Impacts on user acceptance are usually collected (measured) by questionnaires or interviews.

The sub-category user compliance describes whether the user follows the recommendations or advices and transfers them into behaviour. It refers to the percentage of occasions on which the user behaviour complies with the recommendation, incident warning or indication given by the system. Different user preferences and configuration options of the user interface (UI) shall be considered.

The sub-category safety addresses safety aspects although the STREETLIFE system generally does not focus on safety improvements. Only the Berlin pilot considers bike safety as relevant for the envisaged scenarios. Effects on traffic safety can be in the area of STREETLIFE only be derived from questionnaire and interviews. Safety related information coming from the bicycle or STREETLIFE tracking system which is based on sensors (hard braking or steering) will probably not be included in the analysis, since the data quality is not sufficient.

Mobility

The impact category mobility addresses the efficient movement of people and goods. There are two levels to be considered: network and individual. The individual level deals with personal and goods mobility (on the user level), i.e. travel time, delay, number of stops or speed. Mobility seen from the network perspective studies the impacts on the network level, considering indicators such as traffic density (#vehicle/km), traffic volume (#vehicle/h) and total route/network journey time and delay. These criteria often are different for diverse parts of the road network. In STREETLIFE both levels will be considered with different focus.



Table 3: Impact Categories

Impact Category Description

Environment

Environment in transportation covers impacts related to noise, energy consumption (fuel and electric) and exhaust emissions like: HC, CO, NOx, PM, CO2, CH4, NMHC, Pb, SO2, N2O and NH3.

STREETLIFE focuses on potential CO2 savings.

User behaviour User behaviour in transportation refers to the impact of ICT applications and systems in terms of operation and using it. That can be mode choice, user acceptance, user compliance and safety.

STREETLIFE focuses on:

User acceptance: Covering aspects such as ease of use (degree to which a user expects the system to be free of effort to use) and usefulness.

Compliance: describing whether the user transfers the recommendations or advices into behaviour or not, i.e. if he/she has followed a certain advice given by the system. Safety: Is an aspect that needs to be considered as there might be an impact of the STREETLIFE application on safety, primary while intended or secondary due to distraction.

Mobility Mobility in transportation focuses on the efficient movement of people and considers two levels: network and individual. The individual level deals with personal mobility (on the user level), i.e. travel time, number of mode changes, etc. From the network perspective the focus is on the impacts on the network level i.e. traffic volume and total network journey time and delay.

STREETLIFE focuses on both: the individual and network level.

2.5. Measures and Methods used

There are different types of measures and methods that can be acquired during field trials to derive performance indicators. Different data types are needed in order to support the hypotheses validation, these can be acquired by:



Table 4: Measures and Methods used

Type of data Description

User background data:

Data obtained from interviews or questionnaires, concerning the user. The selection of participants and test users for the sites is dependent on the selected indicators and the hypothesis underlying the tests.

Depending upon the research question, there is often a need to select a particular group of participants and ensure that this group is, in some way, representative of those users who will, at the end, interact with the STREETLIFE system (professional users, human factors experts or even potential customers).

Subjective data / Self-reported measures:

Several performance indicators are based on self-reported measures, which can be derived from interviews, focus-group discussions or questionnaires. But also other methods possible were the user or participant feedback provides additional value. These measures are typically not logged continuously, but rather only once or a few times during the course of one study.

Direct measure / real time data

A direct measure is a logged file directly from a sensor and stored without any processing. Data provided can be either by the application under test or the acquisition system. Online data coming from the users and systems to be evaluated. This type of observation must not be intrusive, in order not to affect natural behaviour of users. Real-time public transport or In-vehicle data – coming from vehicle sharing systems - can also be considered under this type

Indirect measures A pre-processed measure is not directly logged from a sensor, but either a variable that has been filtered, for example, or which is a combination of two or several direct or other pre-processed measures.

Situational/Control Variables

The calculation of performance indicators has to consider external influences which are described in the situational variables, e.g. weather conditions or daytime. Besides that, control factors or parameters have to be controlled during testing the hypothesis to ensure that these factors do not disturb the comparability.

A more detailed description can be found in FESTA-methodology (FESTA: 42).

2.5.1. Success Criteria

Success criteria establish the thresholds for performance indicators. That can be done i.e. on the basis of a statistical analysis of baseline and treatment data.



2.5.2. Evaluation Schema

Once, all relevant information in terms of pilot specific research questions, hypothesis and performance indicators are summarized an overview table can be compiled. A first draft overview table of the relevant impact categories and indicators as well as hypotheses can be consulted within the Appendix A and B. By considering the relevant use cases as described in D6.1 Specification of city pilots for the first STREETLIFE operation and evaluation the compiled information provide a comprehensive overview of information within one table. For each RQ different tables shall be provided.

Table 5: Example of the evaluation matrix

RQ-Pilot site-1

Research Question i.e.: Is there a significant change in the mode choice?

Hypothesis Category One of the validation categories: i.e. Mobility

Hypothesis HY-205

Description Streetlife significantly leads to a change in mode choice.

Indicators No.

PI-101

PI-102

Performance Indicator

# Trips per mode,

Success Criteria

Criteria defining when the hypothesis can be confirmed or when it has to be rejected.

i.e.: T-Test (H1>H0)

Situational Variable

Weather Time of day

Use Case

(covered with this hypothesis)

i.e.: BER-PTP-1, BER-PTP-3, BER-CPI-4, BER-BUI-2

Method Questionnaire Tracking



2.6. User groups

The research questions set up require often a selected group of participants to consider for the envisage trials. There is a close correlation to the target group that will be use or interact with the system while market introduction. It is not always a prerequisite to have a representative group of the basic population. While selecting the target group different variables have to be considered for inclusion: • Demographics variables, such as age, gender, social economic variables, • A mobility behaviour which is in align with the envisaged application and system to

be tested • A certain technology experience with various systems • Personality and attitudes. 2.6.1. Sample size

Choosing the right sample size of participants is a matter of expense as well as of discretion. First of all, the core objective of each study should be to assess the functionality and impacts of the application or ICT system on the different target fields i.e. driver behaviour, traffic safety, environment. If the chosen sample size is too small it is difficult to statistically prove effects of the system. Within very large sample sizes the chance of finding an effect increases but in parallel organisational efforts and therefore costs increases as well. The appropriate sample size depends on a number of questions that have to be answered i.e.: How long is the trial period? How often the participants will/should use the ICT system and functionality? How many valuable data sets can be gathered to realize a sound analysis? Is a similar control group foreseen for comparison purpose? From the statistical perspective a minimum number of 30 participants shall result in sound results.

2.7. Types of test trials

According to the STREETLIFE Description of Work (DoW p.34), city pilots are planned in two iterations in order to enable a virtuous loop of development and evaluation. In addition, all pilot cities are dealing with different objectives and different status of mobility-app usage. All that leads to different approaches of trial executions. Therefore the following sections provide an overview of test trials to be considered at each pilot city. Hence, not every city will conduct all of them but select these trials which meet their purpose best.

2.7.1. Initial Survey – usability test

A usability test should be considered if new apps are being developed or if the user interface of an existing app has significantly been changed. The intuitiveness of the user interface has a strong influence on the acceptance of a product. Therefore STREETLIFE applications should have the best possible usability, to reduce the negative influence of a poor interface on service acceptance.



The purpose of the initial survey is to reveal usability problems of the interface in an early phase of interface development. Thus many usability issues can be fixed before the actual testing begins. Additionally information about expected but missing features and further user requirements can be gathered. The gathered information can deal as important input for the development activities within the 2nd iteration of the project.

The approaches for the initial usability test should be executable in early stages of system development, if e.g. only a mock-up or a first prototype is available. Further efforts in time and monetary costs should be kept low. We therefore propose a suitable combination of the following methods:

• Cognitive Walkthrough: a specific scenario within the application is defined and inspected by usability experts and experts of the mobility and human-computer interaction domain [5].

• Heuristic Evaluation: usability experts analyse specific elements of an application under consideration of specific heuristics [6].

• Interview / focus group: a moderator can conduct interviews with single users or focus groups. The questions are developed in advance. Interviews can be combined with other usability methods [7].

• Questionnaires: questionnaires are easy to evaluate and enable quantitative data analysis. For users of an interactive product the AttrakDiff questionnaire measures perceived usability and system design [8].

2.7.2. Field Trials

• Pre-Tests The purpose of the pre-test is to test the system functionality and to test the defined hypothesis as well as the user-friendliness. Another important task within the pre-test is to test all data analysis procedures to ensure appropriate data is collected. As this means a number of changes and improvements might occur, it is recommended to conduct these mainly with friendly user.

• Ex-ante Test: The purpose of the ex-ante test is to gather different kind of data before using a system. This data acquisition means to gather baseline indicator data about the user mobility patterns, their current mobility planning and the level of usefulness perceived by the user prior to usage of multi-modal travel companions. Most of the indicators will be gathered by questionnaires and focus group discussions.

• Ex-post Test The purpose of the ex-post test is to gather indicator data within the treatment phase. Different indicators will be collected and analysed in relation to the indicators on the ex-ante tests. The reactions to both indicators will give useful information for system benefit. A large scale field trial could bring the required statistical relevant number of indicators which allow a



comparison of changes in the user mobility patterns, the level of usefulness perceived by the user and made experiences as well as compliance of the systems recommendations. 2.8. Data analysis process

Different steps within the data analysis chain are required, starting from data quality control to ending up with the impact assessment are required. Different techniques of data analysis and modelling exist which could be used at each presented step. The following short description of the necessary steps is based on the FESTA and STADIUM methodology [9] and adopted to the specific needs of STREETLIFE. STREETLIFE will to some extent collect and store certain user tracking data of several user, such as starting- / end-point and route requests containing information about the selected route. Therefore the data acquisition and analysis process underlies certain quality assurance steps: Step 1: Data quality assurance Since quality assurance precedes data collection, its main focus is “prevention”. Prevention is the most cost-effective activity to ensure the integrity of data collection. It is important to name and solve the problems / failures established for example in an initial survey (at least as early as possible). These failures may be demonstrated in a number of ways:

• Uncertainty about the timing, methods, and identification of person(s) responsible for reviewing data

• Partial listing of indicators to be collected • Vague description of data collection instruments to be used • Failure to identify specific content and strategies for training staff members

responsible for data collection • Obscure instructions for using, making adjustments to, and calibrating data collection

equipment (if appropriate) • Etc.

The description of all the data required to evaluate the indicators chosen for a certain measure should be exhaustive, complete and detailed as much as possible. Step 2: Data collection The primary rationale for preserving data integrity is to support the detection of errors in the data collection process, whether they are made intentionally (deliberate falsifications) or not (systematic or random errors). Each type of data to be collected (on field measurements, models outputs, surveys and results) has its preferred set of instruments. Regardless of the type of data, comprehensive documentation of the collection process before, during and after the activity is essential to preserving data integrity. Step 3: Data quality analysis Data quality analysis is aimed at making sure that data is consistent and appropriate for addressing the hypothesis of interest. Data quality analysis starts from the database and determines whether the specific analyses that were intended to perform on the data to address a specific hypothesis is feasible. Data quality analysis can be performed by following the four sub-steps reported below. The sub-steps for data quality analysis are:



1. Assessing and quantifying missing data (e.g. percentage of data actually collected compared to the potential total amount of data which was possible to collect). 2. Ensuring that data values are reasonable and units of measure are correct (e.g. a mean speed value of 6 may be unreasonable unless speed was actually recorded in m/s instead of km/h). 3. Guaranteeing that measures features satisfy the requirements for the specific data analysis. Please notice, if a failure is encountered, it should then be reported to those, responsible for the database so that the possible technical error behind can be tracked down and solved. Step 4: Data processing Data processing aims to “prepare” the data for addressing specific hypothesis which will be tested in the following steps of data analysis. Data processing includes the following sub-steps:

1. Filtering 2. Deriving new signals from the raw data 3. Event annotation 4. Reorganisation of the data according to different time scale

Data filtering can involve a simple frequency filter, e.g. a low-pass filter to eliminate noise, but also any kind of algorithm aimed at selecting specific parts of the measure. Step 5: Performance Indicators calculation Deriving the Performance Indicators can be realized by five different measures: Direct Measures, Indirect Measures, Events, Self-Reported Measures and Situational Variables. The scale of the dataset and the uncontrolled variation while using STREETLIFE becomes a seriously limiting factor unless efficient calculation methodology is implemented. The choice of which performance indicators and hypotheses to calculate is clearly dependent on the amount of effort required. Efficient calculation methods need to anticipate that performance indicators will be calculated on collected data. There is also the need to consider the fact that imperfect data can be collected. Thoughtfulness and relevant tools have to be given while analysing. The main challenge of a large pilot and its relevant database is to make the collected data manageable for ad-hoc analysis. However, the concept of storing the time history data in a single table has been found to be more efficient from the perspectives of analysis and database performance. As a working strategy a measure equals a column within a table in the database would be beneficial. In order to avoid costly join operations when performing analysis it is also to be considered to keep the database as de-normalized as storage allows. Some data in trips table, such as user ID, can be also included in the time history tables. It might be not preferred from a storage or relational database perspective, but when doing analysis it will reduce complexity of SQL queries and also save computing time. Furthermore functionalities to support data analysis will be recommended, such as:

• database query functionality (e.g. SQL and output in Excel-csv format) • fully customizable analysis and mathematical calculation of performance indicators,

and if possible, application of trigger algorithms to find events of interest (to be done for each pilot)

• grouped analysis of data (e.g. scripts) • export results to tabular format and statistical packages including boxplot diagrams.



From the evaluation perspective a general recommendation for an analysis package is to use SQL software for database queries, mathematical analysis software for computation (such as Matlab), and common statistical software packages (such as SPSS). If other similar software solutions exist, also proprietary solutions, then they might be considered as well. As SQL and some software tools may require a fairly high level of knowledge to use, it may be advantageous to develop proprietary easy-to-use graphical user interfaces. 2.8.1. Synchronisation with interviews and other subjective sensors

Being able to synchronize interviews/questionnaire with sensor data in order to verify and validate the received information, time and date have to be added. It is required for the analysing purpose to verify and validate logged data while tracking or because of route requests. For post-hoc structured comments or questionnaires on events, it is important to define a process of linking the events to absolute time.



3. DEFINITION PHASE – PILOT SITE BERLIN

The relevant performance indicators in STREETLIFE are now grouped in terms of environment, mobility and driver behaviour. The criteria are evaluated under these categories, the impact assessment results will demonstrate whether the implemented system provides benefits to the environment by reducing CO2 emissions, increases traffic efficiency and improves driver`s safety by behaviour

3.1. Objectives

The main objectives of the STREETLIFE Berlin pilot address the following topics:

- With STREETLIFE the carbon footprint of individuals and the carbon emissions within a city caused by land-based transport shall be decreased.

- With STREETLIFE the travel behaviour of citizens shall be sustainably changed into a more eco-friendly behaviour.

- With STREETLIFE an increasing safety for cyclists shall be realized.

- With STREETLIFE the transport performance of a city will not be decreased.

3.2. Research Questions

Identifier Research questions

RQ-BER-1 Is there a significant change in the mode choice?

RQ-BER-2 Which mode benefits most of the change?

RQ-BER-3 Which type of commuters is most willing to change their mobility habits?

RQ-BER-4 What are the affected trips purposes?

RQ-BER-5 Why people change their mobility behaviour?

RQ-BER-6 How do situational variables (i.e weather, day time etc) have an effect regarding the mode change?

RQ-BER-7 How STREETLIFE improves cyclist safety?



3.3. Hypothesis

This section compiles the relevant information as basis for the impact assessment. Based on the defined:

• objectives, • hypotheses, • performance indicators and • success criteria

within the pilot site, the tables below can be derived. The hypothesis categories and indicators can be consulted within the Appendix A and B. Above this, the use cases set up in D6.1 Specification of city pilots for the first STREETLIFE operation and evaluation, are set in context to the RQ, HY and PI to provide value a comprehensive information within one table. For each RQ different tables shall are provided.

Table 6: BER Evaluation Matrix

RQ-BER-1 Research Question Is there a significant change in the mode choice?

Hypothesis Category One of the validation categories: Mobility

Hypothesis HY-205 Description Streetlife significantly leads to a change in mode choice.

Indicators No.

PI-101

PI-102


# Trips per mode

Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case (covered with this hypothesis)

BER-PTP-1, BER-PTP-3, BER-CPI-4, BER-BUI-2

Method Questionnaire Focus Group Tracking



RQ-BER-2 Research Question Which mode benefits most of the change?

Hypothesis Category Mobility

Hypothesis HY-204 Description STREETLIFE does increase the use of “green” transport modes.

Indicators No.

PI-101, PI-102, PI-109, PI-111


Average modal split – user, Average modal split –

trips, Number of mode transfers, Change in trip

length

Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case



Method Tracking Route request Questionnaire Focus Group



RQ-BER-3 Research Question Which type of commuters is most willing to change their mobility habits?


Hypothesis None Description The change in modal choice by STREETLIFE is different base on the type of trip

Indicators No.

PI-101, PI-102, PI-112


Average modal split – user, Average modal

split – trips, Type of trip

Success Criteria



Weather Time of day Trip purpose


BER-PTP-1, BER-PTP-3, BER-CPI-4, BER-BUI-2, BER-PTP-5, BER-PTP-6

Method Tracking Questionnaire (Socio economic)

Route request Focus Group



RQ-BER-4 Research Question What are the affected trips purposes?

Hypothesis Category Mobility, User Behaviour

Hypothesis HY-206- Description The change in modal choice by STREETLIFE is different base on the type of trip

Indicators No.

PI-101, PI-102, PI-111, PI-112


Average modal split – user, Average modal

split – trips, Type of trip

Success Criteria



Weather Time of day

Day of the week

Use Case



Method Tracking Questionnaire Route request



Q-BER-5 Research Question Why people change their mobility behaviour?


Hypothesis HY-201, HY-204, HY-301, HY-302, HY-303, HY-401, HY-501, HY-503, HY-601, HY-602

Description

Indicators No.

PI-202, PI-203, PI-306, PI-301, PI-303, PI-309, PI-310, PI-312, PI-402, PI-403


Total number of collected “Green leaves”

credits

Number of carbon-friendly trips

…

Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case


BER-PTP-1, BER-PTP-3, BER-CPI-4, BER-BUI-2, BER-PTP-5,

BER-PTP-6

Method Questionnaire Tracking and registration



RQ-BER-6 Research Question How do situational variables (i.e. weather, day time etc.) have an effect regarding the mode change?

Hypothesis Category User Behaviour

Hypothesis HY-304 Description Situational variables have an impact on the use of STREETLIFE

Indicators No.

PI-101

PI-102


Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case






RQ-BER-7 Research Question Does STREETLIFE improves cyclist safety?


Hypothesis HY-401

HY-402

Description

Indicators No.

PI-402

PI-403


Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case


BER-PTP-1, BER-PTP-3, BER-CPI-4, BER-BUI-3, BER-BUI-4


3.4. Trials and Surveys

Within the Berlin pilot a comprehensive trial set is envisaged. Due to the new developed and established STREETLIFE application the setting of the test trial foresees an initial survey to mainly to test the usability of the pilot app and gather detailed information about the user requirements also for the 2nd iteration. This will be conducted with about 15 friendly users in a laboratory environment.

Accordingly, a pre-test is planned to test the system functionality, the overall test setting including reliability of route request and data storage as well as testing the hypothesis. A first analysis of the acquired data will provide insights into possible changes in the trial setting and system improvements for the 2nd iteration.



4. DEFINITION PHASE – PILOT SITE ROVERETO

4.1. Objectives

The main objectives of the STREETLIFE Rovereto pilot address the following topics:

– CO2 reduction by modal split change towards “greener” modes (project-level)

– Improve the city transport quality and efficiency (project-level)

– Improve the usage of available parking lots (pilot-level)

– Improve Bike Sharing service quality and efficiency (pilot-level)

– Boost Car Pooling (pilot-level)

– Become a useful instrument in the design of new mobility policies (project level)

– Improve the level of information about mobility to the mobility manager (project-level)

– Get people involved using devices that improve their knowledge base on the mobility situation (project-level)

– Create an incentive system that attracts users involved (project-level)


Identifier Research questions Hypothesis

RQ – ROV1 Is there a significant change in the mode choice? HY – 204; HY - 203

RQ – ROV2 Which mode benefits most of the change? HY – ROV3; HY – ROV16; HY – ROV8

RQ – ROV3 Why do people change their mobility behaviour? HY - 503

RQ – ROV4 Which type of commuters is most willing to change their mobility habits?

HY – ROV1; HY – ROV2

RQ – ROV5 How does an increased amount of information and specification brought by STREETLIFE can improve car-pooling utilization?

HY – ROV8; HY – ROV9; HY – ROV10

RQ – ROV6 How can STREETLIFE improve the utilisation rate HY – ROV5; HY – ROV19



of Bike sharing service on the End User’s side?

RQ – ROV7 If there’s a change in the mode choice, what impact does it have on CO2 emissions?

HY – 102; HY – 104; HY-106

RQ – ROV8 Is there a change in the utilisation rate of parking slots available?


RQ – ROV9 Can the STREETLIFE control panel support with data analysis?

HY – ROV11; HY – ROV12; HY - 402

RQ – ROV10

Can the STREETLIFE control panel support the design of new mobility policies?


RQ – ROV11

Can the STREETLIFE control panel support the management of services?

HY – ROV12; HY – ROV13; HY - 402

RQ – ROV12

Can STREETLIFE use the data contained in users’ profiles efficiently?

HY – 402; HY – ROV9

RQ – ROV13

What is the level of satisfaction from the STREETLIFE users?

HY - 502

RQ – ROV14

What is the level of acceptance from the STREETLIFE users?

HY - 501

RQ – ROV15

What is the level of compliance to the suggested routes from the STREETLIFE users?

HY - 602

RQ – ROV16

Will STREETLIFE reduce time spent in the traffic? HY – 201; HY- 202

RQ – ROV17

If there’s a change in the mode choice, what impact does it have on fuel consumption?

HY – 101; HY – 103; HY-105



4.3. Hypothesis

This section compiles the relevant information as basis for the impact assessment. Based on the defined:

• objectives,

• hypotheses,

• performance indicators and

• success criteria

within the pilot site, the tables below can be derived. The hypothesis categories and indicators can be consulted within the Appendix A and B. Above this, the use cases set up in D6.1 Specification of city pilots for the first STREETLIFE operation and evaluation, are set in context to the RQ, HY and PI to provide value a comprehensive information within one table. For each RQ different tables shall are provided.

Table 7: ROVERETO PILOT SITE SPECIFIC HYPOTHESIS

Hypothesis No: Hypothesis description Performance indicator

HY - ROV1 Commuters to schools will have more confidence using STREETLIFE

users by age, type of trip

HY – ROV2 Commuters to work will have more confidence using STREETLIFE


HY – ROV3 The utilization rate of the bike sharing system will rise

# of trips, km

HY – ROV4 The utilization rate of outer parking slots will grow

% occupancy rate

HY – ROV5 The efficiency of bike sharing system will be higher

turnover % of bikes

HY – ROV6 The mobility manager will select better spots for bike sharing stations

HY – ROV7 The mobility manager will select better spots for parking lots designed for P&R

HY – ROV8 Car-pooling trips will increase # of trips, km



HY – ROV9 Users increase personal info’s on their profiles in order to have a better car-pooling experience

HY - ROV10 Users utilize the STREETLIFE system to create carpooling groups

# of carpooling groups

HY - ROV11 The mobility manager will find the STREETLIFE system useful to analyse actual and previous data

Usefulness

HY - ROV12 The mobility manager will find the STREETLIFE system useful to have a better integrated data vision

Usefulness

HY - ROV13 The mobility manager will find the STREETLIFE system useful to control the services provided

Usefulness

HY – ROV14 The city cop will use the STREETLIFE system for P&R

use of the system

HY – ROV15 The city cop will find the STREETLIFE system useful for P&R

usefulness

HY – ROV16 The utilization rate of buses will grow # of users

HY – ROV18 Users choose trip options that include bike sharing

# of users

HY – ROV19 The utilization rate of parking slots placed in the historical city centre will decrease

% occupancy rate



Table 8: ROV- Evaluation matrix

RQ-ROV-1 Research Question Is there a significant change in the mode choice?


Hypothesis HY-204

HY-203

Description STREETLIFE does increase the use of "green " transport modes

STREETLIFE does increase the travel distances by combining different transport modes (network)

Indicators No. PI-202

PI-203

PI-204


# of carbon friendly Trips, km of carbon friendly trips, total number of collected green leaves

Success Criteria Criteria defining when the hypothesis can be confirmed or when it has to be rejected.

T-Test (H1>H0)


Weather Time of day

Use Case


ROV-BS/3; ROV-CP/3; ROV-PR/2

Method Questionnaire Tracking of bike sharing cards



RQ-ROV-2 Research Question Which mode benefits most of the change?


Hypothesis HY - ROV3

HY – ROV8

HY – ROV16

Description The change in modal choice produced by STREETLIFE is more pronounced towards some means of transportation; The utilisation rate of the bike sharing system will rise

The utilization rate of buses will grow

Car Pooling trips will increase (2nd iteration)


PI-112

PI-122

PI-119

PI-126


Bike sharing distance, turnover of bikes, bike sharing km with P&R


T-Test (H1>H0)


Weather Time of day


ROV-PR/1, ROV-PR/2, ROV-BS/3, ROV-CP/6

Method Questionnaire Registration and tracking from the app



RQ-ROV-3 Research Question Why do people change their mobility behaviour?


Hypothesis HY-503 Description The user will find the STREETLIFE system useful

Indicators No. PI – 305

PI – 309

PI – 310


Users’ perception; service reliability; awareness level



Use Case


ROV-PR/1, ROV-PR/2

Method Questionnaire



RQ-ROV-4 Research Question Which type of commuters is mostly willing to change their mobility habits?


Hypothesis HY – ROV1 HY – ROV2

Description Commuters for going to schools will have more confidence using STREETLIFE; Commuters for work will have more confidence using STREETLIFE

Indicators No. Performance Indicator



Weather Time of day

Use Case


ROV-PR/1, ROV-PR/2




RQ-ROV-5 Research Question How does an increased amount of information and specification brought by STREETLIFE improve car-pooling utilization?



HY – ROV9

HY – ROV10

Description Change in usage of car-pooling

Users increase personal information on their profiles in order to have a better car-pooling experience

Users utilize the STREETLIFE system to create carpooling groups

Indicators No. Performance Indicator

# of car- pooling groups created



Use Case


ROV-CP/1, ROV-CP/2, ROV-CP/3, ROV-CP/4, ROV-CP/5

Method To be tested in 2nd iteration



RQ-ROV-6 Research Question How can STREETLIFE improve the utilisation rate of bike sharing service on the end user’s side?


Hypothesis HY – ROV5

HY – ROV18

Description The efficiency of the bike sharing system will be higher; Users will choose trip options that include bike sharing


PI-112

PI-123

PI-305

PI-309

PI-124

PI-311


# of bikes taken; turnover of bikes; service reliability; user’s perception



Weather Time of day

Day of the week

Use Case


ROV-PR/1, ROV-PR/2, ROV-BS-1

Method Questionnaire Registration and tracking from the app



RQ-ROV-7 Research Question If there is a change in the mode choice, what impact it has on CO2 emissions?

Hypothesis Category Environment

Hypothesis HY – 102

HY – 104

HY – 106

Description STREETLIFE will reduce the carbon footprint of a user for a certain OD

STREETLIFE will reduce the CO2 emissions of all traffic, in (part of a) network.

Indicators No. PI – 201 PI – 204 PI – 205


Carbon emission km of carbon friendly trips Fuel consumption



Weather Time of day

Use Case


ROV-PR/5, ROV-PR/1, ROV-PR/2

Method Questionnaire Detection of # of trips



RQ-ROV-8 Research Question Is there a change in the utilisation rate of parking slots available?



HY – ROV19

Description The utilization rate of outer parking spots will grow

The utilization rate of parking slots placed in the historical city center will decrease


PI-125


% occupancy rate; # of trips that include P&R



Weather Time of day

Day of the week

Use Case


ROV-PR/3, ROV PR/5

Method Registration and tracking from the app § Detection of Parking slots



RQ-ROV-9 Research Question Can the STREETLIFE control panel support with data analysis?



HY – ROV12

Description The mobility manager will find the STREETLIFE system useful to analyse actual and previous data; The mobility manager will find the STREETLIFE system useful to have a better integrated data vision;


PI – 309

PI – 304


Service quality satisfaction; usefulness


T-Test (H1>H0)


Use Case


ROV-PR/4, ROV-PR/5




RQ-ROV-10 Research Question Can STREETLIFE support the design of new mobility policies?



HY – ROV7

Description The mobility manager will select better spots for bike sharing stations

The mobility manager will select better spots for parking lots designed for P+R

Indicators No.



T-Test (H1>H0)


Use Case


ROV-PR/5; ROV-CP/7, ROV-BS/3

Method To be tested in the second pilot iteration



RQ-ROV-11 Research Question Can the STREETLIFE control panel support the management of mobility services?


Hypothesis HY – ROV12;

HY – ROV13;

HY - 402

Description The mobility manager will find the STREETLIFE system useful to have a better integrated data vision; The mobility manager will find the STREETLIFE system useful to control the services provided;

STREETLIFE will warn user for critical traffic situations as i.e. accident hotspots

Indicators No. PI - 304

PI - 309

PI – 311

PI – 305

PI – 308

PI - 127


Service integration; users perception; user satisfaction; number of warnings


T-Test (H1>H0)


Use Case


ROV-PR/3, ROV-PR/4, ROV-CP/6, ROV-BS/2




RQ-ROV-12 Research Question Can STREETLIFE use the data contained in profiles be used in an efficiently?



HY – ROV9

Description STREETLIFE will warn user on critical traffic situations as i.e. accident hotspots

Users increase personal info’s on their profiles in order to have a better car-pooling experience



Number of warnings



Weather Time of day

Day of the week

Use Case


ROV-PR/3, ROV-PR/4

ROV-CP/5 (2nd iteration)


Tracking from the app



RQ-ROV-13 Research Question What is the level of satisfaction of the STREETLIFE users?


Hypothesis HY - 502 Description The user will find the STREETLIFE system easy to use


PI – 302


Ease of use; users’ perception of comfort



Use Case


ROV-PR/1, ROV-PR/2

Method questionnaire



RQ-ROV-14 Research Question What is the level of acceptance of STREETLIFE users?


Hypothesis HY - 501 Description The user will use the STREETLIFE system


PI – 316

PI – 318

PI – 317

PI – 402


User acceptance; user safety perception; # of users, # of active users



Weather Time of day

Day of the week

Use Case


ROV-PR/2, ROV-BS/1, ROV-CP/1, ROV-CP/2





RQ-ROV-15 Research Question What is the level of compliance of the suggested routes from the STREETLIFE users?


Hypothesis HY - 602 Description The user will mostly follow STREETLIFE operational recommendations


PI – 318


# of daily usage; # of active users



Weather Time of day

Day of the week

Use Case


ROV-PR/1, ROV-PR/2, ROV-BS/1, ROV-CP/3, ROV-CP/4





RQ-ROV-16 Research Question Will STREETLIFE reduce the time spent in traffic?

Hypothesis Category Mobility, Environment


HY - 202

Description STREETLIFE does not increase total travel time (individual), of a trip (origin-destination)

STREETLIFE does not increase the total travel time (network)

Indicators No. PI - 307 Performance Indicator

Journey time



Weather Time of day

Day of the week

Use Case


ROV-PR/1, ROV-PR/2, ROV-BS/1




RQ-ROV-17 Research Question If there is a change in the mode choice, what impact it has on fuel consumption?

Hypothesis Category Mobility, Environment


HY – 103

HY - 105

Description STREETLIFE will reduce the fuel consumption of a trip for a certain origin-destination.

STREETLIFE System will reduce the fuel consumption of all traffic, in (part of a) network

Indicators No. PI – 204 PI – 205


km of carbon friendly trips Fuel consumption



Use Case


ROV-PR/5, ROV-PR/1, ROV-PR/2, ROV-CP/7, ROV-CP/6, ROV-BS/1

Method questionnaire Detection of # of trips




In Rovereto two different trials are planned in the 1st iteration:

The first trial (EXP1) will last 6-8 weeks and will involve 50 citizens (commuters). The trials are divided in 3 phases:

o phase 1: use the app to support their standard journeys (define recurrent journey and receive notifications about traffic problems, delays..) and plan/save other itineraries

o phase 2: they will start receiving suggestions about alternative/green means, the real-value (cost, time, green, health) of the different alternatives (as well as their own choice) will be shown

o phase 3: incentives come into play Within the 2nd trial (EXP2) an open-field experiment involving all users of the app within the Christmas markets is foreseen. The setting of the experiment will be the same as phase 3 of EXP1 (real-time traffic info + green/health value + incentives).



5. DEFINITION PHASE – PILOT SITE TAMPERE

5.1. Objectives

The main objectives of the STREETLIFE TRE pilot address the following topics:

– CO2 reduction by modal split change towards “greener” modes (project-level)

– Improve city transport quality and efficiency (project-level)

– Improve the usage of Park & Ride (pilot-level)

– Get people involved using devices that improve their knowledge base on the mobility situation (project-level)

– Create an easy-to-use app that attracts private car users to public transportation (project-level)


Identifier Research questions

RQ-TRE-1 Is there a significant change in the mode choice?

RQ-TRE-2 Why do people change their mobility behaviour?

RQ-TRE-3 Which type of commuters is most willing to change their mobility habits?

RQ-TRE-4 How do situational variables (i.e. weather, day time etc.) have an effect regarding the mode change?

RQ-TRE-5 How an increased amount of information and specification brought by STREETLIFE can improve Park & Ride utilization?

RQ-TRE-6 If there’s a change in the mode choice, what impact does it have on CO2 emissions?

RQ-TRE-7 What is the level of satisfaction from the STREETLIFE users?

RQ-TRE-8 What is the level of acceptance from the STREETLIFE users?

RQ-TRE-9 Will STREETLIFE reduce time spent in the traffic?



RQ-TRE-10 Will STREETLIFE reduce time spent searching for a free parking lot?

RQ-TRE-11 Will STREETLIFE integration to existing IT platforms lead to increased possibilities for further exploitation?

RQ-TRE-12 Will STREETLIFE enable Traffic Manager to manage passenger flows by varying stop goodness values?

RQ-TRE-13 Will STREETLIFE mixed reality interfaces improve perception of public transportation?

RQ-TRE-14 Will STREETLIFE virtual mobility decrease the threshold for using greener means of travel?

5.3. Hypothesis

TRE PILOT SITE SPECIFIC HYPOTHESIS

Hypothesis No:

Hypothesis description Performance indicator

HY - TRE1 Private car users will find Park & Ride useful PI-310

HY – TRE2 Park & Ride utilization increases PI-312%

HY – TRE3 Unnecessary driving in the city centre will reduce

PI-101 Congestion reduction

HY – TRE4 STREETLIFE will stay in production after project has ended

PI-313

City management takes the pilot/parts of pilot into use

HY – TRE5 Decreases congestion in central traffic terminal areas

PI-303

User experience of congestion reduced

HY – TRE6 Users find bus stops and identify buses easier and more accurately than from traditional interfaces

PI-306 Identification speed and accuracy

HY – TRE7 Users can understand a presented route in more comprehensive manner than with traditional interfaces

PI-314

Preference



Table 9: TRE – Evaluation matrix

RQ-TRE-1 Research Question Is there a significant change in the mode choice?


Hypothesis HY-203 HY-204

Description STREETLIFE significantly leads to a change in mode choice.

Indicators No.

PI-101

PI-102

PI-201


#Trips, # User, Trips length, mode

Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case


TRE-02/1 , TRE-04/1




RQ-TRE-2 Research Question Why do people change their mobility behaviour?


Hypothesis HY-503 Description The change in modal choice produced by STREETLIFE is more pronounced towards some means of transportation

Indicators No.

PI-101

PI-102

PI-106


transport mode, P&R usage towards modal shift

Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case


TRE-02/1 , TRE-04/1




RQ-TRE-3 Research Question Which type of commuters is most willing to change their mobility habits?


Hypothesis HY-TRE01

Description Private car users will find Park & Ride useful

Indicators No.

PI-310 Performance Indicator

# Trips, # User, Trips length, mode

Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case


TRE-02/1 , TRE-04/1




RQ-TRE-4 Research Question How an increased amount of information and specification brought by STREETLIFE can improve Park & Ride utilization?


Hypothesis HY-TRE02

Description Park & Ride utilization increases

Indicators No.


# Trips, # User, Trips length, mode, P&R usage

Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case


TRE-04/1, TRE-02/1




RQ-TRE-5 Research Question If there’s a change in the mode choice, what impact it has on CO2 emissions?


Hypothesis HY-102 HY-104

Description STREETLIFE significantly leads to a change in mode choice.

Indicators No.

PI-101

PI-102

PI-106


# Trips, # User, Trips length, mode

Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case


TRE-02/1, TRE-04/1, TRE-05/1




RQ-TRE-6 Research Question What is the level of satisfaction from the STREETLIFE users?


Hypothesis HY-502

HY-503 HY-104

Description User approves the solution is helpful and stable

Indicators No.


User satisfaction

Success Criteria T-Test (H1>H0)

Situational Variable Weather Time of day

Use Case


TRE-02/1, TRE-04/1, TRE-05/1




RQ-TRE-7 Research Question What is the level of acceptance from the STREETLIFE users?


Hypothesis HY-501

Description User is happy about the system



User uses the system

Success Criteria T-Test (H1>H0)

Situational Variable Weather Time of day

Use Case


TRE-02/1, TRE-04/1, TRE-05/1




RQ-TRE-8 Research Question Will STREETLIFE reduce time spent in the traffic?


Hypothesis HY-212

HY-202

Description User approves the solution is helpful and stable

Indicators No.

PI-310

PI-308


User s travelling is faster, journey time reduced

Success Criteria

T-Test (H1>H0)


Weather Time of day

Use Case


TRE-02/1, TRE-04/1




RQ-TRE-9 Research Question Will STREETLIFE reduce time spent searching for a free parking slot?


Hypothesis HY-TRE03

Description Unnecessary driving in the city centre will be reduced.

Indicators No.

PI-101


Mode, P&R usage

Success Criteria


T-Test (H1>H0)


Weather Time of day

Use Case


TRE-02/1, TRE-04/1




RQ-TRE-10 Research Question Will STREETLIFE integration to existing IT platforms lead to increased possibilities for further exploitation?


Hypothesis HY-TRE4

Description STREETLIFE will stay in production after project has ended

Indicators No.


Commercial feasibility

Success Criteria


T-Test (H1>H0)


Use Case


TRE-01




RQ-TRE-11 Research Question Will STREETLIFE enable Traffic Managers to manage passenger flows by varying stop goodness values?


Hypothesis HY-TRE5

Description Decreases congestion in central traffic terminal areas

Indicators No.


Traffic management personnel finds to flow management useful

Success Criteria


T-Test (H1>H0)


Use Case


TRE-03/1




RQ-TRE-12 Research Question Will STREETLIFE mixed reality interfaces improve perception of public transportation?


Hypothesis HY-TRE6

Description Users find bus stops and identify buses easier and more accurately than from traditional interfaces

Indicators No.

PI-306

PI-310


Mobile app is useful for the users

Success Criteria


T-Test (H1>H0)


Use Case


TRE-02/1




RQ-TRE-13 Research Question Will STREETLIFE virtual mobility decrease the threshold for using greener means of travel?


Hypothesis HY-TRE7

Description Users can understand a presented route in more comprehensive manner than with traditional interfaces

Indicators No.


Commercial feasibility

Success Criteria


T-Test (H1>H0)


Use Case


TRE-02/1





In Tampere there is already a mobile application for routing available on the market. Therefore an initial survey as recommended will not be considered. Hence, there will be two types of test trials:

In the 1st iteration the large scale pilot focusses on web based real time multimodal journey planner functionality. This test trial will be conducted with friendly users. CGI Members and ITS Factory community will be the first test user group with restricted access. It is foreseen to ask them to perform tests for new functionality and other controlled experiments. After this early adopter group’s feedback and possible adjustments, the pilot is opened to all public in the TRE journey planner website (City of Tampere web site) with associated marketing. Meaning 20.000 daily users using the existing journey planner today will have a possibility to test STREETLIFE pilot.

Also for the 1st iteration field trials experiments with 3D and augmented reality apps are planned. But here, a small scale controlled user group without public access will be envisaged.



6. CONCLUSION

Existing evaluation frameworks or projects with similar contexts such as FESTA, Amitran and Quartet Plus as well as the CIVITAS have been considered to define the STREETLIFE evaluation plan. It became obvious that there is no one way of conducting a successful field trial studies. The methodologies revealed that many different approaches have been taken into account for planning, running and analysing field trials. There is no ‘Cook book’ but several important information are available to be considered. These provide a formalized and practical framework although the STREETLIFE methodology had to be necessarily adapted to the specific case and questions set up in order to answer the research questions and to increase the efficiency of the approach.

The present ‘Evaluation Plan’ structures and prepares the evaluation process to successfully run the pilots at the three sites. Although all sites focus on different research objectives the methodology covers the prerequisites by additionally taking into account the overall project objectives as stated in the STREETLIFE Description of Work.

Together with the pilot evaluation manager an evaluation plan has been developed which considers the prerequisites of the pilot sites. The pilot site-specific research objectives were then elaborated and defined by setting up research questions, hypotheses and success criteria. Being able to answer these research questions a number of relevant indicators were selected in order to acquire and analyse the required data. All that has been concluded in site specific evaluation matrixes to support the local evaluation manager during the whole process chain of evaluation.



APPENDIX A: IMPACT CATEGORIES AND INDICATORS



Service reliability PI-301

Accuracy of timekeeping/ Travel time predictability

Self reported (questionnaire item)

PI-302User’s perception of comfort


Quality of service PI-303

Service frequency

PI-304 Service integration


PI-305 Service reliabilitySelf reported (questionnaire item)

PI-306 Waiting timesSelf reported (questionnaire item)

PI-308 Service quality satisfaction


PI-309 Users' perception


PI-310 Awareness levelSelf reported (questionnaire item)

Awareness PI-311 User acceptance / satisfaction


Acceptance PI-312 Willingness to pay


PI-313 Ease of useSelf reported (questionnaire item)

PI-314

Number of shared STREETLIFE trips

Compliance PI-315Number of changed mode choices

PI-316Number of daily usage

PI-317

Number of registered STREETLIFE users

PI-318Number active users

Cycle safety PI-401Number of bike accidents

PI-402 User’s safety perception


PI-403Number of selected safe cycle routes

Calculation

User Behaviour

ACCEPTANCE

Safety

TRAFFIC SAFETY



APPENDIX B: HYPOTHESIS AND RELATED IMPACT CATEGORIES

HYPOTHESIS ON MOBILITY

The main research question STREETLIFE wants to answer in the mobility category is: Which impact does STREETLIFE have on transport efficiency of a city? What impacts have STREETLIFE solutions in a cooperative environment for the traffic system of a city/region/network (congestion avoidance, changes in travel distances and travel times)?

Hypothesis No:


HY-101 STREETLIFE does not increase total travel time (individual), of a trip (OD)

Travel time (individual),

in [seconds]

HY-102 STREETLIFE does not increase the total travel time (network).

Travel time (network),

in [seconds]

HY-103 STREETLIFE does increase the travel distances by combining different transport modes.

Travel distances of transport modes in km.

HY-104 STREETLIFE does increase the use of “green” transport modes.

Travel distances with “green” modes in km.

HY-105 Further site specific hypotheses can be added.

Pilot Site specific Hypothesis HY – TRE3

Unnecessary driving in city centre will reduce PI-101 Congestion reduction

HY - ROV1

Commuters for schools will have more confidence using STREETLIFE


HY – ROV2

Commuters for work will have more confidence using STREETLIFE


HY – ROV3

The utilization rate of the bike sharing system will rise

# of trips, km

HY – ROV4

The utilization rate of outer parking slots will grow

% occupancy rate



Hypothesis No:


HY – ROV5

The efficiency of bike sharing system will be higher

turnover % of bikes

HY – ROV6

The mobility manager will select better spots for bike sharing stations

HY – ROV7

The mobility manager will select better spots for parking lots designed for P+R

HY – ROV8

Car-pooling trips will increase # of trips, km

HY – ROV9

Users increase personal info’s on their profiles in order to have a better car-pooling experience

HY - ROV10

Users utilize the STREETLIFE system to create carpooling groups

# of carpooling groups

HY – ROV16

The utilization rate of buses will grow # of users

HY – ROV18

Users choose trip options that include bike sharing

# of users

HY – ROV19

The utilization rate of parking slots placed in the historical city centre will decrease

% occupancy rate



HYPOTHESIS ON ENVIRONMENT

The main research question STREETLIFE wants to answer in the environment category is: To what extent STREETLIFE can decrease the carbon footprint of individuals and the carbon emissions within a city caused by land-based transport?

Hypothesis No:


HY-201 STREETLIFE will reduce the fuel consumption of a trip for a certain OD.

Fuel consumption of a trip (P/Km), in l per OD.

HY-202 STREETLIFE will reduce the carbon footprint of a user for a certain OD.

CO2 emissions of a user in g per OD

HY-203 STREETLIFE will reduce the fuel consumption of the daily trip chain.

Fuel consumption of the daily trip chain in l/km

HY-204 STREETLIFE System will reduce the CO2 emissions of the daily trip chain.

CO2 emissions the daily trip chain in g/d

in g/km

HY-205 STREETLIFE System will reduce the fuel consumption of all traffic, in (part of a) network.

Fuel consumption in (part of a) network,

in l/km, for all traffic

HY-206 STREETLIFE will reduce the CO2 emissions of all traffic, in (part of a) network.

CO2 emissions in (part of a) network,

in g/km, for all traffic


HYPOTHESIS ON USER BEHAVIOUR

The main research question STREETLIFE wants to answer in the user behaviour category is: How can STREETLIFE sustainably change the travel behaviour of citizens into a more eco-friendly behaviour?



HYPOTHESIS ON SAFETY

The main research question STREETLIFE wants to answer in safety is: Which impacts have STREETLIFE solutions on bicycle and traffic safety of a city/region/network (optimization of routing, harmonizing speeds and or even motivating the user to ecological user behaviour)?

Hypothesis No:


HY-401 STREETLIFE will not provide recommendations to the user that will increase his mental workload.

DALI score,

unit less

HY-402 STREETLIFE will warn user for critical traffic situations as i.e. accident hotspots.

Number of warnings (integer)


Table 10 Hypotheses on safety

HYPOTHESIS ON USER ACCEPTANCE

The main research question STREETLIFE wants to answer in the user acceptance category is: Which impacts have STREETLIFE solutions on user acceptance leading to their successful deployment and market development (perception of product quality, maturity, satisfaction, usefulness)?

Hypothesis No:


HY-501 The user will use the STREETLIFE system Use of the system

HY-502 The user will find the STREETLIFE system easy to use

Ease of use

HY-503 The user will find the STREETLIFE system useful

Usefulness

HY-504 The user will understand the symbol messages given by the STREETLIFE system.

Understanding of message (symbol)

HY-505 The user will understand the text messages given by the STREETLIFE System.

Understanding of message (text)




Hypothesis No:


Pilot Site specific Hypothesis HY - TRE1 Private car users will find Park & Ride useful PI-310

HY – TRE2

Park & Ride utilization increases PI-312

HY – TRE3

Unnecessary driving in city center will reduce PI-101 Congestion reduction

HY – TRE4

STREETLIFE will stay in production after project has ended

PI-313

City management takes the pilot/parts of pilot into use

HY – TRE5

Decreases congestion in central traffic terminal areas

PI-303

User experience of congestion reduced

HY – TRE6

Users find bus stops and identify buses easier and more accurately than from traditional interfaces

PI-306 Identification speed and accuracy

HY – TRE7

Users can understand a presented route in more comprehensive manner than with traditional interfaces

PI-314

Preference

HY - ROV11

The mobility manager will find the STREETLIFE system useful to analyse actual and previous data

Usefulness

HY - ROV12

The mobility manager will find the STREETLIFE system useful to have a better integrated data vision

Usefulness

HY - ROV13

The mobility manager will find the STREETLIFE system useful to control the services provided

Usefulness



Hypothesis No:


HY – ROV14

The city cop will use the STREETLIFE system for P+R

use of the system

HY – ROV15

The city cop will find the STREETLIFE system useful for P+R

usefulness

HY – ROV16

The utilization rate of buses will grow # of users

HY – ROV18

Users choose trip options that include bike sharing

# of users

Table 11 Hypotheses on user acceptance

HYPOTHESIS ON USER COMPLIANCE

The following hypotheses related to user compliance are foreseen within the scope of STREETLIFE:

Hypothesis No:


HY-601 The user will mostly follow STREETLIFE strategic recommendations.

User compliance on strategic recommendations

in %.

HY-602 The user will mostly follow STREETLIFE operational recommendations.

User compliance on operational recommendations

in %.


Table 12 Hypotheses on user compliance, for the user behaviour category



APPENDIX C: LITERATURE

[1] FESTA Handbook Version 4 (2011)

[2] JONKER, E. et al. D4.1: Requirements and design of the methodology; Amitran project (2012)

[3] FRANCO, G. et al.: Comprehensive Evaluation of Urban/Regional IRTE: the User View and Suggestions for Exploitation; (1998)

[4] ROOIJEN van T., et al: Applied framework for evaluation in CIVITAS PLUS II (2013)

[5] NIELSEN, Jakob: Usability Engineering. San Francisco, California: Morgan Kaufmann Publishers, 1994

[6] BERTINI, E.; CATARCI, T.; DIX, A.; GABRIELLI, S.; KIMANI, S.; SAN- TUCCI, G.: Appropriating Heuristic Evaluation Methods for Mobile Computing. In: Handbook of Research on User Interface Design and Evaluation for Mobile Technology, IGI Global, S (2008), S. 780–801

[7 HOLZINGER, A.: Usability engineering methods for software developers. In: Communications of the ACM 48 (2005), Nr. 1, S. 71–74

[8] HASSENZAHL, M.: Hedonic, emotional and experiential perspectives on product quality. In: Ghaoui, C. (Hrsg.), Encyclopedia of Human Computer Interaction (2006), S. 266-272

[9] GILKA, P., NEUMANN, E.: Evaluation Framework (D6.1), STADIUM project (2011)

Date post:	10-Mar-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

WP8 – Impact Assessment & Simulations · 2017-04-25 · WP8 – Impact Assessment & Simulations ....

Documents