Pag. 1

EUROFOT: EUROPEAN LARGE-SCALE FIELD OPERATIONAL

TEST ON ACTIVE SAFETY SYSTEMS

Giancarlo Alessandretti1, Angelos Amditis2, Aria Etemad3, Christoph Kessler3 1. ALCOR - Italy 2. ICCS, Institute of Communications and Computer Systems - Greece 3. Ford Research & Advanced Engineering Europe - Germany

ABSTRACT The European project “euroFOT” is developing a large scale assessment program for advanced active safety systems in cars and trucks, based on Field Operational Tests with normal drivers in ordinary traffic. The experiments will provide insight into system performance and driver behaviour, as well as the socio-economic impact, particularly on road safety and traffic efficiency. This paper, after presenting the objectives of the project, outlines on-going activities and some major results for the first year. The topics include research questions, experimental design, test operation and data analysis. Coordination with parallel initiatives is also discussed for this new class of experiments. KEYWORDS ADAS, Field Operational Tests, Evaluation INTRODUCTION AND OBJECTIVES The euroFOT project aims to demonstrate the benefits and to encourage the deployment of Intelligent Vehicle Systems on European roads, focusing on Advanced Driver Assistance Systems (ADAS) which have now reached a good level of maturity. Previous experience worldwide has shown that Field Operational Tests (FOTs) are an excellent method to collect real data, evaluate the impacts and enhance the take-up of new solutions. These tests have also proved to be a powerful tool for gaining insight into the usability of functions, when operated in the real environment, and for a sufficient long time to reach the daily operational and behavioural level. At the same time it is clear that such experiments require considerable resources and efforts, and euroFOT therefore is also aiming to develop and validate a number of effective and harmonised methodologies able to facilitate the operation of FOTs and to ensure a high quality of scientific results. In this context, the main objectives of euroFOT are the following:

• To perform multiple coordinated tests of several Intelligent Vehicle Systems with ordinary drivers in real traffic;

• To investigate performance, driver behaviour, and user acceptance; • To assess the impacts on safety, efficiency, and the environment using data in a

variety of driving scenarios. EuroFOT also intends to contribute to the on-going efforts for the diffusion of advanced Vehicle Technologies and the promotion of a general European perspective. The following two additional goals are therefore particularly important: consolidating a common approach

Pag. 2

for FOTs, and raising awareness in the general public regarding the potentialities of driver support functions, as well as a creating a wider socio-economic acceptance. The nine functions under test in euroFOT are shown in the following figure: in general, they are implemented by reliable systems and sensors, now offered on series production vehicles. For the FOT, the project involves different vehicle fleets, with a total of 1390 passenger cars and 150 trucks; these fleets are coordinated by four Vehicle Management Centres, and expected to travel in different European countries. The duration of each test will be about 12 months, for a total exposure estimated as approximately 27 Millions km.

Longitudinal control functions FCW Forward Collision Warning ACC Adaptive Cruise Control SL Speed Regulation System

Lateral control functions BLIS Blind Spot Information System LDW Lane Departure Warning IW Impairment Warning

Other applications CSW Curve Speed Warning FEA Fuel Efficiency Advisor SafeHMI Safe Human Machine Interaction

Fig. 1: Set of functions under test within euroFOT

EXPECTED RESULTS By such a comprehensive approach, the project aims to obtain vehicle-based and driver-based information that could encourage strategic decisions by different stakeholders, like Manufacturers, Suppliers, Public Authorities, and User Groups. On one side, vehicle manufactures and equipment suppliers can use the results of the project to improve the system design, especially regarding the interaction with the user. On the other hand, authorities and the clients will have a deeper understanding of the potential benefits, especially on safety and efficiency: this will enable a stronger deployment of Intelligent Vehicle Technologies. A third important expected result is the increase of public awareness, gained through a more direct exposure to these new technologies and also through an articulated plan to disseminate the results. This will hopefully contribute to accelerate the market penetration of mature ICT systems. HIGHLIGHTS FOR ON-GOING ACTIVITIES System specifications A fundamental preparatory work has been carried out inside euroFOT on system specifications, providing a description of the functions under test, their operation domain, and especially the basic hypotheses which should be investigated by the experiments. In particular, a set of research hypotheses has been defined, focusing on some important

Pag. 3

issues to be studied (Fig.2). For this process, the FESTA1 methodology has been applied, with specific efforts towards creating a common and harmonised approach for all the vehicle fleets within euroFOT.

Fig. 2: Schematic of the process to define research hypotheses This work has shown the importance of defining quite specific hypotheses, able to be studied in clear and statistical terms. Priorities have been defined, considering that limited time and resources will not permit to cover all the issues; at the same time it is important to establish in advance, and not during the experiments, what are the questions to be addressed. A difficulty in such a large initiative has been providing the same framework for all the functions, allowing at the same time a number of system-specific investigations required by the stakeholders. This trade-off has been addressed by a common structure, coupled to a flexible approach. A data base of the hypotheses specification, open to possible future refinements, is now in place. Another relevant point has been the treatment of integrated functions, since several manufacturers are offering only a combination of driver support systems; the data analysis will take care of these aspects, and will distinguish in particular between longitudinal and lateral control functions. In total, approximately 250 hypotheses have been examined and 84 of them prioritised. The primary relevance of these hypotheses in term of impacts can be divided as follows: driver behaviour (46%), safety (44%), and environment (10%). The specification work has also defined several use cases, showing when the system should be tested and in which driving conditions. For each case, the study has outlined comparison situations, controlled factors and variable factors to be considered in the data analysis. All these results constitute a key point for the subsequent steps addressing the design and the implementation of the experiments. Data Acquisition Systems Data acquisition and data handling are fundamental enabling technologies for FOTs. For this aspect, euroFOT is studying techniques able to provide robustness and high reliability in all the test conditions, considering that no assistance from the experimenters will be provided on a vehicle during the tests. The acquisition of vehicle data is coordinated by a central control unit mounted on-board, with logging and communication capabilities. The loggers are expected to collect data from

1 The FESTA project, under the EC 7th Framework Programme for ICT, addressed several operative aspects regarding Field Operational Tests, and designed a handbook of good practice.

Pag. 4

the vehicle network, as well as from additional subsystems and sensors (e.g. video and radar). In addition to these objective data, euroFOT has defined methods for subjective data acquisition, which are beyond the scope of the present paper. Three types of Data Acquisition Systems are considered within euroFOT:

• CAN only, providing a basic functionality for vehicle data and GPS • CAN + video, since in a significant number of cases cameras will be used to record

both the road and the driver • CAN + video + extra sensors, for a few vehicles equipped with radar/lidar, head/eye

trackers and possibly other sensing devices

A typical example of data sources in a test vehicle is shown in Fig. 3. Besides the standard mechanical and electrical requirements, several functional requirements have been defined for Data Acquisition. The following aspects show some of the peculiarities which a FOT has to address: protection of the proprietary vehicle data, encryption of the data, driver annotation interface, possibility of audio recording, triggered logging, and real-time clock to be synchronised with the GPS. The number of channels to be detected from the CAN bus can be as high as 200 in certain cases. The amount of data from all the low bandwidth data sources is estimated to be 10-20MB/hour. The amount of video data is estimated in the range of a few hundreds MB/hour. Storage capacity on board is designed for about two months of operation. Data are then transferred to a data server periodically. The standard chosen technique is wireless communication, but in some cases it is accepted that high volume data are retrieved by exchanging the hard disk in the vehicle.

ECUECU

Vehicle CAN bus

OBDconnector

GPS & GPRS Antennas multipurpose camera

radar

accelerometer

OR

Standard with vehicleMandatory part of the instrumentation (data)Optional part of the instrumentation

ECUECU

data logger

ECUECUECUECU

Fig. 3: Example of data sources on a test car

Design of Experiment and Data Analysis Based on the systems specifications and the research hypotheses described above, a fundamental step has been the definition of the overall design of experiment (DOE). The driver, vehicle, and environmental elements of the tests have been considered. Drivers will represent as much as possible the population of buyers, with professional drivers considered in the case of trucks. Since it is known that variations between subjects can be large for the metrics normally used in the case of ADAS, a within subject analysis is the preferred approach, where the performance is compared with and without the system for each individual driver. In any case, recording a suitable set of baseline conditions turns

Pag. 5

out to be one of the most important requirements, and a relevant factor for the correctness of data analysis. For several functions, also a control group has been introduced. The vehicles have been chosen based on the present European market, considerations on technical feasibility, and some specific interests from manufacturers. They constitute a representative set of passenger cars of different brands, pertaining to the middle and top classes, and two fleets of heavy commercial vehicles. Providing a suitable driving environment is the most challenging part of the DOE. The approach within euroFOT has been to define a number of events which likely occur during driving, and are specifically relevant to answer the questions of interest. An example of event is an overtaking manoeuvre. Specific situation variables, such as road type, weather, visibility, driver status, have been also specified: they constitute a set of conditions which can characterise the surroundings of a given event. Most importantly, the DOE has identified and specified a number of performance indicators, describing driving behaviour, workload and acceptability (all of these linked to road safety), traffic efficiency, and impacts on the environment. This goes down to a detailed prescription of the actual metrics to be applied for each element. Performance indicators will be the final outcome of the statistical analysis and the basis for the overall evaluation. Examples of performance indicators are: mean speed, mean time headway, frequency of brake pedal press, glance duration off-road. In summary, the driving environment will not be controlled, due to the basic assumptions of the FOT itself, but relevant driving conditions will be monitored and recognised; moreover, the influence of situation variables will be studied. This approach is fundamental in order to extract a manageable volume of data form the very large data base expected. Within euroFOT 25 major events and 27 situation variables have been selected and a total of 81 performance indicators. The DOE has also defined about 110 corresponding measures to be considered in the experiment (i.e. quantities logged from the sensors, derived quantities, or self reported values). A final consideration regarding the DOE concerns the scheduling; it has been judged absolutely necessary to plan an accurate and extensive pilot phase, addressing all the aspects of the FOT. On one side pilot tests can provide important feedbacks to the specifications and to the research questions, as well as to the DOE; on the other side they permit to check a number of technical and organisational issues which must be settled before the definitive experiment. Vehicle Management Centres The practical operation of the FOTs is performed through Vehicle Management Centres (VMCs), established in different countries. There are 4 VMCs in the euroFOT project, in Sweden, Germany, France, and Italy. A scheme for the common structure of these VMCs is shown in Fig. 4. The recruitment of drivers is the first important responsibility of a VMC, to be organised with the support of dealers or marketing departments. A specific consensus form has been prepared to establish a relationship with the subjects under clear terms. During the FOT, a driver liaison will be in charge of contacts with the driver for reasons such as the administration of questionnaires, focus groups, data pick-up, or quality issues (e.g.: camera misplaced, sensor damaged…). At the same time procedures for maintenance will be put in place, and a hotline will be organised so that the driver has a specific reference, in case he/she should contact the project staff.

Pag. 6

Another relevant aspect is the assurance of data quality. This includes the tests on the vehicle before it is delivered, a verification of the equipment installed, and periodic checks, typically by remote monitoring of the data logging and data transfer. Regarding the operational quality, a procedure has been defined consisting of periodic status reports. As a general comment, a basic role of each VMC is to guarantee the best traceability. The chosen approach is to use relational databases containing driver information (with protected access and using anonymous procedures), data information, objective and subjective recordings, hotline reports, and status reports.

MaintenanceMaintenance SalesSalesOperation

Center + HOT LINE

Operation Center

+ HOT LINE

Vehicles Drivers

Data pick-up method

Data storagemethod

Data ServerData Server

Data pick-up method

Data storagemethod

FOT Operations Data

FOT Operations Data

Data storagemethod

Driver liaison during FOT

Data/Operational Quality Assurance

ContractConsent formIncentivesInsurance

Questions / support

Training

Objective data Subjective dataIntervention logging / Documentation

MaintenanceMaintenance SalesSalesOperation

Center + HOT LINE

Operation Center

+ HOT LINE

Vehicles Drivers

Data pick-up method

Data storagemethod

Data ServerData Server

Data pick-up method

Data storagemethod

FOT Operations Data

FOT Operations Data

Data storagemethod

Driver liaison during FOT

Data/Operational Quality Assurance

ContractConsent formIncentivesInsurance

Questions / support

Training

Objective data Subjective dataIntervention logging / Documentation

Fig. 4: Scheme of the operations within a Vehicle Management Centre

International cooperation EuroFOT also generates shared methodologies for the design and operation of FOTs. This is obtained by the trans-national cooperation inside the project, but also by contacts with parallel activities, particularly in US and Japan, focused on different methods such as: design of experiments, synchronisation of performance indicators, common tools for data handling and analysis, comparison of results. Contributions to the definition of a common language are also part of these activities. There is clearly a need for discussions among people who have the practical experience of implementing FOTs, in order to share their knowledge and to solve common problems. EuroFOT is among the participants, and is also encouraging the definition of a coordinated approach via the FOT-net initiative. A particular cooperation is in place with the parallel European project teleFOT2, with the aim to use the same tools for data handling and to harmonise the research questions and the characteristics of data bases. FINAL REMARKS EuroFOT is contributing to the development of Field Operational Tests for Driver Assistance Functions, with the aim to develop and apply a common method across Europe. Important indications are expected from these experiments in order to understand

2 TeleFOT is a large scale project under the 7th Framework Programme, investigating by means of Field Operational Tests the impacts of driver support functions provided by aftermarket and nomadic devices.

Pag. 7

driver behaviour, improve the design of the systems, and evaluate the impacts on traffic safety and efficiency. The experience gained in the first year of the project confirms several aspects under discussion within the scientific community, starting from the consideration that only a FOT can provide specific answers to several key elements for the deployment of Driver Assistance Functions, based on road data. At the same time, it is clear that such experiments require considerable resources for capturing a large amount of data over a period of several months. Therefore, on-going initiatives which are promoting effective and harmonised methods should be certainly encouraged. Some important guidelines for the design and operation of a FOT have been derived from the work performed in euroFOT. A first aspect is the need to be quite specific in the definition of research hypotheses. In addition, when several systems are tested, the experimental design should allow at the same time an harmonised framework and application-specific investigations. In particular, methods dealing with the combination of functions should be considered. A second key element is the specification of baseline conditions in the design of experiment: this is fundamental for the correctness of data analysis. Suitable metrics for the performance indicators and clear definitions of the events to be analysed are also relevant points. From the operational point of view, the study has indicated some basic elements like: procedures for risk management, a strong connection between the staff and the drivers, and robust techniques for data acquisition. The implementation of extensive and accurate pilot tests in the initial phase is considered a necessary step, since it can provide important feedbacks to the specifications and also technical insight for the conduction of the experiment. The euroFOT project is now beginning the second year of work; the next phases will address the implementation of pilot tests, followed by large scale experiments, and in parallel the further refinement of the data analysis plan. ACKNOWLEDGMENTS EuroFOT is a large scale integrated project on Information and Communication Technologies conducted under the Seventh Framework Programme of the European Commission. The authors would like to thank all the project partners who contributed to the topics outlined in the present paper.