TOWARDS REAL-LIFE SAFETY

ASSESSMENT OF ADAS AND ADS

Olaf Op den Camp | TNO Integrated Vehicle Safety | October 2016

AGENDA

Introduction

Process to come to a test protocol for Cyclist-AEB (Autonomous Emergency Braking) systems

Challenges in the assessment of Cyclist-AEB systems (example for ADAS)

Challenges towards the assessment of ADS

Methodology development to collect scenarios from driving on public road

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016 2

SOLUTIONS TO PROTECT CYCLISTS*

* IN CAR TO CYCLIST ACCIDENTS

Collision avoidance / mitigation:

• Forward collision warning • Autonomous Emergency Braking

28 OCTOBER 2016 ' | LINDHOLMEN SOFTWARE DAYS

Time-to-collision (TTC) PoNR Crash

3

DEVELOPMENT OF A CYCLIST-AEB TESTING SYSTEM (CATS)

OBJECTIVE

Prepare the introduction of a protocol for consumer tests of Cyclist-AEB systems on board passenger cars

Propose a test setup (incl. hardware) and test protocol for Cyclist-AEB systems based on technical/scientific considerations

Base the tests on analysis of most relevant cyclist accident scenarios in EU countries

TIMING

Start : 2014 Q1

Finished : 2016 Q2

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016 4

DEVELOPMENT OF A CYCLIST-AEB TESTING SYSTEM (CATS)

APPROACH

Study databases for 6 European countries;

Select severe car-to-cyclists accidents --> fatalities, seriously injured;

Provide overview of distinguished accident scenarios;

Determine the distribution of scenarios in the different countries;

Prioritize scenarios & indicate coverage of fatalities and seriously injured.

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016 5

DEVELOPMENT OF A CYCLIST-AEB TESTING SYSTEM (CATS)

' | LINDHOLMEN SOFTWARE DAYS

FINAL CATS PROTOCOL

CVBL

CVBNO

CVBNU

28 OCTOBER 2016 6

DEVELOPMENT OF A CYCLIST-AEB TESTING SYSTEM (CATS)

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

CYCLIST TARGET: SOFT BICYCLIST DUMMY ON SOFT BIKE DUMMY (version 4activeBS v5)

Changeable handle bar for

Dutch and European bike

White reflector in the front

mounted on the frame

Polymer frame with metal

layer for radar properties

Plastic mud guard

Real rubber tire with

reflecting ring

Rim with reflecting material

Materials and properties of

bicyclist same as Euro

NCAP Pedestrian Target

Adjustable torso-angle

Rotational joint of jip

connected to bike frame

Rotational joint at the knee

point

Rear red reflector mounted

on the luggage rack

Rotating wheels due to

contact to the ground

Preparation for simulating

pedaling behaviour

7

AEB HARDWARE & SOFTWARE IMPLEMENTED IN (TEST) VEHICLE:

• In physical tests on a test track, e.g. using the scenarios from the test matrix

• Determine performance in speed reduction by AEB for varying speed of the test vehicle

AEB SOFTWARE DEVELOPMENT:

• Use virtual simulation tools for prospective effectiveness assessment

CYCLIST-AEB EFFECTIVENESS ASSESSMENT

' | LINDHOLMEN SOFTWARE DAYS

real world/

scenarios

driver

car dynamics

automated

driving

function

sensor

data

vehicle state

view on surrounding

ADF input to car

vehicle response

(v, a, steering angle)

driver input

to car

sensor

system

HMI

ToC

Driver overrule

world

model

28 OCTOBER 2016 8

CYCLIST-AEB EFFECTIVENESS ASSESSMENT

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

FoV:

2 x 24°

VVUT [km/h]

ΔV

AEB

[km

/h]

ΔV

AEB

[km

/h]

VVUT [km/h]

VVUT [km/h]

VVUT [km/h]

VVUT [km/h]

VVUT [km/h]

CVBNU

CVBNU

CVBNO

CVBNO

CVBF

CVBF

FoV:

2 x 45°

Current state-of-the-art

Beyond 2018

9

CYCLIST-AEB EFFECTIVENESS ASSESSMENT

EFFECTIVENESS: BALANCING PERFORMANCE AND ROBUSTNESS

TN

TP

100%

100%

0%

FP

FN

Probability of correct function response

: reduce FN --> Performance

: reduce FP --> Customer acceptance reliability

robustness

TN

TP

Reduce FP by lowering sensor sensitivity

TN

TP

FP

Higher ‘resolution’ by smarter sensor TP: system response

when needed

FN: no system response

when it is needed

TN: no system response

when not needed

FP: system response

when not needed

• Not only aim at

reducing FN to

increase system

performance in critical

situations

• Also aim at reducing

FP to increase

customer acceptance

and performance

28 OCTOBER 2016 ' | LINDHOLMEN SOFTWARE DAYS 10

POSSIBILITIES FOR CYCLIST TO AVOID COLLISION:

Test executed by Subaru to

discuss false positives in the

CATS-project

Prototype vehicle, not mass-

production

Simulated decrease in

performance based on

calculation of PoNR

CYCLIST-AEB EFFECTIVENESS ASSESSMENT

28 OCTOBER 2016 ' | LINDHOLMEN SOFTWARE DAYS

True Positive Possible False Positive (braking bike up to 7 m/s2)

Possible False Positive (turning bike)

VVUT [km/h]

ΔV

AEB

[km

/h]

CVBNU

X

11

CYCLIST-AEB EFFECTIVENESS ASSESSMENT

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

FoV:

2 x 24°

ΔV

AEB

[km

/h]

VVUT [km/h]

VVUT [km/h]

VVUT [km/h]

VVUT [km/h]

VVUT [km/h]

VVUT [km/h]

ΔV

AEB

[km

/h]

CVBNU

CVBNU

CVBNO

CVBNO

CVBF

CVBF

FoV:

2 x 45°

Current state-of-the-art

Beyond 2018

12

FOR ADAS, TRADITIONALLY STRONG FOCUS ON ACCIDENT AND CRASH DATA

To set up test protocol for rating

To determine the performance of systems

INCREASED NEED FOR ADDRESSING NORMAL EVERY DAY DRIVING

To determine the performance on the road, not only in seldom critical/crash situations

Holds for individual ADAS, but increasingly important to deal with the complexity of multiple simultaneously acting ADAS,

especially in view of the transition towards AD.

ADS EFFECTIVENESS ASSESSMENT

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016 13

FOR AUTOMATED AND COOPERATIVE DRIVING FUNCTIONS

Integrated and interrelated functions can no longer be tested independently.

Tests are preferably conducted on public road for real life safety assessment, however safety and legal constraints apply.

The number of relevant scenarios increases strongly: • Multiple targets with individual manoeuvres

• Infrastructure related parameters: type of road, intersection, traffic rules

• Disturbances such as weather/lighting conditions, road works

Scenarios are often not reproducible, ground truth data of targets are unknown or inaccurate

CHALLENGES

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

Most relevant conflict scenarios Relevant interaction scenarios

turning left

going straight

turning right

car

bicycle

6 >36 Many cyclists are present in the scene

14

INTERPRET TRAFFIC AS AN ENDLESS SEQUENCE OF DIFFERENT SCENARIOS

DETERMINE REAL-LIFE PERFORMANCE WITHOUT DRIVING MULTIPLE MILLION KM:

• Data collection under real-life conditions on the road

• Data analysis: scenario identification & classification (big data solutions required)

• Use library of scenarios in virtual testing for development and implementation

SCENARIO:

• A typical maneuver on the road with the complete set of relevant conditions and trajectories of other traffic participants that

have an interaction with the host vehicle over a relevant time period (order of seconds)”

• A ride on the road can in this way be described by a continuous sequence of scenarios; scenarios might overlap in time.

SCENARIO BASED APPROACH

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016 15

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

HOW TO COLLECT RELEVANT SCENARIOS?

16

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

#

RAW DATA

STORAGE - bicycles

- sensors

- GPS

BEHAVIOUR MODELS

- bicyclists

- pedestrians

- vehicles

Real life bicycle data

Develop classification algorithms

(deep learning)

Run identification and classification

algorithms

DEVELOPMENT

OF ADS:

• Integrated

behaviour

prediction models

Real life vehicle data

Reference measurements

SCENARIO DATABASE

incl. all characteristic

parameter distributions

Vehicle data

Sensor data

Vehicle data

Sensor data

Vehicle data

Sensor data

EVALUATION

OF ADS:

• Virtual testing

(extensive set)

• Physical testing

(limited # of km)

Accident

databases UDRIVE Euro NCAP

Calibration based on ground truth

#

RAW DATA

STORAGE - vehicles

- sensors, GPS

- annotations

Develop classification algorithms

(deep learning)

Run identification and classification

algorithms

Big Data No ADAS response

REAL-LIFE SAFETY ASSESSMENT METHODOLOGY

17

TYPICAL ACC SCENARIOS: DATA COLLECTION – SETUP:

VIEW FROM HOST:

SCENARIO IDENTIFICATION & CLASSIFICATION (EXAMPLE)

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016 18

TYPICAL ANALYSIS APPROACH:

SCENARIO IDENTIFICATION & CLASSIFICATION (EXAMPLE)

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016 19

TYPICAL USE OF ANALYSIS RESULTS:

SCENARIO IDENTIFICATION & CLASSIFICATION (EXAMPLE)

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016

* Prepared by: Erwin de Gelder Jan-Pieter Paardekoper

20

REAL-LIFE SAFETY ASSESSMENT OF ADAS AND ADS:

Increased use of data collected on public road during ‘every day driving conditions’ for system evaluation in addition

to laboratory and test track tests. This requires:

• (Automatic) identification and classification of scenarios, and storage into a scenario database.

• Big data / machine learning solutions.

• Interface to scenario simulation tools.

Harmonization of the evaluation approach is a prerequisite to make a safe transition from ADAS to Automated Driving:

• Type approval by the road- and vehicle-authorities

• Clarity of requirements for car manufacturers and automotive suppliers

• Make use of triple helix approach: industry, governments, R&D organizations

OUTLOOK

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016 21

THANK YOU VERY MUCH FOR YOUR ATTENTION

MORE INFORMATION:

TNO Integrated Vehicle Safety W: www.tno.nl

Dr.Ir. Olaf Op den Camp E: olaf.opdencamp@tno.nl

Ir. Sytze Kalisvaart E: sytze.kalisvaart@tno.nl

' | LINDHOLMEN SOFTWARE DAYS 28 OCTOBER 2016 22