Thesis for the Degree of Do tor of Philosophy
Computational Veri� ation Methods
for Automotive Safety Systems
Jonas Nilsson
Department of Signals and Systems
Chalmers University of Te hnology
Göteborg, Sweden 2014
Computational Veri� ation Methods for Automotive Safety Systems
Jonas Nilsson
ISBN 978-91-7385-973-8
© Jonas Nilsson, 2014.
Doktorsavhandlingar vid Chalmers tekniska högskola
Ny serie nr 3654
ISSN 0346-718X
Department of Signals and Systems
Me hatroni s Group
Chalmers University of Te hnology
SE�412 96 Göteborg
Sweden
Telephone: +46 (0)31 � 772 1000
Jonas Nilsson
Telephone: +46 (0)70 � 266 5397
Email: jonas.nilsson�volvo ars. om
jonas.nilsson� halmers.se
Typeset by the author using L
A
T
E
X.
Chalmers Reproservi e
Göteborg, Sweden 2014
to Moni a
Abstra t
This thesis onsiders omputational methods for analysis and veri� ation
of the lass of automotive safety systems whi h support the driver by mon-
itoring the vehi le and its surroundings, identifying hazardous situations
and a tively intervening to prevent or mitigate onsequen es of a idents.
Veri� ation of these systems poses a major hallenge, sin e system de isions
are based on remote sensing of the surrounding environment and in orre t
de isions are only rarely a epted by the driver. Thus, the system must
make orre t de isions, in a wide variety of tra� s enarios. There are two
main ontributions of this thesis. First, theoreti al analysis and veri� ation
methods are presented whi h investigate in what s enarios, and for what
sensor errors, the absen e of in orre t system de isions may be guaranteed.
Furthermore, methods are proposed for analyzing the frequen y of in or-
re t de isions, in luding the sensitivity to sensor errors, using experimental
data. The se ond major ontribution is a novel omputational framework
for determining the errors of mobile omputer vision systems, whi h is one
of the most widely used sensor te hnologies in automotive safety systems.
Augmented photo-realisti images, generated by rendering virtual obje ts
onto a real image ba kground, are used as input to the omputer vision
system to be tested. Sin e the obje ts are virtual, ground truth is readily
available and varying the image ontent by adding di�erent virtual obje ts
is straightforward, making the proposed framework �exible and e� ient.
The framework is used for both performan e evaluation and for training
obje t lassi�ers.
Keywords: Automotive, A tive Safety, Semi-Autonomous Vehi les, Veri-
� ation, Performan e Evaluation, De ision Making, Augmented Reality.
i
ii
A knowledgments
Writing a Ph.D. thesis is a bit like hiking in the mountains. The view is
beautiful and inspiring but as one strives to the top, there is some real
physi al pain involved. You pin your hope on the summit resting right
behind the next rest, only to �nd yet another rest. Anyhow, for a person
appre iating ardiovas ular exer ise, the journey has been truly enjoyable.
Many people have a ompanied me on this journey and deserve my deep-
est gratitude. My a ademi supervisor, Dr. Jonas Fredriksson, has always
been available for dis ussions (often on topi s related to ross- ountry ski-
ing), and has helped me not to loose sight of the bigger pi ture. Thank you
for your ommitment. My industrial supervisor, Dr. Anders Ödblom, has
ontinuously supported me using his impe able eye for details. For this I
thank you. Together I think we have �nally understood what this proje t
is really about. Thank you also to Prof. Jonas Sjöberg for wel oming me
into his resear h group and o asionally allowing me to beat him in a ra e.
From the many olleagues at Volvo Cars and Chalmers ontributing to
a stimulating work environment, I would like to mention a few. My urrent
and former managers, Georgios Minos and Peter Janevik respe tively, for
giving me the freedom needed to work e� iently in between a ademia and
industry. Dr. Mohammad Ali, for fruitful ollaboration and enri hing dis-
ussions. The room got awfully quiet without you. Co-authors Adeel Zafar,
Patrik Andersson and Prof. Irene Gu, I hope we an ollaborate again in
the future. Dr. Erik Coelingh, Dr. Mattias Brännström and Jonas Ekmark,
for indire t ontributions through your on�den e and support. Dr. Fredrik
Persson, Feng Liu and David Hultberg, for your ommitted work on VCAV.
Thank you also to Vinnova, SAFER and Volvo Cars for funding this proje t.
Last but most important I thank my family, starting with my siblings
Fredrik and Pernilla for proofreading this thesis. Moni a, for your everlast-
ing support and your omplementing talents, bringing balan e to our family.
Ludvig and Valdemar, for showing me there is joy in every detail in life and
for being inexhaustible sour es of ideas on new forms of transportation.
Jonas Nilsson
Göteborg, January 2014
iii
iv
Contents
Abstra t i
A knowledgments iii
Contents v
I Introdu tory Chapters
1 Introdu tion 1
1.1 Aims and Obje tives . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 List of Publi ations . . . . . . . . . . . . . . . . . . . . . . . 4
2 Automotive Safety Systems 7
2.1 Tra� A ident Causation . . . . . . . . . . . . . . . . . . . 8
2.2 Vehi le Dynami s Control . . . . . . . . . . . . . . . . . . . 10
2.3 Driver Assistan e . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.1 Sensor Te hnology . . . . . . . . . . . . . . . . . . . 13
2.3.2 De ision-Making and Interventions . . . . . . . . . . 14
2.4 Autonomous Driving . . . . . . . . . . . . . . . . . . . . . . 15
2.5 System E�e tiveness . . . . . . . . . . . . . . . . . . . . . . 16
2.6 Veri� ation Challenges . . . . . . . . . . . . . . . . . . . . . 18
3 Veri� ation Methods 21
3.1 Performan e Metri s . . . . . . . . . . . . . . . . . . . . . . 21
3.2 Method Properties . . . . . . . . . . . . . . . . . . . . . . . 22
3.3 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3.1 System Models . . . . . . . . . . . . . . . . . . . . . 25
3.3.2 Pro ess Models . . . . . . . . . . . . . . . . . . . . . 27
3.4 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.4.1 Real Driving . . . . . . . . . . . . . . . . . . . . . . . 28
v
Contents
3.4.2 Closed-Loop Simulations . . . . . . . . . . . . . . . . 30
3.4.3 Data Replay . . . . . . . . . . . . . . . . . . . . . . . 30
3.4.4 Theoreti al Methods . . . . . . . . . . . . . . . . . . 31
3.5 Method Comparison . . . . . . . . . . . . . . . . . . . . . . 31
4 Summary of In luded Papers 35
5 Con luding Remarks 41
5.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.2 Dire tions of Future Resear h . . . . . . . . . . . . . . . . . 42
Referen es 45
II In luded papers
Paper 1 Worst Case Analysis of Automotive Collision Avoid-
an e Systems 55
1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.1 Collision Avoidan e Systems . . . . . . . . . . . . . . 57
2.2 Performan e Evaluation of Collision Avoidan e Systems 58
3 De�nitions and Problem Formulation . . . . . . . . . . . . . 59
3.1 General CA System Des ription . . . . . . . . . . . . 59
3.2 Problem Formulation . . . . . . . . . . . . . . . . . . 59
4 System Des ription . . . . . . . . . . . . . . . . . . . . . . . 61
5 S enario Des ription . . . . . . . . . . . . . . . . . . . . . . 65
6 Worst Case Performan e . . . . . . . . . . . . . . . . . . . . 67
6.1 Desired Time of Intervention . . . . . . . . . . . . . . 68
6.2 Time of Intervention . . . . . . . . . . . . . . . . . . 69
6.3 Maximum Predi tion Errors . . . . . . . . . . . . . . 70
6.4 Robust Avoidan e S enarios . . . . . . . . . . . . . . 71
7 Numeri al Results . . . . . . . . . . . . . . . . . . . . . . . . 72
7.1 Longitudinal S enarios . . . . . . . . . . . . . . . . . 73
7.2 Lateral S enarios . . . . . . . . . . . . . . . . . . . . 78
7.3 Robust Avoidan e S enarios . . . . . . . . . . . . . . 81
8 Con luding Remarks . . . . . . . . . . . . . . . . . . . . . . 84
Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Paper 2 Veri� ation of Collision Avoidan e Systems using
Rea hability Analysis 91
1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
vi
Contents
2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 92
3 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4 Problem Formulation and Proposed Approa h . . . . . . . . 96
5 Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.1 Coordinate Systems . . . . . . . . . . . . . . . . . . . 98
5.2 Relative Motion . . . . . . . . . . . . . . . . . . . . . 99
5.3 Threat Assessment . . . . . . . . . . . . . . . . . . . 99
6 Veri� ation . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
6.1 Admissible Set . . . . . . . . . . . . . . . . . . . . . 101
6.2 Computing the Safe and Unsafe Sets . . . . . . . . . 101
6.3 Veri� ation . . . . . . . . . . . . . . . . . . . . . . . 101
6.4 Error Robustness . . . . . . . . . . . . . . . . . . . . 102
7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
8 Con luding Remarks . . . . . . . . . . . . . . . . . . . . . . 104
Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Paper 3 Sensitivity Analysis and Tuning for A tive Safety
Systems 109
1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
2 De�nitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
3 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
3.1 Performan e Evaluation . . . . . . . . . . . . . . . . 115
3.2 Input Requirements . . . . . . . . . . . . . . . . . . . 119
3.3 Fun tion Tuning . . . . . . . . . . . . . . . . . . . . 120
4 Appli ation Example . . . . . . . . . . . . . . . . . . . . . . 120
5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . 121
5.1 Input Requirements . . . . . . . . . . . . . . . . . . . 122
5.2 Fun tion Tuning . . . . . . . . . . . . . . . . . . . . 124
6 Con lusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Paper 4 Performan e Evaluation Method for Mobile Com-
puter Vision Systems using Augmented Reality 131
1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
2 Performan e Evaluation Framework . . . . . . . . . . . . . . 133
3 Case study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
3.1 System Des ription . . . . . . . . . . . . . . . . . . . 136
3.2 S enario Des ription . . . . . . . . . . . . . . . . . . 136
3.3 Implementation . . . . . . . . . . . . . . . . . . . . . 136
3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 138
4 Dis ussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.1 Dis ussion on Method . . . . . . . . . . . . . . . . . 139
vii
Contents
4.2 Dis ussion on Case Study Results . . . . . . . . . . . 140
5 Con lusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Paper 5 Reliable Vehi le Pose Estimation using Vision and
Single-Tra k Model 147
1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 149
2.1 Visual Pose Estimation . . . . . . . . . . . . . . . . . 149
2.2 Ground Vehi le Pose Estimation . . . . . . . . . . . . 150
3 Vehi le and Camera Modeling . . . . . . . . . . . . . . . . . 151
3.1 Coordinate Systems . . . . . . . . . . . . . . . . . . . 152
3.2 Pinhole Camera . . . . . . . . . . . . . . . . . . . . . 154
3.3 Lo al Vehi le Motion . . . . . . . . . . . . . . . . . . 155
4 Bundle Adjustment Framework . . . . . . . . . . . . . . . . 157
4.1 Parametrization . . . . . . . . . . . . . . . . . . . . . 158
4.2 Error Modeling . . . . . . . . . . . . . . . . . . . . . 158
5 Re onstru tion Algorithm . . . . . . . . . . . . . . . . . . . 160
5.1 Image Feature Extra tion and Mat hing . . . . . . . 161
5.2 In remental Estimation of Initial Pose . . . . . . . . 161
5.3 Lo al Bundle Adjustment . . . . . . . . . . . . . . . 162
6 Experimental Validation Method . . . . . . . . . . . . . . . 162
6.1 Data Colle tion and Des ription . . . . . . . . . . . . 163
6.2 Performan e Metri s . . . . . . . . . . . . . . . . . . 165
7 Experimental Results . . . . . . . . . . . . . . . . . . . . . . 166
7.1 Bundle Adjustment Performan e . . . . . . . . . . . 167
7.2 Sensitivity to Initial Estimate . . . . . . . . . . . . . 174
7.3 Sensitivity to Model Parameters . . . . . . . . . . . . 177
7.4 Moving Obje ts . . . . . . . . . . . . . . . . . . . . . 180
8 Con lusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
Paper 6 Pedestrian Dete tion using Augmented Training Data189
1 Introdu tion . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 190
3 Pedestrian Dete tion . . . . . . . . . . . . . . . . . . . . . . 191
3.1 Histogram of Oriented Gradients . . . . . . . . . . . 192
3.2 Support Ve tor Ma hine . . . . . . . . . . . . . . . . 192
3.3 Sliding Window Dete tion . . . . . . . . . . . . . . . 193
4 Augmented Data Generation . . . . . . . . . . . . . . . . . . 193
5 Evaluation Methodology . . . . . . . . . . . . . . . . . . . . 194
5.1 Real Training Data . . . . . . . . . . . . . . . . . . . 196
viii
Contents
5.2 Real Test Data . . . . . . . . . . . . . . . . . . . . . 197
5.3 Augmented Training Data . . . . . . . . . . . . . . . 197
6 Experimental Results . . . . . . . . . . . . . . . . . . . . . . 197
7 Con luding Remarks . . . . . . . . . . . . . . . . . . . . . . 200
Referen es . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
ix
x
Part I
Introdu tory Chapters
Chapter 1
Introdu tion
Road tra� a idents are a global problem of epidemi proportions. A -
ording to the World Health Organization (WHO), road tra� injuries are
the leading ause of death globally for young people aged 15 − 29, and
the eight leading ause of death in total, [1℄. In the developed ountries
primarily, road tra� a idents have been on the agenda in the past few
de ades. Governments have invested in infrastru ture and passed laws to
improve road safety. The automotive industry has put emphasis on design-
ing systems that prote t the o upants of the vehi le in ase of a rash, so
alled passive safety systems. Passive safety innovations in lude seat belts,
rumple zones and airbags.
In the 1970s, the introdu tion of Anti-lo k Braking Systems (ABS)
marked a �rst milestone for a tive safety systems, i.e. systems whi h a -
tively intervene to prevent or mitigate onsequen es of a idents. In re ent
years, a tive safety systems whi h monitor the surrounding environment,
using remote sensing te hnologies, have been introdu ed to the market.
By using information on the surrounding tra� environment, systems an
identify hazardous situations, e.g. when the driver has failed to observe a
rossing pedestrian and a ollision is imminent. If and when hazardous
situations are dete ted, the system an a tively intervene to prevent an
a ident either by informing the driver of the up oming danger or by au-
tonomously performing an evasive maneuver su h as Automati Emergen y
Braking (AEB).
This thesis on erns the problem of verifying that a given a tive safety
system a ts orre tly in the wide variety of possible tra� s enarios. There
are two major reasons why this is a hallenging task. First, the variations
in operating onditions are essentially unlimited, a fa t easily a knowledged
when re�e ting and omparing a snowy ountry road in northern Sweden to
downtown Tokyo. Se ond, in orre t de isions by highly intrusive systems,
like AEB, an only be a epted on very rare o asions.
1
Chapter 1. Introdu tion
1.1 Aims and Obje tives
The aim of the work presented in this thesis is to develop omputational
methods for e� ient veri� ation of automotive safety systems. In this on-
text, omputational veri� ation methods are de�ned as methods whi h pre-
di t system performan e by performing omputations with re orded exper-
imental data and/or mathemati al models as input.
In a tive safety systems, de ision fun tions use input from sensors to
de ide how to appropriately support the driver. A vital part of a tive safety
system performan e is the ability to make orre t de isions, also in the
presen e of sensor measurement errors. Consequently, three obje tives are
formulated, namely to develop methods that
I. For a given a tive safety de ision fun tion, identify tra� s enarios
where the fun tion makes in orre t de isions
II. For a given a tive safety de ision fun tion, quantify the robustness to
input errors
III. Generate virtual sensor data with su� ient quality for analysis and
veri� ation
The �rst two obje tives are addressed by Papers 1-3, while the third obje -
tive is treated in Papers 4-6.
1.2 Delimitations
This thesis is on erned with semi-autonomous vehi les where a tive safety
systems monitor the tra� situation and intervene if needed to ensure safety.
Obje tives I and II are delimited to evaluating the orre tness of the inter-
vention de ision as opposed to the hoi e and exe ution of the intervention.
With regards to the same two obje tives, only tra� s enarios with single
moving obje ts are onsidered. In Obje tive II, we primarily onsider in-
put errors whi h are bounded and systemati , where systemati means that
they depend on the spe i� tra� situation. Obje tive III is on erned with
e� iently determining said input errors and is delimited to omputer vision
sensors, whi h is one of the dominating te hnologies used in a tive safety
appli ations.
1.3 Thesis Outline
The thesis is divided into two parts. Part I serves as an introdu tion to
Part II by presenting ba kground information and related work. Part II
2
1.3. Thesis Outline
ontains six s ienti� papers that onstitute the base of the thesis.
Part I provides ontext to the appended papers and is organized as fol-
lows. In Chapter 1, the topi of the thesis is introdu ed and aims, obje tives
and delimitations are des ribed. Chapter 2 gives an overview of in-vehi le
safety systems with a strong emphasis on a tive safety systems. In Chap-
ter 3, an overview of methods for system veri� ation is provided. Chapter 4
brie�y summarizes the papers in luded in Part II while Chapter 5 presents
the main s ienti� ontributions and gives suggestions for future resear h.
3
Chapter 1. Introdu tion
1.4 List of Publi ations
This thesis is based on the following publi ations:
Paper 1
J. Nilsson, A. Ödblom and J. Fredriksson, Worst Case Analy-
sis of Automotive Collision Avoidan e Systems, submitted for
possible journal publi ation.
Paper 2
J. Nilsson, J. Fredriksson and A. Ödblom, Veri� ation of Colli-
sion Avoidan e Systems using Rea hability Analysis, submitted
as invited paper to the 19th IFAC World Congress, Cape Town,
South Afri a, 2014.
Paper 3
J. Nilsson and M. Ali, Sensitivity Analysis and Tuning for A tive
Safety Systems, in Pro eedings of the 13th International IEEE
Conferen e on Intelligent Transportation Systems, 2010, pages
161-167, Madeira Island, Portugal.
Paper 4
J. Nilsson, A. Ödblom, J. Fredriksson, A. Zafar and F. Ahmed,
Performan e Evaluation Method for Mobile Computer Vision
Systems using Augmented Reality, in Pro eedings of the IEEE
Virtual Reality Conferen e, 2010, pages 19-22, Waltham, Mas-
sa husetts, USA.
Paper 5
J. Nilsson, J. Fredriksson and A. Ödblom, Reliable Vehi le Pose
Estimation using Vision and Single-Tra k Model, submitted for
possible journal publi ation.
Paper 6
J. Nilsson, P. Andersson, I. Gu and J. Fredriksson, Pedestrian
Dete tion using Augmented Training Data, submitted to the
22nd International Conferen e on Pattern Re ognition, Sto k-
holm, Sweden, 2014.
4
1.4. List of Publi ations
Other Publi ations
In addition to the publi ations above, the following publi ations by the
thesis author are related to the topi of this thesis:
J. Nilsson, J. Fredriksson, and A. Ödblom, Bundle Adjustment
using Single-Tra k Vehi le Model, in Pro eedings of the IEEE
International Conferen e on Roboti s and Automation, 2013, pp.
2888-2893.
J. Nilsson, A. Ödblom, J. Fredriksson, and A. Zafar, Using Aug-
mentation Te hniques for Performan e Evaluation in Automo-
tive Safety, in Handbook of Augmented Reality, 1st ed., B. Furht,
Ed. Springer, 2011, pp. 631-649.
J. Nilsson, On Performan e Evaluation of Automotive A tive
Safety Systems, Li entiate Thesis R014/2010, ISSN 1403-266X,
Chalmers University of Te hnology, Göteborg, Sweden, 2010.
J. Nilsson and A. Ödblom, On Worst Case Performan e of Col-
lision Avoidan e Systems, in Pro eedings of the IEEE Intelligent
Vehi les Symposium, 2010, pages 1084-1091, San Diego, Califor-
nia, USA.
A. Ödblom and J. Nilsson, Augmented Vision in Image Sequen e
Generated from a Moving Vehi le, Patent pending, EP2639771,
European Patent O� e, 2012.
J. Nilsson, Operating Method and System for Supporting Lane
Keeping of a Vehi le, Patent granted, US8428821, U.S. Patent
and Trademark O� e, 2007.
J. Nilsson, Operating Method and System for Supporting Lane
Keeping of a Vehi le, Patent granted, EP2188168, European
Patent O� e, 2007.
5
6
Chapter 2
Automotive Safety Systems
The over 1 million annual fatalities aused by road tra� a idents are
merely the tip of the i eberg, e.g. the WHO estimates that road tra�
a idents also lead to between 20 and 50 million non-fatal injuries ea h
year, [1℄. On top of that, the e onomi burden linked to road tra� a idents
is signi� ant. In 1998, a rude estimate of the annual global ost was found
to be in the order of US$500 billion, [2℄.
There are large regional di�eren es a ross the world as the variations
in vehi le safety, infrastru ture and driver edu ation are substantial. Re-
markable progress has been made in the developed ountries during the last
de ades, as an be seen in Figure 2.1. Improved vehi le design, road infras-
tru ture investments and road safety poli ies have ontributed to redu ing
the risk of getting killed in tra� , in most developed ountries, by more
than 40% sin e 1990, [3℄.
Su ess in redu ing fatalities has spurred stakeholders in road safety to
set more and more ambitious goals, as des ribed in [6℄. The most ambitious
goal possible, i.e. a vision of zero fatalities in road tra� , has been expressed
in road safety poli ies in Sweden and the Netherlands. The urrent and
future automotive safety systems dis ussed in this hapter have the potential
to ontribute signi� antly to this goal.
We ategorize automotive safety systems into passive safety systems,
whi h prote t the vehi le o upants when ollision has o urred, and three
types of a tive safety systems, whi h are designed to prevent a idents.
The �rst ategory of a tive safety systems, vehi le dynami s ontrol sys-
tems, prevent unwanted dynami al behaviours su h as instability. Driver
Assistan e (DA) systems monitor the vehi le surroundings to assist the
driver. In a not too distant future, Autonomous Driving (AD) systems may
take omplete responsibility for the driving task. The line between these
ategories is by no means sharp, as exempli�ed by the Roadway Departure
Prevention Assist (RDPA) system des ribed in [7℄ whi h in orporate both
7
Chapter 2. Automotive Safety Systems
1970 1975 1980 1985 1990 1995 2000 2005 20100
5
10
15
20
25
30
Year
Fatalities/100
000Inhab
itan
ts
United States
United Kingdom
Germany
Netherlands
Sweden
Figure 2.1: Histori al road tra� fatalities, obtained from [4℄, for some of
the developed ountries. As a referen e, low- and middle-in ome ountries
have annual road tra� fatalities of 18.3 and 20.1 per 100 000 inhabitants
respe tively, [1℄. The sharp in rease in fatalities for Germany in 1990 is an
e�e t of the reuni� ation of Germany, [5℄.
stability ontrol and ollision avoidan e in a single framework.
In this hapter, the di�erent ategories of a tive safety systems are de-
s ribed, after �rst providing some ontext by brie�y dis ussing the auses
of a idents. In the �nal part of the hapter, the e�e tiveness of these sys-
tems is reviewed followed by a dis ussion on the hallenges asso iated with
system veri� ation, whi h is the ore problem addressed in this thesis.
2.1 Tra� A ident Causation
To e� iently prevent a idents, the auses of a idents need to be under-
stood. A ommon approa h for identifying a ident auses is to study a -
ident statisti s. Figure 2.2 shows the a ident distribution in terms of
major rash types, obtained from [8℄. There are numerous ways to lassify
a idents, e.g. by gender, age, type of vehi le, time of day or weather on-
ditions. Extensive reports with a ident lassi� ations based on national
a ident statisti s are published ontinuously, see e.g. [9℄ for the U.S. or [10℄
for Sweden.
Human error plays a major role in a majority of a idents. In an in depth
8
2.1. Traffi A ident Causation
Rear-End
28.4%
Crossing Paths
24.9%
Off Roadway
22.7%
Lane Change
9%
Animal
4%
Opposite Direction
2.6%
Backing
2.1%
Pedestrian
1.1%Pedalcyclist
0.8%
Other
4.4%
Figure 2.2: Distribution in terms of major rash types for all 6 394 000
poli e-reported motor vehi le rashes in the U.S. whi h resulted in
3 189 000 injured people and a total of 41 821 fatalities, [8℄. The �gure
is based on statisti s from the 2000 National Automotive Sampling System
(NASS)/General Estimates System (GES) rash database.
study of real a idents in the 1970s, [11℄, in luding on-s ene investigations,
it was on luded that human parti ipants were solely or partly to blame in
92.6% of the investigated a idents. The orresponding numbers for envi-
ronmental and vehi ular fa tors were 33.8% and 12.6% respe tively. Com-
mon human errors were e.g. ex essive speed, improper evasive a tion and
driver inattention or distra tion. Environmental fa tors were e.g. view ob-
stru tions and slippery road surfa es while vehi ular fa tors in luded brake
failures and inadequate tyre tread depth.
More re ently, in 2005, a Field Operational Test (FOT) known as the
100-Car Study, [12, 13℄, was ompleted. 100 ars were equipped with un-
obtrusive data olle tion instrumentation to olle t naturalisti data from
normal driving. The study rea�rms that drivers are often to blame for
a idents as nearly 80% of all rashes involved the driver looking away from
the forward roadway just prior to the ollision. Driver inattention or dis-
tra tion, e.g. using a mobile phone while driving, does not ne essarily lead
to an a ident but if oin iding with another unfortunate event, e.g. the
vehi le in front suddenly braking, the probability of an a ident in reases
signi� antly. Multiple a ident auses mean that there are multiple possible
preventive measures. As a idents are very diverse, preventing a majority of
9
Chapter 2. Automotive Safety Systems
a idents requires the deployment of a large number of preventive measures.
2.2 Vehi le Dynami s Control
Following advan es in ele troni s te hnology, mass produ tion of ABS sta-
rted on road vehi les in the 1970s but the innovation had been present in
the railway and aviation industries de ades before that. ABS monitors the
rotational speed of the wheels and automati ally redu e the brake for e if
the wheels ease to rotate, thus preventing brake lo k-up. This enables
steering of the vehi le while simultaneously braking hard.
In the 1990s, Ele troni Stability Control (ESC) was introdu ed to han-
dle problems with vehi le instability. ESC dete ts when the vehi le starts
to skid and ountera ts this by automati ally braking the wheels individ-
ually, as illustrated in Figure 2.3. A natural evolution of ESC is to also
prevent the vehi le from rolling over, as presented in [14℄. Roll Stability
Control (RSC) is mostly relevant for vehi les with high enter of gravity,
su h as Sport Utility Vehi les (SUVs) and tru ks, and was �rst introdu ed
in 2002, [15℄.
The interested reader is referred to e.g. [16,17℄, for more omprehensive
treatments of vehi le dynami s ontrol systems.
Figure 2.3: A vehi le drives onto an i e pat h in a urve. Without ESC
the vehi le be omes unstable and starts spinning. With ESC the left front
wheel is braked, thereby ountera ting the rotation, ensuring that stability
is maintained.
10
2.3. Driver Assistan e
2.3 Driver Assistan e
Re ent advan es in remote sensing te hnology have led to the introdu -
tion of several DA systems, see e.g [17, 18℄ for extensive overviews. One of
the �rst examples, laun hed in 1995, is an extension of the ruise ontrol
whi h automati ally maintains a onstant vehi le speed set by the driver.
Adaptive Cruise Control (ACC), thoroughly des ribed in [16,19℄, uses infor-
mation from a forward looking sensor, e.g. a radar, to maintain a onstant
distan e or time gap, set by the driver, to the vehi le in front of the host
vehi le, see Figure 2.4. ACC ontributes to safe driving by assuring that a
safe distan e is kept to the vehi le ahead. Also, ACC an redu e fuel on-
sumption and ongestion through smooth ontrol of the brakes and throttle,
thereby ontributing to a leaner environment.
Utilizing the same forward-looking sensor, Forward Collision Warning
(FCW) indi ates to the driver, as exempli�ed in Figure 2.5a, when im-
minent a tion is needed to avoid a ollision, e.g. when the vehi le ahead
suddenly brakes. If there is insu� ient time or if the driver fails to respond
to warnings, a Collision Avoidan e (CA) system an autonomously ontrol
the vehi le to avoid the impending ollision. A ommon a tion for CA sys-
tems is to automati ally apply the brakes in situations where a ollision
is imminent, so alled AEB, illustrated in Figure 2.5b. If the ollision is
unavoidable, AEB may still be triggered to redu e impa t speed, so alled
Collision Mitigation (CM).
There are also numerous DA systems whi h support the lateral ontrol
of the vehi le, as illustrated in Figure 2.6. If the vehi le rosses a lane
marking a Lane Departure Warning (LDW), [20℄, may be issued to the
driver. A lane guidan e system losely related to LDW is Lane Keeping
Assistan e (LKA), [16℄, where the driver is supported by a torque on the
steering wheel to stay in the urrent lane. In [7℄ the problem of road or
lane departures and vehi le stability are addressed in a ommon framework,
Figure 2.4: ACC automati ally maintains a driver set time gap to the vehi le
in front.
11
Chapter 2. Automotive Safety Systems
(a) (b)
Figure 2.5: (a) FCW displayed in a Head Up Display (HUD). The red light
displayed to the driver in the windshield is designed to resemble the appear-
an e of vehi le brake lights. (b) When the host vehi le enters the red zone,
an imminent ollision is dete ted and an autonomous brake intervention is
initiated.
thereby ombining and enhan ing the fun tionality of lane guidan e systems
and ESC. There are also systems that support the driver when performing
lane hange maneuvers. Lane Change Aid (LCA) systems, [21℄, monitor
adja ent lanes and inform the driver when an obsta le is present in the blind
spot of the rear view mirrors, see Figure 2.6b. In some situations there is
very little, if any, time to warn the driver of a potential hazard, making
it justi�ed for a CA system to ontrol the steering of the host vehi le to
avoid a idents. A system designed to avoid ollisions with on oming tra�
using steering interventions, referred to as Emergen y Lane Assist (ELA),
is presented in [22℄.
Information of host vehi le motion and road geometry an also be used
to assess the present state of the driver. If a driver is fatigued, distra ted
or even impaired by drugs, this will a�e t the driver's ability to maneuver
the vehi le smoothly in the urrent road lane. [23℄ presents a method for
dete ting inadequate driving behaviour, whi h an be used by systems to
e.g. inform the driver when about to fall asleep.
The underlying te hnology for DA systems is dis ussed in the follow-
ing subse tions. DA systems are me hatroni systems and onsist of three
basi layers, namely the per eption, de ision and a tion layers. The ar hi-
te ture for a DA system performing autonomous interventions is illustrated
in Figure 2.7.
12
2.3. Driver Assistan e
(a) (b)
Figure 2.6: (a) A lane guidan e system dete ts the lane markings and warns
the driver (LDW), or applies a steering wheel torque (LKA), when rossing
the lane boundary. (b) The olored zones visualize the blind spots, i.e. the
zones not visible to the driver through the rear view mirrors. LCA indi ates
that an obsta le is present in the blind spot by lighting a small lamp lose
to the rear view mirror.
2.3.1 Sensor Te hnology
A key enabler for DA systems is reliable remote sensing te hnology. In the
per eption layer, see Figure 2.7, sensors olle t observations from the envi-
ronment, driver and host vehi le. Depending on the requirements imposed
by the system, various te hnologies an be hosen to deliver an interpreta-
tion of the surrounding environment.
A frequently used sensor te hnology is omputer vision, whi h dete ts
and lassi�es obje ts in the environment using image data olle ted by am-
eras. Computer vision is the dominant te hnology to retrieve information
on the road geometry and the relative position of the host vehi le to the
road, whi h is done by dete ting the lane markings or the edge of the road.
A tive sensors su h as radar, laser or ultrasoni sensors transmit radio,
opti al or sound signals and evaluate obje t attributes by interpreting the
re�e ted response of the transmitted signal. Also, observations from digital
maps and sensors mounted on other vehi les or infrastru ture an be made
available to the safety system through a ommuni ation devi e.
In many appli ations, system requirements annot be ful�lled by a single
sensor. Sensor observations from multiple sensors are ombined, or fused,
to provide an enhan ed view of the environment. Also, obje ts observed
by sensors are tra ked over time to redu e the in�uen e of noise. General
frameworks for sensor data fusion and tra king are des ribed in [24, 25℄
while [26�29℄ des ribe work tailored to DA systems.
13
Chapter 2. Automotive Safety Systems
Sensor 1
Sensor 2
Sensor n
SensorFusion
&Tracking
ThreatAssessment
DecisionMaking
VehicleControl
DriverInteraction
Perception Decision Action
Figure 2.7: System ar hite ture for an a tive safety system designed to
intervene in ase a riti al situation arises. The per eption layer provides
information used for de ision making in the de ision layer. The de ision is
exe uted in the a tion layer via one or multiple a tuators, e.g. brake system
or driver information displays.
2.3.2 De ision-Making and Interventions
In the de ision layer, see Figure 2.7, input from the per eption layer is used
to de ide if and how to intervene. This de ision fun tion onsists of two
parts. The pro ess of onverting state estimations, e.g. obje t positions,
into measures des ribing whether or not the host vehi le is in a hazardous
situation, i.e. if surrounding road users and obje ts onstitute a threat of
ollision, is termed threat assessment. Based on the threat measures, a
de ision-making algorithm hooses what, if any, a tion should be taken by
the system.
The earlier, relative to the potential a ident, the system intervenes,
the more likely it is to prevent the a ident. Also, the earlier the system
intervenes, the more likely it is that the driver is well aware of the hazard
and thus perfe tly apable of preventing the a ident. If the latter is true
then the driver would onsider the intervention unne essary. Therefore, the
aim of the de ision fun tion is usually to intervene at the latest point in
time when the intervention type is still likely to su eed, where su ess is
de�ned as e.g. preventing or mitigating the onsequen es of an a ident.
A CA system aims to avoid all potential ollisions. For lane guidan e
systems, the aim is not as straightforward to de�ne sin e a lane departure
not ne essarily leads to a dangerous situation. Most LDW systems aim
14
2.4. Autonomous Driving
at issuing warnings ex lusively when lane departures are unintentional. In
situations when the driver intentionally deviates from the urrent lane, it is
assumed that the driver an manage the situation.
There is a range of possible a tions, or intervention types, whi h an be
applied when a hazardous situation is dete ted. If the situation is dete ted
early, the system, e.g. FCW or LDW, an warn the driver by for instan e
audible, visual or hapti feedba k. In ertain situations, there is no time
for the driver to rea t to the feedba k and perform a driving maneuver to
avoid the impending a ident. In those situations the system an, to avoid
the a ident, autonomously ontrol the brakes or the steering.
System interventions are sometimes per eived as intrusive by the driver.
The level of intrusiveness varies between intervention types where warnings
or information to the driver are generally less intrusive than autonomous
vehi le ontrol. The amplitude of the intervention also has an in�uen e as
e.g. a loud warning signal is often onsidered more intrusive than a subtle
warning signal. The possibility for the driver to override an intervention
also a�e ts the level of intrusiveness.
2.4 Autonomous Driving
Automotive safety systems whi h intervene autonomously to prevent a i-
dents are urrently ommer ially available from a large number of vehi le
manufa turers. The systems are evolving to handle more and more oper-
ating s enarios su h as interse tions and night-time driving, and this trend
is likely to ontinue, see Figure 2.8. An enabler for this evolution is the
availability of more a urate, a�ordable remote sensors.
The resear h ommunity has for quite some time fo used on the next ma-
jor step in automotive safety, namely Autonomous Driving systems. These
are systems whi h takes full responsibility for the driving task as opposed
to DA systems whi h still require the driver to monitor the system. In the
2007 DARPA Urban Challenge, [30℄, 35 teams formed from ollaborations
between industry and a ademia ompeted with driverless vehi les in an ur-
ban environment. A total of six self-driving vehi les ompleted the ourse
whi h in luded tasks su h as negotiating interse tions, parking and avoiding
vehi les stalled on the road.
In many ways AD systems are a natural evolution of DA systems and
a number of ompanies, vehi le manufa turers and others, have ommuni-
ated their aim to ommer ialize this te hnology. The potential bene�t of
AD systems is undoubtedly huge, not only in terms of safety, but also in
terms of redu ed fuel onsumption, redu ed ongestion and added driver
onvenien e.
15
Chapter 2. Automotive Safety Systems
(a) (b)
Figure 2.8: (a) Possible sensor setup for future vehi les: 360◦ �eld of view
with ameras and radars. (b) Autonomous vehi les and future driver assis-
tan e systems must handle more tra� s enarios, e.g. night onditions.
2.5 System E�e tiveness
In the last de ades, passive safety systems have made a major ontribution
to road tra� safety through innovations su h as the safety belt, rumple
zones and airbags, see Figure 2.9. Their e�e tiveness has been extensively
studied using a ident statisti s. In the U.S. during 2008, a ording to [31℄,
seat belts saved 13 250 lives, frontal airbags 2 546 and hild restraints 244.In [32℄, it is shown that passive safety improvements have ontributed to a
signi� ant de rease in injury severity between the 1970s and the 1990s, also
when ignoring e�e ts from seat belts and airbags.
The e�e tiveness of passive safety systems is assessed by governments
around the world. In Europe, EuroNCAP has sin e 1997 assessed ars, by
e.g. rash tests, in order to provide onsumers with an independent rating
of safety performan e. A tive safety systems su h as ESC are in luded
in this rating and in 2014 AEB will also be in luded, [33℄. These ratings
are important selling arguments for vehi le manufa turers and thus they
en ourage rapid development of new safety te hnology.
Vehi le dynami s ontrol systems have been widely deployed in the mar-
ket for many years, making it possible to assess their e�e tiveness in im-
proving road safety using a ident statisti s. In [34℄, multiple studies inves-
tigating the safety impa t of ABS are reviewed. A majority of the studies
indi ate that equipping vehi les with ABS signi� antly redu es the o ur-
ren e of a idents involving multiple vehi les. Some studies also indi ate
that ABS in reases the o urren e of run-o�-road a idents. Possible on-
tributing fa tors to this in rease in lude inappropriate use of ABS and driver
behaviour adaption, e.g. when the driver de reases driving safety margins
16
2.5. System Effe tiveness
(a)
(b)
Figure 2.9: (a) The airbag is an example of passive safety te hnology. (b)
Crash tests are used to assess passive safety e�e tiveness.
due to awareness of the positive safety e�e ts of ABS.
A ording to [35℄, use of ESC redu es fatal single-vehi le a idents in-
volving ars and Sport Utility Vehi les (SUVs) by 30-50% and 50-70% re-
spe tively. Considering that single-vehi le rashes stand for 60% of all fatal
rashes in the U.S., [31℄, the potential safety impa t of ESC is signi� ant.
Additionally, the redu tion of rollover a ident fatalities, related to the use
of ESC, is in [35℄ estimated to 70-90%, regardless of vehi le type.
DA systems have, if at all, been introdu ed to the market relatively re-
ently whi h explains why their e�e tiveness has not been studied to the
same extent. An overview on the subje t is given in [17℄ whi h on ludes
that the safety impa ts of DA systems are expe ted to be onsiderable. Due
to the la k of data, several approa hes have been proposed to predi t sys-
tem e�e tiveness su h as re onstru ting real-world a idents from a ident
databases and using simulations to determine if a given system ould prevent
these a idents. Using this method [36℄ predi ts that a newly introdu ed
CA system ould prevent up to 24% of pedestrian fatalities and [37℄ predi ts
that a similar system ould redu e driver fatalities in rear-end rashes by
up to 50%.
17
Chapter 2. Automotive Safety Systems
Driver
Process
EnvironmentVehicle
Active Safety
System
Figure 2.10: The driver monitors both the vehi le and the surrounding envi-
ronment to ontrol the vehi le. The a tive safety system monitors the om-
plete pro ess and ontrol the vehi le either dire tly, or indire tly through
driver intera tion, see Figure 2.7. The swit hes determine if the system is
exe uted in open- or losed-loop, see Se tion 3.2.
2.6 Veri� ation Challenges
This se tion introdu es terminology and dis usses the hallenges asso iated
with system veri� ation. Consider a pro ess, as illustrated by the top part
of Figure 2.10 onsisting of a vehi le, a driver, and an environment. The
environment has in general both stati and dynami ontent where stati
ontent is e.g. roads, trees and tra� signs and dynami ontent is e.g. road
users su h as ars, bi y les and pedestrians. The a tive safety system, in-
tera t with the pro ess a ording to Figure 2.10. The system monitor and
ontrol the pro ess to ensure that the host vehi le is operated safely.
The purpose of system veri� ation is to ensure that the system per-
forman e meets the system requirements. This must be addressed for the
omplete set of operating s enarios, de�ned by the variations in the pro-
ess, i.e. the variations in vehi le, driver and environment behaviour. The
set of operating s enarios is essentially unlimited in size as ombinations
of e.g. weather onditions, road user types, appearan e and motion pat-
terns are in�nite, see Figure 2.11a. De�ning the boundaries of this set is a
hallenge in itself sin e the system is mobile and travels in an environment
whi h is ompletely or partially unknown to the system a priori.
The system relies on real-time remote sensing of e.g. road users and road
geometry to make de isions on when and how to intervene. The sensing
performan e depends on variations in the environment, as illustrated by
Figure 2.11b. For instan e, a amera subje ted to dire t sunlight will exhibit
poor obje t dete tion performan e, mu h like the human eye.
The more intrusive an intervention type is, see Se tion 2.3.2, the less
18
2.6. Verifi ation Challenges
(a) (b)
Figure 2.11: Illustration of system veri� ation hallenges as seen by a vision
sensor. (a) One of the many possible omplex tra� situations. (b) The
sensor is partially blinded when exiting the tunnel.
likely the driver is to a ept an unne essary intervention. Consequently,
the a eptable rate of unne essary interventions is very low for systems per-
forming intrusive interventions. The large quantity of operating s enarios
makes veri� ation of this requirement, on a low rate of unne essary inter-
ventions, espe ially hallenging.
19
20
Chapter 3
Veri� ation Methods
The goal of system performan e evaluation, in the ontext of this thesis,
is to determine the performan e of an a tive safety system in a given set
of operating s enarios. If system performan e evaluation is used for sys-
tem veri� ation, the performan e estimate is ompared to a set of system
requirements, whi h spe ify the a eptable level of system performan e. A -
urate and e� ient methods for system performan e evaluation are needed
for several purposes, e.g. system veri� ation, system tuning or analysing
the system sensitivity to disturban es. For veri� ation purposes it is usu-
ally su� ient to derive or estimate a bound on performan e, to show that
the system requirements are ful�lled. As a onsequen e, some methods
fo us on performan e bounds while some fo us on performan e estimates.
This hapter provides an overview on veri� ation methods used in an a tive
safety ontext.
3.1 Performan e Metri s
In this se tion, performan e metri s des ribing the ability of the system to
make orre t de isions are presented. A ommonly used terminology for de-
s ribing the nature of in orre t de isions omes from statisti al hypothesis
testing, extensively overed in [38℄, and was �rst dis ussed in [39℄. A hy-
pothesis test is lassi�ed with regards to the test out ome, i.e. the de ision
on what hypothesis to a ept, and the true hypothesis, see Figure 3.1. The
default de ision, often a de ision not to perform an a tion, is in statisti al
hypothesis testing represented by the null hypothesis. A test out ome is
said to be negative if the null hypothesis is a epted and positive in the
opposite ase.
In an a tive safety ontext, a test would be e.g. to de ide whether or
not to initiate an autonomous brake intervention, the true hypothesis would
21
Chapter 3. Verifi ation Methods
Missed InterventionType II Error
False Negative
True NegativeIntervention is needed(Null hypothesis is
nottrue)
Intervention is needed(Null hypothesis is false)
System intervene(Null hypothesis is accepted)
does not System intervenes(Null hypothesis is rejected)
Unnecessary InterventionType I Error
False Positive
True Positive
Figure 3.1: Error types for a system de iding on whether or not to intervene.
represent the orre t de ision and the null hypothesis would represent the
de ision not to intervene.
Linked to this, there are two types of errors, ommonly referred to as
Type I and Type II errors. If the null hypothesis is true and is reje ted by
the test, the error is Type I or false positive. If instead the null hypothesis is
false and is a epted by the test, the error is Type II or false negative. False
positives and false negatives are in this thesis referred to as unne essary and
missed interventions respe tively, sin e these terms are more des riptive for
a tive safety appli ations.
3.2 Method Properties
Di�erent methods have di�erent properties and ea h property ontributes
to the overall strength or weakness of the method. Below, relevant method
properties are de�ned and dis ussed.
Coverage
Coverage is a measure used to des ribe the degree to whi h the set of oper-
ating s enarios is evaluated. A major bene�t of theoreti al methods is that
full overage is possible to attain.
By ondu ting experiments, i.e. tests, system performan e an be eval-
uated in a hosen set of s enarios. For a omplex pro ess, generally, the set
of operating s enarios an be des ribed by an unbounded number of param-
eters. As the number of operating s enarios grows exponentially with the
number of s enario parameters, this set is very large. This e�e t, known
as the urse of dimensionality, makes full overage of the set of operating
s enarios unrealisti .
Methods for sele ting a set of s enarios to evaluate are generally referred
22
3.2. Method Properties
to as experimental design or Design of Experiments (DoE), see e.g. [40℄ for
a wide treatment of the subje t. The s enario parameter spa e may for
instan e be overed by drawing random samples, or using a more systemati
approa h, the samples may be hosen su h that the overage is evenly spread
while minimizing the number of evaluated s enarios.
Online/O�ine
Systems are evaluated either online or o�ine, where these terms are used
a ording to the following de�nitions.
De�nition 1 A system is online when for ed to exe ute in real-time.
De�nition 2 A system is o�ine when not for ed to exe ute in real-time.
Online experiments evaluate if the system omply with real-time re-
quirements but have the obvious disadvantages of not being able to exe ute
slower, or faster, than real-time.
Open/Closed-Loop
An a tive safety system monitors a pro ess and use this information to
in�uen e said pro ess, as shown in Figure 2.10. In some experiments the
s enario is partially or ompletely �xed, meaning that the system has limited
or no in�uen e on the pro ess. As a onsequen e, the following de�nitions
are useful.
De�nition 3 A system is exe uted in losed-loop when the ontrol loop
between the system and the pro ess is losed.
De�nition 4 A system is exe uted in open-loop when the ontrol loop be-
tween the system and the pro ess is open.
Note that open-loop exe ution does not equal open-loop ontrol, whi h om-
monly refers to a ontrol system operating without feedba k. Open-loop
exe ution means that the system annot in�uen e the pro ess during exe-
ution. When evaluating the orre tness of de isions, it is in many ases
su� ient to exe ute the system in open-loop. This is valid when the system
does not perform any a tion prior to the de ision to intervene, and will
onsequently not in�uen e the pro ess prior to said de ision.
23
Chapter 3. Verifi ation Methods
E� ien y
The ost in terms of time and money are measures of the method e� ien y.
Online methods are time onsuming as real-time exe ution is required and,
in general, methods involving real world experiments have higher �nan ial
ost than theoreti al analysis and omputer experiments. Also, the ost
asso iated with method development vary between di�erent methods.
Repeatability and Reprodu ibility
Repeatability and reprodu ibility are statisti al terms asso iated with a -
ura y. An experiment is repeatable if it an be performed on two di�erent
o asions with no substantial hange between measured quantities. Re-
peatability only requires this to be possible using the same personnel and
equipment. An experiment is also reprodu ible if it is repeatable using dif-
ferent personnel and equipment when performed on two di�erent o asions.
Repeatability and reprodu ibility ensures that the experiment results are
not signi� antly a�e ted by temporary fa tors.
Ground Truth Data
Ground truth data refers to information that is on�rmed in an a tual �eld
he k at a lo ation, as opposed to information a quired from a distan e.
In remote sensing, the term is ommonly used to des ribe information on-
sidered a urate, relative to information a quired from the remote sensing
system being evaluated. Ground truth data des ribes the true s enario,
e.g. how obje ts move in the s ene, and aids in evaluating sensor and on-
trol system performan e.
Model A ura y
The model a ura y is the ability of the model to generate output equivalent
to output from the real system. Model a ura y should not be onfused with
system a ura y whi h is the ability of the system to generate output with
small errors, e.g. a sensor delivering a urate measurements. If the input
to an a urate model is equivalent to the real operating onditions, the
output is realisti . Output olle ted from real systems in the real operating
environment is realisti per de�nition. A hieving high output realism from
models often indu es high ost, in the form of time, money or both.
24
3.3. Models
S enario Representativeness
For a s enario set to be representative, it must orre tly re�e t the set of op-
erating s enarios in terms of system performan e. Sampling s enarios using
the real system in the real operating environment is the most obvious way
to olle t data with high representativeness. When modeling or re reating
real s enarios, e.g. in omputer simulations or real world test environments,
limitations imposed by pro ess models or test equipment make the s enarios
less representative to a varying degree.
Pro ess Controllability
The ability to ontrol the pro ess during evaluation is referred to as pro-
ess ontrollability. La k of pro ess ontrollability is primarily an issue in
real world experiments, where ontrol of the pro ess related to for instan e
weather or multiple obje t dynami s is hallenging. Also, safety- riti al sit-
uations su h as ollisions and near- ollisions are di� ult to realize in real
world experiments where they are potentially hazardous for involved per-
sonnel and destru tive to the equipment used.
3.3 Models
Many veri� ation methods use mathemati al models to des ribe the a tive
safety system, the pro ess and their intera tion, see Figure 2.10. The om-
plexity of the models vary signi� antly and also depends on the interfa es
to other models, e.g. interfa es to per eption or a tion layer models might
require more or less omplexity in the orresponding pro ess models.
3.3.1 System Models
A tive safety systems onsist of three layers, see Figure 2.7. Commonly, the
de ision layer is an available software intera ting only with the per eption
and a tion layers. The latter two layers intera t physi ally with the pro ess,
e.g. the surrounding environment, and modeling of these layers are dis ussed
below.
Per eption Layer Models
As des ribed in Se tion 2.3, the per eption layer provides input data to the
de ision layer, based on sensor observations of a pro ess. In the per eption
layer, observations from one or several sensors are generally passed through
multiple layers of advan ed signal pro essing, fusing sensor observations into
25
Chapter 3. Verifi ation Methods
estimated states su h as positions and velo ities of dete ted obje ts. Sensor
models des ribe how the pro ess is per eived by the sensors and an be
formulated on many di�erent abstra tion levels.
Low-level sensor models des ribe the transformation between the pro ess
and the unpro essed sensor observations whereas high-level sensor models
des ribes the transformation between the pro ess and the estimated states.
Modeling the physi s of remote sensing te hnologies su h as ameras, lidars
and radars, is a omplex task, espe ially when onsidering situations where
the sensor is observing a omplex environment, see e.g. [41℄ for an overview
of low-level radar models. This is why high-level empiri al models are often
used.
A ommon high-level approa h is to model state estimates, e.g. obje t
position or velo ity, as the true estimate in�uen ed by a noise model. If
noise is ignored, the models represent ideal or perfe t sensors, as used in
e.g. [42℄. A ommon noise model is additive Gaussian noise, as used in
e.g. [43�45℄ and Paper 3. High-level models are in many ases a major
simpli� ation of the sensor and in orporate very limited information on
how the sensor errors depend on the observed pro ess. Nevertheless, they
are useful when studying systems in limited s enario sets, systems with
very a urate sensors, or the aspe ts of system performan e not a�e ted by
sensor errors.
For a omputer vision system, a low-level sensor model des ribes how
a amera per eives the pro ess, i.e. generates a sequen e of images. Te h-
niques for generating images with omputers, known as rendering, are stud-
ied in omputer graphi s. Rendering imagery requires pro ess models whi h
des ribe e.g. the 3D stru ture of obje ts in the environment. Rendered
imagery based on 3D models, in ontrast to real imagery olle ted from
ameras, is denoted virtual imagery while rendered imagery where virtual
obje ts are superimposed on real imagery is denoted augmented imagery.
In [46℄, published in 1995, it is argued that the realism of virtual imagery
is su� ient for evaluation of mobile omputer vision systems. Sin e then
omputer graphi s has evolved rapidly, as an be observed in for instan e the
video gaming and movie industries. Nonetheless, photo-realism in virtual
imagery is not easily a hieved and an overview of the multidis iplinary
hallenges of rendering is found in [47℄. Paper 4 explores the possibility
of rendering augmented imagery for o�ine evaluation of omputer vision
systems.
Online evaluation methods require image rendering in real-time, mak-
ing it more hallenging to attain high realism in rendered images. The
tra� simulation environments des ribed in [48, 49℄ have software modules
available for rendering of virtual imagery in real-time.
26
3.3. Models
A tion Layer Models
Models of e.g. braking and steering systems are needed to des ribe how
the driver and vehi le are in�uen ed by system de isions. Des riptions
and models of automotive systems and omponents, in luding a tive safety
a tuators, are thoroughly des ribed in [50℄. Note that when the system is
exe uted in open-loop, modeling the a tion layer is unne essary.
3.3.2 Pro ess Models
Modeling of the pro ess, i.e. the driver, vehi le and surrounding tra� en-
vironment, is dis ussed in the following se tions.
Driver Models
For evaluation methods based on real or augmented data, the behaviour of
the driver is in orporated in the data. Therefore, only purely model-based
methods require a driver model to generate the driver input to the vehi le,
e.g. steering and braking, based on feedba k from the vehi le, environment
and a tive safety system. Driver modeling is a wide �eld of resear h and
models are often more or less appli ation spe i� . A olle tion of papers
treating driver models in the automotive domain from a variety of perspe -
tives is found in [51℄.
Vehi le Models
Vehi le motion models are needed to des ribe both the motion of the host
vehi le as well as vehi les in the surrounding environment. Vehi le motion
is studied within the �eld of vehi le dynami s whi h is the topi of several
books, e.g. [52℄.
Environment Models
When modeling a dynami tra� environment, ea h obje t in the environ-
ment, e.g. ars, roads and pedestrians, are des ribed by individual models.
Depending on the interfa e to the a tive safety system, e.g. sensor and a -
tuator models, the environment models need to in lude di�erent aspe ts. If
low-level sensor models are used, the level of detail of the environment mod-
els is usually higher ompared to when high-level sensor models are used.
If for instan e virtual imagery is generated by a sensor model, a omplete
3D stru ture of the environment is required.
Presently, there exist several simulations environments for simulating
tra� environments in luding a tive safety systems su h as PreS an, [48℄,
27
Chapter 3. Verifi ation Methods
v-TRAFFIC, [49℄, or the Volvo Cars Tra� Simulator (VCTS), [27℄. These
softwares in lude models of driver, vehi le and environment.
3.4 Methods
This se tion des ribes di�erent types of analysis and veri� ation methods.
The methods evaluate real physi al systems, mathemati al models, or a
ombination thereof.
3.4.1 Real Driving
Online experiments using real vehi les are performed both in real tra�
and on dedi ated test tra ks. They are repeatable to some degree at test
tra ks but to a minor degree in real tra� . If the system is online, it an be
evaluated in losed-loop and sensor data often have the advantage of being
realisti .
Real Tra�
Real tra� experiments are primarily used to estimate the probability of an
unne essary intervention from a set of randomly sampled s enarios. Also,
experiments are ondu ted to estimate the probability of a missed interven-
tion, given that the tested system has relatively frequent and non-intrusive
interventions, whi h is valid for e.g. an LDW system.
Variations between di�erent vehi les and system omponents are hand-
led by using multiple vehi les and omponents in testing. For a randomly
sampled s enario set to be representative, the s enarios available for sam-
pling must also be representative. In [27, 53, 54℄, a Real World User Pro�le
(RWUP) is used to ensure that a representative s enario set is sampled, tak-
ing into a ount for instan e di�erent driving styles, weather and driving
environments.
As dis ussed in Se tion 2.6, the a eptable rate of unne essary interven-
tions for highly intrusive systems is very low, meaning that a large amount of
driving data needs to be olle ted to ensure that the requirement is ful�lled.
The obvious drawba k is that real tra� experiments are both expensive
and time onsuming. Also, ground truth data is hallenging to obtain sin e
the environment is un ontrolled.
Test Tra k
On test tra ks, spe i� types of s enarios are tested in a more ontrolled
setting. Compared to real tra� experiments, test tra k experiments of-
28
3.4. Methods
(a) (b) ( )
Figure 3.2: Non-destru tive tests in ollision and near- ollision s enarios
where (a) shows stationary pedestrian dummies of both adult and hild
size, (b) shows an in�atable moving obje t representing a moving vehi le
and ( ) shows an arti� ial obje t representing a moose.
fer a higher degree of pro ess ontrollability, repeatability and reprodu-
ibility. Motions of involved obje ts an be ontrolled to reate desired
s enarios. Also, ground truth an be obtained by e.g. positioning involved
tra� parti ipants and obje ts with an a urate positioning system su h
as Di�erential Global Positioning System (DGPS). Pro ess ontrollability
on test tra ks is better but not without limitations as for instan e weather,
e.g. snow or rain, and animals rossing the road are still di� ult to repro-
du e on demand.
When re reating ollision and near- ollision situations on test tra ks,
non-destru tive tests are preferred to ensure safety. Therefore, ollisions
are ondu ted between the host vehi le and low-mass obje ts su h as in�at-
able ars, see Figure 3.2. This reates limitations on s enarios possible to
re reate as even state-of-the-art in�atable ar or pedestrian systems annot
re reate all motions possible for real ars or pedestrians. It also degrades
the representativeness of the s enario sin e an in�atable obje t might not
be per eived by the sensors as would an equivalent real obje t.
For highly intrusive systems su h as AEB, the s enarios in whi h the sys-
tem should intervene are very rare, meaning that estimating the probability
of a missed intervention would require an unrealisti amount of driving data
from real tra� onditions. Consequently, the probability of a missed inter-
vention is often, e.g. [27, 53, 54℄, assessed by repli ating ollision situations
on test tra ks. Su h tests are also used to estimate the e�e tiveness of the
system, for instan e impa t speed redu tion.
For veri� ation purposes, s enarios in whi h an in orre t system de ision
is most likely, i.e. the worst ase s enarios, are often repli ated on test tra ks,
thus omplementing tests in real tra� . If the system behaves orre tly in
these worst ase s enarios it an be argued that less hallenging s enarios
29
Chapter 3. Verifi ation Methods
are not likely to pose a problem. Paper 1 presents a theoreti al method for
identifying the worst ase s enarios for a CA system. Examples of s enarios
most likely to ause unne essary interventions are near- ollision situations,
e.g. evasive maneuvers where the time or distan e margins to a potential
ollision are small.
3.4.2 Closed-Loop Simulations
If the pro ess is mathemati ally modeled, the system behaviour an be sim-
ulated in losed-loop with omputer generated inputs. Model-In-the-Loop
(MIL) simulations use a system model while Software-In-the-Loop (SIL)
simulations use an a tual system implementation, whi h not ne essarily
is exe uted on the produ tion hardware. The border between MIL and
SIL is sometimes hard to de�ne but examples of one or the other is found
in [27, 43, 44, 55℄.
MIL/SIL o�ers many bene�ts over real driving when omparing for in-
stan e e� ien y and pro ess ontrollability. Experiments are repeatable and
reprodu ible and these are important properties when omparing di�erent
system on�gurations. MIL/SIL are o�ine methods, meaning there are no
real-time onstraints, making it possible to simulate s enarios with speeds
limited only by the omputational power available. In addition, systems
an be tested before deployment at early stages in development, without
the need of fun tioning hardware.
If system hardware omponents are available, their performan e an
be tested online with omputer generated inputs as Hardware-In-the-Loop
(HIL). The bene�t of HIL, ompared to MIL/SIL, is that the hardware
is also evaluated. The drawba k is the online property, onstraining HIL
simulations to real-time exe ution. In [56℄ a test fa ility where a omplete
vehi le is set up on a hassis dynamometer, with robot vehi les represent-
ing the surrounding environment, is des ribed and referred to as Vehi le
Hardware-In-the-Loop (VeHIL).
3.4.3 Data Replay
In data replay methods, real re orded data is used to evaluate the system
o�ine. If real data is used ex lusively, di�erent system software on�gura-
tions an be evaluated by omputer simulations without any loss in input
data realism, as done in [57℄, but often without the impe able ground truth
data a essible using in-the-loop methods. Limited ground truth, e.g. im-
proved estimates of obje t motion, an be obtained by o�ine pro essing
of real measurement data, as done in [43, 57℄. Data replay methods are
restri ted to open-loop, sin e the s enario is �xed by the olle ted data.
30
3.5. Method Comparison
Another option is to ombine real data with model-based methods thus
generating mixed or augmented data, for example by adding new obje ts or
errors in re orded sensor data. This is exempli�ed in [43℄ where three FCW
algorithms are simulated with input data onsisting of a urate lead vehi le
motion, obtained from real data, and noise, from a radar model. Augmented
data replay has the potential to pi k the best out of two worlds but also
risk pi king the worst. The modeling e�ort ompared to purely model-
based methods is limited and many of the advantages are partly preserved,
e.g. pro ess ontrollability, or ompletely preserved, e.g. repeatability and
la k of real-time onstraints. The downside is that simulation is limited
to open-loop, as some real data is used, and that the data realism is now
dependent on a model-based data augmentation method whi h then requires
validation.
In Paper 4, an augmented data replay framework is formulated, used
for omputer vision systems. This framework uses a low-level sensor model,
dis ussed in Se tion 3.3. If instead high-level sensor models are available,
Paper 3 presents e� ient data replay methods for de ision fun tion tuning
and sensitivity analysis with regards to input perturbations, whi h an be
applied to real, model-based or augmented data.
3.4.4 Theoreti al Methods
The ultimate goal of system veri� ation is to prove that the system meets
the system requirements. Methods for proving system properties, su h as
requirement omplian e, are known as formal methods, see [58℄ for an ex-
tensive survey.
If the a tive safety system and the set of operating s enarios are de-
s ribed mathemati ally it is sometimes possible to derive analyti al expres-
sions des ribing system performan e, as done in Paper 1. Generally, this is
only possible when making quite signi� ant simpli� ations.
For dynami al systems, guarantees of not entering an undesired system
state may be obtained by omputing the set of rea hable states. Paper 2
explores the use of rea hability analysis, [59℄, and viability theory, [60℄, to
formally verify a ollision avoidan e system.
3.5 Method Comparison
This se tion provides a brief omparison of the methods presented in Se -
tion 3.4 with regards to the properties dis ussed in Se tion 3.2. An overview
of the more or less dis rete properties is presented in Table 3.1. Figures 3.3
and 3.4 ompare pro ess ontrollability, sensor data realism and e� ien y
31
Chapter 3. Verifi ation Methods
O�ine
Online
Real TrafficData Replay
Hardware-In-the-Loop
Test Track
Model/Software-In-the-Loop
Vehicle Hardware-In-the-Loop
Real Traffic
AugmentedData Replay
Process Controllability
TheoreticalAnalysis
Eff
icie
ncy
Figure 3.3: A qualitative sket h for the relation between e� ien y and
pro ess ontrollability for di�erent evaluation methods.
for di�erent methods. It should be noted that these properties are appli a-
tion dependent, meaning that the �gures should not be onsidered absolute
truths.
In Figure 3.3 it an be noted how the model-based methods are superior
Vehi le
System
Online
Closed- Ground
Repeatability Reprodu ibility
hardware loop truth
Theoreti al analysis x x x x
Real tra� x x x x
Test tra k x x x x x (x) (x)
Model/software-in-the-loop x x x x
Hardware-in-the-loop x x x x x x
Vehi le hardware-in-the-loop x x x x x x x
Real tra� data replay (x) (x) (x) x
Augmented data replay (x) (x) x x
Table 3.1: Overview of method properties for di�erent methods. The fa t
that data replay methods use vehi les and system hardware indire tly,
i.e. for initial data olle tion, is represented by a tentative "(x)". The
same notation indi ates that, on test tra ks, some aspe ts are repeatable
and reprodu ible while others are not. Ground truth for real data replay is
also tentatively marked as o�ine pro essing an o�er limited ground truth
data.
32
3.5. Method Comparison
TheoreticalAnalysis
Online
O�ineEff
icie
ncy
Sensor Data Realism
Model/Software-In-the-Loop
AugmentedData Replay
Real TrafficData Replay
Real TrafficTest Track
Vehicle Hardware-In-the-Loop
Hardware-In-the-Loop
Figure 3.4: A qualitative sket h for the relation between e� ien y and
sensor data realism for di�erent evaluation methods.
in terms of pro ess ontrollability and also in many ases are very e� ient,
largely due to the o�ine property. The major hallenge for the model-based
methods is related to the sensor model a ura y, as visualized in Figure 3.4.
For the purely model-based methods, e.g. losed-loop simulations, sensor
models generating realisti data are either unexisting or resour e demand-
ing. Methods using ex lusively real tra� data have, by de�nition, realisti
sensor data. Augmented data is relatively realisti but with the drawba k
that augmented data replay is limited to open-loop exe ution of the system,
as shown in Table 3.1.
The omparisons in Table 3.1, Figure 3.3 and Figure 3.4 learly show the
omplementary nature of the presented methods. Thus, veri� ation is often
arried out using a variety of methods, as exempli�ed in [27, 53, 54℄. Meth-
ods whi h require omplete vehi les or system hardware are onstrained
to use in the later stages of the development pro ess. Alternatively, they
may be employed with de reased predi tion a ura y using early hardware
prototypes.
33
34
Chapter 4
Summary of In luded Papers
This hapter provides a brief summary of the papers in luded in the thesis
and also des ribes the ontributions to ea h paper by the author of this
thesis. Full versions of the papers are in luded in Part II.
Paper 1
J. Nilsson, A. Ödblom and J. Fredriksson, Worst Case Analy-
sis of Automotive Collision Avoidan e Systems, submitted for
possible journal publi ation.
As dis ussed in Se tion 2.6, the set of tra� s enarios whi h generates the
input to an a tive safety de ision fun tion is very large. This paper theo-
reti ally identi�es s enarios with a high risk of in orre t system de isions,
i.e. the worst ase s enarios. The main hallenge with this approa h, as
dis ussed in Chapter 3, is to model system and s enarios in su h a way
that performan e an be des ribed analyti ally while still in luding the key
fa tors a�e ting performan e, e.g. sensor errors or obje t motion.
The key idea of this paper is to theoreti ally investigate the fundamental
limitations of a ollision avoidan e system, subje t to systemati measure-
ment errors and unexpe ted future obje t motion, in terms of early and
unne essary interventions. Spe i� ally, we in lude e�e ts of sensor and a -
tuator delays, and derive losed-form expressions for the worst ase perfor-
man e, with regards to longitudinal or lateral predi tion and measurement
errors. For a system example, numeri al results show how de ision timing
and robustness depend on s enario and system parameters. The method an
be used for system veri� ation, tuning or sensitivity analysis with regards
to s enario variations and sensor errors. Also, s enarios with inadequate
performan e an be identi�ed, thus improving existing test methods by di-
re ting testing and analysis e�orts towards relevant s enarios.
35
Chapter 4. Summary of In luded Papers
The thesis author was responsible for the problem formulation, deriva-
tion of the losed-form expressions, implementation and writing the paper.
Paper 2
J. Nilsson, J. Fredriksson and A. Ödblom, Veri� ation of Colli-
sion Avoidan e Systems using Rea hability Analysis, submitted
as invited paper to the 19th IFAC World Congress, Cape Town,
South Afri a, 2014.
The losed-form expressions for performan e derived in Paper 1 are very
useful from a veri� ation perspe tive but for many omplex a tive safety
de ision fun tions, they are not possible to derive. The alternative of eval-
uating state traje tories, as done in traditional simulations and real vehi le
tests, does not provide guarantees for system performan e for all possible
state traje tories.
To address these limitations, Paper 2 des ribes a novel set-based frame-
work for analyzing under what onditions the absen e of in orre t de isions
may be guaranteed for a given ollision avoidan e de ision fun tion. Rea h-
ability analysis and viability theory are used to ompute unsafe and safe
sets, i.e. sets where an ideal system should or should not intervene respe -
tively. In these sets, in orre t de isions for a given de ision fun tion are
identi�ed using optimization te hniques. By separating the dynami s of
the input spa e from the de ision fun tion, non-linear and ad-ho de ision
fun tions are e� iently handled in the proposed framework.
The method is demonstrated on a ollision avoidan e system example
and, given the models used and absen e of measurements errors, we show
that the system does not make in orre t de isions. Furthermore, we des ribe
and demonstrate how to evaluate the robustness to measurement errors,
using the proposed framework.
The thesis author was responsible for the problem formulation, develop-
ment of the proposed methods, implementation and writing the paper.
Paper 3
J. Nilsson and M. Ali, Sensitivity Analysis and Tuning for A tive
Safety Systems, in Pro eedings of the 13th International IEEE
Conferen e on Intelligent Transportation Systems, 2010, pages
161-167, Madeira Island, Portugal.
Papers 1 and 2 are full overage methods, i.e. are on erned with veri�-
ation of the omplete s enario parameter spa e. Full overage methods
36
are desirable but set limitations on the omplexity of the involved math-
emati al models. In ontrast, Paper 3 onsiders veri� ation given that a
representative experimental data set is available.
The design and tuning of an a tive safety de ision fun tion, e.g. how
thresholds are pla ed, will de ide how sensitive the system performan e is
to input errors. Investigating the interplay between input errors, de ision
fun tion and system performan e gives rise to three relevant questions:
i. Given a de ision fun tion and input errors, what is the system perfor-
man e?
ii. Given a de ision fun tion and system performan e requirements, what
are the input requirements?
iii. Given input errors and system performan e requirements, how should
the de ision fun tion be tuned?
This paper proposes a framework for open-loop analysis of de ision fun -
tions, with regards to the above mentioned questions. By introdu ing a ro-
bustness measure, des ribing the robustness to input errors for the de ision
fun tion, e� ient o�ine methods are formulated. The robustness measure
is independent of the input errors, meaning that it needs to be estimated
only on e for ea h de ision fun tion and data set. This allows for e� ient
evaluation of the system performan e as ombinations of de ision fun tion
and input errors an be pro essed without evaluating the de ision fun tion
output for ea h ombination. The framework is applied to data olle ted
in an experimental setting. Also, it is demonstrated how it an be used for
setting input requirements and tuning the de ision fun tion.
The formulation of the presented framework and writing the paper were
jointly ondu ted by both authors of the paper. The author of this thesis is
responsible for the demonstration of the framework while the se ond author
is responsible for the olle tion of experimental data and development of
the de ision fun tion example.
Paper 4
J. Nilsson, A. Ödblom, J. Fredriksson, A. Zafar and F. Ahmed,
Performan e Evaluation Method for Mobile Computer Vision
Systems using Augmented Reality, in Pro eedings of the IEEE
Virtual Reality Conferen e, 2010, pages 19-22, Waltham, Mas-
sa husetts, USA.
37
Chapter 4. Summary of In luded Papers
The methods for analyzing de ision fun tions in Papers 1-3, all rely on
a urate modeling of sensor errors. In Paper 4, a novel framework using
augmented imagery is proposed for determining sensor errors of omputer
vision systems, whi h are widely used in a tive safety systems. The proposed
framework exploits the possibility to add virtual agents into a real data
sequen e olle ted in an unknown environment, thus making it possible to
e� iently reate augmented data sequen es, in luding ground truth, to be
used for performan e evaluation. Varying the ontent in the data sequen e
by adding di�erent virtual agents is straightforward, making the proposed
framework very �exible.
The method has been implemented and tested on a pedestrian dete -
tion system used for ollision avoidan e. Preliminary results show that the
method has the potential to repla e and omplement physi al testing, for
instan e by reating ollision s enarios, whi h are di� ult to test in reality.
The formulation of the novel framework was jointly ondu ted by the
�rst two authors of the paper. The author of this thesis was also responsible
for writing the paper and supervising the ase study implementation done
by authors four and �ve.
Paper 5
J. Nilsson, J. Fredriksson and A. Ödblom, Reliable Vehi le Pose
Estimation using Vision and Single-Tra k Model, submitted for
possible journal publi ation.
The method in Paper 4 relies on an a urate 3D re onstru tion of the am-
era motion in six Degrees of Freedom (6-DoF). Extensive use of this method
requires this to be done without adding additional expensive sensors to the
vehi le. The ore idea of Paper 5 is to use a single-tra k vehi le model in a
lo al bundle adjustment framework to improve the pose estimates obtained
from a standard vehi le sensor setup, i.e. a forward looking mono ular am-
era, wheel speed, yaw rate and steering wheel angle sensors. This means
pose estimates are optimized not only with regards to observed image fea-
tures, but also with respe t to a single-tra k vehi le model and standard
in-vehi le sensors.
The des ribed method has been tested experimentally on hallenging
data sets at both low and high vehi le speeds as well as on a data set with
moving obje ts. The vehi le motion model in ombination with in-vehi le
sensors exhibit good a ura y in estimating planar vehi le motion. Results
show that this property is preserved when ombining these information
sour es with vision. Furthermore, the a ura y obtained from vision-only in
38
dire tion estimation is improved, primarily in situations where the mat hed
visual features are few.
The thesis author was responsible for the problem formulation, devel-
opment of algorithms, implementation, experimental validation and writing
the paper.
Paper 6
J. Nilsson, P. Andersson, I. Gu and J. Fredriksson, Augmented
Training Data for Pedestrian Dete tion, submitted to the 22nd
International Conferen e on Pattern Re ognition, Sto kholm,
Sweden, 2014.
Ma hine learning te hniques are widely used in omputer vision to train
obje t lassi�ers. In many appli ations, e.g. pedestrian dete tion, the dom-
inating approa h in literature is to use supervised learning, e.g. Support
Ve tor Ma hines (SVM), to train a lassi�er using labelled data. This la-
belled data is hosen su h that it represents the environment where the
lassi�er will be used. Thus, for a mobile system operating in a omplex
and un ontrolled environment, e.g. a ar, the training data set must on-
tain a great amount of variation. Colle ting and manually labelling large
amounts of data is an expensive and time onsuming pro ess.
In Paper 6, we propose to repla e or omplement real data with aug-
mented data, using the method presented in Paper 4. Augmented data an
be automati ally labelled while still exhibiting a real, and onsequently real-
isti , ba kground. The proposed solution is evaluated by training pedestrian
lassi�ers using one of the gold-standard methods in pedestrian lassi� a-
tion, spe i� ally a linear SVM and the Histogram of Oriented Gradients
(HOG), [61℄. Experimental validation is performed on real data sets and
the results are ompared to performan e obtained using real training data.
The thesis author was responsible for the problem formulation and writ-
ing the paper. The design of experiments was ondu ted jointly by the
author of this thesis and the se ond author of the paper. Note that the
development and implementation of algorithms were primarily the respon-
sibility of the se ond author, and not the author of this thesis.
39
40
Chapter 5
Con luding Remarks
This hapter states the most important ontributions and provides re om-
mendations for future resear h.
5.1 Contributions
System veri� ation of an automotive safety system must assess the orre t-
ness of system de isions in a vast array of tra� s enarios. These de isions
are based on remote sensing of the surrounding environment and onse-
quently, in luding sensors in the analysis and veri� ation methods is ru ial.
Computational methods have the potential to signi� antly improve the ver-
i� ation pro ess in terms of e.g. e� ien y and overage. This thesis fo us
on omputational methods for both de ision fun tion analysis, in luding
the dependen e on sensor errors, and methods for determining these sensor
errors.
Related to de ision fun tion analysis and veri� ation, the main ontri-
butions of this thesis are:
• Derivation of losed-form expressions for the worst ase de ision tim-
ing, in the presen e of predi tion and measurement errors, for a ol-
lision avoidan e system example. Also, losed-form expressions are
derived for robust avoidan e s enarios, i.e. s enarios whi h are guar-
anteed not to exhibit an unne essary intervention. These results are
presented in Paper 1.
• A novel set-based framework for analyzing under what onditions the
absen e of in orre t de isions may be guaranteed for a given a tive
safety de ision fun tion. In ontrast to evaluating state traje tories,
rea hability analysis and viability theory are used to ompute unsafe
and safe sets, in whi h absen e of in orre t de isions and robustness to
41
Chapter 5. Con luding Remarks
sensor errors may be guaranteed using optimization te hniques. This
framework is presented in Paper 2 and forms a generalization of the
work shown in Paper 1.
• A framework for a tive safety de ision fun tion analysis using re orded
or simulated data. E� ient methods for system performan e evalua-
tion are derived and these an be used to analyze the de ision fun tion
sensitivity to input errors, or for de ision fun tion tuning. This frame-
work is presented in Paper 3.
Related to performan e evaluation of omputer vision systems, the main
ontributions of this thesis are:
• A novel performan e evaluation approa h using augmented imagery
for evaluation of mobile omputer vision systems. Performan e is
evaluated in ollision and near- ollision s enarios, safely and non-
destru tively, while still using a real image ba kground from re orded
data. This on ept is presented in Paper 4 and the use of augmented
data is extended from performan e evaluation to training of a pedes-
trian lassi�er in Paper 6.
• An approa h for 6-DoF vehi le pose estimation using a single vehi le-
based standard amera. Visual features are omplemented by stan-
dard in-vehi le sensors and a single tra k vehi le model in a bundle
adjustment framework. The method has been validated experimen-
tally in hallenging situations at both low and high vehi le speeds.
This method is presented in Paper 5 and is an important module
needed for the framework introdu ed in Paper 4.
5.2 Dire tions of Future Resear h
There is a great need for more e� ient veri� ation methods to handle the
hallenges asso iated with future automotive safety systems. The work
presented in this thesis has inspired multiple ideas on this topi .
Sensor error models
To make full use of the theoreti al methods for performan e estimation,
presented in Papers 1-3, a urate sensor error models are needed. This
requires a quiring and pro essing large amounts of sensor data, with asso-
iated ground truth, but also proper hoi es of model stru tures. The pre-
sented framework for sensor evaluation using augmented data may prove to
be a valuable resour e.
42
5.2. Dire tions of Future Resear h
Extending rea hability methods
The dynami al models used in Paper 2 are linear and low-dimensional,
handling only a single moving obje t. Applying existing methods for rea h-
abality analysis of more omplex systems is an interesting approa h. This
ould enable the analysis of the same problem with more omplex vehi le
dynami s models and/or multiple obje ts.
Augmenting other sensors
The augmentation framework in Paper 4 has been applied primarily on
image data. Many safety systems fuse information from di�erent sensor
te hnologies, e.g. radar, laser. Thus, a natural extension would be to ex-
tend the on ept to in lude also other sensor types. This requires in-depth
knowledge of the sensor te hnology to be added and also a urate and de-
tailed modeling of the spe i� sensor used.
43
44
Referen es
[1℄ �Global status report on road safety 2013: supporting a de ade of a -
tion,� World Health Organization (WHO), Geneva, Switzerland, 2013.
[2℄ G. Ja obs, A. Aeron-Thomas, and A. Astrop, �Estimating global road
fatalities,� Transport Resear h Laboratory, Global Road Safety Part-
nership (GRSP), TRL445, 2000.
[3℄ �IRTAD Annual Report 2009,� International Tra� Safety Data &
Analysis Group (IRTAD), International Transport Forum, 2010.
[4℄ (2013, De .) OECD Statisti s. OECD. [Online℄. Available:
http://stats.oe d.org/
[5℄ F. Winston and R. Menon, �Tra� Mortality in Germany Before, Dur-
ing, and After Reuni� ation,� Annual Pro eedings / Asso iation for the
Advan ement of Automotive Medi ine, pp. 239�250, 1999.
[6℄ �Towards Zero: Ambitious Road Safety Targets and the Safe System
Approa h,� OECD, International Transport Forum, OECD Publishing,
2008.
[7℄ M. Ali, �De ision Making and Control for Automotive Safety,� PhD
Thesis, No 3413, ISSN 0346-718X, Chalmers University of Te hnology,
Göteborg, Sweden, 2012.
[8℄ W. Najm, B. Sen, J. Smith, and B. Campbell, �Analysis of Light Vehi-
le Crashes and Pre-Crash S enarios Based on the 2000 General Esti-
mates System,� U.S. Department of Transportation, National Highway
Tra� Safety Administration, John A. Volpe National Transportation
Systems Center Cambridge, DOT HS 809 573, 2003.
[9℄ �Tra� Safety Fa ts 2011 - A Compilation of Motor Vehi le Crash
Data from the Fatality Analysis Reporting System and the General
Estimates System,� National Highway Tra� Safety Administration,
National Center for Statisti s and Analysis, U.S. Department of Trans-
portation, DOT HS 811 754, Washington, DC, USA, 2012.
45
Referen es
[10℄ �Vägtra�kskador 2012 (Road tra� injuries 2012),� Tra�kanalys,
Statistik 2013:9, Sto kholm, Sweden, 2013.
[11℄ J. Treat, N. Tumbas, S. M Donald, D. Shinar, R. Hume, R. Mayer,
R. Stansifer, and N. Castellan, �Tri-level Study of the Causes of Tra�
A idents: Exe utive Summary,� U.S. Department of Transportation,
National Highway Tra� Safety Administration (Contra t No. DOT
HS 034-3-535), DOT HS 805 099, Washington, DC, 1979.
[12℄ V. L. Neale, T. A. Dingus, S. G. Klauer, J. Sudweeks, and M. Good-
man, �An overview of the 100- ar naturalisti study and �ndings,� in
Pro eedings of the 19th International Te hni al Conferen e on the En-
han ed Safety of Vehi les, 2005, pp. paper 05�0400.
[13℄ T. A. Dingus, S. Klauer, V. L. Neale, A. Petersen, S. E. Lee, J. Sud-
weeks, M. A. Perez, J. Hankey, D. Ramsey, S. Gupta, C. Bu her, Z. R.
Doerzaph, J. Jermeland, and R. Knipling, �The 100-Car naturalisti
driving study: Phase II - Results of the 100-Car �eld experiment,�
U.S. Department of Transportation, National Highway Tra� Safety
Administration, DOT HS 810 593, Washington, DC, USA, 2006.
[14℄ L. Palkovi s, A. Semsey, and E. Gerum, �Roll-Over Prevention System
for Commer ial Vehi les - Additional Sensorless Fun tion of the Ele -
troni Brake System,� Vehi le System Dynami s, vol. 32, no. 4-5, pp.
285�297, Nov. 1999.
[15℄ J. Lu, D. Messih, and A. Salib, �Roll Rate Based Stability Control-The
Roll Stability Control System,� in Pro eedings of the 20th Enhan ed
Safety of Vehi les Conferen e (ESV), paper 07-0136, Lyon, Fran e,
2007.
[16℄ R. Rajamani, Vehi le Dynami s and Control. Springer, 2006.
[17℄ M. Bayly, B. Fildes, M. Regan, and K. Young, �Review of rash ef-
fe tiveness of Intelligent Transport Systems,� TRACE Proje t, No.
027763, Deliverable D4.1.1 - D6.2, 2007.
[18℄ R. Bishop, Intelligent Vehi le Te hnology and Trends. Norwood, MA,
USA: Arte h House, 2005.
[19℄ R. K. Jurgen, Adaptive Cruise Control. SAE International, 2006.
[20℄ D. LeBlan , G. Johnson, P. Venhovens, G. Gerber, R. DeSonia,
R. Ervin, A. Ulsoy, and T. Pilutti, �CAPC: A Road-Departure Pre-
vention System,� IEEE Control Systems Magazine, vol. 16, no. 6, pp.
43�60, 1996.
46
Referen es
[21℄ M. Ruder, W. Enkelmann, and R. Garnitz, �Highway Lane Change
Assistant,� in IEEE Intelligent Vehi le Symposium. IEEE, 2002, pp.
240�244.
[22℄ A. Eidehall, J. Pohl, F. Gustafsson, and J. Ekmark, �Toward Autono-
mous Collision Avoidan e by Steering,� IEEE Transa tions on Intelli-
gent Transportation Systems, vol. 8, no. 1, pp. 84�94, Mar. 2007.
[23℄ W. Birk, M. Brännström, and D. Levin, �Method for determining a
measure for evaluating the behaviour of a driver of a vehi le,� Patent,
EP1674375, European Patent O� e, 2006.
[24℄ S. S. Bla kman and R. Popoli, Design and analysis of modern tra king
systems. Norwood, MA: Arte h House, 1999.
[25℄ B. Risti , S. Arulampalam, and N. Gordon, Beyond the Kalman �lter :
parti le �lters for tra king appli ations. Boston, Mass.: Arte h House,
2004.
[26℄ M. Darms and H. Winner, �A Modular System Ar hite ture for Sensor
Data Pro essing of ADAS Appli ations,� in IEEE Intelligent Vehi les
Symposium. IEEE, 2005, pp. 729�734.
[27℄ E. Coelingh, H. Lind, W. Birk, and D. Wetterberg, �Collision Warning
with Auto Brake,� in FISITAWorld Congress, Yokohama, Japan, 2006.
[28℄ A. Poly hronopoulos, M. Tsogas, A. Amditis, and L. Andreone, �Sensor
Fusion for Predi ting Vehi les' Path for Collision Avoidan e Systems,�
IEEE Transa tions on Intelligent Transportation Systems, vol. 8, no. 3,
pp. 549�562, Sep. 2007.
[29℄ F. Bengtsson and L. Danielsson, �A design ar hite ture for sensor data
fusion systems with appli ation to automotive safety,� in Intelligent
Transportation Systems World Congress, New York, USA, 2008.
[30℄ (2014, Jan.) DARPA Urban Challenge. Defense Advan ed
Resear h Proje ts Agen y (DARPA). [Online℄. Available:
http://ar hive.darpa.mil/grand hallenge/index.asp
[31℄ �Tra� Safety Fa ts 2008 - A Compilation of Motor Vehi le Crash
Data from the Fatality Analysis Reporting System and the General
Estimates System,� National Highway Tra� Safety Administration,
National Center for Statisti s and Analysis, U.S. Department of Trans-
portation, DOT HS 811 170, Washington, DC, USA, 2009.
47
Referen es
[32℄ M. Ri hter, H.-C. Pape, D. Otte, and C. Krettek, �Improvements in
passive ar safety led to de reased injury severity - a omparison be-
tween the 1970s and 1990s.� Injury, vol. 36, no. 4, pp. 484�8, Apr.
2005.
[33℄ (2014, Jan.) The European New Car Assessment Programme
(EuroNCAP). [Online℄. Available: http://www.euron ap. om/
[34℄ D. Burton, A. Delaney, S. Newstead, D. Logan, and B. Fildes, �Ef-
fe tiveness of ABS and vehi le stability ontrol systems,� Report No.
04/01, Royal Automobile Club Of Vi toria (RACV), 2004.
[35℄ S. A. Ferguson, �The e�e tiveness of ele troni stability ontrol in re-
du ing real-world rashes: a literature review.� Tra� injury preven-
tion, vol. 8, no. 4, pp. 329�38, De . 2007.
[36℄ M. Lindman, A. Ödblom, E. Bergvall, A. Eidehall, B. Svanberg, and
T. Lukaszewi z, �Bene�t estimation model for pedestrian auto brake
fun tionality,� in Pro eedings of the ESAR Conferen e, no. 300961,
2010.
[37℄ K. Kusano and H. Gabler, �Safety bene�ts of forward ollision warning,
brake assist, and autonomous braking systems in rear-end ollisions,�
IEEE Transa tions on Intelligent Transportation Systems, vol. 13,
no. 4, pp. 1546�1555, De . 2012.
[38℄ E. L. Lehmann, Testing statisti al hypotheses, 2nd ed. Pa i� Grove,
Calif.: Wadsworth & Brooks/Cole Advan ed Books & Software, 1991.
[39℄ J. Neyman and E. S. Pearson, �On the Use and Interpretation of
Certain Test Criteria for Purposes of Statisti al Inferen e: Part I,�
Biometrika, vol. 20A, no. 1/2, pp. 175�240, 1928.
[40℄ R. O. Kuehl, Design of Experiments: Statisti al Prin iples of Resear h
Design and Analysis, 2nd ed. Pa i� Grove : Duxbury-Thomson
Learning, 2000.
[41℄ L. Danielsson, �Tra king and radar sensor modelling for automotive
safety systems,� PhD Thesis, No 3064, ISSN 0346-718X, Chalmers Uni-
versity of Te hnology, Göteborg, Sweden, 2010.
[42℄ J. Hillenbrand, A. M. Spieker, and K. Kros hel, �A Multilevel Collision
Mitigation Approa h - Its Situation Assessment, De ision Making, and
Performan e Tradeo�s,� IEEE Transa tions on Intelligent Transporta-
tion Systems, vol. 7, no. 4, pp. 528�540, De . 2006.
48
Referen es
[43℄ P. Zheng and M. M Donald, �The e�e t of sensor errors on the perfor-
man e of ollision warning systems,� in IEEE Intelligent Transportation
Systems Conferen e, vol. 1. Pis ataway, NJ, USA: IEEE, 2003, pp.
469�474.
[44℄ L. Yang, J. Yang, E. Feron, and V. Kulkarni, �Development of a
performan e-based approa h for a rear-end ollision warning and avoid-
an e system for automobiles,� in IEEE Intelligent Vehi les Symposium,
2003, pp. 316�321.
[45℄ O. Gietelink, B. De S hutter, and M. Verhaegen, �Probabilisti valida-
tion of advan ed driver assistan e systems,� in Pro eedings of the 16th
IFAC World Congress, vol. 19, Prague, Cze h Republi , 2005.
[46℄ W. Burger, M. Barth, and W. Sturzlinger, �Immersive Simulation for
Computer Vision,� in joint 19th AGM and 1st SDRV workshop Visual
Modules. Maribor, Slovenia: Oldenbourg Press, 1995, pp. 160�168.
[47℄ A. S. Glassner, Prin iples of Digital Image Synthesis, 1st ed. Morgan
Kaufmann, 1995.
[48℄ (2014, Jan.) PreS an. TASS International. [Online℄. Available:
https://www.tassinternational. om/pres an
[49℄ (2014, Jan.) v-TRAFFIC. VIRES Simulationste hnologie GmbH.
[Online℄. Available: http://www.vires. om/Produ ts_ToolChain.htm
[50℄ Bos h Automotive Handbook, 7th ed. Robert Bos h GmbH, 2007.
[51℄ P. C. Ca iabue, Ed., Modelling Driver Behaviour in Automotive En-
vironments. Springer, 2007.
[52℄ H. Pa ejka, Tyre and Vehi le Dynami s, 2nd ed. Elsevier Ltd, 2006.
[53℄ R. Kiefer, D. Leblan e, M. Palmer, J. Salinger, R. Deering, and
M. Shulman, �Development and Validation of Fun tional De�nitions
and Evaluation Pro edures for Collision Warning/Avoidan e System,�
National Highway Tra� Safety Administration, DOT HS 808 964,
Washington, DC, USA, 1999.
[54℄ M. Distner, M. Bengtsson, T. Broberg, and L. Jakobsson, �City Safety
- A system addessing rear-end ollisions at low speeds,� in Pro eedings
of the 21st International Te hni al Conferen e on the Enhan ed Safety
of Vehi les (ESV), Stuttgart, Germany, 2009.
49
Referen es
[55℄ J. Hillenbrand and K. Kros hel, �A Study on the Performan e of Un o-
operative Collision Mitigation Systems at Interse tion-like Tra� Sit-
uations,� in IEEE Conferen e on Cyberneti s and Intelligent Systems.
IEEE, Jun. 2006, pp. 1�6.
[56℄ O. Gietelink, J. Ploeg, B. De S hutter, and M. Verhaegen, �Develop-
ment of advan ed driver assistan e systems with vehi le hardware-in-
the-loop simulations,� Vehi le System Dynami s, vol. 44, no. 7, pp.
569�590, Jul. 2006.
[57℄ K. Lee and H. Peng, �Evaluation of automotive forward ollision warn-
ing and ollision avoidan e algorithms,� Vehi le System Dynami s,
vol. 43, no. 10, pp. 735�751, O t. 2005.
[58℄ J. Wood o k, P. G. Larsen, J. Bi arregui, and J. Fitzgerald, �Formal
methods: Pra ti e and Experien e,� ACM Computing Surveys, vol. 41,
no. 4, pp. 1�36, O t. 2009.
[59℄ I. Mit hell, �Comparing forward and ba kward rea hability as tools
for safety analysis,� in 10th International Workshop, Hybrid systems:
omputation and ontrol. Pisa, Italy: Springer Berlin Heidelberg, 2007,
pp. 428�443.
[60℄ J.-P. Aubin, A. M. Bayen, and P. Saint-Pierre, Viability Theory - New
Dire tions, 2nd ed. Berlin, Heidelberg: Springer Berlin Heidelberg,
2011.
[61℄ N. Dalal, �Finding people in images and videos,� PhD Thesis, Institut
National Polyte hnique de Grenoble / INRIA Rh�ne-Alpes, Grenoble,
2006.
50