FeedMAP Validation Results

Final Workshop 9.10.2008 Trento, Italy

Dr. Bernd Thomas Dr. Jan LöwenauBMW GroupForschung und Technik

Focus of Technical Validation

Validation is ensuring "you built the right product"

Validation is testing to confirm that it satisfies user's needs.

From Specification to Testing to Validation

RequirementsSystem Tests

Test Evaluation

Validation

Specification of Functionaland Non Functional Requirements

Definition of Validation Indicators for most important Requirements

Definition of Indicator Test Methods and Quality Metrics

Analysis and Interpretationof Indicator Test Results

ValidationPlan

TestReport

ValidationReport

RequirementsSpecification

FeedMAP Client Validation Indicators

FR-7 FMC

FR-8 FMC

FR-9 FMC

Det Ind1Is FMC able to detect relevant

deviations?

FR-17 FMC

Det Ind2Are certain FMC able to

detect local warning events?

Comm Ind1Is FMC able to generate

a MDR when a deviation is detected?

FR-16 FMC

Comm Ind3Is FMC able totransmit MDRto the FMSC?

Comm Ind4Is FMSC able to

transmit MDAto the MC?

FR-19 FMC

Comm Ind5Are all MDRs successfully

received at FMSC?

Comm Ind6Are all MDAs successfully

received at MC?

NFR-7

Tim Ind2Do MDR transactions

complete at an acceptableamount of time?

FR-13 FMC FR-11 FMC

QPar Ind1How good are the quality parameters

of MDRs?

Det Ind2Can Public Authorities

provide local warning events?

Detection Communication Quality Timing

FeedMAP Service Centre Validation Indicators

FR-23 FMSC

Comm Ind2Is FMSC able to generate an MDA whenever a conclusion

on a potential mapdeviation has been reached?

FR-21 FMSC FR-22 FMSC

FSCProc Ind1Is FMSC able to pre-process

MDRs according to the specified quality requirements?

FSCProc Ind2Can FMSC increase quality

of deviation data by clustering and statistical

analyses of the MDRs?

NFR-8

Tim Ind3How long does FMSC

take to issue analert (to MC) since the

moment that a deviation actually took place?

FR-24 FMSC

QPar Ind2How good are the

quality parameters of MDAs?

NFR-6

Tim Ind1Can local warning messages reach

FMCs in time?

Communication Quality TimingFMSC Processing

Measures and Metrics for Detection Validation

a positive alarm with correct semantic message content irrespective of geometric accuracy•Semantically Correct (PCA)

a positives alarm with correct semantic message but geometrically inaccurate•Geometrically Inaccurate Alarms (PIAA)

a positive alarm qualified as a valid one according to certain quality criteriaPositive Valid Alarm (PVA)

the total number of alarms reported by the FMC: TA = PA + FATotal Alarms (TA)

the deviations that are known and that are to be detected: TD = MA + PA Reference Deviations (TD)

the FMC reports a MDR for which a reference deviation exists – irrespective of the quality of the message content, i.e. the geometric accuracy or the deviation type may be wrong or right!Positive alarm (PA)

the FMC does not report a MDR for an existing deviation. (i.e. there is a deviation, but no report is generated by the FMC)Missing alarm (MA)

the FMC reports a MDR for a non existing deviation. (i.e. there is no deviation, but the FMC reports one)False Alarm (FA)

ExplanationTerminology

Completeness Rate (CMPR): CMPR = PA / TD

Classification Completeness Rate:CCMPR = PVA / TD

Classification Correctness Rate:CCORR = PVA / TA

Classification Accuracy Rate:CACCR = PVA/(MA+PA+FA)

Classification Quality Approach

Semantic Correctness Rate (SCORR):SCORR = PCA / TD

Semantic Correctness & Accuracy Rate (CSCACCR):CSCACCR = (PCA – PIAA) / TD

Geometric Accuracy (GACCR): e.g. mean distance metric of geometrical object in question

Geodetic Information Quality Approach

Before going into the results for measures and metrics…

Our quantitative results depend strongly on– algorithms, sensors etc.. but also– reference deviations used for testing

» The ‚network context‘ is important: Complex topologies as often the case in cities (‚difficult‘detection) vs. Simple network structures on rural roads (easy detection)

» The magnitude of the errors: large deviations can be more simply detected than small ones

» Sample size, etc.

Test samples in different test sites are different!– e.g. PTV‘s has a focus on the understanding the limits of small scale, innercity

errors. Using as corresponding deviation sample for testing yields ‚poorer results‘ than more ‚favourable‘ deviation samples

Quantitative results across all test sites provide insights with regards to feasibility, but are not

meant as an ‚objective‘ performance measure!

Validation Results on Detection: Wrong Road Geometry

1311563%85%84(11 devs, 7or14

loops dep. on error)

GPS, diff. odometer

Italy(CRF / MMSE)

151866%66%44GPSGothenburg(NAVIGON / VTEC)

15

8

6

FalseAlarms

2131%40%35GPSStuttgart (PTV)

1185%95%240(6 devs, 40 loops)

GPS, gyro, odometer

Stuttgart (DAI)

64%

CCMPR(PVA / TD)

1884%50GPS, gyro, odometer

Munich (BMW / NAVTEQ)

Missing Alarms

CMPR(PA / TD)

ReferenceDeviations

SensorsWRONG GEOMETRY

Wrong Road Geometry is a quite common deviation which can be found in real world due to road improvements changing the road geometry .The geometry modifications can be short but also very long and can have a small or big distance from the original road. It is therefore difficult to define a threshold for deviation detection able to detect Wrong Road Geometry in all instances. Because of several erroneous sources: GPS accuracy, relative and absolute map error. New positioning algorithms and technologies such as Galileo could improve this issue.

Validation Results on Detection: MISSING ROAD

22881%97%77(11 devs, 7 loops)

GPS, diff. odometer

Italy(CRF / MMSE)

1290%90%10GPSGothenburg(NAVIGON / VTEC)

2

1

FalseAlarms

657%84%37GPSStuttgart (PTV)

0100%100%80(2 devs, 40 loops)

GPS, gyro, odometer

Stuttgart (DAI)

CCMPR(PVA / TD)

Missing Alarms

CMPR(PA / TD)

ReferenceDeviations

SensorsMISSING ROAD

Missing Road Detection in general achieved very good resultsSometimes not clear when to consider a road to be a new one or a road with changed geometryUse of standard map matchers for detection sometimes leads to unintended interruption of new road geometry detection because of false matching to near by roads.More elaborated algorithms and modification of standard map matching algorithms will provide improved results.

Validation Results on Detection: Prohibited Turn and Wrong One Way

2

0

FalseAlarms

1345%58%31GPSStuttgart (PTV)

0100%100%80(2 devs, 40 loops)

GPS, gyro, odometer

Stuttgart (DAI)

CCMPR(PVA / TD)

Missing Alarms

CMPR(PA / TD)

ReferenceDeviations

SensorsProhibitedTurn

000%100%17GPS, diff. odometer

Italy(CRF / MMSE)

2

1

FalseAlarms

757%77%30(2 devs, 15 loops)

GPSStuttgart (PTV)

198%98%80(2 devs, 40 loops)

GPS, gyro, odometer

Stuttgart (DAI)

CCMPR(PVA / TD)

Missing Alarms

CMPR(PA / TD)

ReferenceDeviations

SensorsWrongOne Way

Prohibited Turn Detection The false alarm rate is very low for prohibited turn detection. In rural areas, especially at ramps, the detection rate and the correctness rate is very high. In urban areas, the detection rate is not so good due to the use of standard map matcher and false map matching to near by roads.

Wrong One Way Detection By entering a One way road in false direction, a detection of a virtually “Missing Road” is generated instead of the wrong one way. This behaviour could be corrected with a better detection algorithm checking the existence of the road in the other direction.

Validation Results on Detection: Speed Limit, Slope, Travel Time

22 (no radar) 2

9

False Alarms

31 - 43%48GPS, (± radar)Gothenburg(NAVIGON / VTEC)

74%

CCMPR(PVA / TD)

50GPS, gyro, odometer,radar

Munich (BMW / NAVTEQ)

ReferenceDeviations

SensorsSpeedLimit

3 tracks (12km, 7km,32km) => 380 road segments in digital mapExtended coverage of slope information with respect to accuracy (0.2%) and number of slope values per road segment in average +2

GPS, Transmission Control Unit

Gothenburg(NAVIGON / VTEC)

Explanation / ResultsSensorsSlope

11

FalseAlarms

76%

CCMPR(PVA / TD)

50GPS, gyro, odometer

Munich (BMW / NAVTEQ)

ReferenceDeviations

SensorsTravel Time

Speed Limit Changes

3

32

0

11

02

0

5

10

15

20

25

30

35

10 km/h 20 km/h 30 km/h 40 km/h 50 km/h 60 km/h

Speed Limit Detection Assuming the most conservative results are realistic, it is feasible to detect this deviation using FeedMAP probes.The small percentage of map problems detected, translates into large absolute numbers if operated in practice with big number of detection clients. With introduction of advanced systems such as sign post recognition cameras, one can expect the detection rate will be increased dramatically.

Summary Validation Results FMC Det Ind 1

6 Automatically detected deviation types validated according to quality metrics:

– FMC issued alarms in average for 78% of existing reference deviations (CMPR)

– FMC detected in average 72% of deviation types correctly (CCMPR)

– Low False Alarm Rate in average < 8%

Slope Detection showed improved quality

Manual POI Detection 100% successful

0%

20%

40%

60%

80%

100%

Deviation Type

Average Completness Rates - All Test Sites

CMPR 74% 93% 79% 91% 59% 76%

CCMPR 62% 82% 73% 79% 59% 76%

Wrong Road

Missing Road

Prohibited Turn

Wrong One Way

Speed Limit

Travel Time

Det Ind1Is FMC able to detect relevant deviations?

0,00%

5,00%

10,00%

15,00%

20,00%

25,00%

Deviation Type

Average False Alarm Rate - All Test Sites

FAR 7,15% 4,04% 0,90% 8,14% 5,61% 22,00%

Wrong Road

Missing Road

Prohibited Turn

Wrong One Way

Speed Limit

Travel Time

Validation Results on FMSC Indicators: FSCProc Ind2FSCProc Ind2

Can FMSC increase quality of deviation data by clustering and statistical

analyses of the MDRs?

25% to 30%

false alarms filtering

+20%

Deviation Coverage(e.g. different FMC report on slightly different parts of the actual deviation)

0%+6%+8%MDA compared to average of MDRs in cluster

completnessrate

CMPR, CCMPR

semantic accuracy &geometric accuracy

CSCACCR

geometric accuracy

GACCR

FMSC Quality Improvements

Findings:– provision of a confidence value with MDR is a key element to improve the quality of MDA– with fine tuning of FMSC, expectable improved figures– clusters containing only False Alarms can appear in the FMSC. They rarely reach sufficient confidence for MDA creation– in general: accuracy of MDA is better than the individual MDRs

Cluster containing in average at least 400 (MDRs) issues satisfyingly correct MDA:

- covering in average 84% of the deviation

- describing deviations with an average accuracy of 13.5m.

~13m~84%> 400

~14m~67%> 300

~15m~65%> 100

~16m~61%> 40

~19m~33%> 1

Geodetic accuracy of MDA

Percentage of Reference Detection of MDA

Cluster Size(Number of MDR)

Validation Results on FMSC Indicators: FSCProc Ind2

What is the required numberof MDRs per clusterfor MDA creation?

Main influencing factors are:

– number and quality of MDRs needed for MDA creation

– the deviation type, i.e. different deviation types demand different number of MDRs

– FMSC confidence in reporting FMC

– the indivdual FMSC algorithms for MDA creation

Validation Results on FMSC Indicators: Tim-Ind3

300

Traffic FlowVehicles / h

133h1%1400All

FMSC Tele Atlas &Greece(SimCity)(ICCS, TA)

AverageTime MDA alert

AssumedPenetration

MDRsDeviation Type

MDACreation

Tim Ind3How long does FMSC takes to issue an alert (to MC) since the

moment that a deviation actually took place?

Summary on Technical Validation of FeedMAP Concept

The FeedMAP clients and FeedMAP Service Centre show very promising results.

No principal barrier could be identified which would hinder or fully impede the implementation of such a system in practise.

... in dense areas (many very nearby roads) detection of some map deviations types is difficult due to current status of implementation and use of standard system components (e.g. map matcher)

Validation of Economic Feasibility

0

1

2

3

4

5up to date maps

coverage

map accuracy

additional map attributes

update provision time

dynamic events

3D map data

objects with dynamic content

FeedMAP Improved Status Quo Customer Satisfaction

0

1

2

3

4

5up to date maps

coverage

map accuracy

additional map attributes

update provision time

dynamic events

3D map data

objects with dynamic content

FeedMAP Status Quo

„Do the customers want FeedMAP?“„Do we need FeedMAP?“

1,90%7,70%21,20%19,20%30,80%19,20%

additional content (e.g. time tables public

transport )

2,00%11,80%25,50%33,30%17,60%9,80%3D map data

5,80%13,50%13,50%23,10%30,80%13,50%dynamic events (traffic,

weather, etc)

7,70%30,80%21,20%25,00%15,40%0,00%update provision time

(currently 3 month)

6,00%18,00%26,00%28,00%20,00%2,00%map attributes (e.g. truck

attributes)

9,40%30,20%41,50%11,30%7,50%0,00%map accuracy

13,20%43,40%26,40%11,30%5,70%0,00%coverage of maps

3,80%26,40%34,00%20,80%11,30%3,80%up to date maps

543210

Questionnaire ResultsCustomer Satisfaction on existing system features. (0 not satisfied - 5 highly

satisfied)

FeedMAP Value Chain Overview

aggregated value chain

open standard value chain

The Open Standard FeedMAP Value Chain Map

end customer

update serviceoperator

mapsupplier

FMC(system supplier, automotive man.) deviation

analysisoperator

MDA Updates Updates

publicauthority

MDR

•Long chain (many stakeholders involved) reduces potential margins or increases costs for end users •Long chain holds the potential of quality problems due to the high number of integration steps•Increased risk of operation along the supply chain•Expectable longer update provision times since FMSC, update production, and update provision (AMSC) are operated by different parties•Certain companies may use their proprietary solution rather than the FeedMAP approach.

•Number of FMCs can be easily increased by arbitrary system suppliers because of open interfaces and protocols•Expectable faster creation of required amount of MDR reports as all FMCs can contribute together to map improvement•Possible collaboration between many FMSCs (deviation analysis operators)•Possible purchase of MDAs by map suppliers from different FMSCs•Reduce throughput time by operating the FMSC and map centre together

ThreatsOpportunities

•High diversity of different FMCs may harden MDA generation•Map update generation has to consider many different map releases for update production due to many FMCs•Increased administration effort to handle MDAs for different map releases •Possible demand to have several FMSCs for MDRs based on maps from different vendors•Deviation analysis operator is missing map maker knowledge for more complex deviation analysis tasks•Update purchase by end customer from system independent update supplier (no system specific map format delivery) forces system supplier to support formats of update service operators or requires on board map / integration compilation

•Use of open standards allows easy accessibility for additional FMCs, FMSCs, and AMSC•High quality for certain updates provided by public authority•possibilities for public authorities to provide common public information •Rich coverage on different update types due to expectable high number of different FMC•Expectable good geographical coverage due to expectable larger number of FMCs from different system suppliers•Extendible XML schema allows additional deviation types •Modular architecture supports different market scenarios•Support of on the fly location referencing may overcome the limitations of release-specific reporting•Expectable reduced time to obtain MDAs due to expectable larger number of FMCs from different system suppliers

WeaknessesStrengths

FeedMAP Value Chain

mapsupplier

FMC(system supplier, automotive man.)

publicauthority

deviationanalysisoperator

systemsupplier

end customer

MDR

Updates

update serviceoperator

prop. Updates

•Possible deviation from open standard (MDR)•Change of map supplier causes loss of previous FeedMAP community ; adaption necessity of FMCs

•Quality improvement of deviation analysis (FMSC) because of operation by map supplier•Public Authorities possible role as commercial content provider (use of common public information from Public Authorities in commercial organisations)•Business models for system suppliers based on FMC report behaviour•Use of standards at navigation system side will overcome updates tied to a particular format•Use of location referencing techniques can make MDRs more map agnostic

ThreatsOpportunities

•No open MDR sharing community because of FMC (system supplier) binding to particular map supplier•Map update compilation process required by system supplier

•Faster delivery of map updates to end customers•Direct technical and organisational connection from the public authority to the map supplier•Reduced map update integration effort on end customer system by system supplier•MDR analysis makes use of map knowledge and additional source of information (field validation) by map supplier•Reduced complexity of the detection update chain in terms of minimizing purchase costs•System suppliers can keep their own map update formats ; less map update integration effort at the system side

WeaknessesStrengths

End Customer

System Supplier

Map Supplier

AutomotiveManufacturer

Operation of FMSC

Map Update Production

Map Update Delivery Service

System/FMC Provision

Map Update Compilation

Map Update Service

System Purchase

Reception of Updates

Delivery of MDRs

System Purchase

FMC Operation

Adaption of Technical Infrastructure

Hard and Software Development

System (Operation) Maintenance

Human Resources and Training

Billing and Customer Care

Workflow Process Adaption

Sup

port

Act

iviti

esP

rimar

yAc

tiviti

es

Possible Business Model for Mobile Devices

System Supplier Map Supplier

€€€€

End CustomerEnd CustomerEnd CustomerEnd CustomerEnd CustomerEnd Customer

once € €€€

CommunicationProvider

map updates map updates

contract for mobile internet access for systems

MDR

contract between system supplier and telecommunication provider for flat data rates (only for MDR and update transmissions)

no communication costs for end user

one-off payment by end customer plus a contract agreement to receive updates for a fixed price over a fixed period of time

besides telco based internet connectivity a second possibility by WLAN access points is reasonable

Contracts between WLAN hot spot operators (e.g. at petrol stations) and system suppliers are also promising because of:

-reduced communication costs-reduced hardware and administrative costs-possibility of further cost reduction by advertisement (through Hot Spot operator) “Attractive” contracts needed

between System Supplier and Communication Providers

“Attractive” contracts neededbetween System Supplier and Communication Providers

Possible Business Model for Built-In Systems

System Supplier Map Supplier€€€€

End CustomerEnd CustomerEnd CustomerEnd CustomerEnd CustomerEnd Customer

€€€

CommunicationProvider

Automotive Manufacturer

€€€€

+ once €€ systems + map updates map updates

contract for mobile internet access for systems

MDRmap updates and internet access

map updates

internet connectivity already integrated in high class vehicles and fleet management solutions

a similar business model as the proposed model for mobile devices is in use in automotive context

system and map updates are purchased by automotive from OEM on basis of flat rates

communication costs handled by the automotive manufacturer according to existing “connect” agreements with communication providers

map update service operated by system supplier

costs for end customer included in vehicle purchase price

existing models can be used to operate FeedMAP in automotive context

existing models can be used to operate FeedMAP in automotive context

Validation Results Economic Feasibility

Existing Market Awareness:

– System suppliers already brought deviation detection solutions to the market. They are mainly based on manual detection with limited coverage of deviation types.

– FeedMAP provides means for extensions and additional advantages to existing approaches

– Existing solutions are missing transparent and clearly visible interaction between System Suppliers and Map Suppliers, FeedMAP has a transparent and open architecture.

Customer Awareness:

– Customer is demanding up to date and accurate maps

– End customer’s interest in becoming a “map deviation detection” probe is driven by clear benefit for him. Which is: Always up to date maps in the best case with no or very low extra costs.

Validation Results Economic Feasibility

Existing Infrastructure:

– System suppliers offer map updating services for full map releases via internet to their customers. Consequently basic know how and infrastructure is given to extend services with ActMAP similar map update provision techniques

– Incremental or delta map update services are under discussion by the Map Suppliers. FeedMAP can become one building block for a reasonable next step in this context providing new data acquisition techniques, and map quality verification

– Automotive manufacturers offer connected services in their vehicles. The communication infrastructure is available for sending and receiving map data via internet.

– Additional economic effort has to be undertaken for integrating and coupling different sensors with the FMC. But as the FeedMAP tests and technical validation showed, even with a small set of sensors good results are achieved.

Validation Results Economic Feasibility

Challenge Communication Costs:

– offering devices to an attractive price without burdening the end customer with hidden or recurring communication costs

– observable trend: special rates are granted to device suppliers by the telecommunication providers for special mobile phones combining all available internet connectivity technology

– it remains open if mobile navigation devices experience the same end customer acceptance under possible negotiated communication prices as mobiles with internet flat rates

– general open question: does the mobile also become more and more a navigation device or the navigation device more and more a mobile?

Challenge adaption of production processes:

– both System and Map Supplier stakeholders have to adapt their internal production processes to a fully automated process

– Current map releases are provided on 3 monthly basis, hence a 24/7 service is quite a huge step requiring additional technical know how and human resources together with constant system maintenance costs

Validation Conclusion

FeedMAP is effective from a technology and feasible from an economic perspective

FeedMAP is a mean to improve map data acquisition process for the Map Suppliers

FeedMAP is a mean for improving quality of navigation systems by up to date map data

FeedMAP is an enabler for new safety, security, and green driving applications

FeedMAP will obviously not act as unconditional map update generator

In general, the final update verification will remain to be done by the Map Suppliers using their other sources of information.

Thank You

IndicatorRequirement

FSCProc Ind2FR-22 FMSC: Deviation data Analysis. The FMSC shall be able to analyze MDRs.

QPar Ind1FR-11 FMC:Quality parameters on deviation detection. The FMC shall be able to associate quality parameters to deviation detection.

Comm Ind1FR-13 FMC: Generation of map deviation report. The client shall generate a map deviation report including all information necessary for FMSC and MC data processing

Comm Ind3Comm Ind4

FR-16 FMC: Communication management – transmission. The FMC shall queue and transmit each generated deviation report according to predefined rules, including the priority of messages and the availability of information channel.

Comm Ind5Comm Ind6

FR-19:Throughput. All detected deviations can be stored and transmitted, while the system can cope with longer intervals between connections to the FMSC.

Tim Ind2NFR-7:Timing of MDR transmission

Tim Ind3NFR-8:Timing of MDA generation & transmission.

QPar Ind2FR-24 FMSC:Quality parameters on deviation alert. FMSC shall be able to associate quality parameters to a deviation alert.

FSCProc Ind1FR-21 FMSC:Pre-processing. The FMSC shall filter and exclude MDRs from further processing which do not meet certain minimal quality requirements (up-to-dateness of map version, strategy etc.)

Det Ind2Det Ind3

FR-17 FMC: Local warning messages. The system shall allow clients to detect and report local warning deviations (weather, road condition data etc.)

Tim Ind1NFR-6:Timing of local warning messages. With warning messages (for generating dynamic information) the FMC system shall not exceed a maximum processing time. The transmission of dynamic content to the FMSC shall not exceed a time limit . This includes that communication channels need to be made available with a limited delay after the deviation detection.

FMSC

Comm Ind2FR-23 FMSC: Map deviation alert generation. The FMSC shall generate MDAs to be sent to MC.

Det Ind1FR-7 FMC: Operation of autonomous detection. The FMC shall be able to detect relevant deviations by comparing sensors data to in-vehicle map.FR-8 FMC: Operation driver assisted detection. The system shall allow clients to point out detections that fall under their observation.FR-9 FMC: Operation of joint detection. The FMC shall support confirmation of autonomous detection by the driver.

FMC