Skim down AGL Application Framework to bridge AGL with hard realtime subsystems
Dresden AGL/AMM Oct/2018Linux Realtime Technical Lead
Oct-2019Micro Binder Architecture 2
Who are we?
Oct-2019Micro Binder Architecture 3
Where ?
LORIENT
vannes
● https://iot.bzh/en/● http://github.com/iotbzh
Oct-2019Micro Binder Architecture 4
Oct-2019Micro Binder Architecture 5
Bridging AGL Application Framework with hard realtime subsystems
Introduction
Oct-2019Micro Binder Architecture 6
How to broaden AGL Connectivity
● Up to the Cloud– Keeping critical services in realtime contexts
● Down to:– (very) smaller, hard realtime systems
You’re at the right place !
Feb-2018 7
Bridging AGL Application Framework with hard realtime subsystems
Oct-2019Micro Binder Architecture 8
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication● Focus on the can-low-level binding● To infinity (→smaller) and beyond !● Suggested roadmap
Oct-2019Micro Binder Architecture 9
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication● Focus on the can-low-level binding● To infinity (→smaller) and beyond !● Suggested roadmap
Oct-2019Micro Binder Architecture 10
The RT Quest
● Path to RealTime is not straightforward !
Oct-2019Micro Binder Architecture 11
Some urban legends about RealTime
● “I wont’t get RealTime issues, and won’t have to deal with thread priorities, as long as my CPU load keeps low”
Hey buddy, have you ever heard about:
Interrrupts ?
Ping flood ?
Unpredictive (because not tested) time spent in code branches ?
Device latencies with poor driver implementations (eg, SD card) ?
Freezes due to tight loops ?
Feb-2018 12
Some urban legends about RealTime
● “I do not care memory allocations, I have got 4 Go of available RAM !”
Feb-2018 13
What is RealTime in an OS?
● RealTime means “on time” and not “faster”– Realtime is about predictability
● Typically Realtime address following concerns– 80% => Could my execution code be disturbed ?
● If then:– For how much time ?– How often ?– By who ?
– 20% => How big is my latency● Time lost within an external event, and the time my code can
start to handle it ?● Maximum latency fluctuation ?
Feb-2018 14
Linux & Soft/Hard Realtime
● Soft realtime● Periodical tasks/events of several milliseconds● Some acceptable unpredictable delays (10/100 ms)● Often implemented by resource controls● Latency of few ms with exceptionnal unpredictable
fluctuation +-10ms
● Hard realtime● Total Latency magnitude of 10/100 us● Predictable and short delays (< 250ms)
● Current vanilla Linux kernel is Soft RealTime
Feb-2018 15
Which Automotive Apps need RT ?
● Soft Realtime● Data Acquisition● Audio/Video
● Hard Realtime● Cluster● Emergency/Safety signal● Dead reckoning● LIDAR acquisition● Trajectory control● ...
Driv erle ss c ars
Feb-2018 16
RT Options inside AGL
Cluster
Carte handling
Localistion management
POI
CAN GPS
Geopositioning Virtual Signal
Multi ECU & Cloud Aware Architecture
Entertainement
CAN-BUS Virtual Signal
Gyro, AcelerometerCAN-BUS
LIN-BUS
Engine-CAN-BUS
ABS
Transport & ACL
Head Unix
Direction Indication
Cloud
Log Analytics
No-SQL Engine
Statistics & Analytics
Transport & ACL
My Car Portal
Paiement
Subcriptions
Preference
Preferences &
Custumisation
MongoDB Engine
Paiement Service
Cluster Virtual Signal
Transport & ACL
Navigation Service
Maintenance Portal
Know Bugs
Maintenances
Service Packs
Soft RealTimeHard RealTime
Feb-2018 17
Linux RT Application Impact
● Standard Linux: a simple “Ping Flood“ will lag applications.● Linux network IRQs preempt applications too often and for too long, which
significatly increase the latency.
● PREEMPT_RT reduces scheduling latency● Replaces most spinlock by mutexes● Support threaded IRQs● Supports hight resolution timers
Feb-2018 18
Turn “ON” Linux RealTime.
● Objectives● Decrease Application Latencies● Garanty that high priority tasks will not be bothered by
lower priority ones.● Make sure interrupts cannot lag your critical apps.
● Soft Realtime (Standard Kernel)● Container, Ggroups, ...
● Hard Realtime (Kernel must be patched)● PREEMPT-RT● IPIPE+Xenomai
Feb-2018 19
Preempt_RT vs Xenomai
● Xenomai● Xenomai supports to legacy RT non-POSIX applications (eg: VxWorks, PSOS) through
skins● Dual Kernel solution brings more performances when no more than 4 cores run RT
threads● More confidence on the whole RT application (eg: /proc/xenomai statistics)● Integrated debug features● Misses some critical Unix develoment tools (eg: Valgrind, LTTng)
● Preempt_RT● Almost Vanilla Linux (no API/ABI changes)● Continuous testing in OSADL QA farm● No need for extra userspace libraries● Less confidence in app, harder to debug, needs extra code for RT monitoring
Feb-2018 20
Xenomai Dual Kernel Mode
Being replaced by Dovetail
Feb-2018 21
Preempt_RT Latency
Source: http://www.emlid.com/raspberry-pi-real-time-kernel Latency (us)
Oct-2019Micro Binder Architecture 22
Xenomai Latency
Feb-2018 23
Xenomai & Prempt_RT convergence
● Xenomai 3.x offer dual kernel and PREEMPT_RT option (with the same high level API and skins)
● Dual kernel latency remains significantly better● Some options are not available in both
solutions (eg: RtNet only runs on dual kernel)
Feb-2018 24
RT Kernel is only a start (1/4)PREEMPT_RT kernel tuning
● Realtime requires more kernel tuning and clean behaviours on the application side.
● Enable CONFIG_PREEMPT_RT_FULL & CONFIG_HIGHRES_TIMERS to get <1ms precision
● Disable CONFIG_CPU_FREQ !● Might conflict with power management● … and disable THERMAL framework
Feb-2018 25
RT Kernel is only a start (2/4)Impact on Applications
● There are strict rules to follow and actions to take in the application:● Stack pre-faulting● Virtual Memory locking● Fine tuning of threads priorities● malloc() and friends chasing, to avoid page faults (can be difficult with
some C++ libraries, eg BOOST)● Forbidden usage of system(); popen(); execve() … in runtime.● Monitoring run-away threads (ie, tight loops in RT contexts) to prevent
system hanging (and to allow debugging).● clock_nanosleep is your friend, for writing periodic tasks● Carefull initialization parameters of pthread_mutex, default ones do not
have PTHREAD_PRIO_INHERIT !● Fancy some LTTng sessions ? (does not work with IPIPE)
Feb-2018 26
RT Kernel is only a start (3/4)Choose the right clock !
● CLOCK_REALTIME is NOT RealTime ! – Dayligt savings
– NTP adjustements (even THE big jump, when enabled)● CLOCK_MONOTONIC (affected in speed by NTP)
or● CLOCK_MONOTONIC_RAW will only vary depending on the quality
of the quartz and voltage and temperature compensation quality
Feb-2018 27
RT Kernel is only a start (4/4)Impact on Applications :
Last but not least
● Not everything can be RT● Providing high priority to some task means
than the other will inherit of low priority● Base you flow on lock (semaphore) and not on
thread priority● Get rid of any spin lock
Feb-2018 28
Impact on Applications : C specific issues (& workarounds)
● malloc()/realloc() do not lead to pagefault always, (through sys_brk() or sys_mmap_pgoff() because of internal memory pool of the glibc
● Thus, a RT ‘leak’ may be hard to reproduce● Using GDB with a breakpoint on malloc() is usually
sufficient● Another technique, less intrusive, is to use Memory
Allocation Hooks of the glibc.● Some companies allow malloc() for initializations, and
always forbid the free() !
Feb-2018 29
C++ specific issues
● In C++, dynamic allocations are not always explicit
● Example: std::vector growing● In some extra libraries (eg, boost), memory
allocations may be completly out of control (in addition of alien-only-friendly backtraces)
Feb-2018 30
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication● Focus on the can-low-level binding● To infinity (→smaller) and beyond !● Suggested roadmap
Feb-2018 31
Bridging AGL Application Framework with hard realtime subsystems
● Cars are made of several systems with different connectivity or responsiveness constraints
● Communication between cluster and subsystems involves several protocol stacks
● Bringing access security to embedded control systems
● Low-level filtering of devices events is leaner than in high level application(s)
Oct-2019Micro Binder Architecture 32
What is the AGL binder
● Bound to a systemd service● Dynamic loader of applications (bindings)● Embeds:
– a tiny http server
– websockets based on Unix named sockets● Provides an event loop and timers API
Feb-2018 33
Use case of the binder
cloudAPPLI
BINDERA
BINDER
BAPPLI
BINDERC
BINDING
BINDERD
BINDING
Connected carMaster
ECU
HTTPS+WSS/TCP HTTPS+WSS/TCP
WS/UDS
WSS/TCP
WS: WebSocketWSS: WS SecuredUDS: Unix Domain
SocketECU: Engine Control
Unit
BINDERBINDING
Oct-2019Micro Binder Architecture 34
Binder/Binding model
●Binder–Container process–Transport–Security–Standardized Sync/Async API–Expose API through HTTP or WebSocket
● Protected by token●Binding
–One or more API published through binder–Provided as:
● A native library weekly coupled (threading allowed)● A proxy to a remote service
APPLICATION
BINDERafb-daemon
BINDING
BINDING
BINDING
SECURITYCONTEXT
http ws
BINDERafb-daemon
BINDING
SECURITYCONTEXT
A
B
Oct-2019Micro Binder Architecture 35
Some existing binders & bindings in AGL
● agl-service-audio-4a– Highlevel API
– Softmixer
– 4a-HALs
– AlsaCore● agl-service-can-low-level● agl-service-iiodevices
Oct-2019Micro Binder Architecture 36
Porting the Binder to a micro-OS
● Leave the Unix world– No unix sockets
– Inter-binding messages should be lighter, less footprint, no big json data.
– Binary json could be an option
– Avoid copying data when possible● Break the existing dependency to libsystemd
– But keep the timers and event loop API● Need to have an OS abstraction layer for non POSIX systems● Add new communication transport with other system chips
? or shared memory
Oct-2019Micro Binder Architecture 37
Inter SOC communication
● Depends on the subsystem connectivity– Subsystem on same PCB → shared memory or
dedicated hw channel (i2c …)
– Remote subsystem → specific bus, or IP in the best case
● Access token can given by an external authentification server
● Can use the encryption as well
Oct-2019Micro Binder Architecture 38
Defining a boundary
● POSIX may be a too strong prerequisite (not available for all the embedded systems, or only partial support
● Assuming Glibc (or µCLibc) seems fair enough● Incomplete libc implies to have more OS
abstraction
or
µClibc
Glibc
Feb-2018 39
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication
– Security● Focus on the can-low-level binding● To infinity (→small) and beyond !● Suggested roadmap
Oct-2019Micro Binder Architecture 40
Focus on the can-low-levelbinding
41
CAN agent base on AGL framework
● Clear Isolation ● Low level CAN operation only depends on equipment● High level business logic dedicated to applications
● Security Built In● Navigation APP may access GeoLocation but not Telephony● Implement statistic/counter to monitor unexpected behavior
● Leverage AGL framework● API transparency for client applications● Reuse existing technology (faster, cheaper, safer)
42
AGL CAN Mapping
Binder
CAN Low Level Binding(s)
● Decoding / Encoding● Authentication / Crypto / Firewalling● Transaction (set… ack ...)● Stats & Maths● Caching (low freq. Signals, get() call)● Debug
CAN High Level Binding(s)
● Logic● Aggregation (« vehicle.doors.any.open »)● Advanced Ops
CAN BUS
CAN frames - 011010010
Signals - « vehicle.doors.left.open »(Binder Events)
UI
Publish Subscribe
43
Low Level Binding
● Low Level Binding● Binary encoding decoding● Generate Application Friendly Signal Name & Values
● Close to Automatic Code Generation● OpenXC CAN Vector definition in JSON● Other CAN analysers formats, as CANoe XML (TBD)
● Include Basic Filtering & Statistic● GT/GE LT/LE● Timer, Cycle, Timestamps● Counter: last value, average, invalid ID, …
● Can be ship to developer as binary only
44
From OpenXC to AGL CAN Binding
CAN Low Level Binding(Shared Library)
OPENXC Signal
Description(JSON)
Low LevelBinding
Static Code(AGL)
CANDecoding/EncodingC Code(vendor)
OptionalMessageHandlers(vendor)
AGLCode
Generator
C/C++Compiler
OPENXC Signal
Description(JSON)
OPENXC Signal
Description(JSON)
45
Current Binding Capabilities
● Asynchronous signalization● Basic Subscribe/Unsubscribe event model● Transport onto WebSocket/DBus
● AGL App-Framework event signaling API● handle = afb_daemon_make_event (bindif, name)● afb_req_subscribe(request, handle)● afb_req_unsubscribe(request, handle)● afb_event_push(handle, json_object_get(object));
46
Event Subscription
Waiting verb call
[Subscribe/Unsubscribe]
Find corresponding signals name
Create event handler
Make subscription or unsubscription
[No name provided]
[Name provided]
For each signal
Add recurring request to the diagnostic manager
[Diagnostic message]
[Regular CAN message]
Create Diagnostic handle
[No more signals to process]
Take all signals
Vehicle.*.doors.*50<Vehicle.speed<100
Freq=10Hz
47
OpenXC Specification● Messages
● bus - name of initialized CAN buses where this message can be found● name - The name of the CAN message● handler – Function name applied to entire raw CAN message● max_frequency – slow down high level signal frequency (ex: 1/10)● signals - A list of CAN signal objects that are in this message
● Signals● generic_name - The name of the associated generic signal name● bit_position - Staring bit position of this signal within the message● bit_size - The width in bits of the signal● factor - The signal value is multiplied by this if set● offset - This is added to the signal value if set● decoder - type & function name applied to signal bitfield to decode it● encoder - Idem to encode a value in bitfield to send over CAN bus● states - Mapping between descriptive states and values from CAN● max_frequency – slow down high level signal frequency (ex: 1/10)● send_same – if false only send signal when value change● ignore, enabled - Control parameters about activation level for a signal● writable – signal is writable on CAN
48
OpenXC Sample Definition
"0x620": { "bus": "can0",
“name”: “doors”, "signals": { "doors.driver.open": { "generic_name": "doors.driver.open", "bit_position": 78, "bit_size": 1, "factor": 0, "offset": 0, "decoder": "decoder_t::booleanDecoder"}, "doors.passenger.open": { "generic_name": "doors.passenger.open", "bit_position": 79, "bit_size": 1, "factor": 0, "offset": 0, "decoder": "decoder_t::booleanDecoder"},[...]
Oct-2019Micro Binder Architecture 49
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● RT Options inside AGL● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication
– Security● Focus on the can-low-level binding● To infinity (→ small) and beyond !● Suggested Roadmap
Oct-2019Micro Binder Architecture 50
And next ?
● Going to very smaller systems● Scalability and OS-agnosticism will be the
keys● Would eventually imply new transport
implementations (serial, i2c, specific usb ...)
Oct-2019Micro Binder Architecture 51
Plan
● The RT Quest– What is RealTime in an OS ?
– Soft vs Hard Realtime in Linux
– Which Automotive Apps need RT ?
– Impacts on Linux RT Applications
– Turn “ON” Linux RealTime● RT Options inside AGL● Porting to lighter OS
– Binder & bindings
– Impacts on binder
– Defining a boundary to portability
– Inter SOC communication
– Security● Focus on the can-low-level binding● To infinity (→ small) and beyond !● Suggested Roadmap
Oct-2019Micro Binder Architecture 52
Suggested Roadmap
AGLBinder
POSIXBinder
Zephyr / STM32 POC- map POSIX calls to Zephyr syscalls
PREEMPT_RT POC
- Self-monitoring service- Priorities policies
MapSocketcan
Calls toZephyr
CAN driver
Low-levelCAN
on ZephyrBinder on
ZephyrBinder on
Zephyr
RT Binder
RT Low-level
CAN
PortableRT Binder
Lean Memory
allocations
Memfriendly
Remove theDependencyto systemd
pre-RT
Smartmessaging
Xenomai POC
- Priorities policies
RT Binder
RT Low-level
CAN
Oct-2019Micro Binder Architecture 53
Questions ?