1
TI 資深汽車處理器應用工程師Rio Chan
進階駕駛輔助系統解決方案設計簡介
2
Designing Deep Learning Solutions for ADAS
Outline
• What is ADAS?
• What are the important peripherals for ADAS?
• What Deep Learning can do for ADAS?
• TI TDA4 ADAS design concept?
• TI TDA4 road maps?
• TI Deep Learning.
• TIDL Demos?
• What TI can support?
3
ADAS?
4
ADAS level
5
WHAT ARE THE IMPORTANT PERIPHERALS FOR ADAS?
6
What are the important peripherals for ADAS?
TI TDA4
https://www.ti.com/product/TDA4VM
7
What are the important peripherals for ADAS?
TI Radars
https://www.ti.com/sensors/mmwave-
radar/overview.html
8
What are the important peripherals for ADAS?
TI CAN solution - TCANx
https://www.ti.com/solution/drive-assist-ecu
9
What are the important peripherals for ADAS?
TI PGAx U-Sensor AFE
https://www.ti.com/solution/ultrasonic-
park-assist-sensor
TI LMGx DCDC boost https://www.ti.com/solution/automotive-
mechanically-scanning-lidar10
TDA4 ADAS DESIGN CONCEPT
11
TI Processor SDK | Single development kit for all families
12
Cockpit Cluster Amplifier Powertrain Camera Radar LIDAR
Sensor
fusion
100% code
compatibility
TI-RTOS
Gateway
Safety Security
System
Performance
DRA8x TDA4xAM6xAM2x
Jacinto’s DRA benefit for “compute” system
13
Jacinto
DRA
Jacinto’s TDA4 benefit for “ADAS” system
14
Jacinto
TDA
TI Jacinto™ 7 | Heterogeneous compute platform
15
Optimize entire platform around
programmer productivity on the
MPUs
Choose the right core
for the right job
Offload the majority of “work” to
specialized processors. Provide
tools & SW to manage complexity
Heterogeneous
Compute Capabilities
Image Signal
ProcessorDSP for
analytics
Deep Learning
Acceleration
Vision
Accelerators Display Engine
4K
Multi-Media
Accelerators
GPU
Dual-core
RF5 clusterDual-core
RF5 cluster
Jacinto™ automotive
processors
64b ARMv8
MPUs
Common high-performance chassis• High-bandwidth coherent SoC interconnect for main processor & memory
• Packet-based interconnect for latency-sensitive interfaces (Ethernet, CSI)
• Dedicated boot engine (early boot)
Memory
System
L3$
Safety• ASIL-B capable SoC
• Safety HW, SW & collateral
• ECC on all memories & buses
Security• Run-time and secure boot
• SoC-level firewalls, BIST
• Dedicated crypto HWAs
Automotive connectivity• CAN-FD, MOST-150, FlexRay, Ethernet AVB, USB 3.1
• GbE switch, PCIe Gen4 with switch
ASIL-D
Safety MCU
RTOS
Dual-core
A72 cluster
Dual-core
A72 cluster
Dual-core
A72 cluster
Dual-core
A72 cluster
More
Pro
cessin
g
Pe
rform
ance
Unique TI
innovation
Embedded Machine Learning Architectures
16
17
TDA4 ADAS Design concept
18
TDA4
Radar
ADAS Camera mirror
ADAS Domain controller
DMS / In-cabin Sensing
Surround view / Parking
Front Camera
19
20
2121
2222
2323
24
TDA4 ROADMAP
25
Jacinto Vision Analytics Portfolio
26
TI DEEP LEARNING
27
What Deep Learning can do for ADAS?
28
Edge inference software stack on TI SoC
29
TFLite RunTime ONNX RunTime TVM /Neo-AI-DLR
Cortex-AC7x + MMA
(Deep Learning Accelerator)
TIDL Optimizer: Layer fusion | Post Training Quantization | Range Calibration | Memory optimizations
TIDL RunTime: API | TIDL library on C7x & MMA
Model artifacts
30
TIDL RunTime
User ApplicationPython / C / C++
API | interpreter | scheduler
TFLite/ONNX-RT/Neo-AI-DLR
Jacinto 7 processor
Deep learning accelerator
-+ * =
C7x DSP with MMA*
ARM Cortex A72
ARM Cortex A72
IPC
Linux OS
CPU
TIDL Library
Open-source kernels
31
TI Deep Learning | SW Eco System
32
Training Inference
TIDL Importer
Deep Learning Application Execution
OpenVX
Framework
TIDL Inference
Training frameworks
TIDL Graph Compiler
TIDL Importer➢Enables development on Open frameworks
➢Model Parameters Quantization & Calibration
➢Layers Merging/Fusion
TIDL Graph Compiler➢Memory management, Network Execution
Plan/order
➢Models the device & Estimates performance
TIDL Host Emulation➢PC/Host emulation of Inference for offline
Validation and Accuracy benchmarking
TIDL Inference➢Abstracts embedded development
➢High Efficiency Inference Implementation
OpenVX Node ➢Deep learning Use case development
across cores (IPC, Pipelining )
Model Formats
PrototxtCaffemodel
Frozen-pb/TFLite
TIDL Host Emulation
What Deep Learning can do for ADAS?
33
Real-time 8x 2MP camera input demo on single ISP
34
• TDA4VM Camera performance as below.
• 8x 2MP cameras @ 30 fps
• Feeding to single TDA4VM device via CSI-2
• All camera images pre-processed by single
ISP in real-time• ISP still has headroom to process 2 more
cameras
• Flexible video processing sub-system
YouTube link: https://youtu.be/zsYbPPhO0t8
DMPAC: Depth and Motion Perception Accelerator
35
Stereo disparity map
Stereo Depth Engine: for each location <x,y> , estimates depth by
measuring disparity from 2 different views
Assign confidence score for each disparity output
Stereo
Depth
Engine
90 Mpix/s, <0.5WUp to 2MPix
Dense optical flow
Dense
Optical
Flow
150 Mpix/s, <0.5WUpto 2MPix
Dense Optical Flow: for each location <x,y>, estimates 2D
motion vector field given two images
Generate confidence score for each flow vector output
Object Tracking Structure From Motion (3D)
Moving Object Segmentation
Object distance measurementCollision warning, Segmentation,Path planning,….
4x 1MP @ 20 fps
disparity map per core
TDA4 EVM AND DEMOS
36
TDA4V Mid EVM Details
37
Modular Development System
➢Provides complete access to Jacinto™ 7 SoC
➢HW platform for SDK Validation
➢New SOM for each Jacinto™ 7 SoC Variant
➢On-board XDS110 JTAG for debug via CCS
➢XDS560v2 for advanced debug and trace
http://www.ti.com/tool/TMDSEMU560V2STM-U
➢ Processor SOM
➢ Common
Baseboard
Baseline Hardware Kit
Optional Expansion cards for application needs
Gateway Ethernet Switch
Industrial Expansion
Infotainment
Expansion CardFusion Application
Board
additional
expansion
cards plannedhttp://www.ti.com/tool/TDA4VMXEVM
Available Now
Jacinto™ 7 TDA4VM analytics and vision demos
38
Front camera analytics with TDA4 Smarter Parking with TDA4
Deep Learning
Based Forward
Camera Analytics
➢ Semantic Segmentation and Free space ID
➢ Multi-class Object Detection
➢ Custom DL networks built on TI Deep Learning
Libraries (TIDL) and running on C7x+MMA
8MP Front Camera
Perception and
Localization
➢ NCAP ready multi-class object detection using
TIDL Libraries and C7x+MMA
➢ Fusion with IMU and GPS for Localization
➢ Concurrent applications on one TDA4VM SoC
Surround View
with Analytics
➢ High-quality 3D Animated Car Model, overlays
for projected path and distance warning.
➢ Transparent car view using history
➢ Object Detection, Lane ID, Parking spot ID
using DSP based algorithms.
Auto Valet Park
and Surround
View
➢ AVP with Semantic segmentation, object and
parking spot detection based on TIDL
➢ Simultaneous Real-time 4x 2MP camera
Surround view with 3D car model.
➢ All code freely available in TI’s Processor SDK
Real-time 8-Camera
Input to TDA4
➢ Eight 2MP cameras feeding single TDA4VM.
➢ All camera images pre-processed by single
ISP in real-time
➢ All code freely available in TI’s Processor SDK
TDA4: Multi-Camera vision processing
➢ Ruggedized Development platform for
applications on TDA4VM SoC.
➢ Multi-sensor interfaces: camera, radar, lidar…
➢ Software compatible with TI EVM based on
Processor SDK
DesignCore Rugged
Vision Platform for
TDA4VM
Jacinto™ 7 TDA4VM YouTube Videos
39
Front camera analytics with TDA4 Smarter Parking with TDA4
Deep Learning
Based Forward
Camera Analytics
➢ https://youtu.be/F4KNWuTSJds
8MP Front Camera
Perception and
Localization
➢ https://www.businesswire.com/news/home/202
00107006173/en/
Surround View
with Analytics
➢ https://youtu.be/F4KNWuTSJds
Auto Valet Park
and Surround
View
➢ https://youtu.be/PER77CgoHWk
Real-time 8-Camera
Input to TDA4
➢ https://youtu.be/zsYbPPhO0t8
TDA4: Multi-Camera demo
➢ https://youtu.be/bOSmeej9_cQ
DesignCore Rugged
Vision Platform for
TDA4VM
WHAT TI CAN SUPPORT?
40
Software and tools - overview
41
TDA4 AutoSAR
42
TDA4 PinMux tool
43
https://e2e.ti.com/support/processors/f/processors-forum/927526/faq-ccs-tda4vm-pinmux-guide-for-jacinto-processors
TDA4 Cloud AI Tool: https://dev.ti.com/edgeai/
44
TDA4 Cloud AI Tool: https://dev.ti.com/edgeai/
45
TDA4 Cloud AI Tool: https://dev.ti.com/edgeai/
46
TI ADAS system of experienced partners
47
Camera Sensors, Modules
and Tuning Services
MCAL
AutoSAR Classic
Adaptive AutoSAR
SAFETY SOFTWARE, CERTIFICATIONOPERATING SYSTEMS AND SECURITY
ENGINEERING SERVICES
Algorithm Porting and Optimization
Hardware Design
System Testing
Driver Development
Surround and Auto Parking
DMS and In-Cabin Monitoring
Front Camera and Autonomous Drive
ADAS ALGORITHMS
Thank you
48
IMPORTANT NOTICE AND DISCLAIMERTI PROVIDES TECHNICAL AND RELIABILITY DATA (INCLUDING DATASHEETS), DESIGN RESOURCES (INCLUDING REFERENCEDESIGNS), APPLICATION OR OTHER DESIGN ADVICE, WEB TOOLS, SAFETY INFORMATION, AND OTHER RESOURCES “AS IS”AND WITH ALL FAULTS, AND DISCLAIMS ALL WARRANTIES, EXPRESS AND IMPLIED, INCLUDING WITHOUT LIMITATION ANYIMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT OF THIRDPARTY INTELLECTUAL PROPERTY RIGHTS.These resources are intended for skilled developers designing with TI products. You are solely responsible for (1) selecting the appropriateTI products for your application, (2) designing, validating and testing your application, and (3) ensuring your application meets applicablestandards, and any other safety, security, or other requirements. These resources are subject to change without notice. TI grants youpermission to use these resources only for development of an application that uses the TI products described in the resource. Otherreproduction and display of these resources is prohibited. No license is granted to any other TI intellectual property right or to any third partyintellectual property right. TI disclaims responsibility for, and you will fully indemnify TI and its representatives against, any claims, damages,costs, losses, and liabilities arising out of your use of these resources.TI’s products are provided subject to TI’s Terms of Sale (https:www.ti.com/legal/termsofsale.html) or other applicable terms available eitheron ti.com or provided in conjunction with such TI products. TI’s provision of these resources does not expand or otherwise alter TI’sapplicable warranties or warranty disclaimers for TI products.IMPORTANT NOTICE
Mailing Address: Texas Instruments, Post Office Box 655303, Dallas, Texas 75265Copyright © 2021, Texas Instruments Incorporated
