20131102-Towards ECV-輔大

8/13/2019 20131102-Towards ECV-

1/66

Wang, Yuan-Kai()Electrical Engineering,Fu Jen University ()Email: [email protected],URL: http://www.ykwang.twSlideshare: Search Towards Embedded Computer Vision

Towards

EmbeddedComputer Vision

8/13/2019 20131102-Towards ECV-

2/66

Towards Embedded Computer Vision p.

Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai

2

MIT Oxygen

http://oxygen.lcs.mit.edu/ 1999

8/13/2019 20131102-Towards ECV-

3/66



3

The Aware Home

http://awarehome.imtc.gatech.edu/

8/13/2019 20131102-Towards ECV-

4/66



Ambient Intelligence v.s.

Object Intelligence

Embedded system + Sensors

4

Wearable ComputingSmart Home, Smart

8/13/2019 20131102-Towards ECV-

5/66



55

Embedded Systems

"Without" Sensors

MIC ITIS

8/13/2019 20131102-Towards ECV-

6/66



66

Embedded Systems

"With" Sensors

GPS

Navigation

Wii

Roomba

Exoskeleton

8/13/2019 20131102-Towards ECV-

7/66



7

Driver assist

Front view

Night vision

Blind corner

Blind spot

Rear view

Embedded Systems

"With" Image Sensors

8/13/2019 20131102-Towards ECV-

8/66



88

Embedded Systems


Google Glass

8/13/2019 20131102-Towards ECV-

9/66



Smart Watch

9

8/13/2019 20131102-Towards ECV-

10/66



Microsoft

SenseCam

Embedded Systems


10

8/13/2019 20131102-Towards ECV-

11/66



Observation

Miniaturization

High Resolution

3D Capability

11

8/13/2019 20131102-Towards ECV-

12/66



12

Embedded Computer Vision

! Embedded System + Camera+ Computer Vision Algorithm

Image

Processing

Image

Recognition

Image

Capturing

8/13/2019 20131102-Towards ECV-

13/66



13

Embedded Computer Vision

! Embeddedcompute vision" Implements computer vision algorithms

on low-cost, low-power,

constrained hardware! Constrainedhardware" Low-speed CPU" Low capacity memory" No floating-point (FPU)" Low-resolution image sensor

8/13/2019 20131102-Towards ECV-

14/66



A BResearch Status (1/3)

14

8/13/2019 20131102-Towards ECV-

15/66

T d E b dd d C Vi i 1616

8/13/2019 20131102-Towards ECV-

16/66



1616

Research Status (3/3)

DARPA

Grand

Challenge

Google

CarGoogle

Taxi

T d E b dd d C t Vi i 17

8/13/2019 20131102-Towards ECV-

17/66



17

International Activities (1/2)!

Special conferences" IEEE Int. Workshop on Embedded ComputerVision

" IEEE Int. Conf. Distributed Smart Cameras! Special journal issues

" IEEE Journal of Selected Topics in SignalProcessing, vol. 2, no. 4, Aug. 2008

" EURASIP Journal on Embedded Systems! Short courses in important CV conferences

" CVPR07&08: Distributed vision processing insmart camera networks" ESC07: Embedded CV and smart cameras" ICASSP09: Distributed processing in smart

cameras


8/13/2019 20131102-Towards ECV-

18/66



18

International Activities (2/2)

! Research projects and Lab." Princeton Univ./Georgia Tech.:

Embedded Systems Lab., Wayne Wolf" Stanford Univ.

Wireless sensor networks Lab." UCLA, CMU, MIT" Delft Univ. of Technology

SmartCam Project" Graz Univ. of Technology


8/13/2019 20131102-Towards ECV-

19/66



19

Three Ways for ECV

! CPU (Central Processing Unit)" ARM, PowerPC

! DSP (Digital Signal Processor)" TI, ADI, NXP

! FPGA (Field Programmable Gate Array)" Altera, Xilinx

Towards Embedded Computer Vision p 20

8/13/2019 20131102-Towards ECV-

20/66



20

CPU

! Stanford MeshEye" ARM 7 (55MHz), ZigBee node" 3 image sensors" 30x30 grayscale x 2" 640x480 color x 1

Object detection

(http://wsnl.stanford.edu/smartcam.html)

"MeshEye:a hybrid-resolutionsmart camera mote for applicationsin distributed intelligentsurveillance", IPSN-SPOTS, 2007


8/13/2019 20131102-Towards ECV-

21/66



21

DSP

! "Visual surveillance on DSP-basedembedded platform," Graz Univ. ofTechnology, 2008(Phd. dissertation)" TI C6414 (600MHz)" Applications: Adaboost face detection, vehicle

detection, license plate detection


8/13/2019 20131102-Towards ECV-

22/66



22

FPGA (2/3)

"Hardware, Design andImplementation Issues on a

FPGA-Based Smart Camera,"ICDSC, 2007


8/13/2019 20131102-Towards ECV-

23/66



23

Hybrid: CPU+DSP

! "Distributed Embedded Smart Camerasfor Surveillance Applications,"IEEE Computer, 2006.

Developed for traffic surveillance


8/13/2019 20131102-Towards ECV-

24/66



Hybrid: CPU+DSP by SoC

! TI OMAP

24


8/13/2019 20131102-Towards ECV-

25/66



25

Hybrid: CPU+FPGA

! UCLA Cyclops" MCU: Atmega128" CPLD: image processing" ECV app.: Hand gesture recognition

http://research.cens.ucla.edu/projects/2007/

Multiscaled_Actuated_Sensing/

Cyclops/


8/13/2019 20131102-Towards ECV-

26/66



Hybrid: CPU+FPGA by SoC

! Xilinx Zynq

26

Towards Embedded Computer Vision p. 27

8/13/2019 20131102-Towards ECV-

27/66



27

Challenges for ECV

! Algorithm refinement" Parallel computation" Function partition, Multi-threading

" Stream processing" Memory hierarchy optimization

! Hardware design" Pipeline, SIMD, board design

! Optimized programming skills" Fixed-point arithmetic" Memory management" Intrinsic commands


8/13/2019 20131102-Towards ECV-

28/66



28

Our Experiences

Entertainment Robot (CPU)Vision Sensor (CPU)Video Surveillance (FPGA)

Wearable Vision (DSP) Towards Embedded Computer Vision p. 29

8/13/2019 20131102-Towards ECV-

29/66

p p


Case 1Entertainment

RobotCPU


8/13/2019 20131102-Towards ECV-

30/66

p p


Sony AIBO


8/13/2019 20131102-Towards ECV-

31/66

p p


Rolling Dice


8/13/2019 20131102-Towards ECV-

32/66

p p


Face Detection


8/13/2019 20131102-Towards ECV-

33/66

p p


Face Recognition


8/13/2019 20131102-Towards ECV-

34/66


Facial Expression Recognition


8/13/2019 20131102-Towards ECV-

35/66


Case 2Vision Sensor

CPU


8/13/2019 20131102-Towards ECV-

36/66


FJUCam

Powerconsumption:1W

Power sources:5V DC current5V Cell Battery

Weight: 35gmSize:6 x 4.5 x 5 (cm)

(W x H x D)

Three Modules1. Main board, 2. Lens module

3. Storage module


8/13/2019 20131102-Towards ECV-

37/66


Face Detection

FJUCam

Face Detection

Image

Display

Vision Sensor Network


8/13/2019 20131102-Towards ECV-

38/66


Cyclops Robot

! Color tracking


8/13/2019 20131102-Towards ECV-

39/66


Case 3Video Surveillance

FPGA

8/13/2019 20131102-Towards ECV-

40/66


8/13/2019 20131102-Towards ECV-

41/66


Background Subtraction

Current

Frame

-

Background

Image Update

Post Processing

Motion Object Image

Background Image

1+kB

kB

1+kP

1+kM

),(),(

),(

1

1

yxByxP

yxM

kk

k

!=

+

+

),()1(),(

),(

1

1

yxPyxB

yxB

kk

k

+

+

!+= ""

Speed up by (1) Circuit design, (2) Paralization


8/13/2019 20131102-Towards ECV-

42/66


FPGA Speed Up

! Comparisons" PC: 2.8GHz CPU, C implementation" FPGA can speed up 500 times

2.8G

25M

51

10


8/13/2019 20131102-Towards ECV-

43/66


Case 4Wearable Vision

X-EYE

ARM + DSP


8/13/2019 20131102-Towards ECV-

44/66


Goal

44


8/13/2019 20131102-Towards ECV-

45/66


Demo Video


8/13/2019 20131102-Towards ECV-

46/66


Development

SD USB-WIFI

SB-RS232

SB

BeagleBoard

CameraUSBHub


8/13/2019 20131102-Towards ECV-

47/66


Hardware

4747

Peripheral I/O

USB HostJTAGDVI-D video outS-Video outSD/MMC+Stereo in/outRS-232 serial1Alternate powerUSB 2.0 HS OTG

USER

OMAP3530 Processor600MHz Cortex-A8

NEON+VFPv316KB/16KB L1256KB L2

430MHz C64x+ DSP32K/32K L1

48K L1D32K L2

Power VR SGX GPU64K on-chip RAMPOP Memory256MB LPDDR RAM256MB NAND flash

RESET

7.6 cm Towards Embedded Computer Vision p. 48

8/13/2019 20131102-Towards ECV-

48/66


System

2010.04.25

48 Towards Embedded Computer Vision p. 49

8/13/2019 20131102-Towards ECV-

49/66


Software


8/13/2019 20131102-Towards ECV-

50/66


FJUCam2


8/13/2019 20131102-Towards ECV-

51/66


Next : UAV


8/13/2019 20131102-Towards ECV-

52/66


Next : Sleep Monitoring


8/13/2019 20131102-Towards ECV-

53/66


Next : Vehicular

Object Detection


8/13/2019 20131102-Towards ECV-

54/66


Heterogeneous Multicore

! Xilinx Zynq: ARM+FPGA


8/13/2019 20131102-Towards ECV-

55/66


Future ofEmbedded Vision


8/13/2019 20131102-Towards ECV-

56/66


Summary

! Embedded CPU is not appropriate forECV" Pros" Great flexibility" Fast development

" Cons"

No float-point arithmatics"Interrupt-based architecture

! Smart phone with 4/8 cores is OK? Towards Embedded Computer Vision p. 57

8/13/2019 20131102-Towards ECV-

57/66


I Think!

!Embeddedcompute vision"Low-cost, low-power,

minimal hardware

! High-Performancecomputer vision" Fast speed without cost,

power, and hardware

constraintsFrom contradictionto convergence ?!

ConstrainedResource

AbundantResource

Yes by multicore

Contradiction


8/13/2019 20131102-Towards ECV-

58/66


Challenges (1/2)

! Algorithm decomposition" Function decomposition" Partition serial part and parallel part" Data flow analysis and

data dependency analysis" Parallelism" Loop unrolling"

Multithreading" Pipeline


8/13/2019 20131102-Towards ECV-

59/66


Challenges (2/2)

! Performance analysis method" For efficiency improvement! Implementation efforts" Choose a good embedded platform for

computer vision" Software issues" Hardware issues

"Programming skills" Multi-threading


8/13/2019 20131102-Towards ECV-

60/66


Future Research Directions

HeterogeneousMulticore

ARM+ DSP

ARM +FPGA

ARM +GPGPU


8/13/2019 20131102-Towards ECV-

61/66


I Think Again!

! Is there a development platform forembedded computer vision with" Built-in camera" AE, AF, AWB

" High-performance processor" Heterogeneous multicore

" Software development environment:"

C, Matlab" OpenCV" Sample codes" Labs


8/13/2019 20131102-Towards ECV-

62/66


Embedded Vision Alliance


8/13/2019 20131102-Towards ECV-

63/66


! Ramesh RaskarMIT Platform


8/13/2019 20131102-Towards ECV-

64/66


! Andrew AdamsStanford Platform

8/13/2019 20131102-Towards ECV-

65/66


8/13/2019 20131102-Towards ECV-

66/66

!Creative Commons - 3.0 ! (1) (2)!

Date post:	04-Jun-2018
Category:	Documents
Upload:	yuan-kai-wang
View:	228 times
Download:	0 times

20131102-Towards ECV-輔大

Documents