Cooperative Unmanned Vehicles for Vision-based ... · HOG1-Based Human Classification...

Cooperative Unmanned Vehicles for Vision-based Detection and Real-World

Localization of Human Crowds Sponsor:

Air Force Office of Scientific ResearchDDDAS Program (Dr. Darema)

Sara Minaeian, Dr. Jian Liu, Dr. Young-Jun Son

Systems and Industrial Engineering,The University of Arizona

November 3, 2015

Computer Integrated Manufacturing & Simulation Lab

Agenda

• Scope of the Crowd Control Project

• Crowd Detection by UAV and UGV

• Real-world Localization

• System Implementation

• Experiment and Results

• Summary and Ongoing Works

1


Problem Motivation• Motivation: TUS-1 BP Project (23-mile long border area of Sasabe, AZ)

• Problem: highly complex, uncertain, dynamically changing environment

• Main goal: create scalable, robust, multi-scale, and effective surveillance& crowd control strategies using collaborative UVs

• Proposed approach: a comprehensive planning and control frameworkbased on dynamic-data-driven, adaptive multi-scale simulation (DDDAMS)

21.Reuters, FY2000 ~ FY2013 : http://graphics.thomsonreuters.com/14/immigration/index.html

1

1


3

DDDAMS-based Framework

1. Khaleghi, A. M., Xu, D., Lobos, A., Minaeian, S., Son, Y. J., & Liu, J. (2013, December). Agent-based hardware-in-the-loop simulation for UAV/UGV surveillance and crowd control system. In Proceedings of the 2013 Winter Simulation Conference (pp. 1455-1466). IEEE Press.

1


• Objective of Detection Module: vision-based detection ofthe moving targets (crowd) dynamically at each time stamp t

• Objective of Localization: finding the real-world location ofthe detected targets and send them to the Tracking module forpredicting their locations at

4

Crowd Detection Module

Crowd Tracking Module

Motion Planning ModuleDetected crowd locations

at time tPredicted crowd locations

at time .

Scope of the Project

t t

t t

x(t ), y(t )


Agenda







5


UAV-UGV Cooperation in Detection• Cooperation in detection & localization of the moving crowd

• Unmanned Aerial Vehicle (UAV)– Further distance– Lower resolution– Top-down perspective of moving objects– More appropriate for Crowd detection– Optical-flow-based Motion Detection

6

• Unmanned Ground Vehicle (UGV)– Closer distance– Higher resolution– Upright perspective of human– More appropriate for Individual detection– HOG-based Human Classification


1. Extract featured keypoints from frames t using GFTT1

Crowd Detection Module UAV (1)

71. J. Shi and C. Tomasi, “Good features to track,” in Computer Vision and Pattern Recognition, 1994. Proceedings CVPR’94., 1994 IEEE Computer Society Conference on, 1994, pp. 593–600.

Iui

2i Iui

IviiIui

Ivii Ivi

2i

• Autocorrelation matrix of the second derivative images• Good features: 2 eigenvalues greater than a minimum threshold


2. Track (match) keypoints over frames t+1 and t+2 using Optical FlowMatched keypoints over frame t+1 Matched keypoints over frame t+2


3. Perspective warp of frames t+1 and t+2 onto frame t using HomographyWarped frame t+1 over t Warped frame t+2 over t

8


I (T)t1 Ht It1

• (3.1) Apply RANSAC on each pairs of frames for filtering out the outliers (keypoints belonging to the foreground)

• (3.2) Estimate the Homography*

between frames t and t+1; & between frames t and t+2 :

• (3.3) Warp frames t+1 and t+2 onto frame t based on perspective transformation using H matrix:

H

h11 h12 h13

h21 h22 h23

h31 h32 h33

Crowd Detection Module UAV (3)2 Frames: Compensate the background motion and reduce the registration error;

By referencing the same Homography

9

It1 Ht It1 It2 Ht It2

I (T)t2 Ht It2

* In the field of computer vision, any two images ofthe same planar surface are related by a Homography:


• (5.1) Make the silhouette– By applying threshold

• (5.2) Smoothen the image– By applying Gaussian filter

• (5.3) Dilate* the image– To fill small holes in the motion area

• (5.4) Erode* the image– To remove small and separated noise

4. Take absolute differences between transformed frames t+1 and t+2

Dilation

-

Erosion

* Morphological transformation applying a Kernel


10


5. Segment the moving foreground and assign target boundaries: Detection


11


Individual Detection Module UGV

: OpenCV classifier

12

HOG1-Based Human Classification : 3x3 derivative mask[-1, 0 , 1]

: Weighted voting over cells6x6 pixel cells

: Grouping cells into blocks3x3 cell blocks

: L2-norm

Gradient

computation

Orientation

binning

HOG over des. blocks

Block normalization

Classify the

target

1. Dalal, N., & Triggs, B. (2005, June). Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on (Vol. 1, pp. 886-893). IEEE.

Blocks

Cells


Agenda



• Real-World Localization




13


Challenges for Target Localization via UAV

No clue (color, template, size) of the target to detect.

Moving camera and changing background.

Unknown transformation between Image plane and

Earth plane.

Vertical positioning error of the UAV, as well as

Lateral positioning error.

Lateral positioning error of the UGV only.


• Perspective transformation between two planes 1

• Preserves collinearity

151. Criminisi, A., Reid, I., & Zisserman, A. (1999). A plane measuring device. Image and Vision Computing, 17(8), 625-634.

A’ B’

C’D’

D

A B

C

O: center of projection

u ax by rpx qy 1

v cx dy spx qy 1

X=MU(x, y)(u,v) : Image position

: World locationM :Transform matrix

(8 parameters)

Landmark-based Localization


Localization Framework (IEEE SMC) 1

16

Landmarks with: (x,y): known GPS location, (u,v): detected image location.

UAV

UGVs as colored landmarks

UAV’s detection range

• At least 4 coplanar, non-collinear landmarks• UGVs with their real-world location (GPS) and image position (colored) known

• UAV is flying high enough,• UGVs and crowd are close enough,• Disregard any internal depth

differences between the UGVsand crowd on earth’s plane.

UAV’s detection range

UGV’s detection range

Crowd’s individuals

UGV

UAV

Weak Perspective approximation:

1. Minaeian, S, Liu, J., & Son, Y. Vision-based Target Detection and Localization via a Team of Cooperative UAV and UGVs, IEEE Transactions on Systems, Man, and Cybernetics (Accepted).


Localization Algorithm UAV• Landmark ( ) with image position ( ) at time t and

real-world geographic location ( ) at time t• Using homogeneous coordinates• Transformation matrix between two planes:

where: and

• System of linear equations: where

17

i 1,...,ni ui (t ), vi (t )

xi (t ), yi (t )

Ui (t )

Vi (t )

Wi (t )

M

xi (t )

yi (t )

1

M

ui (t ) Ui (t ) Wi (t )

vi (t ) Vi (t ) Wi (t )

M

a b rc d sp q 1

Am k m a, b, r, c, d, s, p, q T


Solving the System of Equations• Three alternatives:

– If Under-determined

– If Exact solution:

– If Linear Least Squares:

Inverse method Not very stable numerically & expensive Gaussian elimination/ Back substitution Less expensive

• For any detected target with image position ( ) andunknown GIS-based location ( ) at time t

18

n 4; n m A-1k

n 4; n m (AT A)1AT k

u (t ), v (t )

x (t ), y (t )

n 4; n


Landmark Assignment Problem• UGVs (as landmarks) are also moving• How to differentiate these landmarks from the moving targets?

Using an Assignment Model:

Mathematical Model(m>n)

Minimizing the Total Errorof Landmark Assignment

Euclidean distance between ith color-detected and jth motion-detected objects

Decision variable to relate corresponding motion-detected objects and landmarks

i-j pair assignment constraints

min Dijzijj1

mi1

ns.t. zijj1

m 1 i 1,...,n

ziji1

n 1 j 1,...,m

zij 0,1 i, j

19


• UGVs also need to transform image coordinates to real-world positions

• Compute camera’s calibrated focal length:

• Estimate target j ’s distance from the UGV:

• UGV’s camera pose:

• Target j ’s real-world pose (location + orientation):

20

Longj , Lat j LongC Dj sin( j ), LatC Dj cos( j )

f D hH

Dj f Hhj

LongC , LatC ,C

j C tan1(devj f ) j

Localization Heuristic UGV

C jDj cos( j )

Dj sin( j )

Lat

Long

(decimal degree)

(decimal degree)

DjLongC , LatC

Target j

UGV’s Camera


Agenda







21


Real Testbed for Detection

Carl Zeiss Tessar HD1080pHD (16:9): 1280x720p @ 30 fpsFOV: 90

22

GoPro HERO 3+ Tarot GimbalHD (16:9): 1280x720p @ 120 ~ 25 fpsFOV(x): 64.4 ; FOV(y): 37.2

DR(x)

DR(y)

FOV (x)

FOV(y)

h

Onboard Computer: ODROID U31.7 GHz quad-core ARM-Cortex-A9 Linux-based operating system


Simulated Testbed for Localization

23

• No adequate number of real UGVs to run the localization algorithm• Higher costs of conducting experiments on the border area of Tucson• UGVs and crowd need to move randomly as independent agents in a simulated

border area (using GIS information)• Platform: Repast Simphony (Open source, Java, NASA World Wind SDK)


Agent-based SimulationRepast Simphony with 3D GIS

UAV(APM:Copter / Arducopter)

UGV(APM:Rover / Ardurover)

Sensory Data (e.g. GPS)

Control Commands (MAVLink Messages)

Hardware Interface

Hardware-in-the-Loop Testbed

HOG-BasedHuman Classification

Optical-Flow-Based Motion Detection

24


Agenda







25


Agent-based Simulation for Localization• Random movements of the crowd in a N-E path in border area of Tucson, AZ• Changing parameters:

– Flight altitude– Number of landmarks– Landmark assignment method

26

Motion-Detected Crowd

Color-Detected

UGVsAgent-based Simulation

Movement speed :1 ~ 3 m/s

Detection & LocalizationAlgorithm


Experiments: Localization (Altitude)

27

31.3318

31.3319

31.3320

31.3321

31.3322

31.3323

31.3324

-110.9046 -110.9045 -110.9044 -110.9043 -110.9042 -110.9041 -110.9040 -110.9039

Lat

itude

(dec

imal

deg

ree)

Longitude (decimal degree)

Simulated Target Detected Target (at 50 m) Detected Target (at 200 m)

0

2

4

6

8

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

Euc

lidea

n E

rror

(M

eter

s)

Simulation Time Stamp (tic)

Error for 50 m Altitude Error for 200 m Altitude

Average Euclidean Error: 3.5 m

Average Euclidean Error: 1.3 m 63% improvements


Experiments: Localization (Landmarks #)

28

31.3318

31.3319

31.3320

31.3321

31.3322

31.3323

31.3324

-110.9046 -110.9045 -110.9044 -110.9043 -110.9042 -110.9041 -110.9040 -110.9039

Lat

itude

(dec

imal

deg

ree)

Longitude (decimal degree)

Simulated Target Detected Target (4 Landmarks) Detected Target (6 Landmarks)

0.000

0.002

0.004

0.006

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Err

or (K

ilom

eter

s)


Error for 4 Landmarks Error for 6 Landmarks



Euc

lidea

n E

rror

(M

eter

s)

6

4

2

0

12% improvements


Experiments: Localization (Assignment)

29

32% improvements:motion-based landmark assignment

versuscolor-based landmark assignment

0.000

0.001

0.002

0.003

0.004

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Err

or (K

ilom

eter

s)


Error for 6 Landmarks; Color-based Error for 6 Landmarks; Motion-based

Error for 4 Landmarks; Color-based Error for 4 Landmarks; Motion-based

Euc

lidea

n E

rror

(M

eter

s)

4

3

2

1

0

2.24

2.49

3.52

3.48

0.0 1.0 2.0 3.0 4.0 5.0

Average Euclidean Error (Meters)

Colored-based assignment: 6 Landmarks

Colored-based assignment: 4 Landmarks

Motion-based assignment: 6 Landmarks

Motion-based assignment: 4 Landmarks


30

Stationary Aerial Non-Stationary Aerial

Camera Pose Cliff et al., 2015 Farmani et al., 2014Redding et al., 2006

Fixed Landmarks Martinez et al., 2011Wang et al., 2014

Dementhon & Davis: POSIT

Moving Landmarks ---

Literature on Target’s Geo-Localization

Contribution of the Work (IEEE SMC) 1

1. Minaeian, S, Liu, J., & Son, Y. Vision-based Target Detection and Localization via a Team of Cooperative UAV and UGVs, IEEE Transactions on Systems, Man, and Cybernetics (Accepted).


Agenda







31


Summary and Ongoing WorksCrowd Detection and Localization using Team of UVs

o Collaborative crowd detection: Motion detection (UAV) Human classification (UGV)

o Collaborative localization: Landmark-based localization (UAV) Heuristic localization (UGV)

o Testbed and experiments: Hardware-in-the-Loop Agent-based simulation Sensitivity Analysis

o Ongoing Work: Human-in-the-loop Hardware-in-the-loop Simulation

32


33

Sara Minaeian: [email protected]. Jian Liu: [email protected]. Young-Jun Son: [email protected]

Date post:	14-Jul-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

Cooperative Unmanned Vehicles for Vision-based ... · HOG1-Based Human Classification...

Documents