MASKS © 2004 Invitation to 3D vision

Lecture 11Vision-based Landing of an

Unmanned Air Vehicle

MASKS © 2004 Invitation to 3D vision

Applications of Vision-based Control

Fire Scout

Global HawkPredator

SR/71

UCAV X-45

MASKS © 2004 Invitation to 3D vision

Goal: Autonomous landing on a ship deck

Challenges• Hostile environments

Ground effect Pitching deck High winds, etc

Why vision?• Passive sensor • Observes relative motion

MASKS © 2004 Invitation to 3D vision

Simulation: Vision in the loop

MASKS © 2004 Invitation to 3D vision

Vision-Based Landing of a UAV

• Motion estimation algorithms Linear, nonlinear, multiple-view Error: 5cm translation, 4° rotation

• Real-time vision system Customized software Off-the-shelf hardware

• Vision in Control Loop• Landing on stationary deck• Tracking of pitching deck

MASKS © 2004 Invitation to 3D vision

Vision-based Motion Estimation

Pinhole Camera

Landing target

Image plane

Feature Points

Current pose

MASKS © 2004 Invitation to 3D vision

Pose Estimation: Linear Optimization

• Pinhole Camera:• Epipolar Constraint:• Planar constraint:

• More than 4 feature points• Solve linearly for• Project onto to recover

MASKS © 2004 Invitation to 3D vision

Pose Estimation: Nonlinear Refinement

• Objective: minimize error

• Parameterize rotation by Euler angles

• Minimize by Newton-Raphson iteration

• Initialize with linear algorithm

MASKS © 2004 Invitation to 3D vision

Multiple-View Motion Estimation

Multiple View Matrix

Rank deficiency constraint

Pinhole Camera

MASKS © 2004 Invitation to 3D vision

Multiple-View Motion Estimation

• n points in m views

• Equivalent to finding s.t.

• Initialize with two-view linear solution

• Least squared solution:

• Use to linearly solve for• Iterate until converge

MASKS © 2004 Invitation to 3D vision

Real-time Vision System

• Ampro embedded Little Board PC Pentium 233MHz running LINUX 440 MB flashdisk HD robust to vibration Runs motion estimation algorithm Controls Pan/Tilt/Zoom camera

• Motion estimation algorithms Written and optimized in C++ using LAPACK Estimate relative position and orientation at

30 Hz

UAV Pan/Tilt Camera Onboard Computer

MASKS © 2004 Invitation to 3D vision

Hardware Configuration

On-board UAVVision System

Vision Computer

RS232

RS232

Vision Algorithm

Frame Grabber

Camera

WaveLAN to Ground

Navigation SystemNavigation Computer

RS232 RS232

Control & Navigatio

n

INS/GPS

WaveLAN to Ground

MASKS © 2004 Invitation to 3D vision

Feature Extraction

• Acquire Image• Threshold Histogram• Segmentation• Target Detection• Corner Detection• Correspondence

MASKS © 2004 Invitation to 3D vision

• Pan/Tilt to keep features in image center Prevent features from leaving field of view Increased Field of View Increased range of motion of UAV

Camera Control

MASKS © 2004 Invitation to 3D vision

Ground Station

Comparing Vision with INS/GPS

MASKS © 2004 Invitation to 3D vision

Motion Estimation in Real Flight Tests

MASKS © 2004 Invitation to 3D vision

Landing on Stationary Target

MASKS © 2004 Invitation to 3D vision

Tracking Pitching Target

MASKS © 2004 Invitation to 3D vision

Conclusions

• Contributions Vision-based motion estimation (5cm accuracy) Real-time vision system in control loop Demonstrated proof of concept prototype:

first vision-based UAV landing• Extensions

Dynamic vision: Filtering motion estimates Symmetry-based motion estimation Fixed-wing UAVs: Vision-based landing on runways Modeling and prediction of ship deck motion Landing gear that grabs ship deck Unstructured environments: Recognizing good

landing spots (grassy field, roof top etc)