www.mtri.org

J. Garbarino, C. Roussi, B. Whitewww.mtri.org/unpaved

ALGORITHM/SYSTEM OVERVIEWRITARS-11-H-MTU1

Technical Advisory CommitteeProject Update – March 24, 2015

Overview

Review– Data Collection System

• Airborne platform(s)• Camera, lens, GPS• Intervalometer

– Data Processing System

• Algorithm Suite• Results

Future steps

2

Unmanned Platform

Bergen Folding Hexacopter– 7kg flight-ready– Gyro-stabilized platform

Nikon D800 w/GPS

Custom Intervalometer

3

Manned Platform

Cesna 152 or equivalent ($170/hr)

Nikon D800 w/ 70-200mm f/2.8 lens ($5100)

Intervalometer ($200)

Tyler mini-gyro stabilized mount ($500)

4

System Specifications

5

Hexacopter

Batteries 26Ah, 14.8V

Weight 4kg empty, 5.5kg w/ batteries

Flight Time 20min hover, 10min full power

Range 5km

Ceiling ~3000m

Misc accessories

Charger, flight controller, tools

Cost $5200

D800

Resolution

36Mp (7360x4912)

Weight 1kg (1.3kg w/ lens)

Frame advance

5 fps max

Speed 1/8000s – 30s

ISO 100 - 6400

Cost $2800

Nikkor 50mm Lens

F-stop f/1.4 – f/16

Weight 0.3kg

Field of view 31 deg

Cost $480

Intervalometer (custom)

Interval Range 5s – 1/4s

Weight 200g

Battery 9V alkaline

Cost $200Costs Unmanned Manned

Equipment $8700 $5800

Operating 1 FTE 1 FTE + $170/hr

Nikkor 70mm-200mm

Lens

F-stop f/2.8 – f/22

Weight 1.5kg

Field of view 12-34 deg

Cost $2400

Tyler Mount gyro

Size 25” x 20” x 13”

Weight 27kg w/ batteries

Cost $500

Unmanned Concept of Operations

Typical Collection involves:– Assemble hexacopter and pre-

flight check – 7min– Determine camera settings and

controller setup – 2min– Flight collection – 2min for

100m– Stow equipment – 5min– Charge batteries – 20min

Typical selection and processing – 4 hrs

6

Collect Data

Select Site

Select Data

Process Data

Evaluate in Roadsoft

Manned Concept of Operations

Typical Collection involves:– Emplacing tyler mount

(may involve a modified aircraft w/ port in hull) – 10min

– Determine camera settings and controller setup – 2min

– Flight collection • 4s for 100m• 2-passes needed for

coverage– Stow equipment – 10min– Charge batteries – 20min

Typical selection and processing – 4 hrs

7

Collect Data

Select Site

Select Data

Process Data

Evaluate in Roadsoft

Platform Comparison

8

Hexacopter Pros Cons

Easy to transport and deploy Can only collect 1km of road on a set of batteries

Data can be collected by a single person May require closing road for safety

Operating costs only involve 1 person’s hourly rate

Some ground-truth can be collected manually, if needed

No recurring costs

Manned Aircraft Pros Cons

Can collect more road segments during a flight

Requires a modified aircraft for gyro-stabilized mount

Can operate over a wider range of weather conditions

Involves a pilot and an operator

Requires multiple passes over the same road to get sufficient coverage for 3D reconstruction

Requires more careful selection of images (manually) for processing

Software Architecture

Because we are incorporating legacy code, third-party tools, and custom code, we need a flexible architecture

– Developed in C, C++, Python, bash– Flexible control, with tools calling each other as needed

9

Algorithm

Use Structure from Motion (SIFT+ Bundler + PVMS) to turn 2D images into 3D point-cloud reconstruction

– SIFT = scale-invariant feature transform– PVMS = patch-based multi-view stereo

Form a surface from the 3D point-cloud– Form grid, Fourier Filter, Marching Cubes to triangulate

Find the depth/height map of the surface– Singular Value Decomposition (SVD)– Rotate so z-axis is “up” (depth)

10

Algorithm

Find and select the road in the scene– Image entropy measure (road is “smoother”)

Rotate extracted road into new coordinate system– Makes it easier to take cuts along and across road

Analyze for features of interest– Gabor Filtering, Circular Hough Transform, Cuts for profiles of

road and drainage

Convert to PASER-like metrics

Generate XML output suitable for RoadSoft processing

11

Algorithm

12

Data Collection

Image Q/C

Preprocessing

SIFT

Bundler

Surface from Point Cloud

SVD to Find Depth Map

Distress Extraction

Characterization

Translation to RoadSoft

PMVS Analysis

RoadSoft Processing

Algorithm Details

13

Data Collection

Road Finding (image entropy metric)

Orient road in image

Ruts, Corrugations (Gabor Filtering)

Potholes (Circular Hough Transform)

Edge Cuts for Berm, Loose Agregate

Apply Detection Maps to Depth Map

Compute Statistics on Distresses

Convert to PASER-like ratings

Output to RoadSoft

Transverse Cuts for Crown Loss

Generate XML

Example Image

Taken from 25m, 2m/s

14

Example Reconstruction

15 images used for reconstruction

15

Bundler output Densified point cloud

3D surface from point cloud Height-field from surface

Road Segmentation from Depth Map

16

Rotated Depth Map Mask of Road Surface from image Entropy

Extracted Road

Depth Map Detail

17

Input to Crown Measurement

18Across Road

Alo

ng

R

oad

Example crossection plot

meters

Pothole Detection

19

XML Report

Ruts and Corrugations

20

Performnace Summary

Crown estimates vary from manual ratings slightly– We measure the crown everywhere; manual inspections

sample the surface poorly

Ruts: Pd = 67%, Pfa = 19%

Corrugations: Pd = 100%, Pfa = 38%

Potholes: Pd = 95%, Pfa = 4%

21

Observations

Deep ruts are sometimes labeled as potholes

Strings of potholes along the driving direction are sometimes labeled as corrugations

Strips of grass on the road surface cause false alarms

Manual scoring is a trade-off between accuracy and time– Spot checks– Spend long enough to get a “good enough” estimate

Automated scoring finds everything– Can be both good and bad

22

Room for Improvement - General

Each step in the process can be addressed– Collection parameters– Image quality– Image pre-processing to enhance– Processor operating points– Algorithm choices for current distresses

• Old can be refined• New can be tried

Expanded Applications

23

Room For Improvement - Specific

Automatic rating/rejection of unsuitable images– Blurred images limit reconstruction accuracy

Tuning of algorithms– Each process has “knobs” to adjust performance– Internal operations can be refined

• E.g. changing entropy estimation routine

– Throughput optimization

Script additions to expose more controls– Adding switches makes it more flexible– Set/reset detection points

Adding data exploitation routines (not all information that can be gleaned from the data has been)– Intersection geometry– Texture changes

24

Example

One way of characterizing an intersection is by abstracting its geometry– Leveraging computer vision morphological tools

Sample intersection w/ non-perfect segmentation, followed by medial-axis transformation– Finding the “skeleton” of the intersection– Imaging a brush-fire starting at the boundary; the place where

the fire meets is the media axis

25