Post on 25-Mar-2020
transcript
Vision-guided Planning and Control for Autonomous Taxiing via Convolutional Neural NetworksPresenter: Chang LiuAdvisor: Silvia FerrariLaboratory for Intelligent Systems and Controls (LISC)Cornell University
AIAA SciTech 2019GNC-18/IS-10, Advances in Adaptive Control Systems I
San Diego, CA, January 8, 2019
1
2
Outline
Motivation and Background Problem Formulation Technical Approach:
Object Recognition using Mask-RCNN ATC-based Path Planning Hybrid Control of Autonomous Taxiing Aircraft
Simulation Results Conclusions
3
Motivation and Background
Aerodromes are becoming increasingly complex and crowded
Dangerous situations seriously affects aerodrome safety
Motivation
Crowded airports Runway incursion incidents
Year2018
Operational Incident
Pilot Deviation
Vehicle Pedestrian Deviation
Other Total
Totals 345 1142 335 10 1832
Incidents of runway incursions in 2018 (from FAA)
4
Autonomous Taxiing:
Automating aircraft taxiing process without human intervention
Take Air Traffic Control (ATC) commands in the loop
Detect obstacles and unforeseen conditions in situ
Generate corrective planning and control to guarantee aircraft safety
Motivation
ATC tower Unforeseen objects
5
Surface Operation Scheduling: schedule the use of taxiways, runways, and gates to reduce the overall travel time and maximize throughput (Morris 2016)
Aircraft Path Planning and Control: Generate energy-efficient and collision-free trajectories based on aircraft motion models (McGee 2007, Coetzee 2011, Chen 2016, Zhang 2018)
Related Work
Situation Awareness for Planning: Uses machine learning algorithms for taxiway feature extraction and unknown obstacle identification (Lu 2016, Lu 2018)
Limitations: Lack of ability to incorporate ATC commands Planning and control ignores environmental perception Unable to handle unexpected dangerous situations
6
7
Problem Formulation
Continuous controller
Problem Formulation
Goal: Develop a vision-guided path planning and control approach for autonomous taxiing under both normal conditions and unforeseen conditions
8
)](),([)1( kkk ξξ usfs =+ )()1( kk µξ =+
)(kξu
Hybrid system modeling:
System discrete mode System continuous state
Objective function Continuous dynamics
Planning horizon Stage cost
sJ
∑−
=
Ψ+=1
))(),(),(())((fk
kjf jjjkJ µϕ ξξ uss
Objective function
,ϕf
],[ fkk
Discrete controller
Normal conditions
Unforeseen conditions
Ψ
Yaw angle Speed
Steering angle Acceleration
Sampling interval Front-rear axle distance
Aircraft Motion Model
The aircraft uses a simple car model, defined as,
Coordinate of the rear-axle center
𝑥𝑥, 𝑦𝑦
𝜃𝜃
𝜙𝜙
𝑣𝑣
9
The simple car model of an aircraft
T
k
kLk
kkkk
kkk
kkk ee
∆
+
=
+++
)(
)(tan)()(sin)()(cos)(
)()()(
)1()1()1(
β
φυθυθυ
υθ
υθ
θ υ
β
LT∆
[ ]Te kkkk )()()()( υθqs =Aircraft stateT
e yx ],[=q
Aircraft control input
Camera Measurement Model
Camera measurement model
Recognizing three semantic classes of incursion objects
“People”
“Animals”
“Ground Vehicles”
objects classification
camera-to-object distance
10Animal VehiclePeople
Onboard RGB-D Camera
Camera View
},2,1,0{)( ∈kl
0)( ≥kdl
Tl kdklk )](),([)( =z
1)( =kl
2)( =kl
3)( =kl0)( =kl means no detected object
Taxiway region
Runway region
Terminal region
Airport Modeling
Taxiway label set
Runway label set
Terminal set
Airport Diagram Airport Satellite Map11
},{ , VA, E, P, FL =A
}33152810{ , , , L =P
},{ 2210310 ,, g, g, gggL =G
,2RWa ⊆ ALa∈
,2RWp ⊆ PLp∈2
0RWg ⊆
,2,1, =igi represents the ith gate
Airport Modeling
A
B
B E
A
hB
h(A,B) h(G,28)
h28
G
28
h(D,g0)
D
Example nodes:
g11
12
Nodes , where connection of two regions terminal gates aircraft current position
},,{ EBAvi =},{ BAv j =}28,{Gvl =
},{ 011 ggvm =},{ 0gDvn =
Taxiway centerline Runway centerline
,0)( =qah,0)( =qph
Taxiway connecting arc ,0)( =qAh AA LLa ×∈
Taxiway-runway arc ,0)( =qPh PA LLP ×∈
Taxiway-terminal arc ,0)( =qTh }{ 0gLT ×∈ A
ALa∈
PLp∈
vi
vj
),( qγ=vGPA LLL ××∈γ
vl
vmvn
Airport Modeling
13
Node set
Edge set Terminal gate region
Runway
Taxiway ATaxiway B
Taxiway C
2002
Simulated airport
Airport Graph: a topological graph representing the connectivity of different airport regions.
Partial graph of the airport
}{vV =
},|{ Vvvvv jiji ∈∩=Ξ
),( Ξ= VG
Nodes, centerlines, and connecting arcs
ATC Commands
Command Category Instruction ATC Command Examples
Cruising Move along certain taxiways to a runway.
• “Runway Three-Six Left, taxi via Taxiway Alpha, hold short of Taxiway Charlie.”• “Cross Runway One-Six Left and Runway One-Six Right at Taxiway Bravo.”
Traffic Following Follow the traffic. • “Follow (traffic), cross Runway Two-Seven Right, at Taxiway Whiskey.”
Holding Hold short of a runway or hold in position.
• “Hold short of runway Two-Seven.”• “Hold in position.”
14
c
15
Technical Approach
Object Recognition using Mask-RCNN
Mask-RCNN: state-of-the-art object detector
16
Consists of a region proposal network and a binary mask classifier
Uses RGB images as input
Outputs a class label, bounding box, segmentation mask, and a confidence level The Mask-RCNN structure (He 17’)
Recognition results using Mask-RCNN
reference waypoint
ATC command sequence
ATC-based Path Planning
ATC command where and
• Example: “Runway Two-Zero, taxi via Taxiway Alpha and Bravo”
Example: node sequence
Path Generation from ATC Commands• Connect graph nodes in accordance with ATC commands
• Turn node sequence into waypoints
),,,,( 121 nn ψψψψ −=Ψ
ALn ∈−121 ,,, ψψψ GPA LLLn ∪∪∈ψ
)20,,( BA=Ψ⇒
),,,,,( 15141398 vvvvvvs
]))(),((,,))1(),1([(* Trr
Trr NyNxyx =q
),( rr yxΨ 17
Hybrid Control of Autonomous Aircraft
The optimization problem in mode is defined as
ξxmin
..ts 0)( ss =k1,,)),(),(()1( −==+ fkkjjjj ξusfs
1,,)( −=∈ fkkjSj s
1,,)( −=∈ fkkjUj ξξu
( ) ( )∑−
=
Ψ+=1
)(),(),(),()(),(),(fk
kjfff jjjyjxkkykxJ ξξ υυϕ u
Reference waypoint Aircraft initial state
State feasible set Control feasible set
),( rr yx 0sS ξU
T
f
TTkkkk )]1(),(),(,),([ −= ξξξ uussx is the optimization variable
Five aircraft modes
18
{cruising (1), traffic following (2), holding (3), incursion (4), idle (5)}
ξ
where:
Continuous State Control
Cruising mode: Follow the reference path and maintain a desired cruise speed
∑−
=
+−+
−=1
2
21
2
2
2
21
)(]),(),([)](),(),([
]),(),([)](),(),([fk
kj
Tcrr
T
Tcfrfr
Tfff
jjyjxjjyjx
kykxkkykxJ
uυυ
υυcυ
Traffic following mode: Follow the reference path and maintain a speed similar to the aircraft in front
∑−
=
+−+
−=1
2
22
2
2
2
22
)(]),(),([)](),(),([
]),(),([)](),(),([fk
kj
Tfrr
T
Tffrfr
Tfff
jjyjxjjyjx
kykxkkykxJ
uυυ
υυ
fυ
Holding mode: Decelerate to stop at the hold-short position
∑−
=
+−+
−=1
2
23
2
2
2
23
)(]0),(),([)](),(),([
]0),(),([)](),(),([fk
kj
Trr
T
Tfrfr
Tfff
jjyjxjjyjx
kykxkkykxJ
uυ
υ
with the additional constraint 0)( =fkυ19
Continuous State Control
Incursion mode: Decelerate to avoid collision with the incursion object
∑−
=
+−+
−=1
2
24
2
2
2
24
)(]0),(),([)](),(),([
]0),(),([)](),(),([fk
kj
Trr
T
Tfrfr
Tfff
jjyjxjjyjx
kykxkkykxJ
uυ
υ
Idle mode: Remain current state
1,,],[)](),([2
2−=≥− fs
Too
T kkjdyxjyjx
with the additional collision avoidance constraint
Object position
Safety distance
),( oo yx
sd
0u =5
20
Discrete State Control Definition
Cruising mode
Traffic following mode
==∈
====
=
otherwise0)( and 0)( if }3,2,1{)( if
)( and 0)( if)( and 0)( if
)(
3
5
4
22
11
µυµ
µµµ
µkkl
klckcklckckl
k
Holding mode
Incursion mode
====
=
otherwise0)( if
)( if)( if)( if
)(
4
5
33
22
11
µυµ
µµµ
µk
ckcckcckc
k
==
=otherwise
)( if)( if
)(
5
22
11
µµµ
µ ckcckc
k
Idle mode
ATC command category
Object classification)(kl321 ,, ccc
∈====
=
otherwise }3,2,1{)( if
)( and 0)( if)( and 0)( if
)(
1
4
33
22
µµµµ
µkl
ckcklckckl
k
21
==∈
====
=
otherwise0)( and 0)( if }3,2,1{)( if
)( and 0)( if)( and 0)( if
)(
2
5
4
33
11
µυµ
µµµ
µkkl
klckcklckckl
k
where: and ii =µ
Visualization of Discrete State Control
),( 5usfs =
5=ξ
),( 1usfs =
1=ξ
),( 2usfs =
2=ξ
),( 3usfs =
3=ξ
),( 4usfs =
4=ξ
2)( and 0)( if ckckl ==
3)( and 0)( ifckc
kl=
=
}3,2,1{)( if ∈kl}3,2,1{)( if ∈kl
}3,2,1{)( if ∈kl
1)( and 0)( ifckc
kl=
=
0)( and 0)( if=
=k
klυ
22
23
Simulation Results
Runway
Taxiway ATaxiway B
Taxiway C
2002
Simulation Setup
A simulated small-sized airport modeled in UnrealEngine®
Simulated airport
Runway and taxiways Ground markings and signs Terminal area 24
Results – Normal Taxiing
Taxis along Alpha and Bravo to Runway Two-Zero following ATC commands
25
Trajectory
y(m
)
x(m)Velocity Profile
v(m
/s)
t(s)
26
Results –Incursion Events
Original plan: taxi along Alpha and Charlie to Runway Two-Zero Incursion object detected and classified as car. New path generated based on ATC commands.
Velocity Profile
v(m
/s)
t(s)
Object Detection
27
Conclusions
Vision-guided Autonomous Taxiing:
Airport modeling using airport diagrams and geographical information.
A systematic approach that enables real-time aircraft perception, obstacle avoidance, and feedback control with ATC commands in the loop
Future Work:
Robustness analysis of the proposed approach
Incorporate action recognition and environmental semantic understanding for airport environments
Conclusions
28
Thank you
Questions?
29
Lu, W., Zhu, P., and Ferrari, S., “A hybrid-adaptive dynamic programming approach for the model-free control of nonlinear switched systems,” IEEE Transactions on Automatic Control, Vol. 61, No. 10, 2016, pp. 3203–3208
Zhu, P., Isaacs, J., Fu, B., and Ferrari, S., “Deep learning feature extraction for target recognition and classification in underwater sonar images,” IEEE 56th Annual Conference on Decision and Control (CDC), 2017, pp. 2724–2731.
Lu, W., Zhang, G., and Ferrari, S., “A randomized hybrid system approach to coordinated robotic sensor planning,” 49th IEEE Conference on Decision and Control (CDC), 2010, pp. 3857–3864.
References
30