DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 11
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited
Autonomy Validation, Introspection, and Assessment (AVIA)
System Description and Lessons Learned
By Miles ThompsonResearch Engineer II
Aerospace, Transportation, and Advanced Systems Lab
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 2
• AVIA is a DARPA project to develop assessment tools for the Anti-Submarine Warfare Continuous Trail Unmanned Vessel (ACTUV) program
• Specifically designed to validate, introspect, and assess autonomous systems (autonomy)
• Developed unique capabilities:
―Metrics for autonomous behaviors (tracking, collision regulations)
― Iterative test selection based on uncertainty
• Designed with modular infrastructure to facilitate future capabilities
―Systems Under Test (SUTs)
―Test-selection methods
―Metrics
Intro to AVIA
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 3
Autonomy Validation, Introspection, and Assessment (AVIA)
Rapidly conduct massive numbers of analytic assessments of ACTUV autonomy in complex, dynamic scenarios
Impact
• Assess capabilities beyond operational envelope: 10x – 100x more scenarios
• Address ACTUV operational concerns
• Provide >10x reduction in time to run & generate results from each simulation, from 42 days to < 1 day
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 4
• Sample entire test domain intelligently and without bias
―Currently using Latin hypercube sampling
―Allow perturbations and emergent conditions to arise
―Upgradeable: can implement future sampling techniques
• Parallel scenario execution
―Demonstrated 1000 1-hour scenarios in < 24 hours (16 hours)
―Automatic metrics evaluation
―Metrics stored in MySQL database, easy to visualize with analysis graphical user interface (GUI)
• Metrics are requirement dependent and include introspection to increase understanding of autonomy logic
How does AVIA accomplish this?
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 5
System Under Test (SUT)
Deliberative Processes
Reflective Processes
Reactive Processes
Assessment Tools
Configurable environment(Automated scenario generator)
0
1
tt 0 tU1
HS1(t)
HU1(t)
Unexpected eventgenerator
Critics (evaluation modules)
Library of evaluation modules
Influences on Actions
Heuristics
Influences on Perception
Exploration tools
Assessment tools
Presentation tools (including uncertainty)
Autonomy Processes
AVIA Handles all Autonomy Processes
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 6
AVIA Architecture
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 7
CreateScenario(s)
RunScenario(s)
AnalyzeResults
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 8
CreateScenario(s)
RunScenario(s)
AnalyzeResults
User creates base scenario
Format parameters in XML file
Generate specific scenarios
Random sample base scenario parameters
Matrix Instance XML
Base Scenario YAML
Launch Run Manager
Format all scenario parameters in one XML file
Spawn a run for each scenario
Scenario Parameters
Create Scenario(s)
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 9
CreateScenario(s)
RunScenario(s)
AnalyzeResults
z
Wrapper NAutonomy ModuleSimulation (NPC, motion)Contact ManagerTime Keeper
Run analysis
Wrapper NAutonomy ModuleSimulation (NPC, motion)Contact ManagerTime Keeper
Run analysis
Wrapper 2Autonomy ModuleSimulation (NPC, motion)Contact ManagerTime Keeper
Run analysis
Wrapper 2Autonomy ModuleSimulation (NPC, motion)Contact ManagerTime Keeper
Run analysis
Wrapper 1Autonomy ModuleSimulation (NPC, motion)Contact ManagerTime Keeper
Wrapper (Perl script) calls all the components for each
instance of simulation
Run analysis
Run Scenario(s)
Performance metrics
Run Log File
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 10
CreateScenario(s)
RunScenario(s)
AnalyzeResults
Performance metrics
Aggregated simulation data
Store data in database
Automated analysis
MySQL database
Automated analysis of performance metrics
Produce test report
Analysis results
Test Report Template
Format analysis results for report template
Analyze Results
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 11
How Data Flows Thru AVIA
Physics Models of Entities & Environment
• Vessel motion dynamics• Vessel signature characteristics• Bathymetry data (Level 0 & 1)
Sensor Models• Representations of contacts
from each sensor on SUT
System Under Test• Constructs worldview• Chooses course of action
Metrics Processor• Compares SUT’s worldview
with “real” world based on sensor pedigree
Entity Poses &
SignaturesEntity Poses
“Real” Worldview
Sensor Pedigree
Perceived Contacts
SUT’s Worldview
Contact List (JDDS format)
SUT
Stee
rin
g &
Sp
eed
Co
mm
an
ds
AVIA
SUT Autonomy
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 12
AVIA is Designed for the Tester
AVIA SIMAVIA GUI
Requirements
Metrics
Parameters
Test Selection
Assessment
Metrics Processor
Scenario Controller
Completion Criteria
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 13
AVIA GUI
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 14
AVIA GUI Evaluator Tab
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 15
AVIA GUI Playback Tab
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 16
Scenario Controller
Other Vessel Behavior Module – uses Finite State Automata (FSA) for non-player characters (NPCs)
FSA for Transit to Waypoint
FSA for Posture behavior
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 17
Scenario Controller
Physics Module – simplified models for speed and heading dynamics
u vt v
Kayref
y
y
+ -
e a aL
Speed model
Heading model
-2000
-1000
0
1000
2000
-2000 -1000 0 1000 2000
Y (
m)
X (m)
Ship track
Route
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 18
Scenario Controller
Sensor Models – random walk noise added to true data
Random walk error added to position data
Example
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 19
Metrics Processor & Automated Analysis
Test Matrix Tool analysis GUI: Scatterplot visualization
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 20
SUT Autonomy
Autonomy communication bridge
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 21
• Added iterative test selection based on performance of previous samples
―Searches the test domain for the highest estimated variances, the highest uncertainty
―Continues to iterate until completion criteria are met
―Estimates calculated using spatial interpolation
―Does not bias to best/worst performance
Iterative Test Selection
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 22
• COLREG classification can fluctuate over time
―Would have been head-on (both give-way), but SUT all-stop so now the non-player character (NPC) is give-way
Lessons Learned
SUT
NPC
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 23
• This SUT will try to pass on the right, even if it is the Stand-On vessel
Lessons Learned
SUT
NPCs
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 24
• Once again, SUT will try to pass on the right, even if it is the Stand-On vessel
• NPC using similar (flawed) COLREG algorithm so also passes on right
Lessons Learned
SUT
NPC
DISTRIBUTION STATEMENT A – Approved for public release: distribution unlimited 25
• Achievements for the AVIA program include:
―Bounded performance for desired scenarios
―Performed 1000 1-hour scenarios within 16 hours
― Identified errant action and attributed to either errant perception or errant behavior (or errant simulation)
―Confirmed stochastic nature of autonomy performance
• Applicable to other programs
―Scalable to massive parallelization on high performance computers (HPCs)
―Architecture supports SUTs, simulations, metrics plugins
―AVIA scenario controller -> sensor, physics, scenario plugins
ANY DOMAIN - HETEROGENEOUS SUTs - REGRESSION TESTING
AVIA Conclusion
