A Hierarchical Approach to Model-based Reactive Planning in Large State Spaces Artificial...

A Hierarchical Approach to Model-based Reactive Planning

in Large State Spaces

Artificial Intelligence & Space Systems Laboratories

Massachusetts Institute of Technology

Brian C. WilliamsJoint with Seung H. Chung

Artificial Intelligence & Space Systems Laboratories Massachusetts Institute of Technology

Outline

• Model-based programming• A Simple model-based executive (Livingstone)• The need for model-based reactive planning• The Burton model-based reactive planner


Objective: Embedded languages that reason from hardware models.

(Reactive Model-based Programming)

Polar Lander Leading Diagnosis:

• Legs deployed during descent.

• Noise spike on leg sensors latched by software monitors.

• Laser altimeter registers 50ft.

• Begins polling leg monitors to determine touch down.

• Latched noise spike read as touchdown.

• Engine shutdown at ~50ft. Mars Mission Failures, 2000:•Climate Orbiter•Polar Lander


Model-based Programs Interact Directly with State

Embedded programs interact withplant sensors and actuators:

• Read sensors

• Set actuators

Model-based programs interact with plant state:

• Read state

• Write state

Embedded Program

SPlant

Obs Cntrl

Model-basedEmbedded Program

SPlant

Problem: Programmer must must map between state and sensors/actuators.

Solution: Model-based executive maps between state and sensors/actuators.

S’Model-based Executive

Obs Cntrl


Orbital Insertion Example

EngineA EngineB

Science Camera

Turn camera off and engine on

EngineA EngineB

Science Camera

Programmer specifiesabstract state evolutions

Model

Temporal plannerTemporal planner

Model-based ExecutiveModel-based Executive

Command

goals

Observations Flight System Control

RT Control Layer

State

Thrust Goals

Attitude Point(a)

Engine OffOff

Delta_V(direction=b, magnitude=200)

Power

Model-based ProgramModel-based ProgramEvolves Hidden StateEvolves Hidden State

ClosedClosed

ValveValve

OpenOpen StuckStuckopenopen

StuckStuckclosedclosed

OpenOpen CloseClose

0. 010. 01

0. 010. 01

0.010.01

0.010.01

inflow = outflow = 0

OrbitInsert()::

(do-watching ((EngineA = Firing) OR (EngineB = Firing))

(parallel

(EngineA = Standby)

(EngineB = Standby)

(Camera = Off)

(do-watching (EngineA = Failed)

(when-donext ( (EngineA = Standby) AND (Camera = Off) )

(EngineA = Firing)))

(when-donext ( (EngineA = Failed) AND (EngineB = Standby) AND (Camera = Off) )

(EngineB = Firing))))

Programmer specifies plant model

Model specifies•Mode transitions•Mode behavior

Reactive Model-based Programming Language: Asserts state Queries state Executes conditionally Preempts Iterates Executes concurrently

Model



Commands

State Goals


RT Control Layer

Thrust Goals

Attitude Point(a)

Engine OffOff


Power

Model-based Executive Model-based Executive Reasons from Plant ModelReasons from Plant Model

State Estimates

State Estimates

Reconfigure & Repair

Estimate & Diagnose

State Goals

s

Observations Commands

Goal: Achieve Thrust

Open fourvalves

Engine Off

Model



Command

goals


RT Control Layer

State

Thrust Goals

Attitude Point(a)

Engine OffOff


Power

Model-based Executive Model-based Executive Reasons from Plant ModelReasons from Plant Model

State Estimates

Reconfigure & Repair

Estimate & Diagnose

State Goals

s

Goal: Achieve Thrust

Diagnose:Valve fails

stuck closed Switch to

backup


Outline

• Model-based programming• A Simple model-based executive (Livingstone)• The need for model-based reactive planning• The Burton model-based reactive planner


A simple model-based executive (Livingstone) commanded NASA’s Deep Space One probe

courtesy NASA JPL

Started: January 1996Launch: October 15th, 1998Remote Agent Experiment: May, 1999

Livingstone [Williams & Nayak, AAAI96]

State estimate

ModeReconfiguration

ModeEstimation

CommandObservations

Model

Flight System Control

RT Control Layer

State goals

s

Thrust

State estimate

ModeSelection

ModeEstimation

CommandObservations

Model


RT Control Layer

State goals

s

Estimate current likely Modes Reconfigure modes to meet goals

State estimate

ModeSelection

ModeEstimation

CommandObservations

Model


RT Control Layer

State goals

s

Mode Selection:

Select a least cost set of allowed component modes that entail the current goal, and are consistent

Mode Estimation:

Select a most likely set of component mode transitions that are consistent with the model and observations

arg max Pt(m’)

s.t. M(m’) ^ O(m’) is consistent

arg min Ct(m’)

s.t. M(m’) entails G(m’)

s.t. M(m’) is consistent

ModeSelection

ModeEstimation

CommandObservations

Model


RT Control Layer

s

OpSat:

arg min f(x)

s.t. C(x) is satisfiable

D(x) is unsatisfiable

State estimate State goals

arg max Pt(m’)

s.t. M(m’) ^ O(m’) is satisfiable

arg min Ct(m’)

s.t. M(m’) entails G(m’)

s.t. M(m’) is satisfiable


Outline

• Model-based programming• A simple model-based executive (Livingstone)• The need for model-based reactive planning• The Burton model-based reactive planner


DS 1 Attitude Control System

z facing thrusters x facing thrusters

1553 bus

Com

mands

Data N2H4

He

PDE

SRU

PDU

GDE

PASM

DSEU

PEPE

BC

FlightComputer

FlightComputer

BC

PDE

Livingstone reconfigured modes using one step commands. But How does the flight computer really open a valve?

• Requires turning on device drivers• Requires repairing bus controllers• Sending commands• Powering down devices . . .


RemoteTerminal

RemoteTerminal

Driver

BusControl

ComputerValve

Driver

Valve• Device modes are changed through indirect commanding.

• Communication paths are established by reconfiguring other devices.

• The task of reconfiguring devices in the proper order generalizes state-space planning to handle indirect effects.

• to achieve reactivity the all possible plans for all possible goal states should be pre-compiled (a generalization of universal plans).

• To achieve compactness we decompose these universal plans according to a goal/sub-goal hierarchy.

How do we reconfigure a valve?

Model-based Execution & Reactive Planning

Burton [Williams & Nayak, IJCAI97]

State estimate

ModeSelection

ModeEstimation

CommandObservations

s goalsReactivePlanner

Model


RT Control Layer

State goals

s


Example: Driver Valve Command Sequence

Valve Driver dr Valve vlv

vcmdindcmdin

Commands Driver State Valve StateME: dr = off, vlv = openMS: dr = off, vlv = closed

MRP dcmdin = onME: dr = on, vlv = openMRP dcmdin = closeME: dr = reset failure, vlv = openMRP dcmdin = resetME: dr = on, vlv = openMRP dcmdin = offME: dr = off, vlv = open

Goal: No thrust


To achieve reactivity we eliminate all forms of search.


Model-based Reactive Planning

Achieved by:

1. Eliminate Indirect Control

. . . through Compilation

2. Eliminate Search for Goal Ordering

. . . through Reversibility and Serialization

3. Eliminate Search to find Suitable Transitions

. . . by Constructing Hierarchical Polices



Achieved by:

1. Eliminate Indirect Control







To Handle Indirect Control . . .dcmdout= vcmdin

off

on

failed

resettable

dcmdin = offdcmdin = on

dcmdin = reset

dcmdin = off

dcmdin = dcmdout

closed

open

stuck closed

stuck open

vcmdin = closevcmdin = open

inflow = outflow

vcmdindcmdin

flowin

flowout


. . . Compile Out Constraints

closed

open

stuck closed

stuck open

vcmdin = closevcmdin = open

off

on

failed

resettable


dcmdin = reset

dcmdin = off

dcmdin = dcmdout inflow = outflowinflow = outflow

dcmdout = vcmdin

driver = ondriver = on

dcmdin = closedcmdin = open

vcmdindcmdin

flowin

flowout


. . . Compile Out Constraints

closed

open

stuck closed

stuck open

off

on

failed

resettable


dcmdin = reset

dcmdin = off



dcmdin


To Compile Out Constraints• Eliminate intermediate variables.

Transitions are conditioned on mode and control variables

• Generate transitions as prime implicates:

i next(yi = ei)

where i is a conjunction of mode and control variable assignments.

• Prime implicates for transitions enumerated using OpSAT

– 40 seconds on SPARC 20 for 12,000 clause spacecraft model.



Achieved by:

1. Eliminate Indirect Effects







ValveDriver

command

• Example– Current State: driver = on, valve = closed– Goal State: driver = off, valve = open– Achieving (driver = off) and then (valve = open) clobbers (driver = off)

Why Search is Needed

1) An achieved goal can be clobbered by a subsequent goal.

Achieve Valve goal before Driver goal


Note: Component schematics tend not to have loops

RemoteTerminal

RemoteTerminal

BusControl

ComputerValve

Valve

Driver

Driver

Work conjunctive goals upstream from outputs to inputs

– Define: Causal Graph G of compiled transition system S • vertices are state variables.

• edge from vi to vj if vj’s transition is conditioned on vi.

dcmdin

Driver

Valve

– Requirement: The causal graph is acyclic.


• The only variables used to set some variable (y7) is its ancestors,

y7 can be changed without affecting its descendants.

Solution

13

12

9

11

8

10

7

4

6

3

5

2

1

UnaffectedAffected

• Safe to achieve goals in an upstream order.

• Simple check: – Number causal graph depth first – achieve goals in order of increasing depth first number.


Latch1

Switch

data

• Example– Latch1 and Latch2 compete for the position of Switch if achieved

concurrently.


2) Two goals can compete for the same variable in their subgoals.

Latch2

1

2


• Sibling goals (7,4) may both need shared ancestors.

13

12

9

11

8

10

7

4

6

3

5

2

1

UnaffectedNot Shared

Shared

13

12

9

11

8

10

7

4

6

3

5

2

1

Unaffected

Not Shared

• But ancestors no longer needed once goal (7) is satisfied.

• Solution: Solve one goal before starting next sibling (Serialization).• Feature: Generates first control action of plan first!


Latch1

Switch

data

• Example– Assume Switch can be used once, – Then Latch1 must be latched before Latch2.


3) A state transition of a subgoal variable has irreversible effect.

Latch2

• But irreversible effects aren’t desirable for reactive planners

Don’t allow irreversible actions. . . Except to repair failure modes

1

2


Solution: Mark Allowed Transitions/Assignments

off

on

failed

resettable


dcmdin = reset

dcmdin = off

closed

open

stuck closed

stuck open



dcmdin

DriverValve

• Mark all control variable assignments allowed:

123



off

on

failed

resettable


dcmdin = reset

dcmdin = off

closed

open

stuck closed

stuck open



dcmdin

DriverValve


123

• For each mode variable v, in decreasing order of DF number:

• Select each transition of v, whose guard has only allowed assignments.



off

on

failed

resettable


dcmdin = reset

dcmdin = off

closed

open

stuck closed

stuck open



dcmdin

DriverValve


123



• Given current assignment v = I for v:

• Mark assignments and transitions in SCC allowed.

• Find strongly connected component of selected transitions that contains I.



off

on


closed

open

stuck closed

stuck open



dcmdin

DriverValve


123








off

on


closed

open

stuck closed

stuck open



dcmdin

DriverValve


123








off

on


closed

open



dcmdin

DriverValve


123








Achieved by:

1. Eliminate Indirect Effects







Solution

• Convert automata into hierarchical policies, one per automaton

closed

open

cmd = closecmd = open

fail

Goal

fail

driver = oncmd = open

idle

idledriver = on

cmd = close

Current

Open

Closed

Stuck

Open Closed


– Policy selects first transition towards achieving each automata goal state, given current state.

– Policy maps goals to subgoals and commands, in proper order– Ensures only reversible transitions are taken,

by only using transitions marked allowed.


Plan by passing sub-goals up causal graph

ValveDriver

fail

Goal

fail


idle

idledriver = on

cmd = close

Current

Open

Closed

Stuck

Open Closed

Goal

cmd = on idle

idle cmd = off

Current

On

Off

Resettable

On Off

Goal: Driver = off, Valve = closed

cmd = reset cmd = off

Current: Driver = off, Valve = open

12



ValveDriver

fail

Goal

fail


idle

idledriver = oncmd = close

Current

Open

Closed

Stuck

Open Closed

Goal

cmd = on idle

idle cmd = off

Current

On

Off

Resettable

On Off




12



ValveDriver

Send:cmd = on

fail

Goal

fail


idle

idledriver = oncmd = close

Current

Open

Closed

Stuck

Open Closed

Goal

cmd = on idle

idle cmd = off

Current

On

Off

Resettable

On Off




12


12

Current: Driver = resettable, Valve = open


ValveDriver

fail

Goal

fail


idle

idle

Current

Open

Closed

Stuck

Open Closed

Goal

idle

idle cmd = off

Current

On

Off

Resettable

On Off



driver = oncmd = close

FailedResettable

cmd = on



ValveDriver

fail

Goal

fail


idle

idle

Current

Open

Closed

Stuck

Open Closed

Goal

cmd = on idle

idle cmd = off

Current

On

Off

Resettable

On Off





12



ValveDriver

fail

Goal

fail


idle

idle

Current

Open

Closed

Stuck

Open Closed

Goal

cmd = on idle

idle cmd = off

Current

On

Off

Resettable

On Off


Sendcmd = reset




12



ValveDriver

fail

Goal

fail


idle

idledriver = on

cmd = close

Current

Open

Closed

Stuck

Open Closed

Goal

cmd = on idle

idle cmd = off

Current

On

Off

Resettable

On Off


Sendcmd = close


Current: Driver = on, Valve = open

12



ValveDriver

fail

Goal

fail


idle

idledriver = on

cmd = close

Current

Open

Closed

Stuck

Open Closed

cmd = reset

Goal

cmd = off

cmd = on idle

idle cmd = off

Current

On

Off

Resettable

On Off


Sendcmd = off

Current: Driver = on, Valve = closed

12



ValveDriver

fail

Goal

fail


idle

idledriver = on

cmd = close

Current

Open

Closed

Stuck

Open Closed

cmd = reset

Goal

cmd = off

cmd = on idle

idle

Current

On

Off

Resettable

On Off

cmd = off

Goal: Driver = off, Valve = closedSuccess

Current: Driver = off, Valve = closed

12


Hierarchical, Model-based Reactive Planning• Compile-time Analysis:

– Compile-out interactions– Confirm schematics are loop free.– Depth first number variables.

• Periodic, Run-time Analysis:– Given initial state

• Identify allowed transitions and assignments

– Given autonomous jump to failure state• Identify allowed transitions and assignments

• Run-time Plan Execution:– Work conjunctive goals from outputs to inputs.– Achieve goals serially.– Only perform reversible transitions.– Lookup control actions and sub-goals in policies


Complexity of Reactive Planning

• Worst Case per action: Depth * Sub-goal branch factor• Average Cost per action: Sub-goal branch factor

Valve1 = open Valve2 = open Driver1 = off Driver2 = off

Driver1 = on

CU = on

CU = on

Driver2 = on

CU = on

CU = on

CU = on CU = on


What If Plan is Not Serializable?

– compose each cycle into a single component.

BusControl

Computer

Antenna

Antenna

AmplifierK-bandTransmitter

AmplifierK-bandTransmitter

• What if causal graph G contains cycles?• Solution:

– Isolate the cyclic components (compute SCCs)

• New causal graph G’ is acyclic, • Goals of G’ are serializable


Composing Cyclic Components

off

on

cmdT = off

Transmitter Amplifier

cmdT = onA = offA = off

off

on

cmdA = offcmdA = on

T = on

onT onA

onT offA

offT offA

offT onA

cmdT = offcmdT = on

cmdA = off

cmdA = on

cmdA = off


Policy for Composed Components

onT onA

onT offA

offT offA

offT onA

cmdT = offcmdT = on

cmdA = off

cmdA = on

cmdA = off

cmdT = on

Goal

cmdT = on

cmdA = on idle

idle cmdA = off

Current

OnT, OnA

OnT, OffA

OffT, OffA

OnT, OnA OnT, OffA

idle

cmdT = off

cmdA = off

OffT, OffA

fail

fail

fail

OffT, OnA

fail fail cmdA = off idleOffT, OnA

• Problem: Composition grows exponential in space usage.

• Solution: Use BDD encoding

(in progress).



1. Compile away constraints from the model

2. Compile away cyclic components

3. Plan serially pursuing causal graph upstream

4. Generate actions using hierarchical policies

Only performs reversible actions

Responds to failure at each step

Average cost per step = subgoal branching factor


Current Demonstration Testbeds• Air Force Tech Sat 21 flight• NASA NMP ST-7 Phase A• NASA Mercury Messenger

on ground.• MIT Spheres on Space Station• NASA Robonaut, X-37, ISPP

• Multi-Rover Testbed• Simulated Air Vehicles

Model-based Programming of Embedded Systems

• To survive decades embedded systems orchestrate

complex regulatory and immune systems.

• Future systems will be programmed with models,

describing themselves and their environments.

• Runtime kernels will be agile, deducing and planning by

solving optimization problems with propositional

constraints. • Model-based reactive planners respond quickly to failure,

while using compile-time analysis of structure to respond quickly and concisely to indirect effects.

Date post:	28-Dec-2015
Category:	Documents
Upload:	brendan-anthony
View:	215 times
Download:	0 times

A Hierarchical Approach to Model-based Reactive Planning in Large State Spaces Artificial...

Documents