Post on 23-Feb-2016
description
transcript
Mining Requirementsfrom
Closed Loop Control Models
Jyotirmoy V. Deshmukh
Xiaoqing Jin Alexander DonzéSanjit A. Seshia
Joint work with:
Mining Temporal Requirements from Control Models
But, you are doing it all wrong!
Design Requirements
2/30
Aren’t you supposed to check if design satisfies requirements/specifications/properties?
Mining Temporal Requirements from Control Models
Challenges Closed-loop models very complex:
nonlinear dynamics look-up tables large amounts of switching components with no models unclear semantics
Requirements too vague, high-level: intake manifold pressure should settle increase fuel efficiency improve ride quality
3/30
Mining Temporal Requirements from Control Models
What this work is all about …
How we could use formal reasoning when all
we have is:
Ability to simulate and test system
Vague idea of what system should satisfy
(Possibly limited) ability to check if system
satisfies property
Requirement
Mining!
4/30
Mining Temporal Requirements from Control Models
‘As-is’ properties of closed-loop design
Mining in Action
5/30
6.25ms
100
Ask designer if mined requirements are OK “Settling time is 6.25 ms” “Overshoot is 100 units”
Mining Temporal Requirements from Control Models
Mine for one version, get many free
Requirement 1Requirement 2Requirement 3
Version 0
Version 1 Version 2
Mine Requirements
Use forV & V
Use forV & V
Use forV & V
6/30
Mining Temporal Requirements from Control Models
Legacy code
It’s working, but I don’t understand why!
Value added by mining: Mined Requirements become
useful documentation Useful for code maintenance
and revision Use requirements during
tuning and testing
7/30
Mining Temporal Requirements from Control Models
Outline
Expressing Requirements in Signal Temporal
Logic
Mining Algorithm
Experimental Results8/30
Mining Temporal Requirements from Control Models
Expressing Requirements in
Signal Temporal Logic
9/30
Mining Temporal Requirements from Control Models
Signal Temporal Logic (STL) Extension of Metric Temporal Logic (MTL) Allows tests over continuous-valued signal
variables Examples:
®®
0 10050
1
3x
0 100
1
-0.1 +0.1
60
x
t
t
10/30
Mining Temporal Requirements from Control Models
Quantitative Semantics of STL
Function that maps STL formula to a numeric
value
Quantifies “how much” a trace satisfies a property
Large positive value : trace easily satisfies
Small positive value: trace close to violating
Negative value: trace does not satisfy
11/30
Mining Temporal Requirements from Control Models
Mining Algorithm
12/30
Mining Temporal Requirements from Control Models
CounterExample Guided Inductive Synthesis
Find “Tightest” Answers
Settling Time is ??Overshoot is ??Upper Bound on x is ??
Are there behaviors that do NOT satisfy
theserequirements?YES
Settling Time is 5 msOvershoot is 5 KPaUpper Bound on x is 3.6
1.
1.
m.
13/30
Mining Temporal Requirements from Control Models
Settling Time is 5.3 msOvershoot is 5.1 KPaUpper Bound on x is 3.8
Settling Time is … msOvershoot is … KPaUpper Bound on x is …
CounterExample Guided Inductive Synthesis
Find “Tightest” Answers
Settling Time is ??Overshoot is ??Upper Bound on x is ??
Are there behaviors that do NOT satisfy
theserequirements?
Counterexamples
1.
m.
1.
n.
YES
14/30
Mining Temporal Requirements from Control Models
CounterExample Guided Inductive Synthesis
Find “Tightest” Answers
Settling Time is ??Overshoot is ??Upper Bound on x is ??
Are there behaviors that do NOT satisfy
theserequirements?
Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1
NO
Settling Time is 6.3 msOvershoot is 5.6 KPaUpper Bound on x is 4.1
Mined Requireme
nt
1.
n.Counterexamples
1.
m.
15/30
Mining Temporal Requirements from Control Models
Parametric STL Constants in STL formula replaced with
parameters Scale parameters Time parameters
Examples: Between some time and 10seconds, x remains greater than some value
After transmissionshifts to gear 2, itremains in gear 2 for at least secs
16/30
Mining Temporal Requirements from Control Models
(v(p)) is an STL formula
Validity domain: {v(p) | i: (xi, t) (v(p))} {xi} : set of traces
Semantics of PSTL formula (p)
p = ( )
Valuation function v assigns values to parameters in p
17/30
j=
Mining Temporal Requirements from Control Models
Parameter Synthesis x -satisfies property if for some i:
(x,t) (v(p)) v(p) = (v1,…vi,…) (x,t) (v(p)) v’(p) = (v1,…v’i,…) |vi v’i| <
Find -tight valuation v such that i: (xi,0) (v(p))
Multi-criteria, nonlinear optimization problem
Solution not unique, need to find Pareto-optimal solution
(I.e. Find the “tightest” value)
18/30
Mining Temporal Requirements from Control Models
Parameter Synthesis
Naïve approach: grid parameter space evaluate satisfaction value at each point pick valuation with smallest satisfaction value
Exponential number of points in parameter space
Could miss optimal values
19/30
Mining Temporal Requirements from Control Models
If upper bound of all signals is 3, any number > 3 is also an upper bound
Sat. value monotonically increasing in ith parameter: x (v(p)) and v(pi) ≤ v’(pi) and j≠i v(pj) =v’(pj) x (v’(p))
Monotonic if either decreasing or increasing
Binary-search in monotonic parameter dimensions Now implemented in tool BREACH
Satisfaction Monotonicity
20/30
0 10050
34
Mining Temporal Requirements from Control Models
Checking Monotonicity
Checking monotonicity is undecidable Encode monotonicity check as SMT
query F.O. Logic with quantifiers + uninterpreted
functions + real arithmetic Return “yes”/ “no” / “unknown” If “yes” – proof of monotonicity If “no” – fall back to naïve procedure
21/30
Mining Temporal Requirements from Control Models
Falsification: any violating behaviors?
u S(u)
Falsification Tool
\
(v(p))
\
22/30
Mining Temporal Requirements from Control Models
Falsification as Optimization Solve
If < 0, found falsifying trace! Use stochastic optimization such as in S-
TALIRO Need clever “parameterization” of input signal
space
Implemented parameterization in Breach-based falsifier
Run-time worsens with more signal parameters
½¤ = minu2U
½(' ;S(u);0)
½¤
Signal parameters: amplitude (A), delay (D)u
23/30
Nonlinear Optimization Problem,
No exact solution, Limited formal
guarantees
Mining Temporal Requirements from Control Models
Mining in a nutshell
BREACH
Template PSTL property
S-TALIRO/BREACHfalsified
Requirement?
Candidate Requirement NO
Mined STL Requirement
1.
n.Counterexamples
1.
m.
YES
24/30
Mining Temporal Requirements from Control Models
Experimental Results
25/30
Mining Temporal Requirements from Control Models
Experimental Results
S-TALIRO for Falsification*
BREACH for Falsification
Time taken
# Simulations
Time Taken
# Simulation
sUpper bounds on speed & rpm 55 s 255 197 s 496Cannot reach 100mph in seconds with rpm < 6422 s 9519 267 s 709Cannot reach 100mph in seconds with rpm < 8554 s 18284 147 s 411Minimum Dwell time in Gear 2 18886 s 130 1015 s 431* We ran S-TALIRO with default options and did not explore signal parameterization
26/30
Mining Temporal Requirements from Control Models
Experimental Results
Found max overshoot with 7000 simulations in 13 hours
Attempt to mine max settling time: Stops after 4 iterations with tsettle = total time for
simulation
27/30
Experimental Engine Control Model
Mining Temporal Requirements from Control Models
Mining can lead to deep bugs
Each iteration produced intermediate requirements Forced falsification to explore trajectories more likely
to altogether violate requirement Discussion with control designer revealed it to be a real bug Root cause identified as wrong value in a look-up table, bug
was fixed Why mining could be useful for bug-finding:
Mining provides better “direction” information to optimizer Looking for bugs Mine for negation of bug
28/30
Experimental Engine Control Model
Mining Temporal Requirements from Control Models
References BREACH & STL: http://www.eecs.berkeley.edu/~donze/breach_page.html
1. Alexander Donzé, Oded Maler. Robust satisfaction of temporal logic over real-valued signals. Formal Modeling and Analysis of Timed Systems, 2010.
2. Alexander Donzé. Breach: A Toolbox for Verification and Parameter Synthesis of Hybrid Systems. CAV, 2010.
3. Eugene Asarin, Alexander Donzé, Oded Maler and D. Nickovic. Parametric identification of temporal properties. Runtime Verification, 2011.
S-TALIRO: https://sites.google.com/a/asu.edu/s-taliro/s-taliro1. Sriram Sankaranarayanan and Georgios Fainekos. Falsification of temporal
properties of hybrid systems using the cross-entropy method. HSCC 2012.2. Y. Annpureddy. C. Liu, G. E. Fainekos, and S. Sankaranarayanan. S-
TaLiRo: A tool for Temporal Logic Falsification for Hybrid Systems: TACAS 2011.
29/30
Mining Temporal Requirements from Control Models
Thank You!
30/30
Mining Temporal Requirements from Control Models
Backup Slides
Mining Temporal Requirements from Control Models
Syntax & SemanticsSyntax
Semantics
Mining Temporal Requirements from Control Models
Quantitative Semantics of STL Following (satisfaction value) does the trick½
Mining Temporal Requirements from Control Models
Quantitative Semantics Demystified
0¹ = x¡ 1:5
1 0.5 -0.5 0.5 -10.1 0.2 0.3 0.4 0.5 0.6 0.7
12
x
t0 0.5¹
11
0.50.5
0.50.5 0.5
sup over each interval
Mining Temporal Requirements from Control Models
Quantitative Semantics Demystified
0¹ = x¡ 1:5
1 0.5 -0.5 0.5 -10.1 0.2 0.3 0.4 0.5 0.6 0.7
12
x
t0 0.5¹
1 1 0.5 0.5 0.5 0.50.5
0.5
= 0.5
inf over result from previous step