Run Time Monitoring of Reactive Systems

Mikhail AugustonNaval Postgraduate School

maugusto@nps.edu

Mark TrakhtenbrotHolon Academic Institute of Technology

ilmarktr@yahoo.com

Outline

Problems in development of reactive systems

Model-based development and analysis

Our approach: TL-based assertion language Automatic generation of monitors for run-time verification

Outline of translation scheme

Further steps

Why development of reactive systems is difficult?

Main problem: complex behavior Intricate event-driven interaction with the environment Concurrency Timing factors

Typically: Critical applications (embedded RT controllers, …) Must fulfill tough requirements (safety, timeliness, … )

Challenges

Capture requirements to express the expected behavior

Check that system meets the requirements ideal: full verification more realistic: testing

* how to ensure a proper coverage?

* how we know the test result is / isn’t correct?

can this task be automated?

So, what to do?Apply formal methods!

Precise spec of behavior: Variants of TL [Pnueli, …] (future/past modalities; timing) Patterns for typical properties [Avrunin, …]

(response, precedence, existence, absence,…)

Model-based development: formal model of system (behavior, structure, …) statecharts to capture behavior; a renown standard executable model; a basis for automated tools model-level analysis (closer to problem domain)

Expressing system properties

Properties expressed using a special language, e.g. Temporal Logic

Examples:

ALWAYS (cross_empty or barrier_closed)

ALWAYS (request EVENTUALLY allocate_resource)

Meaning: the property holds in any computation

More complicated: property holds on part of execution paths (need quantifiers on path)

Time-related properties:

ALWAYS (p EVENTUALLY (5) q)

Q should happen within 5 time units after P.

Expressing system properties

May be difficult to use without a strong logic background

Combination of English with C-like expressions:

Event A follows event B after less than 10 time units

Patterns for typical behavioral properties

E.g. Precedence patterns (S precedes P):

Before R EVENTULLY P (not P Until (S or R))

After Q

Between Q and R

After Q until R

Pattern defines scope in which the property should hold

Statechart for Early Warning System

OFF

ON

IDLE

SETTING_UP

WORKING

COMPARING

DISPLAYING _ALARM

OUT_OF_RANGE dly(15) /

st!(PRINT_ALARM)

RESET

/CONNECT_OFF

[in(CONNECTED) ]

[in(DISCONNECTED) ]

EXECUTE

DONE _SET_UP

DO _SET_UP

S

C

DISCONNECTED

CONNECTED

WAIT SAMPLING

dly(5)

SAMPLING_DONE CONNECT_OFF

CONNECT_ON

IDLE

SETTING_UP

WORKING

COMPARING

DISPLAYING _ALARM

DISCONNECTED

CONNECTED

WAIT SAMPLING

OUT_OF_RANGE dly(15) /

st!(PRINT_ALARM)

RESET

/CONNECT_OFF

[in(CONNECTED) ]

[in(DISCONNECTED) ]

EXECUTE

DONE _SET_UP

DO _SET_UP

dly(5)

SAMPLING_DONE CONNECT_OFF

CONNECT_ON

POWER_OFF

POWER_ON

S

C

Model-based analysis with Statemate

Simulation: check, record and replay test scenarios

Verification: based on predefined assertion templates

Code synthesis: executable C, for host or target OS

Watchdog charts:

- not part of the system model, but co-executed alongside it

- used to drive and monitor system execution (in simulation or generated code run)

Model-based analysis

Simulation:- Analysis of quality attributes (e.g. stability with respect to

changes of the data)

- Analysis of events occurring simultaneously (to resolve scheduling and priorities issues in the system)

- "What if" checks (alternative decisions, "impossible" scenarios)

- Check rarely occurring situations

- Check behavior of the system under extreme values.

- Influence of various uncontrolled and unpredictable factors

- Time-related analysis (performance , very fast / very slow processes)

Model-based analysis

Verification:

Generic properties (e.g. existence of racing, or non-determinism)

Model-specific properties, e.g. reachability of a state in which certain condition is fulfilled (formulated in terms of model elements)

If violation found: a script is generated, driving simulation to the problematic state

Monitor charts

Monitor (watchdog) chart:

- Direct access (visibility) to all model elements

- Support for box black-box and white-box testing

Sample assertion:

Processing of a request must be accomplished within 5 seconds, and before receiving of the next request

Violation: entering into ERROR state 

Wait Process ErrorRequest

ProcessingDone

Request orTimeout(5)

/ViolationNotification

Suggested approach:Statemate-based

run-time verificationGoal: support effective testing

(finding errors rather than validation / showing correctness)

Monitoring of code generated from system model

Main features: Assertion language to specify:

- allowed and forbidden behavior

- actions to be taken (collect and report diagnostic data) Automatic creation of monitor charts from assertions

Used Statemate capabilities: Co-translation of model and monitor charts to C Co-execution of generated code components

 

Advantages of the approach

Flexible assertion language vs. predefined patterns

Smooth transition: assertions executable monitors

Overcome limitations of simulation and verification

Overcome limitations of simulation and verification

Analyze models reflecting design decisions

(tasks/events mapping,…), in realistic target environment

Real time analysis - rather than SIM time

SIM synch time (clock-driven systems):

all reactions assume same duration

SIM asynch time (event-driven systems):

all reactions assume zero duration

No "finite state" limitation; may connect to real input sources

No limitations on model size

Assertion language

Boolean expressions: system state properties in(S) and (x > 5) ; entered(S) started(A) ; ch(d) Regular expressions: describe sequences of states

[SELECT (Open | Read | Write | Close)]

SATISFY Open (Read | Write )* Close Temporal formulas: order properties

ALWAYS (request EVENTUALLY response)

ALWAYS (execute SOMETIME_WAS do_set_up) Timed temporal formulas: real-time restrictions ALWAYS (request EVENTUALLY (10) response) Actions: to trace interesting events (e.g. property violation)

reports, profiling info (based on objects’ attributes)

Example - assertion for EWS

ALWAYS (OUT_OF_RANGE EVENTUALLY (15) (RESET or

started(PRINT_ALARM)))

ON_FAIL [printf(“Violation after occurrence of

OUT_OF_RANGE at time %f “,

OUT_OF_RANGE.occur_time)]

More on temporal properties

Manna & Pnueli Classification (P – past formula) : Safety: ALWAYS (P) , Guarantee: EVENTUALLY (P) Obligation: Boolean combination of safety & guarantee Response: ALWAYS (EVENTUALLY(P)) Persistence: EVENTUALLY (ALWAYS(P)) Reactivity: Bool. combination of response & persistence Any TL formula is equivalent to a Reactivity formula;

others used for convenience & flexibility

Limitations: nesting depth <= 2 for unrestricted future operators unrestricted operator not embedded in a restricted one

Sample monitor construction

ALWAYS (OUT_OF_RANGE EVENTUALLY (15) (RESET or started(PRINT_ALARM)))   

/ ERROR_MESSAGE

Translation of restricted formulaALWAYS (N) P

F D

P

F D

START_P

P

F D

START_P

IDLE

dly(1)

P

F D

START_P

IDLE

dly(2)

P

F D

START_P

IDLE

dly(N)

.....

P

F D

START_P

P

F D

START_P

IDLE

P

F D

START_P

IDLE

P

F D

START_P

IDLE

F D

IDLE dly(1) dly(2) dly(N)

/RES:=false

dly(N) / RES:=true not P

/RES:=false

.....

Maximum time needed to compute value of formula FRM:

t(FRM) = N

Computation may finish in less than N time units

Case 1 : -------

P a is basic formula

Maximum time needed to compute value of formula FRM: t(FRM) = N + t(P)

Case 2 : -------

P contains only restricted temporal operators

All components are in F / RES:=true

At least one component is in D / RES:=false

Translation of unrestricted formula ALWAYS (P)

D

COMP_N

P

F D

START_P

IDLE

RESTART_P_N / RES:=true

dly(N) /TN:=CURR_TIME

COMP_0

P

F D

START_P

RESTART_P_0 / RES:=true

/T0 =: CURR_TIME

COMP_2

P

F D

START_P

IDLE

RESTART_P_2 / RES:=true

dly(2) /T2:=CURR_TIME

COMP_1

P

F D

START_P

IDLE

RESTART_P_1 / RES:=true

dly(1) /T1:=CURR_TIME

...

...

COMP_N

P

F D

START_P

IDLE

COMP_0

P

F D

START_P

COMP_2

P

F D

START_P

IDLE

COMP_1

P

F D

START_P

IDLE

RESTART_P_N / RES:=true

dly(N) /TN:=CURR_TIME

RESTART_P_0 / RES:=true

/T0 =: CURR_TIME

RESTART_P_2 / RES:=true

dly(2) /T2:=CURR_TIME

RESTART_P_1 / RES:=true

dly(1) /T1:=CURR_TIME

/RES:=false

...

...

P is a resticted formula: t(P) = N ===================================

RESTART_P_i = dly(N - mod(CURR_TIME-Ti, N)) At least one component is in D / RES:=false

Translation of unrestricted formula EVENTUALLY (P)

Translation of unrestricted formula ALWAYS (EVENTUALLY (P) )

Semantic issues

Monitoring: deals with finite runs;

semantics of TL – based on infinite runs

[Eisner, et.al.]: reasoning on truncated executions

Assume the run finishes 4 sec after the last p

ALWAYS (p EVENTUALLY(10) q) ; false if no q after the last p

ALWAYS (p ALWAYS(10) q) ; true if always q after the last p

In general: user responsible to provide proper actions

Further stepsImplementation: done - experiments with manually created monitors next step – actual implementation

A more friendly assertion language e.g. combine English with formulas

Optimized translation for certain types of assertions

Apply to other design paradigms using statecharts e.g. in OO: monitor systems with dynamically created

objects (problematic for model checking)