Date post: | 16-Dec-2015 |
Category: |
Documents |
Upload: | earl-simpson |
View: | 216 times |
Download: | 0 times |
SAS_08_Model_Val_Tech_Heimdahl MAC-T IVV-08-152
Model-Validation in Model-Based Development
Kurt Woodham
L-3 Communications
Ajitha Rajan, Mats Heimdahl
University of Minnesota
OSMA SAS ’08OSMA SAS ’08 September 8-12September 8-12
2 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Problem: Model Validation
• Model-Based Development (MBD) is here to stay Use of MBD is accelerating
Estimate 50% of NASA development projects using some form of MBD
Many advantages: model-checking, code generation, desktop testing, closed-loop simulation
Enhances early detection of requirement, design, or implementation defects
“Executable Specifications” enable evaluation of behavior that might otherwise be relegated to Inspections and Testing
• How do we know the models are “right”? Manually develop black-box tests
• When have we validated enough? Measure test coverage on an implementation/model
3 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Problem : Current Practice
• Measure black-box test coverage over the model Indirect measure
Defects of omission in model not exposed.
Executable artifact is necessary Adequacy can only be determined late in the
development process
Incomplete ModelWeak
Black-Box Test set
4 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Goals of Project
• Define metrics for objective, implementation-independent measure of adequacy of a black-box test suite
• Develop tools to measure validation adequacy based on the defined metrics
• Provide capability for autogeneration of black-box test suites
5 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Testing – What does it mean?
Specification
Implementation
Does it implement?
Assertions
Model
Does it implement?
Assertion Based Testing (ABT) to Validate Model
Model
Source Code
Does it implement?
Model-Based Testing (MBT) to Verify Code
Our contribution is in providing novel ABT capabilities
In General
6 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
What are Assertions?
AssertionsProperties/ Formal
Assertions
Defined over
System
in1
ink
out1
outm
Can also be over components, interfaces,...
7 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Contributions - ABT
Assertions Black-Box Tests
Measure Adequacy
Auto-generate
2
Model
Validate
Assess Model and Assertion Completeness
31
We provide the following contributions in the Assertion-Based testing domain (indicated by in the above figure):
1. Objective, implementation-independent measure of adequacy of a black-box test suite
2. Auto-generation of black-box validation tests directly from assertions
3. Objective assessment of completeness of model as well as assertions
8 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
The Idea
…then define structural coverage metrics to directly and objectively describe coverage of
assertions
Write assertions in a formal notation…G (FD_On -> Cues_On);
G((¬ Onside_FD_On Λ ¬ Is_AP_Engaged) → X(Is_AP_Engaged → Onside_FD_On))
Temporal Logic
1
Property_Satisfied
AND
OR
NOTAND
4
Right_FGS_Active
3
Left_FGS_Active
2
Right_Independent_Mode
1
Left_Independent_Mode
Synchronous Observers
microwave_library_temp/mode_logic
Printed 14-Jul- 2006 12:51:47
ON
OFFentry:mode=1;
Okentry: mode=3;
FAILEDentry: mode=2;
[steps_remaining>0]/steps_remaining--;
3
{steps_remaining=steps_to_cook;}
[start && steps_to_cook>0]
[steps_remaining<=0]
2 [door_closed]
1[start && ...door_closed]
1
[clear_off || ...!door_closed]
1
2
[clear_off]/steps_remaining=0;
2
9 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
A
LTL Temporal Operators
Operator Notation Meaning
Globally A G(A) Formula A is true in all states
Future A F(A) Formula A is true in some future state
A until B A U B Formula A is true in every state until B becomes true. B must eventually become true for the property to be true.
Next A X(A) Formula A is true in the next state
S0 S1 S2 S3 Si
A A A
B
A
10 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Formalizing Assertions
“If the onside FD cues are off, the onside FD cues shall be displayed when the AP is engaged”
G((¬ Onside_FD_On ¬ Is_AP_Engaged) → X(Is_AP_Engaged → Onside_FD_On))
• Possible Coverage Metrics Assertion coverage: single test case that
demonstrates that assertion is satisfied Prone to “dumb” tests, e.g., execution in which AP is
never engaged. More rigorous metric is necessary
11 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Task - 1
• Define a collection of assertion coverage criteria
• Formalize the assertion coverage obligations
12 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Antecedent Coverage
• Many of the assertions in the FGS are of the form :
Globally if ‘A’ occurs then ‘B’ will occur
G (A → B) Two ways of satisfying (A → B)
– A is false– A is true and B is true
• Antecedent Coverage – test cases will exercise the antecedent.
S0 S1 Sn
Not A Not A A, B
What if: ACD → B
13 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Modified Condition/Decision Coverage (MC/DC)
• To satisfy MC/DC Every point of entry and exit in the model should be invoked
at least once, Every basic condition in a decision in the model should take
on all possible outcomes at least once, and Each basic condition should be shown to independently
affect the decision’s outcome
A B A or B
F F F
T F T
F T T
Independent effect of AIndependent
effect of B
Basic Conditions
14 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Unique First Cause (UFC) Coverage
“System shall eventually generate an Ack (A) or a Time Out (B)”
Req. LTL property - F(A B) .
Formal UFC obligation for A : ¬(A B) U (A ¬B)
for B : ¬(A B) U (B ¬A)
S0 S1 S2 S3 Si
A, ¬ B¬A, ¬B ¬A, B¬A, ¬B ¬A, ¬B
Path satisfies UFC obligation for A but not B.
S0 S1 Si
To show independence of B, ¬A, B¬A, ¬B ¬A, ¬B
15 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
UFC Coverage
• G(A)+ = {A U (a G(A)) | a є A+} G(A)- = {A U a | a є A-}
• F(A)+ = {¬A U a | a є A+} F(A)- = {¬A U (a G(¬A))| a є A-}
• (A U B)+ = {(A ¬B) U ((a ¬B) (A U B)) | a є A+} {(A ¬B) U b | b є B+}
(A U B)- = {(A ¬B) U (a ¬B) | a є A-} {(A ¬B) U (b ¬(A U B)) | b є B-}
• X(A)+ = {X(a) | a є A+}X(A)- = { X(a) | a є A-}
Michael Whalen, Ajitha Rajan, Mats Heimdahl and Steven Miller. Coverage Metrics for Requirements-Based Testing. In Proceedings of ISSTA 2006.
16 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Task 2 – Validation Adequacy Measurement Tool
Formal Assertions (Eg. LTL
Properties)
Assertion Cov. Criteria
Assertion. Cov. Obligations/formulas
Validation Test Suite
Evaluate and Check formulas
Calculate ratio of formulas that
were true
Run Test Suite
Derive
Assertion Coverage Achieved by Test
Suite
We currently support the following coverage metrics:
• Assertion Coverage
• Assertion Antecedent Coverage
• Assertion UFC Coverage
17 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Task - 3
• Automatically generate requirements-based tests from … Formal assertions Abstract model called Assertion Model created
using assertions and environmental constraints (specified as invariants)
… to provide the defined assertion coverage.
18 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Assertion Model
Formal Assertions (Eg. LTL properties)
Trap Properties (for Cov. Oblig.)
Model Checker
Assertion Cov. Criteria
(eg. UFC)
Counter-examples(Assertion-based test
cases)
Inputs, Ouputs, Environmental
constraints
Generate Generate
Automatically Generating Requirements-Based Tests
Common with the Adequacy Measurement
ToolAssertions and environmental
constraints specified as invariants
19 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
What Are Model Checkers?
• Breakthrough technology of the 1990’s• Widely used in hardware verification
(Intel, Motorola, IBM, …)
• Several different types of model checkers Explicit, Symbolic, Bounded, Infinite Bounded, …
• Exhaustive search of the global state space Consider all combinations of inputs and states Equivalent to exhaustive testing of the model Produces a counter example if a property is not true
• Easy to use “Push button” formal methods Very little human effort unless you’re at the tool’s limits
• Limitations State space explosion
20 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Preliminary Evaluation
Interested in determining:• Feasibility of generating assertion-based tests from a set of
assertions Generated assertion-based tests to provide UFC coverage over the
assertions
• Effectiveness of these test sets in validating the system model
Measured MC/DC achieved by the test sets over the system model
Used three realistic sized examples: • Flight Guidance System (FGS), • and two models related to the Display Window Manager
system (DWM1 and DWM2)
http://w
ww
.umsec.um
n.edu
21
Results
Ajitha Rajan, Michael Whalen, and Mats Heimdahl. Model Validation using Automatically Generated Requirements-Based Tests. In Proceedings of 10th IEEE High Assurance Systems Engineering Symposium, Nov 2007.
22 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Results and Analysis
• UFC test suites achieved high MC/DC coverage over DWM models – well defined set of assertions
• Test-suite generated for UFC achieved very low MC/DC over the FGS model“When the FGS is in independent mode, it shall be active”.
G(m_Independent_Mode_Condition.result → X(Is_This_Side_Active = 1))
RSML–e MacroStructure of Independent_Mode_Condition is not captured in the property
Independent_Mode_Condition = ((Is_LAPPR_Active & Is_VAPPR_Active & IS_Offside_LAPPR_Active & Is_Offside_VAPPR_Active) |
( Is_VGA_Active & Is_Offside_VGA_Active))
23 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Benefits of ABT
• Saves time and effort in generating validation test suites from assertions
• Effective method for generating model validation tests when the assertions are well defined
• Helps in identifying missing assertions and over constrained models
24 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Bonus Task – Adequacy of Conformance Testing
Model
Conformance Tests
Code
Run
Measure Adequacy
Assertions
Measure Adequacy
1. Direct assessment of how well tests exercise the assertions
2. Will expose defects of omission
3. Assertion coverage could necessitate longer test cases than for model coverage
Useful ??
25 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Assertion Coverage as an Adequacy Measure for Conformance Testing
Hypothesis 1(H1): Conformance tests providing assertion UFC coverage are more effective than conformance tests providing MC/DC over the model
Hypothesis 2(H2): Conformance tests providing assertion UFC coverage in addition to MC/DC over the model are more effective than conformance tests providing only MC/DC over the model
26 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Experiment
• Used four industrial systems : Two models from the display window manager Two models representing the mode logic of a
flight guidance system
• Assessed effectiveness of test suites in terms of their fault finding ability
Ajitha Rajan, Michael Whalen, Matt Staats, and Mats Heimdahl. Requirements Coverage as an Adequacy Measure for Conformance Testing. To Appear in Proceedings of 10th International Conference on Formal Engineering Methods, Oct 2008.
27 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Results – Hypothesis 1
MC/DC vs UFC
0
20
40
60
80
100
DWM1 DWM2 Latctl Vertmax
% F
au
lt F
ind
ing
Avg. MC/DC
Avg. UFC
Hypothesis 1 rejected at 5% statistical significance on all but the Latctl system
28 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Analysis – Hypothesis 1
• Model coverage better than assertion coverage for measuring adequacy of conformance test suites
• Assertion UFC coverage is heavily dependent on the nature and completeness of the assertions
• Rigor and robustness of assertion coverage metric used is important UFC metric gets cheated when assertions are structured
to hide the complexity of conditions
29 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Results – Hypothesis 2
MC/DC vs MC/DC + UFC
80
82
84
86
88
90
92
94
96
DWM1 DWM2 Latctl Vertmax
% F
au
lt F
ind
ing
Best MC/DC
Avg. Combined
Hypothesis 2 accepted at 5% statistical significance on all but the DWM2 system
30 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Analysis – Hypothesis 2
UFC Achieved by MC/DC suites
Achievable UFC Rel. Diff
DWM1 28.3% 96.9% 70.8%
DWM2 59.7% 64% 6.7%
Latctl 94.7% 99.5% 4.8%
Vertmax 97.4% 99% 1.6%
Does UFC really help in revealing additional faults?
31 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Summary – Bonus Task
• UFC > MC/DC FALSE 3 of the 4 case examples at 5% statistical significance
• UFC + MC/DC > MC/DC TRUE 3 of the 4 case examples at 5% statistical significance
• Combine rigorous metrics for assertion coverage and model coverage to measure adequacy of conformance test suites
• UFC metric is sensitive to structure of assertions Need assertion coverage metrics that are robust to
structure of assertions
32 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Technology Readiness Level
• “Requirements-Based Test Generation Tool” TRL = 6
System/subsystem model or prototype demonstration in a relevant environment
• “Validation Adequacy Measurement Tool” TRL = 6
System/subsystem model or prototype demonstration in a relevant environment
33 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Relevance to NASA
• MBD is here - estimate one-half of all NASA missions in development or on the books will use model-based subsystem development
Extensive use in avionics industry
• How do we know the models are right? Model validation problem
• We provide the capability to Objectively measure the “quality” of assertion-based black-box
validation tests Objectively assess the completeness of a model
Does the model address all assertions? Objectively assess the adequacy of a set of assertions
Are there enough assertions to adequately describe the model? Automatically generate truly assertion-based tests
34 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Achievements to Date
• Formal assertion notation identified Most work with LTL Extended to work with Live Sequence Charts (LSC)
• Objective validation metrics defined Requirements, Antecedent, Unique First Cause, and Unique Cause
• Test case generation tool developed Developed tool generating tests from LTL
Capable of generating tests to all metrics defined Prototype tool working on LSC developed
• Developed test-adequacy measurement tool for the defined validation metrics
• Evaluation of metrics and tool • 12 papers and one PhD dissertation (Ajitha Rajan)
35 MAC-T IVV-08-152SAS_08_Model_Val_Tech_Heimdahl
Next Steps
• Investigate alternative requirements notations to LTL
• Complete empirical evaluation of the effectiveness in model validation Flight Guidance Sensor (FGS) evaluation
complete Display Manager (DM) evaluation in work Coordinate evaluation on NASA IV&V project
• Coordinate technology transfer