+ All Categories
Home > Documents > 1 A Spectrum of IV&V Modeling Techniques Mats Heimdahl (Co-PI) Jimin Gao (RA) University of...

1 A Spectrum of IV&V Modeling Techniques Mats Heimdahl (Co-PI) Jimin Gao (RA) University of...

Date post: 02-Jan-2016
Category:
Upload: mae-hicks
View: 218 times
Download: 4 times
Share this document with a friend
Popular Tags:
15
1 A Spectrum of IV&V Modeling Techniques Mats Heimdahl (Co-PI) Jimin Gao (RA) University of Minnesota Tim Menzies (Co-PI) David Owen (RA) West Virginia University/NASA IV&V Sanjai Rayadurgam (RA) University of Minnesota—Today’s Speaker
Transcript

1

A Spectrum of IV&V Modeling Techniques

Mats Heimdahl (Co-PI)

Jimin Gao (RA)University of Minnesota

Tim Menzies (Co-PI)

David Owen (RA)West Virginia University/NASA IV&V

Sanjai Rayadurgam (RA)University of Minnesota—Today’s Speaker

http://ww

w.cs.um

n.edu/crisys

2

Model-Based Development

Specification Model

Code

Visualization Prototyping

TestingAnalysis

http://ww

w.cs.um

n.edu/crisys

3

ROI with Model Based Development

Source: Esterel Technologies

Avionics SoftwareHand Coding

Low Level Requirements

20%

Design10%

Coding20%

Testing50%

Avionics SoftwareCoding with Model Based

Techniques

Savings50.0%

Testing25%

Coding5%

Design7.5%

Low Level Requirements

12.5%

- 37.5 %

- 25 %

- 75 %

- 50 %

http://ww

w.cs.um

n.edu/crisys

4

Model-Based DevelopmentComing to projects everywhere—soon

• Model based development in some form will in the near future be the norm in critical systems development Airbus Industries require the use of model based

techniques from all vendors Boeing currently evaluating what to require—

not if they will require something Honeywell and Rockwell Collins are fielding

the capabilities within the next two years Etc., etc.

http://ww

w.cs.um

n.edu/crisys

5

Model-Based Development Tools

• Commercial Products Esterel Studio and

SCADE Studio from Esterel Technologies

Rhapsody from I-Logix Rose Real-Time from

Rational Simulink and Stateflow

from Mathworks Inc.

http://ww

w.cs.um

n.edu/crisys

6

Model-Based Tools-2

• NASA tools STANLEY/

LIVINGSTONE for Integrated (or Intelligent) Vehicle Health Maintenance (IVMS) for second-generation shuttle.

http://ww

w.cs.um

n.edu/crisys

7

RSML-e and Nimbus

RSML-e Formal Models (~20 running concurrently)

Java Simulations of environment

• Integration in MatLab• Test case generation• Model checking• Theorem proving

Project with Rockwell Collins Inc.

http://ww

w.cs.um

n.edu/crisys

8

Formal Model of SUS

Formal Model of SUS

Typical Requirements IV&V Process

Initial Assessment Using Low-Cost Approach

Formal Inspection

Formal Analysis

System Under Study(typically English)

Formal Model of SUS

Inspection

Inspection

InspectionAutomation

ModelExtraction

http://ww

w.cs.um

n.edu/crisys

9

Model-Based IV&V Process

Formal Analysis

Test Inspection

Model v.1 Model

Evolution

Model v.2

Model v.n-1

Model v.n

Formal Analysis

Test Inspection

ModelEvolution

ModelEvolution

Code

Test

Increased Effort and Cost

http://ww

w.cs.um

n.edu/crisys

10

Challenges in the New Process

• Scalability and cost of the formal analysis State space explosion problems in model checking

• Cost effective model evolution Process and guidelines for evolving the model Early and cost effective problem detection

Formal Analysis

Test Inspection

Model v.1 Model

Evolution

Model v.2

Model v.n-1

Model v.n

Formal Analysis

Test Inspection

ModelEvolution

ModelEvolution

Code

Test

http://ww

w.cs.um

n.edu/crisys

11

Scalable Analysis

• Model checking is plagued by state space explosion problems Are there alternative, possibly heuristic, approaches that are effective?

• Alternate representations NAYO: a no-and-yes-or graph

• Hypothesis: NAYOs can be used to evaluate models

Express interesting properties Find interesting problems

NAYO-based evaluation scale NAYO can be used across the model evolution cycle

Work withDavid Owen,

WVU

http://ww

w.cs.um

n.edu/crisys

12

Q: NAYOs can be used to evaluate models?

A: Yes!! (using a novel stochastic

search engine- ISSRE02)

Q: Does NAYO-based evaluation scale?A1: Stochastic search linear time!

A2: NAYO stochastic search always plateau!

A3: False negative rate falls to zero in the plateau

Work withDavid Owen,

WVUSome Results

http://ww

w.cs.um

n.edu/crisys

13

Open Issues

• If the stochastic search does not find problems, are there none?

Compare the stochastic results with full verification on realistic models

Experiments using: RSML-e

Nimbus SMV Stochastic search Flight guidance models

from Rockwell Collins

• How to perform model evolution? Large case study with Rockwell

Collins RSML-e suitable for the full spectrum

of models

• Does finding problem in early models indicate a problem system?

Does elimination of problems early reduce problems in subsequent models (even if substantially different)?

Very difficult to assess No experiment this year Planning for controlled experiment

http://ww

w.cs.um

n.edu/crisys

14

Analysis Experiment

• Available Resources: 6 RSML-e models of Flight

Guidance System from Rockwell Collins Inc.

Collection of desirable properties Translator from RSML-e to

SMV FSM suitable for stochastic

search

• Experimental Method: Seed errors in the FGS models Apply stochastic search as well

as full formal verification Compare performance and

detection capability

RSML-e

Spec.

SMVSpec.

NAYO Graph

Automatic

Translation

Autom

atic

Translation

Work withJimin Gao,

U of Minnesota

http://ww

w.cs.um

n.edu/crisys

15

Summary

• Model based development is here

Or, will be here shortly

• Great potential to improve quality and decrease cost of IV&V

• Must meet some crucial challenges first

Scalability of formal analysis

In particular, state space exploration—model checking

Evolution of models

• Stochastic state space exploration may hold the key

• But, we need to explore The fault detection

capability of stochastic search

The efficiency of stochastic search

• Rigorous experiments are starting as I speak

• We will also evaluate alternative analysis tools

SAL from SRI


Recommended