Research Heaven,West Virginia

FY2003 Initiative:

Hany Ammar, Mark Shereshevsky,Walid AbdelMoez, Rajesh Gunnalan, and Ahmad Hassan

LANE Department of Computer Science and Electrical EngineeringWest Virginia University

The OSMA Software Assurance Symposium July 30, 2003Lakeview, WV

WVU UI: WVU UI: Quantitative Relations Between Static and Dynamic

Software Metrics

NASA OSMA SAS’03 2

Research Heaven,West Virginia

Two main views of software systems: The static view of the structure and composition,

The dynamic view during execution

• Static and dynamic properties of software have been widely studied in the literature.

NASA OSMA SAS’03 3

Research Heaven,West Virginia

So how good is static analysis?

• IV&V testing is mostly focused of static analysis– Software rarely executed at Fairmont.

• Why?– NASA software supports NASA hardware

– So, some software can’t run without its associated hardware

• Traditional view:– Dynamic execution more informative than

static analysis

• This research:– To what extent are static measure

surrogate for dynamic measures?

– Compare insights gained from static and dynamic measures

NASA OSMA SAS’03 4

Research Heaven,West Virginia

This talk

• Propose three hypotheses of possible correlations between selected static and dynamic metrics– Hypothesis I: Static coupling

metrics correlate with error propagation in software architectures.

– Hypothesis II: Static error propagation correlates with dynamic error propagation

– Hypothesis III: Change proneness” correlates with dynamic coupling of components

• Experiments to measure – static metrics

– dynamic metrics

– for selected case studies

• Statistical analysis of the data:– computing correlations

– developing linear or non-linear regression models.

• In summary:– Hypothesis I: not quite

supported– Hypothesis II: supported– Hypothesis III: rejected

NASA OSMA SAS’03 5

Research Heaven,West Virginia

Project Overview

FY03

• Conduct empirical study of the relationship between the static and dynamic metrics.

• Measure Static Metrics • Measure related Dynamic Metrics • Conduct statistical Analysis of the

Data

FY04

• Establish Relationships between Static and Dynamic Metrics

Research Heaven,West Virginia

Tools

NASA OSMA SAS’03 7

Research Heaven,West VirginiaTools for Static Metrics

• We can automatically collect several static metrics (OO metrics,Complexity metrics and Size/Volume metrics).

• We are currently using Understand for Java and C++ by Scientific Toolworks for collecting static metrics.

NASA OSMA SAS’03 8

Research Heaven,West Virginia

Tools for Dynamic Metrics

• Rational Rose-Realtime simulation facility is used to simulate UML models

• At the code level, Rational Purifyplus produces dynamic sequence diagrams at run-time

• JProbe profiler by Quest Software serves a similar purpose for Java code but does not produce sequence diagrams.

Research Heaven,West Virginia

Hypothesis I

Static coupling metrics correlate with error propagation in software architectures.

NASA OSMA SAS’03 10

Research Heaven,West Virginia

Study of Static Coupling Metrics and Error Propagation

• Correlate the experimental data on – error propagation – static information coupling (connector based).

• Correlate:– experimental error propagation – and static NAS (number of associations) coupling measure

(connector Based).

• Correlate:– experimental error propagation – and CBO (connector based).

NASA OSMA SAS’03 11

Research Heaven,West Virginia

NASA OSMA SAS’03 12

Research Heaven,West Virginia

Table Highlighting Non-zero Values Information Flow Vs Ep

From To1 1 5 0.1331 2.3219 10672 1 6 0.128 2.3219 10553 1 3 0.0557 2 5924 3 1 0.0161 1.585 5595 7 1 0.75 2.3219 646 1 4 0.4912 2 577 2 1 0.7838 2.8074 378 1 2 0.5 2.585 369 4 2 0.6429 2 28

10 4 1 0.7083 2 2411 3 2 0.7917 1.585 2412 2 8 0.4286 2 713 10 1 1 1 414 6 2 1 1 415 5 2 1 1 4

EntryConnector EP

(Dynamic)# Fault

InjectionsInformation Flow(Stat)

NASA OSMA SAS’03 13

Research Heaven,West Virginia

Table Highlighting Non-zero Values CBO Vs Ep

From To1 1 5 0.1331 16 10672 1 6 0.128 16 10553 1 3 0.0557 15 5924 3 1 0.0161 3 5595 7 1 0.75 5 646 1 4 0.4912 15 577 2 1 0.7838 31 378 1 2 0.5 6 369 4 2 0.6429 4 28

10 4 1 0.7083 3 2411 3 2 0.7917 4 2412 2 8 0.4286 14 713 10 1 1 3 414 6 2 1 2 415 5 2 1 2 4

EntryConnector EP

(Dynamic)# Fault

InjectionsCBO

NASA OSMA SAS’03 14

Research Heaven,West Virginia

Correlation Values

• Correlation between Information Flow Coupling and Error Propagation is 0.545657

• Correlation between CBO and Error Propagation is 0.130478

• Correlation between NAS and Error Propagation is 0.0• The correlations are only for the values where the # fault

injections >25.

NASA OSMA SAS’03 15

Research Heaven,West Virginia

Discussion of Results

• Information flow coupling metric shows a higher correlation with dynamic error propagation than CBO.

• On further analysis the R2 correlation between Information Flow and EP was 0.297742.

• This low correlation value could be partly due to the small size of our sample (9 data points)

• This result is not sufficient to validate the Hypothesis I

Research Heaven,West Virginia

Hypothesis II:

Static error propagation correlates with dynamic error propagation

NASA OSMA SAS’03 17

Research Heaven,West Virginia

Static and Dynamic Error Propagation

• We will use the experimental data on error propagation and the static error propagation measure(connector Based).

• The static error propagation measure is based on the relation developed in the software architecture metrics USIP using the information flow between components.

NASA OSMA SAS’03 18

Research Heaven,West Virginia

Experimental (Dynamic) Error Propagation Matrix of HCS

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10C1 1 0.11 0.15 0.3 0.76 0.73 0 0 0 0C2 0.37 1 0 0 0 0 0 0.52 0 0C3 0.03 0.32 1 0 0 0 0 0 0 0C4 0.03 0.19 0 1 0 0 0 0 0 0C5 0 0.67 0 0 1 0 0 0 0 0C6 0 0.28 0 0 0 1 0 0 0 0C7 0.32 0 0 0 0 0 1 0 0 0C8 0 0 0 0 0 0 0 1 0 0C9 0 0 0 0 0 0 0 0 1 0

C10 0 0 0 0 0 0 0 0 0 1

B

A

NASA OSMA SAS’03 19

Research Heaven,West Virginia

Static Error Propagation Matrix of HCS

C1 C2 C3 C4 C5 C6 C7 C8 C9 C10C1 1 0.11 0.38 0.44 0.58 0.59 0 0 0 0C2 0.18 1 0 0 0 0 0 0.5 0 0C3 0.11 0.26 1 0 0 0 0 0 0 0C4 0.17 0.15 0 1 0 0 0 0 0 0C5 0 0.57 0 0 1 0 0 0 0 0C6 0 0.18 0 0 0 1 0 0 0 0C7 0.23 0 0 0 0 0 1 0 0 0C8 0 0 0 0 0 0 0 1 0 0C9 0 0 0 0 0 0 0 0 1 0

C10 0.08 0 0 0 0 0 0 0 0 1

B

A

NASA OSMA SAS’03 20

Research Heaven,West Virginia

Correlation Results

• There is a strong correlation between static error propagation and dynamic error propagation.

• Correlation = 0.875098• R2 = 0.765797• This supports Hypothesis II and our conjecture that

some static metrics correlate with related dynamic metrics.

Research Heaven,West Virginia

Hypothesis III:

“Change proneness” correlates with dynamic coupling of components

NASA OSMA SAS’03 22

Research Heaven,West Virginia

Change Proneness and Change Propagation

• Hypothesis III “change proneness correlates with dynamic coupling of components”– from the study of Erik Arisholm June 2002, – Dynamic Coupling Measures for Object-Oriented Software,

– Eighth IEEE Symposium on Software Metrics

• Correlate – dynamic import coupling, – dynamic export coupling metrics – with change proneness measured – based on the software architecture metrics USIP using interface

change propagation probabilities.

NASA OSMA SAS’03 23

Research Heaven,West Virginia

Dynamic Coupling Metrics

• The dynamic coupling metrics we focus on are as follows1. Dynamic Export Coupling

2. Dynamic Import Coupling

3. Total Dynamic Coupling sum of both import and export coupling.

NASA OSMA SAS’03 24

Research Heaven,West Virginia

Experiment

• The experiment was conducted on the SharpTool case study.with 32 components.

• Dynamic coupling was obtained by counting the object interactions dynamically at run time. During the run all functionalities of the tool were executed.

NASA OSMA SAS’03 25

Research Heaven,West Virginia

Experiment Results

Export Import Proneness18 14 0.2

39466 0 0.3757317 20 07182 7179 3.4

75 2 10 0 10 0 1

78 0 2.523 97 0.510 48 08 3 19 25 0.9591 1 03 5 0

23 78 0.85718 14 0.734

123 151 0.56 9 0.151

92 0 0.677 0 25 0 19 5 01 0 0

374 32893 1.541 95 053 3 0.5411 44 0.603

153 15208 0.75181 215 013 5 0.42 158 0.8889 60 1.142

NASA OSMA SAS’03 26

Research Heaven,West Virginia

Discussion of Results

• Zero correlation between – change proneness – and dynamic coupling.

• We also computed CBO and found zero correlation between change proneness and CBO

• This negates Hypothesis III.• The study in the paper by Erik Arisholm was done

using actual project changes data .• In our experiment we used static change proneness data

using our interface change propagation probabilities defined in the architecture metrics USIP

Research Heaven,West Virginia

NASA OSMA SAS’03 28

Research Heaven,West Virginia

So how good is static analysis?

• Hypothesis I: Static coupling metrics correlate with error propagation in software architectures.– Not Sure

• Hypothesis II: Static error propagation correlates with dynamic error propagation– Yes

• Hypothesis III: Change proneness” correlates with dynamic coupling of components– No

• And so, what does all this mean? – IV&V can not run code – And still offer value added to NASA

systems