1

Sustainability Evaluation

of Software Architectures

Heiko Koziolek Industrial Software Systems

ABB Corporate Research, Germany

2 2

3

• Technology – GUI: Microsoft Visual Basic

– Middleware: Classic OPC

– OS: Windows XP

• Functionality (select.) – Basic alarm management

– No system integration

– No online updates

• Code & Documentation – Defined APIs and layers

– Design rationale present

– Up-to-date documentation

• Technology – GUI: Microsoft WPF

– Middleware: OPC UA

– OS: Windows 7

• Functionality (select.) – Rich alarm management

– Multi-system integration

– Online updates

• Code & Documentation – Some layer violations

– Some design rationale lost

– Outdated documentation

2004 2011

Measuring sustainability?

4

5

83%

70%

54%

40%

17%

16%

15%

11%

5%

Experience

Prototyping

Scenarios

Checklists

Simulation

Questionaires

Metrics

Others

Math. models

[Source: AliBabar & Gorton 2009, survey with 88 architects]

State of practice: Techniques for architecture evaluation

Research Questions

• RQ1: Scenario-based methods & sustainability evaluation?

• RQ2: What kind of architecture-level metrics?

• RQ3: Implications for industry & research?

6

Search

• 14 conferences, 12 journals, 10 books

• Included only methods/metrics

from renowned venues

• Excluded dormant or unvalidated

methods/metrics

7

Research Questions

• RQ1: Scenario-based methods & sustainability evaluation?

• RQ2: What kind of architecture-level metrics?

• RQ3: Implications for industry & research?

8

9

Scenario-based Methods

• SAAM (5%)

• ATAM (5%)

• ARID

• SAAMER

• PASA

• SBAR

• SAAMCS

• ISAAMCR

• ALMA (1%)

• ESAAMI

• SAAF

• ASAAM

• SALUTA

• SACAM

• DOSAM

• SBAR

Practitioners

applying

the method

[AliBabar &

Gorton2009]

dormant

Sustainability Evaluation Criteria

Change scenario specification

Ripple effect analysis

Variation point analysis

Architecture improvement

Tradeoff analysis Existing tools &

artifacts

Sustainability Evaluation Criteria

Change scenario specification

Ripple effect analysis

Variation point analysis

Architecture improvement

Tradeoff analysis Existing tools &

artifacts

12

Sustainability evaluation ATAM ALMA

Change

scenario spec.

Ripple effect

analysis

Variation point

analysis

Architecture

improvement

Tradeoff

analysis

Existing

artifacts/tools

Return on

Investment

13

Sustainability evaluation ATAM ALMA

Change

scenario spec.

Quality attribute

scenario template

Top-down, bottom-up,

no template

Ripple effect

analysis

Manual, based on

experience

Manual, based on

experience

Variation point

analysis

n/a n/a

Architecture

improvement

Modifiability Tactics,

Styles

n/a

Tradeoff

analysis

Utility tree with

stakeholder preferences

n/a

Existing

artifacts/tools

No support No support

Return on

Investment

Cost: 30-70 person days

Benefit: not quantified

Cost: unknown

Benefit: not quantified

Findings RQ1: Scenarios

• ATAM: more features

• ALMA: more specific for sustainability

• Manual ripple effect analysis

• No reverse engineering

• No tool integration

• Return on Investment unknown

• In practice: use best of both methods

14

Research Questions

• RQ1: Scenario-based methods & sustainability evaluation?

• RQ2: What kind of architecture-level metrics?

• RQ3: Implications for industry & research?

15

Architectural metrics categories

Similarity of Purpose

Encapsulation Independent

Compil. & Testing

Acyclic Dependencies

Size 16

[Sarkar2007]

Architectural metrics categories

Similarity of Purpose

Encapsulation Independent

Compil. & Testing

Acyclic Dependencies

Size 17

[Sarkar2007]

18

Architecture Metrics

Similarity of

Purpose

Encapsulation

Independent

Compilation

& Testing

Acyclic

Dependencies

Size

19

Architecture Metrics

Similarity of

Purpose

Concept Domination Metric, Concept Coherency Metic,

API Function Usage Index, Concern Diffusion over Arch. Components,

Concern Scope, Concern Overlap

Encapsulation Ratio of Cohesive Interactions, Import Coupling, Export Coupling,

Modularization Quality, Afferent Couplings, Efferent Couplings,

Component-level Interlacing Between Concerns,

Lack of Concern-based Cohesion, Module Interaction Index,

Non-API Function Closedness Index, Implicit Dependency Index,

Base class fragility index, Inheritance-based inter-module coupling,

Not-programming-to-interfaces Index, Association-induced coupling,

State Access Violation Index, Entropy of an architectural slicing,

Architecture Slicing Cohesion, Decision Volatility

Independent

Compilation

& Testing

Cumulative Component Dependency, Average Cumulative Component

Dependency, Normalized Cumulative Component Dependency,

Coupling of a module, Intra-module coupling of a module, Cohesion of a

module, Abstractness, Instability, Distance from the Main Sequence,

Module Interaction Stability Index, Normalized Testability Dependency

Metric, Plugin Pollution Index, Change impact, Independence Level

Acyclic

Dependencies

Package Dependency Cycles, Cyclic Dependencies Index, Layer

Organization Index, Excessive Structural Complexity

Size Module Size Boundness Index, Module Size Uniformity Index

20

Architecture Metrics

Similarity of

Purpose

Concept Domination Metric, Concept Coherency Metic,

API Function Usage Index, Concern Diffusion over Arch. Components,

Concern Scope, Concern Overlap

Encapsulation Ratio of Cohesive Interactions, Import Coupling, Export Coupling,

Modularization Quality, Afferent Couplings, Efferent Couplings,

Component-level Interlacing Between Concerns,

Lack of Concern-based Cohesion, Module Interaction Index,

Non-API Function Closedness Index, Implicit Dependency Index,

Base class fragility index, Inheritance-based inter-module coupling,

Not-programming-to-interfaces Index, Association-induced coupling,

State Access Violation Index, Entropy of an architectural slicing,

Architecture Slicing Cohesion, Decision Volatility

Independent

Compilation

& Testing

Cumulative Component Dependency, Average Cumulative Component

Dependency, Normalized Cumulative Component Dependency,

Coupling of a module, Intra-module coupling of a module, Cohesion of a

module, Abstractness, Instability, Distance from the Main Sequence,

Module Interaction Stability Index, Normalized Testability Dependency

Metric, Plugin Pollution Index, Change impact, Independence Level

Acyclic

Dependencies

Package Dependency Cycles, Cyclic Dependencies Index, Layer

Organization Index, Excessive Structural Complexity

Size Module Size Boundness Index, Module Size Uniformity Index

Example: Module Interact. Idx

𝒎

API Non-API

𝑀𝐼𝐼 𝒎 = 𝐾𝑒𝑥𝑡(𝑓

𝑎)𝑓𝑎∈ 𝑓1 𝑎 … 𝑓𝑛

𝑎

𝐾𝑒𝑥𝑡(𝑚)

=1 + 1

3 ≈ 0.67

call call call

21

[Source: Sarkar2007 IEEE TSE]

Example: Comparing Systems

22

[Source: Sarkar2007 IEEE TSE]

Findings RQ2

• >40 architecture-level source code metrics in literature

• Measures for architecture and modularization quality

• Success factors for application: – Mix of metrics

– Combination with class-level / process metrics

• Limitations: – Empirical validation

– Tool support

23

Research Questions

• RQ1: Scenario-based methods & sustainability evaluation?

• RQ2: What kind of architecture-level metrics?

• RQ3: Implications for industry & research?

24

Findings RQ3: Implications

• More empirical research

• Method integration

• Effective formal models

• Codify experiences

• Explore other

approaches (simulation?)

25

26

Conclusions

• Sustainability Evaluation of Architectures – change scenarios:

basic support for sustainability criteria

– architectural code metrics: measure modularization, lack empirical evidence

• Research opportunities – combined methods

– empirical research

27

28

Existing surveys

• Surveys on scenario-

based method

– Dobrica 2002

– Ali Babar 2004

– Kazman 2005

– Breivold 2010

– ...

• Surveys / papers

on code metrics

– McCabe 1976

– Halstead 1977

– Chidamber 1994

– Riaz 2009

– ...

28

ATAM ALMA

Goals Sensitivity & Tradeoff

Analysis

Change impact analysis,

maint. effort prediction

Architectural

description

Process, data-flow, uses,

physical & module view

Any

Process

support

Comprehensive Limited

Tool support

n/a n/a

Knowledge

repository

Recommended n/a

Validation > 20 industrial case

studies documented

7 industrial case studies

(before 2004) 29