DEVELOPMENT OF MBSE/UML MATURITY

MODEL

ÖZLEM DEMIRCI

MASTER THESIS 2010

INFORMATICS

DEVELOPMENT OF MBSE/UML MATURITY MODEL

ii

SUBJECT

INFORMATION TECHNOLOGIES AND MANAGEMENT IN

INFORMATICS

DEVELOPMENT OF MBSE/UML

MATURITY MODEL

ÖZLEM DEMIRCI

This exam work has been carried out at the School of Engineering in Jönköping in the

subject area IT and Management in Informatics. The work is the two-year university

diploma program of the Master of Science program.

The authors take full responsibility for opinions, conclusions and findings presented.

Examiner: Vladamir Tarasov

Supervisor: Christer Thörn

Abstract

iii

Abstract

Model Based Software Engineering (MBSE) is becoming popular in the industry today. Many

organizations, also including small and medium scales organizations are adopting modeling

approaches. Models are becoming the main artifact of the software development process and

they provide organization to test their product and detect problems before the implementation.

UML (Unified Modeling Language) is the most common modeling language that is used by

organizations during modeling process so that focus is on UML in the organizations.

Therefore organizations that are developing software products in the sense of MBSE and

using UML as a main modeling language are the interests of this paper. These organizations

that are performing modeling approaches are not aware of how advanced or un-advanced they

are performing the related modeling approaches during model based software development

process. They are unaware of measuring and evaluating their modeling practices. This is

because of lack of standards and guidelines regarding model based software development. By

considering this unawareness of organizations, various affects that can effect on model based

development process are examined like modeling process, quality assurance techniques,

personnel competence on modeling practices, training of personnel, design and structure of

the UML Models, modeling tools, transformation issues, and syntax and semantic regulations

of UML. These are considered as most essential aspects regarding the occurrence in literature.

Regarding these aspects, a MBSE/UML Maturity Model is provided where 6 maturity levels

regarding the aim of modeling within the organizations and aspects required to be filled out.

Therefore this Maturity Model different than traditional software development maturity

models consider various aspects at different levels of organization and consider models as the

main development artifacts. The practices and tasks required to be performed are based on

models and modeling approaches. This MBSE/UML Maturity Model provides both

practitioners and researchers to evaluate and measure the proficiency of modeling approaches

performed in the organizations.

Acknowledgments

iv

Acknowledgments

I would like to thank my supervisor Christer Thörn at Jönköping School of Engineering for

helping me to shape the idea behind this thesis work and supporting me every step of the

research process. Thank You.

Key Words

v

Key Words

Model Based Software Engineering (MBSE), Model Driven Development (MDD), UML

(Unified Modeling Process), Maturity Model

Table of Contents

vi

Table of Contents

1. INTRODUCTION ............................................................................................................. 1

1.1 Background .................................................................................................................. 1

1.2 Problem Description .................................................................................................... 2

1.3 Purpose ........................................................................................................................ 3

1.4 Research Questions ...................................................................................................... 4

1.5 Topic Vocabulary ........................................................................................................ 5

1.6 Disposition ................................................................................................................... 5

2. THEORETICAL BACKGROUND ................................................................................... 7

2.1 Previous Research ........................................................................................................ 7

2.2 Models and Model Based Software Engineering (MBSE) .......................................... 8

2.3 Modeling Languages and UML ................................................................................. 10

2.3.1 Types of UML Diagrams ................................................................................... 12

2.3.2 UML Profiles ...................................................................................................... 15

2.3.3 OCL (Object Constraint Language) ................................................................... 17

2.3.4 Meta-Model ........................................................................................................ 18

2.4 DSL (Domain Specific Language) ............................................................................ 18

2.5 Maturity Model .......................................................................................................... 19

3. METHODOLOGY ........................................................................................................... 20

3.1 Data Sources .............................................................................................................. 20

Table of Contents

vii

3.2 Data Collection .......................................................................................................... 21

3.3 Data Analysis ............................................................................................................. 21

3.4 Maturity Model Development Method ...................................................................... 22

3.5 Quality of Science ..................................................................................................... 25

4. EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF ASPECTS 26

4.1 Quality of UML Diagrams in Modeling .................................................................... 26

4.2 Evaluation Criteria for MBSE/UML Maturity Model ............................................... 28

4.2.1 Essential Aspects for MBSE/UML Maturity Models ........................................ 28

4.2.2 Indicators for Each Aspect in MBSE/UML Maturity Model ............................. 32

4.2.3 Different Levels of Organization for MBSE/UML Maturity Model ................. 42

4.2.4 The General Matrix for MBSE/UML Maturity Model ...................................... 43

5. MBSE/UML Maturity Model ........................................................................................... 47

5.1 MBSE/UML Maturity Model Structure and Levels .................................................. 47

5.2 MBSE/UML Maturity Level Dimension ................................................................... 48

5.2.1 Maturity Level 0: No Modeling ......................................................................... 49

5.2.2 Maturity Level 1: Basic Modeling ..................................................................... 49

5.2.3 Maturity Level 2: Initial MBSE/UML ............................................................... 50

5.2.4 Maturity Level 3: Defined MBSE/UML ............................................................ 51

5.2.5 Maturity Level 4: Integrated MBSE/UML ......................................................... 52

5.2.6 Maturity Level 5: Optimized MBSE/UML ........................................................ 53

Table of Contents

viii

6. Assessment of Maturity Levels of MBSE/UML Maturity Model ................................... 54

7. VALIDTY OF MBSE/UML MATURITY MODEL ....................................................... 63

8. CONCLUSION ................................................................................................................ 64

REFERENCES ......................................................................................................................... 66

APPENDIX .............................................................................................................................. 71

Appendix 1: MBSE/UML Modeling Process Evaluation Questionnaire ............................. 71

Appendix 2: MBSE/UML Quality Assurance Techniques Evaluation Questionnaire ........ 78

Appendix 3: MBSE/UML Personnel Competence Evaluation Questionnaire vol. 1........... 85

Appendix 4: MBSE/UML Personnel Competence Evaluation Questionnaire vol. 2........... 92

Appendix 5: MBSE/UML Tools Evaluation Questionnaire ................................................ 94

Appendix 6: MBSE/UML Design Evaluation Questionnaire ............................................ 100

Appendix 7: MBSE/UML Syntax and Semantic Evaluation Questionnaire ...................... 108

Appendix 8: MBSE/UML Transformation Evaluation Questionnaire .............................. 110

Table of Figures

ix

Table of Figures

Figure 1 A sample framework for MBSE/UML Maturity Model ............................................... 4

Figure 2 Disposition ................................................................................................................... 6

Figure 3 MDD Adopting Spectrum, adapted from [1] ............................................................. 10

Figure 4 Krutcher 4+1 view model .......................................................................................... 12

Figure 5 Sample Creating a New Blog Account Use Case Diagram [11] .............................. 14

Figure 6 Sample Activity Diagram [11] .................................................................................. 15

Figure 7 Sample Class Diagram for Blog Account .................................................................. 15

Figure 8 Example of UML Profile ........................................................................................... 17

Figure 9 Sample OCL example for UML Profile ―colors‖ [39] ............................................. 18

Figure 10 Followed Data Analysis Steps ................................................................................. 22

Figure 11 Maturity Model Development Steps ........................................................................ 23

Figure 12 A Sample Declaration of Internal and External Stakeholder .................................. 33

Figure 13 A Sample ATM UML Use Case Diagram ................................................................ 34

Figure 14 Effects of Crossing and Bends [22] ......................................................................... 37

Figure 15 The Structure of MBSE/UML Maturity Model (adapted from [47] and [48]) ........ 47

Figure 16 MBSE/UML Maturity Model Levels ........................................................................ 49

INTRODUCTION

1

1.INTRODUCTION

1.1 Background

“My method is different. I do not rush into actual work. When I get a new idea, I start at once

building it up in my imagination, and make improvements and operate the device in my mind.

When I have gone so far as to embody everything in my invention, every possible

improvement I can think of, and when I see no fault anywhere, I put into concrete form the

final product of my brain” said Nikola Tesla. And in software engineering similar method has

recently been widely used.

Nikola Tesla did not talk about the use of modeling as a process in software development, but

the idea can be considered more or less as same. Even though models are been used in the

software industry for a long time, the activity of using modeling as a process in software

development has recently become widespread since the need to abstract the system and its

behaviors have been considered by both researchers and practitioners. Software engineers did

not start development by implementation of code phase in software life cycle, including steps

like requirement specifications, analysis, design, implementation and maintenance. The aim

of these processes is to improve the productivity and to solve possible problems at early

stages of development. By applying the principles of Model Based Software Engineering

(MBSE) where the models are developed as solutions for particular problems during software

development process, these aims can be achieved much easier. For instance, if the modeling is

performed in an advanced way it is possible to detect errors early in the development life

cycle or to test the software models before the implementation is executed.

Modeling can be performed for different applications in software development. Besides the

large scale enterprises, both medium and small sized enterprises are also interested in

modeling. Due to the wide application of modeling, numerous informal and formal

approaches to modeling have been developed, such as Entity Relationship Diagrams (ERD)

for modeling data, Specification and Description Language (SDL) for modeling

telecommunication systems, formal modeling languages such as Z and B, and the Unified

Modeling Language (UML) which is the modeling language most widely used in industry

today [1].

Modeling a system is not simple. There are various affects and aspects need to be considered.

For instance, modeling a large scale system is complex since it isn‟t possible to model some

of the aspects and elements of such a system because of lack of modeling approaches and a

single modeling language. Moreover, based on literature review, it can be stated that there

exists no single modeling language that can cover all the needs to model a system. However,

UML (Unified Modeling Language) that is proposed by OMG is the most common modeling

language used in industry and also an interest of researchers. Modeling language is the core of

INTRODUCTION

2

using modeling as a process. Therefore the syntax need be well defined and regulated. What

we mean by syntax here is the rules, regulations, and notations to describe a modeling

language and semantic is the meaning of expressions stated in the language. In addition to

these, modeling requires a good quality of design since there exists both technical and un-

technical people, so the design of the models should be clear and comprehensible. The

modeling process need to be well documented and the participants in this process need to be

well trained and skilled. All these issues are considered by the industry but sometimes they do

not give that much importance to one or more of these. Since this is a new field, a proper,

qualified, and sufficient approach to adopt this new process do not exist.

Therefore, in this thesis a maturity model for modeling approaches in organization is

presented by indicating the aspects that need to be considered at different levels of

abstraction. Moreover, the activities and tasks that needs to be performed in order to use

modeling as a process in an advanced way is provided. A focus on UML is provided in this

thesis since it is the most common modeling language in the industry. The maturity model

provided here includes some levels for the proficiency of the organization using modeling

approaches and UML together. The goal of this maturity model is that an organization and

also researchers can assess their proficiency level regarding their modeling approaches and

can get knowledge what else they need to do in order to perform modeling approaches in an

advanced way.

1.2 Problem Description

Regarding the popularity of adopting these new approaches and advantages of modeling,

organizations are started to implement these modeling practices and activities. They face with

lack of standardization on MBSE approaches and lack of standardization on modeling

languages that required to be used while adopting modeling approaches, and several other

lacks, insufficient issues, and problems. Organizations that adopt these MBSE approaches are

aware of how advanced they are performing and what they need to do in order to improve

their modeling practices. In other words, they adopt modeling approaches, but do not know

how to evaluate and measure themselves with respect to their practices regarding their

organizational and modeling goals. This leads to problems and discussions on how they can

improve their modeling practices.

There exists several studies on Model Driven Development and models such as quality goals

that need to be achieved, the lacks of modeling languages utilized in adopting modeling

approaches, different and various issues having an impact on modeling process, and other

researches related to on this field. All this researches are essential for people and

organizations that are involved in adopting modeling approaches. The existing maturity

models and/or frameworks are developed for traditional software development processes, so

organizations which are dealing with MBSE are aware of how mature they are applying these

modeling approaches within their organizations. For instance, CMMI (Capability Maturity

INTRODUCTION

3

Model Integration) - a maturity model proposed by SEI (Software Engineering Institute)

describes process improvement activities required to be performed within the organization.

SPICE (Software Process Improvement and Capability Determination) is also supporting

continuous process improvement activities required to be performed within the organization.

Both are developed for traditional software development process so they are not providing

and/or supporting activities and tasks required to be performed in order to improve the

proficiency of model based approaches. There is a research going on developing a Model

Driven Development (MDD) Maturity Model within the MODELWARE1 project which is

taking CMMI as basis and reference so that only process improvement activities are

considered. Moreover CMMI is too complex for many small and medium size organizations.

Therefore this maturity model can be perceived as too complex for small and medium size

organizations as well.

When these factors and problems are taken into consideration, it seems useful to be able to

evaluate the proficiency level of organizations‟ modeling approaches utilization with regard to

the best practices of the organization and the insufficient practices of the organization. An

improvement on model based approaches can be provided.

1.3 Purpose

The goal of this master thesis is to develop a maturity model for measuring and evaluating the

level of MBSE/UML proficiency in an organization. As discussed in previous section (1.2)

MDD is becoming popular in the industry. However, any standards for proficiency in using

modeling approaches do not exist.

As mentioned previously, there exists several maturity models and each measures and

evaluates different aspects at different levels. However, the aspects that they evaluate are

specific and mostly only one aspect. Moreover these existing maturity models are developed

for traditional software development process so that the tasks and activities that need to be

performed differ from MDD where the models became the key artifact of the process.

Therefore, this thesis aims to develop a maturity model that evaluates and measures the

different aspects of MBSE/UML at different levels of organization like individual, team, etc.

Considering that both practitioners and researchers are interested in to evaluate the

proficiency level of adopting modeling approaches within the organization, these both parties

will be able to benefit from the outcomes of this thesis. The practitioners will be able to

examine their modeling practices and the results for their practices so that they will know

what they are doing at advanced level, what are the best practices of them, and what they need

to do in order to improve. Like CMMI, a different way to improve the development practices

and tasks will be shown in this thesis. The researchers will also have benefit from this thesis.

1 MODELWARE is a project co-funded by the European Commission under the “Information

Society Technologies” Sixth Framework Programme (2002-2006).

INTRODUCTION

4

They will be aware of what practices and tasks an organization should consider while

adopting modeling approaches. Researchers will be able to examine the best practices,

evaluation methods of proficiency of modeling approaches, and essential aspects that need to

be considered during MDD process.

The outcomes and results are;

A maturity model for MBSE/UML approaches with a description of aspects and

levels that are targets for evaluations like shown in Figure 1.

Figure 1 A sample framework for MBSE/UML Maturity Model

Sets of instruments for how to perform the evaluations (ex: questionnaires,

experiments, etc.)

A review of existing approaches to maturity in models, evaluations, etc.

1.4 Research Questions

Even though the aim of this thesis is to develop a maturity model for evaluating the

proficiency usage of modeling approaches, the work was guided by research questions. With

these research questions, it is possible to assign and decide the important aspects, practices

and tasks that require to be performed during modeling.

RQ1: What model quality goals are important in MBSE approaches that use UML as a core

modeling language?

RQ2: How to evaluate the quality and proficiency in usage of UML and MBSE approaches

during a software development process?

INTRODUCTION

5

The results that emerge from these research questions enable us to develop the criteria for

different aspects at different abstraction levels of modeling process in order to evaluate the

proficiency of modeling approaches applied in the organizations.

1.5 Topic Vocabulary

Many abbreviations are used in this paper so here the most used abbreviations are presented

below.

MDD= Model Driven Development

MBSE= Model Based Software Engineering

MDA= Model Driven Architecture

MBD= Model Based Development

UML= Unified Modeling Language

OCL= Object Constraint Language

1.6 Disposition

This thesis is divided into 8 chapters which is shown and described briefly in Figure 2 below.

A general knowledge is provided about the thesis and related topics like MBSE, UML, and

maturity models so that the reader can become familiar with the rest of the paper.

The methodology chapter provides the research process of the thesis, how data is collected

and what kind of data is collected so that the reader will be able to understand how the

research is performed.

The rest of the chapters are about the findings from the data sources and analyzing them

according to the purpose of the thesis. In chapter 4, the results for the research questions are

provided. Moreover how these results are analyzed can be also read in chapter 4. These results

constitute the key areas of the topic.

A MBSE/UML Maturity Model is provided in chapter 5. The development of that is based on

the findings from chapter 4, previous studies, and previous research performed on this field.

Then in chapter 6, it is defined how the maturity of organization will be assessed so that the

instruments designed for this aim is presented there.

Chapter 7 indicates how the MBSE/UML Maturity Model will be validated.

Chapter 8 concludes all tasks performed for the thesis.

INTRODUCTION

6

Figure 2 Disposition

THEORETICAL BACKGROUND

7

2.THEORETICAL BACKGROUND

2.1 Previous Research

Previous research on this field in the industry is mostly addressing the model quality goals

that are required to be considered in order to develop good quality models since models are

the main artifacts of the MDD process. For instance, “Definitions and approaches to model

quality in model-based software development”, Parastoo Mohagheghi et al. [1] examines

various articles published on quality of models and define 6C model quality goals

(correctness, completeness, consistency, etc.) for model quality that is commonly used in the

industry. In [22] the authors define model goals regarding UML structure and MBSE

principles. They define different model characteristics, different modeling purposes, and

relationships among them. Then they provide a UML model quality framework for different

phases of software development process. Moreover, in article [9] it is indicated that model

quality goals require to be defined properly while adopting MBSE principles. [23], [8], [7],

[24], and [5] are other researches that deals with model quality goals. The common thing of

these researches is all state that model goals are essential and require to be defined. Moreover

they all indicate that UML‟s semantic structure is lacking when a MDD process is considered.

Defining the model quality goals is not adequate. It has to be supported by providing some

metrics and measurement techniques so that they can be measure and evaluate if the

developed models meet the model quality goals or not. [25], [26], [27], [5], [50], [29], [30],

and [31] provide some metrics that can be applicable to measure and evaluate whether the

model quality goals are met or not. For instance, Baroni et al. [30] emphasize on defining

metrics by using OCL (Object Constraint Language) for UML model quality. Lange [5]

defines metrics and rules. He also defines model quality characteristic and modeling purposes.

Then he relates them to each model quality characteristics regarding the modeling purposes.

His consideration is related to UML model quality which is the same aim as this thesis.

Since MBSE is not only developing models for communication, analysis, and prediction but

also for transformation and code generation. Therefore, the requirements for developing

models as solution and implementation source require additional activities and tasks to be

performed. Moreover many researches [1], [5], [7], and [8] indicate that UML‟s lack of

semantic is an issue required to be considered while adopting MBSE principles so the

concepts like Meta-Modeling, UML Profiles, OCL, and DSL (Domain Specific Languages)

come up. For instance [32], [33], and [34] emphasizes on meta-modeling and [32] provides a

simple meta-model developed for modeling the blueprints of a software system.

With respect to the research on model quality, some experiment and case studies show that

while adopting modeling approaches, defining the model quality goals and measuring them

are not the only issues require to be performed. For instance, [35] is a case study that

THEORETICAL BACKGROUND

8

examines the Motorola‟s MDD process and encounters some issues like lack of common

tools, lack of abstraction, lack of well-defined semantics, lack of integrated and migration

tools, lack of team experience on modeling, ill-defined processes, and so on. Moreover it

states that Motorola is developing a Modeling Challenge Levels to provide guideline for

MDD process adoption. In addition to these, Parastoo Mohagheghi et al. [36] examines two

organizations that are adopting MDD and states that because of lack of standards, guidelines,

and ill-process definitions, organizations are faced with several problems. The author of [37]

also states the issues require to be performed properly like process definition, training of

personal, maturity of modeling technology and maturity of model related methods. The

common of these case studies and experiment reports is that organizations adopting MDD are

unaware of how mature they are performing these activities and tasks. For this issue, there is

only little research [2] performed as mentioned previously.

2.2 Models and Model Based Software Engineering (MBSE)

It is quite better to know and understand what is model at first. Models are representations of

a (software) system at an abstract level [10]. A model tries to capture and present a specific

aspect of a system so that only the information related to this aspect is represented. A model

can consist of several diagrams in order to represent the part of system clearly. For instance,

UML 2.0 proposes 13 different diagrams and each is described for various usages and

representation styles (will be discussed in 2.2 Modeling Languages and UML). These various

types of diagrams provide modelers and/or designers to develop the models at different

abstraction levels by describing different features and representations of the system so that

everyone involved in the development process can understand what is required and described,

and essentially the roles assigned.

The aim of using models differs and there exist many ways of using models so the

requirements to describe and develop the models also differ. Models can be used to test the

parts of the system before implement it so that the productivity can be improved. Moreover

models can be used to provide a clear and coherent communication between stakeholders,

clients, programmers, and so on; or the other field that models used is to present the

integration and interoperability of distributed systems. In MBSE, they are aimed to perform

transformation and achieve code from the models. The Figure 3 below shows the adoption of

modeling within the organization by indicating the different aims of it. For instance, if the

organization aims to implement the code from the models, the requirements for the code,

transformation, and transformation tool that need to be considered and presented in the

models. Let us assume that an organization will develop software programmed by C++, so the

object-oriented requirements that need to be captured and the models should be developed

based on this. Therefore the concepts like inheritance, classes come up and a specific part of

THEORETICAL BACKGROUND

9

the system requires to be developed regarding this structure. Consequently, there exists

different ways, aims, and different domains that models are presented.

All these fields that models used indicate us that models have been used in software

engineering world for a long time. They mostly have been used at the analysis and design

steps of software development life cycle in order to describe the problem and requirements

but solutions. Now by the proposition of MDD, the models became main artifact of software

development process and they are intended to be developed as solutions.

After understanding what is model and for what aims they are mostly used in industry, now it

will be clearer to understand what MBSE and MDD are. There exist several names for those

concepts. Some states that Model Driven Engineering (MDE) rather than MBSE; and some

utilize Model Based Software Development (MBSD) or Model Based Development (MBD)

rather than Model Driven Development (MDD). In this paper, we will use MBSE and MDD

in order to decrease the complexity of existing terms.

In article [11], model based software engineering is defined as performing software

engineering on the level of abstraction and the authors simply define MBSE as the

functionality of the system is presented by the models where the implementation details are

hidden.

In article [13], the models are described as a main artifact in the software development and

they define MBSE as “Model-based software engineering covers software development

methods, where models are the main artifacts and some or all other artifacts are derived from

the models [13]”.

The main difference between traditional software development and model based development

is that models are the artifacts in the model driven development. The authors in [1] indicates

that MDE (Model Driven Engineering) covers approaches where development is carried out

using models; often at different abstraction levels and from multiple views; and they also

focuses on that the models are the core sources in order to perform transformations. That

basically means that models can be used to develop other models or either way to develop the

source code.

THEORETICAL BACKGROUND

10

Figure 3 MDD Adopting Spectrum, adapted from [1]

Model Driven Development (MDD) approaches are used in the paradigm of MBSE. In MDD,

the complexities of the large software systems are handled by raising the level of abstraction.

[14] MDD is defined as “a software engineering approach consisting of the application of

models and model technologies to raise the level of abstraction at which developers create and

evolve software, with the goal of both simplifying (making easier) and formalizing

(standardizing, so that automation is possible) the various activities and tasks that comprise

the software life cycle”. In article [15], the authors are also defining MDD as model based

software engineering approaches where the parts of the software development process are

executed automatically using model transformations.

In this thesis, the common way of using MBSE and MDD approaches is to evaluate the usage,

adopting proficiency of these concepts in the industry so that the proficiency is dependent on

the different usage principles of these approaches and models regarding different domains and

contexts. The criteria, the evaluation methods, and best practices that need to be applied will

be discussed later in this thesis.

2.3 Modeling Languages and UML

Since modeling is being used in different areas; there exist many modeling languages and

standards. Majority of them emphasizes on graphical notation because modeling is considered

as a graphic. However, a modeling language can also be textual. In addition to these some of

the modeling languages are executable so that a source code can be generated from the

models. Therefore, regarding the type and aim of modeling language, the set of rules and

structure of them differ. For instance, a graphical modeling language consists of set of

diagrams and a set of rules exists to indicate the relationship between diagrams.

The modeling approaches require a modeling language as it mentioned previously and

modeling language provides standards, a set of rules, and conventions in order to improve the

THEORETICAL BACKGROUND

11

quality and develop models. At that point, UML which is the de facto Standard for modeling

software systems in industry is considered since when adopting modeling standards and

guidelines within an organization the first goal is to settle a common notation as Scott W.

Ambler stated in his book “The Elements of UML style [38]”. He states that at that point

UML is a good start since it defines a common notation and semantics. Moreover OMG

(Object Management Group)2 states that modeling is the designing of software applications

before coding and using a model, those responsible for a software development project's

success can assure themselves that business functionality is complete and correct, end-user

needs are met, and program design supports requirements for scalability, robustness, security,

extendibility, and other characteristics, before implementation in code renders changes

difficult and expensive to make. OMG„s UML helps to specify, visualize, and document

models of software systems, including their structure and design, in a way that meets all of

these requirements.

The utilization area of UML in the industry is wide. UML models are used at several stages

during software engineering, such as the early analysis of the problem domain, documenting

the architecture or specifying the detailed design of a system. The abstraction level of the used

models varies for the different stages. A UML model serves one or more specific purposes

depending on the development stage and the stakeholders who are using the model [5].

Moreover, in the book “Learning UML 2.0” [11] authors mention the quotes of Fowler who

identifies the usage of models (UML models) in a precise way and states that UML models

used in industry are assigned to three different roles which are;

UML as Sketch: The models are designed to represent the communication rather than

describing the system specifications so that if the organization is using the UML models as a

sketch than the models that need to be understood and agreed by all users. The models need to

be comprehensible.

UML as Blueprint: UML models are designed detailed and provide detailed information for

the programmers to develop the code. Therefore the UML models need to be correct and

complete.

UML as Programming Language: UML models are designed and developed as executable.

The focus and the aim is developing models first and then implementing them into the code

by using transformation tools.

2 http://www.omg.org

THEORETICAL BACKGROUND

12

2.3.1 Types of UML Diagrams

The latest version of UML which is UML 2.0 consists of 13 different diagrams and each has

different purposes. However, some are related to each other. For instance, the class diagrams

are used to show the static design view of a system while sequence diagrams are used to show

the interaction between the objects. Russell Miles and Kim Hamilton in their book “Learning

UML 2.0 [11]” mentions the views of UML diagrams on overall and he focuses on the views

defined by Philippe Kruchten‟s 4+1 view model where a model is broken into several views

and each view captures specific aspect of the system that is modeled. As shown in Figure 4,

the views are logical, process, development, physical, and use case views.

Figure 4 Krutcher 4+1 view model

The Table 1 below provides basic description about Krutchen‟s 4+1 view model and example

UML diagrams.

Table 1 Krutchen’s View Model Description

View Type Description UML Diagram

Logical View System‟s parts and how they

interact each other are described.

Class, Object, State

Machine, and Interaction

Diagrams

Process View The steps and/or tasks of the

processes are described.

Activity Diagrams

Development The system‟s parts organization

into modules and components is

Package and Component

THEORETICAL BACKGROUND

13

View described. Diagrams

Physical View How the final deployed system is

performed regarding to the 3 other

views of the model is described.

Deployment Diagram

Use Case View What the systems should do from

the outside perspective is described.

Use Case, Description, and

Overview Diagrams

OMG also identifies different perspective of UML diagrams. They provide 3 different

perspectives which are shown and described in Table 2.

Table 2 UML’s Types of Diagrams by OMG

Types of Diagrams Example UML Diagrams

Structural Diagrams Class Diagrams

Object Diagrams

Behavioral

Diagrams

Use Case Diagrams

Sequence diagram

Collaboration diagram

State-chart diagram

Activity diagram

Implementation

Diagrams

Component diagram

Deployment diagram

THEORETICAL BACKGROUND

14

As mentioned previously, even though each diagram is developed for different aims, while

modeling a system these diagrams indicate different issues of the system regarding the other

developed UML diagrams. For instance, let us assume that a modeler is developing a model

which is representing a blog page which allows users to have a blog account in order to post

some entries. The Figure 5 simply illustrates the use case diagram of such a system including

the actors and what they can perform. This use case diagram can be thought as a tool to

provide communication with the client so that the client will be able to comprehend what the

software product will do.

In MBSE, UML diagrams are also used to provide specific features of the system and/or the

software product that will be produced. These specifications will provide developers to

comprehend the problem and then how it is required to be solved. When the example of blog

page system is considered, the UML activity shown in Figure 6 simply illustrates how the

tasks and/or activities of development process will be done. It shows how a decision will be

performed when activities are depending on a condition. For instance, in this example it can

be considered as when a blog account user intends to add a new entry, there are some

limitations to publish his/her entry. It is assumed that if the entry is longer than 1000 words

then it exceeds the word size limit or if the entry is empty then it cannot be published either.

Figure 5 Sample Creating a New Blog Account Use Case Diagram [11]

THEORETICAL BACKGROUND

15

Figure 6 Sample Activity Diagram [11]

Russell Miles and Kim Hamilton states that in their book “Learning UML 2.0 [11]” “Use

cases describe the behavior of your system as a set of concerns. Classes describe the different

types of objects that are needed within your system to meet those concerns. [11]” Regarding

this statement, a sample class diagram of blog account example is shown in Figure 7.

Figure 7 Sample Class Diagram for Blog Account

2.3.2 UML Profiles

It is not necessary to define a new language in order to add some constraints or add new UML

elements by the proposition of UML profiles by OMG. This feature allows customizing the

UML elements such as adding new elements, defining restrictions or constraints. In [39]

authors define UML profiles as set of extensions supporting customizations by using UML‟s

THEORETICAL BACKGROUND

16

extension mechanisms. By this feature developers are enabled to develop UML models in

particular domains.

OMG also defines UML Profiles in “Catalog of UML Profile Specifications3”as “A UML

profile is a specification that does one or more of the following Table 3”:

Table 3 UML Profile Specification by OMG adapted from ―Catalog of UML Profile

Specifications‖

Identifies a subset of the UML meta-model.

Specifies “well-formedness rules” beyond those specified by the identified

subset of the UML meta-model.

“Well-formedness rule” is a term used in the normative UML meta-model

specification to describe a set of constraints written in UML‟s Object

Constraint Language (OCL) that contributes to the definition of a meta-model

element.

Specifies “standard elements” beyond those specified by the identified subset

of the UML meta-model.

“Standard element” is a term used in the UML meta-model specification to

describe a standard instance of a UML stereotype, tagged value or constraint.

Specifies semantics, expressed in natural language, beyond those specified by

the identified subset of the UML meta-model.

Specifies common model elements, expressed in terms of the profile.”

There exist already defined UML Profiles such as UML Profile for CORBA for CCM

(CORBA Component Model) or UML Profile Scheduling. For instance, defining UML

Profile for CORBA is intended to be developed in order to be able to model CORBA

interfaces which support expressing the semantics of CORBA in UML tools also being able to

perform transformations from the models. However, organizations by themselves are also

allowed to define UML Profiles regarding their need. For instance, [39] provides a UML

Profile for MultiTEL (Multimedia Telecommunication Services) in which UML Profile

provides a standard way for designing and documenting systems and application frameworks.

3 Catalog of UML Profile Specifications by OMG

http://www.omg.org/technology/documents/profile_catalog.htm

THEORETICAL BACKGROUND

17

Another UML Profile example is provided by [39] which is a simple example that aims to add

two new elements named weights and colors as shown in Figure 8.

Figure 8 Example of UML Profile

2.3.3 OCL (Object Constraint Language)

OCL is provided by OMG as a modeling specification and OCL is a formal, declarative

language that is used to provide expressions to UML models. In another way it can be defined

as specification of formal constraints in context of a UML model. OCL provides a set of rules

that applicable to UML models. The specifications of constraints are essential since pre- and

post-conditions of operations, invariants, derivation rules for attributes and association etc. are

expressed by constraints. Moreover the restrictions are defined by constraints. Any kind of

UML element can be related with constraints including the UML Profiles that are defined for

new elements. The aforementioned example for adding two new elements which are weights

and colors, the authors of [39] indicates that only classes and associations can be colored. As

a result for this constraint, the following OCL shown in Figure 9 is derived by [39];

THEORETICAL BACKGROUND

18

Figure 9 Sample OCL example for UML Profile ―colors‖ [39]

2.3.4 Meta-Model

A meta-model is a model describing the model. Meta-models capture the different aspects of

the system and make different models to follow the same guideline so that all models

describing the system achieve consistency.

The authors of “Meta-Modeling Semantics of UML[4]” states that “The UML semantics is

described using a meta-model that is presented in terms of three views: the abstract syntax,

well-formedness rules, and modeling element semantics.”

Moreover OMG presents MOF (Meta Object Facility)4 which is a standard used to define the

UML meta-model. It is required to extend the UML meta-model because of its syntax and

semantic lacks. Therefore, MOF is used for this issue.

Therefore meta-modeling is a technique used to provide descriptive modeling language rules

by aligning formal semantics to UML.

2.4 DSL (Domain Specific Language)

DSL is a form of computer languages that are dedicated to a particular domain. Martin

Fowler5 describes it as “The basic idea of a domain specific language (DSL) is a computer

4 http://www.omg.org/mof/

context UML::InfrastructureLibrary::Core::

Constructs::Association

inv : self.isStereotyped(“Coloured”) implies

self.connection->forAll

(isStereotyped(“Coloured”)

implies color=self.color)

THEORETICAL BACKGROUND

19

language that's targeted to a particular kind of problem, rather than a general purpose

language that's aimed at any kind of software problem.” If the wide application areas of

modeling approaches are considered, it can be achieved that DSLs provide better

understanding for different domain problems since each domain has different requirements

and features need to be considered.

2.5 Maturity Model

A maturity model provides a systematic framework to assess how mature the organizations

are developing software products where maturity is defined as “developed fully” by Oxford

Concise Dictionary. Therefore the purpose of a maturity model can be described as

identifying the capabilities and approaches of organization that will allow them to manage

their different processes like personnel competence, test, software development processes

more reliable and successful. There exists several maturity models proposed for different

issues related to software development process in software industry. For instance, there are

maturity models developed by concerning the software process quality like CMM (Capability

Maturity Model), CMMI (Capability Maturity Model Integration)6, SPICE

7 (Software Process

Improvement and Capability Determination), and so on. In software industry, there also exists

maturity models developed for assessing the maturity level of project management, test

processes, architecture, personnel competence, and so on. All these maturity frameworks have

common issues as described by Information Technology and Telecommunications SIG8 as

following;

They describe a number of discrete stages, or levels where each level represents a

progression of overall maturity of the organization. The CMM, and many other models,

are comprised of five levels, from an ad hoc, non-repeatable process (level 1) to an

exceedingly mature, robust and tightly structured capability (level 5).

They are comprised of a number of capability areas.

They explore a set of practices that together collectively define how the overall discipline

They tend to be organizationally focused.

While not processes or process frameworks, they are prescriptive of the processes that an

organization should adopt.

In summary, they are developed to assess the maturity level of organizations on specific

tasks.

5 http://martinfowler.com/

6 http://www.sei.cmu.edu/cmmi/start/faq/related-faq.cfm

7 http://www.isospice.com/categories/ISO{47}IEC-15504-Standard/

8 http://pmi-ittelecom.org/pmtopics/maturity-models-a-framework-for-organizational-improvement/

METHODOLOGY

20

3.METHODOLOGY

A systematic and planned research support finding and getting reliable answers. By keeping

the academic work structured and systematic, the readers will be able to understand the whole

work easier. Moreover, the reliability of the results will be achieved.

3.1 Data Sources

Data sources are the carriers of data (information) [16]. There exist two types of data sources

which are secondary data and primary data. Secondary data are information collected by

others for purposes that can be different from ours. Primary data are original data collected by

us for the research problem at hand [16].

In this thesis, secondary data collection is performed since the sources of primary data like

communication, experiments, and observations are not performed because of time limitations.

However, secondary data sources provided and supported essential and sufficient data to get

the reliable results.

MBSE is a new concept in software engineering. Therefore knowledge about its features,

characteristics, and how it is performed in the industry were required in order to understand

the whole concept. Moreover, since this thesis‟s focus is on UML diagrams usage in MBSE

and UML is known as its complexity because of the multi diagrammatic architecture, UML

by its variety of characteristics and usage fields that need to be examined properly. Therefore

these two knowledge requirement leaded to perform a research by the process of reviewing

literature from previous researches and books published. In order to get these publications,

reliable secondary data sources such as databases exist in Jönköping University Library are

scanned by some keywords. Therefore many publications like e-books and books, academic

articles, course notes, case studies, and experiment reports are scanned. Moreover since UML

is proposed by OMG, the OMG web-site was also another secondary data source that

examined frequently.

Any experiments or interviews are not handled during thesis but existing experiments and

case studies that are performed by others were interest of the this thesis in order to get

answers for the research questions defined in section 1.4.

All these findings are analyzed regarding on mostly research questions so that efficient and

sufficient results are achieved. For instance, the model quality goals and evaluation of them

supported to determine the best practices at different levels of abstraction.

METHODOLOGY

21

3.2 Data Collection

Data sources that are performed in this thesis were secondary data. Therefore qualitative

methods are performed. In qualitative research, the findings are not arrived at by statistical

methods or other procedures of quantification [16]. Even though many researchers believes

that quantitative research is more suitable or scientific; there are some says that methods

and/or techniques can be more scientific and better, it just depends on what the researchers are

looking for and how they will be able to get results for their research questions. Therefore as

long as everything is performed in a systematic way, the results will be sufficient and reliable.

3.3 Data Analysis

In order to analyze the collected data, content analysis method is applied during this process.

Klaus Krippendorff [40] defines content analysis as “an empirically grounded method,

exploratory in process, predictive or inferential in intent.” Also Robert P. Weber [41] states

that “content analysis is a research method that uses a set of procedures to make valid

inferences from text”. The most essential step of qualitative content analysis approach is to

make categorization. In [42] the author briefly indicates that categories are in the center of

analysis and “the aspects of text interpretation, following the research questions, are putted

into categories, which were carefully founded and revised within the process of analysis

[42]”.

In this thesis, the aim is to be able to define the most essential aspects regarding the

MBSE/UML principles within the organization. Therefore, measuring the occurrence of these

factors in the literature (conferences, workshops, experience reports, empirical studies, and

case studies) will provide to make determination of these aspects to be evaluated within the

proposed MBSE/UML Maturity Model.

In order to achieve this, the process shown is Figure 10 is followed.

METHODOLOGY

22

Figure 10 Followed Data Analysis Steps

3.4 Maturity Model Development Method

While developing a maturity model, seven guidelines of [51] is considered as basis in order to

be able develop a reasonable and usable maturity model. Design science addresses research

through the building and evaluation of artifacts designed to meet the identified business need

[51]. Design science aims at the improvement of problem-solving capabilities by creating

innovative artifacts, such as constructs, models, methods and instantiations [52]. Therefore,

developed maturity models require to be considered as innovative artifacts while assessing the

proficiency of organizations‟ model based development process. While adopting these

guidelines in this thesis, some changes are done since this maturity model development

process is supported by some research questions and literature review. The Table 4 below

indicates the description of these seven guidelines that are adapted from [51] and descriptions

of them in this thesis work.

METHODOLOGY

23

Figure 11 Maturity Model Development Steps

Results from the research questions and literature review conducted some of the steps of this

design-science research. As mentioned in previous section, after the determination of aspects,

organization levels, and indicators for model based development process. The maturity levels

are conducted with respect to the organizations‟ aim on model based development. This

process is shown in Figure 11.

Table 4 Design-Science Research Guideline (quoted from [51])

Guideline Description in General Description in This

Thesis

Guideline 1: Design as an

Artifact

Design-science research must

produce a viable artifact in the

form of a construct, a model, a

method, or an instantiation

A maturity model is

developed as a viable

artifact that provides an

assessment of

proficiency level of an

METHODOLOGY

24

[51]. organization on model

based development.

Guideline 2: Problem Relevance The solutions should be

provided for essential and

relevant business problems.

Essential aspects

regarding the model

based development are

retrieved from literature

review by underlying the

occurrence of the aspects

in literature.

Guideline 3: Design Evaluation The utility, quality, and

efficacy of a design artifact

must be rigorously

demonstrated via well-

executed evaluation methods

[51].

The results are required

to be carried out with

scientific methods.

Guideline 4: Research

Contributions

Relevant contributions require

to be performed in the design

areas.

Existing maturity models

relevant with model

based development

require to be examined.

Guideline 5: Research Rigor The selected methods require

to be adjusted rigorously.

The selected and used

methods require being

well-defined.

Guideline 6: Design as a Search

Process

Solutions that are proposed

should be developed,

evaluated, and improved

iteratively.

Maturity Model requires

to be developed step by

step.

Guideline 7: Communication of

Research

Design-science research must

be presented effectively both to

technology-oriented as well as

A usability analysis after

the development of

maturity model requires

METHODOLOGY

25

management-oriented

audiences [51].

to be performed.

3.5 Quality of Science

The method used in this thesis research is qualitative. Therefore, the reliability and validity of

data collected and analyzed are more difficult than the quantitative research. However,

performing a systematic review covers this problem and provides reliable and valid data. As

described in previous section, identifying what exactly are we looking for from the answers of

research questions makes the research question more structured. Therefore, the results

gathered are certain. Then the measuring the occurrence of these data in the literatures ensures

the reliability of the data collected since by applying content analysis, sort of quantitative data

are collected from the qualitative research and at that point, the influence of case studies and

experience reports support that data collected are based on some experiences and real-time

issues.

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

26

4.EVALUATION OF MBSE/UML

APPROACHES AND DEFINITION OF

ASPECTS

4.1 Quality of UML Diagrams in Modeling

Today‟s software systems are complex so modeling is considered to be performed more than

sketches and blueprints and at that point UML‟s multi diagram architecture is also causing a

complexity problems since some of the diagrams and semantic are not described properly.

Therefore it is both complexes to model a large scale system and to use UML in this modeling

process.

In this thesis, we are looking at both UML and modeling concurrently so that the lacks that

will be stated here will be based on using UML as a modeling language in modeling process.

This is also interest of many researchers. However, the variety use of modeling approaches

such as UML as blueprints, sketches, and programming language leads researchers to

examine the UML quality in different modeling approaches.

As it is also discussed and mentioned in articles [1], [17], [18], [5], [6], [4], [5] UML has lack

of semantics and syntax regarding the transformation of software development artifacts from

models. Therefore possible problems and defects arise in the models. For instance, the lack of

formal syntax causes not to be able automated verification of design specifications. There

exist several industrial experiences and researchers on semi-formal semantic and syntax

structure of UML and some provides solutions to decrease the problems that occur because of

these lacks.

It is not only semi-formal structure of UML leads models to have problems, because of multi

diagrammatic architecture of UML many researchers and practitioners state that this

architecture structure leads inconsistency between diagrams in model. Moreover the

interaction between diagrams also cannot be well described.

In article [1], Parastoo Mohagheghi et al. define 6 classes of model quality goals shown in

Table 5.

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

27

Table 5 6C Model Quality Goals

Model Quality Goal Description

Correctness Including right elements and correct relations between them, and

including correct statements about the domain [1]

Not violating rules and conventions; for example adhering to

language syntax, style rules, naming guidelines or other rules or

conventions [1].

Completeness Completeness is defined as having all the necessary information

that is relevant and being detailed enough according to the

purpose of modeling [1].

Consistency Consistency is defined as no contradictions in the model [1].

Comprehensibility Comprehensibility is defined as being understandable by the

intended users; either human users or tools [1].

Confinement Confinement is defined as being in agreement with the purpose of

modeling and the type of system; such as including relevant

diagrams and being at the right abstraction level. A model is a

description from which detail has been removed intentionally [1].

Changeability Changeability is defined as supporting changes or improvements

so that models can be changed or evolved rapidly and

continuously [1].

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

28

In addition to these, the quality of models also depends on aesthetics, the layout of diagrams,

the design, the tools used and so on.

In this thesis, our aim is to provide a maturity model for evaluating the usage of modeling

approaches so that our focus will be on these model quality goals too. These model quality

goals will provide aspects that need to be considered during the modeling process and then

how to achieve these goals or in which proficiency level an organization is performing these

will be examined.

4.2 Evaluation Criteria for MBSE/UML Maturity Model

In this section, the results for research question 2 “How to evaluate the quality and

proficiency in usage of UML and MBSE principles during software development process?” is

examined by literature review regarding UML new features, standards; and MBSE

approaches. The results gathered provide decision making while determining the important

aspects that require to be evaluated at different levels of organization.

Different aspects are considered at different levels of organization since various aspects not

only one have an impact on whole model based development process. Moreover while

evaluating the proficiency of organization on modeling, these aspects require to be considered

in order to support organizations to make improvements on their modeling practices.

In addition to these, since one of the aims of this thesis is to evaluate various aspects at

different levels of organization, in this section, brief information for levels of organization and

aspects required to be considered for each level provided. Moreover, after determining the

aspects and levels of organization; the indicators that will be evaluated and measured

separately are considered and provided in this section.

4.2.1 Essential Aspects for MBSE/UML Maturity Models

This evaluation criterion is important since MBSE approaches and UML are being used for

different purposes in the industry. Some organizations are only applying these to be able

generate code from the models whereas some are just using it to develop sketches. Therefore,

the aim of performing MBSE approaches in organization will differ and requirements for this

adoption will be different. Just to make this part clearer, a simple example can be provided.

For instance, an organization adopts modeling in order to generate code from the models

developed and the domain of the system is a defense system which requiring high security and

safety. In order to manage this, the models that require to provide some implementation

details and the tools that are used should be certified tools. On the other hand, let assume that

another organization is only aiming to use MBSE approaches as sketches so they should not

provide implementation details in their models; otherwise, their models will not be the ones to

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

29

cover the organization‟s purposes so too much unnecessary work and time consuming will

occur.

During the writing of this thesis, the literature examined provided the necessary and essential

aspects require to be estimated while adopting modeling approaches. According to [19], lack

of process development definitions, lack of quality assurance activities, and the design of

models cause a decrease on organization‟s improvements on models. In addition to these,

based on the case studies performed, they found that the characteristics of the designer

(modeler) like skills, motivation, and training has an impact on the quality of the models.

Moreover, time pressure is also examined as another effect on quality of models.

[43], [44], [45], [37], [4-3], and [19] also mention about importance of well-defined modeling

process. For instance, the authors of [44] states that “Processes must be generic enough to be

shared and capitalized at company or department level, but must be customizable enough to

be relevant at project level. Processes describe roles, activities and work products for each

involved stakeholder.” Moreover in [43] the authors also state that modeling process affects

the quality of MBSE so that while evaluating the proficiency of organization on MBSE, the

modeling process should be defined properly. The importance of utilizing a defined process in

software engineering has been known for several years. However, most “tried and tested”

processes are not tailored for MDE, which does not make any assumptions on the software

development process or the design methodology [20].

In [43], [35], and [19] the effects of modeling language, the knowledge and skills of modelers,

and the applied quality assurance techniques are also mentioned. For instance, based on [43]

the used modeling language should be applicable with domain and should be able to represent

the complexity of the system if it is complex. In article [35], team inexperience on modeling,

modeling language used, and modeling tools used stated as a challenge while adopting MBSE

within the organization.

In addition to these, the article [46] focuses on the importance of transformation and examines

the Motorola‟s experience on MDD. It indicates that the automatic generation of code from

developed models requires UML profiles or DSL (Domain Specific Language).

As mentioned several times in this thesis, UML‟s lack of semantic and syntax structure is

another issue to be estimated. For instance, as indicated in [19], [3], and [14] in order to

present and state the UML diagrams precisely a well-defined syntax and semantic is required.

By defining these, the consistency and comprehensibility will be achieved. However, it is

require to enforce the personnel who develops the models to follow these set of rules.

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

30

Table 6 Number of Occurrences of Aspects in Literature

Essential Aspects Occurrence in Literature

Articles, Journals

(n=31)

Case Studies,

Empirical Studies,

Experience Reports

(n= 21)

Modeling Process 13 11

Quality Assurance

Techniques

11 10

Personnel Competence 7 12

Tools 11 13

Design 13 9

Syntax and Semantic 16 12

Transformation 10 10

In total 52 articles are examined while determining the evaluation aspects for MBSE/UML

Maturity Model, the literature consists of two different parts. One is the articles that are

research papers developed systematically; the others are the empirical studies like surveys,

experience reports, etc. The Table 6 above shows the most occurred aspects regarding the

literature review where UML is considered as a main modeling language in MDD process.

The aspects stated in the Table 6 are same as the aspects that are determined as evaluation

criteria for MBSE/UML Maturity Model. However, it is important to state that during the

literature review if they are mentioned about the UML‟s lack of syntax and semantic, it is

related to aspect named syntax and semantic. Or when Meta-Modeling, DSL, and UML

Profiles are also categorized into syntax and semantic; and some are related to transformation

aspect. This categorization depends on how these concepts are mentioned and discussed. Then

these specific aspects like Meta-Models, DSL, and UML Profiles are utilized as indicators for

each aspect.

Therefore in this thesis, the focus will be on aspects which are modeling process, quality

assurance, personnel competence, tools, design, syntax and semantic, and transformation

shown in Table 7.

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

31

Table 7 Definition of Aspects

Aspects Definition

Modeling Process Modeling Process can be defined as a set of steps that need to be

followed during the development process considering the models

are the main artifacts. These steps can be like planning, requirement

specification, and model design like the ones in traditional software

development process but regarding the models.

Quality Assurance

Techniques

These are the techniques or methods that allow organization to

measure its development practices and evaluate if the organization

goals and model goals are gained or not.

Personnel Competence Here personnel competence is considered as motivation of

participants, the knowledge and experience of them, skills of them.

Moreover if it is necessary, training activities need to be included.

Tools Tools are used in modeling approaches and depending on the

modeling aim of organization different types of tools are used. For

instance, tools for graphical view, tools for automatic code

generation.

Design

In this thesis, the design aspect represents the layout, aesthetics of

UML diagrams and models. Moreover it also indicates the different

models require to be developed such as requirement model,

business model, domain model, and design model.

Syntax and Semantic In this thesis, syntax is considered as regulations, notations, and

rules used in order to describe a modeling language and semantic is

considered as meaning of the expressions stated within the

modeling language.

Transformation In this thesis, transformation means being able to perform

transformations from model to model, model to code, or model to

text.

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

32

4.2.2 Indicators for Each Aspect in MBSE/UML Maturity Model

After defining the aspects, it will be clearer to examine how to evaluate these aspects. Based

on the literature review performed, many researchers are discussing why these aspects are

important to be considered and they only focus on these aspects separately. However, in order

to measure and evaluate the proficiency of organization on modeling approaches all these

aspects need to be evaluated so that organizations will be able to examine how advanced or

un-advanced they are performing modeling approaches.

In this thesis, for each aspect some indicators are defined to be evaluated. These indicators are

the general titles and they consist of evaluation methods, some checklists, questionnaires, or

interviews and so on. These evaluation techniques will be discussed later. Now the focus is on

the main indicators that will be evaluated and will be evaluation criteria which are shown in

Table 10 below.

Modeling Process

Because of lack of standardized model based software development process definition,

organizations need to give importance to define a modeling process before they start adopting

modeling approaches. The traditional software development processes can be considered and

based on them; a new process for model based development can be defined regarding the

main artifact as models. The defined modeling process should be applicable to the field that

modeling will be performed. The standards, regulations need to be defined in order to avoid

from inconsistency and controversy that can occur later in the development process.

Moreover, the defined model based development process need to consider the modeling goals

so that it is required to define the modeling goals and document them.

In addition to these, it is important for organization to support and motivate staff to document

everything regarding the standards so that the changes and/or regulations can be tracked and

ordered properly.

Personnel in the model based development process regarding their knowledge, skills, and

experiments need to be evaluated since they are the ones who will develop the models and

final product. Moreover, personnel should agree on the defined modeling process and feel

confident with that. Therefore, it would provide an advantage if the personnel is also involved

in defining the model development process.

The modeling goals, the modeling process, and skills of personnel are considered but it is also

necessary to define quality assurance techniques to be able evaluate the results. This will be

examined with the quality assurance technique aspect. However, while defining the modeling

process and modeling goals, quality assurance techniques also need to be considered.

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

33

Model based development processes include many actors like stakeholders and personnel

from different departments of organization and with different roles. Stakeholders can be both

external and internal as shown in Figure 12.

Figure 12 A Sample Declaration of Internal and External Stakeholder

Each actor within the development process and organization is responsible of sharing

knowledge with others and each has to consider the organization standards, regulations, and

goals. For instance, each modeler can be responsible for different models but which are

supplementary for others too. Therefore, each modeler should know each other‟s work and

should provide consistency interaction and communication with each other. Let assume that a

modeler is responsible for providing a use case diagram to show how the general project will

work and this use case is based on the requirements of the external stakeholder who is client

in this scenario. Let also assume that this sample scenario simply indicates how an ATM

machine works as shown in Figure 13.

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

34

Figure 13 A Sample ATM UML Use Case Diagram

Regarding all these issues that need to be evaluated the indicators that will be used while

evaluating the proficiency level of modeling process aspect in adopting modeling approaches

can be listed like;

Defined Modeling Process,

Organization Workflow Schema,

Documentation,

Traceability,

Communication and Interaction Facilities,

Confidence of Personnel, and

Applicability to Domain.

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

35

Quality Assurance Techniques

If organizations intend to adopt modeling approaches in a proficient level, performing the

quality assurance techniques becomes a must since they are the ones which enable

organizations to evaluate themselves and their works.

The main issue is to evaluate if the organization goals and model goals are fulfilled or not so

that by referring the defined model based software development process and defined goals, a

proper documentation of them and a defined quality assurance plan are necessary. For

instance, a quality assurance plan should consist of goals definition, test plans, training

evaluations, verification and validation, and so on. For instance, quality assurance techniques

should evaluate and validate the 6C model goals [1] that mentioned previously in section 4.1.

In addition to these, quality standards need to be applied so it is necessary to enforce

participants involved in model development process to obey quality standards.

Therefore regarding these the indicators for this aspect are;

Defined Quality Assurance Plan,

Audits and Reviews,

Defined Quality Management Plan,

Defined Test Plan for Model Quality Goals, and

Domain Testability.

Personnel Competence

Personnel competence is another aspect that needs to be considered and evaluated since the

motivation and knowledge of participants have a reasonable effect on the whole development

process. The participants are the ones who develop the models and the final product so that

their affect cannot be underestimated. Therefore it has to be examined that how the staff have

a knowledge about modeling, the tools that will be used, and/or domain. For instance, it is

necessary to know if a modeler has experienced with the modeling tool that now will be used

in model development process or what kind of related certificates and/or participations to

related seminars and/or courses a modeler attended and had can provide knowledge about

training of him. Moreover the knowledge requirement does not only have to be about

modeling approaches, depending on the requirements for the model based development it is

possible to require knowledge on domain.

Regarding the aspects like quality assurance techniques and modeling process, the motivation,

documentation facilities, and communication facilities of personnel need to be considered.

The indicators for Quality Assurance Techniques Aspect are;

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

36

Defined Quality Assurance Plan

Defined Quality Management Plan

Audits and Reviews

Defined Test Plan for Model Quality Goals

Confidence and Compatibility of Personnel in Defined Plans

Domain Testability

Tools

There exist several tools for modeling activities which can be model editors, model

simulators, model executors, transformers, and so on. Regarding the modeling aim, the tool

selection is a critical decision making activity in MBSE. Therefore it is necessary for

organization to evaluate the existing tools including the ones that they already use within the

organization. For instance, if organization intends to transform code from the models then

tools like transformers, transformation editors are needed to be used. Moreover the policy of

organization is also another affect while using and making a decision on using some specific

tools. Therefore the indicators which are;

Policy for Use,

Integration Facility,

Simulation Ability of Use,

Applicability of Domain (certified or secure tools),

Tools Evaluation Facility, and

Repository and Tool Reuse need to be considered.

Design

Even though, most of the requirements for layout and aesthetic rules are required to be

declared by syntax and semantic there are still some other issues that affect the design of

UML diagrams. For instance, as mentioned in [22] and [19] the crossings and bends, size, and

colors are the ones to be estimated during the development of UML diagrams in order to

provide better and comprehensible view of diagrams. As shown in Figure 14, the crossings

and bends have a reasonable effect on comprehensibility of developed UML diagrams. The

most preferable one in this figure is (b) since the view of diagram is not complicated to

understand.

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

37

Figure 14 Effects of Crossing and Bends [22]

Moreover in order to improve the visualization of UML diagrams some perceptual concepts

are required to be performed which are defined in Table 8. These layout and aesthetic rules

are related to semantic structure of UML so that the results from the other aspect “Syntax and

Semantic” that will be presented later will be also indicators for design aspect.

Table 8 Definition of Visualization Principles

The Visualization Principles Definition

Law of Good Figure In this thesis, law of good figure refers to simplicity of UML

diagrams where only necessary information is represented.

Law of Similarity Similar elements (e.g., common shape or color) appear to be

grouped together [22]. In this thesis it refers to same

meaning too.

Law of Continuation As stated in [22] the belonging elements of UML diagrams

should be grouped together.

Law of Connectedness In this thesis, law of connectedness refers to that lines that

connect the belonging UML elements are required to be

connected to each other properly to increase the visualization

of them and support the group of elements.

Law of Familiarity In this thesis it also refers as stated in [22] where law of

familiarity is defined as elements are more likely to be

grouped together if the groups seem familiar or meaningful.

In addition to these, in this thesis design is not considered as layout and aesthetics of UML

diagrams but also as representing the different goals of models where the modeling goals

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

38

should be defined in development of modeling process and quality assurance techniques. In

this thesis, 4 different models will be taken into account which is requirement models, domain

models, design models, and business models where the definitions of them provided in Table

9.

Table 9 Definitions of Different Models in MDD

Different Models Definition

Requirement

Models

The requirement models are the models that are developed in order

to represent the requirements of clients. Therefore the main UML

diagrams that will be developed are use cases.

Business Models Business models represent the business requirements and

information about system platform and technology are not be

shown.

Design Models Design Models are developed in order to provide detailed

information about the system and product that will be developed.

Both functional and non-functional requirements are shown. For

instance, if organization aims to execute code from the models then

in design models these execution requirements and specifications

should be provided. Example UML diagrams are class diagrams,

sequence diagrams, activity diagrams.

Domain Models Domain Model is the model that defines how a business works

without reference to software systems, similarly to OMG‟s

Computation Independent Model [2].

The indicators that will be taken into consideration are;

Law of good figure [22]

Law of similarity [22]

Law of continuation [22]

Law of connectedness [22]

Law of familiarity [22]

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

39

Development of Semantic

Development of Requirement Model

Development of Business Model

Development of Domain Model

Development of Design Model

Skills and Experience

Moreover by indicating these different models, the aim is that each model is responsible to

show another goal and if design model is not related to requirement model then the model

quality cannot be achieved. Moreover if automatic code generation is required then design

model should satisfy all necessary requirements, system and technological information related

to that. That is why, these are considered as indicators.

Syntax and Semantic

Because of lack of definition and regulations on syntax and semantic structure of UML, this

aspect has a great impact on adopting modeling languages. Therefore organizations should

declare syntax and semantic regulations and rules clearly and properly. That will provide

consistency between each model that developed by different modelers. Therefore this

declaration should be understood by personnel and personnel need to be enforced to obey

these regulations. Moreover while organization is defining the syntax and semantic structure,

it is essential to consider the support of modeling language and modeling tool since it is

possible that a modeling tool cannot allow any other notation declared. Therefore the defined

syntax and semantic regulations should be applicable by modeling tools.

Like the other aspects, syntax and semantic evaluation also depend on some results from the

other aspects. For instance, the UML knowledge of personnel which take roles on

development process needed to be examined.

The indicators for this aspect are;

Consistency with Modeling Goals,

Syntax and Semantic Definition,

Modeling Language Support,

Personnel Knowledge and Skills, and

Applicability to Tools

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

40

Transformation

Regarding the definition of transformation, organizations can be willing to perform some

transformation based on their modeling and organization goals. With respect to the aim of

organization and/or modeling approaches, the developed models need to fulfill these

requirements. For instance, if the aim is to be able to generate code automatically in a specific

language then the specifications for this programming language should be indicated in the

models. Moreover, the knowledge and skills of personnel also have an impact on this aspect

too. For instance, in order to generate code automatically from the models UML profiles can

be required to be developed. Therefore the developed models need to fulfill the requirements

for the transformation activities.

The indicators for transformation aspect are;

Automatic Code Generation Requirements [20],

Executable Model [20],

Model Reusability, Changeability,

Generation of Domain Specific Artifacts[20],

Model Representation of Detailed System Specifications, and

Knowledge and Experience of Personnel on Transformation and Transformation

Tools,

The Table 10 below briefly demonstrates the indicators for each aspect.

Table 10 Indicators for Each Aspect

Aspects Indicators

Modeling

Process

Defined Modeling Process [43], [44], [45], [37], [20],

and [19],

Organization Workflow Schema,

Communication and Interaction Facilities [45],

Confidence of Personnel [44], and

Applicability to Domain [43].

Quality

Assurance

Defined Quality Assurance Plan [19], [35]

Audits and Reviews,

Defined Quality Management Plan[35],

Defined Test Plan for Model Quality Goals, and

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

41

Techniques Domain Testability.

Confidence and Compatibility of Personnel in Defined

Plans

Personnel

Competence

Training Certificates and/or Knowledge Level,

Project Skills Requirement Definition

Motivation,

Skills (Tools, Modeling, Domain) [43], [35]

Defined Training Plan,

Domain Knowledge and Skills[43],,

Domain Skills Requirement Definition.

Tools

Policy for Use,

Integration Facility [44], [35], [2]

Simulation Ability of Use [2],

Applicability of Domain (certified or secure tools),

Tools Evaluation Facility,

Repository and Tool Reuse.

Knowledge, experience of Personnel on Tools

Design

Law of good figure [22]

Law of similarity [22]

Law of continuation [22]

Law of proximity [22]

Law of connectedness [22]

Law of familiarity [22]

Development of Semantic

Development of Requirement Model

Development of Business Model

Development of Domain Model

Development of Design Model

Skills and Experience

Syntax and

Semantic

Modeling Language Support,

Formal Language to define syntax and semantic (ex:

meta model [3-24])

Syntax and Semantic Definition[19], [3], and [14],

Personnel Knowledge and Skills,

Applicability to Tools [43],

Consistency with Modeling Goals.

Transformation

Automatic Code Generation Requirements [20]

Executable Model [20]

Model Reusability, Changeability,

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

42

Generation of Domain Specific Artifacts[20],

Model Representation of Detailed System

Specifications.

Knowledge and Experience of Personnel on

Transformation and Transformation Tools,

4.2.3 Different Levels of Organization for MBSE/UML Maturity Model

As mentioned several time in this thesis, depending on the different levels of organization, the

requirements, the activities and practices, and the responsibilities require to be considered and

implemented change. For instance, organizations are responsible for managing and supporting

the modeling approaches as whole. The focus for them is fulfilling the organizational goals,

enforcing the obeying the organizational standards, and some other issues like these.

Regarding the knowledge on the existing maturity models and framework, CMM is developed

for at the organizational level where PCMM developed for at personnel level considering both

individual and team activities and practices. Based on them, the different levels of

organization are determined for this thesis and it is considered essential to examine the whole

activities and practices at different levels of organization which are;

Organizational Level,

Project Level,

Personnel Level, and

Domain Level.

The Table 11 below briefly indicates these levels of organization with a brief description.

Table 11 Levels of Organization

Levels of Organization Definition

Organizational

Management

Support

Organizational Management and Support activities and

practices performed at this level in order to provide and support

organizational change and improvement.

Project At the level of project, a specific project is considered. Based

on that project, the relative activities and practices need to be

performed for advance proficiency in modeling approaches are

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

43

considered.

Personnel

Engineering

Management

Modeler

Stakeholder

Team/ Group

Modeling activities where personnel are involved and perform

are basics for this level of organization. Their relationships,

training, skills, tasks, and so on are the issues that require to be

considered.

Domain

Business and

Financial

Health Care

Communication

Defense

IT

At the level of domain, different domains are considered since

regarding the each project that will be developed different

domains are point at issue. Moreover requirements, structure,

and specifications for each domain differ so that it affects the

whole model development process.

4.2.4 The General Matrix for MBSE/UML Maturity Model

There exist 7 different aspects need to be considered at 4 different levels of organization as

discussed previously and the Table 12 below demonstrates all the aspects that require to be

considered at different levels of organization regarding the indicators that require to be

evaluated.

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

44

Table 12 General Matrix for MBSE/UML Proficiency Framework

Aspects

Modeling Process Quality Assurance

Techniques

Personnel

Competence

Tools Design

Syntax and Semantic Transformation

Levels

Organizational

Management

Support

Defined Modeling

Process,

Organization

Workflow Schema,

Defined Quality

Assurance Plan,

Audits and

Reviews,

Defined Quality

Management Plan,

Defined Training

Plan

Policy for Use,

Tools

Evaluation

Facility,

Repository and

Tool Reuse

Development of

Business Model

Modeling Language

Support,

Syntax and Semantic

Definition,

Applicability to Tools

Transformation Goal

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

45

Project Defined Project/Model

Goals

Defined Test Plan

for Model Quality

Goals

Project Skills

Requirement

Definition

Integration

Facility,

Simulation

Ability of Use

Development of

Requirement

Model

Development of

Design Model

Law of good

figure [22]

Law of

similarity [22]

Law of

continuation

[22]

Law of

connectedness

[22]

Consistency with

Modeling Goals

Automatic Code

Generation

Requirements, [20]

Model

Representation of

Detailed System

Specifications,

Model Reusability,

Changeability

Executable Model

[20]

EVALUATION OF MBSE/UML APPROACHES AND DEFINITION OF

ASPECTS

46

Law of

familiarity [22]

Development of

Semantic

Personnel

Engineering

Management

Modeler

Stakeholder

Team/ Group

Communication and

Interaction Facilities,

Confidence of

Personnel

Confidence and

Compatibility of

Personnel in

Defined Plans

Training

Certificates and/or

Knowledge Level

Motivation,

Skills,

Domain

Knowledge and

Skills

Knowledge,

experience of

Personnel on

Tools

Skills and

Experience

Personnel Knowledge

and Skills

Knowledge and

Experience of

Personnel on

Transformation and

Transformation

Tools,

Domain

Business and

Financial

Health Care

Communication

Defense

IT

Applicability to

Domain

Domain Testability Domain Skills

Requirement

Definition

Applicability of

Domain

(certified or

secure tools)

Development of

Domain Model

Generation of

Domain Specific

Artifacts [20]

MBSE/UML Maturity Model

47

5.MBSE/UML Maturity Model

5.1 MBSE/UML Maturity Model Structure and Levels

The structure of the proposed MBSE/UML Maturity Model (shown in Figure 15) where [47]

and [48] are adapted while developing the maturity model, is built upon regarding different

indications listed as following;

The assessment of essential aspects during MDD process,

The assessment of indicators for each aspect at different levels of organization.

Figure 15 The Structure of MBSE/UML Maturity Model (adapted from [47] and [48])

As mentioned previously, the determination of different aspects provide a general evaluation

criteria to measure how mature organizations are adapting modeling approaches. Therefore

each aspect is considered while determining the maturity levels of MBSE/UML Maturity

Model. Regarding this, the proposed maturity levels will contain these aspects as general

requirements to be performed. However, they will not be the only issue to be considered. The

MBSE/UML Maturity Model

48

indicators for each aspect at different levels of organization will be the ones that the results

for maturity dimension are measured and evaluated. Basically, in addition to the aspects, the

indicators will assess the determination to which maturity level the organizations belong.

Briefly, the MBSE/UML Maturity Level of organization will be assessed regarding these

following issues;

If the aspects assigned for different maturity levels exist or not,

If the results from the indicators evaluations comply the appropriate outcomes

corresponding with the assigned maturity level.

5.2 MBSE/UML Maturity Level Dimension

While determining the maturity levels of MBSE/UML Maturity Model [2], [48], and the

results from literature review which mostly the issues discussed by MDD experience reports,

surveys, and case studies are adapted. It is quite necessary to underline that only while

building up the maturity level of MBSE/UML Maturity Model, existing maturity models for

traditional software development process and MDD process are considered in order to provide

a familiar model for organizations. As mentioned in purpose section in this thesis, different

than the existing maturity models, here the discussion is on MDD process not traditional

software development process; moreover, several aspects at different levels of organization

are the main consideration of this MBSE/UML Maturity Model where the most existing

models are only considering one or few aspects at one organizational level.

After clearing the difference of this proposed maturity model structure, now the maturity level

dimension can be discussed properly.

In this thesis, six MBSE/UML Maturity Model Levels are defined as shown in Figure 16.

MBSE/UML Maturity Model

49

Figure 16 MBSE/UML Maturity Model Levels

5.2.1 Maturity Level 0: No Modeling

No modeling is applied within the organization. Organization is still performing traditional

software development. None of the indicators are applied within the organization with respect

to MBSE/UML approaches.

5.2.2 Maturity Level 1: Basic Modeling

MBSE/UML approaches are not applied within the organization. However, individuals are

developing models for their own help or in order to facilitate communication and interaction

between different stakeholders. In this maturity level, models are developed as sketches so

that MDD‟s main principle which is developing models as main artifacts in software

MBSE/UML Maturity Model

50

development process is not achieved. Therefore organization is still performing traditional

software development process.

5.2.3 Maturity Level 2: Initial MBSE/UML

MDD principles are applied within the organization. However, this is performed at project

level not at organizational level. A modeling process is defined including modeling goals

definition, some quality assurance techniques description but application of them is still not

mature enough. Organization considers their personnel knowledge and skills on MBSE/UML.

However, there are no defined training activities to support personnel‟s level of knowledge on

MBSE/UML. Moreover by development of design models, organization is capable of

implementing the code from the models. However, the code implementation is performed

manually.

The indicators for different aspects at different level of organization in MBSE/UML Maturity

Model Level 2 are shown in Table 13.

Table 13 Aspects and Indicators Assessed in MBSE/UML Maturity Level 2

Aspects Indicators

Modeling Process

Defined Modeling Process

Defined Project/Model Goals

Communication and Interaction Facilities

Quality Assurance Techniques

Defined Quality Assurance Plan

Defined Quality Management Plan

Personnel Competence

Project Skills Requirement Definition

Motivation

Skills

Tools

Tools Evaluation Facility

Repository and Tool Reuse

Design

Development of Requirements Model

Development of Design Models

Law of good figure

Law of similarity

Law of continuation

Law of connectedness

Law of familiarity

MBSE/UML Maturity Model

51

Syntax and Semantic

Consistency with Modeling Goals

Transformation

Model Representation of Detailed System

Specifications

5.2.4 Maturity Level 3: Defined MBSE/UML

Organizations are more familiar with MBSE/UML principles and adapting modeling

approaches at organizational level as well as at project level. In this level, organizations

develop business models where business requirements are represented (see 4.2.2 for

description of business models). After the development of business and design models,

organization is capable of generating code from the models automatically. However, not all

code generation can be achieved automatically. In order to provide a common understanding

and interpreting of models, meta-modeling is performed. Moreover UML Profiles are

developed or organizations use the proposed UML Profiles by OMG in order to enable the

auto-code generation. In addition to these, organizations are considering the personnel

competence in MBSE/UML also.

The indicators that assess the maturity of organization on MBSE/UML Maturity Model Level

3 are shown in Table 14 below. However, the indicators stated previously in maturity level 2

are also required to be assessed.

Table 14 Aspects and Indicators Assessed in MBSE/UML Maturity Level 3

Aspects Indicators

Modeling Process

(All the indicators stated in Maturity Level

2)

Quality Assurance Techniques

Audits and Reviews

Defined Test Plan for Model Quality Goals

Confidence and Compatibility of

Personnel in Defined Plans

Personnel Competence

Defined Training Plan

Training Certificates and/or Knowledge

Level

MBSE/UML Maturity Model

52

Tools

Knowledge and Experience of Personnel

on Tools

Policy for Use

Design

Development of Business Models

Skills and Experience

Syntax and Semantic

Syntax and Semantic Definition

Modeling Language Support

Personnel Knowledge and Skills

Applicability to Tools

Transformation

Transformation Goal

Automatic Code Generation Requirements

Knowledge and Experience of Personnel

on Transformation and Transformation

Tools

5.2.5 Maturity Level 4: Integrated MBSE/UML

Organizations have well-defined meta-models in order to perform generation of models, code,

and/or text from developed models automatically. Domain models are also developed to

represent the specific requirements and concepts of particular domain. The consistency

between different models is achieved. Transformation of development artifacts with respect to

MBSE/UML principles is performed. DSL are utilized and defined.

The indicators that assess the maturity of organization on MBSE/UML Maturity Model Level

4 are shown in Table 15 below. However, the indicators stated previously in maturity level 3

are also required to be assessed.

Table 15 Aspects and Indicators Assessed in MBSE/UML Maturity Level 4

Aspects Indicators

Modeling Process

Organization Workflow Schema

Applicability to Domain

Quality Assurance Techniques

Domain Testability

MBSE/UML Maturity Model

53

Personnel Competence

Domain Knowledge and Skills

Domain Skills Requirement Definition

Tools

Applicability to Domain (certified or secure

tools)

Integration Facility

Simulation Ability of Use

Design

Development of Domain Models

Syntax and Semantic

(All the indicators stated in Maturity Level

3)

Transformation

Generation of Domain Specific Artifacts

5.2.6 Maturity Level 5: Optimized MBSE/UML

Briefly, in addition to the tasks, activities performed on MBSE/UML Maturity Model Level 4,

the developed models are the models that can be executed and reused. Therefore there is a

strict differentiation between business, domain, and design models. To provide this

distinction, some of these models are developed regarding the particular domain and project

whereas some are developed to represent the general structure of the system so that they can

be reused later. In this maturity level, the MBSE/UML aim of organization is completely to be

able to execute the models for final code. Therefore, the main artifact in the software

development process is developing the models as solutions.

In this maturity level, in addition to the indicators stated previously, for transformation aspect

organizations are capable of performing models as reusable and executable as listed in Table

16 below.

Table 16 Aspects and Indicators Assessed in MBSE/UML Maturity Level 5

Aspects Indicators

Transformation

Model Reusability, Changeability

Executable Models

Assessment of Maturity Levels of MBSE/UML Maturity Model

54

6.Assessment of Maturity Levels of

MBSE/UML Maturity Model

In previous chapter, the maturity levels of MBSE/UML Maturity Model are described briefly.

As mentioned there, the indicators are the assessment keys while stating the maturity level of

organization. After organizations start MBSE/UML development, the indicators stated for

each level requires to be achieved in different scales which mean an indicator for one level

does not have to be performed in advanced way. However for maturity levels 4 and 5 all

indicators are required to be resulted advanced.

The instruments are conducted in order to evaluate the indicators that require to be performed

within the organization regarding MDD process. Different questionnaires are designed as

instruments.

The relations of instruments with MBSE/UML Maturity Model‟s aspects and participants who

are intended to answer the questionnaires are provided in Table 17 and the questionnaires are

stated in Appendix part of this thesis.

Table 17 Instruments’ Focus of Aspect and Participants

Aspects Instruments Intended Participants

Modeling Process MBSE/UML Modeling Process

Questionnaire (see Appendix 1)

Project Manager and

Quality Manager

Quality Assurance Techniques MBSE/UML Quality Assurance

Technique Questionnaire (see

Appendix 2)

Quality Manager

Personnel Competence MBSE/UML Personnel

Competence Questionnaire 1

(see Appendix 3)

Developers (designer,

software engineer,

modeler, etc.)

Assessment of Maturity Levels of MBSE/UML Maturity Model

55

MBSE/UML Personnel

Competence Questionnaire 2

(see Appendix 4)

Project Manager and

Quality Manager

Tools MBSE/UML Tool

Questionnaire (see Appendix 5)

Developers, Project

Manager, and Quality

Manager

Design MBSE/UML Design

Questionnaire (see Appendix 6)

Developers, Quality

Manager

Syntax and Semantic MBSE/UML Syntax and

Semantic Questionnaire (see

Appendix 7)

Developer, Quality

Manager

Transformation MBSE/UML Transformation

Questionnaire (see Appendix 8)

Developer

As mentioned previously, 6 maturity levels are proposed and each maturity level requires

being performed different aspects with different indicators at different levels of organization.

Therefore, for each aspect, different questionnaires are conducted to assess if the indicators

are performed or not.

In total 85 questions are designed to be asked. The Table 18 below indicates the total of

questions designed for different indicators. Moreover, it should be noted that some of the

questions that are asked in different aspects are also related to the other aspects. For instance,

while evaluating if the personnel skills are qualified for syntax and semantic aspect, this also

results if the organization is aware of their personnel knowledge and skills which is supposed

to be performed in personnel competence aspect. Therefore for some indicators and aspects,

the results can be also gathered from different questions.

Some of the questions that are designed to be asked are;

Assessment of Maturity Levels of MBSE/UML Maturity Model

56

Do you assure that the granularity of your developed models is sufficient enough to

represent the details of a particular system?

Do you use DSL (Domain Specific Languages) within your MDD process?

Do you separate the domain specific concepts in your domain model from your

business and design model?

Table 18 Number of Questions Designed to be asked for Each Indicators within Different

Aspects

Aspects Indicators Total

Number of

Questions

Asked

Modeling Process

Defined Modeling Process,

5

Organization Workflow Schema,

1

Defined Project/Model Goals,

7

Communication and Interaction

Facilities,

Confidence of Personnel,

8

Applicability to Domain,

2

Documentation,

6

Traceability

2

Quality Assurance

Defined Quality Assurance Plan,

11

Assessment of Maturity Levels of MBSE/UML Maturity Model

57

Techniques Audits and Reviews,

4

Defined Quality Management

Plan 6

Defined Test Plan

5

Confidence and Compatibility of

Personnel in Defined Plans 1

Domain Testability

2

Personnel

Competence

Defined Training Plan,

10

Project Skills Requirement

Definition 1

Training Certificates and/or

Knowledge Level , 1

Motivation,

4

Skills,

6

Domain Knowledge and Skills

3

Domain Knowledge

Requirements Definition 2

Tools

Policy for Use,

5

Assessment of Maturity Levels of MBSE/UML Maturity Model

58

Tools Evaluation Facility,

3

Repository and Tool Reuse,

2

Integration Facility,

2

Simulation Ability of Use,

1

Knowledge and Experience of

Personnel on Tools, 5

Applicability to Domain (Ex.

certified or secure tools) 3

Design

Development of Business Model,

4

Development of Requirement

Model, 5

Development of Design Model,

5

Law of Good Figure,

2

Law of Similarity,

2

Law of Continuation,

2

Law of Connectedness,

2

Assessment of Maturity Levels of MBSE/UML Maturity Model

59

Law of Familiarity,

2

Development Model for

Semantic, 1

Development of Domain Model

5

Meta-Model

5

OCL

1

Syntax and

Semantic

Modeling Language Support,

4

Syntax and Semantic Definition,

5

Applicability to Tools,

2

Consistency with Modeling

Goals, 2

Personnel Knowledge and Skills

3

Transformation

Transformation Goal,

7

Automatic Code Generation

Requirements, 3

Model Representation of Detailed

System Specifications, 2

Assessment of Maturity Levels of MBSE/UML Maturity Model

60

Model Reusability, Changeability,

7

Executable Models,

5

Generation of Domain Specific

Artifacts 2

Most of the answers for questions are assigned with numerical values from 0 to 4. A sample is

shown in Table 19 below.

Table 19 Sample of Assigned Numerical Values for Answer Types

Answer Type 1 Answer Type 2 Assigned

Value

Definitely Yes Complete Support 4

Usually Partly Support 3

Planned but not applied Moderate Support 2

Not Sure Poorly Support 1

Definitely No Not at all 0

Assessment of Maturity Levels of MBSE/UML Maturity Model

61

Moreover, some multiple choice questions are designed in order to examine which of the

practices the organization is performing; then regarding this questions with answers shown

above are asked to measure how mature they are performing those practices. In addition to

these, the answers for some particular questions consist of scales and these scales are also

assigned to numerical values from 0 to 4 to support easy to calculate as shown in Table 20

below.

Table 20 Sample of Assigned Values for Questions

Answer Type Assigned

Value

Between 0% and 20% 4

Between 20% and 40% 3

Between 40% and 60% 2

Between 60% and 80% 1

Between 80% and 100% 0

The evaluation of these answers will be performed regarding the indicators that require to be

performed with respect to MBSE/UML Maturity Levels which are defined in previous

section. For instance, assuming that the evaluation is being performed to assess whether the

organization is in Maturity Level 4 or not where organization is required to separate the

domain concepts in domain model than business and design models. Therefore the answer for

“Do you separate the domain specific concepts in your domain model from your business and

design model?” should have numerical value of 4.

In addition to these, it is important to underline that organizations do not have to be on

Maturity Level 4 or 5 all the time because if their modeling aim is not being able to execute

code from the models or generate model, code, and/or text from the models automatically,

being Level 3 will be adequate for them. However, their advance in modeling still depends on

Assessment of Maturity Levels of MBSE/UML Maturity Model

62

the requirements of Level 3 so that they are required to be performed the modeling practices

with respect to that Level.

In addition to measurable questionnaire questions, structured questions are developed in order

to identify some terminologies within a particular field. For instance, to gather information

about what exactly the organization‟s aim on modeling is. By having information about this,

the questionnaire results can be assessed regarding this issue. An example for a structured

question is “Which of the artifacts like code, model, and/or text do you generate from your

developed models?” If the answer is “they do not generate any of them” then the

questionnaire for transformation aspect is not necessary to be evaluated since the organization

do not purpose for this and this leads that they are performing modeling not in MBSE/UML

Maturity Model 4 and 5. Therefore these types of questions will support the focus of

evaluation which means that for which level the focus should be. The unstructured questions

are most likely to collect some qualitative data on examining how they are performing

modeling within the organization. For instance, a sample question for this is “What do you do

if your personnel do not document their work?” The answer for this is open-ended. Therefore

it will just provide to elicit as much information as possible on documentation maintenance.

When the results from all types of questions within the questionnaire are combined with each

other; more reliable data will be gathered for assessing the organizations‟ Maturity Level on

MBSE/UML.

VALIDTY OF MBSE/UML MATURITY MODEL

63

7.VALIDTY OF MBSE/UML

MATURITY MODEL

A usability analysis is considered for validating the maturity model provided in the thesis. The

goal of a usability analysis is to pinpoint areas of confusion and ambiguity for the user which,

when improved will increase the efficiency and quality of a users‟ experience [49]. Since the

aim of MBSE/UML Maturity Model provide assessment criteria for organizations who are

adopting MBSE/UML principles within their organizations. Therefore the proposed Maturity

Model requires being easy to apply and being satisfied.

To validate these features, as defined in previous chapter, different questionnaires are

conducted for different aspects and these aspects‟ importance is validated by literature review.

For instance, when the occurrence of the aspects are conducted, it is gained that modeling

process is considered more important by researchers than the practitioners but still both state

that modeling process require to be well-defined. Therefore assessing these aspects in

MBSE/UML Maturity Model is providing reliability and validity. The questionnaires

designed will be required to be filled out by the intended personnel within a software

company who adopts MBSE/UML principles within the organization. The results will be

analyzed and evaluated regarding the criteria defined in previous chapters so that the

MBSE/UML Maturity Level of organization will be achieved. During this process, it can be

expected that the feedback can come up regarding the different aims or policies of

organization. However, the results are predicted as sufficient enough to capture many

different issues that come up.

CONCLUSION

64

8.CONCLUSION

The increasing adoption of MBSE/UML principles in software industry require some

standards and proficiency evaluations since many organizations are applying these new

principles and related approaches within the organization but with a lack of knowledge how

advanced or complete they are performing. The existing standards and definitions consider the

traditional software development process. Therefore applying them for MDD process is not

sufficient and effective since in MDD process models are developed as solutions and code is

aimed to be executed from the developed models automatically.

In this thesis, a MBSE/UML Maturity Model is proposed in order to provide organizations to

evaluate themselves with respect to their MBSE/UML approaches. Rather than many maturity

models presented in the software industry, this Maturity Model provide organizations to

assess themselves in different aspects regarding modeling like modeling process, personnel

competence, modeling tools, UML diagrams and models design, modeling quality assurance

techniques, syntax and semantic of UML, and transformation which are determined with

respect to the results from literature review. Therefore organizations will be able to assess

themselves in many essential aspects of MDD adaption process.

While developing this Maturity Model, a systematic literature review is performed and

essential aspects discussed by the researchers and practitioners are examined. In order to

retrieve the essential features of the models, answers for the research question 1 is analyzed so

that it is achieved that quality assurance activities related to models are one of the key roles

while executing the code from the models. The results for this research question is contributed

the indicators for design and syntax and semantic aspects of the MBSE/UML Maturity Model.

Second research question is also provided identification of indicators while evaluating the

proficiency of utilization of UML and modeling approaches. For instance, if organization is

defining their own meta-models this lead us that they are using UML in an advanced way.

Moreover during the writing process of this thesis, MDD Maturity Model [2] is found so that

later in the thesis, this proposed maturity model inspired the process of determining the

dimension of the maturity levels of MBSE/UML Maturity Model.

The most important factor that require to be considered and performed while adapting MDD

is defining a sufficient MDD process which is required to be performed by the organization.

Then the aspects like quality assurance techniques, training, motivation, and skills of

personnel, design of UML diagrams and models, syntax and semantic structure of UML,

modeling tools, and transformation tasks all related to modeling as process that require to be

considered. Some organizations are aware of these aspects. However, it is not known how

competence they are performing related tasks and activities. At that point, this proposed

MBSE/UML Maturity Model will provide organizations to assess themselves in different

aspects with respect to modeling as a process so that it will be easier for the organizations to

CONCLUSION

65

improve themselves in modeling process, modeling quality assurance techniques, modeling

tools, personnel competence based on modeling, transformation facilities of organization,

design issues, and syntax and semantic.

REFERENCES

66

REFERENCES

[1] Parastoo Mohagheghi , Vegard Dehlen, and Tor Neple: Definitions and approaches to

model quality in model-based software development– A review of literature, Elsevier B.V.,

April 2009

[2] Erkuden Rios, Teodora Bozheva, Aitor Bediaga, and Nathalie Guilloreau, MDD Maturity

Model: A Roadmap for Introducing Model-Driven Development, A. Rensink and J. Warmer,

2006

[3] Andy Evans, Robert France, Kevin Lano, and Bernhard Rumpe, The UML as a Formal

Modeling Notation , Computer Standards and Interfaces, Volume 19, Number 7, 1997

[4] Andy Evans¸ Robert France, Kevin Lano, and Bernhard Rumpe, Meta Modeling

Semantics of UML, Proceedings of UML'99, 1999

[5] Christian F.J. Lange, Improving the Quality of UML Models, in Practice International

Conference on Software Engineering Proceedings of the 28th international conference on

Software engineering, available on ACM, 2006

[6] David Harel and Bernhard Rumpe, Modeling Languages Syntax, Semantics and All That

Stuff, Technical report, Jerusalem, Israel, 2000

[7] Christian F.J. Lange and Michel R.V. Chaudron, Effects of Defects in UML Models –An

Experimental Investigation, ICSE‟06 available on ACM, 2006

[8] Helen C. Purchase, Jo-Anne Allder, and David Carrington, Graph Layout Aesthetics in

UML Diagrams: User Preferences, Journal of Graph Algorithms and Applications, 2001

[9] Parastoo Mohagheghi and Vegard Dehlen, Developing a Quality Framework for Model-

Driven Engineering, Springer-Verlag Berlin Heidelberg, 2008

[10] J. Rumbaugh, I. Jacobson, and G. Booch, The Unified Modeling Language Reference

Manual, Addison-Wesley, 1999.

[11] Russell Miles and Kim Hamilton, Learning UML 2.0., O‟Reilly Media, 2006

[12] Christof Lutteroth, AP1: A Platform for Model-Based Software Engineering, Springer-

Verlag Berlin Heidelberg, 2007

REFERENCES

67

[13] Alar Raabe, Feature Matching In Model-Based Software Engineering, Springer, 2006

[14] B. Hailpern and P. Tarr, Model-driven development: The good, the bad, and the ugly,

IBM Systems Journal, 2006

[15] Stefan Winkler and Jens von Pilgrim, A survey of traceability in requirements

engineering and model-driven development, Springer-Verlag, 2009

[16] Pervez Ghauri and Kjell, Gronhaug, Research Methods in Business Studies 3rd

Edition,

Prentice Hall, 2005

[17] Alain April, Jane Huffman Hayes, Alain Abran, and Reiner Dumke, Software

Maintenance Maturity Model (SMmm): The software maintenance process model, John Wiley

& Sons, Ltd, 2004

[18] Christian F.J. Lange, Bart DuBois, Michel R.V. Chaudron, and Serge Demeyer, An

Experimental Investigation of UML Modeling Conventions, Springer-Verlag Berlin

Heidelberg, 2006

[19] Christian F.J. Lange and Michel R.V. Chaudron, Defects in Industrial UML Models – A

Multiple Case Study –, the ACM/IEEE 10th International Conference on Model Driven

Engineering Languages and Systems, 2007

[20] Parastoo Mohagheghi and Vegard Dehlen, Where Is the Proof? - A Review of

Experiences from Applying MDE in Industry, Springer-Verlag Berlin Heidelberg, 2008

[21] Xabier Larrucea, Ana Belen García Díez, and Jason Xabier Mansell, Practical Model

Driven Development process, European Workshop on Model Driven Engineering, European

Software Institute, 2004

[22] Christian F.J. Lange and Michel R.V. Chaudron, Managing Model Quality in UML-

based Software Development, Pre-Proceedings IEEE International Workshop on Software

Technology and Engineering Practice, STEP 2005, 2005

[23] Helen C. Purchase, Matthew McGilI, Linda Colpoys, and David Carrington, Graph

drawing aesthetics and the comprehension of UML class diagrams: an empirical study,

Australian Computer Society, Inc., Proceedings of the 2001 Asia-Pacific symposium on

Information visualization - Volume 9, 2001

[24] John Krogstie, Evaluating UML Using a Generic Quality Framework, Idea Group Inc.,

2003

REFERENCES

68

[25] Norman E. Fenton and Shari Lawrence Pfleeger. Software Metrics, A Rigorous and

Practical Approach, Thomson Computer Press, second edition, 1996

[26] Stephen H. Kan, Metrics and Models in Software Quality Engineering, Addison Wesley

Professional, 2nd edition, September 2002

[27] S. R. Chidamber and C. F. Kemerer. A metrics suite for objectoriented design. IEEE

Transactions on Software Engineering, 20(6):476–493, 1994

[28] Lange, C.: Improving the Quality of UML Models in Practice. In: Proceedings of 28th

International Conference on Software Engineering (ICSE 2006), pp. 993–996, ACM, New

York, 2006

[29] Kim, H. and Boldyreff, C.: Developing Software Metrics Applicable to UMLModels, In

Proceedings of the 6th ECOOP Workshop on Quantitative Approaches in Object- Oriented

Engineering, 2002

[30] Baroni, A., Braz, S., Abreu, and F.B.: Using OCL to Formalize Object-Oriented Design

Metrics Definitions, In Proceedings of the 6th ECOOP Workshop on Quantitative Approaches

in Object-Oriented Software Engineering, 2002

[31] McQuillan, J. and Power, J.: A Metamodel for the Measurement of Object-Oriented

Systems: An Analysis using Alloy, In Proceedings of the 2008 international Conference on

Software Testing, Verification (ICST), pp. 228–297. IEEE Computer Society Press,

Washington, 2008

[32] Tony Bloomfield, MDA, Meta-Modelling and Model Transformation: Introducing New

Technology into the Defence Industry, in: ECMDA-FA 2005, pp. 9 18, Springer, 2005

[33] Evans A., France R., Lano K., and Rumpe B., The UML as a Formal Modeling Notation,

UML'98 Proceedings, Springer-Verlag, 1998

[34] Evans A., France R., Lano K., and Rumpe B., Meta-modelling Semantics of UML,

Behavioural Specifications for Businesses and Systems, Chapter 4, 1999

[35] Baker P., Loh S., and Weil F., Model-Driven Engineering in a Large Industrial Context

— Motorola Case Study, MoDELS 2005. LNCS, vol. 3713, pp. 476–491, Springer-Verlag,

2005

[36] Mohagheghi P., Fernandez M.A., Martell J.A., Fritzsche M., and Gilani W.,MDE

Adoption in Industry: Challenges and Success Criteria, M.R.V. Chaudron (Ed.): MODELS

2008 Workshops, LNCS 5421, pp. 54–59, Springer-Verlag, 2009

REFERENCES

69

[37] Miroslaw Staron, Adopting Model Driven Software Development in Industry – A Case

Study at Two Companies, O. Nierstrasz et al. (Eds.): MoDELS 2006, LNCS 4199, pp. 57 –

72, Springer-Verlag Berlin Heidelberg, 2006

[38] Scott W. Ambler, The Elements of UML Style, Cambridge University Press, 2003

[39] Lidia Fuentes-Fernández and Antonio Vallecillo-Moreno, An introduction to UML

Profiles, The European Journal for the Informatics Professional, Vol. 5, No. 2. , April 2004

[40] Klaus Krippendorff, Content Analysis- An Introduction to its Methodology, Sage

Publications Inc., 2004

[41] Robert Philip Weber, Basic Content Analysis 2nd

Edition, Sage Publications Inc., 1990

[42] Philipp Mayring, Qualitative Content Analysis, Qualitative Social Research [On-line

Journal], avaliable online CiteSeer, 2000

[43] Parastoo Mohagheghi and Jan Aagedal, Evaluating Quality in Model-Driven

Engineering, International Workshop on Modeling in Software Engineering (MISE'07), 2007

[44] Eric Jouenne and Véronique Normand, Tailoring IEEE 1471 for MDE Support, N.

Jardim Nunes et al. (Eds.), UML 2004 Satellite Activities, LNCS 3297, pp. 163-174, Springer

Verlag Berlin Heidelberg, 2005

[45] Mohagheghi P., Fernandez M. A., Martell J.A., Fritzsche M., and Gilani W., MDE

Adoption in Industry: Challenges and Success Criteria, M.R.V. Chaudron (Ed.): MODELS

2008 Workshops, LNCS 5421, pp. 54–59, Springer-Verlag Berlin Heidelberg, 2009

[46] William E. McUmber and Betty H.C. Cheng, A General Framework for Formalizing

UML with Formal Languages, in Proceedings of the 23rd International Conference on

Software Engineering, ICSE 2001, 2001

[47] Beecham, S. and Hall, T., Expert panel questionnaire: validating a requirements process

improvement model, avalaible online

http://homepages.feis.herts.ac.uk/~pppgroup/requirements_cmm.htm, last visit May 2010,

2003

[48] SEI, Capability Maturity Model Integration (CMMISM), Version 1.1. SEI, CMU/SEI-

2002-TR-029, 2002

[49] International Standards Organization, ISO DIS 9241-11: Guidance on Usability, 1994

REFERENCES

70

[50] OMG, Introduction To OMG's Unified Modeling Language (UML), available:

www.omg.org/gettingstarted/what_is_uml.htm, last visit May 2010, 2005

[51] Hevner AR, March ST, Park J, and Ram S, Design science in information systems

research, MIS Quarterly 28(1):75–105, 2004

[52] March ST and Smith G, Design and natural science research on information technology,

Decision Support Systems 15(4):251–266, 1995

APPENDIX

71

APPENDIX

Appendix 1: MBSE/UML Modeling Process Evaluation Questionnaire

This survey is conducted in order to measure the indicators that are essential in Modeling

Process of organization. These indicators are defined in MBSE/UML Maturity Model.

1. Does your organization have a defined MDD Process? If no please leave this survey, if

yes please continue with question 2 in this survey.

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

2. Is MDD Process of your organization documented properly?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

APPENDIX

72

3. Does MDD Process document include description of organization workflow for MDD

projects?

Definitely Yes

Partly

Poorly

Not sure

Definitely No

4. Which of the following does MDD Process document include?

Configuration Management Plan

Training Plan

Project Plan

Project Management Plan

Risk Management

Test Plan

Quality Management Plan

Requirements Specification

Software Description Document

Modeling Goals Document

APPENDIX

73

Modeling Standards Document

Other:

5. Does MDD Process Document provide information about documentation maintenance

regarding facilities like model updates and changes?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

6. Are the tasks, roles, and activities assigned personal defined in MDD Process?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

7. Does your organization consider and state modeling and tools standards?

Definitely Yes

Usually

APPENDIX

74

Planned but not applied

Not sure

Definitely No

8. Are methods and/or techniques regarding measurement of modeling quality the tasks

defined?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

9. Do you estimate the size, effort, and cost for software projects with respect to

adapting modeling as a development process?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

10. Do you enforce your personal to document their work properly?

APPENDIX

75

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

11. Are the commitments and project risks with respect to modeling being traced

according to MDD project plan?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

12. Are you able to trace whether your defined modeling requirements are achieved by

models or not?

Definitely Yes

Usually

Planned but not applied

Not sure

APPENDIX

76

Definitely No

13. Do you define the methods and/or techniques to perform measurement and analysis

of your modeling tasks?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

14. Do you manage the communication and/or interaction facilities between all

participants with respect to the project by means of modeling?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

16. Do you consider the issues and concepts about modeling domain while defining the

MDD Process?

Definitely Yes

Usually

APPENDIX

77

Planned but not applied

Not sure

Definitely No

17. Do you think that your MDD Process is applicable with the project domain?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

18. Which of the following do you define with respect to MDD process??

Definitely

Yes Usually

Planned but

no applied Not sure

Definitely

No

Modeling Goals

regarding particular

project

Modeling Goals

regarding particular

Domain

APPENDIX

78

Definitely

Yes Usually

Planned but

no applied Not sure

Definitely

No

Modeling Goals

regarding

organization goals

Appendix 2: MBSE/UML Quality Assurance Techniques Evaluation

Questionnaire

1. Does your organization apply any quality assurance techniques for modeling process?

If the answer is no, please leave the survey, otherwise please continue to answer survey

questions by question 2.

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

2. Does Quality Assurance Techniques that your organization applies include tasks for

MDD process? If the answer is no, please leave the survey, otherwise please continue to

answer survey questions by question 3.

Definitely Yes

Usually

Planned but not applied

Not sure

APPENDIX

79

Definitely No

3. Does your organization document quality assurance techniques defined for MDD? If

your answer is definitely no, please leave the survey; otherwise please continue to answer the

survey questions by question 4.

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

4. Which of the following does your organization define in Quality Assurance Plan

Document?

Definitely

Yes Partly Moderate Poorly

Definitely

No

Model Quality

Standards

Identification

Modeling Purpose

Identification

Evaluation of

Software (Modeling)

Tools

APPENDIX

80

Definitely

Yes Partly Moderate Poorly

Definitely

No

Evaluation of

Modeling

Requirements

Analysis Process

Evaluation of Model

Design Process

Evaluation of Test

Models

Evaluation of Model

Acceptance Testing

Process

Evaluation of Model

Configuration

Management Process

Perform Model

Configuration Audits

Evaluation of

APPENDIX

81

Definitely

Yes Partly Moderate Poorly

Definitely

No

Training Facilities

Evaluation of

Domain Testability

5. Which of the following documents are expected to be produced during MDD process?

Definitely

Yes Usually

Planned but

not applied

mostly

Not sure Definitely

No

Modeling Needs

Statement

Project Plan

Model Configuration

Management Plan

Quality Assurance

Plan

APPENDIX

82

Definitely

Yes Usually

Planned but

not applied

mostly

Not sure Definitely

No

Feasibility Study

Risk Analysis

System and

Modeling Tools

Support and

Acquisition Plan

System Security and

Privacy Plan

Functional and Non-

Functional

Requirements

Document

Granularity of

Models Document •

(Ex. In design phase,

what is the Level of

abstraction in design

phase)

APPENDIX

83

Definitely

Yes Usually

Planned but

not applied

mostly

Not sure Definitely

No

Domain

Requirements

Document

Semantic Description

Internal Audit Plan

System/Subsystem

and Modeling Tools

Specifications

Validation,

Verification, and

Testing Plan for

Models

Training Plan

Model

Transformation Plan

APPENDIX

84

Definitely

Yes Usually

Planned but

not applied

mostly

Not sure Definitely

No

Modeling Tools,

Transformation

Tools, Modeling

Language

Requirements,

Programing

Language

Specification,

Simulation and

Integration

Identification

Model Checking

Maintenance Manual

6. Which of the following reviews does your organization apply?

Definitely

Yes Usually

Planned but

not applied

mostly

Not sure Definitely

No

Model Requirements

Review

APPENDIX

85

Definitely

Yes Usually

Planned but

not applied

mostly

Not sure Definitely

No

Design Review

Modeling Review

Preliminary Model

Design Review

Model Testing

Review

Model

Transformation

Readiness Review

Model Acceptance

Test Review

Appendix 3: MBSE/UML Personnel Competence Evaluation Questionnaire

vol. 1

This questionnaire is conducted to assess the maturity of the personnel within the organization

regarding the MBSE/UML. Moreover if any activities are performed for improving the

personnel competence like any training programs, this questionnaire also can measure the

APPENDIX

86

competence of these activities held by organization. This is required to be filled out by

developers, modelers, designers, and software engineers.

1. How would you rank your previous knowledge on MBSE approaches? *

1 2 3 4 5

nothing

advanced

2. How would you rank your previous knowledge on UML? *

1 2 3 4 5

nothing

advanced

3. Which of the following do you have? *

Any MBSE related certificates

Academic courses on MBSE

Attendance to conferences, workshops about MBSE

None

Other:

4. How would you rank your current knowledge on MBSE approaches? *

APPENDIX

87

1 2 3 4 5

learned nothing

learned in depth

5. How would you rank your current knowledge on UML? *

1 2 3 4 5

learned nothing

learned in depth

6. How useful were the materials provided to you during this project? *

1 2 3 4 5

not useful

very useful

7. How useful were the materials for your career? *

1 2 3 4 5

not useful

very useful

8. Regarding your experience on MBSE/UML approaches within your organization,

how would you rank the learning curve? *

APPENDIX

88

1 2 3 4 5

insufficient

very sufficient

9. For a particular MBSE/UML project, how would you rank your knowledge on

relevant domain? *

1 2 3 4 5

nothing

very well

10. During a particular MBSE/UML project or MBSE/UML adapting to your

organization, how can you rank your knowledge on the tools that are used? *

1 2 3 4 5

never used

experienced

11. During a particular MBSE/UML project or MBSE/UML adapting to your

organization, were you confident with the tools used? *

1 2 3

not confident

very confident

APPENDIX

89

12. During a particular MBSE/UML project, were the materials are sufficient for a

particular domain? *

1 2 3

not sufficient

very sufficient

13. Which of the following are required to be considered regarding the future training

on MBSE/UML approaches?*

Meta Modeling

UML Profile

OCL

DSL

Round-Trip Engineering Facilities

Other:

14. How useful would it be to learn more about MBSE/UML approaches? *

1 2 3 4 5

not useful

very useful

15. How would you rank the courses given to you by the organization regarding the

MBSE/UML approaches? *

APPENDIX

90

1 2 3 4 5

not given

very sufficient

16. Which of the following UML modes have you worked previously? *

UML as sketches

UML as blueprint

UML as implementation language

None

Other:

17. Which of the following UML modes, did you become familiar currently? *

UML as sketches

UML as blueprint UML as implementation language

None

Other:

18. How did the courses and/or materials provided by your organization improve your

knowledge and/or skills on UML? *

1 2 3

APPENDIX

91

insufficient

sufficient

19. How would you rank your UML knowledge and/or skills currently? *

1 2 3 4 5

nothing

advanced

20. What specific tasks are required to be considered during learning curve regarding

UML? *

Meta Modeling

UML Profiles

OCL

Other:

21. Which of the following MBSE tasks you have knowledge and/or skills? *

Model to Model Transformation

Model to Text Transformation

Model to Code Transformation

Round-Trip Engineering Facilities

Other:

APPENDIX

92

22. How would you rank your familiarity with the modeling process of your

organization *

1 2 3 4 5

not known

well known

23. How would you rank your knowledge on modeling and organization goals of your

organization? *

1 2 3 4 5

do not know

well known

Appendix 4: MBSE/UML Personnel Competence Evaluation Questionnaire

vol. 2

This questionnaire is required to be filled out by the ones who are responsible for personnel

competence activities in addition to the project and quality managers. The aim of this survey

is to evaluate how the personnel competence activities are performed within the organization

regarding MBSE/UML.

1. Do you define a training plan regarding MBSE/UML knowledge requirements of your

personnel?

Definitely Yes

Usually

Planned but not applied

Not sure

APPENDIX

93

Definitely No

2. Are you aware of your personnel's knowledge, skills, and experiences on

MBSE/UML?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

3. Do you analyze the specific MBSE/UML requirements for particular projects?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

4. Do you analyze the requirements for particular domain?

Definitely Yes

Usually

Planned but not applied

Not sure

APPENDIX

94

Definitely No

5. Regarding the requirements of domain and/or project, do you define any training

plans to support knowledge of your personnel?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

Appendix 5: MBSE/UML Tools Evaluation Questionnaire

1. Are your personnel aware of your organization's policy of tool use?

Definitely Yes

Usually

Not sure

Definitely No

2. How does the modeling tools (e.g. model editors, model executors, transformation

editors, and so on) support integration with other tools that support MDD process like

Microsoft Visual Studio, NetBeans, and so on?

1 2 3 4 5

Not Supported

Totally Supported

APPENDIX

95

3. Which of the following modeling tool types is your organization using?

Model Editors

Model Executors

Model Simulators

Model Repositories

Transformers

Transformation Repositories

Other:

4. Are you able to simulate the models developed?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

Other:

5. Are you able to execute code from your developed models by using transformation

tools?

Definitely Yes

Usually

APPENDIX

96

Planned but not applied

Not sure

Definitely No

Other:

6. How would you rank the following issues of your UML modeling tool used for MDD

process?

Not

Supported

Slightly

Supported

Moderately

Supported

Very

Supported

Repository Support

Round-Trip Engineering

HTML Document

Full UML 2.0 Support

(or other versions that is

used within

organization)

Data Model Integration

APPENDIX

97

Not

Supported

Slightly

Supported

Moderately

Supported

Very

Supported

Versioning

Model Navigation

Printing Support

Diagram Views

Exporting Diagrams

Scripting

Robustness

Platform Applicability

Domain Applicability

APPENDIX

98

Not

Supported

Slightly

Supported

Moderately

Supported

Very

Supported

Auto-Generation

XMI Support

7. Which of the following issues are essential for UML modeling tool that will be used

during MDD process?

Not

Essential

Seldomly

Essential

Essential

once in a

while

Essential

most of the

time

Very

Essential

Repository Support

Round-Trip

Engineering

HTML Document

Full UML 2.0 (or

other versions)

Support

APPENDIX

99

Not

Essential

Seldomly

Essential

Essential

once in a

while

Essential

most of the

time

Very

Essential

Data Model

Integration

Versioning

Model Navigation

Printing Support

Diagram Views

Exporting Diagrams

Scripting

Robustness

APPENDIX

100

Not

Essential

Seldomly

Essential

Essential

once in a

while

Essential

most of the

time

Very

Essential

Platform

Applicability

Domain

Applicability

Auto-Generation

XMI Support

Appendix 6: MBSE/UML Design Evaluation Questionnaire

This survey is conducted by regarding the well-formedness rules and modeling conventions

proposed by OMG.

Model Development

This section includes questions to analyze what kind of models are developed by

organization.

1. Do you develop a model or models to represent the following requirements?

Definitely

Yes Usually

Planned but

not applied Not sure

Definitely

No

APPENDIX

101

Definitely

Yes Usually

Planned but

not applied Not sure

Definitely

No

User Requirements

System

Requirements

Business

Requirements

Domain

Requirements

Syntax and Semantic

Requirements

2. Which of the following model are you developing? The requirement models are the

models that are developed in order to represent the requirements of clients. Business models

represent the business requirements and information about system platform and technology

are not be shown. Design Models are developed in order to provide detailed information about

the system and product that will be developed. Both functional and non-functional

requirements are shown. Domain Model is the model that defines how a business works

without reference to software systems, similarly to OMG‟s Computation Independent Model.

Definitely

Yes Usually

Planned but

not applied Not sure

Definitely

No

APPENDIX

102

Definitely

Yes Usually

Planned but

not applied Not sure

Definitely

No

Requirements Model

Design Model

Business Model

Domain Model

3. Do you separate the concepts of business, design, and domain models? If you are not

developing at least two of these models, you can skip this question and continue with

following questions.

Definitely

Yes Usually

Planned but

not applied Not sure

Definitely

No

Requirements Model

Design Model

Business Model

APPENDIX

103

Definitely

Yes Usually

Planned but

not applied Not sure

Definitely

No

Domain Model

UML Diagrams Design Issues

4. Do you apply any metrics to your UML Diagrams to measure the quality of them? ex.

metrics for measuring the complexity of UML Diagrams.

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

This part is conducted for those who apply metrics to their UML Models.

5. Which of the following issues of UML Diagrams' layout do you measure?

Definitely

Yes Usually

Planned but

not applied Not sure

Definitely

No

Complexity

APPENDIX

104

Definitely

Yes Usually

Planned but

not applied Not sure

Definitely

No

Completeness

Modifiability

Familiarity

Similarity

Continuity of points

and lines

Connectedness of

UML Elements

6. Do you think metrics that are defined measure the actual attributes that are intended

in metric definition?

Definitely Yes

Usually

Planned but not applied

APPENDIX

105

Not sure

Definitely No

7. Do you calculate the metrics automatically by means of tool?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

8. With your defined metrics, are you able to measure the model quality goals defined?

(ex. consistency, correctness, comprehensibility, etc. of models)

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

Meta Model Design Issues

This part of survey is conducted for those who perform meta-modeling within their

organization. The questions are derived from the OMG propose for UML well-formedness

rules and modeling conventions.

APPENDIX

106

9. OMG proposes that initial embedded capitals required to be used while naming the

Meta Classes. Do you apply this to your Meta Classes? Please indicate in others, if you do

not have meta models.

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

Other:

10. Are names of meta associations written in the same manner as meta classes (e.g.,

‘ElementReference’).

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

11. Do you define a specific notation for the followings?

Definitely

Yes Usually

Planned but

not applied Not sure

Definitely

No

APPENDIX

107

Definitely

Yes Usually

Planned but

not applied Not sure

Definitely

No

For names that

consist of appended

nouns/adjectives

For Boolean meta

attribute names

For Enumeration

types

12. While referring to meta classes, meta associations, meta attributes, etc. in the text,

the exact names as they appear in the model are always used. Proposed by OMG

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

13. If a mandatory section does not apply for a meta class, the text ‘No additional XXX’

is used, where ‘XXX’ is the name of the heading. If an optional section is not applicable,

it is not included. Proposed by OMG

Definitely Yes

APPENDIX

108

Usually

Planned but not applied

Not sure

Definitely No

OCL Specification

14. Do you define any metrics to measure the structural complexity of OCL expressions?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

We do not have OCL Expressions

Appendix 7: MBSE/UML Syntax and Semantic Evaluation Questionnaire

1. Which of the following is your aim on applying modeling approaches?

No Modeling

Modeling as Sketches

Modeling as Blueprints

Modeling as Transformation

Modeling as Execution

2. How would you rank the support of defined syntax and semantic expressions by your

modeling language (UML) regarding your modeling aim?

APPENDIX

109

Complete Support

Partly Support

Moderate Support

Poorly Support

Not at all

3. Do you require defining additional syntax and semantic declaration for your modeling

process?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

4. How do you consider that UML's support on allowing developers to define additional

syntax and semantic declarations?

Complete Support

Partly Support

Moderate Support

Poorly Support

Not at all

5. Do you think that your defined syntax and semantic regulations are complying to

your modeling goals? If your answer is "Definitely No" for question 3, then you can skip this

question and continue to survey by following questions.

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

6. Are you confident that the modeling tools that you are using are applicable with UML

and UML's additional features like UML Profile, Meta Models?

APPENDIX

110

Definitely Yes

Usually

Moderately

Not sure

Definitely No

7. Is UML's syntax and semantic, profiles, meta-models are adequate to represent the

domain specific regulations?

Definitely Yes

Usually

Moderately

Not sure

Definitely No

8. Which of the following issues do you apply and/or useregarding your MDD process?

Meta-Modeling

UML Profiles

Syntax and Semantic Definition by using formal languages like Z

Modeling Conventions

Well-Formedness Rules

None

Other:

Appendix 8: MBSE/UML Transformation Evaluation Questionnaire

General Statement

General Requirement Analysis for Model Transformation

1. Which of the following artifacts do you generate from your developed models?If your

answer is none, please leave this survey.

Code

Model

APPENDIX

111

Text

None

Other:

2. Do you generate models, text, and/or code from models manually? If your answer is

'Definitely No' and 'Not Sure', please skip the section 'Automatic Generation from Models'

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

Automatic Generation from Models

3. Are you able generating artifacts like code, model, and text from models

automatically?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

Other:

4. Do you assure that the granularity of your developed models is sufficient enough to

represent the details of a particular system?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

5. Which of the following are you using while generating artifacts from your models

automatically?

APPENDIX

112

UML Profile

Meta Models

UML Profile with OCL description

Domain Specific Languages (DSL)

Developing own code generators

Other:

6. What is the percentage of code that you generate from your developed models

automatically?

7. Do you use DSL within your MDD process?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

Other:

Domain Concepts

8. How would you rank the effect of domain for a particular project?

1 2 3 4 5

Not Related

Very Related

9. Do you develop domain models for particular project?

Definitely Yes

Usually

APPENDIX

113

Planned but not applied

Not sure

Definitely No

10. Do you separate the domain specific concepts in domain model from your business

and design models?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

Model Reusability, Changeability, and Executable Model

11. Are you able to do changes on your meta-model and update the models developed

based on your meta-model representations?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

12. Are you able to update all related models when an update is required in one specific

model?

Definitely Yes

Usually

Planned but not applied

Not sure

Definitely No

13. What is the percentage range of your developed models that you can reuse for other

projects?

APPENDIX

114

14. How would you rank your models' reusability, changeability, and executable?

Not at All Slightly

Sufficient

Moderately

Sufficient

Very

Sufficient

Reusability

Changeability

Executable