Intro Integration Evaluation Conclusions

Integration of a Graph-Based Model Indexer inCommercial Modelling Tools

Antonio García-Domínguez, Konstantinos Barmpis,Dimitrios S. Kolovos, Marcos Aurelio Almeida da Silva, Antonin

Abherve and Alessandra Bagnato

Aston University, University of York, SOFTEAM Cadextan

MoDELS’16October 7th, 2016

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 1 / 27

Intro Integration Evaluation Conclusions

Outline

1 Intro

2 Integration

3 Evaluation

4 Conclusions

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 2 / 27

Intro Integration Evaluation Conclusions

Motivation for this work

MONDO EU project: produced Hawk indexerGoal: tackle scalability of MDE in its various facetsOne product was Hawk, a scalable model indexer:https://github.com/mondo-project/mondo-hawk

Hawk mirrors fragmented models into NoSQL graphs

SOFTEAM: Constellation collab server needed searchSOFTEAM has 20+ years experience on UML toolingModelio is their open-source modelling toolJoined MONDO due to scalability challengesCreated Constellation: Modelio collab serverConstellation had no search box – need indexing!

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 3 / 27

Intro Integration Evaluation Conclusions

Motivation for this work

MONDO EU project: produced Hawk indexerGoal: tackle scalability of MDE in its various facetsOne product was Hawk, a scalable model indexer:https://github.com/mondo-project/mondo-hawk

Hawk mirrors fragmented models into NoSQL graphs

SOFTEAM: Constellation collab server needed searchSOFTEAM has 20+ years experience on UML toolingModelio is their open-source modelling toolJoined MONDO due to scalability challengesCreated Constellation: Modelio collab serverConstellation had no search box – need indexing!

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 3 / 27

Modelio: SOFTEAM’s modelling tool

Open source, scriptable, UML/BPMN/SysML...Structure: workspace → projects → fragments (.exml)Projects can import .ramc libraries (zipped fragments)

Intro Integration Evaluation Conclusions

Constellation: SOFTEAM’s collab server

Admin server coordinatesagents and provides web UIAgent nodes (HTTP/SVN)access/host fragmentsNo querying facilities

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 5 / 27

Intro Integration Evaluation Conclusions

Hawk: NoSQL model indexer

We go from these model files...

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 6 / 27

Intro Integration Evaluation Conclusions

Hawk: NoSQL model indexer

... to these NoSQL graphs.

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 6 / 27

Intro Integration Evaluation Conclusions

Hawk: NoSQL model indexer

EPackages → metamodel nodesEClasses → type nodes

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 6 / 27

Intro Integration Evaluation Conclusions

Hawk: NoSQL model indexer

.xmi → file nodesEObjects → element nodes

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 6 / 27

Intro Integration Evaluation Conclusions

Hawk: NoSQL model indexer

MM index: EPackage URI → metamodel nodeFile index: repo + file path → file nodeUsers can define custom indices by attribute/expression

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 6 / 27

Intro Integration Evaluation Conclusions

Outline

1 Intro

2 Integration

3 Evaluation

4 Conclusions

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 7 / 27

Intro Integration Evaluation Conclusions

Integration approach

Key detail in Hawk: plugin-based architectureHawk has abstraction layers for:

Database backends (Neo4j, OrientDB)Model persistence formats (XMI, UML, Modelio...)Query languages (Epsilon Object Language)

Steps taken1 Expose Modelio metamodels to Hawk2 Write .exml/.ramc parser3 Extend Hawk with Modelio specifics4 Embed into Constellation

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 8 / 27

Intro Integration Evaluation Conclusions

Initial legal issues

ProblemHawk is EPL, Modelio is GPLv3: incompatible licensesCannot reuse neither metamodel code nor parsing code

SolutionsSOFTEAM wrote M2T transformation to expose metamodelsas Apache-licensed library (ModelioMetamodelLib)

http://github.com/aabherve/modelio-metamodel-libHawk includes transformation to Ecore

Hawk devs collaborated with SOFTEAM to write new parsersClean-room approach: no looking at Modelio code!

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 9 / 27

Intro Integration Evaluation Conclusions

Initial legal issues

ProblemHawk is EPL, Modelio is GPLv3: incompatible licensesCannot reuse neither metamodel code nor parsing code

SolutionsSOFTEAM wrote M2T transformation to expose metamodelsas Apache-licensed library (ModelioMetamodelLib)

http://github.com/aabherve/modelio-metamodel-libHawk includes transformation to Ecore

Hawk devs collaborated with SOFTEAM to write new parsersClean-room approach: no looking at Modelio code!

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 9 / 27

.exml format

<EXT object="Component" version="3"><DEPS><ID name="Component" mc="Class" uid="2d7b..."/><EXTID name="Element" mc="Class" uid="4ed7..."/>

</DEPS><OBJECT><ID name="Component" mc="Class" uid="2d7b..."/><PID name="Architecture" mc="Package" uid="ea87..."/><ATTRIBUTES><ATT name="Name">Component</ATT>...

</ATTRIBUTES><DEPENDENCIES><COMP relation="OwnedOperation">...</COMP><COMP relation="Parent">...</COMP>

</DEPENDENCIES></OBJECT>

</EXT>

.exml format: version, dependencies

<EXT object="Component" version="3"><DEPS><ID name="Component" mc="Class" uid="2d7b..."/><EXTID name="Element" mc="Class" uid="4ed7..."/>

</DEPS><OBJECT><ID name="Component" mc="Class" uid="2d7b..."/><PID name="Architecture" mc="Package" uid="ea87..."/><ATTRIBUTES><ATT name="Name">Component</ATT>...

</ATTRIBUTES><DEPENDENCIES><COMP relation="OwnedOperation">...</COMP><COMP relation="Parent">...</COMP>

</DEPENDENCIES></OBJECT>

</EXT>

Format versionDependencies

.exml format: version, dependencies

<EXT object="Component" version="3"><DEPS><ID name="Component" mc="Class" uid="2d7b..."/><EXTID name="Element" mc="Class" uid="4ed7..."/>

</DEPS><OBJECT><ID name="Component" mc="Class" uid="2d7b..."/><PID name="Architecture" mc="Package" uid="ea87..."/><ATTRIBUTES><ATT name="Name">Component</ATT>...

</ATTRIBUTES><DEPENDENCIES><COMP relation="OwnedOperation">...</COMP><COMP relation="Parent">...</COMP>

</DEPENDENCIES></OBJECT>

</EXT>

Format versionDependencies

.exml format: root object

<EXT object="Component" version="3"><DEPS><ID name="Component" mc="Class" uid="2d7b..."/><EXTID name="Element" mc="Class" uid="4ed7..."/>

</DEPS><OBJECT><ID name="Component" mc="Class" uid="2d7b..."/><PID name="Architecture" mc="Package" uid="ea87..."/><ATTRIBUTES><ATT name="Name">Component</ATT>...

</ATTRIBUTES><DEPENDENCIES><COMP relation="OwnedOperation">...</COMP><COMP relation="Parent">...</COMP>

</DEPENDENCIES></OBJECT>

</EXT>

Object name, metaclass, UIDSame UID = same objectHappens when importing libraries

Container UIDEMF compatibility would requirederiving containments

.exml format: root object

<EXT object="Component" version="3"><DEPS><ID name="Component" mc="Class" uid="2d7b..."/><EXTID name="Element" mc="Class" uid="4ed7..."/>

</DEPS><OBJECT><ID name="Component" mc="Class" uid="2d7b..."/><PID name="Architecture" mc="Package" uid="ea87..."/><ATTRIBUTES><ATT name="Name">Component</ATT>...

</ATTRIBUTES><DEPENDENCIES><COMP relation="OwnedOperation">...</COMP><COMP relation="Parent">...</COMP>

</DEPENDENCIES></OBJECT>

</EXT>

Object name, metaclass, UIDSame UID = same objectHappens when importing libraries

Container UIDEMF compatibility would requirederiving containments

.exml format: attributes and dependencies

<EXT object="Component" version="3"><DEPS><ID name="Component" mc="Class" uid="2d7b..."/><EXTID name="Element" mc="Class" uid="4ed7..."/>

</DEPS><OBJECT><ID name="Component" mc="Class" uid="2d7b..."/><PID name="Architecture" mc="Package" uid="ea87..."/><ATTRIBUTES><ATT name="Name">Component</ATT>...

</ATTRIBUTES><DEPENDENCIES><COMP relation="OwnedOperation">...</COMP><COMP relation="Parent">...</COMP>

</DEPENDENCIES></OBJECT>

</EXT>

Attributes: name-value pairsDependencies: references

COMPositions (containment?)LINKs (non-containment)

.exml format: attributes and dependencies

<EXT object="Component" version="3"><DEPS><ID name="Component" mc="Class" uid="2d7b..."/><EXTID name="Element" mc="Class" uid="4ed7..."/>

</DEPS><OBJECT><ID name="Component" mc="Class" uid="2d7b..."/><PID name="Architecture" mc="Package" uid="ea87..."/><ATTRIBUTES><ATT name="Name">Component</ATT>...

</ATTRIBUTES><DEPENDENCIES><COMP relation="OwnedOperation">...</COMP><COMP relation="Parent">...</COMP>

</DEPENDENCIES></OBJECT>

</EXT>

Attributes: name-value pairsDependencies: references

COMPositions (containment?)LINKs (non-containment)

.exml format: nested objects

<EXT object="Component" version="3"><DEPS><ID name="Component" mc="Class" uid="2d7b..."/><EXTID name="Element" mc="Class" uid="4ed7..."/>

</DEPS><OBJECT><ID name="Component" mc="Class" uid="2d7b..."/><PID name="Architecture" mc="Package" uid="ea87..."/><ATTRIBUTES><ATT name="Name">Component</ATT>...

</ATTRIBUTES><DEPENDENCIES><COMP relation="OwnedOperation">...</COMP><COMP relation="Parent">...</COMP>

</DEPENDENCIES></OBJECT>

</EXT>

<OBJECT><ID name="..." mc="Generalization" uid="58e6..."/><PID name="Component" mc="Class" uid="2d7b..."/><ATTRIBUTES>...</ATTRIBUTES><DEPENDENCIES><LINK relation="SuperType"><ID name="Element" mc="Class" uid="4ed7..."/>

</LINK></DEPENDENCIES>

</OBJECT>PID here is XML container’sUID: not always the case!LINK to object in another .exml

Intro Integration Evaluation Conclusions

Hawk: tweaks needed for Modelio

References are UID-basedParsed objects/refs can declare they are UID-basedIf so, Hawk will maintain/use UID → object index

Same UID = same objectParsed resources can declare they provide “singletons”If so, Hawk will merge objects with same UID into 1 nodeA single element node can belong then to multiple files

Container-first rather than containment-firstOptional automated derivation of EMF-style containmentsOnly needed for compatibility with EMF-based toolsNot needed for Epsilon compatibility

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 15 / 27

Intro Integration Evaluation Conclusions

Outline

1 Intro

2 Integration

3 Evaluation

4 Conclusions

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 16 / 27

Intro Integration Evaluation Conclusions

Indexing costs on real projects (Neo4j)

File type .exml .ramc

Minimum 229B 23kBMedian 1.51kB 8.45MBMaximum 2.77MB 30.98MB

Count 201 526 23

Stages 2GB 4GB 8GB

Frag ins. 2554s 2502s 2371sFrag conn. 728s 541s 443sCont deriv. 1851s 1234s 1123s

Total 5171s 4287s 3958s

InputsSOFTEAM provided 15in-house projects2GB between .exml and.ramc, 1.23M elems

Memory consumption

W/containment: 2GB+Without: 1GB (3146s)Diminishing returns

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 17 / 27

Embedding into Constellation: advanced queries

Required new OrientDB backend (Neo4j is GPLv3-licensed)SOFTEAM embedded Hawk into web UI in 3 monthsFirst result: EOL query console

Embedding into Constellation: dashboards

SOFTEAM used Hawk queries to feed project dashboardsRegular users do not need to know any queries

Intro Integration Evaluation Conclusions

M2T with Hawk: motivation

Business caseM2T in Modelio was done through Jython scriptingSOFTEAM wanted M2T agents in ConstellationCode generation as a service

Which is more efficient?Modelio + Jython?Hawk + Epsilon Generation Language?

Need to take into account indexing times as well!

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 20 / 27

Intro Integration Evaluation Conclusions

M2T with Hawk: inputs

Script size param m = 1 m = 2 m = 4 m = 8 m = 16

Project size 80MB 150MB 279MB 814MB 1.8GB.exml size 58MB 104MB 179MB 422MB 752MB.exml count 1 255 2 087 3 430 7 849 13 790Element count 47 981 88 451 154 281 367 840 657 228

SOFTEAM wrote simple UML model generator in JythonSize parameter controls number of UML classes and number ofattributes/references per class

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 21 / 27

Intro Integration Evaluation Conclusions

M2T with Hawk: indexing times

1 2 3 4 5 6

·105

200

400

600

800

1,000

1,200

Project size (model elements)

Indexing

time(s)

500

1,000

1,500

2,000

2,500

Orie

ntDBdisk

spaceusage(M

B)

TimeDisk space

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 22 / 27

Intro Integration Evaluation Conclusions

M2T with Hawk: generation times

1 2 3 4 5 6

·105

100

101

102

103

Project size (model elements)

Generationtim

e(s)

MT (run 1) MT (runs 2–5)HT (run 1) HT (runs 2–5)

MT: Modelio M2THT: Hawk M2TCold runs: 10-100xfaster on HawkWarm runs: Hawkscales better

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 23 / 27

Intro Integration Evaluation Conclusions

M2T with Hawk: tradeoff

1 2 3 4 5 6

·105

0

5

10

15

20

1

Project size (model elements)

dMT/(HT

run1+

indexing

)e

How many cold Modelio runs before Hawk makes more sense?For the largest project, the answer is 1!

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 24 / 27

Intro Integration Evaluation Conclusions

Outline

1 Intro

2 Integration

3 Evaluation

4 Conclusions

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 25 / 27

Intro Integration Evaluation Conclusions

Conclusions and future work

Obtained resultsSOFTEAM wanted querying for their model collab serverHawk is now powering it:

Licensing issues guided many design decisionsAvailable as dashboards and advanced manual querying

Hawk+EGL faster than Modelio+Jython for M2T-as-service

Future workSOFTEAM: integrate Hawk in more places and providequerying capabilities for less technical usersFurther performance improvements in Hawk:

Horizontal scalability (more RAM had diminishing returns)On-demand derived features to reduce indexing timeOptimize OrientDB backend (much work done for v1.1!)

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 26 / 27

End of the presentation

Questions?

@antoniogado

A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 27 / 27