Date post: | 14-Jan-2017 |
Category: |
Technology |
Upload: | angela-olasz |
View: | 38 times |
Download: | 0 times |
IQMULUS INFRASTRUCTURE
Michel KrämerFraunhofer IGD
IQmulus, Final WorkshopBergen, September 21, 2016
The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 318787.
MICROSERVICE ARCHIT ECTURE
„In short, the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.“
- Martin Fowler
6
http://martinfowler.com/articles/microservices.html
IQMULUS ARCHIT ECT URE
7
Processingcloud
Virtualmachine Virtualmachine Virtualmachine
…
Service Service
Service Service
Service Service
Service
Service Service
Service
Service
Service Service
Service
Service Service
Service Service
Service
ServiceService
Service Service
D S L TOOLC H A IN , C ON TR OL C OMP ON ENTS
8
User:• Select/write workflow• Upload/select data to process
Infrastructure:• Select suitable processing services
and parameters• Generate executable process chains• Monitor execution and use of
resources
Workflow script(DSL)
Workflow Editor
Interpreter
Job Manager
Job TrackerJob TrackerJob Tracker
Processing serviceProcessing serviceProcessing service
FOR LOOP UNROL L ING
10
[PointCloudCollection]
Placeholder:[File1,File2,File3,…]Interpreter
Loopiteration1
Loopiteration2
Loopiteration3 …
File1 File2 File3 …
JobManager
JobTrackerServices
Node1
JobTrackerServices
Node2
JobTrackerServices
Node3
…
…
FOR LOOP UNROL L ING
11
[PointCloud]
Placeholder:File1Interpreter
Workflow
File1
JobManager
JobTrackerServices
Node1
JobTrackerServices
Node2
JobTrackerServices
Node3
…
…File1File1Fileset
FOR LOOP UNROL L ING
12
[PointCloudCollection]
Placeholder:[File1,File2,File3,…]Interpreter
Loopiteration1
Loopiteration2
Loopiteration3 …
File1 File2 File3 …
JobManager
JobTrackerServices
Node1
JobTrackerServices
Node2
JobTrackerServices
Node3
…
…File1File1Fileset
SOME STATIST ICS
26
• 69 virtual machines• about 70 Docker images• about 570 Docker containers• 4 Spark clusters (24 VMs)• 1 Hadoop cluster (4 VMs)• HDFS (30 TB, 15 TB used, 61 VMs)
TECHNOLOGIES
27
• Docker• Spark• Hadoop• Ansible• Angular JS + Bootstrap• MongoDB/PostgreSQL• Elasticsearch + Logstash + Kibana (ELK)• Artifactory• C/C++/Java/Scala/Groovy/Python/TypeScript/JavaScript