INDUSTRY 4.0 – OPEN ARCHITECTURE FOR DATA COLLECTION AND TRANSPORT

Bologna - 17/05/2019

Agenda

› Industry 4.0 – definition and principles

› Architecture for data collection and transport for Industry 4.0

› Enabling technologies

▪ Apache Kafka

▪ Docker

▪ Kubernetes

2

Industry 4.0 – definition and principles

3

Industry 4.0 - Definition

“Industry 4.0 is a term applied to a group of rapid transformations in the design, manufacture, operation and service of manufacturing systems and products […] everything in and around a manufacturing operation (suppliers, the plant, distributors, even the product itself) is digitally connected, providing a highly integrated value chain”

http://www.europarl.europa.eu/RegData/etudes/BRIE/2015/568337/EPRS_BRI(2015)568337_EN.pdf

4

Industry 4.0 - Definition

1st Industrial revolution

1784 – mid 19° century

Water and steam-poweredmechanical manufacturing

2nd Industrial revolution

Late 19° century -1970s

Electric-powered mass production based on the division of labour

3rd Industrial revolution

1970s - Today

Electronics and information technology drives new levels of automation of complex tasks

4th Industrial revolution – Industry 4.0

Today -

Electronics and information technology drives new levels of automation of complex tasks

5

http://www.europarl.europa.eu/RegData/etudes/BRIE/2015/568337/EPRS_BRI(2015)568337_EN.pdf

I4.0 - Cornerstones

The application of information and communication technology (ICT) to digitise information and integrate systems

Cyber-physical systems that use ICTs to monitor and control physical processes and systems. These may involve embedded sensors, intelligent robots that can configure themselves to suit the immediate product to be created, or additive manufacturing (3D printing) devices

Network communications including wireless and internet technologies that serve to link machines, work products, systems and people, both within the manufacturing plant, and with suppliers and distributors

Simulation, modelling and virtualisation in the design of products and the establishment of manufacturing processes

Collection of vast quantities of data, and their analysis and exploitation, either immediately on the factory floor, or through big data analysis and cloud computing

Greater ICT-based support for human workers, including robots, augmented reality and intelligent tools

6

I4.0 - Cornerstones

The application of information and communication technology (ICT) to digitise information and integrate systems

Cyber-physical systems that use ICTs to monitor and control physical processes and systems. These may involve embedded sensors, intelligent robots that can configure themselves to suit the immediate product to be created, or additive manufacturing (3D printing) devices

Network communications including wireless and internet technologies that serve to link machines, work products, systems and people, both within the manufacturing plant, and with suppliers and distributors

Simulation, modelling and virtualisation in the design of products and the establishment of manufacturing processes

Collection of vast quantities of data, and their analysis and exploitation, either immediately on the factory floor, or through big data analysis and cloud computing

Greater ICT-based support for human workers, including robots, augmented reality and intelligent tools

7

Industry 4.0 – Key factors and challenges

▪ Business practices

▪ Corporate standards

▪ Security/safety policies and procedures

▪ Business and Application requirements

▪ Regulatory compliance

▪ Risk management

8

Cornerstones of a joint collaboration

▪ Industry (SACMI)▪ Business domain expertise (regulations, value stream, etc…)▪ Cyber-physical systems expertise

▪ Academia (UniBO)▪ Private/public infrastructure interoperability▪ Protocol interoperability, efficiency, and performance tuning/evaluation▪ Digital Twins

▪ Enterprise Architecture (Imola)▪ Large-scale system design and integration on private/public infrastructures▪ Software development/delivery process governance▪ Data Governance and Cybersecurity

9

Architecture for data collection and transport for Industry 4.0

10

Project goal

Design and implementation of a data collection and transportplatform/architecture for Industry 4.0 scenarios

11

Project requirements

▪ OT principles and regulations should be guaranteed; (Physical) Safety AND Cybersecurity

▪ Heterogeneity: need to accomodate a large number of (mainly legacy) equipments and data formats

▪ Scaling: need to accomodate ever-increasing volumes of connectedequipments (new plants) and message rates

▪ Near real-time processing requirements: business/safety-critical actions happen at the equipment level

12

Data collection and transport architecture – logical layers

13

• Pluggable output towards multiple data stores

• Multiple data analysis and reporting platformsStorage/analysis layer

• Data format modeling and definition

• Data format validation and transformationInteroperability layer

• Asynchronous, publish-subscribe model

• Scalable message transport

Transport/messaging layer

• HW/SW integration via possibly legacy and/or proprietaryprotocols

• Different equipment models and HW/firmware versionsData collection layer

Data collection and transport architecture – architecture

14

Enabling technologies

Apache Kafka

15

Transport layer – Apache Kafka

Apache Kafka is a streaming platform with three key capabilities

▪ Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system

▪ Store streams of records in a fault-tolerant durable way▪ Process streams of records as they occur

Apache Kafka typical use cases▪ Real-time streaming data pipelines used for data aggregation, processing, and transport;▪ Event-reactive streaming applications used for fraud detection, data validation, email sending

confirmation;▪ Applications for real-time data analytics, stream processing, log aggregation, messaging, audit

trail, sync for cooperative nodes.

16

Publish-subscribe model

17

Publisher Subscriber

Topic SubscriberPublisher

Publisher

Subscriber

Topic SubscriberPublisher

Publisher Subscriber

Subscriber

Apache Kafka Topics and Partitions

18

Publisher

Consumer group 1

Topic – Football Teams

Publisher

Publisher

Partition 1 - A-M Team names

Partition 2 - P-Z Team names

offset:0Name: Boca

Juniors

offset:1Name: Arsenal

offset:0Name:

Real Madrid

offset:1Name:

Borussia Dortmund

Consumer group 2

Consumer process

Consumer process

Consumer process

• Partitions and consumer groups allow for scalability

• Partitions are spread across differentmachines

• Partitions split commit log by given criteria• Partitions preserve message order

Apache Kafka – Brokers and Cluster

19

Publisher

Consumer group 1

Publisher

Publisher

Partition 2 - P-Z Team names

offset:0Name:

Real Madrid

Partition 1 - A-M Team names

offset:0Name: Boca

Juniors

offset:1Name: Arsenal

offset:1Name:

Borussia Dortmund

Consumer group 2

Consumer process

Consumer process

Consumer process

BROKER 2

BROKER 1

KAFKA CLUSTER

Apache Kafka – Apache Zookeeper coordination

20

Publisher

Consumer group 1

Publisher

Publisher

Partition 2 - P-Z Team names

offset:0Name:

Real Madrid

Partition 1 - A-M Team names

offset:0Name: Boca

Juniors

offset:1Name: Arsenal

offset:1Name:

Borussia Dortmund

Consumer group 2

Consumer process

Consumer process

Consumer process

BROKER 2

BROKER 1

KAFKA CLUSTERAPACHE ZOOKEEPER

Apache Kafka commit log

21

Interoperability layer – Kafka Connect

APACHE Kafka Connect – framework to define connectors to stream data into and out of Kafka

▪ A common framework for Kafka connectors

▪ REST interface

▪ Automatic offset management

▪ Distributed and scalable

22

Interoperability layer – Kafka Connect

23

Elasticsearch SinkConnector

KafkaTopic

PostgreSQL DB Source Connector

Amazon S3 SinkConnector

Amazon S3

Enabling technologies

Containers & Docker

24

Infrastructure - Containers

25

PHYSICAL INFRASTRUCTURE

HOST OS

HYPERVISOR

GUEST OS

LIBS/BIN

APP

GUEST OS

LIBS/BIN

APP

GUEST OS

LIBS/BIN

APP

PHYSICAL INFRASTRUCTURE

HOST OS

DOCKER ENGINE

LIBS/BIN

APP

LIBS/BIN

APP

LIBS/BIN

APP

VIRTUAL MACHINE

CONTAINER

Infrastructure - Containers

▪ containers include an application/service together with its

dependencies

▪ containers share kernel with other containers

▪ containers run as isolated processes

▪ higher efficiency w/r to virtualization

▪ images are the cornerstone in crafting declarative/automated, easily

repeatable, and scalable services and applications

26

Infrastructure - Docker

An open platform for distributed applications for developers and sysadmins

Docker allows you to package an application with all of its dependencies into a standardized unit for software development.

https://docs.docker.com/engine/

27

What is Docker?

▪ Docker consists of:▪ The Docker Engine - our lightweight and powerful open

source containerization technology combined with a work flow for building and containerizing your applications.

▪ Docker Hub - our SaaS service for sharing and managing your application stacks.

28

Docker inception

29

▪ 2013: Docker comes to life as an open-source project at dotCloud Inc.

▪ 2014: company changed name to “Docker Inc.” and joined the Linux Foundation

▪ 2015: tremendous increase in popularity

▪ Today:

https://blog.docker.com/2018/03/5-years-later-docker-journey/

Docker - Under the hood

▪ Standard Bodies: Open Container Initiative (OCI), Cloud Native Computing Foundation (CNCF)

▪ OCI Image specification

▪ OCI Runtime Specification

▪ runc runtime (formerly libcontainer)

▪ an abstraction/unification layer to decouple Docker from kernel-specific container features (e.g. LXC, libvirt, ...)

▪ The Docker Images:

▪ copy-on-write filesystems (e.g. AUFS)

▪ The Go programming language

▪ a statically typed programming language developed by Google with syntax loosely based on C

30

Docker Architecture

▪ Docker daemon – The Docker daemon listens for Docker API requests and manages Docker objects such as images,

containers, networks, and volumes.

▪ Docker client – The Docker client (docker) is the primary way that many Docker users interact with Docker. When you

use commands such as docker run, the client sends these commands to the docker daemon, which carries them out.

▪ Docker registries – A Docker registry stores Docker images. Docker Hub and Docker Cloud are public registries that

anyone can use, and Docker is configured to look for images on Docker Hub by default. You can even run your own

private registry. Docker registries are the distribution component of Docker.

Docker objects

Docker imagesA Docker image is a read-only template. For example, an image could contain an

Ubuntu operating system with Apache and your web application installed. Images are used to create Docker containers. Docker provides a simple way to build new images or update existing images, or you can download Docker images that other people have already created. Docker images are the build component of Docker.

Docker containersDocker containers are similar to a directory. A Docker container holds everything

that is needed for an application to run. Each container is created from a Docker image. Docker containers can be run, started, stopped, moved, and deleted. Each container is an isolated and secure application platform. Docker containers are the run component of Docker.

Enabling technologies

Kubernetes Container OrchestrationPlatform

33

Infrastructure orchestration ( Kubernetes, Swarm, Mesos)

▪ Manage the lifecycle of execution environments (containers) in a cluster

▪ Check the state of the containers in the worker nodes

▪ Simplify the implementation of :

▪ High availability (HA) with load balancers

▪ advanced deployment strategies:

▪blue/green deployment, canary release, ...

▪rollback in case of problems

▪ health checks

34

Kubernetes container orchestration platform

Kubernetes is an open source system for managing containerized applications across multiple hosts; providing basic mechanisms for deployment, maintenance, and scaling of applications.

Kubernetes builds upon a decade and a half of experience at Google running production workloads at scale using a system called Borg, combined with best-of-breed ideas and practices from the community.

Kubernetes is hosted by the Cloud Native Computing Foundation (CNCF)

https://github.com/kubernetes/kubernetes

35

Infrastructure - Kubernetes

36

Physical Infrastructure

VM VM VM VM VM VM VM VM

Cluster 1 Cluster 2

Container Orchestration Engine

APP1 APP1

Docker runtime Docker runtime Docker runtime Docker runtime Docker runtime

APP1APP2 APP3APP3

Live demo!

37

Data collection and transport architecture screenshots

38

▪ Più di 20 anni di esperienza nell’Enterprise IT

▪ Consulenza e Skill Transfer su Architetture, Integrazione e Processo

▪ OMG Influence Member, JSR 312 Expert Group, CSI, WWISA, OpenESB Key Partner, NetBeansStrategic Partner

▪ La comunita’ italiana dedicata a Java

▪ 10 anni di articoli, pubblicazioni, libri, eventi, training

▪ Dai programmatori agli architetti

▪ Piu’ di 1.000.000 pagine lette al mese

▪ Business partner in progetti con alto grado di innovazione

▪ Padroni in tecnologie e architetture mobile

▪ Competenti in architetture dell‘informazione, UX e Design

39