54
IoT Protocols Integration
Angelo Corsaro, PhD Chief Technology Officer
with
IaaS
analytics
operational systems
information systems
mobile
desktop
web
embeddedfog computing
messaging / data-sharing
Cloud Messaging
Fog
Fog
Storage
IoT Systems are heterogeneous by nature
IoT architectures have to be designed so to facilitate integration
The danger of O(N2) integration should be addressed architecturally
Integration
Stable Core
Define a Stable Core made by a Reference Platform and Data Model.
Integrate always toward the Stable Core
This approach ensure that the integration problem retains O(N) complexity
Stable Core
Stable Core with Vortex
Vortex is a universal data-sharing backbone
Cop
yrig
ht P
rism
Tech
, 201
5
Vortex supports natively a large number of mobile, enterprise, embedded, cloud, web, and analytics computing platforms
Mobile & Web
Embedded
General Purpose
ClouD
Private Cloud
Analytics
Vortex is programming language independent
Programming Langauges
Programming Langauges
import dds import timeif __name__ == '__main__': topic = dds.Topic("SmartMeter", "Meter") dw = dds.Writer(topic) while True: m = readMeter() dw.write(m) time.sleep(0.1)
import dds._import dds.prelude._import dds.config.DefaultEntities._
object SmartMeter { def main(args: Array[String]): Unit = { val topic = Topic[Meter](“SmartMeter”) val dw = DataWriter[Meter](topic) while (true) { val meter = readMeter() dw.write(meter) Thread.sleep(SAMPLING_PERIOD) } }}
Vortex is data-centric and natively supports the definition of canonical data models
Data-Modeling
Data-Modeling
enum UtilityKind { ELECTRICITY, GAS, WATER }; struct Meter { string sn; UtilityKind utility; float reading; float error; }; #pragma keylist Meter sn
Vortex provide an extremely rich technological base to define the infrastructure of an IoT System
Vortex natively supports the definition of canonical data models
In Summary
Integration with Vortex-Gateway
Vortex integration swiss-army knife
Based on Apache Camel and supporting 200+ connectors to protocols, data stores, analytics, etc.
Supports protocol, format and encoding transformation
Extensible to allow quick integration of proprietary technologies
messaging/Data-Sharing
Data Stores
Putting it all Together
IaaS
Storage
Getting Started with Vortex-Gateway
Cop
yrig
ht P
rism
Tech
, 201
5
Architecture
camel-ospl Component
VortexOpenSplice
camel-ddsi Component
Vortex-Cafe
• Based on DataReader & DataWriter • Typed data
• No native or generated code • No marshalling (raw data buffer)
Camel DDS Processors
• DDSI demarshaller / marshaller • Data transformation • Dynamic Poll Enricher • QoS adaptations
Cop
yrig
ht P
rism
Tech
, 201
5
Components: Provide connectivity to a given technology
Endpoints: Represent a source/destination of messages from/to a components
Exchanges: Encapsulate data and meta coming from and end-point
Routes: Define path from input endpoints to output endpoints
Processors: Allow to perform transformation to the data, its format, etc.
Key Elements
endpoint exchange route
processor
Cop
yrig
ht P
rism
Tech
, 201
5
A component is a factory of Endpoint instances. Over 200 components are supported
Endpoints are specified using URI:
- dds:TopicName:DomainID/TopicType?QoS
- cometd://host:port/channelname
- jms:[topic:]destinationName
- [tcp|udp|vm]:host[:port]
- xmpp://host:port/room
- …
Components and Endpoints
Cop
yrig
ht P
rism
Tech
, 201
5
Sample Route in Scala
val route= new RouteBuilder { ! “ddsi:Foo:1/FooType” to “dds:Bar:2/FooType” !} !
DDSI Endpoint ! Topic Name ! DomainID! TypeName!
val route= new RouteBuilder { ! “dds:Foo:1/FooType” to “dds:Bar:2/FooType” !} !
DDS Endpoint ! Topic Name ! DomainID! TypeName!
Cop
yrig
ht P
rism
Tech
, 201
5
EIP provide pre-specified ways of integrating endpoints
The most common integration patterns are supported, such as, multicast, filtering, correlation, content-routing, load-balancing, etc.
Multicast Ex:
Enterprise Integration Patterns
val shapesRoute = new RouteBuilder { ! “dds:Foo:1/FooType” !
"to (“dds:Bar:2/BarType”, “jms:topic:Foo”) !} !
Recipient List Content Based Router
Message FIlter
A B
Correlation ID
Splitter Aggregator
Data Integration
Cop
yrig
ht P
rism
Tech
, 201
5
Data Integration Patterns are a synthesis of the most commonly used techniques for integrating systems that don’t share the same representation for some given entities
Some common Data Integration Patterns are
- Content Enricher
- Content Filter
- Type Transformation
- Canonical Data Model
Data Integration Patterns
Cop
yrig
ht P
rism
Tech
, 201
5
Enriches a data type by adding additional information required by the target system/subsystem
Example: Compute speed and enrich position
Content Enricher
struct CartesianCoord2D { long x; long y; };
struct Dynamics2D { long x; long y; long dx; long dy;
};System A System B
Content Enricher
Cop
yrig
ht P
rism
Tech
, 201
5Projects a data type by removing information not required by the target system/subsystem
Example: Going from Dynamics to Position
Content Filter
struct CartesianCoord2D { long x; long y; };
struct Dynamics2D { long x; long y; long dx; long dy;
};System A System B
Content Filter
Cop
yrig
ht P
rism
Tech
, 201
5
The Canonical Data Model Pattern is commonly used in DDS systems. In this case, data is always transformed into the representation used by the “Canonical Model”. This approach is very useful as it reduces the integration complexity
Example: Coordinate Systems
Canonical Data Model
Type Transformerstruct CartesianCoord2D { long x; long y; };
struct PolarCoord2D { long rho; float theta; };
System A System B
Canonical Data Model
Cop
yrig
ht P
rism
Tech
, 201
5
When using a DDSI endpoint de-marshaling is lazy. Thus in this example data is never de-marshalled but simply sent-over to the target domain, with the result that domain bridging is very efficient!
Topic/Domain Bridging
VortexDomain 1
VortexDomain 0
Topic“Circle”
Topic“Square”
1 val in = "ddsi:Circle:0/ShapeType"2 val out = "ddsi:Square:1/ShapeType"3 4 val shapesRoute = new RouteBuilder {5 in to out 6 }
Cop
yrig
ht P
rism
Tech
, 201
5
process allows to perform arbitrary modifications to the exchange content
As such it can be used to implement arbitrary type transformations
Type Transformation
VortexDomain 1
VortexDomain 0
Topic“Foo”
Topic“Bar”
1 val shapeRoute0 = new RouteBuilder {2 in unmarshal(cdrData) process ({ 3 e => foo2Bar(e.getIn.getBody(classOf[Foo]))})4 to out5 }
Data Policing
Cop
yrig
ht P
rism
Tech
, 201
5
Content Router 1 // Define endpoints 2 val inEndpoint = "ddsi:"+ inTopic +":"+ inDomain +"/" + shapeType 3 val outEndpoint1 = "ddsi:"+ outTopic1 +":"+ outDomain1 +"/" + shapeType 4 val outEndpoint2 = "ddsi:"+ outTopic2 +":"+ outDomain2 +"/" + shapeType 5 6 val cdrData = new DataFormatDefinition(new CDRDataFormat) 7 val shapesRoute = new RouteBuilder { 8 inEndpoint unmarshal(cdrData) choice { 9 when ( e => {10 val s = e.getIn.getBody(classOf[ShapeType])11 if (s.x > s.y) true else false12 }) to outEndpoint113 14 when (e => {15 val s = e.getIn.getBody(classOf[ShapeType])16 if (s.x < s.y) true else false17 }) to outEndpoint218 19 otherwise { to (outEndpoint1, outEndpoint2) }20 }21 22 }
VortexDomain 2
VortexDomain 0
Topic“Circle”
Topic“Circle”
VortexDomain 1
Topic“Circle”
if (x >= y) if (x <= y)
Cop
yrig
ht P
rism
Tech
, 201
5
Load Balancing
VortexDomain 2
VortexDomain 0
Topic“Circle”
Topic“Circle”
VortexDomain 1
Topic“Circle”
load balance
1 // Define endpoints 2 val inEndpoint = "ddsi:"+ inTopic +":"+ inDomain +"/" + shapeType 3 val outEndpoint1 = "ddsi:"+ outTopic1 +":"+ outDomain1 +"/" + shapeType 4 val outEndpoint2 = "ddsi:"+ outTopic2 +":"+ outDomain2 +"/" + shapeType 5 6 // Define a Route using the Java DSL 7 val shapesRoute = new RouteBuilder { 8 override def configure() = 9 from(inEndpoint).sample(period, TimeUnit.MILLISECONDS) 10 loadBalance() roundRobin() 11 to(outEndpoint1, outEndpoint2)12 }
Impedance Adaptation
Cop
yrig
ht P
rism
Tech
, 201
5
Data Sampling
VortexDomain 1
VortexDomain 0
Topic“Circle”
Topic“Circle”
1 per 500 msec
1 // Define endpoints 2 val inEndpoint = "ddsi:"+ inTopic +":"+ inDomain +"/" + shapeType 3 val outEndpoint = "ddsi:"+ outTopic +":"+ outDomain +"/" + shapeType 4 5 // Define a Route using the Java DSL 6 val shapesRoute = new RouteBuilder { 7 override def configure() = 8 from(inEndpoint) sample(period, TimeUnit.MILLISECONDS) 9 to(outEndpoint)10 }
Technology Integration
Cop
yrig
ht P
rism
Tech
, 201
5
JMS Integration
JMS
VortexDomain
Topic“Circle”
Topic“Circle”
To Json
1 val dds = "ddsi:Circle:0/ShapeType"2 val jms =3 "jms:topic:circle?jmsMessageType=Text&deliveryPersistent=false"4 5 val shapeRoute = new RouteBuider {6 from(dds) unmarshal("cdr") marshal().json() to(jms)7 }
Cop
yrig
ht P
rism
Tech
, 201
5
1 // Define endpoints 2 val inEndpoint = 3 "ddsi:"+ inTopic +":"+ inDomain +"/" + shapeType 4 val outEndpoint = 5 "websocket://"+inTopic.toLowerCase + "?sendToAll=true" 6 7 // Define a Route using the Scala DSL 8 val shapesRoute = new RouteBuilder { 9 override def configure() =10 from(inEndpoint) unmarshal("cdr") marshal() json() to(outEndpoint)11 }
WebSocket Integration
VortexDomain
Topic“Circle”
To JSON
Cop
yrig
ht P
rism
Tech
, 201
4
MQTT Integration
MQTT
VortexDomain
Topic“Circle”
Topic“Circle”
To Json
1 val dds = "ddsi:Circle:0/ShapeType"2 val mqtt =3 "mqtt:topic:circle"4 5 val shapeRoute = new RouteBuider {6 from(dds) unmarshal("cdr") marshal().json() to(mqtt)7 }
Cop
yrig
ht P
rism
Tech
, 201
4
HBase IntegrationVortex
Domain
Topic“Circle”
To Json
1 val dds = "ddsi:Circle:0/ShapeType"2 val hbase =3 "hbase:Circle?mappingStrategyName=body&operation=CamelHBasePut""4 5 val shapeRoute = new RouteBuider {6 from(dds) unmarshal("cdr") marshal().json() to(hbase)7 }
Demo
Vortex provides, probably, the best technology base for defining the stable core of an IoT System
Vortex Gateway makes it trivial to integrate other systems, applications and technologies
Vortex
Cop
yrig
ht P
rism
Tech
, 201
5
