Date post: | 17-May-2015 |
Category: |
Technology |
Upload: | angelo-corsaro |
View: | 1,874 times |
Download: | 5 times |
OpenSplice
DDS
Angelo CORSARO, Ph.D.Chief Technology Officer OMG DDS Sig Co-Chair
DDS Made Simple
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
What I’ll Cover
☐ Understand DDS and its Data Sharing abstraction
☐ Learn about the benefits that Data Sharing provide over plain Messaging
☐ Learn how much simpler is to build distributed systems with DDS than with messaging technologies
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Distributed System Definition
A Distributed System is a collection autonomous software components distributed across a network that coordinate their activities and share system resources to run as a “single system”
N.B. It is useful to remark how this definition does not mention message passing or shared memory, those are implementation details
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Distributed Systems Challenges A number of challenges are inherent to the nature of distributed systems:
☐ Transparency
☐ Scalability
☐ Dependability
☐ Performance
☐ Flexibility
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Building Distributed Systems
Several abstractions have been proposed to build Distributed Systems:
☐ Distributed Shared Memory
☐ Message Passing/Queueing
☐ Distributed Objects
☐ Tuple Spaces
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Building Distributed Systems
Several abstractions have been proposed to build Distributed Systems:
☐ Distributed Shared Memory
☐ Message Passing/Queueing
☐ Distributed Objects
☐ Tuple Spaces
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Message Queueing☐ Message Queueing, as
suggested by its name, provide a coordination abstraction based on the exchange of messages between distributed processes
☐ Message delivery is often mediated by brokers
☐ A Message is composed by a header and a body, where the body is opaque (often binary) data
Broker
BodyHeader
Message
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Tuple Space☐ Tuple Space provide the
abstraction of a Global Tuple Space that can be used by autonomous applications to coordinate by inserting (out), reading (rd) and removing (in) tuples
☐ The Tuple Space model was initially introduced by Gelernter and Carrero’s Linda and extended over the years
<a,b,c>
<x,y, <j, k>><x,y, <j, k>>
out(<a,b,c>) rd(<a,_,_>)
in(<15,_>)eval(...)
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Tuple Spaces vs Messaging☐ Tuple Spaces provides spatial
and temporal decoupling along with content awareness
☐ Tuples remain in the tuple space up to when they are not removed
☐ Tuple Spaces allow application to share information
☐ Overall Tuple Spaces provide a high-level abstraction for building distributed systems
☐ Messaging doesn’t provide content awareness and usually provides limited temporal decoupling
☐ Messages usually disappear once they are delivered to consumers
☐ Messaging allows application to exchange messages. Information has to be reconstructed from messages
☐ Messaging provides a relatively low-level abstraction for building distributed systems
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Why is Messaging so Popular?At this point you may ask yourself:
If Tuple Spaces are so cool, why are messaging technologies so popular?
The answer is actually pretty simple:
Tuple Spaces where originally quite inefficient and this lead people to adopt different approaches, such as messaging and distributed objects.
Nowadays, however, there are plenty of technologies deriving from the tuple space concept that have very high performance (often superior than messaging).
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Why is Messaging so Popular?
At this point you may ask:
OK, that makes sense. Yet there are still plenty of people using messaging technologies. Why?
The answer is again pretty simple:
The adoption of Tuple Space inspired technologies is growing steadily, but many people are still using messaging since this is what they are familiar with. But enlightenment will eventually hit them ;-)
OpenSplice
DDS
DDS
OpenSplice
DDS Ba
DDSics
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Data Distribution Service
☐ Topics: Unit of data sharing
☐ DataWriters: data producers
☐ DataReaders: data consumers
DDS provides a relaxed Tuple Space Abstraction abstraction based on:
DDS Global Data Space
...
TopicA
TopicBTopicC
TopicD
Data Writer
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Reader
For Real-Time Systems
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Data Distribution Service
☐ DataWriters and DataReaders are automatically and dynamically matched by the DDS Dynamic Discovery
☐ A rich set of QoS allows to control existential, temporal, and spatial properties of data
DDS Global Data Space
...
TopicA
TopicBTopicC
TopicD
Data Writer
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Reader
For Real-Time Systems
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Fully Distributed Tuple Space☐ DDS Implements a fully
distributed Tuple Space with relaxed consistency.
☐ DDS provides eventual consistency as opposed to strong consistency
☐ In addition, DDS provides a local “take” and a global “dispose” in place of the “in” operation
<a,b,c>
<x,y, <j, k>>
<x,y, <j, k>>
out(<a,b,c>) rd(<a,_,_>)
in(<15,_>)eval(...)
Traditional Tuple Space
DDS Data SpaceLogical Global Data Space
Physical Local Data Space
The local Data Space is a projection of the Logical Global Data Space. This projection reflects the interest of the given
application.
The Global Data Space is an abstraction in DDS built by
the composition of Local Data Spaces.
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
What DDS Provides☐ Location Transparency => Dynamic Discovery
☐ Anonymity and Temporal Decoupling => Tuple Space
☐ Data Centricity => Tuple Space
☐ Dependability => Fault-Tolerant and Secure
☐ Scalability => Fully distributed architecture easy to scale-out
☐ Portability => OS and Programming Language Independence
☐ Interoperability => Standardized Wire Protocol
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
IDL
DDS Topics☐ A Topic defines a class of
streams ☐ A Topic has associated a user
defined extensible type and a set of QoS policies
☐ The Topic name, type and QoS defines the key functional and non-functional invariants
☐ Topics can be discovered or locally defined
DURABILITY,DEADLINE,PRIORITY,
…
“TVehicleDynamics”
TopicTypeName
QoS
struct VehicleDynamics { long vid; long x; long y; long dx; long dy;};#pragma keylist VehicleDynamics vid
VehicleDynamics
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Instances
☐ Each unique key value identifies a unique topic instance
☐ DDS not only demultiplexes “streams” but provides also lifecycle information
☐ A DDS DataWriter can write multiple instances struct Person {
long ssn; String name; String surname; };#pragma keylist Person ssn
<101,Leslie, Lamport >
<010,Edsger, Dijkstra>
<110,Haskell, Curry>
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Anatomy of a DDS Application
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Running App
☐ During this webcast we’ll be looking at a few DDS aspects through the iShapes applications
☐ The advantage of targeting the iShapes application is that we can actually “see” what our application will do
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
OpenSplice Mobile
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
In Action...
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
iShapes Information Model“Circle”
TopicType
NameQo
S
ShapeType
IDLstruct ShapeType { string color; long x; long y; long shapesize;};#pragma keylist ShapeType color
“Square”
TopicType
Name
QoS
ShapeType
“Triangle”
TopicType
Name
QoS
ShapeType
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Anatomy of a DDS ApplicationDomain Participant
[ISO C++ API]
auto dp = DomainParticipant(domainId);
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Anatomy of a DDS ApplicationDomain Participant
Topics
auto dp = DomainParticipant(domainId);
// Create a Topicauto topic = Topic<ShapeType>(dp, “Circle”);
[ISO C++ API]
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Anatomy of a DDS ApplicationDomain Participant
Topics
auto dp = DomainParticipant(domainId);
// Create a Publisherauto pub = Publisher(dp);
// Create a Topicauto topic = Topic<ShapeType>(dp, “Circle”);
Publisher
// Create a Subscriberauto sub = Subscriber(dp);
Subscriber
[ISO C++ API]
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Anatomy of a DDS ApplicationDomain Participant
Data Writer
Topics
// Create a DataWriterauto writer = DataWriter<ShapeType>(pub, topic);
auto dp = DomainParticipant(domainId);
// Create a Publisherauto pub = Publisher(dp);
// Create a Topicauto topic = Topic<ShapeType>(dp, “Circle”);
Publisher
// Create a Subscriberauto sub = Subscriber(dp);
Subscriber
Data Reader // Create a DataWriterauto reader = DataReader<ShapeType>(sub, topic);
[ISO C++ API]
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Putting it all together 1 int main(int argc, char* argv[]) { 2 try { 3 DomainParticipant dp(0); 4 Topic<ShapeType> topic(dp, "Circle"); 5 Publisher pub(dp); 6 DataWriter<ShapeType> dw(pub, topic); 7 10 for (int i = 0; i < N; ++i) {11 ShapeType bc = {"RED", i, i, 60};12 ShapeType rc = {"BLUE", N-i, N-i, 60};13 dw.write(bc);14 // You can also write with streaming operators!15 dw << rc;16 std::cout << "." << std::flush;17 usleep(sleep_time);18 }19 20 } catch (const dds::core::Exception& e) {21 std::cout << e.what() << std::endl;22 }23 return 0;24 }
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Anatomy of a DDS Application
Domain Participant
Data Writer
Topics
// Create a DataWriterDataWriter<ShapeType> writer = pub.createDataWriter<ShapeType>(topic);
DomainParticipant dp = dpf.createParticipant(domainId);
// Create a PublisherPublisher pub = dp.createPublisher();
// Create a TopicTopic<ShapeType> topic = dp.createTopic(“Circle”,ShapeType.class);
Publisher
// Create a SubscriberSubscriber sub = dp.createSubscriber();
Subscriber
Data Reader
[Java 5 API] BootstrapServiceEnvironment env = ServiceEnvironment.createInstance(this.getClass.getClassLoader);DomainParticipantFactory dpf = DomainParticipantFactory.getInstance(env)
// Create a DataWriterDataReader<ShapeType> reader = sub.createDataReader<ShapeType>(topic);
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Anatomy of a DDS ApplicationDependency Injection through Implicits
Data Writer
Topics
// Create a DataWriterval writer = DataWriter[ShapeType](topic)
[Scala API]
import dds.config.DefaultEntities._
// Create a Topicval topic = Topic[ShapeType](“Circle”)
Data Reader // Create a DataWriterval reader = DataReader[ShapeType](topic)
OpenSplice
DDS
Writing Data
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Writing Data in DDS
writer.write(SpapeType(“RED”, 10, 20, 30));
// -- or -- writer << SpapeType(“RED”, 10, 20, 30);
[ISO C++ API]
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Writing Data in DDS[Java 5 API]
writer.write(new SpapeType(“RED”, 10, 20, 30));
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Writing Data in DDS[Scala API]
writer.write(new ShapeType(“RED”, 10, 20, 30))
// -- Or equivalently:
writer write (new ShapeType(“RED”, 10, 20, 30))
OpenSplice
DDS
Reading Data
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Reading Data// Read Dataauto data = reader.read();
// -- you can also read with streaming operatorsauto data = reader << read();
// Print the received data on the consolestd::for_each(samples.data().begin(), samples.data.end(), demo::printShape)
[ISO C++ API]
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Reading Data// You can read data on your favorite container using iteratorsauto samples = std::vector<Samples<ShapeType>>(MAX_SHAPES);
// Read data into the provided containerreader.read(samples.begin(), MAX_SHAPES);
// Print the received data on the consolestd::for_each(samples.data().begin(), samples.data.end(), demo::printShape)
[ISO C++ API]
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Reading Data
// Read Data Sample.Iterator<ShapeType> iterator = reader.read(); while (iterator.hasNext()) { // Assuming ShapeType ovrrides toString
out.println(iterator.next().getData()); }
[Java API]
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Reading Data
// Read Data reader.read foreach { s => println(s.data()) }
[Scala API]
OpenSplice
DDS
Selecting Samples
OpenSplice
DDS
Data Selectors
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Cherry Picking in DDS
☐ DDS provides some very flexible mechanisms for selecting the data to be read:☐ Data Content☐ Data Status
☐ These mechanisms are composable
OpenSplice
DDS
Content-Based Data Selection
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Filters and Queries☐ DDS Filters allow to control what gets
into a DataReader cache
☐ DDS Queries allow to control what gets out of a DataReader cache
☐ Filters are defined by means of ContentFilteredTopics
☐ Queries operate in conjunction with read operations
☐ Filters and Queries are expressed as SQL where clauses
DataReader Cache
DataReader
...... ... ...
Filter
Query
Application
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Querystruct VehicleDynamics { long vid; long x; long y; long dx; long dy;};#pragma keylist VehicleStatus vid
// Define the query and the parameters std::vector<std::string> p;p.push_back("100");p.push_back("100");dds::core::Query q("x < %0 AND y < %1", p.begin(), p.end());
auto data = reader .select() .content(q) .read();
[DDS C++ API 2010]
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Instances☐ DDS provides a very efficient way of reading data belonging to a
specific Topic Instance
☐ Obviously, one could use queries to match the key’s value, but this is not as efficient as the special purpose instance selector
// C++auto data = reader .select() .instance(handle) .read();
// Scalaval data = reader read(handle)
OpenSplice
DDS
State-Based Selection
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Sample, Instance, and View State☐ The samples included in the DataReader cache have associated
some meta-information which, among other things, describes the status of the sample and its associated stream/instance
☐ The Sample State (READ, NOT_READ) allows to distinguish between new samples and samples that have already been read
☐ The View State (NEW, NOT_NEW) allows to distinguish a new instance from an existing one
☐ The Intance State (ALIVE, NOT_ALIVE_DISPOSED, NOT_ALIVE_NO_WRITERS) allows to track the life-cycle transitions of the instance to which a sample belongs
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
State Selector in Action
// Read only new samplesauto data = reader .select() .state(status::DataState::new_data()) .read()
// Read any samples from live instancesauto data = reader .select() .state(status::DataState::any_data()) .read()
C++// Read only new samplesval data = reader read
// Read any samples from live instancesval data = reader read(SampleSelector.AnyData)
Scala
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Putting all Together
☐ Selectors can be composed in a flexible and expressive manner
C++auto data = reader ! .select() .instance(handle) ! ! .state(status::DataState::new_data()) ! ! .content(q) .read();
OpenSplice
DDS
QoS
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
QoS Model☐ QoS-Policies control local and
end-to-end properties of DDS entities
☐ Local properties controlled by QoS are related resource usage
☐ End-to-end properties controlled by QoS are related to temporal and spatial aspects of data distribution
☐ Some QoS-Policies are matched based on a Request vs. Offered Model thus QoS-enforcement
Publisher
DataWriter
Topic
Type
QoS
Name
writes
QoS
DataWriter
Topic
Typewrites
Subscriber
DataReaderreads
DataReaderreads
...
QoS
Name
QoS
QoS QoS
QoS matching
......
QoS QoS
Type Matching
DomainParticipant DomainParticipant
QoS QoS
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
QoS PoliciesQoS Policy Applicability RxO Modifiable
USER_DATATOPIC_DATA
GROUP_DATADURABILITYDURABILITY
SERVICEHISTORY
PRESENTATIONRELIABILITYPARTITION
DESTINATION ORDER
LIFESPAN
DP, DR, DW N Y
ConfigurationT N Y ConfigurationP, S N Y
Configuration
T, DR, DW Y N
Data AvailabilityT, DW N N
Data Availability
T, DR, DW N N
Data Availability
P, S Y N
Data Delivery
T, DR, DW Y N
Data DeliveryP, S N Y Data DeliveryT, DR, DW Y N
Data Delivery
T, DW N Y
Data Delivery
[T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant]
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
QoS PoliciesQoS Policy Applicability RxO ModifiableDEADLINELATENCY BUDGET
TRANSPORT PRIORITY
TIME BASED FILTER
OWNERSHIPOWNERSHIP STRENGTHLIVELINESS
T, DR, DW Y Y
Temporal/Importance
Characteristics
T, DR, DW Y YTemporal/
Importance Characteristics
T, DW N YTemporal/
Importance Characteristics
DR N Y
Temporal/Importance
Characteristics
T, DR, DW Y NReplicationDW N Y Replication
T, DR, DW Y N Fault-Detection
[T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant]
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Setting QoS Policies
C++// Setting Partition QoS-Policy on Publisherqos::PublisherQos pubQos;pubQos << policy::Partition("Partition");Publisher pub(dp, pubQoS);
// Setting various QoS-Policy on a Topicqos::TopicQos tqos;tqos << policy::Reliability::Reliable() << policy::Durability::Transient() << policy::History::KeepLast(5);
Topic<VehicleDynamics> topic(dp,"Partition", tqos);
OpenSplice
DDS
Ipse Dixit
For non latin speakers: "He, himself, said it"
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
The “Ipse Dixit” Dejavu☐ As an historical reference“Ipse
Dixit” was used by the Holy Inquisition to push back any argument that would contrast with established knowledge
☐ A famous example was the argument around the Geo-Centric model of the Solar System
☐ What has this to do with Middleware... Let me show you!
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
What’s Wrong with this Paper?☐ When looking at DDS the
Authors did not refer to the state of the art
☐ DDS is discarded as being too complicated (really?!?!) -- they did not check out the new APIs available since 2010
☐ Beyond that, I think the authors where comparing a bit apples and oranges since DDS provides far more than ZeroMQ!
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
DDS vs. 0MQ -- What’s Simpler?import dds.config.DefaultEntities._
// Create a Topicval topic = Topic[ShapeType](“Circle”)
// Create a DataWriterval writer = DataWriter[ShapeType](topic)
// Create a DataWriterwriter write (new ShapeType(“RED”, 1,2,3))
import zmqimport time
// Create a Contextcontext = zmq.Context()
// Create a Socketpublisher = context.socket (zmq.PUB)
// Bind the socket to a namepublisher.bind ("ipc://nasdaq-feed")
// Send Datapublisher.send (message)
Sending Data
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
DDS vs. 0MQ -- What’s Simpler?import zmqimport time
// Create a Contextcontext = zmq.Context()
// Create a Socketsubscriber = context.socket (zmq.SUB)
// !!! MANUALLY CONNECT IT !!! What if your // topology changes?!?subscriber.connect ("tcp://192.168.55.112:5556")subscriber.connect ("tcp://192.168.55.201:7721")
// Define subscrptionsubscriber.setsockopt (zmq.SUBSCRIBE, "NASDAQ")
// Receive messagemessage = subscriber.recv()// ... Now decode the msg and do something // with it
Receiving Data
import dds.config.DefaultEntities._
// Create a Topicval topic = Topic[ShapeType](“Circle”)
// Create a DataReaderval reader = DataReader[ShapeType](topic)
// Create a DataWriterreader.read() foreach(s => println(s.data))
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Concluding Remarks
☐ DDS provides a powerful and feature-rich abstraction for building distributed systems
☐ This technology is widely used in mission and business critical systems and it being swiftly adopted in data-centric/big-data systems
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice
DDS
¥@prismtech
¥@acorsaro
¥youtube.com/opensplicetube ¥slideshare.net/angelo.corsaro
¥ [email protected]¥[email protected]
¥opensplice.com ¥forums.opensplice.org
¥opensplice.org ¥[email protected]
:: Connect with Us ::