The Industrial Internet of Things as an enabler for aCircular Economy

Hy-LP: A novel IIoT Protocol, evaluated on a Wind Park’sSDN/NFV-enabled 5G Industrial Network

George Hatzivasilisa,b,∗, Konstantinos Fysarakisa, Othonas Soultatosa, IoannisAskoxylakisa, Ioannis Papaefstathioub, Giorgos Demetriouc

aICS-FORTH, Vassilika Vouton, 70013, Heraklion, Crete, GreecebDepartment of Electrical & Computer Engineering, Technical University of Crete, Akrotiri

Campus, 73100, Chania, Crete, GreececEcole des Ponts Business School, 77 rue de Miromesnil, 75008, Paris, France

Abstract

Smart interconnected devices, including Cyber-Physical Systems (CPS), perme-ate our lives and are now an integral part of our daily activities, paving the waytowards the Internet of Things (IoT). In the industrial domain, these devicesinteract with their surroundings and system operators, while often also inte-grating industrial cloud applications. This 4th Industrial Revolution guides newinitiatives, like the introduction of 5th Generation Mobile Networks (5G), toimplement flexible, efficient, QoS- and energy- aware solutions that are capableof serving numerous heterogeneous devices, bringing closer the vision of a sus-tainable, Circular Economy. However, the lack of interoperable solutions thatwill accommodate the integration, use and management of the plethora of de-vices and the associated services, hinders the establishment of smart industrialenvironments across the various vertical domains. Motivated by the above, thispaper proposes the Hy-LP - a novel hybrid protocol and development frameworkfor Industrial IoT (IIoT) systems. Hy-LP enables the seamless communicationof IIoT sensors and actuators, within and across domains, also facilitating theintegration of the Industrial Cloud. The proposed solution is compared withexisting standardised solutions on a common application, working around theprotocols’ intrinsic characteristics and features to produce each variant. Thedeveloped systems are evaluated on a common testbed, demonstrating that theproposed solution is around 10 times faster for the same CPU usage level, whileconsuming 7 times less memory. Moreover, the applicability of the proposedsolutions is validated in the context of a real industrial setting, analyzing thenetwork characteristics and performance requirements of an actual, operatingwind park, as a representative use case of industrial networks.

∗George HatzivasilisEmail address: hatzivas@ics.forth.gr (George Hatzivasilis)

Preprint submitted to Journal of Computer and Communications networksJanuary 14, 2018

Keywords: Circular Economy, Sustainability, 5G, Industrial Internet ofThings, IIoT, Industrial Cloud, CoAP, DPWS, M2M, MQTT, lightweightinteractions

1. Introduction

By 2020, the Internet of Things (IoT) will have an economic impact of morethan 14 trillion1 with around 34 billion devices being deployed2 in a variety ofdomains (smart grid, city infrastructure, transportation, residential/home au-tomation, industrial systems, military, and healthcare, among others [1, 2, 3, 4]).The convergence of the digital with the physical worlds and the establishmentof Cyber-Physical Systems (CPS) unleashed an entirely new potential for theEuropean industry that can be utilized in the effort of becoming a resilient,competitive and resource efficient economy.

Research and industry struggle to integrate this evolving technology and theexponential growth of IoT. Existing networking [3] and security mechanisms [5]are adapted to handle the vast population of IoT devices. Moreover, as highvolumes of data are produced and processed, the integration of IoT solutionswith cloud computing is becoming imperative [6, 7, 8, 9, 10]. Seamless andhigher level machine to machine (M2M) and human-to-machine (H2M) inter-actions, are another important requirement in order to effectively monitor andmanage the infrastructure, allowing the use of its full potential. The upcom-ing 5G technology tries to tackles these issues. The main design goals towardsthis new shift in technology include among others large network coverage withhigh data speed, energy efficient installations, and increased inter-organizationcommunication between the machines [11, 12, 13, 14].

Typically, end-users do not possess the skills to configure and setup thedevices that may be found in smart environments; in large-scale deployments,individually setting up devices is not even feasible. From the perspective ofimplementers, there is a need for rapid development and deployment, whilesimultaneously tackling issues of scaling and inherent limitations in terms ofresources (CPU, memory, power etc.). However, at its current state, the ubiq-uitous computing landscape is segregated, consisting of numerous proprietarysolutions, which are typically incompatible with each other. This makes settingup, managing and securing a smart device ecosystem, significantly challenging.Various IoT protocols proposed by researchers aim to address these issues, whilestandardization initiatives try to guarantee interoperability, through the wideand structured deployment of the proposed mechanisms.

The interplay of IoT with other domains such as the Circular Economy pro-vides a fertile ground for innovation and value creation. Circular economy value

1GOV2020: http://government-2020.dupress.com/wp-content/uploads/2014/11/Cyber-Physical-+-Sources-9-16-14 AR.pdf

2Business Insider: http://www.businessinsider.com/how-the-internet-of-things-market-will-grow-2014-10

2

drivers include extending the useful life of finite resources and maximising theutilisation of assets, creating an emerging class of looping assets, and regen-erating natural capital for more effective and efficient use. IoT value driversenable for a new window of opportunity to be opened on the economic cycleas it becomes a serving enabler by collating knowledge about asset locations,conditions, quality and performance in real time and over time. IoT has beencatalytic already in presenting solutions to many resource related challengesthat have been faced by circular economy innovators. Moreover, the feedback-rich nature of circular economy models might conversely make them particularlysuitable to help extract value from the large amount of data generated by IoT.

Motivated by the above issues, this paper presents the Hy-LP – a hybridprotocol and development framework for the Industrial IoT. The main contri-butions in the field include:

• The seamless operation of IIoT devices and services within and acrossdomains

• Integration of IoT deployments with the Industrial Cloud and its associ-ated applications

• Lightweight design, considering the limitations of constrained, heteroge-neous embedded devices

• Integration of established and mature communication security solutions(e.g. SSL/TLS)

• Enabling the shift to Green networking and a sustainable, Circular Econ-omy, via the increased efficiency and usability over existing solutions

Other prominent approaches that provide seamless and lightweight IoT inter-actions are also identified, highlighting a representative, standardized protocolfor each of these approaches. Moreover, a CPS featuring M2M interactionsis selected, designing and implementing the required entities and interactions,while adapting them to the intricacies of each protocol. Finally, the developedsolutions are deployed and evaluated on a common testbed, while the perfor-mance is also assessed in the context of the network characteristics recorded onan actual, operating wind park, allowing valuable conclusions to be extracted.

The paper is organized as follows: Section 2 presents and compares widelyused solutions for IoT interconnection and management. Section 3 details theproposed Hy-LP protocol. Section 4 evaluates the performance of our system,compared with other settings, over a real IoT application. Section 5 concludesand refers future work.

2. Current IoT Solutions

Surveying the academic literature and the web, reveals a plethora of proto-cols aiming to unify IoT devices and applications [15, 16, 17, 18]; some are stillin their infancy, some are openly available, some are proprietary, and there are

3

also significant efforts to standardize some protocols. The main approaches aredetailed in the subsections below.

2.1. Synchronous Communication

An interesting approach for IoT interactions is the use of protocols follow-ing the Representational State Transfer (REST) architecture, which currentlydominates the World Wide Web. RESTful implementations typically use theHypertext Transfer Protocol (HTTP), but the latter is not appropriate for IoTapplications, when considering the resource, bandwidth and energy restrictionsof the target devices.

Thus, the Internet Engineering Task Force (IETF) Constrained RESTfulenvironments (CoRE) Working Group presented the Constrained ApplicationProtocol (CoAP [15]), now an IETF standard. CoAP is a specialized web trans-fer protocol for use with constrained nodes and constrained networks in theIoT, aiming to maintain compatibility with the existing Internet infrastructure,through simple proxies. The protocol is often referred to as ”the HTTP for theInternet of Things”. It follows a request/response model, where a client mayinteract with the server using a subset of the HTTP methods, namely usingGET, PUT, POST and DELETE on the server’s resources. CoAP messagesare transported over UDP. The protocol features two layers: the Transactionlayer responsible for single message exchange between end points and the Re-quest/Response layer which is responsible for request/response transmission andresource management; thus providing reliability mechanisms and basic conges-tion control. Moreover, basic publish/subscribe interactions are also supported,as, by extending the HTTP GET method, a client can observe a specific re-source. For security, CoAP applications support DTLS.

There is significant research interest in CoAP, with numerous efforts to lever-age its extremely lightweight interactions in domains such as smart homes [19],mobile IoT deployments [20], cloud services [21], healthcare [22], smart cities[23] and industrial WSNs [24].

2.2. Asynchronous Communication

Message-oriented protocols typically focus on providing asynchronous datatransfers between distributed devices. Their focus is on reliable messaging,including message buffers and Quality of Service (QoS) facilities, controlled bycentralized entities.

The Advanced Message Queuing Protocol (AMQP) implements this func-tionality and is standardized by the Advanced Open Standards for the Infor-mation Society (OASIS)3. It is designed for reliable communication via messagedelivery guarantee primitives, like at-most-once, at-least-once, and exactly-oncedelivery, and it is built upon a reliable transport protocol, such as TCP. Theprotocol consists of two core components that handle communication: the ex-changes and the message queues. Based on pre-defined rules, the exchanges

3OASIS AMQP: https://www.oasis-open.org/committees/tc home.php?wg abbrev=amqp

4

route the messages to appropriate queues, which can store the data and latersend it to the receivers. Moreover, AMQP provides a publish/subscribe commu-nication model by defining a messaging layer on top of the underlying transportlayer. Two message types are supported, the bare messages and the annotatedmessages respectively. The first type is supplied by the sender while the secondtype is seen at the receiver. In comparison with the REST approach, AMQPperforms better under a high volume of message exchanges [25]. The protocolis mainly used in business messaging [26] while in the IoT context it is suitablefor the control plane of the server-based analysis functions [25].

The MQ Telemetry Transport (MQTT [16]) is another message-oriented pro-tocol, introduced by IBM in 1999 and recently standardized by OASIS, as theIoT developments brought it back into the limelight. It is also standardizedas ISO/IEC 209224. MQTT was designed as an extremely lightweight pub-lish/subscribe messaging transport, for small sensors and mobile devices, opti-mized for high-latency or unreliable networks. An MQTT Broker is responsiblefor handling and organizing all communications between the various devices.Messages are published with specific topics, and each client can subscribe to var-ious topics (though the Broker may require username/password authenticationbefore allowing subscription). Topics are organized in a hierarchical manner,like the folder structure in a file system; e.g. home/kitchen/oven/temperaturecould be a topic where a device can subscribe to get updates on the oven’stemperature. When a client publishes a message, the Broker then relays thismessage to all clients subscribed to the message’s topic. Thus, all interac-tions are asynchronous and clients only communicate directly with the Broker.MQTT relies on TCP and secure deployments support the use of TLS. As withthe previous protocol, researchers have already studied MQTT in a variety ofdomains, including eHealth applications [27], [28], WSNs and smart grid [29],smart homes [30] and also mobile IoT contexts [31], among others. In com-parison with AMQP in the IoT domain, MQTT is more suitable as it scalesbetter and requires less computational resources and less effort to implement ona client [32].

The eXtensible Messaging and Presence Protocol (XMPP) is developed forinstant messaging and connects people via text messages over TCP. It is stan-dardized by the IETF as the RFC 6120 [33] and is mainly utilized for chattingand other message exchange applications. In the IoT domain, XMPP imple-ments an easy way to address devices [34] and has been used as a protocol forSoftware-Defined Networking (SDN). However, is was designed for near real-time, human-to-human (H2H) interaction, and it is not practical for M2M set-tings as it does not provide any quality of service guarantees. Moreover, themessages are formatted in XML, which further increases the computational andcommunication overhead due to lots of headers and tag formats. Thus, XMPP israrely utilized in IoT applications due higher latency and resource consumption.

4ISO/IEC 20922: http://www.iso.org/iso/catalogue detail.htm?csnumber=69466

5

2.3. Service-Oriented Approach

Service Oriented Architectures (SOAs) provide an attractive option for IoTnode interactions. This approach has already been successful in business envi-ronments, as web services allow stakeholders to focus on the services themselves,rather than the underlying hardware and network technologies. The OASISstandard Devices Profile for Web Services (DPWS [17]) is the most attractivesolution. It should be noted that DPWS was originally conceived and intro-duced as a successor to UPnP, but nowadays is actively pushed by industrystakeholders as the solution of choice for large-scale enterprise (e.g. industrial)deployments, while UPnP is mostly targeted to the home environment (printers,home entertainment etc. [35]). Also, like UPnP, DPWS is natively integratedinto the various versions of the Windows operating system.

The specification defines a minimal set of implementation constraints toenable secure Web Service messaging, including discovery, description, syn-chronous (via operation invocations) and asynchronous (via subscription andevent-driven changes) interactions on resource-constrained devices. The pro-file’s architecture includes hosting and hosted services. A single hosting serviceis associated with each device while the same device may accommodate varioushosted services. The latter represent the device’s various functional elementsand rely on the hosting service for discovery. For security, DPWS supports theuse of TLS. The mechanisms detailed in the WS-Security specification are alsoapplicable, as with any other Web Services deployment.

DPWS enables the adoption of a SOA approach on embedded and sensordevices with limited resources, allowing system owners to leverage the SOAbenefits across heterogeneous systems that may be found in smart environments.The use and benefits of DPWS have been studied extensively in the contextof various applications areas, which, other than the ones already mentioned,include automotive [37] and railway systems [38], industrial automation [39],eHealth [40], smart cities [41] and smart homes [42] and buildings [43].

2.4. Comparison

CoAP, MQTT, and DPWS share some important characteristics which makethem good candidates for IoT and Industrial IoT (IIoT) applications, and whichmotivated us to consider them for this study. More specifically, all three pro-tocols: are open standards with significant traction in the research communityand the industry; are designed with constrained environments in mind; can of-fer seamless M2M interactions; run on IP; and have a range of implementationsreadily available to developers and researchers alike. While CoAP messagesare transported over UDP, MQTT relies on TCP, and DPWS uses a combina-tion of both (TCP for the bulk of the device interactions, and UDP for devicediscovery and other auxiliary functions); with each protocol inheriting differentcharacteristics from the underlying transport mechanisms. For example, DPWSis constrained for local network communication only, as the implication of re-lying the multicasting features of UDP restricts the standard’s application forInternet-wide. The underlying protocols also affects the available security mech-anisms, with DPWS and MQTT deployments supporting the use of TLS, and

6

CoAP applications supporting DTLS. In the case of DPWS, the mechanismsdetailed in the WS-Security specification are also applicable, as with any otherWeb Services deployment. MQTT is suited, by design, for publish/subscribeinteractions, CoAP also has support for observing resources, partly coveringsuch functionality, but it is better suited for synchronous interactions, insteadof event-based ones. DPWS is more flexible in this regard, as the WS-Eventingspecification enables a feature-rich publish/subscribe functionality, includinginteractions that are triggered at pre-defined intervals and/or when a specificevent takes place. Moreover, QoS is an important aspect in MQTT, with theprotocol supporting three different modes of message delivery (Fire and forget,Delivered at least once and Delivered exactly once), whereas CoAP only offers arudimentary choice between Confirmable and Non-confirmable messages. Theformer have to be acknowledge by the received with an ACK packet, in applica-tions where it is necessary to cater for UDP’s unreliable transport. DPWS hasno such features built-in, relying solely on TCP’s delivery mechanisms. Variousextensions enhance the reliability and QoS features of Web Services (e.g. [35]),but these have not been integrated into DPWS yet.

A detailed and hands-on comparison of the protocols in the context of anactual application, as presented in [36], revealed that, performance-aside, DPWSwas the benchmark in terms of the ease in designing the framework. Its robustand flexible discovery, subscription and eventing mechanisms meant that theentities and their interactions could be designed in an intuitive manner. Thisis also true for the end application, as it is the most hassle-free variant fromthe end users’ perspective; minimal setup is required and the entities discovereach other and interact seamlessly, no matter where they are deployed on thenetwork. CoAP was intuitive to work with, especially considering that as mostdevelopers nowadays have experience with RESTful applications. Still, carefulstudy of the protocol and its limitations (theoretical and/or in terms whatis supported in the existing APIs) is needed, as it is not as mature as theother two protocols considered. Lastly, MQTT’s lack of synchronous interactionsupport, meant that we had to follow a not so elegant approach in designing theentities’ interactions, with too many interactions happening in order to bypassthe limitation of only supporting asynchronous interactions that have to berouted through a Broker.

3. The Hy-LP Protocol

As indicated by the comparison above, the protocol choice necessitates care-ful consideration of the target application, as no ideal protocol exists; someprotocols have more features and are more mature than the alternatives, whileothers are more lightweight, some are ideally suited to aggregating data from avariety of sensors, while others are better suited for end-user (e.g. consumer)applications, etc. Thus, a complex deployment could benefit from the use morethan one protocol or the development of a custom protocol that combines thebenefits and alleviates the disadvantages of the standalone protocols. The latter

7

approach does not fully sacrifice interoperability with existing solutions, com-bining one or more protocols, delegating to each one a task that it’s more suitedfor. Thus, for example, CoAP could be used for the lightweight M2M inter-actions it can provide, MQTT for the cross-domain communications and theSOA-based approach of DPWS could be used for M2H interactions (to leveragethe ubiquity of web service support in all human-operated devices).

Motivated by this insight, we propose the Hy-LP – a novel service-orientedframework for the IoT setting. The framework addresses the main constraintsof the individual protocols, enabling seamless communication of IoT and IIoTdevices and actuators over the Internet. Hy-LP is then directly compared toDPWS, since, as described above, it stood out in previous works as the mostflexible, mature and feature-rich protocol, despite the inherent performance lim-itations and LAN-only operation.

To overcome the latter, instead of HTTP and UDP multicasting of DPWS,Hy-LP leverages the CoAP and MQTT protocols for synchronous and asyn-chronous communication respectively. Moreover, Hy-LP inherits the Broker ar-chitecture of MQTT. Messages are passed through a central server (the Broker),enabling one-to-many and many-to-many communications. A publish/subscribeprotocol is constructed, which accomplishes service/device discovery and event-ing functionality.

Knowledge of the topics and the message format is assumed. As aforemen-tioned, DPWS utilizes SOAP to form messages. Our framework utilizes theJavaScript Object Notation (JSON) open-standard format (a replacement ofXML) to structure the messages’ context. JSON is simpler, human-readableand consumes less resources than XML [44]. The average packet size of Hy-LP

Figure 1: Hy-LP setting and operation mapping.

8

is around six times smaller than the relevant DPWS packets, resulting in lowerprocessing time, memory requirements and energy consumption. Moreover, QoSis an important aspect, with the Hy-LP framework supporting three differentmodes of message delivery, similar with MQTT (Fire and forget, Delivered atleast once, and Delivered exactly once). Figure 1 illustrates a typical Hy-LPdeployment.

First, the devices publish their profile information to the broker, includ-ing the relevant IP address (in contrast to the local network addresses of theDPWS multicast). The broker can be either local or remote, enabling cross-domain interaction. In order to discover a device or service, the actuator sendsa request message to all public devices through the broker, who implements thecorresponding multicasting functionality. The compatible entities respond tothe request by sending descriptive metadata. Figure 2 illustrates the sequencediagram of the discovery operation.

Figure 2: Sequence diagram of Hy-LP’s discovery operation.

For synchronous communication, the actuator invokes the appropriate oper-ations and communicates directly with the entity via CoAP. For asynchronousoperation, subscribe or eventing, the messages are passed through the brokerover MQTT. Figure 3 illustrates the sequence diagram of the event subscriptionoperation.

Figure 3: Sequence diagram of Hy-LP’s event subscription operation.

9

Figure 4, illustrates the equivalent DPWS setting. Actuators send multicastingrequests to discover the active devices and their services. The compatible devicesrespond with their profiles and available services, including the relevant localnetwork addresses. Then, the actuator communicates directly with the selectedservices via HTTP and TCP. For the invoke operation, the device sends therequired information once, when requested. For the eventing operation, thedevice transmits information when an event occurs (e.g. change of monitoredparameters).

Figure 4: DPWS setting and operation mapping.

4. Evaluation

In this section, a comparative performance evaluation is conducted for theproposed Hy-LP and DPWS. To this end, two relevant versions of an IoT systemare implemented on embedded devices and evaluated under a common testbed.

Utilizing the proposed deployment, the user can access the aforementionedfunctionality through the Industrial Cloud. The application is implemented onthe Greek Research and Academic Community cloud service, named okeanos5.With Hy-LP, the user can access the devices directly through Internet. Thebroker is deployed on the cloud to enhance scalability. For the DPWS ver-sion, devices are accessed through a LAN-gateway. Regarding security, all thecommunication of the gateways with any backend systems is encrypted withTLS.

5okeanos: https://okeanos.grnet.gr/home/

10

Moreover, to evaluate the performance on an actual application, the policy-based access control (PBAC) framework presented in [41] is developed and de-ployed on the testbed. Said framework provides access control to the resources ofIoT nodes, based on policy constraints centrally managed by the system owner.The policies are modelled based on the OASIS-standardized eXtensible AccessControl Markup Language (XACML [46]). The policy repository is maintainedin the cloud along with the broker for Hy-LP and in the LAN-gateway forDPWS.

Figure 5 illustrates a typical IIoT deployment spanning different wind parks;a typical use case required in the specific domain. Each distinct wind park in-stallation features several wind turbines, while several sensors, industrial controlsystems and communication equipment exchange data in the wind park’s LAN.Moreover, exchanges take place between the wind parks and a central location,and, in the future, the industrial cloud, through a gateway at the edge of thewind parks’ networks. The wind parks are connected and managed over anSDN and Network Function Virtualization (NFV) -enabled network infrastruc-ture that aggregates all services, providing business applications from centralizedSDN controllers.

The variants of Hy-LP and DPWS are depicted in the upper left corner. TheHy-LP user (green color) can access all the sensory devices through Internet,while the DPWS user’s (orange color) flexibility is constrained as she can gainaccess only to a specific wind park LAN.

4.1. Testbed

We examine the effects of the different approaches and evaluate the perfor-mance of the aforementioned frameworks (Hy-LP and DPWS) over a commonbenchmark suite. We implement two relevant versions of the PBAC framework.

As an example of the framework’s flow of interactions, consider the caseof a wind turbine. The turbine is a device that hosts a service which supportsmultiple operations such selecting operation mode, getting the current rotationsper minute status or even events such as notifications when the tilt of the wingschanges and/or some threshold has been reached. As soon as the availabledevice and its services are discovered, a user can request access to the node thatis of particular interest, e.g. in order to extract the latest values from the sensorattached to it. The request is intercepted by the node’s policy enforcement(PEP) module which then forwards the request to the policy decision point(PDP), the latter running on the wind park’s trusted device. The PDP has toconsider all applicable policies, enriched by any relevant information, from thecentral policy administrator and information repository. The Policy InformationPoint (PIP) and Policy Administrator Point (PAP), which are deployed at thisend, act as a source of attribute values and are used for creating and managingpolicies or policy sets. Once all the required information has been collected,the PDP issues a decision which is sent back to the node’s PEP. Based on thatdecision the PEP may or may not allow the guest to access said nodes data ofinterest.

11

Figure 5: The IIoT wind park setting.

The IoT devices are BeagleBone nodes [47] – ARM Cortex-A8 single coreCPU running at 720MHz (throttled at 500MHz during testing) with 256MBDDR2 RAM. The testbed for the Service Orchestrator was the slightly morepowerful and versatile Beagleboard-xM [48] – 1GHz ARM Cortex-A8 proces-sor (throttled to run at 600MHz during testing) and 512MB DDR2 RAM, alsorunning a minimal Linux-based operating system. The access control infrastruc-ture was deployed on a desktop system as these are expected to run on moreresource-rich devices (e.g. the main system used to control and configure oursmart home) – Core i5 CPU at 3.3GHz, 8GB DDR3 RAM. Finally, all systemswere interconnected via wired Ethernet to minimize the network’s impact onthe reported performance figures. The setup for all variants of the applicationappears in Figure 6.

4.2. Performance

The two frameworks enable the seamless communication among the user,the IoT devices and the central policy repository. We implement the Hy-LP framework with the programming language Golang (a compiled staticallytyped language design by Google). For DPWS, we utilize the novel NodeDPWS

12

Figure 6: Testbed setup.

[49] library we had presented in previous work, built using Node.js (an inter-preted JavaScript-like language for server-side web applications, also designedby Google). Both implementations support IPv6, necessary for IoT applications.

One of the most important aspects compared during benchmarking was theclient-side response time, as this refers to the delay a user would experiencein each case when trying to access the protected resource (e.g. the turbine’srotations per minute). The testbed also featured a client application developedto discover and query the devices, recording response times, for benchmarkingpurposes. In total, 1000 requests were made to each device from our bench-mark client while we measured response times, CPU and memory. Figure 7presents the delay of Hy-LP and NodeDPWS. Random spikes are observed dueto the garbage collector counterparts of both Golang and Node.js. The averageresponse time of Hy-LP is 2.32ms and NodeDPWS is 23.63ms. Based on theresponse time, Hy-LP is significantly faster than the NodeDPWS (one order ofmagnitude faster). This is the result of the design approach for Hy-LP wherea high amount of interactions is performed once and maintained in the broker,like the discovery and eventing operations, while for DPWS has to broadcastthe requested information every time it is needed.

The CPU usage is similar for both frameworks, as illustrated in Figure 8,with Hy-LP exhibiting slightly lower utilization. Figure 9 illustrates the memoryconsumption. The use of lightweight primitives in Hy-LP (e.g. CoAP andJSON) instead of mainstream ones in NodeDPWS (e.g. HTTP and XML)results in significant reduction of memory demands. Our framework consumesaround 7.1 MB where NodeDPWS requires as high as 51MB.

Energy consumption is essential for IoT networks and becomes a major is-sue as the number of devices rises, since they require constant maintenancefor battery charging/replacing. Hy-LP consumes around 1472 mJ on averagewhile NodeDPWS requires around 14963 mJ to operate (an order of magnitude

13

higher), which is strongly affected by the aforementioned performance results.The low energy requirements can enable the employment of wireless energyharvesting techniques [50] to prolong the lifetime of such networks in an envi-ronmental friendly manner.

Figure 7: Execution time of Hy-LP and NodeDPWS.

Figure 8: CPU utilization of Hy-LP and NodeDPWS.

4.3. Performance Analysis – The Wind Park Use Case

As is evident from the analysis above, Hy-LP is more efficient and consumesless resources than the competing DPWS. However, to analyze the protocol’sperformance in the context of an actual vertical application, an SDN/NFV-enabled wind park is examined, as a characteristic use case of next generationindustrial networks operating over 5G communication infrastructures. In thewind park, several sensing devices are deployed (see Figure 10) that transmit

14

Figure 9: Memory Consumption of Hy-LP and NodeDPWS.

periodically information to the backend local Supervisory Control and DataAcquisition (SCADA) servers. The sensing data is used to monitor and/or reacton environmental or other operational parameters. The role of this operationis to read data from analog or digital sensors and transfer it to the backendvia a gateway device. These links are currently wired, but in the context ofupcoming industrial environments [51] and other critical applications [52] theywill be replaced with wireless connections.

In the context of project VirtuWind, we analyzed traces from an actual,operational wind park (in Brande, Denmark), in order to evaluate the speci-ficities of industrial traffic. The examined setting acquires four wind turbines,connected in a redundant star topology. The turbines themselves consist oftwo switches in series, one at the bottom and the other at the top. Numer-ous measurements systems, sensors, and actuators are linked to these switches,communicating with a SCADA server also connected to the star topology. Theconnection between the central switch and the Internet is enabled through arouter. Figure 5 shows the topology of the turbines and the typical networks(Ethernet and Profinet) within a wind turbine. The Park Control System con-sists of two main parts:

• Wind Farm SCADA System – enables the reporting, supervision, ac-quisition, and storage of data from the turbines

• Wind Farm Grid Control System – controls the power output of thedifferent wind turbines and adapts it to the grid operator demands

In the context of IIoT, the traces focus on traffic to/from the SCADA server,which was captured for approximately 1000 seconds of operation. Analyzingthese network traces, in conjunction with the application requirements and howthe wind parks currently function, enhance the interpretation of the evaluationoutcomes in the context of the actual specific application.

15

Several connections with low data rates (about 20.0000) that are containedin these traces can be ignored in the context of IIoT wireless sensor motes andtheir applications, including services such as Network Time Protocol (NTP),Dynamic Host Configuration Protocol (DHCP), and Simple Network Manage-ment Protocol (SNMP) exchanges. The remaining traffic includes TCP andUDP connections between the wind turbines and the SCADA server. The TCPones, though critical, only have end-to-end requirements of 100 ms, 250 ms and500 ms depending on the specific service, while the latter (i.e. instantaneoussingle-packet UDP exchanges) have a more stringent end-to-end delay require-ment of 10 ms. These numbers can be compared with the end-to-end delayobserved by Hy-LP, which is, on average, much lower, as illustrated in Figure 7.The ordinary delays in the round-trip timings dominate the trace communica-tions. As is evident from the observed performance figures, the proposed Hy-LPprotocol, constitutes a viable solution even for the most stringent connectionsin the context of the specific industrial setting examined (i.e. those requiring adelay of maximum of 10ms).

Figure 10: Typical Wind Turbine sensors and data networks.

5. Conclusions

This work presented Hy-LP – a hybrid protocol and framework for IIoT net-works. Hy-LP enables seamless and lightweight interactions with and betweensmart IIoT devices, in inter- and intra-domain deployments. It provides devicesand their associated services with discover, publish/subscribe and eventing (i.e.asynchronous), as well as synchronous (via direct invocation of exposed features)

16

capabilities. Security is also taken into account and all communications can beprotected with mature, robust solutions, such as TLS. Moreover, a policy-basedaccess control framework is supported, as demonstrated, that facilitates the ac-cess rights of different users. As a proof of concept, a CPS is implemented on awind park setting, where IIoT devices and actuators communicate informationthrough cloud. The protocol addresses the main obstacle of the LAN-basedDPWS and accomplishes interactions of devices over Internet. Moreover, Hy-LP is faster and consumes less resources than DPWS. The proposed solutionis lightweight and can be deployed on resource constrained embedded devices,respecting the strict QoS requirements observed in the target, operational windpark. As future work, the overhead imposed by the various security mechanismswill be examined in more detail, while the protocol and development frameworkitself will be made available to the researchers’ and developers’ communities.

Another important aspect of consideration for future work is the impact ofthe 5G communication technologies’ umbrella to the IoT and in particular theexisting technological IoT building blocks, such as the presented hybrid protocol.This impact, should not be examined in a technological isolation, but it shouldalso be seen in the wider context of the collateral interplay that this evolutionwill have in a cross sectoral and cross disciplined manner. IoT and intelligentassets, being considered as key enablers of Circular Economy are highly affectedby the technological advancements that the 5G technology will bring. Theenabling competitiveness, as well as the industrial transformation, will havepositive effects also in the context of the circular economy. New technologieshelp to make products, services, manufacturing and processing cleaner, safer,more secure while contributing in the use of materials and energy as efficientlyas possible given the need for reduced waste and emissions.

6. Acknowledgement

This work has received funding from the European Union’s Horizon 2020research and innovation programme VirtuWind under grant agreement No.671648. The authors would also like to thank the network engineers main-taining the subject wind park in Brande, Denmark for their valuable input ininterpreting the network traces and defining the application requirements.

References

[1] Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, M. Ayyash, Internetof Things: A Survey on Enabling Technologies, Protocols, and Applications,IEEE Commun. Surv. Tutorials, vol. 17, no. 4, pp. 23472376, Jan. 2015.

[2] W. Miao, G. Min, Y. Wu, H. Wang, J. Hu, Performance modelling and anal-ysis of software defined networking under bursty multimedia traffic, ACMTrans. Multimedia Comput. Commun. Appl., vol. 12, issue 5s, article no.77, Dec. 2016.

17

[3] S.L. Toral, F. Barrero, F. Cortes, D. Gregor, Analysis of embedded CORBAmiddleware performance on urban distributed transportation equipments,Computer Standards & Interfaces, vol. 35, issue 1, pp. 150-157, Jan. 2013.

[4] M. Hassanalieragh, A. Page, T. Soyata, G. Sharma, M. Aktas, G. Ma-teos, B. Kantarci, S. Andreescu, Health monitoring and management usingInternet-of-Things (IoT) sensing with cloud-based processing: Opportunitiesand challenges, 12th IEEE International Conference on Services Computing(SCC), IEEE, NY, USA, pp. 285-292, 2015.

[5] Y. Yan, R. Q. Hu, S. K Das, H. Sharif, Y. Qian, A security protocol foradvanced metering infrastructure in smart grid, IEEE Network, vol. 27,issue 4, pp. 64-71, Jul 2013.

[6] A. Botta, W. de Donato, V. Persico, A. Pescape, Integration of Cloud com-puting and Internet of Things: A survey, Future Generation Computer Sys-tems, Elsevier, vol. 56, pp. 684-700, Mar. 2016.

[7] E. Cavalcante, J. Pereira, M. P. Alves, P. Maia, R. Moura, T. Batista, F. C.Delicato, P. F. Pires, On the interplay of Internet of Things and Cloud Com-puting: A systematic mapping study, Computer Communications, Elsevier,vol. 8990, pp. 17-33, 2016.

[8] S. Bera, T. Ojha, S. Misra, M. S. Obaidat, Cloud-based optimal energy fore-casting for enabling green smart grid communication, IEEE Global Commu-nications Conference (GLOBECOM), IEEE, San Diego, CA, USA, pp. 1-6,2015.

[9] E. Borgia, R. Bruno, M. Conti, D. Mascitti, A. Passarella, Mobile edgeclouds for Information-Centric IoT services, IEEE Symposium on Computersand Communication (ISCC), IEEE, Messina, Italy, pp. 422-428, 2016.

[10] G. Hatzivasilis, I. Papaefstathiou, C. Manifavas, SCOTRES: secure routingfor IoT and CPS, IEEE Internet of Things Journal, IEEE, vol. 4, issue 6,pp. 2129-2141, 2017.

[11] A. Antonopoulos, M. D. Renzo, A. S. Lalos, L. Alonso, C. Verikoukis,Cooperation for Next Generation Wireless Networks, Fundamentals of 5GMobile Networks, Wiley, May 2015, pp. 105-124, 2015.

[12] C.-W. Tsai, H.-H. Cho, T. K Shih, J.-S. Pan, J. J.P.C. Rodrigues, Meta-heuristics for the deployment of 5G, IEEE Wireless Communications, IEEE,vol. 22, issue 6, pp. 40-46, 2015.

[13] E. Datsika, A. Antonopoulos, N. Zorba, C. Verikoukis, Green cooperativedevice-to-device communication: A social-aware perspective, IEEE Access,IEEE, vol. 4, pp. 3697-3707, 2016.

18

[14] V. Sciancalepore, V. Mancuso, A. Banchs, S. Zaks, A. Capone, Enhancedcontent update dissemination through D2D in 5G cellular networks, IEEETransactions on Wireless Communications, IEEE, vol. 15, issue 11, pp. 7517-7530, 2016.

[15] Z. Shelby, K. Hartke, C. Bormann, The constrained application protocol(CoAP), IETF, RFC 7252, 2014. https://tools.ietf.org/html/rfc7252.

[16] A. Banks, R. Gupta, OASIS Message Queuing Telemetry Trans-port (MQTT), version 3.1.1, OASIS, pp. 1-81, 2014. http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.pdf.

[17] D. Driscoll, A. Mensch, T. Nixon, A. Regnier, Devices profile for webservices, version 1.1, OASIS, 2009. [Online]. Available: http://docs.oasis-open.org/ws-dd/dpws/wsdd-dpws-1.1-spec.pdf.

[18] D. Corujo, M. Lebre, D. Gomes, R. Aguiar, MINDiT: A framework formedia independent access to things, Computer Communications, Elsevier,vol. 35, issue 15, pp. 1772-1785, 2012.

[19] O. Bergmann, K. T. Hillmann, S. Gerdes, A CoAP-gateway for smarthomes, Computing, Networking and Communications (ICNC), 2012 Inter-national Conference on, Maui, HI, pp. 446-450, 2012.

[20] S. M. Chun, J. T. Park, Mobile CoAP for IoT mobility management, Con-sumer Communications and Networking Conference (CCNC), 2015 12th An-nual IEEE, Las Vegas, NV, pp. 283-289, 2015.

[21] A. Betzler, C. Gomez, I. Demirkol, M. Kovatsch, Congestion controlfor CoAP cloud services, Emerging Technology and Factory Automation(ETFA), IEEE, Barcelona, pp. 1-6, 2014.

[22] H. A. Khattak, M. Ruta, E. Di Sciascio, CoAP-based healthcare sensornetworks: A survey, Applied Sciences and Technology (IBCAST), 2014 11th

International Bhurban Conference on, Islamabad, pp. 499-503, 2014.

[23] A. Zanella, N. Bui, A. Castellani, L. Vangelista, M. Zorzi, Internet of Thingsfor Smart Cities, IEEE Internet of Things Journal, vol. 1, no. 1, pp. 22-32,Feb. 2014.

[24] C. P. Kruger, G. P. Hancke, Implementing the Internet of Things visionin industrial wireless sensor networks, Industrial Informatics (INDIN), 12th

IEEE International Conference on, Porto Alegre, pp. 627-632, 2014.

[25] J. L. Fernandes, I. C. Lopes, J. J. P. C. Rodrigues, S. Ullah, Performanceevaluation of RESTful web services and AMQP protocol, 5th InternationalConference on Ubiquitous and Future Networks(ICUFN), Da Nang, Viet-nam, pp. 810-815, 2013.

19

[26] H. Subramoni, G. Marsh, S. Narravula, P. Lai, D. K. Panda, Design andEvaluation of Benchmarks for Financial Applications using Advanced Mes-sage Queuing Protocol (AMQP) over InfiniBand, Workshop on High Per-formance Computational Finance (WHPCF), Austin, TX, USA, pp. 1-11,2008.

[27] D. Barata, G. Louzada, A. Carreiro, A. Damasceno, System of acquisition,transmission, storage and visualization of Pulse Oximeter and ECG datausing Android and MQTT, Procedia Technology, 9, pp. 1265-1272, 2013.

[28] Y. F. Gomes, D. F. S. Santos, H. O. Almeida, A. Perkusich, IntegratingMQTT and ISO/IEEE 11073 for health information sharing in the Internetof Things, IEEE International Conference on Consumer Electronics (ICCE),Las Vegas, NV, USA, pp. 200-201, 2015.

[29] P. Papageorgas, D. Piromalis, T. Iliopoulou, K. Agavanakis, M. Bar-barosou, K. Prekas, K. Antonakoglou, Wireless Sensor Networking Archi-tecture of Polytropon: An Open Source Scalable Platform for the SmartGrid, Energy Procedia, Vol. 50, Pages 270-276, 2014.

[30] Seong-Min Kim, Hoan-Suk Choi, Woo-Seop Rhee, IoT home gateway forauto-configuration and management of MQTT devices, IEEE Conference onWireless Sensors (ICWiSe), Melaka, Malaysia, pp. 12-17, 2015.

[31] J. E. Luzuriaga, J. C. Cano, C. Calafate, P. Manzoni, M. Perez, P. Boronat,Handling mobility in IoT applications using the MQTT protocol, InternetTechnologies and Applications (ITA), 2015, Wrexham, pp. 245-250, 2015.

[32] J. E. Luzuriaga, M. Perez, P. Boronat, J. C. Cano, C. Calafate, P. Manzoni,A comparative evaluation of AMQP and MQTT protocols over unstableand mobile networks, 12th Annual IEEE Consumer Communications andNetworking Conference (CCNC), IEEE, Las Vegas, NV, USA, pp. 1-6, 2015.

[33] P. Saint-Andre, Extensible Messaging and Presence Protocol (XMPP):Core, IETF, RFC 6120, 2011. https://tools.ietf.org/html/rfc6120.

[34] S. Bendel, T. Springer, D. Schuster, A. Schill, R. Ackermann, M. Amel-ing, A service infrastructure for the Internet of Things based on XMPP,IEEE International Conference on Pervasive Computing and Communica-tions Workshops (PerCom Workshops), IEEE, San Diego, CA, USA, pp.385-388, 2013.

[35] T. Nixon, UPnP Forum and DPWS Standardization Status, 2008. [Online].Available: http://download.microsoft.com/download/f/0/5/f05a42ce-575b-4c60-82d6-208d3754b2d6/UPnP DPWS RS08.pptx.

[36] K. Fysarakis, O. Soultatos, I. Askoxylakis, C. Manifavas, I. Papaefstathiou,V. Katos, Which IoT Protocol? Comparing standardized approaches overa common M2M application, IEEE Global Communications Conference(GLOBECOM), IEEE, Washington, DC, USA, December 4-8, 2016.

20

[37] K. Fysarakis, G. Hatzivasilis, C. Manifavas, and I. Papaefstathiou, RtVMF:A Secure Real-Time Vehicle Management Framework, IEEE Pervasive Com-put., vol. 15, no. 1, pp. 2230, Jan. 2016.

[38] V. Venkatesh, V. Vaithayana, P. Raj, K. Gopalan, R. Amirtharaj, A SmartTrain Using the DPWS-based Sensor Integration, Res. J. Inf. Technol., vol.5, no. 3, pp. 352362, Mar. 2013.

[39] T. Cucinotta, A. Mancina, G. F. Anastasi, G. Lipari, L. Mangeruca, R.Checcozzo, F. Rusina, A Real-Time Service-Oriented Architecture for In-dustrial Automation, IEEE Trans. Ind. Informatics, vol. 5, no. 3, pp. 267277,Aug. 2009.

[40] S. Phlsen, S. Schlichting, M. Strhle, F. Franz, C. Werner, A DPWS-BasedArchitecture for Medical Device Interoperability, World Congress on Med-ical Physics and Biomedical Engineering, September 7-12, 2009, Munich,Germany SE – 22, vol. 25/5, O. Dssel and W. Schlegel, Eds. Springer BerlinHeidelberg, pp. 8285, 2009.

[41] K. Fysarakis, I. Papaefstathiou, C. Manifavas, K. Rantos, O. Sultatos,Policy-based access control for DPWS-enabled ubiquitous devices, Pro-ceedings of the 2014 IEEE Emerging Technology and Factory Automation(ETFA), pp. 18, 2014.

[42] R. R. Igorevich, P. Park, J. Choi, D. Min, iVision based Context-AwareSmart Home system, in The 1st IEEE Global Conference on Consumer Elec-tronics 2012, pp. 542546, 2012.

[43] G. Hatzivasilis, I. Papaefstathiou. D. Plexousakis, C. Manifavas, N. Pa-padakis, AmbISPDM: managing embedded systems in ambient environmentand disaster mitigation planning, Applied Intelligence, Springer, pp. 1-21,2017.

[44] N. Nurseitov, M. Paulson, R. Reynolds, C. Izurieta, Comparison of JSONand XML data interchange formats: A case study, ISCA 22nd InternationalConference on Computer Applications in Industry and Engineering (CAINE2009), San Francisco, California, USA, Nov. 4-6, 2009.

[45] D. Davis, A. Karmarkar, G. Pilz, S. Winkler, U. Yalcinalp, WebServices Reliable Messaging (WS-ReliableMessaging) Version 1.2, Oa-sis Standard, 2009. [Online]. Available: http://docs.oasis-open.org/ws-rx/wsrm/200702/wsrm-1.2-spec-os.pdf.

[46] B. Parducci, H. Lockhart, E. Rissanen, eXtensible Access Control MarkupLanguage (XACML) Version 3.0, OASIS Standard, 2013. [Online]. Available:http://docs.oasis-open.org/xacml/3.0/xacml-3.0-core-spec-cs-01-en.pdf.

[47] BeagleBone System Reference Manual, RevA3 1.0. [Online]. Available:http://beagleboard.org/static/beaglebone/a3/Docs/Hardware/BONE SRM.pdf.

21

[48] BeagleBoard-xM System Reference Manual, Rev. C. [Online]. Available:http://beagleboard.org/static/BBxMSRM latest.pdf.

[49] K. Fysarakis, D. Mylonakis, C. Manifavas, I. Papaefstathiou, Node.DPWS:Efficient Web Services for the Internet of Things, IEEE Software, vol. 33,issue 3, pp. 60-67, 2016.

[50] S. Peng, C. P. Low, Energy neutral directed diffusion for energy harvestingwireless sensor networks, Computer Communications, Elsevier, vol. 63, pp.40-52, 2015.

[51] M. R. Palattella, P. Thubert, X. Vilajosana, T. Watteyne, Q. Wang, T.Engel, 6TiSCH Wireless Industrial Networks: Determinism Meets IPv6, In-ternet of Things: Challenges and Opportunities, S. C. Mukhopadhyay, Ed.Cham: Springer International Publishing, pp. 111141, 2014.

[52] R. N. Akram, K. Markantonakis, K. Mayes, P.-F. Bonnefoi, D. Sauveron,S. Chaumette, An efficient, secure and trusted channel protocol for avionicswireless networks, 2016 IEEE/AIAA 35th Digital Avionics Systems Confer-ence (DASC), pp. 110, 2016.

22