Implementationworkonopensourcewebofthings

serversandgateways

DaveRaggett,W3C<dsr@w3.org>

Monday,11April2016

ReferenceArchitecture

GatewayCloudbasedServices

FirewallAmbientorbatteryoperated IoT devices

Powered,multi-protocol

BrowserforHMI

2/27

Introduction

• IamworkingontwoopensourceWebofThingsserver&gatewayprojects• NodeJS

• Forrelativelypowerdevices,e.g.runningLinux• C++formicrocontrollers

• ForArduinoUnoandWiznet basedEthernetShield• Highlyconstrainedlowcostmicrocontroller• FutureplansforextensiontomorepowerfulMCUs• Andotherkindsofcommunications technologies

• Why?• Nothinglikeworkingcodeforbetterunderstanding• Hugepotentialmarketforlow-endIoT devices

3/27

• ArduinoEthernetShield• 16KBRAM• MicroSD cardslot• ControlledthroughSPIbus• PollingorH/Winterrupt• Cost:4.75GBPoneBay

• ArduinoUnowithATmega328PMCU• 2KBRAM• 1KBEEPROM• 32KBFLASH• LotsofI/Opins• Cost:2.33GBPoneBay

RequiredHardware

4/27EEPROMusefulforstoringsettingsanderror logs,etc.

RequiredSoftware

• ArduinoIDE• Freefromhttps://www.arduino.cc/en/Main/Software

• SerialdriverforCH340USBtoserialchip• RequiredforcheapArduinoclones

• AbitofapaintofindandsetupL

5/27

Components

• ArduinoSketchwithapplicationcode• ThisiswhereyouwriteyourcodetointerfacetosensorsandactuatorsusingtheArduinolibraries• GPIOpins• A2DandPWM• SPIandI2Cbuses• Timers

• WebofThingsLibraryfortheArduinoIDE• InstallablefromGitHub• Smallfootprintwithcustomnetworkingmodule

• Wiznet W5100withnativeIPsupport• Multicastfordiscovery• UDPandTCP• ARP,ICMP,IGMPetc.

6/27

Configuration

• UsestaticIPaddressorDHCPfordynamicaddress• UsesmDNS todiscoverWebofThingsgateway

• Servicetype:_wot._tcp.local (unregistered)• MyMacBook’sfirewallblocksallothermulticastsockets

• RegistersremoteproxyongatewayforlocalthingontheArduino• UsesTCPformessagingwithgateway

• LesscodeduetonativeIPsupportonW5100chip• VerysimilartomyWebSocketimplementationforNodeJS webofthingsserver

• CoAPwouldtakemorespaceandbelesscapable• Notethechoiceofprotocoldependsonthechoiceofhardware,whichinthiscaseisanEthernetcontroller

7/27

Softwarearchitecture• Targetis<28KBfortheLibrarycode

• 2KBforloader,>2KBforapplicationcode• 2KBforRAM• Avoidmalloc andfreeforrobustoperation

• Fragmentationandlongterminstability

• Staticnodepool• JSONfornumbers,booleans,objects,arraysetc.• AverageLengthBinarytrees(objectsandarrays)• Mark/Sweepgarbagecollector• 6bytespernodeontheATmega328P

• JSONobjectpropertynamesreplacedbysymbols• ToreduceRAMfootprintto2KBisverychallenging

• Efficientbinarymessageencoding/decoding• Reliesondeterministic assignmentofsymbols• Designedwithshortpackettechnologies inmind

e.g.NordicnRF24L01+with32bytepayloads

8/27

RAMvsFLASHmemory

• SomeMCUsliketheATmega328PusetheHarvardmemoryarchitecture• Thishasseparatedataandcodeaddressspaces

• StringliteralsarestoredinFLASHandcopiedtoRAMjustbeforeenteringafunction• Thisisobviouslybadifyou’reshortofRAM

• Firststep:tellcompilertokeepliteralsinFLASH• F(“stringliteral”)vs“stringliteral”• Youthenneedtousesystemcallstoaccessthem

• Secondstep:useabstractionlayertohidethedifferencebetweenRAMandFLASH• Signatures:const __FlashStringHelper *vsconst char*• Mergeaddressspacesthroughadding0x8000toFLASHpointersandusestringfunctionsthatknowhowtodealwiththem

9/27

InterruptsandEvents

• Hardwareinterruptsarevaluableforhandlingsensorsandactuatorsetc.• Buttheinterruptserviceroutine(ISR)mustbequicktoavoidblockingotherinterrupts• Eventdrivenbehaviourisveryconvenient,buteventhandlerscantaketimetoexecute• Solution:ISRpusheseventontoqueue,whichisdispatchedfromwithinArduinoloop()method• SamesinglethreadingexecutionmodelasforWebbrowsers– eventhandlersareexecutedsequentially• Avoidsappdevelopersfromhavingtodealwithcomplexsynchronisationissues

10/27

LifeafterDeath

• Eventuallysomethingbadwill happen…• Memoryissues

• RunningoutofRAMfortheprogramstack• RunningoutoffreenodesforusewithJSON• Overflowofcriticalqueues

• Programbugs• Youcanneverbe100%confidentthattherearenobugs

• Baddata• Violationofdesignassumptions

• Softwarerestartondetectionoffatalerrors• Hardwarewatchdogtimerforcesrestartifnotresetintime

• Thisallowsthedevicetorecoverafterbeingstuckinaloop• Gatewaysandotherdevicesmustberesilienttosuchrestarts

11/27

ThingsandProxychains

• Thingsaresoftwareobjectswithproperties,actions,eventsandmetadata• IfyouwanttointeractwithaThingonanotherdeviceyouwillneedaProxyobjectforit• ThingsLayerkeepProxiesinsyncwiththeirThings• ProxiescanbechainedtoformProxytrees rootedintheThingtheystandinfor• Updatingapropertyonaproxyresultsinmessagesthattraveltowardstherootandleavesofthetree

12/27

LocallyScopedIdentifiers

• EachdeviceintheproxytreegenerateslocallyscopedIDsfortheproxies/thingsithosts• Theseareusedtomatchmessagestotheobjectstheyaretargetedat• Messagesaredistinguishedbywhethertheyaresenttoaparentorchildofanodeintheproxytree• ProxyobjectsholdtheirIDforthisdeviceaswellastheirparent’sIDforthecorrespondingThing/Proxyontheparentdevice• Messagehandlersortsthisallouttoroutemessagesbetweenthedesiredsoftwareobjects

13/27

CleanSeparationofLayers

Application Scripts that define thing behaviour in terms of their properties, actions and events, using APIs for control of sensor and actuator hardware

Things Software objects that hold their stateAbstract thing to thing messagesSemantics and Metadata, Data models and Data

Transfer Bindings of abstract messages to mechanisms provided by each protocol, including choice of communication pattern, e.g. pull, push, pub-sub, peer to peer, etc.

Transport REST based protocols, e.g. HTTP, CoAPPub-Sub protocols, e.g. MQTT, XMPPOthers, including non IP transports, e.g. Bluetooth

Network Underlying communication technology with support for exchange of simple messages (packets)Many technologies designed for different requirements

ApplicationDeveloper(WoT focus)

PlatformDeveloper(IoT focus)

Focus on data types, APIs and error handling

14/27

ImplementingtheCommunicationsStack• Designthesoftwarearchitecturetoallowforuseofmultiplecommunicationstechnologies• ThingLayerthathooksintothesoftwareobjectsforthingsandhandlescorrespondingabstractmessages• Transferlayermapsthesetocommunicationpatternssubjecttocommunicationsmetadata• TransportLayerforspecificprotocolsimplementedaspluggablemodulesforthetransferlayer• Thechoiceoftransportisindicatedbytheconnectioninformationforeachparentandchild• Lowenddevicelikelytosupportonlyoneprotocol

15/27

ThingDataModels

• Thingsmayhaveproperties,actions,eventsandmetadata• Eventshaveatypeandavalue• Propertieshavevalues• Actionsmaypassavaluewhentheyareinvoked• Actionsmayreturnasequenceofvaluesasresponses

• zero,one,twoormoreresponsesasappropriate• Valueshavetypes*

• null,true,false,integers,floats,strings• Arraysandobjects(setsofname/valuepairs)• Valuescanbecompound,e.g.nestedarraysandobjects• ThingsandStreamsasfirstclasstypes

*Should enumerations beallowedascoretypes?16/27

LifecycleConsiderations

• Thingscanhaveotherthingsastheirproperties*• Thingsreferencedaspropertiescanbelocalorremote• Forlocalthings,thereferencedthingmaynothavebeencreatedyet• Forremotethings,serverneedstosetupproxies• ProxiestaketimetosetupduetotheneedtogettheThingdatamodeldescription

• Solution:pushthepropertytoapendingqueueandsetthepropertyvaluewhenthereferencedthing/proxyisready

17/27*Exceptforveryconstrainededgedevices

CyclicDependencies

• Withthingsaspropertiesthereisthepotentialforcyclicdependenciesbetweenthings• ThingAhasThingBasaproperty• ThingBhasThingCasaproperty• ThingChasThingAasaproperty

• Thiscouldinvolveamixoflocalandremotethings• IfwestartsuchThingswhenalloftheirpropertieshavebeenstartedthenwewillgetadeadlock!• Solution:startaThing/Proxywhenallofitspropertiesareresolved,andforeachThing/Proxyholdmessagestothatobjectinaqueueuntilithasbeenstarted

18/27

LateBinding

• Valueswhosefulltypeisonlygivenatrun-time• Thetypemaybepartiallyspecifiedatcompiletime

• Canapplytoproperties,actionsandevents• Variantdatatypeswithtagthatdetermineswhichvariantappliestothisvalue(e.g.C++unions)• Thingsaslateboundvalues• Youneedtoretrievethedatamodeltoinstantiatethesoftwareobjectforthethingyou’vebeenpassed• Thevalueincludesareferencetothedatamodel

• ThingLayermanagesthisautomatically

19/27

Pointers

• SayaThinghasapropertywhichisanarray• Ascriptupdatesthe3rd iteminanarray• Thisrequiresameansforupdatemessagestoidentifyareferencetothe3rd iteminthearray• InotherwordssomekindofpointerthatoperatesontheThing’sdatamodel• SimpleexpressionwithinJSONmessages,e.g.

“path”:foo[3]and“path”:foo[3].bar• ForCoAP aURIbasedsyntaxwouldbeappropriate

20/27

RefinementofJSON-LD

• UsedforexpressingThingdatamodels• @contextforbindingshortnamestoRDFURIs

• Implicitcontextforthemostcommonnames• Linkedcontextfordomainspecificnames

• Usedforsemanticsearchandservicecompositions• Canbesafelyignoredbyedgedevices

• Toplevelobjectwith“properties”,“actions”&“events”• Valueseitherdeclaredasname:type

• e.g. “on”:“boolean”,or“level”:“float”• Ordeclaredasname:{annotations}

• e.g.“temperature”:{“type”:“integer”,“min”:-20,“max”:100,“units”:“celcius”}

• Neededfornestedproperties*• Actionsdeclaredwith“in”and“out”forthevaluesthatarepassedtothem,andfortheresponsestheactiongenerates

21/27*Discussiontopics:howtoapplythistonestedarraysandobjects

IntegrityConstraints*

• Enabledetectionofbaddataforincreasedresilienceinthepresenceoffaultsandattacks

• Goodprogrammingpractice:assert(expression)• Basicintegrityconstraintsonasinglevalue

• Upperandlowerboundsfornumbers• Closedsetofvaluesforstrings

• Integrityconstraintsacrossmultiplevalues• Relationships betweenpropertiesofasingleThing• Relationships acrossmultiple Things

• Cardinalityconstraints• e.g.ForeveryXyoumusthaveatleasttwoY

• Metadataconstraints• e.g.forapplicationversioning

• Needforanexpressionlanguagefordefiningconstraints

22/27*Temporalconstraintscanbeexpressedindomainmodels

IntegrityConstraintsinJSON-LD

• Simplevalueconstraints• Asannotationsonthetype,e.g.minandmax

• Complexconstraintsinvolvingexpressions*• OperatorsandFunctions

• Numericcomparisons,Stringcomparisons,Booleanoperators• Extensible setoffunctions

• Complexconstraintsinvolvingpointers• PointersasreferencestoothervaluesinthisThing

• Otherproperties,whichcanincludeotherthings• Otherpartsofthedatainanevent,orpassedto,orreturnedbyanaction

• TooexpensiveforlowenddeviceswithlimitedRAM• Iplantoimplementthisonthegatewayserver

*CanwegeneraliseJSONtoalloweasytounderstandexpressionswithoperatorprecedence?23/27

DomainModels

• Richontologybaseddescriptions• Canexpresssemanticconstraints• Ifxisatemperaturesensorthenxmustdefineitsphysicalunitsfromtheset{Kelvin,Celsius,Fahrenheit}• Temporalconstraints,e.g.onsequencesofactions

• Usefulforsemanticsearchandservicecompositions• Verifycompatibilityofservices

• RequiressupportforOWL,SPARQL,etc.• Notneededforresourceconstrainededgedevices

24/27

DemoProjects

• Aimtoprovidesomeexamplesforhowtousethelibrarywithcommonlyavailablesensorsandactuators,e.g.• Temperature&humidity• Ultrasonicrangesensor• MulticolourLEDs• Servomotor

• Needtoprovidesketch,wiringdiagramandnotes• Betteryettoalsoprovideavideo• Lookingforvolunteerstohelpwiththiseffort!• StartinginJune2016

25/27

Projectsite

• TheArduinoprojectisonGitHub• https://github.com/w3c/wot-arduino

• Stillaworkinprogress• HavingtodothisasabackgroundactivityduetolotsofcompetingtasksL

• CurrentlyworkingonThingLayerfollowingrecentworkongarbagecollectionanddiscovery

• Re-usingvirtuallyallthecodeforgateway• Willenable lowcostmicrocontrollerbasedgateways

• PlantoupdateNodeJS project*tomatch• GoaltohavefullydocumentedprojectwithexamplesintimeforBeijingmeetinginJuly2016

*https://github.com/w3c/web-of-things-framework 26/27

WhereNext?• Morepowerfulprocessors

• ArduinoMega• 8KBRAM,4KBEEPROM,256KBFLASH

• ARMbasedMCUs• MoreRAM,butnoEEPROM

• WorkonStreams,e.g.forelectrocardiograms• Othercommunicationtechnologies

• SensornetworkswithNordicnRF24L01+• BluetoothSmartwithXBee Arduinoshield

• Sleepydevices• Usingtimertowakeupeveryfewminutes• Mainlyrelevanttowirelesstechnologies

• NotethatArduinoboardsarepowerhungryduetoLEDsandlinearvoltageregulators

• Synchronisationacrossclustersofdevices• Synchronisedcontrolofrobotjoints

• Stronghardwarebasedsecurity• Domainmodels• Crowdsourcedinformalsemantics

27/27