Post on 02-Aug-2020
transcript
IoT at Scaleor, "From Little Things, Big Data Grow"
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Christopher Biggs, Accelerando Consulting
@unixbigot @accelerando_au
1 / 74
Who am I?
Christopher Biggs — @unixbigot — @accelerando_au
▷ Brisbane, Australia
▷ 20+ years in IT as developer, architect, manager
▷ Founder, Accelerando Consulting▷ Accelerando is a "full service" consultancy - chips to cloud
▷ IoT, DevOps, Big † Data
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude
† Actual bigness may vary
2 / 74
From little Things, Big Data Grow
"Internet of Things" (IoT) At Scale
▷ Why Even IoT? - my personal motivation
▷ Chips to Cloud - the journey from sensor to senses
▷ The Air Gap - connecting wireless devices to the Internet
▷ Data Tributaries - follow the data from droplet to ocean
▷ Making sense of it all - Managing and visualising IoT data
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude
3 / 74
Overture: Why Even IoT?
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017 4 / 74
"Don't ever invite a vampire into your house, you silly boy. It renders you powerless."
-- "Max", The Lost Boys
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
5 / 74
"The 'S' in IoT stands for 'Security'."
-- Absolutely Everyone, The Internet
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
6 / 74
What is IoT For?
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
7 / 74
It's for not this
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
Prelude Overture
People who shouldn't have to do crummy jobs:
Children
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
9 / 74
People who shouldn't have to do crummy jobs:
People of Colour
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
10 / 74
People who shouldn't have to do crummy jobs:
Women
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
11 / 74
People who shouldn't have to do crummy jobs:
Immigrants
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
12 / 74
People who shouldn't have to do crummy jobs:
Foreigners
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
13 / 74
People who shouldn't have to do crummy jobs:
Robots
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
14 / 74
Wait, what?"Aren't crummy jobs what robots are for?"
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
15 / 74
"Do you want a war of extinction? Because that's how you get wars of extinction"
--Malory Archer, nearly
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
16 / 74
Arti cial nonintelligence
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
17 / 74
Doing all the things no intelligent being should have to be bothered by.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
18 / 74
Give me your tired-of-doing-that, your poorly done jobs, your harried masses yearning to be stress-free
-- Emma Lazarus, kind of
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
19 / 74
IoT's Trinity
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017 20 / 74
ThingsStart with the simplest parts...
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
21 / 74
Internet...connect them together...
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
22 / 74
Of (i.e. context)...to form a synergystic whole.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
23 / 74
My Three Laws of IoT
▷ Devices must cooperate for the bene t of humans
▷ Devices must communicate, and obey instructions
▷ Devices must be as simple and reliable as possible
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
24 / 74
OK, end of setup
-- "Ford Fairlane", (not just a car, but a really awful movie)
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture
25 / 74
CommunicationsCrossing the Air Gap
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017 26 / 74
Fast, Cheap, Battery-ef cient.
Choose any two.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications
27 / 74
PoE (Power-over-Ethernet)
Fast, no-battery, expensive wires.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications
28 / 74
EoP (Ethernet over Powerline)
Cheaper wiring, not so fast.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications
29 / 74
PoDS (Plain ol' Dumb Serial)
Please, no.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications
30 / 74
WiFi
(Wireless Made-up-Acronym)
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications
31 / 74
Bluetooth
(Don't chew on that Biro)
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications
32 / 74
ETPH
You know, telephones
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications
33 / 74
LoRa
Telephones from 1978, but without the wires.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications
34 / 74
IPoSUV
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications
35 / 74
Comms Recap
▷ If you have wires, use 'em
▷ PoDS is not IoT
▷ If you have the power budget for WiFi, great
▷ How about a bluetooth mesh?
▷ "Low-power" equals "really bloody slow"
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications
36 / 74
Structuring for Scalability
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017 37 / 74
Insert magic cloud here
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
38 / 74
Nope
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
39 / 74
Edge Computingaka "Not the cloud but we've spent so long plugging cloud we can't possibly admit that"
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
40 / 74
Web API
HTTP is awful on slow links
HTTPs is aw-over owing
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
41 / 74
Message buses
long-lived connection, lightweight messages
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
42 / 74
Retro t
Websockets
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
43 / 74
Build your own
A bit of orchestration, a bit of docker, a bit of time
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
44 / 74
Owning an architecture is a big responsibility
Maybe you should start with a puppy.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
45 / 74
Don't build your own
Oh, look, google, amazon and microsoft already did it
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
46 / 74
Well, they announced it
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
47 / 74
Microsoft
Messaging in place, provisioning "coming soon"
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
48 / 74
Messaging in place, management in private beta
Device platform (Android Things) supports two vendors only.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
49 / 74
Amazon
Messaging, device management function-provisioning in place.
No news on device provisioning.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
50 / 74
So, tell me again about rolling my own?(I'll tell you on Thursday)
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
51 / 74
Give me a hint?Saltstack. Mosquitto. Docker.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
52 / 74
Leveraging saltstack
▷ ZeroMQ message bus transport
▷ Tree-structured hierarchy
▷ Top level master (or multiple)
▷ Intermediate "syndicate masters" (optional)
▷ Leaf-node "minions"
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
53 / 74
Playing nice with MQTT
▷ Message broker federation (one- or two-way)
▷ Lots of implementations
▷ Mosquitto in Docker
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
54 / 74
What do you mean you want it reliable?
▷ MQTT QoS options (but not end-to-end)
▷ Apache Kafka
▷ Amazon Kinesis
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability
55 / 74
Case study 1 - MQTT and Kinesis
▷ Sensors transmit to on-site MQTT broker
▷ Cloud MQTT broker federates on-site brokers
▷ Cloud ingester writes to Elasticsearch
▷ Cloud ingester forwards certain events to kinesis stream
▷ AWS Lambda functions handle kinesis event
▷ Problems: no recovery of missed messages
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude
56 / 74
Case study 2 - Saltstack and Elasticsearch ( eetvalid.com)
▷ Sensors connect to on-site saltstack gateway
▷ (Potential store-and-forward for disconnected sensor stations)
▷ Saltstack gateway connects to cloud 'master'
▷ Event 'engine' on master relays events to Elasticsearch data store
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude
57 / 74
Case study 3 - Meshes and AWS IoT (evacmate.com.au)
▷ Sensors create a mesh network over bluetooth
▷ One or more gateway nodes have 3G uplinks
▷ Data streamed to cloud over websocket
▷ Gateway nodes managed with AWS IoT
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude
58 / 74
Making sense of data
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017 59 / 74
Layers of data
▷ Device health
▷ Population health
▷ Application data
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation
60 / 74
Put it all in a data bucket
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation
61 / 74
But don't look at it
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation
62 / 74
Except when you do
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation
63 / 74
Arti cial nonintelligence
Machine learning for "this is ne"
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation
64 / 74
Machine Learning in Elastic Search
"this is not ne for 10am on Tuesday"
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation
65 / 74
Managing your thingsBig data bassackwards
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017 66 / 74
Addressing the eet
Saltstack - rich syntax for addressing sets of devices
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation Reaction
67 / 74
Addressing the eet
MQTT comms - bidirectional messaging but little structure
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation Reaction
68 / 74
Addressing the eet
Broadcasts - IP or otherwise
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation Reaction
69 / 74
Reactive architectures
Rule based response to events.
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation Reaction
70 / 74
Coda
▷ IoT (for me) is about relieving Future Shock
▷ IoT's trinity: Hardware, Comms, Data
▷ Crossing the Air Gap
▷ Flowing to the sea
▷ Case studies
▷ Making sense of it all
▷ Knowing when to run
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation Reaction Coda
71 / 74
Homework
▷ Make your systems amenable to cooperation
▷ Try out Amazon GreenGrass - lambda functions on your Raspberry Pi
▷ Keep an eye on beta offerings from Google and Microsoft.
▷ Use a message fabric to instrument your distributed systems
▷ Try out Elastic or Google ML for anomaly detection
▷ What cool things can you do with LoRa and 300bits per second?
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation Reaction Coda
72 / 74
Resources, Questions
Related talks - http://christopher.biggs.id.au/#talks
Me - Christopher Biggs
▷ Twitter: @unixbigot
▷ Email: christopher@biggs.id.au
▷ Slides, and getting my advice: http://christopher.biggs.id.au/
▷ Accelerando Consulting - IoT, DevOps, Big Data - https://accelerando.com.au/
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation Reaction Coda
73 / 74
Links
▷ Open source Bluetooth Mesh on current hardware
▷ Future Standard Bluetooth mesh networksing
▷ Public access LoRaWAN in Sydney
▷ The 'mosquitto' MQTT broker:
▷ Example of how I use SaltStack to provision MQTT
▷ Asynchronous Microservices
Christopher Biggs — IoT at Scale @unixbigot @accelerando_au
#YOWData Sept 2017
Prelude Overture Communications Scalability Interlude Interpretation Reaction Coda
74 / 74