15
Time series IoT data ingestion into Cassandra using Kaa Andrew Shvayka [email protected] kaaproject.org © 2015 CyberVision, Inc. All rights reserved.
Time series IoT data ingestion into Cassandra using Kaa
Andrew Shvayka
[email protected]© 2015 CyberVision, Inc. All rights reserved.
kaaproject.org© 2015 CyberVision, Inc. All rights reserved.
Agenda
➢ Data ingestion challenges➢ Why Kaa?➢ Why Cassandra?➢ Reference architecture overview➢ Hands-on
➢ Sandbox setup➢ Raspberry PI application code walkthrough➢ Cassandra appender configuration➢ Live demo
➢ Q&A
kaaproject.org© 2015 CyberVision, Inc. All rights reserved.
Data ingestion requirements/challenges
Must have:➢ Guaranteed data delivery➢ Scalability➢ Security➢ Performance➢ Low latency
Nice to have:➢ Built-in data structure validation➢ Device platform independent➢ Low footprint➢ Low bandwidth support
kaaproject.org© 2015 CyberVision, Inc. All rights reserved.
kaaproject.org© 2015 CyberVision, Inc. All rights reserved. kaaproject.org© 2015 CyberVision, Inc. All rights reserved.
➢ Fully-featured IoT middleware platform➢ 10 Kb RAM footprint (with C SDK)➢ Guaranteed data delivery and reliable local storage➢ Built-in transport security➢ Efficient data serialization➢ Horizontally scalable and fault tolerant➢ 100% open-source (Apache license 2.0)➢ Rapid application development using C / C++ / Java SDKs➢ Integration with popular device platforms
Why Kaa?
kaaproject.org© 2015 CyberVision, Inc. All rights reserved. kaaproject.org© 2015 CyberVision, Inc. All rights reserved.
➢ Fault tolerant➢ Performant➢ Horizontally scalable➢ Easy deployment➢ Integration with popular analytics platforms
Why Cassandra?
Region 3
Region 1
© 2015 CyberVision, Inc. All rights reserved.
Problem description
kaaproject.org
Region 2
Region 4
© 2015 CyberVision, Inc. All rights reserved.
Reference architecture
kaaproject.org
StructuredDataRaw data
Raspberry Pi
Kaa SDKClient application
Raspberry Pi
Kaa SDKClient application
Kaa cluster/sandbox
Kaa node Cassandra appender
Cassandra ...
Temperature sensor
(DHT11)
Temperature sensor
(DHT11)
Kaa node Cassandra appender
kaaproject.org© 2015 CyberVision, Inc. All rights reserved.
Development environment setup
Sample project repository: https://github.com/kaaproject/kaa-cassandra-sampleKaa Sandbox: http://www.kaaproject.org/download-kaaRaspberry Pi: http://docs.kaaproject.org/display/KAA/Raspberry+Pi
© 2015 CyberVision, Inc. All rights reserved.
Data modeling
kaaproject.org
● Single sensor per row● Single sensor per row, with date partitions and TTL● Sensors per region and model, with date partitions
Kaa data collection schema:{ "type" : "record", "name" : "SensorData", "namespace" : "org.kaaproject.kaa.sample", "fields" : [
{ "name" : "sensorId", "type" : "string"}, { "name" : "model", "type" : "string"}, { "name" : "region", "type" : "string"}, { "name" : "value", "type" : "float"}]
}
© 2015 CyberVision, Inc. All rights reserved.
Single sensor per row
kaaproject.org
● Partition Key: Sensor ID● Clustering Key: Timestamp● Fields: Region, Model, Value, JSON, BLOB
Sensor IDTimestamp 1
Fields
Timestamp 2
Fields
Timestamp N
Fields...
Query:select * from sensor_per_row where sensor_id = ‘Sensor 1’ and ts > 42
© 2015 CyberVision, Inc. All rights reserved.
Single sensor per row, with date partitions and TTL
kaaproject.org
● Partition Key: Sensor ID, Date● Clustering Key: Timestamp● Fields: Region, Model, Value● TTL: 60 sec
Sensor ID,Date
Timestamp N
Fields
Timestamp N-1
Fields
Timestamp 1
Fields...
Query:select * from sensor_per_date where sensor_id = ‘Sensor 1’ and date =
‘2015/09/10’ and ts > 42
© 2015 CyberVision, Inc. All rights reserved.
Sensors per region and model, with date partitions
kaaproject.org
● Partition Key: Region, Model, Date● Clustering Key: Timestamp, Sensor ID● Fields: Value
Region, ModelDate
Timestamp 1,Sensor A
Value
Timestamp 1,Sensor B
Value
Timestamp N,Sensor A
Value...
Query:select * from sensor_per_region where region = ‘Region A’ and model = ‘DHT11’
and date = ‘2015/09/10 17:10’
© 2015 CyberVision, Inc. All rights reserved.
Zookeeper quorum
Endpoints
Control servers
standby
Bootstrap servers
Operations servers
Fault-tolerance and horizontal scalability
kaaproject.org
active
