Building a Smarter Home with Apache NiFi and Spark

Building a Smarter Home with Apache NiFi and Spark

Joseph Niemiec, Systems ArchitectChristopher Gambino, Solutions Engineer

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Our Background

Systems Architect– Hadooping for 4+ Years – War-Driving - Mad max style– Telematics Patent in Parking Floor Detection– Co-Author Apache Hadoop YARN Book

Smart Production Lines– Medical Device Quality Control– FDA Compliant Reporting– Custom IOT devices and framework


Agenda

Use Case Explanation Hardware & Software Summary Architecture NiFi Development Patterns – Multiple Generations Discoveries Challenges //Other Lessons Learned


Use Case Explanation

Hortonworks Data Flow (NiFi) enables easy access and filtering to data that resides on theedge. This data can be transmitted site to site and accessed via data explorationframeworks such as Spark.

Joe was Paranoid Living In Detroit Wanted to investigate the unseen daily activities Understand problems with edge collection and consumption

The Solution?More Technology


Software & Hardware Used

Brand Name Smarthome Sensors + Hub– Door, Motion, Window

NiFi on a Raspberry Pi 2 Model B– Raspbian OS – Python Push based collector

NiFi on EC2 EC2 HDP Cluster BareMetal HDP + HDF Pig & Spark

Aldrin Piri

raspbian, I assume?


Architecture High LevelHub & Spoke Approach

– Hub

• Hadoop Cluster in the DC• Stores All Historical Data• Aimed at making it usable for analysis's• Consumes from Spoke Brokers to land data in DC

– Spoke

• Per Region/Geo• Real Time Stream Processing for Customer Apps• Cleanse data to meet legal requirements for off-geo Hub


Architecture Diagram


Bidirectional Communication

Edge to Cloud– Collection (Push) Site-2-Site– Updates (Pull) HTTP Calls

Cloud – Distributed Cache of Rules for Edges– HTTP Call Edge Updates– Site-2-Site or HTTP Call Rule Updates

Data Center to Cloud– Collection with Site-2-Site pull from the Cloud to HDFS– HTTP Call Rule Updates

• Easily as GET or POST Cache Updates• Bulk Site-2-Site Push


Home Collector High Level

Collection

Monitoring

Redact + Encrypt +Transfer

Aldrin Piri

Are you just using the funnels as endpoints to view the counts for the associated relationship?


1st Gen - Blocking, Sequential Writes

Decrypt->Local Disk -> S3 -> Database

Decrypt

To DiskTo S3To RDS


1st Generation Nifi - Separate Channels

Parallel Decryption = Additional Management

Decrypt

Decrypt


1st Gen - Separate Channels

Parallel Writes = Additional Management

To DiskTo Disk

Merge


2nd Gen - Asynchronous Writes – Still Parallel

Local Disk, S3, and RDS all isolated

Decrypt To Disk

To S3

To RDS


3rd Generation Nifi - Common Architecture

Add Tag

Decrypt

Add Tag


3rd Generation Nifi - Merge on Variable

Extremely Low Overhead

Merge To Disk


3rd Generation Nifi

Routing on Attributes Split Flow

To Disk


What We Discovered

Joe’s wife absolutely hates being spied on Pre-empt Arrival with MAC Addresses Validate Occupancy based on connected MAC Addresses Hackers Attacking the Network High Level of Control vs. Hub Software Motion almost useless as standalone factor.


Challenges // Other Lessons Learned

Oracle vs. Open JDK for ARM Improperly Configured Backpressure FlowFile Names Site to Site Configuration Issues Execute Process has ambiguous failure modes Merging Files Larger SmartHome Hub interface

Aldrin Piri

Just want to additionally underscore how absolutely putrid and terrible OpenJDK ARM is.


Q&ACome hang out with us @ The Dev Café

Date post:	16-Apr-2017
Category:	Technology
Upload:	dataworks-summithadoop-summit
View:	1,609 times
Download:	0 times