Introduction to Apache NiFi - Seattle Scalability Meetup

Introducing #ApacheNiFi

Saptak Sen [@saptak]Technical Product Manager, Hortonworks

© Hortonworks Inc. 2011 – 2015. All Rights Reserved

#seascale

Page 2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Agenda

• New Data Sources and the Rise of the Internet of Anything• Introducing: Hortonworks DataFlow powered by Apache NiFi• Key concepts, architecture, and use cases• Demo• Q&A


IoAT Data Grows Faster Than We Consume It

Much of the new data exists in-flight, between systems and devices as part of the Internet of AnythingNEW

TRADITIONAL

Ability to consume data

The OpportunityUnlock transformational business valuefrom a full fidelity of data and analyticsfor all data.

Geolocation

Server logs

Files & emails

ERP, CRM, SCM

Traditional Data Sources

Internet of Anything

Sensorsand machines

Clickstream

Social media


Interconnectedness Demands User CentricityChanges Organizations into Data Companies

Hortonworks Data Platformfor rich historical insights

from data-at-restNEW Hortonworks DataFlow

for securely collecting, conducting, and curating

data-in-motion while ALSO driving value for data-at-rest

analytics and use cases

Source: Gartner - Architecture Options for Big Data Analytics on Hadoop, July 2015


Simplistic View of IoAT & Data Flow

The Data Flow Thing

Process and Analyze DataAcquire Data

Store Data


Global interactions with customers, business partners, and thingsspanning different volume, velocity, bandwidth, and latency needs

Realistic View of IoAT and Data Flow


Meeting IoAT Edge Requirements

GATHER

DELIVER

PRIORITIZE

Track from the edge Through to the datacenter

Small Footprintsoperate with very little power

Limited Bandwidthcan create high latency

Data Availabilityexceeds transmission bandwidth

Data Must Be Securedthroughout its journey


Hortonworks Acquires Onyara

Turn Internet of Anything Data Into Actionable Insights• Onyara is the creator of and key contributor to Apache NiFi,

an open source solution for processing and distributing data.

• Over the past 8 years, Onyara engineers developed the U.S. government software project called “Niagara Files”, the precursor to Apache NiFi.

• Apache NiFi was made available as an Apache Incubator project through the NSA Technology Transfer Program in the Fall of 2014.

NEW Hortonworks DataFlow offering will securely and easily collect, conduct and curate any data, from anything, anywhere.


The IoAT Data Flow

Hortonworks Data Platformpowered by Apache Hadoop

Hortonworks Data Platformpowered by Apache Hadoop

EnrichContext

Store Data and Metadata

Internetof Anything

Hortonworks DataFlow powered by Apache NiFi

Perishable Insights

HistoricalInsights

Introducing Hortonworks DataFlow powered by Apache NiFi

Hortonworks DataFlow and the Hortonworks Data Platform deliver the industry’s most complete solution for management of Big Data.


Apache NiFi: Three key concepts

• Manage the flow of information

• Data Provenance

• Secure the control plane and data plane


Apache NiFi – Key Features

• Guaranteed delivery• Data buffering

- Backpressure- Pressure release

• Prioritized queuing• Flow specific QoS

- Latency vs. throughput- Loss tolerance

• Data provenance

• Recovery/recording a rolling log of fine-grained history

• Visual command and control

• Flow templates• Pluggable/multi-role

security• Designed for extension• Clustering


Common Apache NiFi Use Cases

Predictive AnalyticsEnsure the highest value data is captured and available for analysisComplianceGain full transparency into provenance and flow of data

IoT OptimizationSecure, Prioritize, Enrich and Trace data at the edge

Fraud DetectionMove sales transaction data in real time to analyze on demand

Big Data IngestEasily and efficiently ingest data into Hadoop

Value ResourcesGain visibility into how data sources are used to determine value


OS/Host

JVM

Flow Controller

Web Server

Processor 1 Extension N

FlowFileRepository

ContentRepository

ProvenanceRepository

Local Storage

OS/Host

JVM

Flow Controller

Web Server


FlowFileRepository

ContentRepository


Local Storage

Architecture

OS/Host

JVM

Flow Controller

Web Server


FlowFileRepository

ContentRepository


Local Storage

OS/Host

JVM

NiFi Cluster Manager – Request Replicator

Web Server

MasterNiFi Cluster Manager (NCM)

OS/Host

JVM

Flow Controller

Web Server


FlowFileRepository

ContentRepository


Local Storage

SlavesNiFi Nodes

High Availability: Control plane vs Data plane…


HDF – Powered by Apache NiFi


Add processor for data intake1 Drag and drop processor icon from the top menu


Choose the specific processor2 Choose one of the processors – currently 90 available – designed for extension


Example: Pick Twitter Processor


Configure the processor

3 Select processor and choose option to Configure

4

Adjust parameters as required


Another processor for data output5 Drag and drop processor icon from the top menu

6 Example: choose PutHDFS processor


Configure second processor7 Configure 2nd processor


Connect processors, configure connection

8


Click Start to begin processing

9


See processors update with real time changes

10As data flows, GUI interface updates in real time.


Dynamically adjust and tune data flow as needed

11 Dynamically adjust and tune dataflow as needed, in real time. Can also replicate data for testing and comparison.


Understand the data path with Data Provenance

14 Select Data Provenance


Trace lineage of a particular piece of data

15

Icon for Data Lineage


Every change to data is tracked: processing, views

16

Provenance event is tracked


Updates as changes happen

17 Updates as data flows


Easily access and trace changes to dataflow


Audit trail of Hortonworks DataFlow User Actions



Operations: Planned




Q & A


Date post:	16-Apr-2017
Category:	Software
Upload:	saptak-sen
View:	2,194 times
Download:	5 times

Introduction to Apache NiFi - Seattle Scalability Meetup

Software