Co-hostedbyA-unityandHortonworksSeptember22,2016

Accelera'ngReal-TimeDataIngestintoHadoop

Today’sSpeakers

Ã AjaySingh–SeniorDirectorofTechnicalAlliances,Hortonworks

Ã JordanMartz–DirectorofTechnologySoluIons,A-unity

TheGrowthofData

44ZBBy2020,upfrom2.8

ZB2012 31%Ofenterprisesexpecttomanage1PBormore

64%OfenterprisesinvesInginbig

data

88%ofdataisnotyetunder

management

IOTisDoublingDataEvery2Years

2.8Gigabytespersecond

26millionImesin12-hoursonebillion

hourspermonth

OpenConnectedDataPlaQorms

DataatRestHDP

DatainMo'onHDF

ACTIONABLEINTELLIGENCE

ModernDataApplica'ons

TheFutureoftheEnterpriseisAboutAllData

TheShiVtoModernDataArchitectureisHappeningNow

DataflowManagement

7 ©HortonworksInc.2011–2016.AllRightsReserved

Simplis'cViewofDataflows:Easy,Defini've

AcquireData

StoreData

DataFlow

ProcessAnalyzeData

8 ©HortonworksInc.2011–2016.AllRightsReserved

Realis'cViewofDataflows:Complex,Convoluted

AcquireData

StoreData

AcquireData

StoreData

StoreData

StoreData

StoreData

Processand

AnalyzeData

DataFlow

AcquireData

AcquireData

9 ©HortonworksInc.2011–2016.AllRightsReserved

HDP HORTONWORKS DATA PLATFORM Powered by Apache Hadoop

HDFMakesBigDataIngestEasyComplicated,messy,andtakesweeksto

monthstomovetherightdataintoHadoop

HDP HORTONWORKS DATA PLATFORM

Streamlined,Efficient,Easy

HDP HORTONWORKS DATA PLATFORM Powered by Apache Hadoop

10 ©HortonworksInc.2011–2016.AllRightsReserved

Connec'ngDataBetweenEcosystemsWithoutCoding:170+Processors

Hash

Extract

Merge

Duplicate

Scan

GeoEnrich

Replace

ConvertSplit

Translate

RouteContent

RouteContext

RouteText

ControlRate

DistributeLoad

GenerateTableFetch

JoltTransformJSON

Priori'zedDelivery

Encrypt

Tail

Evaluate

Execute

HL7

FTP

UDP

XML

SFTP

HTTP

Syslog

Email

HTML

Image

AMQP

MQTT

AllApacheprojectlogosaretrademarksoftheASFandtherespecIveprojects.

Fetch

11 ©HortonworksInc.2011–2016.AllRightsReserved

The Connected Data Architecture & Attunity SO

URC

ES

OLTP ERP CRM MainFrame

DataIntegraIon&Ingests

AgunityReplicateforHDPandHDF

AccelerateIme-to-insightsbydeliveringsoluIonsfaster,withfresherdata,frommanysources-  Automateddataingest-  Incrementaldataingest(CDC)-  Broadsupportformanysources

© 2016 Attunity

You will learn how to

• How to ingest your most valuable data into Hadoop using Attunity Replicate

• About how customers are using Hortonworks DataFlow (HDF) powered by Apache NiFi

• How to combine the real-time change data capture (CDC) technology with connected data platforms from Hortonworks

© 2016 Attunity

• Accelerate data delivery across enterprise and cloud

• Empower rapid utilization of data by the business

• Continually optimize with intelligent insight

Attunity Corporate Overview

Over 2000 Customers in 65 Countries Financial Services Manufacturing / Industrials Government Health Care

Technology / Telecommunications Other Industries

Enterprise Data Management On Premise | Cloud | Across Platforms

Global Organization

USA

EMEA

APAC

© 2016 Attunity

Attunity Platform for Enterprise Data Management

•  Accelerate data delivery

•  Empower rapid utilisation of data

•  Continuously improve the management of data

Attunity Replicate

Attunity Compose Attunity Visibility

Universal Data Availability Data Warehouse Automation Metrics Driven Data Management

Integrate new platforms

Automate ETL/EDW

Optimise performance and cost

On Premises / Cloud

Hadoop Files RDBMS EDW SAP Mainframe

© 2016 Attunity

Success with Fortune 100 manufacturer

•  Centralize data for global analytics

•  Realize faster insights and competitive advantage

Goals

Results 4500 applications

DB2 MF SQL Oracle

200,000 employees >50 plants WW

•  High volumes of varied data across applications, geographies

•  Analytics delayed => opportunities lost

Challenges

•  Consolidating on Hadoop Data Lake with Kafka data brokers

•  Attunity Replicate integrated all sources and targets, minimal costs

Ingest thousands of DBs into HDP with Attunity Replicate

© 2016 Attunity

Hortonworks HDF and Attunity Replicate

© 2016 Attunity

The Connected Data Platform

© 2016 Attunity

The Connected Data Architecture & Attunity SO

URC

ES

OLTP,ERP,CRMSystems

Documents,Emails

WebLogs,ClickStreams

SocialNetworks

MachineGenerated

SensorData

GeolocaIonData

Data Integration & Ingests

Attunity Replicate for HDP and HDF

Accelerate time-to-insights by delivering solutions faster, with fresher data, from many sources -  Automated data ingest -  Incremental data ingest (CDC) -  Broad support for many sources

© 2016 Attunity

Realistic View of Dataflows: Complex, Convoluted

Store Data

Process and Analyze Data

Acquire Data

Store Data Store Data

Store Data

Store Data

Acquire Data

Acquire Data

Acquire Data

Dataflow

© 2016 Attunity

…

Sybase

DB2

SQL

Oracle

Attunity at the Heart of Your Data Strategy

Data Marts Business Units

Operational Systems

(Legacy)

Application Owners

Data Scientists

ETL Developers

Realtime Dashboards

Enterprise Data Warehouse

Enterprise Data Lake

ERP

CRM

POS

Legacy

Logs

Sensor

Files

…

Decision Support Systems

(Legacy)

Cloud Data Systems

Hadoop Kafka

Marketing

Finance

Sales

CxOs

Engineers

© 2016 Attunity

Zero-footprint Architecture

•  Lower impact on IT

•  No software agents on sources and targets for mainstream databases

•  Replicate data from 100’s of source systems with easy configuration

•  No software upgrades required at each database source or target

Hadoop

Files

RDBMS

EDW

Mainframe

•  Log based •  Source specific optimisation

Hadoop

Files

RDBMS

EDW

Kafka

© 2016 Attunity

Data Lake Ingests with Attunity Replicate: On-Prem & Clouds

Transfer

Transform Filter Batch

CDC Incremental

In-Memory

File Channel

Batch

Hadoop

Files

RDBMS

Data Warehouse

Mainframe

Cloud

On-prem

Cloud

On-prem

Hadoop

Files

RDBMS

Data Warehouse

Kafka

Persistent Store

© 2016 Attunity

In Memory and File Optimized Data Transport

Enterprise-class CDC for Data-At-Rest and Data-In-Motion

R1 R1 R2 R1 R2

R1 R2

Batch CDC

Data Warehouse Ingest-Merge

SQL n 2 1

SQL SQL Transactional CDC

Message Encoded CDC

Data Sources

AgunityReplicate–ChangeProcessing

CDC

Many Databases and Data

Warehouses

....

HDP

Gov

erna

nce

&

Inte

grat

ion

Secu

rity

Ope

ratio

ns Data Access

Data Management

YARN

© 2016 Attunity

"table": "table-name",

"schema": "schema-name",

"op": "operation-type",

"ts": "change-timestamp",

"data": [{"col1": "val1"}, {"col2": "val2"}, …., {"colN": "valN"}]

"bu_data": [{"col1": "val1"}, {"col2": "val2"}, …., {"colN": "valN"}],

Easily Create and Manage Kafka Endpoints

Eliminate manual coding • Drag and drop interface for

all sources and targets

• Monitor and control data stream through web console

• Bulk load or CDC

• Multi-topic and multi-partitioned data publication

Attunity Replicate

Command Line

© 2016 Attunity

T1/P0

T2/P1

T3/P0

Broker 1

Attunity Replicate for Kafka - Architecture

M0 M1 M2 M3 M4 M5 M6 M7 M8

M0 M1 M2 M3 M4 M5

M0 M1 M2 M3 M4 M5 M6 M7

T1/P1

T2/P0

Broker 2

M0 M1 M2 M3 M4

M0 M1 M2 M3 M4 M5 M6

© 2016 Attunity

CDC

Demo: Data Streaming into Kafka è HDF è HDP

MSG

n 2 1

MSG MSG

Data Streaming

Transaction logs

In memory optimised metadata management and data transport

Bulk Load

MSG

n 2 1

MSG MSG

Data Streaming

Message broker

Message broker

© 2016 Attunity

Heterogeneous – Broad support for sources and targets

RDBMS

Oracle SQL Server DB2 LUW DB2 iSeries DB2 z/OS MySQL Sybase ASE Informix

Data Warehouse

Exadata Teradata Netezza Vertica Actian Vector Actian Matrix

Hortonworks Cloudera MapR Pivotal

Hadoop

IMS/DB SQL M/P Enscribe RMS VSAM

Legacy

AWS RDS Salesforce

Cloud

RDBMS

Oracle SQL Server DB2 LUW MySQL PostgreSQL Sybase ASE Informix

Data Warehouse

Exadata Teradata Netezza Vertica Pivotal DB (Greenplum) Pivotal HAWQ Actian Vector Actian Matrix Sybase IQ

Hortonworks Cloudera MapR Pivotal

Hadoop

MongoDB

NoSQL

AWS RDS/Redshift/EC2 Google Cloud SQL Google Cloud Dataproc Azure SQL Data Warehouse Azure SQL Database

Cloud

Kafka

Message Broker

targets

sources

© 2016 Attunity

© 2016 Attunity

© 2016 Attunity

© 2016 Attunity

“I have a question… ”

© 2016 Attunity

•  Attunity Replicate enables continuous data ingest from operational databases and complements HDF with CDC to maximize the value of data in motion.

•  Download from Attachments Tab

HDF and Attunity Replicate Data Sheet

© 2016 Attunity

More Next Steps…

Joint Webinars http://hortonworks.com/webinar/accelerating-hadoop-success-rapid-data-integration-modern-data-architecture/ http://hortonworks.com/webinar/accelerate-real-time-data-collection-automate-transfer-apache-hadoop/

Learn More about Attunity & Hortonworks

Attunity.com/hortonworks Hortonworks.com/partner/Attunity/

Whitepaper “The Modern Data Architecture and Automating Data Transfer”

http://learn.attunity.com/whitepaper-modern-data-architecture-with-apache-hadoop