AWS Webcast - Attunity Couchsurfing

transcript

Big Data Solution Showcase

Learn how Couchsurfing transformed 3 months of data

loading into 3 hours

Here is the url for the webinar:

https://connect.awswebcasts.com/p2j3kh6nmx3/

Presenters today

Brad Helicher

Director of Cloud Business

Attunity

Chris Keyser

Partner Solution Architect

Amazon Web Services

Charlie Killian

Founder & Owner

Bytecode IO

Thiago Bruch

Systems Engineer

Attunity

Agenda

• Introduction to AWS and big data

services

• Attunity Cloudbeam: moving the data

that moves your business into the cloud

• Customer case study: Couchsurfing

• Attunity Cloudbeam for Amazon Redshift:

how does it work & demo

• Q & A

What is AWS

AWS Global Infrastructure

Application Services

Networking

Deployment & Administration

DatabaseStorageCompute

Technologies and techniques for working

productively with data, at any scale.

Big Data

Creating Value from Data Assets

Recommendations, Collective Intelligence

Machine Learning

Visualization

DashboardsBusiness Intelligence

Measuring Functionality and Services

Ad Hoc QueriesA/B Testing

Hypothesis Testing & Predictions

Statistical Analysis

Learning from Social Media Conversations

Sentiment Analysis

SOCIAL

BIG DATA

Machine Learning DashboardsBusiness Intelligence

Ad Hoc QueriesA/B Testing

Statistical Analysis

Sentiment Analysis

Big Data AWS Cloud

Potentially Massive Data Sets Massive, virtually unlimited capacity

Iterative, experimental style of data

manipulation and analysis

Iterative, experimental style of

infrastructure deployment/usage

Frequently not a steady-state

workload;

peaks and valleys

Efficient with highly variable

workloads

Time to results is key

Parallel compute clusters from single

data source

Hard to configure/manage

Managed services for data storage

and analysis

AWS Data Services

Velocity

Variety

Volume

Structured, Unstructured, Text, Binary

Gigabytes, Terabytes, Petabytes

Millisecond, Second, Minute, Hour, Day

EC2EBS

Instance Storage

RedshiftRDS

SQL Stores

Hadoop

DynamoDB

Kinesis

Stream

Storage Services

S3Cloud

FrontGlacier

Elasticache

Caching

Pipeline

Orchestrate

Storage Services – Object Store

Amazon S3

Designed for 99.999999999% durability

Stores anything

Lifecycle and Versioning

Fine Grained Access Control

Reduced Redundancy Storage

Fast, simple, petabyte-scale data warehousing for less than $1,000/TB/Year

Amazon Redshift

MySQL, Oracle, SQLServer, Postgres

Backup/Restore, High Availability

Push Button Scalability

Up to 3 TB and 30K IOPS

Amazon

Nokia: 50% Cost Savings with 2x Faster Queries

Hadoop Tools Improving Rapidly

On-demand, Flexible, Big Data Technologies

Cheaper and Faster

Redshift & Hadoop Price-

performance Advantage over

>50% platform cost savings>2x

faster queries Minimal DBA

support

Redshift, Hadoop, S3, EMR, Data Pipeline for ETL

Cost-effective for 10s of TB data sets

AMI-based Services

Internet Speed Report Authoring

Hypothesis testing vs. waterfall

http://aws.amazon.com/marketplace

Big Data Case Studies

Learn from other AWS customers

aws.amazon.com/solutions/case-studies/big-data

AWS Marketplace

AWS Online Software Store

aws.amazon.com/marketplace

Moving the data that moves your

business in the cloud.

Data Transfer throughout On-Premises,

Amazon RDS, S3, & EC2

Brad Helicher

Director of Cloud Business

Brad.Helicher@attunity.com

+1-954-946-2274, ext. 1105

Thiago Bruch

Services Support Engineer

We Move the Data that Moves Your Business

Data Load/Replication

Change Data Capture

Data Access/Federation

Release Automation

Managed File Transfer

Enterprise File Replication

On-Premises to Cloud

Cloud to On-Premises

Cloud to Cloud

Customer Support

Professional Services and

Consulting

Cloud & Data Transfer: Use Cases

» Amazon Redshift: » Data Loading for BI / Analytics

» Amazon RDS / DBs on EC2: » Migrations from legacy / on-prem systems to AWS

» System Cutovers

» Disaster Recovery (on-prem and across AWS regions)

» Amazon S3: » Content Availability

» Disaster Recovery

» Hadoop EMR

» Amazon EC2 (File Replication):» Migrations from legacy / on-prem systems to AWS

» System Cutovers

» Disaster Recovery (on-prem and across AWS regions)

www.attunity.com/cloud

1. Complexity

2. Takes too long

3. Costs too much

4. Not real-time

5. Lack of Developer Resources

Data Replication: Common Challenges

CloudBeam – Value Proposition

» Key Features:

» Move any data, anytime,

anywhere

» Accelerated transfers

» Quick & Easy Setup

» Intuitive Administration

» Extensive Automation

» Real-time Monitoring

» Affordable Pay-per-Use

» Business Benefits:

» Ensure Information

Availability across the

Enterprise and the Cloud

» Overcome Data Transfer

Bottlenecks

» Quick Time-to-Value

Attunity CloudBeam

Customer Success Story: Couchsurfing

High-Performance Information Availability Solutions. Made Radically Simple. 21

Who is Couchsurfing?

Enables a global network of travelers to exchange cultural

experiences via hosted stays and more.

10 millionMembers

120,000Cities

100,000Event Attendees

Who is Bytecode IO?

Data and system integrators specializing in

ETL, APIs, Data Warehousing and Big Data.

charlie.killian@bytecode.io • http://www.bytecode.io/

Problem #1

No data insight.

Millions of hosting requests sent.

But what does it mean?

Rely on data instead of mythology.

Problem #2

Too much data for current architecture.

Solution

Near real-time

ODS(Operational Data Store)

Benefits: Short-Term

ODSonline

Analyst

Benefit: Long-Term

Warehouse

Technology Selection

• ODS

• Data Warehouse

• Data Integration

Warehouse

ODSData

Integration

Amazon Redshift

Warehouse

• No capital outlay

• Offered as a service

• Can be implemented with limited internal resources

• Little operational overhead

• Data sources already in AWS RDS MySQL

• Combine ODS and Data Warehouse

Data Integration Challenges

• Aggregate large amounts of disjointed data

• Map and convert data types

• Build destination tables

• Cleanse data

• Perform Change Data Capture (CDC)

• Limit ongoing maintenance

Simplification and the AWS Marketplace

• Manual development estimated at 3 months

• Development cycle needed to be shortened

• Wanted the whole process to be simplified

• Searched the AWS Marketplace for solutions

Attunity CloudBeam for Amazon Redshift

ODS(Amazon Redshift)

CloudBeam

• End-to-End database to Amazon Redshift loading

• Near real-time

• Incremental CDC

• Easy setup and configuration

• Simple job management

• Affordable

Attunity Solution

AWS Region

Amazon

Redshift

Amazon

RDS MySQLEC2 Machine

Windows Server

EC2 Machine - M3.Large

Windows Server

Attunity AMI (data pass through)

Full Stack Solution

Results

Relying on data instead of mythology.

3 months 3 hours

ATTUNITYMANUAL PROCESS

Data insight!

Solutions for On-Prem Sources

38www.attunity.com/cloud

AWS Region

EC2 Machine

M3.Large

Attunity AMI

RedshiftS3

On-Premises

Source DB

» How it’s Offered:

» Available on-demand via the

Amazon Marketplace

» Licensing: Hourly or

Bring Your Own License

» Supported Source DBs:

» Oracle

» SQL Server

» MySQL… and many others

Solutions for Data Sources within AWS

AWS Region

RedshiftS3

Source DB

On EC2

EC2 Machine

Windows Server

Attunity Replicate Software

EC2 Machine - M3.Large

Windows Server

Attunity AMI (data pass through)

What’s Needed to Get Started?

AWS Region

EC2 Machine

M3.Large

Attunity AMI

RedshiftS3

On-Premises

Source DB

» Windows 64-bit Server» Located in same local network as

source DB(s)

» Hosts ‘Attunity Replicate’ software

» Amazon Redshift Driver & Database

Clients installed

» Ports: 5746 & 5439

» Amazon S3 &

Amazon Redshift instances

» ‘Attunity CloudBeam for

Amazon Redshift’

» Via Amazon Marketplace

» Hourly or BYOL

Web-based Designer and

Management Console

Target

Database

Replication Server

In Memory Processing

Transform

Filter

Persistent Store

Source

Database

Transaction

Bulk Reader

Bulk Loader

Stream Loader

Data / Metadata

Attunity CloudBeam - Replicate-for-Redshift

Method & Processes

How it Works

AWS Region

EC2 Machine

M3.Large

Attunity AMI

RedshiftS3

On-Premises

Source DB

1. Attunity Replicate extracts data

from source database

2. Attunity Replicate applies any

transformations & filters

3. Data files generated

4. Optimized Transfer from Attunity

Replicate server to EC2 machine.

6. ‘Instruction Channel’ used to

execute COPY command.

Data loaded into Amazon Redshift.

7. Incremental Loading via CDC,

applied inside Redshift using merge commands

5. Data passes through, staging in

Amazon S3.

Attunity CloudBeam - Performance

» Optimized transfer protocol

» Data transfer technologies:

» Leverages Amazon multi-part transfers

» Concurrent Sessions / Transfers

» Compression

» Recoverability, Guaranteed Delivery

» SSL Encryption

» Performance Gains:

» 10-12x over standard copy

Thank You

On to the Demo …

Thiago Bruch

Senior Engineer

Get in touch with us…

Attunity Contact:www.attunity.com/cloud

Learn.attunity.com/cloudbeam

Meet us at re:Invent!Attunity – Booth 639

Thank you!