Moving Big Data Into Cloud Object Storage

Matt Yanchyshyn, AWS Solutions Architect

Nelson Hsu, Signiant VP Business Development & Alliances

www.Signiant.com/flight | @signiant

How do we define Big Data?

• When your data sets become so large that you have to start innovating around how to store, organize and transfer them.

• Anytime you are gathering data or using data someone else has gathered for analytics.

www.Signiant.com/flight | @signiant

Unconstrained growth: Big Data is moving fast

From application server logs, web sites and mobile apps to sensor output, high definition film and satellite imagery, data is growing at an unconstrained and exponential rate.

www.Signiant.com/flight | @signiant

• There is no limit on the number of Objects

• Object size can be up to 5TB

• Central data storage for all of your systems

• High bandwidth

• 99.999999999% durability

• Versioning, Lifecycle Policies and Glacier Integration

Why is Amazon S3 good for Big Data?

www.Signiant.com/flight | @signiant

Moving Big Data to and from Amazon S3

Signiant launched a new product last year called Flight that provides an easy way for AWS customers to push large amounts of data into Amazon S3 (and easily pull it back out) without worrying about managing any cloud infrastructure.

www.Signiant.com/flight | @signiant

• When users frequently move large data sets into Amazon S3, like for processing with Amazon EMR and Amazon Redshift.

• For batch-file transfers using manifests. For example, if you’ve pre-aggregated and compressed your data in order to optimize Hadoop.

Signiant Flight is an easy way to move data to and from S3 at high speeds:

AWS Big Data Services

Amazon EMR Amazon S3 Amazon DynamoDB Amazon Glacier Amazon Redshift

AWS Data Pipeline

Amazon Kinesis

AWS Big Data Services

Amazon EMR Amazon S3 Amazon DynamoDB Amazon Glacier Amazon Redshift

AWS Data Pipeline

Amazon Kinesis

www.Signiant.com/flight | @signiant

Enable cloud-based workflows

On-premises storage optimization Large scale ingest

Accelerate big data analytics

A few ways businesses are using Flight

www.Signiant.com/flight | @signiant

High Availability Cost Effectiveness

Global Performance Easy to Deploy & Use

Elasticity Rapid Innovation

24/7

Flight is hybrid Software-as-a-Service (SaaS)

Flight is the only SaaS solution on the market for accelerated file transfers to and from cloud object storage. The SaaS component makes Flight unique for a several reasons:

www.Signiant.com/flight | @signiant

While BYOL (bring your own license) models certainly have their place, there is a significant difference between them and Subscription + Management payment models like Signiant’s.

Namely, with BYOL, you still have to manage, maintain and support your own servers.

With Signiant’s subscription service, they cover all of that for you, significantly reducing both Opex and Capex.

Not all SaaS is created equalSubscription + Management vs. BYOL

www.Signiant.com/flight | @signiant

Signiant Flight eliminates the overhead of managing compute resources in the cloud. Signiant manages the server-side component—the Amazon EC2 instances running Flight servers and the Amazon S3 transfer components—while end users run a lightweight, client-side agent.

All you have to do is:

1. Install the local client

2. Authenticate with AWS and set which Amazon S3 bucket to use

1. Start transferring files

A fully managed service

www.Signiant.com/flight | @signiant

Signiant SaaS Control

Signiant Managed Cloud Servers & Software

Customer’s Cloud Storage Tenancy

Customer’s Network

Cloud Infrastructure

Signiant SaaS Control

www.Signiant.com/flight | @signiant

Highly reliable without a complex setup

When you use Signiant Flight to send files to Amazon S3, its backend automatically scales during high-volume transfer cycles.

Flight’s backend is load-balanced across multiple Amazon EC2 instances spread across multiple AWS Availability Zones, so it is highly reliable without passing the complexity of configuration and management on to you.

www.Signiant.com/flight | @signiant

Importantly, Signiant’s file transfer protocol also supports two features that are not supported in Tsunami UDP:

1. AES-256 bit encryption2. Intelligent file transfer retries

If a transfer is interrupted for any reason, the transfer is restarted (using numerous file retry algorithms) and continues transferring from the point of interruption. If a file already exists in Amazon S3 and hasn’t been changed, Flight won’t upload the file.

Encryption and Checkpoint Restart

www.Signiant.com/flight | @signiant

Signiant’s patented accelerated file transfer protocol is often called UDP acceleration, but it actually implements both an advanced TCP on top of UDP and an advanced FTP. If that intrigues you, read more about it here.

Basically, this minimizes the impact of WAN latency on throughput which results in considerably faster transfers, especially for large files transferred over long distances.

Once files arrive on Signiant Flight’s AWS-based backend, servers managed by Signiant write the data directly into Amazon S3 over HTTPS with the multipart upload API.

Why is Flight so fast?

www.Signiant.com/flight | @signiant

Over long distances, Signiant technology minimizes the impact of latency, while being able to capitalize on increases in bandwidth.

TCP based protocols do not benefit from increases in bandwidth and are very inefficient over long distances due to latency.

www.Signiant.com/flight | @signiant

After you sign-up for Signiant Flight via the Signiant website:

1. Download Flight Client

1. Enter Amazon S3 Account Details

2. Start Moving Files

Setting up Signiant Flight

Note: Flight comes with a command-line interface and other client options. To learn more, check out Flight’s client options.

www.Signiant.com/flight | @signiant

Signiant’s Flight is an easy way to move big data into the cloud at high speed. Because it’s a SaaS solution, its highly available and high performance file transfer system is deployed and maintained for you.

Flight’s encryption in transit and intelligent file transfer guaranteed delivery means that you can send files securely and reliably.

It’s easy to use and to get started! A free trial is available.

Conclusion