Date post: | 15-Apr-2017 |
Category: |
Technology |
Upload: | safe-software |
View: | 1,592 times |
Download: | 2 times |
Ask us anything!
Don Murray Stewart Harper
Connect
Transform
Automate
FME: By Safe Software
AWS Data Storage
AWS Simple Storage
(S3)
AWS RDS / Aurora
Easy to operate, fault tolerant,
scalable Relational Database Service
in the cloud.
Secure, durable, highly-scalable object storage
AWS S3 - Overview
Secure, durable, highly-scalable object
storage
99.999999999% reliability
3+ million requests per second
S3 is highly performant and applications can get 100’s of requests a second
$0.03 per GB of storage (5TB of data $150 a month)
Can store single objects up to 5TB
AWS S3 - How it works
A simple key/value persistent
object store.
It is not a disk and doesn’t have
a directory or folder structure.
It can be presented as a folder structure
though.
Every object is identified with a key.
You have a large
amount of data on
premise.
When might you use S3?
You want to take
advantage of features
that are not available
on premises.You are serving
large amounts of
data and are
having problems
scaling.
You want
programmatic
access to your data
You wish to build
an event driven
architecture.
AWS S3 - Tips & Tricks
Host a static website on S3 for next to nothing. Every object in S3 is web addressable.
For even higher performance content delivery use in combination with CloudFront.
Version your data automatically or transition to Glacier using lifecycle management.
Provide time-limited access to objects via query string authentication.
Easy to operate, fault tolerant,
scalable Relational Database Service
in the cloud.
AWS RDS - OverviewAWS Aurora (MySQL compatible)
Oracle
MariaDB
PostgreSQL (PostGIS)
Microsoft SQL Server
MySQLSu
ppor
ted
Data
Typ
es
The code, applications, and
tools you use today with your existing databases can be used with Amazon
RDS.
The AWS RDS service lets you...
Deploy in minutes
Automate backups
Apply software patches automatically
Scale storage and performance with one click
Replicate to enhance availability and reliability
When might you use RDS?
If you don’t want to
worry about
infrastructure.
If you are looking to run a production
database with minimal administration,
perfectly suited for a small team.
If you don’t want to
worry about scaling,
fault tolerance and
enhanced availability
as you grow.
On-Premises vs RDS PostGIS
Specification: 16 Cores, 122GB of RAM, Linux, 50TB Storage
RDS - Tips and Tricks● You can’t always install all the libraries you want, so check this before you
migrate. For example urllib Python lib is not included.● You don’t have SUPER privileges, which can make it impossible to perform
advanced operations against the DB.
Native Spatial Support Enterprise Geodatabase Support
AWS AuroraDatabase engine runs on EC2 and is compatible with the Open Source MySQL, logging and storage runs on Amazon’s storage layer. This is where most of the gains have come from.
RDS vs Aurora
Aurora only supports the InnoDB storage engine. That rules out spatial index support.
The gains are mainly around fault toleranceand performance. If you aren’t running a high throughput transactional database it might not be worth migrating.
Aurora is only available on larger instances (r3 family), so it is more expensive if you only need a small database.
Cross region replication is not supported by Aurora, it is with RDS.
Other Key AWS Services
Simple Queueing Service
Lambda
Simple Notification Service
API Gateway
Event Driven Data Processing
The event driven design patternDeploying an event-driven
architecture is difficult and requires
custom development.
Lambda, Amazon’s new compute service, coupled with FME gives you an extremely powerful set of tools for the elastic processing of hundreds of datasets.
Amazon S3, Amazon Kinesis and Amazon Dynamodb trigger events.
Automate data processing pattern with AWS Lambda, S3, SNS and SQS
Event Driven Processing Demo
Automatically process data when it is dropped into an S3 bucket, send an email once the data has finished processing and place output in another S3 bucket.
Incoming Data
S3 Bucket S3 Bucket
Pelmorex Example
Pelmorex Example
Data Delivery
AWS Architecture for Data Delivery
AWS perfectly suited for data delivery solutions:
Multiple services for storing large volumes of data cheaply and securely.
Costs grows with the popularity of your data delivery service.
You can build highly performant workflows with the click of a button.
Caching a snapshot of your data on S3
Real time Data Delivery
Architecture - State Arkansas Data Portal
Our rack space costs (real estate on data center floor) are $3200 a month. Add to that the hardware costs, etc and you can see why moving to the cloud was a no brainer for us.
Projected Costs - State Arkansas Data Portal
On Premises AWS
Anthony Davis, State Arkansas
APIs - Next Generation of Data Delivery
● APIs allow businesses to build platforms that partners and customers can use to access core business systems, whenever they want, in a stable and secure way.
● We now have the tools to create a fully functioning scalable API in less than a day, this is going to revolutionize businesses and bring the next wave of API value—creating internal disposable APIs.
Building an API with no infrastructure or code
API Gateway - DemoBuilding an API on top of a database in less than 5 minutes.
Free TrainingHands-on and instructor led.
FME Desktop Basic
● Nov 17-18
FME Server Authoring
● Dec 1-2www.safe.com/training
AWS Doctor's Office
Coming soon - Safe’s AWS experts on hand for 1H.
Q& A