Analytics in the cloud

transcript

Analytics in the CloudNatalino Busa - Head of Data Science

2 Natalino Busa - @natbusa

Distributed computing Machine Learning

Statistics Big/Fast Data Streaming Computing

Head of Applied Data Science at Teradata

On most networks:

@natbusa

Let’s define Cloud Services

Analytics in the cloud: stacking layers

Bare Metal: Physical Machines

IAAS: Virtual Resources

CAAS: Containers,

dPAAS: Datastores, Data Engines iPAAS: Tools Integration, Flows & Processes

CAAS: Containers,

DAAAS: Data Analytics as a Service

Watson ServicesAzure ML

GoogleCloud MLBigML

Analytics in the cloud: today’s talk

CAAS: Containers,

DAAAS: Data Analytics as a Service

“we live in an age of open source datacenters, so we can stack all these things together and we have open source from the ground to ceiling.”

Sam Ramji, CEO of Cloud Foundry

https://www.youtube.com/watch?v=7oCSFcUW-Qk

Containers vs VMs

Techs based on Containers

Containers as a Service

https://aws.amazon.com/ecs/

For example: Amazon ECS

CaaS: 6 offerings

https://www.linux.com/news/5-container-service-tools-you-should-know-about

Project Magnum

Amazon ECSDocker DataCenterGoogle

Container Engine

Most new PaaS solutions are containerized

PaaS: Big Data SQL Queries

Batch OrientedLarge Aggregations

Interactive QueriesData Exploration

Interactive QueriesMachine Learning

Streaming:Micro-batching

Interactive QueriesMachine Learning

Streaming:Event-driven

Advanced Analytics: models and algorithms

PaaS: Advanced Analytics

Graph analytics:

- Cluster items- Extract similarities- Detect patterns

Text analytics:

- Sentiment Analysis- Language Detection- Summarization- Entity extraction

Machine Learning:

- Classification- Regression- Clustering- Forecasting- Anomaly detection

AI and Deep Learning- Unstructured Data- Object Detection- Natural Language Processing- Video Summarization- Speech Recognition

SQL + Graph + Text + Machine Learning + Voice/Image/Video

dPaaS: Machine (deep) Learning

… this are just a few examples ...

Analytics Everywhere

Public Cloud Managed Cloud Private Cloud Private Infra

iPaas: Components for Analytics in the Cloud

SQL : Big Data Data Warehousing

Machine LearningObjects Stores

Streaming Computing

SQL: RelationalTransactional DB

iPaas, dPaaS:

Objects Stores

HDFSGlusterFSCephFSNFSSwiftNovaCassandraRedis

S3 (AWS)Storage (GCP)...

iPaas, dPaaS:

NoSQLObjects Stores

CassandraRedisHBaseAccumulo

Neo4JElasticSearchMongoDBCouchbase

BigTable (GCP)DynamoDB

iPaas, dPaaS:

NoSQLObjects Stores

MySQLPostgreSQLMariaDB

Oracle (AWS MP)

iPaas, dPaaS:

NoSQLObjects Stores

Oracle (AWS MP)

HivePrestoSpark SQLImpala

Redshift (AWS)BigQuery (GCP)Big SQL (IBM)

Teradata (AWS MP)SAP Hana(AWS MP)Vertica (AWS MP)

iPaas, dPaaS:

NoSQL Machine Learning

Objects Stores

Oracle (AWS MP)

Spark MLH2OFlinkAreosolveTheanoTensorflowXGboost

Azure MLAWS MLGoogle MLIBM Watson

iPaas, dPaaS:

NoSQL Machine Learning

Objects Stores

Streaming Computing

Oracle (AWS MP)

Spark MLH2OFlinkAreosolveTheanoTensorflowXGboost

Azure MLAWS MLGoogle MLIBM Watson

Heron (Storm)NiFiSpark StreamingFlinkKafka StreamsLogstashStreamSQL

Google DataFlow (GCP)

iPaaS: Selecting your Analytical Stack

� Flexible. Powerful.- Combinations for this example:

8 * 3 * 4 * 8 * 7 * 7 = 37632

� Right tool for the right job- Fit for purpose- Multi-Genre Analytics

Hard to maintain and upgrade:- Extended Skills and Know-how- Components upgrades must be compatible

Hard to configure: - no matter if cloud or bare or vms- complex stacks with many tools and services

iPaaS: Deploy & Manage your own Analytics

How to simplify? Select a bundle!

iPaaS: bundled recipes & stacks

Select a recipe:- Hortonworks Data Platform- Cloudera Data Platform- Reactive Platform - Smack Stack- Pancake Stack- ELK Stack- Select your own

iPaaS: my favs analytical stacks

Objects Stores

NoSQL SQL : Big Data Data Warehousing

Machine Learning Streaming Computing

All Hadoop (5) HDFS Hbase Hive Spark Storm

Smack stack (2) Cassandra Cassandra Spark Spark Spark

Elastic (5) HDFS ElasticSearch Hive H2O Kafka

Data Science (8) HDFS ElasticSearch Hive, Presto Spark, H2O, Tensorflow Flink

Real Time (2) Cassandra Cassandra Flink Flink Flink

dPaaS: Managed Analytics

This is hard ! Can we access it as a service?

dPaaS: Managed Hadoop & Spark

HDInsight: Hadoop, Spark, and R as services

Managed Spark Clusters, BigInsight (Hadoop)

DataFlow and DataProc: Flink, Spark and Hadoop Clusters as a Service

EMR: Hadoop components a la carte

PaaS: Analytical clusters

Ephemeral

Create then Dispose

Clusters are Short-Lived

Data Exploration

Isolated, Personal

Simple Access Management

Interactive Analytics

Permanent

Clusters are Long Lived

Scheduled Operations

Production ETL

Co-Ordinated

Complex Access Management

Batch Analytics

DAaaS: Microsoft’s Cortana and ML Studio

DAaaS: IBM Watson

DAaaS: Google ML and AI as a service

Cloud Computing forDeep Neural Networks > Train, Score, Data

AI and ML models for:

● Speech (audio)● Language (text)● Vision (images/video)

Summary

• Analytics in the Cloud:

The dawn of a new computing era

• IPaas, dPaas:

complexity vs flexibility, it’s a tradeoff

• Computing clusters:

Ephemeral and Persistent

Head of Applied Data Science at Teradata

Distributed computing Machine Learning

Statistics Big/Fast Data Streaming Computing

Linkedin and Twitter:

natbusa

Analytics in the cloud

Data & Analytics