Big data for cio 2015

transcript

Zohar Elkayam CTO, Brillix

Zohar@Brillix.co.il

Big Data For CIOs

Who am I?

• Zohar Elkayam, CTO at Brillix

• DBA, team leader, and a senior consultant for over 17 years

• Oracle ACE Associate

• Involved with Big Data projects since 2011

• Blogger – www.realdbamagic.com

http://brillix.co.il2

About Brillix

• Brillix is a leading company that specialized in Data Management

• We provide professional services and consulting for Databases, Security and Big Data solutions

Agenda: Big Data

• Big Data • Why • What• Where• Who and How

• A Big Data Solution: Hadoop

• NoSQL vs. RDBMS

4 http://brillix.co.il

What is Big Data?

"Big Data"??

Different definitions

“Big data exceeds the reach of commonly used hardware environments and software tools to capture, manage, and process it with in a tolerable elapsed time for its user population.” -Teradata Magazine article, 2011

“Big data refers to data sets whose size is beyond the ability of typical database software tools to capture, store, manage and analyze.” - The McKinsey Global Institute, 2012

“Big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools.” -Wikipedia, 2014

Success Stories

MORE stories..

• Crime Prevention in Los Angeles

• Diagnosis and treatment of genetic diseases

• Investments in the financial sector

• Generation of personalized advertising

• Astronomical discoveries

Examples of Big Data Use Cases Today

MEDIA/ENTERTAINMENTViewers / advertising effectiveness

COMMUNICATIONSLocation-based advertising

EDUCATION &RESEARCHExperiment sensor analysis

CONSUMER PACKAGED GOODS

Sentiment analysis of what’s hot, problems

HEALTH CAREPatient sensors, monitoring, EHRsQuality of care

LIFE SCIENCESClinical trialsGenomics

HIGH TECHNOLOGY / INDUSTRIAL MFG.Mfg qualityWarranty analysis

OIL & GASDrilling exploration sensor analysis

FINANCIALSERVICESRisk & portfolio analysis New products

AUTOMOTIVEAuto sensors reporting location, problems

RETAILConsumer sentimentOptimized marketing

LAW ENFORCEMENT & DEFENSEThreat analysis - social media monitoring, photo analysis

TRAVEL &TRANSPORTATIONSensor analysis for optimal traffic flowsCustomer sentiment

UTILITIESSmart Meter analysis for network capacity,

ON-LINE SERVICES / SOCIAL MEDIAPeople & career matchingWeb-site optimization

Most Requested Uses of Big Data

• Log Analytics & Storage• Smart Grid / Smarter Utilities• RFID Tracking & Analytics• Fraud / Risk Management & Modeling• 360° View of the Customer• Warehouse Extension• Email / Call Center Transcript Analysis• Call Detail Record Analysis

The Challenge

The Big Data Challenge

Volume

• Big data come in one size: Big.

• Size is measured in Terabyte(1012), Petabyte(1015), Exabyte(1018), Zettabyte (1021)

• The storing and handling of the data becomes an issue

• Producing value out of the data in a reasonable time is an issue

Some numbers

• How much data in the world?• 800 Terabytes, 2000• 160 Exabytes, 2006 (1EB = 1018B)• 4.5 Zettabytes, 2012 (1ZB = 1021B)• 44 Zettabytes by 2020

• How much is a zettabyte?• 1,000,000,000,000,000,000,000 bytes• A stack of 1TB hard disks that is 25,400 km high

Growth Rate

• How much data generated in a day?• 7 TB, Twitter• 10 TB, Facebook

Data grows fast!

Variety

• Big Data extends beyond structured data: including semi-structured and unstructured information: logs, text, audio and videos.

• Wide variety of rapidly evolving data types requires highly flexible stores and handling.

Structured & Un-Structured

Un-Structured Structured

Objects Tables

Flexible Columns and Rows

Structure Unknown Predefined Structure

Textual and Binary Mostly Textual

Big Data is ANY data

• Some has fixed structure

• Some is “bring own structure”

• We want to find value in all of it

Unstructured, Semi-Structure and Structured

Data Types by Industry

Velocity

• The speed in which the data is being generated and collected

• Streaming data and large volume data movement

• High velocity of data capture – requires rapid ingestion

• Might cause the backlog problem

Global Internet Device Forecast

Internet of Things

Veracity

• Quality of the data can vary greatly

• Data sources might be messy or corrupted

So, What Defines Big Data?

• When we think that we can produce value from that data and want to handle it

• When the data is too big or moves too fast to handle in a sensible amount of time

• When the data doesn’t fit conventional database structure

• When the solution becomes part of the problem

Why Big Data Now?

• Because we have data:• Data is born already in digital form• 40% of data growth per year

• Because we can:• 500$ for a drive in which to store all the music of the world• 40 years of Moore's Law = large computational resources

• 64% of organizations have invested in big data in 2013• 34 billion $ invested in big data in 2013

“Because we reached dead end with logic”

How to do Big Data

Big Data in Practice

• Big data is big: technological infrastructure solutions needed

• Big data is messy: data sources must be cleaned before use

• Big data is complicated: need developers and system admins to manage intake of data

Big Data in Practice (cont.)

• Data must be broken out of silos in order to be mined, analyzed and transformed into value

• The organization must learn how to communicate and interpret the results of analysis

Infrastructure Challenges

• Infrastructure that is built for:• Large-scale• Distributed• Data-intensive jobs that spread the problem across clusters of server

Infrastructure Challenges (cont.)

• Storage:• Efficient and cost-effective enough to capture and store terabytes, if

not petabytes, of data• With intelligent capabilities to reduce your data footprint such as:

• Data compression• Automatic data tiering• Data deduplication

Infrastructure Challenges (cont.)

• Network infrastructure that can quickly import large data sets and then replicate it to various nodes for processing

• Security capabilities that protect highly-distributed infrastructure and data

Goals of Analytics

Positions in Big Data management

• DevOps are handling the infrastructure – sys admins and cluster manager

• Data scientists are in charge of producing value from the data

Data Scientist

Hadoop

Apache Hadoop

• Open source project run by Apache (2006)• Hadoop brings the ability to cheaply process large amounts of

data, regardless of its structure• It Is has been the driving force behind the growth of the big

data Industry• Get the public release from:

• http://hadoop.apache.org/core/

Hadoop Creation History

Key points• An open-source framework that uses a simple programming model to

enable distributed processing of large data sets on clusters of computers.

• The complete technology stack includes• common utilities• a distributed file system• analytics and data storage platforms• an application layer that manages distributed processing, parallel

computation, workflow, and configuration management• Cost-effective for handling large unstructured data sets than

conventional approaches, and it offers massive scalability and speed

Why use Hadoop?

Cost Flexibility

Near linear performance up

to 1000s of nodes

Leverages commodity HW & open source SW

Versatility with data, analytics &

operation

Scalability

What Hadoop Is Not?

• Hadoop does not replace DW or relational databases

• Hadoop is not for OLTP or real-time systems

• Very good for large amount, not so much for smaller sets

• Designed for clusters – there is Hadoop monster server (single server)

Hadoop Cluster in Yahoo

Cluster of machine running Hadoop at Yahoo! (credit: Yahoo!)

http://brillix.co.il

Hadoop under the Hood

Hadoop Main Components

• HDFS: Hadoop Distributed File System – distributed file system that runs in a clustered environment.

• MapReduce – programming paradigm for running processes over a clustered environments.

HDFS is...

• A distributed file system• Redundant storage• Designed to reliably store data using commodity hardware• Designed to expect hardware failures• Intended for large files• Designed for batch inserts• The Hadoop Distributed File System

MapReduce is...

• A programming model for expressing distributed computations at a massive scale

• An execution framework for organizing and performing such computations

• An open-source implementation called Hadoop

MapReduce is good for...

• Embarrassingly parallel algorithms

• Summing, grouping, filtering, joining

• Off-line batch jobs on massive data sets

• Analyzing an entire large dataset

MapReduce is OK for...

• Iterative jobs (i.e., graph algorithms)

• Each iteration must read/write data to disk

• IO and latency cost of an iteration is high

MapReduce is NOT good for...

• Jobs that need shared state/coordination• Tasks are shared-nothing• Shared-state requires scalable state store

• Low-latency jobs• Jobs on small datasets• Finding individual records

Spark• Fast and general MapReduce-like engine for large-scale data

processing• Fast

• In memory data storage for very fast interactive queries Up to 100 times faster then Hadoop

• General• Unified platform that can combine: SQL, Machine Learning , Streaming ,

Graph & Complex analytics• Ease of use

• Can be developed in Java, Scala or Python • Integrated with Hadoop

• Can read from HDFS, HBase, Cassandra, and any Hadoop data source.

Key Concepts

Resilient Distributed Datasets• Collections of objects spread

across a cluster, stored in RAM or on Disk

• Built through parallel transformations

• Automatically rebuilt on failure

Operations

• Transformations(e.g. map, filter, groupBy)

• Actions(e.g. count, collect, save)

Write programs in terms of transformations on

distributed datasets

Unified Platform

• Continued innovation bringing new functionality, e.g.:• Java 8 (Closures, LambaExpressions)• Spark SQL (SQL on Spark, not just Hive)• BlinkDB(Approximate Queries)• SparkR(R wrapper for Spark)

Big Data and NoSQL

The Challenge

• We want scalable, durable, high volume, high velocity, distributed data storage that can handle non-structured data and that will fit our specific need

• RDBMS is too generic and doesn’t cut it any more – it can do the job but it is not cost effective to our usages

The Solution: NoSQL

• Let’s take some parts of the standard RDBMS out to and design the solution to our specific uses

• NoSQL databases have been around for ages under different names/solutions

Example Comparison: RDBMS vs. Hadoop

Typical Traditional RDBMS Hadoop

Data Size Gigabytes Petabytes

Access Interactive and Batch Batch – NOT Interactive

Updates Read / Write many times Write once, Read many times

Structure Static Schema Dynamic Schema

Scaling Nonlinear Linear

Query Response

Can be near immediate Has latency (due to batch processing)

Best Used For:

Structured or Not (Flexibility) Scalability of Storage/Compute Complex Data Processing Cheaper compared to RDBMS

Relational Database

Best Used For:

Interactive OLAP Analytics (<1sec)

Multistep Transactions 100% SQL Compliance

Best when used together

Hadoop And Relational Database

The NOSQL Movement

• NOSQL is not a technology – it’s a concept

• We need high performance, scale out abilities or agile structure

• We are willing to sacrifice our sacred database cows: consistency, transactions, durability

• Over 150 different brands and solutions (http://nosql-database.org/).

Is NoSQL a RDMS Replacement?

Well... Sometimes it does…

NoSQL Taxonomy

Type Examples

Key-Value Store

Document Store

Column Store

Graph Store

Key Value Store

• Distributed hash tables• Very fast to get a single value• Examples:

• Amazon DynamoDB• Berkeley DB• Redis• Riak• Cassandra

Document Store

• Similar to Key/Value, but value is a document• JSON or something similar, flexible schema• Agile technology• Examples:

• MongoDB• CouchDB• CouchBase

What is a Column Store Database?

• Column Store databases are management systems that uses data managed in a columnar structure format for better analysis of single column data (i.e. aggregation). Data is saved and handled as columns instead of rows.

• Examples:• HP Vertica• Pivotal (EMC) GreenPlum• Hadoop Hbase• Amazon’s SimpleDB• Cassandra

Query Data

• When we query data, records are read at the order they are organized in the physical structure

• Even when we query a single column, we still need to read the entire table and extract the column

Col 1 Col 2 Col 3 Col 4

Select Col2 From MyTable

Select *From MyTable

How Does Column Stores Keep Data

Organization in row store Organization in column store

Select Col2 From MyTable

Row Format vs. Column Format

Graph Store

• Inspired by the graph theory• Data model: nodes, relationships, properties on both sides• Relational database have a hard time to represent a graph in

the Database• Example:

• Neo4j• InfiniteGraph• RDF

Graph Example

Conclusion• We do Big Data to gain Value. Without value, there is no Big Data

• Handling Big Data is a challenge – we talked about who uses it, when and where

• Hadoop is a solution for Big Data usages but it’s not a magical solution

• NoSQL, NewSQL and RDBMS are all solutions we can integrate for different usages

• New organizational positions: cluster devops and data scientist.

Thank You

Zohar Elkayamtwitter: @realmgicZohar@Brillix.co.il

www.realdbamagic.com

Big data for cio 2015

Technology