Qrious about Insights -- Big Data in the Real World

Qrious about InsightsBig Data in the Real World

AUT DSRG Workshop

Guy Kloss

[email protected] ArchitectQrious Limited

7 February 2017

[email protected]

The Problem Examples The Solution Tools of the Trade Boxing up a Solution Flotsam and Jetsam

Outline

1 The Problem

2 Examples

3 The Solution

4 Tools of the Trade

5 Boxing up a Solution

6 Flotsam and JetsamGuy Kloss | Big Data in the Real World 2/41


Who/What is Qrious?

We help New Zealand businessesand public sector organisations

create valueand solve their most pressing business problems

by turning data into actionable insight.

Guy Kloss | Big Data in the Real World 3/41


Who/What is Qrious?

Backed by SparkApprox. 60 employeesOffices in Auckland & WellingtonSubstantial investment across Data, Platform & PeopleBuilt from the ground up(new generation technology and working principles)One of the largest Data Science teams in the countrywith > 80% qualified to Masters & PhD leveland over 60 years of combined experience years of combined experienceNZs leading data analytics specialist by 2017



Our Capabilities

Advanced analyticsLocation insightsBig Data platformsConsulting servicesBI & Warehousing



Who am I?

Chemical Engineer (Masters)Rocket Scientist (German Aerospace Centre)Computer Scientist (PhD)Former lecturer (AUT)Lead Software Developer and Head Crypto Geek @ MegaEnterprise Architect at QriousDad, baseballer, diver, . . . general geek!



Outline

1 The Problem

2 Examples

3 The Solution





Data size

Number of recordsData volume



An exponentially growing data worldPrimary Memory/Disk Capacity



An exponentially growing data worldRelative Speeds

Source: http://www.cs.cmu.edu/~amarp/cpu-io-gap


http://www.cs.cmu.edu/~amarp/cpu-io-gap


Size Does Matter!

Access/processing beyond a single machine(RAM, disk, CPU)Expensive data transfers at volume(latency, throughput)



Storage Issues

Storage, access, index, findTransfer, manage, prevent data loss



Types of Data

StructuredUnstructuredGraphsFree text. . .



Correlating . . . co-relating . . . mashing . . .

Not single record problemBut an m : n problem



Beyond Exponential

Problems are between exponential and hyperexponential→ Enabling data processing in an exponential world



Outline

1 The Problem

2 Examples

3 The Solution





Number of Records

> 1 trillion (109) records: Spark’s location based data setAnonymised for privacy (on ingest)Fully encrypted (at rest and in transport)Continuous/stream ingestionNormalisation and segmentation on data setCorrelating with external data set

→ Finding insights in this “hay mountain”



Data Volume

100s of TB to PB of “Data Lakes”Not just a backup/data graveFully encrypted (at rest and in transport)Includes data querying and processing capability

→ Capability to “store everything” (every thing and kind)



Outline

1 The Problem

2 Examples

3 The Solution





Divide and Conquer

Massively parallel processing: MPPParallelise: Map-ReducePipelines: Stream processing



Leverage Data Locality

Bring processing to the data



The Right Tools

Don’t re-invent the wheelUse existing high performing tools where possibleAvailable high productivity frameworks, making use of high level languagesThe right tool for the type of dataUse the Source, Luke!(Leverage open source based tooling with a community)



The Right Data Organisation

Row vs. columnar storage→ For analytics often better in columnar format



In, Out, Cha-Cha-Cha

Ingest data from (legacy, external) source systems→ ETL – Extract, Transform, Load

Make sure the rhythm fits (no missing “Out”)



Outline

1 The Problem

2 Examples

3 The Solution





Hadoop

Hadoop and distributionsProcessing tools for relational, streaming, batch, graph, text, search, . . .Allocates cluster resources dynamicallyData distributed (with redundancy),so compute allocated where data is



Hadoop Distributions

Many Hadoop distributions: Similar to Linux distributionsCloudera Partnership with Qrious

“Bronze” partnerAmbitions to become “Silver” partnerand MSP (managed service provider)



Basic Hadoop Tool SuiteExample: Cloudera Hadoop Distribution



MPP Databases

DB for massively parallel processing (MPP)Greenplum database and forks(based on PostgreSQL)



Generic and Specialised DBs

Generic RDBMS (where useful)NoSQLGraph DBOther columnar species



Outline

1 The Problem

2 Examples

3 The Solution





Delivering a Suitable Solution

Includes:System managementConnectivityApplication logicServicesYummy add-ons



System Management Framework

SecurityDedicated sub-networks with specific firewall rulesExternal firewallsUser and credentials managementLog collectorOther security tools . . .

System accessVPNRemote desktop services



Connectivity

API gateways(Reverse) proxiesSFTP



Application Logic

Platfor-as-a-Service (PaaS)Huge benefits of containerising application logic (using Docker)

→ Much reduced cadence for deliveryAPIs, Micro-ServicesOrchestration of Big Data analysis



Services

Solutioning, buildAnalytics and developmentOperation and maintenance



Bonus Points for . . .

Provenance(reproducibility, auditability, compliance)AI and MLBlockchain(non-repudiation, trust, “smart contracts”,identity management, federation, . . . )



Outline

1 The Problem

2 Examples

3 The Solution





In the Qrious Pipeline

Make Big Data a commodity: Don’t buy, pay what you need!→ Big-Data-as-a-Service – BDPaaS

Sliced, diced and configured to your needsStraight on bare metal,not VMs (like most cloud hosters)



Maximising the Jobmarket

What skills do you need?RDBMS?SAS?NoSQL DBs?Maybe Hadoop is a good answer?



Questions?

Parallelise!Guy [email protected]

Just a humble hair–dryer from the 30s:“One of the first machines used forpermanent wave hairstyling back in the1920’s and 1930’s.”Dark Roasted Blend:http://www.darkroastedblend.com/2007/05/

mystery-devices-issue-2.html


[email protected]

http://www.darkroastedblend.com/2007/05/mystery-devices-issue-2.html

http://www.darkroastedblend.com/2007/05/mystery-devices-issue-2.html

Date post:	21-Apr-2017
Category:	Technology
Upload:	guy-k-kloss
View:	104 times
Download:	2 times

Qrious about Insights -- Big Data in the Real World

Technology