Building a Data Lake - An App Dev's Perspective

Building a Data LakeAn App Dev’s Perspective

GeekNight Hyderabad - March 8th 2017

Geetha Balasundaram

[email protected]

© 2017 ThoughtWorks Technologies Pvt. Limited

ABOUT ME

Developer @ ThoughtWorks

Building a data lake in the enterprise ecosystem

Helping a retail business make sense of it ( data guided org )

Been part of web development space ( enterprise rewrite )

Equally startled like everyone else by the data engineering space

Share know-how’s and do-how’s from our team’s experience

[email protected]


AGENDA

What is data in the true sense…

Data Warehouse in an enterprise ecosystem...

What is a data lake...

Data lake implementation in an enterprise ecosystem…

How to make effective use of a data lake: technology+process+people

Cluster Administration tool - Cloudera Manager

Pitfalls to avoid


Question ???

How did R.Ashwin perform in the last Test match?

HIGH LEVEL

PROBLEM STATEMENT


COMPLEX HISTORICAL DATA

Why?

Exploit and derive as much new insights as possible

Match Made

Enterprise systems produce this nature of complexity


DATA WAREHOUSE

https://martinfowler.com/articles/microservices.html

ETL


DID MICROSERVICES CAUSE THIS PROBLEM ?

Decentralised Data

https://martinfowler.com/articles/microservices.html© 2017 ThoughtWorks Technologies Pvt. Limited

MICROSERVICES HELPED

Break down business unit

Break down complexity

Understand the nature of data


Question ???

R.Ashwin performed well ( 6/41 ) in yesterday’s match!

Complex historical data can quantify how well he has performed

Can we say why did he do well in this particular match? What factors affected his enhanced performance?


FACT is a FACT

… even when we don’t know how it can be used


KEY DIFFERENCE

https://martinfowler.com/bliki/DataLake.html© 2017 ThoughtWorks Technologies Pvt. Limited

What is a data lake?


LAKE is...

.. a large body of water in a more natural state.

The contents of the lake, stream in from a source to fill the lake,

and various users of the lake can come to examine, dive in, or

take samples

https://jamesdixon.wordpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/


DATA LAKE is...

.. a large body of water data facts in a more natural state.

The contents of the lake, stream in from a source to fill the lake,

and various users of the lake can come to examine analyse, dive

in build models, or take samples use subset for specific use

cases


KEY DIFFERENCE

https://martinfowler.com/bliki/DataLake.html© 2017 ThoughtWorks Technologies Pvt. Limited

Implementation


OUR IMPLEMENTATION - TECH STACK

DATA SOURCE

DATA INGESTION

DATA LAKE

DATA MARTS DATA ANALYSIS

Staging / Queue



How to make effective use of a data lake:

technology+process+people


Functionality Vs Reality

I need a feature so that I can do this action…..

to

I need this insight so that I can take this action….

eg : I need a functionality to order items anytime before or during a promotion…

to

..I need to know on time, if I have to order items anytime before or during a promotion…

so that I can improve promotion sales

People


Start Simple

There is no data lake yet…

Carve out portions of data which are easy wins yet critical to

arrive at the earlier stated insight..

Set up the infrastructure and pipeline

Get your hands dirty..

eg: Sales is an important factor to analyse / predict anything in retail space..

Technology


How much should I know about the data ?

As a consumer of data (read ‘not a consumer of service’)

How much should I know about it?

Schema ⇔ Contracts

Nature of the data versioned vs latest

transactional vs reference

facts vs aggregate

frequency of change

…..

Technology


DATA INSIGHT - Part 1

Incrementally add

new data to the

lake

Serve data

for analysis

eg: What data wrt promotions do I need to bring into the datalake ??

Sales → improve promotion sales

Technology


DATA INSIGHT - Part 2Sales + Promotions → improve promotion sales

How does adding more data to the lake help arriving at new insights..?

history of past promotions sales = how much to order for this promotion

history of past promotion sales + ‘X’ = how much to order for this promotion

history of past promotion sales + ‘X’ + ‘Y’ …… = how much to order for this promotion

eg: seasonality has a strong correlation with sales

history of past promotion sales + ‘X’ + ‘Y’ …… + ‘A’ = how much to order for this promotion after the start

People


Think Agile

Sales + Promotions + X factor → improve promotion sales

Near perfect list of parameters

Progressive set of parameters

Sales + Promotions → is the quantity arrived from these factors (known to business) ordered on time?

Process


DataMarts

... as a store of bottled water – cleansed and packaged and

structured for easy consumption


DataMarts

... as a store of data subset - curated from meaningful facts

bundled into logical groups for arriving at useful insights


Easy Insight

Sales + Promotions →

is the quantity arrived from these factors (known to business) ordered on time?

System : Tells me what is the quantity that is supposed to be ordered for this promotion..

System : Tells me in realtime what is the quantity that is ordered

Technology


Cluster Administration Tool

Cloudera Manager


Think DevOps

Scale | Performance | Memory | Resource Contention |

Optimization | Stability |

Need for an ecosystem - to monitor how well the different tools

play together without chaos

Tools


QUICK RECAP

What is data in the true sense…

Data Warehouse in an enterprise ecosystem...

What is a data lake...

Data lake implementation in an enterprise ecosystem...

How to make effective use of a data lake…

Cluster Administration tool - Cloudera Manager


PITFALLS TO AVOID

Data envy - Ref:https://martinfowler.com/bliki/Datensparsamkeit.html

Tool envy

Reliable data is a luxury

Understanding the nature of data is a must

Dialogue with the data scientist

Treating the data lake like a RDBMS

Keeping the business involved

Data flow state visibility


Date post:	12-Apr-2017
Category:	Technology
Upload:	geeknighthyderabad
View:	56 times
Download:	1 times