Managing a Multi-Tenant Data Lake

transcript

Managing A Multi-Tenant Data Lake

Agenda Timeline for Evolution Why Governance Multi-Tenancy Anti-Patterns / Warning Signs Instituting Governance Managing through Chaos Monitoring/Metrics Environment Tools SLA Management Support and Staffing Demo - Command Center

Timeline – 2013

2013 – “The Experiment” Started with 10 node cluster

Experimentation with batch processing and enrichment of event data

Team assembled from across organization

Primarily solving single use case

30 nodes by end of trial

2 Racks

Timeline – 2014 (H1)

2014 Production “Honeymoon”

Added 70 more nodes along with lower environments (Dev & QA)

Onboard additional ~20 data sets through batch ETL

Supporting a dozen use cases

5 Racks

Timeline – 2014 (H2)

2014 Production “Tiger’s Tail”

Total of 200 nodes to support additional use cases (data science)

Total of ~30 more data sets through batch ETL

Supporting several dozen use cases and ad-hoc exploration

Starting to have difficulty managing resource requests

9 Racks

Timeline – 2015

2015 Production “Cortez”

Adding 250 more nodes to production environment

Fully embraced governance

Supporting 24x7 production use cases

19 Racks

Timeline – 2016

2016 Production “Planetary”

Adding 1300 more nodes to production environment

Standing up separate 500 node data science cluster

Spinning off critical compute to boundary satellite clusters

Reaping benefits from governance and resource planning

48 Racks

Why Governance?

It’s about establishing acceptable behaviors for the benefit of the community

Minimize user/application impact on cluster

Users will do whatever is technically possible Everyone has been conditioned to work “smarter not harder”

Establishing the guardrails not edicts.

Multi-Tenancy Anti-Patterns

Speculative Execution

Optional User Training

Lack of Resource Isolation

Lack of Testing and Measurement

Ad-hoc Communication Channels

Excessive Resource Utilization/Reservation

Informal Service Level Agreements (SLAs)

Public Domain: Plynn9

Signs of Looming Disaster

Pending Jobs

Queue Fidgeting

Job Rescheduling

Non Predictive Workloads

Cluster Storage Out Of Balance

Public Domain: US DOE

Instituting Governance

Governance is not a technology problem

Governance must be solved using People - Who Processes – What / When / How Policy – Why

Always employ technology to help with enforcement and measurement

Setting Out Governance Standards – Starting Out

Involve the business users to define light-weight policies and processes Onboarding users/applications/tools Resource Utilization Worksheets Deployment checklists Service Level Agreements / Penalties Updates of Governance Standards

You MUST socialize and educate your community on these policies and process

Strive for evolution not revolution

Setting Out Governance Standards – Measurement

Define universally accepted performance measures Storage Compute System Availability Issues and MTTR Average Completion Time Average Pending Apps

Be transparent with results and make them available to entire community

Establish monthly performance reviews with key stakeholders

Setting Out Governance Standards – Enforcement

Lock down as many resources as possible

Monitor resource utilization for compliance

Automate corrective measures

Its all about transitioning from defense to offense and eliminating surprises!

Setting Out Governance Standards – Enforcement

Hadoop provides some base capabilities YARN Queues for compute HDFS Quotas/ACLs for storage

Implement custom solutions for proactive offensive capabilities Job monitoring and migration (Penalty Box) Dynamic Allocation / Queue Flexing Monitor and track leading indicators (Command Center)

Multi-Tenancy: Understanding the Chaos - Monitoring/Metrics

Image Attribution: Pixabay - Creative Commons CC0

Use Case – Extreme Ad Hoc (Data Science)

Challenges? You bet!

Challenges Monitoring and Managing a Multi-tenant Hadoop Environment – Diverse User Community

Images: Creative Commons

Challenges Monitoring and Managing a Multi-tenant Hadoop Environment - SLAs

Challenges Monitoring and Managing a Multi-tenant Hadoop Environment - Governance

Challenges Monitoring and Managing a Multi-tenant Hadoop Environment – Monitoring & Forecasting

Environment

Our Environment - Tools for Monitoring

Standard Hadoop Monitoring

Environment - Tools for Monitoring

Command Center

Pepperdata

SLA Management

Application Timing

SLA Management

Application Timing

Resource Management

SLA Management

Application Timing

Resource Management

Capacity Management

Support & Staffing

Takeaways for DevOps Model in Hadoop

Train Your Teams (!!!)

Measure, Forecast and Model

Automation and Frameworks

Comcast Command Center

The Command Center: Our Focus

Visualizations & Design

Ease Of Use

Extensibility

Ease Of Use

Extensibility

Alerting

The Command Center for Monitoring and Alerting

• Missed SLAs• Guardrails broken

• Definitions• Links

• Containers• Queue capacity

• Status• Measures

• HDFS Usage• Queue Usage

Continuous Evolution

Continuous Engagement

Monitoring and Alerting at Comcast

The Command Center!

Thanks!

Ray HarrisonPrinciple DevOps Architect

Mike FaganPrinciple Big Data Architect

Ray_Harrison@cable.comcast.com Michael_Fagan@cable.comcast.com

We Are Hiring!

Managing a Multi-Tenant Data Lake

Technology