[Kubecon 2017 Austin, TX] How We Built a Framework at Twitter to Solve Service Ownership & Improve...

How built a framework to improve infrastructure resource utilization at scale

★ Sr. Systems Engineer @Twitter★ Proud being a member of @TwitterWomen,

@Techwomen and @WomenWhoCode

I am @VinuCharanya

Hello!

1 2 3 4

History & ContextChargeback @TwitterKite - Service Lifecycle ManagerImpact & Future Work

Agenda

History & Context

Thousands of MicroServices



INFRASTRUCTURE & DATACENTER MANAGEMENT

CORE APPLICATION SERVICES

TWEETS

USERS

SOCIAL GRAPH

PLATFORM SERVICES

SEARCH

MESSAGING & QUEUES

CACHE

MONITORING AND ALERTING

INGRESS & PROXY

FRAMEWORK/

LIBRARIES

FINAGLE(RPC)

SCALDING(Map Reduce in

Scala)

HERON(Streaming Compute)

JVM

MANAGEMENT

TOOLS

SELF SERVE

SERVICE DIRECTORY

CHARGEBACK

CONFIG MGMT

DATA & ANALYTICSPLATFORM

INTERACTIVE QUERY

DATA DISCOVERY

WORKFLOWMANAGEMENT

INFRASTRUCTURESERVICES

MANHATTAN

BLOBSTORE

GRAPHSTORE

TIMESERIESDB

STORAGE

MESOS/AURORA

HADOOP

COMPUTE

MYSQL

VERTICA

POSTGRES

DB/DW

DEPLOY(Workflows)

MESOS/AURORA

HADOOP

MANHATTAN

67%N

umbe

r of S

erve

rs

Number of Servers

MESOS/AURORA

HADOOP

MANHATTAN

67%How to get visibility into resources used by

individual jobs & datasets?

Number of Servers

MESOS/AURORA

HADOOP

MANHATTAN

67%How to attribute resource consumption

to teams/organization?

Number of Servers

MESOS/AURORA

HADOOP

MANHATTAN

67%How do you incentivize the right behavior to

improve efficiency of resource usage?

Chargeback @Twitter

Chargeback @Twitter

Ability to meter allocation & utilization of resources

Chargeback @Twitter

Ability to meter allocation & utilization of resources per service, per project, per engineering team

Chargeback @Twitter

Ability to meter allocation & utilization of resources per service, per project, per engineering team to improve visibility & enable accountability

Features

Supports diverse Infra Services

Chargeback @Twitter

18

Meters abstract resources at daily

granularityDetailed Reports

19

Chargeback @Twitter

1. Resource Catalog: Consistent way to inventory infrastructure resources

Support diverse Infrastructure and Platform Services

20

Chargeback @Twitter


• Resource Fluidity: Support primitive (CPU) and abstract resource (“Tweets / second”). Extend existing resource


21

Chargeback @Twitter


• Resource Fluidity: Support primitive (CPU) and abstract resource (“Tweets / second”). Extend existing resource

2. Resource <> Client Identifier Ownership: Map of client identifier to an owner to enable accountability


OFFER MEASURE COST

RESOURCE CATALOG ENTITY MODEL

OFFER MEASURES

OFFER MEASURE COST

1:N


PROVIDER

INFRASTRUCTURE SERVICE

OFFERINGS

OFFER MEASURES

OFFER MEASURE COST

1:N

1:N

1:N

1:N


TWITTER DC/PUBLIC CLOUD

COMPUTE

CORE-DAYS

$X

PROVIDER


OFFERINGS

OFFER MEASURES

OFFER MEASURE COST

1:N

1:N

1:N

1:N


TWITTER DC/PUBLIC CLOUD

COMPUTE

CORE-DAYS

$X

PROVIDER


OFFERINGS

OFFER MEASURES

OFFER MEASURE COST

1:N

1:N

1:N

1:N

TWITTER DC

STORAGE

GB- RAM

PROCESSING CLUSTER

FILEACCESSES

…

…GB- RAM

FILE ACCESSE

S… …

$X $Y …$M $N… …


{ measures: [{"measure_id": 1,"measure_label": "core-days","measure_unit_label": "per 1 core-day","offering_id": 1,"offering_label": "Compute","infrastructure_id": 1,"infrastructure_name": "Aurora"

},

{"measure_id": 2,"measure_label": "machine-days","measure_unit_label": "per 1 machine-day","offering_id": 2,"offering_label": “zone:tweety","infrastructure_id": 8,"infrastructure_name": "Physical Infrastructure",

},

{

/api/1/measures

Chargeback @Twitter

So, how do you incentivize the right behavior to improve efficiency of resource usage?

Pricing is one way…

Operational Overhead

Headroom

Production Used Cores

Non-Prod Used Cores

Cost of Physical Server($X / day) Total available Cores

Quota Buffer(Underutilized Quota)

Container Size Buffer(Underutilized Reservation)

Total Cost of Ownership for Aurora$X core-day


Headroom


Non-Prod Used Cores




Total used Cores



Headroom


Non-Prod Used Cores




Total used Cores

Excess Cores (incl. DR, Spikes, Overallocation)Total Cost of Ownership for Aurora

$X core-day


Headroom


Non-Prod Used Cores




Total used Cores

Excess Cores (incl. DR, Spikes, Overallocation)

Cores used by platformfor operations &

maintenance


Features

Supports diverse Infra/Platform

Services

Chargeback @Twitter

34



35

Chargeback @Twitter

INFRASTRUCTURE SERVICE 1


INGESTMETRICS

RAWFACT TRANSFORMER RESOLVED

FACT

RESOURCE CATALOG

REPORT

REPORT

Metering Pipeline (ETL Job)

IDENTIFIER OWNERSHIP

MAPPING

Metrics Ingestor

DATA FIDELITY


36

Chargeback @Twitter



INGESTMETRICS


FACT

RESOURCE CATALOG

REPORT

REPORT



MAPPING

Schema(client_identifier, offering_measure, volume, metadata, timestamp)

DATA FIDELITY


37

Chargeback @Twitter




INGESTMETRICS


FACT

RESOURCE CATALOG


MAPPING

REPORT

REPORT

Transformer

DATA FIDELITY


38

Chargeback @Twitter




INGESTMETRICS


FACT

RESOURCE CATALOG


MAPPING

REPORT

REPORT

1. Resolve Ownership

DATA FIDELITY


39

Chargeback @Twitter




INGESTMETRICS


FACT

RESOURCE CATALOG


MAPPING

REPORT

REPORT

2. Cost Computation

DATA FIDELITY


40

Chargeback @Twitter




INGESTMETRICS


FACT

RESOURCE CATALOG DATA FIDELITY

REPORT

REPORT


MAPPING

Data Fidelity & Reporting


41

Chargeback @Twitter




INGESTMETRICS


FACT

RESOURCE CATALOG

REPORT

REPORT


MAPPING

1. Verify Data Integrity & Fidelity

DATA FIDELITY


42

Chargeback @Twitter




INGESTMETRICS


FACT

RESOURCE CATALOG

REPORT

REPORT


MAPPING

2. Alert when things don’t seem the way it should be

DATA FIDELITY


43

Chargeback @Twitter



EXPORTMETRICS


FACT

RESOURCE CATALOG


DATA FIDELITY

REPORT

REPORT


Features


Services

Chargeback @Twitter

44



45

Chargeback @Twitter

Customers

Infrastructure & Platform Operators Overall Cluster GrowthAllocation v/s Utilization of resources by Client/Tenant

Finance & Execs Budget v/s Spend per OrgInfrastructure PnLOverall Efficiency & Trends

Service Owners & Developers Team BillPer Service Allocation vs. Utilization of Resources

Reports

Customers



INFRASTRUCTURE PNL

47

Chargeback @Twitter

Customers



Service Owners & Developers Team BillPer Service Allocation vs. Utilization of Resources

Reports

CHARGEBACK BILL FOR A TEAM

CHARGEBACK DRILLDOWN FOR A TEAM

Features


Services

Chargeback @Twitter

50



51

1 2 3 4

Learnings

Chargeback @Twitter

Invest in data Fidelity

Accurate Ownership Mapping

Logical grouping of resources

Track historical data

• Trust in data is most important.

• Invest in monitoring & alerting for data inconsistencies

• Leverage this for detecting abnormal increase/decrease and notify users

• Static mappings go out of date quickly

• Invest in systems (ex, Kite) for users to manage it themselves

• Identifiers were too granular and teams were too broad.

• Find a good middle ground and invest in system (ex, Kite) to track, understand and maintain

• Unit prices change over time

• Orgs / Teams change over time

• Resources get added / removed

• Change history is essential for consistency which is used for CAP planning

52

1 2 3 4

Learnings

Chargeback @Twitter
















53

1 2 3 4

Learnings

Chargeback @Twitter
















54

1 2 3 4

Learnings

Chargeback @Twitter
















55

1 2 3 4

Learnings

Chargeback @Twitter
















SERVICE IDENTITY MANAGER

RESOURCE PROVISIONING MANAGER

DASHBOARD (SINGLE PANE OF GLASS)

REPORTING

INFRASTRUCTURE SERVICEINFRASTRUCTURE SERVICEINFRASTRUCTURE SERVICEINFRASTRUCTURE & PLATFORM SERVICE

SERVICE LIFECYCLE WORKFLOWS

METADATA RESOURCE QUOTA MANAGEMENT

METERING & CHARGEBACKCLIENT IDENTITY

PROVIDER APIS & ADAPTERS

10,000+ Client Identifiers 1,000+ Projects 100+ Teams 8 Infrastructure Services

58

Kite @Twitter

59

Kite @Twitter

Identity System: Built a consistent way to group client identifiers of different infrastructure services into a project and enabled ownership

• Capture Org Structure: Support org structure changes, project transfer workflows to ensure up-to-date ownership of identifiers

• Unify client identifier provisioning workflow: Enables single source of truth and reduces operator pain around provisioning and managing client identifiers.

Client Identifier Management

IDENTITY ENTITY MODEL

<INFRA, CLIENTID> <Aurora, tweetypie.prod.tweetypie>

<Aurora, ads-prediction.prod.campaign-x>


SERVICE/SYSTEM ACCOUNT

<INFRA, CLIENTID>

1:N

tweetypie

<Aurora, tweetypie.prod.tweetypie>

ads-prediction


BUSINESS OWNER

TEAM

PROJECT


<INFRA, CLIENTID>

1:N

1:N

1:N

1:N

INFRASTRUCTURE

TWEETYPIE

tweetypie

tweetypie


ADS PREDICTION

prediction

ads-prediction


REVENUE


BUSINESS OWNER

TEAM

PROJECT


<INFRA, CLIENTID>

1:N

1:N

1:N

1:N

INFRASTRUCTURE

TWEETYPIE

tweetypie

tweetypie


ADS PREDICTION

prediction

ads-prediction


REVENUE


Entities are time varying dimensions

Impact

10,000+ Client Identifiers

CLAIM OWNERSHIP

PROJECT DISCOVERY

TEAM OVERVIEW

TEAM OVERVIEW

Released unused Resources

TEAM OVERVIEW

Q2 unit price update

TEAM OVERVIEW

New project launch

PROJECT METADATA

AURORA QUOTA MANAGER

Future Work

75

Future Work

Impact & Future Work

1 2Resource provisioning

Enable project deprecation

• Extend Quota Manager and unify the experience into Kite

• Onboard Hadoop, Storage and other systems

• Detect unused resources, notify users, trigger deprecation process based on policy

3Capacity Planning

• Provide historic trends and help with forecast of capacity

76

1 2

Future Work


Resource provisioning





3Capacity Planning


77

1 2

Future Work







3Capacity Planning


79

1 2

Future Work







3Capacity Planning


@VinuCharanya

Date post:	23-Jan-2018
Category:	Technology
Upload:	vinu-charanya
View:	82 times
Download:	0 times

[Kubecon 2017 Austin, TX] How We Built a Framework at Twitter to Solve Service Ownership & Improve...

Technology