PayPal's Private Cloud @ Scale

transcript

PayPal’s Private Cloud @ Scale

Jigar Desai• @jigardesai • December 09 2015

Table of contents Introduction

OpenStack@PayPal

Our Journey

Lessons Learned

Future

Introduction to PayPalA Technology Perspective

About PayPalPayPal is a leading technology platform company that enables digital and mobile payments on behalf of consumers and merchants worldwide. We put our customers at the center of everything we do. We strive to increase our relevance for consumers, merchants, friends and family to access and move their money anywhere in the world, anytime, on any platform and through any device.

Architecture @ PayPalRobust Infrastructure, Reusable Platforms, Payments Operating System, Delightful Experiences

Analytics Platform

HadoopTeradata FPTIBI

Technology Platform

Framework

Data Access

Tools MediaMessagingSOA CrawlerCache

Infrastructure & Operations Layer

Power Data Center Hardware Network Database OperationsToolsCloudStorage Monitoring

Experience

Payments Operating System

LoginIdentity EmailCart CouponsPayment Ship SupportRisk InvoiceWallet POS Notify Merchant

OpenStack@PayPal

x 100K Cores

x 10 PB Storage

OpenStack Cloud

> 10K Physical Servers

x 10K VMs

x 1000engineers

x 10M Lines

x 1000 Releases/year

> 1000 Services

x 1000 Builds/day

x 1000Deploy/day

Key StatisticsDeveloper Statistics

9 Availability Zones

3 Regions

Our OpenStack Journey

2012 2013

• OpenStack POC with 16 servers

• Built 2 AZs with OpenStack Essex release (96 hypervisors)

• 5 apps in production

• Migrated from Essex to Folsom

• Built 3 more Azs • 350 apps in

production• Support for dev

and QA environments

• Upgrade to Havana

• Migration of Nova to Neutron

• 1000+ apps in production

• 100% of web and mid tier apps on OpenStack

• Deployed our largest AZ with 2500 nodes using Nova cells

• Kilo upgrade in progress

The Current Stack

Datacenters (Geographic regions, Availability Zones)

OpenStack Cloud (Infrastructure as a Service)

Platform as a Service

Web/Mid Tier Applications

Common Infrastructure

Common Platforms and Services

• 100% of web and mid tier applications on OpenStack

• Proprietary PaaS on top of OpenStack to automate E2E application development life cycle

• Support for polyglot applications

• Test and production environments available on-demand

Steps to Production 25 Manual Steps Automated

Time to Market

QA Deployment

Build Time

Prod Deployment

Infrastructure

2 years back

8 Weeks

5 Hours

2.5 Hours

4 Hours

1 Month

30 Minutes

5 Minutes

< 30 Minutes

30 Minutes

Impact On Life Of A Developer

Challenges And Lessons Learned

Challenges And Lessons Learned• Operating OpenStack at scale requires building an entire ecosystem of tools and back office

functionality like monitoring, configuration management, log aggregation, hardware remediation etc.

• We faced several control plane reliability issues as our deployment grew from several hundreds of compute to several thousands specially around RabbitMQ and services interaction

• Scaling OpenStack to 10s of thousands of nodes is quite challenging specially around Neutron and RabbitMQ.

• Security is evolving but not mature enough yet • Upgrading to a newer version of OpenStack is not straightforward specially with internal

patches and takes months for larger deployments with multiple regions and availability zones

• Distributed nature of OpenStack results into challenges with debug-ability of system and identifying the root cause

• OpenStack is a journey and commitment. It’s success depends on building a talented team of engineers that takes pride in leveraging and contributing to OpenStack community

The Future

The Future Stack

Datacenters (Geographic regions, Availability Zones)

OpenStack Cloud (Infrastructure as a Service)

Web and Mid Tier Applications

Common Infrastructure

Common Platforms and Services

• New PaaS is being built using Mesos and Docker on top of OpenStack

• Goal is to achieve industry leading datacenter efficiency and utilization

• High application resiliency against infrastructure failures

• Hybrid cloud POC • PaaS support for

stateful applications

Public Clouds

Questions?Jdesai at paypal dot com