© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Mike Kuentz, Senior Solutions Architect, AWSMike Zaleski, Director Emerging and Specialty Technology, SEPTA

Tim Raybould, CEO Ticketleap

June 21, 2016

Helping SEPTA with the Pope’s Visit to Philadelphia

Agenda

• Review of events leading up to Papal visit• Not all technical problems are best solved with technical

solutions• Details of AWS Infrastructure Event Management (IEM)• How leveraging partners can help customers• The importance of load testing

Background Pope Francis plans visit to Philadelphia Friday,

September 25, 2015 Estimates put over 1 million people coming to the

city for his visit Strict security perimeter was in place in downtown

Philadelphia To manage the traffic SEPTA had to limit train tickets to only 350,000

Pilgrims

Transit map vs. event map

Ticket sales website requirements

• Sell tickets on first come, first serve basis, no lottery• No “take a number” queuing systems allowed• No third-party e-commerce (eBay, PayPal, etc.) • Complex rules regarding number of tickets allowed for

various stations• Requirements were based on the premise that tickets

will sell over the course of a week—not in under an hour

Ticket sales website 1.0

Reality by the numbers• Considerable media frenzy leading up to site opening at 9 a.m.• 54,000 visits in the first minute• 900,000 visits in the first 10 minutes• ~1,700 transactions per second• By 11 a.m., ticket sales were suspended, because the architecture

could not meet the load

The Internet responds

Review of first architecture

• Areas of concern• Redis servers running on t2.small• Running out of MySQL connections

• As number of connections increases, memory increases as well• No read replicas• Drupal writing log info into the same DB• Tightly coupled design that had to send an email through

Amazon SES before transaction completed• Amazon SES limit had not been raised

• Load testing didn’t match demand

Infr

astr

uctu

re C

ost

Time

Periodic LargeCapital Expenditures

OpportunityCost

PredictedDemandTraditionalData Center ModelActualDemand

CloudComputing

Shortage: Unable to serve

customers

Scale on demand

How to scale?

• Scaled MySQL vertically and horizontally• Scaled Redis vertically and horizontally• Removed bottlenecks in logic• Removed bottlenecks in code

Load testing

900 transactions per second

(We saw 1,700 transactions per second!)

Requirements are forced to change Move to a lottery based system

Lottery based system

Researched third parties (Ticketmaster, Eventbrite, etc.)— they all refused given the original requirements

Engage AWS IEM and Ticketleap Open access for 24 hours De-duplicate entries and fraud checks Lottery prevents/limits scalpers

Get rid of all the servers!

• AWS customer

• Lightweight online ticketing platform used by thousands of event organizers across the US and Canada

• Case study available• https://aws.amazon.com/solutions/case-studies/ticketleap/

Review of second architecture

Amazon SQS Amazon SESAmazon S3

AWS Lambda

Amazon CloudFront

S3 statistics

Amazon S3

Request rate and performance considerations

• If your workload in an Amazon S3 bucket routinely exceeds 100 PUT/LIST/DELETE requests per second

• Talk to us!• Avoid sequential patterns in key names

• examplebucket/2013-26-05-15-00-00/cust1234234/photo1.jpg• examplebucket/2013-26-05-15-00-00/cust3857422/photo2.jpg• ...• examplebucket/2013-26-05-15-00-01/cust1248473/photo4.jpg• examplebucket/2013-26-05-15-00-01/cust1248473/photo5.jpg

Review of second architecture

Amazon SQS Amazon SESAmazon S3

AWS Lambda

Amazon CloudFront

Results

• Load testing• c4.8xlarge instance and The Grinder• POST 200,000 entries in 85 seconds (~2300 TPS)

AWS Infrastructure Event Management

• Initiation• Planning and execution• Review and closure

• https://aws.amazon.com/premiumsupport/iem/

IEM success stories

Thank you!