How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov

transcript

How To Avoid Epic Web Failure

Lessons Learned from Healthcare.gov

Webinar

What have we learned from recent epic fails?In This Webinar

TODAY’S SOASTA PRESENTERS

Peter Galvin: Brad Johnson:SVP Marketing VP Product Marketing

Agenda: • Poll question• Lessons Learned• Some Best Practices

Questions: Submit in chat box during event

Poll Question

Acceptable Failure?

This October…

“We didn’t have enough testing for a very complicated project”

US Health and Human Services Secretary, Kathleen Sebelius

Impact of Healthcare Site Failures

• Millions of uninsured citizens• Healthcare program mistrust• President’s approval rating dropped• Congress tied up in legislation• Public cynicism• Press and social media circus• Uncertainty for the future

Brought QUALITY and TESTING to the public forefront!

Industry Best Practices

RECOMMENDATIONS

“Name a single implementation leader”

“Lock down” requirements by April 30

“Streamline decision making”

“End-to-end integrated operations and IT testing”

Top down, Home Health Agencies Missed the Mark

http://www.businessweek.com/news/2013-11-19/obamacare-consultants-warned-of-health-website-failure-in-march

WHAT HAPPENED

“No single leader”

“Shifting requirements”

“Multiple definitions of success

“Insufficient time and scope of end-to-end testing”

http://www.npr.org/blogs/alltechconsidered/2013/11/19/246132770/this-slide-shows-why-healthcare-gov-wouldnt-work-at-launch

In March 2013

Outages Are Serious Business

Source: Venture Beat & Ray Wang at Constellation Research 2012

“Twitter’s down…and it has cost the economy $1 billion already?”

Source: Venture Beat & Ray Wang at Constellation Research 2012

OK...reduce Twitter usage assumptions 50%.

Then, only $400,641,025.00 was lost.

“Twitter’s down…and it has cost the economy $1 billion already?”

Last WeekTWO DAYS! (is “too long”)

Extra October Stress for College Applicants 1 million students, 517 colleges, high anxiety

“Mass Panic As Common App Crashes”

But Wait, There’s Moreo Aug. 14: New York Times

• 2 hours - Stock dropped - opened paywall (lost $$) – Twitter storm

o Aug 17: Google

• Less than 5 minutes – lost $500,000 – Internet traffic dropped 40%

o Sept 2: HSBC (UK)

• Internal and external systems – “Weekend of disruption”

o Nov 12: Kayazoonga.com (India)

• Last cricket match of Sachin Tendulkar - 19.7M hits/hour – crashed all day – “Chaos”

o Nov 18: YouTube

• 35 minutes – est. 100,000 users affected

PR Nightmares…

Every Site is SusceptibleEven BatKid’s

SOASTA Helps

London 2012 Results

500 Tests

400,000 concurrent users

431 Million Visitors

4.73 Billion Page Views

Access from 201 Countries

“We began working with SOASTA CloudTest six months prior to the

games.” Paul Bunnell,

Lead Architect, London 2012

SOASTA Helps

Prep for…

Verbalizeit Results “ We were on very

tight timelines…

the SOASTA team

was able to run

multiple tests very

quickly over a two-

day period and get

us going on issues

to immediately

fix.”

Kunal Sarda,

VerbalizeIt

COO and Co-

Founder

Plan based on 6-8 million TV viewers

Tested & tuned for 10 million hits

250 to 30,000 visits/day

7x customer growth

Zero downtime

SOASTA Helps

Dillard’s Results

Tested to 7x Initial load test goals

Reached 9x previous year’s peak

Validated dynamic caching investment

Reduced hardware capacity needs

Validated Akamai to 4x beyond peak

“One of the best investments to our

website this year .” Nancy Jordan,

IT Middleware & Integration

Manager

SOASTA Helps

Hallmark Results“Website performance really

affects your brand image

when you have problems…

The inherent savings to our

brand by using CloudTest is

priceless.”

—Cheryl Davis,

digital technical

operations manager

Hallmark Cards, Inc.

Tested to 120% of expected load

Saved 50% on testing expenses

Reduced staff costs by 66%

Eliminated test software and hw costs

“No amount of testing within reasonable time limits can adequately replicate a live environment of this nature”

About that Healthcare.gov Testing…

“No amount of testing within reasonable time limits can adequately replicate a live environment of this nature”

About that Healthcare.gov Testing…

OBJECTION!

Speed AND QualityThere is no need to compromise

Testing Issues

o 55 Contractors

o Not enough time in schedule to conduct adequate performance testing

o No Integration testing between systems

o No end-to-end testing simulating real life experiences of end users

o Followed waterfall development with parallel/disconnected projects

o Locked at 2000 users at the 1st step

From the HHA press

http://www.washingtonpost.com/politics/house-panel-grills-contractors-on-troubled-health-insurance-web-site/2013/10/24/8f42c748-3ca7-11e3-b7ba-503fb5822c3e_story.html

http://www.cnn.com/2013/10/29/politics/obamacare-warning/

SOASTA Best Practices

o Test Continuously

From ten of thousands of tests

Continuous PerformanceThe path to a performance culture

I JUST MADE A CODE CHANGE

CI system runs load tests against a pre-prod site

…while unit, functional and mobile device tests run in parallel

Daily (or more frequent) updates keep teams “performance focused”

Identify bottlenecks & inform teams, just like unit and functional tests in CI

o Test End To End

From ten of thousands of tests

End-To-End Cloud Testing

Customers

SUT/AUT

Web Server

App Server

Web Server

App Server

Load Balancer

Database

CDNCredit Card

Web analyti

Other 3rd pty svcs

“Conductor”

Analytics

CloudTest

Database

Perf Tester 1

SUT/AUT

Web Server

App Server

Web Server

App Server

Load Balancer

Database

CDNCredit Card

Web analyti

Other 3rd pty svcs

Analytics

CloudTest

Database

Perf Tester 1

AnalyticsLoad

GeneratorLoad GeneratorLoad

Generator

East Coast (AWS)

SUT/AUT

Web Server

App Server

Web Server

App Server

Load Balancer

Database

CDNCredit Card

Web analyti

Other 3rd pty svcs

Analytics

CloudTest

Database

Perf Tester 1

AnalyticsLoad

Generator

East Coast (AWS)

SUT/AUT

Web Server

App Server

Web Server

App Server

Load Balancer

Database

CDNCredit Card

Web analyti

Other 3rd pty svcs

Analytics

CloudTest

Database

Perf Tester 1

SUT/AUT

Web Server

App Server

Web Server

App Server

Load Balancer

Database

Data sources: Native (SSH), JMX, PerfMon, CA Introscope,

AppDynamics, New Relic, CloudWatch,….

“Conductor”

AnalyticsLoad

Generator

East Coast (AWS)

CDNCredit Card

Web analyti

Other 3rd pty svcs

Analytics

CloudTest

Database

Perf Tester 1

AnalyticsLoad

Generator

East Coast (AWS)

SUT/AUT

Web Server

App Server

Web Server

App Server

Load Balancer

Database

AnalyticsLoad

Generator

San Francisco (GoGrid)

AnalyticsLoad

Generator

Chicago (Rackspace)

AnalyticsLoad

Generator

Tokyo (AWS)

AnalyticsLoad

GeneratorLoad Generator

Amsterdam (Azure)

AnalyticsLoad

Virginia (IBM)

SOASTA Global Test

CDNCredit Card

Web analyti

Other 3rd pty svcs

Analytics

CloudTest

Database

Perf Tester 1

AnalyticsLoad

Generator

East Coast (AWS)

AnalyticsLoad

Generator

AnalyticsLoad

Generator

AnalyticsLoad

Generator

Tokyo (AWS)

AnalyticsLoad

Amsterdam (Azure)

AnalyticsLoad

Virginia (IBM)SUT/AUT

Web Server

App Server

Web Server

App Server

Load Balancer

Database

SOASTA Global Test

“Conductor”

CDNCredit Card

Web analyti

Other 3rd pty svcs

Chicago (Rackspace)

Analytics

CloudTest

Database

Perf Tester 1

AnalyticsLoad

Generator

AnalyticsLoad

Generator

Tokyo (AWS)

AnalyticsLoad

Amsterdam (Azure)

SUT/AUT

Web Server

App Server

Web Server

App Server

Load Balancer

Database

AnalyticsLoad

Virginia (IBM)

AnalyticsLoad

Generator

SOASTA Global Test

Access for all

AnalyticsLoad

Generator

East Coast (AWS)

“Conductor”

CDNCredit Card

Web analyti

Other 3rd pty svcs

Chicago (Rackspace)

o Test to Any Scale

From tens of thousands of tests

Memory leaks

Scale of Test

Dev & Test

Release& Deploy

Network& Ops

CDN file placement

Load Balancer configuration

Network bandwidth

Network configuration

DNS routing

Inadequate server resources

Default configuration settings

Unbalanced web serversAuto-scaling failures

Latency between systems

Slow third-party plug-ins

Garbage collection

Database thread counts

Inefficient database queriesSlow pages

Conflict with other apps

Test Lab

Search technology limits

Method-level tuning

Max sockets exceeded

Firewall max capacity

Global latency variance

Security bottlenecks

Scale is relative – bugs are everywhere

Tuning for full-scale…and well beyond

Frequent Testing from Lab to Production

Staging Production (100% +++)

Scale of Test

Dev & Test

Release& Deploy

Network& Ops

Test Lab Staging Production (100% +++)

Where do we typically find the issues?

How to Avoid Epic Web Failure... Lessons Learned from Healthcare.gov

Technology