Continuous Testing

September 9, 2015

“Continuous Testing in DevOps”Dan Boutin – Senior Product Evangelist

[email protected] (404) 304-9529@DanBoutinSOASTA

mailto:[email protected]

© 2014 SOASTA. All rights reserved. 2

In This Discussion Today

Agenda: • SOASTA Introduction• The need for “Continuous”• The “Big 3”• CI with mobile and performance testing


SOASTA – Performance Analytics

100 BillionUser Experiences Tested

10 MillionTests Performed

Actual CloudTest view



Agenda: • SOASTA Introduction• The need for “Continuous”• CI with mobile and performance testing


CI Offers Speed of Delivery & TestingWhat’s not to like?


CI Offers Speed of Delivery

o Small Batch Sizes

What’s not to like?



o Small Batch Sizeso Comprehensive Version Control




o Small Batch Sizeso Comprehensive Version Controlo Simple Branch Strategies




o Small Batch Sizeso Comprehensive Version Controlo Simple Branch Strategieso Automation




o Small Batch Sizeso Comprehensive Version Controlo Simple Branch Strategieso Automation o Continuous Feedback




o Small Batch Sizeso Comprehensive Version Controlo Simple Branch Strategieso Automation o Continuous Feedbacko Emphasis on working builds




o Small Batch Sizeso Comprehensive Version Controlo Simple Branch Strategieso Automation o Continuous Feedbacko Emphasis on working buildso Consistent environments




o Small Batch Sizeso Comprehensive Version Controlo Simple Branch Strategieso Automation o Continuous Feedbacko Emphasis on working buildso Consistent environmentso Developer – Tester Collaboration




Agenda: • SOASTA Introduction• The need for “Continuous”• CI with mobile and performance testing


So, what?

• We can only test as fast as the conveyor belt allows.

• If the conveyor belt is moving slow, testing is delayed.

• The Big 3 drives testing cycle.

o Why is CI critical? https://youtu.be/8NPzLBSBzPI • Speed

• New features, functions, etc. need to get to the customer, ASAP!

“We didn’t have enough testing for a very complicated project”

US Health and Human Services Secretary, Kathleen Sebelius

https://youtu.be/8NPzLBSBzPI

https://youtu.be/8NPzLBSBzPI


s

Memory leaks

Scale of Test

Stag

e / T

eam

Dev & Test

Release& Deploy

Network& Ops

CDN file placement

Load Balancer configuration

Network bandwidth

Network configuration

DNS routing

Inadequate server resourcesDefault configuration settings

Unbalanced web serversAuto-scaling failures

Latency between systems

Slow third-party plug-ins

Garbage collectionDatabase thread counts

Inefficient database queriesSlow pages

Conflict with other apps

Test Lab Staging Production (100% +++)

Search technology limits

Method-level tuning

Max sockets exceeded

Firewall max capacity

Global latency variance

Security bottlenecks

Tuning for full-scale…and well beyond

Issues Found As Scale IncreasesCloud Testing reveals risks at any scale


Laying the FoundationA factory streamlines for quality and speed

Does it work as designed?

Does it perform under

extreme circumstance

s?

Approve materials

Real Drivers

What happens in actual conditions?

Assembly Line


Laying the FoundationCI streamlines for continuous quality

Functional validation

Performance,reliability,scalability tests

Real Users

Build

What happens in actual conditions?

Unit-tested code


Continuous QualitySOASTA’s Lifecycle


Continuous QualitySOASTA’s LifeCycle

I JUST MADE A CODE CHANGE

Jenkins runs any unit tests as usual



Run iterative load tests via Jenkins

Spin up suitable, consistent test environments in the

cloud or the lab






Mobile? Run functional automation on real devices for functional & performance validation


cloud or the lab






Daily (or more frequent) updates keep teams quality focused

Embedded failure details & bottleneck data isolate issues



cloud or the lab








Manage to a performance

baseline



cloud or the lab





Use real user data toiterate dev & test plans




Manage to a performance

baseline



cloud or the lab




Continuous Testing Demo

o Load and Performance Testing with Cloudtesto Mobile Functional Test Automation with TouchTest


Continuous Quality for Continuous Delivery

o SOASTA’s Process Metrics• 3 Main Products

• (2) Builds per day of each product (Dev branch + Customer branch)• 8,000 tests executed per build• For Mobile (TouchTest), Over 300 fully-automated tests each for iOS &

Android

SOASTA - Takeaways


The Path to Continuous Testing

o Understand your requirements

o Identify the cloud advantages for you

o Automate the obvious and most critical

o Connect to a continuous process

o Align teams with actionable information

SOASTA - Takeaways








SOASTA - Takeaways








SOASTA - Takeaways








SOASTA - Takeaways








SOASTA - Takeaways


CONFIDENTIAL – Not for Distribution

Final TakeawayWhy is CI important?

RevenueBrandCompetitive advantage


We will close with some Mobile Trivia

@DanBoutinSOASTA


Questions?

Dan Boutin – Senior Product [email protected]

Mobile (404) 304-9529@DanBoutinSOASTA

Visit our Blog: http://goo.gl/TA2HKP

Questions? @DanBoutinSOASTA


http://goo.gl/TA2HKP

78days

16hours

September 9, 2015

“Continuous Testing in DevOps”Dan Boutin – Senior Product Evangelist

[email protected] (404) 304-9529@DanBoutinSOASTA


Date post:	16-Apr-2017
Category:	Internet
Upload:	soasta
View:	1,798 times
Download:	0 times

Continuous Testing

Internet