Post on 11-Jul-2020
transcript
T16 Performance Testing Thursday, May 3rd, 2018 1:30 PM
Embedding Performance Engineering into the CI/CD Pipeline
Presented by:
Anjeneya Dubey
McGraw-Hill Education
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-- -268- - -8770 ·· 904- --278-- -0524 - info@techwell.com - http://www.stareast.techwell.com/
Anjeneya Dubey McGraw-Hill Education Anjeneya Dubey is the director of performance engineering for McGraw-Hill Education, a learning science company that delivers personalized learning experiences. His responsibilities include ensuring that every product built is high performing, highly scalable, highly available, highly reliable, and fault tolerant. In his past five years with McGraw-Hill, Anjeneya has built automated performance engineering frameworks that detect performance and scalability issues early on in a fast-paced agile environment. Previously he was a technology consultant, focused on providing enterprise quality and performance engineering solutions. Anjeneya has worked with large institutions to set-up enterprise performance and quality engineering solutions.
3/13/18
1
Embedding Performance Engineering Into Continuous Integration & Continuous Delivery Pipeline
By – Anjeneya Dubey
LittleContextaboutMcGraw-HillEducationandMe
AnjeneyaDubeyDirectorofPerformanceEngineeringAnjeneya.dubey@mheducation.com
SoftwareengineeringPerformanceEngineeringCapacityEngineeringInfrastructurePlanningandimplementationAWSCloudArchitecture&OperationsSiteReliabilityEngineering
2
3/13/18
2
Agenda
• ContinuousIntegrationandContinuousDelivery• WhatdoesitmeantoincludeperformanceengineeringintotheCICDPipeline• Challenges• Whatdidwedotoincludeperformanceengineeringinthepipeline
• Processchanges• Performancetesttypes• TestEnvironmentmanagement• TestDatamanagement• ToolsandTechnologiesweuse• Pass/failDecisionMaking
• SelfServicePerformanceEngineering• UsingAIinproduction• Do’sandDon’ts• Summary
3
• AutomatedbuildprocessandbuildverificationtestsforeachenvironmentinContinuousintegration• ExtendContinuousintegrationbyrapidlydeployingcapabilitiestouserstogaincompetitiveadvantage• Reducetestcycletime&timetomarket• Highlyautomatedtesting&release/roll-back• Quickerautomateddecisionmaking&feedbackloop
ContinuousIntegration/ContinuousDelivery
4
3/13/18
3
EmbeddingPerformanceintothePipelineYourpipelineascode• Dev->Test->Prod• Dev->Test->Performance->Prod
Whatdoesitmean?AddingPerformanceenvironmentintothepipelinemeansthatnowtheperformancetestsareblockingyourcodepromotion
Dev Test Production
CodePromotion
Feedback
Performance
5
Challenges- Cultural
• Performanceisanafterthought• Isnotpartoftheagileteams• Isnotpartofthequalityteams• Donotgetincludedintheagileceremonies• Createawarenessonperformancetasks• Empowerdevtotest
6
3/13/18
4
Challenges- Technical
• Automatingtheperformancetestingandanalysis• ReducingTimetoprepareandexecutetest• Quicklyreactingtoperformancemetrics• AutomaticPass/Fail• Scalingtheloadtesttoolforvarietyoftests• Keepthetestingenv/dataconsistent• WhattoShiftleftwhattoshiftright• Costofrunningperformancetestoneverybuild
7
Howdowedoit@MHE?
8
3/13/18
5
Processchanges
• Makenonfunctional/Performancerequirementaspartofthefunctionalrequirement• APIcontracts• Includeperformanceaspartofdefinitionofdoneforsprints• Cleardefinitionofperformancereadyproduct• DiscussPerformanceresultsaspartofthesprintdemoswithallstakeholders
9
PerformanceRequirementsWorkflow
Storieswithacceptancecriteriathatincludesclearperformancerequirements• APIXmusthandleloadofxxtransactionspersecwith95%ileresponsetimeas100ms• AllStoriesmustbeevaluatediftheyrequireperformancecriteria• Performancetestsshouldbecreatedtovalidatethecriteriawithinthesprints• PoorPerformance=FunctionalBug
10
3/13/18
6
TypicalPEProcess
11
TestEnvironment
• Useproductionlikeperformanceenv• Spinuponlywhenyouruntesttosavecost• RefreshDBsfortestdatamanagement
12
3/13/18
7
• Canexpandandcontract- Autoscaling• Infrastructureascode– Terraform,Puppet• Creating&destroyingenvs atease• Createparallelenvs forparallelexecutions
Cloudmakesiteasier
13
Spinupparallelenvs forparallelexecutions
PerfEnv A
PerfEnv B
• Productioncapacityinstance
• ProtocolLevelfullloadtest
• UIperformancetestusingfunctionaltestscripts• Scaleddownenv
• Stress/capacitytest
PerfEnv C
• Testingoutsidethepipeline
• Troubleshooting• Benchmarking/baselining
tests
14
3/13/18
8
PerformanceTesttypesinCICD
• UserExperience- Browsersideperformance• Loadtests• Capacity/StressTests
15
SingleuserperformanceGoodUX=CustomerHappyHowdowemeasurethat?• Collectsingleuserbrowsersideresponsetimes• Leveragefunctionaltestscripts(selenium)• Createscenariosthatyouwanttomeasurethroughourselfserviceautomationframework• AllMethodsinthescriptshavethesnippetthatcollectstheresponsetimes• Executedfromvariousgeolocations• Usabletimevslastbyte• CollectingHAR&Creatingvideosofthetestsforofflineanalysis• UploadthedatatoS3• MHEPerformancePlatformtakesoverfromthere
16
3/13/18
9
Loadtests
• Fullloadtests• ScaledDowntests• Stresstesttofindcapacity
17
FeatureFlags
Whattodowhenyoufindperformanceissues?• Blocktherelease• TurnOffthefeaturethatcreatestheperformanceissue
18
3/13/18
10
TestDatamanagement
• Makeourtestsselfcontained• Create&destroydataaspartofthetestasmuchaspossible
• Fortheonesyoucantcreateduringthetestyoucreateaspartoftheenvironmentbuildout• SpinupparallelAuroraRDSwithpreseededtestdatatospeedupenv buildout
19
Tools&Technologiesweuse
20
3/13/18
11
PerformanceEngineeringPlatform
• Singularplatformtomanageperformancelifecycleforallofourproducts• PowersCICDforPerformanceengineering• Centralrepoforallmetrics• Dynamicthresholds• Passfaildecisionmaking• PowersSelfServicePerformanceEngineering
21
PEPlatformOverview
CollectorServiceAggregatorService
CentralRepository
Reporting&AlertingService
Developer
22
3/13/18
12
PEPlatform– Performancetesttypes
23
Trending– Performancegraphforeachbuild
Build3383on2/28isfailedfor
anAPI
Build3362on2/26isbrokenforanAPIastheresponsetime
degradedalmost30%
24
3/13/18
13
ContainerizeJMeter
• WeuseJMeterheavilyforthealltheCICDtesting• Distributedloadtesting– weneed1master&Nnumberofslavestogeneratehugeload• ScalingtheJMeterforthousandsofuserswasachallenge• Dockerize JMetergivesthescaleneeded• Speedsuptheprovisioning• Partoftheinfraascode– whichmeanswhenthecodegetsdeployedautomaticallyJMeterfarmgetsprovisionedwherethetestgetsexecuted
25
AutomatedPass/Fail
Basedon3basicrules
• Simple&Easy• Implementable• Dependentonthroughput,responsetimesandsystemKPIs
26
3/13/18
14
Thresholdsforpass/fail
• StaticBusinessresponsetimesSLAs• Dynamicuserexperience/APIlevelResponsetimesthresholds• DynamicSystemResourceutilizationthresholds• BasedonhistoricaltrendforeachAPIandalertsifitdeviatedfromlastntests• AllowsseparatethresholdforeachAPI• Doesn’tallowslippageevenwithinthecontract
27
Self-ServicePerformanceEngineering
• Youdon’tneedtobeperformanceengineertoruntest• Automatetheentireperformancecycle• ScriptCreationthroughaUI• ExecutetestaspartofCICDorExecuteitondemandthroughvoiceenabledAlexaorachatBot• AnalysisthroughAPMandMHEbuiltPerformancePlatform• AutomatedNotificationthroughHipchat/Email/PagerDuty• AutomatedDefectcreationwithdetailsinjira
Tester Devops
Developer
PerformanceEngineer
28
3/13/18
15
Self-ServicePerformanceEngineeringTest
Creation
CICD Alexa ChatBots
APM MHEPEPlatform
Hipchat Email Pagerduty
Execution
Analysis
Notificatio
nDe
fect
Jira29
Notifications
• Automateddefectcreation
• Summaryofthetestresult
• APMdashboardlinkswithdrilldown
• Automatedrealtimehipchatnotifications
• WithJiralinkanddetails
30
3/13/18
16
ShiftRight- Anomalydetection
• TwitterAnomalyDetection• Twitter’sBreakoutDetection• PearsonCorrelationAlgorithm• K-MeansClustering• NewRelicRadar
31
Do’s&Don’ts
• Startwithsimple• Perfectitlater• Removefalsepositives- Getitrightfromthebeginning• KnowyourapplicationsKPIs• Runparallel tests• Runcontinuoustests
• Don’trunbenchmark&endurancetestinCI• Dont removethefailingteststopassthroughCD• Don’tkeepincreasingthethresholdstopasstests• Don’treinventyourPEframeworkratherseehowyoucanleverageyourexistingtoolsandframeworkinCICD
32
3/13/18
17
Summary
• IncludeperformanceengineeringinyourCICDpipeline• Automateautomate&automate• Makeyourtestsrepeatable• Collectmetricsalongtheway• Avoidfalsepositives• Keepanalysis&decisionmakingsimple• Empowerdevs totest
33
Questions?
34