Test Driving an Organisation

Paul O’Keeffe

Shaun Loh

We Have a Problem • Coreapplications• Businessideasslowtogettomarket

• Manualtestingtimesdominatedthereleasecycle

• Longtailtestcyclesdelayingrelease

• Defectdetectionlateintheprocess

• Somedefectsgettingtoproduction

• Multipleindependenttestteamssharinglittle

• Lackoftrustintestsduetoinconsistentcoverageandexecution

Quality / Speed Delivery • Qualityinitiative• Morphedinto“SpeedDelivery”

• Cycletimeisperhapsthemostimportantmetricinsoftwaredevelopment

• Reducethetimebetweenchangerequestandproductiondelivery

GUI Test Automation • Webapplications

• Desktopapplication

• Tightcouplingbetweenapplications• Initiallybuildasafetynet

• Thenbuildoutapplicationcoverage

• BusinesslogicintheGUIswasrifesowecouldn’tsensiblytestattheAPIlevelorevenattheHTTPlevel

What Test Pyramid? • Thetestpyramidminusthebaseandmiddle;◦  andthetop

• Webuildthetop,becausethebaseandmiddlecouldn’tbe

• Codewasnotdesignedtobetestable◦  Frameworks!

• Swisscheesepyramid

• Normally,buildingafewhappydaysscenariosatthetopofthepyramidistherightapproach,butwithoutagoodbaseormiddle,fragilityisworse

• Somanythingswouldbreaktests•  Who/whattoblame?

What Could Possibly Go Wrong? • Infrastructure• Environment

• Data• Applications

• Everything…

• StabilisingaGUItestautomationsuiteisincrediblyhardwork

Infrastructure • Cloud(AWS)

• Butproductionstillonpremise,sodifferent

• Longlivedsnowflakes• Provisioningwasgraduallybeingautomated

• Bothfortestingandapplicationinfrastructure

Controllable Environment • Naildownenvironmentonbuildagentsprecisely

• Spreadoutintonailingdowneverythingexternalwhichischangingand/orcouldcauseproblems•  Hardwarespecs•  Manualtesterssharingenvironments•  Batchjobs•  Softwareupdates

• Limitedduetoownershipbyotherteams•  Separationofacceptancetestingfromdevelopmentteams

Data • Don’trelyonacopyofproductiondata•  Becauseyoucan’trelyonit

• Alwaysgeneratethedatayouneedforatest• Productiondataandbatchjobstakingtoolongtoruninunderspecifiedenvironments

Test Isolation • Creatinganewmemberforeachtest

• Naturalisolation•  Datawascompletelyscopedbymember

• Ensureseachtestiscompletelyisolatedfromanyother

Applications (with Real Users) • Usersareapologistsforpoorapplicationbehavior•  Robotsaren’t

• Realusersseemedtohavebeentrainedtohaveahightoleranceforslowapplicationresponses

• Synchronousbusinessprocessesbuiltontopofasynchronoussystems

• Usersjustrefreshuntilthingshavefinished• Realusersworkedaroundapplicationerrorsliketheevery10thindexproblem

• Doesn’tmaketestautomationanyeasier,though

Application Driver Layer • Applicationmodels

• Pagemodels

• Growthisasyouneeditforthetestsyou’vewritten• Don’tputahardcaponittoosoon◦  Cucumber/Gherkin◦  Commercialtools◦  Otheropensourcetools(whoshallremainnameless)

• Domain,notUI

Application Driver Layer • Mosttoolsputahardcaponapplicationmodelling,usuallyatthepagelevel

• Needmoreflexibility.Noteverythingisshapedlikeacucumber

• Renewalsmodelisagoodexample

• Coderetainstheflexibility,sinceithasthenecessarymechanismsforabstraction

• Askyourselfwhetheryourtoolcancompetewithcodeonthisfront?

Page Models • Disappointinglackoffreetoolstodobasicpagemodellingascode

• LearnedalotabouthowtodogoodpagemodellinginJavaandC#

• SomeadvancesonthecurrentlyacceptedstateoftheartIMHO

• DesignofpagemodelDSLpreventedrun-timeerrorsandenabledautocompletion

• Projectcomingsoonwithsomeofourdesigns

• DSLsforAPIsandGUIs.Sometimesthesameonesowecouldswitchthetwoeasily

Asynchronous Behaviour • Nofixeddelays(sleeps!)• Pollingloops

• Events

Weapon of Mass Destruction

Effect on Systems Under Test • Weaponofmassdestruction

• Unexpectedloads/greaterscale•  Runningourtestscausedahigherloadthanproduction•  Duetohaving30buildagentsallrunningtestsfasterthannormalusers

• Buttestenvironmentsaremuchscaleddownversionsofproductiontocutcosts

• Singleserverratherthanclusters(multicastproblem)

• Datasizes?•  Insignificantcomparedtoproduction•  Butgrowthovertimewithoutresetcausedissues

Impact • Manualtestingisusuallyprettylowimpact

• Automatedtestingishighimpact

• ContinuousIntegrationisevenmoreimpact

• Existingsystemsmayonlybedesignedtohandlemanualtests

• Testingperformance/scaledemandsmaybehigherthanproduction!!!•  Especiallyifyouwantfastdevelopmentcycles

• Forcerestoftheorganisationtochange/improve

Test Execution • NeedatoolwiththerightgranularityforGUItestautomation.

• Runeachtestandcollectresultswouldbeaboutright.

• CIservertocoarse.Runalltestsinonebuild.• Testgridtoofine.RuneachUIactionremotelyfromacentralcontroller.

• BrokethingsupbyhavingabuildpertestonCIserver,butnotwellsupported.

Continuous Integration Server • Asopposedtothepracticeofcontinuousintegration• Supportforlongrunningacceptancetests

• Scalabilityofconfiguration• Newgeneration:pipelineascode(oratleastbetterconfiguration)◦  GUIshamperthis

MSHTA • TheUncannyValleyofwebapplications• MSHTAapplicationsareapaintotest

• Nearlyawebappbutnotquitesoyouneedspecialtools• Frameworkdidn’tallowapplicationtoberuninabrowserjustfortesting

• WouldbegoodtohaveafreetoolwhichcoulddothisaswellasSeleniumdoesforweb

• MSHTAhavesomedifferences•  Oftenmoredialogwindowwizards•  Lockeddownbrowser•  Fewercontrols

• Testautomationforcesorganisationawayfromthisstyleofapplication

Recording Evidence • Videorecording>• Sequenceofscreenshots>

• Finalscreenshot>• Logsonly>

• Nothing

• Someproblemsarehardtoreproducejustbyrunningthetestagain

• Observability

• Convincingevidenceforotherswithintheorganization• Otherbenefits

Licensing • Permachinelicensingschemesarepainfulfordesktoptestdevelopment,butmoresoforagridofbuildagentsinthecloud

• Donotbuytoolswithlicenceswhichdiscouragethenumberofpotentialenvironments

• Sitelicencesarefine

Results • Shortenedreleasecyclesbymanyweeks

• Increasedconfidencetoperformreleases,wheretheymightnototherwisehavebeenattempted

• Butotherchangesareslow

New State

Test Driving the Organisation • TDDawholeorganisation•  Conway'sLaw,sortof•  Canmakemovesandreorganisemoreeasilywiththesafetynet

• Removalofredundantdelivery/testingphases

• Smallenvironments

• Fastercycletimes•  Eachchangemakesthisalmostmorepossible•  Loweroverhead•  Costsaving•  Earliertesting•  ShiftLeft

Opportunity: Data Load • Builttoolsformanualtesterstosetupdataforexploratorytesting

• Alsoforothermanualtestswehadn’tyetautomated

• Removeddependenceonproductiondataandfindingappropriateexamples

• Wherepossible,usedAPIsfordataloadinourtestsetup

• Chickenandegg•  Nolongeranyneedforproductiondata,sowechangedtheorganisationinawaywhichsolvedourownproblemandseveralothers

• LearningandDevelopment(trainingdata)

• PusheddevelopmentteamstoopenupAPIs

Automate (Almost) Everything • AnalysistoolslikeDashyandHelpy.• Don’tjustautomateyourtests,automateyouranalysisandanythingelserepetitiveyouhavetodo.

Test Execution • Runyourtestsallthetime?•  Tradeoffs

• Atleastwhenanythingchanges,whichinourcasewasprettymuchallthetime

Deployment and Other Automation • Startedpushingthingstowardsautomateddeployments

• Wewouldhelpoutwiththis,sincewehadautomationskills

• Softwaredevelopmentskillsofteamflowedoverintosmallproductdevelopment

• RobotProcessAutomation(RPA)

Think Different • Developerviewoftestautomation

• Fresheyes

• Embracecode

• Improvedqualityoftestcode

• Bringtestinganddevelopmentclosertogether

Way of Working • Agile,butonlywithinoursmallpartoftheSDLC

• Collaborative

• Pairingondemand◦  Moarpairing!!!◦  Particularlywhereincreaseskillsorshareknowledge,wheretheyareunevenlydistributedwithintheteam

• Honestretrospectives• Continuousimprovementactionsactuallyactedupon

• Attemptedtoinfluenceotherteams

• Asfaraswecouldgowithintheconstraintsoftheorganization

Invest in your People • Super-interns• Andotherswithcomplementaryskillsfromelsewhereintheorganisation• Limitedbudget• Negotiatedtogetreallygoodinternsandthentrainthemwell

• Peopledon'tneedtobeanexactmatchforrequiredskills• Someprogrammingabilityiswelcome,butnoneedforlots• CancomefromotherITbackground•  Goodtohavealternativeperspectives(e.g.support)

• Needtobekeentolearn• Cangetfantasticleverageatlowcost.Wedid.

Invest in your People • Axesoflearning/growth•  Maturesoftwaredevelopment•  Automatedtesting•  Enterprisetraining/constraints

• Highlycollaborativeenvironment/team•  Workwithotherswithdevelopment/testing/domain/applicationexperience•  Balanceofperhaps50%ofteamalreadyexperienced

Invest in Your People • Freedomtofailandlearn•  Psychologicalsafety•  Givespeopleconfidencetotrynewthingsearlier

• Handsonimmediately

• Internsbecamethementors

• Growexistingpeoplefromtheorganisation•  Graftthemontoanewplanttoreignitetheirmotivationandgrownewskills

• Chocolate