Post on 04-Jun-2020
transcript
So#wareTestChallengesinIoTDevices
JonD.Hagar,Consultant,GrandSo4wareTes8ngembedded@ecentral.com
1
• Challengesinbothhardwareandso4waredevelopment– Canweproducequalitywithinscheduleandcostconstraints?– Mergingofphysical,cyber,andnetworkedworlds
• AlltheproblemsofITandMobileSo4ware
– ItonlytakesafewminutesofusinganAppbeforeuserslikeordislikeit
• Worsethanthat...– IoTcanKill:Youmaybeonthenightlynews(badpressisnotgood)
– CompanieswantapieceoftheIoTpie(4-10trillionUSDinnext10years)
Copyright2015JonD.Hagar-“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
2
TheIoTOpportunity
MyTopIoTChallenges(andwhichonesIcoverinred)
• Complexso4wareandhardware(fortes8ng)– Sensorsandthe“realworld”– Howtoconductdevelopment
• Numbersofdevicesandconfigura8ons(andhowtotest)– Configura8onsandcompa8bility– Reliabilityandfaulttolerance
• Bigdataandanaly8cs• PrivacyandSecurity• Connec8vity(systemsandsystemsofsystems)
– Integra8on
• Safety• Lifecycle–unifiedhardware-so4waredev-test-ops(adream)
– Toolstosupportdevelopment,opsandtests– Costandschedule– Concurrentso4wareandhardwaredevelopment
• IntegratedOpera8ons-giventheabove• Interna8onalstandardsfordevicesandprotocols
3Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
BasicDefini5ons
• Test–theactofconduc5ngexperimentsonsomethingtodeterminethequality(s)andprovideinforma5on– Manymethods,techniques,approaches,levels,context– Considera8ons:input,environment,output,instrumenta8on
• Quality(ies)–Valuetosomeone(thattheywillpayfor)– Func8onal– Non-func8onal– It“works”– Doesnoharm
• Arethere(cri8cal)bugs?
4Copyright2015JonD.Hagar-“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
TechnologySpace
PhysicalSystems(circa100,000BC)
5Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
CyberSystems(1950s)
Embedded
IoT
Mobile-Smart Personal Computers
Big Iron (Cloud)
Many Options Huge Numbers of Devices (billions)
Numbers of Devices (millions)
Cyber-PhysicalSystems(today)
WhereareIoTDevicesinComputerSpace?
6Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Embedded
IoT
Mobile-Smart Personal Computers
Big Iron (Cloud)
Many Options Numbers of Devices (billions)
Numbers of Devices (millions)
• Embedded–So4warecontainedin“specialized”hardware…– Minimalnetworking-communica8ons
PLUS
• Mobileandhandheldsmartdevices—small,heldinthehand,highlyconnected(web,cloud,servers,….)
• IoT–InternetofThingsare“tradi8onal”andnewdeviceswithso4wareandcommunica8onadded
WhatisanIoTDevice
Copyright2015JonD.Hagar-“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
Test Brakes
IoT–2.5MainSegments(tonameafew)
8Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Must be Interoperable Across segments
Industrial Value 2x Consumer
Industrial 4.0 Middle Consumer
Home - Security - Monitoring & control - Infotainment Human - Health - Fitness - Info
Vehicles - driverless - monitoring - Infotainment Office - Security - Energy - Worker info Medical - Health monitor/control - Records Retail - Ordering - Checkout - Advertize
Cities/States/Nation - Health - Safety - Info - Control and Monitor Transportation - Vehicles - Navigation - Logistics Worksite/Factories - Ops - Control - Info
Example:IoTConnectLandscape
9Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
IsIoTnew?
Wehavehadembedded,control,M-2-M,theinternetWhyisIoTsodifferent?
Itbringstogether:• Connec8vity• Bigdata• Resourcelimita8ons–size,baXeries,processing,memory,other• Numbersandtypesofdevices• Mixescloud,PC-IT,mobile,embedded,network,anduser• Securityandprivacy
10Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Challenge1:ComplexSo4wareandHardware
forTes8ngInParallelandSuppor8ngDevelopment
11Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
IoTTes8ngOpportuni8es• Requirementsverifica8onchecking
– Necessarybutnotsufficient
• Risk–basedtes8ng– Historicbuttriedandtrue
• PaXernoraXack-basedexploratorytests– PaXern1:Model-basedtes8ng– PaXern2(andChallenge2):Math-basedtes8ng– PaXern3:Skill/experience-basedtes8ng– PaXern4:Standards/process-basedtes8ng
Copyright2015JonD.Hagar-“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
PaXern1:Model-BasedTes-ng
• Addresssystems,so4ware,andhardwaretest
• DeveloperandIndependentModeling
• ImprovedUnderstanding
13Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
• Interestanduseofmodel-basedtes8ngisgrowinginindustrysegments• Telecom,finance,automo8ve,aerospace• EuropeanandUSAinterests
• IoT“highintegrity”areaswillneedit
• Model-basedtes8ngcansupport:• Genera8onoftestcasesfrommodelsintotestautomatedexecu8onengines
directlyusingscriptsorthroughtheuseofkeywords• Earlytes8ngwithimprovedunderstandingofthesystemandrisks• Useofmodelstosupportsimula8onstodrivetestenvironments• Verifica8onviacomparesbetweendevelopmentandtestmodels• Genera8onoftestresultoraclesorjudges• Supportofindependenttes8ngsuchasIndependentV&V(IV&V)• Modelanalysisandformalverifica8on
Model-basedTes5nginIoT
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
15
An Example Test Flow with Modeling for IoT
Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
• UTP1.2(soon2.0)standardinplace
• Toolsupportinplace• Producetestautoma8on• Graphicviewsaidunderstanding• Serveasanoracle
• Aidsinavoidanceand/oriden8fica8onofissuesearlyinlifecycle– Beforecodeorhardwarecomplete
• Considera8onsforgrowthandcon8nuingusage• N-versionproblem• Self-checkingproblemifonlyonemodeliscreated• Skilledmodelersandtestersneeded• Correctdevelopment/testenvironmentmustbeinplace
Model-basedTestAdvantagesandConsidera8ons
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
HowtoAddressPaXern2andChallenge2
withoneconceptHow do we handle
many configurations, options, and even test
data sampling?
17Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Pa1ern2:Math-basedTes-ng
Tes8ngisasamplingproblem: Howcantes8nguseMathtohelp?
• Testsystema8callythenumbersofdevices,configura8ons,
networks,etc.
• Samplinginenvironmentsandqualitycontrol
• Selec8onofdatafromtheinputdomainspace
• BigDataanaly8csfedintotes8ng
18Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
• Andriod
UseMathtoAddressChallenge2:Numbersofdevicesandconfigura8ons
19
• Routers
• IoTdevices
• Data
• CommChannels
• IoTHomeProtocol
How many Tests? to address, data, configurations, devices Comms, resources, integration, resources 10 x 2 x 13 x 6 x 6 x 7 = 65,520 tests
Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
UsingtheACTSCombinatorialTool:Example
20
Parameters:
AndriodAppPlasorm [Device1,Device2,Device3,Device4,Device6,Device7,Device8,Device9,Device10]
IoTProtocolHome [true,false]
IoTDevices
[Refrig,Stove,mircrowave,TV,frontdoor,Garagedoor,Homegaurd,Stereo,TempControl,Lights,Drapes,WaterHeater,windowopeners]
Routers [0,1,2,3,4,5]
Commproviders [Cell1,Broadband,cable,Cell2,Spacebased,Vendorgodzilla]
Data [1,0,-1,99999,-99999,100,-200]
TestCase# AndriodAppPlasorm IoTsHome IoTDevices Routers Commproviders Data
0 Device1 false Refrig 1 Broadband 0 1 Device2 true Refrig 2 cable -1 2 Device3 false Refrig 3 Cell2 99999
3 Device4 true Refrig 4 Spacebased -99999
4 Device6 false Refrig 5 Vendorgodzilla 100
5 Device7 true Refrig 0 Cell1 -200
119 Test Sample
Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Sta8s8calMathToolsGeneralTechniqueConcept ToolExamples
Examplesofwheretechniquecanbeused
Specificsub-techniqueexamples
CombinatorialTes-ng
ACT,HexawiserdExpertPICT
Medical,Automo8ve,Aerospace,Informa8onTech,avionics,controls,Userinterfaces
Pairwise,orthogonalarrays,3-way,andupto6waypairingarenowavailable
DesignofExperiments(DOE)
DOEProXLDOE++JMP
Hardware,systems,andso4waretes8ngwherethereare"unknowns"needingtobeevaluated
TaguchiDOE
RandomTes-ngandFuzztes-ngtools(security)
Randomnumbergeneratorfeatureusedfrommostsystemsorlanguages
Chipmakers,manufacturingqualitycontrolinhardwareselec8on
Tes8ngwithrandomlygeneratednumbersincludes:fuzzinganduseinmodel-basedsimula8ons
Sta-s-calSampling SAS
Mostsciences,engineeringexperiments,hardwaretes8ng,andmanufacturing
Numeroussta8s8calmethodsareincludedwithmoststa8s8caltools
SoMwareBlackboxDomainTes-ng
Mostlyusedinmanualtestdesign,thoughsometoolsarenowcomingavailable
Allenvironmentsandtypesofso4waretests.Theseare“classic”testtechniques,buts8llunderused
EquivalenceClass,BoundaryValueAnalysis,decisiontables
Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Pa1ern3:Skill/Experience-basedTes-ng
22Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
ExploratoryTes8ng-Defini8on
• Quo8ngJamesBach:“Theplainestdefini8onofexploratorytes8ngistestdesignandtestexecu8onatthesame8me.Thisistheoppositeofscriptedtes8ng(predefinedtestprocedures,whethermanualorautomated).Exploratorytests,unlikescriptedtests,arenotdefinedinadvanceandcarriedoutpreciselyaccordingtoplan.”
hXp://www.sa8sfice.com/ar8cles/what_is_et.shtml
CredittoJeanAnnHarison2013Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
Pattern attack–based
Exploration:AnImportantSkillforTesters
• Somepeoplethinkthatalltes8ngisexploratory
• Scien8ficmethods
• Usedatdifferent8mes– Early– Performance– OnHardware– Late
• BasedinpaXernsofa1ack
24Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
PaGern-basedTes5ngWhatisana<ack?
• ApaXern(fortes8ng)basedonacommonmodeoffailureseenoverandover– Maybeseenasanega8ve,whenitisreallyaposi5ve– Goesa4erthe“bugs”thatmaybeintheso4ware– Mayincludeoruseclassictesttechniquesandtestconcepts
• SeeLeeCopeland’sbookontestdesign• Seemanyothergoodtestbooks
• APaXern(morethanaprocess)whichmustbemodifiedforthecontextathandtodothetes8ng
• TesterslearnmentalaXackpaXernswhenworkingovertheyearsinaspecificdomain
Copyright2015JonD.Hagar-“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
• Verifica8onchecking(tests)ofrequirementsiscommon• Expectednecessarytes8ng,butbyitselfisnotsufficient• Singularfocusofmanytestteamsthatmisseserrorsandneeded
informa8on
• IncorporatesriskandaXacktes8ngwithinexploratory,experience-basedtestplanning• Allowsrapidtestexplora8onwithoutlimita8onsofhighlyscriptedtests• Requires“skilled”testteams
• Exploratorytes8ngmustbebalancedwithstrategies– Verifica8onandValida8onusingstandards– Math-based– Model-based
HardwareTestPlanningwithExplora5onConcepts
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
ExploratoryTes5ngInIoT
• Rapidfeedback• Learning• Upfrontrapidlearning
• AXacking• Addressrisk(s)• Coverdata• Reliability• Performance
• Independentassessment• Targetadefect• Prototyping• Needinfofordevelopers• Testbeyondtherequirements
• Cloud• FaultTolerance
CredittoJeanAnnHarison2013Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
MoreExamplesSo4wareAXacksforExploratoryTes8ngExcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
28
SoMwareTestA1ackType A1ackFinds NotesontheA1ack
Developerlevela1acks Codeanddatastructureproblems
Almostaquarteroferrorsinmobileandembeddedcanbefoundbystructuraltes-ng
Controlsystema1acks HardwareandsoMwarecontrolsystemerrors
Manycri-calerrorsinmobileandembeddedarecenteredinthecontrollogic,forexampleanalog-to-digitalanddigital-to-analogcomputa-onproblems
Hardware-soMwarea1acks HardwareandsoMwareinterfaceissues ThesoMwareshouldbetestedtoworkwithanyuniquehardware
Communica-ona1acks Digitalcommunica-onsproblems
SoMwarecommunicateswithhardware,network,andothersoMwarewithcomplexinterfacesthatshouldbetested
Timea1acks Time,performance,sequence,andscenarioerrors
SystemsoMwarecanhavecri-cal-mingandperformancefactorsthattes-ngcanprovidevaluableinforma-onabout
Userinterfacea1acks Problemsbetweenmanandmachine TheusabilityofdevicesandsoMwarearecri-caltosuccess
Smart/Mobile/Hardwarea1acks Issuesspecifictosmartdeviceconfigura-onsincludingcloudissues
Cloud-hybridcompu-ngcomprisesamajorityofthenewsoMwaresystemsbeingdeployed
Securitytesthackinga1acks SoMwareerrorsthatcanexposedevicestosecuritythreats
Securityofdevicesorsystemsisincreasinginimportanceanda1acksinclude,forexample,GPSandiden-tyspoofing
Genericfunc-onalverifica-ona1acks
Requirementsandinteroperabilityerrors BasicchecksthattestersshouldconductonsystemsandsoMware
Sta-ccodeanalysisa1acks Hardtofinderrorsthatclassictes-ngoMenmisses
CanoMenbedonebythedevelopmentgroupbutsome-mesthetestgroupmustrunthisanalysis
Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Pa1ern4:Standards-basedTes-ngForProcesses,NotProducts
29Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
• IEEE1012isastandardthatdefinesV&Vprocesses• Specificac8vi8esandrelatedtasks• AddressesV&Vatsystem,hardwareandso4warelevels• Canbeappliedtoafullsystem,sub-system,orelement
• Featuresinthestandardinclude:
• Integritylevels• MinimalV&Vtasksforeachintegritylevel• Intensityandrigorconsidera8onappliedtoV&Vtasks• DetailedcriteriaforV&Vtasks
IEEE1012-2012Verifica5onandValida5on(V&V)PlanningStandard
30 Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
ISO29119SoVwareTestStandard
31 Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
• AddressedbasicsofModelingearlier
• UTPisa“language”
• ISOisconsideringamodel-basedprocessstandardusingUTP2.0
OMGUTP
32Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Challenge3:IoTDataandAnaly8cs
33Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
TheEvolu8onofComputers=TheEvolu8onofDataUsage
34Copyright2015JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Embedded
IoT
Mobile-Smart Personal Computers
Big Iron (Cloud)
Many Options Huge Numbers of Devices (billions)
Numbers of Devices (millions)
Data Used by The Few
Data Pulled (from whole web)
App that Customizes a Data Slice
Limited Data (if any)
We do not Know yet ? ? ?
IoTtoGenerateHugeAmountsofData(Petabyte,Exabyte,ZeXabyte,OraYoXabyte)
Currentanaly8csfocusisonmarke8ng/sales
Ifuserisatestergenera8ngdata…….
Testerswillneedtousedataanaly8cs
Butforwhat?
35Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
• FromWikipedia:Taxonomyistheprac5ceandscienceofclassifica5on.Thewordfindsits
rootsintheGreekτάξις,taxis(meaning'order','arrangement')andνόμος,nomos('law'or'science').Taxonomyusestaxonomicunits,knownastaxa(singulartaxon).Inaddi5on,thewordisalsousedasacountnoun:ataxonomy,ortaxonomicscheme,isapar5cularclassifica5on("thetaxonomyof..."),arrangedinahierarchicalstructure.
– Fielddatahelpingto“understandandknow”errorstoimproveIoTdevelopmentandtes8ng
36
AnExampleofUsingDataAnaly5csforTes5ng
SoVwareErrors: ABadSitua5ontoAvoid
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
37
ErrorDataTaxonomy(Earlyversionofresearch)SuperCategory
Aero-Space Medsys IoT/Mobile GeneralTime 3 2 3 Interrupted-Satura8on(over8me)
5.5 TimeBoundary–failureresul8ngfromincompa8blesystem8meformatsorvalues
0.5 1 Time-RaceCondi8ons
3 1 Time-Longrunusages 4 1 20Interrupt-8mingorpriorityinversions
0.7 3 Date(s)wrong/causeproblem
0.5 1 Clocks 4 2 Computa8on-Flow 6 23 19Computa8on-ondata 4 1 3 1
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
IoTDataAnaly5cs–OneFuture
SODA–SelfOrganizingDataAnaly8cs
• ThetoolsanddataareorganizedtosupportallaspectsofIoTwithAr8ficialIntelligenceandcustomizedselec8onbasedonnatureofuser
• Users=customer,middlemen,governments,developers,managers,etc.
• Researchtopic38Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Challenge4:
IoTSecurityandPrivacy
(Manyexpertsthinkthesearetoppriority)
39Copyright2015JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
• YourIoTAppgetsonthenightlynews
• Yourteamseessecurityassomeoneelse’sproblem
• YoulosepersonaldataoryourAppmakespersonaldataavailabletoanyone
ExampleofWhatWorriesMeatNight:SecurityandPrivacy
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
• Mobile/IoT–IoTsystemsarehighlyintegratedhardware–so4ware–systemsolu8onswhich:– Mustbehighlytrustworthysincetheyhandlesensi8vedata– O4enperformcri8caltasks
• Securityholesandproblemsabound– CoverityScan2010OpenSourceIntegrityReport-Android
• Sta8canalysistestaXackfound0.47defectsper1,000SLOC• 359defectsintotal,88ofwhichwereconsidered“highrisk”inthesecuritydomain
• Carsandmedicaldeviceshacked
TheCurrentSecuritySitua5on
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
• Fraud–Iden8ty• Worms,virus,etc.
– Faultinjec8on• Processingontherun• Hacksimpact
– Power– Memory– CPUusage
SecurityErrors(refinementoftheso4wareerrordatataxonomy)
• Eavesdropping–“yeseveryonecanhearyou”• Hijacking• Click-jacking• Voice/Screen
• PhysicalHacks• Filesnooping• Lostphone
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
Are you giving away someone else’s keys?
• AXack28Penetra8onAXackTest • AXack28.1Penetra8onSub–AXacks:Authen8ca8on—Password • AXack28.2Sub–AXackFuzzTest• AXack29:Informa8onThe4—StealingDeviceData • AXack29.1SubAXack–Iden8tySocialEngineering • AXack30:SpoofingAXacks • AXack30.1Loca8onand/orUserProfileSpoofSub–AXack• AXack30.2GPSSpoofSub–AXack
SecurityAXacks(from“So4wareTestAXackstoBreakMobileandEmbeddedDevices”)
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
Privacy–RestrictedData• Differentfromsecurity
» Moreofanissueinsomecountries• ExamplesImightnotwantexposed
44Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
• CompaniesmustleveragethedatacomingfromIoT– Sani8zedata
• Bigdataanaly8cs
– ImproveTestandDev-Ops
– HowtomaintainPrivacy?
• Likelywillneedoptin/outwith“mo8va8onfeature”
ALLTHISWILLNEEDTESTING
PrivacyImpactonData
45
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
Challenge5:Connectivity
WhatseparatesIoTfromEmbedded
46Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Connec5vityOpportuni5es
• Tes8ng
– Testthedevice,thenetwork,thecloud,theapp,and????
– Issuesinconnec8vity
– Securityandprivacy(again)
– Protocols–noclearwinnersyet
– Manyop8ons
47Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
SODA
Tes5ngOp5onsforConnec5vityTestEarly• Model-basedtes8ng• Math-basedtes8ng
TestO4en• Testlabs
TestConsistently• Risk-basedtes8ng• Requirementsverifica8onchecking• Automa8on
48Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Connectivity
• TotheUserSo4wareHardwareHuman
• Tothesystem
• Tothesystemofsystem
• Tothedata
49Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Connectivity Concerns
• Timelag
• Datacorrectness
• Differentconfigura8ons
• Datacompleteness
• Privacyandsecurity(yetagain)
50Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
Connectivity Factors
• Testenvironment
• InterfaceProtocolop8ons
• Deviceop8ons
51Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"
IoTTes5ngSummary• Todefeatanenemy,youmustknowthebug
• TheIoTtestdataislimited,– Whatexistshasimplica8ons
• TherearechallengesandpaXernsofopportunaty
• So4warewillbeinverynearlyeverything– Tes8ngmaybealimi8ngfactor
52Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
References(myfavoritebooks)• “SoVwareTestAGackstoBreakMobileandEmbeddedDevices”
–JonHagar– IoTTestsBookin2016
• “How to Break Software” James Whittaker, 2003
– And his other “How To Break…” books • “A Practitioner’s Guide to Software Test Design” Copeland, 2004 • “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al.,
1993 • “Computer Related Risks”, Neumann, 1995 • “Safeware: System Safety and Computers” Leveson, 1995 • Honorable mentions:
– “Systems Testing with an Attitude” Petschenik 2005 – “Software System Testing and Quality Assurance” Beizer,
1987 – “Testing Computer Software” Kaner et. al., 1988 – “Systematic Software Testing” Craig & Jaskiel, 2001 – “Managing the Testing Process” Black, 2002
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
MoreResources
• www.stickyminds.com – Collection of test info • www.embedded.com – info on attacks • www.sqaforums.com - Mobile Devices, Mobile Apps - Embedded
Systems Testing forum • Association of Software Testing
– BBST Classes http://www.testingeducation.org/BBST/
• Your favorite search engine
• My web sites and blogs (see front page)
Copyright2015JonD.Hagarexcerptedfrom“So4wareTestAXackstoBreakMobileandEmbeddedDevices”
ReferencesforSta8s8calMathTools• IEEE1012,StandardforSystemandSo4wareVerifica8onandValida8on-hXp://standards.ieee.org/findstds/standard/1012-2012.html,IEEEpress,2012• ISO29119,So4wareTestStandard-hXp://www.so4waretes8ngstandard.org/• Hagar,J.SoVwareTestAGackstoBreakMobileandEmbeddedDevices,CRCpress,2013• Kuhn,Kacker,Lei,Introduc5ontoCombinatorialTes5ng,CRCpress,2013(includesthetoolACTS)• Tool:Hexawise-app.hexawise.com/• Tool:rdExpert–www.phadkeassociates.com/• Tool:PICT–msdn.microso4.com/en-us/library/cc150619.aspx• Reagan,Kiemele,Tool:DOEProXL-DesignforSixSigma,AirAcademyAssociates,selfpublish,2000• DOE++-www.reliaso4.com/• SAS-www.sas.com/• Kaner,Hoffman,Padmanabhan,TheDomainTes5ngWorkbook,selfpublish,2013• Bailey,DesignofCompara5veExperiments.CambridgeUniversityPress,2008• Kacker,Kuhn,Hagar,Wissink,"IntroducingCombinatorialTes8ngtoaLargeSystem-So4wareOrganiza8on,”scheduled-2014,IEEESoVware• WhiXaker,James2003,HowtoBreakSoVware,PearsonAddisonWesley• WhiXaker,JamesandThompson,Herbert,HowtoBreakSoVwareSecurity,PearsonAddisonWesley,2004• Andrews,WhiXaker,HowtoBreakWebSoVware,PearsonAddisonWesley,2006• Levy,ToolsofCri5calThinking:MetathoughtsforPsychology,1996• Bach,Bolton,“Tes8ngvs.Checking,”www.developsense.com/blog/2009/08/tes8ng-vs-checking/• Hagar,“Whydidn’ttes8ngfindtheembeddedGMTruckfiresystemerror?”-www.breakingembeddedso4ware.wordpress.com/• OMGUTP1.2,www.omg.org/spec/UTP/1.2/• Baker,Dai,Grabowski,Schieferdecker,Williams,“Model-DrivenTes8ng:UsingtheUMLTes8ngProfile,”2008• Green,Hagar,“Tes8ngCri8calSo4ware:Prac8calExperiences,”IFACConference1995• Boden,Hagar,“HowtoBuilda20-YearSuccessfulIndependentVerifica8onandValida8on(IV&V)ProgramfortheNextMillennium,”QualityWeekConference1999
• Port,Nakao,Katahira,Motes,ChallengesofCOTSIV&V,Springerpress,2005
55Copyright2016JonD.Hagar–"So4wareTestAXackstoBreakMobileandEmbeddedDevices"