Post on 09-May-2015
description
transcript
T10 Mobile Testing
5/2/2013 11:15:00 AM
Mobile Testing Methodologies: Trends,
Successes, and Pitfalls
Presented by:
Eran Kinsbruner
Perfecto Mobile
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
Eran Kinsbruner
Eran Kinsbruner is director of product marketing at PerfectoMobile, one of the leading mobile cloud and automation companies. Formerly CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. The co-inventor of a test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems, Eran has experience in the mobile testing world. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and his professional mobile testing blog at ek121268.wordpress.com.
4/16/2013
Perfecto Mobile 1
Mobile Mobile Testing Testing Center of Excellence GroupCenter of Excellence GroupPerfectomobilePerfectomobile
Mobile Testing Methodologies Trends,
Successes, and Pitfalls
StarEast - 2013
Enabling Enterprise mobility
Extending Software Quality to Mobile
About me and my company
Mobile Market Trends
Mobile Quality Trends
Mobile Quality Challenges
Mobile Risk Management
Mobile Testing Best Practices
AgendaAgenda
4/16/2013
Perfecto Mobile 2
∙ Founded 2006
∙ 80+ employees
∙ Office in Boston, London, Israel
∙ Over 600 customers
∙ Fortune 500 enterprises, Telcos, SMBs
∙ SaaS Platform + Licenses
∙ Recurring revenue model
Mobile Application QualityMobile Application Qualityacross the mobile application lifecycle
MonitoringMonitoring TestingTesting AutomationAutomation
ALM Integration
ALM Integration
OffshoringOffshoringContinuous IntegrationContinuous Integration
Public Cloud Private Cloud
Mobile Market Trends
4/16/2013
Perfecto Mobile 3
Global Global Mobile Market Mobile Market SStatstats
Source: nielsen, 2012
New Mobile
OS’s
FireFox OS
Tizen
Samsung/Intel
Ubuntu
SailFish
By Jolla
RIM’s BlackBerry OS
10
Windows Phone
PEAK KEON
Z10 & Q10
Mobile OS market expected Mobile OS market expected
to grow (even more)to grow (even more)
4/16/2013
Perfecto Mobile 4
Cost Of Mobile Quality RisesCost Of Mobile Quality Rises
Mobile accounts for 11% of U.S e-Commerce
Existing Mobile OS’s Existing Mobile OS’s –– End of 2012End of 2012
Samsung accounted for 63 Million out of the 159 Million in Q42012
Source:IDC
4/16/2013
Perfecto Mobile 5
Worldwide smartphone vendors Market share Worldwide smartphone vendors Market share
Source: IDC Worldwide Mobile Phone Tracker, January 24, 2013
Market insights Market insights –– Forecasts and trends Forecasts and trends
Mobile based shopping expected to grow
by 24% in 2013 (U.S)
4/16/2013
Perfecto Mobile 6
Mobile OS battle continues
•New open source competitors trying to influence the eco system
•Windows Phone and BlackBerry OS 10 – critical year for their positioning in the market
Mobile apps become more complex – impact testing and automation
HTML5 and Mobile web expected to grow significantly
EMM establishment – Processes, Apps, Tools, Roles (CMOO)
Mini-tablets and “Phablets” will successfully blur the lines between device types
Market trends Market trends -- 20132013
Mobile Quality Trends
4/16/2013
Perfecto Mobile 7
World Quality report 2012World Quality report 2012--20132013
Mobile Testing is Behind the CurveMobile Testing is Behind the Curve
Source: CapGemini
FragmentedFragmented
DynamicDynamic
ComplexComplex
Business criticalBusiness critical
Mobile EnvironmentMobile Environment
Source: Google, 2012
4/16/2013
Perfecto Mobile 8
Countless screen Countless screen resolutionsresolutions
Source: opensignalmaps, 2012 (Android only)
Consumer Reaction Consumer Reaction –– Less Tolerant than In Desktop Less Tolerant than In Desktop
4/16/2013
Perfecto Mobile 9
HTML5 Trend Involves RisksHTML5 Trend Involves RisksTo help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Mobile Quality Challenges
4/16/2013
Perfecto Mobile 10
Mobile Introduces various challengesMobile Introduces various challenges
Various OS
Types and Versions
Various Device/Tablets
OEM’s
Sample Screen Sizes
(Only Smartphone)
Application Types
On various browsers
AGILE ALM
Native Hybrid Web
Functional
Automation
Performance
2.3.3, 4.0.3
4.1.2, 4.2.2
4.3.3, 5.1
6.1, 6.1.1
QVGA: quarter
VGA (240×320
pixels)
HVGA: half VGA
(320×480 pixels)
WVGA: wide VGA
(480×800 pixels)
FWVGA: full wide
VGA (480×854
pixels)
nHD: one-ninth
high definition
(360×640 pixels)
qHD: one-quarter
high definition
(540×960 pixels)
MMobile obile SSecurity ecurity TThreatshreats
Source: Techcrunch
4/16/2013
Perfecto Mobile 11
Mobile OS Regression bugs Mobile OS Regression bugs –– What does it have to do What does it have to do
with my Mobile Testing?with my Mobile Testing?
Source: LINK
Android defect database
Apple Bug Reporter
Pre-release testing (iOS6.1.1 Beta)
Per Device version
Mobile Web Objects Mobile Web Objects –– Automation ChallengesAutomation Challenges
4/16/2013
Perfecto Mobile 12
Apple iOS Apple iOS –– UI UI GuidelinesGuidelines
SourceSource: :
httphttp://developer.apple.com/library/ios/#://developer.apple.com/library/ios/#documentation/UserExperience/Conceptual/Mobildocumentation/UserExperience/Conceptual/Mobil
eHIG/Introduction/Introduction.htmleHIG/Introduction/Introduction.html
Android OS Android OS –– UI Guidelines UI Guidelines
Source:
http://developer.android.com/design/index.html
4/16/2013
Perfecto Mobile 13
Risks Management
Cost of quality in mobile grows and impacted by
• Availability
• TTM
• Functionality and UX
• New platform support (Mobile OS, Mobile devices)
• Previous app version compatibility
Mobile Quality Risk AnalysisMobile Quality Risk Analysis
4/16/2013
Perfecto Mobile 14
Mobile endMobile end--user Less Tolerantuser Less Tolerant
Obviously Obviously someone forgot someone forgot
to do their regression to do their regression
testing on a Galaxy S3 testing on a Galaxy S3
before releasing this before releasing this
version to market…version to market…
�Listen to the Listen to the
marketmarket
4/16/2013
Perfecto Mobile 15
A clear process as part of the “mobile project” lifecycleA clear process as part of the “mobile project” lifecycle
Define platforms and categoriesDefine platforms and categories
Define coverageDefine coverage
Acceptable not to test - You can’t test everything
Define change frequencyDefine change frequency
Adapt to the market trends as they happen
Define SLA for changes
Define coverage as part of the ALM phaseDefine coverage as part of the ALM phase
Coverage visibility reports to project managers and executivesCoverage visibility reports to project managers and executives
Device governance Device governance
Device Governance is a risk management for mobile projects
You can’t cover everythingYou can’t cover everything
But you should define a target, which is the acceptable
risk
iOS is not as simple as you thinkiOS is not as simple as you think
At least 3 HW platforms (3GS, 4, 4S, 5)
(iPADs, iPODs)
At least 3 SW platforms
Device Governance Device Governance –– Define coverageDefine coverage
Latest
55%
prev
minor
25%
prev
major
15%
Other
5%
4/16/2013
Perfecto Mobile 16
Mobile Malware Mobile Malware -- Trojans, viruses and rootkits migrating from traditional online banking and Trojans, viruses and rootkits migrating from traditional online banking and
designed specifically for the mobile marketplace. Researchers see an increase in mobile designed specifically for the mobile marketplace. Researchers see an increase in mobile
malware development malware development -- in pace with market growth.in pace with market growth.
ThirdThird--Party Apps Party Apps -- Consumers love their smart phone and tablet applications, but often these Consumers love their smart phone and tablet applications, but often these
apps come from third parties with questionable security practices. Or worse, the apps are apps come from third parties with questionable security practices. Or worse, the apps are
created by fraudsters and loaded with malware.created by fraudsters and loaded with malware.
Unsecured WiUnsecured Wi--Fi Fi -- The unsecured wireless network is a tollThe unsecured wireless network is a toll--free highway for free highway for fraudstersfraudsters to gain to gain
access to access to mobile devicesmobile devices, either to seize control of or , either to seize control of or gaingain access to account informationaccess to account information..
User Behavior User Behavior -- Consumers are prone to download Consumers are prone to download thirdthird--party appsparty apps, use unsecured wireless , use unsecured wireless
networks, open and click links in networks, open and click links in SMS text SMS text messages and emessages and e--mails, and mails, and lose their mobile deviceslose their mobile devices..
Mobile testing and Hardware Mobile testing and Hardware (Payment card readers, healthcare units and more) (Payment card readers, healthcare units and more)
Enterprises OWNS the risk for Mobile qualityEnterprises OWNS the risk for Mobile quality
Uncompromised necessity for testing mobile apps
Should adhere to Should adhere to regulationsregulations ((GLBAGLBA//FFIECFFIEC, , RDCRDC, , FCRAFCRA, , PCI DSSPCI DSS) around security, ) around security,
authentications, payment and moreauthentications, payment and more
Usually support complex business work flows (deposit check via camera)Usually support complex business work flows (deposit check via camera)
Multi tier interactions (Databases, trading accounts, other services) Multi tier interactions (Databases, trading accounts, other services)
High Availability and MonitoringHigh Availability and Monitoring
Must address mobile end user actionsMust address mobile end user actions
Using Home and Back button during application run
Network lose during transaction
Roaming during user transaction
Application behavior in various networks Application behavior in various networks
3G, 4G, WiFi
DSS Compliance test case example
4/16/2013
Perfecto Mobile 17
Mobile Mobile IIs s MMandated andated TTo o SStrict trict RRegulationsegulations
Source: LinkSource: LINK
Mobile Testing Best Practices
4/16/2013
Perfecto Mobile 18
This is not a mobile test lab!This is not a mobile test lab!
One Chair
10 Devices
3 Extra Devices Many Cables
3 Workstations
Choose the right tools and platformChoose the right tools and platform
Understand your users device and platform choices
Optimize device and platform testingOptimize device and platform testing
Automate as much as possible using device agnostic scripting
Use mobile cloud based solutions (Collaborative, Governed, Scalable,
Cost Effective)
Use CD/CI solutions as part of your mobile ALM
Do not reDo not re--invent the wheelinvent the wheel
Extend existing ALM tools to Mobile.
Building Building Blocks Blocks OOf f MMobile obile TTesting esting SStrategytrategy
4/16/2013
Perfecto Mobile 19
World Quality report 2012World Quality report 2012--20132013
Source: CapGemini
Consider Consider outsourceoutsource to reduce to reduce mobile manual testing mobile manual testing costscosts
Test for Test for performanceperformance before releasing to the marketbefore releasing to the market
Make sure you do Make sure you do regressionregression on older supported platforms on older supported platforms
Do not have your end users do your QA work
Perform pro-active testing on new devices and mobile OS’s
MonitorMonitor in production to assure end user experience and in production to assure end user experience and
availabilityavailability
ListenListen to your end users and act upon their feedback (Markets to your end users and act upon their feedback (Markets
have significant impact on your business and branding)have significant impact on your business and branding)
Building Building Blocks Blocks OOf f MMobile obile TTesting esting SStrategytrategy
4/16/2013
Perfecto Mobile 20
(True) Mobile (True) Mobile CloudCloud
Real Devices Real Devices + Emulators+ Emulators
Manual + Manual + AutomationAutomation
ALM ALM IntegrationIntegration
EndEnd--toto--End End Mobile ALMMobile ALM
Avoid device management
Secure, Centralized
Real smartphones
Real Tablets
Emulators
Device agnostic
Interactive
Core level extension
Short learning curve
Development
Functional
Performance
Monitoring
Erank@perfectomobile.com