Post on 12-Apr-2017
transcript
with Costa Avradopoulos
Designing a Mobile Device Lab
#MobileDeviceLab
XBOSoft Dedicated to software quality improvement
Founded in 2006
We speed products to market with our expert:
• Software QA consulting
• Software testing
Global team with offices in San Francisco, Beijing and Amsterdam
“Thorough, accurate and fast”
XBOSoft, Inc. All Rights Reserved. 2
House Rules Participants other than the speakers are muted
Questions via the GoToWebinar control on the right side of your screen or through Twitter @XBOSoft
Questions may be asked throughout the webinar - we’ll try to answer them at the end
You will receive info on recording after the webinar
XBOSoft, Inc. All Rights Reserved. 3
Webinar Hashtag: #MobileDeviceLab
Meet Our Speakers
Philip Lew CEO and Founder, XBOSoft • Relevant specialties and
passions o Software quality process,
evaluation, measurement and improvement
o Software quality in use / UX design
o Mobile User Experience and usability
o Cycling and travel
XBOSoft, Inc. All Rights Reserved. 4
Karla Smith Marketing Manager, XBOSoft • Emails you about our
industry hot topic webinars
• Invites you to download our latest trends in software testing whitepapers
XBOSoft Team
Guest Presenter Costa Avradopoulos
XBOSoft, Inc. All Rights Reserved. 5
• Over 25 years’ experience in development, from
requirements, to delivery, and process engineering
• Roles in Development, QA, Automation, Product Management, VP, and CTO. Management of systems up to 15 million users in verticals such as Telecommunications, Financial, Wireless, Transportation, and Retail.
• Much of career dedicated to the mobility space, culminating with a breakthrough
patented mobile technology
• Frequent speaker. Writing book due to be published soon: Winning Mobile Strategies – Bridging the Quality Gap
• Principal for Avracom. Advises organizations in the creation of winning mobile strategies to develop best of breed applications.
• Certified Six Sigma and TPI
costa@avracom.com www.avracom.com
Why We’re Here
XBOSoft, Inc. All Rights Reserved. 6
• Mobile applications – disruption in the market and in the SDLC
• Mobile testing requires a different approach • A good strategy starts with designing a good
mobile device lab • With thousands of variations, traditional
approaches are insufficient
Agenda
XBOSoft, Inc. All Rights Reserved. 7
• Mobile Testing Challenges • Defining Lab Requirements • Building a World Class Device Lab
Out of Scope for Today: • Discussion on specific tools • Automation
XBOSoft, Inc. All Rights Reserved. 8
How Do You Define Mobile?
1 ABLE TO MOVE OR BE MOVED FREELY OR EASILY. 2 OF OR RELATING TO CELLULAR PHONES,
HANDHELD COMPUTERS, AND SIMILAR TECHNOLOGY.
Smartphones, Feature phones, tablets, laptops, ruggedized devices, medical devices, wearables, kiosks, ATMs…
XBOSoft, Inc. All Rights Reserved. 9
World Quality Report 2014
KEY FINDINGS:
The QA function is adapting to business demands by streamlining and centralizing its structure, to achieve efficiency and cost optimization
Significant investment in Test Environments may also be undermined by a lack of specialist expertise
Testing’s late engagement in the application delivery lifecycle and its reliance on basic IT metrics is possibly applying a negative brake on this increasing maturity
Mobile testing has increased in importance as a key discipline within the function, but lacks specialized methods, expertise and environment
XBOSoft, Inc. All Rights Reserved. 10
World Quality Report 2014
KEY FINDINGS (Cont.):
Cloud adoption and cloud-based testing has slowed as organizations continue to manage concerns regarding data security and performance
A higher proportion of overall IT budget is being invested in testing and focused on transformation projects
Agile development is now widely adopted but still gives rise to problems for testing, particularly in relation to specific methodologies and expertise
NOTE: Mobile Testing is a dedicated section in this report To obtain a copy of this report, email costa@avracom.com
Mobile Testing Areas of Focus - 2014
Source: World Quality Report, 6th edition
11
XBOSoft, Inc. All Rights Reserved.
The Cost of Poor Quality Escalates
65% OVER of all apps have a 1 star rating or less
Top quality issues in last 12 months exceed $2 Billion in losses:
• Security: breach at top 10 major retailer ($570M+ loss) • Performance/Outages: Top 3 social media company – 5 major
outages in 2014, and 2 outages in 2015 • Security: breach at top 10 major retailer ($500M+ loss) • Functional: Top 3 bank, double payments issue
12
XBOSoft, Inc. All Rights Reserved.
Impact of Failure
Cost of Poor Quality is Accelerated with Mobile
Visibility is far greater, exposing your brand & reputation:
• Social Media makes failures exponentially more visible, with potential to go viral
• App store is a public report card - you cannot hide or cover up poor ratings
A plethora of choices for customers = competitive risks
• Near zero switching costs for customers • 60% of users will abandon your app/site if does not load in 3 seconds • Less than half will return to give a slow app/site a 2nd chance
13 XBOSoft, Inc. All Rights Reserved.
Mission Critical
Low Criticality
Mobile Testing Challenges
Security
Usability
Performance
Networks
Fragmentation
Due to involvement of GSM/ CDMA carriers identification of threats, risk and determining its business impact is crucial
Thorough understanding of Visual, Accessibility and certification testing is a must
Due to the limited hardware architecture of mobile devices, success and failure of application in market depends on performance of the app
Testing to be performed on different networks like 3G, 4G, Wi-Fi and Edge, etc
When you factor in different OS versions, unique hardware models, and carrier versions, the permutations are in the thousands
Tools Specialized knowledge of tools specific to mobile testing
Lack of Awareness There are many mobile tools and test approaches available in market, but all are not mature
14 XBOSoft, Inc. All Rights Reserved.
Internal Cloud
Definitions of Cloud What do we mean by:
PUBLIC CLOUD
A mobile lab hosted by the testing tool provider. All cloud clients have access to the devices in the lab. A specific tester reserves the device he needs and uses it when its available. Accessible through the internet.
PRIVATE CLOUD
A mobile lab hosted by the testing tool provider dedicated to a specific client. The tester can reserve a device without having to share the devices with other clients. Accessible through the internet.
ON-PREMISE CLOUD
A secure mobile lab set up by the client itself. All the devices are connected to a server inside the company network. The company has complete control over what devices are procured and who uses them.
External Cloud
15 XBOSoft, Inc. All Rights Reserved.
Shared Advantages
• Enables better collaboration between Dev/QA
• Both provide very robust lab/device
management, supporting provisioning and
maintenance
• Resource management. Both provide a
mechanism for scheduling/reserving devices
• Improve efficiency of test execution
• Enable continuous integration for DevOps
16 XBOSoft, Inc. All Rights Reserved.
Using an External Cloud
Pros:
• Scalability - easy access to hundreds of devices on demand
• No need to maintain devices (provider does it)
• Newer devices added to the pool at no extra cost
• Flexible for more advanced solutions (ex. GPS)
Cons:
• Tester session performance tends to be slower, especially with higher
bandwidth apps like video
• Some security concerns –Testing done outside the firewall, while data
is transferred back and forth through the internet
• Different companies use the same devices with
your app and potentially your private test data
17 XBOSoft, Inc. All Rights Reserved.
Using an On-premise Cloud
Types:
• Commercial administration via cart/hub
• DIY lab
Pros:
• Higher performance real-time user sessions
• Increased security over devices and test data
• Allows users to pull devices from lab for hands-on tests
• DIY is least expensive
Cons:
• Limited scalability
• Maintaining devices requires a lab manager
• Procuring/retiring devices can be challenging
• DIY is most difficult to maintain and physically secure
18 XBOSoft, Inc. All Rights Reserved.
Emulators Benefits:
• Most cost effective
• Least infrastructure setup
• Flexible and Scalable
• Support for most test plans
Constraints:
• Cannot execute Usability, Performance, & Security test cases on emulators
• Network and hardware interrupt testing cannot be done on emulators
Mobile Testing Comparison
Physical Devices in Lab Benefits: Covers all test types Safest option Should be used for most
common device types Best for UAT / Usability Constraints: Very costly to support across
all OS, device, and version combinations
Procurement and setup of new devices can delay testing
Physical Devices in Cloud Benefits: Cost effective Fastest start up Flexible and Scalable Can cover approx 90% of test
plan Constraints: Latency during usage and
longer test execution time Lower level of security
excludes this as an option for some clients
XBOSoft, Inc. All Rights Reserved.
Defining Lab Requirements
XBOSoft, Inc. All Rights Reserved. 20
Use Real Mobile Devices the degree to which a test system matches a production system, including hardware, firmware, devices, software, appliances, networks, etc.
TEST INTEGRITY
Emulators/Simulators are tempting due to ease of use and low cost, however:
Proven to cause significant defect leakage (stats & peers)
Lowest level of test integrity Certain functions cannot be replicated
21 XBOSoft, Inc. All Rights Reserved.
Other Device Lab Considerations
LAB SETUP WILL DEPEND ON:
Test objectives on coverage
Manual vs. automation Testing types Security Budget Scale Location of
Teams
22 XBOSoft, Inc. All Rights Reserved.
POLL Approximately how many unique models of mobile
devices exist today?
A. 5,000 B. 10,000 C. 15,000 D. 20,000 E. 35,000
Assumptions: • We are actually looking for approximate total # of permutations • Therefore a device with the same model number, but on a different carrier,
counts as more than 1 “model” or instance • Total active devices in use today globally, including feature phones, tablets,
wearables, but excluding IoT/embedded (cars, kiosks, appliances)
XBOSoft, Inc. All Rights Reserved. 23
Test Coverage
Device type (handheld, tablet, kiosk)
Screen size & resolution – dozens just in the top 80%
Device models – 20,000+ unique variations exist!
Operating system – 27 variations in the top 85%
Connection type (USB, WiFi, Bluetooth, Cellular)
Mobile Web vs. App
Variance in carrier installations – 750 carriers globally
Even using 80/20 rule, you are left with thousands of permutations! So what do we do?
24 XBOSoft, Inc. All Rights Reserved.
Use Analytics to Understand Your Users
Implicit Analytics: Platform usage and user behavior
Omniture Google Analytics
Flurry TestFlight (monitor behavior)
Mixpanel (native vs. mobile web)
Localytics
25 XBOSoft, Inc. All Rights Reserved.
Use Analytics to Understand Your Users
Xyo
Applause (external, sentiment)
Appsfire
Meltwater HootSuite
Explicit Analytics: What your users say
26 XBOSoft, Inc. All Rights Reserved.
Applause Sample Dashboard
27
Customer Feedback: Great place to find those hidden test scenarios! 1. Social Media (Facebook, LinkedIn Groups, Twitter, etc.) 2. App reviews 3. App ratings / comments 4. Comments, forum posts and articles on the web 5. Complaints and support requests 6. Trending analytics (ex via Applause)
Ensuring Proper Test Coverage – Rounding it Out
XBOSoft, Inc. All Rights Reserved.
Lab Definition Exercise
XBOSoft, Inc. All Rights Reserved. 29
First step – review analytics The data says our customers are: • 55% on iOS
– 62% iPhone 6, 15% iPhone 6+, 23% iPad
• 43% on Android – 45% on Samsung S5, 20% on S6, 20% Note 5, 15%
Tab S (tablet)
• 2% on Other (Windows/Blackberry)
Lab Definition Exercise
XBOSoft, Inc. All Rights Reserved. 30
Additional considerations: • 8 screen sizes/resolutions in top 80% • 12 OS versions in top 80% • 4 major U.S. carriers • 1,000 functional test cases
Lab Definition Exercise
XBOSoft, Inc. All Rights Reserved. 31
Tier iOS Android Other Carriers Screens Tests Execution
Total 6 6 2 4 8 1,000 N/A
Tier 1 4 4 2 3 5 1,000 All tests
Tier 2 3 3 0 2 3 750 Mission Critical
Tier 3 2 2 0 1 2 400 Light regression
Tier 4 1 1 1 1 1 100 Exploratory/ Smoke
Test Coverage Sample By Phase
XBOSoft, Inc. All Rights Reserved. 32
Source: Perfecto Mobile
Test Coverage Sample By Index
XBOSoft, Inc. All Rights Reserved. 33
Source: Perfecto Mobile
Test Coverage Sample By OS
XBOSoft, Inc. All Rights Reserved. 34
Source: Perfecto Mobile
Building a World Class Device Lab
XBOSoft, Inc. All Rights Reserved. 35
Overview of Device Lab Components
Desktop Users U.S. or India
Device Management Interface (View device inventory, reserve, etc.)
Real-time Device Viewer (launch sessions)
Remote Control protocol (access to launch apps, perform gestures, text, etc.)
Device interface (USB hub, Desktop/Server, Proprietary Appliance)
Real Devices, On-premise or Cloud
36 XBOSoft, Inc. All Rights Reserved.
U.S. Test Team
Firewall
Example On-premise Cloud setup
India Test Team
Mobile Devices on-premise Cloud
Device access via desktop tool
Wi-Fi Access Point
Carrier Network
Device access via desktop tool
Mobile Devices on-premise Cloud
Wi-Fi Access Point
Carrier Network
Firewall
XBOSoft, Inc. All Rights Reserved. 37
U.S. Test Team
Firewall
Example Public Cloud setup
India Test Team
Mobile Devices Off-site Cloud B Device access
via desktop tool Wi-Fi Access Point
Carrier Network
Device access via desktop tool
Mobile Devices Off-site Cloud A
Wi-Fi Access Point
Carrier Network
Firewall
XBOSoft, Inc. All Rights Reserved. 38
Mobile Device Lab Recap
XBOSoft, Inc. All Rights Reserved. 39
• Define mobile for your organization • Create thorough test objectives • Weigh all factors when defining lab
requirements • Decide on Cloud vs. On-premise • Define test coverage by leveraging analytics • Monitor channels for continuous feedback • Use tier-based approach to optimize coverage
Post your questions on Twitter and we'll answer them @XBOSoft Join us to keep updated on all our webinars, reports and whitepapers: facebook.com/xbosoft +xbosoft linkedin.com/company/xbosoft We post regularly on our blog – check us out! http://xbosoft.com/software-quality-blog/ Download our free Whitepapers, available here: http://xbosoft.com/knowledge-center/ You will receive an email with information on today’s webinar slides and recording. Please email us with any further queries regarding our services or ideas for future webinars! Services@xbosoft.com
Q+A
www.xbosoft.com XBOSoft, Inc. All Rights Reserved.