Date post: | 12-Mar-2018 |
Category: |
Documents |
Upload: | phungquynh |
View: | 273 times |
Download: | 0 times |
PerformanceTesting of Mobile Applications
A White Paper
Architect Testing Practice
By Amit Mohanty
An end-to-end approach to MobileApplication Testing
2Point of View Paper on Performance Testing of Mobile Applications Mphasis
Contents
Introduction........................................................ 3Why Performance Testing?................................ 3The Importance of Performance Testing........................ 3How Performance Testing Helps.................................... 3
Performance Testing Challenges ofMobile Applications............................................ 3Key Parameters - Performance Testing ofMobile Applications............................................ 4Key Focus Areas - Performance Testing of Mobile Applications............................................ 4Performance Testing of Mobile Applications- Process Flow.................................................... 6End-User Perspective (FunctionalPerformance Testing)......................................... 6Overview......................................................................... 6
Benefits.......................................................................... 7Topology......................................................................... 7Device Perspective............................................ 7Overview........................................................................ 7Benefits.......................................................................... 7Topology......................................................................... 8Network Perspective......................................... 8Overview........................................................................ 8Benefits.......................................................................... 10Server Perspective............................................ 10Overview........................................................................ 10Topology......................................................................... 10Benefits.......................................................................... 11The Conclusion.................................................. 11
Introduction
Why Performance Testing?
Performance Testing Challenges of Mobile Applications
There are diverse conditions in which a mobile application is expected to perform at its best:
• A wide diversity in devices and platforms• The need for the best user experience• A variety of network connectivity options and • The ever-growing user base of smartphones
It’s no wonder that Performance Testing of mobile applications is supremely important.
At Mphasis, our Mobility Testing Center of Excellence has developeda specific approach to test various aspects of mobile application performance that covers end user, device network and server parameters.
The Importance of Performance Testing
Performance testing of mobile applications is very important because they are installed on mobile devices based on various mobile platforms/OSs like Android, iOS, Blackberry, Windows, etc. all of which have various hardware configurations like memory, processor, screen-resolution and having varying network connectivity. Yet these applications are expected to work well in all circumstances, no matter how much load there is on the server; whatever the strength of the network may be; and what the device configurations are.
How Performance Testing Helps
Performance testing helps to ensure that the system/application responds consistently within acceptable levels (as per baseline or SLA) and functions correctly under the current load condition or expected load conditions. Performance testing of mobile applications helps in bridging the gap between testing of a mobile application in a test environment with test data and a real world scenario, thereby contributing to business success. It gives confidence to launch an application in this rapidly changing mobile space with users always expecting the best features and a great user experience.
The challenges all deal with how the application performs in all these conditions:• Varying hardware configurations like battery, processor, memory, etc• Varying amounts of disk space• When the same action is performed multiple times• When the same action is performed for long period of time• On varying network conditions like bandwidth, network latency, packet
loss, etc.• When there is heavy load on the backend application server
3Point of View Paper on Performance Testing of Mobile Applications
Performance Testing helps to ensure that the system/application responds consistently within acceptable levels (per baseline or SLAs) and functions correctly under the current load condition or expected load conditions.
Mphasis
Per
form
ance
Tes
ting
of
Mo
bile
Ap
plic
atio
ns
End-UserPerspective
Device Perspective
NetworkPerspective
ServerPerspective
Functional Performance Testing or Testing of the Performance of the application
KPIs - Application Launch time, Navigation Response Time, Screen Transition time, Transaction time measured on the application etc.
Testing the Resource Utilization by the application
KPIs - CPU, Battery, Memory, Data Throughput
Testing the mobile application under varying network conditions
KPIs - Application features tested under varying network conditions like bandwidth, latency and packet loss and 2G and 3G network profiles
Testing the mobile application by loading the backend application server using device emulator or real device
KPIs- Transaction Time of the server by varying the bandwidth, latency and packet loss of the TCP/IP network
Key Parameters - Performance Testing of Mobile ApplicationsThe key performance parameters of mobile applications are as follows:
Figure 1: Key performance parameters of mobile applications
Key Focus Areas - Performance Testing of Mobile ApplicationsThe four main perspectives of Performance Testing of mobile applications and the key focus areas of each can be categorized as below:
Figure 2: Four main perspectives of Performance Testing of mobile applications and the focus areas
4Point of View Paper on Performance Testing of Mobile Applications
Application Response Time
Reliability
Configuration Sizing
Scalability
Transaction Response Time
Bottleneck Identification
Power Management
Resource Utilizations
Does the application respond quickly enough for the intended users?
How stable is the system under a heavy work load?
Which configuration provides the best performance level?
Is the application stable under expected and unexpected user loads?
How long does it take for the transaction to complete?
What is the cause of degradation in performance?
Does the application causes faster battery drain out?
How much CPU and Memory is consumed by theapplication?
Mphasis
Client
Android
ios
Server
End-User Perspective•• Navigation response time• Screen transition time• Transaction time measured
on the application
Application launch timeNetwork Perspective• Application features tested
under varying network conditions like bandwidth, latency and packet loss and 2G, 3G, network profiles.
Server Perspective• Transaction Time of the
server by varying the bandwidth, latency and packet loss of the TCL/IP network.
PerformanceTest Area
Transition Time
ApplicationLaunch Time
Load Time
PageComponentAnalysis
Response Time
KPI
Data inseconds
Data inseconds
Data inseconds
Data inseconds
Data inseconds
PerformanceTest Area
KPI PerformanceTest Area
KPI
Transition Time
ApplicationLaunch Time
Load Time
Pagecomponentanalysis
Response time
Data inseconds under2G/3G networkconditions
Data inseconds under2G/3G networkconditions
Data inseconds under2G/3G networkconditions
Data inseconds under2G/3G networkconditions
Data inseconds under2G/3G networkconditions
BackendServerLoad
Bandwidth,Latency
TransactionResponseTime
Device Perspective• CPU• Memory• Battery• Data throughput
PerformanceTest Area
KPI
PowerManagement
CPU
Performance
Memory
Battey Usage
CPU Kernel
CPU Total
CPU User
Memory Cache
Memory Free
Memory Used
Performance Testing of Mobile Applications- End-To-End Approach
5Point of View Paper on Performance Testing of Mobile Applications
Figure 3: End-To-End Approach For Performance Testing Of Mobile Applications
Mphasis
Understand Performance testRequirements ,system Architecture,Workload & Performance Goals
Develop Performance Test Assets(Test Plan, Test Scripts,Monitoring Strategy etc…)
Execute Performance Tests
Analyze Results and MonitorMetric
Complete testing
PerformanceTuning
Collect Monitoringmetrics
Restartafter tuning
Criterianot met
PerformanceCriteria are met
Performance Testing of Mobile Applications - Process Flow
Figure 4: Process flow of performance testing of mobile applications
End-User Perspective (Functional Performance Testing)Overview
End-user perspective or functional Performance Testing deals with the performance of the application on a real device connected to a real network, when the backend application server is in a loaded state. All this can be achieved by the tester sitting at some geographic location and accessing mobile devices which are remotely located over the cloud. This helps in reducing the total cost of ownership (TCO) as the testing team does not have the overhead of maintaining the devices over the cloud or the server connecting to the device cloud. Since the behavior of the mobile application on a real device is checked when the server is in a loaded condition, similar to a real world environment, it gives a lot of confidence about the user experience in terms of performance before an application is launched for large scale usage.
To check the end-user perspective of the performance of a mobile application, the backend application server of the mobile application is loaded by a number of virtual users hitting the backend server in parallel by using HP Loadrunner. While the load on the backend server is being maintained at a specific level, a functional testing script with various transactions defined to capture the page load time, application launch time, screen transition time, etc. written in HP Unified Functional Testing is run simultaneously from within the HP Loadrunner Controller. Then, the result is observed in the HP Loadrunner Analyzer in terms of how many transactions passed a specific load condition, what time was taken for these transactions to complete, etc. These values are then analyzed against the expected values or the SLA.
6Point of View Paper on Performance Testing of Mobile Applications
End-user perspective or functional performance testing deals with the performance of the application on a real device connected to a real network, when the backend application server is in a loaded state.
Mphasis
Remote Mobile CloudWeb Interface
SoftwaresRequired
Test Teams
Virtual User Load Applied To Server
Remote Mobile Cloud
Central NodeServer
Remote Mobile Cloud DeviceRacks
Devices with Operator SIMs onnetwork
Test Management
Global Delivery Centers Enterprise Client Location
UFT UFT Mobile
LoadrunnerQuality
Center/ ALM
Enterprise Server
Benefits
It simulates a network condition or scenario very closely resembling the real world scenario.
It provides greater confidence to launch the application for large scale usage due to the close resemblance of the test environment with a real world environment.
Since the load generation script and the functional script with various transactions run simultaneously, it helps the in-depth analysis of the reports and uncovers potential performance issues.
It reduces the total cost of ownership for the testing team due to the usage of cloud based device access.
Topology
Figure 5: End-user perspective
Device PerspectiveOverview
The device perspective of Performance Testing of mobile applications helps in gauging the device vitals while an application is. This kind of Performance Testing helps in checking how the mobile application performs under the limited hardware resources of a mobile device and check CPU usage, memory consumption, battery consumption, data throughput, etc. when the application is in use. This also helps in testing a mobile application under stressed memory conditions like low memory and low battery, thus ensuring a great user application.
Benefits
Using any standard web browser, a user can remotely activate a handset located anywhere worldwide.
The solution allows collecting all vital information about the device like memory usage, battery level, battery voltage, CPU usage, etc.
Certain tools installed on mobile devices help accurately collect information about the battery consumption and radio interface.
7Point of View Paper on Performance Testing of Mobile Applications
This kind ofPerformance Testing helps in checking how the mobile application performsunder the limited hardware resources of a mobile device and check CPUusage, memory consumption, battery consumption, data throughput, etc.when the application is in use.
Mphasis
Remote Test Lab
Internet
Single user performance tuning can be done using this analysis.
Topology
Figure 6: Device perspective
Network PerspectiveOverview
The network perspective of Performance Testing of mobile applications deals with testing the application performance on a real mobile device under varying network conditions like bandwidth, latency, packet-loss, etc. which is otherwise called the network profile. Since network profiles similar to 2G, 3G and 3.5G networks can be simulated using different approaches, it is closest to checking the performance of the mobile application in a real world scenario. It enables testers to check the application on various mobile devices in the simulated network condition while they sit remotely at some other location, with the help of mobile device cloud. Also, it gives a lot of confidence to mobile application developers to launch the application for large scale use due to its close resemblance to a real world scenario.
There are two approaches to the network perspective of Performance Testing of mobile applications:
Testing of single user mobile application performance using various network profiles remotely on a real device in the mobile cloud using the network virtualization appliance from Shunra
Testing of mobile application performance remotely on a real device connected to a real network on the mobile cloud while the load is being
8Point of View Paper on Performance Testing of Mobile Applications
The network perspective of Performance Testing of mobile applicationsdeals with testing the application performance on a real mobile deviceunder varying network conditions like bandwidth, latency, packet-loss, etc.which is otherwise called the network profile.
Mphasis
Remote Mobile CloudWeb Interface
Test Teams
Remote Mobile Cloud
Central NodeServer
Remote Mobile Cloud DeviceRacks
Devices with Operator SIMs onnetwork
Test Management
Global Delivery Centres Enterprise Client Location
UFT/NativeScripting
QualityCenter/ ALM
Enterprise Server
NetworkVirtualization
Shunra
applied to the backend application server using specific network profiles which simulate a real world load on the server
Topology of single user network perspective of mobile application Performance Testing
Figure 7: Network perspective- single user
In the case of single user Performance Testing of a mobile application by varying the network profiles, the Shunra appliance is connected to a single device on the cloud via Wi-Fi. So, the network connection of the mobile device can be configured by configuring the Shunra appliance with the help of an application available on the mobile device. While the mobile device is accessing the network with a specific network profile, the performance of the mobile application running on the devices is assessed.
Topology of network perspective of mobile application Performance Testing with loaded backend server
Figure 8: Network perspective- loaded backend server
9Point of View Paper on Performance Testing of Mobile Applications
Remote Mobile CloudWeb Interface
SoftwaresRequired
Test Teams
Virtual User Load Applied To Server
Remote Mobile Cloud
Central NodeServer
Remote Mobile Cloud DeviceRacks
Devices with Operator SIMs onnetwork
Test Management
Global Delivery Centers Enterprise Client Location
UFT UFT Mobile
LoadrunnerQuality
Center/ ALM
Enterprise Server
Shunra NV
Mphasis
Performance TestController
Real DevicesThe Mobile Cloud
Load Generators
Performance Diagnostics
Virtual Mobile Users- Multi user/multi location simulations
MobileApplication
Service
L0
L1
L2
L3
L4
L5
V1
V2
V3
V4
V5
In this kind of Performance Testing, virtual user load is applied on the backend application server using specified network profiles similar to a real world scenario using the Shunra NV Add-In on Loadrunner. While the load is being applied to the backend server, the application’s performance on a real device connected to a real network on the mobile device cloud is assessed with the help of UFT and UFT Add-In from the tester’s computer.
Benefits
Using Network Virtualization solution for real-world load and capacity testing
Virtualized network conditions – unlimited points of presence
Support for accurate simulation of users from any geographical locations/regions
Offer of bandwidth utilization per transaction summary
Server PerspectiveOverview
The server perspective of Performance Testing of mobile application deals with testing the backend server of the mobile application by applying varying the virtual user load and then extrapolating the data for the production server capacity. It gives confidence about the performance of the application at all times during a day/week/month whenever the load on the server varies. It also incorporates the capability through which bandwidth and latency can be varied to create a more life-like scenario. It is the traditional load testing of mobile applications backend server capabilities using Loadrunner.
Topology
Figure 9: Server perspective
10Point of View Paper on Performance Testing of Mobile Applications
Testing the backend server of the mobile application by applying varying the virtual user load.
Mphasis
Benefits
Emulates conditions of a controlled load on the server
Simulates real-world mobile network characteristics
Measures application and server performance under load
Checks where performance delays occur: network or application related delays; CPU performance; I/O delays; and monitors the network and server resources under load
Supports generating load through real devices with QTP integration
The ConclusionA growing number of mobile users are using their mobile devices for their day to day tasks including important transactions like money transfers, paying credit card bills, etc. Due to this, today’s mobile applications are very important and are expected to perform at their best at all times. The surging growth in the number of mobile users using high end mobile devices and relying on them for various kinds of transactions necessitates that all mobile applications are thoroughly tested for performance glitches before their launch. Failing to prepare for these highly variable conditions can lead to customer dissatisfaction and cause damage to the brand reputation.
This whitepaper aims at combining the various solutions and perspectives for Performance Testing of mobile applications and provides a guide to the overall process of Performance Testing.
11Point of View Paper on Performance Testing of Mobile Applications
EFFICIENCY/PERFORMANCE(INCLUDING NETWORK - RELATED)
FUNCTIONALITY
PORTABILITY
USER INTERFACE/EASE OF USE
COMPATIBILITY/ REGRESSION TESTINGSECURITY (PROTECTION OF SENSITIVEDATA OVER THE PHONE OR AIR)
CERTIFICATION OF APPLICATION
64%
48%46%
36%31%
18%14%
Mphasis
About AuthorAmit MohantyArchitect (Testing Practice)
Amit has more than 11 years of experience in Application Testing, especially in the Mobility space and spanning across multiple companies. He has played various roles in Testing, Automation, Application Branding, Requirement Management, Test Management and has led teams located across geographies. He has experience in Performance and Load Testing of mobile applications using leading tools in the market.
With an MS in Quality Management from BITS, Pilani and BE in Computer Science from Utkal University, Bhubaneswar, Amit is also Six Sigma Green Belt certified and ISTQB certified.
12Point of View Paper on Performance Testing of Mobile Applications Mphasis
About Mphasis
For more information, contact: [email protected]
To find out more about Mphasis’s ability to help your organization in thePerformance Testing of Mobile Applications space, please contact:
Amit MohantyArchitect - Testing [email protected]
USA460 Park Avenue SouthSuite #1101, New YorkNY 10016, USATel.: +1 212 686 6655
UK88 Wood StreetLondon EC2V 7RS, UKTel.: +44 20 85281000
UNITED STATES
CANADA
GREAT BRITAIN
FRANCE
IRELAND
NETHERLANDS
BELGIUM
GERMANY
SWITZERLAND
INDIA
JAPAN
CHINA
PHILIPPINES
SRI LANKA
AUSTRALIA
NEW ZEALAND
SINGAPORE
INDONESIA
POLAND
Delivery Footprint Client Footprint India Centre (BPO) Global Service Centre (ITO & APPS) Training AcademyOUR GLOBAL FOOTPRINT
Vadodara
Indore
Raipur
Chennai
Pondicherry
Pune
Mphasis (an HP Company) enables chosen customers to meet the demands of an evolving market place. Mphasis fuels this by combining superior human capital with cutting edge solutions in hyper-specialized areas. Contact with us on www.mphasis.com
INDIABagmane World Technology CentreMarathahalli Outer Ring RoadDoddanakhundi Village, MahadevapuraBangalore 560 048, India.Tel.: +91 080 3352 5000
Copyright Information
Mphasis and the Mphasis logo are registered trademarks of Mphasis Corporation. All other brand or product names are trademarks or registered marks of their respective owners.
Copyright © Mphasis Corporation. All rights reserved.
Bhubaneshwar
Bangalore
Mangalore
Mumbai