User Guide V2 March 2014 - Perfecto

Mobile Performance Testing Using HP LoadRunner

User Guide

V2 March 2014

Copyright

Copyright 2014, Perfecto Mobile Ltd. All rights reserved.

The information in this document is subject to change without notice. No part of this document may be

reproduced, stored or transmitted in any form or by any means, electronic or mechanical, for any

purpose without the express written permission of: Perfecto Mobile Ltd.

Perfecto Mobile Ltd. assumes no liability for any damages incurred, directly or indirectly, from any

errors, omissions or discrepancies between the software and the information contained in this

document.

Mobile Performance User Guide: MobileCloud Automation and HP BSM

Page 2 of 24

Contents 1 Table of Figures ...................................................................................................................................... 4

2 Terms & Definitions ................................................................................................................................ 4

3 Introduction ............................................................................................................................................ 5

4 Setup Pre-Requisites .............................................................................................................................. 6

4.1 Required .......................................................................................................................................... 6

4.2 Optional ........................................................................................................................................... 6

5 Reference Documentation ..................................................................................................................... 7

5.1 Perfecto Mobile ............................................................................................................................... 7

5.2 HP .................................................................................................................................................... 7

6 Examine network conditions impact on mobile user experience .......................................................... 8

6.1 Create MobileCloud Automation Script .......................................................................................... 8

Getting Started with MobileCloud Automation: ................................................................................... 8

Tip: ........................................................................................................................................................ 8

6.2 Device vitals commands ................................................................................................................ 11

6.3 Network virtualization commands ................................................................................................ 11

6.3.1 Network virtualization start ................................................................................................... 11

6.3.2 Network virtualization update ............................................................................................... 12

6.3.3 Network virtualization stop .................................................................................................... 12

6.3.4 Suggested Profiles .................................................................................................................. 12

6.4 Viewing results using Shunra Analytics ......................................................................................... 12

6.4.1 Installing Shunra Analytics...................................................................................................... 13

Installing Wireshark®: ......................................................................................................................... 13

6.4.2 Viewing test results and performance recommendations using Shunra Analytics ............... 13

7 Examine server load impact on mobile user experience ..................................................................... 16

7.1 Creating a VuGen script from the network sniffer (PCAP) file to facilitate the backend server load

16

7.2 Create a LoadRunner Scenario ...................................................................................................... 17

7.3 Best practices configuring real device testing from inside LoadRunner ....................................... 19

8 Reporting .............................................................................................................................................. 20

9 Appendix A- Inside the VuGen template .............................................................................................. 21

9.1.1 Execute MobileCloud Automation Script ............................................................................... 21


Page 3 of 24

9.1.2 Check MobileCloud Automation Script Execution Status ...................................................... 22

9.1.3 Collect and Analyze MobileCloud Script run Report .............................................................. 23


Page 4 of 24

1 Table of Figures

Figure 1: Perfecto Mobile Performance Solution ......................................................................................... 5

Figure 2: Perfecto Mobile Performance solution architecture ..................................................................... 6

Figure 3: MobileCloud Automation Sample Script ..................................................................................... 10

Figure 4: Shunra Analytics welcome ........................................................................................................... 13

Figure 5: Shunra Analytics help Quickpage ................................................................................................. 14

Figure 6: Shunra Analytics high level analysis ............................................................................................. 14

Figure 7: Shunra Analytics optimization suggestions ................................................................................. 15

Figure 8: Shunra Analytics network traffic analysis .................................................................................... 15

Figure 20: HP LoadRunner Analysis - Transaction Summary in Summary Report ...................................... 20

Figure 21: HP LoadRunner Controller - Interactive Schedule Graph .......................................................... 20

Figure 11: Sample Script - Execute MobileCloud Automation .................................................................... 22

Figure 12: Sample Script - Check Execution Status of MobileCloud Automation Script ............................. 23

Figure 13: Sample Script - Collect and Analyze MobileCloud Automation Script ....................................... 24

2 Terms & Definitions

Term Description

HP VuGen HP Virtual User Generator

SaaS Software as a Service

KPI Key Performance Indicator, also known as transactions

SLA Service Level Agreement

CI Configuration Item

MCM Mobile Cloud Manager


Page 5 of 24

3 Introduction

This document details the setup process for implementing the Perfecto mobile performance product for

measuring the mobile user’s experience during impacted network and server load conditions.

The Perfecto MobileCloud provides real devices, connected via Wi-Fi or cellular carriers to the internet,

and spread in different locations. The Perfecto performance product enables real mobile device testing

on simulated network and server load conditions.

This solution enables users to measure response time and availability of key transactions inside a mobile

application, under simulated real-world conditions. This new product handles this complex and

challenging process composed of unpredictable network conditions and varying server loads, alongside

mobile device performance under such conditions.

Figure 1: Perfecto Mobile Performance Solution

Mobile performance testing measures the impact of…

Network conditions on the user experience

Application server load on the user experience


Page 6 of 24

Figure 2: Perfecto Mobile Performance solution architecture

4 Setup Pre-Requisites

Before getting started, be sure to complete the following pre-requisites list. All listed items are required,

and can be obtained from Perfecto Mobile Support.

4.1 Required

Acquired from Perfecto Mobile Support:

Perfecto Mobile Performance License

MobileCloud account, including required devices that are connected to the network

virtualization component via Wi-Fi

MobileCloud Automation License

Acquired from HP:

HP Performance Center and/or LoadRunner

HP Virtual User Generator (VuGen) installation

Download from HP BSM: BSM-Saas > Admin > EUM > Setup and maintenance >

Downloads; OR

Online from here

4.2 Optional

Packet capture analysis tools:

WireShark, Shunra Analytics, http://pcapperf.appspot.com

mailto:[email protected]?subject=Monitoring%20setup%20requirements

http://www8.hp.com/us/en/software-solutions/software.html?compURI=1169460#tab=TAB3

http://pcapperf.appspot.com/


Page 7 of 24

Tip: Open your HP BSM Software as a Service with Internet Explorer or Mozilla Firefox.

5 Reference Documentation

5.1 Perfecto Mobile

Help Knowledgebase

MobileCloud Automation guide

HTTP API guide

HP VuGen Sample Script and LoadRunner Template

Performance data sheet

Mobile Performance Best Practices

5.2 HP

HP LoadRunner user guide

Online from here

HP VuGen user guide here

http://help.perfectomobile.com/category/235/0/10/Guides-Documentation/Performance/

http://help.perfectomobile.com/category/12


http://help.perfectomobile.com/article/AA-02016



http://www8.hp.com/us/en/software-solutions/software.html?compURI=1169460#tab=TAB3

ftp://ftp.itrc.hp.com/applications/HPSoftware/ONLINE_HELP/LoadRunner11.50_User.pdf


Page 8 of 24

6 Examine network conditions impact on mobile user experience

This section includes the following components:

Creating a MobileCloud Automation script

Network conditions commands

Device Vitals commands

Viewing results using Shunra Analytics

6.1 Create MobileCloud Automation Script

To get started, create a MobileCloud Automation script, within the Perfecto MobileCloud.

Once your script is completed, follow the steps and figure 1 below to customize it to Mobile Monitoring.

Getting Started with MobileCloud Automation: Refer to the Getting Started Video and Automation User Guide.

Tip: It is recommended follow the ScriptOnce™1 methodology so your script is applicable for all the devices you are targeting.

Step 1. Start your script by doing some cleanup and proper launch.

It’s very important to clean browser cache/ remove application from resident memory

and ensure a consistent script starting point to ensure script stability and timers

accuracy.

In the case of a mobile application, the recommendation is to use the following

sequence:

- Close application

- Wait(5)

- Start application

In the case of a browser, it is recommended to clean the browser cache. It’s certainly

recommended to use the device generic browser (ex.: safari) as opposed to an

instrumented browser.

Step 2. Identify the KPIs to be measured.

These are the Key Performance indicators (KPIs) you are interested in monitoring.

Note: A KPI is a measurement of response time and availability of a user facing transaction. It can be composed of a single or multiple user action(s) In our example, the CNN page load time is our KPI.

Step 3. Measure each KPI

1 Perfecto Mobile's patented ScriptOnce™ technology allows the creation of device agnostic test objects. With this technology you can write a script and re-use it on multiple platforms and networks with minimal to no modifications to the script.




Page 9 of 24

Define a timer starting point for each KPI. Use the Timer start function one line after the

desired command line. The timer name can be any name. Do this for each KPI you want

to measure.

Step 4. Place a Text checkpoint following the timer and command being measured. Make sure

to ‘Try’ the likelihood of finding the object on the device UI, you want to get a value that’s 98%

or higher. To get accurate timer value, select the “advanced settings” checkbox and set the

“OCR-analysis” mode to “Automatic”. Repeat the “try” and see that the high likelihood is

maintained. It is also possible to use image checkpoints, however text checkpoints will be more

reliable and accurate.

Step 5. Terminate the timer on condition success

Place a Timer stop function within the condition On Success.

If the condition is successful, the timer value will be reported, facilitating the

response time of the aspect of the KPI.

If the condition fails, the timer will not be reported, thus facilitating the

availability aspect of the KPI.

Step 6. Insert a timer checkpoint

1. Set the timer ID to be the same as the one you defined in “timer start”

2. Set the upper bound to a high value (ex.: 1000 seconds)

3. Set the type of timer to be ‘UX’

Step 7. Exit the script on condition failure (Optional)

You may want to close the device(s) and exit the script by placing an Exit function within

the condition On failure.

Step 8. Add network virtualization commands

Network virtualization start

Network virtualization update

Network virtualization stop

Step 9. Add device vitals commands

Device vitals start

Device vitals stop

Device get vitals





Page 10 of 24

Figure 3: MobileCloud Automation Sample Script


Page 11 of 24

6.2 Device vitals commands

The below MobileCloud Automation functions drive the device vitals.

Device vitals start – Begin collection of device vitals for the mobile device. It is recommended to

leave the parameters field as-is (collect-all) and set the collection frequency to run at the highest

possible frequency.

Device vitals stop- this will stop the collection of the device vitals

Get Vitals- this will get an immediate reading of the vitals selected

The script execution report will have a tab labeled ‘performance’, where a CSV file will be available to

download, containing all device vitals.

At the time of writing this document not all commands are supported across all iOS/Android platforms.

Therefore it is recommended to set the error policy for the above commands to ‘catch’ so that they do

not fail the script in case a device is not supported.

6.3 Network virtualization commands

The below MobileCloud Automation functions drive the network conditions.

Network virtualization start – Start network virtualization for the device with the specified

conditions

Network virtualization update – Update network virtualization for the device with the specified

conditions; for any condition that is not specified, the system will continue to use the existing

value

Network virtualization stop – Stop network virtualization for the device and save PCAP file to the

repository

For network conditions testing, the device being used is connected to the Web via WiFi.

Use the above Network virtualization start and the Network virtualization update commands to activate

and update the network conditions. Without this activation there is no effect on the device

communication to the Web.

6.3.1 Network virtualization start

Name Mandatory Type Default Description

Device ID Handset DUT The device for this command.

Latency Optional Number 0 Latency applied on packets in the Network. Effective values are in the range of 0-8000 ms.

Packet loss Optional Number 0 Network packet loss. Effective values are in the range of 0-90%.

Bandwidth in Optional Number unlimited Limitation on the allowed bandwidth into the device in the network.


Page 12 of 24

Effective values are in the range of 3-100,000Kbps, or unlimited.

Bandwidth out Optional Number unlimited Limitation on the allowed bandwidth from the device in the network. Effective values are in the range of 3-100,000Kbps, or unlimited.

6.3.2 Network virtualization update



Latency Optional Number 0 Latency applied on packets in the Network. Effective values are in the range of 0-8000 ms.

Packet loss Optional Number 0 Network packet loss. Effective values are in the range of 0-90%.

Bandwidth in Optional Number unlimited Limitation on the allowed bandwidth into the device in the network. Effective values are in the range of 3-100,000Kbps, or unlimited.

Bandwidth out Optional Number unlimited Limitation on the allowed bandwidth from the device in the network. Effective values are in the range of 3-100,000Kbps, or unlimited.

6.3.3 Network virtualization stop



6.3.4 Suggested Profiles

Network Latency Packet loss Bandwidth in Bandwidth out

3G 75 ms 0% 780 Kbps 330 Kbps

Edge 200 ms 0% 100 Kbps 100 Kbps

LTE 40 ms 0% 10000 Kbps 7500 Kbps

DSL 25ms 0% 2000 Kbps 256 Kbps

100% Loss 0 ms 100% 10000 Kbps 10000 Kbps Very Bad Network 500 ms 10% 1000 Kbps 1000 Kbps

6.4 Viewing results using Shunra Analytics

View the results of your test using Shunra Analytics. Follow the below installation and viewing

procedures.


Page 13 of 24

6.4.1 Installing Shunra Analytics

Download Shunra Analytics from here.

Complete the procedure steps listed in the Licensing Shunra Analytics document.

Installing Wireshark®: Shunra requires that you install Wireshark® – a network protocol analyzer. Follow the installation wizard from within the Shunra Analytics installation wizard.

Visit http://www.wireshark.org for more information.

6.4.2 Viewing test results and performance recommendations using Shunra Analytics

The MobileCloud Automation test will generate a .shunra file once the test is completed. This file can be

downloaded and saved locally from the Perfecto Mobile report or the repository. Open Shunra Analytics

to view the performance results – using waterfall charts, HTTP traffic analysis and much more using this

tool. Below are some examples of performance results.

Figure 4: Shunra Analytics welcome

How to use: For additional details, refer to the Shunra Analytics user manual.

From your desktop, go to: Start > All programs > Shunra (folder) > Analytics (folder) > Shunra Analytics Quickpage

ftp://customerFTP:[email protected]/home/ftp/Customer/Download/SNV_Analytics_8.7/SNV_Analytics_8.7.0.67.zip


http://www.wireshark.org/


Page 14 of 24

Figure 5: Shunra Analytics help Quickpage

From the welcome screen, go to:

> help

Figure 6: Shunra Analytics high level analysis


Page 15 of 24

Figure 7: Shunra Analytics optimization suggestions

Figure 8: Shunra Analytics network traffic analysis


Page 16 of 24

7 Examine server load impact on mobile user experience

This section includes the following components:

Creating a VuGen script to simulate the server load

Creating a VuGen wrapper for the real mobile device

Best practices for configuring real device testing within LoadRunner

This use-case measures the impact of a server load on real mobile devices, leveraging the HP

Performance Center LoadRunner tool to execute the following:

Apply a significant number of virtual users on the backend

The script representing these virtual users may already exist or may be generated from a

previous recording of the device traffic as it communicates with the service.

Execute a real device test and measure KPIs, in parallel, on a sample set of devices

LoadRunner will present reporting for both approaches, as well as the script execution result for each

device from Perfecto Mobile.

7.1 Creating a VuGen script from the network sniffer (PCAP) file to facilitate the backend server load

This step is optional and only required if there is no existing backend load traffic script available; Use the

PCAP file generated in the previous section, and import it into VuGen.

Step 1. Open VuGen

Step 2. Go to File > New Script and Solution

Step 3. Select the Mobile Application - HTTP/HTML protocol

Step 4. Click Create

Step 5. Click Record to open the Recording Wizard

Step 6. Select Analyze Traffic to analyze the traffic using the PCAP file

Step 7. Browse to the PCAP file location to be used to generate a script. The PCAP file is

available by setting the extension of the .shunra file to .zip and unzipping it.

Step 8. Specify the IP address of the host, observed during the PCAP recording

WARNING: No file is generated if the IP address specified here does not match what is in the file. If you've forgotten what the IP addresses are, use a PCAP file reader utility (Wireshark) to view the file

Step 9. Click Finish to complete the process and generate the VuGen script

WARNING: In many cases the network traffic recorded is encrypted. Please refer to VuGen User Guide (here) for more instructions on this topic.



Page 17 of 24

7.2 Create a LoadRunner Scenario

The simplest way to move forward is to adopt the LoadRunner template from Perfecto and the base

VuGen template, available here.

You will notice in the LoadRunner scenario there are two groups, each of them based on a single real

device. The base script is called “Real_Device_Template” and it is meant to drive real devices in the

Perfecto cloud while load is applied on the backend.

You will want to leave the number of virtual users the way it is (1 virtual user) for each of them. You can

change the group name to reflect a logical name you want to call one of the devices in your test. You

may want to extend the duration of the test by clicking on “Duration” at the bottom-left side of the

screen. You then want to set the runtime parameters for your script. You do that by highlighting the

group row and doing a right-click, selecting “Run time Settings…”. Below are all the VuGen parameters:

Name Mandatory Description Example

PerfectoDeviceDUT The ID of the real device to be used

12345

PerfectoDeviceName

Custom name to precede the transaction names in the LoadRunner report

iPhone5

PerfectoScriptName Script name in Perfecto cloud Public:example\demo



Page 18 of 24

PerfectoUserID Your Perfecto cloud username. Usually this would be your email address

[email protected]

PerfectoPassword Your Perfecto Cloud Password. Note you can provide LoadRunner encrypted password or non-encrypted password

abcd

PerfectoCloud Your Perfecto Cloud URL Performance.perfectomobile.com

PerfectoMoreParameters

Optional Additional Parameters you may want to pass to the Perfecto script. Note that the Perfecto script has to have these parameters in the variables set as “runtime”. To read more on this topic please see here

&param.value1=t&param.value2=5

Repeat the same process for the 2nd device in your script. If you do not have a 2nd device to run

your script, delete the group.

Run your scenario once to ensure it works well.

Add to your scenario more groups based on the number of real devices you want to run in

parallel to the load. Ensure the script you are pointing to is the same “Real_Device_Template”

script.

Add to your scenario the script you created that emulates a mobile virtual user, as described

here

Add to your scenario a web load script. See here instructions how to create one.

Eventually you should have a complete scenario as shown below. Run your scenario, observe the

response time(s) for web, virtual mobile user and the various devices. For each Perfecto script execution

you can refer back to the execution center of your cloud to get the complete Perfecto report, .shunra

file and device vitals.

http://help.perfectomobile.com/article/AA-00921/107/Guides-Documentation/Automation/Interface-Guide/Script-Variables.html



Page 19 of 24

Once the scenario is complete, you can click on Results->analyze results to gain access to the

LoadRunner reports

7.3 Best practices configuring real device testing from inside LoadRunner

The following does not replace the LoadRunner manual. This is an explanation how to configure the real

device sampling while emulated load is running.

You may want to create several scripts that will each run on a different device while the emulated load

is in effect. Ensure that for each script:

Only one virtual user is applied at all times

For the duration of the load test, repeat the script execution on the real device


Page 20 of 24

8 Reporting

Refer to the HP LoadRunner user manual for instructions on viewing load session reports. If the load

session was completed successfully, you should be able to obtain KPIs from real devices that are

materially different from those extracted by the emulated load.

Refer to the reporting interface inside Perfecto portal to view specific real device script execution

report.

Figure 9: HP LoadRunner Analysis - Transaction Summary in Summary Report

Figure 10: HP LoadRunner Controller - Interactive Schedule Graph


Page 21 of 24

9 Appendix A- Inside the VuGen template

In the Perfecto Performance Testing solution, the role of the VuGen script is to order the execution of

the Perfecto script on the Perfecto mobile cloud (via Perfecto’s HTTP API). Essentially this is a wrapper

for the MobileCloud script, facilitating the communication between HP LoadRunner and the Perfecto

MobileCloud engine.

The VuGen script includes HTTP API operations that communicate with the MobileCloud.

This section describes all of the VuGen sample script components. These include:

1. Execute MobileCloud Automation Script:

Send an execution query to the MobileCloud MCM to execute the Automation script

2. Check MobileCloud Automation Script Execution Status:

Query the MobileCloud for the Automation script execution status – loop querying every 30

seconds until execution is complete

3. Collect and Analyze MobileCloud Script run Report:

Translate KPIs in MobileCloud report to HP BSM - Report KPI

Return MobileCloud HTML report to HP BSM in case of error - Report Snapshot OnError

9.1.1 Execute MobileCloud Automation Script

Using the target device, execute the desired script.



Page 22 of 24

Figure 11: Sample Script - Execute MobileCloud Automation

How to use: See the Start New Execution Perfecto Mobile Help KB article for more information on how to use this HTTP API operation.

9.1.2 Check MobileCloud Automation Script Execution Status

Now that the script is running, it is possible to loop through the execution status until it has successfully

completed or experienced a timeout.



Page 23 of 24

Figure 12: Sample Script - Check Execution Status of MobileCloud Automation Script

How to use: See the Get Execution Status Perfecto Mobile Help KB article for more information on how to use this HTTP API operation.

9.1.3 Collect and Analyze MobileCloud Script run Report

Once the MobileCloud Automation script has completed, it is possible to retrieve the script report in

XML format, extract the timer(s’) availability and value, and report this data to HP BSM.

In this sample script TimerA is the measured KPI.



Page 24 of 24

Figure 13: Sample Script - Collect and Analyze MobileCloud Automation Script

How to use: See the Download Execution Report Perfecto Mobile Help KB article for more information on how to use this HTTP API operation.


Date post:	17-Nov-2021
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

User Guide V2 March 2014 - Perfecto

Documents