Vmware Io Analyzer Tutorial-V1.5

© 2010 VMware Inc. All rights reserved

VMware I/O Analyzer 1.5 —

I/O Benchmarking Virtual Appliance

VMware Inc.

2

Contents

What is VMware I/O Analyzer?

Deploying I/O Analyzer VM

I/O Analyzer Web UI

Workload Generator

• Use case 1: Single VM IOmeter Test

• Use case 2: Single VM Trace Replay Test

• Use case 3: Multi-VM and Multi-Host IOmeter Test

• Use case 4: Multi-VM and Multi-Host Trace Replay Test

• Use case 5: Listening Mode

Test Scheduler

Trace Repository

FAQ

3

I/O Analyzer Overview

4

I/O Analyzer: An Overview

Integrated framework for storage performance measurement

Includes standard workload generator such as IOmeter and I/O trace replay

Easily extensible framework

• Customizable and configurable I/O tests

• Multi-hosts and multi-VM configurations for various I/O workload scenarios

• Ability to accurately replay I/O traces of real-world applications

Configure

I/O tests

Log esxtop

results

Analyze

storage data

Identify

possible issue

Correct

Storage

Configuration

Traditional Approach (diverse I/O benchmarks, error –prone, manual process)

I/O Analyzer Fully automated

performance measurement

Integrated knowledge base with well-

known storage performance diagnostics

5

I/O Analyzer: Features

Integrated framework for storage performance testing

• Unified web interface for running tests and browsing performance stats

Readily deployable virtual appliance

• On-site diagnosis for well-known problems

Easy configuration and launch of I/O tests on one or more hosts

Built-in workload generators

Synthetic workloads through IOmeter

Real-world application workloads through I/O trace replay

Integrated performance results at both guest and host levels

Enables easy export of performance data for off-site analysis

6

I/O Analyzer 1.5: What‘s New

Test Scheduler

Detailed Storage/Network/CPU Statistics

Annotating Tests and Results

7

I/O Analyzer Deployment

8

Deploying I/O Analyzer: Typical Setup

One controller VM

Depending on your needs, you may deploy more worker VMs

VMFS

Controller

VM

Worker VM

Configure

diagnostic tests

Launch

Tests

Get

Results

Web

Interface

9

Deploying I/O Analyzer: Steps

1. Deploy I/O Analyzer Virtual Appliance

2. Login via vSphere Client

3. Perform Tests via Web UI

10

Unzip downloaded I/O Analyzer archive file (.zip file)

Login to the host via vSphere client

In ―File‖ menu, select ―Deploy OVF Template‖

Deploying I/O Analyzer VM 1/4

11

Select .ova file using ―Browse‖ and click ―Next‖

Review ―OVF Template Details‖ and click ―Next‖


12

Choose ―Name and Locations‖ and click ―Next‖

Choose ―Storage‖ to deploy a VM and click ―Next‖


Note that please do NOT use space or comma for the VM name.

Recommended name should contain only letters, numbers, dots, and

dashes. (The rule applies to worker VMs only.)

13

Select ―Disk Format‖ and click ―Next‖ : Thick provision Eager Zeroed is recommended

Review VM deployment settings and click ―Finish‖ to deploy the appliance


14

Select ―console‖ tab in vSphere client to login to I/O Analyzer VM

Username: root; password: vmware

You cannot access Web UI without appliance login

(Optional) If you need local terminal, click mouse right button and

choose ―xterm‖

Enabling Web UI

15

Find the IP address of the I/O Analyzer VM from vSphere client

―Summary‖ tab

Enter that IP address into a browser (Chrome or Firefox) to access

the user interface

Using Web UI

16

I/O Analyzer Web UI Part 1: Workload Generator

17

I/O Analyzer: Typical Work Flow

Configure and run IOmeter or Trace Replay tests through Web UI

Check unified host and guest results through the same Web UI

Optionally configure new IOmeter configurations or upload new traces to

the appliance for more testing

VMFS

Controller

VM

Worker VM

Configure

diagnostic tests

Launch

Tests

Get

Results

Web

Interface

18

Workload Generator Use Cases

1. Single VM IOmeter Test

2. Single VM Trace Replay Test

3. Multi-VM and Multi-Host IOmeter Test

4. Multi-VM and Multi-Host Trace Replay Test

5. Listening Mode

19

Single VM IOmeter Test 1/6

1. Open Web UI

2. Click “workload configuration” icon

Or alternately, click the large icon.

20


1. Enter ESX host IP and root password

and click “Add New Host”

2. Wait until VM list to be updated

Note that we do support using DNS hostname for ESX host. However,

you should make sure DNS service is working properly in the network

where you deploy I/O Analyzer VM.

Also note that if VM names contain space or comma, they will not be

processed correctly when displaying results.

21


1. Select a worker VM 2. Choose IOMeter

3. Select a workload

4. Add to list

22


1. Specify a name

2. Specify a duration

(recommended at

least 120 seconds) 3. Click “Run Now”

23


Wait until the status text disappears and

go to “Results” page to see test results.

Alternately, you may also go to “Test Scheduler” to

watch ongoing test status.

24

Click ―RESULTS‖ icon after the test is

complete.

Test names of all prior tests are listed

along with time of run. If test is not

listed, try after a few minutes

Select test name to review guest-level

(IOmeter) and host-level I/O

performance stats

Click ―Download Results‖ button to

download archive file for collected

ESXTOP and IOmeter result files

Graphs show ESXTOP results collected

every 20 seconds during the run.


25






5. Listening Mode

26

Single VM Trace Replay Test 1/6

1. Open Web UI



27










28

‗Device‘ is Linux block device

• By default, 100MB disk „sdb‟ comes with appliance

• Bigger disks can be added through vSphere Client

‗Replay Pace‘ is specified to speed-up or slow-down replay

• Values <1 means slow-down factor and values >1 means speed-up factor.

‗Disk Size Adjustment‘ is to specify how to deal with out-of-bound requests.

• „wrap‟ computes mod of disk offsets in trace to fall within disk boundary

• „ignore‟ just ignores requests with offsets outside the disk boundary


1. Select a worker VM 2. Choose Trace Replay

3. Select a trace

4. Add to list

29


1. Specify a name


(recommended at


30






31


complete.




Select test name to review host-level

I/O performance stats



ESXTOP




32






5. Listening Mode

33

Multi-VM and Multi-Host IOmeter Test 1/6

1. Open Web UI



34










Repeat this step to add all hosts that you need.

35


1. Select a worker VM 2. Choose IOMeter

3. Select a workload

4. Add to list

Repeat this step to add all worker VMs you need.

36


1. Specify a name


(recommended at


37






38


complete.




Select test name to review guest-level

(IOmeter) and host-level I/O

performance stats



ESXTOP and IOmeter result files




39






5. Listening Mode

40

Multi-VM and Multi-Host Trace Replay Test 1/6

1. Open Web UI



41










Repeat this step to add all hosts that you need.

42

‗Device‘ is Linux block device

• By default, 100MB disk „sdb‟ comes with appliance

• Bigger disks can be added through vSphere Client

‗Replay Pace‘ is specified to speed-up or slow-down replay

• Values <1 means slow-down factor and values >1 means speed-up factor.

‗Disk Size Adjustment‘ is to specify how to deal with out-of-bound requests.

• „wrap‟ computes mod of disk offsets in trace to fall within disk boundary

• „ignore‟ just ignores requests with offsets outside the disk boundary


1. Select a worker VM 2. Choose Trace Replay

3. Select a trace

4. Add to list

Repeat this step to add all worker VMs you need.

43


1. Specify a name


(recommended at


44






45


complete.








ESXTOP




46






5. Listening Mode

47

Listening Mode – Concept

In listening mode, I/O Analyzer does not invoke any workload

generator automatically.

It simply starts performance stats collection on all added hosts.

It typically works with your own workload generator and is used as

a performance stats collection and visualization tool.

48

Listening Mode 1/5

1. Open Web UI



49

Listening Mode 2/5







Repeat this step to add all hosts that you plan to collect performance

stats from.

50

Listening Mode 3/5

1. Specify a name


(recommended at


When no workload entry is added, it defaults

to “listening” mode.

51

Listening Mode 4/5





52


complete.








ESXTOP



Listening Mode 5/5

53

I/O Analyzer Web UI Part 2: Test Scheduler

54

Test Scheduler

A new component in I/O Analyzer 1.5

Schedule multiple tests to run

Scheduler starts scheduled tests sequentially based on their given

time of run

55

Step 1: Creating Workload Configurations

To make use of the scheduler, you need pre-defined configurations

In ―Workload Configuration‖ page, give the config a name and click

―Save as a New Config‖ when workload composition is done.

Repeat the steps to compose all desired workload configurations.

56

Step 2: Scheduling a Test

1. Click “Scheduler” icon

2. Select a config 3. Specify time of run


(recommended at

least 120 seconds)

5. Click “Schedule Test”

Repeat this step to schedule as many tests as you plan.

To schedule a test to run as early as

possible, click “Get Current Server Time”.

If more than one tests are scheduled to run at the same time, the one

that is scheduled first will run first.

57

Step 3: Test Manager

In test manager, you can see the list of running, pending, and complete

tests.

For running test, the real starting time and the test progress will be shown.

For pending tests, the scheduled time of run will be shown.

• You may cancel a pending test in test manager.

For complete tests, the real starting time will be shown and you may delete

it from the list.

To see the results of complete tests, go to ―Results‖ page.

58

I/O Analyzer Web UI Part 3: Trace Repository

59

Trace Upload

I/O Analyzer comes with some template I/O trace files.

You may collect vScsi traces based on your needs. Please refer to

―Collecting vScsi Trace‖ section in the ―HELP‖ page.

Once you collect your own trace, click ―Upload vScsi Trace‖ to go

to the upload page.

60

Upload from Remote Machines

You may upload from any remote machines that allow SSH connections.

1. Give trace a name: letters, numbers, dashes, and dots only.

2. Provide remote access credentials (hostname, username, and password).

3. Provide the absolute path to where the trace file is stored on the remote machine

(where SCP can retrieve the file).

4. Click ―Upload Remote File‖.

61

Upload from Local Disk

You may also upload the trace file from local disk.

1. Give trace a name: letters, numbers, dashes, and dots only.

2. Click ―Choose File‖ to select a local file.

3. Click ―Upload Local File‖.

62

Trace Characteristics

After uploading a new trace, it will be automatically profiled and

you may see its characteristics in the ―Trace Characteristics‖ page.

63

Summary

Integrated framework for storage performance measurement in

multi-host settings

Supports synthetic I/O workloads through IOmeter and I/O trace

replay of real-world applications

Easily extensible framework

• Customizable and configurable I/O tests

• Extensible I/O test configurations

• Supports upload, characterization and replay of new traces.

Questions and Feedback:

• Mailing list: [email protected]

• Community:

http://communities.vmware.com/community/vmtn/server/performance/ioanalyzer

mailto:[email protected]





http://communities.vmware.com/community/vmtn/server/performance/ioanalyzer

64

FAQ

1. Workload Configuration FAQ

2. Test Scheduler FAQ

3. Test Results FAQ

4. Trace Repository FAQ

5. Advanced Usage FAQ

65

FAQ Part 1: Workload Configuration

66

Can I mix IOmeter tests with Trace Replay tests?

No, we do not support mixing different types of test in the same

configuration. However, you might be able to simulate this by

using multiple controller VMs. Alternately, you may also do it by

running workload generators manually and use listening mode to

collect stats.

67

Can a single VM be composed with multiple workload?

No, each VM in a configuration can only associate with at most one

workload entry.

68

What happen if test duration is longer than a trace?

If, for example, the trace is only 300 seconds and you specify a test

duration of 500 seconds, in the last 200 seconds, trace replayer will

be completely silent. If you need to run a trace in a loop, please

schedule multiple tests with a trace of a proper duration.

69

Can I add a host that has no workers?

Yes, if a host is added but no workers are running on top of it, the

backend will simply run listening mode on that particular host.

70

Test is not running after clicking ―run now‖?

Mostly the problem is because the host credentials (IP or root

password) have changed after you added it to the list.

When a test is scheduled to run, the backend will re-validate all

host credentials. If your host credentials are out-of-date and you

end up having no valid host, scheduler will reject the test.

To fix this problem, please delete the host and re-add it.

Note that re-adding hosts in general does not require re-adding all

workload entries. The only exception is if some VMs have new IP

addresses, then all stale workload entries have to be manually

updated.

71

VMs are not on the list?

Those VMs that are not powered on or do not have a valid IP

address will not be displayed.

If a VM powers on or obtains a valid IP address after you add the

corresponding host to the list, please delete the host and re-add it.

Note that re-adding hosts in general does not require re-adding all

workload entries. The only exception is if some VMs have new IP

addresses, then all stale workload entries have to be manually

updated.

72

VM IP address is inaccurate?

If your VM‘s IP address has changed after you added the host, web

UI will not know. If you are aware of the IP address changes, please

delete the host and re-add it. You will also have to manually all

stale workload entries in all configurations.

Often times, it happens in the way that you are not aware. For

example, if you are scheduling IOmeter or trace replay tests, but do

not see any workload generated in the result page, it is likely that

worker VMs have had their IP addresses changed after you

compose the configuration. Please fix the configuration and rerun

tests.

73

FAQ Part 2: Test Scheduler

74

Can I abort an ongoing test?

Due to the support of multiple distributed workers, there is no

simple way to abort ongoing tests. If you do find a need to do so,

please refer to ―aborting an ongoing test‖ section in the ―HELP‖

page.

Please note that by aborting, the scheduler will lose its states and

thus all pending tests will be lost. You might also lose any

intermediate stats. Also, the immediate following test might as a

result contain some stale stats.

75

Test manager says ―IOmeter guestSummary timeout. Stats will be

omitted for this test.‖ What does it mean?

If you see this status, that means at least one of the worker VMs

that should run IOmeter do not generate an iometer.csv output.

Most likely this is because IOmeter is not correctly invoked on the

worker VM.

Please open the ―Console‖ tab in VI Client of the worker VM and

see whether IOmeter window is open and IOmeter is stuck.

• If you see an EULA term popped up, please read it. If you accept the term,

click agree so that IOmeter can proceed.

• If you have accepted EULA term and IOmeter is running without a valid ICF

spec (i.e., you see an IOmeter window but it is running any spec), please

contact [email protected] for a solution.

• If these are all NOT the case, please make sure the worker VM is reachable by

the controller. You should try to SSH from the controller to make sure it works.






76

Test is stuck in ―xxxxx‖ state for a long time?

The scheduler is equipped with several timeout mechanism. If it

does stuck in a state for an unreasonable long time, please refer to

―aborting an ongoing test‖ section in the ―HELP‖ page.

If the problem persists, please contact io-analyzer-

[email protected].






77

FAQ Part 3: Test Results

78

No workload is generated?

If you are running IOmeter, please note that you have to accept the

Intel EULA the first time you run IOmeter on each worker VM.

Without accepting the term, IOmeter will not proceed but the

controller does not know. (Note that for trace replay tests, the

EULA will not pop up.)

If you have accepted the EULA term but still do not see any

workload generated in the result page, it is likely that worker VMs

have had their IP addresses changed after you compose the

configuration. Please fix the configuration and rerun tests.

Note that this can happen that you schedule a number of tests and

in between tests, some of your worker VMs are assigned a new IP

address via DHCP. In such a case, the controller VM will not be

able to communicate with the worker VM.

79

Why IOmeter did not run?

If you have accepted IOmeter EULA and you have checked that

DHCP did NOT reassign your workers new IP addresses, but

IOmeter still did not run at workers, this most likely means there

are networking issues between the controller and the problematic

worker.

Please ping from the controller to the worker and make sure the

response time (RTT) is under1.5 seconds.

• If RTT is above 1.5 seconds, it can happen that either SCP might timeout

when transferring IOmeter config file or IOmeter starts before the config file

transfer finishes. In either case, IOmeter will not run. Since 1.5 seconds is

uncommonly long, please fix any potential network problems before using I/O

Analyzer.

• If networking problem cannot be addressed, please consider placing the

controller VM and all workers geographically close to each other (say under

the same physical switch).

80

Blank results?

If in the result page, you cannot even see host stats, it is likely that

the backend was unable to connect to any of the hosts. Please

login to the appliance and check /var/www/ioa.log to find out the

root cause.

Another possibility is if you run I/O Analyzer for more than 2 hours,

JVM might exhaust the entire heap. In our lab environment, we

have tested I/O Analyzer for more than 5 hours without running out

of memory. However, we recommend keeping tests under 2 hours.

To find out whether this is the case, please check /var/log/daemon

and look for out of memory exception.

If you do need an extremely long test that I/O Analyzer keeps

failing, you will have to increase tomcat server‘s heap size. Please

contact [email protected] if you need help.






81

I see some charts have gaps. What happened?

I/O Analyzer pulls performance data during a test through

networks. If for any reason a host becomes unreachable during the

test, I/O Analyzer will keep retrying. During the period of retrying,

no samples will be collected and thus you might see a flat line or a

series of zeros until I/O Analyzer recovers from the error.

82

Why do my time-series charts seem to misalign with workload generator?

The stats collection and workload generator are synchronized by a

fixed delay. We would recommend running a test longer to even

out the misalignment during the warm-up and cool-down periods.

If exact timing is a definite requirement in your scenarios, please

contact [email protected] for more information

regarding fine-tuning synchronization delay.






83

Why are some charts missing in the results page?

We use scripts at the backend to generate charts. The scripts

cannot work properly with any VM names that include space or

comma. For those VMs, you won‘t be able to see automatic charts.

However, you can still get their raw time-series data by

downloading the tarball.

84

Some VMs/Devices/Adapters are missing?

If some VMs are missing, it is likely that during the whole test

period, it was totally powered off.

For other counter groups, it is likely that they are ―inactive.‖

In order to be determined as ―active‖

• A disk device or disk adapter must have an average IOPS of at least 0.1.

• A network port must have an average PktTx/Sec of at least 0.1.

If these thresholds are too high/low for you, please consult io-

[email protected] for the mean of configuring these

thresholds.






85

What does ―XXX‖ counter mean?

I/O Analyzer collects performance stats via the same infrastructure

that ESXTOP uses. All counters have the same meaning as in

ESXTOP. Please refer to ESXTOP documentation for the official

definition of those counters.

For storage related counters, the following knowledge base post

might be useful

• http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd

=displayKC&externalId=1008205

For general ESXTOP counters, the following whitepaper might be

useful

• http://communities.vmware.com/docs/DOC-9279

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008205



http://communities.vmware.com/docs/DOC-9279



86

FAQ Part 4: Trace Repository

87

Uploaded trace is not profiled correctly?

The backend trace parsing script accepts only CSV format. Please

make sure you follow the steps specified in the ―Collecting vScsi

Trace‖ section in the ―HELP‖ page. If the format is not compatible,

the scripts might fail generating properly charts.

If ―Trace Characteristics‖ page shows any broken trace files,

please remove it by doing the follows:

1. Login to the appliance

2. Go to “/var/www/traces”

3. Find the problematic trace folder and delete the entire folder

88

FAQ Part 5: Advanced Usage

89

Can multiple users operate the same controller simultaneously?

No. I/O Analyzer does not have a strict synchronization at the

backend. Multi-user scenarios are not supported at the moment.

90

Can I change I/O Analyzer Appliance‘s password?

Changing I/O Analyzer appliance password can only be done via

shell login.

Note that all worker VMs have to change to the same password so

that backend script at the controller is able to communicate with all

workers.

After changing password at all workers, on the controller side,

please edit /root/ioAnalyzerScripts/copy_from_server.pl and

/root/ioAnalyzerScripts/copy_to_server.pl so that they contain the

new password.

91

All the IOs seem to hit the cache. What should I do?

By default, I/O Analyzer comes with a 100MB virtual disk, which is

under /dev/sdb. With such a small disk, it is likely that all disk I/Os

will hit the cache. To test the non-cache performance, please follow

these steps:

1. Shutdown I/O Analyzer appliance (the worker VM)

2. Open VI Client and go to “Summary” tab of the worker VM

3. Click “Edit Settings”

4. Click “Hard Disk 2” and “Remove” it

5. Add another virtual disk with your desired size (and select “Thick Provision

Eager Zeroed”)

6. Power on the worker VM

92

Can I import test results from other I/O Analyzer Appliance?

If you have multiple I/O Analyzer VMs and would like to store all

results on the same one, you may simply transfer all folders under

/var/www/expts to the I/O Analyzer VM you like.

Note that I/O Analyzer 1.5 recognizes all results generated by I/O

Analyzer 1.1+. However, those counters that are not parsed/stored

in previous versions will not show up by importing this way.

Currently, I/O Analyzer does not support importing results that are

not generated by I/O Analyzer.

93

Can I mount a separate disk to store all test results?

Yes. Please mount your desired disk/partition to /var/www/expts at

the controller. The backend server reparses all folders under this

path every time you open the result page.

94

Why are there more than one esxtop files for each host?

Starting I/O Analyzer 1.5, we use a different way to pull

performance stats. This new approach will export all available

counters. If during the run-time, such counters are changed (say

some new VMs are powered on), the backend thread will create a

new esxtop output file to include newly added counters.

95

How to interpret those timeseries* files in the tarball?

For disk related counters (adapter, device, or VM)

• *-iops

• TIMESTAMP\tREADS/s\tWRITES/s\tCMDS/s

• *-throughput

• TIMESTAMP\tMBREADS\tMBWRITES\tMBTOTAL

• *-latency

• For adapter and device

• TIMESTAMP\tDAVG\tKAVG\tGAVG\tQAVG

• For VM

• TIMESTAMP\tLatRd\tLatWr

For network counters (timerseries-network-*)

• *-droprate

• TIMESTAMP\tDropTx\tDropRx

• *-pkps

• TIMESTAMP\tPkpsTx\tPkpsRx

• *-throughput

• TIMESTAMP\tMbpsTx\tMbpsRx

For CPU related counters (esxtop-pcpuutil, esxtop-pcpuused, esxtop-coreutil)

• TIMESTAMP\t#0\t#1\t…\t#N-1

• N is the total number of logical CPUs/cores on that host

Note that the TIMESTAMP is the relative timestamp, in which zero means the starting time instant of the test.

96

Why can‘t ESXPLOT import the generated esxtop*.csv?

Starting I/O Analyzer 1.5, we use a different way of pulling

performance counters. The output CSV files follow the same format

while exporting much more counters than ESXTOP batch output.

Because of these additional counters, ESXPLOT might not

interpret some of them correctly. To fix this problem in ESXPLOT,

please replace IsColZero function in esxplot.py as follows: • def IsColZero(self,index):

"“” Return true if all of the data is essentially zero

"“” for value in self.columns[index]: floatValue = 0.0 try: floatValue = float(value) except ValueError: continue if floatValue != 0.0: return False

return True

Note that you will have to install Python 2.6+, wxPython, and

numPy in order to run the modified esxplot.py.

Please also note that the esxtop output files generated by I/O

Analyzer 1.5+ do not usually follow the same exporting order as

esxtop/resxtop.

97

Is I/O Analyzer appliance safe to expose to public?

Due to several security reasons, we highly discourage exposing I/O

Analyzer appliance to public access.

Also, please be aware that all host credentials are stored in non-

encrypted way. Exposing I/O Analyzer appliance to public might

cause critical security threats to your enterprise networks.

Date post:	01-Nov-2014
Category:	Documents
Upload:	nicholas-king
View:	326 times
Download:	4 times

Vmware Io Analyzer Tutorial-V1.5

Documents