EDITOR’S NOTE CAN HYPERVISOR-AWARE SYSTEMS IMPROVE VIRTUAL SERVER PERFORMANCE? VMWARE SHARED STORAGE ENVIRONMENT OFFERS UNIQUE CHALLENGES THE FUNDAMENTALS OF TROUBLESHOOTING VMWARE PERFORMANCE Tips and Tricks to Make VMware Storage More Efficient Configuring arrays for VMware is far different than working with physical servers. Learn how to troubleshoot storage performance problems with VMware and make the right choices when configuring arrays.
Transcript
EDITOR’S NOTE CAN HYPERVISOR-AWARE SYSTEMS IMPROVE VIRTUAL SERVER PERFORMANCE?
THE FUNDAMENTALS OF TROUBLESHOOTING VMWARE PERFORMANCE
Tips and Tricks to Make VMware Storage More EfficientConfiguring arrays for VMware is far different than working with physical servers. Learn how to troubleshoot storage performance problems with VMware and make the right choices when configuring arrays.
HOME
EDITOR’S NOTE
CAN HYPERVISOR-AWARE
SYSTEMS IMPROVE
VIRTUAL SERVER
PERFORMANCE?
VMWARE SHARED
STORAGE ENVIRONMENT
OFFERS UNIQUE
CHALLENGES
THE FUNDAMENTALS
OF TROUBLESHOOTING
VMWARE
PERFORMANCE
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 2
EDITOR’SNOTE
Get Up to Date (and Speed) With VMware
If you’ve worked in the storage industry long enough, you can remember two comput-ing worlds: the one before VMware and the one after VMware. We are now seated firmly in the after-VMware world, where the technology is likely to be a part of many storage infrastruc-ture decisions.
With all the benefits a virtualization proj-ect can bring to an organization (and there are many), storage managers know that unless application storage traffic is uniquely associ-ated with a particular server, VMware can place new challenges on a storage environment. And managing random I/O traffic patterns can come as a surprise to even the most efficient storage system.
For that reason, storage pros must continu-
ally educate themselves on the ways to design a VMware shared storage infrastructure so their original investment remains intact, and they can maximize the ROI associated with any VMware installation. How will flash help the project? What about traditional tiering? How will VM disk files be distrib-uted? These and many other questions need to be answered. As technology tools and trends continue to shape the VMware land-scape, it’s important to keep up to date. We’ve collected our most practical and concise tech-nical tips on the subject to help you. n
Ellen O’BrienAssociate Editorial Director
SearchVirtualStorage
HOME
EDITOR’S NOTE
CAN HYPERVISOR-AWARE
SYSTEMS IMPROVE
VIRTUAL SERVER
PERFORMANCE?
VMWARE SHARED
STORAGE ENVIRONMENT
OFFERS UNIQUE
CHALLENGES
THE FUNDAMENTALS
OF TROUBLESHOOTING
VMWARE
PERFORMANCE
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 3
BOOST VMWARE STORAGE
Can Hypervisor-Aware Systems Improve Virtual Server Performance?
As virtual servers are introduced into the data center and quickly proliferate, perfor-mance issues with storage systems designed for physical server requirements start to crop up. While physical servers maintain a 1:1 rela-tionship to storage with fairly predictable performance characteristics and a low degree of concurrency, virtualized servers throw that “simple world” into a frenzy of physical servers hosting virtual machines and VMs running a mix of applications with varying workloads.
A high degree of concurrency where VMs and applications are in continuous pursuit of resources, including storage, pushes storage systems to their performance limits and affects the ability of storage administrators to corre-late applications to underlying physical storage resources and components.
To make matters worse, advanced storage techniques such as thin provisioning aggra-vate the problem by enabling multiple VMs
to share storage blocks, requiring the storage system to support an even higher number of IOPS. Addressing these performance challenges effectively is one of the value propositions of hypervisor-aware storage systems, which employ various techniques to achieve accept-able virtual server performance.
n Flash. Without question, solid-state stor-age is one of the key components that enable storage systems to better support the perfor-mance requirements of virtualized environ-ments. While NAND flash supports tens of thousands of IOPS, a single hard disk drive is limited to a few hundred IOPS; for disk-based arrays to scale IOPS, they depend on the paral-lelization of a large of number of disk drives, striping data across them, so that many disks are at work at any given time when data is served. NAND flash is superior to physical disk in its ability to support a large number of
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 4
BOOST VMWARE STORAGE
IOPS in almost all aspects, including cost and complexity.
Since the number of IOPS is typically more relevant than throughput for VM storage—but the type of applications on the VMs can affect this—employing solid-state storage reaps immediate performance benefits. A case in point that perfectly illustrates the relevance of IOPS is the well-known boot storm that occurs when many VMs that share the same physical blocks of the underlying storage all boot at the same time and spike the required IOPS. Flash storage is one of the most effective ways of addressing this.
That solid-state storage should be a com-ponent of hypervisor-aware storage is undis-puted, but where it’s deployed is a relevant consideration when evaluating hypervisor-aware storage systems. Substituting mechani-cal disk drives with solid-state drives (SSDs) is the easiest way of flash-enabling a storage array; the drawback of SSDs is that only data that resides on the SSDs benefits from it. Some hypervisor-aware storage systems deploy flash as cache, with the advantage that all applica-tions benefit from it.
The amount of flash—and whether it’s used for reads only or for reads and writes—is an important consideration. Flash can be deployed on the server, usually in the form of PCI Express cards; its proximity to the CPU and server memory, and reducing the number of transactions that have to traverse to the stor-age system through a relatively performance-constrained interface, makes flash on the server the best performing flash deployment option.
n Storage tiering. While the number of IOPS a storage system can support is very important, the cost of NAND flash is still significantly higher than that of physical disk. Since IOPS requirements fluctuate, a mechanism to sup-port spikes when needed but that’s also able to take advantage of less-expensive mechanical disks where appropriate can help keep the cost of hypervisor-aware storage systems down. This mechanism is available in the form of storage tiering, where mechanical disks and solid-state storage complement each other.
Two schools of thought exist on how to keep active data most effectively in NAND flash. One group of vendors champions policy-based
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 5
BOOST VMWARE STORAGE
data movement, while another espouses the use of NAND flash as cache that sits in front of mechanical disks. The latter is usually the less complex approach since NAND flash simply becomes another cache that sits between the DRAM cache and disk drives, relying on proven caching algorithms to keep active data in the appropriate tier.
n Quality of service (QoS). The ability to priori-tize data by application and data type enables
storage administrators to ensure that critical applications aren’t bogged down by less critical ones. The support of QoS is a critical element of a hypervisor-aware storage system, and it should be considered a must-have criterion on anyone’s evaluation list. QoS is the silver bullet that enables putting any type of data and work-load on a shared storage system with the needed safeguard to protect critical applications; QoS is the arbiter that keeps lower priority data and applications at bay. —Jacob N. Gsoedl
A VMware shared storage architecture offers a wide range of benefits, but also places new demands on your existing storage con-figuration. This article explains the major challenges and chief strategies for building shared storage architectures that are compat-ible with your virtual server infrastructure.
VMware has brought new levels of flex-ibility and efficiency into the data center. Not only can many physical servers be consolidated onto a single VMware host, but those virtual servers are transportable to other hosts, other storage systems and even other data centers. Furthermore, this can all be done transparently without applications experiencing any down-time. For the data center to fully leverage these capabilities, a shared storage architecture must be implemented alongside the virtual server infrastructure.
The shared storage infrastructure that accompanies a virtualization deployment needs
to accomplish several objectives:
n It must provide the shared connectivity that a cluster of virtualization hosts requires to execute functions such as virtual machine (VM) migration and off-site replication.
n The storage system must be able to handle the highly randomized I/O consolidated servers tend to generate.
n It must be cost-effective to purchase, scale and operate.
THE VMWARE STORAGE CHALLENGE
As mentioned above, VMware places unique demands on storage. No longer is application storage traffic uniquely associated with a par-ticular server. The application could be one of dozens of VMs within that server. This creates
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 7
VM STORAGE OPTIONS
a very random and continuous I/O traffic pat-tern that prior generations of storage systems did not have to contend with. This now means the storage system and the network it uses to connect to the physical hosts have added pres-sures on them that directly impact perfor-mance and operation.
These pressure points have caused the shared storage infrastructure to negatively impact the ROI of the virtualization project. As a result, IT planners are looking for new ways to drive down the cost of the storage investment while still meeting these new performance demands. Since it can directly impact the investment costs of the solution, one of the first consider-ations when designing a VMware shared stor-age infrastructure is to decide which protocol to use.
SELECTING A STORAGE PROTOCOL
Vendors have expended a lot of effort trying to convince users which protocol they should use for their VMware storage infrastructure. The typical protocol choices are Fibre Channel (FC), iSCSI or NAS via NFS. While each of these
protocols has its pros and cons, the primary motivation for many buyers is to reduce cost.
Fibre Channel is still, by far, the most domi-nant storage protocol in use in VMware envi-ronments. The latest study by VMware shows FC with a 70% or greater share of the market; however, most industry observers and users consider FC to be more complex and expensive than either of the Internet Protocol (IP)-based protocols. FC tends to win the selection pro-cess not because it delivers on cost savings, but due to its performance-tuning capabilities and universal support for FC among the vendor community.
The IP-based protocols such as iSCSI and NAS (NFS) are growing in popularity because of their cost advantage and potential ease of use. Since both of these protocols run on traditional Ethernet cabling, there is no need to learn a new protocol, implement a different cable design or buy new switches. A virtual LAN that routes only storage traffic can be easily created, and existing server Ethernet networking cards can be utilized.
One challenge is that both protocols have an overhead associated with them as they process
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 8
VM STORAGE OPTIONS
IP traffic. ISCSI has an added latency disadvan-tage because IP traffic needs to be converted to SCSI. The reality is that many data centers do not have performance requirements where this overhead would be a potential problem. However, as data centers continue to grow and VM densities are pushed higher because of ever-increasing processing power, the overhead required by IP-based protocols may become an issue.
Many of these overhead problems can be dealt with by adding network interface cards (NICs) with iSCSI and/or IP offload capabilities. There are also IP NICs that can be installed in servers to provide quality-of-service capabili-ties to each individual VM on that host. This essentially ensures bandwidth to mission-critical VMs. Of course, adding these cards increases costs and complexity.
To sum it up, as the environment scales, so does the complexity of that environment, regardless of the protocol selected. IP-based protocols have a clear advantage when a virtu-alization project kicks off simply because they are already present in the environment. Quite often there is no need to abandon the protocol
as the project scales, but increased cost and complexity will be introduced. In fairness, this should be offset by the savings that a more densely packed VM architecture will deliver.
SSD VERSUS HARD DISK DRIVES
The next key decision is how to leverage solid-state drives (SSDs) in the environment. SSDs allow the storage infrastructure to better respond to the random I/O problem described earlier. These are not rotational devices, so they have direct access to the data and the number of simultaneous requests made to them does not negatively impact performance like it does with a hard disk-based system. Unfortunately, SSD performance comes at a cost premium, so they are typically used sparingly as a caching or tiering strategy.
LOCAL SSD VERSUS SHARED SSD
Almost every virtual environment can benefit from a number of SSDs. The question to ask is, “Where should solid-state storage be lever-aged?” Typically, there are two options: as part
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 9
VM STORAGE OPTIONS
of the shared storage system or installed locally on the servers hosting the VMs.
The location of the SSD is also an architec-tural decision. Where SSD is used can impact how the storage network is designed and what type of storage controller is used. If local SSD is used, it is often combined with caching soft-ware that caches read traffic, essentially elimi-nating 50% or more of the storage I/O traffic that needs to traverse the storage network.
Running the caching application on the local host has its downsides. It consumes CPU resources and local caching may create prob-lems with VM mobility. Second, each host may not effectively use all the available SSD capac-ity. Finally, the caching software has to ensure
eviction occurs prior to the VM being moved.Running SSD on the shared storage device
may require a faster network and a more pow-erful storage controller to fully exploit the per-formance potential of memory-based storage.
Effective VMware storage architecture design plays a large part in a successful virtualized server implementation. Almost every data cen-ter, no matter the size, will eventually encoun-ter storage performance bottlenecks in their virtual server deployments. Indeed, the design of the architecture will have a significant bear-ing on VM and hypervisor host density, and on the number of hypervisors that can be sup-ported by the storage infrastructure itself. —George Crump
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 10
VMWARE PERFORMANCE
TIPS
The Fundamentals of Troubleshooting VMware Performance
A prerequisite to troubleshooting VMware storage performance is confirming whether storage or its infrastructure is the problem. While there are many sophisticated tools available to monitor the virtual envi-ronment, a simple and free way to make this determination is to monitor host CPU and virtual machine (VM) CPU utilization over time. Essentially, you want to know what the utilization of the CPU resource is when the performance problem is most noticeable. If the utilization is above 65%, it’s more than likely the performance problem can be best solved by upgrading the host, allocating more CPU resource to that particular VM or moving the VM to another host.
A simple way to rule out a CPU-related performance issue is to migrate the VM to a more powerful host with more memory, if pos-sible. Assuming the alternate host is on the same shared storage infrastructure, a repeat in
performance loss on a second host certainly begins to make storage performance a top can-didate for resolving the issue.
One of the prime benefits virtualization offers is its role in isolating performance prob-lems. In the past, moving an application to another host meant acquiring server hardware, installing the operating system and application, and then migrating users. With virtualization, a simple vMotion can provide a lot of informa-tion in the troubleshooting process.
TARGETING THE STORAGE NETWORK
Once a performance problem has been better isolated to the storage infrastructure, the next step is to determine where it’s occurring in that infrastructure. Conventional wisdom (and storage vendors) says to “throw hardware” at the problem and buy more disk drives, solid-state drives (SSDs) or a more powerful storage
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 11
VMWARE PERFORMANCE
TIPS
controller. While a faster storage device may be in order, IT planners should first look at the storage network between the VMware hosts and the storage system. If a network problem exists, it doesn’t matter how fast the storage devices in the system are.
A simple way to determine a network per-formance issue is to look at disk performance. Assuming CPU utilization is low, a storage device performance issue should show a rela-tively steady state of IOPS, which means disk I/O has hit a wall. Occasional high spikes or sporadic spikes in disk I/O performance means the device and storage system have perfor-mance to spare, but data isn’t getting to them fast enough. In other words, there is a problem in the network.
IT professionals tend to focus on overall bandwidth as the biggest area of contention in the storage network—for example, when mov-ing from a 1 Gigabit Ethernet (GbE) environ-ment to 10 GbE, or from 4 Gb Fibre Channel (FC) to 8 Gb FC. While an increase in band-width can improve performance, it’s not always the main culprit. Other problem areas, like the quality and capabilities of the network card or
the configuration of the network switch, should also be considered at the outset. Resolving issues at these levels is often far less expensive.
Network interface cards (NICs), whether they’re FC- or Internet Protocol-based, are typically shared across multiple VMs within a host. Even multi-port cards are typically aggre-gated and shared. If a particular VM has a per-
formance problem, dedicating that VM to its own port on a card—or even its own card—may be all that’s needed to resolve the performance problem. If the decision is made to upgrade the NIC to a faster speed, look for cards where specific VM traffic can be isolated or provided a certain quality of service.
You can also upgrade the NIC without upgrading the rest of the network. While it may seem counterintuitive, placing a 16 Gb
Once a performance problem has been better isolated to the storage infrastructure, the next step is to determine where it’s occurring in that infrastructure.
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 12
VMWARE PERFORMANCE
TIPS
FC card into an 8 Gb FC network does two things: It lays the foundation for faster storage infrastructures, and it improves performance even over the old cabling. This is because the processing capabilities of the interface card become more robust with each generation. To move data into and out of a NIC requires pro-cessing power, so the faster this can occur, the better the performance of that card.
SWITCHES CAN GET OVERWHELMED
The second area of the storage network to explore is the switch. Just like a card, a switch can be overwhelmed by the amount of traffic it has to handle; many switches on the market weren’t even designed for a 100% I/O load. For example, some switch designers may have counted on some connections not needing full bandwidth at all times. So while a switch may have 48 ports, it can’t sustain full bandwidth to all ports at the same time. In fairness, in the pre-virtualization days, this was a safe practice. In the modern virtualized infrastructure, how-ever, the thought of idle physical hosts is no longer practical.
Another common problem in switch configu-ration is inter-switch links. It’s not uncommon as switch infrastructures get upgraded to find inter-switch connections hard set to their prior network speed. This configuration error essen-tially locks switch performance to its older performance level.
LOOKING FOR TROUBLE
IN THE STORAGE CONTROLLER
If disk performance measurements show a rela-tively steady state and CPU utilization is low, then it’s more than likely that there’s a problem with the storage system. Again, most tuning efforts tend to focus on the storage device, but the storage controller should be ruled out first. The modern storage controller is responsible for getting data into and out of the system, providing features like snapshots and managing RAID. In addition, some systems now perform even more sophisticated activities, such as data tiering between SSDs and hard disk drives (HDDs).
There are two parts of the storage con-troller that must be ruled out: the network
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 13
VMWARE PERFORMANCE
TIPS
interconnect between the controller and the drives, and the processing resource. Most storage systems will provide a GUI interface that will display the relevant statistics. It’s important to monitor them during the problem period to determine if either one of these are the source of the problem. In the past, these two resources were seldom a concern, but in a virtualized data center, it’s not uncom-mon. Also, if and when SSDs are installed in the storage system, it’s important to recheck those resources to ensure they’re not blocking the SSD from reaching its full potential.
ANALYZING THE STORAGE DEVICE
After all this triage is done, the storage device can finally be analyzed. It’s important to note that most storage tuning efforts start here, when in actuality this is where they should end. Having a fast storage device without an opti-mized infrastructure is a waste of resources. That said, the above modifications (host CPU, storage network and storage controller), while
not optimal, will often prove to be acceptable. The easiest way to confirm a disk I/O perfor-mance problem is when your measurement tool shows a consistent performance result. For example, if IOPS is consistently reporting in the same range while CPU and network utiliza-tion are low.
The fix for device-based performance prob-lems is typically to add additional drives or to migrate to SSD. In the modern era, a move to SSD is almost always more beneficial, provid-ing a better performance improvement for less expense. However, before shifting to more or faster drives, IT professionals should also look at how the VM disk files are distributed. Too many on a single volume can be problematic; moving them to different HDD volumes can help. In the end, SSD should also solve the problem.
Tuning the VMware environment is a step-by-step process. Before you upgrade to high-storage devices, you should go through the above process to ensure your VMware environ-ment will see the maximum benefit from your investment. —George Crump
TIPS AND TRICKS TO MAKE VMWARE STORAGE MORE EFFICIENT 14
ABOUT THE
AUTHORS
JACOB N. GSOEDL is a freelance writer and a corporate director for business systems. He can be reached at [email protected].
GEORGE CRUMP is a longtime contributor to TechTarget, as well as president and founder of Storage Switzerland LLC, an IT analyst firm focused on the storage and virtu-alization segments. Before founding Storage Switzerland, George was chief technology officer in charge of technol-ogy testing, integration and product selection at one of the largest data storage integrators in the U.S.
Tips and Tricks for Making VMware Storage More Efficient is a SearchVirtualStorage.com e-publication.
Rich Castagna | VP of Editorial/Storage Media Group
About TechTarget: TechTarget publishes media for information technology professionals. More than 100 focused websites enable quick access to a deep store of news, advice and analysis about the technologies, products and pro-cesses crucial to your job. Our live and virtual events give you direct access to independent expert commentary and advice. At IT Knowledge Exchange, our social community, you can get advice and share solutions with peers and experts.
