Storage Lunch
Cédric MégrozArchitect
22.11.2012
Why IP StorageIP storage NFS iSCSI SMB3
iSCSI conceptiSCSI architecture
iSCSI namehba or software iscsi
iSCSI dependant independant hardwareiSCSI Configuration
Binding or not binding that is the questionStacked or not stacked that is the question
Masking on the array with iqnMultipathing, Policy & Pluggable Storage Architecture
VMFS 5Miscellaneous
ALUAJumbo Frame
Delay AckEquallogic
Equallogic configurationsHost Software for VMware
Host Software for WindowsQuestions ?
Agenda
Why IP Storage
• Low cost infrastructure with 1 GB Ethernet• No switch and special connectivity to
implement it
• For small infrastructure with less 50 VMs
• BUT not so easy to implement (a lot of best practice to find perf and throughput)
IP Storage NFS iSCSI SMB3
• Two storage family
• Block (SCSI, iSCSI, FC, FCoE, AoE)
• NAS Network Attached storage ( NFS, SMB3 => Hyper-V)
Storage with VMware
Storagetechnology
Datastoretypes
FCoE iSCSIFibre
Channeldirect
attached
filesystem
NAS
NFSVMware vSphere® VMFS
VMware vSphere® ESXi™ hosts
iSCSI – What is it?
• iSCSI specifies a way to “encapsulate” SCSI commands in a TCP/IP network connection:
Explains how to extractSCSI commands and data
Provides information necessary toguarantee delivery
Contains “routing” informationso the message can find itsway through the network
SCSI commands and dataiSCSI Header
TCPHeader
IP Header
GigabitEthernet
Provides Link Addressing, large frame sizes, and flow control
iSCSI concept
• Internet Small Computer Systems Interface (iSCSI)
• Physical network isolation (dedicated sw) Logical network isolation (VLAN)
• Default Port Number: 3260• An optional CHAP Secret (e.g.,
"secretsarefun") do not use for masking• LUN (Logical unit number)
iSCSI architecture
ISCSI name
iSCSI target name:iqn.1992-08.com.mycompany:stor1-47cf3c25 oreui.fedcba9876543210iSCSI alias: stor1IP address: 192.168.36.101
iSCSI initiator name:iqn.1998-01.com.vmware:train1-64ad4c29 or eui.1234567890abcdefiSCSI alias: train1IP address: 192.168.36.88
LUN Masking
HBA or software iSCSI initiator
Network software iSCSI Configuration
• A VMkernel port must be created for ESXi to access software iSCSI.– The same port can be used to access NAS/NFS storage.
• To optimize your vSphere networking setup: Separate iSCSI networks from NAS/NFS networks.
• Physical separation is preferred.• If physical separation is not possible, use VLANs.
iSCSI Target-Discovery Methods
• Two discovery methods are supported:• Static• Dynamic (also known as SendTargets)
By default Dynamic
This functionality is used by Equallogic to redirect the host to the right target
192.168.36.101:3260
iSCSI Security: CHAP
• iSCSI initiators use CHAP for authentication purposes.• By default, CHAP is not
configured.• ESXi supports two types of
CHAP authentication:• Unidirectional • Bidirectional:
– Software iSCSI only
• CHAP is not masking !!!!
Software iSCSI properties > General tab
Target authenticates
host.
Host authenticates
target.
Multipathing with iSCSI
• Hardware iSCSI:
• Use two or more hardware iSCSI adapters.
• Software or dependent hardware iSCSI:
• Use multiple network interface cards (NICs).
• Connect each NIC to a separate VMkernel port.
• Associate VMkernel ports with iSCSI initiator.
Configuration iSCSI 1/4
• Create a dedicated vSwitch with 2 VMKernel and 2 VMNIC connected on the IP storage network
• If same subnet dedicate a VMNIC per VMKernel
• If not do nothing
Configuration iSCSI 2/4
• vmk_iscsi1 with VMNIC5
• vmk_iscsi2 with VMNIC4
• Idem for DVS
Configuration iSCSI 3/4
Configuration iSCSI 4/4
Binding or not binding VMKernel on iSCSI sw
Binding or not binding VMKernel on iSCSI sw
• Rescan times take longer than usual Incorrect • number of paths are seen per device • Unable to see any storage from the storage device• CISCO NEXUS with LACP impose this config. with AA array without ALUA
Stacked or not stacked that is the question
The architecture of the array can have two developement
Multi target: for each Ethernet port you have a iSCSI target => do not stack the switch like the FC word. (HDS or VNX)
Uni target: only one target is generated for all the Ethernet port => stack the switch to have a failover in case of lost connectivity (NetApp, Equallogic)
Dedicated network or not …
For small network yes it’s better
Think IOS update
With 10GB and Nexus… no choice
Generaly network infrastructure is stacked and it’s not a good thinks in case of lost of the master (old switch 15 sec without network)
Pluggable Storage Architecture
Multipathing, Policy
esxcli nmp satp setdefaultpsp -P VMW_SATP_ALUA -S VMW_PSP_RR for esxi 5 esxcli storage nmp satp set –P VMW_PSP_RR –s VMW_SATP_ALUA
What is Asymmetric Logical Unit (ALUA)?
ALUA Allows for paths to be profiled Active (optimized) Active (non-optimized)
Le VMFS 5
• VMFS-5:• Allows concurrent access to shared storage• Can be dynamically expanded• Uses a 1MB block size, good for storing large virtual disk files• Uses subblock addressing, good for storing small files:
– The subblock size is 8KB.• Provides on-disk, block-level locking
hosthost
VMFS datastore
VAAI
• vStorage APIs for Array Integration is a feature introduced ESX/ESXi 4.1 that provides hardware acceleration functionality.
• It enables your host to offload specific virtual machine and storage management operations to compliant storage hardware.
• Used for lock file, clone & copie, svMotion• NFS is supported with a ESXi agent• Need license Ent & Ent +
Datastore size
• Updating the metadata of VMFS requires a SCSI reserve and release
• 1. Starting and stopping a VM• 2. vMotioning a VM• 3. Using VMware snapshots• 4. Using vAPI Dataprotection (use VMware snapshots)
• . => 15 VMs / block datastore• VMware Best Practice 300 – 700 GB• But actually with VAAI => no limit• Use Large capacity datastores Avoid RDMs • NFS: 16TB • VMFS: vSphere 5 = 64TB • vSphere 4 = 2TB • Avoid extents
Miscellaneous
• Never route network IP storage !!! Warning with the loop !!!!
• Ask for compliant subnets with the Network team
• Use always the vCenter Plug-In to apply the best practice on the ESXi host
• Use always the vCenter Plug-In to create and mount the datastore
Jumbo Frame
• What is an Ethernet Jumbo Frame? Ethernet frames with more than 1500 bytes of payload (9000 is common – FCoE is 2240)
• Commonly ‘thought of’ as having better performance • Should I use Jumbo Frames?
• Adds complexity & performance gains (while existent) are relatively marginal with common block sizes
• Stick with the defaults when you can
Delayed Ack mandatory (EMC et HDS hyper-V)
TCP delayed acknowledgment is a technique used by some implementations of the Transmission Control Protocol in an effort to improve network performance. In essence, several ACK responses may be combined together into a single response, reducing protocol overhead. However, in some circumstances, the technique can reduce application performance.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002598
http://support.microsoft.com/kb/981482
http://www.axiomdynamics.com/it-tweaks-and-tricks/iSCSI-best-practices/html/iSCSI-Best-Practices-VMware-EMC-Microsoft.html
Advanced network configurations
• Enable Rapid spanning-tree (portfast)• Enable flow control on all ports in the switch and array• Disable port storm control (broadcast- unicast)• Enable jumbo frames if possible
Alignment is Optimal I/O
Misalignment of filesystems results in additional work on storage controller to satisfy IO request
Affects every protocol, and every storage array VMFS & NFS Datastores
VMDKs & RDMs with NTFS, EXT3, LUN, etc
Filesystems exist in the datastore and VMDK
Create VMFS with GUI not vCLIWindows NT, 2000, 2003, & XP (use diskpart to set to 1MB) Linux (use fdisk expert mode and align on 2048 = 1MB)
Fixing Misalignment
Guest OS filesystem is misaligned
• Step 1: Take an array snapshot/backup • Step 2: Use offline tools to realign EMC
UBERAlign (open, works with all, scheduler, and in-guest reclaim) vSphere Migrator
Alternate: Use online tool to align NetApp Migrate & Optimize (VSC feature)
SIOC NIOC SDRS
• QoS on storage and network dVS
• NIOC offert QoS compatible 802.1p for Virtual machine traffic, Management traffic, iSCSI traffic, NFS traffic, Fault-tolerant traffic, VMware vMotion traffic, User-defined traffic, vSphere replication traffic
• SIOC use queue depth to manage contention. SIOC monitors the latency of I/Os to datastores at each ESX host sharing that device. When the average normalized datastore latency exceeds a set threshold (30ms by default), the datastore is considered to be congested, and SIOC kicks in to distribute the available storage resources to virtual machines in proportion to their shares.
• Need license Ent+ …
iSCSI Storage array
iSCSI Storage array
Switch stacked
Multi target Jumbo Frame 10GB ALUA
EMC VNX no yes yes yes yesNetApp depend Need different
subnet yes yes FC onlyHDS HUS no yes 1GB ? yes AAHDS AMS no yes 1GB no Only AMS
2500AA
Equallogic yes no yes yes -
Equallogic
• Can be combined with other EqualLogic PS Series arrays in the same SAN group, online. Up to 16 total members per group
• Automatic RAID 5, RAID 6, RAID 10 and RAID 50
• 1 mgmt target per SAN Group but each stack has these own tragets.
• The mgmt target redirect to the target with the LUN
• Need to use de MPIO of Dell Equallogic for Windows
Equallogic configurations
• There are three load balancers that operate within a pool:
• The NLB (Network Load Balancer) manages the assignment of individual iSCSI connections to Ethernet ports on the pool members
• The CLB (Capacity Load Balancer) manages the utilization of the disk capacity in the pool
• The APLB (Automatic Performance Load Balancer) manages the distribution of high I/O data within the pool.
Host Software for VMware
• Host Integration Tools for VMware® • Auto-Snapshot Manager/VMware Edition (ASM/VE): Integrates with
VMware VirtualCenter and PS Series snapshots to allow administrators to enable Smart Copy protection of VirtualCenter folders, datastores, and virtual machines.
• EqualLogic Datastore Manager: Allows administrators to create and manage datastores on EqualLogic PS Series arrays from within vCenter.
• Virtual Desktop Deployment Utility: Automates the deployment of virtual desktops in a VMware View™ environment.
• Storage Adapter for Site Recovery Manager (SRM): Allows SRM to understand and recognize PS Series replication for full SRM integration.
• Multipathing Extension Module for VMware® vSphere™: Provides connection awareness enhancements to the existing VMware multipathing functionality that understands PS Series network load balancing and facilitates host connections to PS Series volumes.
Host Software for Windows
• Host Integration Tools for Microsoft® • Remote Setup Wizard (RSW): Initializes new PS Series arrays, configures
host connections to PS Series SANs, and configures and manages multipathing.
• Multipath I/O Device Specific Module (MPIO DSM): Includes a connection awareness-module that understands PS Series network load balancing and facilitates host connections to PS Series volumes.
• VSS and VDS Provider Services: Allows 3rd party backup software vendors to perform off-host backups.
• Auto-Snapshot Manager/Microsoft Edition (ASM/ME): Provides point-in-time SAN protection of critical application data using PS Series snapshots, clones, and replicas of supported applications such as SQL Server®, Exchange Server®, Hyper-V™, and NTFS file shares.
• PowerShell Tools: Provides a comprehensive set of PowerShell cmdlets for managing one or many PS Series groups.
Questions ?
links
http://www.vmware.com/files/pdf/techpaper/vmware-multipathing-configuration-software-iSCSI-port-binding.pdf
http://blogs.vmware.com/vsphere/2011/08/vsphere-50-storage-features-part-12-iscsi-multipathing-enhancements.html
binding or nothttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2038869
PSAhttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011375
Equallogichttp://docs.danielkassner.com/dell/ISCSI_optimization_EQL.pdf
iSCSI best practicehttp://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002598
http://support.microsoft.com/kb/981482
http://www.axiomdynamics.com/it-tweaks-and-tricks/iSCSI-best-practices/html/iSCSI-Best-Practices-VMware-EMC-Microsoft.html
NetApp & EMC VMworld session Best Practicehttp://download3.vmware.com/vmworld/2012/top10/sto2980.pdf