Post on 11-Jan-2017
transcript
© Copyright IBM Corporation 2015
Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
sSY0333
IBM Spectrum Virtualize
Zoning - Best Practices 101
Chuck Laing
Senior technical Staff Member
IBM GTS
sSY0333
© Copyright IBM Corporation
2015
Please fill out an evaluation for sSY0333
2
Some great prizes to be won!
@ IBMtechU
sSY0333
© Copyright IBM Corporation
2015
Session objectives
• By the end of this session you should know:
– Zoning is one of the most important configuration for proper:
• Failover
• Redundancy
• Resiliency
• Performance
• Evolution:
– Understand best practice zoning for 4 port Spectrum Virtualize models
– Understand best practice zoning for 8 port models – CG8
– Understand best practice zoning for 12 port models –DH8
– Understand best practice zoning for the V9000 DH8 w/Flash900
– Understand best practice zoning for Dual Core Fabric to Spectrum Virtualize
3
sSY0333
© Copyright IBM Corporation
2015
IBM Spectrum Virtualize
Transformed Economics of Traditional SAN
4
Overview Highly scalable storage virtualization system providing common functionality,
management, and mobility across heterogeneous storage types Key Capabilities Pools storage from multiple systems Compresses data with high performance with Real-time Compression for
extraordinary efficiency Enables non-disruptive movement of data among storage systems Supports ultra high availability multi-site configurations Benefits Improves storage utilization up to 100% Supports up to 5x as much data in the same physical space Simplifies management of heterogeneous storage systems Enables rapid deployment of new storage technologies for greater ROI Improves application availability with virtually zero storage-related downtime
sSY0333
© Copyright IBM Corporation
2015
Traditional Applications New Generation Applications
Storage Managemen
t
Policy Automation
Analytics & Optimization
Snapshot & Replication
Management
Integration & API
Services
Data Protection
Spectrum Virtualize
Virtualized SAN Block
Spectrum Scale
Global File & Object
Flexibility to use IBM and non-IBM Servers & Storage or Cloud Services
Spectrum Accelerate
Hyperscale Block
IBM Storwize, XIV, DS8000, FlashSystem and Tape
Systems
Non-IBM storage, including commodity servers and media
Data Access
Storage and Data Control
Spectrum Control Spectrum
Protect
Self Service Storage
Spectrum Archive
Data Retention
and non-IBM clouds
IBM Comprehensive Software Defined Storage Capabilities IBM Spectrum Storage Solutions
5
sSY0333
SPECTRUM VIRTUALIZE
ZONING EVOLUTION
IBM Spectrum Virtualize Zoning - Best Practices 101
A Closer Look
© Copyright IBM Corporation 2014
6
sSY0333
© Copyright IBM Corporation
2015
Starting with ≤ CF8 Models
Physical to Logical evolution – 1 slot - 4 port nodes
7
sSY0333
© Copyright IBM Corporation
2015
Spectrum Virtualize CG8 with extra FC HBA
Physical to Logical evolution – - 2 slots - 8 port nodes
8
sSY0333
© Copyright IBM Corporation
2015
Spectrum Virtualize DH8 – 12 Ports
Physical to Logical evolution – 3 slots - 12 port nodes
Understand the physical slot and port positioning
• DH8 - Physical to Logical port numbering for “new builds”.
9
sSY0333
© Copyright IBM Corporation
2015
Spectrum Virtualize CG8 – 8 Ports / DH8 12 Ports
Evolution and Types of Zones
• Think of 4 types of zones per fabric (Best Practice port usage - masking)
– FA = Fabric A; FB = Fabric B
– SPx = Storage Port
– NxPx = Spectrum Virtualize Node x Port x
– … = include multiple remaining node ports the same way
Port Layout provided Keith Williams 10
sSY0333
© Copyright IBM Corporation
2015
V9000 Port Layout – 8/12 Gb Capable –
4 slots -12 port nodes
V9000 1 Building Block – 12 FC ports per node
I/O Group 0
Node 1
1 2
Slot 1
1
1
3 4 5 6
Slot 2
1
2
2
2
2
1
2
4
2
3 Physical
port
number 7 8
Slot 3
3
2
3
1
16G HBA
8G HBA
9 10 11 12
Slot 5
5
1
5
2
5
4
5
3 Logical
port with
wwpn #
embedded
Node 2
1 2
Slot 1
1
1
3 4 5 6
Slot 2
1
2
2
2
2
1
2
4
2
3
7 8
Slot 3
3
2
3
1
9 10 11 12
Slot 5
5
1
5
2
5
4
5
3
11 11
sSY0333
© Copyright IBM Corporation
2015
V9000 Port Layout – 16 Gb Capable – 4 slots -12 port nodes
12
V9000 1 Building Block – 8 FC ports per node
I/O Group 0
Node 1
1 2
Slot 1
1
1
3 4
Slot 2
1
2
2
2
2
1
3
2
3
1 Physical
port
number 5 6
Slot 3
5
2
5
1
16G HBA
7 8
Slot 5
Logical
port with
wwpn #
embedded
Node 2
1 2
Slot 1
3 4
Slot 2
5 6
Slot 3 7 8
Slot 5
1
1
1
2
2
2
2
1
3
2
3
1
5
2
5
1
12
sSY0333
EVOLUTION AND TYPES OF ZONES
IBM Spectrum Virtualize Zoning - Best Practices 101
Backend/Storage to Spectrum Virtualize Zoning
A Closer Look
13
sSY0333
© Copyright IBM Corporation
2015
Storage to SVC Zones – 4 Port Node
Evolution and Types of Zones
• Example of Back-end Storage to SVC Zoning (Cluster Style)
DS8K Right I/O Enclosures
Bay 1
1
3
0
1
3
1
1
3
2
1
3
3
C1R4
1
0
0
1
0
1
1
0
2
1
0
3
C0R2
Bay 3
3
3
0
3
3
1
3
3
2
3
3
3
C1R7
3
0
0
3
0
1
3
0
2
3
0
3
C0R3
Bay 5
5
3
0
5
3
1
5
3
2
5
3
3
C1R8
5
0
0
5
0
1
5
0
2
5
0
3
C0R4
Bay 7
7
3
0
7
3
1
7
3
2
7
3
3
C1R7
7
0
0
7
0
1
7
0
2
7
0
3
C0R3
DS8K Left I/O EnclosuresBay 0
0
0
0
0
0
1
0
0
2
0
0
3
C1L4
0
3
0
0
3
1
0
3
2
0
3
3
C0L2
Bay 2
2
0
0
2
0
1
2
0
2
2
0
3
C1L7
2
3
0
2
3
1
2
3
2
2
3
3
C0L3
Bay 4
4
0
0
4
0
1
4
0
2
4
0
3
C1L8
4
3
0
4
3
1
4
3
2
4
3
3
C0L4
Bay 6
6
0
0
6
0
1
6
0
2
6
0
3
C1L7
6
3
0
6
3
1
6
3
2
6
3
3
C0L3
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
Supported DS8K to SVC ZoningEither ports 1&3 and 2&4 should be
zoned to a fabric or ports as shown on the
next page, both configs are supported
Newer SVC nodes may contain
one HBA card with 4 ports
port1=11052ca port1=11052b7 port1=110529e port1=110528b
port2=12052ca port2=12052b7 port2=120529e port2=120528b
port3=13052ca port3=13052b7 port3=130529e port3=130528b
port4=14052ca port4=14052b7 port4=140529e port4=140528b
sSY0333
© Copyright IBM Corporation
2015
Storage to Spectrum Virtualize Zones – 8 Port Node
Evolution and Types of Zones
• Example excluding heartbeat and MM/GM ports (Cluster style – with masking)
SAN Fabric 1 SAN Fabric 2
15
sSY0333
© Copyright IBM Corporation
2015
New Storage Zoning Schema per Iogrp 12 Port Node
Evolution and Types of Zones – non cluster type
Making 1 zone per Node per Fabric
with the same 8 ports from a single
backend storage unit, will ensure
the max login count of 16 is not
exceeded
Production SAN Fabric
D
STG Zone-1
STG Zone-2
STG Zone-3
STG Zone-4
Production SAN Fabric
C
16
Spectrum Virtualize DH8– 12 FC ports per node
I/O Group 0
Node 1
1 2 3 4
Slot 1
5 6 7 8
Slot 2
Physical
port
number 9 10 11 12
Slot 5
Logical
port with
wwpn #
embedded Node 2
1 2 3 4
Slot 1
5 6 7 8
Slot 2 9 10 11 12
Slot 5
2
2
2
1
2
4
2
3
2
2
2
1
2
4
2
3 5
1
5
2
5
4
5
3
1
2
1
1
1
4
1
3
5
2
5
1
5
4
5
3
1
2
1
1
1
4
1
3
Host/STG
Rep /
Node – Node
16
sSY0333
© Copyright IBM Corporation
2015
New Storage Zoning Schema per Iogrp 12 Port Node
Evolution and Types of Zones – non cluster type
Making 1 zone per Node per Fabric
with the same 6 XIV ports from a
single backend storage unit, will
ensure the max login count of 16 is
not exceeded
Production SAN Fabric
D
STG Zone-1
STG Zone-2
STG Zone-3
STG Zone-4
Production SAN Fabric
C
17
Spectrum Virtualize DH8– 12 FC ports per node
I/O Group 0
Node 1
1 2 3 4
Slot 1
5 6 7 8
Slot 2
Physical
port
number 9 10 11 12
Slot 5
Logical
port with
wwpn #
embedded Node 2
1 2 3 4
Slot 1
5 6 7 8
Slot 2 9 10 11 12
Slot 5
2
2
2
1
2
4
2
3
2
2
2
1
2
4
2
3 5
1
5
2
5
4
5
3
1
2
1
1
1
4
1
3
5
2
5
1
5
4
5
3
1
2
1
1
1
4
1
3
Host/STG
Rep /
Node – Node
17
sSY0333
© Copyright IBM Corporation
2015
Back-end Storage to Spectrum Virtualize Zones
Storage Zone Type – How many Storage zones?
HBA1
P
1
P
2
HBA2
P
1
P
2
Back-end Storage
SAN Fabric 1 SAN Fabric 2
STG Zone-1
STG Zone-2
STG Zone-3
STG Zone-4
STG Zone-5
STG Zone-6
STG Zone-7
STG Zone-8
18
sSY0333
© Copyright IBM Corporation
2015
Definitions and Rules
Three (3) types of Port Masking
• There are three(3) types of Port masking to be aware of with the new code and DH8
• Host port masking (mkhost) • Default port masking, with 64 bits, used to be "1111" now it is "...111111111111". reading from
right to left...1 for each of the 12 ports. Leaving the default of 1 enables the flow of IO to pass through the node port, "0" means to block IO on the port. • Benefits
• zoning changes
• Less complexity for Vdisk IOgrp moves,
• Changes introduced in the SAN, or the hosts, with less complexity
• The MM/GM port mask (partner_fc_mask) • Allows only MM/GM replication traffic to flow through the enable ports,
• Note: Other traffic can flow here from other connected devices, but defeats the purpose of dedicating MM/GM traffic . We dedicate and use port 6&12 for MM/GM (partner_fc_port_mask) so the mask would be 100000100000
• The Intra-cluster communication port mask (local_fc_port_mask) • Also dedicated at 000001000010 for 12 ports meaning ports 2, & 7 are enabled with 1's and the
rest are blocked.
• Bottom-line: We have so much field experience behind these best practices, that this is what we recommend. The most common reoccurring issue is when LUNs are moved around between IOgrps or shared to clusters, the zoning gets redone, but most SME's forget about or don't understand the port mask and so don't change it and then the redundancy is lost when change is made, resulting in IO blocked and outages...this we made it a best practice to reduce the complexity of managing the LUN to host mapping.
19 19
sSY0333
© Copyright IBM Corporation
2015
Definitions and Rules
Calculating Port to Port Logins, not to exceed 16
• Regarding the calculation of the message login limit, here is how it
works:
– This limit applies to any single node X, as seen by another node Y.
– Only 16 port logins are allowed from one node to any other node in an SAN fabric
– New error code 888, alert 801 will be generated if 16 exceeded
– Must correct zoning or masking or a combination of both zoning and masking to clear this issue
– To check whether the limit is exceeded, count the number of distinct ways by which a port
on node X can log into a port on node Y. This must not exceed 16.
– Only ports that are permitted to carry cluster message traffic need be included in the
count.
– Any given two Nodes can see each other for messaging in 8 different ways (4 on each switch)
– Only the ports that can "see" each other (based upon switch zoning) need be included.
• Note : Port logins associated with host/storage ports, strictly speaking, can be disregarded
since they don't carry message traffic. For separate reasons however, Hursley development
recommends that the number of Spectrum Virtualize ports in any given single zone be limited
to no more than 16. Part of the reason for this is to avoid the overheads associated with a
large number of logins.
20 20
sSY0333 V9000 Recommended Options
Two example options : with masking
• Left side considering a 16Gbps capable environment
• Right side considering a 8Gbps capable environment as shown below:
21 21
sSY0333
© Copyright IBM Corporation
2015
V9000 Port Layout – 16 Gb Capable
22
Internal SAN Fabric
B
V9000 1 Building Block – 8 FC ports per node
I/O Group 0
Node 1
1 2
Slot 1
1
1
3 4
Slot 2
1
2
2
2
2
1
3
2
3
1 Physical
port
number 5 6
Slot 3
5
2
5
1
16G HBA
7 8
Slot 5
Logical
port with
wwpn #
embedded
Node 2
1 2
Slot 1
1
1
3 4
Slot 2
1
2
2
2
2
1
3
2
3
1
5 6
Slot 3
5
2
5
1
7 8
Slot 5
Production SAN Fabric
D
Production SAN Fabric
C
Internal Switch
B
Internal Switch
A
22
sSY0333
© Copyright IBM Corporation
2015
V9000 Port Layout – 8 &16 Gb Mix
Internal SAN Fabric
B
V9000 1 Building Block – 12 FC ports per node
I/O Group 0
Node 1
1 2
Slot 1
3 4 5 6
Slot 2
Physical
port
number 7 8
Slot 3
16G HBA
8G HBA
9 10 11 12
Slot 5
Logical
port with
wwpn #
embedded
Node 2
1 2
Slot 1
3 4 5 6
Slot 2
7 8
Slot 3
9 10 11 12
Slot 5
Internal Switch
A
Internal Switch
B
Production SAN Fabric
C
Production SAN Fabric
D
1
1
1
2
2
2
2
1
2
4
2
3
3
2
3
1 5
1
5
2
5
4
5
3
1
1
1
2
2
2
2
1
2
4
2
3
3
2
3
1 5
1
5
2
5
4
5
3
23 23
sSY0333
© Copyright IBM Corporation
2015
New Storage Zoning Schema per IOgrp
Making 1 zone per Node per Fabric
with the same 8 ports from a single
backend storage unit, will ensure
the max login count of 16 is not
exceeded
V9000 1 Building Block – 12 FC ports per node
I/O Group 0
Node 1
1 2
Slot 1
3 4 5 6
Slot 2
Physical
port
number 7 8
Slot 3
16G HBA
8G HBA
9 10 11 12
Slot 5
Logical
port with
wwpn #
embedded
Node 2
1 2
Slot 1
3 4 5 6
Slot 2
7 8
Slot 3
9 10 11 12
Slot 5
Production SAN Fabric
D
1
1
1
2
2
2
2
1
2
4
2
3
3
2
3
1 5
1
5
2
5
4
5
3
1
1
1
2
2
2
2
1
2
4
2
3
3
2
3
1 5
1
5
2
5
4
5
3
STG Zone-1
STG Zone-2
STG Zone-3
STG Zone-4
Production SAN Fabric
C
24 24
sSY0333
TYPES OF ZONES
HOST TO SPECTRUM VIRTUALIZE ZONING EVOLUTION
IBM Spectrum Virtualize Zoning - Best Practices 101
Host Zones a Closer Look
25
sSY0333
© Copyright IBM Corporation
2015
Host Zones Evolution – 4 Port Node
26
sSY0333
© Copyright IBM Corporation
2015
Host Zones evolution – 8 Port Node
27
sSY0333
© Copyright IBM Corporation
2015
Host Zones Evolution – 12 Port Node
• Spectrum Virtualize - DH8
Host
B1 A1
Fabric1 Core1 Fabric2 Core1
28
sSY0333
© Copyright IBM Corporation
2015
Multiple Host - ESX Cluster to
Spectrum Virtualize port zoning
• Create separate zones for each Host node in the ESX cluster or AIX HA cluster
• Big complex picture
Fabric A Fabric B
29
sSY0333
Host_1b Host_ A1
Host_2a Host_1a Host_2b
Multiple Host - ESX Cluster to Spectrum Virtualize
port zoning
30
Fabric 1 Core2
Slide provided by Chuck Laing
B1 A1 B1 A1 B1 A1 B1
Fabric1 Core1 Fabric 2 Core2 Fabric2 Core1
A1
Fabric A Fabric B
• Break down
sSY0333
Host_1b Host_ A1
Host_2a Host_1a Host_2b
Multiple Host - ESX Cluster to Spectrum Virtualize
port zoning
31
Fabric 1 Core2
Slide provided by Chuck Laing
B1 A1 B1 A1 B1 A1 B1 A1
Fabric1 Core1 Fabric 2 Core2 Fabric2 Core1 Fabric A Fabric B
• Break down
sSY0333
Host_1b Host_ A1
Host_2a Host_1a Host_2b
Multiple Host - ESX Cluster to Spectrum Virtualize
port zoning
32
Fabric 1 Core2
Slide provided by Chuck Laing
B1 A1 B1 A1 B1 A1 B1 A1
Fabric1 Core1 Fabric 2 Core2 Fabric2 Core1 Fabric A Fabric B
• Break down
sSY0333
Host_1b A1
Host_2a Host_1a Host_2b
Multiple Host - ESX Cluster to Spectrum Virtualize
port zoning
33
Fabric 1 Core2
Slide provided by Chuck Laing
B1 A1 B1 A1 B1 A1 B1 A1
Fabric1 Core1 Fabric 2 Core2 Fabric2 Core1 Fabric A Fabric B
• Break down
sSY0333
© Copyright IBM Corporation
2015
Zoning Multi HBA hosts for Resiliency
• Right way to zone a multiport Host
• Make four zones (Red, Blue, Orange and Green zones)
• Spread IO across multiple Ports SVC ports
HBA1
P
1
P
2
HBA2
P
1
P
2
Multi Port Host
P
1
2
SAN Fabric 2 SAN Fabric 1
34
sSY0333
© Copyright IBM Corporation
2015
Zoning Multi HBA hosts for Resiliency
• Sys Admins – provide PCI slot to Port WWPN identity to Storage Admins
• Storage Admins – define the Spectrum Virtualize host definitions to match
• Avoid single points of hardware failure at the Host HBA, Fabric and Spectrum Virtualize
• Make four zones, one for each sudo host per fabric(Red, Blue, Orange and Green zones)
HBA1
P
1
P
2
HBA2
P
1
P
2
Physical Host
SAN Fabric 1 SAN Fabric 2
HBA1
P
1
HBA2
P
1
P
2
Spectrum Virtualize defined Pseudo Host1
HBA1
P
1
P
2
HBA2
P
1
Spectrum Virtualize defined Pseudo Host2
35
sSY0333
TYPES OF ZONES - AIX VIOS TO SPECTRUM
VIRTUALIZE ZONING CONCEPTS
IBM Spectrum Virtualize Zoning - Best Practices 101
A Closer Look
36
sSY0333
© Copyright IBM Corporation
2015
Host Zones
Best Practices Concepts
• Planning and design for AIX Live Partition Mobility (LPM) – Create two separate and isolated zones on each fabric for each LPAR
• Do not put both the active and inactive LPAR wwpns in either the same zone
or same Spectrum Virtualize host definition
– Create Psuedo host Spectrum Virtualize host definitions containing only two virtual wwpns, one from each
fabric
• Follow the rules shown in the next few slides
– Map LUNs to the virtual host FC HBA port wwpns not he physical host FCA Adapter wwpn
• Exception to map only boot from SAN LUNs to the physical host FC HBA
wwpn
• Round Robin the Vdisks to the LPAR Pseudo and both the active and inactive
LPARs to enable LPM
– Note: Following these rules will result in not having a degraded path error on the
Spectrum Virtualize, and keep the 4 paths per Vdisk rule
37
sSY0333
© Copyright IBM Corporation
2015
•VFCA
FCA1
P1
P2
FCA2
P3
P4
Spectrum
Virtualize
VIO Server1 VIO Server2
Active
Client
Logical
Partition
(LPAR2)
Fame1 Hypervisor
SAN Active
Client
Logical
Partition
(LPAR1)
VFCA VP1.1a
VFCA VP4.2a
VFCA VP2.2a
VFCA VP3.1a
VP1.1a
VP2.2a
VFCA VP3.1a
VP4.2a VP2.2i
VP1.1i
VFCA
VFCA VP3.1i
VP4.2a VFCA
Dual VIOS to Multiple LPARs
a simplistic view
• Using NPVID, we recommend making no more than a ratio of 1 physical adapter
to 8 virtual ports to ensure IO bandwidths to the physical adapters avoid
oversubscription
38
sSY0333
© Copyright IBM Corporation
2015
•VFCA
FCA1
P1
P2
FCA2
P3
P4
Spectrum
Virtualize
VIO Server1 VIO Server2
Active
Client
Logical
Partition
(LPAR2) Fame1 Hypervisor
SAN Active
Client
Logical
Partition
(LPAR1)
VFCA VP1.1a
VFCA VP4.2a
VFCA VP2.2a
VFCA VP3.1a
VP1.1a
VP2.2a
VP3.1a
VP4.2a VP2.2i
VP1.1i VP3.1i
VP4.2i
Inactive
Client
Logical
Partition
(Pseudo
LPAR1b)
VFCA VP1.1i
VFCA VP4.2i
LPM
LPM
Could go to
Any Frame
Both active and inactive ports will be active during the LPM.
Upon LPM completion the previous active ports will now show
inactive and the previous inactive ports will show active.
inactive & active
vWWPN pairs
Dual VIOS to Multiple LPARs
Showing LPM
39
sSY0333
© Copyright IBM Corporation
2015
FCA2
P3
P4
FCA4
P7
P8
Spectrum
Virtualize
VIO Server1 VIO Server2
Pseudo1
I
Pseudo 2
Active
Client
Logical
Partition
(LPAR2)
Fame1 Hypervisor
SAN
Pseudo1
Pseudo2
Active
Client
Logical
Partition
(LPAR1)
VFCA VP1.1a
VFCA VP7.1a
VFCA VP5.1a
VFCA VP3.1a
VFCA VP2.2a
VFCA VP8.2a
VFCA VP6.2a
VFCA VP4.2a
VP1.1a
VP2.2a
VP5.1a
VP6.2a VP2.2i
VP1.1i VP5.1i
VP6.2a
FCA1
P1
P2
FCA3
P5
P6
VP3.1a
VP4.2a
VP3.1i
VP4.2i
VFCA7.1 VP7.1a
VP8.2a
VP7.1i
VP8.2i
VFCA6.1
VFCA8.1
VFCA5.1
VFCA3.1
VFCA1.1
VFCA2.1
VFCA4.1
Dual VIOS to Multiple LPARs Map LUNs within an LPAR in a round robin fashion to the active pseudo hosts. For example:
• Vdisk1 to Pseudo1 – Vdisk2 to Pseudo2 – Vdisk3 to Pseudo1 – Vdisk4 to Pseudo2
40
sSY0333
© Copyright IBM Corporation
2015
LPM Could go to Frame2 or Frame3 Both active and inactive ports will be active during the LPM. Upon LPM completion the previous active ports will now show inactive and the previous inactive ports will show active.
Map the same Vdisks to the inactive LPAR in the same fashion as the active LPAR
FCA2
P3
P4
FCA4
P7
P8
SVC
VIO Server1 VIO Server2
Pseudo1
I
Pseudo 2
Active
Client
Logical
Partition
(LPAR2)
Fame1 Hypervisor
SAN
Pseudo1
Pseudo2
Active
Client
Logical
Partition
(LPAR1)
VFCA VP1.1a
VFCA VP7.1a
VFCA VP5.1a
VFCA VP3.1a
VFCA VP2.2a
VFCA VP8.2a
VFCA VP6.2a
VFCA VP4.2a
VP1.1a
VP2.2a
VP5.1a
VP6.2a VP2.2i
VP1.1i VP5.1i
VP6.2a
FCA1
P1
P2
FCA3
P5
P6
VP3.1a
VP4.2a
VP3.1i
VP4.2i
VFCA4.1 VP7.1a
VP8.2a
VP7.1i
VP8.2i
Pseudo1
Pseudo2
Inactive
Client
Logical
Partition
(Pseudo
LPAR1b)
VFCA VP1.1i
VFCA VP7.1i
VFCA VP5.1i
VFCA VP3.1i
LPM
•VFCA
P1 P2
P3 P4
P3 … P64
I
Fame2
Hypervisor
SAN
VFCA
.
•VFCA
P1 P2
P3 P4
P3 … P64
I
Fame3
Hypervisor
SAN
VFCA
.
inactive & active
vWWPN pairs
41
sSY0333
LPM Could go to Frame2 or Frame3
During LPM the number of paths double from 4 to 8 Starting with 8 paths per vdisk will render an unsupported 16 paths during this time - could lead to IO interruption
FCA2
P3
P4
FCA4
P7
P8
SVC
VIO Server1 VIO Server2
Pseudo1
I
Pseudo 2
Active Client Logical Partition (LPAR2)
Fame1 Hypervisor
SAN
Pseudo1
Pseudo2
Active Client Logical Partition (LPAR1)
VFCA VP1.1a
VFCA VP7.1a
VFCA VP5.1a
VFCA VP3.1a
VFCA VP2.2a
VFCA VP8.2a
VFCA VP6.2a
VFCA VP4.2a
VP1.1a
VP2.2a
VP5.1a
VP6.2a VP2.2i
VP1.1i VP5.1i
VP6.2a
FCA1
P1
P2
FCA3
P5
P6
VP3.1a
VP4.2a
VP3.1i
VP4.2i
VFCA4.1 VP7.1a
VP8.2a
VP7.1i
VP8.2i
Pseudo1
Pseudo2
Active Client Logical Partition (Pseudo LPAR1b) During LPM
VFCA VP1.1i
VFCA VP7.1i
VFCA VP5.1i
VFCA VP3.1i
LPM
•VFCA
P1 P2
P3 P4
P3 … P64
I
Fame2 Hypervisor
SAN
VFCA
.
•VFCA
P1 P2
P3 P4
P3 … P64
I
Fame3 Hypervisor
SAN
VFCA
.
inactive & active vWWPN pairs
42
sSY0333
© Copyright IBM Corporation
2015
LPM
VFCA6.1
VFCA8.1
VFCA5.1
VFCA3.1
VFCA1.1
VFCA2.1
VFCA4.1
Dual VIOS to Multiple LPARs
Is it resilient? - One VIOS Failure
x
FCA2
P3
P4
FCA4
P7
P8
Spectrum
Virtualize
VIO Server1 VIO Server2
Pseudo1
I
Pseudo 2
Active
Client
Logical
Partition
(LPAR2)
Fame1 Hypervisor
SAN
Pseudo1
Pseudo2
Active
Client
Logical
Partition
(LPAR1)
VFCA VP1.1a
VFCA VP7.1a
VFCA VP5.1a
VFCA VP3.1a
VFCA VP2.2a
VFCA VP8.2a
VFCA VP6.2a
VFCA VP4.2a
VP1.1a
VP2.2a
VP5.1a
VP6.2a VP2.2i
VP1.1i VP5.1i
VP6.2a
FCA1
P1
P2
FCA3
P5
P6
VP3.1a
VP4.2a
VP3.1i
VP4.2i
VFCA7.1 VP7.1a
VP8.2a
VP7.1i
VP8.2i
43
sSY0333
© Copyright IBM Corporation
2015
LPM
VFCA6.1
VFCA8.1
VFCA5.1
VFCA3.1
VFCA1.1
VFCA2.1
VFCA4.1
x
FCA2
P3
P4
FCA4
P7
P8
Spectrum
Virtualize
VIO Server1 VIO Server2
Pseudo1
I
Pseudo 2
Active
Client
Logical
Partition
(LPAR2)
Fame1 Hypervisor
SAN
Pseudo1
Pseudo2
Active
Client
Logical
Partition
(LPAR1)
VFCA VP1.1a
VFCA VP7.1a
VFCA VP5.1a
VFCA VP3.1a
VFCA VP2.2a
VFCA VP8.2a
VFCA VP6.2a
VFCA VP4.2a
VP1.1a
VP2.2a
VP5.1a
VP6.2a VP2.2i
VP1.1i VP5.1i
VP6.2a
FCA1
P1
P2
FCA3
P5
P6
VP3.1a
VP4.2a
VP3.1i
VP4.2i
VFCA7.1 VP7.1a
VP8.2a
VP7.1i
VP8.2i
Dual VIOS to Multiple LPARs
Is it resilient? – One SAN Fabric Failure
44
sSY0333
TYPES OF ZONES
SPECTRUM VIRTUALIZE NODE TO NODE
IBM Spectrum Virtualize Zoning - Best Practices 101
A Closer Look
45
sSY0333
© Copyright IBM Corporation
2015
Node to Node / Heartbeat Zones
The Intra-cluster/Heartbeat/node to node communication zone
– Make the local_fc_port_mask (Intra-cluster) = 000011000000 for 12 ports
– Only make the local_fc_port_mask (Intra-cluster) = 110011000000
• if Write Data Rate > 3GB/sec per Iogrp
• In dual core fabrics, all dedicated heartbeat ports should reside on only one
core
Port Layout provided Keith Williams 46
sSY0333
TYPES OF ZONES
METRO MIRROR / GLOBAL MIRROR ZONES
IBM Spectrum Virtualize Zoning - Best Practices 101
MM/GM - A Closer Look
47
sSY0333
© Copyright IBM Corporation
2015
• One Spectrum Virtualize port from each node on each fabric should be zoned for GM traffic, taking care not to include both
ports that a server might use. This means ports that would also be used for replication would be either ports 1 & 2, or ports 3
& 4
• For each node in a cluster, exactly two fibre channel ports should be zoned to exactly two fibre channel ports from each node
in the partner cluster.
• If dual-redundant ISLs are available, then the two ports from each node should be split evenly between the two ISLs, i.e.
exactly one port from each node should be zoned across each ISL.
• Local cluster zoning should continue to follow the standard requirement for all ports on all nodes in a cluster to be zoned to
one another.
• This is discussed more verbosely on the Flash published for this issue on the IBM website:
– http://www-01.ibm.com/support/docview.wss?uid=ssg1S1003634 and https://www-304.ibm.com/support/docview.wss?uid=ssg1S1003634
Mirror Zones Evolution
MM/GM and Heartbeat – 4 Port Nodes
48
sSY0333
© Copyright IBM Corporation
2015
Mirror Zones Evolution
Spectrum Virtualize GG8 – 8 Port Nodes
• Make the local_fc_port_mask (Node to Node/heartbeat) =
11000000
• Make the partner_fc_port_mask (MM/GM) =
00110000
WAN direct connection to the
FCiP connection to the Spectrum Virtualize ports
GM/MM zone A
WAN direct connection to the
FCiP connection to the Spectrum Virtualize ports
GM/MM zone B
49
sSY0333
© Copyright IBM Corporation
2015
Mirror Zones Evolution
Spectrum Virtualize DH8 – 12 Port Nodes
• Make the local_fc_port_mask(Intra-cluster)=000011000000 for 12 ports
• Make the local_fc_port_mask(Intra-cluster)=110011000000 - Write Data Rate > 3GB/sec per
IOgrp
• Make the partner_fc_port_mask (MM/GM) = 000000110000
WAN direct connection to the
FCiP connection to the Spectrum Virtualize ports
GM/MM zone B
WAN direct connection to the
FCiP connection to the Spectrum Virtualize ports
GM/MM zone A
50
sSY0333
PLANNING DESIGN EXAMPLE FOR
CORRECT “DUAL CORE” FABRIC ZONING
IBM Spectrum Virtualize Zoning - Best Practices 101
Dual Core Design - A Closer Look
51
sSY0333
© Copyright IBM Corporation
2015
Dual core fabric / Multiple Host –
ESX Cluster to Spectrum Virtualize port zoning
• Zone a single HA/ESX cluster to the same core to avoid ISL IO traversing
• Spread Multiple Host clusters evenly across Cores
• Create separate zones for each Host node in the ESX cluster or AIX HA cluster
52
sSY0333
Host_1b Host_ A1
Host_2a Host_1a Host_2b
Dual core fabric / Multiple Host - ESX Cluster to
Spectrum Virtualize port zoning
53
Fabric 1 Core2
Slide provided by Chuck Laing
B1 A1 B1 A1 B1 A1 B1
Fabric1 Core1 Fabric 2 Core2 Fabric2 Core1
A1
• Break down
sSY0333
Host_1b Host_ A1
Host_2a Host_1a Host_2b
Dual core fabric / Multiple Host - ESX Cluster to
Spectrum Virtualize port zoning
54
Fabric 1 Core2
Slide provided by Chuck Laing
B1 A1 B1 A1 B1 A1 B1 A1
Fabric1 Core1 Fabric 2 Core2 Fabric2 Core1
• Break down
sSY0333
Host_1b Host_ A1
Host_2a Host_1a Host_2b
Dual core fabric / Multiple Host - ESX Cluster to
Spectrum Virtualize port zoning
55
Fabric 1 Core2
Slide provided by Chuck Laing
B1 A1 B1 A1 B1 A1 B1 A1
Fabric1 Core1 Fabric 2 Core2 Fabric2 Core1
• Break down
sSY0333
Host_1b A1
Host_2a Host_1a Host_2b
Dual core fabric / Multiple Host - ESX Cluster to
Spectrum Virtualize port zoning
56
Fabric 1 Core2
Slide provided by Chuck Laing
B1 A1 B1 A1 B1 A1 B1 A1
Fabric1 Core1 Fabric 2 Core2 Fabric2 Core1
• Break down
sSY0333
© Copyright IBM Corporation
2015
Correct Dual Core Zoning to Multiple Storage devices
• Zone each device to only one core to avoid extra “ISL IO traversing” cutting
down ISL hops
• Spread Multiple Storage devices as evenly possible across Cores
• Create separate zones for each Storage device
Fabric1 Core1 Fabric 1 Core2 Fabric2 Core1 Fabric 2 Core2
V7000 XIV DS8K_2 DS8K_1
57
sSY0333
© Copyright IBM Corporation
2015
Correct Dual Core Zoning to Multiple Storage devices
Fabric1 Core1 Fabric 1 Core2 Fabric2 Core1 Fabric 2 Core2
V7000 XIV DS8K_2 DS8K_1
• Break down
58
sSY0333
© Copyright IBM Corporation
2015
Correct Dual Core Zoning to Multiple Storage devices
Fabric1 Core1 Fabric 1 Core2 Fabric2 Core1 Fabric 2 Core2
V7000 XIV DS8K_2 DS8K_1
• Break down
59
sSY0333
© Copyright IBM Corporation
2015
Spectrum Virtualize DH8- Mirror Zones
• Make the local_fc_port_mask (Node to Node) = 000010001000
• Make the partner_fc_port_mask (MM/GM = 000001000100
Port Layout provided Keith Williams
Separate IOgrps to separate core
60
sSY0333
© Copyright IBM Corporation
2015
• You should know:
– Zoning is one of the most important configuration for proper:
• Failover
• Redundancy
• Resiliency
• Performance
– Understand best practice zoning for 4 port Spectrum Virtualize models
– Understand best practice zoning for 8 port models – CG8
– Understand best practice zoning for 12 port models –DH8
– Understand best practice zoning for the V9000 DH8 w/Flash900
– Understand best practice zoning for Dual Core Fabric to Spectrum Virtualize
– Hopefully enjoyed learning Right and Wrong Zoning Scenarios by playing along – lets do it!
– Had “Fun”
Session summary
61
sSY0333
Thank you!
For you interest
and attendance
62
sSY0333
RIGHT OR WRONG WAY SCENARIOS
IBM Spectrum Virtualize Zoning - Best Practices 101
Lets Have Some Fun, with Right or Wrong
63
sSY0333 Incorrect vs. correct Dual core fabric Host to Spectrum Virtualize port zoning
64
Host B1 A1
Fabric1
Core1
Fabric 1
Core2
Fabric2
Core1
Fabric 2
Core2
Right or Wrong?
sSY0333 Incorrect vs. correct Dual core fabric Host to Spectrum Virtualize port zoning
65
Host B1 A1
Fabric1
Core1
Fabric 1
Core2
Fabric2
Core1
Fabric 2
Core2
Host B1 A1
Fabric1
Core1
Fabric 1
Core2
Fabric2
Core1
Fabric 2
Core2
Incorrect Correct
sSY0333
© Copyright IBM Corporation
2015
Correct Dual core fabric / Host to Spectrum Virtualize port zoning
Host1
B1 A1
Fabric1 Core1 Fabric 1 Core2
4 Node CG8 SVC
Node 1
12 34
HBA 1
1
0
2
0
4
0
3
0
56 78
HBA 2
7
0
8
0
5
0
6
0
iogrp 02048 LUNs max
Node 2
12 34
HBA 1
1
0
2
0
4
0
3
0
56 78
HBA 2
7
0
8
0
5
0
6
0
Node 1
12 34
HBA 1
1
0
2
0
4
0
3
0
56 78
HBA 2
7
0
8
0
5
0
6
0
iogrp 12048 LUNs max
Node 2
12 34
HBA 1
1
0
2
0
4
0
3
0
56 78
HBA 2
7
0
8
0
5
0
6
0
Fabric2 Core1 Fabric 2 Core2
Host2
B1 A1
Right or Wrong?
Red and Blue cores have ISLs between cores respectively?
66
sSY0333
© Copyright IBM Corporation
2015
Correct Dual core fabric / Host to Spectrum Virtualize port zoning
Host1
B1 A1
Fabric1 Core1 Fabric 1 Core2
4 Node CG8 SVC
Node 1
12 34
HBA 1
1
0
2
0
4
0
3
0
56 78
HBA 2
7
0
8
0
5
0
6
0
iogrp 02048 LUNs max
Node 2
12 34
HBA 1
1
0
2
0
4
0
3
0
56 78
HBA 2
7
0
8
0
5
0
6
0
Node 1
12 34
HBA 1
1
0
2
0
4
0
3
0
56 78
HBA 2
7
0
8
0
5
0
6
0
iogrp 12048 LUNs max
Node 2
12 34
HBA 1
1
0
2
0
4
0
3
0
56 78
HBA 2
7
0
8
0
5
0
6
0
Fabric2 Core1 Fabric 2 Core2
Host2
B1 A1
Correct Correct
67
sSY0333
© Copyright IBM Corporation
2015
Zoning Multi HBA hosts for Resiliency
• Right or Wrong?
HBA1
P
1
P
2
HBA2
P
1
P
2
Multiport Host
SAN Fabric 1 SAN Fabric 2
P
1
2
68
sSY0333
© Copyright IBM Corporation
2015
HBA1
P
1
P
2
HBA2
P
1
P
2
Multiport Host
SAN Fabric 1 SAN Fabric 2
P
1
2
Zoning Multi HBA hosts for Resiliency
• Is this better?
69
sSY0333
© Copyright IBM Corporation
2015
Zoning Multi HBA hosts for Resiliency
• Right way to zone a multiport Host
• Make four zones (Red, Blue, Orange and Green zones)
• Spread IO across multiple Spectrum Virtualize Ports
HBA1
P
1
P
2
HBA2
P
1
P
2
Multi Port Host
SAN Fabric 1 SAN Fabric 2
P
1
2
70
sSY0333
© Copyright IBM Corporation
2015
Right or Wrong Game/Storage - Spectrum Virtualize
• Which example is Right?
DS8K Right I/O Enclosures
Bay 1
1
3
0
1
3
1
1
3
2
1
3
3
C1R4
1
0
0
1
0
1
1
0
2
1
0
3
C0R2
Bay 3
3
3
0
3
3
1
3
3
2
3
3
3
C1R7
3
0
0
3
0
1
3
0
2
3
0
3
C0R3
Bay 5
5
3
0
5
3
1
5
3
2
5
3
3
C1R8
5
0
0
5
0
1
5
0
2
5
0
3
C0R4
Bay 7
7
3
0
7
3
1
7
3
2
7
3
3
C1R7
7
0
0
7
0
1
7
0
2
7
0
3
C0R3
DS8K Left I/O EnclosuresBay 0
0
0
0
0
0
1
0
0
2
0
0
3
C1L4
0
3
0
0
3
1
0
3
2
0
3
3
C0L2
Bay 2
2
0
0
2
0
1
2
0
2
2
0
3
C1L7
2
3
0
2
3
1
2
3
2
2
3
3
C0L3
Bay 4
4
0
0
4
0
1
4
0
2
4
0
3
C1L8
4
3
0
4
3
1
4
3
2
4
3
3
C0L4
Bay 6
6
0
0
6
0
1
6
0
2
6
0
3
C1L7
6
3
0
6
3
1
6
3
2
6
3
3
C0L3
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
port1=11052ca port1=11052b7 port1=110529e port1=110528b
port2=12052ca port2=12052b7 port2=120529e port2=120528b
port3=13052ca port3=13052b7 port3=130529e port3=130528b
port4=14052ca port4=14052b7 port4=140529e port4=140528b
Application Host Server
A2 B2
The same native
backend ports should not be
shared for both direct host connectivity
and SVC connectivity as shown here.
The correct way to bypass SVC is to use
other backend ports not connected to the
SVC
DS8K Right I/O Enclosures
Bay 1
1
3
0
1
3
1
1
3
2
1
3
3
C1R4
1
0
0
1
0
1
1
0
2
1
0
3
C0R2
Bay 3
3
3
0
3
3
1
3
3
2
3
3
3
C1R7
3
0
0
3
0
1
3
0
2
3
0
3
C0R3
Bay 5
5
3
0
5
3
1
5
3
2
5
3
3
C1R8
5
0
0
5
0
1
5
0
2
5
0
3
C0R4
Bay 7
7
3
0
7
3
1
7
3
2
7
3
3
C1R7
7
0
0
7
0
1
7
0
2
7
0
3
C0R3
DS8K Left I/O EnclosuresBay 0
0
0
0
0
0
1
0
0
2
0
0
3
C1L4
0
3
0
0
3
1
0
3
2
0
3
3
C0L2
Bay 2
2
0
0
2
0
1
2
0
2
2
0
3
C1L7
2
3
0
2
3
1
2
3
2
2
3
3
C0L3
Bay 4
4
0
0
4
0
1
4
0
2
4
0
3
C1L8
4
3
0
4
3
1
4
3
2
4
3
3
C0L4
Bay 6
6
0
0
6
0
1
6
0
2
6
0
3
C1L7
6
3
0
6
3
1
6
3
2
6
3
3
C0L3
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
port1=11052ca port1=11052b7 port1=110529e port1=110528b
port2=12052ca port2=12052b7 port2=120529e port2=120528b
port3=13052ca port3=13052b7 port3=130529e port3=130528b
port4=14052ca port4=14052b7 port4=140529e port4=140528b
Application Host Server
A2 B2
The same native
backend ports should not be
shared for both direct host connectivity
and SVC connectivity as shown here.
The correct way to bypass SVC is to use
other backend ports not connected to the
SVC
1 2
71
sSY0333
© Copyright IBM Corporation
2015
Right or Wrong Game/Storage - Spectrum Virtualize
DS8K Right I/O Enclosures
Bay 1
1
3
0
1
3
1
1
3
2
1
3
3
C1R4
1
0
0
1
0
1
1
0
2
1
0
3
C0R2
Bay 3
3
3
0
3
3
1
3
3
2
3
3
3
C1R7
3
0
0
3
0
1
3
0
2
3
0
3
C0R3
Bay 5
5
3
0
5
3
1
5
3
2
5
3
3
C1R8
5
0
0
5
0
1
5
0
2
5
0
3
C0R4
Bay 7
7
3
0
7
3
1
7
3
2
7
3
3
C1R7
7
0
0
7
0
1
7
0
2
7
0
3
C0R3
DS8K Left I/O EnclosuresBay 0
0
0
0
0
0
1
0
0
2
0
0
3
C1L4
0
3
0
0
3
1
0
3
2
0
3
3
C0L2
Bay 2
2
0
0
2
0
1
2
0
2
2
0
3
C1L7
2
3
0
2
3
1
2
3
2
2
3
3
C0L3
Bay 4
4
0
0
4
0
1
4
0
2
4
0
3
C1L8
4
3
0
4
3
1
4
3
2
4
3
3
C0L4
Bay 6
6
0
0
6
0
1
6
0
2
6
0
3
C1L7
6
3
0
6
3
1
6
3
2
6
3
3
C0L3
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
port1=11052ca port1=11052b7 port1=110529e port1=110528b
port2=12052ca port2=12052b7 port2=120529e port2=120528b
port3=13052ca port3=13052b7 port3=130529e port3=130528b
port4=14052ca port4=14052b7 port4=140529e port4=140528b
Application Host Server
A2 B2
The same native
backend ports should not be
shared for both direct host connectivity
and SVC connectivity as shown here.
The correct way to bypass SVC is to use
other backend ports not connected to the
SVC
DS8K Right I/O Enclosures
Bay 1
1
3
0
1
3
1
1
3
2
1
3
3
C1R4
1
0
0
1
0
1
1
0
2
1
0
3
C0R2
Bay 3
3
3
0
3
3
1
3
3
2
3
3
3
C1R7
3
0
0
3
0
1
3
0
2
3
0
3
C0R3
Bay 5
5
3
0
5
3
1
5
3
2
5
3
3
C1R8
5
0
0
5
0
1
5
0
2
5
0
3
C0R4
Bay 7
7
3
0
7
3
1
7
3
2
7
3
3
C1R7
7
0
0
7
0
1
7
0
2
7
0
3
C0R3
DS8K Left I/O EnclosuresBay 0
0
0
0
0
0
1
0
0
2
0
0
3
C1L4
0
3
0
0
3
1
0
3
2
0
3
3
C0L2
Bay 2
2
0
0
2
0
1
2
0
2
2
0
3
C1L7
2
3
0
2
3
1
2
3
2
2
3
3
C0L3
Bay 4
4
0
0
4
0
1
4
0
2
4
0
3
C1L8
4
3
0
4
3
1
4
3
2
4
3
3
C0L4
Bay 6
6
0
0
6
0
1
6
0
2
6
0
3
C1L7
6
3
0
6
3
1
6
3
2
6
3
3
C0L3
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
port1=11052ca port1=11052b7 port1=110529e port1=110528b
port2=12052ca port2=12052b7 port2=120529e port2=120528b
port3=13052ca port3=13052b7 port3=130529e port3=130528b
port4=14052ca port4=14052b7 port4=140529e port4=140528b
Application Host Server
A2 B2
The same native
backend ports should not be
shared for both direct host connectivity
and SVC connectivity as shown here.
The correct way to bypass SVC is to use
other backend ports not connected to the
SVC
The same port on the back-end is being used for host and Spectrum Virtualize
Spectrum Virtualize
Correct Example Spectrum Virtualize Incorrect Example
72
sSY0333
© Copyright IBM Corporation
2015
3 factors determine right/wrong number of paths
per vdisk device
• Example of correct/incorrect number of datapaths with Host to
Spectrum Virtualize zoning
• Which Side is right?
73
sSY0333
© Copyright IBM Corporation
2015
Summary - 3 factors determine right/wrong paths
• Example of correct/incorrect number of datapaths with Host to Spectrum
Virtualize zoning
• Correct datapathing has 3 factors
• Proper zoning
• Proper Spectrum Virtualize Host definitions (Spectrum Virtualize logical
config of the host def)
• Proper redundancy for the Spectrum Virtualize preferred /non preferred
pathing
74
sSY0333
© Copyright IBM Corporation
2015
Right or Wrong ?
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 CF8wwpn
5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
b03vio101NRPOKVIO1A
d1 d3
SVC Host Definitionsid:1
name:P770_1_vio1A
10000000C9C0B3DB
10000000C9C0DC7F
10000000C9C0E0E0
10000000C9C0A984
fscsi0=10000000C9C0A984
fscsi2=10000000C9C0E0E0
fscsi5=10000000C9C0DC7F
fscsi7=10000000C9C0B3DB
port1=10B374 port1=10B363 port1=10B371 port1=10B335
port2=20B374 port2=20B363 port2=20B371 port2=20B335
port3=30B374 port3=30B363 port3=30B371 port3=30B335
port4=40B374 port4=40B363 port4=40B371 port4=40B335
Zone for p770_1_vio1a_d1
10000000c9779a4a
500507680110B374
500507680130B374
500507680110B363
500507680130B363
500507680110B371
500507680130B371
500507680110B335
500507680130B335
Zone for p770_1_vio1a_d3_SVC
10000000C9C0DC7F
500507680120B374
500507680140B374
500507680120B363
500507680140B363
500507680120B371
500507680140B371
500507680120B335
500507680140B335
d2 d4
Zone for p770_1_vio1a_d4_SVC
10000000C9C0B3DB
500507680120B374
500507680140B374
500507680120B363
500507680140B363
500507680120B371
500507680140B371
500507680120B335
500507680140B335
Zone for p770_1_vio1a_d2
10000000C9C0E0E0
500507680110B374
500507680130B374
500507680110B363
500507680130B363
500507680110B371
500507680130B371
500507680110B335
500507680130B335
Over subscribed SVC to Host HBA Zoning causing to many datapaths
DEV#: 3 DEVICE NAME: hdisk3 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 600507680181059BA000000000000005
==========================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 OPEN NORMAL 558254 0
1* fscsi0/path1 OPEN NORMAL 197 0
2* fscsi0/path2 OPEN NORMAL 197 0
3 fscsi0/path3 OPEN NORMAL 493559 0
4 fscsi2/path4 OPEN NORMAL 493330 0
5* fscsi2/path5 OPEN NORMAL 197 0
6* fscsi2/path6 OPEN NORMAL 197 0
7 fscsi2/path7 OPEN NORMAL 493451 0
8 fscsi5/path8 OPEN NORMAL 492225 0
9* fscsi5/path9 OPEN NORMAL 197 0
10* fscsi5/path10 OPEN NORMAL 197 0
11 fscsi5/path11 OPEN NORMAL 492660 0
12 fscsi7/path12 OPEN NORMAL 491988 0
13* fscsi7/path13 OPEN NORMAL 197 0
14* fscsi7/path14 OPEN NORMAL 197 0
15 fscsi7/path15 OPEN NORMAL 492943 0
75
sSY0333
© Copyright IBM Corporation
2015
Wrong - Causes Over Subscription
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 CF8wwpn
5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
b03vio101NRPOKVIO1A
d1 d3
SVC Host Definitionsid:1
name:P770_1_vio1A
10000000C9C0B3DB
10000000C9C0DC7F
10000000C9C0E0E0
10000000C9C0A984
fscsi0=10000000C9C0A984
fscsi2=10000000C9C0E0E0
fscsi5=10000000C9C0DC7F
fscsi7=10000000C9C0B3DB
port1=10B374 port1=10B363 port1=10B371 port1=10B335
port2=20B374 port2=20B363 port2=20B371 port2=20B335
port3=30B374 port3=30B363 port3=30B371 port3=30B335
port4=40B374 port4=40B363 port4=40B371 port4=40B335
Zone for p770_1_vio1a_d1
10000000c9779a4a
500507680110B374
500507680130B374
500507680110B363
500507680130B363
500507680110B371
500507680130B371
500507680110B335
500507680130B335
Zone for p770_1_vio1a_d3_SVC
10000000C9C0DC7F
500507680120B374
500507680140B374
500507680120B363
500507680140B363
500507680120B371
500507680140B371
500507680120B335
500507680140B335
d2 d4
Zone for p770_1_vio1a_d4_SVC
10000000C9C0B3DB
500507680120B374
500507680140B374
500507680120B363
500507680140B363
500507680120B371
500507680140B371
500507680120B335
500507680140B335
Zone for p770_1_vio1a_d2
10000000C9C0E0E0
500507680110B374
500507680130B374
500507680110B363
500507680130B363
500507680110B371
500507680130B371
500507680110B335
500507680130B335
Over subscribed SVC to Host HBA Zoning causing to many datapaths
DEV#: 3 DEVICE NAME: hdisk3 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 600507680181059BA000000000000005
==========================================================
Path# Adapter/Path Name State Mode Select Errors
0 fscsi0/path0 OPEN NORMAL 558254 0
1* fscsi0/path1 OPEN NORMAL 197 0
2* fscsi0/path2 OPEN NORMAL 197 0
3 fscsi0/path3 OPEN NORMAL 493559 0
4 fscsi2/path4 OPEN NORMAL 493330 0
5* fscsi2/path5 OPEN NORMAL 197 0
6* fscsi2/path6 OPEN NORMAL 197 0
7 fscsi2/path7 OPEN NORMAL 493451 0
8 fscsi5/path8 OPEN NORMAL 492225 0
9* fscsi5/path9 OPEN NORMAL 197 0
10* fscsi5/path10 OPEN NORMAL 197 0
11 fscsi5/path11 OPEN NORMAL 492660 0
12 fscsi7/path12 OPEN NORMAL 491988 0
13* fscsi7/path13 OPEN NORMAL 197 0
14* fscsi7/path14 OPEN NORMAL 197 0
15 fscsi7/path15 OPEN NORMAL 492943 0
76
sSY0333
© Copyright IBM Corporation
2015
Right or Wrong?
77
sSY0333
© Copyright IBM Corporation
2015
Right or Wrong?
78
sSY0333
© Copyright IBM Corporation
2015
Right or Wrong?
79
sSY0333
© Copyright IBM Corporation
2015
Wrong, Shows as Degraded Paths
© Copyright IBM
Corporation 2014 80
sSY0333
© Copyright IBM Corporation
2015
What’s wrong?
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
b03vio101b03vio101
A1 A2 A3 B1
SVC Host Definitionsid:2
name:b03vio100_apvg
10000000C97549BD
10000000C97549BC
id:3
name:b03vio100_dbvg
10000000C9771AAF
10000000C9771AAE
Port to Fabrics
not evenly distributed
In this example we show
2 SVC Host definitions, for 1 host,
with uneven port to fabric distribution
and 4 zones
As you can see...A1 will failover to A2,
on the same fabric..this is a Single point
of Falure (SPoF)
See the next page for correct zoning
where there are multiple Host HBAs,
zoned to the SVC
port1=11052ca port1=11052b7 port1=110529e port1=110528b
port2=12052ca port2=12052b7 port2=120529e port2=120528b
port3=13052ca port3=13052b7 port3=130529e port3=130528b
port4=14052ca port4=14052b7 port4=140529e port4=140528b
Fabric_A Zone Definitions
Zone 101_a Zone_101_dA1=10000000c97549bc B2=10000000c9771aae
P1=50050768011052ca P1=50050768011052ca
P1=50050768011052b7 P1=50050768011052b7
P1=500507680110529e P1=500507680110529e
P1=500507680110528b P1=500507680110528b
Zone_101_bA2=10000000c97549bd
P1=50050768013052ca
P1=50050768013052b7
P1=500507680130529e
P1=500507680130528b
Fabric_B Zone Definitions
Zone 101_cB1=10000000c9771aaf
P1=50050768014052ca
P1=50050768014052b7
P1=500507680140529e
P1=500507680140528b
81
sSY0333
© Copyright IBM Corporation
2015
Wrong
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
b03vio101b03vio101
A1 A2 A3 B1
SVC Host Definitionsid:2
name:b03vio100_apvg
10000000C97549BD
10000000C97549BC
id:3
name:b03vio100_dbvg
10000000C9771AAF
10000000C9771AAE
Port to Fabrics
not evenly distributed
In this example we show
2 SVC Host definitions, for 1 host,
with uneven port to fabric distribution
and 4 zones
As you can see...A1 will failover to A2,
on the same fabric..this is a Single point
of Falure (SPoF)
See the next page for correct zoning
where there are multiple Host HBAs,
zoned to the SVC
port1=11052ca port1=11052b7 port1=110529e port1=110528b
port2=12052ca port2=12052b7 port2=120529e port2=120528b
port3=13052ca port3=13052b7 port3=130529e port3=130528b
port4=14052ca port4=14052b7 port4=140529e port4=140528b
Fabric_A Zone Definitions
Zone 101_a Zone_101_dA1=10000000c97549bc B2=10000000c9771aae
P1=50050768011052ca P1=50050768011052ca
P1=50050768011052b7 P1=50050768011052b7
P1=500507680110529e P1=500507680110529e
P1=500507680110528b P1=500507680110528b
Zone_101_bA2=10000000c97549bd
P1=50050768013052ca
P1=50050768013052b7
P1=500507680130529e
P1=500507680130528b
Fabric_B Zone Definitions
Zone 101_cB1=10000000c9771aaf
P1=50050768014052ca
P1=50050768014052b7
P1=500507680140529e
P1=500507680140528b
82
sSY0333
© Copyright IBM Corporation
2015
Right
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn500507680
DIR1 SAN Fabric DIR2 SAN Fabric
b03vio101_a
A1 B1
SVC Host Definitionsid:2
name:b03vio101_a
10000000C97549BC
10000000C9771AAF
id:3
name:b03vio101_b
10000000C97549BD
10000000C9771AAE
Fabric_A Zone Definitions
Zone 101_aA1=10000000c97549bc
P1=50050768011052ca
P1=50050768011052b7
P1=500507680110529e
P1=500507680110528b
Zone_101_b
A2=10000000c97549bd
P1=50050768013052ca
P1=50050768013052b7
P1=500507680130529e
P1=500507680130528b
The host definitions in the SVC
should be defined as 2 hosts
and 4 zones in the Fabrics
In this example we show
2 seperate SVC Host definitions and
2 Fabric zones, per host per Fabric
Totaling 4 zones
This allows proper failover from Primary
to Alternate nodes within the SVC iogrp.
b03vio101_b
A2 B2
port1=11052ca port1=11052b7 port1=110529e port1=110528b
port2=12052ca port2=12052b7 port2=120529e port2=120528b
port3=13052ca port3=13052b7 port3=130529e port3=130528b
port4=14052ca port4=14052b7 port4=140529e port4=140528b
Fabric_B Zone Definitions
Zone 101_aB1=10000000c9771aaf
P1=50050768014052ca
P1=50050768014052b7
P1=500507680140529e
P1=500507680140528b
Zone_101_bB2=10000000c9771aae
P1=50050768012052ca
P1=50050768012052b7
P1=500507680120529e
P1=500507680120528b
83
sSY0333
© Copyright IBM Corporation
2015
Right or Wrong?
b03vio210
A1A2A3A4 B1B2B3B4
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
SVC Host Definitionsid:8
name:b03vio210_allvg
10000000C942B618
10000000C942B44B
10000000C9428016
10000000C94289AC
10000000C93FF72F
10000000C93F7416
10000000C93F8054
10000000C93F7537
Fabric_A Zone Definitions
A1=10:00:00:00:c9:3f:75:37
A2=10:00:00:00:c9:3f:80:54
A3=10:00:00:00:c9:42:89:ac
A4=10:00:00:00:c9:42:80:16
Fabric_B Zone Definitions
B1=10:00:00:00:c9:3f:74:16
B2=10:00:00:00:c9:3f:f7:2f
B3=10:00:00:00:c9:42:b4:4b
B4=10:00:00:00:c9:42:b6:18
In this example we show
1 SVC Host definitions and
2 Fabric zones, per host per Fabric.
This does not allow proper failover from
Primary to Alternate nodes within the
SVC iogrp.
See the next page for proper zoning
84
sSY0333
© Copyright IBM Corporation
2015
Right
b03vio210_2
A2 B2
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
SVC Host Definitionsid:1 name:b03vio210_1
10000000C93F7537
10000000C9428016
id:2 name:b03vio210_2
10000000C93F8054
10000000C93FF72F
id:3 name:b03vio210_3
10000000C94289AC
10000000C942B44B
id:4 name:b03vio210_4
10000000C93F7416
10000000C942B618
In this example we show
4 seperate SVC Host definitions and
2 seperate Fabric zones, per host
per Fabric
This does allow proper failover from
Primary to Alternate nodes within the
SVC iogrp.
b03vio210_3
A3 B3
b03vio210_4
A4 B4
b03vio210_1
A1 B1
Fabric_A Zone Definitions
A1=10:00:00:00:c9:3f:75:37Fabric_B Zone Definitions
B4=10:00:00:00:c9:42:b6:18
Fabric_A Zone Definitions
A4=10:00:00:00:c9:42:80:16Fabric_A Zone Definitions
A3=10:00:00:00:c9:42:89:ac
Fabric_A Zone Definitions
A2=10:00:00:00:c9:3f:80:54
Fabric_B Zone Definitions
B3=10:00:00:00:c9:42:b4:4b
Fabric_B Zone Definitions
B2=10:00:00:00:c9:3f:f7:2f
Fabric_B Zone Definitions
B1=10:00:00:00:c9:3f:74:16
85
sSY0333
SPECTRUM VIRTUALIZE
ZONING EVOLUTION
IBM Spectrum Virtualize Zoning - Best Practices 101
Backup Slides
86
sSY0333
© Copyright IBM Corporation
2015
Most common zoning questions - BP recommendations
• What is the max number of WWNN/WWPNs devices?
– Always check the Max Limit configuration URL for the most current updates
• Search for the term "configuration limits and restrictions" in the Search support and downloads search box at the
following website:
– www.ibm.com/storage/support/2145
– Currently 1024 WWNN per cluster and 1024 WWPNs per cluster
• Recommendation - The more wwpns per wwnn, the more throughput up to 16
• How many IO connections/zones per storage device to the Spectrum
Virtualize should I zone?
– 16 from any “one” storage device unit zoned with all Spectrum Virtualize node ports
• All Spectrum Virtualize nodes must see same set of LUNs from
disk controller – Otherwise degraded mode on controller and/or MDisks
sSY0333
© Copyright IBM Corporation
2015
Most common zoning questions - BP recommendations
• Should I mask host connections through zoning or Spectrum Virtualize
configurations?
– Best practice is to use the default value of …..111111111111 (all ports enabled) and control masking
through zoning
– Use host type Target Port Group (tpgs) for Solaris hosts, (hpux) for HP/UX, (openvms) for HP
AlphaServer/HP Intergrity and (generic) for everything else
– Separate disk and tape IO on host HBAs
• What works better, 4 or 8 paths per Vdisk?
– Recommendation is 4 paths per Vdisk
sSY0333
© Copyright IBM Corporation
2015
Most common zoning questions - BP recommendations
• How many Iogrps should I map to a host? 4?
– Recommendation is to size per throughput and number of hosts per cluster
– May need to isolate/sement host activity/chatter from one Iogrp to another
– For more horsepower – spread IO across more LUNs from more Iogrps
• Can a host have only one connection to the Spectrum Virtualize ?
– Dual host HBA connections
sSY0333
© Copyright IBM Corporation
2015
• Maximum of 1024 WWNNs
– EMC DMX/SYMM, All HDS and SUN/HP HDS clones use one WWNN per port; each appears as a
separate controller to Spectrum Virtualize
• Map LUNs through up to 16 FA ports – Results in 16 WWNNs/WWPNs used out of the max of 1024
– IBM, EMC Clariion, HP, etc. use one WWNN per subsystem; each appears as a single controller with
multiple ports/WWPNs
• Maximum of 16 ports/WWPNs per WWNN using 1 out of the
max of 1024
• Greater Qdepth is archived with fewer Mdisks
90
Disk Controller Best Practices
sSY0333
91
• DS4K/5K – EMC Clariion/CX
– Both have preferred controller architecture
• Spectrum Virtualize honors this configuration
– Use minimum of 4 and preferably 8 ports or more up to maximum of 16
– More ports equate to more concurrent I/O driven by Spectrum Virtualize
– Support for mapping controller A ports to Fabric A and controller B ports to Fabric B or cross connecting ports to both fabrics from both controllers
• IBM® DS4000® does not support a storage system A and storage system B in the same zone
• Later is preferred to avoid AVT/Trespass occurring if a fabric or all paths to a fabric fail
– Spectrum Virtualize supports Spectrum Virtualize queue depth change for CX models
• Drives more I/O per port per Mdisk
Disk Controller Best Practices
sSY0333
© Copyright IBM Corporation
2015
Storage Zones - General Spectrum Virtualize Best Practice Zoning
Concepts
• Create two cluster zones (do not include MM/GM and intra-cluster traffic
ports)
• Never span zones to include more than one Backend storage device!
• Never put Host OS ports, Spectrum Virtualize ports and Backend
Storage ports together in the same zone
– Instead • Create zones with Host ports and Spectrum Virtualize ports
• Create zones with Backend and Spectrum Virtualize ports
– Never use the same DS8K ports or any native back-end port for
connectivity to Spectrum Virtualize and an attached host
• If Spectrum Virtualize is attached to the DS8K or other native back-end
devices and the DS8K or other back-end device is using native GM (not
Spectrum Virtualize GM) then dedicate appropriate back-end ports
specifically for GM, not to be used for attaching any other device, whether
Host Server, Spectrum Virtualize or other connectivity relationships.
sSY0333
© Copyright IBM Corporation
2015
Host Zones
Best Practices Concepts
• Too many paths to a Vdisk – If the recommended number of paths to a Vdisk are exceeded, path failures may not be recovered in the
required amount of time
• Causes excessive I/O waits, resulting in application failures
• Under certain circumstances, it can reduce performance
– Note: 8 paths are supported but 4 are optimum for SDD/SDDDSM/SDDPCM
• Spectrum Virtualize host zones – There must be a single zone for each host port. This zone must contain the host port, and one port from
each Spectrum Virtualize node that the host will need to access. While there are two ports from each node
per SAN fabric in a usual dual-fabric configuration, make sure that the host only accesses one of them.
• Check with the Host System Administrators to receive output from the SDD or
PCM path query device commands on each host to verify proper pathing
between the Spectrum Virtualize nodes and host systems
– Note: It is a supported configuration to have eight paths to each VDisk, but this design
provides no performance benefit, and it does not improve reliability or availability by any
significant degree
sSY0333
© Copyright IBM Corporation
2015
Host Zones
Spectrum Virtualize Preferred Node Scheme
• Hosts with four (or more) Host Bus Adapters (HBAs)
– Takes a little more planning.
– Because eight paths are not an optimum number, you must instead configure your Spectrum
Virtualize Host Definitions (and zoning) as though the single host is two or more separate hosts
– During Vdisk assignment, alternate which Vdisk is assigned to one of the “pseudo-hosts”, in a round
robin fashion (a pseudo-host is nothing more than another regular host definition
in the Spectrum Virtualize host config. Each pseudo-host will contain 2
unique host WWPNs, 1 WWPN mapped to each fabric) – Note: A pseudo-host, is not a defined function or feature of the Spectrum Virtualize. If you need to define a pseudo-host,
you are simply adding another host id to the Spectrum Virtualize host config. Instead of creating one host id with 4
WWPNs, you would define 2 hosts with 2 WWPNs. This is now the reference for the term pseudo-host.
– Note: Be careful not to share the Vdisk to more than two adapters per host, so as to not oversubscribe the number of
datapaths per vdisk per host
sSY0333
© Copyright IBM Corporation
2015
General Spectrum Virtualize Best Practice Zoning Concepts
• Dual Core Fabrics require a breakout in cluster zoning to separate
Spectrum Virtualize node ports on core1 from core2
– Prevents/Minimizes IO from spanning/routing across ISLs and Edge switches in a Core/Edge design
• Spectrum Virtualize node ports plugged into dir1 core1 should reside in a separate zone from Spectrum
Virtualize node ports plugged into dir1 core2.
• For MM/GM zones, take advantage of spreading the IO across both cores through careful zoning
• Ensure the Intra-cluster/heartbeat zone is also isolated properly between each fabrics dual cores (don’t span
heartbeat across cores in one fabric)
– Note: - Never make zoning changes on redundant Fabrics at the same time
• Make changes on one fabric and wait 30 min in-between
• Please see the following link for zoning BP’s
– Cisco:
http://g25aciwas09.con.can.ibm.com:9080/ram/assetDetail/generalDetails.faces?guid=5EB2DDF9-7568-368A-7F99-
1BCFFF85E472
– Brocade:
http://g25aciwas09.con.can.ibm.com:9080/ram/assetDetail/generalDetails.faces?guid=57A9649D-A04B-5931-05DA-
F389F697D4D3
sSY0333
96
Correct way to make MM/GM Zone, new implementation 7.1.x
Make the local_fc_port_mask (Node to Node/heartbeat) = 000010001000
Make the partner_fc_port_mask (MM/GM = 000001000100
sSY0333
© Copyright IBM Corporation
2015
New Storage Zoning Schema per Iogrp 12 Port Node
Evolution and Types of Zones – non cluster type
Making 1 zone per Node per Fabric
with the same 6 XIV ports from a
single backend storage unit, will
ensure the max login count of 16 is
not exceeded
Production SAN Fabric
D
STG Zone-1
STG Zone-2
STG Zone-3
STG Zone-4
Production SAN Fabric
C
97
Spectrum Virtualize DH8– 12 FC ports per node
I/O Group 0
Node 1
1 2 3 4
Slot 1
5 6 7 8
Slot 2
Physical
port
number 9 10 11 12
Slot 5
Logical
port with
wwpn #
embedded Node 2
1 2 3 4
Slot 1
5 6 7 8
Slot 2 9 10 11 12
Slot 5
2
2
2
1
2
4
2
3
2
2
2
1
2
4
2
3 5
1
5
2
5
4
5
3
1
2
1
1
1
4
1
3
5
2
5
1
5
4
5
3
1
2
1
1
1
4
1
3
Host/STG
Rep /
Node – Node
97
sSY0333
© Copyright IBM Corporation
2015
Back-end Storage to Spectrum Virtualize Zones
1 storage zone per node/8 for 8 nodes
XIV Storage
SAN Fabric 2
STG Zone4
STG Zone5
STG Zone6
STG Zone7
STG Zone8
98
Module1 HBA1
P
1
P
2
HBA2
P
3
P
4
Module2 HBA1
P
1
P
2
HBA2
P
3
P
4
Module3 HBA1
P
1
P
2
HBA2
P
3
P
4
Module4 HBA1
P
1
P
2
HBA2
P
3
P
4
Module5 HBA1
P
1
P
2
HBA2
P
3
P
4
Module6 HBA1
P
1
P
2
HBA2
P
3
P
4
STG Zone2 SAN Fabric 1 STG
Zone1 STG
Zone3
sSY0333
© Copyright IBM Corporation
2015
Back-end Storage to Spectrum Virtualize Zones
Storage Zone Type – How many Storage zones?
XIV Storage
SAN Fabric 1 SAN Fabric 2
STG Zone3
STG Zone4
STG Zone5
STG Zone6
STG Zone7
STG Zone8
99
Module1 HBA1
P
1
P
2
HBA2
P
3
P
4
Module2 HBA1
P
1
P
2
HBA2
P
3
P
4
Module3 HBA1
P
1
P
2
HBA2
P
3
P
4
Module4 HBA1
P
1
P
2
HBA2
P
3
P
4
Module5 HBA1
P
1
P
2
HBA2
P
3
P
4
Module6 HBA1
P
1
P
2
HBA2
P
3
P
4
STG Zone1
STG Zone2
sSY0333
© Copyright IBM Corporation
2015
XIV Storage to Spectrum Virtualize Zone Example
Storage Zone Type
• Example XIV Storage to Spectrum Virtualize Zoning Ports 1 and 3 only
sSY0333
V7000 Storage-Spectrum Virtualize - 12 Ports
Storage Zone Type
101 Slide provided by Tron Pryor
CAUTION
3
1
4
2
Disconnect allsupply power forcomplete isolation
Disconnect allsupply power forcomplete isolation
4
2
3
1
CAUTION
3
1
4
2
4
2
3
1
CAUTION
Disconnect allsupply power forcomplete isolation
CAUTION
Disconnect allsupply power forcomplete isolation
2
3
1
4
21
1 2
2 1
1
3
2
412
2 14 3 2 14 3
21 4321 43
12
1 2
SAN Fabric 1 SAN Fabric 2
sSY0333
EMC Symmetrix Storage
to Spectrum Virtualize zoning with 12 ports
IBM, EMC Clariion, HP, etc. use one WWNN per subsystem; each appears as a single controller with
multiple ports/WWPNs
• Maximum of 16 ports/WWPNs per WWNN using 1 out of the max of 1024
102
Slide provided by Tron Pryor
Slot 10
Slot 11
Slot 12
Slot 13
Slot 14
Slot 15
Slot 16
Slot 17
A
B
Dir3
A
B
Dir4
A
B
Dir5
A
B
Dir6
A
B
Dir7
A
B
Dir8
A
B
Dir2
A
B
Dir1
BE
BE
FE
FE
FE
FE
FE
FE
A
B
Dir11
A
B
Dir12
A
B
Dir13
A
B
Dir14
A
B
Dir15
A
B
Dir16
A
B
Dir10
A
B
Dir9
FE
FE
FE
FE
FE
FE
BE
BE
Rule of 17
SAN Fabric 1 SAN Fabric 2
sSY0333
HDS USP Storage to Spectrum Virtualize Zones
with 12 ports HDS uses one WWNN per subsystem; each appears as a single controller with multiple ports/WWPNs
• Maximum of 16 ports/WWPNs per WWNN using 1 out of the max of 1024 or 2014 depending on model
P/K CHPxx Port-Label
LOC MPID HardID Port-Label
40 40 2A / 6A
41 41 4A / 8A
42 42 2B / 6B
43 43 4B / 8B
44 44 2C / 6C
45 45 4C / 8C
46 46 2D / 6D
47 47 4D / 8D
CL 2 - REAR CHAs
2Q
P/K CHPxx Port-Label
LOC MPID HardID Port-Label
00 00 1A / 5A
01 01 3A / 7A
02 02 1B / 5B
03 03 3B / 7B
04 04 1C / 5C
05 05 3C / 7C
06 06 1D / 5D
07 07 3D / 7D
CL 1 - FRONT CHAs
1E
SAN Fabric 1 SAN Fabric 2
Slide provided by Tron Pryor
sSY0333
© Copyright IBM Corporation
2015
Spectrum Virtualize Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Even SAN Fabric 13 14 15 16 09 10 11 12
05 06 07 08 01 02 03 04
ODD SAN Fabric 13 14 15 16 09 10 11 12
05 06 07 08 01 02 03 04
CHA P
A B C D E F G H
1P 1Q
CHA P
J K L M N P Q R
2X 2Y
CHA P
A B C D E F G H
2V 2W
CHA P
J K L M N P Q R
1R 1S
Right Controller(2) Left Controller(1)
Hitachi 9980 connected to Spectrum Virtualize
L H P C B F K
Storage Zones Hitachi Example
Storage Zone Type HDS uses one WWNN per subsystem; each appears as a single controller with multiple ports/WWPNs
• Maximum of 16 ports/WWPNs per WWNN using 1 out of the max of 1024 or 2048 depending on model
sSY0333
105
SAN Fabric A SAN Fabric B
Node 3
1 2
3 4
I/O G-1
1 2
3 4
Node 4
Node 1
1 2
3 4
I/O G-0
1 2
3 4
Node 2
MD
isk1
0 / A
rray1
0
MD
isk8
/ Arra
y8
MD
isk7
/ Arra
y7
MD
isk6
/ Arra
y6
MD
isk9
/ Arra
y9
MD
isk5
/ Arra
y5
MD
isk3
/ Arra
y3
MD
isk2
/ Arra
y2
MD
isk1
/ Arra
y1
MD
isk4
/ Arra
y4
MD
isk1
1 / A
rray1
1
MD
isk1
2 / A
rray1
2
MD
isk1
3 / A
rray1
3
MDisk Group 1 / DS5K_1
VDisk 1
VDisk 2
VDisk 3
VDisk 4
SVC
Cluster
Channels 1
and 3
HOST ZONING
Create a SVC/Host zone for each server that receives storage from the SVC cluster.
Example:
Zone Server 1 port A (RED) with all SVC node port 3's.
Zone Server 1 port B (BLUE) with all SVC node port 2's. Zone Server 2 port A (RED) with all SVC node port 1's.
Zone Server 2 port B (BLUE) with all SVC node port 4's.
*** NOTE *** SVC supports a maximum of 256 host objects per I/O group thus a maximum of 1024 per cluster. The above host zoning results in each server being seen by every I/O group and the default host object creation behavior results in each host object counting as one towards this 256 maximum.
To create more then 256 host objects in the cluster you must zone a host to a subset of the I/O groups, you must assign the host object at host creation time to that same subset of I/O groups and then you must assign that host’s VDisks to one of those I/O groups in that same subset.
Server 1
A B
Server 2
A B
SVC ZONING
Create one zone in the RED fabric with all the SVC node ports cabled to Fabric A and create one zone in the BLUE fabric with all the SVC node ports cabled to Fabric B.
Example:
All odd (RED) SVC node ports in one zone and all even (BLUE) SVC node ports in one zone.
Note: For a cluster to be created and to operate correctly all node ports must be zoned together.
STORAGE
ZONING
Create a SVC/Storage zone for each storage subsystem virtualized by the SVC cluster.
Example: Zone DS5K_1 controller A and B daughter card channel ports 1 and 3 with all SVC node ports 1 and 3 in the RED fabric.
Zone DS5K_1 controller A and B daughter card channel ports 2 and 4 with all SVC node ports 2 and 4 in the BLUE fabric.
Cntrl A
Channels 2
and 4
Channels 2
and 4
SVC Cabling
and Zoning
Cntrl B
Channels 1
and 3
Best Practice
Storage to Spectrum Virtualize
Zones DS4k/5K
Storage Zone Type
sSY0333
© Copyright IBM Corporation
2015
Dual Core Fabric
Storage Back-end Problem Scenario • Problem: A single back-end zone had ports p1 and p2 from Spectrum Virtualize 7 and port a3 from XIV (example).
Spectrum Virtualize 7 will form paths from p1 to a3 (purple) and from p3 to a3 (green). Purple paths traverse 2 cores and 1 edge switch – interfering with host edge-to-core traffic. This issue is pervasive. Also, eliminate (unnecessary) separate heartbeat zone.
• Fix: Split zones so that only common-core switch (green paths) can occur. Reduce paths, contention, latency. Also group XIVn array ports attaching common core switch in common zone.
Slide provided by Kirby Dahman
sSY0333
© Copyright IBM Corporation
2015
Dual Core Fabric
Host Front-end Problem Scenario • Problem: Many host HBA ports are in a single zone with Spectrum Virtualize ports cabled to different core
switches. This does not hurt the host directly, but as a side-effect it permits Spectrum Virtualize inter-node cache updates (after each host write) from preferred to non-preferred node in an IOgroup to cross from core-switch-to-edge switch-to-core switch (see purple path).
• Fix: Split zones so that only common-core switch (green paths) can occur for inter-node cache updates.
sSY0333
© Copyright IBM Corporation
2015
Incorrect “Dual Core” fabric zoning example for a 2 node Spectrum
Virtualize Cluster
Slide provided by John Locke
sSY0333
© Copyright IBM Corporation
2015
Correct “Dual Core” fabric zoning example for a 2 node Spectrum
Virtualize Cluster
Slide provided by John Locke
sSY0333
Spectrum Virtualize Cluster zone for a
“Single vs. Dual Core” per fabric
• A Single core cluster “zone” Brocade
Example
– CLI View of one Fabric /8 node Spectrum
Virtualize • zone: lsan_allSpectrum Virtualize_zone_a
• 50:05:07:68:01:10:54:28
• 50:05:07:68:01:40:54:28
• 50:05:07:68:01:10:54:62
• 50:05:07:68:01:40:54:62
• 50:05:07:68:01:10:51:07
• 50:05:07:68:01:40:51:07
• 50:05:07:68:01:10:54:73
• 50:05:07:68:01:40:54:73
• 50:05:07:68:01:10:4e:c3
• 50:05:07:68:01:40:4e:c3
• 50:05:07:68:01:10:45:64
• 50:05:07:68:01:40:45:64
• 50:05:07:68:01:10:4f:67
• 50:05:07:68:01:40:4f:67
• 50:05:07:68:01:10:4f:66
• 50:05:07:68:01:40:4f:66
• A Dual Core cluster “zone” Brocade Example
– CLI View of one Fabric /8 node Spectrum Virtualize “Fabric 1 Core1”
• zone: lsan_allSpectrum Virtualize_zone_a
• 50:05:07:68:01:10:54:28
• 50:05:07:68:01:10:54:62
• 50:05:07:68:01:10:51:07
• 50:05:07:68:01:10:54:73
• 50:05:07:68:01:10:4e:c3
• 50:05:07:68:01:10:45:64
• 50:05:07:68:01:10:4f:67
• 50:05:07:68:01:10:4f:66
– CLI View of one Fabric /8 node Spectrum Virtualize “Fabric Core2”
• 50:05:07:68:01:40:54:28
• 50:05:07:68:01:40:54:28
• 50:05:07:68:01:40:54:62
• 50:05:07:68:01:40:51:07
• 50:05:07:68:01:40:54:73
• 50:05:07:68:01:40:4e:c3
• 50:05:07:68:01:40:45:64
• 50:05:07:68:01:40:4f:67
• 50:05:07:68:01:40:4f:66
110 Slide provided by Chuck Laing
sSY0333
© Copyright IBM Corporation
2015
V9000/DH8 – 12 Ports
Physical to Logical evolution
Understand the physical slot and port positioning
• DH8 - Physical to Logical port numbering for “new builds”.
sSY0333
© Copyright IBM Corporation
2015
DH8 - Physical to Logical Port numbering for “existing” clusters
Spectrum Virtualize DH8 – 12 Ports
Physical to Logical evolution
sSY0333
© Copyright IBM Corporation
2015
Spectrum Virtualize CG8- Port Positioning
Physical to Logical evolution
Logical port #’s reflecting the embedded WWPN
Blue ports go to one Fabric
Red Ports go to the other Fabric
Logical port with wwpn # embedded
Physical port number
Port Layout provided Keith Williams
sSY0333
© Copyright IBM Corporation
2015
Storage to Spectrum Virtualize Zones – 4 Port Node
Evolution and Types of Zones
• Example of Back-end Storage to Spectrum Virtualize Zoning (Cluster Style)
DS8K Right I/O Enclosures
Bay 1
1
3
0
1
3
1
1
3
2
1
3
3
C1R4
1
0
0
1
0
1
1
0
2
1
0
3
C0R2
Bay 3
3
3
0
3
3
1
3
3
2
3
3
3
C1R7
3
0
0
3
0
1
3
0
2
3
0
3
C0R3
Bay 5
5
3
0
5
3
1
5
3
2
5
3
3
C1R8
5
0
0
5
0
1
5
0
2
5
0
3
C0R4
Bay 7
7
3
0
7
3
1
7
3
2
7
3
3
C1R7
7
0
0
7
0
1
7
0
2
7
0
3
C0R3
DS8K Left I/O EnclosuresBay 0
0
0
0
0
0
1
0
0
2
0
0
3
C1L4
0
3
0
0
3
1
0
3
2
0
3
3
C0L2
Bay 2
2
0
0
2
0
1
2
0
2
2
0
3
C1L7
2
3
0
2
3
1
2
3
2
2
3
3
C0L3
Bay 4
4
0
0
4
0
1
4
0
2
4
0
3
C1L8
4
3
0
4
3
1
4
3
2
4
3
3
C0L4
Bay 6
6
0
0
6
0
1
6
0
2
6
0
3
C1L7
6
3
0
6
3
1
6
3
2
6
3
3
C0L3
iogrp 02048 LUNs max
Node 1
HBA 1
P1 P2 P3 P4
HBA 2
Node 2
HBA 1
P1 P2 P3 P4
HBA 2
Node 3
HBA 1
P1 P2 P3 P4
HBA 2
Node 4
HBA 1
P1 P2 P3 P4
HBA 2
iogrp 12048 LUNs max
4 Node SVC MAX Vdisk 4096 wwpn5005076801
DIR1 SAN Fabric DIR2 SAN Fabric
Supported DS8K to SVC ZoningEither ports 1&3 and 2&4 should be
zoned to a fabric or ports as shown on the
next page, both configs are supported
Newer SVC nodes may contain
one HBA card with 4 ports
port1=11052ca port1=11052b7 port1=110529e port1=110528b
port2=12052ca port2=12052b7 port2=120529e port2=120528b
port3=13052ca port3=13052b7 port3=130529e port3=130528b
port4=14052ca port4=14052b7 port4=140529e port4=140528b
sSY0333
© Copyright IBM Corporation
2015
Storage to Spectrum Virtualize Zones - 12 Port Node
Evolution and Types of Zones
• Example excluding node- node and MM/GM ports – 4 Storage zones
Spectrum Virtualize DH8– 12 FC ports per node
I/O Group 0
Node 1
1 2 3 4
Slot 1
5 6 7 8
Slot 2
9 10 11 12
Slot 5
Node 2
1 2 3 4
Slot 1
5 6 7 8
Slot 2 9 10 11 12
Slot 5
2
2
2
1
2
4
2
3
2
2
2
1
2
4
2
3 5
1
5
2
5
4
5
3
1
2
1
1
1
4
1
3
5
2
5
1
5
4
5
3
1
2
1
1
1
4
1
3
SAN Fabric 1 SAN Fabric 2
sSY0333
© Copyright IBM Corporation
2015
VIOS Simplistic Dual Path Overview
sSY0333
© Copyright IBM Corporation
2015
Dual VIOS to Single LPAR Example
H
B
A
1
P1
P2
VHBA1
VP1
VP2
H
B
A
2
P3
P4
VHBA2
VP4
VP3
Spectrum Virtualize
VIO Server1 VIO Server2
1
2
4
3
sSY0333
© Copyright IBM Corporation
2015
N_Port ID Virtualization
c0:50:76:07:0f:5f:00:20 c0:50:76:07:0f:5f:00:30
sSY0333
© Copyright IBM Corporation
2015
Zoning Multi HBA LPARs for AIX VIOS LPM Resiliency • Sys Admins – provide PCI slot to Port WWPN identity to Storage Admins
• Storage Admins – define Spectrum Virtualize pseudo host definitions with active and inactive WWPNs respectively – Avoid single points of hardware failure at the Host HBA, Fabric and Spectrum Virtualize
– Make eight zones, one for each pseudo host per fabric with active and inactive wwpns
SAN Fabric 1 SAN Fabric 2
P
1
2
Active LPAR Fame1
HBA1
P
1
HBA2
P
1
P
2
Spectrum Virtualize defined Pseudo Host1
HBA1
P
1
P
2
HBA2
P
1
Spectrum Virtualize defined Pseudo Host2
P
1
2
Inactive LPAR Fame 2
HBA1
P
1
HBA2
P
1
P
2
Spectrum Virtualize defined Pseudo Host1
HBA1
P
1
P
2
HBA2
P
1
Spectrum Virtualize defined Pseudo Host2
sSY0333
© Copyright IBM Corporation
2015
Zoning Multi HBA LPARs for AIX VIOS LPM Resiliency
• Create a zone to contain the one active Host HBA initiator port to multiple Spectrum Virtualize target ports (shown with solid lines)
• Create another zone for the inactive Host HBA port to exactly the same multiple Spectrum Virtualize target ports as the active counterpart
(dashed lines)
• Separate the Spectrum Virtualize host definitions to contain two active pseudo Host HBA ports in one Spectrum Virtualize host ID and the
other two active pseudo Host HBA ports in another Spectrum Virtualize Host ID.
SAN Fabric 1 SAN Fabric 2
P
1
2
Physical Active Host Fame1
HBA1
P
1
HBA2
P
1
P
2
Spectrum Virtualize defined Pseudo Host1
HBA1
P
1
P
2
HBA2
P
1
Spectrum Virtualize defined Pseudo Host2
P
1
2
Virtual Inactive Host Fame 2
HBA1
P
1
HBA2
P
1
P
2
Spectrum Virtualize defined Pseudo Host1
HBA1
P
1
P
2
HBA2
P
1
Spectrum Virtualize defined Pseudo Host2
a1 a2 a3 a4 i1 i2 i3 i4
sSY0333
© Copyright IBM Corporation
2015
Multiport AIX VIOS HBAs for LPM Resiliency-
- Active Zones & definitions Fabric_red Zone Definitions
Zone 101_a
a1=C0000000c97549a1
N1P1=50050768011052ca
N2P1=50050768011052b7
N3P1=500507680110529e
N4P1=500507680110528b
SAN Fabric 1 SAN Fabric 2
P
1
2
Physical Active Host Fame1
HBA1
P
1
HBA2
P
1
P
2
Spectrum Virtualize defined Pseudo Host1
HBA1
P
1
P
2
HBA2
P
1
Spectrum Virtualize defined Pseudo Host2
P
1
2
Virtual Inactive Host Fame 2
HBA1
P
1
HBA2
P
1
P
2
Spectrum Virtualize defined Pseudo Host1
HBA1
P
1
P
2
HBA2
P
1
Spectrum Virtualize defined Pseudo Host2
a1 a2 a3 a4 i1 i2 i3 i4
Fabric_red Zone Definitions
Zone 101_b
a4=C0000000c97549a4
N1P3=50050768011052cc
N2P3=50050768011052b8
N3P3=500507680110529f
N4P3=500507680110528bc
Fabric_blue Zone Definitions
Zone 102_a
a2=C0000000c97549a2
N1P9=50050768011052dd
N2P9=50050768011052b9
N3P9=500507680110529g
N4P9=500507680110528d
Fabric_blue Zone Definitions
Zone 102_b
a3=C0000000c97549a3
N1P11=50050768011052ee
N2P11=50050768011052r6
N3P11=500507680110529j
N4P11=500507680110528be
Spectrum Virtualize Host Definitions
id:1 name:Pseudo Host_1
C0000000c97549a1
C0000000c97549a2
Spectrum Virtualize Host Definitions
id:2 name: Pseudo Host_2
C0000000c97549a4
C0000000c97549a3
sSY0333
© Copyright IBM Corporation
2015
Multiport AIX VIOS HBAs for LPM -Inactive Zones & Definitions
SAN Fabric 1 SAN Fabric 2
P
1
2
Physical Active Host Fame1
HBA1
P
1
HBA2
P
1
P
2
Spectrum Virtualize defined Pseudo Host1
HBA1
P
1
P
2
HBA2
P
1
Spectrum Virtualize defined Pseudo Host2
P
1
2
Virtual Inactive Host Fame 2
HBA1
P
1
HBA2
P
1
P
2
Spectrum Virtualize defined Pseudo Host1
HBA1
P
1
P
2
HBA2
P
1
Spectrum Virtualize defined Pseudo Host2
a1 a2 a3 i1 i2 i3 i4
Spectrum Virtualize Host Definitions
id:3 name:Pseudo Host_1
C0000000c97549i1
C0000000c97549i2
Spectrum Virtualize Host Definitions
id:4 name: Pseudo Host_2
C0000000c97549i4
C0000000c97549i3
Fabric_red Zone Definitions
Zone 101_a
i1=C0000000c97549i1
N1P1=50050768011052ca
N2P1=50050768011052b7
N3P1=500507680110529e
N4P1=500507680110528b
Fabric_red Zone Definitions
Zone 101_b
i4=C0000000c97549i4
N1P3=50050768011052cc
N2P3=50050768011052b8
N3P3=500507680110529f
N4P3=500507680110528bc
Fabric_blue Zone Definitions
Zone 102_a
i2=C0000000c97549i2
N1P9=50050768011052dd
N2P9=50050768011052b9
N3P9=500507680110529g
N4P9=500507680110528d
Fabric_blue Zone Definitions
Zone 102_b
i3=C0000000c97549i3
N1P11=50050768011052ee
N2P11=50050768011052r6
N3P11=500507680110529j
N4P11=500507680110528be
sSY0333 V9000 Options to choose from
Option 1 – 16Gb Capable
123
sSY0333 V9000 Options to choose from
Option 2 – 8Gb Capable
124
sSY0333
© Copyright IBM Corporation
2015
Types of Zones
Host ESX to Spectrum Virtualize Zones
sSY0333
© Copyright IBM Corporation
2015
Types of Zones
Host ESX to Spectrum Virtualize Zones
2+2 =4 Paths per LUN