ccie-dc-full-scale-labs.pdf

CCIE Data Center Full-Scale Labs - Bootcamp Members - CCIE Data Center Full-Scale Lab 1

CCIE DC Full-Scale Lab 1 TasksThis workbook is not yet compatible with current DC racks for self-paced study.

Introduction1. Data Center Infrastructure2. Data Center Storage Networking3. Unified Computing4. Data Center Virtualization

Introduction

All devices used in this scenario, with the exception of the UCS and Nexus 7K, will be pre-configured for you with a basic initial configuration before starting. Do not modify or remove this initial configuration, such as pre-configured MGMT0 IP addresses, pre-configured VRFs, pre-configured routing, etc. These initial configs are required to successfully complete this scenario.NX-OS device logins are admin with the password Cciedc01. The UCS Management VM's login is Administrator with the password cisco. Do not modify the admin role on any platform, change the console speed, configure AAA, or make any other configuration changes that would potentially lock you out of the CLI interface. Rack rental tokens will not be refunded in cases where configuration errors on your part cause you or the automation system to be locked out of the devices.Pre-configured MGMT0 addresses for this scenario are as follows:

N5K1 192.168.101.51/24N5K2 192.168.101.52/24MDS1 192.168.101.61/24MDS2 192.168.101.62/24N7K1 192.168.101.71/24

Any references to "Y" in this scenario refer to the last octet of the MGMT0 interface.

1. Data Center Infrastructure1.1 UCS Initialization

Connect to UCS Fabric Interconnect A's CLI and use the following options for the initial configuration dialog:

Enforce strong passwords: yesAdmin password: Cciedc01Cluster: yesSwitch fabric: ASystem name: UCS-FIMGMT0 IP address: 192.168.101.201Netmask: 255.255.255.0Default gateway: 192.168.101.1Cluster IP address: 192.168.101.200

Configure UCS FI B to join the cluster and use the IP address 192.168.101.202/24.Enable both Telnet and SSH access to the Fabric Interconnects.

Score: 3 Points

1.2 Nexus 7K VDC InitializationCreate three VDCs on N7K1 as follows:

VDC 2 named N7K2VDC 3 named N7K3VDC 4 named N7K4

Do not inherit the VDC hostname from the default VDC's hostname.Allocate the interfaces to these VDCs according to the diagram. Any unneeded interfaces should be assigned to VDC 0.Connect to these VDCs from the console and configure the admin user with the password Cciedc01.Configure the MGMT0 IP addresses of the VDCs as follows:

VDC 2: 192.168.101.72/24VDC 3: 192.168.101.73/24VDC 4: 192.168.101.74/24

Enable both telnet and SSH access to all VDCs.

Score: 5 Points

1.3 Initial IP AddressingConfigure the higher-numbered M1 port in the diagram between N7K1 and N7K3 as a native layer 3 routed interface using the addresses 10.71.73.Y/24.Configure the M1 ports between N7K2 and N7K4 as layer 3 Port-Channel10. Use LACP for the Port-Channel, and the addresses 10.72.74.Y/24.Configure N7K3 and N7K4's links to the Data Center Interconnect as layer 2 access edge ports in VLANs 1050 and 1051, respectively. Configure interfaces VLAN 1050 and 1051 on N7K3 and N7K4, respectively, with addresses 10.50.73.0/31 and 10.51.74.0/31.

Score: 3 Points

1.4 Layer 3 RoutingConfigure N7K1 and N7K2 to default to N7K3 and N7K4, respectively.

Configure N7K3 and N7K4 to peer BGP with the DCI provider. The provider uses BGP AS 100, whereas N7K3 and N7K4 have been allocated BGP ASes 65001 and 65002, respectively. The DCI provider also requires MD5 authentication using the password DCIPROVIDER.Do not modify any DCI-related configuration on N5K1 or 3750G.When complete, N7K1 and N7K2 should have IP reachability to each other over the DCI.

Score: 5 Points

1.5 FabricPathN5K1 and N7K4 should form Port-Channel20 using LACP on the links connecting them according to the diagram.Configure FabricPath on the port channel as well as the link connecting N7K4 and N5K2 according to the diagram.Create VLANs 200299 as FabricPath VLANs on these switches.Authenticate all FabricPath IS-IS adjacencies using an MD5 hash of the password FPAUTH.

Score: 6 Points

1.6 vPC+Configure UCS-FI-A to form Port-Channel201 up to N5K1 and N5K2 using the links in the diagram.Configure UCS-FI-B to form Port-Channel202 up to N5K1 and N5K2 using the links in the diagram.From N5K1 and N5K2's perspective, these links should be vPC 201 and 202.vPC 201 and 202 should be 802.1Q trunk links, STP edge ports, and only allow VLANs 200299.Use the vPC Domain ID 500 and the FabricPath Switch-ID 501.

Score: 6 Points

1.7 FabricPath Traffic EngineeringEnsure that N7K4 can use both N5K1 and N5K2 to reach their southbound Classical

Ethernet peers in VLANs 200299.

Score: 5 Points

1.8 Spanning-Tree Protocol OptimizationModify N5K1 and N5K2's Classical Ethernet configuration so that they run the minimum number of spanning-tree instances necessary to deliver traffic from the northbound FabricPath domain into the southbound UCS domain.Any new switches that are attached to the Classical Ethernet domain of N5K1 or N5K2 that have a non-zero STP priority should not be able to be elected the STP root bridge.

Score: 6 Points

1.9 Fabric ExtendersN7K3 has two links to each N2K1 and N2K2, which are then used to dual-home to the UCS C200 server. Configure N7K3 to pair with N2K1 and N2K2 as FEX 131 and 132, respectively. Use Port-Channel 131 and 132, respectively.

Score: 5 Points

1.10 OTVConfigure OTV on N7K1 and N7K2 to bridge VLANs 200299 over the Data Center Interconnect.N7K1 should use the Site VLAN and Identifier 3001, and N7K2 should use the Site VLAN and Identifier 3002.Trunk the minimum number of necessary VLANs between N7K1 and N7K3, and N7K2 and N7K4.N7K3 and N7K4 should use PIM Sparse Mode for multicast routing with the DCI, and use the RP address 10.0.0.51, which is hosted by the provider.Multicast Control Plane traffic for the OTV should be tunneled over the DCI using the group 224.71.72.0.Multicast Data Plane traffic originating from N7K1 should use the group range 232.71.71.0/24.Multicast Data Plane traffic originating from N7K2 should use the group range

232.72.72.0/24.Authenticate the IS-IS adjacency between N7K1 and N7K2 using an MD5 hash of the password OTVAUTH.Create Interface VLAN 200 on N7K3 and N7K4 with the IP addresses 192.168.200.Y/24.When complete, N7K3 and N7K4 should be able to ping each other over the DCI through the OTV tunnel, as well as the VMKernel interfaces of the ESXi instances on UCS Blades 1 and 2, and the C200 server. The ESXi addresses are 192.168.200.101, 192.168.200.102, and 192.168.200.104, respectively.

Score: 7 Points

2. Data Center Storage Networking2.1 Fibre Channel Initialization

Configure N5K1, N5K2, UCS-FI-A, and UCS-FI-B's Unified Ports in Fibre Channel mode as shown in the diagram.N5K1's links to MDS1 and N5K2's links to MDS2 should be configured as Port-Channel101 and 102, respectively. The port channels should use dynamic negotiation and be configured as Trunking Expansion ports.N5K1's links to UCS-FI-A and N5K2's links to UCS-FI-B should be configured as Port-Channel 103 and 104, respectively. The port channels should use dynamic negotiation and be configured as non-trunking Fabric ports on the N5K1 and N5K2 sides.

Score: 5 Points

2.2 VSANs and TrunkingThe SAN A side of the UCS blade servers will use VSAN 103, and the SAN B side will use VSAN 104. Internal to UCS, these should map to VLANs 1103 and 1104, respectively.UCS-FI-A's Port-Channel103 to N5K1 and UCS-FI-B's Port-Channe104 to N5K2 should be non-trunking NP ports in VSANs 103 and 104, respectively.N5K1's Port-Channel101 to MDS1 and N5K2's Port-Channel102 to MDS2 should be TE ports that only forward VSANs 103 and 104, respectively.

MDS1 and MDS2's link to the SAN should be F ports in VSANs 103 and 104, respectively.

Score: 6 Points

2.3 Fibre Channel ZoningConfigure Enhanced Zoning and Enhanced Device Aliases on both the SAN A and SAN B sides of the UCS blade server.Device Aliases in SAN A should be configured as follows:

Alias "FC-SAN-A" pWWN 21:00:00:1b:32:04:5e:dcAlias "BLADE1-SAN-A" pWWN 20:00:00:cc:1e:dc:01:0aAlias "BLADE2-SAN-A" pWWN 20:00:00:cc:1e:dc:02:0a

Device Aliases in SAN B should be configured as follows: Alias "FC-SAN-B" pWWN 21:01:00:1b:32:24:5e:dcAlias "BLADE1-SAN-B" pWWN 20:00:00:cc:1e:dc:01:0bAlias "BLADE2-SAN-B" pWWN 20:00:00:cc:1e:dc:02:0b

Configure Zoning for SAN A so that both blades can reach "FC-SAN-A" on the A side.Configure Zoning for SAN B so that both blades can reach "FC-SAN-B" on the B side.Use the minimum amount of zones necessary to accomplish this.

Score: 5 Points

2.4 iSCSI Virtual TargetThe UCS C200 is preconfigured to mount its VMware ESXi Datastores via iSCSI. Configure the network as follows to allow for this.The C200 uses VLAN 202 and the initiator IP address 192.168.202.104/24 for iSCSI, and has the target address configured as 192.168.202.61.The 3750G is preconfigured with VLAN 202 trunking toward N7K3, and an access VLAN 202 assignment toward MDS1.Configure N7K3 so that it trunks only VLAN 202 traffic received from the C200 server toward MDS1.Configure MDS1 so that the C200 server is assigned the pWWN 20:00:00:cc:1e:dc:03:0a.Target LUNs reachable via MDS1's link in VSAN 103 to the FC SAN should be represented with the IQN "iqn.1987-05.com.cisco:05.mds1.01-01.01234567890abcde".

Ensure that the C200 is the only initiator that can use this target.Do not add any additional zones to accomplish this.

Score: 6 Points

3. Unified Computing3.1 Address Pools

Configure default pools in the Root ORG on UCS as follows: UUIDs 0000-000000000001 - 0000-000000000080MAC Addresses 00:CC:1E:DC:00:01 00:CC:1E:DC:00:FFnWWNs 20:01:00:CC:1E:DC:01:01 - 20:01:00:CC:1E:DC:01:FFManagement IPs 192.168.101.210 - 192.168.101.219 (GW 192.168.101.1)

Score: 5 Points

3.2 UCS Service Profile TemplatesCreate a Service Profile Initial Template that will be used for Blades 1 and 2 called PROFILE.UUIDs, MAC Addresses, nWWNs, and Management IPs should be pulled from the previously created default pools.For SAN connectivity, there should be two vHBAs, fc0 on SAN A using VSAN 103, and fc1 on SAN B using VSAN 104.For LAN connectivity, create five vNICs as follows:

vNIC0 named VMKernelA on Fabric A in VLAN 200vNIC1 named VMKernelB on Fabric B in VLAN 200vNIC2 named vMotion on Fabric B in VLAN 201vNIC3 named VMGuestsA on Fabric A with VLANs 202 - 210vNIC4 named VMGuestsB on Fabric B with VLANs 202 - 210

Ensure that if FI-B loses upstream connectivity that the vMotion NIC does not lose reachability to the rest of the network.If a change in this service profile in the future requires re-association to apply the change, ensure that the administrator is notified before the blade is automatically rebooted.

Score: 6 Points

3.3 Service ProfilesCreate two Service Profiles from the previously created template called PROFILE1 and PROFILE2 for Blade 1 and Blade 2, respectively.PROFILE1 should be customized as follows:

Assign vHBA FC0 the pWNN 20:00:00:cc:1e:dc:01:0a.Assign vHBA FC1 the pWNN 20:00:00:cc:1e:dc:01:0b.Boot to LUN 0 on the SAN target 21:00:00:1b:32:24:5e:dc via FC0 as the primary, and then to LUN 0 on the SAN target 21:01:00:1b:32:24:5e:dc via FC1 if booting via FC0 fails.

PROFILE2 should be customized as follows: Assign vHBA FC0 the pWNN 20:00:00:cc:1e:dc:02:0a.Assign vHBA FC1 the pWNN 20:00:00:cc:1e:dc:02:0b.Boot to LUN 0 on the SAN target 21:01:00:1b:32:24:5e:dc via FC1 as the primary, and then to LUN 0 on the SAN target 21:00:00:1b:32:24:5e:dc via FC0 if booting via FC1 fails.

Associate PROFILE1 to Blade 1 and PROFILE2 to Blade 2. If successful, the blades should boot their ESXi instances from the SAN.

Score: 6 Points

4. Data Center Virtualization4.1 Nexus 1000v

Nexus 1000v VSMs are pre-installed on the ESXi instances for Blade 1 and Blade 2. The VSM's MGMT0 IP address is 192.168.200.200, and it has a login of admin with the password Cciedc01.Modify the existing N1Kv configuration so that the VEM on Blade 1's ESXi host (192.168.200.101) appears as module 10.The VEM on Blade 2's ESXi host (192.168.200.102) should appear as module 20.The C200's ESXi host (192.168.200.104) should dynamically choose any available VEM slot.

Score: 5 Points

4.2 Private VLANsVirtual Machines (VMs) Win2k8-www-1 through 6 are preconfigured with IP addresses 192.168.255.1 through 6, and they have a pre-defined port-group on the Nexus 1000v. These VMs can be reached through the VMware Console of the vSphere Client and have the username/password combination Administrator/Cciedc01.Create Interface VLAN 204 on N7K3 with the IP address 192.168.255.73/24.Configure Private-VLANs in such a way that all VMs can ping N7K3's VLAN 204 interface, but cannot ping each other.Do not make changes to any other devices besides the Nexus 1000v and N7K3 to accomplish this, including the vCenter server.

Score: 5 Points

CCIE Data Center Full-Scale Labs - Bootcamp Members - CCIE Data Center Full-Scale Lab 1

CCIE DC Full-Scale Lab 1 Solutions1. Data Center Infrastructure2. Data Center Storage Networking3. Unified Computing4. Data Center Virtualization

1. Data Center Infrastructure1.1 UCS InitializationConfiguration

UCS-FI-A:

Enter the configuration method. (console/gui) ?console

Enter the setup mode; setup newly or restore from backup. (setup/restore) ?setup

You have chosen to setup a new Fabric interconnect. Continue? (y/n):y

Enforce strong password? (y/n) [y]:y

Enter the password for "admin":Cciedc01

Confirm the password for "admin":Cciedc01

Is this Fabric interconnect part of a cluster(select 'no' for standalone)? (yes/no) [n]:yes

Enter the switch fabric (A/B) []:A

Enter the system name:UCS-FI

Physical Switch Mgmt0 IPv4 address :192.168.101.201

Physical Switch Mgmt0 IPv4 netmask :255.255.255.0

IPv4 address of the default gateway :192.168.101.1

Cluster IPv4 address :192.168.101.200

Configure the DNS Server IPv4 address? (yes/no) [n]:

Configure the default domain name? (yes/no) [n]:

Following configurations will be applied:

Switch Fabric=A

System Name=UCS-FI

Enforced Strong Password=yes

Physical Switch Mgmt0 IP Address=192.168.101.201

Physical Switch Mgmt0 IP Netmask=255.255.255.0

Default Gateway=192.168.101.1

Cluster Enabled=yes

Cluster IP Address=192.168.101.200

NOTE: Cluster IP will be configured only after both Fabric Interconnects are initialized

Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no):yes

Applying configuration. Please wait.

Configuration file - Ok

UCS-FI-B:

Enter the configuration method. (console/gui) ?console

Installer has detected the presence of a peer Fabric interconnect. This Fabric interconnect will be added to the cluster. Continue (y/n) ?

y

Enter the admin password of the peer Fabric interconnect:Cciedc01

Connecting to peer Fabric interconnect... done

Retrieving config from peer Fabric interconnect... done

Peer Fabric interconnect Mgmt0 IP Address: 192.168.101.201

Peer Fabric interconnect Mgmt0 IP Netmask: 255.255.255.0

Cluster IP address : 192.168.101.200

Physical Switch Mgmt0 IPv4 address :192.168.101.202

Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no):yes

Applying configuration. Please wait.

Configuration file - Ok

Like Nexus, UCS allows SSH access by default. Telnet can be enabled from the UCSM GUI, or from the CLI as follows.

UCS-FI-A#scope system

UCS-FI-A /system #scope services

UCS-FI-A /system/services #enable telnet-server

UCS-FI-A /system/services* #commit-buffer

UCS-FI-A /system/services #end

UCS-FI-A#exit

1.2 Nexus 7K VDC InitializationConfiguration

First remove all interfaces from the default VDC by allowing only F2 ports. This will force all M1 and F1 ports to be allocated to VDC 0:

N7K1#config t

N7K1(config)#feature telnet

N7K1(config)#vdc N7K1

N7K1(config-vdc)#limit-resource module-type f2

This will cause all ports of unallowed types to be removed from this vdc. Continue (y/n)? [yes]yes

N7K1(config-vdc)#show vdc membership

vdc_id: 0 vdc_name: Unallocated interfaces:

Ethernet1/1 Ethernet1/2 Ethernet1/3










Ethernet1/31 Ethernet1/32












vdc_id: 1 vdc_name: N7K1 interfaces:

Now change the default VDC back to allow both M1 and F1 ports, create the other

VDCs, and allocate the needed ports.

N7K1(config)#no vdc combined-hostname

N7K1(config)#vdc N7K1

N7K1(config-vdc)#limit-resource module-type m1 f1 m1xl

This will cause all ports of unallowed types to be removed from this vdc. Continue (y/n)? [yes]yes

N7K1(config-vdc)#allocate interface Ethernet1/1-8

Moving ports will cause all config associated to them in source vdc to be removed. Are you sure you want to move the ports (y/n)? [yes]

yes

N7K1(config-vdc)#vdc N7K2 id 2

Note: Creating VDC, one moment please ...

N7K1 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 2 has come online N7K1(config-vdc)#allocate interface Ethernet1/25-32


yes



yes





yes



yes





yes

N7K1(config-vdc)#allocate interface Ethernet2/5-8,Ethernet2/13-14,Ethernet2/19-20


yes

N7K1(config-vdc)#end

Now "switchto" the VDCs to configure the admin password as well as the MGMT0 IP address.

N7K1#switchto vdc N7K2

---- System Admin Account Setup ----

Do you want to enforce secure password standard (yes/no) [y]:y



---- Basic System Configuration Dialog VDC: 2 ----

This setup utility will guide you through the basic configuration of

the system. Setup configures only enough connectivity for management

of the system.

Please register Cisco Nexus7000 Family devices promptly with your

supplier. Failure to register may affect response times for initial

service calls. Nexus7000 devices must be registered to receive

entitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime

to skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): n

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.

The copyrights to certain works contained in this software are

owned by other third parties and used and distributed under

license. Certain components of this software are licensed under

the GNU General Public License (GPL) version 2.0 or the GNU

Lesser General Public License (LGPL) Version 2.1. A copy of each

such license is available at

http://www.opensource.org/licenses/gpl-2.0.php and

http://www.opensource.org/licenses/lgpl-2.1.php N7K2#config t

Enter configuration commands, one per line. End with CNTL/Z. N7K2(config)#feature telnet

N7K2(config)#interface mgmt0

N7K2(config-if)#ip address 192.168.101.72/24

N7K2(config-if)#end

N7K2#switchback









of the system.


















http://www.opensource.org/licenses/lgpl-2.1.php N7K3#conf t


N7K3(config)#int mgmt0


N7K3(config-if)#end

N7K3#switchback









of the system.


















http://www.opensource.org/licenses/lgpl-2.1.php N7K4#config t


N7K4(config)#interface mgmt 0


N7K4(config-if)#end

N7K4#switchback

N7K1#copy running-config startup-config vdc-all

[####### ] 17%

[############ ] 29%

[###################### ] 53%

[############################ ] 69%

[##################################### ] 90%

[########################################] 100%

Verification

N7K1#show vdc membership

vdc_id: 0 vdc_name: Unallocated interfaces:





















Ethernet2/24








Some interfaces not listed on the diagram must still be allocated to VDCs 1 - 4 due to the port-group boundaries. Port-groupings can be verified as shown below.

N7K1#show interface capabilities | include "Ethernet|Group"Ethernet1/1 Port Group Members: 1,3,5,7

Ethernet1/2 Port Group Members: 2,4,6,8

Ethernet1/3

Port Group Members: 1,3,5,7

Ethernet1/4


Ethernet1/5


Ethernet1/6


Ethernet1/7


Ethernet1/8


1.3 Initial IP AddressingConfiguration

N7K1:

interface Ethernet1/2

ip address 10.71.73.71/24

no shutdown

N7K2:

feature lacp

!


channel-group 10 mode active

no shutdown

!



no shutdown

!

interface port-channel10

ip address 10.72.74.72/24

N7K3:

feature interface-vlan

!

vlan 1050

!


ip address 10.71.73.73/24

no shutdown

!


switchport access vlan 1050

spanning-tree port type edge

no shutdown

!

interface Vlan1050

no shutdown

ip address 10.50.73.0/31

N7K4:

feature interface-vlan

!

feature lacp

!

vlan 1051

!



no shutdown

!



no shutdown

!


ip address 10.72.74.74/24

!


switchport access vlan 1051

spanning-tree port type edge

no shutdown

!

interface Vlan1051

no shutdown

ip address 10.51.74.0/31

Verification

N7K2#show port-channel summary

Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)

s - Suspended r - Module-removed

S - Switched R - Routed

U - Up (port-channel)

M - Not in use. Min-links not met

--------------------------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------------------------

10 Po10(RU) Eth LACP Eth1/25(P) Eth1/26(P)

N7K2#show ip route direct

IP Route Table for VRF "default"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%' in via output denotes VRF

10.72.74.0/24

, ubest/mbest: 1/0, attached *via 10.72.74.72, Po10

, [0/0], 21:47:09, direct

N7K2#ping 10.72.74.72

PING 10.72.74.72 (10.72.74.72): 56 data bytes

64 bytes from 10.72.74.72: icmp_seq=0 ttl=255 time=0.597 ms





--- 10.72.74.72 ping statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss

round-trip min/avg/max = 0.295/0.422/0.597 ms

1.4 Layer 3 RoutingConfiguration

N7K1:

ip route 0.0.0.0/0 10.71.73.73

N7K2:

ip route 0.0.0.0/0 10.72.74.74

N7K3:

feature bgp

!

router bgp 65001

address-family ipv4 unicast

network 10.71.73.0/24

neighbor 10.50.73.1

remote-as 100

password 0 DCIPROVIDER


N7K4:

feature bgp

!

router bgp 65002

log-neighbor-changes


network 10.72.74.0/24

neighbor 10.51.74.1

remote-as 100

password 0 DCIPROVIDER


Verification

N7K3#show ip bgp neighbors

BGP neighbor is 10.50.73.1, remote AS 100, ebgp link, Peer index 1

BGP version 4, remote router ID 10.0.0.50

BGP state = Established, up for 21:47:57

Peer is directly attached, interface Vlan1050 TCP MD5 authentication is enabled

N7K3#show bgp ipv4 unicast summary

BGP summary information for VRF default, address family IPv4 Unicast

BGP router identifier 10.71.73.73, local AS number 65001

BGP table version is 8, IPv4 Unicast config peers 1, capable peers 1

4 network entries and 4 paths using 496 bytes of memory

BGP attribute entries [4/512], BGP AS path entries [2/16]

BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.50.73.1 4 100 1301 1310 8 0 0 21:45:52 3

N7K3#show bgp ipv4 unicast

BGP routing table information for VRF default, address family IPv4 Unicast

BGP table version is 8, local router ID is 10.71.73.73

Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best

Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist

Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath

Network Next Hop Metric LocPrf Weight Path

*>e10.0.0.50/32 10.50.73.1 0 0 100 i

*>e10.0.0.51/32 10.50.73.1 0 100 i

*>l10.71.73.0/24 0.0.0.0 100 32768 i *>e10.72.74.0/24

10.50.73.1 0 100 65002 i

N7K3#show ip route bgp

IP Route Table for VRF "default"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%' in via output denotes VRF

10.0.0.50/32, ubest/mbest: 1/0

*via 10.50.73.1, [20/0], 21:46:03, bgp-65001, external, tag 100

10.0.0.51/32, ubest/mbest: 1/0

*via 10.50.73.1, [20/0], 21:46:03, bgp-65001, external, tag 100 10.72.74.0/24

, ubest/mbest: 1/0 *via 10.50.73.1, [20/0], 21:45:14, bgp-65001

, external, tag 100

N7K2#ping 10.71.73.71

PING 10.71.73.71 (10.71.73.71): 56 data bytes








1.5 FabricPathConfiguration

N5K1:

install feature-set fabricpath

feature-set fabricpath

feature lacp

!

vlan 200-299

mode fabricpath

!

key chain FABRICPATH

key 1

key-string 0 FPAUTH

!


switchport

switchport mode fabricpath

fabricpath isis authentication-type md5

fabricpath isis authentication key-chain FABRICPATH

!




no shutdown

!




no shutdown

N5K2:



!

vlan 200-299

mode fabricpath

!


key 1

key-string 0 FPAUTH

!





no shutdown

N7K1:


N7K4:


!

vlan 200-299

mode fabricpath

!


key 1

key-string 0 FPAUTH

!


switchport




!





no shutdown

!




no shutdown

!




no shutdown

Verification

N7K4#show port-channel summary






M - Not in use. Min-links not met

--------------------------------------------------------------------------------


Channel

--------------------------------------------------------------------------------

10 Po10(RU) Eth LACP Eth1/17(P) Eth1/18(P)

20 Po20(SU) Eth LACP Eth2/7(P) Eth2/13(P)

N7K4#show fabricpath isis adjacency Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:System ID SNPA Level State Hold Time Interface

N5K1 N/A 1 UP 00:00:24 port-channel20

N5K2 N/A 1 UP 00:00:29 Ethernet2/6

N7K4#show fabricpath isis interface port-channel 20

Fabricpath IS-IS domain: default

Interface: port-channel20

Status: protocol-up/link-up/admin-up

Index: 0x0002, Local Circuit ID: 0x01, Circuit Type: L1 Authentication type MD5

Authentication keychain is FABRICPATH

Authentication check specified

Extended Local Circuit ID: 0x16000013, P2P Circuit ID: 0000.0000.0000.00

Retx interval: 5, Retx throttle interval: 66 ms

LSP interval: 33 ms, MTU: 1500

P2P Adjs: 1, AdjsUp: 1, Priority 64 Hello Interval: 10, Multi: 3, Next IIH: 00:00:04

Level Adjs AdjsUp Metric CSNP Next CSNP Last LSP ID 1 1 1 20 60 00:00:55 ffff.ffff.ffff.ff-ff

Topologies enabled:

Topology Metric MetricConfig Forwarding

0 20 no UP

1.6 vPC+Configuration

N5K1:

feature vpc

!

vpc domain 500

peer-keepalive destination 192.168.101.52

fabricpath switch-id 501

!




no shutdown

!




no shutdown

!


switchport mode trunk

switchport trunk allowed vlan 200-299


no shutdown

!





no shutdown

!



vpc peer-link



!




spanning-tree port type edge trunk

vpc 201

!





vpc 202

N5K2:

feature vpc

feature lacp

!

vpc domain 500

peer-keepalive destination 192.168.101.51

fabricpath switch-id 501

!




no shutdown

!




no shutdown

!





no shutdown

!





no shutdown

!



vpc peer-link



!





vpc 201

!





vpc 202

Connect to the UCSM using the credentials that you previously configured. Next, under the Fabric Interconnects on the Equipment tab, configure the Ethernet links connecting northbound to the N5Ks in the diagram as Uplink Ports.

Now under the LAN tab, create and enable Port-Channels 201 and 202 on FI-A and FI-B respectively.

Verification

N5K1# show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 500

vPC+ switch id : 501

Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive

vPC fabricpath status : peer is reachable through fabricpath

Configuration consistency status: success

Per-vlan consistency status : success

Type-2 consistency status : success

vPC role : secondary

Number of vPCs configured : 2

Peer Gateway : Disabled

Dual-active excluded VLANs : -

Graceful Consistency Check : Enabled

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ -------------------------------------------------- 1 Po500 up 200-299

vPC status

---------------------------------------------------------------------------

id Port Status Consistency Reason Active vlans vPC+ Attrib

-- ---------- ------ ----------- ------ ------------ -----------

201 Po201 up success success 200-299 DF: Partial

202 Po202 up success success 200-299 DF: Partial

UCS-FI-A:

UCS-FI-A#connect nxos

UCS-FI-A(nxos)#show run interface ethernet 1/4 - 5


description U: Uplink

pinning border


switchport trunk allowed vlan 1,200-299


no shutdown


description U: Uplink

pinning border


switchport trunk allowed vlan 1,200-299


no shutdown

UCS-FI-A(nxos)#show port-channel summary






--------------------------------------------------------------------------------


Channel

--------------------------------------------------------------------------------

201 Po201(SU) Eth LACP Eth1/4(P) Eth1/5(P)

1.7 FabricPath Traffic EngineeringConfiguration

N7K4:


fabricpath isis metric 40

Verification

N5K1 and N5K2 share the emulated FabricPath Switch-ID 501 for the vPC+, as shown below:

N7K4#show fabricpath switch-id

FABRICPATH SWITCH-ID TABLE

Legend: '*' - this system

=========================================================================

SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED

----------+----------------+------------+-----------+--------------------

501 547f.ee79.137c Primary Confirmed No Yes

501 547f.ee7a.4d7c Primary Confirmed No Yes

*645 64a0.e742.8dc4 Primary Confirmed No No

1207 547f.ee79.137c Primary Confirmed No No

3550 547f.ee7a.4d7c Primary Confirmed No No

Total Switch-ids: 5

The port channel between N7K4 and N5K1 has an IS-IS metric of 20, whereas the single 10GigE link from N7K4 to N5K2 has an IS-IS metric of 40. This means that the shortest path from N7K4 to Switch-ID 501 (the vPC+ pair) is only via N5K1.

N7K4#show fabricpath route

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]

ftag 0 is local ftag

subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/645/0, number of next-hops: 0

via ---- , [60/0], 0 day/s 22:19:30, local 1/501/0, number of next-hops: 1

via Po20, [115/20]

, 0 day/s 20:30:58, isis_fabricpath-default


via Po20, [115/40], 0 day/s 20:30:58, isis_fabricpath-default

via Eth2/6, [115/40], 0 day/s 22:19:16, isis_fabricpath-default



To allow for Equal Cost Multipath (ECMP), the port channel to N5K1 and the single link to N5K2 must have equal costs. This can be configured either by raising the cost of the port channel or by lowering the cost of the link to N5K2.

N7K4#config t

Enter configuration commands, one per line. End with CNTL/Z. N7K4(config)#interface port-channel20

N7K4(config-if)#fabricpath isis metric 40

N7K4(config-if)# end

Now Switch-ID 501 is reachable via both N5K1 and N5K2 with a metric of 40.

N7K4#show fabricpath route

FabricPath Unicast Route Table

'a/b/c' denotes ftag/switch-id/subswitch-id

'[x/y]' denotes [admin distance/metric]

ftag 0 is local ftag

subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default


via ---- , [60/0], 0 day/s 22:19:58, local 1/501/0, number of next-hops: 2

via Po20, [115/40]

, 0 day/s 20:31:26, isis_fabricpath-default via Eth2/6, [115/40]

, 0 day/s 00:00:06, isis_fabricpath-default


via Eth2/6, [115/40], 0 day/s 22:19:44, isis_fabricpath-default



1.8 Spanning-Tree Protocol OptimizationConfiguration

N5K1:

spanning-tree mode mst

spanning-tree mst 0 priority 0

spanning-tree mst configuration

name MST0

revision 1

N5K2:

spanning-tree mode mst

spanning-tree mst 0 priority 0

spanning-tree mst configuration

name MST0

revision 1

Verification

In the below output, we can see that both N5K1 and N5K2 have collapsed all of their STP instances into the single default MST0 instance. Additionally, both switches in the vPC+ pair should always appear as the root of the Spanning-Tree, and share the Bridge-ID c84c.75fa.6000. Note that Spanning-Tree only forwards southbound toward the Classical Ethernet domain, and not northbound toward the FabricPath domain.

N5K1#show spanning-tree mst 0

##### MST0 vlans mapped: 1-4094

Bridge address c84c.75fa.6000 priority 0 (0 sysid 0)

Root this switch for the CIST

Regional Root this switch

Operational hello time 2 , forward delay 15, max age 20, txholdcount 6

Configured hello time 2 , forward delay 15, max age 20, max hops 20

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po201 Desg FWD 200 128.4296 (vPC) Edge P2p


N5K2#show spanning-tree mst 0

##### MST0 vlans mapped: 1-4094

Bridge address c84c.75fa.6000 priority 0 (0 sysid 0)

Root this switch for the CIST

Regional Root this switch

Operational hello time 2 , forward delay 15, max age 20, txholdcount 6

Configured hello time 2 , forward delay 15, max age 20, max hops 20

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------



Eth1/11 Desg FWD 20000 128.139 P2p Bound(PVST)

1.9 Fabric Extenders

N7K1:

install feature-set fex

N7K3:

feature-set fex

!


switchport

switchport mode fex-fabric

fex associate 131

!


switchport


fex associate 132

!


switchport


fex associate 131

channel-group 131

no shutdown

!interface Ethernet1/14

switchport


fex associate 131

channel-group 131

no shutdown

!


switchport


fex associate 132

channel-group 132

no shutdown

!


switchport


fex associate 132

channel-group 132

no shutdown

!

interface Ethernet131/1/1

switchport



no shutdown

!

interface Ethernet132/1/1

switchport



no shutdown

Verification

N7K3#show fex

FEX FEX FEX FEX

Number Description State Model Serial

------------------------------------------------------------------------

131 FEX0131 Online

N2K-C2232PP-10GE FOC17100NHX 132 FEX0132 Online

N2K-C2232PP-10GE FOC17100NHU

N7K3#show fex detail

FEX: 131 Description: FEX0131 state: Online

FEX version: 6.0(2) [Switch version: 6.0(2)]

FEX Interim version: 6.0(2.9)

Switch Interim version: 6.0(2)

Extender Model: N2K-C2232PP-10GE, Extender Serial: FOC17100NHX

Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:00:42, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/14

Fabric interface state: Po131 - Interface Up. State: Active

Eth1/13 - Interface Up. State: Active


Fex Port State Fabric Port Eth131/1/1 Up Po131

FEX: 132 Description: FEX0132 state: Online

FEX version: 6.0(2) [Switch version: 6.0(2)]

FEX Interim version: 6.0(2.9)

Switch Interim version: 6.0(2)

Extender Model: N2K-C2232PP-10GE, Extender Serial: FOC17100NHU

Part No: 73-12533-05

Card Id: 82, Mac Addr: f0:29:29:ff:02:02, Num Macs: 64

Module Sw Gen: 12594 [Switch Sw Gen: 21]

pinning-mode: static Max-links: 1

Fabric port for control traffic: Eth1/15

Fabric interface state: Po132 - Interface Up. State: Active



Fex Port State Fabric Port Eth132/1/1 Up Po132

1.10 OTVConfiguration

The OTV Site VLAN is in decimal, but the OTV Site Identifier is in hex, which means that a decimal to hex conversion is required.

N7K1:

feature otv

!

vlan 200-299,3001

!

otv site-vlan 3001

otv site-identifier 0xbb9

!

spanning-tree vlan 3001 priority 0

!

key chain OTV

key 1

key-string 0 OTVAUTH

!

interface Overlay1

otv isis authentication-type md5

otv isis authentication key-chain OTV

otv join-interface Ethernet1/2 otv control-group 224.71.72.0

otv data-group 232.71.71.0/24

otv extend-vlan 200-299

no shutdown

!


switchport


switchport trunk allowed vlan 200-299,3001

no shutdown

!


ip igmp version 3

N7K2:

feature otv

!

vlan 200-299,3002

!

otv site-vlan 3002

otv site-identifier 0xbba

!

key chain OTV

key 1

key-string 0 OTVAUTH

!


ip igmp version 3

!

interface Overlay1

otv isis authentication-type md5

otv isis authentication key-chain OTV

otv join-interface port-channel10 otv control-group 224.71.72.0

otv data-group 232.72.72.0/24

otv extend-vlan 200-299

no shutdown

!




no shutdown

N7K3:

feature pim

!

vlan 200-299,3001

!

interface Vlan200

no shutdown

ip address 192.168.200.73/24

!

interface Vlan1050

ip pim sparse-mode

!


switchport



no shutdown

!


ip pim sparse-mode

ip igmp version 3

!

ip pim rp-address 10.0.0.51 group-list 224.0.0.0/4

ip pim ssm range 232.0.0.0/8

N7K4:

feature pim

!

vlan 3002

!

spanning-tree vlan 200-299 priority 0

!

interface Vlan200

no shutdown

ip address 192.168.200.74/24

!

interface Vlan1051

ip pim sparse-mode

!


ip pim sparse-mode

ip igmp version 3

!




no shutdown

!

ip pim rp-address 10.0.0.51 group-list 224.0.0.0/4

ip pim ssm range 232.0.0.0/8

Verification

To establish the OTV tunnel, the AEDs must have multicast reachability to each other with the control group. The first step in verification, then, is to ensure that the tree for the control multicast group is built in the DCI core. Both N7K3 and N7K4 should see the (S,G) entries for the control group 224.71.72.0.

N7K3#show ip mroute

IP Multicast Routing Table for VRF "default"

(*, 224.71.72.0/32), uptime: 00:11:06, igmp ip pim

Incoming interface: Vlan1050, RPF nbr: 10.50.73.1

Outgoing interface list: (count: 1)

Ethernet1/10, uptime: 00:11:06, igmp

(10.71.73.71/32, 224.71.72.0/32)

, uptime: 00:12:45, ip pim mrib Incoming interface: Ethernet1/10

, RPF nbr: 10.71.73.71 Outgoing interface list

: (count: 2)

Ethernet1/10, uptime: 00:11:06, mrib, (RPF) Vlan1050

, uptime: 00:12:34, pim

(10.72.74.72/32, 224.71.72.0/32)

, uptime: 00:11:03, ip mrib pim Incoming interface: Vlan1050


: (count: 1) Ethernet1/10

, uptime: 00:11:03, mrib

(*, 232.0.0.0/8), uptime: 00:12:54, pim ip

Incoming interface: Null, RPF nbr: 0.0.0.0


N7K4#show ip mroute


(*, 224.71.72.0/32), uptime: 00:13:47, igmp ip pim

Incoming interface: Vlan1051, RPF nbr: 10.51.74.1


port-channel10, uptime: 00:13:47, igmp

(10.71.73.71/32, 224.71.72.0/32)

, uptime: 00:13:39, ip mrib pim Incoming interface: Vlan1051


: (count: 1) port-channel10

, uptime: 00:13:39, mrib

(10.72.74.72/32, 224.71.72.0/32)

, uptime: 00:13:44, ip mrib pim Incoming interface: port-channel10


: (count: 2) Vlan1051

, uptime: 00:12:18, pim

port-channel10, uptime: 00:13:44, mrib, (RPF)

(*, 232.0.0.0/8), uptime: 00:13:53, pim ip

Incoming interface: Null, RPF nbr: 0.0.0.0


Ensure that the Site VLAN is up on both AEDs.

N7K1#show otv

OTV Overlay Information

Site Identifier 0000.0000.0bb9

Overlay interface Overlay1

VPN name : Overlay1

VPN state : UP

Extended vlans : 200-299 (Total:100)

Control group : 224.71.72.0

Data group range(s) : 232.71.71.0/24

Join interface(s) : Eth1/2 (10.71.73.71) Site vlan : 3001 (up)

AED-Capable : Yes

Capability : Multicast-Reachable

N7K2#show otv

OTV Overlay Information

Site Identifier 0000.0000.0bba

Overlay interface Overlay1

VPN name : Overlay1

VPN state : UP

Extended vlans : 200-299 (Total:100)

Control group : 224.71.72.0

Data group range(s) : 232.72.72.0/24

Join interface(s) : Po10 (10.72.74.72) Site vlan : 3002 (up)

AED-Capable : Yes

Capability : Multicast-Reachable

Now the AEDs should be able to form an IS-IS adjacency over the OTV tunnel.

N7K1#show otv isis adjacency OTV-IS-IS process: default VPN: Overlay1

OTV-IS-IS adjacency database:System ID SNPA Level State Hold Time Interface Site-ID

N7K2 64a0.e742.8dc2 1 UP 00:00:08 Overlay1 0000.0000.0bba

Verify that MD5 authentication for IS-IS is enabled on the Overlay1 interface.

N7K1#show otv isis interface overlay 1

OTV-IS-IS process: default VPN: Overlay1

Overlay1, Interface status: protocol-up/link-up/admin-up

IP address: none

IPv6 address: none

IPv6 link-local address: none

Index: 0x0001, Local Circuit ID: 0x01, Circuit Type: L1

Level1

Adjacency server (local/remote) : disabled / none Adjacency server capability : multicast Authentication type is MD5Authentication keychain is OTV

Authentication check specified

LSP interval: 33 ms, MTU: 1400

Level Metric CSNP Next CSNP Hello Multi Next IIH

1 40 10 Inactive 10 3 00:00:03

Level Adjs AdjsUp Pri Circuit ID Since 1 1 1 64 N7K2.01 00:15:55

N7K3 and N7K4 should now be able to reach each other's VLAN 200 interfaces, and the OTV AEDs should learn the routes to these MAC addresses.

N7K4#show interface vlan 200 | include ddress Hardware is EtherSVI, address is 64a0.e742.8dc4

Internet Address is 192.168.200.74/24

N7K3#ping 192.168.200.74

PING 192.168.200.74 (192.168.200.74): 56 data bytes








N7K1#show otv route

OTV Unicast MAC Routing Table For Overlay1

VLAN MAC-Address Metric Uptime Owner Next-hop(s)

---- -------------- ------ -------- --------- -----------

200 000c.29bb.9b82 42 00:18:25 overlay N7K2

200 64a0.e742.8dc3 1 00:18:15 site Ethernet1/1

200 64a0.e742.8dc4 42 00:18:14 overlay N7K2

200 d48c.b5bd.460c 1 00:18:23 site Ethernet1/1

N7K2#show otv route

OTV Unicast MAC Routing Table For Overlay1

VLAN MAC-Address Metric Uptime Owner Next-hop(s)

---- -------------- ------ -------- --------- -----------

200 000c.29bb.9b82 1 00:19:03 site Ethernet2/3

200 64a0.e742.8dc3 42 00:18:24 overlay N7K1

200 64a0.e742.8dc4 1 00:18:24 site Ethernet2/3

200 d48c.b5bd.460c 42 00:18:32 overlay N7K1

Multicast tunneling can be verified by joining a multicast group on one of the switches and then sending ICMP pings from the remote OTV site. If successful, a new OTV multicast tunnel should form using the OTV multicast data groups.

N7K3#config t

Enter configuration commands, one per line. End with CNTL/Z. N7K3(config)#interface vlan 200

N7K3(config-if)#ip pim sparse-mode

N7K3(config-if)#ip igmp join-group 224.1.1.1N7K4#ping multicast 224.1.1.1 interface vlan 200

PING 224.1.1.1 (224.1.1.1): 56 data bytes






--- 224.1.1.1 ping multicast statistics ---

5 packets transmitted, From member 192.168.200.73: 5 packets received, 0.00% packet loss

--- in total, 1 group member responded ---

N7K3#show ip mroute 232.72.72.0


(10.72.74.72/32, 232.72.72.0/32)

, uptime: 00:02:44, igmp ip pim Incoming interface: Vlan1050


: (count: 1) Ethernet1/10

, uptime: 00:02:44, igmp

2. Data Center Storage Networking2.1 Fibre Channel InitializationConfiguration

N5K1:

feature fcoe

feature npiv

feature fport-channel-trunk

!

slot 1

port 28-32 type fc

!

interface fc1/28

channel-group 101

no shutdown

!

interface fc1/29

channel-group 101

no shutdown

!

interface fc1/30

switchport mode F

switchport trunk mode off

channel-group 103

no shutdown

!

interface fc1/31

switchport mode F


channel-group 103

no shutdown

!

interface san-port-channel 101

channel mode active

!


channel mode active

switchport mode F


N5K2:

feature fcoe

feature npiv

feature fport-channel-trunk

!

slot 1

port 28-32 type fc

!

interface fc1/28

channel-group 102

no shutdown

!

interface fc1/29

channel-group 102

no shutdown

!

interface fc1/30

switchport mode F


channel-group 104

no shutdown

!

interface fc1/31

switchport mode F


channel-group 104

no shutdown

!


channel mode active

!


channel mode active

switchport mode F


MDS1:

interface fc1/3

channel-group 101

no shutdown

!

interface fc1/4

channel-group 101

no shutdown

!

interface port-channel 101

channel mode active

MDS2:

interface fc1/3

channel-group 102

no shutdown

!

interface fc1/4

channel-group 102

no shutdown

!


channel mode active

In UCSM, go to the Equipment tab, and then, under the Fabric Interconnects, go to Configure Unified Ports. Just like on the 5Ks, changing the port type from Ethernet to Fibre Channel requires a reboot, so to save time, start with FI-B first, and then configure FI-A.

When the FIs have rebooted, go to the SAN tab and configure FC uplinks on FI-A and FI-B as SAN-Port-Channels 103 and 104, respectively. Remember to enable the port channels when created, because like on the 5Ks, they are in the shutdown state when created.

Verification

Changing Unified Port types between Ethernet and Fibre Channel requires a reload of the Nexus 5000 or the UCS Fabric Interconnect on which the change was made.

N5K2#config t

Enter configuration commands, one per line. End with CNTL/Z. N5K2(config)#feature fcoe

FC license checked out successfully

fc_plugin extracted successfully

FC plugin loaded successfully

FCoE manager enabled successfully

FC enabled on all modules successfully

Enabled FCoE QoS policies successfully N5K2(config)#feature npiv

N5K2(config)# ! N5K2(config)#slot 1

N5K2(config-slot)# port 28-32 type fc

N5K2(config-slot)#end

N5K2#copy running-config startup-config

[########################################] 100%

Copy complete, now saving to disk (please wait)...

N5K2# reload

WARNING: This command will reboot the system Do you want to continue? (y/n) [n]y

Shutdown Ports..

writing reset reason 9,

When the SAN port channels are configured, you may need to flap the links for the port channels to come up, as shown below.

N5K2#show san-port-channel database

san-port-channel 102

Last membership update is successful

2 ports in total, 2 ports up

First operational port is fc1/28

Age of the port-channel is 0d:00h:10m:14s Ports: fc1/28 [up] *

fc1/29 [up]




Age of the port-channel is 0d:00h:10m:14s Ports: fc1/30 [down]

fc1/31 [down]

N5K2#conf t

Enter configuration commands, one per line. End with CNTL/Z. N5K2(config)#int san-port-channel 104

N5K2(config-if)#shut

N5K2 %$ VDC-1 %$ %PORT-5-IF_DOWN_ADMIN_DOWN: %$VSAN 1%$ Interface san-port-channel 104 is down (Administratively down) N5K2(config-if)#no shut

N5K2(config-if)#end

N5K2 %$ VDC-1 %$ %PORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: %$VSAN 1%$ Interface san-port-channel 104 is down (No operational members) N5K2 %$ VDC-1 %$ Apr 6 20:48:00 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=1,rctl:0x23,type:0x1,oxid 0x6,rxid:0x1f7 - kernelN5K2 %$ VDC-1 %$ Apr 6 20:48:00 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=1,rctl:0x23,type:0x1,oxid 0x7,rxid:0x1f8 - kernelN5K2 %$ VDC-1 %$ %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on console0N5K2 %$ VDC-1 %$ Apr 6 20:48:10 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=1,rctl:0x23,type:0x1,oxid 0xe,rxid:0x204 - kernelN5K2 %$ VDC-1 %$ Apr 6 20:48:20 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=1,rctl:0x84,type:0x0,oxid 0xd,rxid:0x1fe - kernelN5K2 %$ VDC-1 %$ %PORT-5-IF_UP: %$VSAN 1%$ Interface san-port-channel 104 is up in mode F

N5K2 %$ VDC-1 %$ Apr 6 20:48:30 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=1,rctl:0x23,type:0x1,oxid 0x23,rxid:0x225 - kernelN5K2#show san-port-channel database






fc1/29 [up]





Age of the port-channel is 0d:00h:11m:15s Ports: fc1/30 [up]

fc1/31 [up] *

On the UCS side, the SAN port channels are configured in Proxy Node Port (NP) mode, for Node Port Virtualizer (NPV), or in other words, Fibre Channel End Host Mode.


UCS-FI-A(nxos)#show run interface fc1/31 - 32

interface fc1/31 switchport mode NP

channel-group 103 force

no shutdown

interface fc1/32 switchport mode NP

channel-group 103 force

no shutdown

UCS-FI-A(nxos)#show run interface san-port-channel 103


channel mode active switchport mode NP

UCS-FI-A(nxos)#show san-port-channel database






fc1/32 [up]

2.2 VSANs & TrunkingConfiguration

N5K1:

vsan database

vsan 103

vsan 103 interface san-port-channel 103

!


switchport trunk allowed vsan 103

N5K2:

vsan database

vsan 104

vsan 104 interface san-port-channel 104

!



MDS1:

vsan database

vsan 103

vsan 103 interface fc1/7

!

interface fc1/7

no shutdown

!



MDS2:

vsan database

vsan 104

vsan 104 interface fc1/7

!

interface fc1/7

no shutdown

!



UCS-FI-A:


UCS-FI-A(nxos)#show run | section "vsan database"vsan database

vsan 103

UCS-FI-A(nxos)#show run | section "vlan 1104"vlan 1103

fcoe vsan 103

name fcoe-vsan-1103

UCS-FI-B:

UCS-FI-B#connect nxos

UCS-FI-B(nxos)#show run | section "vsan database"vsan database

vsan 104

UCS-FI-B(nxos)#show run | section "vlan 1104"

vlan 1104

fcoe vsan 104

name fcoe-vsan-1104

In UCSM, browse to the SAN tab, and then, under SAN Cloud, right-click VSANs to create new VSANs. Ensure that VSAN 103 is on the Fabric A side and VSAN 104 is on the Fabric B side.

To assign the VSANs to the SAN-Port-Channels, go back to the SAN tab, and under SAN Cloud, right-click the appropriate FC interface and click Show Navigator. SAN-Port-Channel 103 should be in VSAN 103, and Port-Channel 104 should be in VSAN 104.

Verification

When all the VSANs are created and assigned, check N5K1 and N5K2 to ensure

that the UCS FIs have performed a Fabric Login (FLOGI) on the SAN port channel interfaces.

N5K1#show flogi database vsan 103

--------------------------------------------------------------------------------

INTERFACE VSAN FCID PORT NAME NODE NAME

--------------------------------------------------------------------------------

San-po103 103 0xbc0000 24:67:00:2a:6a:15:66:80 20:67:00:2a:6a:15:66:81

Total number of flogi = 1.

N5K2#show flogi database vsan 104

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

San-po104 104 0x6e0000 24:68:00:2a:6a:15:05:00 20:68:00:2a:6a:15:05:01


On MDS1 and MDS2, ensure that the Fibre Channel SAN has performed FLOGI.

MDS1#show flogi database vsan 103

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

fc1/7 103 0x0d0000 21:00:00:1b:32:04:5e:dc 20:00:00:1b:32:04:5e:dc


MDS2#show flogi database vsan 104

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

fc1/7 104 0xaa0000 21:01:00:1b:32:24:5e:dc 20:01:00:1b:32:24:5e:dc


Both N5K1 and MDS1 on the SAN A side and N5K2 and MDS2 on the SAN B side should agree on the Fibre Channel Name Service (FCNS) database. This verifies that both the initiators and targets are logged in and have been assigned Fibre Channel Identifiers (FCIDs) and that VSAN trunking in the fabric is end to end.

N5K1#show fcns database vsan 103

VSAN 103:

--------------------------------------------------------------------------

FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE

--------------------------------------------------------------------------

0x0d0000 N 21:00:00:1b:32:04:5e:dc (Qlogic)

0xbc0000 N 24:67:00:2a:6a:15:66:80 (Cisco) npv

Total number of entries = 2

N5K2#show fcns database vsan 104

VSAN 104:

--------------------------------------------------------------------------

FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE

--------------------------------------------------------------------------

0x6e0000 N 24:68:00:2a:6a:15:05:00 (Cisco) npv

0xaa0000 N 21:01:00:1b:32:24:5e:dc (Qlogic)


2.3 Fibre Channel ZoningConfiguration

N5K1:

device-alias mode enhanced

device-alias database

device-alias name FC-SAN-A pwwn 21:00:00:1b:32:04:5e:dc

device-alias name BLADE1-SAN-A pwwn 20:00:00:cc:1e:dc:01:0a


!

device-alias commit

!

zone mode enhanced vsan 103

!

zone name VSAN_103_ZONE vsan 103

member device-alias FC-SAN-A

member device-alias BLADE1-SAN-A

member device-alias BLADE2-SAN-A

!

zoneset name VSAN_103_ZONESET vsan 103

member VSAN_103_ZONE

!

zoneset activate name VSAN_103_ZONESET vsan 103

zone commit vsan 103

N5K2:

device-alias mode enhanced


device-alias name FC-SAN-B pwwn 21:01:00:1b:32:24:5e:dc

device-alias name BLADE1-SAN-B pwwn 20:00:00:cc:1e:dc:01:0b

device-alias name BLADE2-SAN-B pwwn 20:00:00:cc:1e:dc:02:0b

!

device-alias commit

!

zone mode enhanced vsan 104

!


member device-alias FC-SAN-B

member device-alias BLADE1-SAN-B

member device-alias BLADE2-SAN-B

!


member VSAN_104_ZONE

!



Verification

Devices on the SAN A side should agree on the Device Alias database and zoneset for VSAN 103.

MDS1#show device-alias status

Fabric Distribution: Enabled

Database:- Device Aliases 3 Mode: Enhanced

Checksum: 0x252e3d5059933b2826cabfe0ee148

MDS1#show device-alias database

device-alias name FC-SAN-A pwwn 21:00:00:1b:32:04:5e:dc




MDS1#show zone status vsan 103

VSAN: 103 default-zone: deny distribute: active only Interop: default mode: enhanced

merge-control: allow

session: none

hard-zoning: enabled broadcast: enabled

Default zone:

qos: none broadcast: disabled ronly: disabled

Full Zoning Database :

DB size: 224 bytes

Zonesets:1 Zones:1 Aliases: 0 Attribute-groups: 1 Active Zoning Database

:

DB size: 148 bytes Name: VSAN_103_ZONESET Zonesets:1 Zones:1

Status: Activation completed at 20:55:21 UTC May 26 2013

MDS1 learned the zoning configuration applied on N5K1, but it does not yet see an FCID for the UCS blades. This is because we haven't configured the service profiles for the blades, which means they're not yet logged in to the fabric. When the SP association is complete, we should see the FCIDs of the blades get dynamically assigned, as well as the pWWNs we manually configure on them logged in to the fabric.

MDS1#show zoneset active vsan 103



* fcid 0x0d0000 [device-alias FC-SAN-A]

device-alias BLADE1-SAN-A


2.4 iSCSI Virtual TargetConfiguration

N7K3:



switchport trunk allowed vlan 202

no shutdown

MDS1:


device-alias name UCS-C200-SAN-A pwwn 20:00:00:cc:1e:dc:03:0a

!

device-alias commit

!

feature iscsi

iscsi enable module 1

!

vsan database

vsan 103 interface iscsi1/1

!

iscsi virtual-target name iqn.1987-05.com.cisco:05.mds1.01-01.01234567890abcde

pWWN 21:00:00:1b:32:04:5e:dc

initiator ip address 192.168.202.104 permit

!

iscsi initiator ip-address 192.168.202.104

static pWWN 20:00:00:cc:1e:dc:03:0a

!


member device-alias UCS-C200-SAN-A

!



!

interface GigabitEthernet1/1

ip address 192.168.202.61 255.255.255.0

no shutdown

!

interface iscsi1/1

no shutdown

Verification

When the iSCSI configuration is complete, MDS1 should see the UCS C200 server log in as an iSCSI Initiator. The nWWN can be dynamic, but because zoning and LUN Masking on the SAN is done based on the pWWN, this needs to be manually assigned to the iSCSI Initiator.

MDS1#show iscsi initiator

iSCSI Node name is 192.168.202.104

iSCSI Initiator name: iqn.1998-01.com.vmware:localhost-7463f71b

iSCSI alias name:

Configured node (iSCSI)

Node WWN is 21:01:00:0d:ec:4a:21:02 (dynamic)

Member of vsans: 103

Number of Virtual n_ports: 1 Virtual Port WWN is 20:00:00:cc:1e:dc:03:0a (configured)

Interface iSCSI 1/1, Portal group tag: 0x3000

VSAN ID 103, FCID 0x0d0100

From the iSCSI Initiator's point of view, the MDS is an iSCSI Target. Note that only the C200's IP address is allowed to use this target.

MDS1#show iscsi virtual-target

target: iqn.1987-05.com.cisco:05.mds1.01-01.01234567890abcde

* Port WWN 21:00:00:1b:32:04:5e:dc

Configured node (iSCSI)

No. of initiators permitted: 1 initiator 192.168.202.104/32 is permitted

All initiator permit is disabled

Trespass support is disabled

Revert to primary support is disabled

MDS1 should see the C200 server registered to the fabric in the FLOGI database.

MDS1#show flogi database

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

fc1/7 103 0x0d0000 21:00:00:1b:32:04:5e:dc 20:00:00:1b:32:04:5e:dc

[FC-SAN-A] iscsi1/1 103 0x0d0100 20:00:00:cc:1e:dc:03:0a

21:01:00:0d:ec:4a:21:02 [UCS-C200-SAN-A]


Adding the C200's pWWN to the already defined zone for VSAN 103 will allow it access to the LUNs that the SAN is presenting for this initiator.

MDS1#show zoneset active



* fcid 0x0d0000 [device-alias FC-SAN-A]


device-alias BLADE2-SAN-A * fcid 0x0d0100 [device-alias UCS-C200-SAN-A]

The final verification for this task is to ensure that the ESXi instance has actually

mounted the iSCSI LUNs. To check this, go to the vSphere client, select the C200 host on the left, click the Configuration tab, and then click Storage Adapters. Under the iSCSI Software Adapter, you should see the LUNs appear as shown below.

3. Unified Computing3.1 Address PoolsUUID Pools in UCSM are configured under the Servers tab, Pools, then UUID Suffix Pools, as shown below.

MAC Address Pools are under the LAN tab, Pools, then MAC Pools.

Node World Wide Name Pools are under the SAN tab, Pools, then WWNN Pools.

Management IP Address Pools are under the Admin tab, Communication Management, then Management IP Pool. Note that the default gateway here is arbitrary, because the task did not ask for a specific value, but it is still a required field.

3.2 UCS Service Profile Templates

Create a new Service Profile Template under the Servers tab, then Service Profile Templates. The task requires that this be an Initial Template and get its addresses from the default pools that were created in the previous task.

Under Storage, ensure that the vHBAs are assigned to VSANs 103 and 104 on Fabric A and Fabric B, respectively.

For vNICs, use the Expert option, and add the five new vNICs according to the task requirements. The VLANs needed are created in this step to save time, but could also be configured as a separate step under the LAN Cloud.

Ensure that the vMotion vNIC has Fabric Failover enabled according to the task requirements.

The vNICs for the VMGuests are trunks that carry the rest of the VLANs.

The Maintenance Policy is where we define that the administrator must acknowledge a change that would cause the blade to reboot.

The Operational Policies define where the Management IP addresses of the Service Profiles come from.

3.3 Service ProfilesTo assign the service profiles, we must first enable the southbound links from the

FIs to the Blade Chassis. To do so, configure them as Server ports under the Fabric Interconnects on the Equipment tab.

Create two copies of the Service Profile Template previously created.

Before we customize the boot options for the individual service profiles, a QoS policy is created that will apply to the vHBAs. Note that this is just for clarity of the configuration, so that we know for certain that the vHBAs are being assigned to a no-drop QoS policy.

Modify the vHBAs to have the appropriate pWWNs according to the task. Note that if these values are incorrect, the blades will fail to boot from the SAN, because the LUN masking on the SAN only allows specific initiating pWWNs to access their LUNs.

We need to create a Boot Policy that tells the blade which SAN target it needs to boot to.

Again, ensure 100% accuracy, because an incorrect pWWN value will cause the blade to be unable to boot.

Repeat the above steps, but now for the backup boot target.

Don't forget to actually assign the Boot Policy to the service profile after it is successfully created.

Repeat the above steps for the second service profile that will be assigned to blade 2.

Finally, associate the service profiles to the blades.

When the blades begin to boot, you can track their progress by connecting to their KVMs. When the blades are fully booted, you should see the console screen for the ESXi instances, as shown below.

4. Data Center Virtualization4.1 Nexus 1000v

Configuration

First we need to determine which UUIDs were dynamically assigned to the blades, and which VEMs they are currently inserted as. The below output shows us the module number (VEM number), the UUID, and the IP address.

N1Kv#show module

Mod Ports Module-Type Model Status

--- ----- -------------------------------- ------------------ ------------

1 0 Virtual Supervisor Module Nexus1000V active *

2 0 Virtual Supervisor Module Nexus1000V ha-standby

4 248 Virtual Ethernet Module NA ok



Mod Sw Hw

--- ------------------ ------------------------------------------------

1 4.2(1)SV2(1.1) 0.0

2 4.2(1)SV2(1.1) 0.0

4 4.2(1)SV2(1.1) VMware ESXi 5.1.0 Releasebuild-799733 (3.1)



Mod MAC-Address(es) Serial-Num

--- -------------------------------------- ----------

1 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA

2 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA

4 02-00-0c-00-04-00 to 02-00-0c-00-04-80 NA

5 02-00-0c-00-05-00 to 02-00-0c-00-05-80 NA

6 02-00-0c-00-06-00 to 02-00-0c-00-06-80 NA

Mod Server-IP Server-UUID Server-Name

--- --------------- ------------------------------------ --------------------

Date post:	08-Nov-2015
Category:	Documents
Upload:	naveedrana
View:	35 times
Download:	4 times

ccie-dc-full-scale-labs.pdf

Documents