© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS Storage Integration, Technologies, and Topologies
Craig Ashapa, Technical Marketing Engineer
BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
Cisco UCS 3rd Generation Fabric Interconnect
FC/FCoE Uplink ConnectivitySAN Uplinks, FCoE Multi-Hop, Brocade, Hyper-VFC/FCoE
Modes of Operation, Direct Connectivity of Storage, Local Zoning
IP-Based Storage
Storage Profiles
UCS C3000 Series
Monitoring, Troubleshooting, Advanced CLI
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco UCS 3rd Generation Fabric Interconnect
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FI 6332• 32 x 40GbE QSFP+ ports
• 2.56Tbps switching performance
• 1RU fixed form factor, two power supplies & four fans
FI 6332-16UP• 24 x 40GbE QSFP+ & 16 x UP ports (1/10GbE or 4/8/16G FC)
• 2.43Tbps switching performance
• 1RU fixed form factor, two power supplies & four fans
IOM 2304• 8 x 40GbE server links & 4 x 40GbE QSFP+ uplinks
• 960Gbps switching performance
• Modular IOM for UCS 5108
UCS FI & IOM ModelsFI 6300 Series and IOM 2304
BRKCOM-2007 5
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FI 6300 Series Hardware Overview
FI 6332 (Front View) FI 6332-16UP (Front View)
26 x 40G QSFP+ *
or
98 x 10G SFP+ **
6 x 40G QSFP+
* QSA module required on ports 13-14 to provide 10G support
** Requires QSFP to 4xSFP breakout cable
18 x 40G QSFP+
Or
72 x 10G SFP+ *
6 x 40G QSFP+16 x UP
16 x 1/10G SFP+
or
16 x 4/8/16G FC
2 x Hot Swappable
Power Supplies (1+1)
4 x Hot Swappable
Fans (3+1)
6300 Series
(Rear View) Serial Ports
L1 & L2
High
avail
ports
L1 & L2
High
avail
ports
BRKCOM-2007 6
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
6332 Switch on Chip (SOC)
Storage Protocols
FCoE
iSCSI
Ethernet
40G only ports (Uplink Ports)
Storage Protocols
FCoE
iSCSI
Ethernet
6332-16UP Unified Port Controller (UPC)
Storage Protocols
FC
FCoE
iSCSI
Ethernet
Switch on Chip (SOC)
Same as 6332
40G only ports (Uplink Ports)
Same as 6332
6332 & 6332-16UP ASICs/Storage Protocols
BRKCOM-2007 7
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
6332-16UP ASIC Port Mapping
6332 ASIC Port Mapping
UPC SOC 40G Only
40G OnlySOC
BRKCOM-2007 8
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
6332/6332-16UP QSFP Breakout Ports
Each SOC QSFP Can be Breakout Port
Breakout Port Creates (4) 10Gb Ports
Port Modes Same as on Non-Breakout Ports
Each Port of Breakout Port Configured Individually
Change to/from Breakout Port Requires FI Reboot
Note: If 2nd No-Drop Class Configured with Jumbo MTU, QSFP
Breakout Ports limited to 4 (ie, 16 individual 10Gb ports) -
Remainder of QSFP Ports will be Non-Breakout 40Gb ports.
BRKCOM-2007 9
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
ASIC Type Server
Port
Network
Uplink
Port
FC
Uplink
Port
FCoE
Uplink
Port
FC
Storage
Port
FCoE
Storage
Port
Appliance
Port
Unified
Port
UPC
6332-16UP
SOC
6332-16UP/6332
40G Only
6332-16UP/6332
Port Type vs. ASIC Type
Port Types vs. ASIC Type(Green Means Qualified)
BRKCOM-2007 10
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
6332-16UP UP Port Groups
16 Unified SFP Ports
Configured as 6/6/4 Port Groups
FI Reboots Upon Change
All Groups FC/All Ethernet/ Or Mix
UCSM Configure Unified Ports
Slider - FC Left / Ethernet Right6 FC / 12 Ethernet
BRKCOM-2007 11
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Interface
• NIF
• 4 x 40G QSFP
• Connects only to FI63xx
• HIF
• 32 Interfaces
• Support 10G or 4 ports can combine to a single 40G
2304 IO Module
Templeton ASIC
Internal backplane ports to blades
No Local Switching – ever!
Traffic goes up to FI
Fabric Ports to FI63xx
1 32
BRKCOM-2007 12
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
`
VIC1340
23X
X-B
Mezz 1
(empty)
23X
X-A
CPU # 1 CPU # 0
Active KR Lane
Blade Server
PCIe Lanes
QPI
PCIe Lanes
20G (2x10G)
IOM 2304 and Adapter ConnectionVIC1340 Only
BRKCOM-2007 13
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
`
VIC1340
23X
X-B
Port Expander
Card
23X
X-A
Blade Server
CPU # 1 CPU # 0
PCIe Lanes
QPI
PCIe Lanes
40G (native)
2304 IOM and Adapter ConnectionVIC1340 Plus Port Expander
BRKCOM-2007 14
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Adapter Resiliency
• 2 independent Adapters
• vCon placement
• 4 20G connections
• 20G are 2x10
• 6 tuple hashing`
VIC1340
23X
X-B
VIC1380
23X
X-A
Blade Server
CPU # 1 CPU # 0
PCIe Lanes
QPI
PCIe Lanes
2304 IOM and Adapter ConnectionVIC1340 Plus VIC1380
BRKCOM-2007 15
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FC/FCoE Uplink Connectivity,SAN Uplinks, FCoE Multi-Hop, Brocade, Hyper-V
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Block Storage Uplink Versatility
Fibre Channel (FC) Up to 16Gb FC Connectivity with UCS Gen3
Cisco MDS
Cisco Nexus N5k
Brocade (See Cisco HCL)
Fibre Channel over Ethernet (FCoE) Up to 40Gb FCoE Connectivity with Gen3
Cisco MDS
97xx with 48-Port 10/40 Gbps FCoE Modules
9250i Multiservice Fabric Switch - 10 Gbps FCoE
Cisco Nexus
N5k - 10/40 Gbps FCoE
N7k with F-Series 10/40 Gbps Modules
17BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FC/FCoE Port Trunking / Channeling
Port Trunking
Multiple VSANs per Link
Help consolidate Infrastructure
vHBAs can be on different VSANs
All VSANs trunked on all FC/FCoE uplinks – No Pruning
Scalability Per UCS System:
Max of 32 VSANs (62xx/6332/6324)
Max of 15 VSANs (6332-16UP)
Port Channeling
Up to 16 FC/FCoE ports can be aggregated together in a
single FC/FCoE port channel (62xx//6332)
VSANs can be trunked over the port channel
Port Trunking/Channeling in NPV and FC Switch mode
Trunking/Channeling is not available for direct connect
FC/FCoE Storage Port types
18BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
How FC/FCoE Trunking is Enabled
Per UCS Fabric Setting
Default is Not enabled
Enabling Flaps Ports
Maintenance Window
End Host Mode – TNP
FC Switch Mode – TE
Don’t Forget Northbound Device Configs!
19BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Hop FCoE
End-to-End 10/40Gb FCoE
MDS, N5K, N7K FCoE Capable Switches Supported Upstream
“Unified Uplink” Port Type
Carries FCoE and Normal Ethernet Traffic
Fabric Interconnect in NPV Mode Primary use Case
VNP Ports
FI in FC Switching Mode
VE Ports also Supported
MDS/N5K/N7K
Unified Uplinks
UCS B-Series
UCS FI UCS FI
FCoE STORAGE
MDS/N5K/N7K
NPV/EHM
FCoE FCoE
FCoE/Ethernet FCoE/Ethernet
20BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Hop FCoE
Pure FCoE or Converged FCoE/Ethernet Uplinks Northbound Switch Dependent
FCoE Uplinks Individual
Port Channels (62xx/6332): Max 12, Max 16 Members Per)
FCoE Uplink Trunking/Port Channels Trunking is Global
All VSANs Available on All Uplinks
FCoE Uplink Port is a “Border Port” NPV Mode – NP/TNP
FC Switching Mode – E/VE
21BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Hop FCoENorthbound Connectivity Comparison
MDS N5k N7k
FCoE Uplinks
Converged Uplinks N/A
Converged vPC N/A
FCoE Uplinks Carry Pure FCoE Storage Traffic
FCoE Uplinks Individual or Port-Channel
Converged Uplinks Carry FCoE AND Ethernet LAN traffic
Converged Uplinks Individual or Port-Channel
22BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS to Brocade FC Connectivity
Brocade NPIV Mode / Cisco UCS End-Host (NPV) Mode
Link Aggregation Not Possible
Cisco – Port Channeling / Brocade – Port Trunking
Cisco Trunking and VSANs Northbound of FIs Not Supported
See Cisco HCL for Latest Supported Brocade FOS Versions
http://www.cisco.com/en/US/products/ps10477/prod_technical_reference_list.html
Cisco to Brocade Traffic Engineering
Common VSANs
SAN Pin Groups
Unique VSANs
See Cisco UCS to Brocade Connectivity Guide for Additional Information
http://www.cisco.com/en/US/prod/collateral/ps10265/ps10281/whitepaper_C07-730016.pdf
23BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS – Hyper-V Synthetic Fibre Channel
SAN-ANPIV Mode
UCS FI-ANPV Mode
vHBA – UCS Virtual HBAsHBA – Hyper-V Synthetic HBA
sHBA1
vHBA1
vHBA2
sHBA3
sHBA5
sHBA7
sHBA2
sHBA4
sHBA6
sHBA8
Hyper-V Server 2012
VM1
VM2
VM3
VM4
UCS FNIC NPIV Enabled
SAN-BNPIV Mode
UCS FI-BNPV Mode
sHBA WWPNs: C0:03:FF:XX:XX:XX:XX:XX
vHBA WWPNs: 20:00:00:25:b5:XX:XX:XX
20:00:00:25:b5:XX:XX:XXC0:03:FF:XX:XX:XX:XX:XX
UCS FNIC NPIV Enabled
24BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS – Hyper-V Synthetic Fibre Channel
Benefits Security – FC LUN Masked Direct to VM
Multi-Pathing – Individual Multi-Pathing Policies per VM
Shared Clustered FC storage
Live Migration Resiliency – Uninterrupted FC Access via Microsoft’s Synthetic HBAs
Details NPIV on the UCS FNIC
Enabled by Default, Not Configurable
Requires NPIV Enabled Upstream SAN (Pre-UCSM 3.1.2)
Supports Windows Server 2008, 2008R2, 2012, 2012R2 VMs
4 Microsoft Synthetic HBAs per VM Maximum
128 Maximum NPIV Ports per Hyper-V Server Example: Microsoft Hyper-V (32) virtual machines each with (4) Virtual Fibre Channel
Adapters = 128 NPIV ports (128 is the maximum)
25BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS – Hyper-V Synthetic Fibre Channel
Caveats Not supported with UCS Local Zoning (Pre-UCSM 3.1.2)
No Persistent Binding
No Boot from SAN from VM via Microsoft Synthetic FC Adapters
26BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Modes of Operation, Direct Connectivity of Storage,Port Types, Local Zoning
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28BRKCOM-2007
Operation Mode for FC/FCoE
End-Host (NPV) Mode - Default
UCS Functions as Node Port (initiator)
Required for Connecting FC to Non-MDS FC Switches
FC Switching Mode
Upstream MDS or Nexus FC Switch Supported
Required for UCS Local Zoning Feature
Direct Connect from Fabric Interconnect to FC/FCoE Storage Target
Operation Mode for Ethernet/iSCSI/NAS
End-Host Mode - Default
Appliance Ports which allow Direct Connect Ethernet/iSCSI/NAS Storage Targets
Ethernet Switch
No Storage Based Reasons to use this Mode
Operation Mode vs. Features
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Four UCS port types related to direct storage capability
Storage FC Port – direct connect this port to FC port on array
Storage FCoE Port – direct connect this port to FCoE port on array
Appliance Port - direct connect this port to 10G/40G Ethernet port on array
Unified Storage Port (Combined Appliance and FCoE) – direct connect this port to NetApp Unified Target Adapter on array
UCS Direct Connect Storage
29BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Port Types vs. FI Operating Mode (Green Means Valid in Mode of Operation)
FI Operational Role Server
Port
Uplink
Port
Uplink
FC Port
FCoE
Uplink
Port
FC
Storage
Port
FCoE
Storage
Port
Appliance
Port
Unified
Port
Ethernet: EHM
FC: EHM
Ethernet: EHM
FC: Switching
Ethernet: Switching
FC: EHM
Ethernet: Switching
FC: Switching
FI Mode vs. Valid Port Type in that Mode
Valid Port Types Available with UCS 2.1 and Above Release
End Host Mode is the Default Mode for the Fabric Interconnects
30BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Local Zoning OR Upstream Zoning
UCS Configured in FC Switch Mode
Ethernet and FC Switching modes are Independent
Hybrid Topology - Upstream Zoning Provided by MDS/Nxk Switches
See UCS HCL for Supported Arrays
Support to Directly Connect FC/ FCoE Storage to Fabric Interconnects
End to End FC/FCoE Topologies Possible
Lower Cost Point for Small Deployments (no Access Layer FC/FCoE Switches Required)
Customer benefits
Feature details
FCoE Storage FC Storage
Direct Connection of FC/FCoE Storage
UCS B-Series
UCS 62xx/63xx UCS 62xx/63xx
31BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FC/FCoE Local Zoning
Direct connect FC/FCoE storage to FI with noupstream SAN switch
FIs in FC Switch Mode
Implicit zone creation No need to create zones
manually
Single-Initiator-Single-Target and Multi-Target Zones
Migrating from Upstream to Local Zoning
CLI Commands to Remove Residual Upstream Zoning
FCoE Storage FC Storage
UCS B-Series
UCS 62xx/63xx UCS 62xx/63xx
32BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCSM User Defined Zoning
New to UCSM 3.1.2
Custom Zoning Creation without Upstream FC Switch FC Target to FC Target Zoning
Replication
Microsoft Hyper-V Virtual Fibre Channel Zoning of Microsoft Synthetic HBAs to UCS
Direct Attached Storage
Live Migration
New FC Zone Profile Construct Logical Container for FC Zones
Each Zone can be in Different VSAN
Enable/Disable FC Zone Profile Enables/Disables all Zones within that Profile
33BRKCOM-2007
sHBA to FC Storage
Target to Target
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCSM User Defined Zoning
34BRKCOM-2007
CC25-Antwerp-A# scope fc-storage
CC25-Antwerp-A /fc-storage # show fc-zone-profile
FC Zone Profile:
Name Descr Admin State Oper State
---------- ---------- ----------- -----------------------
hyper-v-zone Enabled Ok
NetApp_Repl Enabled Ok
CC25-Antwerp-A /fc-storage # scope fc-zone-profile NetApp_Repl
CC25-Antwerp-A /fc-storage/fc-zone-profile # show fc-user-zone detail expand
FC User Zone:
Name: Net-T1_Net-T2
Zone Name: ucs_CC25-Antwerp_A_6_NetApp_Repl_Net-T1_Net-T2
Vnet: 1500
Path: A
Config State: Applied
Oper State: Active
Zone Member:
Member wwpn address: 50:0A:09:84:8F:9B:4C:B2
Member wwpn address: 50:0A:09:88:8F:3B:02:C5
Target VSAN:
Name: 1500
VSAN ID: 1500
Operational VSAN: fabric/fc-estc/net-1500
Fabric ID: Dual
FC Zoning: Enabled
Zone Name: ucs_<system
name>_<switch id>_<zone_id>_<Zone
Profile Name>_<FC Zone name>
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
IP-based StorageAppliance Ports, iSCSI
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS Manager Appliance Ports
Appliance Port
Utilized for:
Direct Connect IP Storage
Direct Connect Unmanaged non-Switch/non-Router IP Devices (Servers, Appliances)
iSCSI/NFS/SMB
Qualified Scale – 4 Ports per FI
Ethernet End-Host and Switching Mode
Does Not Run STP
36BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Appliance Port Exposed Settings
QoS per port settings,
normal UCS QoS
constructs
Manual (static) pinning
using pin groups for border
port selection
Select which VLANs can
traverse this port
Optionally specify the
destination MAC address
of the filer . Some Filers
do not broadcast their
MAC address
QoS Settings
VLAN Trunking
Optional – If Not
then MAC
Learning
37BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VLANs and Appliance Ports
Similar to VSAN concept, there are two scopes
Traditional, LAN Cloud
Appliance Cloud with scope restricted to appliance ports and associated VLANs
Use the same VLAN ID in both scopes
LAN Scope
Appliance
Scope
38BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Appliance Ports Have Uplink Ports
Appliance ports like server ports have an uplink/border port assigned by static or dynamic pinning
Loss of last uplink port results in UCS appliance port being taken down
Default Network Control policy
Modifiable to Warning
39BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCSM iSCSI Feature Overview
Primary purpose is to support booting via iSCSI
Object called an “iSCSI vNIC” is created as a child object to the parent vNIC
IP Assignment – DHCP, Pools, Static
IQN Assignment – Manual, Pools, Per vNIC, Single Profile IQN (RFC3720)
CHAP / Mutual-CHAP Security
Initiator Authentication (CHAP), Target Authentication (M-CHAP)
Password (CHAP Secret) 12-16 Characters, $ = ? Invalid
iSCSI Failover and Redundancy
MPIO / Host Multi-Pathing Drivers Similar to FC – No Network Link Aggregation
40BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
iSCSI Boot Flow
Create iSCSI vNICs
Create iSCSI boot policy
Provide UCSM with iSCSI boot information Target ip, iqn
Initiator ip/mask/gw, iqn
vMedia map the OS and drivers
Adapter successfully initializes
Install OS and Drivers (if required)
41BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Unified Appliance Support
File and block data over a single
port/cable
FCoE, iSCSI, NFS, SMB
Port and cable consolidation
New port type: Unified Appliance Port
Appliance port of today + FCoE
Initial support for NetApp storage and
their Unified Target Adapter
Storage
FCoEiSCSINFSCIFSUnified Appliance Port
UCS B-Series
UCS 61xx/62xx UCS 61xx/62xx
42BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Storage Profiles
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Storage Profiles – Local Storage
44
Legacy – Initial Disk Configuration and Virtual Drive Creation
BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Storage Profiles / MultiLUN
Today - UCSM Profile Driven Local Storage
Local LUN Provisioning from Within UCSM
Storage Profiles
Virtual Drives
Advanced Disk Geometries and Configurations
Disk Group Policies
Direct Communication to Local PCIe Storage Controller
PCH Controller Integration
C240M4 – Internal SSDs
Modification of Local Storage Without Reboot
Consumed by Service Profiles and Templates
UCSM Managed M3 and M4 (B,C)
BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Storage Profiles – Local Storage
Disk Group Policies
Each Storage Profile Defined LUN Mapped to a Disk Group
Disk Group Configuration Defines RAID Level, Disks and
Quantities
Automatic Configuration
Chooses from Available Disks
Disk Qualifiers
Manual Configuration – Choose Specific Disks
Virtual Drive Configuration Defines LUN Parameters Within
Disk Group
46BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Storage Profiles – Local Storage
Local LUN Creation
Define Individual LUNs
Claim Existing Local LUN (Orphan LUN)
Expand To Available
Use All Remaining Space in Disk Group
Auto / No Auto Deploy
Automatic or Manual LUN Deployment
Disk Group
Consume or Create Disk Group for LUN
BRKCOM-2007 47
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Storage Profiles – Local Storage Advanced Disk Geometries and Configurations
Multiple Disk Groups and Virtual Drives (LUNs
48
Virtual Drives on Server
Physical Disks in Selected Virtual Drive
BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Storage Profiles – Provisioning
BRKCOM-2007 49
Local Storage Provisioning Flexibility
Specific Storage Profiles
Single System Use
Legacy Local Disk Configuration
Storage Profile Policies
Multi-Use
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 50BRKCOM-2007
UCSM Control of PCH Wellsburg Controller
Responsible for the (2) Internal C240M4 SSDs
Models UCSC-C240-M4L/M4SXX
AHCI and SWRAID Modes
RAID0 and RAID1
Integrated into Storage Profiles Construct
Scrub Policy and Protect Configuration Support
UCSM PCH/Internal SSD Support
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Storage Profiles – Local Storage
51
Boot Policies Boot From Local LUN
Enter Defined Boot LUN Name
PCH Controller Support C240M4 Internal SSDs
SWRAID - Add Embedded Local LUN
AHCI – Add Embedded Local Disk
Local LUN Image Path Secondary Option – M Series Only
BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS C3000 High Density Server
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Compute Resources
Storage Resources
Shared Local Resources Designed for large unstructured data repositories, backup and
archiving, cloud storage and long term retention
Full-Featured Modular Two-Socket Xeon Server
Stand-alone CIMC, IMC Supervisor and UCSM
Massive Local Storage
Up to 560TB of dense storage in a compact 4U Form Factor that fits in a standard rack
Network Resources
High I/O Bandwidth
Powered by Cisco VIC and the latest LSI 12-SAS Lane Technology
UCS C3000 Storage
BRKCOM-2007 53
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS C3000 Chassis ArchitectureDrives
4 Rows of hot-swappable HDD
4/6/8/10TB with up to 2 rows
of 400/800GB/1.6TB SSD
Total top load: 56 drives
FAN8 hot-pluggable
fans
Server Node2x E5-2600 V2 or v4 CPUs
128/256/512GB RAM
1GB/4GB RAID Cache
SAS HBA Node
Optional Slot 2Server Node, I/O Expansion or
Drive Expansion
Four 120GB/480GB
SSDsHot-Plug OS/Boot
System I/O Controller (SIOC)Cisco mLOM Slot or VIC 1300 on Chip
Power Supply4 hot-pluggable PSUs
Form Factor4U / Standard
Depth Rack <32”
BRKCOM-2007 54
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Broadwell Refresh Server Node SPECS:
1 or 2 CPU per Server Node: E5-2620v4: 8C/2.1G/85W,
E5-2650v4: 12C/2.2G/105W,
E5-2680v4: 14C/2.4G/120W
E5-2695v4: 18C/2.1G/120W
4 or 8 DDR4 2400Mhz DIMMs per CPU: 16G, 32G, 64G (Post-FCS)
RAID with 4G Cache: LSI SAS3316/24(Intruder) 12-CH
Up to 1, 15mm NVMe on board: 800G or 1.6TB
HW RAID for Boot Drives
BRKCOM-2007 55
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Extending UCS C3260 Infrastructure with 2nd Slot
Storage
Optional 2nd Server Node
Network
Optional 2X PCIe Expansion Board
Compute
Optional 4X LFF Disk Expansion
BRKCOM-2007 56
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
PCIe Expansion Board Stonycreek
Occupies Top Slot
2x PCIe Slots: Half Height/Half Width
Ethernet: Intel x520, Intel Quad port
Fiber Channel: Emulex, Qlogic
PCIe Flash Storage: Fusion I/O
Coming Soon:
2nd RAID/HBA Controller in single server node
2x NVMe On board
BRKCOM-2007 57
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS C3260 Management
Standalone
CIMC WebUI - HTML5 Interface
CLI
XML API
IPM
Dynamic Storage RAID Card – Dedicated
Disks
HBA Card – Dedicated and Shared Disks
Global Hot Spares
58BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS C3260 Management
• UCSM 3.1.2 Integration
Fully Managed by 2nd and 3rd
Generation Fabric Interconnects
Connects via FI Server Ports to 3260 SIOC Ports
Each 3260 Physical Box is a Chassis
Chassis-Wide and Per Server Node Management
Inventory, Compute and Storage Configuration, FW Mgmt, Pools, Policies, Profiles, Templates, vNICs/vHBAs, and Much More
Storage Profiles – Disk Group (RAID) and LUN Configuration
59BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 60BRKCOM-2007
UCS C3260 UCSM Integration
New Chassis Profile Object
Container for Chassis Firmware Package and Disk Zoning Policy
Assigned to Each 3260 Chassis to Apply Specified FW Package and Disk Zoning Policy
Initial and Updating Templates
New Parent Navigation Bar Chassis Icon
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 61BRKCOM-2007
UCS 3260 UCSM Integration
Disk Zoning Policy
Defines Disk Ownership from Chassis Disk Slots for Each Server Node
Unassigned, Dedicated, Shared (with HBA Card), Global Hot Spares
One Policy per Chassis
Consumed by Chassis Profile
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring, Troubleshooting and Advanced CLI
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring the Storage Path - Identification
Adapter Port:
VIC to IOM
HIF Port:
IOM to VIC
NIF Port:
IOM to FI
Server Port:
FI to IOM
Uplink Port:
FI to SAN
63BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring the Storage Path - GUI
GUI – VIF Paths Tab
vHBA to IOM to FI to Uplink
GUI – Statistics Tab
vHBA
IO Module Backplane Port
Fabric Interconnect Server Port
Fabric Interconnect Uplink Port
64BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring the Storage Path – GUIVIF Screen – With Gen2/3 VIC to FI Port Channels (220x/2304)
Adapter Port: Statistics Under Service Profile vHBA
Fex/Network/FI Server Port: Find Port Channel in Equipment/IOM/Fabric Ports/, Click on one of the Ports and follow
to Peer which is the Fabric Interconnect Server Port – View Statistics
FI Uplink Port: Find Port Channel in SAN\SAN Cloud\Fabric A or B\FC Port Channels – View Statistics
65BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring the Storage Path - CLI
CLI
vHBA - /org/service-profile/vhba # show stats
IOM Module “HIF” Port (Backplane Port)
connect IOM x (x designates IOM number)
For 2104 IOM: show platform software redwood sts
show platform software redwood rmon 0 hifx (x designates hif number)
For 220x IOM: show platform software woodside sts
show platform software woodside rmon 0 hifx (x designates hif number)
For 2304 IOM: show platform software tiburon sts
show platform software tiburon rmon 0 hifx (x designates hif number)
66BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring the Storage Path - CLI
Map “HIF” to “NIF” (Not Possible if IOM to FI Port Channels being Used)
connect nxos
show fex x detail (replace x with fex #)
show interface fex-fabric
IOM Module “NIF” Port (Fabric Port)
Connect IOM x (x designates IOM number)
For 2104 IOM: show platform software redwood rmon 0 nifx (x designates nif number)
For 220x IOM: show platform software woodside rmon 0 nifx (x designates nif number)
For 2304 IOM: show platform software tiburon rmon 0 nifx (x designates nif number)
67BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring the Storage Path - CLI
Map Backplane Port to FI Server Ports Interface/Port Channel
Connect NXOS
Find IOM/FI Port Channel associated with FEX Port(s)
show fex # detail (replace x with fex #)
Find interfaces associated with IOM/FI Port Channel and Statistics
show interface po # (replace x with po #)
Find FI FC PO /Uplinks and Show PO/Uplink Statistics
connect nxos
show npv flogi-table (match vfc/fcid/wwpn of vHBA to External Interface/po)
show interface san-port-channel # (replace # with po #)
or
From UCS CLI prompt: show service-profile circuit name x (replace x with SP name)
connect nxos
show interface san-port-channel # (replace # with po # found in show circuit output)
68BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring the Storage Path - CLIFor 220x IOM: show platform software woodside sts
69BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring the Storage Path - CLI
For 2304 IOM: show platform software tiburon sts
70BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring the Storage Path - CLIFor 220x IOM: show platform software woodside/tiburon rmon 0 hifx (x designates hif number)
71BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring the Storage Path - CLI
From UCS CLI prompt: show service-profile circuit name x (replace x with SP name)
72BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS Internal FC/FCoE Port Channel HashingIOM to FI / FI to IOM
220x/2304 IO Modules – Port Channels Enabled
IOM Inherits Load-Balance Hash from FI
To Display:
connect IOM x (x designates IOM number)
show platform software fwmctrl nifport
Scroll down a bit to see the hash parameters:
l2_da: 1 l2_sa: 1 l2_vlan: 0l3_da: 1 l3_sa: 1l4_da: 1 l4_sa: 1FCoE l2_da: 1 l2_sa: 1 l2_vlan: 0FCoE l3_did: 1 l3_sid: 1 l3_oxid: 1
Transmit (I’ve left MAC addresses of my system as an
example)
Source = 00:05:73:b8:fa:1d FCF-MAC of FC NP uplink port
Destination = 0e:fc:00:7b:09:0f = fpma (0e:fc:00 = FC-MAP
, 7b:09:0f = FC_ID of VFC of FC NP uplink port)
Did = FC_ID of VFC of FC NP Uplink Port
Sid = FC Target FC_ID
OX_ID = Unique Exchange ID
Receive (I’ve left MAC addresses of my system as an
example)
Source = 0e:fc:00:7b:09:0f = fpma (0e:fc:00 = FC-MAP ,
7b:09:0f = FC_ID of VFC of FC NP uplink port)
Destination = 00:05:73:b8:fa:1d = FCF-MAC of FC NP
uplink port
Did = FC Target FC_ID
Sid = FC_ID of VFC of FC NP Uplink Port
OX_ID = Unique Exchange ID
73BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS Internal FC/FCoE Port Channel HashingIOM to FI / FI to IOM
74BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS Internal FC/FCoE Port Channel HashingVIC to IOM / IOM to VIC
Sereno/Cruz ASIC – 12xx/13xx Port Channels
To Display Hash:
connect adapter x/y/z (x=chassis, y=blade, z=adapter)
connect
attach-mcp
echash_cfg
UIF: 0
eth fields: MAC_SA, MAC_DA, SRC_PORT, DST_PORT, SRC_IP, DST_IP
fcoe fields: MAC_SA, MAC_DA, OX_ID, RX_ID, S_ID, D_ID
hash bins: 1,3,1,3, 1,3,1,3, 1,3,1,3, 1,3,1,3
crc8 hash poly: polynomial 0x12F
75BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS Internal FC/FCoE Port Channel HashingVIC to IOM / IOM to VIC
76BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
UCS Internal vHBA and VNIC ExposedShow vHBA/vNIC State, Speed, MTU
granada-A# connect adapter 1/6/1 (chassis/server/adapter)
adapter 1/6/1 # connect
adapter 1/6/1 (top):6# attach-mcp
adapter 1/6/1 (mcp):3# vnic
Fibre Channel and Ethernet vnics configured on VIC will display13 vnic_1 enet 0 UP 2 UP =>0 746 57 147 UP
14 vnic_2 fc 0 UP 3 UP =>0 747 59 300 UP
15 vnic_3 enet 1 UP 4 UP =>0 751 58 25 UP
16 vnic_4 fc 1 UP 5 UP =>1 748 43 301 UP
adapter 1/6/1 (mcp):4# vnic 14 notify
.
.
link_state: UP
port_speed: 40000
mtu: 2158
77BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
iSCSI Boot Troubleshooting – Cisco VIC
78BRKCOM-2007
If your SP and iSCSI config is correct,
you will see this during POST
cae-sj-ca1-A# conn adapter 1/8/1
adapter 1/8/1 # connect
adapter 1/8/1 (top):1# attach-mcp
adapter 1/8/1 (mcp):1# iscsi_get_config
vnic iSCSI Configuration:
----------------------------
vnic_id: 5
link_state: Up
Initiator Cfg:
initiator_state: ISCSI_INITIATOR_READY
initiator_error_code: ISCSI_BOOT_NIC_NO_ERROR
vlan: 0
dhcp status: false
IQN:iqn.2012-03.com.cisco.cma:palo-iscsi-boot:0
IP Addr: 172.25.183.142
Subnet Mask: 255.255.255.0
Gateway: 172.25.183.1Target Cfg:
Target Idx: 0State: ISCSI_TARGET_READY
Prev State: ISCSI_TARGET_DISABLEDTarget Error: ISCSI_TARGET_NO_ERROR
IQN:iqn.199208.com.netapp:sn.101202278IP Addr: 172.25.183.49
Port: 3260Boot Lun: 0
Ping Stats: Success (9.698ms)Session Info: session_id: 0host_number: 0bus_number: 0target_id: 0
Cisco VIC
Initialization
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
FC Boot Troubleshooting – Cisco VIC
Looking Good – Assigned Service Policy
can see Targets and LUNs on Boot
Using LUNLIST to Troubleshoot
FIELD-TME-EL-CAP-A# connect adapter 3/7/1
adapter 3/7/1 # connect
adapter 3/7/1 (top):3# attach-fls
adapter 3/7/1 (fls):1# vnic
---- ---- ---- ------- -------
vnic ecpu type state lif
---- ---- ---- ------- -------
15 1 fc active 4
16 2 fc active 5
adapter 3/7/1 (fls):2# lunlist 15
vnic : 15 lifid: 4
- FLOGI State : flogi est (fc_id 0x340005)
- PLOGI Sessions
- WWNN 50:06:01:69:08:64:08:12 fc_id 0x3403ef
- LUN's configured (SCSI Type, Version, Vendor, Serial No.)
(0x0, 0x4, DGC , FNM00130800204)
- REPORT LUNs Query Response
LUN ID : 0x0000000000000000
LUN ID : 0x0001000000000000
- WWNN 50:06:01:60:08:64:08:12 fc_id 0x3402ef
- LUN's configured (SCSI Type, Version, Vendor, Serial No.)
(0x0, 0x4, DGC , FNM00130800204)
- REPORT LUNs Query Response
LUN ID : 0x0000000000000000
LUN ID : 0x0001000000000000
- Nameserver Query Response
- WWPN : 50:06:01:60:08:64:08:12
- WWPN : 50:06:01:69:08:64:08:12
79BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Demo
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.
81BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
82BRKCOM-2007
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Please join us for the Service Provider Innovation Talk featuring:
Yvette Kanouff | Senior Vice President and General Manager, SP Business
Joe Cozzolino | Senior Vice President, Cisco Services
Thursday, July 14th, 2016
11:30 am - 12:30 pm, In the Oceanside A room
What to expect from this innovation talk
• Insights on market trends and forecasts
• Preview of key technologies and capabilities
• Innovative demonstrations of the latest and greatest products
• Better understanding of how Cisco can help you succeed
Register to attend the session live now or
watch the broadcast on cisco.com
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Thank you
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public