IBM Blade Center Deployment Guide - Part 2

IBM BladeCenter Deployment Guide

IBM® BladeCenter™ Deployment Guide Part 2 – Embedded Switches

Chris Gillen, IBM

Sam Litenatsky, IBM Kevin Conover, IBM

Version 3.0, 3/10/2006 http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564The BladeCenter Deployment Guide is intended as a personal productivity tool. It is not intended to be comprehensive and is provided for

guidance only, on an 'as is basis' without warranty of any kind. Please be aware that its contents have not been certified by IBM.

1


2 http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564The BladeCenter Deployment Guide is intended as a personal productivity tool. It is not intended to be comprehensive and is provided for


© 2006 IBM Version 3.0, 3/10/2006


Table of Contents 1 Introduction........................................................................................................................................5

1.1 Overview ...................................................................................................................................5 1.2 BladeCenter Subsystems..........................................................................................................6 1.3 Notes .........................................................................................................................................6

1.3.1 Contributors.......................................................................................................................6 Installation and Configuration .......................................................................................................................7

2 Preparatory Steps .............................................................................................................................7 2.1 Initial Configuration ...................................................................................................................7 2.2 Network Configuration...............................................................................................................7 2.3 BladeCenter Management Module ...........................................................................................8 2.4 User Configuration ....................................................................................................................8 2.5 BladeCenter Firmware ..............................................................................................................9

Switch Configuration ...................................................................................................................................11 3 Embedded Ethernet switches .........................................................................................................11

3.1 Configuring the Ethernet switch ..............................................................................................11 3.1.1 Changing the IP addresses.............................................................................................11 3.1.2 Enabling the external ports .............................................................................................11

3.2 Access and log on to the management module......................................................................11 3.2.1 Externally managing the Nortel GbE switch module.......................................................11 3.2.2 Externally Managing Cisco CIGESM ..............................................................................12 3.2.3 Testing Procedures – Ethernet Connectivity...................................................................12 3.2.4 Problem Documentation Requirements ..........................................................................12

3.3 Updating the Firmware on the Nortel GbESM ........................................................................13 3.4 Ethernet Port Numbering ........................................................................................................15

4 Embedded Fibre Channel Switches................................................................................................16 4.1 SAN Configuration ..................................................................................................................16 4.2 Disabling ports on the QLogic switch......................................................................................16

4.2.1 Using Telnet ....................................................................................................................16 4.2.2 Using SANSurfer GUI .....................................................................................................17

4.3 Updating the Firmware on the QLogic Switch.........................................................................18 4.3.1 Updating via SANSurfer GUI...........................................................................................18 4.3.2 Updating via Telnet .........................................................................................................19

5 SAN Configuration ..........................................................................................................................21 5.1 Description ..............................................................................................................................21 5.2 Firmware Revision Levels .......................................................................................................21 5.3 Fabric Domain IDs ..................................................................................................................21

5.3.1 Determining the Domain ID for McData ..........................................................................21 5.3.2 Determining the Domain ID for Qlogic ............................................................................22

5.4 Principle Switch.......................................................................................................................23

Appendix A. Firmware Versions ..............................................................................................26 Firmware Levels for Chassis Components .....................................................................................26



3


Table of Tables Table 1 Recommended settings for the QLogic QLA23xx adapter card for Windows 2000 or 2003.........20 Table 2 Management Module BIOS Information.........................................................................................26 Table 3 Nortel Networks Layer 2/3 GbE Switch Module.............................................................................26 Table 4 Nortel Switch Firmware Information...............................................................................................26 Table 5 Cisco GbESM Firmware Information .............................................................................................26 Table 6 QLogic 2pt Fibre Channel Switch BIOS Information......................................................................27 Table 7 QLogic 6pt Fibre Channel Switch BIOS Information......................................................................27 Table 8 Brocade SAN Switch Module Firmware Information......................................................................27 Table 9 QLogic Firmware Information.........................................................................................................27

Table of Figures Figure 1 BladeCenter Chassis – Rear View .................................................................................................6 Figure 2 BladeCenter internal network .........................................................................................................8 Figure 3 Firmware Levels from Management Module ................................................................................10 Figure 4 Updating the firmware on the Nortel switch..................................................................................14 Figure 5 Storage Connection Diagram .......................................................................................................16 Figure 6 Port Status Online.........................................................................................................................17 Figure 7 Port Status Offline.........................................................................................................................17 Figure 8 SANSurfer Switch Faceplate ........................................................................................................18 Figure 9 Port Properties ..............................................................................................................................18 Figure 10 Firmware Update via Telnet........................................................................................................20 Figure 11 Switch Parameters (McData)......................................................................................................22 Figure 12 Change domain ID (McData) ......................................................................................................22 Figure 13 Launch SANbox Manager...........................................................................................................23 Figure 14 Determine Qlogic domain ID.......................................................................................................23 Figure 15 Setting Switch Priority.................................................................................................................24 Figure 16 Setting Switch to Principle ..........................................................................................................24



© 2006 IBM Version 3.0, 3/10/2006


1 Introduction 1.1 Overview The IBM BladeCenter Deployment Guide is intended to present the general steps necessary to successfully deploy IBM blades in the IBM® BladeCenter™. Its primary focus is to detail the steps required for setting up remote boot on IBM Blades. The information provided in this document is based on technical observations. Due to the breadth of content related to this topic, this guide has now been split into 3 parts.

1. Management Module – information specific to the initial setup and ongoing usage of the Management Module.

2. Embedded Switches – information related to the configuration of the embedded Ethernet and Fibre channel switches.

3. Blades – blade configuration and installation information with a primary focus of installing Linux in a boot to SAN environment.

Remote boot, root boot, or boot from SAN, is the name referred to the server configuration where the server operating system is installed on a logical drive (LUN) that does not reside inside the server chassis. This document will describe in detail the process that one must go through to setup a remote boot for HS20 or HS40 blade servers. The blade servers utilize the IBM BladeCenter Fibre Channel Expansion Card connected to the BladeCenter Fibre Channel Switch Modules via the BladeCenter backplane. The switch modules in turn are connected to the core fibre channel switches that host the SAN Fabric. This document is a supplement the IBM® BladeCenter™ publications provided with the BladeCenter products. In addition to the publications being provided with the BladeCenter products, they also are available in Portable Document Format (PDF). The latest versions of the publications are available from the IBM web site at http://www.ibm.com/pc/support/.

BladeCenter publications include the following: • IBM BladeCenter Type 8677 Planning and Installation Guide: This manual provides physical

planning information for the IBM BladeCenter products. It describes the BladeCenter components, explains deployment and installation considerations, and provides worksheets that you can use to determine the configuration, power, weight, and cabling requirements for your BladeCenter unit. (http://www-1.ibm.com/support/docview.wss?uid=psg1MIGR-53670)

• IBM BladeCenter Type 8677 Installation and User’s Guide. This document contains general information about your BladeCenter unit, including information about features, how to configure your BladeCenter unit, and how to get help. It is provided with the BladeCenter unit and is on the IBM BladeCenter Documentation CD.

• IBM BladeCenter Management Module User Guide: This guide contains information about configuring the management module and managing components installed in BladeCenter chassis. (http://www-1.ibm.com/support/docview.wss?rs=0&uid=psg1MIGR-45153)

• IBM xSeries and BladeCenter Server Management (SG24-6495): This document covers the hardware side of the IBM systems management solution including the BladeCenter management module. It is available for download from the IBM Web site at http://www.ibm.com/redbooks/.

• IBM BladeCenter Switch Interoperability Guide: Provides the details needed to configure and deploy multi-vendor switched fabrics. Detailed switch configuration data and step-by-step configuration procedures are provided to merge the IBM eServer BladeCenter with Brocade, Cisco, McDATA, and QLogic Fibre Channel switched fabrics that comply with the second revision of the Fibre Channel switch standard (FC-SW-2).

• Nortel Networks L2/3 Ethernet Switch Module for IBM BladeCenter. This IBM Redpaper will help you install, tailor, and configure BladeCenter and the Nortel Layer L2/3 Switch Module in various network environments. (REDP3586)

• IBM BladeCenter Layer 2-7 Network Switching. This IBM Redpaper will help you install, tailor, and configure BladeCenter and the Layer 2-7 GbE Switch Module (GbESM) in various network environments. (REDP3755)



5

http://www.ibm.com/pc/support/

http://www-1.ibm.com/support/docview.wss?uid=psg1MIGR-53670

http://www-1.ibm.com/support/docview.wss?rs=0&uid=psg1MIGR-45153&loc=en_US


• Cisco Systems Intelligent Gigabit Ethernet Switch Module for IBM BladeCenter. This Redpaper helps plan for, install, and configure the Cisco CIGESM in several network topologies. The topology examples are provided in this Redpaper for the purpose of demonstrating a number of ways to perform the integration of the switch module into different networks. (REDP3869)

1.2 BladeCenter Subsystems The various BladeCenter Subsystems include: • Server Blades - provide the processors, memory, hard disk(s) and firmware of an industry standard

server. In addition, they will normally incorporate keyboard, video, and mouse interfaces, and an onboard service processor.

• Management Modules – incorporate the chassis-level management processor and Keyboard/Video/Mouse (KVM) access for the blades.

• Switch Modules - provide multiple high-speed Ethernet ports in Switch Module slots 1 and 2 for use by all of the Server Blades to interconnect internally as well as to external network devices. Switch Module slots 3 and 4 are used for Fibre Channel or other switch types and connect to the Server Blades as well as external devices.

• Power - Four modular power supplies provide primary and backup power to all BladeCenter subsystems. Power supplies 1 and 2 are responsible for powering blade 1 through 6 plus all of the BladeCenter components such as the switches, management modules, media tray, etc, and power supplies 3 and 4 are responsible for power blades 7 through 14.

• Cooling - Dual blowers provide forced-air cooling of BladeCenter components and subsystems. • Media Tray - Removable media subsystem shared by all blades. Includes CD/DVD and floppy media.

Figure 1 BladeCenter Chassis – Rear View

1.3 Notes This is currently a living document and will continue to grow as we have more experience with IBM Blades in the boot-to-SAN environment. This document is intended as a personal productivity tool. It is not intended to be comprehensive, and is provided for guidance only, on an 'as is basis' without warranty of any kind. Please be aware that its contents have not been certified by IBM.

1.3.1 Contributors We would like to acknowledge the contributions of the following people:

1. Richard Gebhardt, IBM Lab Services – Appendix E Error! Reference source not found., portions of Section Error! Reference source not found. Error! Reference source not found..

2. Fred Rabert, Nortel Networks – Assistance with Section 3 Embedded Ethernet switches plus BroadCom Driver installation instructions.



© 2006 IBM Version 3.0, 3/10/2006


Installation and Configuration

2 Preparatory Steps The BladeCenter Chassis components should be prepared by updating the firmware and assigning the appropriate IP addresses to the management and switch modules. The IP addresses of the various modules as well as the userids and passwords for each will be needed during the installation. A sample worksheet can be found in 1.Appendix A.

2.1 Initial Configuration To use the BladeCenter Management Module to perform the tasks in the coming sections remotely, use the following steps: • Open a browser and input the IP address of the MM. Note, the management module defaults to using

DHCP. If no DHCP server is found, the management module uses a default IP address of 192.168.70.125 with subnet mask 255.255.255.0. The host name will be MMxxxxxxx. The default hostname is MMxxxxxxxxxxxx where xxxxxxxxxxxx is the burned-in MAC address. This number is printed below the reset button. It is possible to connect directly into the management module by using a crossover cable between a laptop and the Ethernet port of the management module. Change the ip address of the laptop to an address on the same subnet as the management module.

• The logon screen will display, and you will be prompted to log onto the MM web server. The default user ID and password are USERID and PASSW0RD (0 is a zero). These are case sensitive.

• Once you enter your user ID and password, you will presented with a welcome window that displays where you are connected and, for security purposes, where you set the duration that your connection can be idle before being automatically disconnected.

• Click <Continue>. The BladeCenter Management and Configuration window opens. From the Management and Configuration program main menu, you can select settings that you want to view or change.

2.2 Network Configuration The two switch modules provide redundant paths to each of the server blades for reaching other server blades within the same chassis or for communication to external devices via the network infrastructure. (Figure 2 BladeCenter internal network) The Ethernet switch modules can be managed via telnet or a Web interface. Both the Web and telnet interfaces can be started by accessing the switch directly or starting a session from the management module’s Web interface. Note: The external ports on the switch must be enabled, and enabled for management as well, to manage the switch directly without going through the management module. Because the management interface is on VLAN 4095 and cannot be moved, a separate interface from the management interface must be created with an IP address valid on the production network to manage the switch directly.



7


Figure 2 BladeCenter internal network Note: For switch communication through the management module external Ethernet port, the switch module internal network interface and the management module internal and external interfaces must be on the same subnet.

2.3 BladeCenter Management Module To use the BladeCenter Management Module to perform the tasks in the coming sections remotely, use the following steps: • Open a browser and input the IP address of the MM. The default IP address of the external port of the

management module is 192.168.70.125, the default subnet mask is 255.255.255.0, and the default hostname is MMxxxxxxxxxxxx where xxxxxxxxxxxx is the burned-in MAC address. It is possible to connect directly into the management module by using a crossover cable between a laptop and the Ethernet port of the management module. Change the ip address of the laptop to an address on the same subnet as the management module. 192.168.70.124 is a good choice.

• The logon screen will display, and you will be prompted to log onto the MM web server. The default user ID and password are USERID and PASSW0RD (0 is a zero). These are case sensitive.

• Once you enter your user ID and password, you will presented with a welcome window that displays where you are connected and, for security purposes, where you set the duration that your connection can be idle before being automatically disconnected.

• Click <Continue>. The BladeCenter Management and Configuration window opens. From the Management and Configuration program main menu, select settings that you want to view or change.

2.4 User Configuration Authority level Select the authority level(s) associated with this login id and transfer them to the "Assigned roles" box. The ">>" and "<<" buttons may be used to transfer selected items from one box to the other. • Supervisor: User has no restrictions except for assigned scope. • Operator: User has only read-only access, and cannot perform any save, modify, clear, or state

affecting operations (eg. restart ASM, restore defaults, upgrade firmware, etc.).



© 2006 IBM Version 3.0, 3/10/2006


• Custom: User may or may not have any restrictions, depending on which custom authority level is assigned to the user.

If you select the Custom option, you must select one or more of the following custom authority levels: • Chassis Operator User may browse status and properties of chassis components (MM, blowers,

midplane, power modules, media tray). Ability to backup MM configuration. • Chassis User Account Management User may add, modify, and delete MM login profiles. • Chassis Log Administration User may clear the MM event log and change log policy settings. • Chassis Configuration User may modify and save any chassis configuration parameter (except user

profiles and event log settings). For example, general MM settings, MM port assignments, MM network interfaces, MM network protocols, MM security. Ability to restore MM configuration. Ability to restore MM factory defaults if the user also has Chassis Administration permissions.

• Chassis Administration User may do MM firmware updates, restore MM factory defaults if the user also has Chassis Configuration permissions, or restart the MM.

• Blade Operator User may browse status and properties of blades. • Blade Remote Presence User may access the Remote Control web panel and the functions

provides on the panel: remote console (KVM) and remote disk. The CLI "console" command to start an SOL session to a blade will also require this authority.

• Blade Configuration User may modify and save and blade configuration parameters. For example, parameters in the Blade Configuration web panel, and blade SOL parameters on the Serial Over LAN web panel.

• Blade Administration User may power on/off and restart blades, activate standby blade servers, do firmware updates.

• Switch Operator User may browse status and properties of I/O modules, and may ping the I/O modules.

• Switch Configuration User may modify the switch module IP address, configure switch Advanced Management parameters, and restore factory defaults.

• Switch Administration User may power on/off and restart switch modules, update switch module firmware, and enable/disable the Fast POST and External Ports of a switch. Note: in order to be able to launch the switch module telnet or web UI, supervisor access is required for that switch module. This means that both Switch Configuration and Administration roles must be assigned.

Note: In order to prevent a situation where none of the users have read/write access, login profile #1 must be set with at least the ability to modify the login profiles. This means that this user must be given either Supervisor access, or User Account Management access. This guarantees that at least one user can perform actions or make configuration changes, or at the very least, add users to the login profiles who can.

2.5 BladeCenter Firmware The tables in 1.Appendix A Firmware Versions contain both the minimum versions required deploy on the BladeCenter and its various components, and the current versions at the time this document was published. Please visit the IBM support site for the most current firmware versions for all components.

www-306.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-54597 There is also a URL for registering for proactive emails regarding firmware updates.

https://www-1.ibm.com/support/mysupport/us/en/ If first time select register. After you have logged into the site, select edit profile and add the servers that you would like receive notification on. The tables in that appendix will also help translate between build ids and release versions. You can see the current firmware levels from the management module, shown in Figure 3 Firmware Levels from Management Module.



9

http://www-306.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-54597


Figure 3 Firmware Levels from Management Module



© 2006 IBM Version 3.0, 3/10/2006


Switch Configuration

3 Embedded Ethernet switches

3.1 Configuring the Ethernet switch

3.1.1 Changing the IP addresses Establish a TCP/IP session for the GbE switch through the management module by completing the following steps: 1. Access and log on to the management module. For more information, see Section 2.1 2. Select Management on the I/O Module Tasks menu on the left side of the BladeCenter Management

Module window. 3. From the I/O Module Management portion of the window, click the bay number that corresponds to

the location of the GbE switch module that you installed. The applicable bay number (Note: Bay 1 is the designated primary Ethernet switch, Bay 2 is designated as the standby Ethernet) appears in the bottom portion of the window, followed by other related switch-module information, including the IP address. The switch-module information is divided under two headings: Current IP Configuration and New Static IP Configuration.

4. To change the IP address of the GbE switch module, use the Tab key to reach the IP address data field under the New Static IP Configuration heading; then, type the new IP address and click Save. Click Advanced Management under the applicable bay information (in this example, bay 1) in the Management Module window. You can now start a browser-based interface session, a Telnet session, or a Secure Shell (SSH) session.

3.1.2 Enabling the external ports To access and manage your GbE switch module from an external environment, you must enable certain features, such as the external ports (ports) and external management over all ports, through the management module. After you enable these features, you can establish a TCP/IP session for the switch through the external GbE switch module connection. as described in Establishing a TCP/IP session for the switch through the external switch-module connectors. Complete the following steps to configure the external Ethernet interface and to enable the external ports.

3.2 Access and log on to the management module. 1. For more information, see Section 2.1. 2. Select Management on the I/O Module Tasks menu on the left side of the BladeCenter

Management Module window. 3. Make sure that the following GbE switch-module features are enabled (I/O Module Tasks

Management Advanced Management in the management module browser-based user interface): a. Select external ports b. Select external management over all ports

The default value is disabled for both features. If these features are not already enabled, change the value to Enabled. Then, click Save.

3.2.1 Externally managing the Nortel GbE switch module The following procedure describes how to configure the GbE switch module to enable management through the external ports (EXT1 to EXT4). Reference the following configuration for enabling external management of the GbE switch module.



11


Start a Telnet session with the GbE switch module from your laptop connected directly to any of the IBM Nortel switch external ports. USERID: admin Passwd: admin The following example shows how to create this configuration using the CLI:

/cfg/13/if 1/mask 255.255.255.0 /cfg/13/if 1/addr 10.10.1.1 /cfg/13/if 1/vlan 1 /cfg/13/if 1/ena ena apply save yes

Critical Note: Please remember the following acronym: (EASY) E= Enable A= Apply S= Save Y= Yes after making your CLI changes to avoid losing your changes.

3.2.2 Externally Managing Cisco CIGESM See Section 5.3 in Cisco Redpaper REDP3869.

3.2.3 Testing Procedures – Ethernet Connectivity • Open a Telnet session from the Management Module login, and issue a ping command to the switch

IP address & Gateway address. If successful attempt to launch a telnet session with to the external Cisco Catalyst switch. If successful launch a Telnet session from the Cisco Catalyst to the BladeCenter Ethernet switch, if successful basic Ethernet connectivity is complete.

• Open a remote console session to one of the servers, login and open up a DOS window. From the DOS prompt ping the servers NIC IP address and IP Gateway address. If successful ping the BladeCenter Ethernet switch IP address. If successful ping the IP address of the external Cisco Catalyst switch, and if successful then attempt to Telnet to the Cisco External switch. Once logged into the Cisco switch ping the server IP address.

• To test the L2 Trunk failover with the Standby Switch in Bay2, a. Remove the switch in Bay 1, b. Then repeat the steps in step 1 to verify traffic has moved from the the standby switch. c. After test completion, reinstall the switch in Bay1, slide switch 2 in Bay2 out then reseat switch

2, bay 2. d. Then repeat tests from step 1 to verify that switch 1 is now the operational.

3.2.4 Problem Documentation Requirements The scope of this section is to insure that in the event we have an issue with BladeCenter, all appropriate documentation will be gathered the first time to aid in a timely resolution of a problem. Nortel GbESM The command to use for dumping the switch is:

/info/sys/tsdmp Note: Dump the information to a TFTP server, or Laptop Cisco CIGESM The command to use for dumping the External Switch show tech Note: dump the information to a TFTP server or Laptop.



© 2006 IBM Version 3.0, 3/10/2006


e-Gatherer The process to gather the server's information begins by downloading a copy of e-gatherer. 1. Download a copy of e-Gatherer from the following link for Windows or Linux and then follow the

instructions below. http://www-1.ibm.com/support/docview.wss?rs=113&context=HW180&q1=e-gatherer&uid=psg1MIGR-4R5VKC&loc=en_US&cs=utf-8&lang=en

2. Double-click on the file to run the e-Gatherer program. At the DOS window type Y to accept the terms and conditions of the disclaimer.

3. When the progress bar indicates that the e-Gatherer program has completed its information gathering operation, close the DOS window.

For Linux, use the following commands: 1. Log in as root 2. Download or copy egather2-2.09.linux to a directory of your choice, for example /tmp 3. Enter the command: chmod u+x egatherer2-2.09.linux 4. Enter the command: ./egatherer2-2.09.linux to run eGatherer. When e-Gatherer has successfully executed, an output file is generated. This file resides in the same directory as your e-Gatherer program file. The name of this output file takes has the format IBM-xxxxxxx-ABCDEFG.eg2 where: "xxxxxxx" denotes a seven digit machine type number and "ABCDEFG" denotes the serial number of your system. This output file contains all of your system's configuration information. Attach this output file (IBM-xxxxxxx-ABCDEFG.eg2) to an e-mail and send this e-mail back to IBM's support center.

3.3 Updating the Firmware on the Nortel GbESM A TFTP server application is used to upgrade the software of the switch module. The TFTP server application must be installed on you server. Once that is done the software image for the switch module can be applied. Note: The TFTP server, management module, and switch IP addresses MUST all be on the same subnet in order for the firmware update to work. 1. Obtain the latest Nortel GbESM firmware zip file. 2. Unzip the file obtained above into a temporary directory. 3. Copy the two image files, GbESM-AOS-<ver>-os.img and GbESM-AOS-<ver>-boot.img, from

the temporary directory to the root of the TFTP server. 4. Connect to the Nortel ESM telnet interface

a. Click on the “Management” link under the “I/O Module Tasks” section b. Click on the “Advanced Management” link under the “Bay 1 (Ethernet SM)*” c. Scroll down to the bottom of the page and click the “Start Telnet Session” button in the “Start

Telnet/Web Session” section to launch a telnet app and connect to the switch. d. Log in to the switch (the default password is “admin”) e. When prompted to run “Setup Up” to configure the switch, type “n” and press enter

5. Update the Nortel GbESM OS image Note: There are two OS images (image1 and image2) that reside in the Nortel ESM. Normally, the Nortel GbESM will boot image1. However, you must determine which OS image is being used before updating the Nortel GbESM firmware. a. Type “/boot/cur” at the telnet prompt to determine which OS image is being used. The output

from the “/boot/cur” command will state the name of the image (image1 or image2) that is currently being used. The text will be something like “Currently set to boot software image1”.

Note: You MUST use image2 for the command in the next step if the output from the /boot/cur command states that image2 is currently empty



13

http://www-1.ibm.com/support/docview.wss?rs=113&context=HW180&q1=e-gatherer&uid=psg1MIGR-4R5VKC&loc=en_US&cs=utf-8&lang=en

http://www-1.ibm.com/support/docview.wss?rs=113&context=HW180&q1=e-gatherer&uid=psg1MIGR-4R5VKC&loc=en_US&cs=utf-8&lang=en


b. Type “/boot/gtimg imageX TFTPAddr GbESM-AOS-<ver>-os.img” at the telnet prompt and press enter where

• imageX is the name of the OS image (image1 or image2) that was determined in previous step.

• TFTPAddr is the IP address of the TFTP server on your production network where the image files reside.

See Figure 4 for an example. The image file name must match file downloaded in Steps 1 through 3 above.

c. Type “y” and press enter when prompted to “Confirm download operation” d. Wait for the flashing operation to complete.

Note: Type “y” and press enter to boot from the new image if you were forced to update image2 because it was previously empty.

e. Type “exit” and press enter to terminate the telnet interface. f. Close the window in which the telnet session was running.

6. Restart the Nortel GbESM a. From the Management Module Web Interface, click on the “Power/Restart” link under the “I/O

Module Tasks” section. b. Check the box next to the Nortel GbESM and click on the “Power Off Module(s)” link. c. Click the “OK” button on the confirmation screen. d. Check the box next to the Nortel GbESM and click on the “Power On Module(s)” link e. Click the “OK” button on the confirmation screen. You must wait approximately 60 seconds for

the swtitch to fully restart before continuing. 7. Reconnect to the Nortel GbESM telnet interface by repeating Step 4. 8. Update the Nortel GbESM boot image

Type “/boot/gtimg boot TFTPAddr GbESM-AOS-<ver>-boot.img” at the telnet prompt and press enter where TFTPAddr is the IP address of the TFTP server on your production network where the image files reside. The image file name must match file downloaded in Steps 1 through 3 above.

9. Type “y” and press enter when prompted to “Confirm download operation” 10. Wait for the flashing operation to complete. 11. Type “exit” and press enter to close the telnet interface, close the window in which the telnet session

was running. 12. Restart the Nortel GbESM by repeating Step 6 above.

Figure 4 Updating the firmware on the Nortel switch



© 2006 IBM Version 3.0, 3/10/2006


3.4 Ethernet Port Numbering In most cases, for Windows 2000/2003, the connection named Local Area Connection goes to the Ethernet switch in switch bay 1, and the connection named Local Area Connection 2 goes to the Ethernet switch in switch bay 2. We use the phrase most cases, because this is not always the case. For Windows 2000, the order of the Local Area Connection names assigned is based on the order in which the drivers for each NIC are installed. The drivers necessary to support the NICs on a blade server are not part of a standard Windows 2000/2003 install, and the NICs will be generically listed in Windows 2000 Device Manager as two or more Ethernet Controllers until the necessary drivers are loaded. For these NICs to become active, a third-party driver, supplied by IBM, needs to be installed. The normal procedure most users follow is to install the drivers on the first Ethernet Controller in the list, and then install the drivers on the second Ethernet Controller in the list (and so on). The end result of this is the most-cases scenario previously mentioned, where the Windows 2000/2003 connection named Local Area Connection goes to switch 1 and the one named Local Area Connection 2 goes to switch 2. If, however, the drivers are installed on the second Ethernet Controller in the list first, and then the first Ethernet Controller in the list, the connection names are reversed, and the connection named Local Area Connection is now the one going to switch 2, and the connection named Local Area Connection 2 is now going to switch 1. In older versions of Linux it was observed that eth0 went to the Ethernet switch in switch bay 2 and eth1 goes to the Ethernet switch in switch bay 1. This was reversed from a normal Windows 2000 install, as previously mentioned, and also proves to be the source of some confusion. However, current testing with Linux (Red Hat Version 3 Update 6) and SLES 9 has shown that eth0 goes to the Ethernet switch in bay 1 and eth1 goes to the switch in bay 2. On an HS40 blade, 0 & 2 should be connected to top switch module and 1 & 3 connect to the bottom switch module. The designations can be verified by browsing the MAC address table on the Ethernet switch and match the values to the MAC addresses show in the ifconfig output on the blade. The following example shows the MAC address of Eth 0/1 devices. Log into the top switch module (bay 1) and search for the MAC address on Internal port 13 (Bay 13) corresponding to Eth0 device. It shows the same MAC address of Eth0. For Eth1, do the same by logging into switch module bay2 (bottom) and searching for the MAC address on internal port 13, it also shows the same MAC address for Eth1.

[root@hs20-8843 root]# cat /proc/version Linux version 2.4.21-27.ELsmp ([email protected]) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-47)) #1 SMP Wed Dec 1 21:59:02 EST 2004 [root@hs20-8843 root]# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:11:25:4A:00:4C inet addr:9.42.166.82 Bcast:9.42.166.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 ... Browse MAC Address Table – Top Switch VID MAC Address Port Learned 9 00-11-25-4a-00-4c Bay 13 Dynamic

eth1 Link encap:Ethernet HWaddr 00:11:25:4A:00:4D inet addr:10.0.200.32 Bcast:10.0.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 ... Browse MAC Address Table – Bottom Switch VID MAC Address Port Learned 100 00-11-25-4a-00-4d Bay 13 Dynamic



15


4 Embedded Fibre Channel Switches

4.1 SAN Configuration Shown in Figure 5 is an example of a dual path SAN configuration that we will be using as a reference in this document. Two McData directors steer traffic through separate fabrics to the IBM BladeCenters, which use Qlogic HBAs (Host Bus Adapters). Separate fabrics allow full redundancy in the storage path. The IBM Blade Server boots to drives physically located on the Hitachi subsystem over the SAN. Qlogic HBAs must be specifically configured to allow remote booting. See diagram 1 for a visual representation of this architecture.

Figure 5 Storage Connection Diagram

4.2 Disabling ports on the QLogic switch Here are the steps necessary to disable ports on the QLogic switch. This can be done either via a telnet session or a browser session.

4.2.1 Using Telnet 1. Start a telnet session with the IP address of the QLogic Switch Module. 2. After logging in, enter the command ”show port” to view the current status of the ports.



© 2006 IBM Version 3.0, 3/10/2006


Figure 6 Port Status Online

3. Enter admin mode by entering “admin start” 4. Disable the target port by entering the command “set port x state offline” where x is the

number of the port. For this example, we used “4” 5. Confirm the status by entering “show port”again.

Version 3.0, 3/10/2006 http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564 17

Figure 7 Port Status Offline

4.2.2 Using SANSurfer GUI 1. Start the SANSurfer GUI. If the fabric has not already been discovered, click Fabric-Add Fabric

and enter the IP address of the target QLogic switch in the window that will pop up - That should open SAN Browser.

2. Click the handle in the left pane to drop down the switch - highlight it and the faceplate will appear in the right pane.

Admin Operational Login Config Running Link Link

Port State State Status Type Type State Speed

---- ----- ----------- ------ ------ ------- ----- -----

Ext1:0 Online Online LoggedIn GL F Active 1Gb/s

Ext2:15 Online Offline NotLoggedIn GL Unknown Inactive Auto

Bay1 Online Online LoggedIn F F Active 2Gb/s

Bay2 Online Offline NotLoggedIn F Unknown Inactive 2Gb/s











Bay13 Online Offline NotLoggedIn F Unknown Inactive 2Gb/s Bay14 Online Offline NotLoggedIn F Unknown Inactive 2Gb/s

QLogic: USERID> admin start

QLogic (admin): USERID> set port 4 state offline

QLogic (admin): USERID> show port

Admin Operational Login Config Running Link Link

Port State State Status Type Type State Speed

---- ----- ----------- ------ ------ ------- ----- -----

Ext1:0 Online Online LoggedIn GL F Active 1Gb/s

Ext2:15 Online Offline NotLoggedIn GL Unknown Inactive Auto




Bay4 Offline Offline NotLoggedIn F Unknown Inactive 2Gb/s

Bay5 Online Offline NotLoggedIn F Unknown Inactive 2Gb/s Bay6 Online Offline NotLoggedIn F Unknown Inactive 2Gb/s

The BladeCenter Deployment Guide is intended as a personal productivity tool. It is not intended to be comprehensive and is provided for guidance only, on an 'as is basis' without warranty of any kind. Please be aware that its contents have not been certified by IBM.


Figure 8 SANSurfer Switch Faceplate

3. Right click on the desired port and choose "port properties" from the menu. 4. At the top of port properties put a bullet next to "offline" or "down". 5. Repeat steps 1-4 and reverse step 5 to online the port.

Figure 9 Port Properties

4.3 Updating the Firmware on the QLogic Switch 4.3.1 Updating via SANSurfer GUI

Below are the instructions for performing the update of the QLogic Switch firmware via the SANSurfer GUI. You can also find this information in the SANSurfer help.

Installing firmware involves loading, unpacking, and activating the firmware image on the switch. The application does this in one operation.

To install firmware on a switch, do the following from the SANSurfer GUI: 18 http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564


© 2006 IBM Version 3.0, 3/10/2006


1. In the faceplate display, open the Switch menu and select Load Firmware. 2. In the Firmware Upload dialog, click the Select button to browse and select the firmware file to be

uploaded. 3. Click the Start button to begin the firmware load process. 4. The application prompts you to activate the new firmware, non-disruptively, if possible. Click the OK

button to reset the switch and activate the new firmware.

Note: For firmware versions that follow the initial release of version 2.0, you have the ability to install and activate new firmware without resetting the switch, and therefore, without disrupting traffic. The application will attempt a nondisruptive firmware installation and activation whenever possible.

Important: During a hotreset operation, fabric services will be unavailable for a short period (30-75 seconds depending on switch model). To ensure that a Non-Disruptive Code Load and Activation (NDCLA) operation is successful, verify that all administrative changes to the fabric (if any) are complete. Changes to the fabric may disrupt the NDCLA process. Common administrative operations that change the fabric include:

1. Zoning modifications 2. Adding, moving or removing devices attached to the switch fabric. This includes powering up or

powering down attached devices. 3. Adding, moving or removing ISLs or other connections.

After an NDCLA operation is complete, management connections must be re-initiated. Sessions will re-connect automatically Telnet sessions must be restarted manually.

Applicable Code Versions • NDCLA capability is available starting with version 2.0 of the switch code. (N/A for 20-Port 10-Gbps

switches) • Upgrading to version 2.0 from previous releases will be disruptive. (N/A for 20-Port 10-Gbps

switches) • An NDCLA operation to previous switch code release is not supported.

4.3.2 Updating via Telnet 1. After downloading new firmware from QLogic, change directory to the directory where the new

firmware is located. 2. ftp to the switch 3. login using username images, password images 4. enter the following commands: > bin > put firmware_xxx (where xxx is version information) > quit

5. telnet to the switch 6. login using username: <userid>, password: <password>, and enter admin mode by entering

“admin start” 7. Confirm that the new image upload was successful:

QLogic (admin): USERID> image list firmware_2.0.0.22

> image unpack firmware_xxx

8. Unpack the firmware image, then perform a hot reset of the switch as shown in Figure 10 Firmware Update via Telnet. Wait for the unpack to complete before issuing the hot reset command.



19


20 http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100564 © 2006 IBM Version 3.0, 3/10/2006

Figure 10 Firmware Update via Telnet

http://knowledge.storage.ibm.com/servers/storage/support/hbasearch/interop/hbaSearch.do Parameters Recommended settings BIOS Disabled Frame size 2048 Loop reset delay 5 (minimum) Adapter hard loop ID Disabled Hard loop ID 0 Execution throttle 100 Fast command posting Enabled >4 GB addressing Disabled (for 32-bit systems) LUNs per target 0 Enable LIP reset No Enable LIP full login Yes Enable target reset Yes Note: Enable LIP reset, Enable LIP full login, and Enable target reset

parameters control the behavior of the adapter when Windows tries to do a SCSI bus reset. You must perform a target reset to make cluster failovers work. Use the SCSI bus device reset option to clear SCSI reservations.

Login retry count 20 (minimum) Port down retry count 20 (minimum) Driver load RISC code Enabled Enable database updates No Disable database load No IOCB allocation 256 Extended error logging Disabled

Table 1 Recommended settings for the QLogic QLA23xx adapter card for Windows 2000 or 2003

QLogic (admin): USERID> image unpack firmware_2.0.0.22

Processing the command ...

A notification with the command result will appear

on the screen when the command has completed.

QLogic (admin): USERID>

image unpack command result: Passed.

The firmware will become active and the SWImageVersion

information will be updated after the next switch reset.

QLogic (admin): USERID> hotreset


http://knowledge.storage.ibm.com/servers/storage/support/hbasearch/interop/hbaSearch.do


5 SAN Configuration

5.1 Description To merge the QLogic and McData switches. The McData switch will handle the zoning, while the QLogic switch will operate in a passive mode. Since there are two separate fabrics, the following rules apply to each fabric: • Firmware Revision Levels – Verify the correct version of firmware is installed on the McData

and Qlogic products and also the correct versions of SANavigator, EFCM, and SANbox Manager are installed.

• Fabric Domain IDs – Ensure there are no duplicate domain ID’s in each fabric. • Principle Switch – Select a McData Director as the Principle Switch. • Zone Names – Ensure there are no duplicate zone/zone set names. • Soft Zoning – Ensure the McData switch has fabric zoning configured with WWN. • Zone Naming Conventions – Ensure McData and QLogic switches comply with proper zone

naming. • Operating Mode – Ensure the fabrics are in the proper operating mode. • Rerouting Delay – Ensure rerouting delay is disabled on McData. • Auto Save – Ensure the QLogic “Auto Save” is disabled. • Qlogic Zoning – Ensure default zoning is disabled on QLogic. • Merge Fabrics – Add the switches to the fabric one at a time. • Complete Zoning – Use SANavigator for Zoning Activities. • Save Zone Set – After fabrics are merged, save your current active zone set.

5.2 Firmware Revision Levels The correct versions of firmware and the correct versions of SANavigator, EFCM, and SANbox Manager must be installed before attempting to merge the fabrics. Since updates are frequent, contact your local McDATA or QLogic customer service representative to determine what versions are needed and if necessary, obtain the latest versions of applications and firmware.

5.3 Fabric Domain IDs Each switch in a fabric has a domain ID. These domain IDs are assignable and must be unique to each switch. QLogic offsets domain IDs by a factor of 96; in other words a domain ID of 1 on a McData switch will be the same as domain ID 97 on a QLogic switch. Before attempting to merge the fabrics, domain IDs must be changed to ensure unique addresses.

5.3.1 Determining the Domain ID for McData If multiple McData switches are to work in a fabric, each must have a unique domain ID. The switches must be configured with IP addresses and be powered on and accessible from the IP network. From a workstation running SANavigator:

1. Launch EFCM from SANavigator. 2. From EFCM, select Configure, Operating Parameters, Switch Parameters.



21


Figure 11 Switch Parameters (McData)

3. From the Configure Switch Parameters dialog box, change the number in Domain ID Preferred box to the appropriate domain ID.

Figure 12 Change domain ID (McData)

5.3.2 Determining the Domain ID for Qlogic 1. Launch the SANbox Manager from SANavigator.



© 2006 IBM Version 3.0, 3/10/2006


Figure 13 Launch SANbox Manager

2. Once SANbox Manager is launched, you can change the domain number in the Domain ID box from the Switch Properties SANbox Manager dialog.

Note: Subtract 96 from the domain ID of the Qlogic switches to determine whether there is a domain ID conflict with the McData switches.

Figure 14 Determine Qlogic domain ID

5.4 Principle Switch Since the Qlogic switches are in passive mode, the McData switches are used as the principle switch. This reduces build fabric traffic and hops, and ensures that Fabric Address Zoning is not used.

Setting McData as the principle switch To select McData as the Principle Switch, perform these steps,

Note: This is an offline operation, which requires a switch outage. Traffic may be moved to the alternate SAN if available. Please plan accordingly.



23


1. Use SANavigator to launch EFCM. Select Configure Operating Parameters, Fabric Parameters.

Figure 15 Setting Switch Priority

2. On the Configure Fabric Parameters dialog box, change Switch Priority to Principal.

Figure 16 Setting Switch to Principle

Disabling Qlogic Principle Switch Option For the Qlogic switch, make sure the switch does not become the Principle Switch by setting its Principal Priority to 253. This is accomplished by using TELNET. Required manual responses are in bold.

login: admin password: xxxxxxxxx Switch-2 #> admin start Switch-2 (admin) #> config edit Switch-2 (admin-config) #> set config switch A list of attributes with formatting and current values will follow. Enter a new value or simply press the ENTER key to accept the current value. If you wish to terminate this process before reaching the end of the list, press ’q’ or ’Q’ and the ENTER key to do so.



© 2006 IBM Version 3.0, 3/10/2006


AdminState (1=Online, 2=Offline, 3=Diagnostics) [Online] BroadcastEnabled (True / False) [True] InbandEnabled (True / False) [True] DefaultDomainID (decimal value, 1-239) [111] DomainIDLock (True / False) [True] SymbolicName (string, max=32 chars) [Switch-2] R_T_TOV (decimal value, 1-1000 msec) [100] R_A_TOV (decimal value, 100-100000 msec) [10000] E_D_TOV (decimal value, 10-20000 msec) [2000] FS_TOV (decimal value, 100-100000 msec) [5000] DS_TOV (decimal value, 100-100000 msec) [5000] PrincipalPriority (decimal value, 1-255) [254] 253 ConfigDescription (string, max=64 chars) [QLogic SANbox2 FC Switch] Finished configuring attributes. This configuration must be saved with the config save command and activated with the config activate command before the configuration can take effect. To discard this configuration use the config cancel command. Switch-2 (admin-config) #> config save Switch-2 (admin) #> config act The configuration will be activated. Please confirm (y/n): [n] y



25


Appendix A. Firmware Versions Firmware Levels for Chassis Components All values current as of Feb 28, 2006. Management Module

Minimum Req Most Current

Version 1.13 1.19H

Release Date 2005/02/11 2005/12/22

Revision (Main App) BRET73H BRET82H

Table 2 Management Module BIOS Information

A minimum of BRET82A is highly recommended since it contains several important fixes: 1) Repeating keystrokes fix 2) Gratuitous arp fix 3) Improved failover and firmware transfer handling

Ethernet switches Nortel Networks Layer 2/3 GbE Switch Module


Version 1.1.1.1 1.1.1.1

Release Date 2005/10/12 2005/10/12

Build Id 1.1.1.1 1.1.1.1

Table 3 Nortel Networks Layer 2/3 GbE Switch Module

Nortel Networks Layer 2-7 GbE Switch Module (GbESM) Firmware Minimum Req Most Current

Version 20.1.2.2 20.2.2.7

Release Date 2004/07/23 2005/11/05

Build Id WM01000 WM01001

Build Date 2004/03/09 10/15/2004

Revision 2001 2002

Table 4 Nortel Switch Firmware Information

Cisco Systems Intelligent Gigabit Ethernet Switch Module firmware Minimum Req Most Current

Version 12.1(22)AY1 12.1(22)EA6a

Release Date 2005/03/25 2005/01/03

Build Id BRCSMI12.1 BRCSMI12.1

Build Date tbd TBD

Revision 22AY EA6

Table 5 Cisco GbESM Firmware Information



© 2006 IBM Version 3.0, 3/10/2006


Fiber switches QLogic 2pt Fibre Channel Switch Module Firmware


Version 2.0.1.05 2.0.1.09

Release Date 2004/06/04 tbd

Build Id BRFSM BRFSM

Revision 2005 tbd

Table 6 QLogic 2pt Fibre Channel Switch BIOS Information

QLogic 6pt Fibre Channel Switch Module Firmware Minimum Req Most Current

Version 4.1.0.22 5.2.0.26

Release Date 2004/08/06 2005/08/03

Build Id 4122 5226

Revision tbd tbd

Table 7 QLogic 6pt Fibre Channel Switch BIOS Information

Notes regarding the QLogic 2pt and 6pt switches 1. Obtain this firmware from the QLogic website:

http://www.qlogic.com/support/oem_detail_all.asp?oemid=3242. Firmware versions that follow the initial release of version 2.0 support the ability to install and

activate new firmware without resetting the switch, and therefore, without disrupting traffic. The application will attempt a non-disruptive firmware installation and activation whenever possible. Important: During a hotreset operation, fabric services will be unavailable for a short period. To ensure that a Non-Disruptive Code Load and Activation (NDCLA) operation is successful, verify that all administrative changes to the fabric (if any) are complete. Changes to the fabric may disrupt the NDCLA process. See 4.3 for more information on this process.

• Brocade Entry/Enterprise SAN Switch Module Firmware Minimum Req Most Current

Version 4.2.1 4.4.1a

Release Date 2004/04/08 2005/02/15

Build Id BREFSM BREFSM

Revision 421 441a

Table 8 Brocade SAN Switch Module Firmware Information

QLogic HBA Firmware Minimum Req Most Current

Version 1.43 1.43

Table 9 QLogic Firmware Information

Note: we are now recommending v1.43 for the QLogic BIOS. Old minimum was v1.38.



27

http://www.qlogic.com/support/oem_detail_all.asp?oemid=324


Index Brocade SAN Switch

Firmware level ..............................................27 Cisco Ethernet Switch Module

Firmware Levels...........................................26 Management ................................................12

Firmware Levels Brocade SAN Switch Module .......................27 Cisco Ethernet Switch Module .....................26 Download page ..............................................9 Management Module ...................................26 Nortel Networks Switches

Layer 2/3 GbE Switch Module..................26 Layer 2-7 GbE Switch Module .................26

QLogic Fibre Channel Switch Module..........27 QLogic HBA..................................................27

Management Module Connecting To............................................7, 8 View firmware levels ......................................9

Management Module Failover .........................26

Nortel Networks Switches Configuration................................................11 Firmware Update .........................................13 Layer 2/3 Switch

Firmware Levels.......................................26 Layer 2-7 Switch

Firmware Levels.......................................26 Management ................................................11

QLogic HBA Firmware levels............................................27

QLogic Switch Disabling ports .............................................16 Firmware level..............................................27 Firmware update ....................................18, 27 SANSurfer..............................................17, 18

Related Documents ...........................................5 Repeating keystrokes ......................................26 SAN Configuration ...........................................21



© 2006 IBM Version 3.0, 3/10/2006

Date post:	20-Feb-2015
Category:	Documents
Upload:	werner-welgens
View:	199 times
Download:	1 times