Configuring a MetroCluster System With SAS Disk

Configuring a MetroCluster system with SAS disk shelves andFibreBridge 6500N bridges

FibreBridge 6500N bridges are used in MetroCluster systems whenever SAS disk shelves are used. You can install the bridgesas part of a new MetroCluster installation, as a hot-add to an existing MetroCluster system with SAS or FC disk shelves, or as ahot-swap to replace a failed bridge. You can also hot-add a SAS disk shelf to an existing stack of SAS disk shelves, and hot-remove disk shelves.

This document assumes you are installing a stretch or fabric-attached MetroCluster configuration. For installing a shared-switchconfiguration, see the Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode.The Best Practices for MetroCluster Design and Implementation TR-3548 document has more information about using bridges.All documents can be found on the NetApp Support Site at support.netapp.com.

For the most current information about supported platforms, disk shelves, cables, and Data ONTAP releases, see the IBMSystem Storage N series Introduction and Planning Guide at the N series support website (accessed and navigated as describedin Websites).MetroCluster configurations with FibreBridge 6500N bridges do not support the following:

SNMP (Simple Network Management Protocol) ACP (Alternate Control Path) SAS optical cables

This document contains the following procedures and information:

Installing bridges and SAS disk shelves as part of a new MetroCluster installation Hot-adding a stack of SAS disk shelves and bridges to an existing MetroCluster system Hot-swapping a FibreBridge Hot-adding a SAS disk shelf to a stack of SAS disk shelves Hot-removing disk shelves in systems running Data ONTAP 8.2.1 or later Converting an HA pair or stretch MetroCluster system with SAS disk shelves to a fabric-attached MetroCluster system Replacing SAS copper cables Other FibreBridge management interfaces FibreBridge error messages

MetroCluster configuration examplesThe following illustrations are examples of fully cabled stretch and fabric-attached MetroCluster systems.

1215-05845_E0 Copyright 2014 NetApp, Inc. All rights reserved.Web: web.netapp.com Feedback: [email protected]

IOM A IOM BFirst shelf

Last shelf

FC

FC1

SAS A

FC2

M1

FC FC FC

Controller 1

FC FC FC FC

Controller 2

FibreBridge A FC1

SAS A

FC2

M1FibreBridge B

FC1

SAS A

FC2

M1FibreBridge A

FC1

SAS A

FC2

M1FibreBridge B

Site 1 Site 2

Stretch MetroCluster system

Stack of SAS shelvesIOM A IOM B

First shelf

Last shelf

Stack of SAS shelves

HA interconnect

Connect each FibreBridge management 1 port to the network

Network

2 Configuring a MetroCluster system with SAS disk shelves and FibreBridge 6500N bridges

FC switch 1 FC switch 2 FC switch 3 FC switch 4


Last shelf

FC

FC1

SAS A

FC2

M1

FC FC FC

Controller 1

FC FC FC FC

Controller 2

FibreBridge A FC1

SAS A

FC2

M1FibreBridge B

FC1

SAS A

FC2

M1FibreBridge A

FC1

SAS A

FC2

M1FibreBridge B

Site 1 Site 2

Fabric MetroCluster system

Stack of SAS shelvesIOM A IOM B

First shelf

Last shelf


Inter-Switch Links (ISLs)

Connect each FibreBridge management 1 port to the network

HA Interconnect (FC-VI adapter connections)

Network

Installing bridges and SAS disk shelves as part of a new MetroClusterinstallation

You can install and cable bridges and SAS disk shelves as part of a new MetroCluster installation.

About this task

This procedure is written with the assumption that you are using the recommended bridge management interfaces: the ATTOExpressNAV GUI and ATTO QuickNAV utility.You use the ATTO ExpressNAV GUI to configure and manage a bridge, and to update the bridge firmware. You use the ATTOQuickNAV utility to configure the bridge Ethernet management 1 port.You can use other management interfaces instead, if needed, such as a serial port or Telnet to configure and manage a bridgeand to configure the Ethernet management 1 port, and FTP to update the bridge firmware. If you choose any of thesemanagement interfaces, you must meet the applicable requirements in the Other FibreBridge management interfaces section.

Related references

Other FibreBridge management interfaces on page 30

Installing bridges and SAS disk shelves as part of a new MetroCluster installation 3

Preparing for the installationPreparing to install the bridges as part of your new MetroCluster system involves ensuring that your system meets certainrequirements, including meeting setup and configuration requirements for the bridges. Other requirements include downloadingthe necessary documents, the ATTO QuickNAV utility, and bridge firmware.

Before you begin

Your system must be a supported configuration and running a supported version of Data ONTAP. For fabric-attached MetroCluster configurations, each FC switch must have one FC port available for one bridge to connect

to it. For stretch MetroCluster configurations, each controller must have one QLogic 8-Gb or 4-Gb FC port available for each

bridge to connect to it.Only QLogic 8-Gb and 4-Gb ports are supported. These ports can be onboard or on PCIe HBAs. For example, in aconfiguration with one stack of SAS disk shelves and two bridges at each site, a total of four FC ports is used on eachcontroller.

Note: For improved performance, you should use 8-Gb ports when possible. The computer you are using to set up the bridges must be running an ATTO-supported web browser to use the ATTO

ExpressNAV GUI. The ATTO-supported web browsers are Internet Explorer 8 and 9, and Mozilla Firefox 3.The ATTO Product Release Notes have an up-to-date list of supported web browsers. You can access this document usingthe following information

Steps

1. Download the following documents from the NetApp Support Site at support.netapp.com:

Interoperability Matrix Tool (IMT) Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode The Fabric-attached MetroCluster Systems Brocade Switch Configuration Guide or Fabric-attached MetroCluster

Systems Cisco Switch Configuration Guide if you are installing a fabric-attached MetroCluster system. Hardware and Service Guide for your disk shelf model

2. Go to the ATTO FibreBridge Description page on the NetApp Support Site at support.netapp.com and complete thefollowing substeps:

a) Navigate to the ATTO FibreBridge Description page by clicking Software, scrolling to Protocol Bridge, choosingATTO FibreBridge from the drop-down menu, clicking Go!, and then clicking View & Download.

b) Access the ATTO web site using the link provided and download the following: ATTO FibreBridge 6500N Installation and Operation Manual ATTO QuickNAV utility (to the computer you are using for setup)

c) Go to the ATTO FibreBridge 6500N Firmware Download page and do the following: Navigate to the ATTO FibreBridge 6500N Firmware Download page by clicking Continue at the end of the ATTO

FibreBridge Description page. Download the bridge firmware file using Steps 1 through 3 of that procedure.

You will update the firmware on each bridge later, when instructed in the Hot-adding the stack of shelves section. Make a copy of the ATTO FibreBridge 6500N Firmware Download page and release notes for reference when you

are instructed to update the firmware on each bridge.

3. Check the Interoperability Matrix Tool to verify that your system meets all configuration requirements for the bridge.4. Check the Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode to verify that your system

meets all applicable MetroCluster requirements as defined in the MetroCluster installation section.

5. Gather the hardware and information needed to use the recommended bridge management interfacesthe ATTOExpressNAV GUI and ATTO QuickNAV utilityby completing the following substeps:


a) Acquire a standard Ethernet cable (which connects from the bridge Ethernet management 1 port to your network).b) Determine a non-default user name and password (for accessing the bridges).

It is recommended that you change the default user name and password.c) Obtain an IP address, subnet mask, and gateway information for the Ethernet management 1 port on each bridge.d) Disable VPN clients on the computer you are using for setup.

Active VPN clients cause the QuickNAV scan for bridges to fail.6. Acquire four screws for each bridge (to flush-mount the bridge "L" brackets securely to the front of the rack).

The openings in the bridge "L" brackets are compliant with rack standard ETA-310-X for 19-inch (482.6 mm) racks.

Installing the new systemAfter ensuring that the system meets all the requirements in the Preparing for the installation section, you can install your newsystem.

About this task

Multipath HA is the only supported configuration for MetroCluster systems that use bridges.Both controllers must have access through the bridges to the disk shelves in each stack.

You should use an equal number of disk shelves at each site. The system connectivity requirements for maximum distances for disk shelves, FC switches, and backup tape devices using

50-micron, multimode fiber-optic cables, also apply to FibreBridges.The IBM System Storage N series Introduction and Planning Guide has detailed information about system connectivityrequirements.

Steps

1. Properly ground yourself.

2. Install the platforms and, if applicable, the FC switches.

For instructions on installing the platforms, see the Installation and Setup Instructions that came with your platform. Forinstructions on installing FC switches, see the Fabric-attached MetroCluster Systems Brocade Switch Configuration Guideor Fabric-attached MetroCluster Systems Cisco Switch Configuration Guide.

3. Install the disk shelves, power them on, and set the shelf IDs.

For instructions, see the Hardware and Service Guide for your disk shelf model.Note: You must power-cycle each disk shelf.

Note: Shelf IDs must be unique for each SAS disk shelf within the entire storage system.

4. Install each bridge by completing the following substeps:

a) Secure the "L" brackets on the front of the bridge to the front of the rack (flush-mount) with the four screws you acquiredin the Preparing for the installation section.

For more information and an illustration of the installation, see the ATTO FibreBridge 6500N Installation and OperationManual.

b) Connect each bridge to a power source that provides a proper ground.c) Power on each bridge.

Note: For maximum resiliency, ensure that bridges attached to the same stack of disk shelves are connected todifferent power sources.

The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge has completed its power-onself test sequence.


For information about power requirements and LEDs, see the ATTO FibreBridge 6500N Installation and OperationManual.

5. Connect the Ethernet management 1 port on each bridge to your network using an Ethernet cable.

Note: The Ethernet management 1 port enables you to quickly download the bridge firmware (using ATTO ExpressNAVor FTP management interfaces) and to retrieve core files and extract logs.

6. Configure the Ethernet management 1 port for each bridge by following the procedure in the ATTO FibreBridge 6500NInstallation and Operation Manual, section 2.0.

Note: When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that isconnected by the Ethernet cable is configured. For example, if you also wanted to configure the Ethernet management 2port, you would need to connect the Ethernet cable to port 2 and run QuickNAV.

7. Configure the bridges.

Be sure to make note of the user name and password that you designate.

a) Configure the data rate/speed of the bridge FC ports.When you set the bridge FC port speed, you must set it to the appropriate speed for the length of cable you are using: upto 150 meters is 8 Gb, up to 270 meters is 4 Gb, up to 500 meters is 2 Gb.

The most current information on supported distance can be found in the Interoperability Matrix Tool on the NetAppSupport Site at support.netapp.com.

If you have a fabric-attached MetroCluster system, the following applies:

If you are using the Cisco 9222i switch, you must set the FC port on each bridge to the same speed as the Cisco9222i switch port to which the bridge will connect.For example, if the switch port is 8 Gb, you need to set the bridge port to 8 Gb. If the switch port is 4 Gb, you need toset the bridge port to 4 Gb.

If you are using a Brocade switch, the bridge port must be set to 8 Gb.b) Configure the connection mode that the bridges use to communicate across the FC network.

If you have a fabric-attached MetroCluster system, you must set the bridge connection mode to ptp (point-to-point). If you have a stretch MetroCluster system, you must set the bridge connection mode to loop.

Note: If you are connecting bridges to an X1143A FC adapter in the controller, you must set the bridge connectionmode to ptp.

For example, if you were to use the command line interface (CLI) to set the bridge FC 1 port speed to 8 Gb and theconnection mode to ptp, you would enter the following three commands; the third command saves the configurationchanges:Set FCDataRate 1 8Gb

Set FCConnMode 1 ptp

SaveConfiguration

Result: You are prompted to restart the bridge.

The ATTO FibreBridge 6500N Installation and Operation Manual has the most current information on available commandsand how to use them.

8. Update the firmware on each bridge to the latest version by following the instructionsstarting with Step 4on theFibreBridge 6500N Download page.

9. Cable the disk shelves to the bridges by completing the following substeps:

a) Daisy-chain the disk shelves in each stack.For information about daisy-chaining disk shelves, see the Installation and Service Guide for your disk shelf model.

b) For each stack of disk shelves, cable IOM A square port of the first shelf to SAS port A on FibreBridge A.


c) For each stack of disk shelves, cable IOM B circle port of the last shelf to SAS port A on FibreBridge B.Each bridge has one path to its stack of disk shelves; bridge A connects to the A-side of the stack through the first shelf, andbridge B connects to the B-side of the stack through the last shelf.

Note: The bridge SAS port B is disabled.

The following illustration shows a set of bridges cabled to a stack of three disk shelves:


Last shelf

FC1

SAS A

FC2

M1

FibreBridge A FC1

SAS A

FC2

M1

FibreBridge B


10. Verify that each bridge can detect all disk drives and disk shelves it is connected to.

If you are using the... Then...

ATTO ExpressNAV GUI a. In a supported web browser, enter the IP address of a bridge in the browser box.You are brought to the ATTO FibreBridge 6500N home page, which has a link.

b. Click the link and enter your user name and the password that you designated when you configured thebridge.The ATTO FibreBridge 6500N status page appears with a menu to the left.

c. Click Advanced in the menu.

d. Enter the following command, and then click Submit:

sastargets

Serial port connection Enter the following command:

sastargets

Example

The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines are sequentiallynumbered so you can quickly count the devices. For example, the following output shows that 10 disks are connected.

Tgt VendorID ProductID Type SerialNumber 0 NETAPP X410_S15K6288A15 DISK 3QP1CLE300009940UHJV 1 NETAPP X410_S15K6288A15 DISK 3QP1ELF600009940V1BV 2 NETAPP X410_S15K6288A15 DISK 3QP1G3EW00009940U2M0 3 NETAPP X410_S15K6288A15 DISK 3QP1EWMP00009940U1X5 4 NETAPP X410_S15K6288A15 DISK 3QP1FZLE00009940G8YU 5 NETAPP X410_S15K6288A15 DISK 3QP1FZLF00009940TZKZ 6 NETAPP X410_S15K6288A15 DISK 3QP1CEB400009939MGXL 7 NETAPP X410_S15K6288A15 DISK 3QP1G7A900009939FNTT 8 NETAPP X410_S15K6288A15 DISK 3QP1FY0T00009940G8PA 9 NETAPP X410_S15K6288A15 DISK 3QP1FXW600009940VERQ


Note: If the text response truncated appears at the beginning of the output, you can use Telnet to connect to thebridge and enter the same command to see all the output.

11. Verify the command output shows the bridge is connected to all disks and disk shelves in the stack that it is supposed to beconnected to.

If the output is... Then...

Correct Repeat Step 10 for each remaining bridge.

Not correct a. Check for loose SAS cables or correct the SAS cabling by repeating Step 9.

b. Repeat Step 10.

12. Cable each bridge to the controllers or FC switches, depending on your MetroCluster configuration:

If you have a... Then...

Stretch MetroClusterconfiguration

a. Cable FC port 1 of bridge A to an 8-Gb or 4-Gb FC port on the controller at Site 1.

b. Cable FC port 2 of bridge A to the same speed FC port on the controller at Site 2.

c. Repeat Substeps a and b for bridge B.

Fabric-attachedMetroClusterconfiguration

a. Cable FC port 1 of bridge A to FC switch 1 at Site 1.

b. Cable FC port 1 of bridge B to FC switch 2 at Site 1.

c. Repeat Substeps a and b for Site 2.

Note: You can connect the bridge to any available port on the FC switch. Whichever port you choose, youmust use the same port consistently on each FC switch that you are connecting at both sites. This is for ease ofmanageability. For example, if you choose FC switch port 3, use only that port on each FC switch.

Note: The FC port 2 on each bridge remains unused.

13. Connect and configure the controllers, and if applicable the FC switches, following the procedure for your MetroClusterconfiguration in the Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode:a) If applicable, configure the FC switches.

Note: Each FC switch port connected to the bridge must be set to G_Port.b) Cable the FC-VI and inter-switch link as appropriate for your configuration.c) Check whether your system has disk autoassignment enabled by entering the following command at the console of either

controller.options disk.auto_assign

If disk autoassignment is enabled, the output shows disk.auto_assign on.d) If your system does not have disk autoassignment enabled or if disk drives in the same stack are owned by both

controllers, assign disk drives to the appropriate pools by using the information in the Data ONTAP High Availabilityand MetroCluster Configuration Guide for 7-Mode; otherwise, go to the next step.

Note: If you are splitting a single stack of disk shelves between two controllers, you must disable disk autoassignmentbefore assigning disk ownership; otherwise, when you assign any single disk drive, the remaining disk drives might beautomatically assigned to the same controller and pool.

To disable disk autoassignment, run the options disk.auto_assign off command at the console of bothcontrollers.

e) Configure the system and enable licenses as needed using the information about configuring an HA pair in the DataONTAP High Availability and MetroCluster Configuration Guide for 7-Mode.

14. Verify disk shelf connectivity by entering the following command at the system console of either controller:


sysconfig -v

The output shows the following:

Each bridge on a separate line. Additionally, for fabric-attached MetroCluster configurations, each bridge appears undereach FC port to which it is visible. For stretch MetroCluster configurations, each bridge is directly connected.For example, the output for a fabric-attached MetroCluster configuration would look like the following:

FC-to-SAS Bridge:Brocade300-1:9.126L0 : ATTO FibreBridge6500N 1.11 FB6500N100189Brocade300-2:1.126L0 : ATTO FibreBridge6500N 1.11 FB6500N100162

For example, the output for a stretch MetroCluster configuration would look like the following:

FC-to-SAS Bridge:125L0 : ATTO FibreBridge6500N 1.10 FB6500N100053125L0 : ATTO FibreBridge6500N 1.10 FB6500N100048

Each disk shelf is on a separate line under each FC port to which it is visible (for fabric-attached or stretch MetroClusterconfigurations).For example:

Shelf 0: IOM6 Firmware rev. IOM6 A: 0111 IOM6 B: 0111Shelf 1: IOM6 Firmware rev. IOM6 A: 0111 IOM6 B: 0111

Each disk drive is on a separate line under each FC port to which it is visible (for fabric-attached or stretch MetroClusterconfigurations).For example, the output for a fabric-attached MetroCluster configuration would look like the following:

Brocade300-1:9.126L1 : NETAPP X421_HCOBD450A10 NA01 418.0GB (879097968 520B/sect) Brocade300-1:9.126L2 : NETAPP X421_HCOBD450A10 NA01 418.0GB (879097968 520B/sect)


125L1 : NETAPP X421_HCOBD450A10 NA01 40.0GB (82262416 520B/sect) 125L2 : NETAPP X421_HCOBD450A10 NA01 40.0GB (82262416 520B/sect)

Related concepts

FibreBridge error messages on page 31

Hot-adding a stack of SAS disk shelves and bridges to an existingMetroCluster system

You can hot-add a stack of SAS disk shelves and set of bridges to an existing MetroCluster system that has SAS or FC diskshelves.

About this task

This procedure is written with the assumption that you are using the recommended bridge management interfaces: the ATTOExpressNAV GUI and ATTO QuickNAV utility.You use the ATTO ExpressNAV GUI to configure and manage a bridge, and to update the bridge firmware. You use the ATTOQuickNAV utility to configure the bridge Ethernet management 1 port.

Hot-adding a stack of SAS disk shelves and bridges to an existing MetroCluster system 9

You can use other management interfaces instead, if needed, such as a serial port or Telnet to configure and manage a bridgeand to configure the Ethernet management 1 port, and FTP to update the bridge firmware. If you choose any of thesemanagement interfaces, you must meet the applicable requirements in the Other FibreBridge management interfaces section.

Related references


Preparing for the hot-addPreparing to hot-add a stack of SAS disk shelves and bridges involves downloading documents as well as the disk drive anddisk shelf firmware.

Before you begin

Your system must be a supported configuration and running a supported version of Data ONTAP. All disk drives and disk shelves in the system must be running the latest firmware version. For fabric-attached MetroCluster configurations, each FC switch must have one FC port available for one bridge to connect

to it.

Note: If you are hot-adding to a system that contains FC disk shelves, you might need to upgrade the FC switchdepending on the FC switch compatibility.

For stretch MetroCluster configurations, each controller must have one QLogic 8-Gb or 4-Gb FC port available for eachbridge to connect to it.Only QLogic 8-Gb and 4-Gb ports are supported. These ports can be onboard or on PCIe HBAs. For example, in aconfiguration with one stack of SAS disk shelves and two bridges at each site, a total of four FC ports are used on eachcontroller.

Note: For improved performance, you should use 8-Gb ports when possible. The computer you are using to set up the bridges must be running an ATTO supported web browser to use the ATTO

ExpressNAV GUI: Internet Explorer 8 or 9, or Mozilla Firefox 3.The ATTO Product Release Notes have an up-to-date list of supported web browsers. You can access this document usingthe information in the steps.

Steps


Interoperability Matrix Tool Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode Hardware and Service Guide for your disk shelf model






You will update the firmware on each bridge later, when instructed in the Hot-adding the stack of shelves section.


Make a copy of the ATTO FibreBridge 6500N Firmware Download page and release notes for reference when youare instructed to update the firmware on each bridge.

3. Check the Interoperability Matrix Tool to verify that the disk shelf you are hot-adding is supported.4. Download the latest disk and disk shelf firmware:

Note: In this step you are only completing the download portion of the instructions provided in the links. You also need tocopy the installation portion of the instructions in the links for use later. The installation portion of the instructions isfound in the Hot-adding the stack of shelves section.

a) Download the disk firmware from support.netapp.com/NOW/download/tools/diskfw and make a copy of the diskfirmware instructions for reference later.

b) Download the disk shelf firmware from support.netapp.com/NOW/download/tools/diskshelf and make a copy of the diskshelf firmware instructions for reference later.

5. Gather the hardware and information needed to use the recommended bridge management interfacesthe ATTOExpressNAV GUI and ATTO QuickNAV utilityby completing the following substeps:a) Acquire a standard Ethernet cable (which connects from the bridge Ethernet management 1 port to your network).b) Determine a non-default user name and password (for accessing the bridges).


Active VPN clients cause the QuickNAV scan for bridges to fail.6. Acquire four screws for each bridge (to flush-mount the bridge "L" brackets securely to the front of the rack).

The openings in the bridge "L" brackets are compliant with rack standard ETA-310-X for 19-inch (482.6 mm) racks.

Hot-adding the stack of shelvesAfter ensuring that the system meets all the requirements in the Preparing for the hot-add section, you can hot-add a stack ofSAS disk shelves and bridges.

About this task

This is a nondisruptive procedure if all bridge requirements in the Interoperability Matrix Tool are met. Multipath HA is the only supported configuration for MetroCluster systems using bridges.

Both controllers must have access through the bridges to the disk shelves in each stack. You should hot-add an equal number of disk shelves at each site. The system connectivity requirements for maximum distances for disk shelves, FC switches, and backup tape devices using

50-micron, multimode fiber-optic cables also apply to bridges.The IBM System Storage N series Introduction and Planning Guide has detailed information about system connectivityrequirements.

Attention: Hot-adding a disk shelf requires you to update the disk drive firmware on the hot-added disk shelf by running thedisk_fw_update command. Running this command can be disruptive if the firmware on existing disk drives in your systemis an older version.

Steps


2. Follow Steps 3 through 11 of the Installing bridges and SAS disk shelves as part of a new MetroCluster installation sectionof this document.

3. Check whether your system has disk autoassignment enabled by using the options disk command at the console of eithercontroller.

Hot-adding a stack of SAS disk shelves and bridges to an existing MetroCluster system 11

If disk autoassignment is enabled, the output shows disk.auto_assign on.

4. If your system does not have disk autoassignment enabled, assign disk drive ownership by using the information in the DataONTAP High Availability and MetroCluster Configuration Guide for 7-Mode.



Note: Do not add disk drives to aggregates or volumes until after the disk drive and disk shelf firmware has been updated.

5. Update the disk drive firmware to the most current version by entering the following command at the system console:disk_fw_update

You must run this command on both controllers.

If needed, refer to the instructions you copied when you downloaded the firmware as part of the preparation for this task.

6. Update the disk shelf firmware to the most current version by using the instructions you copied when you downloaded thefirmware as part of the preparation for this task.

You can run the commands in the procedure from the system console of either controller.

7. If applicable, repeat this procedure for the remote site.

Related concepts


Hot-swapping a FibreBridgeWhen a bridge fails, you can hot-swap the bridge without disrupting the storage system when the system is cabled correctly forMultipath HA.

About this task


Related references


Preparing for the hot-swapPreparing to hot-swap a bridge as part of your new MetroCluster system involves downloading documents and bridge firmware.

Steps

1. Verify that the system is cabled for Multipath HA by entering the following command at the system console of bothcontrollers:


sysconfig

Note: It might take up to a minute for the system to complete discovery.

For example, the following output states that the system is cabled for Multipath HA:

NetApp Release 8.1 7-Mode: Thu Sep 22 02:50:01 PDT 2011 System ID: 1231231231 (stqa-3270-2); partner ID: 4564564564 (stqa-3270-1) System Serial Number: 700000123123 (stqa-3270-2); partner Serial Number: 700000456456 (stqa-3270-1) System Rev: B0 System Storage Configuration: Multi-Path HASystem ACP Connectivity: NA

2. Verify that the command output shows that the system is cabled for Multipath HA.

If the System StorageConfiguration states that it is...

Then...

Multipath HA Go to Step 3.

Mixed-path or single-path HA You must correct the SAS and FC cabling because not all disk drives are accessible through the newbridge.

Attention: If you must correct cabling on a system that is serving data, you can perform hot-cable moves only if one path to the disk drives is maintained.

Attention: When bridges are not cabled as Multipath HA, restarting a bridge might cause loss ofaccess to the disk drives and result in a multi-disk panic.








4. Complete the following substeps to use the recommended bridge management interfaces: the ATTO ExpressNAV GUI andATTO QuickNAV utility:

a. Determine a non-default username and password (for accessing the bridge).It is recommended that you change the default username and password.

b. Disable VPN clients on the computer you are using for setup.Active VPN clients cause the QuickNAV scan for bridges to fail.

Note: For the new bridge, use the IP address, subnet mask, and gateway information for the Ethernet management 1 portfrom the failed bridge..

Hot-swapping a FibreBridge 13

Hot-swapping the FibreBridgeAfter ensuring that the system meets all the requirements in the Preparing for the hot-swap section, you can hot-swap a bridge.

Steps


2. Power off the failed bridge and disconnect it from the power source.

3. Disconnect the cables connected to the failed bridge.

You should make note of the port each cable was connected to.

4. Remove the failed bridge from the rack.

5. Install the new bridge into the rack.

6. Reconnect the power cord and Ethernet cable.

Note: Do not reconnect the SAS or FC cables at this time.

7. Connect the bridge to a power source and turn it on.

The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge has completed its power-on selftest sequence.


Note: When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that isconnected by the Ethernet cable is configured. For example, if you also wanted to configure the Ethernet management 2port, you would need to connect the Ethernet cable to port 2 and run QuickNAV.










For example, if you were to use the command line interface (CLI) to set the bridge FC 1 port speed to 8 Gb and theconnection mode to ptp, you would enter the following three commands; the third command saves the configurationchanges:


Set FCDataRate 1 8Gb


SaveConfiguration



10. Update the firmware on the bridge to the latest version by following the instructionsstarting with Step 4on theFibreBridge 6500N Download page.

11. Reconnect the SAS and FC cables to the same ports on the new bridge.

12. Verify that each bridge can see all disk drives and disk shelves it is connected to.


ATTO ExpressNAV GUI a. In a supported web browser, enter the IP address of the bridge in the browser box.You are brought to the ATTO FibreBridge 6500N home page, which has a link.




sastargets


sastargets

Example

The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines are sequentiallynumbered so that you can quickly count the devices. For example, the following output shows that 10 disks are connected:


Note: If the text response truncated appears at the beginning of the output, you can use Telnet to access the bridgeand enter the same command to see all the output.

13. Verify that the command output shows that the bridge is connected to all disks and disk shelves in the stack it is supposed tobe connected to.


Correct Repeat Step 12 for each remaining bridge.

Hot-swapping a FibreBridge 15


Not correct a. Check for loose SAS cables or correct the SAS cabling by repeating Step 11.

b. Repeat Step 12.

14. Verify that both controllers have access through the new bridge to the disk shelves (that the system is cabled for MultipathHA), by entering the following command at the system console of both controllers:sysconfig


For example, the following output states that the system is cabled for Multipath HA:

NetApp Release 8.1 7-Mode: Thu Sep 22 02:50:01 PDT 2011 System ID: 1231231231 (stqa-3270-2); partner ID: 4564564564 (stqa-3270-1) System Serial Number: 700000123123 (stqa-3270-2); partner Serial Number: 700000456456 (stqa-3270-1) System Rev: B0 System Storage Configuration: Multi-Path HASystem ACP Connectivity: NA

15. Verify that the command output shows that the system is cabled for Multipath HA.

If the System StorageConfiguration states that it is...

Then...

Multipath HA You are done.

Mixed-path or single-path HA You must correct the SAS and FC cabling because not all disk drives are accessible through the newbridge.

Attention: If you must correct cabling on a system that is serving data, you can perform hot-cable moves only if one path to the disk drives is maintained.

Attention: When bridges are not cabled as Multipath HA, restarting a bridge might cause loss ofaccess to the disk drives and result in a multi-disk panic.

16. Return the failed part to NetApp as described in the RMA instructions shipped with the kit. Contact technical support at support.netapp.com, 888-463-8277 (North America), 00-800-44-638277 (Europe), or +800-800-80-800 (Asia/Pacific) if youneed the RMA number or additional help with the replacement procedure.

Related concepts


Hot-adding a SAS disk shelf to a stack of SAS disk shelvesYou can hot-add one or more SAS disk shelves to a stack of SAS disk shelves.

Preparing for the hot-addPreparing to hot-add a disk shelf involves downloading documents as well as the disk drive and disk shelf firmware.

Before you begin

The disk shelf you are installing must be supported. All disk drives and disk shelves in the system must be running the latest firmware version.


Steps


Interoperability Matrix Tool Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode Hardware and Service Guide for your disk shelf model

2. Download the latest disk and disk shelf firmware:

Note: In this step, you are only completing the download portion of the instructions provided in the links. You also needto copy the installation portion of the instructions in the links for use later. The installation portion is found in the Hot-adding the shelf section.

a) Download the disk firmware from support.netapp.com/NOW/download/tools/diskfw and make a copy of the diskfirmware instructions for reference later.

b) Download the disk shelf firmware from support.netapp.com/NOW/download/tools/diskshelf and make a copy of the diskshelf firmware instructions for reference later.

3. Check the Interoperability Matrix Tool to verify that the disk shelf you are hot-adding is supported.

Hot-adding the shelfAfter ensuring that the system meets all the requirements in the Preparing for the hot-add section, you can hot-add a disk shelf.

About this task

This procedure is for hot-adding a disk shelf to the logical last disk shelf in a stack.The logical last disk shelf in a stack is connected from IOM B to bridge B.

This is a nondisruptive procedure. You should hot-add an equal number of disk shelves at each site. If you are hot-adding more than one disk shelf, you must hot-add one disk shelf at a time.

Attention: Hot-adding a disk shelf requires you to update the disk drive firmware on the hot-added disk shelf by running thedisk_fw_update command. Running this command can be disruptive if the firmware on existing disk drives in your systemis an older version.

Steps


2. Install and cable the disk shelf by completing the following substeps:

a) Install the disk shelf, power it on, and set the shelf ID.For instructions, see the Hardware and Service Guide for your disk shelf model.

Note: You must power-cycle the disk shelf.

Note: Shelf IDs must be unique for each SAS disk shelf within the entire storage system.b) Disconnect the SAS cable from the IOM B circle port of the last shelf in the stack and reconnect it to the same port in the

new shelf.

The other end of this cable remains connected to bridge B.c) Daisy-chain the new disk shelf by cabling the new shelf IOM square ports (of IOM A and IOM B) to the last shelf IOM

circle ports (of IOM A and IOM B).For an example of a cabled stack of disk shelves, you can reference any of the illustrations in this document.

3. Verify disk shelf connectivity by entering the following command at the system console of either controller:sysconfig -v

Hot-adding a SAS disk shelf to a stack of SAS disk shelves 17

The output shows the following:

Each bridge on a separate line and, for fabric-attached MetroCluster configurations, under each FC port to which it isvisible; otherwise, for stretch MetroCluster configurations, directly connected.For example, the output for a fabric-attached MetroCluster configuration would look like the following:

FC-to-SAS Bridge:Brocade300-1:9.126L0 : ATTO FibreBridge6500N 1.11 FB6500N100189Brocade300-2:1.126L0 : ATTO FibreBridge6500N 1.11 FB6500N100162


FC-to-SAS Bridge:125L0 : ATTO FibreBridge6500N 1.10 FB6500N100053125L0 : ATTO FibreBridge6500N 1.10 FB6500N100048

Each disk shelf on a separate line under each FC port to which it is visible (for fabric-attached or stretch MetroClusterconfigurations).For example:

Shelf 0: IOM6 Firmware rev. IOM6 A: 0111 IOM6 B: 0111Shelf 1: IOM6 Firmware rev. IOM6 A: 0111 IOM6 B: 0111

Note: Depending on the version of Data ONTAP your system is running, the output might appear differently. Each disk drive on a separate line under each FC port to which it is visible (for fabric-attached or stretch MetroCluster

configurations).For example, the output for a fabric-attached MetroCluster configuration would look like the following:

Brocade300-1:9.126L1 : NETAPP X421_HCOBD450A10 NA01 418.0GB (879097968 520B/sect) Brocade300-1:9.126L2 : NETAPP X421_HCOBD450A10 NA01 418.0GB (879097968 520B/sect)


125L1 : NETAPP X421_HCOBD450A10 NA01 40.0GB (82262416 520B/sect) 125L2 : NETAPP X421_HCOBD450A10 NA01 40.0GB (82262416 520B/sect)

4. Check that your system has disk autoassignment enabled by entering the following command at the console of eithercontroller:options disk.auto_assign

If disk autoassignment is enabled, the output shows options disk.auto_assign on.

5. If your system does not have disk autoassignment enabled or if disk drives in the same stack are owned by both controllers,assign disk drives to the appropriate pools by using the information in the Data ONTAP High Availability and MetroClusterConfiguration Guide for 7-Mode; otherwise, go to the next step.



Note: Do not add disk drives to aggregates or volumes until after the disk drive and disk shelf firmware has been updated.

6. Update the disk drive firmware to the most current version by entering the following command at the system console:disk_fw_update


You must run this command on both controllers.

If needed, refer to the instructions you copied when you downloaded the firmware as part of the preparation for this task inthe Preparing for the hot-add section.

7. Update the disk shelf firmware to the most current version by using the instructions you copied when you downloaded thefirmware as part of the preparation for this task.

You can run the commands in the procedure from the system console of either controller.

8. If you are hot-adding more than one disk shelf, repeat Steps 1 through 7 for each disk shelf that you are hot-adding.

Hot-removing disk shelves in systems running Data ONTAP 8.2.1 or laterIf your system is running Data ONTAP 8.2.1 or later, you can hot-remove disk shelvesphysically remove disk shelves thathave had the aggregates removed from the disk drivesin a multipath HA stretch or fabric MetroCluster configuration withFibreBridge 6500N bridges that is up and serving data. You can hot-remove one or more disk shelves from anywhere within astack of disk shelves or remove a stack of disk shelves.

Before you begin

Your storage system must be running Data ONTAP 8.2.1 or later. Your storage system must be a multipath HA system. You must have already removed all aggregates from the disk drives in the disk shelves you are removing.

Attention: Do not attempt this procedure unless you have already removed all aggregates from the disk shelves you areremoving.

For MetroCluster configurations, removing aggregates involves splitting the mirrored aggregates on the disk shelves you areremoving and then re-creating the mirrored aggregates with another set of disk drives. For procedures, see Splitting amirrored aggregate and Rejoining split aggregates in the Data ONTAP Data Protection Online Backup and RecoveryGuide for 7-Mode.

As a best practice, you should remove disk drive ownership after you remove the aggregates from the disk drives in the diskshelves you are removing.

Note: This procedure follows the best practice of removing disk drive ownership; therefore, steps are written with theassumption that you have removed disk drive ownership.

For information about removing disk drive ownership, see the Removing ownership from a disk procedure in the StorageManagement Guide for 7-Mode. This document is available on the NetApp Support Site at support.netapp.com.

Note: The procedure for removing ownership from disk drives requires you to disable disk autoassignment. You reenabledisk autoassignment when prompted at the end of this shelf hot-remove procedure.

Multipath HA configurations cannot be in a takeover state. If you are removing one or more disk shelves from within a stack, you must have factored the distance to bypass the disk

shelves you are removing; therefore, if the current cables are not long enough, you need to have longer cables available.The IBM System Storage N series Introduction and Planning Guide at the N series support website (accessed and navigatedas described in Websites) contains information about supported SAS cables.

About this task

This procedure follows cabling best practices; therefore, references to IOMs and IOM SAS square and circle ports align withthe best practices. If your storage system is cabled differently from what is prescribed as best practice, the IOM and/or IOMports might be different.

Path A refers to the A-side disk shelf I/O module (IOM A) located in the top or left of the disk shelf depending on your diskshelf model.

Path B refers to the B-side disk shelf I/O module (IOM B) located in the bottom or right of the disk shelf depending on yourdisk shelf model.

The first disk shelf in the stack is the disk shelf with a SAS square port directly connected to a bridge.

Hot-removing disk shelves in systems running Data ONTAP 8.2.1 or later 19

The interim disk shelf in the stack is the disk shelf directly connected to other disk shelves in the stack. The last disk shelf in the stack is the disk shelf with a SAS circle port directly connected to a bridge. The next disk shelf is the disk shelf downstream of the disk shelf being removed, in depth order. The previous disk shelf is the disk shelf upstream of the disk shelf being removed, in depth order. After completing this procedure for the local site, you are instructed to repeat this procedure for the remote site if you are

also removing disk shelves for the remote site.

Steps

1. Verify that your system configuration is Multi-Path HA by entering the following command from the console of eithercontroller:sysconfig

It might take up to a minute for the system to complete discovery.

The configuration is listed in the System Storage Configuration field.

Attention: If your system configuration is shown as something other than Multi-Path HA, you cannot continue with thisprocedure. Your system must meet the prerequisites stated in the Before you begin section of this procedure.

2. Verify that no aggregates are on the disk drives in the disk shelves you are removing, by completing the following substeps:

a) Enter the following command at the console of either controller:aggr status -r

If you have traditional volumes, you enter the vol status -r command.b) Check the output SHELF column to verify that the disk shelves from which you have removed aggregates do not appear.

Attention: If any disk drives in the disk shelves you are removing are anything other than spares or failed, you cannotcontinue with this procedure. Your system must meet the prerequisites stated in the Before you begin section of thisprocedure.

Example

The following aggr status -r output is for a stretch MetroCluster system with disk shelf 3 being removed. Becauseaggregates have been removed from disk shelf 3, only disk shelves in the stack with aggregates (disk shelves 2 and 4) appearin the SHELF column:

fas6200> aggr status -r

RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)--------- ------ ------------- ---- ---- ---- ----- --------------

--------------

...

dparity 2c.125L3 2c 2 2 FC:A 1 BSAS 7200 847555/1735794176 847884/1736466816 parity 2c.125L12 2c 2 11 FC:A 1 BSAS 7200 847555/1735794176 847884/1736466816data 2c.125L15 2c 4 14 FC:A 1 BSAS 7200 423111/866531584 423946/868242815 ...

3. Verify that ownership is removed from the disk drives on the disk shelves you are removing:

a) Enter the following command at the console of either controller:disk show -v

b) Check the output and take the appropriate action:


If the output shows... Then...

Not Owned or FAILED Go to the next step.The disk drives in the disk shelves you are removing are in the correct state.

Any disk drives in the diskshelves you are removinghave ownership

You can use the Removing ownership from a disk procedure referenced in the Beforeyou begin section of this procedure.

Example

The following output for the disk show -v command on a stretch MetroCluster system shows that disk drives on thedisk shelf being removed (disk shelf 3) are inOWNER a correct state for removal. Two disk drives have the ownershipremoved; therefore, Not Owned appears in the column. Two disk drives are failed; therefore, FAILED appears in thePOOL column:

fas6200> disk show -v

DISK OWNER POOL SERIAL NUMBER DR HOME CHKSUM---------- ------------- ----- ------------- -------------

-------

...

2c.125L24 Not Owned NONE N11DNLML dg-8020-03(536871641) Block2b.125L27 Not Owned NONE 9QMC2777 dg-8020-04(536871573) Zoned2c.125L43 dg-8020-04(536871573) FAILED 9QMCPPWP dg-8020-04(536871573) Block2c.125L9 dg-8020-03(536871641) FAILED 9QMCDZQZ dg-8020-03(536871641) Block

4. Physically identify and note the disk shelf IDs on the front of the disk shelves you are removing.

You need to be certain of which disk shelves you are removing so that you can correctly recable path A and path B later inthis procedure.

5. If you are removing an entire stack of disk shelves, complete the following substeps; otherwise, go to the next step:

a) Remove all SAS cables on path A and path B.This includes bridge-to-shelf cables and shelf-to-shelf cables for all disk shelves in the stack you are removing.

b) Go to Step 9.6. If you are removing one or more disk shelves from a stack (but keeping the stack), recable the applicable path A stack

connections to bypass the disk shelves you are removing by completing the applicable set of substeps:

If you are removing more than one disk shelf, complete the applicable set of substeps one disk shelf at a time.

Hot-removing disk shelves in systems running Data ONTAP 8.2.1 or later 21

If you areremoving...

Then...

The first disk shelfin a stack

For path A, do the following:

a. Remove the cable connecting the IOM A SAS circle port of the first disk shelf and the IOM A SAS square portof the second disk shelf in the stack and set it aside.

b. Move the cable connecting the bridge to the IOM A SAS square port of the first disk shelf to the IOM A SASsquare port of the second disk shelf in the stack.

Later, when you are instructed to repeat this procedure for path B, do the following:

a. Remove the cable connecting the IOM B SAS circle port of the first disk shelf and the IOM B SAS square portof the second disk shelf in the stack and set it aside.

b. Go to the next step in this procedure.You have completed the recabling for path B.

An interim diskshelf in a stack


a. Remove the cable connecting the IOM A SAS circle port of the disk shelf being removed and the IOM A SASsquare port of the next disk shelf in the stack and set it aside.

b. Move the cable connecting the IOM A SAS square port of the disk shelf being removed to the IOM A SASsquare port of the next disk shelf in the stack.

Later, when you are instructed, repeat these substeps for path B.

The last disk shelfin a stack


a. Remove the cable connecting the IOM A SAS square port of the last disk shelf and the IOM A SAS circle portof the previous disk shelf in the stack and set it aside.

b. Go to the next step in this procedure.You have completed the recabling for path A.

Later, when you are instructed to repeat this procedure for path B, do the following:

a. Remove the cable connecting the IOM B SAS square port of the last disk shelf and the IOM B SAS circle portof the previous disk shelf in the stack and set it aside.

b. Move the cable connecting the bridge to the IOM B SAS circle port of the last disk shelf to the IOM B SAScircle port of the previous disk shelf in the stack.

7. Verify that the cabling on path A has successfully bypassed the disk shelves you are removing and all disk drives on the diskshelves you are removing are still connected through path B, by entering the following command from the console of eithercontroller:storage show disk -p

It might take up to a minute for the system to complete discovery.

Example

In this example of how the output should appear for a stretch MetroCluster system, the disk shelf being removed is disk shelf3. One line item appears for each disk drive connected through path B (now the primary path); therefore, the disk drives arelisted in the PRIMARY column and B appears in the first PORT column. There is no connectivity through path A for any of thedisk drives in the disk shelf being removed; therefore, no information is shown in the SECONDARY or second PORT columns:

fas6200> storage show disk -p

PRIMARY PORT SECONDARY PORT SHELF BAY-------- ---- --------- ---- ---------

2b.125L27 B 3 02b.125L28 B 3 12b.125L3 B 3 22b.125L30 B 3 32b.125L31 B 3 4


2b.125L6 B 3 52b.125L7 B 3 62b.125L34 B 3 72b.125L36 B 3 82b.125L37 B 3 92b.125L38 B 3 102b.125L39 B 3 112b.125L43 B 3 122b.125L44 B 3 132b.125L45 B 3 14...

Attention: If the output shows anything other than all the disk drives connected only through path B, you must correct thecabling by repeating Step 6.

8. Complete the following substeps:

a) Repeat Step 6 and Step 7 for path B.b) Repeat Step 1 to confirm that your system configuration is the same as before you began this procedure.c) Go to the next step.

9. If, when you removed ownership from the disk drives as part of the preparation for this procedure, you disabled diskautoassignment, then reenable disk autoassignment by entering the following command; otherwise, go to the next step:options disk.auto_assign on

Enter the applicable command from the console of each controller.

10. Power off the disk shelves you disconnected and unplug the power cords from the disk shelves.

11. Remove the disk shelves from the rack or cabinet.

To make disk shelves lighter and easier to maneuver, remove the power supplies and IOMs. Avoid removing the disk drivesif possible, because excessive handling can lead to internal damage.

It is recommended that you use a mechanical hoist or lift if you are moving a fully populated 4U disk shelf. For example, afully populated DS4243 or DS4246 disk shelf can weigh approximately 110 lbs (49.9 kg).

12. If you are removing disk shelves at the remote site, repeat this procedure.

Converting an HA pair or stretch MetroCluster system with SAS diskshelves to a fabric-attached MetroCluster system

You can convert an HA pair or stretch MetroCluster system that has SAS disk shelves to a fabric-attached MetroCluster system.

About this task


Related references


Converting an HA pair or stretch MetroCluster system with SAS disk shelves to a fabric-attached MetroCluster system 23

Preparing for the conversionPreparing to convert an HA pair or stretch MetroCluster system with SAS disk shelves to a fabric-attached MetroCluster systeminvolves ensuring that your system meets certain requirements. These requirements include meeting setup and configurationrequirements for the bridges, for downloading documents, the ATTO QuickNAV utility, and bridge firmware.

Before you begin

Your system must be a supported configuration and running a supported version of Data ONTAP. The computer you are using to set up the bridges must be running an ATTO-supported web browser to use the ATTO

ExpressNAV GUI: Internet Explorer 8 or 9, or Mozilla Firefox 3.The ATTO Product Release Notes have an up-to-date list of supported web browsers. You can access this document usingthe information in the steps.

If you are converting an HA pair, you must have two bridges for each stack of shelves in your configuration.

Steps


TR-3517, MetroCluster Upgrade Planning Guide Interoperability Matrix Tool Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode Fabric-attached MetroCluster Systems Brocade Switch Configuration Guide or Fabric-attached MetroCluster Systems

Cisco Switch Configuration Guide Hardware and Service Guide for your disk shelf model








3. Check the Interoperability Matrix Tool to verify that your system meets all configuration requirements for the bridge.4. Check the Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode to verify that your system

meets all applicable MetroCluster system requirements as defined in the MetroCluster system section.

5. Gather the hardware and information needed to use the recommended bridge management interfacesthe ATTOExpressNAV GUI and ATTO QuickNAV utilityby completing the following substeps:a) Acquire a standard Ethernet cable (which connects from the bridge Ethernet management 1 port to your network).b) Determine a non-default user name and password (for accessing the bridges).



Active VPN clients cause the QuickNAV scan for bridges to fail.6. Acquire four screws for each bridge to flush-mount the bridge "L" brackets securely to the front of the rack.

The openings in the bridge "L" brackets are compliant with rack standard ETA-310-X for 19-inch (482.6 mm) racks.7. Update the following to the latest versions for each system as described in the Data ONTAP Upgrade and Revert/

Downgrade Guide for 7-Mode: Data ONTAP

Note: Both controllers must be running the same supported Data ONTAP version. Storage system firmware Disk shelf firmware Disk drive firmware

8. If you have a chassis with two controllers, complete the following substeps so that there is only one controller in eachchassis:

a) Move one controller to a new chassis to form the partner node of the MetroCluster system.b) Obtain and install the FC-VI interconnect card on both systems.

For details about the conversion process, see TR-3517, MetroCluster Upgrade Planning Guide.

Converting the systemAfter ensuring that the system meets all the requirements in the Preparing for the conversion section, you can convert your HApair or stretch MetroCluster system.

About this task

If you have a standard HA pair, you must make it a mirrored HA pair (in Step 19), which requires you to add disk shelves,disable disk autoassignment, assign the new disk drives to pool 1, and then reboot each controller.

Multipath HA is the only supported configuration for MetroCluster systems using bridges.Both controllers must have access through the bridges to the disk shelves in each stack.

You should use an equal number of disk shelves at each site. You must have two bridges for each stack of disk shelves at each site. Each FC switch must have one FC port available to allow one bridge to connect to it. The system connectivity requirements for maximum distances for disk shelves, FC switches, and backup tape devices using

50-micron, multimode fiber-optic cables also apply to bridges.The IBM System Storage N series Introduction and Planning Guide has detailed information about system connectivityrequirements.

Steps

1. Make note of the shelf IDs of the disk shelves that are connected to the system.

You use this information later when performing the disk autoassignment.

2. Move the NVRAM adapter and FC-VI adapter to the correct slots for your model, as shown in the IBM System Storage Nseries Introduction and Planning Guide at the N series support website (accessed and navigated as described in Websites).

3. Determine your switch and general configuration by completing the Planning the fabric-attached MetroCluster installationworksheet in the Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode.

4. If needed, update the firmware on each FC switch to the latest supported version for the Data ONTAP release that yoursystem is running.

The list of supported FC switch firmware is found in the Interoperability Matrix Tool.5. Set up and configure the FC switches, and verify your FC switch licenses, as described in the Brocade Switch Configuration

Guide for Fabric-attached MetroClusters or Fabric-attached MetroCluster Systems Cisco Switch Configuration Guide.


Make sure of the following:

Always refer to MetroCluster documentation when installing and configuring your FC switches. If you are using Brocade switches, the switch configuration and firmware requirements for a MetroCluster environment

are different from the requirements for SAN environments. If you are using switches that were previously used for another purpose, you must restore the switches to the factory

defaults. Each switch port connected to a bridge must be set to G_Port.

6. If you are converting a stretch MetroCluster system, got to Step 11; otherwise, install the bridges by completing thefollowing substeps, and then continue to Step 8:

a) Secure the "L" brackets on the front of each bridge to the front of the rack (flush-mount) with the four screws youacquired in the Preparing for the installation section.

b) Connect each bridge to a power source that provides a proper ground.c) Power on each bridge.

Note: For maximum resiliency, ensure that bridges attached to the same stack of disk shelves are connected todifferent power sources.

The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge has completed its power-onself test sequence.

For information about power requirements and LEDs, see the ATTO FibreBridge 6500N Installation and OperationManual.

7. Connect the Ethernet management 1 port on each bridge to your network using an Ethernet cable.

Note: The Ethernet management 1 port enables you to quickly download the bridge firmware (using ATTO ExpressNAVor FTP management interfaces) and to retrieve core files and extract logs.


Note: When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that isconnected by the Ethernet cable is configured. For example, if you also want to configure the Ethernet management 2port, you would need to connect the Ethernet cable to port 2 and run QuickNAV.











For example, if you were to use the command line interface (CLI) to set the bridge FC 1 port speed to 8 Gb and theconnection mode to ptp, you would enter the following three commands; the third command saves the configurationchanges:Set FCDataRate 1 8Gb


SaveConfiguration



10. Update the firmware on each bridge to the latest version by following the instructionsstarting with Step 4on theFibreBridge 6500N Download page.

11. Cable the disk shelves to the bridges by completing the following substeps:

Note: The disk shelves in the stack remain daisy-chained.

a) For each stack of disk shelves, cable IOM A square port on the first shelf to SAS port A on FibreBridge A.b) For each stack of disk shelves, cable IOM B circle port on the last shelf to SAS port A on FibreBridge B.Each bridge has one path to its stack of disk shelves; bridge A connects to the A-side of the stack through the first shelf, andbridge B connects to the B-side of the stack through the last shelf.

Note: The bridge SAS port B is disabled.

The following illustration shows a set of bridges cabled to a stack of three disk shelves:


Last shelf

FC1

SAS A

FC2

M1

FibreBridge A FC1

SAS A

FC2

M1

FibreBridge B


12. Verify that each bridge can see all the disk drives and disk shelves that it is connected to.



ATTO ExpressNAV GUI a. In a supported web browser, enter the IP address of the bridge in the browser box.You are brought to the ATTO FibreBridge 6500N home page, which has a link.




sastargets


sastargets

Example

The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines are sequentiallynumbered so you can quickly count the devices. For example, the following output shows that 10 disks are connected:


Note: If the text response truncated appears at the beginning of the output, you can use Telnet to access the bridgeand enter the same command to see all the output.

13. Cable the bridges to the FC switches by completing the following substeps:

a) Cable FC port 1 on bridge A to FC switch 1.b) Cable FC port 1 on bridge B to FC switch 2.

Note: You can connect the bridge to any available port on the FC switch. Whichever port you choose, you must usethe same port consistently on each FC switch that you are connecting for both sites. This is for ease of manageability.For example, if you choose FC switch port 3, only use that port on each FC switch.

Note: The FC port 2 remains unused on each bridge.

14. Cable the controllers to the FC switches using the information in the Data ONTAP High Availability and MetroClusterConfiguration Guide for 7-Mode.

15. Check whether your system has disk autoassignment enabled by entering the following command at the console of eithercontroller:options disk.auto_assign

If disk autoassignment is enabled, the output shows options disk.auto_assign on.

16. If your system does not have disk autoassignment enabled or the disk drives in the same stack are owned by both controllers,assign disk drives to the appropriate pools by using the information in the Data ONTAP High Availability and MetroClusterConfiguration Guide for 7-Mode; otherwise, go to the next step.You can use the disk drive information that you gathered in Step 1.




17. Install the Data ONTAP licenses on each controller in the following order:

a) syncmirror_localb) cf_remote

18. Reboot each controller.

19. Configure the controller by using the information in the Data ONTAP High Availability and MetroCluster ConfigurationGuide for 7-Mode:

If you are converting a... Then...

Standard HA pair with SAS storage Set up mirroring, and then configure the controller.

Note: You need to add disk shelves, disable disk autoassignment, assign the newdisk drives to pool 1, and then boot each controller.

Stretch MetroCluster system with SAS storage Configure the controller.

20. Verify that the aggregate Snapshot copy reserve for mirroring is 5 percent or larger by entering the following command atthe console of each controller:aggr options aggrname

21. Configure the MetroCluster system by using the information in the Data ONTAP High Availability and MetroClusterConfiguration Guide for 7-Mode.

22. Transport the partner controller, FC switches, disk shelves, and bridge to the remote site.

23. Install and set up the remote site system by repeating this procedure.

Related concepts


Replacing SAS copper cablesYou can nondisruptively replace SAS copper cables with other SAS copper cables in a multipath HA configuration when acable has failed or you need a different-length cable.

Before you begin

You must have met the requirements given at the beginning of this document.

About this task

Replacing a SAS copper cable means that you are replacing a cable using the same exact ports for bridge-to-shelf or shelf-to-shelf connections.

Attention: After your storage system is up and serving data, you cannot move SAS copper cables (change the SAS portsto which a cable is connected) nondisruptively. If you need to correct system cabling, you can use a maintenance period todo so.

You cannot change any disks, disk shelves, or components of a controller module as part of these procedures.

Replacing SAS copper cables 29

Steps

1. Verify that your MetroCluster system is Multi-Path HA by running the following command at the console of bothcontrollers:sysconfig


The configuration is listed in the System Storage configuration field. It should be the fourth line of output.

Caution: If your system configuration is shown as something other than Multi-Path HA, you cannot continue with thisprocedure.

2. Replace SAS cables by completing the following substeps:

Note: When replacing a SAS cable, wait a minimum of 10 seconds before plugging in the new cable so that the systemcan detect the cable change.

You can ignore cabling messages that might appear on the console.

a) Replace cables on Side A one cable at a time.The Side A cables are the cables connected to IOM A of each disk shelf.

b) Verify that you have correctly replaced the SAS cables by entering the following command at the system console:sysconfig

For HA pairsoperation in 7-Mode, you can run the command from either controller. For clustered systems, you must runthis command from the nodeshell of the target controller.


The output should be the same as Step 1: the system should be Multi-Path HA, and the SAS port and attached diskshelf information should be the same.

If the output is something other than Multi-Path HA, you must identify the cabling error, correct it, and run thesysconfig command again.

c) Repeat substeps a and b for Side B.The Side B cables are the cables connected to IOM B of each disk shelf.

Other FibreBridge management interfacesYou can use a combination of serial port, Telnet, and FTP instead of the recommended management interfaces. Your systemmust meet the requirements for the applicable interface before you install the bridges.

You can use a serial port or Telnet to configure the bridge and Ethernet management 1 port, and to manage the bridge. You canuse FTP to update the bridge firmware.

Note: The ATTO FibreBridge 6500N Installation and Operation Manual has more information about management interfaces.You can access this document on the ATTO web site by using the link provided on the ATTO FibreBridge 6500NDescription page on the NetApp Support Site at support.netapp.com. From this web site, navigate to the ATTO FibreBridgeDescription page by clicking Software, scrolling to Protocol Bridge, choosing ATTO FibreBridge from the drop-downmenu, clicking Go!, and then clicking View & Download.

Serial portWhen using the serial port to configure and manage a bridge, and configure the Ethernet management 1 port, your system mustmeet the following requirements:

A serial cable (which connects from the bridge serial port to a serial (COM) port on the computer you are using for setup)


The bridge serial port is RJ-45 and has the same pin-out as the controllers. A terminal emulation program such as Hyperterminal, Teraterm, or PuTTY to access the console

The terminal program should be capable of logging screen output to a file. An IP address, subnet mask, and gateway information for the Ethernet management 1 port on each bridge

TelnetWhen using Telnet to configure and manage a bridge, your system must meet the following requirements:

A serial cable (which connects from the bridge serial port to a serial (COM) port on the computer you are using for setup)The bridge serial port is RJ-45 and has the same pin-out as the controllers.

(Recommended) A non-default user name and password (for accessing the bridge) A terminal emulation program such as Hyperterminal, Teraterm, or PuTTY to access the console

The terminal program should be capable of logging screen output to a file.

FTPWhen using FTP to update bridge firmware, your system must meet the following requirements:

A standard Ethernet cable (which connects from the bridge Ethernet management 1 port to your network) (Recommended) A non-default user name and password (for accessing the bridge)

FibreBridge error messagesBridge error messages inform you of bridge configuration errors that Data ONTAP has detected when cabling bridges or whendownloading disk shelf and disk drive firmware.

Attention: If you must correct cabling on a system that is serving data, you can perform hot-cable changes only if one path tothe disk drives is maintained.

config.bridge.adapter.invalidMessage config.bridge.adapter.invalid

Severity ERROR

Description This message occurs when an FC-to-SAS bridge is attached to an unsupported FC adapter port.

Corrective action Attach the FC-to-SAS bridge to a QLogic 4-Gb/s or 8-Gb/s adapter (which are the only supportedadapters).FC ports can be onboard and/or on PCIe HBAs.

SNMP trap ID N/A

config.bridge.extra.portMessage config.bridge.extra.port

Severity ERROR

Description This message occurs when both FC ports of an FC-to-SAS bridge are attached to the same controller.

Corrective action Correct the cabling by following the defined stretch or fabric-attached MetroCluster cabling informationand illustrations in this Configuring a MetroCluster system with SAS disk shelves and FibreBridge 6500Nbridges document.

SNMP trap ID N/A

FibreBridge error messages 31

config.bridge.fw.mismatchMessage config.bridge.fw.mismatch

Severity ERROR

Description This message occurs when two or more attached FC-to-SAS bridges are running

Date post:	18-Dec-2015
Category:	Documents
Upload:	rafer79
View:	26 times
Download:	1 times

Configuring a MetroCluster System With SAS Disk

Documents