215-05500_B0

Universal SAS and ACP Cabling Guide

NetApp, Inc.495 East Java DriveSunnyvale, CA 94089 USATelephone: +1 (408) 822-6000Fax: +1 (408) 822-4501Support telephone: +1 (888) 4-NETAPPDocumentation comments: [email protected] Web: www.netapp.com

Part number: 215-05500_B0November 2011

Copyright and trademark information

Copyright information

Copyright © 1994-2011 NetApp, Inc. All rights reserved. Printed in the U.S.A.

No part of this document covered by copyright may be reproduced in any form or by any means—graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic retrieval system—without prior written permission of the copyright owner.

Software derived from copyrighted NetApp material is subject to the following license and disclaimer:

THIS SOFTWARE IS PROVIDED BY NETAPP “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

NetApp reserves the right to change any products described herein at any time, and without notice. NetApp assumes no responsibility or liability arising from the use of products described herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product does not convey a license under any patent rights, trademark rights, or any other intellectual property rights of NetApp.

The product described in this manual may be protected by one or more U.S.A. patents, foreign patents, or pending applications.

RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.277-7103 (October 1988) and FAR 52-227-19 (June 1987).

Trademark information

NetApp, the NetApp logo, Network Appliance, the Network Appliance logo, Akorri, ApplianceWatch, ASUP, AutoSupport, BalancePoint, BalancePoint Predictor, Bycast, Campaign Express, ComplianceClock, Cryptainer, CryptoShred, Data ONTAP, DataFabric, DataFort, Decru, Decru DataFort, DenseStak, Engenio, Engenio logo, E-Stack, FAServer, FastStak, FilerView, FlexCache, FlexClone, FlexPod, FlexScale, FlexShare, FlexSuite, FlexVol, FPolicy, GetSuccessful, gFiler, Go further, faster, Imagine Virtually Anything, Lifetime Key Management, LockVault, Manage ONTAP, MetroCluster, MultiStore, NearStore, NetCache, NOW (NetApp on the Web), Onaro, OnCommand, ONTAPI, OpenKey, PerformanceStak, RAID-DP, ReplicatorX, SANscreen, SANshare, SANtricity, SecureAdmin, SecureShare, Select, Service Builder, Shadow Tape, Simplicity, Simulate ONTAP, SnapCopy, SnapDirector, SnapDrive, SnapFilter, SnapLock, SnapManager, SnapMigrator, SnapMirror, SnapMover, SnapProtect, SnapRestore, Snapshot, SnapSuite, SnapValidator, SnapVault, StorageGRID, StoreVault, the StoreVault logo, SyncMirror, Tech OnTap, The evolution of storage, Topio, vFiler, VFM, Virtual File Manager, VPolicy, WAFL, Web Filer, and XBB are trademarks or registered trademarks of NetApp, Inc. in the United States, other countries, or both.

ii Copyright and trademark information

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. A complete and current list of other IBM trademarks is available on the Web at www.ibm.com/legal/copytrade.shtml.

Apple is a registered trademark and QuickTime is a trademark of Apple, Inc. in the U.S.A. and/or other countries. Microsoft is a registered trademark and Windows Media is a trademark of Microsoft Corporation in the U.S.A. and/or other countries. RealAudio, RealNetworks, RealPlayer, RealSystem, RealText, and RealVideo are registered trademarks and RealMedia, RealProxy, and SureStream are trademarks of RealNetworks, Inc. in the U.S.A. and/or other countries.

All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.

NetApp, Inc. is a licensee of the CompactFlash and CF Logo trademarks. NetApp, Inc. NetCache is certified RealSystem compatible.

iv Copyright and trademark information

Table of Contents

Chapter 1 Cabling requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 2 SAS cabling for new storage systems. . . . . . . . . . . . . . . . . . . . . . 3

SAS cabling rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Cabling SAS connections. . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

SAS cabling examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Chapter 3 ACP cabling for new storage systems . . . . . . . . . . . . . . . . . . . . 29

ACP cabling rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Cabling ACP connections . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

ACP cabling examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Appendix A Terms and symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Appendix B Document update record . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Table of Contents v

vi Table of Contents

Chapter 1: Cabling requirements

1
Cabling requirements
Before you begin To cable a new storage system with SAS disk drives, your storage system must meet the following requirements:

◆ Your storage system must be running the minimum software release required to support your model of SAS disk shelves.

See the Installation and Service Guide for your particular disk shelf model or the System Configuration Guide at support.netapp.com.

◆ If your storage system was not shipped in a system cabinet, the controllers and disk shelves must already be installed in a rack.

◆ If your storage system will be using one or more PCIe SAS HBAs, they must already be installed and available for cabling the disk shelves.

◆ If you are cabling for the ACP capability and your storage system does not have a native onboard ACP port, you must be prepared to designate a network interface for each controller at system setup.

If you are using a four-port Ethernet card for this purpose, this card must already be installed and available in each controller.

1

2 Cabling requirements

Chapter 2: SAS cabling for new storage systems

2
SAS cabling for new storage systems
About this chapter This chapter provides information about cabling the disk shelf SAS connections for a new storage system installation.

This chapter does not provide cabling information for hot-adding or replacing disk shelves. For those procedures, see the Installation and Service Guide at support.netapp.com for your disk shelf model.

NoteReferences to the top and bottom disk shelves in a stack are to the logical top and bottom of a stack. Your physical stack arrangement might be different.

Topics in this chapter

This chapter discusses the following topics:

◆ “SAS cabling rules” on page 4

◆ “Cabling SAS connections” on page 11

◆ “SAS cabling examples” on page 16

3

SAS cabling rules

Topics in this section

The following topics are covered in this section:

◆ “General SAS cabling rules” on page 4

◆ “Onboard SAS port rule” on page 5

◆ “Controller-to-controller rule (2240 systems only)” on page 6

◆ “Shelf-to-shelf connection rule” on page 6

◆ “Controller-to-shelf connection rules” on page 7

General SAS cabling rules

The following SAS cabling rules apply to all supported storage systems using SAS HBAs and onboard SAS ports, except where otherwise noted:

Supported configurations (except single-controller 2240 systems):

Multipath HA is the supported configuration, with the following exceptions:

❖ 2040 storage systems use single-path; all 2040 systems in an HA pair configuration use single-path HA.

❖ 2240 systems in an HA pair with no external SAS storage may use single-path HA, which requires no external cabling.

However, the recommended configuration is to enable Multipath HA for the internal disks by connecting the SAS ports of one controller module to the SAS ports of the partner.

Supported configurations (single-controller 2240 systems): In 2240 systems in a single-controller configuration, the internal disks are single-path but the system can use either single- or dual-path connectivity to the external shelves. Both are supported.

Dual-path provides greater resiliency, but when dual-path is used Data ONTAP 8.1 and 8.1.1 issues occasional warnings to the console because mixed paths are detected. To avoid the warnings, single-path connections to the external SAS storage are a supported option in these configurations.

SAS cables used: QSFP to QSFP SAS cables are used to connect disk shelves together and to connect disk shelves to the SAS ports on the controller.

4 SAS cabling rules

QSFP to mini-SAS cables are used for 2050 configurations only, and connect disk shelves to the dual-port mini-SAS HBAs on the controller.

SAS port designations: All SAS ports on the disk shelf are designated by a square symbol or a circle symbol. The following illustration shows square and circle labeling on the IOM ports.

NoteSAS connectivity does not use the concept of In ports and Out ports.

Onboard SAS port rule

To ensure that the SAS cabling rules apply to all SAS HBAs on a controller, whether in a physical PCI slot (slot 1-N) or on board the base controller, an on board HBA is defined as occupying PCI slot 0. Each port is named with sequentially increasing letters (0a, 0b, 0c, and so on).

For example, the following illustration shows a controller with one onboard SAS HBA in slot 0 and three SAS HBAs in physical PCI slots 1, 2, and 3.

ACP SAS

IOM A IOM B

ACP SAS

Chapter 2: SAS cabling for new storage systems 5

Exception: The only exception is the 2040 storage system. Treat the onboard SAS port labeled 0d as if it were port 0a. Treating the port as 0a enables you to apply all the SAS cabling rules. (Data ONTAP continues to recognize this port as 0d.)

Controller-to-controller rule (2240 systems only)

On 2240 systems in HA configurations with no external storage, external cables should connect the SAS ports on the two controller modules to enable Multipath HA for the internal disks.

Shelf-to-shelf connection rule

All the disk shelves in a stack are daisy-chained when you have more than one disk shelf in a stack.

◆ IOM A circle port is connected to the next IOM A square port.

◆ IOM B circle port is connected to the next IOM B square port.

B

A

C

D

Slot 1

B

A

C

D

Slot 3

B

A

C

D

Slot 2

B

A

Slot 0

2040 controller

0a

0d

Bottom shelf

Top shelf

ACP SAS ACP SAS

IOM A IOM B

Stack of three shelves

6 SAS cabling rules

Controller-to-shelf connection rules

Each controller connects to each stack of disk shelves in the system through the controller SAS ports, which can be A, B, C, and D, (and can be on a SAS HBA in a physical PCI slot [slot 1-N] or on board the base controller).

For quad-port SAS HBAs, the controller-to-shelf connection rules ensure resiliency for the storage system based on the ASIC chip design. Ports A and B are on one ASIC chip, and ports C and D are on a second ASIC chip. By using ports A and C to connect to the top shelf in each stack, and using ports B and D to connect to the bottom shelf in each stack, the controllers maintains connectivity to the disk shelves if an ASIC chip fails.

The following illustration shows a quad-port SAS HBA with the two ASIC chips and their designated ports.

SAS port A and C connections: SAS ports A and C on a controller are connected to the stacks of disk shelves in this way:

◆ SAS ports A and C are connected to the top disk shelf square port in each stack of disk shelves.

❖ The A and C ports on controller 1/A connect to IOM A.

❖ The A and C ports on controller 2/B connect to IOM B.

Quad-port SAS HBA

ASIC chip

ASIC chip

C portD port

B portA port

PCI-e chip


◆ SAS ports A and C are connected to the disk shelf stacks in sequential order of the HBA slots (slot 0, slot 1, slot 2, and so on).

For example, a controller with slots 1, 2, and 3 occupied with quad-port SAS HBAs is cabled in order of HBAs and ports 1a, 2a, 3a, 1c, 2c, 3c, as shown below.

SAS port B and D connections: SAS ports B and D on the controller are connected to the stacks of disk shelves in this way:

◆ Ports B and D are connected to the bottom disk shelf circle port in each stack of disk shelves.

❖ The B and D ports on controller 1/A connect to IOM B.

❖ The B and D ports on controller 2/B connect to IOM A.

Controller 1/A

Slot 1 Slot 1

Controller 2/B

Stack 1 Stack 2

A

B B

C C

D D

A

ACP SAS ACP SAS ACP SAS ACP SAS

IOM A IOM B IOM A IOM B

Top shelf

Bottom shelf

Slot 1

B

D

Slot 2 Slot 3

Controller 1/A

A

C

B

D

A

C

B

D

A

C

8 SAS cabling rules

◆ SAS ports B and D are connected to the disk shelf stacks by offsetting the order of the HBA slots by one so that the first HBA port is cabled last. The cabling offset ensures system resiliency.

For example, a controller with three quad-port SAS HBAs in slots 1, 2, and 3 is cabled in order of HBAs and ports 2b, 3b, 1d, 2d, 3d, 1b, as shown below.

Controller 1/A

Slot 1 Slot 1

Controller 2/B

Stack 1 Stack 2

A

B B

C C

D D

A



Top shelf

Bottom shelf

Slot 1

B

D D

Slot 2 Slot 3

Controller 1/A

A

C

B

D D

A

C

B

A

C


A controller with one HBA is cabled in order of port 1d and then 1b, as shown below.

Stack cabling sequence: Controllers are cabled to the stacks in sequence of the stacks, such as stack 1, stack 2, stack 3, and so on.

20xx storage system connections: For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

Controller 1/A

Slot 1

B

D

A

C

10 SAS cabling rules

Cabling SAS connections



◆ “Completing the SAS cabling worksheet” on page 11

◆ “SAS cabling worksheet” on page 13

◆ “Example worksheet” on page 13

◆ “Cabling SAS ports” on page 14

Completing the SAS cabling worksheet

By making a list of the SAS ports for each controller that need to be cabled to the disk shelf IOMs, you can simplify the cabling task of your storage system.

Step Action

1 Make a list of the top connections from the controller by listing all SAS HBA A ports and then all SAS HBA C ports on each controller.

You can use the “SAS cabling worksheet” on page 13 to write your information.

NoteIf you have an HA pair configuration, repeat this list for the second controller.

Examples:

A controller with three quad-port SAS HBAs is listed as 1a, 2a, 3a, 1c, 2c, 3c.

A controller with one quad-port SAS HBA is listed as 1a, 1c.

A controller with one dual-port SAS HBA is listed as 1a.

A 2040 controller is listed as 0a. 2040 system configurations only have a top connection to the disk shelves.


2 Make a list of the bottom connections from each controller.

You can use the “SAS cabling worksheet” on page 13 to write your information.

a. List the SAS HBA B ports and then all SAS HBA D ports.

Examples:

A controller with three quad-port SAS HBAs is listed as 1b, 2b, 3b, 1d, 2d, 3d.

A controller with one quad-port SAS HBA is listed as 1b, 1d.

A controller with one dual-port SAS HBA is listed as 1b.

b. Copy the first port in your list to the end of your list, and then cross out the first port in your list.

Offsetting the list by one (so that the bottom connection cabling begins on the second HBA) ensures system resiliency after you cable your storage system.

Examples:

A controller with three quad-port SAS HBAs is listed as 2b, 3b, 1d, 2d, 3d, 1b.

A controller with one quad-port SAS HBA is listed as 1d, 1b.

A controller with one dual-port SAS HBA is listed as 1b.

NoteIf you have an HA pair configuration, repeat this list for the second controller.

3 Go to “Cabling SAS ports” on page 14 to cable your storage system.

Step Action

1b, 2b, 3b, 1d, 2d, 3d, 1bXFor example:

12 Cabling SAS connections

SAS cabling worksheet

By filling in this worksheet as instructed in the “Completing the SAS cabling worksheet” on page 11, you identify the SAS ports for each controller that need to be cabled to the disk shelf IOMs.

Example worksheet This is an example of a completed worksheet for an HA pair configuration with three quad-port SAS HBAs and three stacks of disk shelves.

NoteThe grayed out SAS ports are unused because of the limited number of stacks in this configuration. If there were six stacks, all SAS ports would be used.

Connection type Controller SAS port cabling sequence Cable to disk shelf IOM

Top connections

C1/A To top shelf IOM A

C2/B To top shelf IOM B

Bottom connections

C1/A To bottom shelf IOM B

C2/B To bottom shelf IOM A

Connection type Controller SAS port cabling sequence Cable to disk shelf IOM

Top connections

C1/A 1a, 2a, 3a, 1c, 2c, 3c To top shelf IOM A

C2/B 1a, 2a, 3a, 1c, 2c, 3c To top shelf IOM B

Bottom connections

C1/A 2b, 3b, 1d, 2d, 3d, 1b To bottom shelf IOM B

C2/B 2b, 3b, 1d, 2d, 3d, 1b To bottom shelf IOM A


Cabling SAS ports To cable the SAS ports, complete the following steps.

Step Action

1 If you have more than one disk shelf in a stack, cable the shelf-to-shelf SAS connections; otherwise, go to Step 2.

Using the QSFP to QSFP SAS cables that came with your system:

a. Connect IOM A circle port to the next IOM A square port.

b. Connect IOM B circle port to the next IOM B square port.

NoteThe QSFP connectors are keyed and click into place when oriented correctly into a SAS port. Do not force a connector into a port.

2 Use your completed worksheet to cable the top connections for each controller.

NoteIf you are cabling a 2050 storage system, use the QSFP to mini-SAS cables for all controller to shelf connections; otherwise, use the QSFP to QSFP SAS cables.

Bottom shelf

Top shelf

ACP SAS ACP SAS

IOM A IOM B

Stack of three shelves

14 Cabling SAS connections

3 Use your completed worksheet to cable the bottom connections for each controller.

NoteFor quad-port SAS HBAs, because ports A and C never connect to the same stack of disk shelves and ports B and D never connect to the same stack of disk shelves, you might have unused HBA ports depending on the number of stacks in your configuration. For example, a single-controller with one quad-port SAS HBA and one stack of shelves only uses HBA ports A and D.

4 Verify the SAS connections by entering the applicable command:

◆ For 7-Mode, enter the following command at the system console:

sasadmin expander_map

For HA pair configurations, you can run this command on either node.

◆ For Cluster-Mode, enter the following command at the clustershell prompt:node run -node <node name> -command “sasadmin expander_map”

Result: The output shows information about SAS expanders attached to SAS channels on the controllers.

5 The next step depends on the output.

If... Then...

The output lists all of the IOMs The IOMs have connectivity. Go to Step 6.

Any of the IOMs are not shown - either the output will not show the IOM that is cabled incorrectly, or will not show all of the IOMs downstream from it.

Repeat Steps 1 through 5 to correct cabling errors.

6 The next step depends on whether you are using the ACP capability.

If... Then...

You are using the ACP capability Go to “Cabling ACP connections” on page 32.

You are not using the ACP capability Boot the storage system and begin setup.

Step Action


SAS cabling examples



NoteThe cabling examples show cabling to the logical top and logical bottom of a stack of disk shelves. Your physical stack arrangement might be different.

◆ “Examples of HA pair quad-port SAS HBA cabling” on page 16

◆ “Examples of single-controller quad-port SAS HBA cabling” on page 19

◆ “Examples of 2240 system SAS cabling” on page 21

◆ “Examples of 2040 SAS cabling” on page 24

◆ “Examples of 2050 SAS cabling” on page 26

Examples of HA pair quad-port SAS HBA cabling

The following cabling examples show the SAS cabling sequence for HA pair configurations using quad-port SAS HBAs.

One quad-port SAS HBA and one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 1a; Bottom: 1d.

Stack 1

Controller 2/B

Ethernet portB

A

C

D

Slot 1

Controller 1/A

Ethernet portB

A

C

D

Slot 1

1. 2.

3.

4.

ACP SAS

IOM A IOM B

ACP SAS

16 SAS cabling examples

Two quad-port SAS HBAs and two stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a; Bottom: 2b, 1d.

4.

1. 2.

5.

3.

6.

7.

8.

Controller 2/B

Ethernet portB

A

C

D

Slot 1

B

A

C

D

Slot 2

Controller 1/A

Ethernet portB

A

C

D

Slot 1

B

A

C

D

Slot 2


Stack 1 Stack 2


Top shelf

Bottom shelf


Three quad-port SAS HBAs and three stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a; Bottom: 2b, 3b, 1d.

Four quad-port SAS HBAs and four stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a; Bottom: 2b, 3b, 4b, 1d.

Four quad-port SAS HBAs and eight stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 1c, 2c, 3c, 4c; Bottom: 2b, 3b, 4b, 1d, 2d, 3d, 4d, 1b.

Five quad-port SAS HBAs and five stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 5a; Bottom: 2b, 3b, 4b, 5b, 1d.

Five quad-port SAS HBAs and 10 stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 5a, 1c, 2c, 3c, 4c, 5c; Bottom: 2b, 3b, 4b, 5b, 1d, 2d, 3d, 4d, 5d, 1b.

Controller 2/BController 1/A

B

A

C

D

Slot 1

B

A

C

D

Slot 2

B

A

C

D

Slot 3

B

A

C

D

Slot 1

B

A

C

D

Slot 2

B

A

C

D

Slot 3

3.

1.4.

6.

2.

8.9.

7.

11.

12.


Stack 1 Stack 2


Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 3

IOM A IOM B

5. 10.


Examples of single-controller quad-port SAS HBA cabling

The following cabling examples show the SAS cabling sequence for single-controller configurations using quad-port SAS HBAs.

One quad-port SAS HBA and one stack of disk shelves: The SAS port cabling sequence is 1a, 1d.

B

A

C

D

Slot 1

Controller 1/A

Stack 1

1.

2.

Top shelf

Bottom shelf

IOM A IOM B

ACP SAS ACP SAS


One quad-port SAS HBA and two stacks of disk shelves: The SAS port cabling sequence is Top: 1a, 1c; Bottom: 1d, 1b.

Two quad-port SAS HBAs and two stacks of disk shelves: The SAS port cabling sequence is Top: 1a, 2a; Bottom: 2b, 1d.

Controller 1/A

B

A

C

D

2.

4.

3.

Slot 1

Stack 1 Stack 2



Top shelf

Bottom shelf

1.

1.

2.

4.

3.

Controller 1/A

Ethernet portB

A

C

D

Slot 1

B

A

C

D

Slot 2


Stack 1 Stack 2


Top shelf

Bottom shelf


Examples of 2240 system SAS cabling

The following cabling examples show the SAS cabling sequence for 2240 HA pair and single-controller configurations.

NoteThese examples show a 2240-2, but the cabling also applies to the 2240-4, even though the controllers are in a different position in the 2240-4.

2240 HA pair configuration with one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 0a, 0b.

ACP SAS ACP SAS

PSU

0b 0a

PSU

0b 0a

SAS SAS

ACP SAS ACP SAS

ACP SAS ACP SAS


2240 single-controller with one stack of disk shelves (dual-path):

The SAS port cabling sequence is Top: 0a.

NoteDual-path provides greater resiliency, but when dual-path is used Data ONTAP 8.1 issues occasional warnings to the console because mixed paths are detected. To avoid the warnings, single-path connections to the external SAS storage are a supported option in these configurations.

ACP SAS ACP SAS

0b 0a

SAS

ACP SAS ACP SAS

ACP SAS ACP SAS

ACP


2240 single-controller with one stack of disk shelves (single-path):

The SAS port cabling sequence is Top: 0a.

2240 HA pair configuration with no external disk shelves: The SAS port cabling sequence for each controller is Top: 0a, 0b.

0b 0a

10101

e0b e0d

e0a e0c

SAS

ACP SAS ACP SAS

ACP SAS ACP SAS

ACP SAS ACP SAS

0b 0a 0b 0a

SAS SAS


Examples of 2040 SAS cabling


NoteFor single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

NoteThe 2040 storage system’s onboard SAS port labeled 0d is treated as port 0a for cabling purposes. When the onboard port is treated as port 0a, all the SAS cabling rules can be applied.


2040 HA pair configuration with one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 0a.

2040 single-controller with one stack of disk shelves: The SAS port cabling sequence is Top: 0a.

Controller 1/A

0a

Controller 2/B

0a

1.

2040 configuration

0d 0d

Ethernet port

e0PEthernet port

e0P

Stack 1 2.

Top shelf

Bottom shelf

ACP SAS ACP SAS

IOM A IOM B

Controller 2/B

Ethernet port

0a

1.

2040 configuration

0d

e0P

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B


Examples of 2050 SAS cabling


NoteFor single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

2050 HA pair configuration with one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 1a; Bottom: 1b.

Stack 1

Controller 1/A

Slot 1

BA

Ethernet port

Controller 2/B

Slot 1

B

A

Ethernet port

4.

3.

1. 2.

2050 configuration

Bottom shelf

Top shelf

ACP SAS ACP SAS

IOM A IOM B


2050 HA pair configuration with one disk shelf: The SAS port cabling sequence for both controllers is Top: 1a; Bottom: 1b.

Controller 1/A

Ethernet portB

A

Slot 1

Controller 2/B

Ethernet portB

A

Slot 1

4.

1.

3.

2.

2050 configuration

ACP SAS ACP SAS

IOM A IOM B

Stack 1


2050 single-controller configuration with one stack of disk shelves:

The SAS port cabling sequence is Top: 1a; Bottom:1b.

2050 single-controller configuration with one disk shelf: The SAS port cabling sequence is Top: 1a; Bottom: 1b.

Controller 2/B

Slot 1AB

1.

2.

2050 configuration

Stack 1

Top shelf

Bottom shelf

ACP SAS ACP SAS

IOM A IOM B

Controller 2/B

Ethernet portB

A

Slot 1

1.2.

2050 configuration

ACP SAS ACP SAS

IOM A IOM B

Stack 1


Chapter 3: ACP cabling for new storage systems

3
ACP cabling for new storage systems
About this chapter This chapter provides information about cabling the disk shelf ACP connections for a new storage system installation.

This chapter does not provide cabling information for hot-adding or replacing disk shelves. For those procedures, see the Installation and Service Guide at support.netapp.com for your disk shelf model.

NoteReferences to the top and bottom disk shelves in a stack are to the logical top and bottom of a stack. Your physical stack arrangement might be different.

Topics in this chapter

This chapter discusses the following topics:

◆ “ACP cabling rules” on page 30

◆ “Cabling ACP connections” on page 32

◆ “ACP cabling examples” on page 38

29

ACP cabling rules



◆ “General ACP cabling rules” on page 30

◆ “ACP connectivity rules” on page 30

General ACP cabling rules

These ACP cabling rules apply to all supported storage systems using SAS storage:

◆ You must use CAT6 Ethernet cables with RJ-45 connectors for ACP connections.

◆ If your storage system does not have a dedicated on board network interface for each controller, you must dedicate one for each controller at system setup. You can use a quad-port Ethernet card.

◆ All ACP connections to the disk shelf are cabled through the ACP ports, which are designated by a square symbol or a circle symbol.

ACP connectivity rules

All supported storage systems using the ACP capability are cabled in this way:

Shelf-to-shelf connections: All the disk shelves in a stack are daisy-chained when you have more than one shelf in a stack.

◆ IOM A circle port is connected to the next IOM A square port.

◆ IOM B circle port is connected to the next IOM B square port.

Intrastack connections: The bottom shelf and top shelf in each stack are connected to provide resiliency within each stack of disk shelves.

The bottom shelf IOM A circle port is connected to the top shelf IOM B square port. This applies to stacks containing one or more disk shelves.

Stack-to-stack connections: When you have two or more stacks of disk shelves, the last shelf in a stack is connected to the first shelf in the next stack until all stacks are connected.

The bottom shelf IOM B circle port of a stack is connected to the top shelf IOM A square port of the next stack.

Controller-to-stack connections: Each storage system controller is connected to each stack of disk shelves through a dedicated Ethernet port:

30 ACP cabling rules

◆ Controller 1/A always connects to the top shelf IOM A square port in a stack.

◆ Controller 2/B always connects to the bottom shelf IOM B circle port in a stack.

Controller-to-controller connections (2240 systems with no external SAS shelves): Each storage system controller is connected to the partner controller through a dedicated Ethernet port marked with the locked wrench icon. This rule applies to 2240 storage systems only.

20xx storage system connections: For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

Chapter 3: ACP cabling for new storage systems 31

Cabling ACP connections

Cabling ACP in an HA pair with no external SAS storage

The following cabling procedure applies to 2240 HA storage systems with no external SAS storage:

Step Action

1 Cable the locked wrench ACP port on one controller module to the same port on the partner controller module.

NoteThese examples show a 2240-2, but the cabling also applies to the 2240-4, even though the controllers are in a different position in the 2240-4.

PSU

0b 0a

PSU

0b 0a

SAS SASACP

(Locked wrench)ACP

(Locked wrench)

32 Cabling ACP connections

Cabling ACP on systems with external SAS storage

Any storage system supporting the ACP capability and having external SAS storage can be cabled using this cabling procedure.

Step Action

1 If you have more than one disk shelf in a stack, cable your shelf-to-shelf ACP connections; otherwise, go to Step 2.

Using the Ethernet cables that came with your system:

a. Connect each shelf IOM A circle port to the next shelf IOM A square port until all shelves in each stack are connected.

b. Connect each shelf IOM B circle port to the next shelf IOM B square port until all shelves in each stack are connected.

Daisy-chain connections

Top shelf

Bottom shelf

ACP SAS ACP SAS

IOM A IOM B


2 Cable the intrastack connection.

If... Then...

You have more than one disk shelf in a stack of disk shelves

Connect the bottom shelf IOM A circle port to the first shelf IOM B square port and repeat for all stacks.

You have only one disk shelf in a stack of disk shelves

Connect the IOMs within the disk shelf by connecting IOM A circle port to IOM B square of the disk shelf.

Step Action

Intrastack connection

Top shelf

Bottom shelf

ACP SAS ACP SAS

IOM A IOM B

Intrastack connection

ACP SAS ACP SAS

IOM A IOM B


3 If you have more than one stack of disk shelves, cable the stack-to-stack connections; otherwise, go to Step 4.

a. Beginning with Stack 1, connect the bottom shelf IOM B circle port to the next stack top shelf IOM A square port.

b. Repeat Step a until all stacks are connected.

Step Action

Stack-to-stack connections

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B

ACP SAS ACP SAS

Stack 2

IOM A IOM B

ACP SAS ACP SAS

Stack 3

IOM A IOM B


4 Cable controller-to-stack connections.

If... Then...

If your system has a controller 1/A Cable controller 1/A to the Stack 1 top shelf IOM A square port.

If your system has a controller 2/B Cable controller 2/B to the last stack bottom shelf IOM B circle port.

For example, an HA pair configuration with two stacks of disk shelves has the controller-to-stack connections cabled as follows:

Note20xx storage systems use the e0P port on each controller.

5 Verify that the ACP cabling is correct by entering the applicable command:

◆ For 7-Mode, enter the following command from the system console:

storage show acp

If you have an HA pair configuration, run this command on both nodes.

◆ For Cluster-Mode, enter the following command from the clustershell:node run -node local -command “storage show acp”

Step Action

Controller 2/B

Ethernet port

Controller 1/A

Ethernet port


Stack 1 Stack 2


Top shelf

Bottom shelf


6 The next step depends on what the command output is for each node.

If the result in the output shows ACP connectivity status... Then...

Full Connectivity ACP cabling is correct.

The number of shelf IOMs detected out-of-band (through ACP) and in-band (through SAS) are the same.

No Connectivity No ACP ports on the shelf IOMs are connected to the storage controllers.

Repeat Step 4.

Partial Connectivity Repeat Steps 1 through 6 to see what ACP connections you missed.

Fewer shelf IOMs are detected out-of-band (through ACP) than in-band (through SAS).

Additional Connectivity Repeat the “Cabling SAS connections” on page 11 to see what SAS connections you missed.

More shelf IOMs are detected out-of-band (through ACP) than in-band (through SAS).

7 Boot the storage system and begin setup.

See the Installation and Setup Instructions that came with your storage system.

NoteIf your storage system does not have a dedicated ACP on board Ethernet port on each controller for ACP, you will designate them at setup.

Step Action


ACP cabling examples



NoteThe cabling examples show cabling to the logical top and logical bottom of a stack of disk shelves. Your physical stack arrangement might be different.

◆ “Examples of HA pair configuration ACP cabling” on page 38

◆ “Examples of ACP cabling for 2240 systems with no external storage” on page 40

◆ “Examples of ACP cabling for all systems except 20xx” on page 41

◆ “Examples of 20xx single-controller ACP cabling” on page 42

Examples of HA pair configuration ACP cabling

The following cabling examples show ACP cabling for all supported HA pair configurations.

HA pair configuration with one disk shelf:

Controller 2/B

Ethernet port

Controller 1/A

Ethernet port

Top shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B

38 ACP cabling examples

HA pair configuration with two stacks of disk shelves:

HA pair configuration with three stacks of disk shelves:

Controller 1/A

Ethernet port

Controller 2/B

Ethernet port


Stack 1 Stack 2


Top shelf

Bottom shelf

Controller 1/A

Ethernet port

Stack 2Stack 1 Stack 3

Controller 2/B

Ethernet port

ACP SAS ACP SAS

IOM A IOM B

ACP SAS ACP SAS

IOM A IOM B

ACP SAS ACP SAS

IOM A IOM B

Top shelf

Bottom shelf


Examples of ACP cabling for 2240 systems with no external storage

The following examples show ACP cabling for 2240 systems with no external disk shelves:

2240-2 HA configuration with no external disk shelves:

2240-4 HA configuration with no external disk shelves:

PSU

0b 0a

PSU

0b 0a

SAS SASACP

(Locked wrench)ACP

(Locked wrench)

ACPSAS

Controller A

Controller B


Examples of ACP cabling for all systems except 20xx

The following cabling examples show ACP cabling for all supported storage systems other than 20xx systems.

Single-controller configuration with one stack of disk shelves (except the 20xx systems):

Controller 1/A

Ethernet port

Non 20xx configuration

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B


Single-controller configuration with two stacks of disk shelves (except the 20xx systems):

Examples of 20xx single-controller ACP cabling

The following cabling examples show ACP cabling for 20xx single-controller configurations.

NoteFor single-controller 20xx configurations, the controller is in Slot B (the bottom slot); therefore, the controller needs to be cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

Controller 1/A

Ethernet port

Non 20xx configuration

ACP SAS ACP SAS

Stack 2

IOM A IOM B

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B


20xx single-controller configuration with one stack of disk shelves:

Controller 2/B

Ethernet port

20xx configuration

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B


20xx single-controller configuration with two stacks of disk shelves:

Controller 2/B

Ethernet port

20xx configuration

ACP SAS ACP SAS

Stack 2

IOM A IOM B

Top shelf

Bottom shelf

ACP SAS ACP SAS

Stack 1

IOM A IOM B


Appendix A: Terms and symbols

A
Terms and symbols
About this appendix This appendix defines terms and port icon symbols particular to SAS disk shelves and also includes common terms about connectivity.

The terms are organized by relationship to each other instead of alphabetical order.

List of terms and symbols

You should be familiar with the following terms:

◆ Square and circle symbols:

The SAS ports and ACP ports on the disk shelf IOMs are designated by square and circle symbols. All cabling is done in reference to connecting to a square port or a circle port.

NoteSAS connectivity does not use the concept of In ports and Out ports.

◆ IOM:

The SAS shelf I/O module is located in the back of the disk shelf. It connects the individual disk drives to the rest of the storage system and controls the disk shelf operator display panel LEDs. Each disk shelf has two IOMs: IOM A and IOM B. These are also referred to as SBB A and SBB B on the slot map label on the back of the disk shelf.

◆ SAS stack:

Also referred to as stack. A group of one or more SAS disk shelves connected (daisy-chained) together and connected to the controller through the top disk shelf in the stack and the bottom disk shelf in the stack (as needed). The maximum number of disk shelves in a stack of disk shelves and the number of disk shelf stacks in a configuration are dependent on the type of storage system.

◆ ACP:

Alternate Control Path is a protocol that enables Data ONTAP to manage and control the disk shelf storage subsystem. It uses a separate network from the data path, so management communication is not dependent on the data path being intact and available. Use of ACP requires that all disk shelf IOMs and storage system controllers connect through the ACP ports on the IOMs and the designated network interface on each controller.

◆ QSFP:

The standard SAS cable connector for all NetApp®SAS systems. QSFP to QSFP SAS cables are used to daisy-chain SAS disk shelf ports and connect

45

to the quad-port SAS host bus adapter (HBA) or onboard SAS ports. QSFP to mini-SAS cables are used to connect disk shelves to the dual-port mini-SAS HBA.

◆ Software-based disk ownership:

Software-based disk drive ownership is stored on the disk drive rather than determined by the topology of the storage system’s physical connections. It gives you increased flexibility and control over disk drive use. SAS and SATA disk drives use software-based disk drive ownership.

Therefore, the hardware-based disk ownership concept that the local node owns all direct-attached disk drives connected to it through the A channel (the loop attached to the A module on the disk shelf) and its partner owns the disk drives connected to the local node on the B channel, does not apply.

This also means that the hardware-based disk drive ownership terms do not apply: primary path, standby path, redundant path, and redundant standby path. Instead, connectivity between the controller and SAS disk shelf is addressed in terms of: top connections and bottom connections.

◆ Top connection:

The cabling connection from the controller to the top disk shelf in a stack of disk shelves.

References to the top disk shelf in this document are to the logical top of a stack of disk shelves. Your physical disk shelf stack might be arranged differently.

◆ Bottom connection:

The cabling connection from the controller to the bottom disk shelf in a stack of disk shelves.

References to the bottom disk shelf in this document are to the logical bottom of a stack of disk shelves. Your physical disk shelf stack might be arranged differently.

◆ Shelf-to-shelf (daisy-chain):

The cabling connection between disk shelves in a stack of more than one disk shelf. Each stack of disk shelves is daisy-chained through its SAS ports and, if you are using the ACP capability, each disk shelf is also daisy-chained through its ACP ports.

◆ Controller:

The component of your storage system that runs Data ONTAP software and manages the storage subsystem.

The controller located in the top slot in a storage system chassis can be referred to as Controller 1 or Controller A. The controller located in the bottom slot in a storage system chassis can be referred to as Controller 2 or Controller B.

46 Terms and symbols

In this document controllers are referred to as Controller 1/A and Controller 2/B.

◆ Single-controller configuration:

A storage system having one controller.

◆ HA pair configuration:

A storage system having two controllers. This can be two controllers in one chassis or it can be two controllers, each in a separate chassis. The two controllers serve data for each other if one of the controllers becomes impaired.

◆ Multipath HA:

Each controller in the HA pair configuration has multiple ways to connect to a disk drive. Multipath HA cabling is the most resilient and preferred configuration for HA pair configurations. This is because it takes full advantage of the resiliency capability of SAS, which means that the storage system continues to have access to disk drives in the event of cable, HBA, and IOM failure. A single failure of a cable, HBA, or IOM does not result in a controller failover.

◆ Dual-path:

The single-controller storage system has two ways to connect to a disk drive. This is the preferred configuration for single-controller configurations.

◆ Single-path HA:

Each controller in the HA pair configuration has one way to connect to a disk drive. This means an IOM or cable failure requires a controller failover.

◆ Single-path:

The single-controller storage system has one way to connect to a disk drive. This configuration has multiple single points of failure and is not a preferred configuration.

◆ Configured systems:

A new storage system that ships with SAS disk shelves and preinstalled SAS HBAs (if applicable).

◆ Add-on disk shelves:

SAS disk shelves that are shipped individually—not shipped as part of a configured system.

Appendix A: Terms and symbols 47

48 Terms and symbols

Appendix B: Document update record

B
Document update record
Document update history

The following table lists and describes the history of changes made to this document. Changes are not dictated by Data ONTAP releases.

Feature updatesFeature release date

◆ Updated with rules and exceptions for 2240 systems. November 2011

◆ Initial release of the SAS and ACP cabling information as a stand-alone document and updates.

❖ The SAS disk shelf illustrations are now generic so that they apply to any NetApp SAS disk shelf. A generic disk shelf is shown as two IOMs side-by-side (no power supplies are shown).

❖ HBAs (1, 2, 3 etc.) are now referred to as slots (1, 2, 3 etc.)

❖ Introduces the new rule for referring to onboard SAS ports as belonging to slot 0 and the 2040 system exception - the single onboard port is 0a.

❖ Introduces the new rule that for single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

❖ The term active/active is updated to HA pair throughout.

❖ The term Multipath is updated to Multipath HA.

September 2010

◆ Initial release of the SAS and ACP cabling information was part of the DS4243 Installation and Service Guide.

September 2009

49

50 Document update record

Index

AACP cabling procedure 33ACP capability 30

Ccabling ACP

examples 42procedure 33verifying ACP connections 36

cabling SASexamples 19rules 4verfiying SAS connections 15worksheet 13worksheet procedure 11

Tterminology 45

Index 51

52 Index

Date post:	14-Oct-2014
Category:	Documents
Upload:	02j
View:	103 times
Download:	1 times

215-05500_B0

Documents