+ All Categories
Home > Documents > Kraemer Gpfs Aix Faq

Kraemer Gpfs Aix Faq

Date post: 30-Oct-2014
Category:
Upload: vijay729
View: 33 times
Download: 0 times
Share this document with a friend
Popular Tags:
24
GPFS for AIX 5L Questions and Answers Overview The General Parallel File System (GPFS) for AIX 5L is a high-performance shared-disk file system that can provide data access from nodes in an AIX ® cluster environment. Parallel and serial applications can readily access shared files using standard UNIX ® file system interfaces, and the same file can be accessed concurrently from multiple nodes. GPFS is designed to provide high availability through logging and replication, and can be configured for failover from both disk and server malfunctions. GPFS has been widely accepted on the IBM ® RS/6000 ® SP system to scale file system I/O which can help meet the requirements of a wide range of applications including seismic data processing, digital library file serving, and data mining in business intelligence. GPFS for AIX 5L supports the IBM ERserver Cluster 1600 as well as clusters of IBM Eserver pSeries ® nodes or IBM Eserver p5 nodes. It is suggested you always check GPFS common FAQ question What is the current service information for GPFS? Updates to this FAQ include: Date Question October 2004 What disk support guidelines must be followed when running GPFS in an sp cluster type? What disk support guidelines must be followed when running GPFS in an rpd cluster type? What are the disk support guidelines that must be followed when running GPFS in an hacmp cluster type? What disks support hardware fencing? What versions of GPFS support EtherChannel? IBM TotalStorage ® FAStT has been renamed IBM TotalStorage DS4000 series DS4100 formerly FAStT100 DS4300 formerly FAStT600 DS4300 Turbo formerly FAStT600 Turbo DS4400 formerly FAStT700 DS4500 formerly FAStT900 | | | |
Transcript
Page 1: Kraemer Gpfs Aix Faq

GPFS for AIX 5L

Questions and

Answers ���

Overview

The General Parallel File System (GPFS) for AIX 5L™ is a high-performance shared-disk file system that

can provide data access from nodes in an AIX® cluster environment. Parallel and serial applications can

readily access shared files using standard UNIX® file system interfaces, and the same file can be accessed

concurrently from multiple nodes. GPFS is designed to provide high availability through logging and

replication, and can be configured for failover from both disk and server malfunctions.

GPFS has been widely accepted on the IBM® RS/6000® SP™ system to scale file system I/O which can

help meet the requirements of a wide range of applications including seismic data processing, digital

library file serving, and data mining in business intelligence.

GPFS for AIX 5L supports the IBM ERserver Cluster 1600 as well as clusters of IBM Eserver pSeries®

nodes or IBM Eserver p5 nodes.

It is suggested you always check GPFS common FAQ question What is the current service information for

GPFS?

Updates to this FAQ include:

Date Question

October 2004 What disk support guidelines must be followed when running GPFS in an sp cluster type?

What disk support guidelines must be followed when running GPFS in an rpd cluster type?

What are the disk support guidelines that must be followed when running GPFS in an hacmp

cluster type?

What disks support hardware fencing?

What versions of GPFS support EtherChannel?

IBM TotalStorage® FAStT has been renamed IBM TotalStorage DS4000 series

DS4100 formerly FAStT100

DS4300 formerly FAStT600

DS4300 Turbo formerly FAStT600 Turbo

DS4400 formerly FAStT700

DS4500 formerly FAStT900

||||

Page 2: Kraemer Gpfs Aix Faq

Questions & Answers

1. What is an sp cluster type?

2. When running GPFS in an sp cluster type, what are the software requirements?

3. What disk support guidelines must be followed when running GPFS in an sp cluster type?

4. What is an rpd cluster type?

5. When running GPFS in an rpd cluster type, what are the software requirements?

6. What disk support guidelines must be followed when running GPFS in an rpd cluster type?

7. What is an hacmp cluster type?

8. When running GPFS in an hacmp cluster type, what are the software requirements?

9. What are the disk support guidelines that must be followed when running GPFS in an hacmp

cluster type?

10. How can I implement disaster recovery for my GPFS cluster?

11. What are the coexistence rules for GPFS clusters?

12. What type of disk fencing mechanisms does GPFS support?

13. How can the fencing technique in operation for a cluster be verified?

14. What is AIX MPIO and does GPFS support it?

15. What disks support hardware fencing?

16. What should I do if commands such as mmcrlv, mmcrfs, mmadddisk, mmrpldisk, or mount fail to

complete due to problems with non-IBM disks?

17. What cluster specific single-node quorum restrictions are there?

18. Why would I want to use EtherChannel in my GPFS cluster?

19. What versions of GPFS support EtherChannel?

20. What’s the difference between using an ESS with or without SDD or SDDPCM installed on the host?

21. Is a Fibre Channel Switch required for all Fibre Channel disk attachments which are used by GPFS?

22. What Fibre Channel Switches are qualified for GPFS usage and is there a FC Switch support chart

available?

23. What performance tuning suggestions are there for AIX nodes?

24. What configuration and performance tuning suggestions are there for GPFS when used primarily for

Oracle databases?

25. What is the largest file system size supported by GPFS?

26. Are there any special concerns regarding the use of mksysb images?

27. Are there any considerations when using NIS automount?

28. How can I ask a more specific question about GPFS?

2

Page 3: Kraemer Gpfs Aix Faq

Q1: What is an sp cluster type?

A1: The GPFS cluster type sp is based on the IBM Parallel System Support Programs (PSSP) licensed

product and the shared disk concept of the IBM Virtual Shared Disk component of PSSP. In the

GPFS cluster type sp (a PSSP environment), the nodes which are members of the GPFS cluster

depend on the network switch type being used. In a system with an SP Switch, the GPFS cluster

is equal to all of the nodes in the corresponding SP partition which have GPFS installed. In a

system with an SP Switch2, the GPFS cluster is equal to all of the nodes in the system which

have GPFS installed. That is, the cluster definition is implicit and there is no need to run the

GPFS cluster commands. Within the GPFS cluster, you define one or more nodesets within which

your file systems operate. Disk requirements within your GPFS sp cluster require that all disks

must be:

1. Attached to a virtual shared disk server within the nodeset.

2. Processed via either the mmcrvsd command or the directions in the Parallel System Support

Programs for AIX: Managing Shared Disks book to create virtual shared disk useable by GPFS.

3. Conform to the disk support guidelines listed in this FAQ.

Q2: When running GPFS in an sp cluster type, what are the software requirements?

A2: In an sp cluster type, GPFS requires the Parallel System Support Programs (PSSP) licensed

product and its IBM Virtual Shared Disk and IBM Recoverable Virtual Shared disk components

for uniform disk access and recovery.

Table 1. Supported software levels for an sp cluster type

GPFS Version AIX Level PSSP Level GPFS End of Service

GPFS v2.2.1 AIX 5L v5.2 PSSP v3.5

GPFS v2.1 AIX 5L v5.1 or AIX 5L v5.2 PSSP v3.5 30April2006

Note: Ensure you are at the current service level for each software level listed.

Q3: What disk support guidelines must be followed when running GPFS in an sp cluster type?

A3: In an sp cluster type, a typical GPFS cluster configuration uses recoverable virtual shared disks

(RVSD). Deviations from the typical case are GPFS clusters which use concurrent virtual shared

disks (CVSD). Use of CVSD may have an impact on the types of disks supported. This distinction

is noted in the Cluster Properties section in Table 2 on page 4, Table 3 on page 5, Table 4 on page 5,

and Table 5 on page 6. There are no specific disk limitations in a two-node sp cluster running

single-node quorum (see the question What considerations are there for two node nodesets?).

For additional disk configuration information and examples, please see

ftp.software.ibm.com/storage/fastt/fastt500/PSSP-GPFS_config_info.pdf

3

Page 4: Kraemer Gpfs Aix Faq

Table 2. GPFS v2.2.1 and PSSP v3.5 running AIX 5L v5.2

Cluster Properties: Typical GPFS cluster configuration

Disks Supported: IBM TotalStorage® DS4100 with Serial Advanced Technology Attachment (SATA)

disk drives (SM8.42 microcode and APAR IY62752)

IBM TotalStorage FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)

IBM DS4300 (SM8.33 microcode)

IBM DS4300 Turbo (SM8.40 microcode)

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit

(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR

IY62752)

IBM DS4400 (SM8.21and SM8.3 microcode)

IBM DS4500 (SM8.3 microcode)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit

(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR

IY62752)

IBM TotalStorage® Enterprise Storage Server® (ESS) (2105-F20 or 2105-800 with

Subsystem Device Driver (SDD) - disk leasing mode)

IBM TotalStorage ESS (2105-F20 or 2105-800 using AIX Multiple Path I/O (MPIO)

and Subsystem Device Driver Path Control Module (SDDPCM) - persistent reserve

mode)

IBM SAN Volume Controller (APARs IY47267 and IY42355)

EMC Symmetrix Direct Matrix Architecture (DMX) Storage Subsystems (FC attach

only)

IBM Serial Storage Architecture (SSA) (all disk sizes)

Hitachi Lightning 9900™(9910, 9960, 9970V and 9980V (Hitachi Dynamic Link

Manager™) (HDLM) required)

Restrictions: Fibre Channel disks must be FC switch attached.

No disk can be larger than 1 TB.

4

||

|

|

|||

|

|

|||

Page 5: Kraemer Gpfs Aix Faq

Table 3. GPFS v2.2.1 and PSSP v3.5 running AIX 5L v5.2

Cluster Properties: CVSD required

Disks Supported: IBM TotalStorage® DS4100 with Serial Advanced Technology Attachment (SATA)

disk drives (SM8.42 microcode and APAR IY62752)

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit

(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR

IY62752)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit

(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR

IY62752)

IBM TotalStorage Enterprise Storage Server (ESS) (2105-F20 or 2105-800 using AIX

MPIO and SDDPCM - persistent reserve mode)

IBM SAN Volume Controller (APARs IY47267 and IY42355)

EMC Symmetrix DMX Storage Subsystems (FC attach only)

IBM SSA (all disk sizes)

Restrictions: Fibre Channel disks must be FC switch attached.

No disk can be larger than 1 TB.

Table 4. GPFS v2.1 and PSSP v3.5 running AIX 5L v5.1

Cluster Properties: Typical GPFS cluster configuration

Disks Supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR IY61688)

IBM FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)

IBM DS4300 (SM8.33 microcode)

IBM DS4300 Turbo (SM8.40 microcode)

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit

(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR

IY61688)

IBM DS4400 (SM8.21and SM8.3 microcode)

IBM DS4500 (SM8.3 microcode)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit

(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR

IY61688)

IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD)

IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD - persistent reserve mode)

IBM SAN Volume Controller (APARs IY47267 and IY42355)

EMC Symmetrix DMX Storage Subsystems (FC attach only)

IBM SSA (all disk sizes)

Hitachi Lightning 9900 (9910, 9960, 9970V and 9980V ) (HDLM required)

Restrictions: Fibre Channel disks must be FC switch attached.

No disk can be larger than 1 TB.

5

||

|||

|||

|

|

|

|||

|

|

|||

Page 6: Kraemer Gpfs Aix Faq

Table 5. GPFS v2.1 and PSSP v3.5 running AIX 5L v5.1

Cluster Properties: CVSD required

Disks Supported: IBM TotalStorage® DS4100 with Serial Advanced Technology Attachment (SATA)

disk drives (SM8.42 microcode and APAR IY62752)

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit

(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR

IY62752)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion Unit

(1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode and APAR

IY62752)

IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD - persistent reserve mode)

IBM SAN Volume Controller (APARs IY47267 and IY42355)

EMC Symmetrix DMX Storage Subsystems (FC attach only)

IBM SSA (all disk sizes)

Restrictions: Fibre Channel disks must be FC switch attached.

No disk can be larger than 1 TB.

Q4: What is an rpd cluster type?

A4: The GPFS cluster type rpd is based on the Reliable Scalable Cluster Technology (RSCT)

subsystem of AIX 5L. The GPFS cluster is defined within an existing RSCT peer domain. The

nodes which are members of the GPFS cluster are defined via the mmcrcluster, mmaddcluster,

and mmdelcluster commands. With an RSCT peer domain, all nodes in the GPFS cluster have the

same view of the domain and share the resources within the domain. Within the GPFS cluster,

you define one or more nodesets within which your file systems operate.

Disk requirements within your GPFS rpd cluster utilizing an IP network, require that all disks

must be:

1. Logical volumes:

v SAN-attached to all nodes in the cluster.

v Processed via the mmcrlv command to create logical volumes useable by GPFS.2. Virtual shared disks:

v Accessed via the IBM Virtual Shared Disk component of RSCT.

v Processed via the mmcrvsd command to create or register disks for use by GPFS.3. Conform to the disk support guidelines listed in this FAQ.

Disk requirements within your GPFS rpd cluster utilizing the IBM Eserver pSeries High

Performance Switch (HPS), require that all disks must be:

1. Accessed via the IBM Virtual Shared Disk component of RSCT.

2. Processed via the mmcrvsd command to create or register disks for use by GPFS.

6

||

|||

|||

Page 7: Kraemer Gpfs Aix Faq

Q5: When running GPFS in an rpd cluster type, what are the software requirements?

A5: In an rpd cluster type, GPFS requires the RSCT component of AIX. The GPFS cluster is defined

within an existing RSCT peer domain via the mmcrcluster command.

Table 6. Supported levels of AIX in an rpd cluster type

GPFS Version AIX Level GPFS End of Service

GPFS v2.2.1 AIX 5L Version 5 Release 2

For use of the Eserver p5, APAR IY58249, or later

modifications

For use of the pSeries HPS and IBM Virtual Shared Disks,

AIX 5L Version 5 Release 2 with APAR IY56532, or later

modifications.

GPFS v2.1 AIX 5L Version 5 Release 2 30April2006

For use of the pSeries HPS and IBM Virtual Shared Disks,

AIX 5L Version 5 Release 2 with APAR IY47306, or later

modifications.

30April2006

AIX 5L v5.1 5100-03 Recommended Maintenance package

(APAR IY30258), or later

30April2006

Note: Ensure you are at the current service level for each software level listed.

Q6: What disk support guidelines must be followed when running GPFS in an rpd cluster type?

A6: In an rpd cluster type, a typical GPFS cluster configuration uses multi-node quorum. Deviations

from the typical case are two-node GPFS clusters which have single-node quorum enabled.

Single-node quorum may have an impact on the types of disks supported. This distinction is

noted in the Cluster Properties section in Table 7 on page 8, Table 8 on page 8, Table 9 on page 9,

and Table 10 on page 9. For additional disk configuration information and examples, please see

ftp.software.ibm.com/storage/fastt/fastt500/PSSP-GPFS_config_info.pdf

7

Page 8: Kraemer Gpfs Aix Faq

Table 7. GPFS v2.2.1 running AIX 5L v5.2

Cluster properties: Typical GPFS cluster configuration (2 - 32 nodes)

Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR

IY62752)

IBM FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)

IBM DS4300 (SM8.33 microcode)

IBM DS4300 Turbo (SM8.40 microcode)

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage

Expansion Unit (1742-60U and 1742-60X ) with SATA disk drives

(SM8.42 microcode and APAR IY62752)

IBM DS4400 (SM8.21 and SM8.3 microcode)

IBM DS4500 (SM8.3 microcode)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion

Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42

microcode and APAR IY62752)

IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD - disk leasing

mode)

IBM TotalStorage ESS (2105-F20 or 2105-800 using AIX MPIO and

SDDPCM - persistent reserve mode)

IBM SAN Volume Controller (APARs IY47267 and IY42355)

EMC Symmetrix DMX Storage Subsystems (FC attach only)

IBM SSA (all disk sizes)

Hitachi Lightning 9900 (9910, 9960, 9970V and 9980V ) (HDLM

required)

Restrictions: Fibre Channel disks must be FC switch attached

Maximum cluster size is 8 nodes if using SSA disks

Table 8. GPFS v2.2.1 running AIX 5L v5.2

Cluster properties: Single-node quorum (two node cluster only)

Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR

IY62752)

IBM SSA (all disk sizes)

IBM TotalStorage Enterprise Storage Server (ESS) (2105-F20 or 2105-800

using AIX MPIO and SDDPCM - persistent reserve mode)

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage

Expansion Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR

IY62752)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion

Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR IY62752)

Restrictions: For single-node quorum - no disk leasing mode disks can be present.

Fibre Channel disks must be FC switch-attached.

8

||

|

|

|||

|

|

|||

||

|||

||

Page 9: Kraemer Gpfs Aix Faq

Table 9. GPFS v2.1 running AIX 5L v5.1

Cluster properties: Typical GPFS cluster configuration (2 - 32 nodes)

Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR

IY61688)

IBM FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)

IBM DS4300 (SM8.33 microcode)

IBM DS4300 Turbo (SM8.40 microcode)

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage

Expansion Unit (1742-60U and 1742-60X ) with SATA disk drives

(SM8.42 microcode and APAR IY61688)

IBM DS4400 (SM8.21 and SM8.3 microcode)

IBM DS4500 (SM8.3 microcode)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion

Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42

microcode and APAR IY61688)

IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD - disk leasing

mode)

IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD - persistent

reserve mode)

IBM SAN Volume Controller (APARs IY47267 and IY42355)

EMC Symmetrix/DMX Storage Subsystems (FC attach only)

IBM SSA (all disk sizes)

Hitachi Lightning 9900 (9910, 9960, 9970V and 9980V ) (HDLM

required)

Restrictions: Fibre Channel disks must be FC switch attached.

Maximum cluster size is 8 nodes if using SSA disks

Table 10. GPFS v2.1 running AIX 5L v5.1

Cluster properties: Single-node quorum (two node cluster only)

Disks supported: IBM SSA (all disk sizes)

IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD) - persistent

reserve mode

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage

Expansion Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR

IY61688)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion

Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR IY61688)

Restrictions: For single-node quorum - no disk leasing mode disks can be present.

Fibre Channel disks must be FC switch-attached.

9

||

|

|

|||

|

|

|||

|||

||

Page 10: Kraemer Gpfs Aix Faq

Notes:

1. You may have up to 1024 external shared disks, disk arrays, or logical units (LUNs) with the

adapters configured to allow each disk, connectivity to each node in the GPFS cluster. No disk

can be larger than 1 TB.

2. You cannot protect your file system against disk failure by mirroring data at the LVM level.

You must use GPFS replication or RAID devices to protect your data

1.

Q7: What is an hacmp cluster type?

A7: The GPFS cluster type hacmp is based on IBM’s High Availability Cluster Multi-Processing for

AIX (HACMP™) licensed product. The GPFS cluster is defined over an existing HACMP cluster.

The nodes which are members of the GPFS cluster are defined via the mmcrcluster,

mmaddcluster, and mmdelcluster commands.

Within the GPFS cluster, you define one or more nodesets within which your file systems operate.

Disk requirements within your GPFS hacmp cluster require that all disks must be:

1. SAN-attached to all nodes in the cluster.

2. Processed via the mmcrlv command.

3. Follow the disk support guidelines listed in this FAQ.

Q8: When running GPFS in an hacmp cluster type, what are the software requirements?

A8: In an hacmp cluster type, GPFS requires IBM’s HACMP licensed product over which the GPFS

cluster is defined via the mmcrcluster command.

Table 11. Supported software levels in an hacmp cluster type

GPFS Version AIX Level HACMP Level GPFS End of

Service

GPFS v2.2.1 AIX 5L v5.2 HACMP v5.1

For use of the Eserver p5, APAR IY58249, or later modifications

GPFS v2.1 AIX 5L v5.2 HACMP v4.5 / v5.1 30April2006

GPFS v2.1 AIX 5L v5.1 HACMP/ES v4.4.1/4.5 30April2006

Notes:

1. Ensure you are at the current service level for each software level listed

2. If running with HACMP/ES v4.5, the minimum APAR level required is IY32874

Q9: What are the disk support guidelines that must be followed when running GPFS in an hacmp

cluster type?

A9: In an hacmp cluster type, a typical GPFS cluster configuration uses multi-node quorum.

Deviations from the typical case are two-node GPFS clusters which have single-node quorum

enabled. Single-node quorum may have an impact on the types of disks supported. This

distinction is noted in the Cluster Properties section in Table 12 on page 11, Table 13 on page 11,

Table 14 on page 12, and Table 15 on page 12. For additional disk configuration information and

examples, please see ftp://ftp.software.ibm.com/storage/fastt/fastt500/PSSP-GPFS_config_info.pdf

1. IBM General Parallel File System for AIX 5L in an RSCT Peer Domain: Concepts, Planning, and Installation Guide Number GA22-7974

(July 2004), International Business Machines Corporation, 2455 South Road, Poughkeepsie, NY 12601-5400, USA.

10

Page 11: Kraemer Gpfs Aix Faq

Table 12. GPFS v2.2.1 running AIX 5L v5.2

Cluster properties: Typical GPFS cluster configuration (2- 32 nodes)

Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR

IY62752)

IBM FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)

IBM DS4300 (SM8.33 microcode)

IBM DS4300 Turbo (SM8.40 microcode)

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage

Expansion Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42

microcode and APAR IY62752)

IBM DS4400 (SM8.21 and SM8.3 microcode)

IBM DS4500 (SM8.3 microcode)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion

Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode

and APAR IY62752)

IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD - disk leasing

mode)

IBM TotalStorage ESS (2105-F20 or 2105-800 using AIX MPIO and

SDDPCM - persistent reserve mode)

IBM SAN Volume Controller (APARs IY47267 and IY42355)

EMC Symmetrix DMX Storage Subsystems (FC attach only)

IBM SSA (all disk sizes)

Hitachi Lightning 9900 (9910, 9960, 9970V and 9980V ) (HDLM required)

Restrictions: Fibre Channel disks must be FC switch attached.

Maximum cluster size is 8 nodes if using SSA disks

Table 13. GPFS v2.2.1 running AIX 5L v5.2

Cluster properties: Single-node quorum (two node cluster only)

Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR

IY62752)

IBM SSA (all disk sizes)

IBM TotalStorage Enterprise Storage Server (ESS) (2105-F20 or 2105-800

using AIX MPIO and SDDPCM - persistent reserve mode)

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage

Expansion Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR

IY62752)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion

Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR IY62752)

Restrictions: For single-node quorum no disk leasing mode disks can be present.

Fibre Channel disks must be FC switch attached.

11

||

|

|

|||

|

|

|||

||

|||

||

Page 12: Kraemer Gpfs Aix Faq

Table 14. GPFS v2.1 running AIX 5L v5.1

Cluster properties: Typical GPFS cluster configuration (2- 32 nodes)

Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR

IY61688)

IBM FAStT500 (SM7.10, SM8.21, and SM8.3 microcode)

IBM DS4300 (SM8.33 microcode)

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage

Expansion Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42

microcode and APAR IY61688)

IBM DS4400 (SM8.21 and SM8.3 microcode)

IBM DS4500 (SM8.3 microcode)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion

Unit (1742-60U and 1742-60X ) with SATA disk drives (SM8.42 microcode

and APAR IY61688)

IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD - disk leasing

mode)

IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD - persistent

reserve mode)

IBM SAN Volume Controller (APARs IY47267 and IY42355)

EMC Symmetrix DMX Storage Subsystems (FC attach only)

IBM SSA (all disk sizes)

Hitachi Lightning 9900 (9910, 9960, 9970V and 9980V ) (HDLM required)

Restrictions: Fibre Channel disks must be FC switch attached.

Maximum cluster size is 8 nodes if using SSA disks.

Table 15. GPFS v2.1 running AIX 5L v5.1

Cluster properties: Single-node quorum (two node cluster only)

Disks supported: IBM DS4100 with SATA disk drives (SM8.42 microcode and APAR

IY61688)

IBM SSA (all disk sizes)

IBM TotalStorage ESS (2105-F20 or 2105-800 without SDD) - persistent

reserve mode

IBM DS4300 Turbo with IBM TotalStorage DS4000 EXP100 Storage

Expansion Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR

IY61688)

IBM DS4500 with IBM TotalStorage DS4000 EXP100 Storage Expansion

Unit (1742-60U and 1742-60X ) (SM8.42 microcode and APAR IY61688)

Restrictions: For single-node quorum no disk leasing mode disks can be present.

Fibre Channel disks must be FC switch attached.

12

||

|

|||

|

|

|||

||

|||

||

Page 13: Kraemer Gpfs Aix Faq

Notes:

1. You may have up to 1024 external shared disks, disk arrays, or LUNs with the adapters

configured to allow each disk, connectivity to each node in the GPFS cluster. No disk can be

larger than 1 TB.

2. You cannot protect your file system against disk failure by mirroring data at the LVM level.

You must use GPFS replication or RAID devices to protect your data

2.

Q10: How can I implement disaster recovery for my GPFS cluster?

A10: Beginning with GPFS 2.2.1 and APAR IY59339, disaster recovery for your GPFS cluster can be

implemented as described in Disaster Recovery with General Parallel File System located at

http://www.ibm.com/servers/eserver/pseries/software/whitepapers/gpfs_recovery.html

Q11: What are the coexistence rules for GPFS clusters?

A11: You must be concerned with the coexistence rules for each type of GPFS cluster. Updates to the

coexistence rules for GPFS clusters listed in this FAQ supersede the product documentation.

Table 16 and Table 17 identify release levels of additional clustering software which may coexist

on the same node. This does not imply interaction with GPFS if the software is not required for

the particular GPFS cluster type. Please note that PSSP and Cluster System Management (CSM)

may not coexist and therefore the information is presented in two separate tables. There are also

special considerations noted in Table 18 on page 14.

Table 16. Coexistence of clustering software with PSSP

GPFS cluster type GPFS level AIX level PSSP level HACMP level

rpd (IP network) 2.2.1 5.2 3.5 5.1/4.5

rpd (IP network) 2.1 5.2/5.1 3.5 5.1/4.5/4.4.1

hacmp 2.2.1 5.2 3.5 5.1/4.5

hacmp 2.1 5.1 3.5 4.5/4.4.1

sp 2.2.1 5.2 3.5 5.1/4.5

sp 2.1 5.1 3.5 4.5/4.4.1

Table 17. Coexistence of clustering software with CSM

GPFS cluster type GPFS level AIX level CSM level HACMP level

rpd (IP network) 2.2.1 5.2 1.4 / 1.3.x.x 5.1/4.5

rpd pSeries HPS 2.2.1 5.2 1.4 / 1.3x.x 5.1/4.5

rpd (IP network) 2.1 5.2/5.1 1.1.x.x 5.1/4.5/4.4.1

rpd pSeries HPS 2.1 5.2 1.4 / 1.3.x.x 5.1/4.5/4.4.1

hacmp 2.2.1 5.2 1.4 / 1.3.x.x 5.1/4.5

hacmp 2.1 5.2/5.1 1.1. x.x 4.5/4.4.1

Notes:

1. The fact that GPFS and CSM may coexist does not imply that there is interaction between

GPFS and CSM.

2. For the CSM levels, x.x refers to the current modification and fix levels.

3. PSSP does not support the pSeries HPS, therefore a GPFS rpd cluster utilizing the pSeries HPS

may not coexist with PSSP

2. IBM General Parallel File System for AIX 5L in an HACMP Cluster: Concepts, Planning, and Installation Guide Number GA22-7971-00

(December 2003), International Business Machines Corporation, 2455 South Road, Poughkeepsie, NY 12601-5400, USA.

13

Page 14: Kraemer Gpfs Aix Faq

Table 18. Special considerations which must be adhered to when the products coexist

GPFS Cluster Type CSM RSCT peer domain PSSP HACMP

rpd 2 2 1,2 2

sp 2 2 2 2

hacmp 2 2 1,2 2

1. When running in either a GPFS rpd or hacmp cluster type that contains nodes that are also

within an sp cluster, you need to be concerned with the daemon to daemon authentication

method for the common nodes. GPFS, as an SP trusted service, can have its daemon to

daemon authentication method set by the system administrator. The authentication method

for all nodes in your GPFS nodeset must be compatible. If nodes within a cluster are not

configured for a compatible authentication method, an error condition will be written

to/var/adm/ras/mmfs.log.latest when there is an attempt to make a connection between them.

See the section on Security in the GPFS for AIX 5L in a PSSP Cluster: Clusters Concepts,

Planning, and Installation Guide and search on Security.

2. The level of RSCT is determined by your AIX service level.

Q12: What type of disk fencing mechanisms does GPFS support?

A12: In order to preserve data integrity in the event of certain system failures, GPFS is designed to

prevent a node that is down from accessing the file system until it returns to the available state.

GPFS is said to fence the failing node from the disks.

There are two types of fencing mechanisms:

v Fencing that takes advantage of hardware assist

v Fencing that is done in GPFS software without hardware assist

Depending upon the types of disks that are being using, there are three possible ways for the

fencing to occur:

SSA fencing

An SSA disk subsystem specific fencing mechanism. SSA fencing takes advantage of

hardware assistance.

persistent reserve

SCSI-3 standard reserve mechanism. Persistent reserve takes advantage of hardware

assistance. See the question Does GPFS support AIX MPIO?

Disk leasing

A GPFS specific fencing mechanism for disks which do not support either SSA fencing or

persistent reserve. Disk leasing does not rely on any hardware assistance.

GPFS can use different types of disks in the same cluster. GPFS will determine the disks

capability and prepare the disks for proper use during the setup phase. This preparation is

performed by the GPFS command mmcrlv and mmcrvsd, and can also be performed by the

RVSD command createvsd. No options need to be provided to direct the GPFS codes to perform

this function.

GPFS will always attempt to use fencing with hardware assistance if possible.

There can only be one fencing mechanism active in a cluster, hardware assisted or disk leasing.

If all the disks in a configuration support fencing with hardware assistance, that configuration

will be running in a useDiskLeasing no environment. In other words, disk leasing is not being

used in this environment to perform fencing, the fencing is being done with the assistance of the

disk subsystem hardware and firmware.

If any disk in a GPFS configuration does not support hardware fencing, all disks in that same

environment revert to using disk leasing. If any or all disks in a configuration do not support

14

Page 15: Kraemer Gpfs Aix Faq

hardware fencing, the cluster will be running in a useDiskLease yes environment. In other

words, disk leasing is being used and none of the capabilities of the disk subsystem and

firmware are being exploited for fencing.

Single-node quorum is not supported when disk leasing is in effect.

The only manual action that needs to be performed by the administrator is in the SSA disk types,

the ODM attribute ssar needs to be setup on each node in the cluster. For directions on using the

mmlsfenceid and mmsetfenceid commands:

v For the GPFS cluster type rpd, please see the General Parallel File System for AIX 5L in an RSCT

Peer Domain: Problem Determination Guide .

v For a GPFS cluster type of hacmp, please the GPFS for AIX 5L in an HACMP Cluster General

Parallel File System for AIX 5L in an HACMP Cluster: Problem Determination Guide .

Q13: How can the fencing technique in operation for a cluster be verified?

A13: The current fencing mode for a cluster is either useDiskLease yes or useDiskLease no. The

current state of disk leasing in a cluster is determined by using the mmlsconfig command.

Q14: What is AIX MPIO and does GPFS support it?

A14: Beginning with AIX 5.2, GPFS can use the Multiple Path I/O (MPIO) feature of AIX to define

alternate paths to a device for failover purposes. Failover is a path-management algorithm that

improves the reliability and availability of a device because the system automatically detects

when one I/O path fails and re-routes I/O through an alternate path. All SCSI SCSD disk drives

are automatically configured as MPIO devices. Other devices can be supported, providing the

device driver is compatible with the MPIO implementation in AIX. For more information about

MPIO, see the:

v AIX 5L Version 5.2 System Management Concepts: Operating System and Devices book and search

on Multi-path I/O.

v AIX 5L Version 5.2 System Management Guide: Operating System and Devices book and search on

Multi-path I/O.

As GPFS supports disk subsystems exploiting the AIX MPIO function, this FAQ will be updated

and the new support statements will be communicated to the IBM Sales representatives. Please

see:

v Q3: What disk support guidelines must be followed when running GPFS in an sp cluster type?

v Q6: What disk support guidelines must be followed when running GPFS in an rpd cluster

type?

v Q9:What are the disk support guidelines that must be followed when running GPFS in an

hacmp cluster type?

Q15: What disks support hardware fencing?

A15: The current set of disks that support hardware fencing are:

v IBM SSA – all models

v IBM TotalStorage ESS using AIX MPIO and SDDPCM

v Selected members of the IBM TotalStorage DS4000 family of subsystems with microcode at

versions 8.40 or greater. See the individual cluster types (sp, rpd, hacmp) for details.

Q16: What should I do if commands such as mmcrlv, mmcrfs, mmadddisk, mmrpldisk, or mount

fail to complete due to problems with non-IBM disks?

A16: Certain GPFS disk commands issue multiple AIX varyonvg -u commands in parallel to the nodes

in the cluster. For some non-IBM disks, when many varyonvg -u commands are issued in

parallel, some of the AIX varyonvg -u invocations do not complete and the GPFS disk command

hangs. This is recognized by the GPFS disk command not completing after a long period of time

and the persistence of the varyonvg processes as shown by the output of the ps -ef command on

some of the nodes of the cluster. In such cases, kill the varyonvg processes issued by the disk

15

||

Page 16: Kraemer Gpfs Aix Faq

command on the nodes of the cluster to allow the disk command to complete. Before mounting

the file system in question on any node where a varyonvg process was killed, issue the varyonvg

-u vgname command on the node to make the disk available to GPFS. Do this on each of the

nodes in question one by one until all of the GPFS volume groups are varied on.

Q17: What cluster specific single-node quorum restrictions are there?

A17: Restrictions when using single-node quorum include:

1. Single-node quorum is not supported when:

v The GPFS cluster type lc consists of only AIX nodes.

v Disk leasing is in effect. Disk leasing is a GPFS-specific fencing mechanism for disks which

do not support either SSA fencing or persistent reserve. Disk leasing is activated if any disk

in any file system in the nodeset is not using SSA fencing or hardware assisted persistent

reserve. See the question What type of disk fencing mechanisms does GPFS support?

v SDD (the non-MPIO version) is installed.2. If a disk is using persistent reserve, this needs to be both supported at both the disk

subsystem level and the AIX device driver level.

3. When adding nodes to a nodeset:

v The GPFS daemon must be stopped on all of the nodes.

v If after adding the nodes, the number of nodes in the nodeset exceeds two, the quorum

algorithm is automatically changed to the multi-node quorum algorithm.4. When deleting nodes from a nodeset:

v A node cannot be deleted from the nodeset without stopping the GPFS daemon on both

nodes.

v If the number of nodes remaining in the nodeset falls below three, you must run the

mchconfig command if you want to change the quorum algorithm to single-node quorum.

See the question, What considerations are there for two node nodesets?

Q18: Why would I want to use EtherChannel in my GPFS cluster?

A18: The main benefit of EtherChannel and IEEE 802.3ad Link Aggregation is increased bandwidth:

the aggregated adapter has the network bandwidth of all its adapters in a single network

presence. In addition, if an adapter fails, the packets are automatically sent on the next available

adapter without disruption to existing user connections. The adapter is automatically returned to

service on the EtherChannel or Link Aggregation when it recovers.

EtherChannel and IEEE802.3ad each requires support within the Ethernet switch. Refer to the

product documentation for your switch to detemine if EtherChannel is supported. To determine if

your version of GPFS supports EtherChannel, see the question, What versions of GPFS support

EtherChannel?

For details on how to configure EtherChannel and IEEE 802.3ad Link Aggregation:

1. Go to

publib16.boulder.ibm.com/pseries/en_US/aixbman/commadmn/tcp_etherchannel.htm#yu528frokferg

2. Search on Configuring EtherChannel

Hint: Make certain that the switch ports are configured for LACP (the default is PAGP).

For details on how to verify whether the adapter and the switch are operating with the correct

protocols For IEEE 802.3ad:

1. Go to

publib16.boulder.ibm.com/pseries/en_US/aixbman/commadmn/tcp_etherchannel.htm#yu528frokferg

2. Search on Troubleshooting IEEE 802.3ad.

For additional service updates regarding the use of EtherChannel:

1. Go to www.ibm.com/support

2. Click Search technical support

3. Enter the search term EtherChannel

16

Page 17: Kraemer Gpfs Aix Faq

4. Click Submit

Hint: A useful command for troubleshooting, where device is the Link Aggregation device, is:

entstat -d device

Q19: What versions of GPFS support EtherChannel?

A19: GPFS support for EtherChannel includes:

v GPFS 2.2 on cluster types rpd or hacmp on AIX 5.2 and maintenance package 5.2.0.13 with

APAR IY48170 and APAR IY44290

v GPFS 2.1 on cluster types rpd or hacmp on AIX 5.1 and maintenance package 5.1.0.50 with

APAR IY48331

For the latest service level, go to www-912.ibm.com/eserver/support/fixes/fcgui.jsp

1. For your server type, select pSeries family.

2. For product or fix type select AIX OS, Java™, compilers.

3. For your option, select the type of fixes you are searching for.

4. Select your OS level.

5. Click continue.

6. If choosing Latest fixes or Critical fixes, the GPFS fixes will be bundled with all of the file

sets.

7. If you choose Specific fixes, enter mmfs as your search string

Q20: What’s the difference between using an ESS with or without SDD or SDDPCM installed on

the host?

A20: The use of SDD or SDDPCM gives the AIX host the ability to access multiple paths to a single

LUN within an ESS. This ability to access a single LUN on multiple paths allows for a higher

degree of data availability in the event of a path failure. Data can continue to be accessed within

the ESS as long as there is at least one available path. Without one of these installed, you will

lose access to the LUN in the event of a path failure.

However, your choice of whether to use SDD or SDDPCM impacts your ability to use single-node

quourm:

v Single-node quorum is not supported if SDD is installed.

v Single-node quorum is support if SDDPCM is installed.

To determine the GPFS disk support guidelines for SDD and SDDPCM for your cluster type, see

v Q3: What disk support guidelines must be followed when running GPFS in an sp cluster type?

v Q6: What disk support guidelines must be followed when running GPFS in an rpd cluster

type?

v Q9:What are the disk support guidelines that must be followed when running GPFS in an

hacmp cluster type?

Q21: Is a Fibre Channel Switch required for all Fibre Channel disk attachments which are used by

GPFS?

A21: Yes, all Fibre Channel disks require attachment to AIX hosts using a SAN FC switch when used

by GPFS.

Q22: What Fibre Channel Switches are qualified for GPFS usage and is there a FC Switch support

chart available?

A22: There are no special requirements for FC switches used by GPFS other than the switch must be

supported by AIX. For further information see www.storage.ibm.com/ibmsan/index.html

Q23: What performance tuning suggestions are there for AIX nodes?

A23: Performance tuning suggestions for AIX nodes include:

ipqmaxlen

In a GPFS cluster type sp or rpd (utilizing the pSeries HPS), the ipqmaxlen network

17

||||

Page 18: Kraemer Gpfs Aix Faq

option controls the number of incoming packets that can exist on the IP interrupt queue.

Since both GPFS and the IBM Virtual Shared Disk use IP, the default value of 128 is often

insufficient. This is especially important if your virtual shared disks are configured over

IP rather than KLAPI. The recommended setting is 512:

no -o ipqmaxlen=512

Buddy buffers

In a GPFS cluster type sp or rpd (utilizing the pSeries HPS), the buddy buffer is pinned

kernel memory. The virtual shared disk server node uses the buddy buffer to temporarily

store data for I/O operations originating at a client node. The data in a buddy buffer is

purged immediately after the I/O operation completes. The values associated with the

buddy buffer are:

v Minimum buddy buffer size allocated to a single request

v Maximum buddy buffer size allocated to a single request

v Total number of maximum-size buffers

Total number of maximum-size buddy buffers that the system will attempt to

dynamically allocate

IBM suggests using the following values:

v Minimum buddy buffer size: 4096 (4 KB)

v Maximum buddy buffer size: 262144 (256 KB)

If your application uses the fastpath option of asynchronous I/O, the maximum buddy

buffer size must be greater than or equal to 128 KB. Otherwise, you will receive

EMSGSIZE Message too long errors.

v Total number of maximum-size buffers:

– 2000 for a virtual shared disk server node

– One for a virtual shared disk client-only node

You can either set these values using the vsdnode command, or update the values using

the updatevsdnode command. IBM suggests, however, that you configure the

recommended amount of buddy buffer space, unless your system is memory-constrained

and you want to restrict the amount of buddy buffer space.

When the device driver is configured, the total buddy buffer space is not pinned; instead,

approximately one-quarter of the total space requested is pinned when the device driver

is configured. This initial amount of spaced pinned is limited to a maximum of 64 MB for

a 32-bit kernel, or 128 MB for a 64-bit kernel. After configuration, the device driver

attempts to dynamically expand and contract additional buddy buffer space up to the

maximum specified, or until AIX can no longer satisfy the memory request. If a buddy

buffer cannot be obtained, then the request is queued at the virtual shared disk server

until a buddy buffer is available.

max_coalesce

The max_coalesce (SSA) parameter defines the maximum number of bytes that the SSA

disk device driver attempts to transfer to or from an SSA logical disk in one operation.

This parameter is very important when dealing with writes on SSA RAID systems

3. The

suggested setting is 256KB for a RAID-5 4+P configuration:

chdev -l hdiskx -a max_coalesce=0x40000

KLAPI

In the GPFS cluster type sp, the default IBM Virtual Shared Disk communication protocol

is IP. KLAPI is an alternate high-performance communication interface that can be used

to avoid some data copies (which would have been in the IP path) in handling the I/O

request. Avoiding the data copies reduces CPU utilization on both the client and server

nodes. In order to switch from IP to KLAPI, the RVSD system has to be stopped and IBM

3. Dominique Heger, Gautam Shah: General Parallel File System (GPFS v1.4) for AIX Architecture and Performance, November 2001

4. Dominique Heger, Gautam Shah: General Parallel File System (GPFS v1.4) for AIX Architecture and Performance, November 2001

18

Page 19: Kraemer Gpfs Aix Faq

Virtual Shared Disk component has to be unconfigured. In the case that IBM Virtual

Shared Disk over KLAPI encounters a communication problem, the system will switch to

IBM Virtual Shared Disk over IP. Choosing KLAPI will not affect the number of

user-space SP Switch2 windows that are available to user applications

4:

ctlvsd -l on

SP Switch, SP Switch2, and pSeries HPS subsystem parameters

Depending upon which switch you are using, the communications subsystem parameters

should be configured differently. These are especially important if you configure your

virtual shared disks over IP.

If you are using an SP Switch, the rpoolsize and spoolsize values should be set to

16777216:

chgcss -l css0 -a rpoolsize=16777216 -a spoolsize=16777216

If you are using an SP Switch2, the rpoolsize and spoolsize values should be set to

33554432:

chgcss -l css0 -a rpoolsize=33554432 -a spoolsize=33554432

If you are using the pSeries HPS, the rpoolsize and spoolsize values should be set to

67108864:

chgsni -l sn0 –a rpoolsize=67108864 –a spoolsize 67108864

For further performance information on the SP Switch or the SP Switch2, see the GPFS

white papers at www.ibm.com/servers/eserver/clusters/library/wp_aix_lit.html

Q24: What configuration and performance tuning suggestions are there for GPFS when used

primarily for Oracle databases?

A24: Oracle uses the Asynchronous I/O (AIO) and Direct I/O (DIO) features of AIX to do their own

scheduling of I/O directly to disks, bypassing most of GPFS caching and prefetching facilities.

Most of these settings are to make sure GPFS is out of the way once a database has been

formatted, so Oracle can get the best performance:

v When setting up your LUNS it is important to:

1. Create the logical volumes such that they map one to one with a volume group and a

volume group be one to one with a LUN which is a single RAID device.

2. Not stripe logical volumes across multiple RAIDs for I/O parallelism when using raw

logical volumes because:

– GPFS puts each block of a file on different LUNs spread across all LUNs

– Logical volume striping makes removing a bad RAID more difficultv For file systems holding large Oracle databases, set the GPFS file system block size via the

mmcrfs command using the -B option, to a large value:

– 512 KB is generally suggested.

– 256 KB is suggested if there is activity other than Oracle using the filesystem and many

small files exist which are not in the database.

– 1 MB is suggested for file systems 100 TB or larger.

The large block size makes the allocation of space for the databases manageable and has no

affect on performance when Oracle is using AIO/DIO.

v Set the GPFS worker threads via the mmchconfig - worker1Threads command to allow the

maximum parallelism of the Oracle AIO threads:

– On a 64-bit AIX kernel, the setting can be as large as 548.

The GPFS prefetch threads must be adjusted accordingly via the mmchconfig -

prefetchThreads command as the sum of those two classes of threads must be 550 or less.

– On a 32-bit kernel, the setting can be as large as 162.

The GPFS prefetch threads must be adjusted accordingly via the mmchconfig -

prefetchThreads command as the sum of those two classes of threads must be 164 or less.

19

Page 20: Kraemer Gpfs Aix Faq

When requiring GPFS sequential I/O, set the prefetch threads between 50 and 100 (the default

is 64), and set the worker threads to have the remainder.

Note: These changes via the mmchconfig command take effect upon restart of the GPFS

daemon.

v The number of AIX AIO kprocs to create should be approximately the same as the GPFS

worker1Threads setting. For the AIO maxservers setting:

– On AIX 5.1 systems, it is the total number of AIO kprocs.

– On AIX 5.2 systems, it is the number of kprocs PER CPU. It is suggested to set is slightly

larger than worker1Threads divided by the number of CPUs.

For example if worker1Threads is set to 500 on a 32-way SMP:

– On an AIX 5.1 system, set maxservers to 640

– On an AIX 5.2 system, set maxservers to 20v Setting the GPFS striping method to balanced random via the mmcrfs -s balancedRandom or

the mmchfs -s balancedRandom command may prove to make disk access patterns slightly

more random than the system default. This may avoid creating hotspots thus creating less

work for the GPFS administrative operations that rebalance data across all LUNs such as

mmdeldisk, mmadddisk, and mmrestripefs.

v Set the Oracle database block size equal to the LUN segment size or a multiple of the LUN

pdisk segment size.

v Set the Oracle read-ahead value to prefetch one or two full GPFS blocks. For example, if your

GPFS block size is 512 KB, set the Oracle blocks to either 32 or 64 16 KB blocks.

v Do not use the dio option on the mount command as this forces DIO when accessing all files.

Oracle automatically uses DIO to open database files on GPFS.

v There are fixes in the GPFS PTF stream necessary for correct AIO/DIO operation for Oracle. If

using GPFS release 2.1, the mmfs.base.rte fileset should be at least 3.5.0.11.

Q25: What is the largest file system size supported by GPFS?

A25: On 32- bit AIX machines, the largest supported file system size depends on the blocksize of your

file system:

v If your file system has a 16 KB block size, you may have one or more file systems with a total

size of 1 TB mounted.

v If your file system has a 64 KB block size, you may have one or more file systems with a total

size of 10 TB mounted

v If your file system has a 256 KB or greater block size, you may have one or more file systems

mounted with a total size of not greater than 200 TB where no single file system exceeds 100

TB mounted.

The maximum supported file system size of 100 TB is a testing limitation rather than an

architectural limitation of the GPFS licensed product.

Q26: Are there any special concerns regarding the use of mksysb images?

A26: Using a mksysb image to install or reinstall the IBM Virtual Shared Disk licensed product on a

node presents some challenges to be aware of.

In a GPFS cluster type sp:

v The RVSD version number that the post-installation scripts would normally save in the System

Data Repository can not be saved. This may prevent the RVSD subsystem from starting

because it thinks that there are back-level nodes within the SP partition. This problem can be

prevented by using the rvsdrestrict command to explicitly set the restriction level before

starting RVSD. For example, on the control workstation issue:

rvsdrestrict -s RVSD3.5

v The RVSD subsystem expects that the underlying disks and volume groups are available to

AIX. Therefore, if you are reinstalling an existing server node from a mksysb image, you may

need to use the AIX importvg command to make AIX aware of the pre-existing volume groups.

20

Page 21: Kraemer Gpfs Aix Faq

Note: All virtual shared disk volume groups must have the auto varyon attribute turned off. To

accomplished this, after the importvg, issue:

chvg -a -n volume_group_name

In a GPFS cluster type rpd, duplicate node ids may be created. The lsnodeid command (available

in /usr/sbin/rsct/bin) has been provided for you to verify whether or not node ids are duplicated

within the cluster. If a duplicate node id is found, see the General Parallel File System for AIX 5L in

an RSCT Peer domain: Administration and Programming Reference and search on Changing IP

addresses, node names, node numbers and reliable hostnames

Q27: Are there any considerations when using NIS automount?

A27: The default file system type when using the automounter daemon is NFS. When the -fstype

option is not specified, and the server is the local node, a soft-mount of the local directory is

done at the desired mount point. JFS is assumed as the only handler of local directories. A GPFS

file system local soft-mount does not work implicitly, since the mount request is passed to JFS

which then produces an error. When specifying -fstype mmfs the local soft-mount works because

the mount is then passed to GPFS instead of JFS.

A GPFS soft-mount does not automatically unmount. Setting -fstype nfs3 causes the local server

mounts to always go through NFS. This allows you to have the same auto.map file on all nodes

whether the server is local or not and the automatic unmount will occur. If you want local

soft-mounts of GPFS file systems while other nodes perform NFS mounts, you should have

different auto.map files on the different classes of nodes. This should improve performance on

the GPFS nodes as they will not have to go through NFS.

Q28: How can I ask a more specific question about GPFS?

A28: If you have a question that can benefit other GPFS users, then please post it to the public GPFS

community mailing list ([email protected]). You can sign up for this list at

oss.software.ibm.com/developerworks/mail/?group_id=91

You can also send a note directly to the GPFS development team at [email protected]. However,

this mailing list is informally monitored as time permits and should not be used for priority

messages to the GPFS team.

If your query concerns a potential software error in GPFS and you have an IBM software

maintenance contract, then please contact 1-800-IBM-SERV in the United States or your local IBM

Service Center in other countries. IBM Scholars Program users should notify the GPFS

development team of potential software bugs via [email protected].

Note: This FAQ is continually being enhanced. Please contribute questions or answers by sending

them to [email protected]

21

Page 22: Kraemer Gpfs Aix Faq

Notices

This information was developed for products and

services offered in the U.S.A.

IBM may not offer the products, services, or

features discussed in this document in other

countries. Consult your local IBM representative

for information on the products and services

currently available in your area. Any reference to

an IBM product, program, or service is not

intended to state or imply that only IBM’s

product, program, or service may be used. Any

functionally equivalent product, program, or

service that does not infringe any of IBM’s

intellectual property rights may be used instead.

However, it is the user’s responsibility to evaluate

and verify the operation of any non-IBM product,

program, or service.

IBM may have patents or pending patent

applications covering subject matter described in

this document. The furnishing of this document

does not give you any license to these patents.

You can send license inquiries, in writing, to:

IBM Director of Licensing

IBM Corporation

North Castle Drive

Armonk, NY 10594-1785

USA

For license inquiries regarding double-byte

(DBCS) information, contact the IBM Intellectual

Property Department in your country or send

inquiries, in writing, to:

IBM World Trade Asia Corporation

Licensing

2-31 Roppongi 3-chome, Minato-ku

Tokyo 106, Japan

The following paragraph does not apply to the

United Kingdom or any other country where such

provisions are inconsistent with local law:

INTERNATIONAL BUSINESS MACHINES

CORPORATION PROVIDES THIS PUBLICATION

“AS IS” WITHOUT WARRANTY OF ANY KIND,

EITHER EXPRESS OR IMPLIED, INCLUDING,

BUT NOT LIMITED TO, THE IMPLIED

WARRANTIES OF NON-INFRINGEMENT,

MERCHANTABILITY OR FITNESS FOR A

PARTICULAR PURPOSE. Some states do not

allow disclaimer of express or implied warranties

in certain transactions, therefore, this statement

may not apply to you.

This information could include technical

inaccuracies or typographical errors. Changes are

periodically made to the information herein; these

changes will be incorporated in new editions of

the publication. IBM may make improvements

and/or changes in the product(s) and/or the

program(s) described in this publication at any

time without notice.

Any references in this information to non-IBM

Web sites are provided for convenience only and

do not in any manner serve as an endorsement of

those Web sites. The materials at those Web sites

are not part of the materials for this IBM product

and use of those Web sites is at your own risk.

IBM may use or distribute any of the information

you supply in any way it believes appropriate

without incurring any obligation to you.

Licensees of this program who wish to have

information about it for the purpose of enabling:

(i) the exchange of information between

independently created programs and other

programs (including this one) and (ii) the mutual

use of the information which has been exchanged,

should contact:

IBM Corporation

Intellectual Property Law

2455 South Road,P386

Poughkeepsie, NY 12601-5400

USA

Such information may be available, subject to

appropriate terms and conditions, including in

some cases, payment or a fee.

The licensed program described in this document

and all licensed material available for it are

provided by IBM under terms of the IBM

Customer Agreement, IBM International Program

License Agreement or any equivalent agreement

between us.

Any performance data contained herein was

determined in a controlled environment.

Therefore, the results obtained in other operating

environments may vary significantly. Some

22

Page 23: Kraemer Gpfs Aix Faq

measurements may have been made on

development-level systems and there is no

guarantee that these measurements will be the

same on generally available systems.

Furthermore, some measurements may have been

estimated through extrapolation. Actual results

may vary. Users of this document should verify

the applicable data for their specific environment.

This information contains examples of data and

reports used in daily business operations. To

illustrate them as completely as possible, the

examples include the names of individuals,

companies, brands, and products. All of these

names are fictitious and any similarity to the

names and addresses used by an actual business

enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application

programs in source language, which illustrates

programming techniques on various operating

platforms. You may copy, modify, and distribute

these sample programs in any form without

payment to IBM, for the purposes of developing,

using, marketing or distributing application

programs conforming to the application

programming interface for the operating platform

for which the sample programs are written. These

examples have not been thoroughly tested under

all conditions. IBM, therefore, cannot guarantee or

imply reliability, serviceability, or function of

these programs. You may copy, modify, and

distribute these sample programs in any form

without payment to IBM for the purposes of

developing, using, marketing, or distributing

application programs conforming to IBM’s

application programming interfaces.

If you are viewing this information softcopy, the

photographs and color illustrations may not

appear.

Trademarks

The following terms are trademarks of the

International Business Machines Corporation in

the United States or other countries or both:

v AFS

v AIX

v AIX 5L

v Eserver

v Enterprise Storage Server

v HACMP

v IBM

v IBMLink

v LoadLeveler

v Netfinity

v pSeries

v POWER

v PowerPC

v RS/6000

v Redbooks

v SANergy

v Scalable POWERParallel Systems

v ServerProven

v SP

v Tivoli

v TotalStorage

v xSeries

Dynamic Link Manager, Lightning 9900, Thunder

9200, and Thunder 9500 are trademarks of Hitachi

Data Systems Corporation.

Intel, Intel Inside (logos), MMX and Pentium are

trademarks of Intel Corporation in the United

States, other countries, or both.

Java and all Java-based trademarks and logos are

trademarks or registered trademarks of Sun

Microsystems, Inc. in the United States, other

countries, or both.

Linux is a trademark of Linus Torvalds in the

United States, other countries, or both.

Microsoft, Windows, Windows NT, and the

Windows logo are trademarks of Microsoft

Corporation in the United States, other countries,

or both.

Red Hat, the Red Hat ″Shadow Man″ logo, and

all Red Hat-based trademarks and logos are

trademarks or registered trademarks of Red Hat,

Inc., in the United States and other countries.

UNIX is a registered trademark of the Open

Group in the United States and other countries.

Other company, product, and service names may

be the trademarks or service marks of others.

23

Page 24: Kraemer Gpfs Aix Faq

October 2004

© Copyright International Business Machines Corporation 2003, 2004. All rights reserved.

US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract

with IBM Corp.


Recommended