2.3 What is the IBM Spectr um Scale support position r ... · IBM Spectrum Scale ™ Frequently...

IBM Spectrum Scale™ Frequently AskedQuestions and Answers

IBM Spectrum Scale Overview

IBM Spectrum Scale™, based on technology from IBM General Parallel File System (hereinafter referred toas IBM Spectrum Scale or GPFS™), is a high performance shared-disk file management solution thatprovides fast, reliable access to data from multiple servers. Applications can readily access files usingstandard file system interfaces, and the same file can be accessed concurrently from multiple servers andprotocols. IBM Spectrum Scale is designed to provide high availability through advanced clusteringtechnologies, dynamic file system management, and data replication. IBM Spectrum Scale can continue toprovide data access even when the cluster experiences storage or server malfunctions. IBM SpectrumScale scalability and performance are designed for data intensive applications such as cloud storageengineering design, digital media, data mining, relational databases, financial analytics, seismic dataprocessing, scientific research and scalable file serving.

IBM Spectrum Scale is supported on AIX®, Linux and Windows Server operating systems. It is supportedon IBM® POWER®, Intel or AMD Opteron based servers, and IBM Z. For more information on thecapabilities of IBM Spectrum Scale and its applicability to your environment, see the IBM Spectrum Scale:Concepts, Planning, and Installation Guide.

Announcement

The immutability function in IBM Spectrum Scale Version 4.2 has been assessed for compliance inaccordance to the US SEC17a-4f rules, German and Swiss laws and regulations by a recognized auditor.The detailed assessment report can be downloaded here: http://www.kpmg.de/bescheinigungen/RequestReport.aspx?41742. The software certificate can be downloaded here: https://www.kpmg.de/bescheinigungen/RequestReport.aspx?41743.

Important References

v “Functional Support Matrices” on page 11v “Supported Upgrade Paths” on page 42v Software Version Recommendation Preventive Service Planning

IBM Spectrum Scale FAQ

These IBM Spectrum Scale Frequently Asked Questions and Answers provides you the most up-to-dateinformation on topics including ordering IBM Spectrum Scale, supported platforms and supportedconfiguration sizes and capacities. This FAQ is maintained on a regular basis and must be referencedbefore any system upgrades or major configuration changes to your IBM Spectrum Scale cluster. Wewelcome your feedback, if you have any comments, suggestions or questions regarding the informationprovided here send email to [email protected].

Updates to this FAQ include:

Table 1. FAQ June 2018 updates

June 2018

2.2 Which Linux distributions are supported by IBM Spectrum Scale?

2.22 What are the current limitations common to both the Active File Management (AFM) and AFMDR functions?

http://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html


http://www.kpmg.de/bescheinigungen/RequestReport.aspx?41742

http://www.kpmg.de/bescheinigungen/RequestReport.aspx?41742

https://www.kpmg.de/bescheinigungen/RequestReport.aspx?41743

https://www.kpmg.de/bescheinigungen/RequestReport.aspx?41743

http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009703

mailto: [email protected]

Table 1. FAQ June 2018 updates (continued)

June 2018

14.4 What are the current advisories for all platforms supported by IBM Spectrum Scale?

14.5 What are the current advisories for IBM Spectrum Scale on AIX?

14.6 What are the current advisories for IBM Spectrum Scale on Linux?

Table 2. FAQ May 2018 updates

May 2018


2.22 What are the current limitations common to both the Active File Management (AFM) and AFMDR functions?

10.1 What considerations are there when using OpenStack Software with IBM Spectrum Scale?

10.2 What are the new features of IBM Spectrum Scale for Object Storage?

10.4 What is the level of compatibility between S3 API in IBM Spectrum Scale and Amazon S3?



Table 3. FAQ April 2018 updates

April 2018


Table 4. FAQ March 2018 updates

March2018

2.3 What is the IBM Spectrum Scale support position regarding clone Linux distributions (CentOS,ROCK, White box Linux, etc.) ?

13.1 How are IBM Spectrum Scale V4 and V5 licensed?

13.4 How are IBM Spectrum Scale V4 and V5 priced?



Table 5. FAQ February 2018 updates

February2018


5.13 What are the current limits on the number of filesets in an IBM Spectrum Scale file system?

5.14 What are the current limits on the number of snapshots in an IBM Spectrum Scale file system?


Table 6. FAQ January 2018 updates

January2018

2.36 What are the current limitations of quota management?

6.16 What considerations are there when using IBM Spectrum Archive with IBM Spectrum Scale?

2

Table 7. FAQ December 2017 updates

December2017

2.1 Which levels of the AIX OS are supported by IBM Spectrum Scale?


2.7 What levels of the Windows OS are supported by IBM Spectrum Scale?

2.15 What are the requirements/limitations for using native encryption in IBM Spectrum Scale V4Advanced Edition or Data Management Edition?

2.28 What are the current requirements/limitations for IBM Spectrum Scale for Linux on Z?

2.31 What are the requirements/limitations for using the IBM Spectrum Scale GUI?

3.1 What are the minimum hardware requirements for a IBM Spectrum Scale cluster?

9.3 What Hadoop distributions are supported by IBM Spectrum Scale?

9.4 Can IBM Spectrum Scale support any Hadoop distribution vendor?

10.1 What considerations are there when using OpenStack Software with IBM Spectrum Scale?

10.2 What are the new features of IBM Spectrum Scale for Object Storage?

10.10 What are the limitations for IBM Spectrum Scale Object Storage with Unified File Access enabled?


16.4 What are the requirements and limitations for file audit logging?

Table 8. FAQ November 2017 updates

November2017

2.1 Which levels of the AIX OS are supported by IBM Spectrum Scale?

2.2 What Linux distributions are supported by IBM Spectrum Scale?

2.15 What are the requirements/limitations for using native encryption in IBM Spectrum Scale V4Advanced Edition or Data Management Edition?

2.28 What are the current requirements/limitations for IBM Spectrum Scale for Linux on Z?

5.12 What is the limit on the maximum number of groups a user can be a member of when accessing aGPFS file system?

16.1 Which IBM Spectrum Scale editions is file audit logging available in?

16.2 Which platforms is file audit logging supported on?

16.3 How many nodes are needed to enable file audit logging?

Table 9. FAQ October 2017 updates

October2017


4.1 What disk hardware has IBM Spectrum Scale been tested with?

4.5 What devices does IBM Spectrum Scale support with SCSI-3 Persistent Reservations?




3

Table 10. FAQ September 2017 updates

September2017



Table 11. FAQ August 2017 updates

August2017


4.1 What disk hardware has IBM Spectrum Scale been tested with?

4.12 What are the considerations for using thinly provisioned or compressed volumes with GPFS?




Table 12. FAQ July 2017 updates

July 2017


7.8 Does IBM Spectrum Scale support PowerVM Shared Storage Pool (SSP)?

8.3 What are some configuration considerations when deploying the protocol functionality?

13.14 Can I trade-up my IBM Spectrum Scale V4 licenses between Editions?



4

Questions & Answers

1. General questions:1.1 How do I order IBM Spectrum Scale?1.2 Where can I find ordering information for IBM Spectrum Scale?1.3 Where can I find the documentation for IBM Spectrum Scale?1.4 What is there beyond the standard documentation that can help me learn more about and use IBMSpectrum Scale?1.5 How can I ask a more specific question about IBM Spectrum Scale?1.6 Does IBM Spectrum Scale participate in the IBM Academic Initiative Program?1.7 Is IBM Spectrum Scale available in IBM PartnerWorld®?1.8 Does IBM Spectrum Scale have a trial program?1.9 Where can I find the documentation for IBM Spectrum Scale RAID?

2. Software questions:2.1 Which levels of the AIX OS are supported by IBM Spectrum Scale?2.2 Which Linux distributions are supported by IBM Spectrum Scale?2.3 What is the IBM Spectrum Scale support position regarding clone Linux distributions (CentOS,ROCK, White box Linux, etc.) ?2.4 What are the current restrictions on IBM Spectrum Scale Linux kernel support?2.5 What Linux distributions are supported by the integrated protocols access methods in IBMSpectrum Scale V4.1.1 and later?2.6 What is the impact on the /dev, /proc/mounts, /etc/mtab directories and the mount commandfor IBM Spectrum Scale for Linux due to the recent changes in systemd? What happened to the blockdevice in /dev? Why is the /dev/ prefix missing from the output of the mount command and alsofrom /proc/mounts and /etc/mtab?2.7 What levels of the Windows OS are supported by IBM Spectrum Scale?2.8 Does IBM Spectrum Scale V4.1 still require the Windows SUA feature?2.9 What are the limitations of IBM Spectrum Scale support for Windows?2.10 What are the requirements for the use of OpenSSH on Windows nodes?2.11 Can different IBM Spectrum Scale maintenance levels coexist?2.12 Are there any requirements for Clustered NFS (CNFS) support in IBM Spectrum Scale?2.13 Does IBM Spectrum Scale support NFS V4?2.14 Are there any requirements for Persistent Reserve support in IBM Spectrum Scale ?2.15 What are the requirements/limitations for using native encryption in IBM Spectrum Scale V4Advanced Edition or Data Management Edition?2.16 Are there any considerations when utilizing the Simple Network Management Protocol(SNMP)-based monitoring capability in IBM Spectrum Scale?2.17 What are the current limitations and advisories for using the mmbackup command?2.18 What are the current limitations and advisories for using Scale Out Backup and Restore(SOBAR)?2.19 What are the current limitations for using the File Placement Optimizer (FPO) function?2.20 What are the current limitations for using the Active File Management (AFM) Async DRfunction?2.21 What are the current limitations for using the Active File Management (AFM) function?

5

2.22 What are the current limitations common to both the Active File Management (AFM) and AFMDR functions?2.23 What is the currently recommended transport protocol for AFM and AFM DR data transfers?2.24 What is the current limitation for using the snapshot restore function?2.25 What are the current requirements when using local read-only cache?2.26 What are the current requirements/limitations for using the Cluster Configuration Repository(CCR) ?2.27 What are the current requirements/limitations for using Ubuntu?2.28 What are the current requirements/limitations for IBM Spectrum Scale for Linux on Z?2.29 What is the Highly-Available Write Cache (HAWC) function?2.30 What are the current requirements/limitations for the deadlock amelioration function in IBMSpectrum Scale?2.31 What are the requirements/limitations for using the IBM Spectrum Scale GUI?2.32 What are the requirements/limitations for using the compression function?2.33 What are the requirements/limitations for using the Quality of Service (QoS) function?2.34 What considerations are there when running on SELinux?2.35 What enhancements to quota management are available?2.36 What are the current limitations of quota management?2.37 What are the limitations of the Snapshot ILM feature?

3. Machine questions:3.1 What are the minimum hardware requirements for a IBM Spectrum Scale cluster?3.2 On what servers is IBM Spectrum Scale supported?3.3 What interconnects are supported for IBM Spectrum Scale daemon-to-daemon communication inmy cluster?

4. Disk questions:4.1 What disk hardware has IBM Spectrum Scale been tested with?4.2 What Fibre Channel (FC) Switches are qualified for IBM Spectrum Scale usage and is there a FCSwitch support chart available?4.3 Can I concurrently access disks from both AIX and Linux nodes in my IBM Spectrum Scalecluster?4.4 What disk support failover models does IBM Spectrum Scale support for the IBM TotalStorageDS4000® family of storage controllers with the Linux operating system?4.5 What devices does IBM Spectrum Scale support with SCSI-3 Persistent Reservations?4.6 What considerations are there when setting up DM-MP multipath service?4.7 Are there any steps that need to be taken before disks are used by IBM Spectrum Scale on AIX?4.8 Does IBM Spectrum Scale support Logical Volumes (LVs) ?4.9 Does IBM Spectrum Scale support AIX raw hdisks (rhdisk) ?4.10 Does IBM Spectrum Scale for Linux on Z support Direct Attached Storage Devices (DASD)?4.11 Does IBM Spectrum Scale support 4K disk sectors?4.12 What are the considerations for using thinly provisioned or compressed volumes with GPFS?4.13 Does IBM Spectrum Scale for Linux on Z work with disk hardware replication?4.14 Does IBM Spectrum Scale for Linux on Z support HyperSwap or SVC Stretch Clusters?

5. Scaling questions:5.1 What are the GPFS cluster size limits?

6

5.2 What are some scaling considerations for the protocols function?5.3 What is the current maximum tested limit for SMP scaling?5.4 What is the current limit on the number of nodes that may concurrently join a cluster?5.5 What is the limit of remote clusters that a client node can join?5.6 What is the limit of remote clusters that can join a local cluster?5.7 What are the current file system size limits?5.8 What is the current limit on the number of mounted file systems in a GPFS cluster?5.9 What is the architectural limit of the number of files in a file system?5.10 What is the architectural limit of the number of disks in a file system?5.11 What are the limitations on GPFS disk size?5.12 What is the limit on the maximum number of groups a user can be a member of when accessinga GPFS file system?5.13 What are the current limits on the number of filesets in an GPFS file system?5.14 What are the current limits on the number of snapshots in an GPFS file system?5.15 What is the current limit for the number of data and metadata replicas?5.16 What is the limitation on pathname length?

6. Configuration and tuning questions:6.1 What specific configuration and performance tuning suggestions are there?6.2 What configuration and performance tuning suggestions are there for GPFS when used primarilyfor Oracle databases?6.3 What configuration requirements exist for utilizing Remote Direct Memory Access (RDMA)?6.4 What configuration and performance tuning suggestions are there for the Active File Managementfunction of GPFS?6.5 Sometimes GPFS appears to be handling a heavy I/O load, for no apparent reason. What could becausing this?6.6 What considerations are there when using IBM Spectrum Protect with IBM Spectrum Scale?6.7 Does IBM Spectrum Scale V4.1 use OpenSSL ?6.8 How do I get OpenSSL to work with GPFS V3?6.9 What ciphers are supported for use by IBM Spectrum Scale V4?6.10 When I allow other clusters to mount my file systems, is there a way to restrict accesspermissions for the root user?6.11 How do I determine the maximum size of the extended attributes allowed in my file system?6.12 What are configuration considerations when using IPv6?6.13 How should IBM Spectrum Scale V4 Advanced Edition or Data Management Edition beconfigured to only use FIPS 140-2-certified cryptographic engines?6.14 What are the configuration requirements for the use of different IBM Spectrum Scale Editions?6.15 What are the configuration requirements for the use of the integrated protocols access methods?6.16 What considerations are there when using IBM Spectrum Archive with IBM Spectrum Scale?

7. Virtualization questions:7.1 How do I determine the number of licenses required when running IBM Spectrum Scale in VMs ina virtualization environment?7.2 How do I determine whether a server license or a client license is required when running IBMSpectrum Scale in VMs in a virtualized environment?7.3 Is IBM Spectrum Scale on Linux (x86 and Power®) supported in a virtualization environment?7.4 Is IBM Spectrum Scale for Linux on Z supported in a virtualization environment?

7

7.5 Is IBM Spectrum Scale on Windows supported in a virtualization environment?7.6 Can IBM Spectrum Scale run in a Workload Partitioning (WPAR) environment?7.7 Does IBM Spectrum Scale support exploitation of the Virtual I/O Server (VIOS) features ofPOWER processors?7.8 Does IBM Spectrum Scale support PowerVM Shared Storage Pool (SSP)?7.9 Are there any virtualization considerations when using an Oracle databases?7.10 Can Linux on Z with IBM Spectrum Scale run in logical partition (LPAR) mode or on z/VM® asguest operating system?

8. Integrated Protocol Server questions8.1 What are the requirements to use the protocol access methods integrated with IBM Spectrum ScaleV4.1.1 and later?8.2 What is the minimum hardware requirement for a protocol node?8.3 What are some configuration considerations when deploying the protocol functionality?8.4 What is the guidance for NFS clients to be configured with the CES NFS function?8.5 Can we configure protocol nodes with an ESS/GSS?8.6 Are there any known limitations and what are considerations should I be aware before using theintegrated CES Protocol function?8.7 Where can I find additional information about protocols?8.8 How do I reduce the logging level for performance monitoring of Swift?8.9 What are the limitations when multiple file systems are exported by NFS?8.10 How to effectively use VMWare with IBM Spectrum Scale?8.11 What are the limitations when upgrading the SMB service?

9. Hadoop Support questions:9.1 What platforms are supported for Hadoop on IBM Spectrum Scale?9.2 What versions of Hadoop are supported by IBM Spectrum scale version?9.3 What Hadoop distributions are supported by IBM Spectrum Scale?9.4 Can IBM Spectrum Scale support any Hadoop distribution vendor?9.5 What IBM Spectrum Scale licenses do I need to use the IBM Spectrum Scale Hadoop connector?9.6 Can Hadoop connector be used for IBM Spectrum Scale that has shared storage (including theElastic Storage Server)?9.7 Can Hadoop connector be used for IBM Spectrum Scale that has shared storage (including ESS)and internal storage (FPO pool) in the same file system?9.8 What open-source Hadoop components are certified using IBM Spectrum Scale connector?9.9 Why do you need special IBM Spectrum Scale connector instead of using Hadoop local file system( file:///) if not using FPO/internal storage?9.10 What are the differences between the Hadoop connector and the HDFS Transparency connector ?9.11 What are the requirements/limitations for using the IBM Spectrum Scale HDFS Transparencyconnector?9.12 What are the current advisories for the IBM Spectrum Scale Hadoop connector?

10. Object Protocol Support questions:10.1 What considerations are there when using OpenStack Software with IBM Spectrum Scale?10.2 What are the new features of IBM Spectrum Scale for Object Storage?10.3 How should I ensure that unauthorized users cannot access my object data when using SpectrumScale for object storage?10.4 What is the level of compatibility between S3 API in IBM Spectrum Scale and Amazon S3?

8

10.5 How can I ensure secure data in flight between object client and Spectrum Scale object storage?10.6 What authentication schemes does IBM Spectrum Scale Object Storage support?10.7 When using Unified File and Object Access feature, when should I use the local_mode vsunified_mode for identity management?10.8 Can I have SMB/NFS export over object data when using Unified File and Object Access feature?10.9 Can I have read/write access from file interface as well as object interface on the same data,when using Unified File and Object Access feature?10.10 What are the limitations for IBM Spectrum Scale Object Storage with Unified File Accessenabled?10.11 What are the considerations when using HAProxy load balancer with Spectrum Scale ObjectStorage?10.12 Are there known issues with OpenStack software?10.13 Are there any additional limitations or restrictions when using Spectrum Scale compression orencryption with Object protocol?10.14 What are the limitations when configuring objects to use CES Groups?

11. Installation Toolkit questions:11.1 Can I upgrade an IBM Spectrum Scale cluster with protocols directly from 4.1.1.x to 4.2.0.x or4.2.1.x?11.2 What are the limitations when I use the Installation Toolkit to install Spectrum Scale 4.2.0.0 orupgrade from Spectrum Scale 4.1.1.x to 4.2.0.0 or 4.2.1.0 with Object Protocol?11.3 Can GUI nodes be added during an upgrade to IBM Spectrum Scale V4.2.0.1?11.4 What are the limitations when I use the to upgrade Spectrum Scale from 4.1.1 to 4.2 with NFS orSMB Protocol?11.5 How can I determine if the Installation Toolkit successfully upgrades from IBM Spectrum ScaleV4.1.1 to V4.2?11.6 Can I have EPEL repos enabled when using the spectrumscale installation toolkit for install orupgrade?11.7 What are the functions that are not supported by the installation toolkit?11.8 What are the potential limitations when using the Installation Toolkit with SLES12 SP1 or SP2nodes?

12. Transparent cloud tiering questions:12.1 What platforms is transparent cloud tiering supported on?12.2 How many transparent cloud tiering bridge nodes are supported?12.3 What are the networking requirements for transparent cloud tiering?12.4 What Cloud Object Storage Providers are supported?12.5 Can transparent cloud tiering be used with other IBM Spectrum Scale functions?12.6 Is transparent cloud tiering supported with ESS?12.7 Are there restrictions on viewing or listing files?

13. Licensing and Pricing questions:13.1 How are IBM Spectrum Scale V4 and V5 licensed?13.2 How can I tell which Edition of IBM Spectrum Scale I am running?13.3 How is GPFS V3.5 licensed?13.4 How are IBM Spectrum Scale V4 and V5 priced?13.5 How is GPFS V3.5 priced?13.6 Can I migrate my GPFS V3.5 licenses to V4?

9

13.7 Is there an additional cost to upgrade or migrate from V3 to V4?13.8 I am in the middle of my SWMA or S&S contract, my anniversary date for renewal is 4 monthsaway, will I have to pay again?13.9 My licenses were purchased with a Power server and my entitlements and SWMA are managedby CHIS contracts. I have purchased SWMA for 3 years and have considerable time remaining. Howwill my SWMA be managed going forward? Will my existing SWMA change?13.10 For my Power servers, will my machine type model (MTM) and serial numbers (SN) bemigrated with my entitlements?13.11 I am not familiar with Passport Advantage, is there anything special I will need to do?13.12 I am interested in the new IBM Spectrum Scale Advanced Edition, can I upgrade to AdvancedEdition from V3?13.13 My S&S or SWMA has expired and now I want to move to V4, can I migrate using theAfter-License option?13.14 Can I trade-up my IBM Spectrum Scale V4 licenses between Editions?13.15 Can I transfer my IBM Spectrum Scale licenses between machines?13.16 Do I require additional IBM Spectrum Scale licenses for my Disaster Recovery (DR)environment?

14. Service questions:14.1 What support services are available for IBM Spectrum Scale?14.2 How do I download fixes for IBM Spectrum Scale?14.3 How do I migrate from GPFS V4.1 or IBM Spectrum Scale V4.1.1 to IBM Spectrum Scale V4.2?14.4 What are the current advisories for all platforms supported by IBM Spectrum Scale?14.5 What are the current advisories for IBM Spectrum Scale on AIX?14.6 What are the current advisories for IBM Spectrum Scale on Linux?14.7 What are the current advisories for IBM Spectrum Scale on Windows?14.8 What Linux kernel patches are provided for clustered file systems such as GPFS?14.9 Where can I find the IBM Spectrum Scale Software License Agreement?14.10 Where can I find End of Market (EOM) and End of Service (EOS) information?14.11 Where can I locate IBM Spectrum Scale code to upgrade from my current level ?14.12 Are there any items that will no longer be supported?

15. Scale Management Server (REST API) questions:15.1 Does IBM Spectrum Scale support RESTful APIS for file system management?15.2 What are the changes made to the API implementation in the 4.2.3 release?

16. File audit logging questions:16.1 Which IBM Spectrum Scale editions is file audit logging available in?16.2 Which platforms is file audit logging supported on?16.3 How many nodes are needed to enable file audit logging?16.4 What are the requirements and limitations for file audit logging?

10

Functional Support MatricesTable 13. IBM Spectrum Scale V5.0.1 Functional Support Matrix (Advanced or Data Management Edition)

Advanced or Data Management Edition

Linuxx86andLoP

LinuxonZ

AIX

Encryptionof Dataat Rest

x

x4.2.1.1 or

lateronly

x

AsynchDisasterRecoverywithAFM

xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04

(x86_64 and Power LE)

xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04

x

GUI xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04

xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04

IntegratedProtocolsSupportCES (SMB,NFS,OBJ, iSCSI)

xRHEL 7.1 or later

x86_64Power BE,Power LE

SMB/NFSonlywith

SLES 12SP1SP2

x86_64

Ubuntu16.04

11

||

|

|

||||

|||

|

||||

||||

|

|||||

|||||||||||

||||||||||

|

|||||||||||

||||||||||

|

||||||

||||||||||||||||

||

Table 13. IBM Spectrum Scale V5.0.1 Functional Support Matrix (Advanced or Data ManagementEdition) (continued)


Linuxx86andLoP

LinuxonZ

AIX

InstallationToolkit(allows theinstallationof NFSservices,CESframework,SMB andObject)

xRHEL 7.1 or later

Power BE,Power LE

x86_64SLES 12

SLES 12 SP2SLES 12 SP3

x86_64Ubuntu 16.04

Ubuntu 16.04.3x86_64

Call Home xRHEL 7.1 or later


SLES 12 xx86_64

Ubuntu16.04

xRHEL 7.1 or later

SLES12.x

Ubuntu 16.04

PerformanceMonitoringTool

xRHEL 7.1 or later

SLES 12Debian7/8.1

Ubuntu 16.04x86_64

Power LERHEL 7SLES12

Power BERHEL 7.1 or later

SLES 11

xRHEL 7.1 or later

SLES 12SP1SP2

Ubuntu16.04

HealthMonitoring

x x x

ActiveFileMgt(AFM)

xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04 (x86_64 and Power LE)

xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04

x

12

||

|

|

||||

|||

|

||||||||||

||||||||||||

||

|||||||||

||

||||||

|

|||

|||||||||||||||

|||||||||

|

|||||

||||

||||||||||

||||||||||

|



Linuxx86andLoP

LinuxonZ

AIX

QualityofService(QoS)

x x x

IntegratedLifecycleMgt(ILM)policysupport

x x x

Backupof a

filesystem

x x x

FilePlacementOptimizer

x x

Hadoopincludingsharedstorage

x

ClusteredNFS

x

Filesets x x x

File clones x x x

Quotas x x x

REST API xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04

xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04

TCT Client:RHEL 7.1 or later

SLES 12Debian 7/8

x86_64RHEL 7.2 on Power8 LE

Server:RHEL 7.1 or later


13

||

|

|

||||

|||

|

||||

|||

||||||

|||

||||

|||

||||||

||||

|||

|||||

||||

||||

||||

|||||||||||

||||||||||

|

||||||||||||

||



Linuxx86andLoP

LinuxonZ

AIX

File audit logging RHEL 7.1 or laterx86_64

Power LE

Ubuntu 16.04x86_64

Table 14. IBM Spectrum Scale V5.0.1 Functional Support Matrix (Standard Edition)

Standard Edition

Linuxx86andLoP

LinuxonZ

AIX Win




orSLES 12

SP1SP2SP3

Ubuntu16.04

xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04


xRHEL 7.1 or later


SMB/NFSonlywith

SLES 12SP1SP2

x86_64

Ubuntu16.04

14

||

|

|

||||

|||

|

|||||||

||

|

||

|

|

||||

|||

||

|||||||

|||||

||||

|||||||||||

||||||||||

||

||||||

||||||||||||||||

|||

Table 14. IBM Spectrum Scale V5.0.1 Functional Support Matrix (Standard Edition) (continued)

Standard Edition

Linuxx86andLoP

LinuxonZ

AIX Win


xRHEL 7.1 or later

Power BE,Power LE

x86_64SLES 12


x86_64Ubuntu 16.04

Ubuntu 16.04.3x86_64



SLES12 xx86_64

Ubuntu 16.04

xRHEL 7.1 or later

SLES12.x

Ubuntu 16.04


xRHEL 7.1 or later

SLES 12Debian7/8.1

Ubuntu 16.04x86_64



xRHEL 7.1 or later

SLES 12SP1SP2

Ubuntu16.04

HealthMonitoring

x x x

ActiveFileMgt(AFM)

xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04


xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04

x

15

|

|

|

||||

|||

||

||||||||||

||||||||||||

|||

||||||||||

||||||

||

|||

||||||||||||||

||||||||

||

||||||

||||

|||||||||||

||||||||||

||


Standard Edition

Linuxx86andLoP

LinuxonZ

AIX Win


x x x x


x x x x

Backupof a

filesystem

x x x


x x


x

ClusteredNFS

x

Filesets x x x x

File clones x x x

Quotas x x x


orSLES 12

SP1SP2SP3

Ubuntu16.04

xRHEL 7.1 or later

orSLES 12

SP1SP2SP3

Ubuntu16.04

TCT

16

|

|

|

||||

|||

||

||||

||||

||||||

||||

||||

||||

|||||||

||||

||||

||||||

|||||

|||||

|||||

|||||||||||

||||||||||

||

||||||

Table 15. IBM Spectrum Scale V5.0.0 Functional Support Matrix (Advanced or Data Management Edition)


Linuxx86andLoP

LinuxonZ

AIX


x

x4.2.1.1 or

lateronly

x


xRHEL 7.1 or later1

orSLES 12

SP1SP2SP31

Ubuntu16.04


xRHEL 7.1 or later1

orSLES 12

SP1SP2SP31

Ubuntu16.04

x


orSLES 12

SP1SP2

Ubuntu16.04

xRHEL 7.1 or later

orSLES 12

SP1SP2

Ubuntu16.04


xRHEL 7.1 or later2


SMB/NFSonlywith

SLES 12SP1SP2

x86_64

Ubuntu16.04

17



Linuxx86andLoP

LinuxonZ

AIX


xRHEL 7.1 or later3

Power BE,Power LE

x86_64SLES 12


x86_64Ubuntu 16.04

Ubuntu 16.04.33

x86_64



SLES 12 xx86_64

Ubuntu16.04

xRHEL 7.1 or later

SLES12.x

Ubuntu 16.04


xRHEL 7.1 or later

SLES 12Debian7/8.1

Ubuntu 16.04x86_64



SLES 11

xRHEL 7.1 or later

SLES 12SP1SP2

Ubuntu16.04

HealthMonitoring

x x x

ActiveFileMgt(AFM)

xRHEL 7.1 or later1

orSLES 12

SP1SP2SP31

Ubuntu16.04 (x86_64 and Power LE)

xRHEL 7.1 or later1

orSLES 12

SP1SP2SP31

Ubuntu16.04

x

18



Linuxx86andLoP

LinuxonZ

AIX


x x x


x x x

Backupof a

filesystem

x x x


x x


x

ClusteredNFS

x

Filesets x x x

File clones x x x

Quotas x x x


orSLES 12

SP1SP2

Ubuntu16.04

xRHEL 7.1 or later

orSLES 12

SP1SP2

Ubuntu16.04

TCT Client:RHEL 7.1 or later

SLES 12Debian 7/8


Server:RHEL 7.1 or later


19



Linuxx86andLoP

LinuxonZ

AIX

File audit logging RHEL 7.1 or laterx86_64

Power LE

Ubuntu 16.04x86_64

Note:

1. AFM and Asynch Disaster Recovery with AFM are supported on RHEL 7.4 and SLES 12 SP3 in IBMSpectrum Scale V5.0.0.1.

2. Integrated Protocols is supported on RHEL 7.4 in IBM Spectrum Scale V5.0.0.1.3. The Installation Toolkit is supported on RHEL 7.4, SLES 12 SP3, and Ubuntu 16.04.3 in IBM Spectrum

Scale V5.0.0.1.


Standard Edition

Linuxx86andLoP

LinuxonZ

AIX Win




orSLES 12

SP1SP2

Ubuntu16.04

xRHEL 7.1 or later

orSLES 12

SP1SP2

Ubuntu16.04

20


Standard Edition

Linuxx86andLoP

LinuxonZ

AIX Win


xRHEL 7.1 or later1


SMB/NFSonlywith

SLES 12SP1SP2

x86_64

Ubuntu16.04


xRHEL 7.1 or later2

Power BE,Power LE

x86_64SLES 12


x86_64Ubuntu 16.04

Ubuntu 16.04.32

x86_64



SLES12 xx86_64

Ubuntu 16.04

xRHEL 7.1 or later

SLES12.x

Ubuntu 16.04


xRHEL 7.1 or later

SLES 12Debian7/8.1

Ubuntu 16.04x86_64



xRHEL 7.1 or later

SLES 12SP1SP2

Ubuntu16.04

21


Standard Edition

Linuxx86andLoP

LinuxonZ

AIX Win

HealthMonitoring

x x x

ActiveFileMgt(AFM)

xRHEL 7.1 or later3

orSLES 12

SP1SP2SP33

Ubuntu16.04


xRHEL 7.1 or later3

orSLES 12

SP1SP2SP33

Ubuntu16.04

x


x x x x


x x x x

Backupof a

filesystem

x x x


x x


x

ClusteredNFS

x

Filesets x x x x

File clones x x x

Quotas x x x


orSLES 12

SP1SP2

Ubuntu16.04

xRHEL 7.1 or later

orSLES 12

SP1SP2

Ubuntu16.04

22


Standard Edition

Linuxx86andLoP

LinuxonZ

AIX Win

TCT

Note:

1. Integrated Protocols is supported on RHEL 7.4. in IBM Spectrum Scale V5.0.0.1.2. The Installation Toolkit is supported on RHEL 7.4, SLES 12 SP3, and Ubuntu 16.04.3 in IBM Spectrum

Scale V5.0.0.1.3. AFM is supported on RHEL 7.4 and SLES 12 SP3 in IBM Spectrum Scale V5.0.0.1.



Linuxx86andLoP

LinuxonZ

AIX


x

x4.2.1.1 or

lateronly

x


x x x

GUI xRHEL 7.x

orSLES 12

SP1SP2

Ubuntu2

16.04

xRHEL 7.x

orSLES 12

SP1SP2

Ubuntu2

16.04


xRHEL 7.

xx86_64

Power BE,Power LE

SMB/NFSonlywith

SLES 12SP1SP2

x86_64

23



Linuxx86andLoP

LinuxonZ

AIX


xRHEL 7.

xPower BE,Power LERHEL 6.8

x86_64Power BESLES 12x86_64

Call Home xRHEL 7.

xx86_64

Power BE,Power LE

SLES 12 xx86_64


xRHEL 6.x

7.xSLES 11/12

Debian7/8.1

Ubuntu14.04/16.04

x86_64


Power BERHEL 6, 7

SLES 11

xRHEL 6.5

6.66.77.x

SLES 11SP3SP4

SLES 12SP1

Ubuntu2

16.04

HealthMonitoring

x x x

ActiveFileMgt(AFM)

x x x


x x x

24



Linuxx86andLoP

LinuxonZ

AIX


x x x

Backupof a

filesystem

x x x


x x


x

ClusteredNFS

x

Filesets x x x

File clones x x x

Quotas x x x

REST API1 xRHEL 7.x

orSLES 12

SP1SP2

Ubuntu2

16.04

xRHEL 7.x

orSLES 12

SP1SP2

Ubuntu2

16.04

TCT Client:RHEL 6/7SLES 11/12Debian 7/8

Ubuntu 14.04x86_64

RHEL 7.2 on Power8 LE3

Server:RHEL 7x86_64

RHEL 7.2 on Power8 LE3

Note:

1. The REST API that is delivered with IBM Spectrum Scale V4.2.3 is based on the GUI stack, so itsupports the same architectures and operating systems as the GUI.

25

2. Ubuntu 16.04.1 support is delivered with IBM Spectrum Scale V4.2.3.1.3. TCT support on Power8 LE is delivered with IBM Spectrum Scale V4.2.3.1.


Standard Edition

Linuxx86andLoP

LinuxonZ

AIX Win



GUI xRHEL 7.x

orSLES 12

SP1SP2

Ubuntu2

16.04

xRHEL 7.x

orSLES 12

SP1SP2

Ubuntu2

16.04


xRHEL 7.

xx86_64

Power BE,Power LE

SMB/NFSonlywith

SLES 12SP1SP2

x86_64


xRHEL 7.


x86_64Power BESLES 12/x86_64

26


Standard Edition

Linuxx86andLoP

LinuxonZ

AIX Win

Call Home xRHEL 7.

xx86_64

Power BE,Power LE

SLES12 xx86_64


xRHEL 6.x

7.xSLES 11/12

Debian7/8.1

Ubuntu14.04/16.04

x86_64


Power BERHEL 6, 7

SLES 11

xRHEL 6.5

6.66.77.x

SLES 11SP3SP4

SLES 12SP1

Ubuntu2

16.04

HealthMonitoring

x x x

ActiveFileMgt(AFM)

x x x


x x x x


x x x x

Backupof a

filesystem

x x x


x x

27


Standard Edition

Linuxx86andLoP

LinuxonZ

AIX Win


x

ClusteredNFS

x

Filesets x x x x

File clones x x x

Quotas x x x

REST API1 xRHEL 7.x

orSLES 12

SP1SP2

Ubuntu2

16.04

xRHEL 7.x

orSLES 12

SP1SP2

Ubuntu2

16.04

TCT

Note:

1. The REST API that is delivered with IBM Spectrum Scale V4.2.3 is based on the GUI stack, so itsupports the same architectures and operating systems as the GUI.

2. Ubuntu 16.04.1 support is delivered with IBM Spectrum Scale V4.2.3.1.



Linuxx86andLoP

LinuxonZ

AIX


x

x4.2.1.1 or

lateronly

x


x3 x3 x3

GUI xRHEL 7.x

orSLES 12

SP0SP1

xRHEL 7.x

orSLES 12

SP0SP1

28



Linuxx86andLoP

LinuxonZ

AIX


xRHEL 7.

xx86_64

Power BE,Power LE

SMB/NFSonlywith

SLES 12x86_64


x4

RHEL 7.x

Power BE,Power LERHEL 6.8


Call Home xRHEL 7.

xx86_64

Power BE,Power LE

SLES12 xx86_64


xRHEL 6.x

7.xSLES 11/12

Debian7/8.1

Ubuntu14.04/16.04

x86_64


Power BERHEL 6, 7

SLES11

xRHEL 6.5

6.66.77.x

SLES 11SP3SP4

SLES 12SP0SP1

HealthMonitoring

x x x5

29



Linuxx86andLoP

LinuxonZ

AIX

ActiveFileMgt(AFM)

x6 x x


x x x


x x x

Backupof a

filesystem

x x x


x x


x

ClusteredNFS

x

Filesets x x x

File clones x x x

Quotas x x x

REST API RHEL 7.xx86_64

TCT Client:RHEL 6/7SLES 11/12Debian 7/8

Ubuntu 14.04x86_64

Server:RHEL 7x86_64

Note:

1. Functions that are supported by all editions on all platforms are not listed in the matrix.

30

2. Unless noted otherwise, the functions listed are supported on all platforms supported by IBMSpectrum Scale.

3. Our initial feedback from the field suggests that success of a disaster recovery solution depends onadministration discipline, including careful design, configuration, and testing. Considering this, IBMhas decided to disable the Active File Management- based Asynchronous Disaster Recovery feature(AFM DR) by default and require that customers deploying the AFM DR feature first review theirdeployments with IBM Spectrum Scale development. You should contact IBM Spectrum Scale Supportat [email protected] to have your use case reviewed. IBM will help optimize your tuning parametersand enable the feature. Please include this message while contacting IBM Support.

4. The Installation Toolkit will not install the extra packages from the advanced package. The advancedRPMs must be installed manually.

5. For Health monitoring on AIX, you need to install Python 2.7.5 or later from the AIX Toolbox forLinux Applications.

6. The Active File Management function is not supported on Debian.7. The mmapplypolicy command does not run on Windows.8. If you are running SLES 12 SP0, the gpfs.gss.pmsensors-*.sles12.1.s390x.rpm packages can be used

with the Systems Performance Monitoring Tool.9. The Data Management Edition is available in IBM Spectrum Scale V4.2.2 and later.

Table 20. IBM Spectrum Scale V4.2.2 Functional Support Matrix (Standard Edition and Express Edition)

Standard Edition Express Edition

Linuxx86andLoP

LinuxonZ

AIX Win Linuxx86andLoP

LinuxonZ

AIX Win



GUI xRHEL 7.x

orSLES 12

SP0SP1

xRHEL 7.x

orSLES 12

SP0SP1


xRHEL 7.

xx86_64

Power BE,Power LE

SMB/NFSonlywith

SLES 12x86_64

xRHEL 7.

xx86_64

Power BE,Power LE

SMB/NFSonlywith

SLES 12x86_64

31


http://www-03.ibm.com/systems/power/software/aix/linux/


Table 20. IBM Spectrum Scale V4.2.2 Functional Support Matrix (Standard Edition and Express Edition) (continued)


Linuxx86andLoP

LinuxonZ


LinuxonZ

AIX Win


xRHEL 7.



Call Home xRHEL 7.

xx86_64

Power BE,Power LE

SLES12 xx86_64

xRHEL 7.

xx86_64

Power BE,Power LE

SLES12 xx86_64


xRHEL 6.x

7.xSLES 11/12

Debian7/8.1

Ubuntu14.04/16.04

x86_64


Power BERHEL 6, 7

SLES11

xRHEL 6.5

6.66.77.x

SLES 11SP3SP4

SLES 12SP0SP1

xRHEL 6.x

7.xSLES 11/12

Debian7/8.1

Ubuntu14.04/16.04

x86_64


Power BERHEL 6, 7

SLES11

xRHEL 6.5

6.66.77.x

SLES 11SP3SP4

SLES 12SP0SP1

HealthMonitoring

x x x5 x x x5

ActiveFileMgt(AFM)

x6 x x


x x x x x x x x

32

Table 20. IBM Spectrum Scale V4.2.2 Functional Support Matrix (Standard Edition and Express Edition) (continued)


Linuxx86andLoP

LinuxonZ


LinuxonZ

AIX Win


x x x x7

Backupof a

filesystem

x x x


x x x x


x

ClusteredNFS

x

Filesets x x x x

File clones x x x x x x

Quotas x x x x x x

REST API RHEL 7.xx86_64

TCT

Note:

1. Functions that are supported by all editions on all platforms are not listed in the matrix.2. Unless noted otherwise, the functions listed are supported on all platforms supported by IBM

Spectrum Scale.3. Our initial feedback from the field suggests that success of a disaster recovery solution depends on

administration discipline, including careful design, configuration, and testing. Considering this, IBMhas decided to disable the Active File Management- based Asynchronous Disaster Recovery feature(AFM DR) by default and require that customers deploying the AFM DR feature first review theirdeployments with IBM Spectrum Scale development. You should contact IBM Spectrum Scale Supportat [email protected] to have your use case reviewed. IBM will help optimize your tuning parametersand enable the feature. Please include this message while contacting IBM Support.



6. The Active File Management function is not supported on Debian.7. The mmapplypolicy command does not run on Windows.

33




8. If you are running SLES 12 SP0, the gpfs.gss.pmsensors-*.sles12.1.s390x.rpm packages can be usedwith the Systems Performance Monitoring Tool.

Table 21. IBM Spectrum Scale V4.2.1 Functional Support Matrix

Advanced Edition Standard Edition Express Edition

Linuxx86andLoP

LinuxonZ

AIX Linuxx86andLoP

LinuxonZ


LinuxonZ

AIX Win


xx

4.2.1.1 orlater only

x


x3 x3 x3

GUI xRHEL 7.x

orSLES 12

SP0SP1

xRHEL 7.x

orSLES 12

SP0SP1

xRHEL 7.x

orSLES 12

SP0SP1

xRHEL 7.x

orSLES 12

SP0SP1


xRHEL 7.

xx86_64

Power BE

SMB/NFSonlywith

SLES 12x86_64

xRHEL 7.

xx86_64

Power BE

SMB/NFSonlywith

SLES 12x86_64


x4

RHEL 7.x

xRHEL 7.

x

Call Home xRHEL 7.

xx86_64

Power BE

SLES12 xx86_64

xRHEL 7.

xx86_64

Power BE

SLES12 xx86_64

xRHEL 7.

xx86_64

Power BE

SLES12 xx86_64

34

Table 21. IBM Spectrum Scale V4.2.1 Functional Support Matrix (continued)


Linuxx86andLoP

LinuxonZ

AIX Linuxx86andLoP

LinuxonZ


LinuxonZ

AIX Win


xRHEL 6.x

7.xSLES 11/12

Debian7/8.1

Ubuntu14.04

x86_64


Power BERHEL 6, 7

SLES11

xRHEL 6.5

6.66.77.x

SLES 11SP3SP4

SLES 12SP0SP1

xRHEL 6.x

7.xSLES 11/12

Debian7/8.1

Ubuntu14.04

x86_64


Power BERHEL 6,7

SLES11

xRHEL 6.5

6.66.77.x

SLES 11SP3SP4

SLES 12SP0SP1

xRHEL 6.x

7.xSLES 11/12

Debian7/8.1

Ubuntu14.04

x86_64


Power BERHEL 6, 7

SLES11

xRHEL 6.5

6.66.77.x

SLES 11SP3SP4

SLES 12SP0SP1

HealthMonitoring

x x x5 x x x5 x x x5

ActiveFileMgt(AFM)

x6 x x x6 x x


x x x x x x x x x x x


x x x x x x x7

Backupof a

filesystem

x x x x x x


x x x x x x


x x

ClusteredNFS

x x

Filesets x x x x x x x

File clones x x x x x x x x x

35

Table 21. IBM Spectrum Scale V4.2.1 Functional Support Matrix (continued)


Linuxx86andLoP

LinuxonZ

AIX Linuxx86andLoP

LinuxonZ


LinuxonZ

AIX Win

Quotas x x x x x x x x x

TCT Client:RHEL 6/7

SLES 11/12Debian 7/8

Ubuntu14.04x86_64

Server:RHEL 7x86_64

Note:

1. Functions that are supported by all editions on all platforms are not listed in the matrix.2. Unless noted otherwise, the functions listed are supported on all platforms supported by IBM

Spectrum Scale.3. Our initial feedback from the field suggests that success of a disaster recovery solution depends on




6. The Active File Management function is not supported on Debian.7. The mmapplypolicy command does not run on Windows.8. If you are running SLES 12 SP0, the gpfs.gss.pmsensors-*.sles12.1.s390x.rpm packages can be used

with the Systems Performance Monitoring Tool.

Table 22. IBM Spectrum Scale V4.2 Functional Support Matrix


Linuxx86andLoP

LinuxonZ

AIX Linuxx86andLoP

LinuxonZ


LinuxonZ

AIX Win


x x


x2 x2 x2

36




Table 22. IBM Spectrum Scale V4.2 Functional Support Matrix (continued)


Linuxx86andLoP

LinuxonZ

AIX Linuxx86andLoP

LinuxonZ


LinuxonZ

AIX Win

GUI xRHEL 7.x

as listedin the

RedHatsupport

tableor

SLES 12only

xRHEL 7.x

as listedin the

RedHatsupport

tableor

SLES 12only

IntegratedProtocolsSupport(CES, SMB,and NFS)

xRHEL 7

or 7.1x86_64

Power BE

xRHEL 7

or 7.1x86_64

Power BE


x3

RHEL 7or 7.1only

xRHEL 7

or 7.1only

Call Home xRHEL 7

or 7.1only

x86_64Power BE

xRHEL 7

or 7.1only

x86_64Power BE


xRHEL 6.x

7.xSLES 12Debian7/8.1

Ubuntu14.04

x86_64

RHEL 7.xPower BE

xRHEL 6.x


Ubuntu14.04

x86_64

RHEL 7.xPower BE

xRHEL 6.x


Ubuntu14.04

x86_64

RHEL 7.xPower BE

ActiveFileMgt(AFM)

x4 x x x4 x x

37

Table 22. IBM Spectrum Scale V4.2 Functional Support Matrix (continued)


Linuxx86andLoP

LinuxonZ

AIX Linuxx86andLoP

LinuxonZ


LinuxonZ

AIX Win


x x x x x x x x


x x x x x x x5

Backupof afilesystem

x x x x x x


x x x x x x


x x

ClusteredNFS

x x

Filesets x x x x x x x

File clones x x x x x x x x x

Quotas x x x x x x x x x

Note:

1. Functions that are supported by all editions on all platforms are not listed in the matrix.2. Our initial feedback from the field suggests that success of a disaster recovery solution depends on



4. The Active File Management function is not supported on Debian.5. The mmapplypolicy command does not run on Windows.

38


Table 23. IBM Spectrum Scale V4.1.1 Functional Support Matrix

AdvancedEdition


Linuxx86andLoP

AIX Linuxx86andLoP

LinuxonZ

AIX Windows Linuxx86andLoP

LinuxonZ

AIX Windows


x x

AsynchronousDisasterRecoverywith AFM

x2 x2

IntegratedProtocolsSupport(CES, SMB,and NFS)

xRHEL 7

or 7.1only

xRHEL 7

or 7.1only

Active FileManagement(AFM)

x x x x x

IntegratedLifecycleManagement(ILM)policy support

x x x x x x3

Backupof afilesystem

xV4.1.1.1or later

xV4.1.1.1or later

xV4.1.1.1or later

xV4.1.1.1or later

IntegratedInstaller(allows theinstallationof NFSservices,CESframework,SMB andObject)

x4

RHEL 7or 7.1only

xRHEL 7

or 7.1only


x x x x x x

Hadoopsharedstorage

x x

ClusteredNFS

x x

Filesets x x x x x x

File clones x x x x x x x x

Quotas x x x x x x x x

39

Note:

1. Functions that are supported by all editions on all platforms are not listed in the matrix.2. Our initial feedback from the field suggests that success of a disaster recovery solution depends on


3. The mmapplypolicy command does not run on Windows.4. The Integrated Installer will not install the extra packages from the advanced package. The advanced

RPMs must be installed manually.


AdvancedEdition


Linuxx86andLoP

AIX Linuxx86andLoP

LinuxonZ

AIX Windows Linuxx86andLoP

LinuxonZ

AIX Windows


x


x x x x


x x x x x x2

Backupof a

filesystem

seeV4.1.1

seeV4.1.1

seeV4.1.1

seeV4.1.1

FilePlacementOptimizerand Hadoop

x x x x x x

ClusteredNFS

x x

Filesets x x x x x x

File clones x x x x x x x x

Quotas x x x x

Note:

1. Functions that are supported by all editions on all platforms are not listed in the matrix.2. The mmapplypolicy command does not run on Windows.

40



Linux x86 and LoP AIX Windows


x x


x x x

Backupof a

filesystem

x x


x x

ClusteredNFS

x

Filesets x x x

File clones x x

Quotas x x

Note: Functions that are supported on all platforms are not listed in the matrix.

41

Supported Upgrade PathsTable 26. IBM Spectrum Scale supported online upgrade paths

Upgradingfrom: To: 3.5 To: 4.1.0.x To: 4.1.1.x To: 4.2.0.x To: 4.2.1.x To: 4.2.2.x To: 4.2.3.x To: 5.0.0.x To: 5.0.1.x

3.5 U U U X X X X X X

4.1.0.x -- U U U U U U X X

4.1.1.x -- -- U U U U U X X

4.2.0.x -- -- -- U U U U U U

4.2.1.x -- -- -- -- U U U U U

4.2.2.x -- -- -- -- -- U U U U

4.2.3.x -- -- -- -- -- -- U U U

5.0.0.x -- -- -- -- -- -- -- U U

5.0.1.x -- -- -- -- -- -- -- -- U

Remember:

v Online upgrade support for protocol nodes depends on which services have been enabled anddeployed. Different protocol services have different levels of online upgrade support. For moreinformation, see Online upgrade support for protocols and performance monitoring.

v Offline upgrade can be performed from all supported versions to all supported versions.v IBM Spectrum Scale installation toolkit can be used for upgrades from version 4.1.1.x onwards.

General QuestionsQ1.1: How do I order IBM Spectrum Scale?A1.1: IBM Spectrum Scale V4 is distributed by Edition (Express, Standard, or Advanced):

v For x86 and Power platforms, each Edition contains binaries for all the supported platforms:– To order through the AAS ordering system (find contact information for your country at

http://www.ibm.com/planetwide/ ), order 5641-GPF. Orders placed through AAS arefulfilled through PPA.- Orders can be placed by either the Power or Storage configurators.- Linux on Z is not available to order via AAS.

– To order through Passport Advantage® at http://www.ibm.com/software/lotus/passportadvantage/, order 5725-Q01

v For Linux on Z:– Ordered through Passport Advantage at http://www.ibm.com/software/lotus/

passportadvantage/ via 5725-S28v Also see the question, How is IBM Spectrum Scale V4 licensed?

For GPFS V3.5:v To order GPFS on POWER for AIX or Linux , find contact information for your country at

http://www.ibm.com/planetwide/ to place your order for 5765-G66v To order GPFS for Linux or Windows on x86 Architecture:

– To order in the same IBM fulfillment system as GPFS on POWER find contact informationfor your country at http://www.ibm.com/planetwide/ to place your order for 5765-XA3

– Go to the Passport Advantage site at http://www.ibm.com/software/lotus/passportadvantage/ where you may order 5724-N94

Q1.2: Where can I find ordering information for IBM Spectrum Scale?A1.2: You can view ordering information for IBM Spectrum Scale in the Announcement Letters at

http://www.ibm.com/common/ssi/index.wss

42

https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1ins_cesupgradeconsiderations.htm

http://www.ibm.com/planetwide

http://www.ibm.com/software/lotus/passportadvantage/









1. In the keyword, letter or product number bar, search for IBM Spectrum Scale in title, orsearch for <one of the product numbers> in product or document. Choose Announcementletters, Software.

2. For IBM Spectrum Scale V5, choose the corresponding product number to enter in the Searchfor field:v IBM Spectrum Scale Standard Edition: 5737-F33 (Passport Advantage), 5641-ST1, 5641-ST3,

5641-ST5 (eConfig/AAS)v IBM Spectrum Scale Data Management Edition: 5737-F34 (Passport Advantage), 5641-DM1,

5641-DM3, 5641-DM5 (eConfig/AAS)3. For IBM Spectrum Scale V4, choose the corresponding product number to enter in the Search

for field:v For orders placed through AAS ordering system, enter 5641-GPF.v For orders placed through Passport Advantage, enter 5737-F33 or 5737-F34.

Q1.3: Where can I find the documentation for IBM Spectrum Scale?A1.3: Documentation is available on the IBM Knowledge Center

v For IBM Spectrum Scale V4.2.2, at http://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.htmlb

v For GPFS V3.5 and V4.1, at http://www-01.ibm.com/support/knowledgecenter/SSFKCN/gpfs_welcome.html

.Q1.4: What is there beyond the standard documentation that can help me learn more about and use

IBM Spectrum Scale™?A1.4: Additional resources include:

v The paper On Making GPFS Truly General at http://researcher.watson.ibm.com/researcher/files/us-dhildeb/login_hildebrand.pdf

v The IBM Spectrum Scale wiki at https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)

v GPFS forums:– The GPFS technical discussion forum at https://www.ibm.com/developerworks/

community/forums/html/forum?id=11111111-0000-0000-0000-000000000479&ps=100 willhelp answer your questions on installing and running GPFS.

v GPFS Technotes:– GPFS Internal and External mount states at http://www.ibm.com/support/

docview.wss?uid=isg3T1021965v GPFS Web pages:

– The IBM Support Portal at (login with your IBM ID) https://www-947.ibm.com/support/entry/myportal/product/cluster_software/general_parallel_file_system?productContext=-1420716648

– The GPFS Wiki at https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/GPFS%20Wiki

– The IBM Spectrum Scale page at http://www.ibm.com/systems/storage/spectrum/scale/index.html

– The GPFS Support Portal at https://www-947.ibm.com/systems/support/myview/subscription/css.wss/

v The IBM Systems Magazine site at http://www.ibmsystemsmag.com/ and search on IBMSpectrum Scale and GPFS.

v The IBM Redbooks® and Redpapers™ site at www.redbooks.ibm.com and search on IBMSpectrum Scale and GPFS.

v The IBM Technical Sales Library at http://www-03.ibm.com/support/techdocs/atsmastr.nsf/Web/Techdocs

v For IBM Spectrum Scale for Linux on Z, see the white paper Getting started with IBMSpectrum Scale for Linux on Z.

v Classes:

43



http://www-01.ibm.com/support/knowledgecenter/SSFKCN/gpfs_welcome.html


http://researcher.watson.ibm.com/researcher/files/us-dhildeb/login_hildebrand.pdf

http://researcher.watson.ibm.com/researcher/files/us-dhildeb/login_hildebrand.pdf

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)

https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479&ps=100


http://www.ibm.com/support/docview.wss?uid=isg3T1021965


https://www-947.ibm.com/support/entry/myportal/product/cluster_software/general_parallel_file_system?productContext=-1420716648



https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/GPFS%20Wiki



http://www.ibm.com/systems/storage/spectrum/scale/index.html

http://www.ibm.com/systems/storage/spectrum/scale/index.html

https://www-947.ibm.com/systems/support/myview/subscription/css.wss/


http://www.ibmsystemsmag.com/

http://www.ibm.com/redbooks

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/Web/Techdocs

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/Web/Techdocs

http://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=WH&infotype=SA&appname=STGE_ZS_ZS_USEN&htmlfid=ZSW03272USEN&attachment=ZSW03272USEN.PDF

http://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=WH&infotype=SA&appname=STGE_ZS_ZS_USEN&htmlfid=ZSW03272USEN&attachment=ZSW03272USEN.PDF

Table 27. Classes

Course Code Course Title Course Type

H005G IBM Spectrum Scale Basic Administration forLinux

System Administration classroom

IBM training provides education to support many IBM offerings. Descriptions of courses for ITprofessionals and managers are on the IBM training website http://www.ibm.com/services/learning/

Q1.5: How can I ask a more specific question about IBM Spectrum Scale?A1.5: Depending upon the nature of your question, you may ask it in one of several ways.

v If you want to correspond with IBM regarding IBM Spectrum Scale :– If your question concerns a potential software error in IBM Spectrum Scale and you have an

IBM software maintenance contract, please contact 1-800-IBM-SERV in the United States oryour local IBM Service Center in other countries.

– If you have a question that can benefit other IBM Spectrum Scale users, you may post it tothe GPFS technical discussion forum at http://www.ibm.com/developerworks/forums/forum.jspa?forumID=479

– This FAQ is continually being enhanced. To contribute possible questions or answers, pleasesend them to [email protected]

v If you want to interact with other GPFS users, the San Diego Supercomputer Center maintainsa GPFS user mailing list. The list is [email protected] and those interested can subscribe tothe list at lists.sdsc.edu/mailman/listinfo/gpfs-general

v If you want to submit or view Request for Enhancements for IBM Spectrum Scale, go to IBMdeveloperWorks at https://www.ibm.com/developerworks/rfe/execute?use_case=changeRequestLanding&BRAND_ID=0&PROD_ID=1026&x=15&y=16

If your question does not fall into the above categories, you can send a note directly to the IBMSpectrum Scale development team at [email protected]. However, this mailing list is informallymonitored as time permits and should not be used for priority messages to the IBM SpectrumScale team.

Q1.6: Does IBM Spectrum Scale participate in the IBM Academic Initiative Program?A1.6: IBM Spectrum Scale does not participate in the IBM Academic Initiative Program. Work with

your IBM client representative to determine what educational discount may be available for IBMSpectrum Scale. See www.ibm.com/planetwide/index.html

Q1.7: Is IBM Spectrum Scale available in IBM PartnerWorld?A1.7: IBM Spectrum Scale is available in IBM PartnerWorld. Search for "IBM Spectrum Scale", "General

Parallel File System", or "GPFS" in the Software Access catalog https://www-304.ibm.com/jct01004c/partnerworld/partnertools/eorderweb/ordersw.do

Q1.8: Does IBM Spectrum Scale have a trial program?A1.8: For trials:

v A free 90-day trial program is available to prospective customers of IBM Spectrum Scale.Contact your IBM sales representative to apply for this trial evaluation.You can also use the IBM Spectrum Scale Trial Virtual Machine, which can be downloadedfrom http://www-03.ibm.com/systems/storage/spectrum/scale/trial.html. The IBM SpectrumScale Trial VM offers a fully preconfigured single node IBM Spectrum Scale instance in avirtual machine along with step-by-step guides for certain IBM Spectrum Scale features. TheIBM Spectrum Scale Trial VM can be used with VirtualBox, KVM, and VMware ESXhypervisor. The download bundle includes the virtual image and required guides (Quick StartGuide, Explore guide, and Advanced guide), so that you can try key features. The Explorer andAdvanced guides provide step-by-step instructions for GUI, Information Life Cycle (ILM),Unified file, object, protocols access, and file compression functionality.For further questions, use the reference forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479&ps=25.

ISVs can obtain the software and licenses from the IBM Partner World at https://www-356.ibm.com/partnerworld/wps/servlet/ContentHandler/isv/sac.

44

https://www-03.ibm.com/services/learning/ites.wss/zz/en?pageType=course_description&courseCode=H005G&cc=

https://www-03.ibm.com/services/learning/ites.wss/zz/en?pageType=course_description&courseCode=H005G&cc=

http://www.ibm.com/services/learning/

http://www.ibm.com/services/learning/


http://www.ibm.com/developerworks/forums/forum.jspa?forumID=479



http://www.sdsc.edu/

mailto:[email protected]

https://lists.sdsc.edu/mailman/listinfo/gpfs-general

https://www.ibm.com/developerworks/rfe/execute?use_case=changeRequestLanding&BRAND_ID=0&PROD_ID=1026&x=15&y=16

https://www.ibm.com/developerworks/rfe/execute?use_case=changeRequestLanding&BRAND_ID=0&PROD_ID=1026&x=15&y=16



https://www-304.ibm.com/jct01004c/partnerworld/partnertools/eorderweb/ordersw.do

https://www-304.ibm.com/jct01004c/partnerworld/partnertools/eorderweb/ordersw.do

http://www-03.ibm.com/systems/storage/spectrum/scale/trial.html



https://www-356.ibm.com/partnerworld/wps/servlet/ContentHandler/isv/sac

https://www-356.ibm.com/partnerworld/wps/servlet/ContentHandler/isv/sac

Q1.9: Where can I find the documentation for IBM Spectrum Scale RAID?A1.9: The documentation for IBM Spectrum Scale RAID can be found:

v The FAQ at http://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/SSYSP8/gnrfaq.html

v The Elastic Storage Server 4.0 IBM Knowledge Center at http://www.ibm.com/support/knowledgecenter/SSYSP8_4.0.0/sts40_welcome.html



v The GPFS Storage Server 2.0 IBM Knowledge Center at http://www.ibm.com/support/knowledgecenter/SSYSP8_2.0.0/sts20_welcome.html

45

http://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/SSYSP8/gnrfaq.html

http://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/SSYSP8/gnrfaq.html

http://www.ibm.com/support/knowledgecenter/SSYSP8_4.0.0/sts40_welcome.html








Software questionsQ2.1: Which levels of the AIX OS are supported by IBM Spectrum Scale?A2.1: IBM Spectrum Scale supports AIX V7.2, V7.1, and V6.1.

Table 28. IBM Spectrum Scale for AIX

IBM Spectrum Scale AIX 7.2 AIX V7.1 AIX V6.1

IBM Spectrum Scale V5.0 V5.0.0.0 or later V5.0.0.0 or later (AIX 7.1 TL3or later)

IBM Spectrum Scale V4.2 V4.2.0.0 or later V4.2.0.0 or later V4.2.0.0 or later(AIX 6.1.4.0 or later)

IBM Spectrum Scale V4.1 V4.1.1.4 or later V4.1.0.0 or later V4.1.0.0 or later(AIX 6.1.4.0 or later)

GPFS V3.5 V3.5.0.29 or later V3.5.0.0 or later V3.5.0.0 or later

Note:1. It is highly suggested that customers apply the latest service updates for their operating

system. For links to the latest IBM Spectrum Scale service updates, see the question How do Idownload fixes for IBM Spectrum Scale?

2. The version of OpenSSL that is shipped with some versions of AIX 7.1, AIX V6.1, and AIXV5.3 does not work with GPFS V3.5 or V3.4 due to a change in how the library is built. Toobtain the level of OpenSSL that works with GPFS, see the question How do I get OpenSSL towork with GPFS?. As of V4, IBM Spectrum Scale does not use OpenSSL for securecommunication across nodes. Instead, it uses the GSKit toolkit, which is shipped asgpfs.gskit in all editions of IBM Spectrum Scale V4 and later.

3. Service is required for GPFS to work with some levels of AIX. For more information, see thequestion What are the current advisories for IBM Spectrum Scale on AIX?

4. Restrictions apply to certain protocols. For more information, see “Functional SupportMatrices” on page 11.

Q2.2: Which Linux distributions are supported by IBM Spectrum Scale?A2.2: IBM Spectrum Scale supports the distributions in the following tables. See subsequent tables for

kernel specific information. See the Functional Support Matrices at the beginning of the FAQ forfunctional limitations regarding distribution support.

While IBM Spectrum Scale runs with many different Linux kernel levels, it is highly suggestedthat customers apply the latest kernel service updates for their operating system. For links to thelatest IBM Spectrum Scale service updates, see the question How do I download fixes for IBMSpectrum Scale?.

Note: For information on RHEL support for protocols, see the question What are the requirementsto use the protocol access methods integrated with IBM Spectrum Scale V4.1.1 and later?.

Table 29. Linux distributions supported by IBM Spectrum Scale on x86_64 (RHEL and SLES)

RHEL SLES

7 6 5 12 11 10

IBMSpectrumScale V5.0

V5.0.0 or later(RHEL 7.1 or

later)

V5.0.0 or later(SLES12 SP1 or

later)


V4.2.0.0 orlater

V4.2.0.0 or later(RHEL 6.4 or later)

V4.2.0.0 or later V4.2.0.0 or later(SLES11 SP3 or

later)


V4.1.0.4 orlater

V4.1.0.0 or later(RHEL 6.2 or later)

V4.1.1.0 or later V4.1.0.0 or later(SLES11 SP2 or

later)

46

Table 29. Linux distributions supported by IBM Spectrum Scale on x86_64 (RHEL and SLES) (continued)

RHEL SLES

7 6 5 12 11 10

GPFS V3.5 V3.5.0.21 orlater

V3.5.0.0 or later V3.5.0.0 orlater

V3.5.0.0 or later V3.5.0.0or

later

Table 30. Linux distributions supported by IBM Spectrum Scale on x86_64 (Debian and Ubuntu)

Debian Ubuntu

8 7 6 16.04 14.04

IBM SpectrumScale V5.0

V5.0.0 or later


V4.2.0.0 or later V4.2.0.0 or later V4.2.1.1 or later V4.2.0.0 or later


V4.1.1.0 or later V4.1.0.0 or later V4.1.0.0 or later V4.1.1.9 or later V4.1.0.5 or later

GPFS V3.5 V3.5.0.0 or later V3.5.0.22 or later

Table 31. Linux distributions supported by IBM Spectrum Scale on Power

RHEL SLES Ubuntu

7 6 5 12 11 10 16.04 14.04


V5.0.0 orlater

(RHEL 7.1or later)

V5.0.0 orlater

(SLES 12SP1 orlater)

V5.0.0 orlater (LE)


V4.2.0.0or later

(RHEL 7.1or later,BE and

LE. RHEL7.0, BE)

V4.2.0.0or later

(RHEL 6.4or later)

V4.2.0.0or later

(LE)

V4.2.0.0or later(SLES11SP3 orlater)

V4.2.1.1or later

(LE)

V4.2.0.0 or later (LE)


V4.1.1.0or later

(RHEL 7.1or later,BE and

LE)V4.1.0.4or later(RHEL7.0, BE)

V4.1.0.0or later

(RHEL 6.2or later)

V4.1.1.0or later

(LE)

V4.1.0.0or later(SLES11SP2 orlater)

V4.1.1.9or later

(LE)

V4.1.0.5 or later (LE)

GPFSV3.5

V3.5.0.21or later

(RHEL 7.0or later,

BE)

V3.5.0.0or later

V3.5.0.0or later

V3.5.0.0or later

V3.5.0.0or later

Note: GPFS V3.5 does not support LE.

47

Table 32. IBM Spectrum Scale for Linux RedHat kernel support

RHEL DistributionLatest Kernel LevelTested1

Minimum KernelLevel Required2

Minimum IBMSpectrum ScaleLevel Tested3

Minimum IBMSpectrum ScaleLevel Supported4

7.5 for IBM Power LE(POWER9)

4.14.0-49.el7a 4.14.0-49.el7a From V4.2.3.9 in the4.2 release

From V5.0.1.1 in the5.0 release



7.5 3.10.0-862.el7 3.10.0-862.el7 From V4.1.1.20 in the4.1.1 release



From V4.1.1.20 in the4.1.1 release



7.4 for IBM Power LE(POWER9)

4.11.0-44.6.1.el7a 4.11.0-44.6.1.el7a From V4.2.3.8 in the4.2 release

From V5.0.0 in the 5.0release



7.46, 7 x86: 3.10.0-693.11.6

Power: 3.10.0-693.17.1

3.10.0-693 From V4.1.1.17 in the4.1.1 release






7.3 x86: 3.10.0-514.36.5

Power: 3.10.0-514.41.1







7.2 x86: 3.10.0-327.62.4

Power: 3.10.0-327


From V4.2



From V4.2


7.1 3.10.0-229 3.10.0-229 From V3.5.0.26 in the3.5 release

From V4.1.1

From V4.2



From V4.1.1

From V4.2


48

||||||

||

||

||

|||||

||

||

||

||

||

||||||

||

||

||

Table 32. IBM Spectrum Scale for Linux RedHat kernel support (continued)







From V4.2



From V4.2

6.9 x86: 2.6.32-696.18.7

Power: 2.6.32-696.23.1










From V4.2

6.7 x86: 2.6.32-573.49.3

Power: 2.6.32-573

2.6.32-573 From V3.5.0.27 in the3.5 release


From V4.2



From V4.2



From V4.2



From V4.2


From V4.1

From V4.2


From V4.1

From V4.2

6.4 2.6.32.358.2.1 2.6.32.358 From V3.5.0.8 in the3.5 release

From V4.1

From V4.2


From V4.1

From V4.2

6.3 2.6.32.279.14.1 2.6.32.279 From V3.5.0.1 in the3.5 release

From V4.1


From V4.1

6.2 2.6.32-220.4.1 2.6.32-220 From V3.5.0.1 in the3.5 release

From V4.1


From V4.1

49

Table 32. IBM Spectrum Scale for Linux RedHat kernel support (continued)





6.1 2.6.32-279.2.1 2.6.32-131.0.15 From V3.5.0.1 in the3.5 release
















Note:1. The Latest Kernel Level Tested column indicates the latest kernel level that full testing is done

on.2. The Minimum Kernel Level Required column indicates the minimum kernel level that is

required or the distribution version GA kernel level.3. The Minimum IBM Spectrum Scale Level Tested column indicates the IBM Spectrum Scale

level that full testing is done on.4. The Minimum IBM Spectrum Scale Level Supported column is the minimum IBM Spectrum

Scale level that supports the Linux distribution version.5. The Elastic Storage Server (ESS) is supported on RHEL 7.1 with V4.1.0.8 if efix 25 is applied.6. The installation toolkit, AFM, Asynch Disaster Recovery with AFM, SMB, NFS, and Object

require IBM Spectrum Scale V4.2.3.7 or later.7. The installation toolkit, AFM, Asynch Disaster Recovery with AFM, SMB, NFS, and Object

require IBM Spectrum Scale V5.0.0.1 or later.

Table 33. IBM Spectrum Scale for Linux SLES kernel support

SLES DistributionLatest Kernel LevelTested1




12 SP3 x86: 4.4.103-6.38.1

Power: 4.4.114-94.11







50

||||

||

|

|||

||

||

||

||

||

Table 33. IBM Spectrum Scale for Linux SLES kernel support (continued)





12 SP2 x86: 4.4.103-92.56.1

Power: 4.4.114-92.64

4.4.21-69-default From V4.1.1.14 in the4.1.1 release






12 SP1 3.12.74-60.64.69requires 4.1.1.19,4.2.3.7, or 5.0.0.1

3.12.74-60.64.82requires 4.1.1.20,4.2.3.8, or 5.0.1.0







12 Power: 3.12.51-52.31

x86: 3.12.61-52.111requires 4.1.1.19 or4.2.3.7

x86: 3.12.61-52.119requires 4.1.1.20 or4.2.3.8

3.12.51-52.31 From V4.1.1

From V4.2

From V4.1.1

From V4.2

11 SP4 x86: 3.0.101-108.21.1

Power: 3.0.101-108.35



From V4.2



From V4.2

11 SP3 x86:3.0.101-0.47.106.11.1

Power: 3.0.76-0.11

3.0.76-0.11 From V3.5.0.11 in the3.5 release

From V4.1

From V4.2


From V4.1

From V4.2

11 SP2 3.0.101-0.21.1 3.0.42-0.7.3 From V3.5.0.6 in the3.5 release

From V4.1


From V4.1

11 SP1 2.6.32.59-0.7 2.6.32.59-0.7 From V3.5.0.1 in the3.5 release


11 2.6.32.43-0.4 2.6.32.43-0.4 From V3.5.0.1 in the3.5 release


10 SP4 2.6.16.60-0.84.1 2.6.16.60-0.84.1 From V3.5.0.1 in the3.5 release


10 SP3 2.6.16.60-0.59.1 2.6.16.60-0.59.1 From V3.5.0.1 in the3.5 release


Note:

51

||||

|||

|||

||

||

||

||

||

||

|||

|||

||

|

|

|

1. The Latest Kernel Level Tested column indicates the latest kernel level that full testing is doneon.

2. The Minimum Kernel Level Required column indicates the minimum kernel level that isrequired or the distribution version GA kernel level.

3. The Minimum IBM Spectrum Scale Level Tested column indicates the IBM Spectrum Scalelevel that full testing is done on.

4. The Minimum IBM Spectrum Scale Level Supported column is the minimum IBM SpectrumScale level that supports the Linux distribution version.

Table 34. IBM Spectrum Scale for Linux Debian kernel support

Debian DistributionLatest Kernel LevelTested1




8.6 3.16.0-4-amd643.16.36-1+deb8u1

3.16.0-4-amd643.16.36-1+deb8u1



From V4.1.1


8.5 3.16.0-4amd643.16.7-ckt25-2

3.16.0-4amd643.16.7-ckt25-2



From V4.1.1


8.4 3.16.0-4amd643.16.7-ckt25-1

3.16.0-4amd643.16.7-ckt25-1



From V4.1.1


8.3 3.16.0-4amd643.16.7-ckt20-1+deb8u3

3.16.0-4amd643.16.7-ckt20-1+deb8u3



From V4.1.1


8.1 3.16.7-ckt11-1 3.16.7-ckt11-1 From V4.1.1.1 in the4.1.1 release

From V4.2

From V4.1.1

From V4.2

7.11 3.2.0-4amd64 3.2.81-1 3.2.0-4amd64 3.2.81-1 From V4.1.1.9 in the4.1.1 release


From V4.1.1


7.9 3.2.0-4amd643.2.68-1+deb7u3

3.2.0-4amd643.2.68-1+deb7u3



From V4.1.1


7.1 3.2.0-4-amd64 3.2.0-4-amd64 From V4.1

From V4.2

From V4.1

From V4.2

6.0.8 3.2.5.1-1 3.2.5.1-1 From V3.5.0.15 in the3.5 release

From V4.1


From V4.1

6.0.3 2.6.32-5-amd64 2.6.32-5-amd64 From V3.5.0.1 in the3.5 release


52





Scale level that supports the Linux distribution version.

Table 35. IBM Spectrum Scale for Linux Ubuntu kernel support

Ubuntu DistributionLatest Kernel LevelTested1




16.04.4 (HWE kernel) 4.13.0-37 4.13.0-37 From V4.2.3.9 in the4.2 release




16.04.4 4.4.0-116 4.4.0-116 From V4.1.1.20 in the4.1.1 release






16.04.3 (HWE kernel) x86: 4.10.0-37

Power: 4.13.0-36





16.04.3 x86: 4.4.0-112

Power: 4.4.0-116







16.04.2 4.4.0-77/4.8.0-51 4.4.0-62/4.8.0-36 From V4.1.1.16 in the4.1.1 release






16.04.1 4.4.0-47 4.4.0-47 From V4.1.1.12 in the4.1.1 release






53

|||||

||

||

||

||

||

Table 35. IBM Spectrum Scale for Linux Ubuntu kernel support (continued)





16.04 4.4.0-31 4.4.0-21 From V4.1.1.9 in the4.1.1 release






14.04.5 (HWE kernel) x86: 4.4.0-111

Power: 4.4.0-116





14.04.5 x86: 3.13.0-141

Power: 3.13.0-142

x86: 3.13.0-141

Power: 3.13.0-142





14.04.4 4.2.0-42 4.2.0-27 From V3.5.0.32 in the3.5 release








From V4.2



From V4.2



From V4.2










level that full testing is done on.

54

4. The Minimum IBM Spectrum Scale Level Supported column is the minimum IBM SpectrumScale level that supports the Linux distribution version.

Note:v When issuing make World for Ubuntu 14.04.1, this warning appears but can be disregarded

because kdump-kern-dummy.ko is not used by IBM Spectrum Scale.WARNING: ".TOC." [/usr/lpp/mmfs/src/gpl-linux/kdump-kern-dummy.ko] undefined!

v Support for Ubuntu requires IBM Spectrum Scale V4.1.1 or the installation of base RPMs datedDecember 2014.– For products ordered through AAS/eConfig, log in to the Entitled Software page at:

https://www-05.ibm.com/servers/eserver/ess/OpenServlet.wss.– For products ordered through Passport Advantage, log in to the site at:

http://www.ibm.com/software/lotus/passportadvantage/.

Table 36. Linux distributions supported by IBM Spectrum Scale on IBM Z (RHEL and SLES)

RHEL SLES

7 6 12 11


V5.0.0 or later(RHEL 7.1 or

later)

V5.0.0 or later (SLES 12SP1 or later)


V4.2.0.0 or later V4.2.0.0 or later (RHEL6.5 or later)

V4.2.0.0 (SLES 12 SP1 orlater)

V4.2.0.0 or later(SLES 11 SP4 or later)


V4.1.0.5 or later V4.1.0.5 or later (RHEL6.5 or later)

V4.1.1.0 or later V4.1.0.5 or later(SLES11 SP3 or later)

Table 37. Linux distributions supported by IBM Spectrum Scale on IBM Z (Ubuntu)

Ubuntu

16.04

IBM Spectrum Scale V5.0 V5.0.0 or later

IBM Spectrum Scale V4.2 V4.2.3.1 or later

Table 38. IBM Spectrum Scale for Linux on Z RHEL kernel support



Minimum IBMSpectrum Scale LevelTested3

Minimum IBMSpectrum Scale LevelSupported4

7.5 3.10.0-862.3.2.el7.s390x

3.10.0-862.el7.s390x From V4.2.3.9 in the4.2 release




7.46 3.10.0-693.17.1 3.10.0-693.5.2.el7.s390x




7.3 3.10.0-514.10.2.el7.s390x

3.10.0-514.10.2.el7.s390x





55

||||||

||

||

||

https://www-05.ibm.com/servers/eserver/ess/OpenServlet.wss


Table 38. IBM Spectrum Scale for Linux on Z RHEL kernel support (continued)





7.2 3.10.0-327.28.3.el7.s390x5

3.10.0-327.4.5.el7.s390x





7.1 3.10.0-229.4.2.el7.s390x

3.10.0-229.el7.s390x From V4.1.1


From V4.1.1


7.0 3.10.0-123.6.3.el7.s390x



6.9 2.6.23-696.23.1.el6.s390x


From 4.2.3.1 in the 4.2release

6.8 2.6.32-642.6.2.el6.s390x5

2.6.32-573.18.1.el6.s390x



6.7 2.6.32-573.18.1.el6.s390x

2.6.32-573.18.1.el6.s390x

From V4.2 in the 4.2release


6.6 2.6.32-504.8.1.el6.s390x

2.6.32-504.1.3.el6.s390x



6.5 + Errata UpdateRHSA-2014-0328

2.6.32-431.29.2.el6.s390x

2.6.32-431.11.2 From V4.1.0.5 in the4.1 release


Table 39. IBM Spectrum Scale for Linux on Z SLES kernel support





12 SP3 4.4.103-94.6.1 4.4.103-94.6 From V4.2.3.7 in the4.2 release



12 SP2 4.4.38-93-default 4.4.21-69-default From V4.2.3 in the 4.2release




12 SP1 3.12.62-60.64.8-default 3.12.51-60-20-default From V4.2.1 in the 4.2release




12 3.12.60-52.54-default 3.12.36-38-default From V4.1.1 in the 4.1release


11 SP4 3.0.101-84-default 3.0.101-63 From V4.2 in the 4.2release


11 SP3 + MaintwebUpdate

3.0.101-0.47.71-default 3.0.101-0.15 From V4.1.0.5 in the4.1 release


56

Table 40. IBM Spectrum Scale for Linux on Z Ubuntu kernel support





16.04.4

16.04.47

4.4.0-119

4.4.0-119/4.13.0-38

4.4.0-116

4.4.0-116/4.13.0-38

From 4.2.3.8 in the4.2 release

From 5.0.1 in the5.0 release

From 4.2.3.8 in the4.2 release

From 5.0.1 in the5.0 release

16.04.3 4.4.0-112 4.4.0-93/4.10.0-33 From V4.2.3.7 in the4.2 release



16.04.2 4.4.0-75 / 4.8.0-49 4.4.0-75 / 4.8.0-49 From V5.0.0 in the 5.0release










Scale level that supports the Linux distribution version.5. In regard to the latest Kernel for RHEL 6.8 and RHEL 7.2, see the following Technote

http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009622.6. AFM and Asynch Disaster Recovery with AFM (Data Management and Advanced editions)

require IBM Spectrum Scale V5.0.0.1 or later.7. IBM Spectrum Scale supports HWE kernel 4.13 on Ubuntu 16.04.4 on V5.0.1.0 for IBM Z.

Note: Support for IBM Spectrum Scale on IBM Z requires an order for 5725-S28.

Also see the questions:v What are the current requirements/limitations for IBM Spectrum Scale for Linux on Z?v What are the current requirements/limitations for using Ubuntu?v What are the current restrictions on IBM Spectrum Scale Linux kernel support?v Is IBM Spectrum Scale on Linux supported in a virtualization environment?v What are the current advisories for all platforms supported by IBM Spectrum Scale?v What are the current advisories for IBM Spectrum Scale on Linux?

Q2.3: What is the IBM Spectrum Scale support position regarding clone Linux distributions (CentOS,ROCKS, White box Linux, etc.) ?

A2.3: There are many Linux distributions out there. It would not be practical to try to test and supportthem all. The IBM Spectrum Scale team focuses on testing Enterprise Linux distributions (RHELand SLES) and Ubuntu. However, there are some popular distributions in the Linux communitythat are created by essentially building the code from source packages corresponding to one ofthe enterprise distributions, usually with some cosmetic changes. IBM Spectrum Scale code maybe able to work correctly on such a distribution, since it very closely resembles a supported one.However, we do not test IBM Spectrum Scale explicitly on such clone distributions, and will not

57


be able to provide support for any problems specific to the use of the latter. If a problem isreported in such an environment, we will investigate it, but if the problem is suspected to berelated to the type of distribution used, we may request that the problem be recreated on asupported distribution. Note that other IBM products may have a different support policy. Werecommend that a supported distribution is used on NSD servers and other nodes that have SANconnectivity, to make it possible to get support with storage-related issues.

Note: IBM Spectrum Scale for Linux on Z is only supported on the distributions and kernellevels as documented in the question What Linux distributions are supported by IBM Spectrum Scale?

Q2.4: What are the current restrictions on IBM Spectrum Scale Linux kernel support?A2.4: Current restrictions on IBM Spectrum Scale Linux kernel support include:

v For IBM Spectrum Scale on RHEL 7 or SLES 12 SP1 (kernel versions later than 3.7) to run onthe Broadwell processors, the IBM Spectrum Scale version needs to be at version 4.1.1.10 orlater on the 4.1 release and version 4.2.1.1 or later on the 4.2 release. On IBM Spectrum Scalereleases earlier than version 4.1.1.10 on the 4.1 release and earlier than version 4.2.1.1 on the 4.2release, it is necessary to follow the steps outlined below:– Disable the Supervisor Mode Access Prevention (smap) kernel parameter– Reboot the node before using GPFS– For more information, see http://www-01.ibm.com/support/

docview.wss?uid=ssg1S1009287.v Systemd is replacing traditional sysVinit in many Linux distributions. Though systemd should

still support traditional sysVinit without any changes, starting with systemd version 219-19,this support is not working properly, causing IBM Spectrum Scale services not to startup atboot time. If you are experiencing this problem, you can do one of the following:– Upgrade your nodes to IBM Spectrum Scale V4.2.0.1.

IBM Spectrum Scale uses systemd to start IBM Spectrum Scale services starting withV4.2.0.1.

– Apply the following workaround:rm /etc/init.d/gpfscp /usr/lpp/mmfs/bin/gpfsrunlevel /etc/init.d/gpfs

– If systemd is upgraded to version 219 after IBM Spectrum Scale V4.2.0.1 was alreadyinstalled, you can apply the workaround in the second option or take the following step toenable IBM Spectrum Scale services to use systemd:systemctl enable /usr/lpp/mmfs/lib/systemd/gpfs.service

v GPFS V3.4 and V3.5 on Linux do not support standard disk partitioning commands, and donot label disks in a way that is recognized by standard disk utilities. To these utilities, GPFSdisks will generally appear to be unused devices. The mmlsnsd command allows systemsadministrators to list GPFS disks. Administrators should verify the disposition of a disk beforeusing parted, fdisk, or another partitioning command so as not to inadvertently partition aGPFS disk.

Note: IBM Spectrum Scale V4.1 writes a GUID Partition Table (GPT) label on newly createdNSDs, which should make it safer to use standard disk partitioning tools on servers with blockaccess to GPFS NSDs

v As of V3.5.0.3, GPFS provides mmksh and the KSH related issues below do not apply:– GPFS has experienced memory leak issues with various levels of KSH. In order to address

this issue, please ensure that you are at the minimum required level of KSH or later:- RHEL 5 should be at ksh-20100202-1.el5_6.3, or later- SLES 10 should be at ksh-93t-13.17.19 (shipped in SLES 10 SP4), or later- SLES 11 should be at ksh-93t-9.9.8 (shipped in SLES11 SP1)

Some GPFS commands may experience a premature termination bug with various levels ofKSH. In order to address this issue, please ensure that you are at the required level of KSH:– RHEL 6.1 and RHEL 6.2 should be at ksh-20100621-12.el6_2.1 or later– SLES 11 SP2 should be at ksh-93u-0.8.1 or later

v GPFS has the following restrictions on RHEL support:

58



– The modutils package has been renamed module-init-tools in RHEL7. As a result, to installthe gpfs.base rpm for V3.5 on a RHEL 7 system, you need to specify the --nodeps option.

– For RHEL 7, in order to have GPFS work properly in Secure Boot enabled system:- Disable Secure Boot in BIOS- Sign GPFS kernel module manually.

– On RHEL6.x, GPFS does not support the Transparent Huge Page (THP) feature for kernelsprior to RHEL 6.4 (kernel version 2.6.32-358.el6), RHEL 6.3 (kernel version 2.6.32-358.2.1),and RHEL 6.2 (kernel version 2.6.32-220.4.1.e;6.x86_64). On unsupported kernels, THPshould be disabled at boot time by appending transparent_hugepage=never to the kernelboot options.

– GPFS does not currently support the following kernels:- RHEL hugemem- RHEL largesmp- RHEL uniprocessor (UP)

– There is required service for support of RHEL, please see the question What are the currentadvisories for GPFS on Linux?

v GPFS has the following restrictions on SLES support:– There is required service for support of SLES, please see the question What are the current

advisories for GPFS on Linux?v GPFS for Linux on POWER does not support mounting of a file system with a 16KB block size

when running on either RHEL 5, or later, or SLES 11, or later.v GPFS has the following restrictions on Debian support:

– Clustered NFS is not supported on Debian.– The File Placement Optimizer function is not supported on Debian.– These nodes can only operate as NSD clients and do not support direct attached disks.

Please also see the questions:v What Linux distributions are supported by IBM Spectrum Scale?v Is IBM Spectrum Scale on Linux supported in a virtualization environment?v What are the current advisories for all platforms supported by IBM Spectrum Scale?v What are the current advisories for IBM Spectrum Scale on Linux?

Q2.5: What Linux distributions are supported by the integrated protocols access methods in IBMSpectrum Scale V4.1.1 and later?

A2.5: The integrated protocols access methods available with IBM Spectrum Scale V4.1.1 and latersupport RHEL 7 and 7.1. IBM Spectrum Scale V4.2.2.2 and later supports RHEL 7.3.

After upgrading to IBM Spectrum Scale V4.2.2.2, if you upgrade to RHEL 7.3, there can be someissues while running NFS and Object services. To avoid these issues, perform the following stepsafter you upgrade to RHEL 7.3:

For NFS:

Because the IBM Spectrum Scale version of the NFS server must be used, the NFS service mustrun under service instead of systemctl.1. Determine if the nfs.service is active on any CES node by running the command mmdsh -N

cesnodes systemctl status nfs.service. If any of the nodes report that the service is active,you will need to stop and disable the nfs.service:v mmdsh -N cesNodes systemctl stop nfs.servicev mmdsh -N cesNodes systemctl disable nfs.service

2. You can then restart the IBM Spectrum Scale NFS service:v mmces service start nfs -a

For Object:

You will need to reload the spectrum-scale-object-selinux module if the CES nodes are inSELinux Enforcing mode. To check whether they are in SELinux Enforcing mode, run thecommand: mmdsh -N cesnodes getenforce. If the mode is returned as Enforcing or Permissive,then run the following commands:v mmces service stop obj -a

59

v mmdsh -N cesNodes semodule -i /usr/share/selinux/packages/spectrum-scale-object-selinux.pp

v mmces service start obj -a

Note: Also, see the question 14.6 What are the current advisories for IBM Spectrum Scale on Linux?Q2.6: What is the impact on the /dev, /proc/mounts, /etc/mtab directories and the mount command

for IBM Spectrum Scale for Linux due to the recent changes in systemd? What happened tothe block device in /dev? Why is the /dev/ prefix missing from the output of the mountcommand and also from /proc/mounts and /etc/mtab?

A2.6: Starting with Spectrum Scale 4.2.1, GPFS on Linux no longer creates a block device in /dev forthe corresponding GPFS file system. As the result, the prefix /dev/ does not appear before theGPFS device name in the output of the Linux mount command and in files /etc/fstab , /etc/mtab,and /proc/mounts. On the other hand, commands that accept /dev/file-system-name as input willcontinue doing so. A few commands will still display the file system name as/dev/file-system-name.

For example:c13c1apv7:~ # awk ’$3 == "gpfs" { print }’ /proc/mounts/gpfs/automountdir/fs2 /gpfs/automountdir/fs2 gpfs rw,relatime 0 0fs1 /gpfs/fs1 gpfs rw,relatime 0 0fs5mpathd /fs5mpathd/a/few/level/mount/point gpfs rw,relatime 0 0remote /remote gpfs rw,relatime 0 0/gpfs/automountdir/fs3mpathb /gpfs/automountdir/fs3mpathb gpfs rw,relatime 0 0/fs4mpathc /gpfs/automountdir/fs4mpathc gpfs rw,relatime 0 0/gpfs/automountdir/autofs1 /gpfs/automountdir/autofs1 gpfs rw,relatime 0 0/autofs2 /gpfs/automountdir/autofs2 gpfs rw,relatime 0 0/autofs3 /gpfs/automountdir/autofs3 gpfs rw,relatime 0 0

c13c1apv7:~ # mount | awk ’/type gpfs/ { print }’/gpfs/automountdir/fs2 on /gpfs/automountdir/fs2 type gpfs (rw,relatime)fs1 on /gpfs/fs1 type gpfs (rw,relatime)fs5mpathd on /fs5mpathd/a/few/level/mount/point type gpfs (rw,relatime)remote on /remote type gpfs (rw,relatime)/gpfs/automountdir/fs3mpathb on /gpfs/automountdir/fs3mpathb type gpfs (rw,relatime)/fs4mpathc on /gpfs/automountdir/fs4mpathc type gpfs (rw,relatime)/gpfs/automountdir/autofs1 on /gpfs/automountdir/autofs1 type gpfs (rw,relatime)/autofs2 on /gpfs/automountdir/autofs2 type gpfs (rw,relatime)/autofs3 on /gpfs/automountdir/autofs3 type gpfs (rw,relatime)

Q2.7: What levels of the Windows OS are supported by IBM Spectrum Scale?A2.7: IBM Spectrum Scale V4 and V5 support Windows Server 2008 x64 (SP2), Windows Server 2008

R2, Windows Server 2012 (Datacenter and Standard), Windows Server 2012 R2 (Datacenter andStandard), Windows 7 (Enterprise and Ultimate editions) x64 SP1, and Windows 8.1 (Enterpriseedition) in both heterogeneous (Windows and non-Windows nodes) and homogenous(Windows-only nodes) clusters.

GPFS V3.5 supports Windows Server 2008 x64 (SP 2), Windows Server 2008 R2 and Windows 7(Enterprise and Ultimate editions) x64 SP1 in both heterogeneous (Windows and non-Windowsnodes) and homogenous (Windows-only nodes) clusters. GPFS V3.5.0.11 and later, supportsWindows Server 2012 (Datacenter and Standard) in homogeneous clusters only.

Also see the questions:1. What are the limitations of IBM Spectrum Scale support for Windows?2. What are the current advisories for all platforms supported by IBM Spectrum Scale?3. What are the current advisories for IBM Spectrum Scale on Windows?

Q2.8: Does IBM Spectrum Scale V4 still require the Windows SUA feature?A2.8: IBM Spectrum Scale V4 does not require the Subsystem for UNIX-based Applications (SUA)

feature in Windows. SUA was deprecated by Microsoft in Windows Server 2012 (Windows 8) andhas been completely removed beginning with Windows Server 2012 R2 (Windows 8.1). IBMSpectrum Scale V4 removed SUA as a prerequisite. Instead, IBM Spectrum Scale now requiresCygwin from Red Hat (http://www.cygwin.com):

60

http://www.cygwin.com

v For V4.1.0 thru V4.1.0.8, Cygwin 32-bit (version 1.7 or later) must be installed before installingIBM Spectrum Scale on Windows.

v For V4.1.1 and later, Cygwin 64-bit (version 1.7 or later) must be installed before installing IBMSpectrum Scale on Windows.

Please see the Installing GPFS on Windows nodes chapter of the IBM Spectrum Scale 4.1: Concepts,Planning and Installation Guide, on how to install and configure Cygwin.

Rolling upgrade of GPFS V3.5 (SUA based) to IBM Spectrum Scale V4.1 (Cygwin based) issupported. Adding a new IBM Spectrum Scale V4.1 Windows node to an existing GPFS clusterhaving V3.5 nodes is also supported. Please refer to the Migration, coexistence and compatibilitychapter of the IBM Spectrum Scale 4.1: Concepts, Planning and Installation Guide.

Q2.9: What are the limitations of IBM Spectrum Scale support for Windows ?A2.9: Current® limitations include:

v IBM Spectrum Scale for Windows does not support disks with 4K sector size.v IBM Spectrum Scale for Windows does not support OpenLdap for Windows-Unix

user-mapping. The only way to achieve Windows-Unix user-mapping in GPFS is via IdentityMapping for Unix (IMU) from Microsoft.

v Exporting IBM Spectrum Scale file systems as Server Message Block (SMB) shares (also knownas CIFS shares) from IBM Spectrum Scale Windows nodes is not supported.

v GPFS V3.5.0.11 and later versions of 3.5, support Windows Server 2012 (Datacenter andStandard) in homogeneous clusters only. IBM Spectrum Scale V4.1 has no such limitation.

v NFS serving (any version of NFS) by GPFS Windows nodes is not supported.v The mmapplypolicy command is not supported.v If installing GPFS V3.5 on Windows 7, only the Enterprise and Ultimate x64 SP1 editions are

supported as they include Subsystem for Unix-based Applications (SUA), a component of theoperating system required for GPFS on Windows. In order to install SUA on Windows 7, go toControl Panel, click on Programs and then Turn Windows features on or off.

v Since Windows 7 Enterprise and Ultimate and Windows 8.1 Enterprise are desktop operatingsystems, these nodes can only operate as NSD clients and do not support direct attached disks.It is recommended that most GPFS administrative commands be executed on non-Windows 7nodes.

v IBM Spectrum Scale for Windows is not supported in any environment where CitrixProvisioning Services are deployed.

v In IBM Spectrum Scale homogeneous Windows clusters, the Windows nodes can perform mostof the management and administrative operations. The exceptions include:– Certain GPFS commands to apply policies, administer quotas and administer ACLs.– Support for the native Windows Backup utility.Please refer to the IBM Spectrum Scale Concepts, Planning and Installation Guide for a full list oflimitations.

v In IBM Spectrum Scale V4, the following user commands require Administrative privileges.They can only be run by a user who is a member of the Administrators group: mmchfileset,mmcrsnapshot, mmdelsnapshot, mmdf, mmlsdisk, mmlsfileset, mmlsfs, mmlspolicy,mmlspool, mmlssnapshot, and mmsnapdir.

Note: This restriction does not apply to GPFS V3.5. Ordinary users can execute the abovecommands in the SUA environment.

v Encryption is not supported on Windows. The encryption function in the Advanced Editionshould be disabled when Windows nodes are in the cluster.

v IPv4 subnets are not supported on a cluster that is defined with IPv6 primary addresses(hostname) that contains Windows nodes.

v With V4, V3.5 and V3.4.0.14 (and later), a DMAPI-enabled file system may be mounted on aWindows node, with certain restrictions.

61





Note: Beginning with Version 7.1.3, IBM Tivoli® Storage Manager (TSM) is now IBM SpectrumProtect™.For TSM V7.1.1, only supported with IBM Spectrum Scale V4.1, please refer to– IBM Tivoli Storage Manager V7.1.1 Knowledge Center at http://www-01.ibm.com/support/

knowledgecenter/SSGSG7_7.1.1/com.ibm.itsm.tsm.doc/welcome.html– TSM support page at https://www-947.ibm.com/support/entry/myportal/product/tivoli/

tivoli_storage_manager?productContext=-2105539168v With GPFS V3.4.0.13 or earlier version of GPFS, a DMAPI-enabled file system may not be

mounted on a Windows node.v For GPFS V3.5 and V3.4, Windows only supports the AUTHONLY cipher.

Please also see questions:v What are the requirements for the use of OpenSSH on Windows nodes?v Is IBM Spectrum Scale on Windows supported in a virtualization environment?

Q2.10: What are the requirements for the use of OpenSSH on Windows nodes?A2.10: IBM Spectrum Scale requires the use of OpenSSH to support its administrative functions when

the cluster includes Windows nodes and UNIX nodes.

IBM Spectrum Scale V4

For IBM Spectrum Scale V4, install the Cygwin openssh package as described in the InstallingIBM Spectrum Scale on Windows nodes chapter of the IBM Spectrum Scale : Concepts, Planning andInstallation Guide. If using an openssh package from another vendor, make sure that it iscompatible with the Cygwin namespace and environment.

GPFS V3.4 and V3.5

Microsoft does not provide SSH support in the SUA Utilities and SDK, and the remote shellservice included with SUA has limitations that make it unsuitable for GPFS.

Interop Systems Inc. previously hosted the SUA Community Web site which included SUACommunity packages. This site is no longer available.

For GPFS V3.5.0.11 and later, an alternative for obtaining OpenSSH separately is available. GPFSprovides the Open Secure Shell (OpenSSH) for GPFS on Windows V3.5 package for downloadfrom FixCentral at http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster+software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=Windows&function=all. This version of OpenSSH is not supported onWindows 2012 at this time.v Extract the contents of the ZIP archive so that the .msi file it includes is directly accessible on

your system.v Uninstall the system's current version OpenSSH using Programs and Features in Control Panel.

Also uninstall any OpenSSH packages from other vendors, such as OpenSSH from SUACommunity. If Interop SUA Community OpenSSH is installed, remove it by using thepkg_delete utility.

v Install the OpenSSH package using the msiexec command so that you can create an installationlog file. For example:msiexec /i C:\openssh*.msi /log c:\openssh-install.log

v Complete the procedure as noted in the GPFS V3.5.0.11 Concepts, Planning, and InstallationGuide under the heading "Installing and configuring OpenSSH".

v The OpenSSH suite for GPFS on Windows gets installed under /usr/lpp/mmfs/platform inthe SUA environment. Utilities such as ssh-keygen must be invoked directly from under thebin subdirectory. For example, /usr/lpp/mmfs/platform/bin/ssh-keygen.

v The sshd daemon that comes with the OpenSSH suite for GPFS on Windows is started by theGPFS Administration Service (mmwinserv) upon a reboot. For sshd to work properly,mmwinserv must be configured to run under the root account.

v The sshd daemon can be manually started or stopped using the mmdeployssh command. Run/usr/lpp/mmfs/platform/bin/mmdeployssh without any arguments for usage. This is inaddition to starting or stopping sshd via the /etc/init.d/sshd command.

62

http://www-01.ibm.com/support/knowledgecenter/SSGSG7_7.1.1/com.ibm.itsm.tsm.doc/welcome.html

http://www-01.ibm.com/support/knowledgecenter/SSGSG7_7.1.1/com.ibm.itsm.tsm.doc/welcome.html

https://www-947.ibm.com/support/entry/myportal/product/tivoli/tivoli_storage_manager?productContext=-2105539168

https://www-947.ibm.com/support/entry/myportal/product/tivoli/tivoli_storage_manager?productContext=-2105539168



http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster+software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=Windows&function=all



Q2.11: Can different IBM Spectrum Scale maintenance levels coexist?A2.11: Certain levels of IBM Spectrum Scale can coexist, that is, be active in the same cluster and

simultaneously access the same file system. This allows for rolling upgrades. Further, it allows themounting of IBM Spectrum Scale file systems from other clusters that may be running a differentmaintenance level. The current maintenance level coexistence rules are:v All IBM Spectrum Scale V4.2 maintenance levels can coexist with each other and with V4.1

maintenance levels, unless otherwise stated in this FAQ.v All IBM Spectrum Scale V4.1 maintenance levels can coexist with each other and with GPFS

V3.5 maintenance levels, unless otherwise stated in this FAQ.– In order to use the new function in IBM Spectrum Scale V4.2:

- The mmchconfig release=LATEST command must be issued.- To use the Async DR function, all nodes in both clusters must be at a minimum of V4.1.1.

– The fileset snapshot restore function has a coexistence limitation which currently requires allnodes participating in the restore to be at the same level (4.1.0.0, 4.1.0.1, 4.1.0.2, or 4.1.0.3).The workaround is a two-step process: (assuming 4.1.0.3 is installed in at least one node inthe cluster):1. Upgrade the file system manager node to V4.1.0.3 or later2. Only restore a snapshot from a node running V4.1.0.3 or later, by the mmrestorefs -N

command.

Note: This problem does not exist in V4.1.0.4 and later.v All GPFS V3.5 maintenance levels can coexist with each other and with GPFS V3.4

maintenance Levels, unless otherwise stated in this FAQ.– Nodes running GPFS V3.5.0.3 or later, should not mount a remote file system hosted by a

cluster using a tie-breaker disk if any of the nodes in that cluster or that remote-mount thatsame file system are running levels of GPFS code equal to or less than GPFS V3.4.0.14, orGPFS V3.5.0.2.

– Nodes running 3.5.0.3 or later, on a cluster which uses a tie-breaker disk are incompatiblewith nodes from remote clusters which attempt to mount a file system provided by thiscluster, if the remote nodes are running levels of GPFS code equal to or less than GPFSV3.4.0.14, or GPFS V3.5.0.2.

– GPFS V3.5.0.3 to GPFS V3.5.0.6 cannot coexist with previous versions of GPFS V3.5 and allversions of GPFS V3.4. The problem is addressed in GPFS V3.5.0.7. Please upgrade to thatlevel or later.

– GPFS V3.5.0.1 cannot coexist with all versions of GPFS older than V3.4.0.7. Seehttp://www.ibm.com/developerworks/forums/thread.jspa?messageID=14818640 for details.

Q2.12: Are there any requirements for Clustered NFS (CNFS) support in IBM Spectrum Scale?A2.12: IBM Spectrum Scale supports Clustered NFS (CNFS) on SLES and RHEL levels supported by

your version of IBM Spectrum Scale. However there are limitations:v Exporting using NFS V4 is supported starting with IBM Spectrum Scale V4.1 or later.v CNFS over IPV6 is only supported with IBM Spectrum Scale V4.1 or later.v It is very important to make all the nodes in the same group as identical as possible - from the

hardware and software running on them to the configuration of IBM Spectrum Scale, NFS, andthe network.

v CNFS is not supported on the Debian or Ubuntu distribution.v CNFS is not able to export a remotely mounted filesystem.v CNFS lock failover and failback do not work properly on RHEL 6.5 and earlier due to Linux

kernel issues. After replacing /usr/sbin/sm-notify with one of version 1.2.5 or higher, lockfailover and failback will work only if there is no lock competition among clients. If there islock competition, lock failover and failback will not work. Please reference Bugzilla 959006 -infinite loop of resends until the blocked lock is satisfied after server reboot athttps://bugzilla.redhat.com/index.cgi. and https://patchwork.kernel.org/patch/2469651/

v NFS v3 exclusive byte-range locking works properly only on clients of:– x86-64 with SLES 10 SP2 or later, SLES 11, SLES 12, RHEL 5.4 or later, RHEL 6, and RHEL 7– ppc64 with SLES 11, SLES 12, RHEL 5.4 or later, RHEL 6, and RHEL 7

63

http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14818640

https://bugzilla.redhat.com/index.cgi

https://patchwork.kernel.org/patch/2469651/

v Kernel patches are required for distributions prior to SLES 10 SP2 and RHEL 5.2 (no kernelpatches are required for SLES 11 or later, and RHEL5.4 or later) :– If NLM locking is required, until the code is included in the kernel, a kernel patch for lockd

must be applied. This patch is currently available at http://sourceforge.net/tracker/?atid=719124group_id=130828func=browseThe required lockd patch is not supported on RHEL 4 ppc64.

– For SUSE distributions:- portmap must be installed on CNFS nodes- use of the common NFS utilities (sm-notify in user space) is required. The specific

patches required within util-linux are:v support statd notification by name (patch-10113) http://support.novell.com/techcenter/

psdb/2c7941abcdf7a155ecb86b309245e468.htmlv specify a host name for the -v option (patch-10852) http://support.novell.com/

techcenter/psdb/e6a5a6d9614d9475759cc0cd033571e8.htmlv allow selection of IP source address on command line (patch-9617)

http://support.novell.com/techcenter/psdb/c11e14914101b2debe30f242448e1f5d.html– For Red Hat distributions, use of nfs-utils 1.0.7 is required for rpc.statd fixes. Please contact

your Red Hat support representative. Go to https://www.redhat.com/

See also What Linux kernel patches are provided for clustered file systems such as IBM Spectrum Scale?Q2.13: Does IBM Spectrum Scale support NFS V4?A2.13:

Note: NFSv4 can be supported in the following ways:v Clustered NFSv Integrated protocols (CES and NFS), which support NFSv3 and NFSv4.0.

Enhancements to the support of Network File System (NFS) V4 are available on:v AIX V6.1 or AIX V7.1.v The following Linux distributions:

– RHEL 5.5 and later, 6.x, and 7x.– SLES 11 SP1 and later, and SLES 12

Note: Currently, NFS V4.1 is not supported

Restrictions include:v To support NFsv4 ACLs, the package nfs4-acl-tools must be installed.v Windows-based NFSv4 clients are not supported with Linux/NFSv4 servers because of their

use of share modes.v If a file system is to be exported over NFSv4/Linux, then it must be configured to support

POSIX ACLs (with -k all or -k posix option). This is because NFSv4/Linux servers will onlyhandle ACLs properly if they are stored in GPFS as posix ACLs.

v Starting with Linux kernel version 2.6, an fsid value must be specified for each GPFS filesystem that is exported on NFS. For example, the format of the entry in /etc/exports for theGPFS directory /gpfs/dir1 might look like this:/gpfs/dir1 cluster1(rw,fsid=745)

For further details see the Linux export considerations at http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.1.1/com.ibm.spectrum.scale.v4r11.adm.doc/bl1adm_nfslin.htm

v Concurrent AIX/NFSv4 servers, Samba servers and GPFS Windows nodes in the cluster areallowed. NFSv4 ACLs may be stored in GPFS filesystems via Samba exports, NFSv4/AIXservers, GPFS Windows nodes, ACL commands of Linux NFSv3 and ACL commands of GPFS.However, clients of Linux v4 servers will not be able to see these ACLs, just the permissionfrom the mode.

For more information on the support of NFS V4, please see the Spectrum Scale documentationupdates file at

64

http://sourceforge.net/tracker/?atid=719124group_id=130828func=browse


http://support.novell.com/techcenter/psdb/2c7941abcdf7a155ecb86b309245e468.html

http://support.novell.com/techcenter/psdb/2c7941abcdf7a155ecb86b309245e468.html

http://support.novell.com/techcenter/psdb/e6a5a6d9614d9475759cc0cd033571e8.html

http://support.novell.com/techcenter/psdb/e6a5a6d9614d9475759cc0cd033571e8.html

http://support.novell.com/techcenter/psdb/c11e14914101b2debe30f242448e1f5d.html

https://www.redhat.com/

http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.1.1/com.ibm.spectrum.scale.v4r11.adm.doc/bl1adm_nfslin.htm

http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.1.1/com.ibm.spectrum.scale.v4r11.adm.doc/bl1adm_nfslin.htm

v For IBM Spectrum Scale V4.1.1 and later, at http://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html

v For GPFS V3.5 and V4.1, at http://www-01.ibm.com/support/knowledgecenter/SSFKCN/gpfs_welcome.html

Q2.14: Are there any considerations for the use of the Persistent Reserve support in IBM SpectrumScale?

A2.14: Considerations for the use of Persistent Reserve include:v Support for Persistent Reserve requires:

– For V3.5 support on AIX V6.1 requires APAR IZ57224AIX 6.1 TL7 + Service Pack 4 is required to support Persistent Reserve without SDDPCM.

– For V3.5 , V4.1 or V4.2 support on AIX V7.1, refer to the storage documentation to installthe correct multipath driver.

v The use of Persistent Reserve is supported on GPFS tie-breaker disks with GPFS V3.5.0.21, orlater, and IBM Spectrum Scale V4.1.0.4, or later.

Note: See the question What are the current requirements/limitations for using the ClusterConfiguration Repository (CCR) ?

v For the Activate Persist Through Power Loss (APTPL) feature:– On Linux, if the storage is capable of supporting APTPL, GPFS V3.5.0.15, or later supports

this feature.v Starting with 3.5.0.16, it is possible to have a descOnly disk that resides on a device that does

not support SCSI-3 Persistent Reserve while allowing Persistent Reserve to be used on otherdisks in the same file system. The lack of Persistent Reserve support for the descOnly disk willnot result in fast failover being disabled.

Also see the question What devices does GPFS support with SCSI-3 Persistent Reservations?Q2.15: What are the requirements/limitations for using native encryption in IBM Spectrum Scale V4

Advanced Edition or Data Management Edition?A2.15: Considerations for the use of native encryption (encryption of data at rest on GPFS disks) in IBM

Spectrum Scale V4 Advanced Edition include:v The installation and use of either IBM Security Key Lifecycle Manager (ISKLM) V2.5.0.1, or

later, or Vormetric Data Security Manager (DSM) V5.2.3 or later, is required for each node thatacts as a key server. Key server nodes are not required to be members of the IBM SpectrumScale cluster(s) that use them.For ISKLM:– ISKLM is not shipped with nor licensed with IBM Spectrum Scale and must be purchased

separately.– ISKLM V2.5.0.1, or later of the server software (D0887LL) must be installed on each node

that acts as a key server.– One entitlement for IBM Security Key Lifecycle Manager for IBM Spectrum Scale Server

Node Client Device License (D1JWDLL) is required for each GPFS server node that is aNetwork Shared Disk (NSD) node, protocol node, or gateway. Nodes that are licensed asserver nodes that do not serve data (for example, quorum-only nodes or configurationmanagement nodes in remote clusters) do not require an ISKLM license.

– One entitlement for IBM Security Key Lifecycle Manager for Spectrum Scale FPO NodeClient Device License (D1JWFLL) is required for each GPFS FPO node.

– Consult the following link for considerations when using ISKLM V2.7: http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010026.

Note:1. ISKLM is not shipped with nor licensed with IBM Spectrum Scale and must be purchased

separately.2. IBM Spectrum Scale Client nodes do not require an ISKLM license.

For Vormetric:

65





http://www-03.ibm.com/software/products/en/key-lifecycle-manager

http://www.vormetric.com/products/certificate-and-key-vault/data-security-manager



– Vormetric Data Security Manager is not shipped with nor licensed with IBM Spectrum Scaleand must be purchased separately. Contact Vormetric directly to purchase.

– Vormetric Data Security Manager is not supported on nodes installed with Linux on Z.v Every node that accesses the encrypted data must be running the Advanced Edition or Data

Management Edition of IBM Spectrum Scale.v Every node that accesses the encrypted data must have network connectivity to the key server.v IBM Spectrum Scale Client nodes do not require a key server license.

Current limitations of the IBM Spectrum Scale encryption function include:v Only user data is encrypted. The encryption of directories or other metadata is not supported.v Extended attributes are not encrypted.v Data, which is backed up, is in cleartext unless encryption is supported by the backup system.v Data, which is migrated to tape using software such as IBM Spectrum Protect or IBM Spectrum

Archive, is in cleartext unless the tape system and the connection between them (if Ethernet orInfiniBand) provide encryption.

v Encryption is not supported on Windows. The encryption function should be disabled whenWindows nodes are in the cluster.

v FIPS mode is supported on the POWER8 processor in little endian mode in IBM SpectrumScale V4.2.1 and later.

v Booting an RHEL 7.0 node in FIPS mode is not supported.v FIPS mode is not supported on IBM z14 and LinuxONE Emperor II.v The contents of encrypted files are placed into a local read-only cache (LROC) based on the

settings of the lrocEnableStoringClearText configuration option. For more information, see the"Encryption and local read-only cache (LROC)" section.

v LROC is not supported on Linux on Z.Q2.16: Are there any considerations when utilizing the Simple Network Management Protocol

(SNMP)-based monitoring capability in IBM Spectrum Scale?A2.16: Considerations for the use of the SNMP-based monitoring capability include:

v The SNMP collector node must be a Linux node in your GPFS cluster. GPFS utilizes Net-SNMPwhich GPFS does not support on AIX.

v Support for ppc64 requires the use of Net-SNMP 5.4.1. Binaries for Net-SNMP 5.4.1 on ppc64are not available. You will need to download the source and build the binary. Go tohttp://net-snmp.sourceforge.net/download.html

v If the monitored cluster is relatively large, you need to increase the communication time-outbetween the SNMP master agent and the GPFS SNMP subagent. In this context, a cluster isconsidered to be large if the number of nodes is greater than 25, or the number of file systemsis greater than 15, or the total number of disks in all file systems is greater than 50. For moreinformation see Configuring Net-SNMP in the IBM Spectrum Scale: Advanced AdministrationGuide.

v SNMP-based monitoring has not been tested in clusters composed of more than 127 nodes.Q2.17: What are the current limitations and advisories for using the mmbackup command?A2.17: Current limitations and advisories include:

v Beginning with IBM Spectrum Scale V4.2, in file systems that are managed by an HSM system,mmbackup will skip over candidates for backup that are migrated offline to avoid causing arecall storm. Instead, records for these files will be added to a file in the root of the filesetcalled mmbackup.hsmMigFiles.name of server. System managers should recall these changedfiles online to allow mmbackup to properly protect them in the next invocation.

v File systems with IBM Spectrum Protect for Space Management that have unlinked filesets,will be required to link all filesets when issuing the mmbackup command for the first timeafter you upgrade GPFS cluster from GPFS 3.5.0.10 or lower, to GPFS 3.5.0.11 or higher. If youhave any concerns regarding this requirement, please contact GPFS service.– In the United States contact us toll free at 1-800-IBM-SERV (1-800-426-7378)– In other countries, contact your local IBM Service Center

v Use of the IBM Spectrum Protect Backup-Archive client option SKIPACLUPDATECHECK withthe mmbackup command requires IBM Tivoli Storage Manager release 6.4.1.0 or later.

66

http://net-snmp.sourceforge.net/download.html




Note: Beginning with Version 7.1.3, IBM Tivoli Storage Manager is now IBM Spectrum Protect.v The GPFS mmbackup command is not integrated with the IBM Spectrum Protect for Space

Management-Multi HSM Server feature. See Managing a file system with multiple Tivoli StorageManager servers.

v Restoring a file via a node that has a different architecture than the one used to do the backupcould cause the associated ACL to be corrupted.For example, if a file was backed up using an x86_64 node, and then restored using a ppc64node, this could cause its ACL to be corrupted (caused by differences in endianness of thearchitectures which is not supported by the GPFS APIs used in the restore operation). It isrecommended that backup and restore operations be done on similar types of nodes.

v The mmbackup command supports backup of a whole file system from a global snapshot.The mmbackup -S snapshot command option is supported with IBM Spectrum Scale V4.1.1 oneither a global snapshot for the whole file system or for a fileset backup if the fileset wascaptured in that snapshot. It is also supported with a fileset snapshot for a fileset backup,providing the name of the snapshot is unique among all snapshot names. Do not use the samesnapshot name for multiple snapshots.The mmbackup -S snapshot command option is supported with GPFS V3.5.0.3 or later. GPFSV3.4 and GPFS V3.5.0.2 or lower do not support backup from a snapshot. If the snapshotdirectory for global snapshots and the directory for fileset level snapshots are different, thenGPFS V3.5.0.4 or higher level is required.– Doing backup from a snapshot in an IBM Spectrum Protect for Space Management managed

file system could cause recall of migrated files.In an HSM managed file system such as IBM Spectrum Protect for Space Management,using mmbackup to back up from a snapshot could cause the recall of migrated files if themigration was done after the snapshot was taken. This is due to the fact that a snapshot is astatic view of the file system which does not reflect migration state changes. To avoidrecalling data from migrated files, create the snapshot and complete the backup operationbefore migrating files or make sure that migration is done before the snapshot is taken for abackup operation. Backup operations will not recall files if the snapshot captured the files intheir migrated state. Until and unless the migrated file stubs are removed from the live filesystem. In this case a recall will be required to populate the contents of the snapshot view ofthe files. If a snapshot exists, consider recalling files by using the IBM Spectrum Protect forSpace Management "tape optimized recall" function before deleting migrated files from theactive file system.

– Doing backup from a snapshot in an IBM Spectrum Protect for Space Management managedfile system could cause failure to backup migrated files.

v The use of unsupported characters in the names of files or directories will cause failures.mmbackup uses the IBM Spectrum Protect Backup Archive client to backup data to the IBMSpectrum Protect server. As IBM Spectrum Protect currently does not support all specialcharacters in file or directory names, they cannot be supported by mmbackup. If specialcharacters are used in the names of files or directories backed up by the mmbackup command,failures will result. Known special characters which can cause problems include: *, ?, ", ',control-X, control-Y, carriage return and the new line character. Use of the IBM SpectrumProtect options QUOTESARELITERAL and WILDCARDSARELITERAL along with the--noquote command line option to mmbackup will allow support for all special charactersexcept carriage return, new line, control-X, and control-Y.

v Beginning with IBM Spectrum Scale V4.1.1, backup of either the entire file system or a selectedfileset is supported. Nested fileset arrangements where one fileset is linked inside another arenot supported by mmbackup on a fileset. Nesting remains supported for whole file systemmmbackup. The first mmbackup of any fileset must be made using the option -t full to avoidcausing accidental invalidation of existing backups that may exist of previously existing nestedfilesets.

v Differences in the way the mmbackup command and IBM Spectrum Protect process theinclude and exclude statements in the dsm.sys configuration file may cause files or directoriesto be included or excluded unexpectedly.

67

http://www-01.ibm.com/support/knowledgecenter/SSGSG7_7.1.0/com.ibm.itsm.hsmul.doc/c_multi_ovw.html

http://www-01.ibm.com/support/knowledgecenter/SSGSG7_7.1.0/com.ibm.itsm.hsmul.doc/c_multi_ovw.html

Known differences in processing include, but are not limited to:– The mmbackup command does not support exclude.archive, exclude.file.spacemgmt,

exclude.spacemgmt, exclude.fs.– Whether or not there is a / at the end of an exclude.dir affects the way mmbackup decides

what files or directories are excluded.– exclude.file may cause incorrect files to be backed up if the pattern presents a wildcard at

the end.Q2.18: What are the current limitations and advisories for using Scale Out Backup and Restore

(SOBAR) ?A2.18: Current limitations and advisories include:

v IBM Spectrum Scale Image Backup and Restore (SOBAR) has been tested in a standalonemanner; but must be tested with Data Management/HSM products before deployment bycustomers with such products in production environments. Customers who are interested inmaking use of this function should contact [email protected].

v SOBAR does not support the backup or restore of a file system with Active File Management(AFM) filesets

v SOBAR supports backup from a global snapshot. Independent fileset snapshots are notsupported.

v SOBAR is dependant on the IBM Spectrum Protect for Space Management V6.4.0.1, or later .v SOBAR is not supported on ppc64 because there are no qualified HSM systems currently

supported on ppc64.Q2.19: What are the current limitations for using the File Placement Optimizer (FPO) function?A2.19: Current limitations include:

v The File Placement Optimizer (FPO) function is supported on IBM Spectrum Scale GPFS V4and V3.5 for both the Linux and AIX operating systems. For V3.5, the Linux (x86 and Power)operating system requires APAR IV28687 and the AIX operating system requires APARIV40108.

v AFM ADR (primary/secondary filesets) is not supported on an FPO enabled file system.v With the AFM function, if you want to maintain data locality on both home and cache, they

must have the same Failure Group configuration. Additionally, block placement policy must beset via write-affinity-failure-group at both sites.

v Twin-tailed disks are supported in an FPO pool only when a single NSD server is defined foreach disk.

v The FPO function is not supported on the Debian distribution or Linux on IBM Z.v Nodes running the GPFS File Placement Optimizer feature cannot coexist or interoperate with

nodes running GPFS V3.4 or earlier releases of GPFS.v The architectural scaling limit for the FPO function is thousands of nodes. However, the tested

limit is lower. Contact [email protected] if you plan to deploy a cluster larger than 732 nodes.v If a storage pool is FPO-enabled (allowWriteAffinity=yes), then layoutMap=cluster must also

be specified.v With GPFS V3.5, use of the mmrestripefile, mmadddisk -r and the mmrestripefs commands

will break the original FPO file's placement.v With GPFS V4.1, use of the mmrestripefile -b, mmadddisk -r and the mmrestripefs -b

commands will break the original FPO file's placement.v With GPFS V4.1, use of the mmrestripefile -r and the mmrestripefs -r commands is supported

with locality awareness. Use of the commands with clones and snapshots will break theoriginal FPO file's placement.

v On clusters with the FPO function enabled, in order to utilize the mmrestorefs command, youmust specify the write-affinity-failure-group policy.

v If the size of a file is less than the value of the blocksize divided by 32, the write affinity depthpolicy and the write affinity failure group policy will not be followed. Data is widely stripedinstead.

v The setXattr function cannot set the FPO extended attributes writeAffinityDepth,write-affinity-failure-group, and BlockGroupFactor for a clone file or the policy MIGRATErule. Respectively, setWAD, setWADFG, and setBGF should be used.

68



v The extended attributes writeAffinityDepth, write-affinity-failure-group, andBlockGroupFactor are for use only on an FPO pool.

Q2.20: What are the current limitations for using the Active File Management (AFM) Async DRfunction?

A2.20: Async DR specific limitations include:v Our initial feedback from the field suggests that success of a disaster recovery solution

depends on administration discipline, including careful design, configuration and testing.Considering this, IBM has decided to disable the Active File Management- based AsynchronousDisaster Recovery feature (AFM DR) by default and require that customers deploying the AFMDR feature first review their deployments with IBM Spectrum Scale development. You shouldcontact Spectrum Scale Support at [email protected] to have your use case reviewed. IBM willhelp optimize your tuning parameters and enable the feature. Please include this messagewhile contacting IBM Support.

Note: This limitation does not apply to base AFM support. This applies only to Async DRavailable with the IBM Spectrum Scale Advanced Edition V4.1.1 and later.Have this information available when requesting a development led feasibility assessment forAsync DR related proposals.– Detailed use case description including any application description for applications running

in DR filesets.– Network latency and bandwidth between sites.– Scale of the solution including the number of filesets, number of files, type of files (small or

large), any special file usage (clones or encryption).– RPO time.– Rate of change of data that needs to be transferred between two RPOs from Async DR

primary site to secondary.– Number of Async DR filesets.– Any plans to use snapshots in Async DR filesets.

v If you change an AFM DR Secondary export from Kernel NFS or CNFS to the new IBMSpectrum Scale CES in 4.1.1 or later, AFM will not recognize the secondary. The best way tomanage this situation is to run the mmafmctl changeSecondary command. See the IBM SpectrumScale documentation for more details.

v DR administration using the mmafmctl command is not supported when run from the AIX OS.v AFM ADR (primary/secondary filesets) is not supported on an FPO enabled system.v AFM features such as revalidation, eviction, prefetch, show home snapshots, partial file

caching, expire and unexpire, mmafmctl resync and failover, and IW failback are not applicablefor AFM DR filesets.

v Administrative operations such as fileset conversion to primary, failover, and failback can berun at the same time on multiple filesets; however, it is not recommended because commandsmay fail with a "busy" error in cases where the system runs out of the resources that arerequired to perform multiple operations at the same time. If this happens, rerun the failedcommands after the conflicting command completes.

Q2.21: What are the current limitations for using the Active File Management (AFM) function?A2.21: AFM limitations include:

v If you change an AFM SW/IW home export from Kernel NFS or CNFS to the new IBMSpectrum Scale CES in 4.1.1 or later, AFM will not recognize the home. The best way tomanage this situation is to run the mmafmctl failover command. See the IBM Spectrum Scaledocumentation for more details.

v AFM RO/LU home export cannot be changed from Kernel NFS or CNFS to new SpectrumScale CES in 4.1.1. or later.

v Customers who have enabled DMAPI/HSM at the AFM home cluster should be running atRHEL 6.3, or later, or SLES 11 SP2, or later, on the cache cluster. There has been a bugidentified in earlier levels of RHEL, which requires the exported path at home to be excludedfrom DMAPI/HSM. See http://www-01.ibm.com/support/docview.wss?uid=ssg1S1004310 .

v Code upgrade limitations:

69



– If there are AFM filesets using the Kernel NFS backend to communicate with a GPFS homethat is running V4.1 or earlier, upgrading home to IBM Spectrum Scale V4.1.1 or later willcause the AFM filesets to disable synchronization to home with a message as follows:GPFS: 6027-3218 Change in home export detected. Synchronization with homeis suspended until the problem is resolved

Note: This only affects AFM filesets that were originally created with a GPFS home runningV4.1 or earlier. This does not affect AFM filesets that were created with a home running IBMSpectrum Scale V4.1.1 or later. This issue is fixed in 4.1.1.4 (IBM Fix central for 4.1.1.4) and4.2.0.1 (IBM Fix central for 4.2.0.1) available at Fix Central.

v Creating an AFM association to the same local file system is not supported using the GPFS™

protocol.v In the current phase, due to a known issue with AFM SMB integration, the cache cluster might

experience re-queued messages on gateway nodes. As a result, SMB clients might experience aperformance impact on IOs.

Q2.22: What are the current limitations common to both the Active File Management (AFM) and AFMDR functions?

A2.22: The common limitations for both AFM and AFM DR functions include:v The OS/architecture support matrix is given below:

Table 41. Availability and OS/architecture supported

Arch/OS Fileset Supported Not Supported

Linux for Power Systemsand x86

AFM SW/IW/RO/LUAFM primary/secondary

GatewayApplicationFilesystem Manager

Linux on Z AFM SW/IW/RO/LUAFM primary/secondary

GatewayApplicationFilesystem Manager

AIX AFM SW/IW/RO/LUAFM primary/secondary

ApplicationFilesystem Manager

Gateway AIX as NFS serverat secondary

v 3rd party NFS servers at home/secondary:– Unsupported for AFM DR secondary.– Untested for all AFM modes. The IBM Spectrum Scale support team will help customers

who are using other file systems at home to solve problems directly related to GPFS orSpectrum Scale, but will not be responsible for solving problems deemed to be issues withthe other file systems.

v AIX NFS servers at home/secondary:– Unsupported for AFM DR secondary.– Tested and supported for all AFM modes, but running mmafmconfig on the home cluster is

not supported, which means, EA's, ACLs, sparseness of files, and psnap will not besupported.

v The mmclone command is not supported on AFM cache and AFM DR primary filesets. Clonescreated at home for AFM filesets are treated as separate files in the cache.

v Quality of Service and Compression are not tested on AFM and AFM Async DR filesets.v AFM and AFM DR is not tested with IPv6 on gateway nodes.v AFM and AFM DR filesets on Linux for Systems Z is not supported with HSM.v AFM and AFM DR filesets on Linux for Systems Z is not supported with the NSD protocol for

communication.v Connecting AFM cache or home or AFM primary or secondary to IBM Spectrum Archive is not

supported.v Linking of a dependent or an independent fileset is not allowed inside the inode space of any

AFM or AFM DR fileset (in other words, linking them under the junction path within thefileset is not allowed).

v AFM and Async DR is not supported on clusters that have Windows nodes.v Cascading relationships with AFM caches and AFM primary filesets are not tested.

70

http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=All&function=all


v Fileset snapshot restore is not supported for AFM and AFM DR filesets.v Parallel data transfers will not work if the home/secondary NFS server is AIX.v Immutability and appendonly features are not supported on AFM and AFM DR filesets.v AFM and AFM DR filesets do not replicate security, system, or trusted extended attributes.v AFM and AFM DR are known to have compatibility issues with SMB oplocks. These issues

might result in denied or delayed locks for AFM, which would cause delayed synchronizationof the data to the secondary site.

v dm_write_invis() is not supported in AFM and AFM DR filesets.v dm_read_invis() is not supported in AFM and AFM DR filesets if the file is not already cached.v The file audit logging feature is not supported on AFM or AFM DR filesets.

Q2.23: What is the currently recommended transport protocol for AFM and AFM DR data transfers?A2.23: For the current recommendations regarding the transport protocol for AFM and AFM DR data

transfers, please see http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1ins_NFSvsGPFSAFM.htm.

Q2.24: What is the current limitation for using the snapshot restore function?A2.24: For mixed clusters containing GPFS V3.5, IBM Spectrum Scale 4.1.0, 4.1.0.4, 4.1.1, or V4.2 nodes, it

is recommended that for both fileset and global snapshot restore, the mmrestorefs command isissued from a node running the most recent version of Spectrum Scale. Version 4.1.1 and later,provides some new snapshot restore functionality so IBM Spectrum Scale attempts to intelligentlyuse the latest features.

Note:1. IBM Spectrum Scale V4.2 only interoperates with IBM Spectrum Scale V4.12. The mmrestorefs command is not supported on the Express Edition.v To improve performance support for the –N parameter on the mmrestorefs command has been

phased in over the past few releases.– In GPFS 3.5 and earlier there is no –N parameter– In GPFS 4.1 –N can be used for fileset snapshot restore only– In IBM Spectrum Scale 4.1.1 and later –N can be used for both fileset and global snapshot

restoresv For global snapshot restore:

– If the mmrestorefs command is issued from a pre-4.1.1 node:1. The file system must be unmounted.2. The file system manager performs the restore.

– If the mmrestorefs command is issued from a 4.1.1 or later node:- The file system must be mounted.- By default the restore is performed on all nodes running the latest level of code.

v For fileset snapshot restore:– If the mmrestorefs command is issued from a V3.5 node:

1. The file system must be unmounted.2. The file system manager performs the restore.

– If the mmrestorefs command is issued from a 4.1 or later node:- The file system must be mounted.- By default the restore is performed on all nodes running the latest level of code.

Q2.25: What are the current requirements when using local read-only cache?A2.25: The current requirements/limitations for using local read-only cache include:

v A minimum of IBM Spectrum Scale V4.1.0.1.v Local read-only cache is only supported on Linux x86 and Power.v The minimum size of a local read-only cache device is 4 GB.v The local read-only cache requires memory equal to 1% of the local read-only device's capacity.

Note: Use of local read-only cache does not require a server licenseQ2.26: What are the current requirements/limitations for using the Cluster Configuration Repository

(CCR) ?A2.26: The current requirements/limitations for using the Cluster Configuration Repository (CCR)

include:

71

|||

http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1ins_NFSvsGPFSAFM.htm

http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1ins_NFSvsGPFSAFM.htm

v IBM Spectrum Scale V4.1.0: The Disaster Recovery procedures described in the AdvancedAdministration Guide are not supported in a cluster with CCR enabled:– Do not run mmchcluster --ccr-enable for existing clusters– Use mmcrcluster --ccr-disable for new clusters

v IBM Spectrum Scale V4.1.1, or later, there are no limitations for using CCR.Q2.27: What are the current requirements/limitations for using Ubuntu?A2.27: The current requirements/limitations for using Ubuntu include:

v The minimum level of Ubuntu supported is 14.04.1.v Only IBM Spectrum Scale 4.1.0.8 or later, is supported with 14.04.2.v Only IBM Spectrum Scale 4.1.1.9/4.2.1.1 or later is supported with 14.04.4/16.04.v The minimum kernel level supported is 3.13.0.32.v P8 is supported with Little Endian only and only with GPFS for Linux on System p base RPMs

dated January 2015 (GPFS V4.1.0.5 or later).If you have Software Maintenance Agreement (SWMA) for your products ordered throughAAS/eConfig or IBM Subscription and Support (S&S) for orders placed through PassportAdvantage, you may log into the respective systems and upgrade your level of GPFS:– For products ordered through AAS/eConfig, please log into the Entitled Software page at:

https://www-05.ibm.com/servers/eserver/ess/OpenServlet.wss– For products ordered through Passport Advantage, please log into the site at:

http://www.ibm.com/software/lotus/passportadvantage/v GPFS V3.5.0.22 or later is only supported on x86_64 architecturev When issuing make World for Ubuntu 14.04.1, this warning will appear but can be disregarded

because kdump-kern-dummy.ko is not utilized by GPFS.

WARNING: ".TOC." [/usr/lpp/mmfs/src/gpl-linux/kdump-kern-dummy.ko] undefined!v As Tivoli Storage Manager (TSM) does not support Ubuntu, GPFS commands that utilize TSM

are not supported on Ubuntu.

Note: Beginning with Version 7.1.3, IBM Tivoli Storage Manager is now IBM Spectrum Protect.v If you use CNFS or CES features under Ubuntu, verify that the iputils-arping package is

installed. See the Software requirements page for more information.Q2.28: What are the current requirements/limitations for IBM Spectrum Scale for Linux on Z?A2.28: The current requirements/limitations for IBM Spectrum Scale for Linux on Z include:

Note: For a complete overview of supported functions:v For V5.0.0, see the Table 15 on page 17 or Table 16 on page 20v For V4.2.3, see the Table 17 on page 23 or Table 18 on page 26v For V4.2.2, see the Table 19 on page 28 or Table 20 on page 31v For V4.2.1, see the IBM Spectrum Scale V4.2.1 Functional Support Matrixv For V4.2, see the IBM Spectrum Scale V4.2 Functional Support Matrixv For V4.1.1, see the IBM Spectrum Scale V4.1.1 Functional Support Matrixv IBM Spectrum Scale for Linux on Z supports all editions (Express, Standard, and Advanced)

– Beginning with the Advanced Edition of IBM Spectrum Scale for Linux on Z V4.2.1,encryption of data at rest is supported.

Note: Central Processor Assist for Cryptographic Function (CPACF) is supported. CPACF isIBM Z hardware encryption acceleration.

It is incorporated in the central processors that are shipped with IBM Z. To benefit from theCPACF, you must install LIC internal feature 3863 (Crypto Enablement feature), which isavailable free of charge. By default, IBM Z is delivered to customers without this featureunless it is ordered explicitly by the customer. The installation of this feature at a future timeis nondisruptive.

– The Advanced Edition of IBM Spectrum Scale for Linux on Z includes AsynchronousDisaster Recovery (Asynch DR)

v Functions are limited to Server or Client licenses. File Placement Optimizer is not supported.

72



v Support of backup and restore functions with IBM Spectrum Scale V4.2 or later Standard andAdvanced Editions:– For IBM Spectrum Scale V4.2, IBM Spectrum Protect backup-archive and Space Management

client are only supported starting with Spectrum Protect V7.1.4 running 64-bit versions ofRHEL 7.1 and SLES 12.

– For IBM Spectrum Scale V4.2.1 or later, IBM Spectrum Protect backup-archive and SpaceManagement client are only supported starting with Spectrum Protect V7.1.6 or laterrunning 64-bit versions of RHEL 7.1 or later and SLES 12 or later.

– The Spectrum Scale policy engine is utilized to rapidly identify changed files and usesparallel execution of backup processes on multiple nodes to deliver scalability of the dataprotection process.

– Space Management functions extend the size of a Spectrum Scale file system beyond thedisk-based storage pools by defining external storage pools on sequential access media andmigrating seldom used data.

– For IBM Spectrum Scale V4.2.3.x only, IBM Spectrum Protect backup-archive and SpaceManagement client are supported with Spectrum Protect V8.1.2.

v Regarding applications supporting IBM Spectrum Scale for Linux on Z, refer to the respectiveapplication's documentation.

v IBM Z firmware MCLs N49686.003 and 004, which are provided in [D15 Bundle 30], must beinstalled on EC12 machines to prevent [machine checks/disable waits] on zGPFS (LPAR andz/VM) images.

v Supported with IBM Systems Storage DS8000, IBM Storwize V7000, IBM FlashSystem, IBM XIV,IBM SVC, IBM ESS, HDS, or EMC (see Disk hardware tested with IBM Spectrum Scale for Linux onZ) . Also, see the question Does IBM Spectrum Scale for Linux on Z support Direct Attached StorageDevices (DASD)?

v Support for stretched cluster with synchronous mirroring utilizing block-level replication:– For V4.2.1 and later, with distances up to 300 km.– For V4.2 and previous, with distances less than 100 km.

v Cluster Export Service (CES) is not supported. (Monitoring capabilities, Object, CIFS, Userspace implementation of NFS)

v Kernel NFS (v3 and v4) is supported. Clustered NFS is not supported.v Prior to V4.2, running several admin commands in a scripted way consecutively, there is a

possibility to get a command hang due to a 100% CPU consumption of a mmccr process. Thehang process can be found by using the ps or top command. In such a case it was observedthat this process disappeared after a short while but in some cases it remains for more thanone hour. It is possible to get out of this hang, by killing this mmccr process. This results in afailure of the last executed admin command.

Note: The limitation on z14 support in IBM Spectrum Scale V5 has been lifted. For moreinformation, see the question What are the minimum hardware requirements for a IBM SpectrumScale cluster?

Q2.29: What is the Highly-Available Write Cache (HAWC) function?A2.29: The Highly-Available Write Cache (HAWC) function available with IBM Spectrum Scale V4.1.1.1

or later, reduces the latency of bursts of small write requests by buffering them in fast storagesuch as SSDs.

Note: If you plan on using the HAWC function on client nodes, V4.1.1.2 is required.

The HAWC function can benefit numerous applications such as VMs, appending to logs andmany more. If a file system's metadata is already stored on fast storage such as SSDs, then thefeature can be simply enabled with very little effort. If not, then a new 'fast storage' pool must becreated on either one or more NSD servers or on the clients themselves. HAWC is controlled viathe file system parameter write-cache-threshold and can also be used with existing as well as newfile systems. For more information, see the IBM Spectrum Scale Advanced Administration Guide athttp://www-01.ibm.com/support/knowledgecenter/STXKQY/411/com.ibm.spectrum.scale.v4r11.adv.doc/bl1adv_hawc.htm?lang=en

73

http://www-01.ibm.com/support/knowledgecenter/STXKQY/411/com.ibm.spectrum.scale.v4r11.adv.doc/bl1adv_hawc.htm?lang=en

http://www-01.ibm.com/support/knowledgecenter/STXKQY/411/com.ibm.spectrum.scale.v4r11.adv.doc/bl1adv_hawc.htm?lang=en

Q2.30: What are the current requirements/limitations for the deadlock amelioration function in IBMSpectrum Scale?

A2.30: The current requirements/limitations for use of the deadlock amelioration function include:v Deadlock amelioration functions are fully supported in IBM Spectrum Scale V4.v In a cluster with minReleaseLevel below 4.1.0, that consists of all GPFS 4.1 nodes or a mixture

of 4.1 and 3.5 nodes, the deadlock amelioration functions may still work partially. In order toavoid a problem of tracing not being turned off after GPFS code turns it on make sure to have3.5.0.24 or later, or 4.1.0.7 or later, or have APAR IV69797 applied to all nodes. Running withtracing on could have performance implications.

Q2.31: What are the requirements/limitations for using the IBM Spectrum Scale GUI?A2.31: Considerations for using the IBM Spectrum Scale GUI include:

v The GUI is available with the Standard and Advanced Editions for Linux on x86, Linux on Z,and Power (Big Endian and Little Endian).

v The GUI is supported on RHEL 7.1 or later, SLES 12 SP1 and SP2, and Ubuntu 16.4 on Linuxx86, Linux on Power, and Linux on z platforms. For more information, see the Table 15 on page17 and the Table 16 on page 20.

v The maximum number of nodes supported is:– With V4.2.1 and later, 1000 nodes– With V4.2, 128 nodes

v When planning to add GUI nodes with the Installation Toolkit, add them via spectrumscaleinstall or spectrumscale deploy, either before performing an upgrade to 4.2.0.1 or later.Attempting to add GUI nodes during the upgrade itself may result in a failure during theUpgrading Performance Monitoring step.

v The GUI works with either a client or server license.v The GUI depends on a pre-installed PostgreSQL server usually installed already with the

operating system installation but if this is not the case if you install the operating system fromscratch you need to do this before you install the GUI or the installation fails.

Q2.32: What are the requirements/limitations for using the compression function?A2.32: Compression support excludes the following:

v Compressing files in snapshots.v Compressing clones and cloning compressed files.v Compressing files on AFM cache site and AFM-based asynchronous DR filesets.v Small file compression (files consuming less than 2 sub-blocks, compressing small files into

inode).v Compression of non-regular files, such as directories.v Compression of files in Windows hyper allocation mode.

Additionally:v Compression is supported in an FPO environment or horizontal storage pools with IBM

Spectrum Scale V4.2.1 and later.v Compression in this release is optimized for cold data or write-once objects and files. It uses

the zlib data compression library and favors saving space over speed. Usage on other types ofdata may result in performance degradation.

v On Windows:– Compression of a file on Windows is only enabled via the mmchattr command.– The following Windows APIs are not supported:

- FSCTL_SET_COMPRESSION to enable/disable compression on a file- FSCTL_GET_COMPRESSION to retrieve compression status of a file

– In Windows Explorer, in the Advanced Attributes window, the compression feature is notsupported.

Q2.33: What are the requirements/limitations for using the Quality of Service (QoS) function?A2.33: Considerations for using the Quality of Service function include:

v Only two classes are supported: maintenance and other.v All user applications are being placed in the same class. There is no support for different

applications being throttled differently.v For Linux on Z, Quality of Service is only supported with V4.2.1 and later.

74

v No throttling for applications which perform direct I/O.v Not supported on AFM cache and AFM-based asynchronous DR filesets.v For IBM Spectrum Scale V4.2.1 and later, QoS is supported in an FPO environment.v A Flash for QoS

Abstract:In an IBM Spectrum Scale V4.2 file system with multiple storage pools, Quality ofService (QoS) settings should be set for all storage pools to avoid performancedegradation for unspecified storage pools.

Problem Summary:In an IBM Spectrum Scale V4.2 file system with multiple storage pools, if the userspecifies Quality of Service for I/O operations (QoS) settings (for the maintenance andother classes) only for one storage pool then the I/O allocations for the unspecifiedpools will be set to a very low value, resulting in severe performance degradationwhen I/O is performed to the unspecified storage pool(s).

See the complete Flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1005464Q2.34: What are the considerations when running on SELinux?A2.34: Considerations when running on SELinux include:

v For file system support, see the IBM Spectrum Scale wiki article on Using GPFS with SELinux athttps://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Using%20GPFS%20with%20SElinux

Note: SELinux in enforcing mode is not supported by other Spectrum Scale components.v When using the Installation Toolkit, the IBM Spectrum Scale Object protocol functionality

requires the following SELinux packages to be installed:– selinux-policy-base at 3.13.1-23 or higher– selinux-policy-targeted at 3.12.1-153 or higher

v When utilizing Object Protocol functionality, enabling SELinux after Spectrum Scale has beeninstalled is not supported. Please contact Spectrum Scale Support at [email protected] if youhave questions about this restriction

Q2.35: What enhancements to quota management are available?A2.35: Enhancements to quota management available with IBM Spectrum Scale V4.1 and later, allow for

quota clients to dynamically acquire and relinquish quotas based upon consuming rate; that is,the quota manager grants quota shares based upon global quota information such as theremaining quota limit and the number of mounted clients. Decisions based upon existinginformation provides for greater efficiency in managing quotas, however, this could result in anincrease in the in-doubt values from earlier releases when running with a heavy IO workload. Inorder to get more accurate usage, issuing the mmlsquota and mmrepquota commands with the -eoption might help.

Note: Quota report accuracy is mainly affected by hardware errors, such as node/networkfailures. Having a large number of nodes could increase the chances of having these types offailures.

Q2.36: What are the current limitations of quota management?A2.36:

The aggregate total number of quota records in user, group, and fileset quota files is limited to200K records per file system. This limitation is due to the maximum amount of data that can beexchanged between the quota manager and quota command client such as the mmrepquotacommand.

Note: A large number of quota records per file system can result from the following scenarios:v There are a very large number of users, groups, or filesets.v If the --perfileset-quota option is enabled, the number of possible quota records is the

number of filesets times number of users (and groups).Q2.37: What are the limitations of the Snapshot ILM feature?

75

http://www.ibm.com/support/docview.wss?uid=ssg1S1005464

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Using%20GPFS%20with%20SElinux

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Using%20GPFS%20with%20SElinux


A2.37: The Snapshot ILM feature, supported on file system version 4.2.2 or later and on new snapshots,has not been tested with AFM and AFM DR filesets.

76

Machine questionsQ3.1: What are the minimum hardware requirements for an IBM Spectrum Scale cluster?A3.1: The minimum hardware requirements are:

v IBM z14 and LinuxONE Emperor II are dependent on the IBM Spectrum Scale version used:– IBM Spectrum Scale V4.2.3 PTF 6 or later– IBM Spectrum Scale V5.0.0 or later

v IBM Spectrum Scale for Linux on Z support is dependent upon the OS in use:– SLES 11 is supported on System z9®, System z10™, and zEnterprise®

– SLES 12 is supported on zEnterprise– RHEL 6 is supported on System z9, System z10, and zEnterprise– RHEL 7 is supported on zEnterprise

For more details, see the System z® test matrix at http://www-03.ibm.com/systems/z/os/linux/resources/testedplatforms.html

v IBM Spectrum Scale V4/IBM GPFS V3 running on POWER:– AIX on POWER is supported by IBM POWER5 processors, or higher processors , supported

by your level of AIX, with a minimum of 2 GB of system memory.– Linux on POWER is supported on IBM POWER5, or higher, processors, with a minimum of

2 GB of system memoryv IBM Spectrum Scale V4/IBM GPFS V3 on x86 Architecture is supported on:

– Intel EM64T processors, with 2 GB of memory– AMD Opteron processors, with 2 GB of memory– Because of their sometimes unique communications fabric, or other elements of the system

architecture, support of GPFS in any Cray system environment requires GPFS developmentreview and approval. Please contact [email protected] to arrange for such a review.

Additionally, it is highly suggested that a sufficiently large amount of swap space is configured.While the actual configuration decisions should be made taking into account the memoryrequirements of other applications, it is suggested to configure at least as much swap space asthere is physical memory on a given node.

IBM Spectrum Scale is supported on systems which are listed in, or compatible with, the IBMhardware specified in the Hardware requirements section of the Sales Manual for IBM SpectrumScale.

To access the Sales Manual:1. Go to http://www.ibm.com/common/ssi/index.wss2. On Information Type, choose HW&SW Desc (sales manual,RPQ).3. To view a IBM Spectrum Scale V4 sales manual, choose the corresponding product number to

enter in the search field:v For orders placed through the AAS fulfillment system, enter 5641-GPFv For orders placed through Passport Advantage, enter 5725-Q01v For orders for GPFS for Linux on Z, enter 5725-S28

4. To view a GPFS V3 sales manual, choose the corresponding product number to enter in thesearch field:v For General Parallel File System for POWER, enter 5765-G66v For General Parallel File System x86 Architecture, enter the appropriate order number;

either 5724-N94, 5765-XA3, 5641-A07, 5641-B07, or 5641-C075. The Hardware Requirements section which is part of the Technical Description section.

Q3.2: On what servers is IBM Spectrum Scale supported?A3.2:

1. IBM Spectrum Scale for Linux on Z is supported:a. with the distributions and kernel levels as listed in the question What Linux distributions are

supported by IBM Spectrum Scale?b. on servers that meet the minimum hardware model requirements as listed in the question

What are the minimum hardware requirements for a IBM Spectrum Scale cluster?

77

http://www-03.ibm.com/systems/z/os/linux/resources/testedplatforms.html

http://www-03.ibm.com/systems/z/os/linux/resources/testedplatforms.html



c. IBM Spectrum Scale for Linux on Z does not rely on the z/VM versions as long as thez/VM can support the required Linux distributions, kernel levels and hardware.

d. z/VM V6.1 and later will operate on: IBM zEnterprise System (zEC12, zBC12, z196 orz114), IBM System z10 or equivalent server.

e. Customers using older hardware (IBM System z9, IBM eServer™ zSeries 990 (z990), 900(z900), 890 (z890), and 800 (z800) servers ) have to use z/VM 5.4.

2. IBM Spectrum Scale for AIX is supported:a. with levels of AIX as listed in the question What levels of the AIX OS are supported by IBM

Spectrum Scale ?b. on servers that meet the minimum hardware model requirements as listed in the question


Note: IBM Spectrum Scale runs on the POWER8 processor in default, POWER8, andPOWER7 compatibility modes.

3. IBM Spectrum Scale for Linux on POWER is supported:a. with the distributions and kernel levels as listed in the question What Linux distributions are



Note: IBM Spectrum Scale runs on the POWER8 processor in default, POWER8, andPOWER7 compatibility modes.

4. IBM Spectrum Scale for Linux on x86 Architecture is supported:a. with the distributions and kernel levels as listed in the question What Linux distributions are


What are the minimum hardware requirements for a IBM Spectrum Scale cluster?c. for additional support statements, see the question What are the current restrictions on IBM

Spectrum Scale Linux kernel support?5. IBM Spectrum Scale for Windows on x86 Architecture is supported:

a. with the levels of Windows Server as listed in the question What levels of the Windows OSare supported by IBM Spectrum Scale ?

b. on servers that meet the minimum hardware model requirements as listed in the questionWhat are the minimum hardware requirements for a IBM Spectrum Scale cluster?

Q3.3: What interconnects are supported for GPFS daemon-to-daemon communication in a GPFScluster?

A3.3: The interconnect for GPFS daemon-to-daemon communication depends upon the types of nodesin your cluster.

Note: This table provides the list of communication interconnects which have been tested by IBMand are known to work with GPFS. Other interconnects may work with GPFS but they have notbeen tested by IBM. The IBM Spectrum Scale support team will help customers who are usinginterconnects that have not been tested to solve problems directly related to GPFS, but will not beresponsible for solving problems deemed to be issues with the underlying communicationinterconnect's behavior including any performance issues exhibited on untested interconnects.

78

Table 42. GPFS daemon -to-daemon communication interconnects

Nodes in your cluster Supported interconnect Supported environments

Linux(x86 , Power)/AIX/Windows

Ethernet All supported IBM Spectrum Scaleenvironments

10-Gigabit Ethernet

40-Gigabit Ethernet

All supported IBM Spectrum Scaleenvironments

InfiniBand All supported IBM Spectrum Scaleenvironments

IP only

Linux (x86 and Power)


10-Gigabit Ethernet

40-Gigabit Ethernet


InfiniBand IP and optionally VERBS RDMANote: See the question Are there anyconsiderations when utilizing the RemoteDirect Memory Access (RDMA) offered byInfiniBand?

Linux on Z

Ethernet

10-Gigabit EthernetHipersocketsVswitch (z/VM only)GuestLAN (z/VM only)


AIX


10-Gigabit Ethernet

40-Gigabit Ethernet



IP only

Windows


10-Gigabit Ethernet

40-Gigabit Ethernet



IP only

79

Disk questionsQ4.1: What disk hardware has IBM Spectrum Scale been tested with?A4.1: This set of tables displays the set of disk hardware which has been tested by IBM and known to

work with IBM Spectrum Scale . Other disk devices may work with IBM Spectrum Scale usingNSD disk leasing, though they have not been tested by IBM. The IBM Spectrum Scale supportteam will help customers who are using devices outside of this list of tested devices, using NSDdisk leasing only, to solve problems directly related to IBM Spectrum Scale , but will not beresponsible for solving problems deemed to be issues with the underlying device's behaviorincluding any performance issues exhibited on untested hardware. Untested devices should notbe used with GPFS assuming SCSI-3 PR as the fencing mechanism, since our experience hasshown that devices cannot, in general, be assumed to support the SCSI-3 Persistent Reservemodes required by GPFS.

These test statements apply to all current releases of IBM Spectrum Scale unless specifiedotherwise.

It is important to note that:v Each individual disk subsystem requires a specific set of device drivers for proper operation

while attached to a host running GPFS. The prerequisite levels of device drivers are notdocumented in this GPFS-specific FAQ. Refer to the disk subsystem's web page to determinethe currency of the device driver stack for the host's operating system level and attachmentconfiguration.For information on IBM disk storage subsystems and their related device drivers levels andOperating System support guidelines, go to www.ibm.com/servers/storage/support/disk/index.html

v Microcode levels should be at the latest levels available for your specific disk hardware.For the IBM System Storage®, go to www.ibm.com/servers/storage/support/allproducts/downloading.html

DS4000 customers: Please also seev The IBM TotalStorage DS4000 Best Practices and Performance Tuning Guide at

publib-b.boulder.ibm.com/abstracts/sg246363.html?Openv For the latest firmware and device driver support for DS4100 and DS4100 Express Midrange

Disk System, go to http://www.ibm.com/systems/support/supportsite.wss/selectproduct?brandind=5000028&familyind=5329597&osind=0&oldbrand=5000028&oldfamily=5345919&oldtype=0&taskind=2&matrix=Y&psid=dm

v For the latest storage subsystem controller firmware support for DS4200, DS4700, DS4800, goto:– https://www.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-

5075581&brandind=5000028– https://www.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-

5073716&brandind=5000028

Table 43. Disk hardware tested for AIX on POWER

XtremIO 4.0.10

VMAX 3 Hudson 5977.810.784 and Trinity 5977.932.887

AIX 6.1 TL9 SP6, AIX 7.1 TL4 and AIX 7.2 or later, with IBM Spectrum Scale V4.1.0.0 or later

IBM FlashSystem® 900 Minimal Firmware Level: 1.2.0.11

This storage subsystem has been tested onAIX 7.1.3.16 with IBM Spectrum Scale V4.1.0.8 or later

80

http://www.ibm.com/servers/storage/support/disk/index.html

http://www.ibm.com/servers/storage/support/disk/index.html

http://www.ibm.com/servers/storage/support/allproducts/downloading.html

http://www.ibm.com/servers/storage/support/allproducts/downloading.html

http://publib-b.boulder.ibm.com/abstracts/sg246363.html?Open

http://www.ibm.com/systems/support/supportsite.wss/selectproduct?brandind=5000028&familyind=5329597&osind=0&oldbrand=5000028&oldfamily=5345919&oldtype=0&taskind=2&matrix=Y&psid=dm



https://www.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5075581&brandind=5000028




http://www-03.ibm.com/systems/storage/flash/index.html

Table 43. Disk hardware tested for AIX on POWER (continued)

IBM FlashSystem 840

Minimal Firmware Level: 1.1.1.2AIX 6.1(6100-09) and AIX 7.1(7100-02-03-1334) with GPFS V3.5.0.19 or later,and IBM Spectrum Scale V4.1 or later

IBM FlashSystem 820

Minimal Firmware Level: 6.3.0.6

AIX 6.1(6100-06) and AIX 7.1(7100-01) with GPFS V3.5.0.11 or later, and IBM Spectrum Scale V4.1 or later

IBM Storwize® V7000/V3500/V3700/SVCNote: Placing GPFS metadata on thinly provisioned or compressed volumes is not supported.

AIX 6.1 and 7.1 with GPFS V3.5 or later, and IBM Spectrum Scale V4.1 or later

IBM XIV® 2810

Minimum Firmware Levels: 10.1, 10.2AIX 6.1 and 7.1 with GPFS V3.5 or later, and IBM Spectrum Scale V4.1 or laterFor more information, directions and recommended settings for attachment please referto the latest Host Attach Guide for Linux located at the IBM XIV Storage System Knowledge Center go tohttp://publib.boulder.ibm.com/infocenter/ibmxiv/r2/index.jsp

IBM System Storage DS8000® using SDDPCM

IBM System Storage DCS3700

AIX 6.1 and 7.1 with GPFS V3.5 or later, and IBM Spectrum Scale V4.1 or later

IBM System Storage DS5000 all supported expansion drawer and disk types including SSD

This includes models: DS5100, DS5300 and DS5020 Express.

on AIX V7.1 with GPFS V3.5 or later, and IBM Spectrum Scale V4.1 or later

on AIX V6.1 with a minimum level of TL2 with SP2 and APAR IZ49639 GPFS V3.5 or later,and IBM Spectrum Scale V4.1 or later

Firmware levels:7.60.28.007.83.22.007.77.34.00

IBM System Storage DS3400 (1726-HC4)

See http://www-07.ibm.com/storage/includes/pdf/TSD03007USEN.pdf.

IBM TotalStorage ESS (2105-F20 or 2105-800 with SDD)

IBM System Storage Storage Area Network (SAN) Volume Controller (SVC) V2.1 and V3.1

See www.ibm.com/support/docview.wss?rs=591&uid=ssg1S1002471 for specific advice on SAN Volume Controllerrecommended software levels.

Hitachi Virtual Storage Platform(VSP)Hitachi Storage Platform(VSP G200,G400,G600,G800)Hitachi Storage Platform (VSP G1000)Hitachi Unified Storage VM

on AIX 7.1 TL03 (7100-03-03-1415) with IBM Spectrum Scale V4.1 or laterwith HDLM v7.3.1-00

81



http://www-03.ibm.com/systems/storage/disk/storwize_v7000/index.html

http://www-03.ibm.com/systems/storage/disk/xiv/index.html

http://publib.boulder.ibm.com/infocenter/ibmxiv/r2/index.jsp

http://www-03.ibm.com/systems/storage/disk/ds8000/

http://www-03.ibm.com/systems/storage/disk/dcs3700/

http://www.ibm.com/systems/storage/disk/ds5000/index.html

http://www-07.ibm.com/storage/includes/content/disk/ds3400/

http://www-07.ibm.com/storage/includes/pdf/TSD03007USEN.pdf

http://www.storage.ibm.com/disk/ess/index.html

http://www.ibm.com/servers/storage/support/software/sdd/

http://www.ibm.com/servers/storage/software/virtualization/svc/index.html

http://www.ibm.com/support/docview.wss?rs=591&uid=ssg1S1002471

Table 43. Disk hardware tested for AIX on POWER (continued)

Hitachi Universal Storage Platform (USP V)Hitachi Virtual Storage Platform (VSP)Hitachi Adaptable Modular Storage (AMS)

- AMS Series (includes 2100, 2300 and 2500 models)Note:1. In all cases Hitachi Dynamic Link Manager™ (HDLM) (multipath software) or MPIO (default PCM - failover

only) is required.2. AIX ODM objects supplied by Hitachi Data Systems (HDS) are required for all above devices.3. Customers should consult with HDS to verify that their proposed combination of the above components is

supported by HDS.

EMC Symmetrix VMAX and DMX Storage Subsystems (FC attach only)

Device driver support for Symmetrix includes both MPIO and PowerPath.

Selected models of EMC CLARiiON CX/CX-3 family including CX300, CX400, CX500 CX600, CX700 and CX3-20,CX3-40 and CX3-80Note: CX/CX-3 requires PowerPath.See http://www.emc.com.

Customers should consult with EMC to verify that their proposed combination of the above components issupported by EMC.

HP XP 128/1024HP StorageWorks Enterprise Virtual Arrays (EVA) 4000/6000/8000and 3000/5000 models that have been upgraded to active-activeconfigurationsNote: HDLM multipath software is required

HPE 3PAR OS 3.3.1Minimum Firmware Level: HPE 3PAR OS 3.3.1RHEL 6.7 with GPFS V4.2.3.4 or later

Table 44. Disk hardware tested with Linux on x86 servers

Hitachi Virtual Storage Platform (VSP)Hitachi Storage Platform (VSP G200,G400,G600,G800, microcode 83-03-24-00/00)Hitachi Virtual Storage Platform (VSP) (F400, F600, F800, microcode 83-03-24-00/00)Hitachi Storage Platform (VSP G1000, VSP G1500, microcode 80-04-22-00/00 or later)Hitachi Unified Storage VM

RHEL 7.1 or later, with IBM Spectrum Scale V4.2.1.1 or later

IBM FlashSystem A9000/A9000R Firmware version 12.0.1

RHEL 7.2 or later, with IBM Spectrum Scale V4.2.1.0 or later

See Q4.12 What are the considerations for using thinly provisioned or compressed volumes with GPFS? forconsiderations on thin provisioning and compression on these storage subsystems.

IBM FlashSystem 900 Minimal Firmware Level: 1.2.0.11

RHEL 6.5, or later, and SLES 11 SP3, or later, with IBM Spectrum Scale V4.1.0.8 or later

IBM FlashSystem 840

Minimum Firmware Level: 1.1.1.2

RHEL 6.5 and SLES 11 SP with GPFS V3.5.0.20 or later, and IBM Spectrum Scale V4.1.0.2 or later

82

http://www.emc.com




Table 44. Disk hardware tested with Linux on x86 servers (continued)

IBM FlashSystem 820

Infinband and FC attachMinimal Firmware Level: 6.3.1.0

RHEL 6.3 or later and SLES 11 SP 1 or later, with GPFS V3.5.0.11 or laterRHEL 6.3 or later and SLES 11 SP 2 with IBM Spectrum Scale V4.1 or later

IBM Flex Systems V7000

Firmware Level: SVC 6.4.1.4

RHEL 6.4 or later, and SLES 11 SP2 or later, using multipath Device MapperGPFS V3.5.0.9 or later, and IBM Spectrum Scale V4.1 or later

RHEL 5.9 and SLES 10 SP4GPFS V3.5.0.9 or later

IBM Storwize V7000/V3500/V3700/SVC

RHEL 6.x and 5.x with levels of GPFS that support the distributionSLES11 SP2 or later with GPFS V3.5 or later, and IBM Spectrum Scale V4.1 or later

SLES 10 SP1 and SP2 with GPFS V3.5 or later

IBM XIV 2810

Minimum Firmware Level: 10.0.1RHEL5.1 and greater with GPFS V3.5 or later, and IBM Spectrum Scale V4.1 or laterSLES 10.2 with GPFS V3.5 or later, and IBM Spectrum Scale V4.1 or later

For more information, directions and recommended settings for attachment please refer to the latestHost Attach Guide for Linux located at the IBM XIVStorage System Knowledge Center go tohttp://publib.boulder.ibm.com/infocenter/ibmxiv/r2/index.jsp

IBM System Storage DS8000

IBM System Storage DS3500/DS5300

RHEL 6.2 or later and SLES 11 SP2 or later with IBM Spectrum Scale V4.1 or laterRHEL 6.x, 5.x with GPFS V3.5 or laterSLES 11, 10 with GPFS V3.5 or later

Firmware level 7.84.44.00


RHEL 6.0, 5.6 and 5.5SLES 11.1, 10.4 and 10.3


This include:Models: DS5100, DS5300 and DS5020 Express.Firmware levels:7.60.28.007.83.22.007.77.34.00

IBM System Storage DS3400 (1726-HC4)

83


http://www-03.ibm.com/systems/storage/disk/storwize_v7000/

http://www-03.ibm.com/systems/storage/disk/xiv/index.html

http://publib.boulder.ibm.com/infocenter/ibmxiv/r2/index.jsp

http://www-03.ibm.com/systems/storage/disk/ds8000/



http://www-07.ibm.com/storage/includes/content/disk/ds3400/

Table 44. Disk hardware tested with Linux on x86 servers (continued)

IBM TotalStorage Enterprise Storage Server® (ESS) models 2105-F20 and 2105-800, with Subsystem Device Driver(SDD)

EMC ScaleIO V2.0 without persistent reserve

RHEL 7.1, or later, with IBM Spectrum Scale V4.2.0.1, or later

EMC Symmetrix Direct Matrix Architecture (DMX) Storage Subsystems 1000 with PowerPath v 3.06 and v 3.07

IBM System Storage Storage Area Network (SAN) Volume Controller (SVC) V2.1 and V3.1

See www.ibm.com/support/docview.wss?rs=591&uid=ssg1S1002471 for specific advice on SAN Volume Controllerrecommended software levels.

IBM DCS9550 (either FC or SATA drives)FC attach onlyminimum firmware 3.08bQLogic drivers at 8.01.07 or newer andIBM SAN Surfer V5.0.0 or newer

IBM DCS9900 (either FC or SATA drives)FC attach only

Table 45. Disk hardware tested with Linux on POWER

IBM FlashSystem™ 900

Minimal Firmware Level: 1.2.0.11RHEL 6.5 or later, and SLES 11 SP3 or later, with IBM Spectrum Scale V4.1.0.8 or later

IBM FlashSystem 840

Minimal Firmware Level: 1.1.1.2 using default DM-MPRHEL 6.4 or later, and SLES 11 SP2 or later

GPFS V3.5.0.18 or later, and IBM Spectrum Scale V4.1 or later

IBM Flex System® V7000Firmware Level: SVC 6.4.1.4, 7.1, 7.2RHEL 6.4 or later and SLES 11 SP2 or later using multipath Device MapperIBM Spectrum Scale V4.1 or later

RHEL 5.9 or later, RHEL 6.4 or later, SLES 10 SP4 or later, SLES 11 SP2 or later,using multipath Device Mapper with GPFS V3.5.0.9 or later

IBM Storwize V7000/V3500/V3700/SVCNote: Placing GPFS metadata on thinly provisioned or compressed volumes is not supported.

RHEL 6.x and 5.x with levels of GPFS that support the distribution

SLES11 SP2 or later, with IBM Spectrum Scale V4.1 or laterSLES 10 SP1 or later, and SLES11 SP1 or later with GPFS V3.5 or later

IBM System Storage DS3500/DS5300

RHEL 6.2 or later and SLES 11 SP2 or later with IBM Spectrum Scale V4.1 or laterRHEL 6.x, 5.x with GPFS V3.5 or laterSLES 11, 10 with GPFS V3.5 or later

Firmware level 7.84.44.00


RHEL 6.0, 5.6 and 5.5SLES 11.1, 10.4 and 10.3

84

http://www.storage.ibm.com/disk/ess/index.html



http://www.emc.com/storage/scaleio/index.htm

http://www.ibm.com/servers/storage/software/virtualization/svc/index.html

http://www.ibm.com/support/docview.wss?rs=591&uid=ssg1S1002471

http://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=897&letternum=ENUS107-294

http://www-01.ibm.com/common/ssi/rep_ca/3/897/ENUS108-793/ENUS108-793.PDF




http://www-03.ibm.com/systems/storage/disk/storwize_v7000/index.html


Table 45. Disk hardware tested with Linux on POWER (continued)


This include:Models: DS5100, DS5300 and DS5020 Express.Firmware levels:7.60.28.007.83.22.007.77.34.00

IBM System Storage DS8000

Table 46. Disk hardware tested with IBM Spectrum Scale for Linux on Z

HDS USPV/VSP/VSP G1000 without persistent reserve

EMC without persistent reserve

IBM DS8000 Series

IBM FlashSystem

IBM Storwize V7000

IBM XIV

IBM SVC

IBM Elastic Storage Server (ESS) V4.5.0

Note:1. Ensure you are running with the latest firmware levels available.2. DS4000 is not supported3. See the question Does IBM Spectrum Scale for Linux on Z support Direct Attached Storage Devices (DASD)?4. IBM Spectrum Scale for Linux on Z is supported with EMC without persistent reserve. Customers should

consult with EMC to verify if their proposed solution is supported by EMC.

Q4.2: What Fibre Channel Switches are qualified for IBM Spectrum Scale usage and is there a FCSwitch support chart available?

A4.2: There are no special requirements for FC switches used by IBM Spectrum Scale other than theswitch must be supported by AIX or Linux or Windows. For further information seewww.storage.ibm.com/ibmsan/index.html

Q4.3: Can I concurrently access SAN-attached disks from both AIX and Linux (x86 and Power) nodesin my IBM Spectrum Scale cluster?

A4.3: While the architecture of IBM Spectrum Scale would generally allow LUNs to be shared betweendifferent operating systems (Linux (x86 and Power), AIX, and Windows), the actualimplementation of various OS specific features preclude this from being exploited at the currenttime. There are differences in how disks are labeled, how partitions are created and managed,and how multi-pathing managers react to error conditions between the various OS such that thissupport isn't offered in IBM Spectrum Scale today.

Note:1. For IBM Spectrum Scale for Linux on Z, SAN-attached disks can only be accessed from Linux

on Z cluster nodes.Q4.4: What disk support failover models does IBM Spectrum Scale support for the IBM System

Storage DS4000 family of storage controllers with the Linux operating system?A4.4: IBM Spectrum Scale has been tested with both the Host Bus Adapter Failover and Redundant

Dual Active Controller (RDAC) device drivers.

To download the current device drivers for your disk subsystem, go to http://www.ibm.com/servers/storage/support/.

Note: IBM Spectrum Scale for Linux on Z does not support the DS4000 family.

85


http://www.ibm.com/servers/storage/disk/ds8000/index.html

http://www.emc.com/

http://www.ibm.com/servers/storage/san/

http://www.ibm.com/servers/storage/support/

http://www.ibm.com/servers/storage/support/

Q4.5: What devices does IBM Spectrum Scale support with SCSI-3 Persistent Reservations?A4.5: The following devices are supported with SCSI-3 Persistent Reservations:

v EMC XtremIO 4.0.10, VMAX 3 Hudson 5977.810.784 and Trinity 5977.932.887 using NativeMPIO running AIX 6.1 TL9 SP6, AIX 7.1 TL4 and AIX 7.2 or later, through Fiber Channelconnection (IBM Spectrum Scale V4.1.0.0 or later).

v Hitachi Storage Platform (VSP G1000) (microcode 80-04-22-00/00) using default DM-MP on x86Linux running RHEL7.1, or later, through Fiber Channel connection (IBM Spectrum ScaleV4.2.1.1 or later).

v DCS3700 (firmware 08.20.12.00) using IBM RDAC driver or the default DM-MP on x86 Linuxrunning RHEL7.2, SLES 11 SP3 (IBM Spectrum Scale V4.2.0.3 or later)

v IBM FlashSystem A9000/A9000R (firmware version 12.0.1) using default DM-MP on x86 Linuxrunning RHEL7.2, or later through Fiber Channel connection (IBM Spectrum Scale V4.2.1.0 orlater)

v IBM FlashSystem 900 (firmware 1.2.0.11) on x86 Linux running RHEL 6.5, or later, or SLES 11SP3, or later, through Infiniband or Fiber Channel connection

v IBM FlashSystem 900 (firmware 1.2.0.11) on power Linux running RHEL 6.5, or later, or SLES11 SP3, or later, through Infiniband or Fiber Channel connection

v IBM FlashSystem 900 (firmware 1.2.0.11) using default AIX PCM on AIX 7.1.3.16 through FiberChannel and Infiniband connections (GPFS V4.1.0.8 or later)

v IBM FlashSystem 840 (firmware 1.1.1.2) on x86 Linux running RHEL 6.5, or later, or SLES 11SP3, or later, through Infiniband or Fiber Channel connection

v IBM FlashSystem 840 (firmware 1.1.1.2) on power Linux running RHEL 6.5, or later, or SLES 11SP3, or later, through Infiniband or Fiber Channel connection

v IBM FlashSystem 840 (firmware 1.1.1.2) using default AIX PCM on AIX 6.1.9.0 and AIX 7.1.2.3through Fiber Channel and Infiniband connections (GPFS V3.5.0.19 or later and IBM SpectrumScale V4.1.0.0 or later)

v IBM Storwize V7000 (firmware SVC 7.1.0.3) using default DM-MP on Power Linux runningRHEL 6.4, or later, or SLES 11 SP2 (GPFS V3.5.0.16 or later and GPFS V3.4.0.20 or later)

v IBM Flex V7000 (firmware SVC 6.4.1.4) using SDDPCM (2.6.3.2) on AIX 6.1.8 or AIX 7.1.2(GPFS V3.5.0.16 or later and GPFS V3.4.0.20 or later)

v IBM Flex V7000 (firmware SVC 6.4.1.4) using default DM-MP on Power Linux running RHEL5.9/6.4 or SLES 10.4/11.2 (GPFS V3.5.0.16 or later and GPFS V3.4.0.20 or later)

v IBM Flex V7000 (firmware SVC 7.1.0.0) using default DM-MO on x86 Linux running RHEL5.9/6.4 or SLES 10.4/11.2 (GPFS V3.5.0.16 or later and GPFS V3.4.0.20 or later)

v IBM Storwize V7000 (firmware SVC 7.1.0.3) using SDDPCM on AIX 6.1.0 and AIX 7.1.0 throughFiber Channel connection

v IBM FlashSystem 820 (firmware 6.3.1 SP1) using default DM-MP on Power Linux runningRHEL 6.3 or SLES 11 SP2 through Infiniband or Fiber Channel connections (GPFS V3.5.0.21 orlater, and IBM Spectrum Scale V4.1.0.4 or later)

v IBM FlashSystem 820 (firmware 6.3.1 SP1) on x86 Linux running RHEL 6.3 or SLES 11 SP2through Infiniband or Fiber Channel connection (GPFS V3.5.0.21 or later, and IBM SpectrumScale V4.1.0.4 or later)

v IBM FlashSystem 820 using default AIX PCM on AIX 6.1.0 and AIX 7.1.0 through FiberChannel connection (GPFS V3.5.0.21 or later, and IBM Spectrum Scale V4.1.0.4 or later)

v IBM Storwize V7000/V3500/V3700 (SVC firmware) on x86 Linux running SLES 10 SP4, SLES11 SP2, RHEL 5.8, or RHEL 6.2

v DS5000 using SDDPCM or the default AIX PCM on AIXv DS8000 (all 2105 and 2107 models) using SDDPCM or the default AIX PCM on AIXv DS4000 subsystems using the IBM RDAC driver and AIX MPIO on AIX.

(devices.fcp.disk.array.rte or MPIO)v DS3500 using IBM RDAC driver or the default DM-MP on Linuxv DS4800 using IBM RDAC driver or the default DM-MP on Linuxv DS5020 using IBM RDAC driver or the default DM-MP on Linuxv DS5300 using IBM RDAC driver or the default DM-MP on Linuxv DS8000(2107 models) using IBM SDD driver or the default DM-MP on Linuxv EMC VMAX using EMC PowerPath 5.5 P04 B003 and EMC AIX ODM Package 5.3.0.6.

86










http://www.redbooks.ibm.com/abstracts/tips1039.html







http://www-03.ibm.com/systems/storage/storwize/

Please check EMC PowerLink for support details, and consult EMC to verify that the proposedconfiguration is supported by EMC. Note: The use of AIX MPIO is also supported in thisenvironment.

v HPE 3PAR (HPE 3PAR OS 3.3.1) using the default DM-MP on x86 Linux running RHEL6.7(IBM Spectrum Scale V4.2.3.4 or later)

The most recent versions of the device drivers are always recommended to avoid problems thathave been addressed.

Note: For a device to properly offer SCSI-3 Persistent Reservation support for GPFS, it mustsupport SCSI-3 PERSISTENT RESERVE IN with a service action of REPORT CAPABILITIES.The REPORT CAPABILITIES must indicate support for a reservation type of Write Exclusive AllRegistrants. Contact the disk vendor to determine these capabilities.

Also see the question Are there any requirements for Persistent Reserve support in GPFS ?Q4.6: What considerations are there when setting up DM-MP multipath service?A4.6: To setup up the DM-MP multipath service, depending on node distribution and storage controller

firmware level, you may need to modify the /etc/multipath.conf file to fit your individual storagerequirements. A default copy of the multipath.conf file can be copied from the /usr/share/docdirectory.

As an example, the following attributes are tested with the IBM products DS3500 (1746), DS5020(1814), DS4800 (1815), and DS5300 (1818) :

device {vendor "IBM"product "1746"getuid_callout "/sbin/scsi_id -g -u -s /block/%n"prio_callout "/sbin/mpath_prio_rdac /dev/%n"features "0"hardware_handler "1 rdac"path_selector "round-robin 0"path_grouping_policy group_by_priofailback immediaterr_weight uniformno_path_retry failrr_min_io 1000path_checker rdac

}

Note: In order for GPFS failover to take place, the following steps must be taken:1. The following parameters must be set:v features "0"v failback immediatev no_path_retry fail

2. The mmnsddiscover -a command must be issued in order for the NSD server to rediscoverthe disks.

Additionally, seev The IBM Spectrum Scale wiki at https://www.ibm.com/developerworks/community/wikis/

home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Device%20Naming%20and%20Discovery%20in%20GPFS

v Please refer to each distribution's multipath document for details. For instance:– https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/

DM_Multipath/– https://www.suse.com/documentation/sles11/stor_admin/?page=/documentation/sles11/

stor_admin/data/bookinfo.htmlQ4.7: Are there any steps that need to be taken before disks are used by IBM Spectrum Scale on

AIX?

87

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Device%20Naming%20and%20Discovery%20in%20GPFS



https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/

https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/

https://www.suse.com/documentation/sles11/stor_admin/?page=/documentation/sles11/stor_admin/data/bookinfo.html

https://www.suse.com/documentation/sles11/stor_admin/?page=/documentation/sles11/stor_admin/data/bookinfo.html

A4.7: Yes. Most of the following specifics are for IBM disks. If you have non-IBM disks, commentsbelow help explain how you would need to adjust the commands that are shown. The lsattr andchdev commands are used for all disk types:v Set all disks that will be used as NSDs to the no_reserve reservation policy.

– This may require chdev commands to modify the attributes if they are found not set tono_reserve as the default.

– The reserve_policy needs to be checked on all nodes with access to the disks even if theyare not specially coded as NSD servers.- The lsattr -El hdiskX -a reserve_policy command should show no_reserve- Issue the chdev -El hdiskx -a reserve_policy=no_reserve command to update if necessary.

v Please note this advice pertains to all disks being used by IBM Spectrum Scale on AIX, not justdisks that will be used in a persistent reserve model.

v If there are any issues with the chdev commands contact AIX support for assistance.

Disks from other manufacturers should have their own unique attributes that similarly controlsdisk reserves. A specific example is the reserve_lock attribute, which needs to have the value no.The lsattr -El device command would show all the attributes that the device supports, along withthe current value of each. If it is not obvious which attribute controls reserves on the disk, contactthe manufacturer for that information. The lsattr -R -l device -a attribute command can be used tofind out all the legal values for the specified attribute. For example lsattr -R -l hdiskx -areserve_policy.

Q4.8: Does IBM Spectrum Scale support Logical Volumes (LVs) ?A4.8: Logical Volumes (LVs) are minimally supported under these conditions:

v The customer must maintain LV availability. IBM Spectrum Scale does not support themanagement of the export/import or varying on/off of LVs between nodes.

v Conventional LVs can be used when only attached to a single node as a descOnly disk.v Starting with GPFS 3.5.0.16, Concurrent Mirrored LVs can be used as descOnly disks.

Q4.9: Does IBM Spectrum Scale support AIX raw hdisks (rhdisk) ?A4.9: Input to the mmcrnsd command requires the use of the hdisks format, not rhdisks. Internal logic

converts the hdisk format to rhdisk.Q4.10: Does IBM Spectrum Scale for Linux on Z support Direct Attached Storage Devices (DASD)?A4.10: The DASD device driver provides access to real or emulated Direct Access Storage Devices

(DASD) that can be attached to the channel subsystem of an IBM Z. This device driver supportsthe ECKD™ (Extended Count Key Data) and FBA (Fixed Block Access) devices.

Note: Prior to IBM Spectrum Scale V4.2.1, an ECKD device has to have all the same Bus-ID onall NSD server nodes.

To enable the usage of FBA devices, the cluster needs to run IBM Spectrum Scale V4.2.2 or later.

It is recommended to set the failfast parameter of the DASD device so the device driverimmediately returns "failed" for an I/O operation when the last path to a DASD is lost. If thefailfast parameter is not set, GPFS might hang until the path to the DASD is restored. See theDevice Drivers, Features, and Commands documentation for your Linux distribution:v For SLES, see http://www.ibm.com/developerworks/linux/linux390/

documentation_suse.html.v For RHEL, see http://www.ibm.com/developerworks/linux/linux390/

documentation_red_hat.html.Q4.11: Does IBM Spectrum Scale support 4K disk sectors?A4.11: Yes, 4K disk sector support requires IBM Spectrum Scale V4.1.0.5 or later. The following disk

subsystems with 4K sector size have been tested by IBM:v ECKD disk devices (Linux on Z only)v IBM FlashSystem 820.v IBM FlashSystem 840v IBM FlashSystem 900

Note:

88

http://www.ibm.com/developerworks/linux/linux390/documentation_suse.html

http://www.ibm.com/developerworks/linux/linux390/documentation_suse.html

http://www.ibm.com/developerworks/linux/linux390/documentation_red_hat.html

http://www.ibm.com/developerworks/linux/linux390/documentation_red_hat.html




1. Other disk devices may work with IBM Spectrum Scale, though they have not been tested byIBM. See the question What disk hardware has IBM Spectrum Scale been tested with?

2. IBM Spectrum Scale on Windows does not support 4K disk sectors.Q4.12: What are the considerations for using thinly provisioned or compressed volumes with GPFS?A4.12: When using thinly provisioned (with compression constituting a form of thin provisioning)

volumes with GPFS, the following important configuration aspects must be considered:

Metadata: Placing GPFS metadata on an NSD backed by a thinly provisioned volume isdangerous and not supported. If the real disk space that backs a thinly provisioned virtual volumeis exhausted, it is possible that a write to a previously allocated disk sector could fail. If thisvolume hosts GPFS metadata, in certain scenarios a GPFS file system could become temporarilyor even permanently unavailable. When the volume runs out space, the file system will unmountand might not remount if the log recovery is unable to write metadata and thus failed tocomplete.

Data: Placing GPFS data on a thinly provisioned volume is supported, but with certainlimitations. Specifically, if the real disk space that backs a thinly provisioned virtual volume isexhausted, a failing data write could make the GPFS file system temporarily unavailable. Inaddition, placing replicated data on a thinly provisioned volume is not supported. Spaceexhaustion while writing to replicated data might result in the subsequent log recovery failing,which could lead to the file system remaining inaccessible. Presently, GPFS does not communicateblock deallocation events to the block device layer, so freeing space on the file system layer doesnot free up space on the block device layer. Thus, the efficiency of thin provisioning will degradeover time as blocks are allocated and freed.

Q4.13: Does IBM Spectrum Scale for Linux on Z work with disk hardware replication?A4.13:

Storage subsystems, such as IBM DS8000, offer data replication mechanisms. For example, MetroMirror provides synchronous data replication; whereas, Global Mirror provides asynchronousdata replication over distance. IBM Spectrum Scale supports disk hardware replication providedthat the disks (FCP LUNs or ECKD volumes) are within a consistency group. The IBM SpectrumScale configuration needs to be set up in a way that handles the different device addresses usedfor primary and secondary devices.

Disk replication can be managed by products such as IBM GDPS (Geographically DispersedParallel Sysplex) or IBM CSM (Copy Services Manager).

Q4.14: Does IBM Spectrum Scale for Linux on Z support HyperSwap or SVC Stretch Clusters?A4.14:

IBM Spectrum Scale supports HyperSwap, SVC Stretch Clusters, or similar technologies on IBM Zbeginning in IBM Spectrum Scale V4.2.3.1. The actual swap from one device to another results ina pause to IO while the necessary storage level actions are taken. The failureDetectionTime andleaseRecoveryWait tunables need to be set accordingly. It is suggested to use a value of 1.5X theexpected pause time. The user should contact their provider to discuss expected IO pause timefor their particular configuration as the actual values depend on the HyperSwap/stretch clustertechnology, as well as the storage subsystem and configuration. For more information aboutHyperSwap with IBM DS8000, see the following White Paper: https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102640.

89

https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102640

https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102640

Scaling questionsQ5.1: What are the IBM Spectrum Scale cluster size limits?A5.1: The current maximum tested IBM Spectrum Scale cluster size limits are:

Table 47. IBM Spectrum Scale maximum tested cluster sizes

IBM Spectrum Scale for Linux (x86, Power, and IBM Z) 9620 nodes

IBM Spectrum Scale for AIX 1530 nodes

IBM Spectrum Scale for Windows on x86_64 Architecture 64 Windows nodes

FPO-enabled 732 nodes

IBM Spectrum Scale for Linux and IBM Spectrum Scale forAIX

3906 (3794 Linux nodes and 112 AIX nodes)

Note: Contact [email protected] if you intend to exceed:v Configurations with Linux nodes exceeding 512 nodesv Configurations with AIX nodes exceeding 128 nodesv Configurations with Windows nodes exceeding 64 nodesv FPO-enabled configurations exceeding 732 nodes

Although IBM Spectrum Scale is typically targeted for a cluster with multiple nodes, it can alsoprovide high performance benefit for a single node so there is no lower limit. For a given I/Oconfiguration, typically multiple nodes are required to saturate the aggregate file systemperformance capability. If the aggregate performance of the I/O subsystem is the bottleneck, thenIBM Spectrum Scale can help achieve the aggregate performance even on a single node.

Q5.2: What are some scaling considerations for the protocols function?A5.2: Scaling considerations for the protocols function include:

v The number of protocol nodes.If you are using SMB in any combination of other protocols you can configure only up to 16protocol nodes. This is a hard limit and SMB cannot be enabled if there are more protocolnodes. If only NFS and Object are enabled, you can have 32 nodes configured as protocolnodes.

v The number of client connections.A maximum of 3,000 SMB connections is recommended per protocol node with a maximum of20,000 SMB connections per cluster. A maximum of 4,000 NFS connections per protocol node isrecommended. A maximum of 2,000 Object connections per protocol nodes is recommended.The maximum number of connections depends on the amount of memory configured andsufficient CPU. We recommend a minimum of 64GB of memory for only Object or only NFSuse cases. If you have multiple protocols enabled or if you have SMB enabled we recommend128GB of memory on the system.

Q5.3: What is the current maximum tested limit for SMP scaling?A5.3: The largest SMP scale tested to date is 64-way. If you require a configuration with a larger SMP

machine, please contact [email protected]: What is the current limit on the number of nodes that may concurrently join a cluster?A5.4: As of GPFS V3.4.0.18 and GPFS V3.5.0.5, the total number of nodes that may concurrently join a

cluster is limited to a maximum of 16384 nodes.

A node joins a given cluster if it is:v A member of the local GPFS cluster (the mmlscluster command output displays the local

cluster nodes).v A node in a different GPFS cluster that is mounting a file system from the local cluster.

For example:v GPFS clusterA has 2100 member nodes as listed in the mmlscluster command.v 500 nodes from clusterB are mounting a file system owned by clusterA.

In this example clusterA therefore has 2600 concurrent nodes.Q5.5: What is the limit of remote clusters that a client node can join?

90



A5.5: The maximum number of remote clusters that a client node can join is 31 (32 when counting thelocal cluster).

Q5.6: What is the limit of remote clusters that can join a local cluster?A5.6: There is not really a limit. The smallest cluster possible is a single node cluster, which means that

16,383 clusters can join a local cluster (16384 - 1).Q5.7: What are the current file system size limits?A5.7: The current file system size limits are:

Table 48. Current file system size limits

GPFS 2.3 or later, file system architectural limit 2^99 bytes

GPFS 2.2 file system architectural limit 2^51 bytes (2 Petabytes)

Current tested limit approximately 18 PB

Q5.8: What is the current limit on the number of mounted file systems in an IBM Spectrum Scalecluster?

A5.8: The current limit on the number of mounted file systems in an IBM Spectrum Scale cluster is 256on all supported OSs except for Windows. On Windows, the limit is the number of unused drivesin the range A-Z.

Q5.9: What is the architectural limit of the number of files in a file system?A5.9: The architectural limit of the number of files in a file system is determined by the file system

format:v For file systems created with GPFS V3.4 or later, the architectural limit is 264.

The current tested limit is 9,000,000,000.v For file systems created with GPFS V2.3 or later, the limit is 2,147,483,648.v For file systems created prior to GPFS V2.3, the limit is 268,435,456.

Please note that the effective limit on the number of files in a file system is usually lower thanthe architectural limit, and could be adjusted using the mmchfs command (GPFS V3.4 and lateruse the --inode-limit option).

Q5.10: What is the architectural limit of the number of disks in a file system?A5.10: The architectural limit of the number of disks in a file system is 2048.Q5.11: What are the limitations on IBM Spectrum Scale disk size?A5.11: The maximum disk size is only limited by the OS kernel and device driver support

Table 49. Maximum disk size supported

OS kernel Maximum supported GPFS disk size

AIX, 64-bit kernel >2TB, up to the device driver limit

Linux 2.6 64-bit kernels >2TB, up to the device driver limit

Windows >2TB, up to the device driver limit

Note:1. On systems running with the Linux kernel 3.0, both processor.max_cstate and

intel_idle.max_cstate should be set to zero.2. IBM Spectrum Scale supports 16MB file system block size.

Q5.12: What is the limit on the maximum number of groups a user can be a member of whenaccessing a GPFS file system?

A5.12: Each user may be a member of one or more groups, and the list of group IDs (GIDs) that thecurrent user belongs to is a part of the process environment. This list is used when performingaccess checking during I/O operations. Due to architectural constraints, GPFS code does notaccess the GID list directly from the process environment (kernel memory), and instead makes acopy of the list, and imposes a limit on the maximum number of GIDs that may be smaller thanthe corresponding limit in the host operating system. The maximum number of GIDs supportedby GPFS depends on the platform and the version of GPFS code. Note that the GID list includesthe user primary group and supplemental groups.

91


Table 50. Maximum number of GIDs supported

Platform Maximum number of GIDs supported

AIX 2,0481

Linux with 4K page size (all supported platforms exceptthe two below)

1,020

Linux with 64K page size (PPC64/RHEL5/RHEL6/RHEL 7 platforms)

16,380

Windows Windows OS limit (no limit in GPFS code)

Note:1. The table reflects the maximum value that can be achieved with IBM Spectrum Scale 5.0.0 or

later and AIX 7.1 or later. On earlier versions of IBM Spectrum Scale or AIX, the limit is 128.For more information about configuring the Number of Groups allowed, seehttps://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.security/number_groups_allowed.htm.

Q5.13: What are the current limits on the number of filesets in an IBM Spectrum Scale file system?A5.13:

Table 51. Maximum number of filesets

Maximum number of filesets (dependent +independent) Maximum number of independent filesets

10,000 1,000

Note: The listed numbers include the root fileset. The root fileset is an independent fileset.Q5.14: What are the current limits on the number of snapshots in an IBM Spectrum Scale file system?A5.14:

Table 52. Maximum number of snapshots

Global snapshotsMaximum number of snapshots of eachindependent fileset

256 256

Q5.15: What is the current limit for the number of data and metadata replicas?A5.15: The maximum supported number of data and metadata replicas is 3 for GPFS V3.5.0.7 and later,

and 2 for older versions.Q5.16: What is the limitation on pathname length?A5.16: The maximum supported pathname length (directory tree path) must not exceed 4096 bytes.

92

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.security/number_groups_allowed.htm

https://www.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.security/number_groups_allowed.htm

Configuration and tuning questionsPlease also see the questions:v What are the current advisories for all platforms supported by IBM Spectrum Scale?v What are the current advisories for IBM Spectrum Scale on AIX?v What are the current advisories for IBM Spectrum Scale on Linux?v What are the current advisories for IBM Spectrum Scale on Windows?

Q6.1: What specific configuration and performance tuning suggestions are there?A6.1: In addition to the configuration and performance tuning suggestions in the IBM Spectrum Scale:

Concepts, Planning, and Installation Guide for your version of IBM Spectrum Scale:v With IBM Spectrum Scale V4.2.0.3 and later, the mmchconfig command supports the

workerThreads attribute to control the maximum number of concurrent file operations at anyone instant, as well as the degree of concurrency for flushing dirty data and metadata in thebackground and for prefetching data and metadata. This attribute may be used instead ofworker1Threads and prefetchThreads as a simpler and more comprehensive way to tune thefile system for systems capable of handling higher sequential as well as random read/writeworkloads and small file activity.See the mmchconfig command as documented in the IBM Spectrum Scale: Administration andProgramming Reference

v On systems running with the Linux kernel 3.0, both processor.max_cstate andintel_idle.max_cstate should be set to zero.

v IBM Spectrum Scale supports 16MB block size.

Note:1. For support of 8MB block size with SLES, the minimum level of support is SLES 10 SP4

with this patch http://download.novell.com/Download?buildid=lOqqokqjuQQ2. For support of 8MB block size with RHEL, the minimum level of RHEL 6.0 shipped with

coreutils-8.4 is needed to take full advantage of block sizes larger than 4MB. Using RHEL5with coreutils-5.97 is supported, but can result in degraded performance from basicoperations including but not limited to the cp command.

v If your IBM Spectrum Scale cluster is configured to use SSH/SCP, it is suggested that youincrease the value of MaxStartups in sshd_config to at least 1024.

v You must ensure that when you are designating nodes for use by IBM Spectrum Scale youspecify a non-aliased interface. Utilization of aliased interfaces may produce undesired results.When creating or adding nodes to your cluster, the specified hostname or IP address must referto the communications adapter over which the GPFS daemons communicate. When specifyingservers for your NSDs, the output of the mmlscluster command lists the hostname and IPaddress combinations recognized by IBM Spectrum Scale. Utilizing an aliased hostname notlisted in the mmlscluster command output may produce undesired results.

v On Linux systems it is recommended you adjust the vm.min_free_kbytes kernel tunable. Thistunable controls the amount of free memory that Linux kernel keeps available (i.e. not used inany kernel caches). When vm.min_free_kbytes is set to its default value, on someconfigurations it is possible to encounter memory exhaustion symptoms when free memoryshould in fact be available. Setting vm.min_free_kbytes to a higher value (Linux sysctl utilitycould be used for this purpose), on the order of magnitude of 5-6% of the total amount ofphysical memory, but no more than 2GB, should help to avoid such a situation.Also, see the following GPFS Redpapers:– GPFS Sequential Input/Output Performance on IBM pSeries 690 at www.redbooks.ibm.com/

redpapers/pdfs/redp3945.pdfQ6.2: What configuration and performance tuning suggestions are there for IBM Spectrum Scale

when used primarily for Oracle databases?A6.2: In addition to the performance tuning suggestions in the IBM Spectrum Scale: Concepts, Planning,

and Installation Guide for your version of GPFS:

93



http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.2.0/ibmspectrumscale42_welcome.html


http://download.novell.com/Download?buildid=lOqqokqjuQQ

http://www.redbooks.ibm.com/redpapers/pdfs/redp3945.pdf

http://www.redbooks.ibm.com/redpapers/pdfs/redp3945.pdf



v When running Oracle RAC 10g, it is suggested you increase the value forOPROCD_DEFAULT_MARGIN to at least 500 to avoid possible random reboots of nodes.In the control script for the Oracle CSS daemon, located in /etc/init.cssd the value forOPROCD_DEFAULT_MARGIN is set to 500 (milliseconds) on all UNIX derivatives except for AIX. ForAIX this value is set to 100. From a GPFS perspective, even 500 milliseconds may be too low insituations where node failover may take up to a minute or two to resolve. However, if duringnode failure the surviving node is already doing direct IO to the oprocd control file, it shouldhave the necessary tokens and indirect block cached and should therefore not have to waitduring failover.

v Using the IBM Spectrum Scale is attractive for RAC environments because executables, tracefiles, and archive log files are accessible on all nodes. However, care must be taken to properlyconfigure the system in order to prevent false node evictions, and to maintain the ability toperform rolling upgrades of the Oracle software. Without proper configuration GPFS recoveryfrom a node failure can interfere with cluster management operations resulting in additionalnode failures.If you are running GPFS and Oracle RAC 10gR2 and encounter false node evictions:– Upgrade the CRS to 10.2.0.3 or newer.

The Oracle 10g Clusterware (CRS) executables or logs (the CRS_HOME) should be placed ona local JFS2 filesystem. Using GPFS for the CRS_HOME can inhibit CRS functionality on thesurviving nodes while GPFS is recovering from a failed node for the following reasons:- In Oracle 10gR2, up to and including 10.2.0.3, critical CRS daemon executables are not

pinned in memory. Oracle and IBM are working to improve this in future releases of10gR2.

- Delays in updating the CRS log and authorization files while GPFS is recovering caninterfere with CRS operations.

- Due to an Oracle 10g limitation rolling upgrades of the CRS are not possible when theCRS_HOME is on a shared filesystem.

– CSS voting disks and the Oracle Clusterware Registry (OCR) should not be placed on GPFSas the IO freeze during GPFS reconfiguration can lead to node eviction, and the inability ofCRS to function. Place the OCR and Voting disk on shared raw devices (hdisks).

– Oracle Database 10g (RDBMS) executables are supported on GPFS for Oracle RAC 10g.However, the system should be configured to support multiple ORACLE_HOME’s so as tomaintain the ability to perform rolling patch application. Rolling patch application issupported for the ORACLE_HOME starting in Oracle RAC 10.2.0.3.

– Oracle Database 10g data files, trace files, and archive log files are supported on GPFS.

Note:1. Only a subset of releases are certified for use in Oracle environments:

Table 53. Releases certified for use in Oracle environments

IBM Spectrum Scale or GPFS Versions

Oracle RAC AIX/VIOS 4.11 3.5

12cR1 AIX 7.1 AIX 7.1-TL3-SP03IBM Spectrum Scale 4.1.1.2GPFS V4.1.0.6RAC 12.1.0.2.3

AIX 7.1-TL1-SP03GPFS 3.5.0.8RAC 12.1.0.1, 12.1.0.2

AIX 6.1 AIX 6.1-TL9IBM Spectrum Scale 4.1.1.2GPFS V4.1.0.6RAC 12.1.0.2.3

AIX 6.1-TL7-SP03GPFS 3.5.0.8RAC 12.1.0.1, 12.1.0.2

VIOS2 2.2.3.3 2.2.1.3

94

Table 53. Releases certified for use in Oracle environments (continued)

IBM Spectrum Scale or GPFS Versions

11gR2 AIX 7.1 AIX 7.1-TL2-SP04GPFS 4.1RAC 11.2.0.4

AIX 7.1-TL1-SP02GPFS 3.5.0.1RAC 11.2.0.1, 11.2.0.2, 11.2.0.3

AIX 6.1 AIX 6.1-TL8-SP04GPFS 4.1RAC 11.2.0.4

AIX 6.1-TL7-SP03GPFS 3.5.0.1RAC 11.2.0.1, 11.2.0.2, 11.2.0.3

VIOS2 VIOS 2.2.1.7 VIOS 2.2.1.3

Note:a. Oracle certification is for storing Database in the IBM Spectrum Scale direct access model.

Configuring an Oracle database to access through Protocol Nodes is not certified.b. If AIX, GPFS, IBM Spectrum Scale or Oracle RAC minimum certified levels are different

while using a VIOS , the levels are specially noted under VIOS rows. Otherwise, for theVIOS rows that don’t specify levels of AIX, please see corresponding RAC and AIX releaserows to determine minimum certified levels.

v For the latest GPFS certification information:– Login to Oracle support (https://support.oracle.com/ ) and search for [ID 1376369.1.– IBM Techdocs:

- IBM GPFS 4.1 is available on Oracle Database 11gR2 with AIX 7.1 and AIX 6.1 athttp://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FLASH10850

- IBM GPFS 3.5 available on Oracle Database 11gR2 with AIX 7.1 and AIX 6.1 athttp://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FLASH10791

- IBM’s PowerVM® LPM is available for Oracle DB RAC for 11gR2 with GPFS 3.5 & 3.4at http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FLASH10799

v For AIX go to, http://www.oracle.com/technetwork/database/clustering/tech-generic-unix-new-166583.html

v For Linux go to, http://www.oracle.com/technetwork/database/clustering/tech-generic-linux-new-086754.html

Note:a. There are no currently supported levels of IBM Spectrum Scale qualified with Linux on

Power.b. Oracle has not been certified with IBM Spectrum Scale on Linux on Intel and there are

no current plans to do so.c. Oracle as a workload is not supported with IBM Spectrum Scale Express® Edition V4.1

for Linux on Z.2. For the list of virtualization and partitioning technologies supported by Oracle, go to

http://www.oracle.com/technetwork/database/virtualizationmatrix-172995.htmlQ6.3: Are there any considerations when utilizing the Remote Direct Memory Access (RDMA)?A6.3: IBM Spectrum Scale supports RDMA on Linux only. IBM Spectrum Scale uses the VERBS

programming interface to provide RDMA support. While IBM Spectrum Scale uses the VERBSprogramming interface for RDMA support, the underlying implementation of RDMA isvendor-specific.

IBM Spectrum Scale supports RDMA in the following configurations:v RDMA over Infiniband fabrics is supported on the following Linux RDMA stacks, provided

that the Distribution version and kernel are supported by IBM Spectrum Scale:– Mellanox RDMA stacks on ppc64 and x86_64, provided that the Mellanox HCA, Distribution

version, and kernel are supported by the Mellanox RDMA stack.– Linux Distro RDMA stacks on ppc64 and x86_64 provided that the Mellanox HCA,

Distribution version, and kernel are supported by Mellanox.

95

https://support.oracle.com/

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/FLASH10850



http://www.oracle.com/technetwork/database/clustering/tech-generic-unix-new-166583.html

http://www.oracle.com/technetwork/database/clustering/tech-generic-unix-new-166583.html

http://www.oracle.com/technetwork/database/clustering/tech-generic-linux-new-086754.html

http://www.oracle.com/technetwork/database/clustering/tech-generic-linux-new-086754.html

http://www.oracle.com/technetwork/database/virtualizationmatrix-172995.html

v RDMA over Omni-Path fabrics is supported on the following Linux RDMA stacks, providedthat the Distribution version and kernel are supported by IBM Spectrum Scale:– Intel RDMA stacks on x86_64, provided that:

- The Intel HFI, Distribution version, and kernel are supported by the Intel RDMA stack.- Spectrum Scale V4.2.1, or later, is required to enable Omni-Path 8K path MTU support.

Omni-Path 8K MTU support is enabled with the mmchconfigverbsRdmaQpRtrPathMtu=8192 command

– Linux Distro RDMA stacks on x86_64, provided that the Intel HFI, Distribution version, andkernel are supported by Intel.

v RDMA over Converged Ethernet (RoCE) is supported on the following Linux RDMA stacksprovided that the Distribution version and kernel are supported by IBM Spectrum Scale:– Mellanox RDMA stacks on ppc64 and x86_64, provided that:

- The Mellanox HCA, Distribution version, and kernel are supported by the MellanoxRDMA stack.

- RDMA Connection Manager (RDMA-CM) must be enabled with the mmchconfigverbsRdmaCm=enable command.

The following restrictions apply for IBM Spectrum Scale RDMA support:v The protocols export over CES does not utilize RDMA.v A single IB subnet is supported.

Clusters that make use of multiple fabrics that are not connected should use the mmchconfigverbsPorts=Device/Port/Fabric option to ensure proper RDMA connections are created.

v Infiniband and Omni-Path partition keys are not supported by Spectrum Scale.GPFS uses partition key index zero by default.

v Support for single port HCAs or HFIs using RHEL 6.6 or later must use IBM SpectrumScale/GPFS V3.5.0.20 or later.

v Mellanox Connect-IB restrictions:– GPFS pagepool size must be 3840MB or less for Mellanox OFED version less than V2.3.– GPFS pagepool size greater than 3840MB is supported with GPFS V4.1.0.6 or V3.5.0.23 or

later.– Connect-IB is supported with IBM Spectrum Scale V4.1.0.6 or later, and V3.5.0.23 or later.– IBM Spectrum Scale on ppc64 does not support Connect-IB with Mellanox OFED version

2.4-1.0.4.v RDMA over Converged Ethernet (RoCE) restrictions:

– All nodes must use IBM Spectrum Scale V4.1.0.4 or later.– If a node is using multiple ports for RoCE, the IPv4 IP address for each port must belong to

a different IP subnet.– IPv6 must be enabled to use RoCE, if interfaces are selected using the port name.

v RDMA stacks based on OFED 1.1 and OFED 1.2 are not supported.v RDMA is not supported on a node when both Mellanox HCAs and Intel Omni-Path HFIs are

enabled for RDMA.

Note:1. Ensure you are at the latest firmware level for both your switch and adapter.2. When enabling Infiniband on AMD64 hardware, iommu=soft may be required in grub boot

options to permit allocations greater than 1GB to the VERBS RDMA device. This may impactperformance and CPU utilization.

3. See the question What are the current advisories for IBM Spectrum Scale on Linux?Q6.4: What configuration and performance tuning suggestions are there for the Active File

Management function of GPFS?A6.4: In addition to the performance tuning suggestions in the IBM Spectrum Scale: Advance

Administration Guide:v There is a known TCP performance issue with the NFS server in certain kernel releases. It is

suggested for best performance to use RHEL 6.1 (or later) or SLES 11 SP2 (or later) for the NFSserver in a cache relationship.

96



Q6.5: Sometimes GPFS appears to be handling a heavy I/O load, for no apparent reason. What couldbe causing this?

A6.5: On some Linux distributions the system is configured by default to run the file system indexingutility updatedb through the cron daemon on a periodic basis (usually daily). This utilitytraverses the file hierarchy and generates a rather extensive amount of I/O load. For this reason,it is configured by default to skip certain file system types and nonessential file systems.However, the default configuration does not prevent updatedb from traversing GPFS file systems.

In a cluster this results in multiple instances of updatedb traversing the same GPFS file systemsimultaneously. This causes general file system activity and lock contention in proportion to thenumber of nodes in the cluster. On smaller clusters, this may result in a relatively short-livedspike of activity, while on larger clusters, depending on the overall system throughput capability,the period of heavy load may last longer. Usually the file system manager node will be thebusiest, and GPFS would appear sluggish on all nodes. Re-configuring the system to either makeupdatedb skip all GPFS file systems or only index GPFS files on one node in the cluster isnecessary to avoid this problem.

Q6.6: What considerations are there when using IBM Spectrum Protect with IBM Spectrum Scale?A6.6: Considerations when using IBM Spectrum Protect with IBM Spectrum Scale include:

v When using IBM Spectrum Protect with IBM Spectrum Scale, verify the supportedenvironments:– IBM Spectrum Protect for Space Management technotes:

- For Linux x86 at http://www.ibm.com/support/docview.wss?uid=swg21248771- For AIX at http://www.ibm.com/support/docview.wss?uid=swg21248419- For Linux on Z at http://www.ibm.com/support/docview.wss?uid=swg21966164

– General overview on the integration between IBM Spectrum Scale and Spectrum Protect:https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Integrating%20IBM%20Tivoli%20Storage%20Manager%20with%20IBM%20Elastic%20Storage

– Tivoli Field Guide for TSM for Space Management for UNIX-GPFS Integration athttp://www-01.ibm.com/support/docview.wss?uid=swg27018848

– IBM Spectrum Protect Requirements for IBM AIX Client at http://www.ibm.com/support/docview.wss?uid=swg21052226

– IBM Spectrum Protect Linux x86 Client Requirements at http://www.ibm.com/support/docview.wss?uid=swg21052223

– IBM Spectrum Protect Linux on Z at http://www-01.ibm.com/support/docview.wss?rs=663&context=SSGSG7&q1=clientrequirements&uid=swg21066436

– To search IBM Spectrum Protect support information go to www.ibm.com/software/sysmgmt/products/support/IBMTivoliStorageManager.html and enter GPFS as the searchterm

v When configuring IBM Spectrum Scale Active File Management, see https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Configuring%20IBM%20Spectrum%20Scale%20Active%20File%20Management

v Quota limits are not enforced when files are recalled from the backup using IBM SpectrumProtect . This is because dsmrecall is invoked by the root user who has no allocationrestrictions according to the UNIX semantics.

v IBM Spectrum Protect Backup Archive 7.1.3 client is the only supported version to work withIBM Spectrum Scale 4.1.1

v IBM Spectrum Protect Backup Archive 7.1.1 client is only verified to work with IBM SpectrumScale V4.1 or later.

v IBM Spectrum Protect Backup Archive 7.1.0 client is only verified to work with GPFS V3.5 orlater. IBM Spectrum Scale V4.1 is not supported at this level.

v IBM Spectrum Protect Backup Archive 6.3 client is only verified to work with GPFS V3.4.0.4 orlater and V3.5. IBM Spectrum Scale V4.1 is not supported at this level.

v A DMAPI-enabled file system may be mounted on a Windows node, with certain restrictions.For more information, refer to the following resources:

97

http://www.ibm.com/software/tivoli/products/storage-mgr/

http://www.ibm.com/support/docview.wss?uid=swg21248771



https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Integrating%20IBM%20Tivoli%20Storage%20Manager%20with%20IBM%20Elastic%20Storage



http://www-01.ibm.com/support/docview.wss?uid=swg27018848





http://www-01.ibm.com/support/docview.wss?rs=663&context=SSGSG7&q1=clientrequirements&uid=swg21066436

http://www-01.ibm.com/support/docview.wss?rs=663&context=SSGSG7&q1=clientrequirements&uid=swg21066436

http://www-01.ibm.com/support/docview.wss?rs=663&tc=SSGSG7&uid=swg21248771


https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Configuring%20IBM%20Spectrum%20Scale%20Active%20File%20Management



– IBM Spectrum Protect Version 6.3 Information Center at http://publib.boulder.ibm.com/infocenter/tsminfo/v6r3/index.jsp

– IBM Spectrum Protect support page at http://www-01.ibm.com/support/docview.wss?rs=663&tc=SSGSG7&uid=swg21248771

Q6.7: Does GPFS V4 use OpenSSL ?A6.7: GPFS V4 does not use OpenSSL for secure communication across nodes. Instead, it uses the

GSKit toolkit, which is shipped in all Editions of GPFS V4.1 and later, as gpfs.gskit.

Q6.8: How do I get OpenSSL to work with GPFS V3?

A6.8: With GPFS V3, to help enhance the security of mounts using Secure Sockets Layer (SSL) aworking version of OpenSSL must be installed. This version must be compiled with support forthe Secure Hash Algorithm (SHA).

On Linux, OpenSSL libraries available with your OS distribution can be used. On AIX, OpenSSLpackages are available on the AIX OS media (AIX 6 and newer).

On Windows, GPFS versions prior to and including 3.5.0.1 do not require OpenSSL to beseparately installed. However, beginning with version 3.5.0.2, GPFS on Windows requires the 2DLLs SSLEAY32.DLL and LIBEAY32.DLL to be installed on the system. An end user can eitherdownload the code from www.OpenSSL.org and build the Release version of these 2 DLLs. Or,you can install a 3rd party prebuilt package, such as Win64 OpenSSL v1.0.1m Light from ShiningLight Productions at http://slproweb.com/products/Win32OpenSSL.html.

Note: If the installation of the package from Shining Light Productions warns about any missingVisual C++ Redistributables (x64), the user is encouraged to install the same by following thelink to the Microsoft download site.

In many cases, GPFS will be able to locate and use OpenSSL libraries by default. However, insome cases defaults may not be sufficient, and GPFS configuration needs to be changed to pointat the right set of libraries, for example:v On AIX 64-bit kernel:

mmchconfig openssllibname="/usr/lib/libssl.a(libssl64.so.0.9.8)"

v On Linux:mmchconfig openssllibname="/usr/lib64/libssl.so.0.9.8"

v On Windows:For GPFS to locate and use these 2 necessary DLLs (ssleay32.dll and libeay32.dll), at least oneof the following must be true:– The 2 DLLs are copied to the Windows System directory C:\Windows\System32).– The Shining Light installer gives the options:– The 2 DLLs are copied into the GPFS bin directory C:\Windows\SUA\usr\lpp\mmfs\bin.– The GPFS parameter openssllibname is set as:

mmchconfig openssllibname="C:/OpenSSL64/bin/ssleay32.dll;C:/OpenSSL64/bin/libeay32.dll"

Here C:/OpenSSL64 is the folder where the Shining Path package is installed.

Note: The full path to both of these DLLs must be specified using forward '/'s. Asemi-colon ';' must be used as the name separator.

Q6.9: What ciphers are supported for transport security by IBM Spectrum Scale V4?A6.9:

TLS has an inherent limitation in that the protocol does not periodically refresh the key materialthat is used to protect the data that is exchanged. Depending on the cipher suite that is used, andthe amount of data that is transmitted between two nodes, if the key is not updated after thethreshold for that cipher is reached, the data might be at risk to an attack that would compromisethe confidentiality or integrity of the transmitted data. The AES-GCM cipher suite (only availableon TLS 1.2) is affected by this issue because the number of bytes that can be safely exchanged on

98

http://publib.boulder.ibm.com/infocenter/tsminfo/v6r3/index.jsp

http://publib.boulder.ibm.com/infocenter/tsminfo/v6r3/index.jsp



http://www.openssl.org

http://slproweb.com/products/Win32OpenSSL.html

a single TLS session is the lowest among cipher suites. This limit is on the order of hundreds ofGiB. For more information, see the following links:v http://dx.doi.org/10.6028/NIST.SP.800-38Dv http://www.isg.rhul.ac.uk/~kp/TLS-AEbounds.pdf

This issue could be exploited by an attacker with the capability to collect large quantities ofnetwork traffic exchanged between two nodes and then perform sophisticated cryptanalysis todecrypt part of the traffic exchanged, or be able to inject messages in the encryptedcommunications between two nodes (with partial control over their content). IBM Spectrum Scaleuses long-lived TLS connections and when using the AES-GCM cipher suite it might exchangeenough data to increase the risk of this type of weakness being exploited.v For environments where nistCompliance=off

– AES128-SHA– AES256-SHA

v For environments where nistCompliance=SP800-131A– AES128-SHA– AES128-SHA256– AES256-SHA– AES256-SHA256

Note:v When a cluster contains both GPFS V3 (use of OpenSSL) and IBM Spectrum Scale V4 (use of

GSKit) nodes, ensure the use of a cipher that is supported by all nodes:– AES128-SHA– AES256-SHA

v IBM Spectrum Scale also supports the keywords DEFAULT, EMPTY, and AUTHONLY inplace of a cipher list. DEFAULT, EMPTY, and AUTHONLY are not affected by this issue. Thedefault security mode is EMPTY in IBM Spectrum Scale V4.1 or earlier and is AUTHONLY inIBM Spectrum Scale V4.2 or later. When EMPTY is specified, IBM Spectrum Scale does notauthenticate or check authorization for network connections, or encrypt transmitted data. WhenAUTHONLY is specified, IBM Spectrum Scale checks network connection authorization, butdata that is sent over the connection is not encrypted, therefore not protected.

Q6.10: When I allow other clusters to mount my file systems, is there a way to restrict accesspermissions for the root user?

A6.10: Yes. A root squash option is available when making a file system available for mounting by otherclusters using the mmauth command. This option is similar to the NFS root squash option. Whenenabled, it causes GPFS to squash superuser authority on accesses to the affected file system onnodes in remote clusters.

This is accomplished by remapping the credentials: user id (UID) and group id (GID) of the rootuser, to a UID and GID specified by the system administrator on the home cluster, for example,the UID and GID of the user nobody. In effect, root squashing makes the root user on remotenodes access the file system as a non-privileged user.

Although enabling root squash is similar in spirit to setting up UID remapping, there are twoimportant differences:1. While enabling UID remapping on remote nodes is an option available to the remote system

administrator, root squashing need only be enabled on the local cluster, and it will beenforced on remote nodes.

2. While UID remapping requires having an external infrastructure for mapping between localnames and globally unique names, no such infrastructure is necessary for enabling rootsquashing.

When both UID remapping and root squashing are enabled, root squashing overrides the normalUID remapping mechanism for the root user. See the mmauth command man page for furtherdetails.

99

http://dx.doi.org/10.6028/NIST.SP.800-38D

http://www.isg.rhul.ac.uk/%7Ekp/TLS-AEbounds.pdf

Note: Administrators who use UID remapping to configure users with many group membershipsare advised to ensure that ID remapping helper functions (IRHF) scale appropriately. For moreinformation about UID remapping, see https://www.ibm.com/support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_uid/uid_gpfs.html.

Q6.11: How do I determine the maximum size of the extended attributes allowed in my file system?

A6.11: As of GPFS 3.4, the space allowed for extended attributes for each file was increased and theperformance to get and set the extended attributes was improved. To determine which version ofextended attribute your file system uses, issue the mmlsfs --fastea command. If the new fastextended attributes are enabled, yes will be displayed on the command output. In this case, thetotal space for user-specified extended attributes has a limit of 50K out of 64K and the size ofeach extended attribute has a limit of 16K, otherwise the total space limit is 8K out of 16K andthe size of each extended attribute has a limit of 1022 bytes.

Q6.12: What are configuration considerations when using IPv6?

A6.12: Considerations when using IPv6 include:v IPv4 subnets are not supported on a cluster that is defined with IPv6 primary addresses

(hostname) that contains Windows nodes.v IBM Spectrum Scale does not support IPv6 with

– Clustered NFS– The integrated protocol access methods available with IBM Spectrum Scale V4.1.1 or later– The GUI– The Vormetric key server– Transparent cloud tiering

For additional information, please see the IBM Spectrum Scale: Advanced Administration Guide

Q6.13: How should IBM Spectrum Scale V4 Advanced Edition or Data Management Edition beconfigured to only use FIPS 140-2-certified cryptographic engines?

A6.13: To only use FIPS 140-2-certified cryptographic engines, you need to perform the following twosteps before performing any configuration steps to enable encryption:1. On IBM Security Key Lifecycle Manager (ISKLM), turn the FIPS configuration parameter on.

See the ISKLM installation guide for more information at https://www.ibm.com/support/knowledgecenter/SSWPVP_2.7.0/com.ibm.sklm.doc/welcome.htm.

If you installed Vormetric Data Security Manager, see the Configuring encryption with theVormetric DSM key server topic for information about using FIPS 140-2.

2. Issue the command mmchconfig FIPS1402Mode=yes.

Note:

v Core GPFS components (communication across nodes and native encryption) use GSKit andcan be configured to operate in FIPS-compliant modes; integrated protocol components useother cryptographic libraries and are currently not FIPS-compliant.

v FIPS mode is supported on the POWER8 processor in little endian mode in IBM SpectrumScale V4.2.1 and later.

Q6.14: What are the configuration requirements for the use of different IBM Spectrum Scale Editions?

A6.14: The following configuration rules must be followed for the various Editions of GPFS V4:v All nodes in a cluster must be running the same Edition.v In a multicluster environment, Express clusters and Standard clusters may interact.v In a multicluster environment, if encryption is required, all clusters must be at the Advanced

Edition.v IBM Spectrum Scale Advanced Edition is not supported on Windows.

100

https://www.ibm.com/support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_uid/uid_gpfs.html

https://www.ibm.com/support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_uid/uid_gpfs.html


https://www.ibm.com/support/knowledgecenter/SSWPVP_2.7.0/com.ibm.sklm.doc/welcome.htm

https://www.ibm.com/support/knowledgecenter/SSWPVP_2.7.0/com.ibm.sklm.doc/welcome.htm

https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.2/com.ibm.spectrum.scale.v4r22.doc/bl1adv_encryptionenv_vormetric.htm

https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.2/com.ibm.spectrum.scale.v4r22.doc/bl1adv_encryptionenv_vormetric.htm

Q6.15: What are the configuration and tuning considerations when using the IBM Spectrum Scaleintegrated protocols access methods?

A6.15: Configuration considerations for using the integrated protocols access methods include:v Client nodes can access integrated protocols with integrated NFS, SMB, and Object services

with the CES infrastructure using only IPV4.v For additional information, see the wiki at https://www.ibm.com/developerworks/

community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Protocols

Q6.16: What considerations are there when using IBM Spectrum Archive with IBM Spectrum Scale?

A6.16: For the latest support information about using IBM Spectrum Archive with IBM Spectrum Scale,see the Required software for Linux systems topic in the IBM Spectrum Archive EnterpriseEdition (EE) Knowledge Center.

Note: To see support information about a specific release of IBM Spectrum Archive EnterpriseEdition, select from the options in the drop-down menu on the upper left corner of the page.

101

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Protocols



https://www.ibm.com/support/knowledgecenter/en/ST9MBR_1.2.5/ltfs_ee_sw_req.html

Virtualization questionsQ7.1: How do I determine the number of licenses required when running IBM Spectrum Scale in

VMs in a virtualization environment?A7.1:

IBM Spectrum Scale V4 for Power and x86_64 is licensed by socket. However, IBM virtualizationlicensing rules are defined by the number of virtual cores allocated to the program. Thevirtualization environment might allow different virtual cores to be assigned to different physicalsockets at different times; therefore, it is not always possible to determine the number of physicalsockets in use by IBM Spectrum Scale.

Instead, to determine the number of socket licenses required, the number of virtual coresallocated to IBM Spectrum Scale is converted into a corresponding number of socket licenses asfollows:1. Count the total number of cores and sockets available in the virtualization environment.

Consider the following examples:v Environment A has 10 machines each with 2 sockets. Each socket has 4 cores. In total, A has

20 sockets and 80 cores.v Environment B has 10 machines each with 2 sockets. On 5 of the machines, each socket has

4 cores. On the other 5, each socket has 8 cores. In total, B has 20 sockets and 120 cores.2. Determine the average number of cores per socket.v For A, the average is 4 cores per socket. Since all the sockets in A are identical, this is the

same as the actual number of cores for each socket.v For B, the average is 6 cores per socket.

Note: If the average is not a whole integer, round down. For example, 5.7 cores per socketbecomes 5 cores per socket.

3. Count the number of virtual cores allocated to the program. Divide by the average number ofcores per socket to determine the required number of socket licenses.

Note: If the number of socket licenses is not a whole number, round up. For example, 8.3socket licenses becomes 9 socket licenses

4. Finally, IBM virtualization licensing rules specify that the total licenses required can neverexceed the licenses required for the entire physical environment. If more cores are allocatedthan physically available, the number of licenses is capped to the physical limit. Consider thefollowing example:v Environment C has 5 machines each with 2 sockets and 4 cores for a total of 40 cores. 40

instances of IBM Spectrum Scale are pinned to these machines, and each is allocated 2virtual cores for a total of 80 virtual cores. You would license only 40 cores, the physicalmaximum, which is equivalent to 10 sockets.

Note:1. For all nodes (LPARs) configured as uncapped, the number of virtual processors must be

taken into account.2. For all nodes (LPARs) configured as capped, the value of entitled capacity must be taken into

account.

With IBM Spectrum Scale V4 for Linux on Z, licenses are required for the cores/IFLs available toGPFS. See the following links for more information:v https://www-112.ibm.com/software/howtobuy/passportadvantage/valueunitcalculator/

vucalc.wss?jadeAction=GUIDE_TREE#achorv http://www-03.ibm.com/systems/z/resources/swprice/subcap/linux.htmlv http://www-03.ibm.com/systems/z/os/linux/solutions/ifl.htmlv http://www-03.ibm.com/systems/z/resources/swprice/zipla/

With GPFS V3, the number of processors for which licenses are required is the smaller of thefollowing:

102

https://www-112.ibm.com/software/howtobuy/passportadvantage/valueunitcalculator/vucalc.wss


http://www-03.ibm.com/systems/z/resources/swprice/subcap/linux.html

http://www-03.ibm.com/systems/z/os/linux/solutions/ifl.html

http://www-03.ibm.com/systems/z/resources/swprice/zipla/

v The total number of activated processors in the machine.v When GPFS nodes are in partitions with dedicated processors, then licenses are required for

the number of processors dedicated to those partitions.v When GPFS nodes are LPARs that are members of a shared processing pool, then licenses are

required for the smaller of:– the number of processors assigned to the pool or– the sum of the virtual processors of each uncapped partition plus the entitled capacity in

each capped partitionAn LPAR is defined as one or more virtualized images of a hardware computing system thatcan include shared and dedicated resources assigned from the pool of resources available on aphysical server. Each image appears to the operating system running within it to be a uniqueinstance of a physical server.

For example:1. One GPFS node is in a partition with .5 of a dedicated processor → license(s) are required for 1

processor2. 10 GPFS nodes are in partitions on a machine with a total of 5 activated processors → licenses

are required for 5 processors3. LPAR A is a GPFS node with an entitled capacity of say, 1.5 CPUs is set to uncapped in a

processor pool of 5 processors.LPAR A is used in a way that requires server licenses.LPAR B is a GPFS node that is on the same machine as LPAR A and is also part of the sharedprocessor pool as LPAR A.LPAR B is used in a way that does not require server licenses so client licenses are sufficient.LPAR B has an entitled capacity of 2 CPUs, but since it too is uncapped, it can use up to 5processors out of the pool.For this configuration server licenses are required for 5 processors.

Note: Any fractional part of a processor in the total calculation must be rounded up to a fullprocessor.

For WPARs, please see question Can GPFS run in a Workload Partitioning (WPAR) environment?

For Linux virtualized NSD clients, the number of licenses required is equal to the physical coresavailable to GPFS.

When the same processors/cores are available to both GPFS Server nodes and GPFS Client nodes,GPFS Server licenses are required for those processors/cores

Please referencev Counting Software licensesv http://public.dhe.ibm.com/software/passportadvantage/SubCapacity/

Scenarios_Power_Systems.pdfv ftp://ftp.software.ibm.com/software/passportadvantage/SubCapacity/

Eligible_Virtualization_Technology.pdfQ7.2: How do I determine whether a server license or a client license is required when running IBM

Spectrum Scale in VMs in a virtualized environment?A7.2: Whether you need a server license or a client license is determined by the function of the virtual

node. Virtual hosts that perform management functions such as cluster configuration manager,quorum node, manager node, Network Shared Disk (NSD) server, and protocol node require aserver license. Virtual hosts/servers only providing a disk image to a local virtual machine (aguest) may be licensed via a client license as no management functions are performed.

With a client license, IBM Spectrum Scale can also execute in the hypervisor or in a VM and thenexport data to VMs or daemons executing on the same physical server via a protocol such as NFSas long as the client license covers all sockets available to all the VMs on that physical server.

Q7.3: Is IBM Spectrum Scale on Linux (x86 and Power) supported in a virtualization environment?

103

http://www-01.ibm.com/software/lotus/passportadvantage/Counting_Software_licenses_using_specific_virtualization_technologies.html

http://public.dhe.ibm.com/software/passportadvantage/SubCapacity/Scenarios_Power_Systems.pdf

http://public.dhe.ibm.com/software/passportadvantage/SubCapacity/Scenarios_Power_Systems.pdf

ftp://ftp.software.ibm.com/software/passportadvantage/SubCapacity/Eligible_Virtualization_Technology.pdf

ftp://ftp.software.ibm.com/software/passportadvantage/SubCapacity/Eligible_Virtualization_Technology.pdf

A7.3: In a virtualization environment. the level of support depends on whether an individual node isan NSD server (has direct-attached or SAN-attached disks) or an NSD client. The following arethe support matrices when GPFS is run in a virtualization environment:

Note: IBM Spectrum Scale is only supported as an NSD client on a XEN guest

Table 54. KVM support matrix on Virtual Machine (VM) Guest

ConfigurationKVMVersion OS Distribution Supported Configurations Known Limitations

GPFS nodeswith no directdisk access

RHEL6.2-x86_64 orhigher

Linux distributions aresupported by both KVMand GPFS

N/A v Live migration is notsupported

v KVM high availability isnot supported

v Local read-only cache isnot supported

GPFS nodeswith directdisk access

RHEL6.2-x86_64 orhigher

Linux distributions aresupported by both KVMand GPFS

v PCI passthroughv Devices supported by

GPFS and KVMv Virtio SCSI is supported

with RHEL 6.4 or higher

v Live migration is notsupported

v KVM high availability isnot supported

v SCSI-3 PR on Virtio is notsupported


Table 55. PowerKVM (pKVM) support matrix on VM Guest

ConfigurationpKVMVersion OS Distribution Supported Configurations Known Limitations


IBMPowerKVMreleases2.1.0,2.1.13.1.x

Linux distributions aresupported by both pKVMand GPFS

N/A v Live migration is notsupported

v pKVM high availabilityis not supported



IBMPowerKVMrelease 2.1.13.1.x

Linux distributions aresupported by both pKVMand GPFS

v PCI passthrough onlyv Devices supported by

GPFS and pKVM

v Live migration is notsupported

v Virtio SCSI is notsupported

v pKVM high availabilityis not supported

Table 56. GPFS support on PowerKVM (pKVM) Host

ConfigurationIBM PowerKVM HostVersion Supported Configurations

GPFS nodes with no diskaccess

IBM PowerKVM Release 3.1 N/A

GPFS nodes with diskaccess

IBM PowerKVM Release 3.1 Disk hardware supported by both pKVM and GPFS.

104

Table 57. VMware support matrix on VM Guest

ConfigurationVMwareVersion OS Distribution Supported Configurations Known Limitations


VMware ESX5.xand6.x

Linux distributions aresupported by bothVMware and GPFS

vSphere vMotion issupported

v vSphere Fault Tolerance(FT) is not supported



VMware ESX5.xand6.x

Supported only withRHEL 6.2 or later, andSLES 11 SP1 or later

Pass-through Raw DeviceMapping (RDM) withphysical compatibilitymode.

For details on how toconfigure RDM, refer toSection 14.3 Raw DeviceMapping with VirtualMachine Clusters athttp://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-storage-guide.pdf

v vSphere vMotion is notsupported

v vSphere Fault Tolerance(FT) is not supported

v Use of Persistent Reserveis not supported

v VMDK disk are notsupported, since EIOerrors are not passedfrom ESX host to guest inAPD(all paths down)condition.


Also see the VMware Compatibility Guide at http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=san&productid=39659&deviceCategory=san&details=1&partner=43&keyword=spectrum%20scale&isSVA=0&page=1&display_interval=500&sortColumn=Partner&sortOrder=Asc

Q7.4: Is IBM Spectrum Scale for Linux on Z supported in a virtualization environment?A7.4: For hardware virtualization, IBM Spectrum Scale for Linux on Z is supported to run in logical

partition (LPAR) mode. For software virtualization, IBM Spectrum Scale for Linux on Z issupported on z/VM as guest operating systems. IBM Spectrum Scale for Linux on Z does notrely on the z/VM versions as long as the z/VM can support the required Linux distributions,kernel levels, and hardware.

See questions:v What Linux distributions are supported by IBM Spectrum Scale?v What are the minimum hardware requirements for an IBM Spectrum Scale cluster?v On what servers is IBM Spectrum Scale supported?

Q7.5: Is IBM Spectrum Scale on Windows supported in a virtualization environment?A7.5: In a virtualization environment, the level of support depends on whether an individual GPFS

node has direct-attached or SAN-attached disks. IBM Spectrum Scale on Windows is supported inthe following virtualization environment:

Table 58. HyperV support matrix on VM Guest

HyperV Version OS DistributionSupportedConfigurations Known Limitations

GPFSnodes withno directdisk access

Windows 2012 R2Windows 2012Windows 2008 R2

Windows 2012 R2Windows 2012Windows 2008 R2Windows 2008 R2Windows 7

v HyperV Live migration isnot supported


GPFSnodes withdirect diskaccess

None None None

105

http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-storage-guide.pdf





http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=san&productid=39659&deviceCategory=san&details=1&partner=43&keyword=spectrum%20scale&isSVA=0&page=1&display_interval=500&sortColumn=Partner&sortOrder=Asc




Table 59. VMware support matrix on VM Guest

VMware Version OS DistributionSupportedConfigurations Known Limitations

GPFSnodes withno directdisk access

VMware ESX 5.1 Windows 2008 R2Windows Server 2012

v vSphere vMotion isnot supported

v vSphere FaultTolerance (FT) is notsupported

v Local read-only cacheis not supported

GPFSnodes withdirect diskaccess

None None None

IBM Spectrum Scale for Windows does not support any kind of raw disk I/O when running as aVM guest.

Q7.6: Can GPFS run in a Workload Partitioning (WPAR) environment?A7.6: GPFS can only be run in the global environment. It is not possible to run the GPFS subsystem or

mount a GPFS file system in a WPAR. A GPFS file system can be made available to a WPARusing namefs.

By definition, a global instance is each AIX operating system that is running. The instanceconsists of all the program and services that compose AIX. If WPARs are inside of an instance ofAIX, the parent AIX is referred to as the global instance. The global instance can share resourceswith the WPARs, but WPARs cannot directly share resources with other WPARs(http://public.dhe.ibm.com/software/passportadvantage/SubCapacity/Scenarios_Power_Systems_AIX_System_WPARs.pdf).

As GPFS does not run in a WPAR, licenses are not required for the WPAR, only the globalinstance. The type of GPFS license required by the global instance depends on what functions theinstance is performing. See questions:v How is IBM Spectrum Scale V4.1 licensed?v How is GPFS V3.

Q7.7: Does GPFS support exploitation of the Virtual I/O Server (VIOS) features of POWERprocessors?

A7.7: Yes, GPFS allows exploitation of POWER VIOS configurations. N_Port ID Virtualization(NPIV),Virtual SCSI (VSCSI), LPM (Live Partition Mobility) and Shared Ethernet Adapter (SEA)are supported in single and multiple Central Electronics Complex (CEC) configurations. Thissupport is limited to GPFS nodes that are using the AIX V7.1, V6.1 or V5.3 operating system or aLinux distribution that is supported by both VIOS (see www14.software.ibm.com/webapp/set2/sas/f/vios/documentation/datasheet.html) and GPFS (see What levels of the AIX OS are supportedby GPFS? and What Linux distributions are supported by GPFS?).

There is no GPFS fix level requirement for this support, but it is recommended that you be at thelatest GPFS level available. For information on the latest levels, go to the GPFS page on FixCentral

For further information on POWER VIOS go to www14.software.ibm.com/webapp/set2/sas/f/vios/documentation/datasheet.html

For VIOS documentation, go to www14.software.ibm.com/support/customercare/sas/f/vios/home.html

Q7.8: Does IBM Spectrum Scale support PowerVM Shared Storage Pool (SSP)?A7.8: Yes, it is supported on PowerVM 2.2.5 and later, IBM Spectrum Scale 4.2.2 and later, and RedHat

7.2 and later on ppc64 NSD server.Q7.9: Are there any virtualization considerations when using an Oracle databases?A7.9: Ensure the virtualization or partitioning technology you are utilizing is supported by both GPFS

106

http://public.dhe.ibm.com/software/passportadvantage/SubCapacity/Scenarios_Power_Systems_AIX_System_WPARs.pdf

http://public.dhe.ibm.com/software/passportadvantage/SubCapacity/Scenarios_Power_Systems_AIX_System_WPARs.pdf

https://www14.software.ibm.com/webapp/set2/sas/f/vios/documentation/datasheet.html


http://www-933.ibm.com/support/fixcentral/




http://www14.software.ibm.com/support/customercare/sas/f/vios/home.html

http://www14.software.ibm.com/support/customercare/sas/f/vios/home.html

and Oracle. For the list of virtualization and partitioning technologies supported by Oracle, go tohttp://www.oracle.com/technetwork/database/virtualizationmatrix-172995.html

Q7.10: Can Linux on Z with GPFS run in logical partition (LPAR) mode or on z/VM as guestoperating system?

A7.10: Yes, GPFS cluster nodes on Linux on Z can be configured either as a NSD server or as a NSDclient. All of the cluster nodes can run on LPAR directly or on z/VM as a guest operating system.

107

http://www.oracle.com/technetwork/database/virtualizationmatrix-172995.html

Integrated Protocol Server questions

Note: Additionally, see the questions:v What are the scaling considerations for the integrated protocols access methods?

v What Linux distributions are supported by the integrated protocols access methods in IBM Spectrum ScaleV4.1.1?

v What are the configuration requirements for the use of the integrated protocols access methods?Q8.1: What are the requirements to use the protocol access methods integrated with IBM Spectrum

Scale V4.1.1 and later?A8.1: Requirements to utilize the integrated protocols access methods include the following:

v With IBM Spectrum Scale V4.2.1 and later, protocols can be served from RHEL 7, 7.1, and 7.2on Intel servers or big endian POWER servers. With V4.2.2 and later, protocols can also beserved from RHEL 7, 7.1, and 7.2 on little endian POWER servers.

v With IBM Spectrum Scale V4.1.1 and V4.2, protocols can be served from RHEL 7 and 7.1 onIntel servers or (Big Endian) on POWER servers.

v SMB/NFS may run on SLES 12 on Intel servers.

Note: The IBM Spectrum Scale cluster itself can contain nodes with other operating systems.v Protocol functionality is available in the Standard and Advanced editions of IBM Spectrum

Scale.v Nodes configured as a protocol node must have an IBM Spectrum Scale server license

designation.v The IBM Spectrum Scale cluster must be configured to use the Cluster Configuration

Repository (CCR).– For the mmhealth command to work, the cluster must be configured to use the CCR.

v All nodes in the cluster should be at 4.1.1 or later and should have run the mmchconfigrelease=LATEST command successfully.

v Oracle certification is for storing Database in the IBM Spectrum Scale direct access model.Configuring an Oracle database to access through Protocol Nodes is not certified.

Q8.2: What is the minimum hardware requirement for a protocol node?A8.2: The protocol functionality is software only delivery, so the capability and performance is based

on the configuration you choose.v If you are going to enable only one of either NFS or Object, we recommend a minimum of 1

CPU socket server of the latest POWER or Intel variety with at least 64 GB of memory.v If you are going to enable multiple protocols or if you enable SMB, then we recommend a

minimum of 2 CPU socket server of the latest POWER or Intel variety with at least 128 GB ofmemory.

v Network configuration is important so we recommend at least a 10Gb Ethernet connection forprotocol client access.

Q8.3: What are some configuration considerations when deploying the protocol functionality?A8.3: Configuration considerations include:

v When using the Installation Toolkit, the IBM Spectrum Scale Object protocol functionalityrequires the following SELinux packages to be installed:– selinux-policy-base at 3.13.1-23 or higher– selinux-policy-targeted at 3.12.1-153 or higher

v As with basic IBM Spectrum Scale functionality, the protocol function also relies onadministrator of the cluster to setup networking appropriately. This includes ensuring that theappropriate firewall ports are opened as well as ensuring that the Domain Name Service (DNS)is configured for hostname lookups as well as reverse hostname lookups

v The NFS functionality that is provided with Cluster Export Services (CES) cannot coexist withthe Clustered NFS function (CNFS). If you want to use the SMB and Object functionsintegrated with CES, you have to migrate from CNFS to CES NFS. As you plan for thatmigration, note that CES NFS failover group functionality is not completely equivalent withCNFS.

108

v If NFS stack, on IBM Spectrum Scale home, is migrated to the integrated protocol export, anyremote cluster caching data needs to clear cache and have it repopulated.

v IBM Spectrum Scale Clustered NFS (CNFS) and integrated protocol support using the clusterexport services are not available on the same cluster.

v SMB1 is not supported.v IPV6 protocol support does not extend to exporting the NFS/SMB/Swift protocols.v While the IBM Spectrum Scale cluster uses RDMA, the NFS, SMB, and Object protocols do not

utilize RDMA.v Several GPFS configuration aspects have not been explicitly tested with the protocol function:

– Local Read Only Cache– Protocol as well as NSD serving functions can coexist on the same systems if the hardware

is capable of handling both workloads in terms of network, CPU and memory. In largerscale deployments its advised to separate the functions on separate hardware.

– If you have an FPO configuration and if you want to use integrated protocol function, theprotocol nodes should be nodes that are not FPO disk servers.

v The protocol software includes open source components of NFS server (Ganesha), SMB(Samba) and Openstack Swift (from IBM Cloud Manager). You should only use the versions ofthese components provided for use in the integration with CES (failover/monitoring) providedby IBM.

Q8.4: What is the guidance for NFS clients to be configured with the CES NFS function?A8.4: It is generally recommended that the NFS client mount with the options mount -o hard,intr.

Mounting with -o soft is strongly discouraged because of the risk of data loss or corruption.Hard mounts and intr (interruptible) enables the application to be sure of a successful write. Inaddition, it is advised that the GPFS filesystem used for the NFS export have the syncnfs option.Use the mmlsfs command to display if the sync option is set:mmlsfs gpfs_file_system -o

and the mmchfs command to change the syncfs option:mmchfs gpfs_file_system -o syncfs

The AIX NFS client is unable to reestablish a connection with the NFS server after NFS serverfailover. To resolve this problem, complete the following steps:1. Ensure that the AIX version of the NFS client is 6.1 or 7.1.2. Install any of the following IBM APARs from the IBM AIX support site:v AIX 6.1: AIX 6.1 TL7 SP4 or earlier versions up to AIX 6.1 TL6 SP0 with an ifix for IV07784

and IV07918v AIX 7.1: AIX 7.1 TL0 or later versions with ifix for IV04555, IV08311, and IV08310

Use the NFS mount options hard,intr,timeo=1000,dio,noac on the AIX client. For example:mount -o hard,intr,timeo=1000,dio,noac spectrumScaleCESIP:/path/to/exportedDirectory

/localMountPoint

NFSv4.0 server uses 64-bit cookies for readdir and the IBM AIX NFS client truncates them into32-bits. This causes the readdir from the AIX NFS clients to fail. To resolve this problem, installthe following APARs from the IBM AIX support site:v AIX 6.1 TL6 SP10 : IV28464v AIX 6.1 TL7 SP6 : IV28372v AIX 6.1 TL8 : IV25166v AIX 7.1 TL0 SP8 : IV26554v AIX 7.1 TL1 SP6 : IV28894v AIX 7.1 TL2 : IV24863

Q8.5: Can we configure protocol nodes with an ESS/GSS?A8.5:

Configuration requirements to run with an ESS/GSS with V4.2 and later:v Prior to running the 4.2.1.x Installation Toolkit for protocol deployment on a cluster containing

an ESS, all servers in a cluster need to have IBM Spectrum Scale at V4.2.0.0 or later, code level

109

to use the protocol function. To allow the use of protocols in this special case of using ESS NSDservers (since the IBM Spectrum Scale version on ESS is not entirely under the control of theuser) along with other servers running V4.2.0.0 or later, we have additional configurationrequirements:1. The ESS/GSS has to run latest ESS/GSS level that supports V4.2.0.0 or later.2. Install IBM Spectrum Scale manually on the protocol nodes using rpms from the

/usr/lpp/mmfs/4.2.x.x/gpfs_rpmsdirectory?3. Join the protocol nodes to the existing ESS cluster using the mmaddnode command4. The cluster should have CCR enabled. Issue the mmlscluster command to determine if

CCR is enabled. Issue the mmchcluster -ccr-enable command to enable CCR if needed.5. Any nodes designated as quorum or manager nodes must be running 4.2.0.0 or later code.

Depending upon the configuration, this may mean movement of quorum and/or managerfunction to higher level nodes within the cluster.

6. ESS nodes need to be in the same cluster as the protocol nodes that export ESS filesystems.

7. We expect a node class called gss or gss_ppc64 (mmlsnodeclass --all)8. Input the protocol nodes into the Installation Toolkit (do not input the ESS IO nodes nor

EMS node).9. Configure the protocols using the Installation Toolkit.

10. Proceed with a protocol deployment using the Installation Toolkit .11. Run protocol CLI commands from a protocol node if other nodes in the cluster are at a

lower level.

Configuration requirements to run with an ESS/GSS at pre V4.2.0.0 levels:v Prior to running the Installation Toolkit for protocol deployment on a cluster containing an

ESS, all servers in a cluster need to have IBM Spectrum Scale at V4.1.1 or later, code level touse the protocol function. To allow the use of protocols in this special case of using ESS NSDservers (since the IBM Spectrum Scale version on ESS is not entirely under the control of theuser) along with other servers running V4.1.1 or later, we have additional configurationrequirements:1. The ESS/GSS has to run latest ESS/GSS level that supports V4.1.1.2. Install IBM Spectrum Scale manually on the protocol nodes using rpms from the

/usr/lpp/mmfs/4.x.0.0/gpfs_rpms directory3. Join the protocol nodes to the existing ESS cluster using the mmaddnode command4. The cluster should have CCR enabled. Issue the mmlscluster command to determine if

CCR is enabled. Issue the mmchcluster –ccr-enable command to enable CCR if needed.5. None of the V4.1.0.8 nodes can have quorum or management – any nodes that are

designated as quorum or manager nodes should be running V4.1.1 or later code (note thatthe ESS management function and GUI can be on a node that runs V4.1.0.8).

6. ESS nodes need to be in the same cluster as the protocol nodes that export ESSfilesystems.

7. We expect a node class called gss or gss_ppc64 (mmlsnodeclass --all)8. Input the protocol nodes into the Installation Toolkit (do not input the ESS nodes) .9. Configure the protocols using the Installation Toolkit.

10. Proceed with a protocol deployment using the Installation Toolkit11. No protocol CLI will be run from ESS nodes. The CLI only runs on nodes that are at

V4.1.1 or later.

Note:1. Asynchronous Disaster Recovery function of V4.1.1or later, requires a V4.1.1 filesystem

format and therefore cannot be used with ESS 3.0 (and by implication ESS+Protocols) ESSdoes not allow any function to reside on ESS nodes including protocol node functionality.

2. Expected sequence for configuring Protocols with ESSv Install and configure an ESS using standard procedures.v Perform a standard install of IBM Spectrum Scale on additional nodes that will be added to

the cluster with ESS nodes (include any Protocol nodes).

110

v Add these nodes (including protocol nodes) to the cluster created during ESS installationand configuration.

v Change node roles to ensure none of the ESS NSD servers are designated manager orquorum.

v Create any prerequisite file systems (including shared-root).v Configure protocol nodes for CES use and enable protocols.

Q8.6: Are there any known limitations and what are considerations should I be aware before usingthe integrated CES Protocol function?

A8.6: The current set of limitations and release notes are documented in the DeveloperWorks Wiki athttps://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Protocols

Q8.7: Where can I find additional information about protocols?A8.7: Additional information can be found at:

v The IBM Spectrum Scale Knowledge Center (http://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html) introduces the function, provides guidance on theset-up, administration and management of protocols including information on logs.

v The developer works wiki (https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Protocols) should contain the latest release notes, protocols quick overview, migration guide andproblem determination guide. We will continue to add tuning and configurationrecommendations to this wiki page.

Q8.8: How do I reduce the logging level for performance monitoring of Swift?A8.8: To reducing the logging level for performance monitoring of Swift, run the following commands

on each of the Object protocol node.v To clear them immediately issue the commands:

perl -p -i -e "s/PMS\_LOG\_LEVELS\[\’DEBUG\’\]/PMS\_LOG\_LEVELS\[\’ERROR\’\]/g" /usr/local/pmswift/pmswiftparams.pyrm -f /var/log/pmswift/pmswift*systemctl restart pmswiftd.service

v To automatically clear the logs after seven days issue the commands:perl -p -i -e "s/PMS\_LOG\_LEVELS\[\’DEBUG\’\]/PMS\_LOG\_LEVELS\[\’ERROR\’\]/g" /usr/local/pmswift/pmswiftparams.pysystemctl restart pmswiftd.serviceQ8.9: What are the limitations when multiple file systems are exported by NFS?A8.9: If a file system that was previously exported successfully by NFS on a CES node becomes

unavailable, the NFS daemon exits and the CES node becomes unhealthy. If the file systembecomes unavailable on all CES nodes, the whole CES cluster becomes unhealthy. In this case, theNFS daemons need to be restarted on all nodes.

Q8.10: How to effectively use VMWare with IBM Spectrum Scale?A8.10: IBM Spectrum Scale 4.2.2 does not support NFS v4.1. Therefore, it is recommended that NFSv3 be

used with VMWare ESX with IBM Spectrum Scale 4.2.2.Q8.11: What are the limitations when upgrading the SMB service?A8.11: All protocol nodes that are running the SMB service must have the same version of gpfs.smb

installed at any time. Upgrading the SMB service also requires an outage. For a manual upgrade,it is recommended that you upgrade all of the other parts of the system first before taking anoutage to upgrade the gpfs.smb package on the protocol nodes. For more information, see theprocedure at https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1ins_updatingsmb.htm. If you use the toolkit for theupgrade, a similar process is followed to ensure a proper SMB upgrade.

111







https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1ins_updatingsmb.htm

https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1ins_updatingsmb.htm

Hadoop Support questionsQ9.1: What platforms are supported for Hadoop on IBM Spectrum Scale?A9.1: Hadoop can run with IBM Spectrum Scale over x86_64, Power/BigEndian and

Power/LittleEndian. The verified operating system includes Redhat Enterprise Linux (RHEL),SuSe Enterprise Linux Server (SLES), and Ubuntu. Please refer to the question What Linuxdistributions are supported by IBM Spectrum Scale?

Note:1. For IBM Spectrum Scale Hadoop supported code on the Ubuntu or Debian distributions,

please contact IBM Service referencing APAR IV73885.2. IBM Spectrum Scale for AIX, Linux on Z, and Windows are not supported for Hadoop.

Q9.2: What versions of Hadoop are supported by IBM Spectrum Scale version?A9.2:

Table 60. Versions of Hadoop are supported by IBM Spectrum Scale version

GPFS 3.5(connector 1.1)

IBM SpectrumScale 4.1.0(connector 2.4)



HDFS Transparencyconnector

Hadoop1.x

Supported Supported Not supported Not supported Not supported

Hadoop2.4

Not supported 4.1.0.4 or later 4.1.1.0 4.1.1.1 or later 4.1/4.1.1/4.2, or later

Hadoop2.5

Not supported 4.1.0.7 or later(1) 4.1.1.0 4.1.1.1 or later 4.1/4.1.1/4.2, or later

Hadoop2.6

Not supported 4.1.0.7 or later(1) 4.1.1.0(1) 4.1.1.1 or later 4.1/4.1.1/4.2, or later

Hadoop2.7

Not supported 4.1.0.7 or later(1) 4.1.1.0(1) 4.1.1.1 or later 4.1/4.1.1/4.2, or later

Note:1. If applications don’t exploit file system APIs introduced in the corresponding Hadoop version,

the IBM Spectrum Scale connector will work with the specified Hadoop version. Referhttp://wiki.apache.org/hadoop/Roadmap to see what new features are introduced incorresponding Hadoop version.

2. If there are multiple connector versions that support the Hadoop version you have, the latestconnector is recommended. For example, with IBM Spectrum Scale 4.1.1, you should useconnector 2.7 even if the version you are running Hadoop2.4 version.

Q9.3: What Hadoop distributions are supported by IBM Spectrum Scale?A9.3: IBM BigInsights V4.0/4.1/4.2/V4.2.5, Open Apache distributions, and HortonWorks HDP 2.6.x

are certified. For more information about IBM BigInsights integration with IBM Spectrum Scale,see IBM Open Platform with Apache Hadoop.

Limited testing has been done with Cloudera, but it is not officially supported yet. If you needsupport for these distributions, please send mail to [email protected].

Q9.4: Can IBM Spectrum Scale support any Hadoop distribution vendor?A9.4: It depends on whether the Hadoop vendor does specific changes in its Hadoop distribution. If

the Hadoop Vendor does not do any specific changes, IBM Spectrum Scale supports thecomponents. For more information, see the question What open-source Hadoop components arecertified using IBM Spectrum Scale connector?

Q9.5: What IBM Spectrum Scale licenses do I need to use the IBM Spectrum Scale Hadoopconnector?

A9.5: The Spectrum Scale Hadoop connector can be used with all license types (Client, Server, FPO)and all Editions (Express, Standard, and Advanced). There is no additional license requirementfor Hadoop access.

112

http://wiki.apache.org/hadoop/Roadmap

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/IBM%20Open%20Platform%20with%20Apache%20Hadoop


Q9.6: Can Hadoop connector be used for IBM Spectrum Scale that has shared storage (including theElastic Storage Server (ESS) )?

A9.6: Hadoop applications can run over shared storage since IBM Spectrum Scale 4.1.1 with theHadoop connector 2.5. The customer needs to install the package IBM Spectrum Scale Hadoop-2.5connector (IBM Spectrum Scale 4.1.1) or Hadoop-connector 2.7 (IBM Spectrum Scale 4.1.1.1 orlater) available on developerWorks at https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/1st%20generation%20Hadoop%20Connector

If you want to utilize the HDFS Transparency connector, you need to install gpfs.hdfs protocol2.7.0-1 or later available on developerWorks at https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/2nd%20generation%20HDFS%20Transparency%20Protocol

Q9.7: Can Hadoop connector be used for IBM Spectrum Scale that has shared storage (includingESS) and internal storage (FPO pool) in the same file system?

A9.7: Yes. Both the Hadoop connector (gpfs.hadoop-connector) and the HDFS Transparency connector(gpfs.hdfs-protocol) are able to work with both types of storage and leverage locality when thefiles are stored on the FPO pool. You can also use IBM Spectrum Scale's ILM feature to movedata between the FPO and shared/ESS pool.

Q9.8: What open-source Hadoop components are certified using IBM Spectrum Scale connector?A9.8: IBM Spectrum Scale development certifies core Hadoop components such as HBase, Hive, Pig,

Oozie, Avro, Spark, Flume, Sqoop, Kafka, Knox, Slider, Solr, Ranger, and Accumulo for bothshared and shared nothing-storage using FPO. Other components are expected to work if they areusing Hadoop FileSystem API. If you have specific questions about other components, pleasesend mail to [email protected].

Q9.9: Why do you need special IBM Spectrum Scale connector instead of using Hadoop local filesystem ( file:///) if not using FPO/internal storage?

A9.9: As compared with file:///, the IBM Spectrum Scale connector does more than handle I/O fromHadoop applications. This includes reporting data block location for data awareness schedulingin scheduling engineer; large data chunk size support for scheduling; optimization for workloads,such as HBase and Hive; supporting both FPO and shared storage within the same cluster etc.Additionally, using a cluster file system as a local - per node file system will introduce challengesin Yarn scheduling as a map reduce split will map to the entire file versus creating multiple splitsper file which is handled in the IBM Spectrum Scale connector.

Q9.10: What are the differences between the Hadoop connector and the HDFS Transparencyconnector?

A9.10: The Hadoop connector (gpfs.hadoop-connector) implements the Hadoop File System API. It doesnot support Kerberos authentication or the WebHDFS REST API.

The HDFS Transparency connector implements the Hadoop HDFS RPC and supports widerHadoop workloads including full Kerberos, WebHDFS, and distcp.

If you are running IBM Spectrum Conductor, use the Hadoop connector. Otherwise, use HDFSTransparency.

Q9.11: What are the requirements/limitations for using the IBM Spectrum Scale HDFS Transparencyconnector?

A9.11: Considerations for using the IBM Spectrum Scale HDFS Transparency connector support include:v The Hadoop Distributed File System (HDFS) Transparency connector supports both FPO mode

and shared storage including ESS since the HDFS Transparency connector 2.7.0-1.v The HDFS Transparency connector is available from IBM developerWorks at

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/2nd%20generation%20HDFS%20Transparency%20Protocol

v The HDFS Transparency connector has no dependencies on the level of IBM Spectrum Scale.However, it's fully tested with IBM Spectrum Scale V4.1.1 or later and it is recommended atthese levels.

v Linux on Z is not supported.

113

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/1st%20generation%20Hadoop%20Connector



https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/2nd%20generation%20HDFS%20Transparency%20Protocol







Q9.12: What are the current advisories for the IBM Spectrum Scale Hadoop connector?A9.12: The current advisory is:

Abstract:IBM Spectrum Scale (GPFS) Hadoop connector is affected by a security vulnerability(CVE-2015-7430)

Summary:A security vulnerability has been identified in the IBM Spectrum Scale (GPFS) Hadoopconnector which could allow an unprivileged user the ability to read, write, modify, ordelete any data in a GPFS file system (CVE-2015-7430)

See the complete bulletin at either http://www-01.ibm.com/support/docview.wss?uid=isg3T1022979 or http://www.ibm.com/support/docview.wss?uid=ssg1S1005461

114

http://www-01.ibm.com/support/docview.wss?uid=isg3T1022979



Object Protocol Support questionsQ10.1: What considerations are there when using OpenStack Software with IBM Spectrum Scale?A10.1: IBM Spectrum Scale includes support for Object Protocol:

v For IBM Spectrum Scale V4.1.1 and V4.2, this is built with the IBM Cloud Manager forOpenStack distribution (ICM).

v For IBM Spectrum Scale V4.2.1, this is built with the Liberty release of OpenStack. TheOpenStack Liberty release notes can be found at https://wiki.openstack.org/wiki/ReleaseNotes/Liberty.

v For IBM Spectrum Scale V4.2.3 and V5.0.0, this is built with the Mitaka release of OpenStack.The OpenStack Mitaka Swift details can be found at https://releases.openstack.org/mitaka/#mitaka-swift.

v For IBM Spectrum Scale V5.0.1, this is built with the Pike release of OpenStack. The OpenStackPike Swift details can be found at https://docs.openstack.org/releasenotes/swift/pike.html.

IBM Spectrum Scale includes the Swift and Keystone components to provide a complete objectstorage capability that is tightly integrated with Spectrum Scale.

Other resources for using OpenStack software with IBM Spectrum Scale include:v IBM Spectrum Scale in an OpenStack Environment at http://www.redbooks.ibm.com/

abstracts/redp5331.html.v The OpenStack Cinder documentation for configuring the IBM Spectrum Scale volume driver

at http://docs.openstack.org/mitaka/config-reference/block-storage/drivers/ibm-gpfs-volume-driver.html.

v The GPFS support in OpenStack Cinder Havana release blog at http://www.ibm.com/developerworks/community/blogs/storage_redbooks/entry/gpfs_support_in_openstack_cinder_havana_release.

v A Deployment Guide for Elastic Storage Object at http://www.redbooks.ibm.com/redpieces/abstracts/redp5113.html.

v IBM Private, Public, and Hybrid Cloud Storage Solutions at http://www.redbooks.ibm.com/abstracts/redp4873.html.

v Customers who are interested in learning more about IBM Spectrum Scale and OpenStackSoftware should contact [email protected] for additional guidance.

Q10.2: What are the new features of IBM Spectrum Scale for Object Storage?A10.2:

The new features of Object Storage in V5.0.1 include:v OpenStack Pike Release, including Swift 2.15.1 and Keystone 12.0.1.v Swift3 release 1.12, including minimum segment size for multi-part upload in S3 protocol.

The new features of Object Storage in V5.0.0 include:v Support on Ubuntu 16.04.v Cumulative upgrades from older releases directly to V5.0.0.

The new features of Object Storage in V4.2.3 include:v OpenStack Mitaka Release, including Swift 2.7.3 and Keystone 9.3.1.

The new features of Object Storage in V4.2.1 include:v OpenStack Liberty Release, including Swift 2.5.0 and Keystone 8.0.0.v Storage policy support for encryption. Storage polices allow encryption to be enabled on a per

container basis.v Support for issuing mmobj commands on GPFS client nodes.v Improved problem determination documentation.v Improved documentation for configurations using an external Keystone identity service.v Simplified enablement of S3 API support.v Simplified enablement of Unified File and Object access support.v Monitoring of AD and LDAP services used with Keystone.v Support for object configuration using CES groups.

The new features of Object Storage in V4.2 include:

115

|||||

|||

https://wiki.openstack.org/wiki/ReleaseNotes/Liberty

https://wiki.openstack.org/wiki/ReleaseNotes/Liberty

https://releases.openstack.org/mitaka/#mitaka-swift

https://releases.openstack.org/mitaka/#mitaka-swift

https://docs.openstack.org/releasenotes/swift/pike.html

http://www.redbooks.ibm.com/abstracts/redp5331.html


http://docs.openstack.org/mitaka/config-reference/block-storage/drivers/ibm-gpfs-volume-driver.html

http://docs.openstack.org/mitaka/config-reference/block-storage/drivers/ibm-gpfs-volume-driver.html

http://www.ibm.com/developerworks/community/blogs/storage_redbooks/entry/gpfs_support_in_openstack_cinder_havana_release



http://www.redbooks.ibm.com/redpieces/abstracts/redp5113.html

http://www.redbooks.ibm.com/redpieces/abstracts/redp5113.html




v Storage policy support for compression, uniform file and object access and multi-region activeobject storage. Storage polices allow these features to be enabled on a per container basis.

v Compression allows object data to be compressed in the background after being committed tostorage.

v Unified file and object access allows object data to be ingested from the object interface andthen be accessed (read/update/delete) from the file interface, as well as data to be ingestedfrom the file interface and then accessed from the object interface.

v Multiregion active-active object storage allows you to configure containers that have datareplicated between multiple sites.

v S3 emulation support has added support for S3 ACLs on the object interface, and support forS3 multi part uploads.

Q10.3: How should I ensure that unauthorized users cannot access my object data when usingSpectrum Scale for object storage?

A10.3: To ensure against unauthorized access to your object data:v It is extremely important to set up firewall rules to limit access to the ports used by object

storage services.v Shell access by non-root users must be restricted on IBM Spectrum Scale protocol nodes where

the object services are running to prevent unauthorized access to object data.v See the IBM Spectrum Scale Advanced Administration Guide for your level of code at

http://www.ibm.com/support/knowledgecenter/en/STXKQY/ibmspectrumscale_welcome.html. Refer to the section on Object port configuration.

Q10.4: What is the level of compatibility between S3 API in IBM Spectrum Scale and Amazon S3?A10.4: IBM Spectrum Scale uses the OpenStack Swift3 code to implement the S3 API. The compatibility

level is documented at https://review.openstack.org/#/c/504281/11/doc/source/s3_compat.rst.Q10.5: How can I ensure secure data in flight between object client and Spectrum Scale object

storage?A10.5: IBM Spectrum Scale Object Storage should be configured with suitable load balancer (for

example, HAProxy) enabled in SSL mode to ensure secure data in flight between object client andthe object storage system. It is the customer's responsibility to provide and configure the loadbalancer.

Q10.6: What authentication schemes does IBM Spectrum Scale Object Storage support?A10.6: Spectrum scale for object storge must be configured with OpenStack Keystone (either installed on

Spectrum Scale protocol nodes or using an external Keystone instance). Keystone supportsintegration with Microsoft Active Directory, LDAP or can use a local Postgres repository for userdata. The Unified File and Object Access feature can be configured to use either local or unifiedidentity management mode. When using unified mode, the object identity back end must be thesame as that used for file. If using local mode, these can be different. See the supportAuthentication Matrix for Object for your version of IBM Spectrum Scale at http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.ins.doc/bl1ins_authconcept.htm

Q10.7: When using Unified File and Object Access feature, when should I use the local_mode vsunified_mode for identity management?

A10.7: The use of identity management mode depends upon your use case. Generally:v local_mode: Suitable when authentication schemes for file and object are different and file

access is required for applications and file ownership of data ingested via object interface is notof interest.

v unified_mode: Suitable for unified file and object access for end users. File ownership for dataingested via object interface is required and one can leverage features like common ILMpolicies for file and object data based on data ownership. In this mode, Object and File areexpected to use a common authentication back end coming from the same directory service (AD+RFC 2307 or LDAP)

Q10.8: Can I have SMB/NFS export over object data when using Unified File and Object Accessfeature?

A10.8: Yes, but it is important to note that the file authorization is independent from objectauthorization, and that changing a file ACL does not impact the object access of the data and viceversa. Also, the ownership of the data seen on file interface depends upon the ID management

116



https://review.openstack.org/#/c/504281/11/doc/source/s3_compat.rst

http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.ins.doc/bl1ins_authconcept.htm



mode (local_mode or unified_mode) being used. We recommend creating exports at thecontainer/bucket level in the Unified File and Object directory hierarchy.

Q10.9: Can I have read/write access from file interface as well as object interface on the same data,when using Unified File and Object Access feature?

A10.9: Yes, but it is important to consider the use case for this data. Object semantics do not supportlocking of objects. From the object interface, every PUT operation creates a new object atomically.From the file interface, files can be created and then updated many times. Even thoughsimultaneous read/write access is possible between file interface and object interface, doing thiswill lead to unpredictable results. We recommend a serial work flow where any object or any fileis only accessed from one interface at any point in time. One way to achieve this is to have eitherof the interfaces (file interface or object interface) to be read only and the other to be read/writeat any point in time.

Q10.10:What are the limitations for IBM Spectrum Scale Object Storage with Unified File Accessenabled?

A10.10:Limitations for IBM Spectrum Scale Object Storage with Unified File Access enabled include:v We see acceptable results in tests with up to 10 containers and 400,000 files/objects per

container when running the objectizer at its default interval of 30 minutes. If you require alarger number of containers, we recommend using an objectizer interval of 120 minutes orlonger. The interval can be changed as shown here, for example, setting it to 2 hours (inseconds):mmobj config change --ccrfile spectrum-scale-objectizer.conf --section DEFAULT \

--property objectization_interval --value 7200v In some situations, the objectizer process can complete before all of the new files are added to

the container listing but are still queued as asynchronous operations. In this case, the files arevisible and can be accessed from the object interface, but they do not show up in the containerlisting for some time. These objects eventually show up in a container listing when thisasynchronous queue is processed.

v It is possible when stopping GPFS that the ibmobjectizer service may not be stoppedautomatically. You can verify if this is the case and force it to stop using the systemctlcommand:mmdsh -N cesnodes systemctl status ibmobjectizer -n 0mmdsh -N cesnodes systemctl stop ibmobjectizer

v Additional limitations for Unified File and Object Access are documented in the KnowledgeCenter. See the Administration and Programming Reference guide, Managing Object Storagesection for your level of IBM Spectrum Scale at http://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html.

Q10.11:What are the considerations when using HAProxy load balancer with Spectrum Scale ObjectStorage?

A10.11:When using HAProxy as a load balancer to distribute Swift Object workloads across multipleprotocol nodes, users need to be aware:v The default HAProxy timer values may interfere with communication between the Swift client

and protocol node. With HAProxy default timer values that are typically lower than either thedefault object server or default client settings, HAProxy can timeout and terminate atransaction before either the client or the server timers expire. This may result in the serverlogging an "unexpected client disconnect" indicating to the object server administrator that theclient disconnected when actually HAProxy terminated the connection.

v The recommended debug method for HAProxy environments is to either remove HAProxyfrom the configuration (preferred) or disable the timers in /etc/haproxy.cfg (or set theHAProxy timers to very large values, ex. 60 minutes), investigate the problem between theclient and server, and then re-instate HAProxy as the load balancer.

Q10.12:Are there known issues with OpenStack Software?

117



A10.12:The following are the known issues with documented workarounds:

In Release 4.2.1 and earlier:v Authentication fails when openstackclient prompts for a password: see https://

bugs.launchpad.net/python-openstackclient/+bug/1473862. The workaround is to include thepassword in your openrc or environment settings, or pass on the command line.

v The IBM Spectrum Scale object protocols functionality on the Linux (standard and advanced)platform is affected by security vulnerabilities in the TLS and SSL protocols. For theworkaround, see http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009336

In Release 4.2.2 and later:v Credentials required for s3 access are created and stored locally in Keystone even when

Keystone is configured to use an external identity source. Using an external identity source forcredentials required for s3 access is not supported.

Q10.13:Are there any additional limitations or restrictions when using Spectrum Scale compression orencryption with Object protocol?

A10.13:No, there are no additional limitations or restrictions for these with Object Protocol. Anylimitations or restrictions that exist with Spectrum Scale compression or encryption also apply toObject protocol.

Q10.14:What are the limitations when configuring objects to use CES groups?

A10.14:Limitations when configuring objects to use CES groups include:v Configuring objects to use CES groups is supported in IBM Spectrum Scale V4.2.1 or later.v If the CES group feature is used as described in the Configuration of object for isolated node and

network groups section of the Advanced Administration Guide the following limitation needs to beconsidered:If a CES IP address that has the database or singleton attribute assigned is changed by either ofthe following, it needs to be ensured that the selected CES IP address is within the objectgroup:– Removed via the command mmces address remove– Any of the attributes are changed to a different CES IP via the command mmces address

change

Use the mmces address list command to list the current group and attribute assignment.If the assignment is incorrect so that an attribute is assigned to a CES IP that is not part of theobject group, use the mmces address change --ces-ip IP --attribute Attribute command tochange the attribute to CES IP address assignment. For example, for ces_ip8 within the objectgroupmmces address change --ces-ip ces_ip8 --attribute object_database_node

118

https://bugs.launchpad.net/python-openstackclient/+bug/1473862

https://bugs.launchpad.net/python-openstackclient/+bug/1473862


http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/ibmspectrumscale421_welcome.html

Installation Toolkit questionsQ11.1: Can I upgrade an IBM Spectrum Scale cluster with protocols directly from 4.1.1.x to 4.2.0.x or

4.2.1.x?A11.1: IBM Spectrum Scale V4.1.1.x clusters with protocols must first upgrade to V4.2.0.0 and then to

V4.2.0.x or 4.2.1.x. See question What are the limitations when I use the Installation Toolkit to upgradeSpectrum Scale from 4.1.1 to 4.2 with NFS or SMB Protocol? for help with the first step of this dualupgrade.

Q11.2: What are the limitations when I use the Installation Toolkit to install Spectrum Scale 4.2.0.0 orupgrade from Spectrum Scale 4.1.1.x to 4.2.0.0 or 4.2.1.0 with Object Protocol?

A11.2: Current limitations for the Installation Toolkit include:v The Installation Toolkit does not support installing the IBM Spectrum Scale GUI during an

upgrade of a cluster which has the Object protocol enabled. If you happen to do this, the GUIwill not start properly after the upgrade completes. Before installing the IBM Spectrum ScaleGUI, (and after completing the cluster upgrade) you will need to first remove or rename thepostgresql.service file from /etc/systemd/system/postgresql.service. After that, install the IBMSpectrum Scale GUI by running the spectrumscale install command. It is generally advisableto check for this file if the GUI reports database issues after install or upgrade. For example:mv /etc/systemd/system/postgresql.service /etc/systemd/system/postgresql.service.sav.4.1.1

v The Installation Toolkit does not support installing or upgrading a configuration that uses anexternal keystone. This limitation will be corrected in an upcoming refresh of InstallationToolkit. The install or upgrade can be accomplished by using the IBM Spectrum Scale CLI. Seethe IBM Spectrum Scale Administration and Programming Reference Guide section onConfiguring object authentication with external Keystone server at http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.2.0/ibmspectrumscale42_welcome.html

v In some cases, the Installation Toolkit may fail during upgrade of the performance monitoringcomponent. The reason for this is that the Installation Toolkit fails to stop pmswiftd.servicebefore upgrade, leaving the pmswiftserver running and the corresponding ports open. Thiscauses a failure when starting pmswiftd.service after upgrade. A workaround for this problemis to stop all running pmswiftserver processes on ALL protocol nodes and then manually startpmswiftd.service. To do this, use the following commands on each protocol node:– Check pmswiftserver process, this step is optional

$ ps aux|grep pmswiftserver– Stop pmswiftserver process

$ kill -9 $(pgrep pmswiftserver)– Start pmswiftd.service

$ systemctl start pmswiftd.service– Check the status of pmswiftd.service

$ systemctl status pmswiftd.service– You may be required to restart pmsensors.service if systemctl status shows FAILED or

corresponding pmswiftproxy is not active (this step may or may not be required). Youshould look for something like /usr/bin/python2.7 /usr/local/pmswift/pmswiftproxy in theoutput.$ systemctl restart pmsensors.service

– Check the status of pmsensors.service to make sure that corresponding pmswiftproxy isactive. You should look for something like /usr/bin/python2.7 /usr/local/pmswift/pmswiftproxy in the output.$ systemctl status pmsensors.service

Q11.3: Can GUI nodes be added during an upgrade to IBM Spectrum Scale V4.2.0.1?A11.3: When planning to add GUI nodes with the Installation Toolkit, add them via spectrumscale

install or spectrumscale deploy, either before performing an upgrade to 4.2.0.1 or afterwards.Attempting to add GUI nodes during the upgrade itself may result in a failure during theUpgrading Performance Monitoring step.

Q11.4: What are the limitations when I use the Installation Toolkit to upgrade Spectrum Scale from4.1.1 to 4.2 with NFS or SMB Protocol?

119



A11.4: Current limitations for using the Installation Toolkit to upgrade include:v An intermittent failure may occur during Deploy while 'enabling gpfs fileset quota sensors'.

This failure may result in SMB being down and CTDB being unhealthy. The following solutionwill recover the cluster if this issue is hit:1. This problem will create core files on the CES nodes. Typically the core files are in and

may have even filled up the entire root file system. The files will be named similar to/core.1734. Remove all core files:rm -rf /core.*

2. Stop SMB services on all CES nodes:/usr/lpp/mmfs/bin/mmces service stop SMB -a

3. Stop pmsensors on all CES nodes. ssh to each CES node and issue the followingcommand:systemctl stop pmsensors

4. Locate SMBStats.cfg and SMBGlobalStats.cfg on the installer node in/usr/lpp/mmfs/4.2.0.0/installer/cookbooks/zimon_on_gpfs/files/default Copy these files to/opt/zimon on all CES nodes. If the installer node is a CES node, also copy these files to/opt/zimon on the installer node.

5. Check GPFS cluster state on all nodes (double check the CES nodes since they may bedown)/usr/lpp/mmfs/bin/mmgetstate -a

6. Start GPFS on any down nodes/usr/lpp/mmfs/bin/mmstartup -a

7. Verify GPFS becomes active on all nodes:/usr/lpp/mmfs/bin/mmgetstate -a

8. Restart SMB on all CES nodes/usr/lpp/mmfs/bin/mmces service start SMB -a

9. Start the pmsensors service on all CES nodes. ssh to each CES node and execute:systemctl start pmsensors

10. Verify cluster state and service state on all nodes/usr/lpp/mmfs/bin/mmgetstate -a/usr/lpp/mmfs/bin/mmces service list -a

11. Resume the deploy/usr/lpp/mmfs/4.2.0.0/installer/spectrumscale deploy

12. Verify CES service states on all nodes after the deploy is successful./usr/lpp/mmfs/bin/mmces service list -a/usr/lpp/mmfs/bin/mmces state cluster

v Occasionally the cluster-wide knowledge of the state of the protocol nodes (viewable with themmces state cluster command ) may become out of sync with the local state (viewable withthe mmces state show command ) for some nodes. The most common case where this mayoccur is upgrading a system that has SMB enabled and uses Active Directory authentication. Inorder to bring the nodes back to the correct state the monitors on the affected nodes need to berestarted. This can by done by running the mmcesmoncontrol restart command on the nodesthat have inconsistent state information.

v There is also a known issue that it is possible to hit on clusters when performing an upgradewith SMB enabled. This issue occurs very rarely but is most frequent if using Active Directorybased authentication. What this issue looks like and the steps to fix it are:1. If the upgrade fails during SMB upgrade the system state will look something like this:

$ mmlscluster --cesNode Daemon node name IP address CES IP address list

-----------------------------------------------------------------------3 node01 172.31.132.1 node failed4 node02 172.31.132.2 node failed5 node03 172.31.132.3 Node suspended6 node04 172.31.132.4 node failed7 node05 172.31.132.5 Node suspended8 node06 172.31.132.6 node failed

120

2. Recovery from this failure requires completing the SMB upgrade manually. The first step isto stop SMB on all nodes$ mmdsh -N cesNodes /usr/lpp/mmfs/bin/mmces service stop SMB

3. Check which nodes have already been upgraded to the newer gpfs-smb level$ mmdsh -N cesNodes "rpm -qa | grep grep gpfs.smb"node01: gpfs.smb-4.3.0_gpfs_8-1.el7.x86_64node02: gpfs.smb-4.2.2_gpfs_31-1.el7.x86_64node03: gpfs.smb-4.3.0_gpfs_8-1.el7.x86_64node04: gpfs.smb-4.2.2_gpfs_31-1.el7.x86_64node05: gpfs.smb-4.3.0_gpfs_8-1.el7.x86_64node06: gpfs.smb-4.2.2_gpfs_31-1.el7.x86_64

4. Manually upgrade the rpm on each of the nodes that are down-level.a. Copy the newer rpm to each node (replace /usr/lpp/mmfs/4.2.0.0/ with the directory

you extracted the GPFS self-extracting package to if you chose a different location)$ scp /usr/lpp/mmfs/4.2.0.0/smb_rpms/gpfs.smb-4.3.0_gpfs_8-1.el7.x86_64.rpm node02:/tmp/

b. Use rpm to upgrade the package on each node$ rpm -U /tmp/gpfs.smb-4.3.0_gpfs_8-1.el7.x86_64.rpm

c. Check that the level is as expected using the rpm query above. If all nodes are now atthe same gpfs.smb version then you can restart SMB on all nodes$ mmdsh -N cesNodes /usr/lpp/mmfs/bin/mmces service start SMB

d. Determine which are the suspended nodes$ mmlscluster --cesNode Daemon node name IP address CES IP address list

-----------------------------------------------------------------------3 node01 172.31.132.1 node failed4 node02 172.31.132.2 node failed5 node03 172.31.132.3 Node suspended6 node04 172.31.132.4 node failed7 node05 172.31.132.5 Node suspended8 node06 172.31.132.6 node failed

e. Resume the suspended nodes.$ mmces node resume -N node03,node05

Q11.5: How can I determine if the Installation Toolkit successfully upgrades from IBM SpectrumScale V4.1.1 to V4.2?

A11.5: To check if the Installation Toolkit successfully upgraded from IBM Spectrum Scale V4.1.1 to V4.2,issue the following command:./spectrumscale upgrade -po

If the only error condition displayed is the following, the upgrade completed successfully"TypeError: sequence item 0: expected string, Node found"

Q11.6: Can I have EPEL repos enabled when using the spectrumscale installation toolkit for install orupgrade?

A11.6:

EPEL repos must be disabled on all nodes that have been added to the Spectrum ScaleInstallation Toolkit when attempting to install, deploy or upgrade.

See the Flash at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009275.Q11.7: What are the functions that are not supported by the installation toolkit?A11.7: For detailed information on functions that are not supported by the installation toolkit, see:

Limitations of the spectrumscale installation toolkit.Q11.8: What are the potential limitations when using the Installation Toolkit with SLES12 SP1 or SP2

nodes?A11.8: There is a potential issue with IBM Spectrum Scale packaged with SMB and SLES version of

samba-winbind that can lead to an installation failure. If this occurs, remove the samba-winbindrpms and continue the installation and deployment. For more information, see Package conflicton SLES 12 SP1 and SP2 nodes while doing installation, deployment, or upgrade usinginstallation toolkit.

121


http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.2/com.ibm.spectrum.scale.v4r22.doc/bl1ins_limitationsofthetoolkit.htm

https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1pdg_packageconflictsles12sp.htm



Transparent cloud tiering questionsQ12.1: What platforms is transparent cloud tiering supported on?A12.1: The transparent cloud services node must be installed on CES protocol nodes or NSD nodes

running RHEL 7 on x86 or Power8 Little Endian. In order to transparently recall a file that hasbeen migrated via transparent cloud tiering, a node must be running: RHEL, SLES, Debian, orUbuntu on x86 or RHEL on Power8 Little Endian. The IBM Spectrum Scale cluster might includenodes with other platforms or operating systems, but these nodes will not be able to migrate orrecall files directly.

Note: To enable transparent cloud tiering nodes, you must first enable the transparent cloudtiering feature. These nodes must have GPFS server licenses enabled. This feature provides a newlevel of storage tiering capability to the IBM Spectrum Scale customer. Please contact your IBMClient Technical Specialist (or send an email to [email protected]) to review your use case of thetransparent cloud tiering feature and to obtain the instructions to enable the feature in yourenvironment.

Q12.2: How many transparent cloud tiering bridge nodes are supported?A12.2: Up to four cloud services nodes can be configured per file system. Up to four file systems can be

managed in a cluster.Q12.3: What are the networking requirements for transparent cloud tiering?A12.3: Transparent cloud tiering utilizes the cloud services node in order to communicate with an

external storage cloud. Typically, this communication will utilize standard HTTP or HTTPS TCPports (port 80 or 443). However, some storage cloud providers may use other TCP ports. Pleasecheck with your cloud provider for details. The bridge node must be able to communicate to thestorage cloud. Prior to migrating files to a cloud provider, ensure that there is sufficientbandwidth to both send and receive files as needed. The bandwidth required will vary based onworkload and user requirements.

Q12.4: What Cloud Object Storage Providers are supported?A12.4: Transparent cloud tiering is compatible with Openstack Swift and the Amazon S3 object APIs.

The following public cloud providers have been explicitly tested with transparent cloud tiering:v IBM Softlayerv Amazon S3

The following private object stores have been explicitly tested with transparent cloud tiering:v IBM Spectrum Scale Object servicesv IBM Cloud Object Storage

Q12.5: Can transparent cloud tiering be used with other IBM Spectrum Scale functions?A12.5: Considerations for using transparent cloud tiering with IBM Spectrum Scale include:

v IBM Spectrum Archive™ - Linear Tape File Systems (LTFS) and IBM Spectrum Protect for SpaceManagement (HSM)Running IBM Spectrum Archive and transparent cloud tiering on the same file system is notsupported. However, both HSM and transparent cloud tiering can coexist on the same systems(as long as they are configured with different file systems)

v AFM– Running transparent cloud tiering service on the AFM gateway nodes is not supported.– Data from the AFM and AFM DR filesets must not be accessed by transparent cloud tiering.

v FPOTransparent cloud tiering is supported on an FPO cluster.

v IBM Spectrum Scale ObjectTransparent cloud tiering can be configured on IBM Spectrum Scale Object fileset(s) only.Support for native object storage is not provided.

v Snapshots– Transparent cloud tiering cannot be used to migrate/recall snapshots.– Space contained in snapshots will not be freed if files are migrated to cloud object storage.

v Sparse Files

122


Transparent cloud tiering can be used to migrate and recall sparse files, but sparseness will notbe retained. Full blocks will be allocated.

v Native EncryptionTransparent cloud tiering can be used with Spectrum Scale native encryption. All data migratedto Cloud Object Storage will be migrated with the encryption key configured in transparentcloud tiering. When the file is read from the filesystem, the data will be unencrypted andre-encrypted using transparent cloud tiering encryption algorithms prior to being sent to cloudstorage.

v CompressionTransparent cloud tiering can be used along with Scale file system level compression capability.When the file is read from file system, the file will be uncompressed, transparent cloud tieringwill transfer the uncompresssed file to cloud storage. Recalled files will be uncompressed onthe file system.

v CES nodes (Protocol Services)Transparent cloud tiering can coexist along with NFS, SMB or Object Services on the CESnodes.

v IBM Spectrum Protect.

Note: Beginning with Version 7.1.3, IBM Tivoli Storage Manager is now IBM Spectrum Protect.Files should be backed up prior to transferring them to cloud storage via transparent cloudtiering. Failure to do so will cause files to be recalled in order to perform the back up.

Q12.6: Is transparent cloud tiering supported with ESS?A12.6: Transparent cloud tiering cannot be deployed directly on ESS nodes, however it can be deployed

on other nodes in the Spectrum Scale cluster that meet the hardware and software requirements.Q12.7: Are there restrictions on viewing or listing files?A12.7: Standard Unix tools and windows utilities such as ls or dir can be used to view files that have

been migrated to the cloud. Some file viewers, such as Windows Explorer and GNOME Fileviewer utilize preview functions which will open files in order to generate a preview. Thesefunctions may result in files being unintentionally recalled from the cloud.

123

Licensing and Pricing QuestionsQ13.1: How are IBM Spectrum Scale V4 and V5 licensed?A13.1: IBM Spectrum Scale V4 and V5 are available in the following editions, which are licensed

individually:v IBM Spectrum Scale Standard Edition includes the base function plus ILM and AFM. With

V4.1.1 or later, support is provided for integrated protocol access methods (NFS server, SambaServer, and Object with OpenStack Swift).

v IBM Spectrum Scale Advanced Edition includes all of the features in the Standard Edition aswell as native encryption for secure storage and secure deletion. With V4.1.1, support isprovided for Asynchronous Disaster Recovery.

v For V4.2.2 or later, IBM Spectrum Scale Data Management Edition is available. This editionprovides identical functionality to IBM Spectrum Scale Advanced Edition under capacity-basedlicensing. For more information, see Capacity-based licensing.

Note:1. All nodes in a cluster must be running the same edition.2. In a multi-cluster environment:v An IBM Spectrum Scale V4 Advanced or Data Management Edition cluster cannot be a

client cluster to other editions. Every node accessing encrypted data must be running theAdvanced or Data Management Edition of IBM Spectrum Scale.

3. IBM Spectrum Scale Advanced Edition and Data Management Edition are not supported onthe Windows operating system. See the question What are the limitations of IBM Spectrum Scalesupport for Windows?

4. IBM Spectrum Scale for Linux on Z is supported on all editions. See the question What are thecurrent requirements/limitations for IBM Spectrum Scale for Linux on Z?

5. Licenses can be traded up to a higher priced edition, but not downgraded.6. Asynchronous Disaster Recovery requires the Advanced or Data Management Edition. Only

the primary cluster requires the Advanced or Data Management Edition; a secondary clustermay be Advanced, Data Management, or Standard Edition.

Each Standard edition of IBM Spectrum Scale V4 or later, with the exception of Linux on Z, hasthree types of licenses: a Server license, a Client license, and a File Placement Optimizer license.Licenses are priced per socket with the exception of Linux on Z. According to the functionsperformed on the virtual servers where IBM Spectrum Scale Server, IBM Spectrum Scale Client,or IBM Spectrum Scale File Placement Optimizer is installed, the licensee must obtain entitlementfor each socket on the virtual servers made available to IBM Spectrum Scale.

Note:1. IBM Spectrum Scale for Linux Standard Edition has Server and Client licenses only.2. IBM Spectrum Scale for Linux Standard Edition is priced per 10 PVU.

The IBM Spectrum Scale Server license permits the licensed virtual server to perform IBMSpectrum Scale management functions such as cluster configuration manager, quorum node,manager node, and Network Shared Disk (NSD) server. In addition, the IBM Spectrum ScaleServer license permits the licensed virtual server to share IBM Spectrum Scale data directlythrough any application, service protocol, or method such as Network File System (NFS),Common Internet File System (CIFS), File Transfer Protocol (FTP), Hypertext Transfer Protocol(HTTP), or OpenStack Swift.

Note: The specific use of local read-only cache function does not require a server license.

For requirements in a virtualization environment, see the following questions:v How do I determine the number of licenses required when running IBM Spectrum Scale VMs in a

virtualization environment?v How do I determine whether a server license or a client license is required when running IBM Spectrum

Scale VMs in a virtualized environment?

124

http://www.ibm.com/support/knowledgecenter/STXKQY_4.2.2/com.ibm.spectrum.scale.v4r22.doc/bl1ins_capacitylicense.htm

The IBM Spectrum Scale Client license permits exchange of data between virtual servers thatlocally mount the same IBM Spectrum Scale file system or OpenStack services such as Cinder,Glance, Nova, and Manila. No other export of the data is permitted. The IBM Spectrum ScaleClient may not be used for virtual servers to share IBM Spectrum Scale data directly through anyapplication, service protocol or method, such as NFS, CIFS, FTP, HTTP, or OpenStack Swift.. Forthese functions, a IBM Spectrum Scale Server license would be required. One exception is sharingof IBM Spectrum Scale data to VMs on the same server, which is described in more detail in thequestion: How do I determine whether a server license or a client license is required when running IBMSpectrum Scale in VMs in a virtualized environment?

The IBM Spectrum Scale FPO license permits the licensed virtual servers to perform NSD serverfunctions for sharing IBM Spectrum Scale data with other virtual servers that have a IBMSpectrum Scale FPO or IBM Spectrum Scale Server license. This license cannot be used to sharedata with virtual servers that have a IBM Spectrum Scale Client license or non-IBM SpectrumScale virtual servers.

Computers which are directly attached to storage and meet all of the following criteria onlyrequire an IBM Spectrum Scale client license:v Do not perform IBM Spectrum Scale NSD server functions.v Are not defined to the IBM Spectrum Scale cluster as NSD servers.v Do not perform IBM Spectrum Scale cluster management functions.v Do not share IBM Spectrum Scale data directly through any application, service protocol or

method (other than specified in the question How do I determine whether a server license or a clientlicense is required when running IBM Spectrum Scale VMs in a virtualized environment?).

v Do not access an IBM Spectrum Scale FPO filesystem.

See the Concepts, Planning, and Installation Guide for your level of IBM Spectrum Scale for moreinformation.

Q13.2: How can I tell which Edition of IBM Spectrum Scale I am running?A13.2: There are a couple of ways to determine which Edition of IBM Spectrum Scale V4 you are

running:v You can run the mmlslicense command. It will tell you which Edition is running on the local

node.v You can view the GPFS License files at /usr/lpp/mmfs/4.x. Each edition of IBM Spectrum Scale

is separately licensed.v You can issue the appropriate command rpm -aq|grep gpfs or lslpp -l |grep -i gpfs

– If gpfs.base is listed, you have the Express Edition.– If gpfs.ext is additionally listed, you have the Standard Edition.– If gpfs.crypto and gpfs.adv are additionally listed, you have the Advanced Edition.

Q13.3: How is GPFS V3.5 licensed?A13.3: Editions were introduced in V4.1 and do not apply to V3.5. GPFS V3.5 has three types of licenses,

a Server license, a Client license, and a File Placement Optimizer license. Licenses are priced perprocessor core. For each node in a GPFS cluster, the customer determines the appropriate numberof GPFS Server licenses, GPFS Client licenses or GPFS FPO licenses that correspond to the wayGPFS is used on that node. A node is defined as an individual operating system image that mayappear on a single computer within a cluster, on a system within a cluster, or on a partition.

The GPFS Server license permits the licensed node to perform GPFS management functions suchas cluster configuration manager, quorum node, manager node, and Network Shared Disk (NSD)server. In addition, the GPFS Server license permits the licensed node to share GPFS data directlythrough any application, service protocol or method such as Network File System (NFS),Common Internet File System (CIFS), File Transfer Protocol (FTP), or Hypertext Transfer Protocol(HTTP).

Note:1. For requirements for a Server license in a virtualization environment, please see the question

How do I determine whether a server license or a client license is required when running GPFS inVMs in a virtualized environment?

125


2. Regarding OpenStack services, only sharing data through Swift would require a GPFS Serverlicense. Other OpenStack services such as Cinder, Glance, Nova, and Manila do not shareGPFS data and therefore do not require a GPFS Server license.

The GPFS Client license permits exchange of data between nodes that locally mount the sameGPFS file system. No other export of the data is permitted. The GPFS Client may not be used fornodes to share GPFS data directly through any application, service, protocol or method, such asNFS, CIFS, FTP, or HTTP. For these functions, a GPFS Server license would be required. Oneexception is sharing of GPFS data to VMs on the same server, which is described in more detailin the question: How do I determine whether a server license or a client license is required when runningGPFS in VMs in a virtualized environment?

Computers which are directly attached to storage and meet all of the following criteria onlyrequire a GPFS client license:v Do not perform GPFS NSD server functionsv Are not defined to the GPFS cluster as NSD serversv Do not perform GPFS cluster management functionsv Do not share GPFS data directly through any application, service protocol or method (other

than specified in the question How do I determine whether a server license or a client license isrequired when running IBM Spectrum Scale VMs in a virtualized environment?)

v Do not access a GPFS-FPO filesystem

For GPFS on Linux, beginning with GPFS V3.5.0.7, and GPFS on AIX beginning with V3.5.0.11,we introduce the GPFS FPO license. The GPFS FPO license permits the licensed node to performNSD server functions for sharing GPFS data with other nodes that have a GPFS FPO or GPFSServer license. This license cannot be used to share data with nodes that have a GPFS Clientlicense or non-GPFS nodes.

Note: Also, see the question What are the current limitations for using the File Placement Optimizer(FPO) function?

Q13.4: How are IBM Spectrum Scale V4 and V5 priced?A13.4:

For V4.2.2 or later, IBM Spectrum Scale Data Management Edition is priced by Capacity (perbinary TB). IBM Spectrum Scale Standard Edition V4.1 or later licenses for x86 and Power arepriced per socket.

IBM Spectrum Scale V4.1 or later for Linux on Z utilizes the per 10 PVU pricing metric. Firstdetermine the cores/IFLs available to IBM Spectrum Scale, then utilize the PVU calculator athttps://www-112.ibm.com/software/howtobuy/passportadvantage/valueunitcalculator/vucalc.wss.

For pricing details, ask your IBM representative or Business Partner. General pricing informationis provided in the Announcement letters. To access the Announcement letters for IBM SpectrumScale:1. Go to http://www.ibm.com/common/ssi/index.wss.2. On Information Type, choose Announcement letters.3. Enter the appropriate product number in the search field:v For orders placed through the AAS fulfillment system, enter 5641-GPF.v For orders placed through Passport Advantage, enter 5725-Q01.v For orders for IBM Spectrum Scale for Linux on Z, enter 5725-S28.

Q13.5: How is GPFS V3.5 priced?A13.5: GPFS V3.5 is orderable through multiple methods at IBM. One of these uses Processor Value

Units (PVUs) and the other uses small, medium and large. Your IBM sales representative can helpyou determine which method is appropriate for your situation.

Pricing examples include:

GPFS for POWER (5765-G66) and GPFS on x86 Architecture (5765-XA3)

126




Licenses continue to be priced per processor core.

Common small commercial Power Systems™ cluster where virtualization is used:v You have a cluster that consists of four Power systems. Each system has eight processor cores

per physical system and is partitioned into two LPARs with four processor cores per LPAR fora total of 8 LPARS running GPFS. All of the nodes access the disk through a SAN.

v Three of the LPARs are configured as quorum nodes. Since these nodes are running GPFSmanagement tasks (i.e. quorum) they require a GPFS Server license. Three nodes with fourCPUs each means you will need 12 Server licenses.

v Five of the LPARs are configured as non-quorum nodes. These nodes do not run GPFSmanagement tasks. So five nodes with four CPUs each means you will need 20 Client licenses.

See http://www-03.ibm.com/systems/power/hardware/reports/quick-reference.html

GPFS on x86 Architecture (5724-N94, 5641-A07, 5641-B07, and 5641-C07)

Licenses continue to be priced per 10 Processor Value Units (PVU). For example, 1 AMD Opteroncore requires 50 PVUs.

PROCESSOR VALUE UNIT

PVU is the unit of measure by which this program is licensed. PVU entitlements are based onprocessor families (vendors and brands). A Proof of Entitlement (PoE) must be obtained for theappropriate number of PVUs based on the level or tier of all processor cores activated andavailable for use by the Program on the server. Some programs allow licensing to less than thefull capacity of the servers activated processor cores, sometimes referred to as sub-capacitylicensing. For programs which offer sub-capacity licensing, if a server is partitioned utilizingeligible partitioning technologies, then a PoE must be obtained for the appropriate number ofPVUs based on all activated processor cores available for use in each partition where the programruns or is managed by the program. Refer to the International Passport Advantage AgreementAttachment for Sub-Capacity Terms or the program's License Information to determine applicablesub-capacity terms. The PVU entitlements are specific to the program and may not be exchanged,interchanged, or aggregated with PVU entitlements of another program.

For general overview of PVUs for processor families (vendors and brands), go tohttp://www.ibm.com/software/lotus/passportadvantage/pvu_licensing_for_customers.html

To calculate the exact PVU entitlements required for the program, go to https://www-112.ibm.com/software/howtobuy/passportadvantage/valueunitcalculator/vucalc.wss

Common x86 HPC setup with no virtualization:v You have four x3655 systems with eight cores each. In addition you have 32 x3455 systems

each with four processor cores. Each physical machine is a GPFS node (no virtualization).v The four x3655 nodes are configured as NSD servers and quorum nodes. Therefore they are

serving data and providing GPFS management services so they require a GPFS Server license.Four nodes each with eight AMD Opteron cores means you have a total of 32 cores. Each AMDOpteron core is worth 50 PVUs and each Server license is worth 10 PVUs so you will need 160GPFS Server licenses. (32 AMD Opteron cores*50 PVUs)/10 PVUs per Server License = 160GPFS Server licenses.

v The 32 x3455 nodes are all configured as NSD clients. So you have 32 nodes each with fourcores for a total of 128 cores. Each AMD Opteron core is worth 50 PVUs and each Clientlicense is worth 10 PVUs so you will need 640 GPFS Client licenses. (128 AMD Opteroncores*50 PVUs)/10 PVUs per Client license = 640 client licenses.

For further information contact:v [email protected] In the United States, please call 1-888-SHOP-IBMv In all other locations, please contact your IBM Marketing Representative. For a directory of

worldwide contact, see www.ibm.com/planetwide/index.htmlQ13.6: Can I migrate my GPFS V3.5 licenses to IBM Spectrum Scale V4?A13.6: Yes, if you have maintained your software maintenance (SWMA) or subscription and support

127

http://www-03.ibm.com/systems/power/hardware/reports/quick-reference.html

http://www.ibm.com/software/lotus/passportadvantage/pvu_licensing_for_customers.html





(S&S), you are entitled to move from V3 to V4. However, due to the change in charge metric fromprocessor cores or processor value units to sockets, it will require IBM interaction. Please contactyour Client or Sales representative to have the process initiated to migrate your licenses toversion 4 and Passport Advantage.

PVU and Core based entitlements are converted based on existing usage and hardware. The IBMrepresentative will need to know how many cores per socket are in each of the servers whereyour GPFS V3 licenses are today. They will also need to know the breakdown by license type(GPFS client, server and/or FPO). For each license type they will divide the number of processorentitlements by the number of cores per socket, rounding up if a fraction. This will be your newentitled quantity.v For each physical server where GPFS V3 is deployed, determine the number of cores that are

licensed, divide by the number of cores/physical sockets, rounding up if a fraction.v Example: If you have GPFS V3 Server licenses for 10 2-socket systems with 4 cores/socket, you

will be entitled to 20 IBM Spectrum Scale V4 Server licenses.

Passport Advantage (PPA) Customers:v If your entitlements are not already migrated to sockets, the IBM Client Representative will

need to process a License Exchange.

Advanced Administrative System (AAS) Customers:v Clients with CHIS contracts for software maintenance may remain with CHIS and move to V4.

You will need to notify your CHIS contact that you have upgraded to V4. You will need tomigrate to Passport Advantage before you can add any new licenses. When your entitlementshave been moved to Passport Advantage, you will need to cancel GPFS from your CHIScontract.

v In order to migrate from V3 to V4, the IBM Client Representative will need to process an AASto PA migration.

Note: If the rpm - U command is used, you must additionally install the gpfs.ext package.

Full install images are available for entitled customers on Fix Central beginning with IBMSpectrum Scale V4.2.v V4.2.0 is available at https://www-945.ibm.com/support/fixcentral/swg/

selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.0&platform=All&function=all

v V4.2.1 is available at https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.1&platform=All&function=all

Note: AIX images are not licensed except for V4.2.1.0 Standard Edition https://www-945.ibm.com/support/fixcentral/swg/downloadFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.1&platform=All&function=fixId&fixids=Spectrum_Scale_Standard-4.2.1.0-ppc64-AIX-install&includeRequisites=1&includeSupersedes=0&downloadMethod=http&login=true

Customers moving to either IBM Spectrum Scale V4.1.1 and V4.1.0 must already have a license ordownload DVD / e-image from Passport Advantage:v 4.1.1 is available at https://www-945.ibm.com/support/fixcentral/swg/


v 4.1.0 is available at https://www-945.ibm.com/support/fixcentral/swg/selectFixes?product=ibm/power/IBM+General+Parallel+File+System+Standard+Edition&release=4.1.0&platform=AIX&function=all

Q13.7: Is there an additional cost to upgrade or migrate from V3 to V4?A13.7: There is no additional cost to migrate licenses, standard renewal charges will be applied.Q13.8: I am in the middle of my SWMA or S&S contract, my anniversary date for renewal is 4

months away, will I have to pay again?

128

https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.0&platform=All&function=all






https://www-945.ibm.com/support/fixcentral/swg/downloadFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.1&platform=All&function=fixId&fixids=Spectrum_Scale_Standard-4.2.1.0-ppc64-AIX-install&includeRequisites=1&includeSupersedes=0&downloadMethod=http&login=true








https://www-945.ibm.com/support/fixcentral/swg/selectFixes?product=ibm/power/IBM+General+Parallel+File+System+Standard+Edition&release=4.1.0&platform=AIX&function=all



A13.8: No, the migration will take your outstanding S&S into account.v The License Exchange will migrate your entitled licenses to v4 and will maintain your current

anniversary dates. When your renewal comes due, you will need to renew V4 based on thenew entitlements.

v If your entitlements for V3 are in Passport, and your anniversary date is 2 months out,Passport will send a renewal request based on V3. This automatic renewal request should betaken into consideration when processing the License Exchange.

v There is no cost for processing the License Exchange.Q13.9: My licenses were purchased with a Power server and my entitlements and SWMA are

managed by CHIS contracts. I have purchased SWMA for 3 years and have considerable timeremaining. How will my SWMA be managed going forward? Will my existing SWMA change?

A13.9: Any remaining SWMA will be accounted for in your Passport Advantage anniversary date. YourContract Handling International System (CHIS) contract will end, but it is not considered atermination; your SWM will continue in PA.

Q13.10:For my Power servers, will my machine type model (MTM) and serial numbers (SN) bemigrated with my entitlements?

A13.10:The IBM Spectrum Scale V4 entitlements are managed in Passport Advantage (PA), and PA doesnot track specific machines. As such, when you call in for support, you will provide yourcustomer number, not your MTM SN.

Q13.11:I am not familiar with Passport Advantage, is there anything special I will need to do?

A13.11:If you do not currently have an account in PA, you will be asked to sign a Passport AdvantageAgreement prior to your licenses being migrated. See https://www-01.ibm.com/software/passportadvantage/pa_agreements.html.

Q13.12:I am interested in the new IBM Spectrum Scale Advanced Edition, can I upgrade to AdvancedEdition from V3?

A13.12:You will first need to have your licenses exchanged to V4 Standard Edition, then you can processa trade-up to Advanced Edition. The general migration is to IBM Spectrum Scale StandardEdition.

Q13.13:My S&S or SWMA has expired and now I want to move to V4, can I migrate using theAfter-License option?

A13.13:You must first reinstate your V3 licenses using the after-license process for your expiredentitlements. Then you can migrate those new license entitlements to V4 via the license exchangeprocess.

Q13.14:Can I trade-up my IBM Spectrum Scale V4 licenses between Editions?

A13.14:IBM Spectrum Scale V4 Licenses are traded-up through Passport Advantage regardless of theordering system where the original order was placed. It is possible to trade-up from IBMSpectrum Scale Standard Edition to IBM Spectrum Scale Data Management Edition; however, dueto the change in pricing metric, it must be calculated on a case by case basis. For assistance,contact your IBM Client representative. Trade-up is not available for other editions of IBMSpectrum Scale.

Q13.15:Can I transfer my IBM Spectrum Scale licenses between machines?

A13.15:IBM Spectrum Scale Licenses may be transferred between machines based on the version:

129

https://www-01.ibm.com/software/passportadvantage/pa_agreements.html

https://www-01.ibm.com/software/passportadvantage/pa_agreements.html

v IBM Spectrum Scale V4 licenses for x86 and Power are transferable between OSs and x86 orPower machines as needed as IBM Spectrum Scale V4 licenses are ordered based on Editionand function (Server, Client, FPO) only.

v IBM Spectrum Scale V4 for Linux on Z licenses are transferable between supported Linux OSsand associated supported IBM Z hardware. See What are the minimum hardware requirements for aIBM Spectrum Scale cluster?

v GPFS V3.5 licenses may not be transferred between Power and x86 machines.v GPFS V3.5 licenses may be transferred between Power and Power machines.v For Disaster Recovery sites:

– For Non-Passport Advantage programs, the International Agreement for Acquisition ofSoftware Maintenance located at http://www-03.ibm.com/software/sla/sladb.nsf/sla/iaasm/

– For Distributed Software, the Distributed Software Licensing Guide at ftp://ftp.software.ibm.com/software/passportadvantage/License_Management/IBM_Distributed_Software_Licensing_Reference_Guide.pdf. Reference the section on Backupor Standby Systems.

– Where IBM Spectrum Scale is running and active (Hot), licenses are required. Additionallicenses may need to be purchased if the destination site is of larger capacity.

– For LPAR migration, additional licenses are required if you are migrating to a machine withlarger capacity.

Q13.16:Do I require additional IBM Spectrum Scale licenses for my Disaster Recovery (DR)environment?

A13.16:

Regardless of whether your IBM Spectrum Scale is licensed based on PVUs, sockets, or capacity,the determination on whether additional licenses are needed depends on whether "work" is beingdone and whether the DR location is "cold," "warm," or "hot". Since that answer is specific toyour situation, it is important to see how IBM's policy lines up with your relevant dynamics.

IBM's policy on backup or standby systems to apply is described in the following paragraphs:

The intent of backup licensing is to enable the customer to continue to be operational if anemergency occurs. This is frequently referred to as backup use, disaster recovery, or abbreviatedas BRS when a backup and recovery service is involved.

For programs or residents running on backup machines, IBM defines three types of backupsituations: cold, warm, and hot. In cold and warm situations, a separate entitlement for the copyon the backup machine is typically not required and no additional charge applies. In a hotbackup situation, the customer must acquire license entitlements sufficient for that machine. Allprograms running in backup mode must be solely under the customer's control, even if runningat another enterprise's location.

As a practice, the following table contains definitions and allowable actions concerning the copyof the program used for backup purposes:

Table 61. Backup licensing for Disaster Recovery environments

Backup Category Program Status Licensing

Cold Not started No additional entitlements

Warm Started but idle No additional entitlements

Hot Started and doing work Additional entitlements required

"Doing work" includes, but is not limited to activities such as:v Programmingv Developmentv Program maintenancev Testing

130

http://www-03.ibm.com/software/sla/sladb.nsf/sla/iaasm/

http://www-03.ibm.com/software/sla/sladb.nsf/sla/iaasm/

ftp://ftp.software.ibm.com/software/passportadvantage/License_Management/IBM_Distributed_Software_Licensing_Reference_Guide.pdf



v Mirroring of transactionsv Updating of filesv Synchronization of programs, data, or other resources (for example, active linking with another

machine, program, database, or other resource)v Any activity or configurations that would allow an active hot switch or other synchronized

switch over between programs, databases, or other resources to occur.

If a program or system configuration that is designed to support a high availability environmentby using various techniques (for example, duplexing, mirroring of files or transactions,maintaining a heartbeat, or active linking with another machine, program, database, or otherresource), the program is considered to be doing work in both the warm and hot situation andlicense entitlements must be acquired.

131

Service questionsQ14.1:

What support services are available for IBM Spectrum Scale?A14.1: Support services include:

v Please see the Software Support Handbook at http://www14.software.ibm.com/webapp/set2/sas/f/handbook/home.html

v Forums– Technical discussion forum at www.ibm.com/developerworks/forums/

dw_forum.jsp?forum=479&cat=13 will help answer your questions on installing and runningGPFS.

– For the latest announcements and news, please subscribe to the IBM Spectrum Scale /GPFSAnnounce Forum at http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1606 .

v Customize your support portal at http://www-01.ibm.com/software/support/einfo.html.Update your notifications to include IBM Spectrum Scale and/or General Parallel File Systemat https://www-947.ibm.com/systems/support/myview/subscription/css.wss/ :– For V4.1.1 and later, search on IBM Spectrum Scale– For earlier releases, search on General Parallel File System

v IBM Global Services - Support Line for LinuxA 24x7 enterprise-level remote support for problem resolution and defect support for majordistributions of the Linux operating system. Go to www.ibm.com/services/us/index.wss/so/its/a1000030.

v IBM Systems Lab ServicesIBM Systems Lab Services can help you optimize the utilization of your data center and systemsolutions.Lab Services has the knowledge and deep skills to support you through the entire informationtechnology race. Focused on the delivery of new technologies and niche offerings, Lab Servicescollaborates with IBM Global Services and IBM Business Partners to provide complementaryservices that will help lead through the turns and curves to keep your business running at topspeed.Go to http://www.ibm.com/systems/services/labservices/.

v Software maintenanceDefect resolution for current holders of IBM software maintenance contracts:– In the United States contact us toll free at 1-800-IBM-SERV (1-800-426-7378)– In other countries, contact your local IBM Service Center

You may also visit the Support Portal online at https://www-947.ibm.com/support/entry/portal/Open_service_request/Software/Other_Software/General_Parallel_File_System?category=2&locale=en_us&lnk=mhsd

Contact [email protected] for all other services or consultation on what service is best for yoursituation.

Q14.2: How do I download fixes for IBM Spectrum Scale?

A14.2: To download fixes, go to the IBM Spectrum Scale page on Fix Central at http://www-933.ibm.com/support/fixcentral/

Note: To receive notifications for fix availability, customize your support portal athttp://www-01.ibm.com/software/support/einfo.html. Update your notifications to include IBMSpectrum Scale and/or General Parallel File System at https://www-947.ibm.com/systems/support/myview/subscription/css.wss/ :v For V4.1.1 and later, search on IBM Spectrum Scalev For earlier releases, search on General Parallel File System

132

http://www14.software.ibm.com/webapp/set2/sas/f/handbook/home.html

http://www14.software.ibm.com/webapp/set2/sas/f/handbook/home.html





http://www-01.ibm.com/software/support/einfo.html


http://www.ibm.com/services/us/index.wss/so/its/a1000030

http://www.ibm.com/services/us/index.wss/so/its/a1000030

http://www.ibm.com/systems/services/labservices/


https://www-947.ibm.com/support/entry/portal/Open_service_request/Software/Other_Software/General_Parallel_File_System?category=2&locale=en_us&lnk=mhsd







http://www-01.ibm.com/software/support/einfo.html



For V4.2 go to http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.0&platform=All&function=all

Direct links for V4.1.1v AIX http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software

%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=AIX&function=all

v Linux on Power BE http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=Linux+64-bit,pSeries&function=all

v Linux on x86 http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=Linux+64-bit,x86_64&function=all

v Linux on Power LE http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=Linux+PPC64LE&function=all

v Linux on Z http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=Linux390+64-bit&function=all

v Windows http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=Windows&function=all


Note: On Fix Central, after you have selected the fix you want, if you change the downloadoption (on the right-hand navigation) to Download using your web browser (HTTPS) (asopposed to FTP or Download Director), then right-click on the package to be downloaded andselect Save Link As and provide the target location.

Direct links prior to V4.1.1:v IBM Spectrum Scale V4.1 Advanced Edition http://www-933.ibm.com/support/fixcentral/

swg/selectFixes?product=ibm/power/IBM+General+Parallel+File+System+Advanced+Edition&release=4.1.0&platform=All&function=all

v IBM Spectrum Scale V4.1 Standard Edition http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%2Bsoftware&product=ibm/power/IBM+General+Parallel+File+System+Standard+Edition&release=4.1.0&platform=All&function=all

v IBM Spectrum Scale V4.1 Express Edition http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%2Bsoftware&product=ibm/power/IBM+General+Parallel+File+System+Express+Edition&release=4.1.0&platform=All&function=all

v GPFS V3.5– AIX http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster+software

&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=AIX&function=all

– Linux on Power http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster+software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=Linux+64-bit,pSeries&function=all

– Linux on x86 Architecture http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster+software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=Linux+64-bit,x86_64&function=all

133




http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=AIX&function=all



http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=Linux+64-bit,pSeries&function=all



http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=Linux+64-bit,x86_64&function=all



http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=Linux390+64-bit&function=all






http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.1.1&platform=Windows&function=all






http://www-933.ibm.com/support/fixcentral/swg/selectFixes?product=ibm/power/IBM+General+Parallel+File+System+Advanced+Edition&release=4.1.0&platform=All&function=all



http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%2Bsoftware&product=ibm/power/IBM+General+Parallel+File+System+Standard+Edition&release=4.1.0&platform=All&function=all




http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%2Bsoftware&product=ibm/power/IBM+General+Parallel+File+System+Express+Edition&release=4.1.0&platform=All&function=all



http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster+software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=AIX&function=all



http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster+software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=Linux+64-bit,pSeries&function=all




http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster+software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=Linux+64-bit,x86_64&function=all




– Windows http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster+software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=Windows&function=all

Q14.3: How do I migrate from GPFS V4.1 or IBM Spectrum Scale V4.1.1 to IBM Spectrum Scale V4.2?A14.3: IBM Spectrum Scale supports node-at-a-time migration if the nodes in the cluster are running the

previous release. Nodes can coexist and interoperate with nodes that are running the previousrelease; however, some new functions that depend on format changes are not available until allnodes are migrated.v To migrate from V4.1.0.x to IBM Spectrum Scale 4.1.1.x, see the IBM Spectrum Scale: Concepts,

Planning, and Installation Guide at http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.1.1/ibmspectrumscale411_welcome.html

v To migrate from V4.1.1.x to IBM Spectrum Scale 4.2 , see the IBM Spectrum Scale: Concepts,Planning, and Installation Guide at http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.2.0/ibmspectrumscale42_welcome.html

Full install images are available for entitled customers on Fix Central beginning with IBMSpectrum Scale 4.2.v V4.2.0 is available at https://www-945.ibm.com/support/fixcentral/swg/


v V4.2.1 is available at https://www-945.ibm.com/support/fixcentral/swg/selectFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.1&platform=All&function=all

Note: AIX images are not licensed except for V4.2.1.0 Standard Edition https://www-945.ibm.com/support/fixcentral/swg/downloadFixes?parent=Software%2Bdefined%2Bstorage&product=ibm/StorageSoftware/IBM+Spectrum+Scale&release=4.2.1&platform=All&function=fixId&fixids=Spectrum_Scale_Standard-4.2.1.0-ppc64-AIX-install&includeRequisites=1&includeSupersedes=0&downloadMethod=http&login=true

Customers moving to either IBM Spectrum Scale V4.1.1 or V4.1.0 must already have a license ordownload DVD / e-image from Passport Advantage:v 4.1.1 is available at https://www-945.ibm.com/support/fixcentral/swg/


v 4.1.0 is available at https://www-945.ibm.com/support/fixcentral/swg/selectFixes?product=ibm/power/IBM+General+Parallel+File+System+Standard+Edition&release=4.1.0&platform=AIX&function=all

Note: If IBM Spectrum Scale Protocols are enabled on a cluster, a direct migration from V4.1.1.xto V4.2.0.1 is not supported. You must go to V4.2.0.0 first, then apply V4.2.0.1. Attempting adirect migration will produce an error message similar to the following:...Attempt 4 of 5Waiting 30 seconds before recheckingChecking state of OBJ on all nodesOBJ not healthy on all nodesAttempt 5 of 5OBJ is not healthy on: nodeA, nodeD, nodeA, nodeB.OBJ is not active on all protocol nodes...One or more post upgrade checks detected errors. See above for more details [root@nodeA installer]

Q14.4: What are the current advisories for all platforms supported by IBM Spectrum Scale?A14.4: The current general advisories are:

134

























Note: It is recommended that you are at the latest fix level available.v Attention: If you have V4.2.1 or V4.2.2.0 installed, you must migrate to V4.2.2.1 or later at

your earliest convenience. For more information, see the Flash (Alert) here:http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009965

v For V4.1.1 and later, see the flashes, alerts, and bulletins at https://www-947.ibm.com/support/entry/myportal/alerts/system_storage/storage_software/software_defined_storage/ibm_spectrum_scale?productContext=1538918429

v For V4.1 and prior, see the flashes, alerts, and bulletins for IBM Spectrum Scale athttps://www-947.ibm.com/support/entry/myportal/alerts/cluster_software/general_parallel_file_system?productContext=444130266

v IBM Spectrum Scale Software/IBM Elastic Storage Server: Release Recommendation

Abstract:

IBM Spectrum Scale customers running prior versions are encouraged to upgrade to V4.2.3.8 inorder to benefit from numerous quality improvement fixes included in this release. IBM ElasticStorage Server (ESS) V5.2.2.1 also leverages IBM Spectrum Scale V4.2.3.8 to provide additionalrobustness and reliability. IBM also strongly recommends that ESS systems running priorversions be upgraded to ESS 5.2.2.1 or later releases.For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1012386.

v Security Bulletin: A vulnerability has been identified in IBM Spectrum Scale with CESstack enabled that could allow sensitive data to be included with service snaps. This datacould be sent to IBM during service engagements (CVE-2018-1512)

Summary:

A security vulnerability has been identified in IBM Spectrum Scale with CES stack enabled thatcould allow sensitive data to be included with service snaps. This data could be sent to IBMduring service engagements (CVE-2018-1512).For more information, see the complete security bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1012325.

v Security Bulletin: A vulnerability has been identified in IBM Spectrum Scale that couldallow a local unprivileged user access to information located in dump files. User data couldbe sent to IBM during service engagements (CVE-2017-1654)

Summary:

A vulnerability has been identified in IBM Spectrum Scale that could allow a local unprivilegeduser access to information located in dump files. User data could be sent to IBM during serviceengagements (CVE-2017-1654).For more information, see the complete security bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010869.

v Flash (Alert): IBM Spectrum Scale (GPFS) V4.1 and 4.2 levels: network reconnect functionmay result in file system corruption or undetected file data corruption

Abstract:

IBM has identified a problem with IBM Spectrum Scale (GPFS) V4.1 and V4.2 levels. Resendingan NSD RPC after a network reconnect function might result in file system corruption orundetected file data corruption.For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010668.

v Flash (Alert): IBM Spectrum Scale (GPFS) V4.2.3: failures in scanning file system metadatamay result in file system data or metadata corruption

Abstract:

IBM has identified a problem with the GPFS file sys:tem metadata scanning function in IBMSpectrum Scale V4.2.3.0 through V4.2.3.3, which might result in silent file system datacorruption or metadata corruption on certain failures.

135

|

|

|||||

||


https://www-947.ibm.com/support/entry/myportal/alerts/system_storage/storage_software/software_defined_storage/ibm_spectrum_scale?productContext=1538918429



https://www-947.ibm.com/support/entry/myportal/alerts/cluster_software/general_parallel_file_system?productContext=444130266

https://www-947.ibm.com/support/entry/myportal/alerts/cluster_software/general_parallel_file_system?productContext=444130266









For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010487.

v Flash (Alert): IBM Spectrum Scale (GPFS): On I/O with file size change on metanodetakeover, the file size change may not be committed to disk

Abstract:

IBM has identified an issue with IBM GPFS and IBM Spectrum Scale in which a file sizechange, which happens during a small timing window at the non-metanode, might not becommitted to the disk during metanode takeover.For more information, see the complete flash at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010293.

v Flash (Alert): IBM Spectrum Scale: O_SYNC/O_DSYNC not honored, potentially causingsome writes to be lost

Abstract:

IBM has identified a problem with IBM Spectrum Scale V4.2.0.1 through V4.2.2.3, in which theO_SYNC/O_DSYNC flag to the open() system call could be ignored, which might result inundetected data corruption due to writes to the file (that appear to have been completed) beinglost if a node failure occurs before data is written to disk.For more information, see the complete flash at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010130.

v Technote: IBM Spectrum Scale Support: SKLM 2.7 UUID Length Workaround

Abstract:

IBM Spectrum Scale versions up to and including 4.2.2.2 only support master encryption keyswith identifiers up to 42 characters in length. With V2.7, IBM Security Key Lifecycle Manager(SKLM) generates encryption keys that are by default up to 48 characters long. Therefore, suchkeys cannot be used for encryption by IBM Spectrum Scale.For more information, see the complete technote at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010026.

v Flash (Alert): IBM Spectrum Scale V4.2.1/4.2.2 parallel log recovery function may result inundetected data corruption

Abstract:

IBM identified a problem with the IBM Spectrum Scale parallel log recovery function inV4.2.1/V4.2.2, which might result in undetected data corruption during a file system recovery.For more information, see the complete flash at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009965.

v Security Bulletin: IBM Spectrum Scale and IBM GPFS are affected by a securityvulnerability (CVE-2016-6115)

Summary: A security vulnerability has been identified in IBM Spectrum Scale (GPFS) thatcould allow a remote authenticated attacker to overflow a buffer and execute arbitrary codeon the system with root privileges or cause the server to crash. This vulnerability is onlyapplicable if:– file encryption is being used– the key management infrastructure has been compromisedSee the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009639

v Security Bulletin: A security vulnerability has been identified in GSKit shipped with IBMSpectrum Scale V4 (CVE-2016-2183)

Summary: A security vulnerability has been identified in one of the cipher suites supportedby GSKit.

See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009579v Abstract:

IBM Spectrum Scale and IBM GPFS are affected by security vulnerabilities (CVE-2016-2985 andCVE-2016-2984)

136













Problem Summary:

Security vulnerabilities have been identified in all levels of IBM Spectrum Scale and IBM GPFSthat could allow:– a local attacker to execute commands as root by setting environment variables processed by

setuid programs (CVE-2016-2985)– a local attacker to execute commands as root by supplying command line parameters to

setuid programs (CVE-2016-2984)See the complete bulletin at either http://www.ibm.com/support/docview.wss?uid=ssg1S1007994 or http://www.ibm.com/support/docview.wss?uid=isg3T1023945

v Abstract:

IBM Spectrum Scale and IBM GPFS are affected by a security vulnerability (CVE-2016-0392)Problem Summary:

A security vulnerability has been identified in all levels of IBM Spectrum Scale and IBM GPFSthat could allow a local attacker to inject commands into setuid file parameters and executecommands as root.See the complete bulletin at either http://www-01.ibm.com/support/docview.wss?uid=isg3T1023763 or http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005781

v Abstract:

IBM Spectrum Scale is affected by a security vulnerability (CVE-2015-7488)Problem Summary:

A security vulnerability has been identified in the current levels of IBM Spectrum Scale V4.1.1thru 4.1.1.3 and V4.2.0.0 that could allow a local, unprivileged user or a user with networkaccess to the IBM Spectrum Scale cluster, access to the LDAP directory bind user passwordwhen File protocol is deployed with LDAP / LDAP with Kerberos based authentication.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005580

v Abstract: Multiple vulnerabilities in IBM Java SDK affect IBM Spectrum Scale(CVE-2015-4843, CVE-2015-4805, CVE-2015-4810, CVE-2015-4806, CVE-2015-4871,CVE-2015-4902)

Problem Summary:

There are multiple vulnerabilities in IBM® SDK Java™ Technology Edition, Version 8 that isused by the IBM Spectrum Scale GUI. These issues were disclosed as part of the IBM Java SDKupdates for October 2015.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005579

v Abstract:

IBM Spectrum Scale is affected by a security vulnerability (CVE-2015-7456)Problem Summary:

A security vulnerability has been identified in the current levels of IBM Spectrum Scale V4.1.1thru 4.1.1.3 and V4.2.0.0 that could allow a local unprivileged user, or a user with networkaccess to the IBM Spectrum Scale cluster, to access admin passwords for object storageinfrastructure. This vulnerability only affects clusters which have installed and deployed theObject protocol.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005476

v Abstract:

In an IBM Spectrum Scale V4.2 file system with multiple storage pools, Quality of Service(QoS) settings should be set for all storage pools to avoid performance degradation forunspecified storage pools.Problem Summary:

In an IBM Spectrum Scale V4.2 file system with multiple storage pools, if the user specifiesQuality of Service for I/O operations (QoS) settings (for the maintenance and other classes)

137












only for one storage pool then the I/O allocations for the unspecified pools will be set to avery low value, resulting in severe performance degradation when I/O is performed to theunspecified storage pool(s).See the complete bulletin at http://www.ibm.com/support/docview.wss?uid=ssg1S1005464

v Abstract:

IBM has identified a failure in the Spectrum Scale daemon which may occur when a givenNetwork Shared Disk (NSD) object is deleted and then recreated via the mmdelnsd andmmcrnsd commands. When this failure occurs the Spectrum Scale mmfsd daemon will failwith the following assert message: logAssertFailed: cfgP->getNsdId() == cfgNP->getNsdId()Problem Summary:

As a result of a timing hole that may occur in the propagation of changes to the configurationdata, an mmfsd assert failure may occur after the mmcrnsd command is run for an NSD thatwas recently removed by the mmdelnsd command.See the complete bulletins at either http://www.ibm.com/support/docview.wss?uid=ssg1S1005460 or http://www.ibm.com/support/docview.wss?uid=isg3T1022972

v Abstract:

A problem has been identified with AFM filesets using the NFS backend to communicate witha GPFS home that is running V4.1 or earlier when upgrading home to IBM Spectrum ScaleV4.1.1 or later.Problem Summary:

If there are AFM filesets using the NFS backend to communicate with a GPFS home that isrunning V4.1 or earlier, upgrading home to IBM Spectrum Scale V4.1.1 or later will cause theAFM filesets to disable synchronization to home with a message as follows: GPFS: 6027-3218Change in home export detected. Synchronization with home is suspended until the problem isresolved. Please do not upgrade home from GPFS V4.1 or earlier to IBM Spectrum Scale V4.1.1or later if continued AFM functionality is desired. Note: This only affects AFM filesets thatwere originally created with a GPFS home running V4.1 or earlier. This does not affect AFMfilesets that were created with a home running IBM Spectrum Scale V4.1.1 or later.See the bulletins at either http://www.ibm.com/support/docview.wss?uid=ssg1S1005459 orhttp://www.ibm.com/support/docview.wss?uid=isg3T1022926

v Security Bulletin: IBM Spectrum Scale and IBM GPFS are affected by securityvulnerabilities (CVE-2015-4974, CVE-2015-4981)

Summary Security vulnerabilities have been identified in the current levels of IBMSpectrum Scale V4.1.1, IBM GPFS V4.1 and V3.5:– could allow a local non privileged attacker to execute commands with root privileges

(CVE-2015-4974)– could allow a local non privileged attacker to read system memory contents (CVE-2015-4981)See the complete bulletin at either http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005366 or http://www-01.ibm.com/support/docview.wss?uid=isg3T1022637

v Security Bulletin: Vulnerability in OpenSSL affects IBM GPFS V4.1 and IBM SpectrumScale V4.1.1 (CVE-2015-1788)

Summary: An OpenSSL denial of service vulnerability disclosed by the OpenSSL Projectaffects GSKit. IBM GPFS V4.1 and IBM Spectrum Scale V4.1.1 use GSKit and addressed theapplicable CVE.

See the complete bulletin at either http://www-01.ibm.com/support/docview.wss?uid=isg3T1022618 or http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005364

v Abstract:

138
















IBM has identified a problem with the GPFS (IBM Spectrum Scale) Rapid Repair function,which is in use by default on GPFS 4.1 format file systems wherever data replication is in use,and may result in undetected data corruption.Problem Summary:

As a result of an incorrect calculation in GPFS code, an update to an internal data structuremay result in modifications being made in the wrong memory location. If this locationcorresponds to a user data or metadata block, this block may become corrupted on disk. Inaddition, the same issue may result in back-level replicas of data blocks being present afterrecovery from down disks.See the complete bulletins at either http://www-01.ibm.com/support/docview.wss?uid=isg3T1022582 or http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005352

v Abstract:

GPFS Product Support has received reports of errors accessing directories created sinceupgrading old GPFS file systems to use the new features of the V4.1 release. The issue isconfined to the rmdir and readdir directory access paths on GPFS file systems originallycreated at the 3.1, or earlier, levels.Problem Summary:

The DirV2 feature of GPFS V4.1 does not work correctly on file systems created prior to V3.2(original format version less than 1000). Those early file systems use a different file namefolding function (foldName) that does not support case-insensitive lookup for Windows (thekey feature in V3.2, for these purposes), and DirV2 changes to hashing names within directoryblocks does not handle this older foldName correctly. The two main effects of this problem areproblems finding the ".." entry and properly sequencing readdir cookies. This leads to rmdirfailing with EEXIST even when the direct is empty. It also causes directory enumerationwithreaddir (getdents) to repeat or skip entries and even to loop, repeatedly returning thesame batch of names indefinitely. These problems only occur after updating the file system toV4.1 and only for directories created since the upgrade (i.e., only DirV2 directories).See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=isg3T1022368

v Security Bulletin: IBM General Parallel File System V4.1 is affected by a securityvulnerability (CVE-2015-1890)

Summary: A security vulnerability has been identified in GPFS V4.1 where the private keyof TLS client certificates used by GPFS nodes may be contained in a gpfs.snap file(CVE-2015-1890).

See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=isg3T1022077v Security Bulletin: IBM General Parallel File System is affected by security vulnerabilities

(CVE-2015-0197, CVE-2015-0198, CVE-2015-0199)

Summary: Security vulnerabilities have been identified in current levels of GPFS V4.1, V3.5,and V3.4

See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=isg3T1022062v Abstract:

IBM has identified a problem with the online replica compare/repair function invoked via themmrestripefs -c command.Problem Summary:

IBM has identified a problem with the online replica compare/repair function invoked via themmrestripefs -c command, when invoked with any disks having status other thanready/replacement, for example with any disks in suspended state. This function may causefile system corruption, with potential loss or corruption of user files, if any replica is be copiedor moved for reasons other than replica mismatch. This problem affects customers running anyPTF level of GPFS 3.5 from GPFS 3.5.0.11 through 3.5.0.20, or any level of IBM Spectrum Scale4.1 from IBM Spectrum Scale 4.1.0.0 through IBM Spectrum Scale 4.1.0.3. The function providedby the mmrestripefs -c command is disabled in PTFs 3.5.0.21 and 4.1.0.4.

139








Users affected :

This problem affects customers running any PTF level of GPFS 3.5 from GPFS 3.5.0.11 through3.5.0.20, or any level of IBM Spectrum Scale 4.1 from IBM Spectrum Scale 4.1.0.0 through IBMSpectrum Scale 4.1.0.3.This problem can only occur when the mmrestripefs -c command is run while there are diskswith status other than ready/replacement. For a file system with data replication, the problemmay only occur if none of the replicas of a data block are on a disk with status ofready/replacement.Users can check if they may have been affected by running the following command todetermine if mmrestripefs -c was ever issued in the cluster:mmdsh -N all grep "mmrestripefs.*-c" /var/adm/ras/mmfs.log\*

If the result of the grep indicates that the command has been run, contact IBM Service.Recommendation:– Avoid running the mmrestripefs command with the -c option until a fix is made available

by IBM. A fix is made available for GPFS V3.5 with APAR IV66437 and IBM Spectrum ScaleV4.1 with APAR IV66123.

– Contact IBM for an efix to disable the code; APAR IV66270 for GPFS V3.5 and APARIV66271 for IBM Spectrum Scale V4.1.

v Abstract:

GPFS may incorrectly permit a disk that is too large for the file system to be added to anexisting FPO storage pool, resulting in undetected data corruption of the file systemProblem Summary:

GPFS is designed to impose a maximum allowable disk size for disks added after the filesystem has been created. An integer overflow problem has been discovered in the GPFS blockmap allocation module that allows GPFS to incorrectly add disks too large to the file system. Adisk that should have failed this maximum size check could be incorrectly added to an FPOenabled storage pool. If a disk is added with size equal to or larger than 8 times the size of thedisks used at the time the file system was originally created and the file system is utilizingFPO enabled storage pools, data blocks will be corrupted when data is written to the new disk.Users affected:

File systems that have added a disk or disks that are equal to or larger than 8 times the size ofthe disk used at the time the file system was originally created and GPFS has FPO enabledstorage pools.Please check the following conditions to assess if your file system is at risk:1. Is the file system FPO enabled? Use the following command to determine. If it's yes, then

this is a FPO pool.mmlspool fsname poolname -L | grep allowWriteAffinity

2. Is the file system metadata block size larger than 256KB? Use the following command todetermine.mmlspool fsname system -L |grep blockSize

3. Have you added disks (or plan to add disks) to the file system that are equal to or largerthan 8 times of the original disks via mmadddisk command? The new disks must be equalto or larger than 8 times in capacity than the largest disk existing at file system creationtime. Use the following mmdf command to review sizes of disks belonging to the filesystem. Review the "disk size" column to determine disks that are equal to or larger than 8times of size of the original disks in the file system.mmdf filesystem

Below is an example from an affected system. The disk data01node04 was added after thefile system was created and its size (3.5TB) is more than 8 times of the original largest disksize (296 GB).

140

# mmdf gpfs1disk disk size failure holds holds free KB free KBname in KB group metadata data in full blocks in fragments--------------- ------------- -------- -------- ----- -------------------- -------------------Disks in storage pool: system (Maximum disk size allowed is 9.0 TB)data01node01 296874976 1001 yes yes 293232640 ( 99%) 2144 ( 0%)data02node01 296874976 1001 yes yes 293307392 ( 99%) 2784 ( 0%)data02node02 296874976 1002 yes yes 296857600 (100%) 1984 ( 0%)data01node02 296874976 1002 yes yes 296856576 (100%) 1984 ( 0%)data02node03 296874976 1003 yes yes 293314560 ( 99%) 2784 ( 0%)data01node03 296874976 1003 yes yes 293234688 ( 99%) 2144 ( 0%)data01node04 3512693760 1004 yes yes 1306125312 ( 37%) 404892224 (12%)

------------- -------------------- -------------------(pool total) 5293943616 3072928768 ( 58%) 404906048 ( 8%)

4. If you are still unable to determine, you will need to unmount the file system and runmmfsck. Please contact IBM support for assistance and further details to run mmfsck.

Problem Description:

See Problem Summary.Required Actions:

IBM recommends that GPFS FPO enabled users apply a fix as soon as it is available and beforeadding new disks to the file system. For GPFS 3.5, the fix is available in GPFS 3.5.0.19(IV60817) on Fix Central. For GPFS 4.1, the fix is available in GPFS 4.1.0.2 (IV62418) on the FixCentral site.If you have determined that you are affected, please call IBM support as soon as possible forassistance with data recovery.

v Abstract:

Conflicting advisory locks may result in undetected data corruption.Problem Summary:

Conflicting advisory locks may be granted when applications use fcntl() to acquire advisorylocks on a GPFS file. After a node acquires an advisory lock, if another node takes any actionwhich triggers an inode token revoke (such as running the chmod command on the file), thefcntl token may be released while the fcntl lock is still held. If another application process thenasks for a conflicting advisory lock for the same file, that request could be granted. User datacould then be corrupted, because the two different application processes believe they haveexclusive access to the same file range.GPFS metadata will not be affected. Only customers using fcntl advisory locks at the affectedGPFS levels could be impacted.Users affected (both of the following conditions must apply for customer to be affected):– Customers running GPFS service levels 4.1.0.0, or 4.1.0.1.– Customer workload includes use of fcntl advisory locks on GPFS files.Problem Description:

See Problem Summary.Recommendation:

Affected customers should contact upgrade to V4.1.0.2 (APAR IV62043).v Abstract:

IBM has identified an issue with GPFS version 3.5.0.16 and later releases that may affectinstallations that use both a non-English language locale settings and also have PersistentReserve enabled for the cluster.Problem Summary:

On such systems, it is possible that the disk usage information recorded in the mainconfiguration file (/var/mmfs/gen/mmsdrfs) is not correct. This may result in an improperhandling of the PR settings and inability to mount the affected file systems. The root cause forthis problem is corrected in GPFS 3.5.0.19 and GPFS 4.1.0.1 (APAR IV61323)Fix

141

http://www.ibm.com/eserver/support/fixes/

To see if your system is susceptible to this problem, run the following commandgrep SG_DISKS /var/mmfs/gen/mmsdrfs | awk -F : ’{ print $3 " " $5 " " $8 }’

and examine the reported disk usage information. If you see descOnly shown for disks that aresupposed to contain data or metadata, then your system is affected and you need to correct theproblem using the following procedure:1. install GPFS 3.5.0.19 or GPFS 4.1.0.12. for each of the affected file systems run

mmcommon recoverfs deviceName

If the mmcommon recoverfs command fails because it cannot read the file system descriptor,then you will need to temporarily disable the Persistent Reserve feature:1. mmshtudown -a2. mmchcconfig usePersistentReserve=no3. mmstartup -a4. mmcommon recoverfs deviceName

Note: repeat for all affected file systems5. mmshtudown -a6. mmchcconfig usePersistentReserve=yes7. mmstartup -a

v Abstract:

GPFS directory corruption with possible undetected data corruptionProblem Summary:

When multiple nodes are updating a shared directory concurrently, the problem could causeincorrect results from directory operations issued on one node, leading to orphaned inodes(files inaccessible from any directory entry), or directory entries pointing to deleted or incorrectfiles. This problem could also cause silent data corruption, if any disk contains both GPFSmetadata and data, and a stale buffer is written to a disk address that has been freed andreallocated for some other purpose.Users affected (both of the following conditions must apply for customer to be affected):– GPFS service levels 3.4.0.24, 3.4.0.25, 3.4.0.26, 3.4.0.27, 3.5.0.13. 3.5.0.14, 3.5.0.15, or 3.5.0.16.– Workload consists of concurrent directory updates from multiple nodes.Problem Description:

See https://www14.software.ibm.com/webapp/set2/subscriptions/onvdq?mode=7&heading=CL_CLUSTER&path=/201404/CL_GPFS/20140410/datafile123513&label=GPFSdirectory corruption with possible undetected data corruptionRecommendation:

Customers who have run the affected service levels should upgrade to GPFS 3.5.0.17 or 3.4.0.28service level updates (go to Fix Central). Customers who have seen FSSTRUCT 1124 or 1122messages, or EIO errors during directory operations, should also run off-line fsck to identifyand repair possible directory damage.

v Abstract:

A delete of a migrated file that has a copy in a snapshot will fail to recall the migrated datafrom HSM and the failure is not detected by GPFS.Problem:

Under certain conditions, a delete of a migrated file that has a copy in a snapshot will fail torecall the migrated data from HSM and the failure is not detected by GPFS. A subsequent HSMreconcile operation initiated by the administrator may cause the HSM system to delete the lastremaining copy of the file data that was thought to still remain on disk in the snapshot view.Root cause:

The issue was discovered during testing of a scenario where many files were deleted from afile system that is managed by HSM. A GPFS snapshot of the file system had been created after

142

https://www14.software.ibm.com/webapp/set2/subscriptions/onvdq?mode=7&heading=CL_CLUSTER&path=/201404/CL_GPFS/20140410/datafile123513&label=GPFS directory corruption with possible undetected data corruption




HSM migration but before file deletion, so migrated files' data were not resident on disk. Thedeletion should cause a recall of each deleted file's data from the HSM to populate thesnapshot view of the files. IBM is not aware of any occurrences of this issue in customerenvironments or under any other circumstances. Since the issue is specific to accessing only thesnapshot copies of migrated data for already deleted files, it does not affect applications usingthe active (non-snapshot) view of the file system. If an administrator were to issue an HSMreconcile command, it is possible the HSM system may delete the last remaining copy of thefile data for those files deleted, but believed to still be preserved in the GPFS snapshot.Fix:

The fix is available in GPFS 3.5.0.18 (APAR IV58500) and GPFS 3.4.0.29 (APAR IV59300 ).Customers using HSM should update to 3.5.0.18 or 3.4.0.29 available on Fix Central.Users affected (all of the following conditions must apply for customer to be affected):

Users of GPFS 3.4.0.16 and later, or GPFS 3.5.0.3 and later, who are utilizing HSM and thefollowing conditions exist:– Files are migrated.– A GPFS snapshot is made of the file system.– Files are deleted from the active file system.– HSM Reconcile operation is initiated by the administrator.– Access of deleted files is attempted through the snapshot view of the file system.

v Issue Description:

IBM has identified a memory leak in GPFS 3.5.0.14. Users of GPFS 3.5.0.14 may experience outof memory errors (ENOMEM) or a GPFS crash as a result of a memory allocation failure.Fix and Recommendations:

IBM provided a fix for this problem now available in GPFS 3.5.0.15 (APAR IV51893). Allcustomers using 3.5.0.14 should update to 3.5.0.15 or later, as soon as possible.

v Abstract:

Under certain conditions, an issue with GPFS 3.5.0.10 - 3.5.0.12 using an AFM fileset may resultin undetected in-memory data corruption.Issue Description:

Under certain conditions involving an AFM fileset using any mode, a read to an uncachedblock in AFM cache while background prefetch is running may return wrong data (e.g., abuffer filled with zeros). The AFM background prefetch is triggered automatically when anapplication reads more than 2 blocks or based on the afmPrefetchThreshold value set on thefileset. The file data on disk is always correct, so a reread of the file will return correct dataonce background prefetch of the file completes.Users Affected:

Only users of GPFS version 3.5.0.10 or later who are utilizing AFM cache and haven't explicitlydisabled background prefetch.Users not Affected:– Users not running AFM.– GPFS users running 3.5.0.11 or later and having set afmPrefetchThreshold to 100.– GPFS users running AFM using 'SW' (Single Writer) mode, where all data is generated by

the cache site ( e.g. there are no read operations to the home cluster).Fix and Recommendations:

The fix is available in GPFS 3.5.0.13 (APAR IV48136). Customers using AFM should update to3.5.0.13 or later.Workaround until fix is applied:

Both of the below conditions must be met.1. Set afmPrefetchThreshold to 100 (Note: requires GPFS 3.5.0.11 or later).2. If the AFM prepop command (mmafmctl Device prefetch -j FilesetName) is running, ensure

that the command completes before issuing writes to the fileset.v Abstract:

143


Under certain conditions, an issue with IBM GPFS 3.5.0.11 using AFM Single Writer orIndependent Writer modes may result in undetected data loss.Issue Description:

Under certain conditions involving an AFM fileset using Single Writer or Independent Writermodes, a write to an uncached block in the AFM cache may be overwritten by datasubsequently arriving from the AFM home resulting in undetected data loss. This may occurwhen an application reads enough blocks of an uncached file to trigger AFM backgroundprefetch, which will read data from home and write to cache. While this prefetch is running, anapplication write to an uncached portion of the file (file blocks not yet pre-fetched to the AFMcache) may succeed, but at a later point the block so written may be overwritten with datafrom the AFM home which had not yet arrived in the AFM cache.Users Affected:

Only users of GPFS version 3.5.0.11 who are utilizing AFM Single Writer or Independent Writercache starting with non-empty AFM home.Users not Affected:

The following uses of GPFS version 3.5.0.11 should not be affected by this issue.1. Users not running AFM2. Read Only or Local Update AFM caches3. Single Writer or Independent Writer AFM caches with empty AFM home4. Files created in AFM Single Writer or Independent Writer cachesFix and Recommendations:

The fix is available in GPFS 3.5.0.12 (APAR IV46085). Customers using AFM should update to3.5.0.12 or later.Workaround until fix is applied:

Both of the below conditions must be met.1. Set afmPrefetchThreshold to 1002. If the AFM prepop command (mmafmctl Device prefetch -j FilesetName) is running, ensure

that the command completes before issuing writes to the fileset.v In internal testing on an unsupported configuration, IBM has identified a potential data

integrity issue in GPFS 3.4.0.9 and later releases of GPFS 3.4, and with GPFS 3.5. This issue hasnot, to IBM's knowledge, been experienced in the field. The issue occurs only when writing viaNFS to a GPFS node running on the Linux operating system. This issue may manifest as eithera data loss issue from a transient read error, returning zeros rather than the expected data, oras incorrect data returned following a write that is applied to the wrong file.The fix for this issue is in GPFS 3.4.0.15 (APAR IV24937) and GPFS 3.5.0.3 (APAR IV24942).While IBM cannot rule out the possibility that this problem may impact any customer writingvia NFS in Linux GPFS environments at the stated levels, empirical evidence suggests that thelikelihood of customers encountering this issue is quite limited. To avoid this issue customersshould upgrade to GPFS 3.4.0.15 or GPFS 3.5.0.3 or later.

v IBM has identified an issue with GPFS file systems at version 3.5 which currently use, or havepreviously used independent filesets and Access Control Lists (ACLs). This issue can occur ifindependent filesets have been created with the mmcrfileset --inode-space command and anyfiles or directories in the GPFS file system have ACLs added or modified with commands suchas mmputacl or mmeditacl. File systems which have previously used but have now deletedindependent filesets are still susceptible to this issue.The issue can result in a loss of data access to any file or directory within the GPFS file systemwhich has an ACL.The fix is available in GPFS version 3.5.0.3 (APAR IV24426).Customers who have created independent filesets, even if the filesets are now deleted, arerequested to not create or modify GPFS ACLs on any file or directory until the IBM Service fixis applied.

144

Customers who have not created independent filesets are requested to not do so until the IBMService fix is applied.If you are already affected by loss of access to files or directories with ACLs, then you arerequested to call IBM Service and reference this problem. The IBM Service person will walkyou through a fix for correcting the issue.Please follow the steps below to determine if your GPFS file system may be susceptible to thisissue.1. On a system with GPFS running, invoke the mmlsconfig command to determine the device

names of all file systems2. For each file system device listed in step 1, use the following command to determine if

your system may be affected:echo desc | /usr/lpp/mmfs/bin/tsdbfs fs-device | grep inodeSpaceMask

where fs-device is the device output from step 1 (e.g. /dev/gpfs in step 1 output example)If there is any non-zero value from the output of the command, your system is exposed tothe issue.In the following example, the file system /dev/gpfs is safe (no independent filesets werecreated) but files in file system /dev/trouble with ACLs are at risk.# echo desc | /usr/lpp/mmfs/bin/tsdbfs /dev/trouble | grep inodeSpaceMaskinodeSpaceMask 0000000000000100 0000000000000000000000000000000000000000000000000000000100000000

# echo desc | /usr/lpp/mmfs/bin/tsdbfs /dev/gpfs | grep inodeSpaceMaskinodeSpaceMask 0000000000000000 0000000000000000000000000000000000000000000000000000000000000000

v IBM has identified an issue with GPFS file systems at versions 3.4 or 3.5 which were migratedfrom file systems created with GPFS versions earlier than 3.4. This issue can occur only afterusing the mmmigratefs command with the [--fastea] option.The issue can result in a loss of data, requiring the restoration of data from a backup source.GPFS file systems created with versions earlier than 3.4 should not be migrated using themmmigratefs command with the [--fastea] option unless you are at GPFS version 3.5.0.3(APAR IV24151) or later, or 3.4.0.15 (APAR IV24150) or later.If customers have already migrated file systems from GPFS versions earlier than 3.4, IBMService has a fix. Please follow the steps below to determine if your system may be affected.To determine if your system may be affected:1. Ensure your GPFS file systems are mounted.2. As a user with GPFS administrator privileges on a machine where your GPFS file systems

are mounted, issue the command:/usr/lpp/mmfs/bin/mmfsadm dump stripe | grep "inode 0"

The command will produce output that identifies locations for the "inode 0" file for allcurrently mounted GPFS file systems. Example output for a file system configured with twoway metadata replication would be in the form:inode 0: 3:4098 1:4098

For a file system with no metadata replication the output would be in the form:inode 0: 3:4098

The relevant information to look for to see if you may experience a problem are the fieldsdenoting disk:sector for each inode 0 replica (e.g. 3:4098 and 1:4098 in these examples).If each disk:sector replica only denotes 4098 for the sector field then you are notexperiencing this problem. If however there is a number other than 4098 in the sectoroutput then you are requested to immediately call IBM service and reference thisproblem. The IBM Service person will walk you through a fix for correcting the issue.

v When installing or migrating GPFS, the minimum levels of service you must have applied are:– GPFS V3.5 you must apply APAR IV15333 (GPFS V3.5.0.1)– GPFS V3.4 you must apply APAR IZ78460 (GPFSV3.4.0.1)If you do not apply these levels of service and you attempt to start GPFS, you will receive anerror message similar to:

145

mmstartup: Required service not applied. Install GPFS 3.5.0.1 or latermmstartup: Command failed Examine previous error messages to determine cause

Q14.5: What are the current advisories for IBM Spectrum Scale on AIX?A14.5: The current AIX-specific advisories are:


Abstract:


v Security Bulletin: Vulnerabilities in GSKit affect IBM Spectrum Scale (CVE-2018-1431,CVE-2017-3736, CVE-2017-3732, CVE-2016-0705)

Summary:

Vulnerabilities in GSKit affect IBM Spectrum Scale where:– A local attacker could obtain control of the IBM Spectrum Scale daemon to access and

modify files in the IBM Spectrum Scale file system, and possibly to obtain administratorprivileges on the node (CVE-2018-1431).

– OpenSSL could allow a remote attacker to obtain sensitive information, which is caused by acarry propagation flaw in the x86_64 Montgomery squaring function bn_sqrx8x_internal().An attacker with online access to a system that is not patched could exploit thisvulnerability to obtain information about the private key (CVE-2017-3736).

– OpenSSL could allow a remote attacker to obtain sensitive information, which is caused by acarry propagating bug in the x86_64 Montgomery squaring procedure. An attacker couldexploit this vulnerability to obtain information about the private key (CVE-2017-3732).

– OpenSSL is vulnerable to a denial of service, which is caused by a double-free error whenparsing DSA private keys. An attacker could exploit this vulnerability to corrupt memoryand cause a denial of service (CVE-2016-0705).

For more information, see the complete security bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1012049.

v Technote (troubleshooting): IBM Spectrum Scale support: mmchfs -V full command will failon upgrade from file system 2.2

Problem (Abstract):

If your file system version is still at the 2.2 level, then running mmchfs -V full on certain levelsof IBM Spectrum Scale will cause the mmchfs command to fail. The file system will be unusableuntil the fix is applied.For more information, see the complete technote at http://www.ibm.com/support/docview.wss?uid=ssg1S1010654.

v Flash (Alert): IBM Spectrum Scale (GPFS) AFM incorrectly replicates data when write andtruncate operations are interleaved

Abstract:

IBM has identified an issue with AFM in IBM GPFS (V3.5.0.0 through V3.5.0.34 or V4.1.0.0through V4.1.0.8) and IBM Spectrum Scale (V4.1.1.0 through V4.1.1.16 or V4.2.0.0 throughV4.2.3.4) levels where AFM might not transfer write operations completely when a file istruncated. This might cause a data mismatch between cache (or primary) and home (orsecondary). This issue might result in undetected data corruption at home (or secondary).For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010629.

v Flash (Alert): IBM Spectrum Scale v4.2.3.1 and v4.2.3.2 installation toolkit on ppc64platform(/usr/lpp/mmfs/4.2.3.x/installer/spectrumscale) might fail when attempting to run./spectrumscale install, deploy, or upgrade

146

|

|

|||||

||||

|

||||||||||||||

||









Abstract:

IBM has identified an issue with IBM Spectrum Scale v4.2.3.1 and v4.2.3.2 of the installationtoolkit on ppc64 in which the toolkit might fail if it is run multiple times. This can occurduring install, deploy, or upgrade and will give a FATAL message to the user and prevent thedesired task from completing.For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010505.

v Flash (Alert): IBM Spectrum Scale: AFM incorrectly replicates rename operations

Abstract:

IBM has identified an issue with AFM in IBM Spectrum Scale V4.1.1.12 through V4.1.1.15 andV4.2.2.0 through V4.2.3.2 levels where AFM might incorrectly replicate the rename operations.This issue might cause undetected data loss due to some files missing at the target site.For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010426.

v Flash (Alert): IBM Spectrum Scale (GPFS) V4.2.3 AFM with DMAPI-enabled file systemcauses mmfsd daemon failure on gateway node

Abstract:

IBM identified an issue with AFM on a DMAPI-enabled filesystem in IBM Spectrum ScaleV4.2.3 where health monitoring is enabled by default, which causes an mmfsd daemon assert onthe gateway node. This issue might cause the mmfsd daemon on the gateway nodes to fail andrestart repeatedly.For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010103.

v Flash (Alert): IBM Spectrum Scale Active File Management (AFM) and AFM AsynchronousDisaster Recovery (DR)

Abstract:

IBM identified certain situations with respect to Active File Management (AFM) and AFMAsynchronous Disaster Recovery (DR) in IBM Spectrum Scale that may result in undetecteddata corruption:– AFM may intermittently read files from the home cluster incorrectly, which could result in

undetected data corruption due to Direct IO usage.– AFM may have undetected data corruption when eviction and read operations run in

parallel on the same file.– AFM cache may incorrectly read a file from the home cluster due to the incorrect calculation

of the file sparseness information, potentially resulting in undetected data corruption.– If parallel IO is enabled, AFM and AFM Asynchronous DR may experience undetected data

corruption with failover, resync and changeSecondary commands.– AFM Asynchronous DR failback may read HSM migrated files from the acting AFM

Primary cluster (originally the AFM Secondary cluster) as sparse files, potentially resultingin the AFM cache to return incorrect data (all zeros) to an application on a read.

See the complete bulletin at either http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009244 or http://www-01.ibm.com/support/docview.wss?uid=isg3T1024249.

v Flash (Alert): IBM Spectrum Scale (GPFS) V4.2 and V4.1.1 AFM Async DR requirement forplanning

Abstract:

Our initial feedback from the field suggests that success of a disaster recovery solutiondepends on administration discipline, including careful design, configuration and testing.Considering this, IBM has decided to disable the Active File Management- based AsynchronousDisaster Recovery feature (AFM DR) by default and require that customers deploying the AFMDR feature first review their deployments with IBM Spectrum Scale development. You should

147











contact Spectrum Scale Support at [email protected] to have your use case reviewed. IBM willhelp optimize your tuning parameters and enable the feature. Please include this messagewhile contacting IBM Support.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005817

v Flash (Alert): IBM Spectrum Scale (GPFS) V4.2 AFM cache reads an HSM migrated file fromhome as a sparse file

Abstract:

IBM has identified an issue with AFM in IBM Spectrum Scale V4.2.0.0 through 4.2.0.2 levelswhen the AFM cache reads the HSM migrated files from home (running IBM Spectrum ScaleV4.2 or later) as a sparse file. This issue may cause the cache to have undetected datacorruption and may return incorrect data (all zeros) to an application on read.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005766

v Security Bulletin: IBM Spectrum Scale is affected by a security vulnerability(CVE-2016-0263)

Summary:

A security vulnerability has been identified in the current levels of IBM Spectrum Scale V4.2,V4.1 and IBM General Parallel File System V3.5, that could allow a local user, under specialcircumstances, to escalate their privileges or cause a denial of service when the mmapplypolicycommand is issued with certain options and syntax.See the complete bulletin at either http://www-01.ibm.com/support/docview.wss?uid=isg3T1023450 or http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005708

v Security Bulletin: IBM Spectrum Scale V4.1.1, IBM GPFS V4.1, and IBM V3.5 for AIX areaffected by a security vulnerability (CVE-2015-7403)

Summary:

A security vulnerability has been identified in the current levels of IBM Spectrum Scale V4.1.1,IBM GPFS V4.1 and V3.5 that could allow a local attacker to cause the node they are on tocrash.See the complete bulletin at either http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005452 or http://www.ibm.com/support/docview.wss?uid=isg3T1022940

v In order for tracing to function properly on a system running the levels of AIX listed below,appropriate service must be installed. If you are running GPFS without the appropriate servicelevel installed and have AIX tracing enabled (such as by using the GPFS mmtracectlcommand), you will experience a GPFS memory fault (coredump) or node crash with kernelpanic.– AIX V7.1 with the 7100-00 Technology Level, you must either install AIX 7100-00-02 Service

Pack or open a PMR to obtain an iFix for APAR IZ84576 from IBM Service.– AIX V6.1 with the 6100-06 Technology Level, you must either install AIX 6100-06-02 Service

Pack or open a PMR to obtain an iFix for APAR IZ84729 from IBM Service.Q14.6: What are the current advisories for IBM Spectrum Scale on Linux?A14.6: The current Linux-specific advisories are:


Abstract:


v Technote (troubleshooting): The pre-built SELinux policy within RHEL7.x conflicts withIBM Spectrum Scale NFS Ganesha

148

|

|

|||||

||||














Problem (Abstract):

Ganesha running on CES nodes with SELinux in enforcing mode and selinux-policy-targeted-3.13.x and later installed causes the start of Ganesha to fail and thus all CES nodes getUNHEALTHY. See https://bugzilla.redhat.com/show_bug.cgi?id=1383784.

Note: IBM Spectrum Scale does not support CES with SELinux in enforcing mode.For more information, see the complete technote at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009947.

v Security Bulletin: A vulnerability in IBM Java SDK affects IBM Spectrum Scale

Summary:

There is a vulnerability in IBM SDK Java Technology Edition, Version 8 used by IBM SpectrumScale. This issue was disclosed as part of the IBM Java SDK updates in April 2018.For more information, see the complete security bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1012400.

v Security Bulletin: Vulnerabilities in GSKit affect IBM Spectrum Scale (CVE-2018-1431,CVE-2017-3736, CVE-2017-3732, CVE-2016-0705)

Summary:

Vulnerabilities in GSKit affect IBM Spectrum Scale where:– A local attacker could obtain control of the IBM Spectrum Scale daemon to access and

modify files in the IBM Spectrum Scale file system, and possibly to obtain administratorprivileges on the node (CVE-2018-1431).

– OpenSSL could allow a remote attacker to obtain sensitive information, which is caused by acarry propagation flaw in the x86_64 Montgomery squaring function bn_sqrx8x_internal().An attacker with online access to a system that is not patched could exploit thisvulnerability to obtain information about the private key (CVE-2017-3736).

– OpenSSL could allow a remote attacker to obtain sensitive information, which is caused by acarry propagating bug in the x86_64 Montgomery squaring procedure. An attacker couldexploit this vulnerability to obtain information about the private key (CVE-2017-3732).

– OpenSSL is vulnerable to a denial of service, which is caused by a double-free error whenparsing DSA private keys. An attacker could exploit this vulnerability to corrupt memoryand cause a denial of service (CVE-2016-0705).


v Security Bulletin: A vulnerability has been identified in IBM Spectrum Scale with CESstack enabled that could allow sensitive data to be included with service snaps. This datacould be sent to IBM during service engagements (CVE-2018-1512)

Summary:

A security vulnerability has been identified in IBM Spectrum Scale with CES stack enabled thatcould allow sensitive data to be included with service snaps. This data could be sent to IBMduring service engagements (CVE-2018-1512).For more information, see the complete security bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1012325.

v Security Bulletin: Vulnerabilities in Samba affect IBM Spectrum Scale SMB protocol accessmethod (CVE-2017-14746, CVE-2017-15275)

Summary:

Vulnerabilities in Samba affect IBM Spectrum Scale SMB protocol access method in a mannerthat could result in the following issues:– Allowing a remote attacker to execute arbitrary code on the system, caused by a

use-after-free memory error (CVE-2017-14746).– Allowing a remote attacker to obtain sensitive information, caused by a heap memory

information leak (CVE-2017-15275).

149

|

|||

|

|||

|

||

||||

|

||||||||||||||

||

https://bugzilla.redhat.com/show_bug.cgi?id=1383784










v Flash (Alert): IBM Spectrum Scale (GPFS): Undetected corruption of archived sparse files(Linux)

Abstract:

IBM has identified an issue with IBM GPFS and IBM Spectrum Scale for Linux environments inwhich a sparse file may be silently corrupted during archival, which could result in the filebeing restored incorrectly.For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1012054.

v Technote (troubleshooting): IBM Spectrum Scale: Using O_DIRECT and fork(2) in the sameprocess in Linux

Problem (Abstract):

IBM Spectrum Scale: Using O_DIRECT and fork(2) in the same process in Linux.Symptom:

System call read() might fail to record data into the user buffer when direct I/O is used, that is,when specifying the O_DIRECT flag when opening the file.For more information, see the complete technote at http://www.ibm.com/support/docview.wss?uid=ssg1S1010878.

v Technote (troubleshooting): IBM Spectrum Scale v4.2.x may experience cluster hangs,unacceptably high CPU load spikes, or high Command Line Interface command executiontime on IBM System z14

Problem (Abstract):

Core components of IBM Spectrum Scale initialize and use encrypted communication. Thiscommunication is happening whenever certain IBM Spectrum Scale commands are performed,even when the system appears to be idle. Depending on the IBM System z14 configuration,cluster hangs might occur. In addition, encryption initialization might temporarily spike CPUusage to 100% and response times will slow down to unreasonable speeds.IBM has corrected the issues in IBM Spectrum Scale v4.2.3.6 and later releases for zLinux only.For more information, see the complete technote at http://www.ibm.com/support/docview.wss?uid=ssg1S1010859.

v Security Bulletin: Vulnerabilities in Samba affect IBM Spectrum Scale SMB protocol accessmethod (CVE-2017-12163, CVE-2017-12151, CVE-2017-12150)

Summary:

Vulnerabilities in Samba affect IBM Spectrum Scale SMB protocol access method that:– Could allow a remote authenticated attacker to obtain sensitive information, caused by a

memory leak over SMB1 (CVE-2017-12163).– Could provide weaker than expected security, caused by the failure to properly sign and

encrypt DFS redirects when the max protocol for the original connection is set as ''SMB3''(CVE-2017-12151).

– Could allow a remote attacker to obtain sensitive information, caused by the failure torequire SMB signing in SMB1/2/3 connections (CVE-2017-12150).

For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010703.

v Technote (troubleshooting): IBM Spectrum Scale support: mmchfs -V full command will failon upgrade from file system 2.2

Problem (Abstract):

If your file system version is still at the 2.2 level, then running mmchfs -V full on certain levelsof IBM Spectrum Scale will cause the mmchfs command to fail. The file system will be unusableuntil the fix is applied.

150











For more information, see the complete technote at http://www.ibm.com/support/docview.wss?uid=ssg1S1010654.

v Flash (Alert): IBM Spectrum Scale: Quick restart of ctdb under SMB load can lead to a racecondition

Abstract:

IBM has identified an issue with IBM Spectrum Scale V4.2.0.x in which a quick restart of ctdbunder SMB load can lead to a race condition that keeps ctdb stuck in an endless recovery. Thiscan occur during upgrade or by rapidly bringing the SMB service online and offline.For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010618.

v Technote (troubleshooting): IBM Spectrum Scale: SMB cluster export services (CES) must notbe upgraded concurrently

Problem (Abstract):

SMB cluster export services (CES) within IBM Spectrum Scale must not be upgradedconcurrently nor have different versions of the gpfs.smb rpm active across multiple CES nodesat once.For more information, see the complete technote at http://www.ibm.com/support/docview.wss?uid=ssg1S1010619.

v Flash (Alert): IBM Spectrum Scale (GPFS) AFM incorrectly replicates data when write andtruncate operations are interleaved

Abstract:

IBM has identified an issue with AFM in IBM GPFS (V3.5.0.0 through V3.5.0.34 or V4.1.0.0through V4.1.0.8) and IBM Spectrum Scale (V4.1.1.0 through V4.1.1.16 or V4.2.0.0 throughV4.2.3.4) levels where AFM might not transfer write operations completely when a file istruncated. This might cause a data mismatch between cache (or primary) and home (orsecondary). This issue might result in undetected data corruption at home (or secondary).For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010629.

v Flash (Alert): IBM Spectrum Scale NFS Cluster Export Services may experience NFSv3 accessloss during upgrade from v4.2.1.x

Abstract:

IBM has identified an issue with IBM Spectrum Scale v4.2.1 cluster export services (CES) thatoccurs during upgrades from IBM Spectrum Scale v4.2.1.x to IBM Spectrum Scale v4.2.2.0. Theissue results in NFSv3 access loss if the SHORT_FILE_HANDLE option is set to True.For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010628.

v Security Bulletin: IBM Spectrum Scale Object Protocols functionality is affected by asecurity vulnerability in Python (CVE-2017-2592)

Summary:

IBM Spectrum Scale Object Protocols functionality is affected by a security vulnerability inPython that could allow a local authenticated attacker to obtain sensitive information, which iscaused by including sensitive data in the CatchError class. A local attacker could exploit thisvulnerability to obtain sensitive information.For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010471.

v Flash (Alert): IBM Spectrum Scale v4.2.3.1 and v4.2.3.2 installation toolkit on ppc64platform(/usr/lpp/mmfs/4.2.3.x/installer/spectrumscale) might fail when attempting to run./spectrumscale install, deploy, or upgrade

Abstract:

151













IBM has identified an issue with IBM Spectrum Scale v4.2.3.1 and v4.2.3.2 of the installationtoolkit on ppc64 in which the toolkit might fail if it is run multiple times. This can occurduring install, deploy, or upgrade and will give a FATAL message to the user and prevent thedesired task from completing.For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010505.

v Security Bulletin: A vulnerability in Samba affects IBM Spectrum Scale SMB protocol accessmethod (CVE-2017-9461)

Summary:

A Samba vulnerability affects IBM Spectrum Scale SMB protocol access method, which couldallow denial of service that is caused by improper handling of dangling symlinks in smbd. Aremote attacker could exploit this vulnerability to cause a fd_open_atomic infinite loop withhigh CPU usage and memory consumption on the system.For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010376.

v Flash (Alert): IBM Spectrum Scale: AFM incorrectly replicates rename operations

Abstract:

IBM has identified an issue with AFM in IBM Spectrum Scale V4.1.1.12 through V4.1.1.15 andV4.2.2.0 through V4.2.3.2 levels where AFM might incorrectly replicate the rename operations.This issue might cause undetected data loss due to some files missing at the target site.For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010426.

v Flash (Alert): IBM Spectrum Scale and IBM Elastic Storage Server: Performance monitoringcomponent potentially running fileset quota sensor on more than one node causingadditional load

Abstract:

IBM has identified a configuration issue with the performance monitoring component in IBMSpectrum Scale V4.2.2 and V4.2.3 and IBM Elastic Storage Server (ESS) 5.0 and 5.1. The defaultconfiguration is monitoring the fileset quota usage hourly from all nodes in the cluster insteadof just from one node. This issue might cause more load on the system than needed and cancontribute to an overload situation.Problem Summary:

The GPFSFilesetQuota sensor is by default not restricted to run on a single node so that it isexecuted every hour from all cluster nodes. This command needs to communicate with allother nodes that have been mounted on the filesystem.For more information, see the complete flash at http://www.ibm.com/support/docview.wss?uid=ssg1S1010423.

v Flash (Alert): IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on theNSD server may result in file IO error

Abstract:

IBM has identified an issue with all IBM GPFS and IBM Spectrum Scale versions where theNSD server is enabled to use RDMA for file IO and the storage used in your GPFS cluster isaccessed via NSD servers (not fully SAN accessible). IBM Elastic Storage Server (ESS) andGPFS Storage Server (GSS) are not affected. Under these conditions, when the RDMA-enablednetwork adapter fails, the issue might result in undetected data corruption for file write orread operations.For more information, see the complete flash at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010233.


Summary:

152











A Samba vulnerability affects IBM Spectrum Scale SMB protocol access method, which couldallow a remote authenticated attacker to launch a symlink attack, caused by a race condition.An attacker could exploit this vulnerability using SMB1 unix extensions or NFS to create asymbolic link from a temporary file to various files on the system, which could allow theattacker to view non-exported files.For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010155.

v Security Bulletin: IBM Spectrum Scale Object Protocols functionality is affected by securityvulnerabilities in OpenStack (CVE-2015-1852 and CVE-2015-7546)

Summary:

IBM Spectrum Scale Object Protocols functionality is affected by security vulnerabilities inOpenStack that could allow:– A man-in-the-middle attack, caused by an error in the api-paste.ini configuration file. A

remote attacker could exploit this vulnerability using a specially-crafted handshake toconduct man-in-the-middle attacks to decrypt and modify traffic (CVE-2015-1852)

– A remote attacker to bypass security restrictions, caused by an error when using the PKI orPKIZ token providers. By manipulating the token contents of a revoked token, therevocation check will improperly consider the token as valid. An attacker could exploit thisvulnerability using a revoked token to gain unauthorized access to cloud resources(CVE-2015-7546)

For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010157.


Summary:

A Samba vulnerability affects IBM Spectrum Scale SMB protocol access method, which couldallow a remote authenticated attacker to execute arbitrary code on the system. It is caused byimproper access to named pipe endpoints. By uploading a specially-crafted shared library to awriteable share, an attacker could exploit this vulnerability to execute arbitrary code on thesystem.For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010317.

v Flash (Alert): IBM Spectrum Scale (GPFS): Asynchronous I/O write: file size change may notbe updated

Abstract:

IBM has identified an issue with IBM Spectrum Scale V4.1.0.4 through V4.1.1.14 and V4.2.0.0through V4.2.3.0 levels when asynchronous Direct I/O is used to write to a file on LINUX,using the io_submit interface.Problem Summary:

As a result of an asynchronous Direct I/O write using the io_ submit interface, a file sizechange may not be updated correctly, which could possibly lead to undetected loss of datawhen a node fails before the file size change is committed to disk.For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010223.

v Security Bulletin: Multiple vulnerabilities in IBM Java SDK affect IBM Spectrum Scale

Summary:

There are multiple vulnerabilities in IBM® SDK Java™ Technology Edition, Version 8 used byIBM Spectrum Scale. These issues were disclosed as part of the IBM Java SDK updates inJanuary 2017.For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010033.

153











v Flash (Alert): IBM Spectrum Scale (GPFS) V4.2.3 AFM with DMAPI-enabled file systemcauses mmfsd daemon failure on gateway node

Abstract:

IBM identified an issue with AFM on a DMAPI-enabled filesystem in IBM Spectrum ScaleV4.2.3 where health monitoring is enabled by default, which causes an mmfsd daemon assert onthe gateway node. This issue might cause the mmfsd daemon on the gateway nodes to fail andrestart repeatedly.For more information, see the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010103.

v Technote: IBM Spectrum Scale support: Kernel crashes on Ubuntu 16.04.1 with kernelversion 4.4.0-57 or later

Summary:

When using Ubuntu 16.04.1 and upgrading the kernel version to 4.4.0-57 or later, you mightencounter a kernel crash issue when there are some operations involving setting and removingan extended file attribute (for example, using the setxattr and removexattr syscall).For more information, see the complete technote at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1010071.

v Security Bulletin: Vulnerabilities in Samba affect IBM Spectrum Scale SMB protocol accessmethod (CVE-2016-2126, 2016-2125)

Summary:

Samba vulnerabilities affect IBM Spectrum Scale SMB protocol access method, which couldallow the following events to occur:– A remote authenticated attacker to gain elevated privileges on the system, caused by

forwarding a Ticket Granting Ticket (TGT) to other service when using Kerberosauthentication. An attacker could exploit this vulnerability to impersonate the authenticateduser and gain elevated privileges on the system (2016-2125).

– A remote authenticated attacker to gain elevated privileges on the system, caused by thefailure of handling the PAC checksum. By using a specially-crafted Kerberos ticket, anauthenticated attacker could exploit this vulnerability to gain privileges or cause thewinbind process to crash (2016-2126).

See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009714.v Technote: IBM Spectrum Scale support: 'blk_cloned_rq_check_limits: over max size limit'

errors

Abstract:

If you are using a GPFS block size larger than 512 Kbytes, you may encounter'blk_cloned_rq_check_limits: over max size limit' errors, which lead to dm-multipath pathfailure. GPFS then will not be able to access the underlying block device.See the complete technote at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009622.

v Security Bulletin: IBM Spectrum Scale Object Protocols functionality (Linux Standard andAdvanced) is affected by security vulnerabilities in the TLS and SSL protocols(CVE-2015-2808 and CVE-2014-3566)

Summary:

The IBM Spectrum Scale object protocols functionality on the Linux (standard and advanced)platform is affected by security vulnerabilities in the TLS and SSL protocols.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009336.

v Flash (Alert): IBM Spectrum Scale Active File Management (AFM) and AFM AsynchronousDisaster Recovery (DR)

Abstract:

IBM has identified certain situations with respect to Active File Management (AFM) and AFMAsynchronous Disaster Recovery (DR) in IBM Spectrum Scale that may result in undetecteddata corruption:

154









– AFM may intermittently read files from the home cluster incorrectly which could result inundetected data corruption due to Direct IO usage.

– AFM may have undetected data corruption when eviction and read operations run inparallel on the same file.

– AFM cache may incorrectly read a file from the home cluster due to the incorrect calculationof the file sparseness information, potentially resulting in undetected data corruption.

– If parallel IO is enabled, AFM and AFM Asynchronous DR may experience undetected datacorruption with failover, resync and changeSecondary commands.

– AFM Asynchronous DR failback may read HSM migrated files from the acting AFMPrimary cluster (originally the AFM Secondary cluster) as sparse files, potentially resultingin the AFM cache to return incorrect data (all zeros) to an application on a read.

See the complete bulletin at either http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009244 or http://www-01.ibm.com/support/docview.wss?uid=isg3T1024249.

v Flash (Alert): IBM Spectrum Scale Installation Toolkit

Abstract:

IBM Spectrum Scale installation toolkit may fail due to a package conflict if EPEL repos areenabledProblem Summary:

You must disable all configured EPEL repositories on all nodes added in the spectrumscaleinstallation toolkit before proceeding with an install, deploy, or upgrade.See the Flash at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009275

v Flash (Alert): IBM Spectrum Scale (GPFS) V4 Asynchronous I/O write

Abstract:

IBM has identified an issue with IBM Spectrum Scale V4.1.0.4 through V4.1.1.7 and V4.2.0.0through V4.2.0.3 levels when asynchronous Direct I/O is used to write to a file on LINUX,using the io_submit interface.Problem Summary:

As a result of an asynchronous Direct I/O write using the io_ submit interface, a user file maycontain undetected data corruption via writing of stale data to disk.See the complete bulletin at either http://www-01.ibm.com/support/docview.wss?uid=ssg1S1007917 or http://www.ibm.com/support/docview.wss?uid=isg3T1023951

v Security Bulletin: Vulnerabilities in OpenStack affect IBM Spectrum Scale V4.2 and V4.1.1(CVE-2015-8466 and CVE-2016-0738)

Summary:

OpenStack vulnerabilities that could allow: - with OpenStack Swift 3, a remote attacker tolaunch a replay attack affects IBM Spectrum Scale (CVE-2015-8466) - with OpenStack Objectstorage(Swift), a remote authenticated attacker could exploit this vulnerability to consume allavailable proxy-server resources (CVE-2016-0738)See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005833

v Flash (Alert): IBM Spectrum Scale (GPFS) V4.2 and V4.1.1 AFM Async DR requirement forplanning

Abstract:

Our initial feedback from the field suggests that success of a disaster recovery solutiondepends on administration discipline, including careful design, configuration and testing.Considering this, IBM has decided to disable the Active File Management- based AsynchronousDisaster Recovery feature (AFM DR) by default and require that customers deploying the AFMDR feature first review their deployments with IBM Spectrum Scale development. You shouldcontact Spectrum Scale Support at [email protected] to have your use case reviewed. IBM willhelp optimize your tuning parameters and enable the feature. Please include this messagewhile contacting IBM Support.

155












See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005817v Flash (Alert): IBM Spectrum Scale (GPFS) V4.2 AFM cache reads an HSM migrated file from

home as a sparse file

Summary:

IBM has identified an issue with AFM in IBM Spectrum Scale V4.2.0.0 through 4.2.0.2 levelswhen the AFM cache reads the HSM migrated files from home (running IBM Spectrum ScaleV4.2 or later) as a sparse file. This issue may cause the cache to have undetected datacorruption and may return incorrect data (all zeros) to an application on read.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005766

v Security Bulletin: IBM Spectrum Scale, with the Spectrum Scale GUI installed, is affectedby a security vulnerability (CVE-2016-0361)

Summary:

A security vulnerability has been identified in the current levels of IBM Spectrum Scale V4.2.0.0thru V4.2.0.1, with the Spectrum Scale GUI installed, that could allow a remote unprivilegeduser to obtain sensitive information including ADMIN passwords used to access othercomponents of the system.See the complete bulletin at http://www.ibm.com/support/docview.wss?uid=ssg1S1005742

v Security Bulletin: Vulnerability in Samba affects IBM Spectrum Scale SMB protocol accessmethod (CVE-2016-2119)

Summary:

A Samba vulnerability which could allow a remote attacker to conduct spoofing attacks affectsIBM Spectrum Scale SMB protocol access method.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009255

v Security Bulletin: Multiple vulnerabilities in Samba – including Badlock - affect IBMSpectrum Scale SMB protocol access method

Summary:

Samba vulnerabilities were disclosed on April 12, 2016. Samba is used by IBM Spectrum ScaleSMB protocol access method. IBM Spectrum Scale has addressed the applicable CVEs includingthe vulnerability commonly referred to as “Badlock”.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005740

v Security Bulletin: IBM Spectrum Scale is affected by a security vulnerability(CVE-2016-0263)

Summary:

A security vulnerability has been identified in the current levels of IBM Spectrum Scale V4.2,V4.1 and IBM General Parallel File System V3.5, that could allow a local user, under specialcircumstances, to escalate their privileges or cause a denial of service when the mmapplypolicycommand is issued with certain options and syntax.See the complete bulletin at either http://www-01.ibm.com/support/docview.wss?uid=isg3T1023450 or http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005708

v Security Bulletin: Vulnerability in Samba affects IBM Spectrum Scale SMB protocol accessmethod (CVE-2015-5252)

Summary:

A Samba vulnerability which could allow a remote attacker to launch a symlink attack affectsIBM Spectrum Scale SMB protocol access method.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005689

v Security Bulletin: Vulnerability in IBM Java SDK affect IBM Spectrum Scale GUI(CVE-2015-7575)

Summary:

There is a vulnerability in IBM® SDK Java™ Technology Edition, Version 8 that is used by theIBM Spectrum Scale GUI.

156











See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=ssg1S1005690v Systemd is replacing traditional sysVinit in many Linux distributions. Though systemd should

still support traditional sysVinit without any changes, starting with systemd version 219-19,this support is not working properly, causing IBM Spectrum Scale services not to startup atboot time. If you are experiencing this problem, you can do one of the following:– Upgrade your nodes to IBM Spectrum Scale V4.2.0.1.

IBM Spectrum Scale uses systemd to start IBM Spectrum Scale services starting withV4.2.0.1.

– Apply the following workaround:rm /etc/init.d/gpfscp /usr/lpp/mmfs/bin/gpfsrunlevel /etc/init.d/gpfs

– If systemd is upgraded to version 219 after IBM Spectrum Scale V4.2.0.1 was alreadyinstalled, you can apply the workaround in the second option or take the following step toenable IBM Spectrum Scale services to use systemd:systemctl enable /usr/lpp/mmfs/lib/systemd/gpfs.service

v Abstract:

In the Advanced Edition of GPFS V4.1.0 or later, under certain circumstances, files may beincorrectly encrypted when the AES:ECB wrapping mode is used, possibly resulting inundetected data corruption of those files.Problem Summary:

File encryption (available only in the Advanced Edition of GPFS 4.1.0 or later releases) in GPFSis controlled by encryption policy rules, which determine which files are encrypted and withwhat keys and algorithms. At the time an encrypted file is created, a File Encryption Key (FEK)is generated and then encrypted ("wrapped") with the Master Encryption Key (MEK), and theresult is recorded as one of the file's extended attributes, along with the identity of the MEKand the encryption parameters needed to decrypt the FEK later on. The contents of the file areencrypted with the FEK being used as encryption key. A problem has been discovered in theCLiC toolkit, which is used by GPFS to perform basic encryption/decryption. The problemresults in the FEK being encrypted incorrectly before getting stored as one of the file's extendedattributes. The outcome is that an incorrect FEK may be produced upon opening the file.Though the file may still be read correctly on nodes of the same type as the one used when thefile was created, it may be read incorrectly from other nodes. In addition, if the file is writtenby one of those other nodes, the file may then be retrieved incorrectly from any of the nodes,since different portions of the file may have been written using different FEKs. The problemoccurs only when FIPS mode is disabled (the FIPS1402mode configuration variable is set to'no') and the AES:ECB wrapping mode is used on a Linux x86 node with AESNI encryptionacceleration. Note that AES:ECB is not the default wrapping mode and is not the one used ifthe default DEFAULTNISTSP800131A algorithm is configured.Users affected:

Users are affected only if all of the following conditions are true:– The file encryption capability (available only in the Advanced Edition of GPFS 4.1.0 or later

releases) is used, as described in the Encryption chapter in the Advanced AdministrationGuide publication at http://www.ibm.com/support/knowledgecenter/STXKQY/ibmspectrumscale_welcome.html

Note: Linux on Z nodes are not affected.– The AES:ECB wrapping mode is chosen when providing the encryption policy rules for a

file system. Note that AES:ECB is not the default wrapping mode and is not the one used ifthe default DEFAULTNISTSP800131A algorithm is configured.The following command (and its sample output) shows the existing policy for the filesystem. It will clearly indicate whether the affected wrapping mode is being used:mmlspolicy file system name -L

[...]

157




RULE ’F128’ ENCRYPTION SET ’F128’ALGO ’AES:128:CBC:FEK:HMACSHA512’COMBINE ’XOR’ WRAP ’AES:ECB’KEYS(’KEY-6aaa3451-6a0c-4f2e-9f30-d443ff2ac7db:RKMKMIP3’)

Note that ALGO 'DEFAULTNISTSP800131A' is a safe setting, since it corresponds to theAES:KWRAP wrapping mode.The following command (see sample output) will show the parameters used for a given file:mmlsattr -n gpfs.Encryption xyz2.enc

file name: xyz2.encgpfs.Encryption:"EAGC???N▒V]0????????????????D▒▒?▒▒;▒??????????????"▒_▒?▒46?Z? }??1?fooBar1?"EncPar ’AES:128:CBC:FEK:HMACSHA512’

type: wrapped FEK WrpPar ’AES:ECB’ CmbPar ’XOR’KEY-6aaa3451-6a0c-4f2e-9f30-d443ff2ac7db:RKMKMIP3

Note that changing the encryption policy does not affect the wrapping mode of existingfiles.

– The length of the file encryption key (as specified in the encryption parameter string) is setto a value longer than 128 bits (192 or 256 bits)

– The FIPS1402mode configuration parameter is set to no, which will result in the CLiCtoolkit being used for encryption/decryption.The FIPS mode in use can be displayed with the following command:mmlsconfig FIPS1402mode

– The cluster includes nodes which run on Linux x86 (any distro)– At least one node in the cluster supports the AESNI instruction. The following command

can be used to determine whether that CPU instruction is present:grep aes /proc/cpuinfo

If the command above produces some output then the AESNI instruction is present on thenode.

Required Actions:

GPFS 4.1 Advanced Encryption customers should avoid use of the AES:ECB wrapping mode.If the existing encryption policy rules specify AES:ECB then the encryption rules should bechanged (with mmchpolicy) to use a different wrapping mode. That will ensure that filescreated from that point on will not use the AES:ECB wrapping mode.The following applies to the users for which all the conditions under Users affected are true.It is important to act promptly if your cluster may be affected by the problem (if all conditionsunder Users Affected are true). The steps described below aim to minimize the chance of anyfurther data loss, as opposed to fixing any damage to the existing files. The existing files mayor may not be recoverable (and backup copies may have been damaged as well), depending ona complex set of circumstances.For the affected users, the following steps should be taken:1. Run the following command and save the output. This contains the current policy rules

(including any encryption rules) in place for the file system. The output should be providedto IBM Support.mmlspolicy file system name -L

2. The following steps above should prevent any new file from being created using AES:ECBas wrapping mode, and should avoid producing additional damaged files.a. Change the encryption policy and replace any instances of the wrapping parameter

string AES:ECB with wrapping parameter string AES:KWRAPb. Apply the resulting policy by running

mmchpolicy file system file containing the modified policy -I yes

158

3. Call IBM support as soon as possible for assistance with data recovery.

Note: In a cluster which may be affected by the problem, do not change the value of theFIPS1402mode variable . Changing that value (which will alter the toolkit used toencrypt/decrypt data) may cause file content to be incorrectly decrypted, since an incorrectFEK may be derived. In addition, writing to existing files -- but not reading from them -- maycause them to be damaged irreparably. For the very same reasons, also do not add nodes to thecluster which are affected by the problem differently than the original nodes in the cluster. Thatis, if all nodes in the cluster are affected (all nodes are Linux x86 with AESNI enabled) then donot add nodes which are not affected, or vice-versa: if the cluster is not affected then do notadd nodes which are affected.

v After applying the glibc fix provided by your OS distribution for CVE-2015-0235 - GHOST:glibc gethostbyname buffer overflow (http://www.openwall.com/lists/oss-security/2015/01/27/9), a rebuild of the gpl layer is not necessary.

v Abstract:

In Linux environments, GPFS may incorrectly fail writev() with EINVAL resulting in the userapplication failing during write.Problem Summary:

IBM has identified a problem with GPFS 3.5.0.20 and GPFS 4.1.0.2 where GPFS may fail tocorrectly handle multiple vectors passed via the writev() system call. When a {NULL, 0} ispassed as the first vector, an EINVAL error may be incorrectly returned. This would cause theuser application to fail unexpectedly when writev() is called to write to a GPFS file. User dataare not affected. The writev() call is most likely to have been automatically generated by thelibrary or compiler.Users Affected:

Only customers running the affected level on Linux and have applications which use thewritev() system call for writes to a GPFS file. Note: The writev() call is most likely to have beenautomatically generated by the library or compiler. For example, using C++ stream class towrite more than 1023 byte to a file will generate a writev() call that could fail with an EINVALerror.The following sample program shows an example for which a write using stream class mayfail unexpectedly:#include<cassert>#include<cstdio>#include>

int main (int argc, char** argv) {assert(argc == 2);

char* data = new char[1000000];std::ofstream f(argv[1], std::ios_base::binary);f.write(data, 1023); // this would succeedperror("write call");f.flush();

f.write(data, 1024); // this would failperror("write call");f.flush();

f.write(data, 1025); // this would failperror("write call");f.flush();

f.write(data, 1023); // this would succeedperror("write call");f.write(data, 1024); // this would succeedperror("write call");f.flush();

159

http://www.openwall.com/lists/oss-security/2015/01/27/9

http://www.openwall.com/lists/oss-security/2015/01/27/9

f.write(data, 1024); // this would failperror("write call");f.write(data, 1023); // this would succeedperror("write call");f.flush();

f.write(data, 512); // this would succeedperror("write call");f.write(data, 512); // this would succeedperror("write call");f.write(data, 1024); // this would succeedperror("write call");f.flush();

f.close();delete[] data;return 0;}

Recommendations:

Affected V3.5 customer should contact IBM Service for an efix containing APAR IV64863; thisfix is available in GPFS 3.5.0.21(APAR IV64863) or later. V4.1 customers should upgrade toGPFS 4.1.0.3 (APAR IV64862) or later at Fix Central http://www.ibm.com/eserver/support/fixes/

v Abstract:

GPFS on clusters enabled for RDMA use may experience server crashes, RDMA failures, hangs,or undetected data corruption.Problem Summary:

IBM has identified a problem with GPFS versions 3.5.0.17 efix18 and efix19, 3.5.0.19 and 4.1.0.2,for clusters enabled for GPFS RDMA when the value configured for verbsRdmasPerNode isless than the value configured for nsdMaxWorkerThreads for any NSD server. Under certainconditions, the NSD server thread may get indication that the RDMA completed successfullybefore the RDMA actually completes. This problem may result in NSD server crashes, RDMAfailures, hung NSD server threads, or undetected data corruption.Problem Description:

See Problem Summary.Users Affected:

Only customers running the affected levels, configured to use RDMA, with a value ofverbsRdmasPerNode that is less than the value configured for nsdMaxWorkerThreads for theNSD servers, are vulnerable to the problem.To verify if the NSD servers are vulnerable to the problem, run the following command oneach NSD server:mmfsadm test verbs config | grep -e "Max RDMAs per node"

In the examples below:– The value for "Max RDMAs per node max" corresponds to nsdMaxWorkerThreads.– The value for "Max RDMAs per node curr" corresponds to verbsRdmasPerNode (which may

be adjusted by GPFS).An example of an NSD server that is not vulnerable to the problem:

In this example, "Max RDMAs per node max" reports the same value (512) as"Max RDMAs per node curr" (512):

mmfsadm test verbs config | grep -e "Max RDMAs per node"Max RDMAs per node max : 512Max RDMAs per node curr : 512

An example of an NSD server that is vulnerable to the problem:

160



In this example, "Max RDMAs per node max" reports a value (512)that is greater than "Max RDMAs per node curr" (128):mmfsadm test verbs config | grep -e "Max RDMAs per node"

Max RDMAs per node max : 512Max RDMAs per node curr : 128

Recommendations:

IBM recommends that customers vulnerable to the problem immediately install an efixincluding IV63698; for the affected levels, the relevant efixes are:– If 3.5.0.17 efix18 or efix19 is installed, then install 3.5.0.17 efix20– If 3.5.0.19 is installed, then install 3.5.0.20 or later– If 4.1.0.2 is installed, then install 4.1.0.3 or laterCustomers vulnerable to the problem but unable to immediately apply the above fix levelsshould run the following command to change the value of verbsRdmasPerNode to equal thevalue of nsdMaxWorkerThreads for each NSD Server, as a workaround. The customer mayexperience performance impacts while this workaround is in effect.

In this example, N = the value configured for nsdMaxWorkerThreads.mmchconfig verbsRdmasPerNode=N

Customers vulnerable to the problem, after applying the service or workaround above, shouldcontact GPFS support for instructions to run mmfsck to detect and repair any metadatadamage.

v Abstract:

A recent change in the UEFI driver update for the SAS HBA can result in damage to any disksused for GPFS which previously contained a GPT partition table (due to non-GPFS use) but arenow assigned to GPFS, on upgrade.Content:

The UEFI firmware includes a function for Disk GPT Table Recovery. This function will restorethe GPT table from the backup GPT table which was stored at the end of the disk, and it is thedefault function. When a disk contains a backup GPT table but is later used as a GPFS NSD,the GPT Table Recovery action could rewrite GPFS NSD and Disk descriptor headers with thebackup GPT table. Thus such NSDs will be lost after the GPT Table Recovery action.Recommendation:

An nsdcheck script is available to run against NSD devices to determine if there is a validbackup GPT table on the device. An NSD disk is at risk if the remarks displayhasPrimaryGpt=no,hasSecondaryGpt=yes. If the backup table is not valid, the script can thenbe used to clear the backup GPT table on the NSD device, prior to any firmware updates, assoon as is possible.Running this script is recommended for all Linux customers, as a precaution. The script whenused to remove the secondary GPT will only remove it IF AND ONLY IF there is a GPTsignature on the last sector of the NSD device but not at the beginning.The script is available:– In the samples directory with GPFS V3.4.0.29 and V3.5.0.19 from FixCentral

- http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%2Bsoftware&product=ibm/power/IBM+General+Parallel+File+System&release=3.4.0&platform=All&function=all

- http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%2Bsoftware&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=All&function=all

– From the GPFS developerWorks® wiki at https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/GPFS%20Service%20Bulletins

161

http://www-933.ibm.com/support/fixcentral/swg/selectFixes?parent=Cluster%2Bsoftware&product=ibm/power/IBM+General+Parallel+File+System&release=3.4.0&platform=All&function=all






Note: If you need to restore the secondary GPT signature from GptBackupFile to disk, contactIBM Service. This should not be done without guidance.

v NFS access to an HSM enabled GPFS file system running on levels of Linux below RHEL 6.3and SLES 11 SP2, may hit the issue described here http://www-01.ibm.com/support/docview.wss?uid=ssg1S1004310

v Upgrading GPFS to a new major release on Linux:When migrating to a new major release of GPFS (for example, from GPFS 3.5 to GPFS 4.1), thesupported migration path is to install the GPFS base images for the new release, then applyany required service updates. GPFS will not work correctly if you use rpm -U command toupgrade directly to a service level of a new major release without installing the base imagesfirst. If this should happen you must uninstall and then reinstall the gpfs.base package.

RHEL specificv Some versions of glibc released as updates of RHEL6 on x86_64 change the behavior of the

memcpy routine: calling memcpy with overlapping range produces unexpected behavior. Whilethese changes do not violate any standards, they constitute a departure from previous behavior,and in doing so expose significant problems in GPFS code, leading to a variety of symptoms.The memcpy behavior change has turned out to be problematic in some other aspects, andRedHat has since released an updated version of glibc that restores the old memcpy behavior.For details, please see http://rhn.redhat.com/errata/RHSA-2012-0393.html.The list of affected glibc versions includes (but is not limited to):– glibc-2.12-1.47.el6– glibc-2.12-1.47.el6_2.5Version glibc-2.12-1.47.el6_2.9 and later do not have the problem. Be sure to not upgrade anynodes running any version of GPFS to a problematic version of glibc, and install a fixedversion if such an upgrade has already occurred.

v GPFS autoload may not work if /usr/lib/lsb/install_initd is not installed or if the commanddoes not work as expected, as we have found in some RHEL 6 platforms. The workaround isto run "/sbin/chkconfig --add gpfs" after installing or upgrading GPFS.

v If you get errors on RHEL5 when trying to run GPFS self-extractor archive from the installationmedia, please run export _POSIX2_VERSION=199209 first.

SLES specificv On SLES 11 SP2, the minimum kernel level supported is 3.0.42-0.7.3. With Linux 3.0 kernel

versions prior to this, under certain workloads (in particular when NFS export of GPFS filesystems is involved) a deadlock in kernel code may occur. This is fixed on kernel.org in 3.0.45.

v Due to a problem in system utilities shipped with SLES 11 SP2, tail -f of a file in a GPFS filesystem may not work. A temporary workaround is to use the undocumented command tail -f---disable-inotify. Newer versions should have this fixed. For details, please seehttp://lists.gnu.org/archive/html/bug-coreutils/2011-10/msg00080.html .

Q14.7: What are the current advisories for IBM Spectrum Scale on Windows?A14.7: For IBM Spectrum Scale V4.1, no hotfix updates are required. It is recommended that you:

v Install the latest OS Service Pack available from Microsoft and stay current with WindowsUpdates.

v Cygwin be updated periodically and stay relatively current.

Security Bulletin: GPFS V3.5 for Windows is affected by the following vulnerabilities:v Security Bulletin: Vulnerabilities in OpenSSH and OpenSSL affect GPFS for Windows V3.5

Summary:

OpenSSH vulnerabilities were disclosed on December 23, 2016 by the OpenSSH Project.OpenSSL vulnerabilities were disclosed on November 10, 2016 and January 26, 2017 by theOpenSSL Project. OpenSSH and OpenSSL are used by GPFS V3.5 for Windows. GPFS V3.5 forWindows has addressed the applicable CVEs.See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=isg3T1024968

v OpenSSH vulnerabilities were disclosed on July 22, August 10, and October 19, 2016 by theOpenSSH Project. OpenSSL vulnerabilities were disclosed on September 22 and 26, 2016 by the

162



http://rhn.redhat.com/errata/RHSA-2012-0393.html

http://lists.gnu.org/archive/html/bug-coreutils/2011-10/msg00080.html


OpenSSL Project. OpenSSH and OpenSSL are used by GPFS V3.5 for Windows. GPFS V3.5 forWindows has addressed the applicable CVEs. See the complete bulletin athttp://www-01.ibm.com/support/docview.wss?uid=isg3T1024394

v A Vulnerability in OpenSSH and Multiple Vulnerabilities in OpenSSL affect IBM GPFS V3.5 forWindows. OpenSSH could allow a remote authenticated attacker to execute arbitrarycommands on the system, caused by improper validation of user-supplied X11 authenticationcredentials by the sshd server. OpenSSL vulnerabilities were disclosed on May 3, 2016 by theOpenSSL Project. OpenSSL is used by GPFS V3.5 for Windows. IBM GPFS V3.5 for Windowshas addressed the applicable CVEs. See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=isg3T1023814

v OpenSSL vulnerabilities were disclosed on March 1, 2016 by the OpenSSL Project. OpenSSL isused by GPFS for Windows V3.5. GPFS for Windows V3.5 addressed the applicable CVEs withthe details provided in http://www-01.ibm.com/support/docview.wss?uid=isg3T1023478. Anadditional CVE, CVE-2016-2842, was also fixed but was not initially included in the March 1,2016 OpenSSL Project announcement or in the associated security bulletin for this product. Thisbulletin only addresses CVE-2016-2842. Please see the bulletin linked above for the other CVEsthat were addressed by the March 1, 2016 OpenSSL Project. See the complete bulletin athttp://www-01.ibm.com/support/docview.wss?uid=isg3T1023631

v Multiple vulnerabilities in OpenSSL affect IBM GPFS V3.5. OpenSSL vulnerabilities weredisclosed on December 3, 2015, January 28, 2016, and March 1, 2016 by the OpenSSL Project.OpenSSL is used by GPFS for Windows V3.5. GPFS for Windows V3.5 has addressed theapplicable CVEs including the “DROWN: Decrypting RSA with Obsolete and WeakenedeNcryption" vulnerability. See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=isg3T1023478

v Vulnerabilities in OpenSSH affect GPFS V3.5 for Windows (CVE-2016-0777, CVE-2016-0778). Aninformation leak flaw and buffer overflow flaw in the way the OpenSSH client roaming featurewas implemented affects OpenSSH for GPFS V3.5. See the complete bulletin athttp://www-01.ibm.com/support/docview.wss?uid=isg3T1023321

v OpenSSL vulnerabilities were disclosed by the OpenSSL Project and affect IBM GPFS V3.5 .This includes the Logjam Attack on TLS connections using the Diffie-Hellman (DH) keyexchange protocol (CVE-2015-4000). This also includes the alternate chains certificate forgeryvulnerability (CVE-2015-1793). IBM GPFS V3.5 has addressed the applicable CVEs.http://www-01.ibm.com/support/docview.wss?uid=isg3T1022527

v A vulnerability in RC4 stream cipher affects GPFS V3.5 for Windows (CVE-2015-2808) /Enabling weak cipher suites for IBM General Parallel File System is NOT recommended. TheRC4 "Bar Mitzvah" Attack for SSL/TLS affects OpenSSH for GPFS V3.5 for Windows.Additionally, with the recent attention to RC4 "Bar Mitzvah" Attack for SSL/TLS, this is areminder to NOT enable weak or export-level cipher suites for IBM General Parallel FileSystem (GPFS). See the complete bulletin at http://www-01.ibm.com/support/docview.wss?uid=isg3T1022137

v Vulnerabilities in Open Secure Shell for GPFS V3.5 on Windows (CVE-2014-2653,CVE-2014-2532) http://www-01.ibm.com/support/docview.wss?uid=isg3T1021316

v GPFS V3.5 for Windows is affected by OpenSSL vulnerabilities (CVE-2014-3512, CVE-2014-3509,CVE-2014-3506, CVE-2014-3507, CVE-2014-3511, CVE-2014-3505, CVE-2014-3510,CVE-2014-3508, CVE-2014-5139) http://www-01.ibm.com/support/docview.wss?uid=isg3T1021317

For GPFS V3.5 and earlier versions of GPFS, which require the SUA feature in Windows, theadvisories are as follows:v Required Windows hotfix updates for GPFS consist of:

– For Windows 7 Enterprise or Ultimate:- For GPFS V3.4.0.12, or later, and GPFS V3.5, KB article 2639164 at http://

support.microsoft.com/kb/2639164– For Windows Server 2008 R2 SP1:

- For GPFS V3.4.0.12, or later, and GPFS V3.5, KB article 2639164 at http://support.microsoft.com/kb/2639164

163















http://support.microsoft.com/kb/2639164




– For Windows Server 2008 R2:- KB article 978048 at http://support.microsoft.com/kb/978048.

– For Windows Server 2008:- There are currently no hotfix updates required for Windows Server 2008 SP2. All required

updates are contained in SP2. Ensure you are running with that level.Q14.8: What Linux kernel patches are provided for clustered file systems such as GPFS?A14.8: The Linux kernel patches provided for clustered file systems are expected to correct problems

that may be encountered when using GPFS with the Linux operating system. The suppliedpatches are currently being submitted to the Linux development community but may not beavailable in particular kernels. It is therefore suggested that they be appropriately applied basedon your kernel version and distribution.

A listing of the latest patches, along with a more complete description of these patches, can befound at the General Parallel File System project on SourceForge®.net at sourceforge.net/tracker/?atid=719124group_id=130828func=browse:1. Click on the Summary description for the desired patch.2. Scroll down to the Summary section on the patch page for a description of and the status of

the patch.3. To download a patch:

a. Scroll down to the Attached Files section.b. Click on the Download link for your distribution and kernel level.

Q14.9: Where can I find the IBM Spectrum Scale Software License Agreement?A14.9: Licensing information may be viewed at http://www.ibm.com/software/sla/sladb.nsf . To search

for a specific program license agreement:

To view an IBM Spectrum Scale V4 license, choose the corresponding product number to enter inthe search field:v For orders placed through the AAS fulfillment system, enter 5641-GPFv For orders placed through Passport Advantage, enter 5725-Q01

To view a GPFS V3.5 license, choose the corresponding product number to enter in the searchfield:v For GPFS V3.5 on POWER, enter 5765-G66v For GPFS on x86 hardware, enter 5724-N94. 5765-XA3, 5641-A07, 5641-B07, or 5641-C07

Q14.10:Where can I find End of Market (EOM) and End of Service (EOS) information?

A14.10:IBM Spectrum Scale and GPFS follow the Standard IBM Support Lifecycle Policy as described athttps://www.ibm.com/software/support/lifecycle/lc-policy.html which includes:v Provide a minimum of 3 years of product technical support beginning at the planned

availability date of the version/release of the product.v Ensure support is available for all IBM components of a product or until the product or bundle

is withdrawn from support. In addition, all components of a bundle have a common End ofService date.

v Publish a notice of support discontinuance ("End of Service") for a product at least 12 monthsprior to the effective date.

v Align the effective date of support discontinuance ("End of Service") to occur on common dateseither in the months of April or September.

v Make product support extensions available, where possible, that are designed to allowmigration to the current release to be completed. For additional information on producttechnical support extensions beyond the three-year minimum period, contact your IBMrepresentative. For details see the announcement letter USA Ann# 203-204 effective August 8,2003.

Current applicable announcement letter information includes:

GPFS V3.5 End of Market is April 15, 2016 and End of Service is April 30, 2017:

164




http://www-03.ibm.com/software/sla/sladb.nsf

https://www-01.ibm.com/software/support/lifecycle/lc-policy.html

Note: For additional languages, search on the PIDs notes below atv GPFS on Power (5765-G66) and GPFS for Linux on x86 Architecture (5765-XA3)

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=897&letternum=ENUS916-013

v GPFS for x86 Architecture (5724-N94) http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=897&letternum=ENUS916-014

GPFS V3.4 End of Market was April 18, 2014 and End of Service was April 30, 2015:

Note: For additional languages, search on the PIDs notes below atv GPFS on Power (5765-G66) http://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/

common/ssi/rep_ca/2/897/ENUS914-002/index.html&lang=en&request_locale=env GPFS on x86 Architecture (5724-N94) http://www-01.ibm.com/common/ssi/

ShowDoc.wss?docURL=/common/ssi/rep_ca/3/897/ENUS914-003/index.html&lang=en&request_locale=en

v GPFS on x86 Architecture (5641-A07, 5641-B07, 5641-C07) http://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/9/899/ENUSLP14-0009/index.html&lang=en&request_locale=en

GPFS V3.3 End of Service was April 30, 2013:v http://www-01.ibm.com/common/ssi/rep_ca/7/649/ENUSA12-0007/ENUSA12-0007.PDFv http://www-01.ibm.com/common/ssi/rep_ca/8/649/ENUSA12-0008/ENUSA12-0008.PDF

GPFS V3.2 End of Service was September 30, 2011:v http://www-01.ibm.com/common/ssi/rep_ca/3/897/ENUS910-243/ENUS910-243.PDFv http://www-01.ibm.com/common/ssi/rep_ca/0/897/ENUS910-210/ENUS910-210.PDF

Announced product EOM and EOS dates are available from the:v IBM Software Support Lifecycle page at http://www-01.ibm.com/software/support/lifecycle/

index_g.html.v IBM Spectrum Scale Sales Manual:

1. Go to http://www.ibm.com/common/ssi/index.wss2. Select your language preference and click Continue.3. In the Information Type, choose HW&SW Desc (Sales Manual,RPQ).4. To view a GPFS sales manual, choose the corresponding product number to enter in the

Search for field.For IBM Spectrum Scale V4:– For orders placed through the AAS fulfillment system, enter 5641-GPF.– For orders placed through Passport Advantage, enter 5725-Q01.– For IBM Spectrum Scale for Linux on Z, enter 5725-S28.For GPFS V3.5:– For GPFS V3.5 on POWER, enter 5765-G66.– For GPFS on x86 Architecture, enter 5724-N94, 5765-XA3, 5641-A07, 5641-B07, or

5641-C07.Q14.11:

Where can I locate IBM Spectrum Scale code to upgrade from my current level?A14.11:

If you have Software Maintenance Agreement (SWMA) for your products ordered throughAAS/eConfig or IBM Subscription and Support (S&S) for orders placed through PassportAdvantage, you may log into the respective systems and upgrade your level of IBM SpectrumScale:v For products ordered through AAS/eConfig, please log into the Entitled Software page at:

https://www-05.ibm.com/servers/eserver/ess/OpenServlet.wssv For products ordered through Passport Advantage, please log into the site at:

http://www.ibm.com/software/lotus/passportadvantage/Q14.12:

Are there any items that will no longer be supported?

165





http://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/2/897/ENUS914-002/index.html&lang=en&request_locale=en





http://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/9/899/ENUSLP14-0009/index.html&lang=en&request_locale=en



http://www-01.ibm.com/common/ssi/rep_ca/7/649/ENUSA12-0007/ENUSA12-0007.PDF

http://www-01.ibm.com/common/ssi/rep_ca/8/649/ENUSA12-0008/ENUSA12-0008.PDF



http://www-01.ibm.com/software/support/lifecycle/index_g.html

http://www-01.ibm.com/software/support/lifecycle/index_g.html




A14.12:GPFS V3.5 is the last release to support.v RHEL 5v SLES 10

GPFS V3.4 was the last release to support:v Itanium Servers through a special Programming Request for Price Quotation (PRPQ).v 32-bit kernelsv AIX V5.3

GPFS V3.3 was the last release to support:v IBM Virtual Shared Disk

With GPFS V3.3 and later, new file systems must be created utilizing network shared disksonly. IBM Virtual Shared Disks are not supported on new file systems. Please see the GPFS:Concepts, Planning, and Installation Guide for details on creating Network Shared Disks.

v Data shipping modev 32-bit AIX kernelsv The High Performance Switch

166



Scale Management Server (REST API) questionsQ15.1: Does IBM Spectrum Scale support RESTful APIs for file system management?A15.1: In v4.2.2 and later, IBM Spectrum Scale supports a REST API for configuring, managing, and

monitoring various components of IBM Spectrum Scale system. The IBM Spectrum Scale RESTAPI is an HTTP programming interface for performing IBM Spectrum Scale management tasks.With the REST API, you can automate storage management operations and integrate IBMSpectrum Scale capabilities into your applications. The APIs are installed on the GUI stack of theIBM Spectrum Scale cluster. The GUI installation and setup takes care of the API installation. Youdo not need to perform any additional steps to set up APIs. Clients communicate using HTTPSprotocol and JSON syntax is used to frame data inside HTTP requests and responses.

Q15.2: What are the changes made to the API implementation in the 4.2.3 release?A15.2: The API Version 1 was introduced with the IBM Spectrum Scale 4.2.2 release. The implementation

was based on Python and the deployment was limited only to the manager nodes that run onRHEL7. The API Version 2 is introduced in the 4.2.3 release. The current implementation is basedon GUI stack. That is, GUI server manages and processes the API requests and commands.Version 2 has the following features:v Reuses the GUI deployment's backend infrastructure, which makes introduction of new API

commands easier.v Uses the same role-based access feature that is available to authenticate and authorize the GUI

users. No additional configuration is required for the API users.v Makes deployment easier as the GUI installation takes care of the basic deployment.v Supports filtering of objects and paging if several thousand objects are retrieved.v Highly scalable and can support large clusters with thousands of nodes.v The APIs are driven by the same lightweight WebSphere Liberty server and object cache that is

used by the IBM Spectrum Scale GUI.

Note: Although the REST API delivered with IBM Spectrum Scale V4.2.3 still supports version 1requests, it is highly recommended that you switch to REST API version 2 requests at yourearliest convenience since version 1 is deprecated and will not be enhanced.

167

File audit logging questionsQ16.1: Which IBM Spectrum Scale editions is file audit logging available in?A16.1: File audit logging is available in the IBM Spectrum Scale Advanced Edition and Data

Management Edition only.Q16.2: Which platforms is file audit logging supported on?A16.2: File audit logging can be enabled on clusters with nodes that are running RHEL 7 on x86, RHEL

7 on Power8 Little Endian, or Ubuntu 16.04/16.04.01 on x86.

Note:v RHEL 7 Linux Kernel must be greater than or equal to 3.10.0-0123.v Clusters with mixed operating systems can still enable file audit logging; however, nodes that

do not meet the previously listed requirements cannot participate in auditing. Any IOgenerated from those nodes is not recorded.

Q16.3: How many nodes are needed to enable file audit logging?A16.3: File audit logging can be enabled on clusters that have at least three Linux quorum nodes

defined. At least three nodes need to be available to act as the message queue servers (alsoknown as broker nodes). These can be the same quorum nodes, but they do not have to be.

Q16.4: What are the requirements and limitations for file audit logging?A16.4: For more information, see Requirements and limitations for file audit logging in the IBM Spectrum

Scale: Concepts, Planning, and Installation Guide.

168

https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1ins_adlgreqlim.htm

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries.Consult your local IBM representative for information on the products and services currently available inyour area. Any reference to an IBM product, program, or service is not intended to state or imply thatonly IBM's product, program, or service may be used. Any functionally equivalent product, program, orservice that does not infringe any of IBM's intellectual property rights may be used instead. However, itis the user's responsibility to evaluate and verify the operation of any non-IBM product, program, orservice.

IBM may have patents or pending patent applications covering subject matter described in thisdocument. The furnishing of this document does not grant you any license to these patents. You can sendlicense inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10594-1785USA

For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual PropertyDepartment in your country or send inquiries, in writing, to:

IBM World Trade Asia CorporationLicensing2-31 Roppongi 3-chome, Minato-kuTokyo 106-0032, Japan

The following paragraph does not apply to the United Kingdom or any other country where suchprovisions are inconsistent with local law:

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION “AS IS”WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOTLIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY ORFITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or impliedwarranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodicallymade to the information herein; these changes will be incorporated in new editions of the publication.IBM may make improvements and/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not inany manner serve as an endorsement of those Web sites. The materials at those Web sites are not part ofthe materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate withoutincurring any obligation to you.

Licensees of this program who wish to have information about it for the purpose of enabling: (i) theexchange of information between independently created programs and other programs (including thisone) and (ii) the mutual use of the information which has been exchanged, should contact:

169

IBM CorporationIntellectual Property Law2455 South Road,P386Poughkeepsie, NY 12601-5400USA

Such information may be available, subject to appropriate terms and conditions, including in some cases,payment of a fee.

The licensed program described in this document and all licensed material available for it are providedby IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement orany equivalent agreement between us.

Any performance data contained herein was determined in a controlled environment. Therefore, theresults obtained in other operating environments may vary significantly. Some measurements may havebeen made on development-level systems and there is no guarantee that these measurements will be thesame on generally available systems. Furthermore, some measurements may have been estimated throughextrapolation. Actual results may vary. Users of this document should verify the applicable data for theirspecific environment.

This information contains examples of data and reports used in daily business operations. To illustratethem as completely as possible, the examples include the names of individuals, companies, brands, andproducts. All of these names are fictitious and any similarity to the names and addresses used by anactual business enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustratesprogramming techniques on various operating platforms. You may copy, modify, and distribute thesesample programs in any form without payment to IBM, for the purposes of developing, using, marketingor distributing application programs conforming to the application programming interface for theoperating platform for which the sample programs are written. These examples have not been thoroughlytested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, orfunction of these programs.

If you are viewing this information softcopy, the photographs and color illustrations may not appear.

TrademarksIBM, the IBM logo, and ibm.com® are trademarks or registered trademarks of International BusinessMachines Corporation in the United States, other countries, or both. If these and other IBM trademarkedterms are marked on their first occurrence in this information with a trademark symbol ( ® or ™), thesesymbols indicate U.S. registered or common law trademarks owned by IBM at the time this informationwas published. Such trademarks may also be registered or common law trademarks in other countries. Acurrent list of IBM trademarks is available on the Web at Copyright and trademark information atwww.ibm.com/legal/copytrade.shtml

Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, othercountries, or both and is used under license therefrom

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon,Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or itssubsidiaries in the United States and other countries.

Java™ and all Java-based trademarks and logos are trademarks or registered trademarks of SunMicrosystems, Inc. in the United States, other countries, or both.

170

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Red Hat, the Red Hat "Shadow Man" logo, and all Red Hat-based trademarks and logos are trademarksor registered trademarks of Red Hat, Inc., in the United States and other countries.

UNIX is a registered trademark of the Open Group in the United States and other countries.

Microsoft, Windows, Windows NT, and the Windows logo are registered trademarks of MicrosoftCorporation in the United States, other countries, or both.

Other company, product, and service names may be the trademarks or service marks of others.

June 2018

© Copyright IBM Corporation 2004, 2018.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Date post:	20-May-2018
Category:	Documents
Upload:	nguyentuyen
View:	231 times
Download:	4 times

2.3 What is the IBM Spectr um Scale support position r ... · IBM Spectrum Scale ™ Frequently...

Documents