Deploying DB2 pureScale with EMC Symmetrix...

Deploying the DB2 pureScale Feature

with EMC Symmetrix VMAX

Aruna De Silva (IBM) Testing and solutions specialist, DB2 Quality Assurance

Aslam Nomani (IBM) Manager, DB2 Quality Assurance

Roger E. Sanders (EMC) Senior Consultant Corporate Systems Engineer

Kevin Richards (EMC) Consultant Systems Integration Engineer

IBM

EMC Corporation and IBM Corporation

Deploying the DB2 pureScale Feature with EMC Symmetrix VMAX 2

Last updated 2012-08-31



Table of contents

Executive summary.............................................................................................. 5

Introduction .......................................................................................................... 6

DB2 pureScale Feature exploitation of storage.................................................... 8

DB2 cluster services tiebreaker disk............................................................... 8

Ultra-fast failover using SCSI-3 persistent reserve ......................................... 9

Device drivers and storage protocols.............................................................. 9

Multipath I/O drivers ............................................................................................................9 Supported storage subsystems.......................................................................... 11

Configuring EMC Symmetrix VMAX for pureScale storage .......................... 11

Validation matrix ........................................................................................... 12

Advantages of using the DB2 pureScale Feature .............................................. 12

Addition and removal of members ................................................................ 13

Automatic workload balancing ...................................................................... 13

Stealth maintenance ..................................................................................... 14

Continuous availability .................................................................................. 14

Disaster recovery.......................................................................................... 15

Deploying the DB2 pureScale Feature on an EMC Symmetrix VMAX storage array ...................................................................................................... 16

Validating the configuration of a DB2 pureScale deployment ....................... 18

Configuring multipath I/O with EMC Symmetrix VMAX in a pureScale environment ....................................................................................................... 20

Configuring AIX MPIO for VMAX .................................................................. 20

Introduction ........................................................................................................................20 Configuring AIX MPIO to support EMC Symmetrix devices in pureScale configurations ....................................................................................................................21 A note on AIX MPIO device attribute requirements ..........................................................24

Configuring the Device Mapper Multipath module on Linux operating systems ................................................................................................... 26

Introduction ........................................................................................................................26 The DM-MP configuration file ............................................................................................27 Setting up DM-MP on the pureScale cluster hosts ...........................................................28 DM-MP configuration file examples...................................................................................29 Applying DM-MP configuration file changes......................................................................32

Enabling and testing SCSI-3 PR with Symmetrix EMC VMAX storage controllers........................................................................................................... 34



Introduction ................................................................................................... 34

Configuration overview ................................................................................. 35

Storage and operating system requirements for supporting SCSI-3 PR....... 36

Enabling SCSI-3 PR on the EMC Symmetrix VMAX controller..................... 36

Using the Symmetrix service processor ............................................................................36 Using the SYMCLI .............................................................................................................37 Enabling the SCSI-3 PR flag on the required Symmetrix hyper volumes .........................44

Validating that the storage subsystem meets pureScale SCSI-3 PR requirements ........................................................................................... 45

Checking SCSI-3 PR device report capabilities on Linux pureScale clusters...................45 Checking SCSI-3 PR device report capabilities on AIX pureScale clusters .....................46

Enabling SCSI-3 PR on the pureScale cluster configuration ........................ 46

Verifying that SCSI-3 PR is enabled in the pureScale cluster configuration.....................46 SCSI-3 PR key management in a DB2 pureScale cluster ................................................47

Conclusion ......................................................................................................... 49

Appendix A. Storage provisioning in Symmetrix VMAX with Enginuity code 5874 and higher ................................................................................................. 50

Auto-provisioning example: provisioning a shared disk to a pureScale Linux test cluster...................................................................................... 51

Examples of querying the Auto-provision Groups and the masking views ....................................................................................................... 55

Creating Symmetrix hyper devices ............................................................... 57

Appendix B. List of references for more information .......................................... 60

Acknowledgements ............................................................................................ 61

Notices ............................................................................................................... 62

Trademarks................................................................................................... 63



Executive summary

In today’s highly competitive marketplace, it is important to deploy a data processing

architecture that not only meets your immediate tactical needs but also can grow and

change to adapt to your future strategic requirements. In December 2009, IBM®

introduced the DB2® pureScale® Feature for Enterprise Server Edition software (DB2

pureScale Feature). The DB2 pureScale Feature builds on familiar and proven design

features from the IBM DB2 for z/OS® database software (DB2 for z/OS), bringing the

industry-leading technology and reliability of DB2 for z/OS to the open systems platform.

The DB2 pureScale Feature helps to meet the availability and capacity needs of your

organization by providing:

• Extreme capacity. The ability to scale out your system by easily adding additional

server systems to your cluster.

• Application transparency. The ability to use your existing applications with

minimal or no changes.

• Continuous availability. The inherent redundancy that an active-active

architecture provides.

• Reduced total cost of ownership (TCO). Savings through simplified deployment

and management of advanced technology.

A key aspect of the DB2 pureScale solution is related to the capabilities of the underlying

storage. The storage must not only provide best-of-breed performance but must also

possess best-of-breed availability characteristics. In addition, the DB2 pureScale Feature

takes advantage of specific storage capabilities to enhance the solution. This paper takes

you through the value proposition of deploying DB2 pureScale databases on EMC®

Symmetrix® VMAX® storage arrays, demonstrating a solution that meets the needs of

even the most demanding businesses.

The steps that are outlined in the paper apply to both DB2 9.8 with Fix Pack 4 and IBM

DB2 Version 10.1 for Linux®, UNIX®, and Windows® (DB2 10.1).



Introduction

The DB2 pureScale Feature provides an active-active, shared-disk database

implementation that leverages proven technology from DB2 for z/OS data-sharing

architecture. In addition, the DB2 pureScale Feature offers the following key benefits:

Extreme capacity. The DB2 pureScale Feature provides practically unlimited capacity by

allowing for the addition and removal of members on demand. It can scale to 128

members and has a highly efficient, centralized management facility that allows for

superior scale-out compared to peer-to-peer models. The DB2 pureScale Feature also uses

a technology called Remote Direct Memory Access (RDMA), which provides highly

efficient inter-node communication that also facilitates superior scaling.

Application transparency. An application that runs in a DB2 pureScale environment

does not need to have any knowledge of the different members in the cluster nor be

concerned about partitioning data. The DB2 pureScale Feature automatically routes

applications to the members that it deems the most appropriate. It also provides native

support for a great deal of syntax that is used by non-IBM database vendors. Therefore,

applications that you wrote for non-IBM databases can run in a DB2 pureScale

environment with minimal or no changes.

Continuous availability. With the fully active-active configuration, if one member goes

down, processing can continue at the remaining active members. During a failure, only

data that is being modified on the failing member is temporarily unavailable and only until

database recovery is completed for that set of data, which happens very quickly. This

behavior is in direct contrast to some competing solutions, where an entire system freeze

can occur as part of the database recovery process.

Reduced TCO. You can easily deploy the DB2 pureScale Feature. The DB2 pureScale

Feature interfaces handle the deployment and maintenance of components that are

integrated within the DB2 pureScale Feature. This reduces the steep learning curves that

are associated with some of the competing technologies.

Symmetrix VMAX, the newest member of the Symmetrix family from EMC Corporation,

is a revolutionary new storage system that meets virtual data center requirements. Based

on the Virtual Matrix Architecture and new Enginuity™ capabilities, Symmetrix VMAX

scales performance and capacity, delivers non-disruptive operations, and greatly simplifies

and automates the management and protection of information. A basic overview of

Symmetrix VMAX specifications is shown in Figure 1.



Figure 1. The Symmetrix VMAX platform

The Symmetrix VMAX design is based on individual high-availability engines with

redundant CPU, memory, and connectivity on two directors for fault tolerance. VMAX

engines connect to and scale out linearly through the Virtual Matrix Architecture, which

allows resources to be shared within and across VMAX engines. To meet growth

requirements, you can add VMAX engines non-disruptively for efficient and dynamic

scaling of capacity and performance that is available to any application on demand.

� 1 - 8 redundant VMAX engines

� Up to 2.1 PB usable capacity

� Up to 128 FC FE ports

� Up to 64 FICON FE ports

� Up to 64 GigE or iSCSI FE ports

� Up to 1 TB global memory (512 GB usable)

� 48 - 2400 drives

� 200/400 GB enterprise flash drives

� 146/300/450 GB, 15k RPM FC drives

� 300/450/600 GB, 10k RPM FC drives

� 1 TB, 7.2k RPM SATA drives



DB2 pureScale Feature exploitation of storage

The DB2 pureScale Feature uses capabilities that are provided by the storage to enhance

availability. The two key items that the feature exploits are as follows:

• The ability to create a DB2 cluster services tiebreaker disk

• The ability to provide ultra-fast failover and recovery by fencing virtually

instantaneously any host that is deemed not healthy

DB2 cluster services tiebreaker disk

With the DB2 pureScale Feature, if sets of machines are partitioned from others because

of events such as network errors, the group of machines that is greater than half of the

cluster is deemed the healthy side. The machines on the healthy side stay online, while the

group of machines that is less than or equal to half the cluster goes offline. The DB2

pureScale Feature implements this technique to ensure data integrity—to protect against

the case where two disjoint portions of the cluster proceed on the assumption that the

other side is offline.

However, a mechanism is required to ensure that at least one cluster partition can host

resources when there is an even-split. For example, consider a case where there are four

machines in a pureScale configuration. Assume that the machines have a network partition

such that the first two machines can talk only to each other and the remaining two

machines can talk only to each other. In this case, neither side of the cluster constitutes

greater than half the machines; therefore, both sides must go offline unless there is a disk

tiebreaker.

The DB2 installer automatically configures the tiebreaker disk technology if it is

supported within the environment. The disk tiebreaker is a disk resource that acts as an

arbitrator in the cluster. Each disjoint side of the cluster tries to acquire the disk

tiebreaker; however, by definition, only one side can acquire the disk tiebreaker at any

point in time. The side that acquires the disk tiebreaker is considered to have a majority

portion of the cluster and thus is deemed to be the portion of the cluster that remains

online. Therefore, with a disk tiebreaker resource, half the machines in the pureScale

cluster can fail or be separated, yet the pureScale cluster can safely stay online to handle

database requests.



Ultra-fast failover using SCSI-3 persistent reserve

The DB2 pureScale Feature supports SCSI-3 persistent reserve [type code 0x7]

technology (SCSI-3 PR), to allow for very fast eviction and fencing of storage access for

failed nodes in a cluster. By ensuring that failed nodes are fenced from the cluster, the

DB2 pureScale Feature help provide the highest level of data integrity. If a failed node

cannot access the storage, the failed node cannot cause corruption.

A key aspect of a reliable clustering solution is its ability to fence failed nodes and to do it

in a timely manner. By using SCSI-3 PR technology, the DB2 pureScale Feature can

fence failed members in just a few seconds, whereas other technologies can take 60

seconds or more. DB2 automatically enables SCSI-3 PR technology if it is supported in

the environment.

Device drivers and storage protocols

When shared disks in a DB2 pureScale configuration are backed by an EMC Symmetrix

VMAX storage controller, the multipath I/O solutions that you can use are AIX® Multi-

Path I/O (MPIO), Linux Device Mapper Multipath I/O (DM-MP), and EMC PowerPath.

You can use these multipath I/O solutions with either Fibre Channel or Fibre Channel

over Ethernet (FCoE) protocols.

Multipath I/O drivers

AIX MPIO

AIX MPIO is part of the base kernel that provides native support for multipath Fibre

Channel storage attachment. MPIO automatically discovers, configures, and makes

available every storage device path. The storage device paths are managed to provide high

availability and load balancing of storage I/O.

For AIX pureScale clusters using EMC Symmetrix VMAX hyper volumes, you must

install the EMC Symmetrix FCP MPIO file set (EMC.Symmetrix.fcp.MPIO.rte) on each

node in the cluster. This AIX Object Data Manager (ODM) fileset contains all the ODM

attributes and PCM components that are required to support SCSI-3 PR and MPIO with

EMC Symmetrix Storage arrays.

In addition, any EMC hyper devices that are detected on the host must support the setting

of the following ODM device attributes:

• For the device that you use as the pureScale tiebreaker disk:

o algorithm = fail_over

o reserve_policy = single_path

• For devices that you will use for pureScale cluster file systems, such as instance

home, data, and log file systems:

o reserve_policy = no_reserve



PR_shared

Important: The DB2 pureScale Feature initially expects all devices that you use

for cluster file systems to have a reserve_policy=no_reserve setting.

Linux DM-MP

The Device Mapper Multipath module provides multipath I/O capability for hosts running

Linux operating systems. DM-MP is the default MPIO solution for key Linux server

platforms.

For Linux pureScale clusters using EMC Symmetrix VMAX hyper volumes, you should

use the following settings in the Linux DM-MP configuration:

• polling_interval=5

Specifies the interval, in seconds, between two path checks.

For correctly functioning paths, the interval between checks

gradually increases to 4 * the value of the polling_interval setting.

• no_path_retry=5

Specifies the number of times that DM-MP attempts to use a failed path before

disabling queuing.

For detailed configuration steps for AIX MPIO and Linux DM-MP, refer to the

instructions provided in the section “Configuring multipath I/O with EMC Symmetrix

VMAX in a pureScale environment”.

EMC PowerPath

PowerPath® combines multipath I/O capabilities, automatic load balancing, and path-

failover functions into one integrated offering that works across heterogeneous server and

storage environments. For enhanced control across the data center, PowerPath provides

the following features:

• Load balancing. PowerPath uses sophisticated algorithms to boost application-I/O

rates for EMC and non-EMC arrays with automatic data-path load balancing,

allowing for the greater efficiency and throughput.

• Proactive path management. Automated load balancing applies a number of

methods to test and manage active and inactive data paths and detects path failures

that might threaten application availability. Dynamic enablement of inactive or

failed paths is re-enabled, and paths are put back into use after they regain

acceptable performance levels.

• Application availability and performance optimization. PowerPath ensures

continuous data access while optimizing server and path usage by protecting the

I/O path and applications in the event of a path failure.



The EMC PowerPath family includes the following members:

• PowerPath multipathing for Windows, Linux, Solaris, HP-UX, and AIX physical

environments and for Linux, AIX, and Solaris virtual environments

• PowerPath/VE multipathing for VMware vSphere and Microsoft Hyper-V virtual

environments

EMC PowerPath version 5.5 P04 B003 when used with EMC Enginuity version

5876.82.57 or higher on the EMC Symmetrix VMAX supports DB2 pureScale fast I/O

fencing capabilities – thus, can be used instead of AIX MPIO as the multipath I/O solution

for AIX pureScale clusters.

For hdiskpower devices that you will use for pureScale cluster file systems, such as

instance home, data, and log file systems: • reserve_policy = PR_shared

Reserve policy setting after the cluster file systems are created, i.e when SCSI-3

Persistent Reserve is enabled.

• reserve_policy = no_reserve

Initial reserve policy setting for all hdiskpower devices that will be used for cluster

file systems.

Please refer to the section on GPFS clusters in EMC Host Connectivity Guide for IBM

AIX (P/N 300-000-608 REV A42) for details on how to configure EMC PowerPath for

DB2 pureScale. This AIX host connectivity guide can be accessed from EMC Powerlink®

site under Support > Technical Documentation and Advisories > Host Connectivity/HBAs

> Installation/Configuration.

Supported storage subsystems

The DB2 pureScale Feature supports a wide range of storage arrays, including the IBM

DS3000, DS5000, DS8000, v7000, and XIV lines of storage. The feature also supports

storage that is provided by vendors such as EMC, Netapp, and Hitachi. To use the DB2

pureScale rapid fencing that is required for ultra-fast failover times, you should validate

the disk storage supports SCSI-3 PR with the underlying clustered file system.

Configuring EMC Symmetrix VMAX for pureScale storage

Provisioning storage from an EMC VMAX to a pureScale cluster so that it can use fast

I/O failure capabilities involves performing the following steps:

1. Create required storage devices on the EMC Symmetrix VMAX array. Typically a

1 GB hyper volume is sufficient for the tiebreaker disk. You can create the rest of

the devices based on the needs of your organization.



2. Enable SCSI-3 flag (SC3) for all the hosts in the cluster. You can perform this step

at one of two levels:

• At a global level by turning on SC3 flags on the frontend adapter (FA)

ports that are connected to the SAN fabric on the VMAX

• For each host initiator in the pureScale cluster that is seen by the VMAX

The key differences between the two approaches are as follows:

• The first approach turns on SCSI-3 support for all hosts that are zoned

through the FA ports. However, those FA ports must be first brought

offline before enabling SCSI-3 support.

• The second approach is a fully non-disruptive method and affects only the

World Wide Port Name (WWPN) s of the fibre channel initiators in the

pureScale cluster.

3. On the VMAX, enable SCSI-3 PR flags for each Symmetrix hyper volume that

you are provisioning as a shared disk to the pureScale cluster.

Detailed instructions on how to configure EMC VMAX are provided in the section

“Enabling and testing SCSI-3 PR with Symmetrix EMC VMAX storage

controllers”.

Validation matrix

For a listing of the storage subsystems that have been validated with SCSI-3 PR, go to

Cluster Products Information Center

(http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp) and search for GPFS

faqs.

For a complete list of storage configurations that have been validated with the DB2

pureScale Feature, go to DB2 Version 10.1 Information Center

(http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/index.jsp) and search for Shared storage

considerations for DB2 pureScale environments.

Advantages of using the DB2 pureScale Feature

As pointed out in the “Introduction” section, the DB2 pureScale Feature offers many

advantages. The following sections contain common use cases that further demonstrate

the added value.

All sample DB2 commands in this section are based on DB2 9.8 Fix Pack 4. The syntax

might be different in future releases. If you are using a different DB2 release, see the DB2

Information Center for that release for the applicable command syntax.



Addition and removal of members

With the DB2 pureScale Feature, you can add members to the configuration quickly and

without requiring any type of data redistribution. The DB2 installation binary files are

automatically stored on the installation-initiating host (IIH); you do not need the

installation media when you add members. You can add a member by running the

db2iupdt command, as shown in the following example:

db2iupdt –add –m ServerX:ServerX-ib0 db2sdin1

Similarly, db2iupdt –drop –m command issued from a node that will still belong to the

cluster, can be used to drop a member. In addition, you can also start or quiesce members

transparently to applications.

Automatic workload balancing

With the DB2 pureScale Feature, you can dynamically distribute a workload across all

active members, based on the utilization levels of the different machines. Multithreaded

CLI applications, by default, use connection-level workload balancing without requiring

any changes. You can change the connection-level workload balancing to transaction-

level workload balancing. For multi-threaded Java applications, you can use the following

setting in the connection string to take advantage of transaction-level workload balancing:

enableSysplexWLB=true

For detailed information about the JCC configuration parameters and additional ways that

you can tune workload balancing, go to DB2 Version 10.1 Information Center

(http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/index.jsp) and search for Workload

balancing support for Java clients.

As additional members are started, clients are automatically routed to the new members

without any interruption of service. Also, you can stop members as described in the

instructions in the “Stealth maintenance” section of this paper without users and

applications knowing that this operation occurred.

You can also configure clients to have a preference for which member they should

connect to. This feature is referred to as client affinity and can be beneficial if a partitioned

workload exists. Java applications can take advantage of client affinity by setting the

following JCC configuration attribute in the connection string:

enableClientAffinitiesList=DB2BaseDataSource.YES

For details on all the JCC settings that are required to enable client affinity in Java

applications, go to DB2 Version 10.1 Information Center and search for Configuration of

client affinities for Java clients for DB2.

Prerequisite: To take advantage of DB2 pureScale features such as transaction-level



workload balancing or client affinity, the minimum DB2 client level is DB2 9.7 Fix Pack

3 or the corresponding JCC driver level. To determine JCC level packaged with a given

DB2 fix pack, see

http://www-01.ibm.com/support/docview.wss?rs=71&uid=swg21363866.

Stealth maintenance

In many cases, it is critical to apply maintenance to a system, but you don’t want to

negatively affect users and applications. Stealth maintenance allows all transactions on a

member to be completed or drained and then transparently routes connected applications

to another member. For example, to drain member 1, you can run the following command,

which lets the applications finish running before the member is stopped:

db2stop member 1 quiesce

You might encounter situations where a user session for a UOW was started but was not

committed or rolled back. In such situations, the db2stop quiesce command must wait

for that UOW to be completed before stopping that member unless you specify a timeout

value. For example, you can specify a timeout value of 10 minutes, which gives an

application 10 minutes to complete the UOW. If after 10 minutes the UOW is not

completed, DB2 automatically forces off the application. Any applications that are

completed within 10 minutes are transparently and automatically rerouted to active

members when they complete their UOWs.

To drain member 1 with a 10-minute timeout, you can run the following command:

db2stop member 1 quiesce 10

Continuous availability

One of the significant value propositions of the DB2 pureScale Feature is the continuous

availability characteristics that are inherently integrated into the architecture. All

necessary resources are automatically monitored by the DB2 pureScale cluster services

and are restarted as needed. Applications that are connected to a failing member are

automatically rerouted to an active member where the application can reissue any failed

transactions. Applications connected to a non-failing component are not affected beyond

what might be seen as a short blip in performance.

One of the distinguishing factors of the DB2 pureScale Feature as compared to competing

technologies is that if a node fails, no cluster-wide freeze occurs. In fact, only data in the

process of being updated on the failing member is temporarily unavailable, and only until

recovery is completed. The recovery is completed in tens of seconds so that data

availability during a member failure looks similar to that shown in the following graph:



Figure 2. Impact on data availability during member failures

In addition, the DB2 pureScale Feature supports multiple cluster interconnect network

adapters called host channel adapters (HCAs) on the cluster caching facilities (CFs).

Having multiple HCAs on InfiniBand networks that are connected to multiple switches

enables the DB2 pureScale Feature to deliver increased throughput while having a higher

resiliency to hardware failures.

Disaster recovery

The DB2 pureScale Feature offers a local high availability solution, but you might require

a disaster recovery solution to meet your business continuity requirements. The DB2

pureScale Feature can use remote disk mirroring technology (such as EMC’s SRDF®

technology) and also works with database replication products. These products include

IBM InfoSphere® Replication Server and the IBM Homogeneous Replication Feature for

DB2 9.7 Linux, UNIX, and Windows, Enterprise Server Edition. If the entire primary site

that is running a DB2 pureScale instance fails, you can use the remote site to allow

business operations to continue. The DB2 pureScale Feature can also use the traditional

database backup, restore, and rollforward functionality to facilitate disaster recovery.



Deploying the DB2 pureScale Feature on an EMC Symmetrix VMAX storage array

You can deploy the DB2 pureScale Feature on shared disks from an EMC Symmetrix

VMAX storage array. The following process is typically required to deploy the IBM DB2

pureScale Feature on EMC Symmetrix VMAX storage with fast I/O failover support:

1. Verify that the hardware, software, networking, and storage that you will use to

deploy the DB2 pureScale Feature satisfy the pre-installation requirements. The

pureScale Feature pre-installation tasks can be found by searching for the topic

Installing the DB2 pureScale Feature in the DB2 Version 10.1 Information Center

(http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/index.jsp).

2. Complete the pureScale pre-installation tasks, such as verifying firmware levels,

creating required users, setting up OpenSSH, and modifying kernel parameters (on

Linux operating systems). For detailed information about pureScale pre-

installation checklist for AIX and Linux, go to DB2 Version 10.1 Information

Center and search for the topic Preinstallation checklist for DB2 pureScale

Feature.

3. Perform the initial setup and configuration tasks to provision the shared disks to

the pureScale cluster nodes. Tasks that you must do include following ones:

• If not already done, create the required Symmetrix hyper devices, and use

those devices to create the Auto-provisioning Groups and the masking

view. An introduction to Symmetrix Auto-provisioning and device creation

has been provided in Appendix A.

• On each host, install and configure multipath I/O settings to support the

DB2 pureScale Feature:

o AIX operating systems: Install or update the EMC MPIO ODM

file sets. The minimum supported version is 5.3.0.5.

o Linux operating systems: Install and configure DM-MP. Linux

hosts running either Red Hat Enterprise Linux (RHEL) 5 (or

higher) or SUSE Linux Enterprise Server (SLES) 11 have DM-MP

built in and configured as a kernel module.

For detailed instructions on how to configure multipath I/O settings to

support the DB2 pureScale Feature, see the section “Configuring multipath

I/O with EMC Symmetrix VMAX in a pureScale environment”.

• Discover the devices on the pureScale cluster hosts.



4. Configure the EMC Symmetrix VMAX storage controller to enable SCSI-3 PR

support. For detailed instructions, see the instructions provided in the section

“Enabling and testing SCSI-3 PR with Symmetrix EMC VMAX storage

controllers”.

On Linux pureScale clusters, the device that you use as the DB2 cluster services

tiebreaker disk must have SCSI-3 PR “WRITE EXCLUSIVE REGISTRANTS

ONLY [type code 0x5]” enabled. On AIX pureScale clusters, this is not a

requirement, because on AIX clusters, tiebreaker mechanism uses SCSI-2 reserve-

release commands.

5. Install the DB2 pureScale Feature on the cluster hosts. For information about

installing the IBM DB2 pureScale Feature for Enterprise Server Edition, go to

DB2 Version 10.1 Information Center

(http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/index.jsp) and search for

Preparing to install the DB2 pureScale Feature.

6. Linux operating systems only: Create a configuration file to indicate to the DB2

pureScale Feature that the EMC disks are SCSI-3 PR capable:

a. Determine the disk type by issuing the tsprinquiry command, as shown

in the following example:

# /usr/lpp/mmfs/bin/tsprinquiry dm-1

EMC :SYMMETRIX :5874

b. On each pureScale cluster host, create a prcapdevices file

(/var/mmfs/etc/prcapdevices) by using the device information that you

obtained in the previous substep. For example, create the following entry in

the file:

EMC :SYMMETRIX :5874

7. Create a pureScale instance. Go to DB2 Version 10.1 Information Center and

search for Creating a DB2 pureScale instance for more information.

8. Complete the post-installation tasks, as described in DB2 Version 10.1

Information Center under the topic Taking the first steps after installing the DB2

pureScale Feature.

Tip: You can use the following resources as end-to-end deployment guidelines:

• Deploy the DB2 pureScale Feature on Linux

http://www.ibm.com/developerworks/data/library/techarticle/dm-

1104purescale/index.html



• DB2 best practices: SAP applications with the DB2 pureScale Feature on SUSE

Linux Enterprise Server and IBM System x

http://www.ibm.com/developerworks/data/bestpractices/purescalesapsuselinux/

Validating the configuration of a DB2 pureScale deployment

After successfully deploying a DB2 pureScale configuration, you can perform the

following steps to validate the state of the deployed environment:

1. Log in to a pureScale member node or a CF node as the DB2 instance owner and

change to the ~/sqllib/bin directory.

2. Verify that the DB2 cluster services tiebreaker disk is correctly set by issuing the

following command:

# ./db2cluster -cm -list -tiebreaker

Sample output follows:

The current quorum device is of type Disk with the following

specifics: WWID=36005076304ffc21f0000000000001125.

3. Verify the pureScale cluster file system configuration:

a. Issue the following command:

# ./db2cluster -cfs -list -configuration

b. Ensure that the cluster file system is using SCSI-3 PR, that is, the

usePersistentReserve option is set to yes, as shown in the following sample

output: OPTION VALUE

---------------------- ---------------------------------

...

clusterId 655940652452386548

clusterName db2cluster_20120201144806.torolab.ibm.com

...

failureDetectionTime 35

...

tiebreakerDisks gpfs2nsd

...

usePersistentReserve yes

...

verifyGpfsReady yes

leaseRecoveryWait 35

4. Verify that all the pureScale nodes (members and CFs) are active:

# db2instance -list

Sample output follows:



ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME

---- ----- --------- ------------ ----- ---------------- ------------ -------

0 MEMBER STARTED coralxib62 coralxib62 NO 0 0 coralxib62-eth4

1 MEMBER STARTED coralxib63 coralxib63 NO 0 0 coralxib63-eth4

128 CF PEER coralxib64 coralxib64 NO - 0 coralxib64-eth8,coralxib64-eth5, coralxib64-

eth4,coralxib64-eth6

129 CF PRIMARY coralxib65 coralxib65 NO - 0 coralxib65-eth4,coralxib65-eth6,coralxib65-

eth5,coralxib65-eth7

HOSTNAME STATE INSTANCE_STOPPED ALERT

-------- ----- ---------------- -----

coralxib65 ACTIVE NO NO




For detailed information about how to manage a pureScale cluster by using the db2cluster

command, search for the topic Manage DB2 cluster services command from DB2 Version 10.1

Information Center (http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/index.jsp).

In the next two sections we deal with the main topic addressed in this paper, how to

configure multipath device attributes to support DB2 pureScale Feature and, how to

enable SCSI-3 PR on a VMAX controller to deliver pureScale fast I/O fencing

capabilities.



Configuring multipath I/O with EMC Symmetrix VMAX in a pureScale environment

This section describes how to configure AIX Multiple Path I/O (MPIO) or Linux DM-MP

to work correctly for a pureScale cluster using VMAX SAN disks.

Configuring AIX MPIO for VMAX

This section provides a brief introduction to the AIX MPIO driver and describes how you

can configure it to manage EMC Symmetrix VMAX devices.

Introduction

With AIX MPIO, a device can be uniquely detected through one or more physical

connections or paths and managed through a path-control module (PCM). The AIX PCM

consists of more than one user-configurable I/O routing algorithm. The PCM can select

the optimal path to send I/O to a device based on a variety of criteria, such as load

balancing, connection speed, and connection failure.

An MPIO-capable device driver can control more than one type of target device.

Similarly, a PCM can support one or more specific devices. The following figure

illustrates the interaction between device drivers, ODM definitions, and PCM components

that make up the AIX MPIO framework. In this figure, the MPIO device driver controls

multiple target device types, each requiring a different PCM.

Figure 3. AIX MPIO component interaction (In the diagram, KE stands for kernel

extension, and RTL stands for runtime loadable.)



Before a device can take advantage of MPIO, you must modify the driver, methods, and

predefined attributes of the driver in the AIX ODM to support detection, configuration,

and management of multiple paths. For EMC Symmetrix devices, installing Symmetrix

FCP MPIO ODM fileset on the host takes care of the AIX ODM modification process

described previously.

Configuring AIX MPIO to support EMC Symmetrix devices in pureScale configurations

The following process is required to fully support pureScale shared-disk configurations

that use SAN devices from EMC Symmetrix VMAX storage arrays:

1. Install the EMC Symmetrix AIX ODM FCP MPIO file sets, which contains all the

ODM attributes and the PCM components that are required to support SCSI-3 PR

and MPIO for EMC hyper devices:

a. Download the filesets. Go to EMC PowerLink, and click Support >

Software Downloads and Licensing > Downloads A-B > AIX ODM

Definitions.

EMC AIX ODM definitions are bundled as a single tar file that contains

file sets for Symmetrix, CELERRA, CLARiiON, and INVISTA storage

controllers, with support for FCP, FCP MPIO, and iSCSI device types.

This tar file is versioned as follows:

• EMC.AIX.6.x.x.x.tar, for hosts running AIX 7.1

• EMC.AIX.5.3.x.x.tar, for hosts running AIX 6.1

b. Extract the contents of the file containing the EMC AIX ODM definitions.

c. From the location into which you extracted the tar file, generate the table of

contents (.toc file) by using the “inutoc .” command. The AIX installp command uses the .toc file to detect which filesets are included

in the source.

d. From the ODM source location, install the following two filesets on each

pureScale cluster node: • EMC.Symmetrix.aix.rte

• EMC.Symmetrix.fcp.MPIO.rte

To install, issue the smitty installp command, or run the following

installp command at an AIX korn shell prompt as root:

root> installp -d '.' -f

'EMC.Symmetrix.aix.rte

5.3.0.5,EMC.Symmetrix.fcp.MPIO.rte 5.3.0.5'

'-c' '-N' '-g' '-X' '-G' '-Y' '-M'

e. Reboot the host to update the Base Operating System (BOS) boot image.



f. Verify that the filesets were correctly installed by issuing an lslpp command, as shown in the following example:

root@coralpib11b:/> lslpp -L EMC.Symm* Fileset Level State Type Description (Uninstaller)

----------------------------------------------------------------------------

EMC.Symmetrix.aix.rte 5.3.0.5 C F EMC Symmetrix AIX Support

Software

EMC.Symmetrix.fcp.MPIO.rte

5.3.0.5 C F EMC Symmetrix FCP MPIO Support

Software

State codes:

A -- Applied.

B -- Broken.

C -- Committed.

E -- EFIX Locked.

O -- Obsolete. (partially migrated to newer version)

? -- Inconsistent State...Run lppchk -v.

Type codes:

F -- Installp Fileset

P -- Product

C -- Component

T -- Feature

R -- RPM Package

E -- Interim Fix

2. Provision the required EMC hyper volumes to the hosts in the cluster. For more

details, see Appendix A.

3. Enable SCSI-3 PR flags on the storage controller. For more detail, refer to the

instructions provided in the section “Enabling and testing SCSI-3 PR with

Symmetrix EMC VMAX storage controllers”.

4. Check that all the required EMC hyper devices were correctly detected on the

correct paths:

a. List all the EMC MPIO devices that are detected by the host:

root> lsdev -Ccdisk | awk '/EMC/'

b. Display path information for each of EMC MPIO devices that are found:

• Issue the following command as root for each device to query the

path information:

root> lspath -l hdisk#

- F"connection:parent:path_id:path_status:status"

• To query the path information of multiple devices and display a

formatted output, run the following script as root

root> lsdev -Ccdisk | awk '/EMC/ {print $1}' |

while read disk; do echo ""; echo



"======================== Path Info for ${disk}

======================="; lspath -l ${disk}

-F"connection:parent:path_id:path_status:status";

done; echo ""

For example, when we issued the previous script on an AIX

pureScale cluster host with EMC MPIO disks that we used in our

test environment, the following output was produced:

root@coralpib11b:/> lsdev -Ccdisk | awk '/EMC/ {print

$1}' | while read disk; do echo ""; echo

"======================== Path Info for ${disk}

======================="; lspath -l ${disk}

-F"connection:parent:path_id:path_status:status";

done; echo ""

==================== Path Info for hdisk14 =================

50000972c0088d19,0:fscsi0:0:Available:Enabled

50000972c0088d9c,0:fscsi1:1:Available:Enabled







5. Verify that the EMC hyper devices that were detected support the settings of the

necessary ODM device attributes:

a. Check the possible values that you can specify for the algorithm and reserve_policy ODM device attributes by issuing the following

commands:

root> lsattr -l hdisk# -a algorithm –R

root> lsattr -l hdisk# -a reserve_policy -R

For example, assume that you have an AIX host on which the EMC MPIO

ODM file set is installed and that uses EMC hyper devices from a

Symmetrix VMAX with all the required SCSI-3 PR flags enabled on the

controller. If you issue the previous commands, you should see the

following values for the algorithm and reserve_policy ODM device

attributes:

root@coralpib11b:/> lsattr -l hdisk14 -a algorithm -R

fail_over

round_robin

root@coralpib11b:/> lsattr -l hdisk14 -a reserve_policy -R

no_reserve

single_path

PR_exclusive



PR_shared

For EMC FCP MPIO devices, the default value for the algorithm ODM

attribute is fail_over, and the default value for the reserve_policy ODM attribute is single_path. You can query the default value for any

ODM attribute by specifying the -D option instead of the -R option in an

lsattr command.

b. From the lsattr –R command output, for the device that you will use as

the DB2 cluster services tiebreaker disk, ensure that the algorithm attribute can be set to fail_over and the reserve_policy attribute can be

set to single_path.

c. From the lsattr –R command output, for disks that you will use for

pureScale cluster file systems, ensure that the reserve_policy attribute

can be set to no_reserve or PR_shared.

The DB2 pureScale Feature initially expects all devices that you will use to

create pureScale cluster file systems to have the reserve_policy attribute

set to no_reserve.

6. For the EMC hyper devices that you are using for a pureScale cluster, on each

node, set the required initial values for the algorithm and reserve_policy

device ODM attributes:

• For the hdisk that will act as the DB2 cluster services tiebreaker device,

issue the following command:

root> chdev –l hdisk# -a “algorithm=fail_over

reserve_policy=single_path”

• For each hdisk that you will use to create cluster file systems, issue the

following command:

root> chdev –l hdisk# -a reserve_policy=no_reserve

A note on AIX MPIO device attribute requirements

As described in the previous section, there are specific AIX MPIO device attribute

requirements to support DB2 pureScale cluster services tiebreaker and cluster file system.

The following is brief explanation on why those specific settings are required when AIX

MPIO is used.

Cluster services tiebreaker

The cluster services tiebreaker device depends on the SCSI-2 reserve-release reservation

mechanism on the AIX operating system. This mechanism is inherently tied to the

initiator (path) that the device is opened on or the path that the SCSI-2 reserve is issued

from. Therefore, if I/O is being routed via multiple paths and being switched from one



path to another (that is, by using a round-robin approach), a node might be unable to get a

reservation on the tiebreaker device because it is locked by another host.

For example, assume that hostA issued a SCSI-2 reserve through path1 and successfully

acquired a lock on the cluster services tiebreaker device. The pureScale cluster manager

keeps re-reserving the tiebreaker device periodically, as defined by the HeartbeatPeriod parameter for the IBM.TieBreaker device class. If a node already acquired a lock, a re-

reserve succeeds. Assume that MPIO is now switched to path2 from hostA. At this point,

hostA loses the SCSI-2 reservation; however, the cluster manager has no way of knowing

that. At the next reserve, none of the nodes can acquire a lock on the tiebreaker device

because there is already a SCSI-2 reserve from hostA through path1. In such a scenario,

the cluster manager declares a loss of quorum, and all nodes in the cluster might be

rebooted.

For the cluster services tiebreaker device, if the algorithm ODM attribute is set to fail_over and the reserve_policy ODM attribute is set to single_path, MPIO opens

all paths to the underlying device. However, SCSI-2 reserve-release is always done using

only one path. When this path fails, reserve is released, another path is selected, and SCSI-

2 reserve is reissued using this new path. All of this is transparent to the upper layer,

hence guaranteeing that the reserve is not lost. With MPIO, all physical paths are internal

to MPIO and only the virtual path, which is externalized as an hdisk, is visible to upper-

layer applications such as the pureScale cluster manager.

Cluster file system

In a DB2 pureScale environment, all data and logs are created on an IBM General Parallel

File System (GPFS™) cluster and GPFS file systems that are managed by the DB2

product. Typically, you do not have to change the default values of the ODM attributes of

the devices that are used as GPFS shared disks; that is, you can retain the default

fail_over and single_path values for algorithm and reserve_policy attributes.

However, sometimes the GPFS product cannot create a shared disk on the devices. Before

enabling SCSI-3 PR by changing the reserve_policy attribute to PR_Shared and setting

PR keys, the GPFS product opens the disks with "no reserve." For example, assume that

another node has an ODM entry with the reserve_policy attribute set to single_path and has the disk open before GPFS product tries to open it. At this point, the GPFS

daemon on this node cannot open the disk. Potentially, something as fundamental as a

health check running against the disk from a node using a reserve policy other than

no_reserve can cause a similar scenario. Therefore, it is highly recommended to set the

reserve_policy attribute to no_reserve for all the devices that you will use to create

GPFS shared disks on all the nodes in the pureScale cluster. By doing so, you avoid

potential DEVICE_BUSY and RESERVATION_CONFLICT errors.



Configuring the Device Mapper Multipath module on Linux operating systems

The following section provides a brief introduction to the Device Mapper Multipath (DM-

MP) module and explains how to configure and use it in a pureScale environment with

EMC Symmetrix VMAX.

Introduction

The DM-MP module provides MPIO capability for hosts running Linux operating

systems. DM-MP is the default MPIO solution for Linux server platforms such as Red Hat

Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES).

The Linux kernel detects each path for a multipath device as a separate SCSI device, that

is, /dev/sd[a-z]*. DM-MP then defines a single multipath device on top of those

underlying device paths (for the same device) with a globally unique World Wide

Identifier (WWID). By default, the name of a multipath device is set to its WWID.

DM-MP supports up to eight paths per device and features automatic configuration of the

MPIO subsystem for configurations such as the following ones:

• Active/passive, where one path is active and the others are passive

• Active/active, where all paths are active with round-robin load balancing

DM-MP performs automatic path discovery and grouping. DM-MP also performs

automated path retesting, so that a previously failed path is automatically reinstated when

it becomes healthy again. However, DM-MP does not protect against failures in the device

itself. If one of the active paths is lost because of a host bust adapter (HBA) failure, a

SAN port failure, or a FC cabling problem, for example, I/O is redirected to the remaining

paths. Depending on the path failover policy that you use, the path fails over to one of the

passive paths (if you use active/passive) or the traffic is balanced across the remaining

healthy paths (if you use active/active).



Typically, the following steps must be performed on a Linux host, to set up DM-MP:

1. If required, install the device-mapper-multipath RPMs.

• You can query if DM-MP RPMs are already installed by issuing, the

following command as root:

root> rpm –qa | grep -i dm

• If DM-MP RPMs are installed, check if the device-mapper module is

loaded and the multipathd (DM-MP daemon) is running, by issuing the

following two commands as root:

root> lsmod | grep -i dm_multipath

root> chkconfig --list | grep -i multipathd

2. Create the DM-MP configuration file, /etc/multipath.conf, using the default

DM-MP configuration.

3. Start the DM-MP daemons to enable multipathing.

4. If necessary, edit the multipath.conf configuration file to modify default values.

The compiled-in default settings that are included with the DM-MP module are

suitable for most of the common multipath configurations.

5. Start the multipath daemon.

The changes to the default DM-MP configuration to enable multipath I/O for devices from

Symmetrix VMAX storage in a pureScale cluster, are discussed in detail in the rest of this

section.

The DM-MP configuration file

The DM-MP configuration file, /etc/multipath.conf, has the following structure:

default {

Multipath tools default settings. You do not have to define this stanza unless you

must override the DM-MP complied-in defaults, which are in the multipath.conf.defaults file.

}

blacklist {

List of device names to discard as not multipath candidates.

You can identify devices by their device node name (devnode), WWID (wwid), or

vendor and product string (device). }

multipaths {

Settings that apply to one specific multipath that you specify by using the wwid of

the path.

multipath {



Settings for a single path. }

}

devices {

List of per-storage controller settings that override default settings and are

overridden by multipath settings. Because many devices from all major storage

vendors that support multipathing are included with complied-in defaults, typically

you do not have to modify the devices stanza unless you must override those

defaults.

device {

Settings for a single storage controller. }

}

Setting up DM-MP on the pureScale cluster hosts

The following instructions are provided as a quick reference to setting up DM-MP on

hosts running RHEL 6 or SLES 11. For more detailed configuration information and

options, see the DM-MP documentation that the distribution vendors provide.

1. Enable the multipath daemon to run at boot time:

• RHEL 6: root> mpathconf –enable

On RHEL, this command generates the DM-MP configuration file if it

does not exist.

• SLES 11: root> insserv multipathd

2. SLES only: Create the DM-MP configuration file, multipath.conf, by copying

the /usr/share/doc/packages/multipath-tools/multipath.conf.synthetic file to the /etc/multipath.conf file.

3. Configure user-friendly names for multipath devices by setting the user_friendly_names option to yes in the /etc/multipath.conf file. If you set

the option to yes, DM-MP uses the /var/lib/multipath/bindings file to assign

a persistent and unique name to the device in the form of mpath<n>. If the option

is set to no (the default), DM-MP uses the WWID as the name of the device.

Important: Enabling user-friendly names does not guarantee that the multipath

device names are unique across all nodes in the cluster. The user_friendly_names configuration option sets unique names for multipath

devices only on the node from which you enable the configuration. If you want the

system-defined user-friendly names to be consistent across all nodes in the cluster,

use the following procedure:

a. Set up all of the multipath devices on one cluster node.

b. On your other machines, disable all your multipath devices.



c. Copy the /etc/multipath/bindings file from the first machine to all the

other machines in the cluster.

d. If you add a new device, you must repeat this process.

4. Ignore local (non-multipathed) devices. The /etc/multipath.conf file contains a

blacklist section, where all non-multipathed devices, such as local hard drives and

floppy drives, are listed. To blacklist all local devices from being managed by

DM-MP, ensure that the blacklist section looks similar to the following example:

blacklist {

devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda)[0-9]*"

devnode "^hd[a-z][0-9]*"

}

Tip: RHEL 6 introduced the find_multipath DM-MP configuration parameter.

When this parameter is set to yes, DM-MP automatically blacklists all single-path

devices. For more details on how to use this new configuration parameter, see the

RHEL 6 DM Multipath Configuration and Administration Guide.

5. To re-create the initial RAM disk (initrd) on your system, issue the mkinitrd command.

6. To have the changes take effect, reboot. After you do this, the local devices should

no longer be listed in the multipath maps if you issue the multipath -ll command.

7. Perform a dry run with the updated multipath DM-MP configuration by issuing the

following command and evaluating the setup:

root> multipath -v2 -d

8. Start the DM-MP daemon:

• RHEL 6: root> service multipathd start

• SLES 11: root> chkconfig multipathd on

root> chkconfig boot.multipath on

DM-MP configuration file examples

You can use the following sample configurations in DB2 pureScale Feature–EMC

Symmetrix VMAX environments:

• RHEL 6: # Multipath default configuration attributes

# for a DB2 pureScale shared-disk configuration.

defaults {



user_friendly_names yes

polling_interval 5

checker_timeout 25

}

blacklist {

devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda)[0-9]*"

devnode "^hd[a-z][0-9]*"

}

devices {

# Multipath device configuration attributes

# for EMC Symmetrix storage arrays in a

# DB2 pureScale shared-disk configuration.

device {

vendor "EMC"

product "SYMMETRIX"

getuid_callout "/lib/udev/scsi_id --

whitelisted --page=pre-spc3-83 --device=/dev/%n"

path_grouping_policy multibus

failback immediate

path_selector "round-robin 0"

path_checker tur

no_path_retry 5

# fast_io_fail_tmo 5

# dev_loss_tmo 90

hardware_handler "0"

prio const

rr_weight uniform

rr_min_io 1000

}

}

Notes:

• On RHEL 6, all complied-in DM-MP configuration files are located in /usr/share/doc/device-mapper-multipath-0.4.9 directory. For a

description of the DM-MP parameters in the previous sample, see the multipath.conf.annotated file.

• Before enabling the fast_io_fail_tmo and dev_loss_tmo DM-MP

configuration parameters, see the topic on how to use them, later in this

section.

• RHEL 5:

Use the same defaults stanza as in the previous example, and change the devices stanza as follows:

device {

vendor "EMC"

product "SYMMETRIX"

getuid_callout "/lib/udev/scsi_id --whitelisted

--device=/dev/%n"


failback immediate




path_checker tur

no_path_retry 5


rr_weight uniform

rr_min_io 1000

}

Notes:

• The checker_timeout and prio DM-MP configuration parameters are not

supported on RHEL 5 distributions. • On RHEL 5.7, all complied-in DM-MP configuration files are located in

/usr/share/doc/device-mapper-multipath-0.4.7 directory.

• SLES 11:

device {

vendor "EMC"

product "SYMMETRIX"

getuid_callout "/lib/udev/scsi_id

--whitelisted --page=pre-spc3-83 --device=/dev/%n"


failback immediate


path_checker tur

no_path_retry 5

# fast_io_fail_tmo 5

# dev_loss_tmo 25


prio const

rr_weight uniform

rr_min_io 1000

}

Notes:

• The checker_timeout DM-MP configuration parameter is not supported

on SLES11 distributions.

• Before enabling the fast_io_fail_tmo and dev_loss_tmo DM-MP

configuration parameters, see the topic on how to use them, later in this

section.

DM-MP behavior with path failures in RHEL 6 or SLES 11

Typically, during a path failure, DM-MP marks the path as failed but retains the bad path

in the path list. The bad path is marked as failed in the output of the multipath -l command.

However, in Linux distributions with higher-level kernels, such as RHEL 6 or SLES 11,

when a path fails, all references to that path and the devices are completely removed from

the OS kernel. This behavior is due to a change in the behavior of the Linux kernel.

Therefore, if a path fails, it is not displayed in the output of the multipath -l command.



In addition, when restoring failed paths, the device trespass (switching the ownership of a

device in a path failure) will occur only after all paths are restored. This change in

behavior applies only to the most recent versions of DM-MPIO.

Using the fast_io_fail_tmo and dev_loss_tmo DM-MP configuration parameters on RHEL 6 and SLES 11

The dev_loss_tmo parameter sets the number of seconds to wait before a multipath link

is marked as bad. The default value varies according to the underlying device driver that

you use. The valid range of values is 0 - 600 seconds. If the value is 0 or greater than 600,

the internal timeout value of the driver is used instead.

The fast_io_fail_tmo parameter sets the length of time to wait before failing I/O after a

link problem is detected. If I/O is in a blocked queue, the I/O does not fail until the dev_loss_tmo time elapses and the queue is unblocked.

When host-addressable SAN devices are configured with multiple paths using DM-MP,

and the multipath no_path_retry setting is active (that is, queuing is enabled), modify

the dev_loss_tmo setting of the storage subsystem accordingly to ensure that no devices

are removed if all paths fail. The recommended setting for the dev_los_tmo parameter for

the storage subsystem is as follows:

dev_loss_tmo setting = no_path_retry * polling_interval

where:

• no_path_retry is the number of retries for multipath I/O until the path is

considered to be lost and queuing of I/O is stopped.

• polling_interval is the time in seconds between path checks.

Applying DM-MP configuration file changes

Any changes that you make to the /etc/multipath.conf file are picked up by the DM-

MP daemon (multipathd) only when it’s starting up. Therefore, after making the

recommended changes to the DM-MP configuration file, as described previously, perform

the following steps to apply those changes:

• RHEL 5 or SLES 11

1. Stop the multipathd service:

root> /etc/init.d/multipathd stop

2. Clear old multipath bindings:

root> /sbin/multipath –F

3. Create new multipath bindings:

root> /sbin/multipath -v2 -l



4. Start the multipathd service:

root> /etc/init.d/multipathd start

5. To re-create the initrd on your system, issue the mkinitrd command.

6. To have the changes take effect, reboot.

• RHEL 6

RHEL 6 supports reloading the multipath configuration file without recycling the

multipath daemon and the multipath bindings. To reload, issue the following

command:

root> service multipathd reload



Enabling and testing SCSI-3 PR with Symmetrix EMC VMAX storage controllers

This appendix details the process that is required to configure EMC Symmetrix VMAX to

support DB2 pureScale ultra-fast failover, specifically using SCSI-3 persistent

reservations.

Introduction

PR refers to a set of SCSI-3 standard commands and command options that provide SCSI

initiators with the ability to establish, preempt, abort, query, and reset a reservation policy

with a specified target device. PR commands are not compatible with the legacy

reserve/release mechanism, and target devices can support only one reservation

mechanism at a time.

When a device is opened, the device driver either checks the ODM for a reserve_policy and a PR_key_value (on AIX operating systems) or issues a SCSI inquiry command (on

Linux operating systems) before hand to open the device accordingly. SCSI-3 PR supports

device access from multiple systems or from multiple paths from a single system. At the

same time, SCSI-3 PR blocks access to the device from other systems or other paths.

Therefore, for PR to work, each host that is attached to the shared disk must use a unique

registration key (PR key) value which is shared among all the device paths on the same

host.

The DB2 pureScale Feature can use the SCSI-3 PR standard to provide fast failover with

improved recovery times. To use this functionality, you must create the pureScale file

systems on SCSI-3 PR–capable disks.

The DB2 pureScale Feature requirements for SCIS-3 PR capability on the target devices

are as follows:

• For AIX only: The ODM must support setting PR_shared for reserve_policy

attribute with a PR key value for the PR_key_value attribute.

• The device must support “REPORT CAPABILITIES service action 2h” and a

reservation type of “WRITE EXCLUSIVE ALL REGISTRANTS [type code 7h]”.

• For Linux only: The device that you select as the DB2 cluster services tiebreaker

must support a SCSI-3 PR type of “WRITE EXCLUSIVE REGISTRANTS

ONLY [type code 5h]”. With SCSI-3 PR type 0x5, there is only one persistent

reservation holder, which is the node that acquires the quorum device.

In the SCSI-3 specification, there is a Type Mask Valid (TMV) bit that is returned when a

Report Capabilities SCSI inquiry is sent to a device. A TMV bit set to 1 indicates that the

PERSISTENT RESERVATION TYPE MASK field contains a bit map indicating which



PR types the device supports. The DB2 pureScale disk management subsystem checks this

TMV bit to figure out whether reservation type masks were set before verifying whether

the device supports the “WRITE EXCLUSIVE ALL REGISTRANTS ONLY type code

7h” reservation type.

Configuration overview

To enable SCSI-3 PR, perform the following configuration steps on the EMC VMAX

storage array and on pureScale cluster hosts:

1. Check the storage and operating system requirements for supporting SCSI-3 PR

with the DB2 pureScale Feature.

2. Perform the initial setup and configuration tasks to provision the shared-storage to

the pureScale cluster nodes. These tasks include the following ones:

a. Zone the DB2 pureScale cluster hosts with the EMC storage controller FA

ports.

b. If not already done, create EMC Symmetrix hyper devices on the VMAX

with the required size for your organization’s data.

c. Create the Auto-provision Groups and the masking view. Rescan the SCSI

bus to discover those new devices on all nodes in the DB2 pureScale

cluster. For detailed instructions on how to do this, see the section “Storage

Provisioning in Symmetrix VMAX with Enginuity 5874 Code and higher”.

d. On each pureScale cluster host, configure multipath I/O settings to support

pureScale MPIO requirements. For detailed instructions on how to do this

on AIX or Linux operating systems, see the section “Using multipath I/O

with EMC Symmetrix VMAX in a pureScale environment”

3. Enable SCSI-3 PR on the EMC VMAX storage array.

4. Validate that the storage subsystem meets pureScale SCSI-3 PR requirements

5. Enable SCSI-3 PR on the pureScale cluster file system.

The example outputs provided in the rest of this section are mainly from a Linux

pureScale test topology that we used during the development of this paper, which has four

cluster nodes that are zoned with an EMC VMAX storage array:

• Cluster nodes: coralxib62, coralxib63, coralxib64, coralxib65

• EMC storage array: VMAX SE (Enginuity code 5874)

• EMC storage admin host: emcaix1



Storage and operating system requirements for supporting SCSI-3 PR

The storage and operating system requirements for SCSI-PR are as follows:

• On the EMC Symmetrix VMAX (Enginuity code 5874), you need Symm Code

5874.207.166 or higher. You can use symcfg command to query this information.

For example, issuing the following command on our topology shows:

root@emcaix1:/> symcfg -sid 547 list -v | grep -i Microcode

Microcode Version (Number) : 5875 (16F30000)

Microcode Registered Build : 1

Microcode Date : 06.06.2011

Microcode Patch Date : 06.06.2011

Microcode Patch Level : 198

• On AIX operating systems: The EMC Symmetrix FCP MPIO file set contains all

the ODM attributes that are required to support SCSI-3 PR on Symmetrix storage.

To download the EMC FCP MPIO AIX ODM file sets, go to EMC PowerLink.

• On Linux operating systems: You must install DM-MP and the SG Utilities

RPMs, and you must configure DM-MP and have it running. Linux hosts running

either Red Hat Enterprise Linux (RHEL) v5 or better or running SUSE Linux

Enterprise Server (SLES) v11 have DM-MP built in and configured as a kernel

module.

• You must enable SCSI-3 PR flags on the DA/FA ports and on each hyper device

that you assign to the pureScale hosts.

Enabling SCSI-3 PR on the EMC Symmetrix VMAX controller

You can enable SCSI-3 PR flags on the VMAX by using either of the following tools:

• Symmetrix service processor. This requires an EMC CE to do a site visit or use

the ESRS Gateway and remotely make the changes by using Service Processor

menus.

• Symmetrix Solutions Enabler (SYMCLI). Enablement using SYMCLI can be

done on any host that is attached to the Symmetrix and configured with SYMCLI

v7.1 or better.

Using the Symmetrix service processor

1. Ensure that, on the DirEdit menu in the impl.bin file, the following flags are

enabled on the FA ports:

SC3, SPC2, OS2007, PP, UWN, ACLX, EAN

2. Ensure that, on the VolEdit menu in the impl.bin file, the following flag is

enabled for each hyper volume that the pureScale cluster file systems use:



SCSI Persistent

Using the SYMCLI

There are two approaches for enabling SCSI-3 port flags by using the SYMCLI:

• Set the SCSI-3 flags by HBA WWPN. HBA port flags always override the FA

flags, hence this change will only affect HBA with the specified WWPN.

• Set the SCSI-3 flags by FA port. This enables SCSI-3 on any HBA attached to

these FA ports. You can have multiple HBAs on these FA ports connected through

a SAN switch.

In addition to enabling SCSI-3 flags either at the FA port level or the HBA port level (per

WWPN) you must enable them at the hyper device level as well.

Method 1: Setting SCSI-3 flags at the HBA/WWPN level (per host)

This online configuration method enables SCSI-3 flags for only the host initiator that you

specify, not at the FA port level. Enabling flags at the FA port level turns on SCSI-3 flags

for all the hosts that are attached through that port.

1. Log in to the EMC storage administration host that is configured with SYMCLI

and can communicate with the Symmetrix VMAX.

2. Display detailed information about the Auto-provision masking view that was

created for this pureScale cluster to determine the host initiator group names. For

example, in the test configuration, we obtained this information by issuing the

following command:

root@emcaix1:/> symaccess -sid 547 show view db2ps_v

Symmetrix ID : 000194900547

Masking View Name : db2ps_v

Last updated at : 10:39:14 AM on Tue Aug 09,2011

Initiator Group Name : db2ps_ig

Host Initiators

{

IG : db2ps_coralxib62_ig




}

Port Group Name : db2ps_pg

Director Identification

{

FA-7E:1



FA-8G:0

}

Storage Group Name : db2ps_sg

Sym Dev Host

Name Dir:P Physical Device Name Lun Attr Cap(MB)

------ ----- ----------------------- ---- ---- -------

00C9 07E:1 Not Visible 0 10240

08G:0 Not Visible 0

00CA 07E:1 Not Visible 1 102400

08G:0 Not Visible 1

00CB 07E:1 Not Visible 2 102400

08G:0 Not Visible 2

00FB 07E:1 Not Visible 3 (M) 409603

08G:0 Not Visible 3 (M)

-------

Total Capacity 615427

3. Query the current HBA flags that are set for each host IG. For example, we issued

the following command to query host IG db2ps_coralxib62_ig in our test

configuration:

root@emcaix1:/> symaccess -sid 547 show db2ps_coralxib62_ig -

type initiator -detail



Initiator Group Name : db2ps_coralxib62_ig

Originator Port wwn : 2100001b321d800a

User-generated Name : 2100001b321d800a/2100001b321d800a

FCID Lockdown : No

Heterogeneous Host : No

Port Flag Overrides : No

CHAP Enabled : N/A

Type : Fibre



FCID Lockdown : No


Port Flag Overrides : No

CHAP Enabled : N/A

Type : Fibre

4. For each host IG in the Auto provision masking view, enable SCSI-3 flags on the

HBAs. In the test configuration, we had two HBAs mapped to each host IG, with

four IGs in total in the Auto provision view. Therefore, we used the following

commands:

root@emcaix1:/> symaccess -sid 547 -wwn 2100001b321d800a set hba_flags on SC3 -enable



root@emcaix1:/> symaccess -sid 547 -wwn 2101001b323d800a set hba_flags on SC3 -enable root@emcaix1:/> symaccess -sid 547 -wwn 2100001b321d850a set hba_flags on SC3 -enable root@emcaix1:/> symaccess -sid 547 -wwn 2101001b323d850a set hba_flags on SC3 -enable root@emcaix1:/> symaccess -sid 547 -wwn 2100001b321d540b set hba_flags on SC3 -enable root@emcaix1:/> symaccess -sid 547 -wwn 2101001b323d540b set hba_flags on SC3 -enable root@emcaix1:/> symaccess -sid 547 -wwn 2100001b3280b327 set hba_flags on SC3 -enable root@emcaix1:/> symaccess -sid 547 -wwn 2101001b32a0b327 set hba_flags on SC3 –enable

5. Verify that the HBAs in the IG are updated. For example, we issued the following

command to query host IG db2ps_coralxib62_ig in our test configuration:

root@emcaix1:/> symaccess -sid 547 show db2ps_coralxib62_ig -

type initiator -detail



Initiator Group Name : db2ps_coralxib62_ig



FCID Lockdown : No


Port Flag Overrides : Yes

Enabled : SCSI_3(SC3)

Disabled : N/A

CHAP Enabled : N/A

Type : Fibre



FCID Lockdown : No


Port Flag Overrides : Yes

Enabled : SCSI_3(SC3)

Disabled : N/A

CHAP Enabled : N/A

Type : Fibre

Method 2: Setting SCSI-3 flags at the FA port level (for all hosts that are connected to an FA port)

If you have multiple hosts that are connected to same EMC FA ports by using a SAN

switch in the middle, setting SCSI-3 flags by FA dir:port affects all hosts that are attached

through those FA ports. In addition, before changing SCSI-3 flags for any FA port, you

must take that port offline. The only benefit of using this method is that it is a one-time



operation. That is, after you enable the FA dir:ports for SCSI-3 PR, any new host that you

zone with those ports will not require any changes on the Symmetrix to support the DB2

pureScale Feature.

1. Identify the Symmetrix FA dir:port for which you must enable the SCSI-3 PR

flags by using the auto storage provisioning masking view. For example, we issued

the following command to query the masking view db2ps_v in our test

configuration:






Host Initiators

{





}



{

FA-7E:1

FA-8G:0

}


Sym Dev Host


------ ----- ----------------------- ---- ---- -------


08G:0 Not Visible 0


08G:0 Not Visible 1


08G:0 Not Visible 2



-------


2. Create a configuration file with the following commands, to enable SCSI_3 flags

on the FA dir:ports that you use on the EMC VMAX to zone the pureScale

cluster. For example, in our test environment, the EMC VMAX was connected to



the SAN through FA dir:ports 7E:1 and 8G:0; hence, we used the following

configuration file (named as enableSCSI3):

set port 7E:1 SCSI_3=ENABLE;

set port 8G:0 SCSI_3=ENABLE;

3. Validate the syntax of the configuration file that you created in the previous step:

root@emcaix1:/> symconfigure -file enableSCSI3 preview

Establishing a configuration change

session...............Established.

Processing symmetrix 000194900547

Performing Access

checks..................................Allowed.

Checking Device

Reservations..............................Allowed.

Locking

devices...........................................Locked.

Validating configuration

changes..........................Validated.

Closing configuration change

request......................Closed.

Terminating the configuration change

session..............Done.

The configuration change session has completed successfully.

4. Validate the context of the configuration file. That is, verify that you can run the

requested operations on the Symmetrix.

root@emcaix1:/> symconfigure -file enableSCSI3 prepare




Performing Access

checks..................................Allowed.

Checking Device


Locking

devices...........................................Locked.

Validating configuration

changes..........................Validated.

Closing configuration change

request......................Closed.



The configuration change session has completed successfully.

5. Bring the affected FAs offline, because changing FA port flags is an offline

operation:



root@emcaix1:/> symcfg -sid 547 offline -SA 7E -P 1 -noprompt -v

A port 'Offline' operation execution is

in progress for Symmetrix unit '000194900547'. Please wait...

Offline FE director(s) on local Symmetrix..................

The 'OFFLINE' control operation SUCCEEDED.

The port 'Offline' operation successfully executed for

Symmetrix Unit '000194900547'.

root@emcaix1:/> symcfg -sid 547 offline -SA 8G -P 0 -noprompt -v

A port 'Offline' operation execution is


Offline FE director(s) on local Symmetrix..................

The 'OFFLINE' control operation SUCCEEDED.

The port 'Offline' operation successfully executed for


6. Apply the changes in the configuration file:

root@emcaix1:/> symconfigure -file enableSCSI3 commit




Performing Access

checks..................................Allowed.

Checking Device


Locking

devices...........................................Locked.

Initiating COMMIT of configuration

changes................Queued.

COMMIT requesting required

resources......................Obtained.

Step 020 of 044

steps.....................................Executing.

Step 051 of 105

steps.....................................Executing.

Step 051 of 105

steps.....................................Executing.

Step 057 of 105

steps.....................................Executing.

Step 061 of 105

steps.....................................Executing.

Step 066 of 105

steps.....................................Executing.

Step 084 of 105

steps.....................................Executing.

Step 087 of 105

steps.....................................Executing.

Step 089 of 105

steps.....................................Executing.



Step 090 of 105

steps.....................................Executing.

Step 093 of 105

steps.....................................Executing.

Local:

COMMIT............................................Done.



The configuration change session has successfully completed.

7. Bring the FAs back online:

root@emcaix1:/> symcfg -sid 547 online -SA 7E -P 1 -noprompt -v

A port 'Online' operation execution is


Online FE director port(s) on local

Symmetrix..................Done.

The 'ONLINE' control operation SUCCEEDED.

The port 'Online' operation successfully executed for


root@emcaix1:/> symcfg -sid 547 online -SA 8G -P 0 -noprompt -v

A port 'Online' operation execution is


Online FE director port(s) on local

Symmetrix..................Done.

The 'ONLINE' control operation SUCCEEDED.

The port 'Online' operation successfully executed for


8. Verify that FA port flags are correctly set:

root@emcaix1:/> symcfg -sid 547 list -FA 7E -v | awk '/Director

Identification/ || /Director Port:/ || /SC3/'

Director Identification: FA-7E

Director Port: 0

SCSI_3(SC3) : Enabled

Director Port: 1


root@emcaix1:/> symcfg -sid 547 list -FA 8G -v | awk '/Director

Identification/ || /Director Port:/ || /SC3/'

Director Identification: FA-8G

Director Port: 0


Director Port: 1




Enabling the SCSI-3 PR flag on the required Symmetrix hyper volumes

Irrespective of the method that you used to enable SCSI-3 flags on the ports, you must

also enable SCSI-3 PR flags on each hyper device that you use.

The following steps and the sample outputs provided are from our Linux pureScale test

configuration.

1. Using the Auto storage provision masking view, identify the Symmetrix hyper

volumes that you need to enable SCSI-3 PR flags for:






Host Initiators

{





}



{

FA-7E:1

FA-8G:0

}


Sym Dev Host


------ ----- ----------------------- ---- ---- -------


08G:0 Not Visible 0


08G:0 Not Visible 1


08G:0 Not Visible 2



-------


2. Create the a configuration file with the commands to set the following Symmetrix

hyper device flags (named it as enableSCSI3pr_hypers, for example):



set dev 0C9:0CB attribute=SCSI3_persist_reserv;

set dev 0FB attribute=SCSI3_persist_reserv;

3. Validate the syntax and the context of the configuration file that you created in

Step 2. and then commit those changes by executing the following commands:

root@emcaix1:/> symconfigure -file enableSCSI3pr_hypers preview

root@emcaix1:/> symconfigure -file enableSCSI3pr_hypers prepare

root@emcaix1:/> symconfigure -file enableSCSI3pr_hypers commit

4. Verify that SCSI-3 PR is enabled on the required Symmetrix hyper devices.

Repeat the following command for each hyper device address range that you

updated with the SCSI3_persist_reserv flag:

root@emcaix1:/> symdev -sid 547 list -RANGE 0C9:0CB -v | awk

'/Device Symmetrix Name/ || /SCSI-3/'

Device Symmetrix Name : 00C9

SCSI-3 Persistent Reserve: Enabled

Device Symmetrix Name : 00CA


Device Symmetrix Name : 00CB


root@emcaix1:/> symdev -sid 547 list -RANGE 0FB:0FB -v | awk

'/Device Symmetrix Name/ || /SCSI-3/'

Device Symmetrix Name : 00FB


Validating that the storage subsystem meets pureScale SCSI-3 PR requirements

After provisioning the storage to the pureScale cluster hosts and enabling SCSI-3 PR on

those devices, verify that the storage subsystem meets pureScale SCSI-3 PR requirements.

On Linux clusters, you can do this validation before starting the DB2 pureScale

installation. However, on AIX clusters, you can do this validation only after the

installation.

Checking SCSI-3 PR device report capabilities on Linux pureScale clusters

On Linux hosts with the sg_utils RPMs installed, you can use the sg_persist command to query the device capabilities. The command output must show that both the

Type Mask Valid (TMV) bit and Write Exclusive All Registrants bit are enabled.

Sample output from running the sg_persist command on one of the devices in our

cluster is as follows:

# sg_persist -c /dev/dm-1 EMC SYMMETRIX 5874 Peripheral device type: disk Report capabilities response:



Compatible Reservation Handling(CRH): 0 Specify Initiator Ports Capable(SIP_C): 0 All Target Ports Capable(ATP_C): 0 Persist Through Power Loss Capable(PTPL_C): 1 Type Mask Valid(TMV): 1 Allow Commands: 0 Persist Through Power Loss Active(PTPL_A): 0 Support indicated in Type mask: Write Exclusive, all registrants: 1 Exclusive Access, registrants only: 1 Write Exclusive, registrants only: 1 Exclusive Access: 1 Write Exclusive: 1 Exclusive Access, all registrants: 1

Checking SCSI-3 PR device report capabilities on AIX pureScale clusters

You can validate SCSI-3 PR device capabilities on AIX pureScale hosts by using the

tsprrpcap utility, which is installed during pureScale setup. The command output must

show that both the Type Mask Valid (TMV) bit and the Write Exclusive All Registrants

bit are enabled.

Following is an example of running the tsprrpcap utility. The output shows the SCSI bit

mask followed by the interpretation of all the bits that are on.

# /usr/lpp/mmfs/bin/tsprrpcap hdisk5

PR Type Mask = ea 01

Write_Exclusive_All_Registrants

Exclusive_Access_Registrants_Only

Write_Exclusive_Registrants_Only

Exclusive_Access

Write_Exclusive

Exclusive_Access_All_Registrants

Enabling SCSI-3 PR on the pureScale cluster configuration

When you run the DB2 pureScale installer, it queries the underlying storage subsystem

and detects, through the Report Capabilities SCSI inquiry command, whether the specified

shared disk meets the pureScale SCSI-3 PR requirements. If the storage subsystem

satisfies pureScale SCSI-3 PR requirements, the pureScale installer automatically enables

the cluster configuration to support SCSI-3 PR.

Verifying that SCSI-3 PR is enabled in the pureScale cluster configuration

If a pureScale cluster file system has the usePersistentReserve attribute enabled, that

pureScale cluster is configured to use SCSI-3 PR. For example, in our test cluster, we

queried the pureScale cluster file system configuration by issuing the db2cluster –cfs

command:

# ./db2cluster -cfs -list -configuration

OPTION VALUE



---------------------- ---------------------------------

...

tiebreakerDisks gpfs2nsd

...

usePersistentReserve yes

...

verifyGpfsReady yes

SCSI-3 PR key management in a DB2 pureScale cluster

As part of the process to enable SCSI-3 PR in the cluster configuration, pureScale cluster

file system layer generates a set of PR keys that are unique to each host. The DB2

pureScale Feature also distributes them among all the hosts in the cluster and to the

storage controller. That is, after you enable SCSI-3 PR, all the hosts and the storage

controller that provisioned the shared disks are aware of the PR keys that will be used.

Because the uniqueness of a PR key is per host, the same PR key is used on each path

even if there are multiple paths to a single device.

You can query the PR keys that are registered for a device by issuing the tsprreadkeys

command on AIX or Linux operating systems. Before issuing this command, you must

discover which devices are used for cluster file systems. The following command, which

we issued on one of the test cluster hosts, shows shared disk-to-device mappings that are

local to that host, for each pureScale cluster file system.

[root@coralxib62 ~]# /usr/lpp/mmfs/bin/mmlsnsd -m

Disk name NSD volume ID Device Node name Remarks

---------------------------------------------------------------------------------

gpfs1nsd 091A5DE34F4B9DFF /dev/dm-1 coralxib62.torolab.ibm.com

gpfs2nsd 091A5DE34F4BA3CF /dev/dm-3 coralxib62.torolab.ibm.com

gpfs3nsd 091A5DE34F4BA480 /dev/dm-2 coralxib62.torolab.ibm.com

The following results were returned when we issued the tsprreadkeys command on our

Linux test cluster:

[root@coralxib62 ~]# /usr/lpp/mmfs/bin/tsprreadkeys dm-1

Registration keys for dm-1

1. 00006d0000000004

2. 00006d0000000004

3. 00006d0000000003

4. 00006d0000000003

5. 00006d0000000002

6. 00006d0000000002

7. 00006d0000000001

8. 00006d0000000001

The pureScale cluster file systems on AIX operating systems and on Linux operating

systems use a different PR key format. For example, on an AIX cluster, you would see

something similar to the following output, which was returned when we issued the tsprreadkeys command on our AIX test cluster:

root@coralpib93:/> /usr/lpp/mmfs/bin/tsprreadkeys hdisk5



Registration keys for hdisk5

1. 00006d00091ab6b1

2. 00006d00091ab6b1

3. 00006d00091a05cd

4. 00006d00091a05cd

5. 00006d00091ab6b2

6. 00006d00091ab6b2

7. 00006d00091a05ce

8. 00006d00091a05ce

The pureScale cluster file system layer generates a single PR key for each member node,

and that key must be registered on each shared device with SCSI-3 PR enabled. Therefore,

in a cluster with four members using shared disks, each having two paths to the storage,

the number of keys is as follows:

• Number of unique PR keys = 4

• Total number of keys that are registered on a shared LUN

= Number of cluster nodes * number of paths

= 4 * 2 = 8

On AIX operating systems, you can also query the ODM attributes of a device to look at

the PR key and reserve policy that pureScale cluster file system layer has set, as shown in

the following example:

root@coralpib12:/> lsattr -El hdisk18 -a "PR_key_value reserve_policy"

PR_key_value 0x6d0009170148 Persistant Reserve Key Value True

reserve_policy PR_shared Reserve Policy True

Similarly, on Linux operating systems, you can use the sg_persist command to dump

the keys that were set on a device, as shown in the following example:

[root@coralxib62 ~]# sg_persist -d /dev/dm-1 -i -k

EMC SYMMETRIX 5874

Peripheral device type: disk

PR generation=0x2f2, 4 registered reservation keys follow:

0x6d0000000004

0x6d0000000004

0x6d0000000003

0x6d0000000003

0x6d0000000002

0x6d0000000002

0x6d0000000001

0x6d0000000001



Conclusion

The DB2 pureScale Feature for DB2 Enterprise Server Edition, along with EMC

Symmetrix VMAX storage, can scale to meet the growing and dynamic needs of different

organizations. You can start additional members in a DB2 pureScale environment, without

affecting existing applications, to meet the demands of peak processing times and

additional storage capacity requirements. This solution also provides leading-edge high

availability by leveraging SCSI technology, allowing for best-of-breed availability. If a

DB2 member fails, applications are automatically rerouted among the active members

until the failed member comes back online. The design and capabilities of the DB2

pureScale Feature have a lower total cost of ownership than those of other solutions, by

using a simplified deployment and maintenance model.



Appendix A. Storage provisioning in Symmetrix VMAX with Enginuity code 5874 and higher

Starting with Symmetrix VMAX Enginuity code 5874, Auto Provisioning Groups provide

you with a simplified model for storage provisioning. Auto provisioning allows for

logically grouping HBAs (initiators), Symmetrix devices, and FA ports on the Symmetrix

and associating all three entities through masking views. When you create a masking

view, all the required device mappings and masking steps are performed automatically to

provision the storage. In addition, any changes to the groupings of initiators, storage

devices, or ports are propagated throughout the view, and any affected mappings and

maskings are updated automatically. This new approach to storage provisioning decreases

the complexity of all aspects of provisioning. These aspects include performing initial

storage provisioning, adding or removing hosts, changing HBAs, adding or removing

storage devices, and changing FA ports on the Symmetrix.

The following diagram shows an example of an auto-provisioning logical configuration.

Figure 4. EMC Symmetrix Auto-provisioning logical configuration



Auto-provisioning example: provisioning a shared disk to a pureScale Linux test cluster

In this section we show how to create and also to query Auto-provision Groups and the

masking views. The command and example outputs are from running SYMCLI

commands from the Symmetrix storage management host in our test environment. ,

Perform the following steps from a host with SYMCLI installed and connected to the

EMC Symmetrix:

1. Get the serial number ID (SID) of the Symmetrix:

root@emcaix1:/> symcfg list

Sample output is as follows:

S Y M M E T R I X

Mcode Cache Num Phys Num Symm

SymmID Attachment Model Version Size (MB) Devices Devices

000194900547 Local VMAX-1SE 5875 28672 23 649

2. Select the list of devices to add to the storage group by using the symdev list

command. If there are no unassigned hyper volumes available, create new hyper

devices in the back end. For more information, see the brief introduction to

creating Symmetrix hyper devices at the end of this appendix.

3. Create a storage group by using the selected Symmetrix hyper devices. The sample

commands in this step use the hyper devices that are indicated by the 4-digit

hexadecimal values in the following list:

• A 1 GB disk for the DB2 cluster services tiebreaker device - 00C9

• A 100 GB disk for sqllib_shared directory - 00CA

• A 400 GB disk for DB2 data - 00FB

• A 100 GB disk for DB2 logs - 00CB

root> symaccess -sid 547 create -name db2ps_sg -type storage devs C9:CA

root> symaccess -sid 547 add -name db2ps_sg -type storage devs FB

root> symaccess -sid 547 add -name db2ps_sg -type storage devs CB

4. Create a port group that defines the directors to which to map the devices and

through which the host can access the devices as defined in the storage group. The

following sample command is based on our test configuration. We used two FA

ports (7E:1 and 8G:0) on the EMC VMAX that was connected to the SAN fabric,

through which all the hosts in our pureScale cluster were zoned.



root> symaccess -sid 547 create -name db2ps_pg -type port -dirport 7E:1,8G:0

5. To create the host initiator groups, get the WWNs of HBAs on each host that is

zoned with the VMAX. You can get the WWNs in either of the following ways:

• By querying the zone tables on the SAN fabric switch or switches

• By issuing the appropriate commands on the AIX or Linux operating

system on each host in the pureScale cluster:

AIX operating systems:

a) List all the FC adapters:

root> lsdev –Ccadapter | awk '/^fcs/'

b) For each available adapter, list the vital product data (VPD)

information, and parse the network address attribute:

root> lscfg –vl fcs<#> | awk '/Network/'

Linux operating systems:

a) Issue the following systool command against the fc_host class:

root> systool -av -c fc_host | grep -i port_name

b) If the systool command does not work, look in the

/sys/class/fc_host/host#/port_name file, for example:

root> cat /sys/class/fc_host/host*/port_name

6. Check whether all the host initiators that you identified in the previous step are

logged into the Symmetrix FAs by issuing the following command:

root> symaccess -sid SID list logins

7. If you don’t see all the required WWNs logged into the Symmetrix FAs, perform

the following steps:

a. Rediscover the HBAs and devices into the Access Control Logix (ACLX)

by issuing the following command:

root> symaccess discover hba

If HBA rediscovery does not work, you may have to look at the SAN

fabric, FC cabling, or the FC adapter state or settings on the hosts with the

problem.

b. Refresh the Symmetrix device database by issuing the following command:



root> symcfg discover

8. Create a host initiator group for each host in the pureScale cluster that groups the

WWNs of the HBAs that are zoned with the VMAX. For example, in our test

configuration, we had four hosts, each with two HBAs. Hence, we created four

initiator groups, one per host (to group the two HBAs), as follows:

root> symaccess -sid 547 create -name db2ps_coralxib62_ig -type

initiator -wwn 2100001b321d800a

root> symaccess -sid 547 add -name db2ps_coralxib62_ig -type







initiator -wwn 2100001b321d540b


initiator -wwn 2101001b323d540b


initiator -wwn 2100001b3280b327


initiator -wwn 2101001b32a0b327

Because all the hosts in the pureScale cluster will access the same Symmetrix

hyper devices, defining a single parent initiator group over all the host initiator

(cascading initiator) groups simplifies the creation and the management of auto

provision groups.

root> symaccess -sid 547 create -name db2ps_ig -type initiator

-ig db2ps_coralxib62_ig

root> symaccess -sid 547 add -name db2ps_ig -type initiator -ig

db2ps_coralxib63_ig


db2ps_coralxib64_ig


db2ps_coralxib65_ig

Tip: Having a consistent naming convention is helpful when using Auto

Provisioning. For example, we used db2 pureScale cluster name_hostname_ig for host initiator groups and db2 pureScale cluster name_ig for the parent initiator groups in our test environment.

9. Verify that the storage group, director group, and initiator groups were created. For

example in our test environment, the following output was displayed:

root@emcaix1:/> symaccess -sid 547 list




Group Name Type

-------------------------------- ---------

db2ps_ig Initiator

db2ps_coralxib62_ig Initiator




db2ps_pg Port

db2ps_sg Storage

10. Define an auto provision masking view that binds the previously defined storage

group, port group, and (parent) initiator group together.

root > symaccess -sid 547 create view -name db2ps_v -storgrp

db2ps_sg -portgrp db2ps_pg -initgrp db2ps_ig

The creation of the masking view causes the Symmetrix to automatically execute

mapping and masking operations as necessary to make the LUNs available to the

host, using the specified WWNs and FA ports.

On each host in the pureScale cluster, as root user, perform the following steps:

AIX operating systems:

a) Discover configuration changes and refresh the ODM by issuing the following

command:

root> cfgmgr –v

b) Verify that the new EMC LUNs are visible by issuing the following command:

root> lsdev –Ccdisk | awk ‘/EMC/’

Linux operating systems:

a) Initiating a SCSI bus scan on a Linux operating system is tricky and depends

on the kernel version and other distribution specifics. You can rescan the SCSI

bus and present the new LUNs to a Linux host by issuing the following

command for the SCSI hosts that are associated with the FC HBAs on the

server:

echo "- - -" > /sys/class/scsi_host/scsi_host_name/scan

The "- - -" value stands for “c t l”, where c is the channel on the HBA, t is

the SCSI target ID, and l is the LUN.



Because each initiator is defined with a unique SCSI host on Linux operating

systems, you can use the following simple shell script to issue a rescan on all

the initiators that you configured on a server:

root> for h in `ls -1 /sys/class/scsi_host`; do echo "- - -

" > /sys/class/scsi_host/${h}/scan; done

b) If you are using DM-MP, refresh the DM-MP daemon by issuing the following

command:

root> multipath

c) List the newly added EMC LUNs by issuing the following commands:

root> lsscsi | awk ‘/EMC/’

root> multipath –l

Examples of querying the Auto-provision Groups and the masking views

The following examples executed on our test environment show how to query masking

views and the Auto-provision Groups in a view.

1. List all the storage, port ,and initiator groups that you created on the VMAX by

issuing the following command:

root@emcaix1:/> symaccess -sid 547 list


Group Name Type

-------------------------------- ---------

db2ps_ig Initiator





db2ps_pg Port

db2ps_sg Storage

Tip: If you want to look at only one type of an auto-provision group, use the

option –type [storage | port | initiator].

2. List all the auto-provision storage masking views on the VMAX by issuing the

following command:

root@emcaix1:/> symaccess -sid 547 list view




Masking View Name Initiator Group Port Group Storage Group

------------------- ------------------- ------------------- --------------

db2ps_v db2ps_ig db2ps_pg db2ps_sg

3. Display detailed information about an auto provision masking view by issuing the

following command:






Host Initiators

{





}



{

FA-7E:1

FA-8G:0

}


Sym Dev Host


------ ----- ----------------------- ---- ---- -------


08G:0 Not Visible 0


08G:0 Not Visible 1


08G:0 Not Visible 2



-------


Tip: If you want to look at details about the host initiator groups, such as the

WWNs in each group, append the –detail option to the symaccess command. In

addition, you can view details of each auto provision group by using a SYMCLI

command similar to the following one:



root> symaccess -sid SID show auto_provisioning_group_name -

type [storage | port | initiator]

Creating Symmetrix hyper devices

The following process is generally required to create Symmetrix hyper volumes. Steps are

illustrated with sample commands and output, if applicable.

1. List the available Symmetrix devices so that you can choose which Symmetrix

volumes are available to be presented as LUNs to the pureScale cluster hosts:

root> symdev list

You can use the symdev list -inventory command to show a summary of

hyper devices, grouped by device type. For example, on our VMAX, we issued the

following command:

root@emcaix1:/> symdev list -inventory

Symmetrix ID: 000194900547

Device Config FBA CKD3390 CKD3380 AS400 CELERRA

----------------- ----- ------- ------- ----- -------

2-Way Mir 354 N/A N/A N/A N/A

RAID-5 188 N/A N/A N/A N/A

RAID-6 32 N/A N/A N/A N/A

TDEV 20 N/A N/A N/A N/A

VDEV 15 N/A N/A N/A N/A

2. Collect additional information, as follows:

• Identify the available subsystem IDs (SSIDs) on the VMAX by issuing the

following SYMCLI command:

symcfg list -ssid

• Query the disk groups on the backend DAs, grouping by the drive

technology, by issuing the following SYMCLI command:

symdisk list -by_diskgroup

3. Create new Symmetrix hyper devices. The number and size of hyper devices

created depends on the needs of your organization.

For example, we wanted to create the following hyper devices in our pureScale

test cluster:

• A 1 GB disk for the DB2 cluster services tiebreaker device

• A 100 GB disk for the DB2 instance shared home (sqllib_shared)

• A 400 GB disk for DB2 data

• A 100 GB disk for DB2 logs



The following sub-steps illustrate how to create devices, with examples based on

our test cluster:

a. Create a configuration file. We specified the following information in a

configuration file called crtDevs:

# 1GB RAID-1 devs

create dev count=5,

emulation=fba,

size=1 GB,

config=2-Way-Mir,

mvs_ssid=2,

disk_group=1;

# 100GB RAID-5 devs

create dev count=30,

emulation=fba,

size=100 GB,

config=RAID-5,

data_member_count=3,

mvs_ssid=2,

disk_group=2;

Tip: Typically, it’s best to define hyper devices of a few select sizes with

the required protection level (that is, RAID level) to match organizational

needs and to create more devices than you immediately require. Defining

and creating devices in this way helps ensure that you can quickly handle

future requests to add new storage.

b. Validate the syntax and the context of the configuration file and apply the

configuration changes in the file to create the hyper volumes:

root> symconfigure -file crtDevs preview

root> symconfigure -file crtDevs prepare

root> symconfigure -file crtDevs commit

In our test configuration, we created the following hyper volumes by using

the crtDevs configuration file:

• Five 1 GB hyper volumes (00C5: 00C9)

• Thirty 100 GB hyper volumes (00CA: 00FE)

4. Record the new hyper device IDs, since they will be required when you create the

Auto-provision Groups. However, you can always run the symdev list command

to list all the hyper devices in the backend. All unassigned hyper devices are

displayed with ???:? in the SA:P sub-column under Directors in the symdev list command output.



5. In our environment, we had to create a large (400 GB) hyper device for DB2 data.

We did that by forming a concatenated meta device using four of the previously

created RAID-5 hyper devices, with hyper device 00FB forming the meta head:

root> symconfigure –sid 547 -cmd "form meta from dev 00FB,

config=concatenated, count=4;” –noprompt preview


config=concatenated, count=4;” –noprompt prepare


config=concatenated, count=4;” –noprompt commit

Tip: If the meta-members are already striped (like RAID-5), it is best to avoid

another striping level by using a concatenated meta-configuration rather than a

striped meta-configuration.



Appendix B. List of references for more information

• IBM DB2 10.1 Information Center:

http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/index.jsp

• IBM DB2 pureScale Feature Information Center:

http://publib.boulder.ibm.com/infocenter/db2luw/v9r8/index.jsp

• Complete list of storage configurations validated with the DB2 pureScale Feature:

http://publib.boulder.ibm.com/infocenter/db2luw/v9r8/index.jsp?topic=/com.ibm.d

b2.luw.sd.doc/doc/c0059360.html

• Deploy the DB2 pureScale Feature on Linux

http://www.ibm.com/developerworks/data/library/techarticle/dm-

1104purescale/index.html

This is an excellent paper on how to deploy a DB2 pureScale cluster from end-to-

end in a Linux environment.

• DB2 best practices: SAP applications with the DB2 pureScale Feature on SUSE

Linux Enterprise Server and IBM System x:

http://www.ibm.com/developerworks/data/bestpractices/purescalesapsuselinux/

• EMC product and technical documentation:

o Symmetrix CLI Quick Reference Guide

o Symmetrix Array Controls CLI

o Host Connectivity Guide for Linux

You can access these documents from the EMC Powerlink site at

http://powerlink.emc.com.

• Storage Provisioning: Transitioning to EMC Symmetrix VMAX Auto-provisioning

groups

This white paper, available on EMC Powerlink, provides a comprehensive

introduction to the VMAX storage provisioning model using Auto-provisioning

Groups.



Acknowledgements

The authors would like to thank the following people for their contributions in doing a

comprehensive review and providing many helpful comments to improve the quality and

the usefulness of this white paper.

IBM DB2 Development and Quality Assurance:

Alan Y. Lee

Gopikrishnan Varadarajulu

Pandu Mutyala

Information Management Technology Ecosystem:

Yvonne Chan

DB2 Emerging Technologies:

Raanon Reutlinger

Maria N. Schwenger

IBM Reliable Scalable Cluster Technology (RSCT) Development:

Myung Bae

IBM General Parallel File System (GPFS) Development:

Brian Herr

Lei C. Chen

The authors are especially grateful to the DB2 Information Development, IBM

developerWorks, and IBM Legal teams for their help in editing and publishing this paper:

Leslie I. McDonald

Karen Ng

Nancy Miller

Farzana Anwar

Peter K. Wang



Notices Copyright © 2012 EMC Corporation and IBM Corporation.

Printed 2012

The authors believe the information in this publication to be accurate as of its publication date. The

information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” - NEITHER EMC CORPORATION NOR IBM

CORPORATION MAKE ANY REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE

INFORMATION IN THIS PUBLICATION, AND BOTH SPECIFICALLY DISCLAIM IMPLIED WARRANTIES OF

MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

The furnishing of this document does not imply giving license to any IBM or EMC patents.

For the most up-to-date regulatory document for your product line, go to the Technical

Documentation and Advisories section on EMC Powerlink.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

References in this document to IBM products, Programs, or Services do not imply that IBM intends to

make these available in all countries in which IBM operates. Consult your local IBM representative for

information on the products and services currently available in your area. Any reference to an IBM

product, program, or service is not intended to state or imply that only that IBM product, program, or

service may be used. Any functionally equivalent product, program, or service that does not infringe

any IBM intellectual property right may be used instead. However, it is the user's responsibility to

evaluate and verify the operation of any non-IBM product, program, or service.

Use, copying, and distribution of any EMC or IBM software described in this publication requires an

applicable software license. IBM may have patents or pending patent applications covering subject

matter described in this document. The furnishing of this document does not give you any license to

these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing

IBM Corporation

North Castle Drive

Armonk, NY 10504-1785

U.S.A.

The following paragraph does not apply to the United Kingdom or any other country/region where

such provisions are inconsistent with local law: INTERNATIONAL BUSINESS

MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT

WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT

LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT,

MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of

express or implied warranties in certain transactions; therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically

made to the information herein; these changes will be incorporated in new editions of the publication.

IBM may make improvements and/or changes in the product(s) and/or the program(s) described in

this publication at any time without notice.

This document may provide links or references to non-IBM websites and resources. IBM makes no

representations, warranties, or other commitments whatsoever about any non-IBM websites or third-

party resources that may be referenced, accessible from, or linked from this document. A link to a non-

IBM website does not mean that IBM endorses the content or use of such website or its owner. In

addition, IBM is not a party to or responsible for any transactions you may enter into with third parties,

even if you learn of such parties (or use a link to such parties) from an IBM site. Accordingly, you

acknowledge and agree that IBM is not responsible for the availability of such external sites or

resources, and is not responsible or liable for any content, services, products, or other materials on or

available from those sites or resources. Any software provided by third parties is subject to the terms

and conditions of the license that accompanies that software.



IBM may use or distribute any of the information you supply in any way it believes appropriate without

incurring any obligation to you.

Licensees of the DB2 programs referenced in this document who wish to have information about it for

the purpose of enabling: (i) the exchange of information between independently created programs

and other programs (including this one) and (ii) the mutual use of the information that has been

exchanged, should contact:

IBM Canada Limited

Office of the Lab Director

8200 Warden Avenue

Markham, Ontario

L6G 1C7 CANADA

Such information may be available, subject to appropriate terms and conditions, including in some

cases payment of a fee. The licensed program described in this document and all licensed material

available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International

Program License Agreement, or any equivalent agreement between us.

Information concerning non-IBM products was obtained from the suppliers of those products,

their published announcements, or other publicly available sources. IBM has not tested those products

and cannot confirm the accuracy of performance, compatibility, or any other claims related to non-

IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers

of those products.

All statements regarding IBM's future direction or intent are subject to change or withdrawal

without notice, and represent goals and objectives only.

This information may contain examples of data and reports used in daily business operations.

To illustrate them as completely as possible, the examples include the names of individuals,

companies, brands, and products. All of these names are fictitious, and any similarity to the

names and addresses used by an actual business enterprise is entirely coincidental.

If you are viewing this information softcopy, the photographs and color illustrations may not appear.

Trademarks EMC, EMC2, Symmetrix, VMAX, SRDF, PowerPath, Powerlink and Enginuity are either registered

trademarks or trademarks of EMC Corporation in the United States and other countries.

IBM, the IBM logo, AIX, DB2, GPFS, ibm.com, InfoSphere, and z/OS are trademarks or registered

trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. A

current list of IBM trademarks is available on the Web at “Copyright and trademark information” at

www.ibm.com/legal/copytrade.shtml.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

UNIX is a registered trademark of the Open Group in the United States and other countries.

Windows is a trademark of Microsoft Corporation in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Date post:	11-Aug-2020
Category:	Documents
Upload:	others
View:	19 times
Download:	0 times

Deploying DB2 pureScale with EMC Symmetrix...

Documents