+ All Categories
Home > Documents > High Availability for SAP V1.5

High Availability for SAP V1.5

Date post: 21-Jul-2016
Category:
Upload: sachinshukla
View: 80 times
Download: 10 times
Share this document with a friend
Description:
High Availability for SAP V1.5
71
High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 High Availability Solution Design Document for SAP
Transcript
Page 1: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012

High Availability Solution Design Document for SAP

Page 2: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 2 of 71

Preface Edition Notice (November 2012): This document describes an example of a solution design for a high availability implementation for a SAP NetWeaver system. Scope and Audience This document is intended for IBM and Customer personnel involved in the implementation of a high availability solution for SAP on IBM Power Systems. The document is derived from recent experiences in client implementations, and can be used as an example template for a custom implementation document. Author

Edmund Haefele, IBM eBusiness Technical Sales (eTS)

The implemented solution described is based on the framework “Using IBM PowerHA SystemMirror for AIX to implement high availability (HA) and disaster recovery (DR) for SAP applications”, which was developed by:

Katharina Probst, IBM Research and Development

Walter Orb, IBM SAP International Competence Center

The covered solution stack includes

IBM Power Systems using AIX operating system

IBM PowerHA SystemMirror for AIX

SAP System based on SAP NetWeaver 7.x with DB2 UDB LUW database o DB2 high availability disaster recovery (DB2 HADR)

Disclaimer This document is subject to change without notification and will not comprehensively cover all the issues encountered in every customer situation. It should be used only in conjunction with the product literature accompanying the products listed above. The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS.

Page 3: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 3 of 71

Document history Version Date Author Comments

V1.0 2011-09-02 Edmund Haefele Initial version

V1.1 2011-10-07 Edmund Haefele Update on DB2 HADR (Chapter 1.4.2)

V1.2 2011-11-22 Edmund Haefele Updates according to feedback from <CLIENT>

V1.3 2012-03-26 Edmund Haefele Update for AIX FC parameters. Add section about DB2 HADR performance considerations

V1.4 2012-08-07 Edmund Haefele Update chapter 1.4.2.2 (DB2 HADR Start Script considerations), 1.4.2.7 (DB2 HADR Peer-window) and 1.4.4.1 (Description of the high-level recovery procedure)

V1.5 2012-09-01 Edmund Haefele Update chapter 1.4.1.4 (SAP Start Profile settings) for clarification. Add description of timeout behaviour (1.4.2.8).

Page 4: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 4 of 71

Table of Contents

1 INFRASTRUCTURE COMPONENT: HIGH AVAILABILITY .......................................... 7

1.1 INTRODUCTION............................................................................................................................ 7 1.2 DEFINITION OF APPLICATION CATEGORIES ................................................................................... 9

1.2.1 Mapping of the SAP components to the application categories ..................................................... 9 1.3 SOLUTION OVERVIEW ................................................................................................................ 10

1.3.1 General cluster considerations ...................................................................................................... 11 1.3.2 Cluster Type 1: “HADR for DB2 plus takeover for SAP ASCS” ........................................................ 15 1.3.3 Cluster type 2: “Takeover for all DB and SAP components” .......................................................... 18 1.3.4 Comparison for cluster type 1 and cluster type 2 ......................................................................... 21 1.3.5 <CLIENT> decision for the implementation of the cluster types ................................................... 23

1.4 SETUP & CONFIGURATION PRE-REQUISITES ............................................................................... 24 1.4.1 Common PowerHA cluster setup .................................................................................................. 24 1.4.2 Additional configuration steps for cluster type 1 .......................................................................... 42 1.4.3 DB2 HADR performance considerations ....................................................................................... 57 1.4.4 DB2 Log File considerations .......................................................................................................... 60 1.4.5 PowerHA Planning Sheet .............................................................................................................. 65 1.4.6 Interdependencies on other Infrastructure Components .............................................................. 67

1.5 ROLES & RESPONSIBILITIES (SETUP, OPERATIONS) ..................................................................... 68 1.5.1 Considerations for Continuous Availability ................................................................................... 69

Page 5: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 5 of 71

List of Figures Figure 1 HA for SAP WebDispatcher running in a DMZ ....................................................................... 14 Figure 2 HA for SAP WebDispatcher running within the same Security zone ...................................... 14 Figure 3 Overview diagram for cluster type 1 “HADR for DB2 plus takeover for SAP ASCS” ............. 17 Figure 4 Overview diagram for cluster type 2 “One common cluster for DB and SAP” ........................ 19 Figure 5 Overview diagram for cluster type 2 (SAP BW only) “Takeover for all DB and SAP components” ......................................................................................................................................... 19 Figure 6 Storage layout considerations for split mirror backup ............................................................. 28 Figure 7 Resource Groups for the PowerHA cluster ............................................................................ 31 Figure 8 PowerHA cluster NFS mounts – resource group offline ......................................................... 33 Figure 9 PowerHA cluster NFS mounts – resource group online, standard operation ......................... 33 Figure 10 PowerHA cluster NFS mounts – resource group online, takeover operation ....................... 34 Figure 11 State transition for the standby database and DB2 HADR takeover command arguments . 43 Figure 12 The different states of the DB2 HADR standby database .................................................... 48 Figure 13 HADR timeout illustration (HADR_PEER_WINDOW = 0) .................................................... 52 Figure 14 HADR timeout illustration (HADR_PEER_WINDOW > 0) .................................................... 53 Figure 15 DB2 Registry DB2_HADR_PEER_WAIT_LIMIT (HADR_PEER_WINDOW = 0) ................. 55 Figure 16 AIX LVM mirror of DB2 online logs – normal situation ......................................................... 61 Figure 17 Crash of <STORAGE SYSTEM>-1 ...................................................................................... 62 Figure 18 Manual activation .................................................................................................................. 63

Page 6: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 6 of 71

List of Tables Table 1 High Availability patterns ............................................................................................................ 7 Table 2 Key non-functional requirements for the three application categories ....................................... 9 Table 3 Mapping of the SAP components to the application categories ................................................ 9 Table 4 Software Stack for the takeover cluster ................................................................................... 11 Table 5 Comparison between cluster type 1 and type 2 for different failure scenarios ........................ 21 Table 6 Mapping between SAP Systems and the cluster types ........................................................... 23 Table 7 Entries for PowerHA in /etc/services ........................................................................................ 25 Table 8 Local AIX filesystems ............................................................................................................... 25 Table 9 AIX volume groups layout ........................................................................................................ 26 Table 10 Filesystem layout for SAP and DB2 UDB database .............................................................. 27 Table 11 PowerHA cluster name .......................................................................................................... 29 Table 12 PowerHA node names ........................................................................................................... 29 Table 13 PowerHA networks................................................................................................................. 29 Table 14 Network interfaces and Service IP addresses ....................................................................... 29 Table 15 Resource Group attributes in PowerHA ................................................................................. 30 Table 16 Power HA Resource Groups for Database and SAP ............................................................. 31 Table 17 PowerHA application server ................................................................................................... 34 Table 18 Base settings for PowerHA application monitors ................................................................... 36 Table 19 PowerHA application monitor definitons ................................................................................ 36 Table 20 Conventions for SAP Instance numbers ................................................................................ 38 Table 21 Additional TCP/IP communication ports used in the SAP systems ....................................... 38 Table 22 Java Instance Parameters ..................................................................................................... 40 Table 23 Parameterization for the DB2 scripts ..................................................................................... 42 Table 24 Return Codes from rc.hadr.monitor ....................................................................................... 42 Table 25 DB2 Takeover command executed on standby (w/o “BY FORCE” clause) .......................... 49 Table 26 DB2 Takeover command executed on standby using the “BY FORCE” clause .................... 50 Table 27 Example for Cluster Configuration (<SID>) ........................................................................... 66 Table 28 Dependencies of HA to other infrastructure components ...................................................... 67 Table 29 Roles & responsibilities for Infrastructure Component High Availability ................................ 68 Table 30 Regular maintenance topics .................................................................................................. 70

Page 7: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 7 of 71

1 Infrastructure Component: High Availability

1.1 Introduction

The high availability solution helps to ensure that the failure of any component of the solution does not cause the SAP application and its data to be inaccessible to the business users. To achieve high availability for the most critical business processes all the supporting components need to be identified in a first step. Each of these components then is a candidates for protection through a HA solution. Within the individual high availability solutions any single points of failure (SPOF) need to be eliminated through appropriate design, planning, selection of hardware, configuration of software, and carefully controlled change management discipline. Eliminating SPOFs for a single SAP system includes deployment of

Redundant hardware components (Server, storage system, network, etc.)

Internal protection for each storage system (RAID technology)

Redundant communication paths to the system (network components and network routes

Redundant access paths to the data (multi-path driver, redundant storage area network)

Redundant middleware components (multiple load balancers, multiple web servers, etc.)

Multiple SAP application servers and strict implementation of SAP logon groups

Failover cluster solutions for SAP database system and SAP central services

This chapter has main focus on the failover cluster solution for the SAP database system and SAP Central Services. The design presented here covers all needs in respect of availability, investment, reliability, ease of use and operation and last but not least manageability for the SAP systems. The design takes into account that some of the information at the current stage is still estimated and some decisions are still open. However, design adoptions on special needs can be made easily and may then be reflected in the detailed design documents of each individual SAP system. Table 1 gives a high-level overview about the two different HA patterns discussed in this chapter:

Pattern Name

High Level Description

Cluster Type 1

HADR plus Takeover for SAP ASCS / ERS Fast failover for the database due to DB2 HADR technology: Automatic launch of SAP Central Services to a takeover LPAR. Takeover is managed by cluster software PowerHA. Failover time in the order of magnitude of five minutes. HA application start/stop scripts are able to handle ABAP, Java and Dual-Stack systems, Deployment of SAP Replicated Enqueue server.

Cluster Type 2

Takeover for all DB and SAP components Simpler cluster design (simple failover combined with LVM mirroring). Automatic launch of DB and SAP Central Services to a takeover LPAR. Takeover is managed by cluster software PowerHA. Slightly higher failover times (depending on the load about 10 – 40 min). HA application start/stop scripts are able to handle ABAP, Java and Dual-Stack systems, Deployment of SAP Replicated Enqueue server.

Table 1 High Availability patterns

Page 8: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 8 of 71

Both cluster solutions are based on IBM PowerHA SystemMirror for AIX V6.1 Standard Edition (PowerHA). Each of the cluster patterns is built as a two node PowerHA cluster. The HA clusters protects the SAP key components database (DB), Central Services (SCS) and the NFS service. SAP Application Servers are not considered as SPOFs: They’re installed redundantly on the IBM Power blades outside the cluster nodes. From an end-to-end view, the business process will not depend solely on the SAP systems, but may include additional components, like e.g.

Job-Scheduling

(e.g. critical interface or data processing jobs need to be scheduled timely and with a given

periodicity)

Backup/ Restore Infrastructure

(e.g. for the backup of database archive logs)

User-Authentication

(e.g. LDAP)

Even though workarounds may be available for handling the outage of a single of these additional components in the whole process chain, it may be reasonable to invest in a HA solution for it also. An impact to the business process may already appear when one of these components is unavailable for a certain small time window. It is advised to include all critical components in the end-to-end HA concept, e.g. implement additional takeover clusters to ensure availability for job scheduling, Net-backup servers, etc. Additional requirements for the robustness of the overall HA solution, especially related to facilities and the underlying IT infrastructure e.g. power supply, cooling, cabling, SAN, network, etc. need to be considered in addition.

Page 9: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 9 of 71

1.2 Definition of Application Categories

<CLIENT> has defined three different application categories for the SAP systems. Table 2 lists the key NFRs for the three different application categories

Category 1 Category 2 Category 3

Simple (hardware) failure Leverage HA cluster solution / HA cluster takeover and restart time [minutes]

< 5 < 15 < 25

Full database restore and recovery [hours]

1 1 4

Local disaster Disaster recovery [hours]

2 2 2

Handle Logical errors within the system [hours]

12 12 12

Table 2 Key non-functional requirements for the three application categories

1.2.1 Mapping of the SAP components to the application categories

Table 3 contains the mapping between the SAP components to these application categories. The most critical components are SAP BPP and the SAP ECC systems.

Category 1 Category 2 Category 3

ECC (ABAP)

PI (Dual-Stack)

(own SLD, replicated from central)

BW (ABAP)

BPP (ABAP)

BPM (Java)

SolMan (Dual Stack)

CE

(Java) SLD (Central)

(Java)

EP

(Java)

CPS

CUA

(ABAP)

Table 3 Mapping of the SAP components to the application categories

Page 10: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 10 of 71

1.3 Solution Overview

Based on the definitions of the key KPIs from <CLIENT> two different cluster types (combinations of different cluster technologies) are implemented. The items below highlight the basic cornerstones for HA design valid for both cluster types: POWER Systems For the DB2 database and the SAP Central Services <CLIENT> are deployed IBM Power Systems 780 and IBM Power Systems 750 servers. The LPARs on these systems are created with dedicated CPU cores and with dedicated I/O adapters: I/O adapters for each LPAR are redundant (multiple network adapters, multiple fibre channel adapters). The redundant adapters are located in different drawers. No PowerVM virtualization is implemented. The SAP Application Servers are deployed on IBM Power Blades. AIX operating system

The AIX operating system is mirrored to redundant local disks (AIX LVM mirroring for “rootvg”)

All Ethernet network connections are setup redundant. (EtherChannel, Link Aggregation for higher bandwidth and path failover)

The SAP/ DB2 administration users, interface users, personal users need to be implemented with same UID and GID on all LPARs of the SAP environment (PowerHA cluster nodes, all SAP Application servers on the Power blades.)

SAN and Storage

SAN-attached <STORAGE SYSTEM> storage is used for all SAP systems.

All SAN connections between the LPARs and the <STORAGE SYSTEM> systems must be fully redundant and must provide sufficient bandwidth: The <STORAGE SYSTEM> systems are attached via a redundant SAN (two fully redundant fabrics) to all LPARs on the managed systems

OS based mirroring (AIX LVM) is implemented for all SAP and some of the database filesystems. Data is mirrored to two different <STORAGE SYSTEM> storage systems.

PowerHA Clusters

All HA clusters are setup as two-node clusters. Each cluster node of the PowerHA cluster is deployed on redundant server hardware.

The two cluster nodes have same resources for CPU amount and memory configuration. Each node must have sufficient resources to handle the eventually additional application workload in case of a takeover situation.

If a production system is deployed in a HA cluster setup, then the pre-production system should be deployed on the same OS platform and as HA cluster also. Background for clustering of the pre-production system is not based on its high-availability requirements, but on testing considerations: Test of new software stacks or configuration changes should always be tested in a non-production environment first before they’re applied to the production systems.

All AIX volume groups for the shared disks are created in “Enhanced Concurrent Mode” (ECM). This enables fast disk takeover and allows using the VGs for the disk heartbeat.

IPAT (IP Address Takeover) using IP aliasing is used in PowerHA to protect the service IP addresses.

Each node has a persistent IP address (an IP alias address bound to the LPAR and always available while the node is up) defined. The persistent IP address corresponds to the “physical hostname” of the LPAR. The additional service IP addresses (“virtual hostnames”) are controlled by PowerHA and are associated to PowerHA resource groups.

SAP

The SAP Replicated Enqueue Server is implemented

Two clusters are deployed: One for the SAP Database, and one for the SAP Central Services

Page 11: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 11 of 71

Additional SAP Application servers are deployed in further IBM POWER blades outside the PowerHA cluster

Only the SPOFs of a SAP system (Database, SAP Central Services) are controlled by PowerHA. The management of the SAP application servers: Start-/Stop of the SAP application servers on the POWER blades is beyond the scope of the HA cluster and is managed by the SAP basis team.

There is a shared responsibility between the IT infrastructure team and the SAP basis team for start and stop of database and central services in the cluster. See chapter ‎1.6 “Roles & Responsibilities (setup, operations)” for details.

Table 4 lists the software stack of the two takeover clusters. Both cluster types are based on the same software components:

Component SW Version

Operating System AIX 6.1 TL6 SP5 AIX 6.1 TL7 SP4 with IV22062

Cluster software PowerHA 6.1 SP5 HACMP 6.1 SP08 with ifix IV15812, IV17089

DB2 UDB DB DB2 V9.5 DB2 V9.7

SAP Kernel SAP Kernel 7.11 SAP Kernel 7.20

Table 4 Software Stack for the takeover cluster

1.3.1 General cluster considerations

Main goal of the HA solution is to make the SAP application highly available for the end-users and 3rd

party systems. The solution design needs to take into account all the different layers in the IT infrastructure. In this chapter general aspects in the context of AIX, Power HA and SAP are presented in more detail.

1.3.1.1 PowerHA

Two main areas need to be specified during the definition of the PowerHA cluster 1.3.1.1.1 CLUSTER TOPOLOGY

The cluster topology describes the realization of the cluster on the LPARs. It includes the definition of the cluster, describes the cluster nodes and the communication paths between the nodes. The cluster topology is purely in the responsibility of the IT infrastructure team. 1.3.1.1.2 CLUSTER RESOURCES

The cluster resources describe the mapping of the application to the takeover units (resource groups). The cluster resources are a shared responsibility as they include both attributes from the IT infrastructure team and the SAP basis team. From an IT infrastructure perspective cluster resources include

Virtual hostnames (as Service IP addresses in the resource group)

AIX Volume Groups and File systems

NFS File systems From an application (SAP Basis perspective) Cluster Resources include

Start-and Stop of the DB2 database and SAP application

Page 12: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 12 of 71

The management of dependencies between the application components and their mapping to resource group dependencies

Prerequisites that allow to operate the application components on both cluster nodes

1.3.1.2 SAP

1.3.1.2.1 SAP APPLICATION SERVER LAYOUT

For the HA design the previous existing central instance of each SAP system is split into Central Services instances (ASCS, SCS) and the “primary SAP Application server“. To protect the Enqueue data, the SAP Enqueue Replication Server is deployed on the takeover node. The SAP tool “sapcpe” is used to check (and eventually automatically copy) the SAP kernel programs during SAP instance startup from the “shared” executable directory (/usr/sap/<SID>/SYS/exe/run/DIR_CT_RUN) locally to each SAP instance directory. For the SAP Central Service (and ERS) the SAP instance directory is located in the shared VG and is part of the PowerHA resource groups. The advantage of the “sapcpe” implementation is that the application servers always have a local copy of the SAP kernel programs and are not dependent on an NFS mounted directory for their executables. The situation of an unavailable NFS server is handled and implemented in stop scripts ensuring that the SAP instances (ASCS, SCS) can be stopped normally. 1.3.1.2.2 SAP REPLICATED ENQUEUE SERVER

The Enqueue service handles the locks in the application servers. In the event that the service fails and locks are not replicated, running processes in the SAP application servers will still rely on the old locks which are now lost due to the failure. To maintain integrity, the SAP application server restarts the work process as soon as it detects such a situation. By using the enqueue replication server (ERS) the enqueue locks are “mirrored” to a second node: The lock table can be rebuild during failover of the Enqueue Service and the integrity of the locks can be assured under all circumstances. There is no need to restart the SAP work processes on the SAP application servers: Additional impact and downtime for the users is avoided. SAP provides different options to realize the handling for the SAP Replicated Enqueue Server in a HA cluster:

“Local” Installation of SAP ERS on all cluster nodes and implementation of the “polling” mechanism: The SAP ERS will be active on that node on which the enqueue server is not active.

Installation of SAP ERS in a shared Resource Group. Implementation of resource group dependencies (Anti-collocation: “Online on different nodes only”).between the RG for ERS and the RG for SCS. PowerHA handles the RG intended behavior fully automatic.

Installation of SAP ERS in a shared Resource Group managed by PowerHA. Implementation of the dependencies between the RG for ERS and RG for SCS within the PowerHA application server scripts.

In the implemented solution the SAP ERS is deployed in a shared Resource Group. Management of the dependencies and management of the timing behavior between the SAP Enqueue Server and the SAP Replicated Enqueue Server are handled within the PowerHA application scripts. This provides the required functionality, and scripting effort is minimal as this functionality is already included in the SAP application script templates provided by the ISICC. 1.3.1.2.3 SAP START PROFILE CONSIDERATIONS FOR SCS, ASCS AND ERS

For the Enqueue Server (EN) and the Enqueue Replication Server (ERS), the „Restart“ option needs to be disabled in the SAP Start Profile:

Page 13: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 13 of 71

In case the EN fails, the appropriate action is to initiate a takeover to a node which has a replica of the enqueue locks in memory: This table was prepared by the ERS running on the takeover node. The PowerHA application monitor will initiate the takeover of the EN.

When the EN is acquired on the node, the ERS is stopped during the startup procedure of the EN. When a second Cluster Node is available, the PowerHA application monitor of the ERS will move the ERS to the 2nd node and start it there: The protection for the EN is restored.

Both actions need to be controlled by PowerHA only, so a restart out of „sapstart“ could interfere with this logic.

The Message Server (MS) is stateless and can be restarted locally without causing any data loss. The „Restart“ option needs to be enabled in the SAP Start Profile for the message server.

The PowerHA Application Monitor of the SCS/ ASCS doesn’t probe for the MS status, so a failure of the MS will not trigger a takeover/ notification.

1.3.1.2.4 SAP SOLUTION MANAGER DIAGNOSTICS AGENT (SMDA)

A „saphostagent“ is installed locally on each AIX operating system image (LPAR) that hosts SAP components. The „saphostagent“ will not be taken over by the PowerHA cluster, it is not part of any of the PowerHA Resource Groups. An own SMDA instance is installed for each IP address/ virtual IP address to which a SAP component is assigned to:

(primary) Database

SAP Central Services

SAP Application Servers (Blade) No SMDA will be deployed for the ERS and the standby DB (DB2 HADR). For the DB2 (primary) database and the SAP Central Services the SMDA instance is part of the PowerHA resource group (rg_<sid>_db and rg_<sid>scs). The start and stop of SMDA needs to be included in the application start/stop scripts. 1.3.1.2.5 SAP SPOOL CONSIDERATIONS

The print output devices in the SAP system need to be grouped: This allows for a high-available definition and removes the SAP spool service as single point of failure. These definitions will need to be done on the SAP Application servers outside of the PowerHA cluster.

Each logical spool server in the SAP (ABAP) system needs to be defined with a primary and an additional alternate spool server.

The primary spool server and the alternate spool server need to be hosted on different SAP application servers; in addition they should be deployed to different POWER blades.

If the primary spool server is down, the print jobs are automatically processed by the alternate server. If UNIX printing is used, both LPARs hosting SAP instances for the primary and the alternate spool server need to have both all required print queue definitions.

1.3.1.2.6 JAVA CONSIDERATIONS

In a Java environment, the SAP WebDispatcher may need to be installed and made highly available too. There are two different implementation scenarios:

Web-Dispatcher is running in a DMZ

Page 14: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 14 of 71

From a networking perspective the Web Dispatcher needs to be fully isolated from the SAP system. An own two node Power HA cluster needs to be deployed in the DMZ.

Figure 1 HA for SAP WebDispatcher running in a DMZ

WEB Dispatcher is running in the same Security zone

If the Web Dispatcher is running in the same Security Zone as the SAP System, it can be integrated into the same PowerHA cluster. For simplicity the SAP WebDispatcher can be grouped together with the PowerHA resource group for the SAP Central Services.

Figure 2 HA for SAP WebDispatcher running within the same Security zone

1.3.1.2.7 DUAL-STACK CONSIDERATIONS

For dual-stack systems both central services instances for ABAP and JAVA (ASCS and SCS) are grouped together in one single resource group. Also, the two ERS for ABAP and JAVA are grouped together. As both instances are handled during the takeover, the overall takeover time will be increased. Some parameters of the Java instance need to be adjusted (See SAP Note # 1121900). See also chapter 1.4.1.4 for the discussion of these parameters.

Page 15: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 15 of 71

1.3.1.2.8 POWERHA RESOURCE GROUP SUMMARY

Given the considerations above, the following rules apply for the implementation of the PowerHA resource groups (switchover units):

SAP Database instance and Central Services Instance (ASCS/ SCS) are located in different PowerHA clusters/ resource groups

o Central services switchover time is very fast (typically less than a minute) o Database instance has a higher switchover time due to the processing of open

transaction logs during crash recovery. o Decoupling of both components

The Central services instances for Dual-Stack systems (ASCS and SCS) can be grouped together in one single resource group

SAP enqueue replication server o The SAP ERS is installed in a shared Resource Group o If two nodes are available, then ERS RG gets anti-collocated to SCS resource. This

behaviour is controlled by the PowerHA application monitor scripts

The NFS server for the shared SAP filesystems (SAPMNT, SAPTRANS) and interface filesystems is located in its own resource group.

SAP Application servers are not placed into PowerHA resource groups and are managed outside of PowerHA

1.3.2 Cluster Type 1: “HADR for DB2 plus takeover for SAP ASCS”

The section gives a high-level summary for the HA design of the “DB2 HADR” takeover cluster. Two HA clusters will be deployed: A cluster for the DB2 database (“DB2 HADR” cluster) plus a second cluster for the takeover of the SAP Central Services.

1.3.2.1 DB2 HADR Overview

HADR is a DB2 replication feature to make the database server highly available (shared nothing approach). In the HADR scenario two separate DB2 database servers are deployed: a primary and a standby database server. Both database servers have their own storage and are up and running. One server has the role of the primary server. All clients (SAP application servers) connect to this server. The transactions of the client connections are written to the database logs. The log data is transferred via TCP/IP to the standby database server: The standby server updates the local database by rolling forward the transferred log records. This continuous roll forward operation ensures that the standby server keeps in sync with the primary server. HADR is a replication feature only. There is no failure detection and no automation facilities for the takeover activation of the primary. To achieve an automated failover process the two database servers are monitored by PowerHA: In case of a crash of the primary database server, PowerHA initiates the HADR takeover on the standby server and ensures that the virtual IP address is assigned to the new primary server also. If the primary server crashes, the takeover activation of the standby takes only a short time since the second database server is already running and the database is in a consistent state. 1.3.2.1.1 DB2 HADR SYNCHRONIZATION MODES

Three different synchronization modes are available for DB2 HADR. The synchronization algorithm has some impact on performance dependent on the synchronization mode:

Page 16: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 16 of 71

SYNC (synchronous) This mode provides the greatest protection against transaction loss, and using it results in the longest transaction response time among the three modes. In this mode, log writes are considered successful only if logs have been written to log files on the primary database and if the primary database has received acknowledgment from the standby database that the logs have also been written to log files on the standby database. The log data is guaranteed to be stored at both sites.

NEARSYNC (near synchronous) While this mode has a shorter transaction response time than the synchronous mode, it also provides slightly less protection against transaction loss. In this mode, log writes are considered successful only if the log records have been written to the log files on the primary database and if the primary database has received acknowledgment from the standby system that the logs have also been written to the main memory on the standby system. Data is only lost if both sites fail simultaneously and if the target site has not transferred all of the log data that it has received to non-volatile storage.

ASYNC (asynchronous) This mode has the highest chance of transaction loss if the primary system fails. It also has the shortest transaction response time among the three modes. In this mode, log writes are considered successful only if the log records have been written to the log files on the primary database and have been delivered to the TCP layer of the primary system's host machine. Since the primary system does not wait for acknowledgment from the standby system, transactions might be considered committed when they are still on their way to the standby server.

Recommendation is to use NEARSYNC as HADR synchronization mode for the SAP clusters.

1.3.2.2 Restrictions for HADR Setup

The following restrictions apply for the DB2 HADR setup

HADR is not supported in a partitioned database environment.

The primary and standby databases must have same operating system version and the same version of the DB2 database system, except for a short time period during a rolling upgrade.

The DB2 database system release on the primary and standby databases must be of the same bit size (32 or 64 bit).

Log archiving can only be performed by the current primary database.

Self-Tuning Memory Manager (STMM) can only be implemented on the current primary database.

Backup operations are only supported on the primary database. Backup operations are not supported on the standby database.

Non-logged operations, such as changes to database configuration parameters and to the recovery history file, are not replicated to the standby database.

Load operations with the COPY NO option specified are not supported.

HADR does not support the use of raw I/O (direct disk access) for database log files. If HADR is started via the START HADR command or the database is activated (restarted) with HADR configured and raw logs are detected, the associated command fails.

Page 17: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 17 of 71

1.3.2.3 PowerHA cluster setup for DB2 HADR

The takeover logic to perform the state change is imbedded within the PowerHA high availability cluster to provide for an unattended HADR failover. The PowerHA integration for DB2 HADR is based on the start/ stop scripts which are delivered as samples part of the DB2 software image (<DB2 instdir>/sqllib/samples/hacmp/rc.hadr*).

PowerHA Cluster

PowerHA Cluster

LPAR #3

takeover

takeover

LPAR #1

SAP Central Services

ASCS10 / SMDA95

NFS Server

LPAR #2

SAP ERS

ERS19

LPAR #4

takeover

saphostagent saphostagent

saphostagentsaphostagent

DB2 UDB

(primary)

DB2 UDB

(standby)

takeoverSMDA96

Storage 2Storage 1

DB2 HADR

LVM mirroring

(for ASCS, NFS, SMDA*)

*) SAP ABAP system. Some slight modifications for JAVA or DualStack Apply

*)

*)

Figure 3 Overview diagram for cluster type 1 “HADR for DB2 plus takeover for SAP ASCS”

By using those scripts, PowerHA will manage only the takeover: The DB Instances need to be started manually before the PowerHA cluster start. Future scripts developed by ISICC may include the DB2 start/ stop for primary and standby. Required Start Sequence:

Start SAP database o Start the Standby DB on Node B o Start the Primary DB on Node A o Check for Peer State between o Start the PowerHA Cluster for the database

Start SAP Central Services o Start the PowerHA Cluster for the SAP Central services

(and Enqueue Replication server)

Start the SAP Application servers on the POWER blades Resource Groups:

Database Cluster

One PowerHA Resource Group for managing the DB2 HADR takeover. During normal operation node 1 hosts the primary database; node 2 hosts the standby database. All DB2 database filesystems are local on both cluster nodes (same layout, same size) except the log

Page 18: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 18 of 71

archive directory (which is shared)

o The resource group contains the virtual IP address for the SAP database connection („SAPDBHOST“) as well as the SMDA agent for the database. The log archive directory is also part of the resource group.

o The PowerHA application server manages the activation of the primary database:

During a takeover, the standby database on the second node is activated as new primary.

SAP Cluster

One PowerHA resource group manages the SAP Central Services This resource group contains the virtual IP address for the SAP client connection via LOGON groups („Message Server“) as well as the SMDA agent for the database. During normal operation, the resource group is active on node 1. In case of a takeover, the resource group is activated on node 2.

One PowerHA resource group manages the NFS server. This NFS filesystems are exported to all SAP application servers. During normal operation, the resource group is active on node 1. In case of a takeover, the resource group is activated on node 2.

One PowerHA resource group manages the ERS. As long as two cluster nodes are available, the ERS is anti-collocated to the SAP Central Services: During normal operation, the resource group is active on node 1. In case of a takeover of the SCS to node 2, the resource group of the ERS stays active on node 2 unless a new, free cluster node is available: If a new node is available, the ERS will be moved to that node. This anti-collocation is not a resource group dependency, but is managed out of the logic of the PowerHA application monitoring script for the ERS.

1.3.2.4 PowerHA cluster scripts

1.3.2.4.1 MANAGEMENT OF TAKEOVER FOR DB2 HADR

As application scripts, the script package delivered as sample with the DB2 UDB software is used. (/db2/<SID>/db2<sid>/sqllib/samples/hacmp/rc.hadr.*). The application start, and stop scripts are deployed to each cluster locally to each node of the cluster. Currently, a script package for DB2 HADR is under development in the ISICC (International SAP IBM Competence Center): When this package gets available, this may replace the DB2 UDB sample package later on. 1.3.2.4.2 MANAGEMENT OF TAKEOVER FOR SAP

The cluster scripts for the integration of the start/stop of the SAP systems are based on templates developed by the ISICC (International SAP IBM Competence Center). The application scripts are deployed to each cluster locally. The scripts are copied to both cluster nodes during the cluster setup, but there will be no automatic distributions for the scripts to the HA clusters later on. Potential changes to the scripts are managed within the change management process and policies for each SAP landscape individually. As a long term solution the out-of-the-box integration for SAP in PowerHA (“Smart Assist for SAP” feature) may be used for the future: “Smart Assist for SAP” is not available for PowerHA 6.1, but is only supported for higher PowerHA versions (PowerHA 7.1).

1.3.3 Cluster type 2: “Takeover for all DB and SAP components”

The takeover logic to perform the state change is imbedded within the PowerHA high availability cluster to provide for an unattended of both DB2 database and SAP components. The cluster type 2 is realized with two different characteristics:

Figure 4: One common takeover cluster for both takeover of DB2 and SAP components

Page 19: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 19 of 71

Figure 5, SAP BW System only: Two separate clusters for takeover of DB2 and takeover of the SAP components

PowerHA Cluster

takeover

takeover

LPAR #1

SAP Central Services

ASCS10 / SMDA95

NFS Server

LPAR #2

SAP ERS

ERS19

takeover

saphostagent saphostagent

Storage 2Storage 1

LVM mirroring

(for all components

including DB)

DB2 UDB

takeoverSMDA96

*) SAP ABAP system. Some slight modifications for JAVA or DualStack Apply

*)

*)

Figure 4 Overview diagram for cluster type 2 “One common cluster for DB and SAP”

PowerHA Cluster

PowerHA Cluster

LPAR #3

takeover

takeover

LPAR #1

SAP Central Services

ASCS10 / SMDA95

NFS Server

LPAR #2

SAP ERS

ERS19

LPAR #4

takeover

saphostagent saphostagent

saphostagentsaphostagent

Storage 2Storage 1

LVM mirroring

(for all components

including DB)

DB2 UDB

takeoverSMDA96

*) SAP ABAP system. Some slight modifications for JAVA or DualStack Apply

*)

*)

Figure 5 Overview diagram for cluster type 2 (SAP BW only) “Takeover for all DB and SAP components”

The cluster scripts for integration of the start/stop of the SAP systems are based on templates developed by the ISICC (International SAP IBM Competence Center). By using those scripts, PowerHA will manage database and SAP start as well as database and SAP takeover.

Page 20: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 20 of 71

Required Start Sequence:

Start SAP database o Start the PowerHA Cluster for the database

Start SAP Central Services o Start the PowerHA Cluster for the SAP Central services

(and Enqueue Replication server)

Start the SAP Application servers on the POWER blades Resource Groups:

Database

One PowerHA Resource Group for managing the DB2 database This resource group contains the virtual IP address for the SAP database connection („SAPDBHOST“) as well as the SMDA agent for the database. All database filesystems are located in a shared volume group and are so accessible for both cluster nodes. During normal operation, the resource group is active on node 1. In case of a takeover, the resource group is activated on node 2.

SAP

One PowerHA resource group manages the SAP Central Services This resource group contains the virtual IP address for the SAP client connection via LOGON groups („Message Server“) as well as the SMDA agent for the database. During normal operation, the resource group is active on node 1. In case of a takeover, the resource group is activated on node 2.

NFS

One PowerHA resource group manages the NFS server. This NFS filesystems are exported to all SAP application servers. During normal operation, the resource group is active on node 1. In case of a takeover, the resource group is activated on node 2.

Enqueue Replication Server

One PowerHA resource group manages the ERS. As long as two cluster nodes are available, the ERS is anti-collocated to the SAP Central Services: During normal operation, the resource group is active on node 1. In case of a takeover of the SCS to node 2, the resource group of the ERS stays active on node 2 unless a new, free cluster node is available: If a new node is available, the ERS will be moved to that node. This anti-collocation is not a resource group dependency, but is managed out of the logic of the PowerHA application monitoring scripts for the ERS.

1.3.3.1.1 MANAGEMENT OF TAKEOVER FOR BOTH DATABASE AND SAP

The cluster scripts for the integration of the start/stop of the SAP systems are based on templates developed by the ISICC (International SAP IBM Competence Center). The application scripts are deployed to each cluster locally. The scripts are copied to both cluster nodes during the cluster setup, but there will be no automatic distributions for the scripts to the HA clusters later on. Potential changes to the scripts are managed within the change management process and policies for each SAP landscape individually. As a long term solution the out-of-the-box integration for SAP in PowerHA (“Smart Assist for SAP” feature) may be used for the future: “Smart Assist for SAP” is not available for PowerHA 6.1, but is only supported for higher PowerHA versions (PowerHA 7.1).

Page 21: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 21 of 71

1.3.4 Comparison for cluster type 1 and cluster type 2

Table 5 shows a comparison for the coverage against certain failure scenarios for the two different cluster types. Both clusters protect versus a component or service failure. Cluster type 1 will ensure a faster failover for the database during those events. Cluster type 1 has advantages in certain failure scenarios for Storage/ SAN failures and block corruptions/ technical data integrity issues. Both cluster types don’t protect against a physical disaster or logical errors: To protect the SAP systems against those types, additional D/R protection must be taken into account.

Cat Error Category Error scenario Cluster Type 1 Cluster Type 2

1 Component or service failure

server or LPAR failure (DB, CS) y y

process/ service failure (DB, CS) y y

ERS failure y y

failure of AS / J2EE engine y y

database hanging (archiver stuck, deadlock)

admin admin

2 Storage/ SAN failure

single disk failure y y

failure of more than 1 disk (raid group failure)

y y (AIX LVM)1

failure of or failure within 1 storage array

y y (AIX LVM)1

failure of both storage systems <STORAGE SYSTEM>-1 and <STORAGE SYSTEM>-2

DR DR

failure of storage connectivity to one server

y y (AIX LVM)

bug in storage software y DR

3 Block Corruptions/ Technical Data Integrity Issue

DB block corruptions y DR

corruption in LVM y DR

deletion of volume groups, logical volume or filesystem (DB files)

y DR

deletion of volume groups, logical volume or filesystem (non-DB files)

n n

File corruptions (other than DB) n n

4 Physical Disaster

primary data center is inoperable DR DR

power failure ? ?

network failure ? ?

5 Logical Errors/ Data Inconsistencies

deletion of a single file, deletion of a DB table, application data has been overwritten or falsified, wrong transports have been imported

n n

data inconsistencies between systems

n n

Table 5 Comparison between cluster type 1 and type 2 for different failure scenarios

Protecting the data is one base foundation for High Availability. To cover storage and SAN failures, data mirroring on the two different <STORAGE SYSTEM> systems is required. In the following, the data protection rules are summarized:

1 <CLIENT> hasn’t implemented AIX LVM for Cluster Type 2 at the moment. Implementation of AIX LVM is however still recommended to protect against certain storage failure scenarios.

Page 22: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 22 of 71

For Type 1 clusters SAP Cluster: The data needs to be mirrored with AIX LVM

AIX LVM provides the capability to write the data synchronously using two copysets

AIX LVM mirroring: One copyset on <STORAGE SYSTEM>-1 (Site A), and another on <STORAGE SYSTEM>-2 (Site B) for all filesystems

DB HADR Cluster:

Primary DB2 Database on <STORAGE SYSTEM>-1 (not-mirrored by AIX LVM)

Standby Database on <STORAGE SYSTEM>-2 (not mirrored by AIX LVM)

AIX LVM Mirroring for online log filesystems

AIX LVM Mirroring for archive log filesystem For Type 2 clusters:

SAP Cluster: The data should be mirrored with AIX LVM

AIX LVM provides the capability to write the data synchronously using two copysets

AIX LVM mirroring: One copyset on <STORAGE SYSTEM>-1 (Site A), and another on <STORAGE SYSTEM>-2 (Site B) for all filesystems

DB takeover Cluster: The data should be mirrored with AIX LVM

AIX LVM provides the capability to write the data synchronously using two copysets

AIX LVM mirroring: One copyset on <STORAGE SYSTEM>-1 (Site A), and another on <STORAGE SYSTEM>-2 (Site B) for all filesystems

Page 23: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 23 of 71

1.3.5 <CLIENT> decision for the implementation of the cluster types

Table 6 summarizes the intended cluster types for the different SAP systems. Only the SAP ECC and SAP BPP systems will be implemented with DB2 HADR. All other SAP systems are implemented as “standard” takeover clusters.

Category SAP System Cluster Type KPIs met?

Category 1

ECC (ABAP)

Cluster Type 1 “HADR for DB2 plus

takeover for SAP ASCS”

Yes BPP

(ABAP)

Category 2

PI (Dual-Stack)

(own SLD, replicated from central)

Cluster type 2 “Takeover for all DB

and SAP components”

No

BPM (Java)

CE (Java)

EP (Java)

CPS

CUA (ABAP)

Category 3

BW (ABAP)

Cluster type 2 “Takeover for all DB

and SAP components”

Partially SolMan

(Dual Stack)

SLD (Central) (Java)

Table 6 Mapping between SAP Systems and the cluster types

Additional recommendations for the implementation of the clusters include:

Mirror data across 2 storage systems for all HA clusters

o Set up 2 identical storage systems (<STORAGE SYSTEM>1 plus a second storage system of the same type) as the basis for HADR as well as for AIX LVM mirroring.

Ensure sufficient bandwidth for the access of both storage systems Bandwidth of storage access is crucial for performance and availability of the systems

Distribute the servers across the 2 rooms

o Consider a symmetric distribution of all components – storage system as well as servers across the 2 rooms F and G. (current concept foresees all servers in Site A and only the second storage system in Site B).

Page 24: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 24 of 71

1.4 Setup & Configuration Pre-requisites

1.4.1 Common PowerHA cluster setup

In the following section general considerations are discussed which are valid for both cluster types

1.4.1.1 AIX

In the following some AIX specific requirements and items for the PowerHA cluster are discussed: 1.4.1.1.1 AIX FIBRECHANNEL SETTINGS

Enable „Dynamic tracking“ and „Fast Error Recovery“ for all fibre channel SCSI adapters. This will improve error recovery time in case of a failure of a SAN/ storage component: # chdev -a dyntrk=yes -a fc_err_recov=fast_fail -l fscsiX –P

1.4.1.1.2 AIX 6.1 FILESETS

The AIX filesets listed below are a prerequisite for installing PowerHA. These filesets need to be included in the <CLIENT> standard built for AIX:

bos.adt.lib

bos.adt.libm

bos.adt.syscalls

bos.net.tcp.client

bos.net.tcp.server

bos.rte.SRC

bos.rte.libc

bos.rte.libcfg

bos.rte.libcur

bos.rte.libpthreads

bos.rte.odm

bos.rte.lvm.

bos.clvm.enh 1.4.1.1.3 RSCT NETWORK OPTIONS SETTINGS

PowerHA requires that the network options nonlocsrcroute, ipsrcroutesend, ipsrcrouterecv, and ipsrcrouteforward are all set to 1. The parameters are set during the startup of the “topsvcs” startup script. The topsvcs startup is part of AIX RSCT. 1.4.1.1.4 POWERHA NETWORK OPTIONS SETTING

PowerHA ensures that the value for each of the following network options is consistent across all the active cluster nodes:

tcp_pmtu_discover

udp_pmtu_discover

ipignoreredirects If these settings are Out-of-sync on any of the cluster nodes the setting is automatically adjusted by PowerHA: The change of IP addresses during a takeover operation may cause network routes to be changed or deleted: The AIX network option routerevalidate has to be set to 1 (no -o routerevalidate=1). This setting ensures proper communication between all the cluster nodes. During a PowerHA cluster verification run (with corrective action) the routerevalidate setting is automatically adjusted on all active nodes in the cluster.

Page 25: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 25 of 71

1.4.1.1.5 SNMP V3 CONSIDERATIONS

The PowerHA commands “clstat” or “cldump” will not work properly if the internet MIB tree is not enabled in the snmpdv3.conf file. This behavior is seen since AIX 6.1, where this internet MIB entry was disabled per default. (As it may potentially be a security exposure). The internet MIB entry is required to view/resolve the risc6000clsmuxpd (1.3.6.1.4.1.2.3.1.2.1.5) MIB sub tree used by “clstat” or “cldump” functionality. To enable the MIB sub tree for risc6000clsmuxpd the line: VACM_VIEW defaultView 1.3.6.1.4.1.2.3.1.2.1.5 - included is added to the /etc/snmpdv3.conf file. The snmp daemon must be restarted after enablement of the MIB entry. 1.4.1.1.6 /ETC/SERVICES SETTINGS FOR POWERHA

Table 7 contains the required entries for PowerHA in the /etc/services file. If these entries are not present, they will be added during the installation of the PowerHA software.

Name Port Protocol

topsvcs 6178 udp

grpsvcs 6179 udp

clinfo_deadman 6176 udp

clcomd 6191 tcp

Table 7 Entries for PowerHA in /etc/services

1.4.1.1.7 SETUP OF FILE SYSTEMS IN ROOTVG

The filesystems /db2 respectively /usr/sap are part of the OS image in the rootvg. These filesystems are the “container” filesystems for the filesystems dedicated to the database and SAP components.

AIX VG

Content Filesystem Logical Volume

Size [GB]

Comment

rootvg SAP base /usr/sap/ lvsap 40 AIX LVM mirroring

DB2 UDB base

/db2 lvdb2 20 AIX LVM mirroring

Table 8 Local AIX filesystems

1.4.1.1.8 VOLUME GROUPS FOR THE APPLICATION

Table 9 describes the storage layout for the volume groups for the DB2 and SAP application in the PowerHA cluster. DB2 Database: The storage layout fulfills the boundary conditions for the implementation of split-mirror backups: DB2 UDB database container files, DB2 active logs, and DB2 archived logs are located within three different AIX Volume Group sets (<sid>db{N}vg for data, <sid>logvg, and <sid>arclogvg). The VG for database container files (<sid>db{N}vg) shouldn’t contain any other filesystem for non-database data: These data might be overwritten during a split-mirror restore. The DB2 database directory /db2/<SID>/db2<sid> needs to be located in one of the VGs for database data files too. SAP Instances: VG <sid>scsvg hosts the filesystems for the SAP Central Services, and VG <sid>ersvg hosts the filesystems for the SAP Enqueue Replication server. VG <sid>nfsvg hosts the shared filesystems PowerHA disk heartbeat: Additional VGs are required for the disks for non-IP cluster communication through the SAN (“disk heartbeat”)

Page 26: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 26 of 71

VG Name Type Content Cluster Type 1 Cluster Type 2

<sid>db{N}vg N = 1,2,3,4,5

Database DB2 UDB files

Sapdata directories /db2/<SID>/sapdata{N} Database directory /db2/<SID>/db2<sid>

DB2 HADR Cluster: No AIX LVM mirror non-shared VG: An own VG both on primary and standby

DB2 Takeover Cluster: AIX LVM mirrored shared VG

<sid>logvg Database Active Logs

/db2/<SID>/log_dir DB2 HADR Cluster: AIX LVM mirror, non-shared VG: An own VG both on primary and standby

DB2 Takeover Cluster: AIX LVM mirrored shared VG

<sid>arclogvg Database archive logs

/db2/<SID>/log_arc DB2 HADR Cluster: AIX LVM mirrored shared VG

DB2 Takeover Cluster: AIX LVM mirrored shared VG

<sid>nfsvg shared filesystems (NFS server)

SAP Kernel, SAP Profiles, Job Logs, Batch Input files, Interface directories, …

AIX LVM mirrored shared VG

NFS export to all SAP AppServers

<sid>ascsvg SAP Central Services

SAP directories SCS, ASCS /usr/sap/<SID>/ASCSxx /usr/sap/<SID>/SCSxx

AIX LVM mirrored shared VG

<sid>ersvg SAP Enqueue Replication Server

SAP directories ERS AIX LVM mirrored shared VG

hbvg Disk heartbeat PowerHA disk heartbeat devices

shared VG

Table 9 AIX volume groups layout

All “shared” volume groups are defined as “Enhanced Concurrent Mode” (ECM) Volume Groups. Implementation of ECM allows a faster disk takeover: The “Enhanced concurrent” VG is varied on in “active” mode (read-write access) on the HA cluster node that has the resource group online: All JFS2 filesystems within the VG are mounted on that node. On the 2

nd PowerHA cluster node the volume

group is varied-on in “passive” mode (read-only access for the VGDA): The JFS2 filesystems cannot be mounted on this node until a takeover takes place. Then the VG is varied-on in active mode there. Accidental use/ access to the data on the 2

nd cluster node is so prevented.

The major number for each volume group should be unique and should be defined with the same value on both cluster nodes. The “hbvg” is required for the additional “non-IP” disk heartbeat for PowerHA: “hbvg” should contain two disk volumes of the two different <STORAGE SYSTEM> systems to allow for located within the two different rooms.

Page 27: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 27 of 71

Table 10 lists the different file systems that are created in the VGs:

AIX VG

Content Filesystem Logical Volume

Size [GB]

NFS Mount

2

rootvg SAP base /usr/sap/ lvsap 40

DB2 UDB base

/db2 lvdb2 20

<sid>nfsvg SAP Executables und Profiles

/export/sapmnt/<SID> lvsapmnt 80 /sapmnt/<SID>

Transport directory

/export/usr/sap/trans lvtrans 80 /usr/sap/trans

Interface data /export/forsap/<SID> lvforsap 1000 /forsap/<SID>

<sid>ascsvg ABAP Central Services

/usr/sap/<SID>/ASCS10 lvASCS10 20

JAVA Central Services

/usr/sap/<SID>/SCS20 lvSCS20 1

<sid>dbvg

Database Archive Logs

/db2/<SID/log_arc

<sid>logvg

Database Online Logs

/db2/<SID/log_dir

<sid>db{N}vg

Database data /db2/<SID>/sapdata1

?

<sid>ersvg Enqueue Replication Server ASCS

/usr/sap/<SID>/ERS11 lvERS11 10

Enqueue Replication Server SCS

/usr/sap/<SID>/ERS21 lvERS21 10

Table 10 Filesystem layout for SAP and DB2 UDB database

The AIX volume group layout and filesystem layout will support a split-mirror backup, as indicated in Figure 6. Beside the tablespace containers, also the database directory /db2/<SID>/db2<sid> needs to be part of the FlashCopy target volumes: During the “set write suspend” execution before initiation of the “split-mirror” in the storage system, some database information is written to this directory. This information then is required during the execution of “db2inidb” on the FlashCopy volumes. The database logs should not be part of the set of FlashCopy volumes for backup operations: Otherwise, if the online logs are part of the FlashCopy set latest online log data on the system will get overwritten during a split-mirror restore activity with logs in the state of the timestamp the split-mirror backup was taken. If it is intended to “clone” e.g. the pre-production system from the production system with the split-mirror technique, then the logs however should be part of this image. Other filesystems should not be part of in any of the FlashCopy sets, as they might overwrite filesystems during a split-mirror restore.

2 It should be avoided to mount the NFS mount point directly in the root directory ‘/’. For that reason a directory structure below ‘/’ should be created for the mount point.

Page 28: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 28 of 71

All other DB2 filesystems

/db2

/db2/<SID>/log_arc

/db2/<SID>/db2dump

AIX VGs for

database data

SnapShot/

FlashCopy(Backup,

Cloning)

Filesystem for DB2 logs

/db2/<SID>/log_dir

Filesystem for DB2 database

/db2/<SID>/db2<sid>

/db2/<SID>/sapdata1

/db2/<SID>/sapdata2

/db2/<SID>/sapdata3

/db2/<SID>/sapdata4

...

No

SnapShot/

FlashCopy

SnapShot/

FlashCopy(Cloning only,

eg. Q-System

refresh)

AIX VGs for

database logs

AIX VGs for other

database filesystems

Figure 6 Storage layout considerations for split mirror backup

1.4.1.2 PowerHA

Setting up a PowerHA cluster requires the following steps:

Install PowerHA software on both cluster nodes

Define cluster topology o Define Cluster and add nodes to the cluster. o Configure the PowerHA network and decide the method for IP address takeover o Configure the communication interface and devices.

A RS-232 link or disk heartbeat is used for non-IP communication. o Configure the high availability service IP address.

Define Cluster resources o Configure the PowerHA application server. o Configure the PowerHA resource group.

In the following section, several of these aspects will be discussed for the two cluster types. 1.4.1.2.1 POWERHA SOFTWARE

PowerHA version 6.1 is deployed. A NIM bundle can be is used to install the PowerHA software filesets. The NIM bundle then needs to include the file sets

cluster.es.cspoc

cluster.es.nfs

cluster.es.server

cluster.es.worksheets

cluster.license

cluster.man.en_US.es The following filesets need to be installed in addition for the RSCT component:

rsct.compat.basic.hacmp

rsct.compat.clients.hacmp

rsct.core.sec

rsct.core.rmc

Page 29: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 29 of 71

1.4.1.2.2 HA CLUSTER TOPOLOGY

Each PowerHA cluster is set-up as a two-node-cluster. Cluster name is e.g. cl_sap_<SAP Component> respectively cl_db_<SAP Component>

Cluster Name

cl_sap_< SAP Component > respectively cl_db_< SAP Component>

Table 11 PowerHA cluster name

The PowerHA cluster node names have the same name as the hostnames of the two LPARs (“physical hostname”), but are defined in UPPER CASE.

Cluster Nodes

Communication path to node

<node 1> <IP address of node 1 (persistent)>

<node 2> <IP address of node 2 (persistent)>

Table 12 PowerHA node names

Three different network types are defined in PowerHA: The access network provides the communication path for the end users and for system-to-system communication to the SAP system and database. The other networks, serial network and disk-heartbeat provide intra-cluster communication between the two nodes for cluster heartbeat purpose.

Network description Network Name Nettype

Access net for SAP and DB

<SAP Component>_net ether

Serial RS232 Network (Non-IP)

<SAP Component>_rs232 RS232

Disk Heartbeat #01 <SAP Component>_diskhb1 diskhb

Disk Heartbeat #02 <SAP Component>_diskhb2 diskhb

Table 13 PowerHA networks

On the Access net, the persistent IP addresses (always bound to the node) and the service IP addresses (bound to the PowerHA resource group) are defined.

Name Type Network Nettype Shared IP Address Node

<node 1> Persistent Access_net Ether No n/a <node 1>

<node 2> Persistent Access_net Ether No n/a <node 2>

<SAP Component>nfs Service Access_net Ether Yes n/a n/a

<SAP Component>scs Service Access_net Ether Yes n/a n/a

<SAP Component>db Service Access_net Ether Yes n/a n/a

Table 14 Network interfaces and Service IP addresses

1.4.1.2.3 POWERHA CLUSTER RESOURCES

PowerHA provides an interface to identify all the cluster resources which are essential for the operation of an application and combines them in a resource group. Cluster resources include disk resources, AIX volume groups and their logical volumes, filesystems, service IP labels (virtual hostnames), applications and the dependencies between these items. PowerHA enables resource group management by defining resource group policies and attributes for the typical cluster operations like startup, failover and fallback dependencies. The Cluster resource groups can be concurrent or non-concurrent. Concurrent resource groups are available simultaneously on multiple

Page 30: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 30 of 71

nodes in the cluster. Non-concurrent resource groups are available on one node in the cluster online at any point in time. For the DB2 database and the SAP systems all resource groups are defined as non-concurrent resource groups. Dependent on the resource group policy a home node is defined for each resource group. The home node is the first node in the node list for a particular resource group: Typically the resource group is activated on the home node first. Failover is a term used to refer to the movement of the resource group: In case the node currently hosting the resource group experiences a failure, the resource group is moved to another node in the cluster. The resource group move from the node that acquired the resource group during the failover back to the failed node (that rejoined the cluster) is called a fallback operation. For the SAP systems the Fallback settings of the resource group are set to “Never Fallback”: A fallback operation will always be manually triggered from the infrastructure team during a planned action. All resource groups are defined with the settings as shown in Table 15:

Resource Group Startup Behavior

Failover Behavior

Fallback Behavior

Participating Nodes

rg_<SAP Component>_db Home Node

Next Priority Node Never Fallback <node 1> <node2>

rg_< SAP Component >_cs Home Node

Next Priority Node Never Fallback <node 1> <node2>

rg_< SAP Component >_nfs Home Node

Next Priority Node Never Fallback <node 1> <node2>

rg_< SAP Component >_er Home Node

Next Priority Node Never Fallback <node 2> <node1>

Table 15 Resource Group attributes in PowerHA

Figure 7 gives an overview of the Resource Groups in the PowerHA cluster. The example is shown for cluster type 2, but is similar for cluster type 1. In total, four Resource Groups are created for the database and SAP application in PowerHA.

Resource Group for DB2 UDB Database instance

Resource Group for SAP Central services instance(s)

Resource group for SAP Enqueue Replication Server

Resource Group for NFS Service

Page 31: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 31 of 71

Figure 7 Resource Groups for the PowerHA cluster

The SAP Application servers are not placed into PowerHA resource groups and are fully managed outside of PowerHA.

RG RG Name Service IP Volume Groups

NFS Server

DB2 UDB rg_<sid>_db <sid>db <sid>db{N}vg <sid>logvg <sid>dbvg

No

SAP Central Services

rg_<sid>_scs <sid>scs <sid>scsvg No

NFS Server rg_<sid>_nfs <sid>nfs <sid>nfsvg Yes*)

SAP ERS rg_<sid>_ers n/a <sid>ersvg No *) Attribute “File systems Mounted before IP configured” in the Resource Group attributes needs to be set to “true”.

Table 16 Power HA Resource Groups for Database and SAP

The resource group definition in PowerHA leads to the following behavior: Database Cluster:

o During normal operation, rg_<sid>_db and so the (primary) database is online on node 1. No resource group for the database is active on node 2. However, in case of a DB2 HADR cluster, the standby database is running on node 2.

o Assuming node 1 goes down then PowerHA acquires the RG rg_<sid>_db on node 2.

In case of a DB2 takeover cluster, the database is restarted on node 2. Automated crash recovery is initiated.

In case of a DB2 HADR cluster, the standby database is activated as new primary on node 2.

Page 32: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 32 of 71

If the database engine on node 1 still can be reached (“graceful takeover”), a normal DB2 “takeover” command is sufficient.

If the database engine on node 1 is no longer available, then the DB2 “takeover by force” needs to be executed.

o When node 1 is repaired/ rebooted, the IT infrastructure team (re-)starts PowerHA on node 1.

In case of a DB2 takeover cluster, redundancy is restored after node 1 is up again. There is no absolute need to move the resource group rg_<sid>_db back to node 1, but this can be done (if desired) during the next available downtime window.

In case of a DB2 HADR cluster, the IT infrastructure team and the SAP Basis team need to take appropriate actions to restart the DB2 standby database on node 1.

SAP Cluster:

o During normal operation rg_<sid>_cs (SAP SCS) is online on node 1. rg_<sid>_ers is online on node 2.

o Assuming node 1 goes down then PowerHA acquires the RG for the SAP SCS on node 2: RG SCS gets online on node 2. The start of the SAP Central Services in RG SCS stops the ERS process running on node 2: RG rg_<sid>_ers stays online on Node 2, but the SAP Replicated Enqueue Server is stopped.

o When node 1 is repaired/ rebooted, the IT infrastructure team (re-)starts PowerHA on node 1.

After PowerHA is up on node 1 the PowerHA application monitor of the ERS recognizes that there is a new cluster node available: The application monitor initiates a RG move for RG rg_<sid>_ers to node 1. rg_<sid>_scs and so the SAP SCS continue to run on node 2. While rg_<sid>_ers is acquired on node 1 the SAP Enqueue Replication server is started on node 1. Redundancy is restored after the ERS is active on node 1 again. There is no absolute need to move the resource group rg_<sid>_scs back to node 1 in this situation. However this can be done (if desired) nearly online during the next available window by a RG move of the rg_<sid>_scs to node 1: There will be no disruption, however the SCS will be down and freezing the system during the short time of the RG move. The application monitoring of the ERS will then afterwards automatically move the ERS to node 2.

A few dual-stack systems (ABAP + Java) are in scope of <CLIENT>: In HA clusters for dual stack systems both the ABAP Central Services and the JAVA Central Services are located together in one and the same resource group rg_<sid>_scs. As the cluster reacts on infrastructure related incidents only a failure will impact always both components at the same time. 1.4.1.2.4 POWERHA NFS CLUSTER MOUNTS

NFS cross-mounting is a PowerHA specific NFS configuration where each node in the cluster may act both as NFS server and NFS client. A file system is being exported from one node, and the file system is mounted via NFS on all nodes to which the Resource Group is assigned to. Also the node currently exporting the filesystem remounts the filesystem via NFS. Applications so can access the NFS file systems on any of the nodes which is assigned to the resource group. IP address takeover needs to be configured for the resource group. In a failover situation

The takeover node acquires the IP service address for the NFS server

The NFS file system is locally mounted on the takeover node and re-exported.

All other nodes in the resource group maintain their NFS mounts.

Page 33: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 33 of 71

The following set of figures illustrates the behavior for the example of /sapmnt: In the initial state (Figure 8) AIX is started on both cluster nodes and PowerHA is either not started, or the resource group for the NFS server is offline. As a result the NFS file system is not mounted on any of the PowerHA cluster nodes.

Figure 8 PowerHA cluster NFS mounts – resource group offline

After start of PowerHA (or set online the resource group for the NFS server), the underlying JFS2 file system /export/sapmnt gets mounted on node 1 and the service address of the NFS server is acquired on the primary node. PowerHA then (re-) exports the JFS2 filesystem: The details of the NFS export options can be specified in the file /usr/es/sbin/cluster/etc/exports. The filesystem is mounted via NFS automatically by PowerHA to /sapmnt on both cluster nodes. (Figure 9)

Figure 9 PowerHA cluster NFS mounts – resource group online, standard operation

In case of a takeover (or a resource group move) to the second node, the underlying JFS2 file system gets unmounted on the primary node and remounted on the secondary node. The service address of the NFS server is released on the primary node and acquired on the secondary one. The NFS export is refreshed, and all the NFS clients are able to access the file system again. (Figure 10).

Page 34: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 34 of 71

Figure 10 PowerHA cluster NFS mounts – resource group online, takeover operation

1.4.1.2.5 SAP APPLICATION START/ STOP SCRIPTS

Table 17 contains an example for PowerHA application server definitions. The PowerHA application server is member of the resource group and is the link to the application start-/ stop scripts. As Start-Script and Stop-Script in the PowerHA application server the SAP scripts provided by the ISICC respectively the DB2 HADR scripts delivered as samples with the DB2 software are referenced. For specific details of the scripts please refer to the documentation in the SAP and PowerHA developerworks WIKI: http://www.ibm.com/developerworks/wikis/display/WikiPtype/SAP+and+PowerHA

Application Server Start-Script Stop-Script as_sap_<sid>_ascs /usr/es/sbin/cluster/sap/cl_sap_start

ASCS10 <SID> /usr/es/sbin/cluster/sap/cl_sap_stop ASCS10 <SID>

as_sap_<sid>_aers /usr/es/sbin/cluster/sap/cl_sap_start ERS11 <SID>

/usr/es/sbin/cluster/sap/cl_sap_stop ERS11 <SID>

Table 17 PowerHA application server

PowerHA application Start and Stop scripts should verbosely output to the PowerHA log file in the same format like the “internal” PowerHA event scripts. This is achieved by including the following line in the header of each of the scripts: set -x && PS4="${0##*/}"'[$LINENO]' The Start and Stop scripts for the SAP provide following functionality:

DB2 UDB Database (takeover) o Start Database o Stop Database

SAP Central Services o Start SAP Central Services o Stop SAP Central Services

SAP Replicated Enqueue Server o Start Enqueue Replication Server o The stop script for the SAP Replicated Enqueue Service is a wrapper only: no

“stopsap” command will be issued. The SAP Replicated Enqueue server is dependent on the SAP Central Services: The stop of the SAP Replicated Enqueue Server will be controlled from there.

o If the SAP Central Services are acquired on the node where the SAP Replicated Enqueue server is currently running, the SAP Replicated Enqueue server will be stopped automatically after the start of the Enqueue Server.

Page 35: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 35 of 71

o Check if the SAP Replicated Enqueue Server RG is online on the same node like the SCS: Eventually move ERS to another node

When the 2nd

cluster node is available again and the RGmove for the SAP ERS is issued to restore redundancy

For the additional application components, further scripts beyond the ISICC package may be required:

SAP Web Dispatcher If the SAP Web Dispatcher is part of the PowerHA cluster (SAP WebDispatcher is running in the same security zone like the SAP system), the WebDispatcher is added to the resource group for the SAP Central Services. The start-/ stop of the SAP WebDispatcher need to be added there: A new PowerHA application server is defined.

SAP Solution Manager Diagnostics Agent (SMDA) If the SMDA is added to the resource group for the database and SAP Central Services, the start-/ stop of the SMDA need to be added there also. Either the start/ stop of the SMD Agent is integrated into the existing PowerHA application server start and stop script, or an additional PowerHA application server can be defined especially for the SMD agent.

Additional 3rd

party components If further components need to be started together with a SAP or database component in a PowerHA resource group (e.g. backup agents, monitoring agents, …) then their start/ stop needs to be added also.

1.4.1.2.6 POWERHA APPLICATION MONITORS

PowerHA allows implementation of so-called “application monitors” to detect the current state of the clustered application. PowerHA Application Monitoring can monitor the application processes running on the OS level and/or regularly query the application directly about its state. Depending on the return code of these “status checks” an automated restart/recovery or a takeover action is started in control of PowerHA. The implementation of PowerHA application monitors helps to improve overall application availability: For the SAP Central Services (ASCS/ SCS) a failed enqueue server process can be detected and automatically initiate a failover action. A seamless failover is achieved for the end users: The restart of the SAP Central Services on the 2nd cluster node benefits from the replicated enqueue table already existing there. For the ERS, the PowerHA application monitor (“NODE_JOIN”) will automatically recover from a one node situation: In case the node currently hosting the SCS fails, the SCS is taken over to the remaining node. The ERS stays online on this node; however the ERS process will be stopped with the start of the SCS. When the failed node gets online again, the PowerHA application monitoring of the ERS will initiate a Resource Group move to ensure the ERS can be restarted (and protection for the SCS is restored). The stabilization interval in Table 18 reflects suggested settings for the long-running monitoring mode. In this mode, the application monitor periodically checks that the application is running. The checking starts the first time after the specified stabilization interval has passed. The monitor then is run periodically based on the monitoring interval. If the application monitor returns a zero return code, the application is running successfully. A non-zero return code indicates that the application has failed, which will then lead to the configured action (Failover or Notify).

DB2 HADR takeover

DB2 takeover

ASCS takeover

SCS takeover

Stabilization interval

300 900 350 - 900 500 - 900

Monitor interval

10 - 30 10 - 60 30 - 90 30 – 90

Possible Fallover/ Notify Fallover/ Notify Fallover/ Notify Fallover/ Notify

Page 36: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 36 of 71

DB2 HADR takeover

DB2 takeover

ASCS takeover

SCS takeover

Action

Benefit Failing DB is restarted

Failing DB is restarted

Automatic failover if en.* process dies: will automatically

acquire the replicated enqueue table and hence a seamless failover

Automatic failover if en.* process dies: will automatically

acquire the replicated enqueue table and hence a seamless failover

Risk False failover or notification

False failover or notification

False failover or notification

False failover or notification

Table 18 Base settings for PowerHA application monitors

Application monitors are implemented for

SAP Central Services In case that the SAP Central Services fail the PowerHA application monitor script will recognize this failure and will return a non-zero return code to the PowerHA event processing. A fallover action is triggered (RC <> 0) and the SAP Central Services are restarted on the other cluster node running the SAP Enque Replication server.

Enqueue Replication Server In case that the SAP Enqueue Replication Servers fails the PowerHA application monitor script will recognize this failure. In case that the SAP Central Services are not running on the actual node, the Enqueue Replication Server will be restarted. In case that the SAP Central Services are running on that node (“takeover situation”) and a 2

nd node is available in the

cluster (“the failing node was repaired and re-joined the cluster”) then a resource group move of the ERS will be initiated to the available node.

Table 19 contains an overview of the required entries for the PowerHA application monitor definition:

Application Monitor mon_sap_<sid>_ascs mon_sap_<sid>_aers Application Server as_sap_<sid>_ascs as_sap_<sid>_aers Type Customer Customer Monitor Method /usr/es/sbin/cluster/sap/cl_sap_monit

or_scs /usr/es/sbin/cluster/sap/cl_sap_monitor_ers

Monitor Interval 60 60 Hung Monitor Signal 9 9 Stabilization Interval 300 360 Restart Count

3 0 2 Restart Interval

4 0 792 Action on App Failure Fallover Notify Notify Method n/a n/a Cleanup Method n/a n/a Restart Method n/a /usr/es/sbin/cluster/sap/cl_sap_start_er

s

Table 19 PowerHA application monitor definitons

3 The retry count specifies the number of times to try restarting the application before taking any other actions. 4 The restart interval dictates the number of seconds that the restart application must remain stable before the retry count is set to zero, thus completing the monitor activity until the next failure occurs. Do not set this value smaller than the product of <Restart Count> * <Stabilization Interval> (The default setting is 10% larger than that value).

Page 37: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 37 of 71

No application monitor is implemented for the DB2 UDB database. A takeover for the database / activation of the DB2 standby database as new primary will be initiated on infrastructure related incidents only: It is too difficult to predict the exact state/ root cause in all cases, and so a takeover initiated out of the database application monitor wouldn’t improve the availability for all cases. In case of a database application issue the database administrator needs to get involved to react accordingly. For the settings of the “Stabilization Interval” for the ASCS Application Monitor the following considerations need to be obeyed:

The start of the ASCS instance is dependent on the availability of the NFS Filesystem (e.g. start of ASCS requires access to the Instance profiles on /sapmnt/<SID>/profile).

Assuming the NFS service is unavailable, and the HA cluster attempts to bring up the ASCS on node A. The Application Monitor gets started (after stabilization interval) also, and will report a failure of ASCS (as NFS is unavailable, and ASCS couldn’t get started).

The HA cluster now attempts to bring up ASCS on node B, the Application Monitor will be started again, reports a start failure (as NFS is unavailable on node B also), and the HA cluster will attempt to bring up ASCS on node A, etc.

To ensure that this behaviour is suppressed in case a takeover of NFS happens at the same time, it needs to be ensured that the “stabilization interval” is sufficiently high so that NFS has successfully settled also.

1.4.1.3 Further considerations for SAP components in a PowerHA environment

1.4.1.3.1 PREPARATION OF PRIMARY AND TAKEOVER NODE

Ensure that all SAP and DB2 UDB specific entries in /etc/services are identically available on both cluster nodes and on the additional SAP application servers.

In the home directories for both the users db2<sid> and <sid>adm: o Rename the files for

.sapenv_<hostname>.csh

.sapenv_<hostname>.sh

.dbenv_<hostname>.csh

.dbenv_<hostname>.sh to .sapenv.csh / .sapenv.sh respectively .dbenv.csh / .dbenv.sh

Align the structure of the /usr/sap/ directories and their contents (except for the contents of the shared volume groups (like /usr/sap/<SID>/SCS10) and those belonging to local directories of node1 (e.g. primary application server) between primary and secondary node. Ensure to keep permissions, owner and group the same.

Copy the /etc/rc.d/init.d/sapinit from node1 to node2. This will automatically restart sapstartsrv after reboot

Install saplicense for both primary and secondary node.

The SAP kernel is installed in location /usr/sap/<SID>/SYS/exe/run (as indicated by the instance profile parameter DIR_CT_RUN). This directory is finally linked to the NFS shared directory /sapmnt/<SID>/exe. During SAP instance start the tool “sapcpe” copies the SAP kernel from the shared directory to an instance specific directory on the local disks (indicated by the instance profile parameter DIR_EXECUTABLE). For each SAP instance of the system the directory $(DIR_INSTANCE)/exe needs to exist:

Page 38: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 38 of 71

/usr/sap/<SID>/<Instance Name>/exe. The directory needs to be owned by user <sid>adm, group sapsys, and with permission 755 (drwxr-xr-xr). To ensure that not only the SAP Kernel executables, but also all the shared libraries for it are loaded from the “local” directory, the environment variable DIR_LIBRARY need to reflect the “local” path. Solution is to add/ modify the START profile (START_<Instance name>_<virtual hostname>) of the instance: SETENV_xx = DIR_LIBRARY=/usr/sap/$(SAPSYSTEMNAME)/$(INSTANCE_NAME)/exe After instance start, the usage of the “local” libraries can be verified with the AIX “procldd” command. By identifying the UNIX process id <pid> of a running disp+work workprocess and executing the “procldd <pid>, the output need to reflect the “local” directories only.

1.4.1.3.2 SAP INSTANCE NUMBER RANGES

The following table contains the convention for the SAP instance number ranges:

Role Instance Number Range

Description Power HA Resource Group

ABAP 10 SAP Central Services (ABAP) rg_<SID>_scs

11 Enqueue Replication Server (ABAP) rg_ <SID>_ers

Java 20 SAP Central Services (Java) rg_<SID>_scs

21 Enqueue Replication Server (Java) rg_ <SID>_ers

Table 20 Conventions for SAP Instance numbers

The SAP instance number defines which TCP/IP ports are used for the SAP internal communication and the communication to the end users and interfaces. All TCP/IP ports need to be aligned so that proper communication is ensured both in “normal” and “takeover” situation. Further TCP/IP ports need to be considered for SAP communication in addition:

Role Description

ABAP SAP Message Server Port sapms<SID> in /etc/services rdisp/msserv

Secure the SAP Message server communication: Prevent against unwanted clients pretending to the message server to be application servers.

rdisp/msserv_internal

SAP System Log collector rslg/collect_daemon/listen_port

Table 21 Additional TCP/IP communication ports used in the SAP systems

The SAP configuration in the SAP instance profiles need to ensure that these ports are unique and not used for any other application communication on all the LPARs.

1.4.1.4 SAP Start Profile settings

By default, during SAP installation the instance profiles are created using “Restart_Program” statements for some of the critical processes. This needs to be adapted:

Ensure that the Message Server process is started with “Restart_Program” in the instance profile: The “sapstart” process will monitor the “Message Server” child process and will restart it, if necessary.

Page 39: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 39 of 71

Ensure that the Enqueue Server process is started with “Start_Program” in the instance profile. The PowerHA Application Monitor will monitor the Message Server process, and initiate a Fallover (to the node where the ERS is running), if necessary. (Restarting the Enqueue Server locally would lead to data loss, as the enqueue table in memory would be lost).

#-----------------------------------------------------------------------

# Start SAP message server

#-----------------------------------------------------------------------

_MS = ms.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)

Execute_01 = local rm -f $(_MS)

Execute_02 = local ln -s -f $(DIR_EXECUTABLE)/msg_server$(FT_EXE) $(_MS)

Restart_Program_00 = local $(_MS) pf=$(_PF)

#-----------------------------------------------------------------------

# Start SAP enqueue server

#-----------------------------------------------------------------------

_EN = en.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)

Execute_03 = local rm -f $(_EN)

Execute_04 = local ln -s -f $(DIR_EXECUTABLE)/enserver$(FT_EXE) $(_EN)

Start_Program_01 = local $(_EN) pf=$(_PF)

Ensure that the ERS process is started with “Start_Program” in the instance profile. The PowerHA Application Monitor will monitor the Enqueue Replication Server process, and initiate a Restart or Fallover dependent on the situation.

#-----------------------------------------------------------------------

# Start enqueue replication server

#-----------------------------------------------------------------------

_ER = er.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME)

Execute_02 = local rm -f $(_ER)

Execute_03 = local ln -s -f $(DIR_EXECUTABLE)/enrepserver$(FT_EXE) $(_ER)

Start_Program_00 = local $(_ER) pf=$(_PFL) NR=$(SCSID)

1.4.1.5 Java Instance Parameters

The Java instance parameter as listed in Table 22 Java Instance ParametersTable 22 may need to be adjusted to ensure that the Java instances can reconnect to the SAP Central Services in the failover situation. It may be required to tune the AS Java with respect to timeout settings, etc. to handle increased takeover times e.g. for Dual stack systems: In a lot of cases reconnect issues in a cluster setup failover can be solved by increasing ms.reconnect.timeout and decreasing ms.keepalive only. Sometimes it is may be required to increase the ms.notification.timeout also.

Page 40: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 40 of 71

Parameter Description Default Value

New Value

ms.confirmation.timeout Amount of time that a joining cluster element waits for the confirmation replies from the cluster elements. If one or more cluster elements do not send a confirmation message, the server process will restart

30000

ms.keepalive If there has been no communication between a server process and the message server for the specified amount of time, the server process triggers an aliveness check (ping/pong protocol). This ping/pong mechanism is important for the server process to detect that the connection to the message server is broken. Without sending messages to the message server, the connection failure cannot be detected

20000

ms.notification.timeout Amount of time for receiving the acknowledgment from the message server that the message has been delivered. If the message server fails to deliver the acknowledgment, the server process will restart

180000 600000

ms.reconnect.timeout Specifies the timeout after which a reconnection to the message server is no longer possible. If the switchover software is not able to detect the need for a switch and start the central services instance on a different host within this time, the server process(s) will restart

180000

ms.backup.message.number Length of internal message backup queue. It specifies the number of one-way and reply messages that will be sent again after a message server crash

10

ms.reconnect.sync.timeout The cluster architecture requires resending of messages that might have got lost during a crash of the message server. (Duplicated messages will be sorted out on the receiver-side). In order to deliver broadcast messages to all cluster elements, each server process has to check whether all server processes were able to reconnect. This is the only mechanism to guarantee the reliable message delivery of broadcast messages. This parameter specifies how long this reconnect could take at most. If set to 0 (default), the following formula will specify the timeout value: ms.reconnect.sync.timeout = 6 * ms.keepalive + 60000

0

Table 22 Java Instance Parameters

Page 41: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 41 of 71

1.4.1.5.1 SAP DEPLOYMENT

The SAP installation steps include

Install SAP Database Server o The resource group rg_<SID> db needs to be online on the primary node to be able

to install the Database Server o Setup Standby Database on the 2

nd DB node

Install SAP Central Services o The resource group rg_<SID>_scs needs to be online on the primary node to be able

to install the SAP SCS o A SAP Gateway process may need to be added to the SCS.

Install Primary SAP Application server on a POWER blade (outside the cluster) o The NFS filesystems (e.g. /sapmnt/<SID>) need to be available to be able to install

the primary SAP Application Server.

Install additional SAP Application Server on POWER blades (outside the cluster) o The NFS filesystems (e.g. /sapmnt/<SID>) need to be available to be able to install

the primary SAP Application Server.

Setup Enqueue Replication Server o The resource group rg_<sid>_scs needs to be online on the primary node and the

resource group rg_<sid>_ers needs to be online on the secondary node to be able to setup the Enqueue Replication Server on the secondary node.

Install the SAP component using the virtual hostnames for the appropriate virtual hostname (SAPINST_USE_HOSTNAME=).

1.4.1.6 Perform Takeover tests

The full failover testing needs to be executed according to the testing documents

Page 42: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 42 of 71

1.4.2 Additional configuration steps for cluster type 1

1.4.2.1 Configuring DB2 HADR for PowerHA

As application scripts, the script package delivered as sample with the DB2 UDB software is used. The application start, stop and monitoring scripts need to be copied from the DB2 samples directory to a local directory on each node of the cluster # cp /db2/<SID>/db2<sid>/sqllib/samples/hacmp/rc.hadr.* /<local directory> The following parameters (see Table 23) need to be specified for the call of the scripts:

Argument Variable Value Comment

$1 DB2HADRINSTANCE1 db2<sid> Database instance on the primary

$2 DB2HADRINSTANCE2 db2<sid> Database instance on the standby (same)

$3 DB2HADRNAME <SID> Database name

Table 23 Parameterization for the DB2 scripts

Add the DB2 HADR start and stop scripts from above to the PowerHA application server. The DB2 HADR start script conditionally reacts on the current status internally:

Started on

State RC from the monitor

Comment

PRIMARY Peer State 10

Not in Peer State 20

STANDBY Peer State 30

Not in Peer State 40 Start script needs to be modified, see sections 1.4.2.2, 1.4.3 and 1.4.4

Other 100

Table 24 Return Codes from rc.hadr.monitor

Page 43: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 43 of 71

1.4.2.2 DB2 HADR Start Script considerations

Figure 11 illustrates the state transitions between the different states on the standby, and the command arguments for the DB2 takeover command:

State transition for the Standby database

Database Startup

Local catchup

Remote catchup pending

Disconnected Peer

connected

Connection lost

HADR_PEER_WINDOW > 0

Connection lost

HADR_PEER_WINDOW = 0

Remote catchup

Connection

restored or

Peer Window

expired

DB2 HADR takeover command arguments

Discon-

nected

Peer

Peer

Remote

catchup

Remote

catchup

pending

Local

catchup

State

„takeover by

force peer

window only“

„takeover by

force“

„takeover“

Allowed -

high

assurance of

data

consistency

Allowed - no

assurance of

data

consistency

Not allowed

(SQL1770N)

n/aOld primary

marked as

invalid, prevent

commit of new

transactions.

Primary

database and

standby

database switch

roles.

n/aIn

SUPERASYNC

mode only

In

SUPERASYNC

mode only

n/aAllowed, data

loss

Not allowed

n/aNot allowedNot allowed

DB2 HADR takeover command arguments

Discon-

nected

Peer

Peer

Remote

catchup

Remote

catchup

pending

Local

catchup

State

„takeover by

force peer

window only“

„takeover by

force“

„takeover“

Allowed -

high

assurance of

data

consistency

Allowed - no

assurance of

data

consistency

Not allowed

(SQL1770N)

n/aOld primary

marked as

invalid, prevent

commit of new

transactions.

Primary

database and

standby

database switch

roles.

n/aIn

SUPERASYNC

mode only

In

SUPERASYNC

mode only

n/aAllowed, data

loss

Not allowed

n/aNot allowedNot allowed

Peer

Connection lost

Figure 11 State transition for the standby database and DB2 HADR takeover command arguments

The takeover commands are executed out of the script rc.hadr.start (or a copy of it) as part of

the PowerHA application server definition in the cluster configuration: All the commands for DB2 database start and DB2 takeovers are executed out of this script. During execution the script checks

for the status of the database by calling the rc.hadr.monitor command.

In case that the standby database is in non-peer state, rc.hadr.monitor returns return code 40:

The original rc.hadr.start script then has the following code sequence:

elif [ $rc -eq 40 ]; then

# this instance is standby not peer

# Must takeover by force to bring up HADR in Peer state

logger -i -p notice -t $0 "NOTICE: Takeover by force issued"

su - ${instance_to_start?} -c "db2 takeover hadr on db ${DB2HADRDBNAME?} by force"

So in case that the standby database is not in Peer State and a takeover is executed, by default a “takeover by force” is issued. This will lead to data loss. The command sequence in the script needs to be adapted to avoid that situation:

Instead of the default “takeover by force”, a “takeover by force within peer window” (together with a HADR_PEER_WINDOW greater than 0: See also chapter 1.4.2.7) can avoid data loss in some situations. Disadvantage of the peer window is that the availability of the primary

Page 44: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 44 of 71

database is reduced: If the standby is disconnected (e.g. due to a network failure), then the primary pauses log write/ commit activities until the peer window is expired: This ensures that no committed transactions are lost if the primary database fails during the peer window (e.g. in the case of multiple or cascading failures).

There are situations in which data loss can still happen in case of “takeover by force peer window only”:

If the original primary database remains active past the time when the peer window expires, and if the original primary database still has no connection to the standby database, the original primary database will move out of disconnected peer state and resume processing transactions independently. In a PowerHA environment this risk is mitigated by the PowerHA cluster, which has the “SAPDBHOST” service address as part of the resource group: The SAPDBHOST IP address is active only on one cluster node (except in “split-brain” situations), so it is moved away from the original primary to the new “primary” on the previous standby node.

In NEARSYNC mode: If the standby database fails after it has acknowledged the reception of the transaction logs from the primary database, but before it has written that transaction log information to disk. Then that transaction log information might be lost.

This risk can be mitigated by using the SYNC mode.

Example for a code change in rc.hadr.start (to be adapted): …

elif [ $rc -eq 40 ]; then

# this instance is standby not peer

# Must takeover by force to bring up HADR in Peer state

logger -i -p notice -t $0 "NOTICE: Alert: Attempt for DB2 HADR takeover, but

Standby is not in peer state”

# “Disconnected peer”? Then a “takeover by force peer window only” will help

su - ${instance_to_start?} -c "db2 takeover hadr on db ${DB2HADRDBNAME?} by force

peer window only"

if [[ $? –ne 0 ]]

then

# Otherwise Alert (and don’t execute “takeover by force”)

mail –s “EMERGENCY: Attempt for DB2 HADR takeover on ${hostname}, Standby not

in peer state!” ${ALERT_MAIL} <<EOF

Emergency: There is an attempt for a DB2 HADR takeover on ${hostname}, but the Standby is not

in peer state. Please check and handle accordingly

EOF

exit 0

fi

Typically such a change will help to cover situations where “rolling” error conditions occur: At first there is a network failure (and so the state of the standby database changes to “Disconnected Peer”) followed by a server failure of the primary server.

The “duration” t of the “Disconnected Peer” state ranges from

(HADR_PEER_WINDOW – HADR_TIMEOUT) < t < (HADR_PEER_WINDOW) So the HADR_PEER_WINDOW should be set to a higher value than the HADR_TIMEOUT. If the TAKEOVER BY FORCE PEER WINDOW ONLY command is executed when the HADR pair is not in a peer or disconnected peer state (the peer window has expired), an error and a non-zero return code is raised. One potential solution would be to have a follow-on attempt with the TAKEOVER BY FORCE command, which then would lead to data loss. Instead an alert needs to be triggered, and manual interventions are required (See also chapter 1.4.4.1).

Page 45: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 45 of 71

1.4.2.3 Starting the DB2 HADR cluster

Start HADR in Peer state prior to starting the PowerHA service

Make sure that the primary HADR database will initially be running on the cluster node that is supposed to keep the resource group when PowerHA is started (Node1).

To start the standby HADR DB, issue the following command on Node2 o su – db2<sid> o db2 start hadr on db <SID> as standby

To start the primary HADR DB, issue the following command on Node1: o su – db2<sid> o db2 start hadr on db <SID> as primary

The standby HADR DB should always be started prior to starting the primary HADR DB. Then start the PowerHA service on both cluster nodes

smitty clstart, select the node(s) to start the service, press enter.

1.4.2.4 DB2 database configuration parameters

Two parameters will influence the behaviour of DB2 HADR for rebuilding invalid indexes 1.4.2.4.1 LOGINDEXBUILD DATABASE CONFIGURATION PARAMETER

General recommendation for HADR databases is to set the LOGINDEXBUILD database configuration parameter to ON. This ensures that the complete information is logged during index creation, recreation, and reorganization on the primary. The index builds might take longer and will require more log space when having LOGINDEXBUILD set to ON (compared to having LOGINDEXREBUILD not set). However all the indexes will be rebuilt on the standby system directly during HADR log replay and will so be immediately available when a failover takes place. If the index builds on the primary system are not logged and a failover occurs, then any invalid index will have to be rebuilt before the index can be accessed. While the indexes are being recreated, they cannot be accessed by any applications. The LOGINDEXBUILD database configuration parameter is superseded by the table attribute LOG INDEX BUILD: If the LOG INDEX BUILD table attribute is set to its default value of NULL, then DB2 uses the value specified for the LOGINDEXBUILD database configuration parameter. If the LOG INDEX BUILD table attribute is set to ON or OFF, then the value specified for the LOGINDEXBUILD database configuration parameter will be ignored.

If the LOG INDEX BUILD table attribute is set to OFF on one or more tables, any index build operation on those tables might cause the indexes to be recreated any time a takeover operation occurs. Similarly, if the LOG INDEX BUILD table attribute is set to its default value of NULL, and the LOGINDEXBUILD database configuration parameter is set to OFF, any index build operation on a table might cause the indexes on that table to be recreated any time a takeover operation occurs.

Set the LOG INDEX BUILD table attribute to ON, or set the LOG INDEX BUILD table attribute to NULL and the LOGINDEXBUILD configuration parameter to ON on the standby database to ensure that the index recreation will be logged.

1.4.2.4.2 INDEXREC DATABASE CONFIGURATION PARAMETER

General recommendation for HADR databases is to set the INDEXREC database configuration parameter to RESTART (the default) on both the primary and standby databases. This will cause invalid indexes to be rebuilt after a takeover operation is complete. If any index builds have not been

Page 46: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 46 of 71

logged, this setting allows DB2 to check for invalid indexes and to rebuild them. This process takes place in the background, and the database will be accessible after the takeover operation has completed successfully. If a transaction accesses a table that has invalid indexes before the indexes have been rebuilt by the background recreate index process, the invalid indexes will be rebuilt by the first transaction that accesses it. 1.4.2.4.3 DB2BPVARS CONFIGURATION PARAMETER PREC_NUM_WOCB

The DB2 registry variable DB2BPVARS can be set to a full path to a configuration file. In this DB2BPVARS configuration file additional DB2 configuration parameters can be set. The DB2BPVARS configuration parameter PREC_NUM_WOCB controls the number of waiter control blocks (WOCBs). A WOCB is a 64-bytes memory control block which is used by redo workers for synchronization purpose. It is allocated by the DB2 redo master (redom) only for those log records which require some types of synchronization in log replay, e.g. tablespace level synchronization, page level synchronization, … . In some situations the redom process will need to wait for a free entry in the tablespace blocking waiter control block list: This then will impact the HADR standby rollforward performance. If WOCBs are exhausted, the redom has to wait for the redo workers to free some WOCB. The default number of WOCB is 128: Increasing the number of WOCB decreases the likelihood of blocking of redo master, so in these cases the redo performance is improved. For high performance databases, the <CLIENT> value for PREC_NUM_WOCB is set to 2048: Beside additional memory consumption there is no further disadvantage in increasing this parameter. 1.4.2.4.4 DB2BPVARS CONFIGURATION PARAMETER PREC_NUM_AGENTS

The DB2 registry variable DB2BPVARS can be set to a full path to a configuration file. In this DB2BPVARS configuration file additional DB2 configuration parameters can be set. The DB2BPVARS configuration parameter PREC_NUM_AGENTS controls the number of redo workers. Per default, DB2 derives them out of the number of cpus assigned to the LPAR. For large LPARs the default number may result in a too high number of redo workers, which then adversely impacts rollforward performance. To change the number of redo workers the DB2BPVARS variable PREC_NUM_AGENTS can be set accordingly to reduce the overall number.

1.4.2.5 Initial setup of the HADR Standby Database

For the cluster based on HADR, a standby database need to be created first as a copy of the primary database. The SAP homogeneous system copy can be used for setting up the standby database server. For local users such as db2<sid>, <sid>adm, sap<sid>, or sap<sid>db, set the passwords of these users on the second node to the same value as on the primary node (using AIX passwd command).

Log on to the primary node as user db2<sid>. Enable archiving for the logs files: db2 “update db cfg for <SID> using LOGARCHMETH1 DISK:/db2/<SID>/log_arc/”

Both the primary and the standby database needs to be able to retrieve log files from all the log archive locations to which either of the databases might archive log files. To configure log archiving for HADR it is recommended that both the primary and the standby database are configured to have automatic log retrieval capability from all the log archive locations.

Create a directory for a disk backup (e.g. /db2/<SID>/backup) mkdir /db2/<SID>/backup

Create a full database backup to disk (or backup to the EMC net backup)

Page 47: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 47 of 71

db2 “backup db <SID> to /db2/<SID>/backup compress”

Log on to the second node as user root and create the directory /db2/<SID>/backup mkdir /db2/<SID>/backup

Transfer the backup from the first node to the directory /db2/<SID>/backup on the 2nd

node

Start SAPinst on the secondary to install a new DB server

o Use SAPinst parameter SAPINST_USE_HOSTNAME=<VIRTUAL HOSTNAME> to specify the virtual host name that SAPinst uses for the installation. The virtual IP address must reflect to the persistent IP address on the secondary node.

o Select Software Lifecycle Tasks System Copy Target System Distributed System Database Instance in the SAPinst menu.

o For the copy method, choose Homogeneous System Copy so that the backup/ restore method can be used to restore the backup on the standby server.

o On the Summary screen, revise the database communication parameters for the database communication port and the password phrase to ensure that you have the same values as on the primary database server.

o Start the installation process. When the exit step to restore the database for the homogeneous system copy is reached, then exit SAPinst.

All required SAPInst steps are complete now: The subsequent SAPinst phases have already been executed during the SAPinst install of the primary database server.

Restore the database backup from the primary host on the secondary node. Restore the database backup using the command: db2 “restore db <SID> from /db2/<SID>/backup replace history file“

Verify that the database is in rollforward pending state db2 “get db cfg for <SID>” | grep Rollforward

Page 48: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 48 of 71

1.4.2.6 States of the DB2 HADR

Figure 12 illustrates the transition between the different states of the DB2 HADR standby database:

Figure 12 The different states of the DB2 HADR standby database

After the standby database is started on the node, it processes all the log files which are available in the local log directories (“Local catchup”). Then it starts the connection to the primary database (“Remote catchup pending”). If the connection to the primary is established, the standby database receives and processes the log files from the primary (“Remote catchup”) until the log gap is closed. Now the standby database receives and processes the actual log files from the primary (“Peer State”). To safeguard continuous operation and the ability for the activation of the standby without any data loss, it is very important that the standby database is always kept in peer state. The DB2 HADR state needs to be monitored and included into the alerting procedures. The table summarizes the behaviour while issuing the TAKEOVER command on the standby database.

If the standby database is in “local catchup” or “remote catchup” state, then it cannot be activated using the takeover HADR command, even if the “by force” clause is used. An error message is returned.

If the standby database is in “disconnected peer” state, it can be activated only using the “by force clause”. A greater degree of data consistency can be assured by implementing a peer window, and using the “within peer window” clause for the takeover command. As data consistency is improved with this option, and then a standstill of the standby database will impact the production database also.

If the standby database is in “remote catchup pending” state, it can be activated only using the by force clause.

If the standby database is in peer state, then the preferred activation of the standby shouldn’t use the “by force” clause. Only if the first attempt fails, then the “by force” clause should be executed afterwards in a second attempt.

Page 49: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 49 of 71

Table 25 shows an overview of the behaviour when the “TAKEOVER HADR” command is executed on the standby node:

Standby state BY FORCE option used

Takeover behavior

Disconnected peer No NOT ALLOWED (SQL1770N)

Local catchup or remote catchup

No Error message returned

Peer No Primary database and standby database switch roles. If no failure is encountered during takeover, there will be no data loss. However, if failures are encountered during takeover, data loss might occur and the roles of the primary and standby might or might not have been changed. The following is a guideline for handling failures during a takeover in which the primary and standby switch roles: If a failure occurs during a takeover operation, the roles of the HADR databases might or might not have been changed. If possible, make sure both databases are online. Check the HADR role of the available database or databases using the Snapshot Monitor, or by checking the value of the database configuration parameter hadr_db_role. If the intended new primary is still in standby role, and takeover is still desired, re-issue the TAKEOVER HADR command (see the next guideline regarding the BY FORCE option). It is possible to end up with both databases in standby role. In that case, the TAKEOVER HADR command with the BY FORCE option can be issued at whichever node should now become the primary. The BY FORCE option is required in this case because the two standbys cannot establish the usual HADR primary-standby connection.

Remote catchup pending

No Error message returned.

Disconnected peer No NOT ALLOWED (SQL1770N)

Table 25 DB2 Takeover command executed on standby (w/o “BY FORCE” clause)

Table 25 DB2 Takeover command executed on standby (w/o “BY FORCE” clause)Table 26 shows an overview of the behaviour when the “TAKEOVER HADR BY FORCE” command is executed on the standby node:

Standby state BY FORCE option used

Takeover behavior

Disconnected peer Yes, just BY FORCE

ALLOWED – NO ASSURANCE OF DATA CONSISTENCY Note: A "no transaction loss" takeover is also possible using the TAKEOVER BY FORCE command without the PEER WINDOW ONLY option, i.e., unconditional failover, as long as the necessary conditions hold. Such a failover can be executed even long after the expiration of the peer window that was in effect when the primary failed.

Disconnected peer Yes, BY FORCE PEER WINDOW ONLY

ALLOWED - GREATER DEGREE OF DATA CONSISTENCY There are situations in which data loss can still happen: If the primary database remains active past the time when the peer window expires, and if the primary database still has no connection to the standby database, the primary database will move out of disconnected peer state and resume

Page 50: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 50 of 71

Standby state BY FORCE option used

Takeover behavior

processing transactions independently. In NEARSYNC mode, if the standby database fails after acknowledging receipt of transaction logs from the primary database but before writing that transaction log information to disk, then that transaction log information, in the log receive buffer, might be lost.

Local catchup or remote catchup

Yes Error message returned

Peer Yes Old primary marked as invalid, preventing it from writing any more logs or committing any more transactions. The next log write attempt brings down the database. However, if sessions on the old primary only run read-only queries, the old primary might stay up indefinitely. Existing client connections stay open as long as they perform read-only operations and new client connections might be accepted. To avoid a situation of dual-primary, it is recommended that you stop transaction processing on old primary first before

issuing a TAKEOVER HADR command with the BY FORCE

option.

Remote catchup pending

Yes The standby becomes a primary.

Table 26 DB2 Takeover command executed on standby using the “BY FORCE” clause

1.4.2.7 DB2 HADR Peer-window

DB2 HADR has introduced a new database configuration parameter “HADR_PEER_WINDOW”, which affects the behaviour in case primary and standby database loose their connection If the HADR_PEER_WINDOW database configuration parameter is set to a non-zero value and the primary database loses connection to the standby database then the DB2 HADR primary and standby database pair will still behave as if it would still be in peer state during this time window. In “peer state”, any transaction is not considered committed until the primary database receives a confirmation from the standby database: In “Nearsynch” mode the standby database confirms that the database logs have been written to memory, in “Synch” mode the standby database confirms that the database logs have been written to the local log path. This helps to ensure data consistency: If there is a failure on the primary database, then all transaction information that was logged in the database logs on the primary database is available in the database logs on the standby database also. When the primary and standby databases are in peer state, and the primary database loses the connection to the standby database, then transactions cannot be committed because the primary database cannot receive the confirmation from the standby database for any transactions any more. In previous versions of IBM DB2, the primary database automatically moved into “disconnected” state by itself and then continued to process and commit application transactions now fully independent of the standby database. The standby database got out of sync (“remote catchup pending”). If the primary database failed while it was processing transactions independently of the standby database, transaction information on the primary database could be lost during an activation of the standby “by force” If the HADR_PEER_WINDOW database configuration parameter is set to a non-zero value, and the primary loses connection with the standby database then the primary database will move from “peer state” to a new “disconnected peer state”. When the primary database is in “disconnected peer state”, it continues to behave as if it was in peer state: It still will waiting for a confirmation from the standby database before it is committing any transaction. The time period for which the primary database

Page 51: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 51 of 71

remains in “disconnected peer state” is called the peer window. If communication between primary and standby is re-established within the peer window, then primary and standby database will stay in sync.

The HADR_PEER_WINDOW value is considered only in SYNC or NEARSYNC mode

HADR_PEER_WINDOW need to be set to the same value on both primary and standby databases.

A recommended minimum value is 120 seconds.

To avoid impacting the availability of the primary database when the standby database is intentionally shut down, for example, for maintenance, the peer window is not invoked if the standby database is explicitly deactivated while the HADR pair is in peer state.

The TAKEOVER HADR command with the PEER WINDOW ONLY option will launch a takeover operation only if the HADR standby is presently inside the defined peer window.

The takeover operation with the PEER WINDOW ONLY option may behave incorrectly if the primary database clock and the standby database clock are not synchronized to within 5 seconds of each other. A time synchronization service (for example, NTP) should be used to keep the clocks synchronized.

On the standby databases, the peer window end time is based on the last heartbeat message received from the primary database rather than disconnection. Therefore, the standby database's remaining time in disconnected peer state before transition to remote catchup pending state ranges from (HADR_PEER_WINDOW - hadr_timeout) seconds to (HADR_PEER_WINDOW) seconds, depending on when and how the disconnection occurred.

Although the availability of the primary database is reduced during the peer window, no committed transaction is lost if the primary database failed during the peer window as in the case of multiple or cascading failures.

1.4.2.8 DB2 HADR parameters affecting timeouts

There are different DB2 HADR parameters which play a role in communication and timeouts between primary and standby database:

HADR_TIMEOUT database configuration parameter

HADR_PEER_WINDOW database configuration parameter (See also chapter 1.4.2.7).

Registry variable DB2_HADR_PEER_WAIT_LIMIT 1.4.2.8.1 HADR_TIMEOUT DATABASE CONFIGURATION PARAMETER

The HADR_TIMEOUT configuration parameter determines how long an HADR database waits before considering its connection with the partner database as broken. If a HADR database does not receive any communication from its partner database for longer than the time interval defined by the HADR_TIMEOUT parameter, then the database concludes that the connection with the partner database is lost. If the database is in peer state when the connection is lost and

the HADR_PEER_WINDOW database configuration parameter is greater than zero then the database changes to disconnected peer state.

the HADR_PEER_WINDOW database configuration parameter is zero then the database changes to remote catchup pending state

Page 52: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 52 of 71

The state change applies to both primary and standby databases. Figure 13 illustrates the behaviour for a temporary connection loss when the HADR_PEER_WINDOW is set to 0:

In the example the last HADR heartbeat is exchanged at t0, and the HADR timeout window starts to decay. HADR mode is SYNC or NEARSYNC mode, so the primary database is blocked for updates/ deletes as the Standby doesn't process any logs for now.

At t1 the HADR timeout is expired: The standby falls back into "Remote Catchup Pending" state. The primary is no longer blocked, and continues production operation.

In "Remote Catchup Pending" state, the Standby Database attempts to reconnect to the primary DB once again. (Re-)Connection is successful at t2

HADR standby database changes its state to "Remote Catchup ": All the log records generated since t1 on the primary are applied to the Standby

At t3 the Standby database has caught-up all out-standing logs, and is in synch with the primary: Peer State is reached again.

Setting the HADR_TIMEOUT value too high may block the Primary database unnecessary long. (HADR will break the connection immediately as soon as a “low level” network error is detected on the TCP socket during a send, receive, or poll operations: HADR polls the socket every 100 milliseconds. This allows responding quickly to a network error detected by the OS. Only in the worst case HADR will wait until HADR_TIMEOUT expiry to break a bad connection). If the HADR_TIMEOUT value is set too small, then the state change from “Peer State” to “Remote Catchup pending” may happen unnecessary often, and may each time contribute with a "blocking for HADR timeout".

DB Server HA Protection Standby not in sync

Primary active Primary blocked

Peer State

Remote Catchup Pending

Peer State

Remote Catchup

t0 t1 t2 t3

t

HADR timeout HADR timeout

0 0 0 1 2 0 0 0 1 2 3 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0

hadr_heartbeat monitor value:

HADR heartbeats; The heartbeat interval is one quarter of the value of the hadr_timeout database

Configuration parameter, or 30 seconds, whichever is shorter.

No heartbeat

received Attempting to reconnect

Retrieve logs from

primary and process

them on the standby

DB Server Availability

Figure 13 HADR timeout illustration (HADR_PEER_WINDOW = 0)

Page 53: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 53 of 71

1.4.2.8.2 HADR_PEER_WINDOW DATABASE CONFIGURATION PARAMETER

The HADR_PEER_WINDOW configuration parameter does not replace the HADR_TIMEOUT configuration parameter. If the primary database loses connection with the standby database and the HADR_PEER_WINDOW parameter is set to a non-zero time value, then a HADR primary-standby database pair continues to behave as though still in peer state (disconnected peer state) for the configured amount of time. In this case, a database application that is running at the time of failure can be blocked for a period of time equal to the sum of the HADR_TIMEOUT and HADR_PEER_WINDOW database configuration parameters. This helps to ensure data consistency. Figure 14 illustrates the behaviour for a temporary connection loss when the HADR_PEER_WINDOW is set greater than 0:

In the example the last HADR heartbeat is exchanged at t0, and the HADR timeout window starts to decay. HADR mode is SYNC or NEARSYNC mode, so the primary database is blocked for updates/ deletes as the Standby doesn't process any logs for now.

At the t1 the HADR timeout is expired. The standby database changes into disconnected peer state. HADR_PEER_WINDOW starts to decay.

o In case of a potential takeover in disconnected peer state, the standby remains fully consistent: No data loss will occur.

o In case the connection with the primary is re-established within the HADR_PEER_WINDOW, then the standby database state changes via "Remote Catchup Pending" and “Remote Catchup State” back into “Peer State”. As the standby is still in sync this sequence of state changes is very fast. Back in “Peer State” the primary is no longer blocked, and fully continues production operation.

o At t2 the HADR_PEER_WINDOW is expired: The database changes into “Remote Catchup Pending”

In "Remote Catchup Pending" state, the Standby Database attempts to reconnect to the primary DB once again. (Re-)Connection is successful at t3

HADR standby database changes its state to "Remote Catchup ": All the log records generated since t2 on the primary are applied to the Standby

At t4 the Standby database has caught-up all out-standing logs, and is in synch with the primary: Peer State is reached again.

DB Server Availability

DB Server HA Protection Standby not in sync

Primary active Primary blocked

Peer State

Remote Catchup

Pending

Peer State

Remote

Catchup

t0 t1 t3 t4

t

HADR timeout

No heartbeat

receivedAttempting

to reconnect

HADR Peer Window

Disconnected Peer

t2

Figure 14 HADR timeout illustration (HADR_PEER_WINDOW > 0)

Page 54: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 54 of 71

It is desirable to keep the waiting time that a database application experiences to a minimum. Setting the HADR_TIMEOUT and HADR_PEER_WINDOW configuration parameters to small values reduces the “blocking time” that a database application must wait in case the HADR standby databases loses its connection with the primary database. However two additional details should be considered when choosing values for the HADR_TIMEOUT and HADR_PEER_WINDOW configuration parameters:

The HADR_TIMEOUT database configuration parameter should be set to a value that is long enough to avoid false alarms on the HADR connection caused by short, temporary network interruptions. Default value of HADR_TIMEOUT is 120 seconds, which is a reasonable value for many networks.

The HADR_PEER_WINDOW database configuration parameter should be set to a value that is long enough to allow the system to perform automated failure responses. If the HA system, for example a cluster manager, detects primary database failure before disconnected peer state ends, a failover to the standby database takes place. Data is not lost in the failover as all data from old primary is replicated to the new primary. If HADR_PEER_WINDOW is too short, the HA system may not have enough time to detect the failure and respond.

1.4.2.8.3 REGISTRY VARIABLE DB2_HADR_PEER_WAIT_LIMIT

When the DB2_HADR_PEER_WAIT_LIMIT registry variable is set, HADR primary database breaks out of peer state if logging on the primary database has been blocked for the specified number of seconds because of log replication to the standby. When this limit is reached, the primary database breaks the connection to the standby database. If peer window is disabled, the primary enters disconnected state and logging resumes. Figure 15 illustrates the behaviour for the DB2_HADR_PEER_WAIT_LIMIT (HADR_PEER_WINDOW is zero in the example)

In the example the logging on the primary database gets blocked at t0 because of log replication to the standby, and the DB2_HADR_PEER_WAIT_LIMIT timeout window starts to decay.

At the t1 the DB2_HADR_PEER_WAIT_LIMIT is expired: The primary forces a disconnect of the standby. The standby database changes into “Remote Catchup Pending” State

In "Remote Catchup Pending" state, the Standby Database attempts to reconnect to the primary DB once again. (Re-)Connection is successful at t2

HADR standby database changes its state to "Remote Catchup ": All the log records generated since t1 on the primary need to be applied to the Standby to close the log gap

At t3 the Standby database has caught-up all out-standing logs, and is in synch with the primary: Peer State is reached again.

Page 55: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 55 of 71

DB Server HA Protection Standby not in sync

Primary active Primary blocked

Peer State

Remote Catchup

Pending

Peer State

Remote Catchup

t0 t1 t2

t

Force

Disconnect

Standby

Attempting to

reconnect

Retrieve logs from

primary and process

them on the standby

DB Server Availability

DB2_HADR_PEER_WAIT_LIMIT

Figure 15 DB2 Registry DB2_HADR_PEER_WAIT_LIMIT (HADR_PEER_WINDOW = 0)

If peer window is enabled (HADR_PEER_WINDOW > 0), the primary database enters disconnected peer state, in which logging still continues to be blocked. The primary leaves disconnected peer state upon re-connection or expiration of the peer window. Logging resumes after the primary leaves disconnected peer state. Honouring peer window transition when breaking out of peer state ensures peer window semantics for safe takeover in all cases. If the primary fails during the transition, normal peer window protection still applies (safe takeover from standby as long as it is still in disconnected peer state). On the standby side, after disconnection, the database continues replaying already received logs. After the received logs have been replayed, the standby reconnects to the primary. Upon re-connection, normal state transition follows (first remote catchup state, then peer state) Relationship to HADR_TIMEOUT The HADR_TIMEOUT database configuration parameter does not break the primary out of peer state if the primary keeps receiving heartbeat messages from the standby while blocked. HADR_TIMEOUT is a timeout for the HADR network layer. An HADR database breaks the connection to its partner database if it has not received any message from its partner for the HADR_TIMEOUT period. It does not control timeout for higher layer operations such as log shipping and acknowledgement. If log replay on the standby database is stuck on a large operation such as load or reorganization, the HADR component still sends heartbeat messages to the primary database on normal schedule. In such a scenario, the primary is blocked as long as the standby replay is blocked, unless DB2_HADR_PEER_WAIT_LIMIT is set. DB2_HADR_PEER_WAIT_LIMIT unblocks primary logging regardless of connection status. Note that even if DB2_HADR_PEER_WAIT_LIMIT is not set, the primary always breaks out of peer state when a network error is detected or the connection is closed (possibly as result of HADR_TIMEOUT).

1.4.2.9 Update SAPDBHOST to a new virtual IP

If a new virtual IP address shall be implemented for the DB2 HADR database, the following steps need to be checked/ performed.

Check the SAP DEFAULT profile:

Page 56: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 56 of 71

In the default profile /sapmnt/<SID>/profile/DEFAULT.PFL the values for SAPDBHOST and j2ee/dbhost need to reflect the virtual IP address of the database server.

Check the DB2 CLI driver settings in file db2cli.ini virtual IP address of the database server

Check environment files of users <sid>adm and sap<sid> The environment of the users <sid>adm and sap<sid> has to be adapted because most of the environment source files contain the host name as part of their file name.

Check the catalog entries on all SAP NetWeaver instances: o Log on to the host of an SAP NetWeaver instance as user db2<sid>.

Generate a list of all catalog entries and look for an entry NODE<SID> by entering the following command:

db2 “list node directory”

Uncatalog the reference to the old host name using the following command: db2 “uncatalog node NODE<SID>”

Catalog the reference to the new virtual host name using the command: db2 “catalog tcpip node NODE<SID>remote <virtual_hostname> server <servicename>”

<servicename> stands for sapdb2<SID> (as reflected in the /etc/services for the port)

Repeat this procedure for all SAP NetWeaver instances.

Check the JDBC URL: NOTE The following steps only apply for ABAP+Java and Java systems.

o Log on to the host of an SAP NetWeaver instance as user root. o Change to the directory of the configuration tool using the following command:

cd /usr/sap/<SID>/<Instance>/j2ee/configtool

o Run the configuration tool using the following command:./configtool.sh In the left frame, choose security store. In the right frame, choose the key jdbc/pool/<SID>/url. Change the host name in the JDBC URL to the virtual host name.

EXAMPLE

URL with the old host name:jdbc:db2://db_hostname:5912/TSP:deferPrepares=0 URL with the new virtual host name:jdbc:db2://db_virtual_ip:5912/TSP:deferPrepares=0

Choose Add. To save your changes, click the disk icon in the upper left corner. Close the configuration tool.

o Restart the Java instance.

1.4.2.10 Check Failure Detection Rate of heartbeat device(s)

An important tuning parameter for PowerHA network modules is the Failure Detection Rate, which determines how quickly a connection is considered to have failed. The Failure Detection Rate of a

Page 57: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 57 of 71

Network Module can be changed for each network module either to predefined values of Fast, Normal and Slow, or to custom values. The Failure Detection Rate consists of two components:

Cycles to fail (cycle) The number of heartbeats missed before detecting a failure

Heartbeat rate (hbrate). The number of seconds between heartbeats.

The time needed to detect a failure can be calculated using this formula: (heartbeat rate) * (cycles to fail) * 2 The Failure Detection Rate should be set lower than 15 seconds for each heartbeat device. Check the failure detection rate with the “cllsnim” command: /usr/es/sbin/cluster/utilities/cllsnim -g -n'ether'

Current values for Network Module <ether>:

Description [Ethernet Protocol]

Parameters []

Grace Period 60

Supports gratuitous arp [True]

Type Address (IP)

Entry type [adapter_type]

Next generic type [transport]

Next generic name [Generic_UDP]

Supports source routing True

Failure Cycle 3

Heartbeat Rate (seconds) 2.00

Failure Detection Rate (Custom)

Failure rate is 3 * 2.00 * 2 = 12 seconds Check the following devices also: /usr/es/sbin/cluster/utilities/cllsnim -g -n'rs232' /usr/es/sbin/cluster/utilities/cllsnim -g -n'diskhb' The Failure Detection Rate can be adjusted via:

# smitty hacmp Extended Configuration

Extended Topology Configuration Configure HACMP Network Modules

Change a Network Module using Custom Values

1.4.3 DB2 HADR performance considerations

General DB2 HADR performance advices include:

Adjust DB2 HADR Receive Buffer By default, the log receive buffer size on the standby database will be two times the value specified for the LOGBUFSZ configuration parameter on the primary database. If the primary database is experiencing a high transaction load, then the log receive buffer on the standby

Page 58: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 58 of 71

database might fill to capacity and the log shipping operation from the primary database might stall. In case the HADR standby receive buffer gets close to its limit during peak load then increase its size on the standby database (DB2_HADR_BUF_SIZE registry variable). This will improve the insert/update rate and reduce the number of congestion states.

Ensure that network bandwidth is greater than the database log generation rate In remote catch-up state, the primary database reads logs from log files in the log path or the log archive path and sends them onto the network. It is common that

o the log reading rate is higher than the network send rate, or o the minimum of log read rate and network send rate is higher than the log replay rate

on the standby In both cases the primary database send will end up with congestion. It is normal to see congestion going on and off in remote catch-up state. In most systems the logging capability will not be driven to its limit during peer state: Even in SYNC mode, there might not be an observable slow down on the primary database (See also “Handle network congestion”).

Carefully consider the DB2 HADR synchronization mode Network delays affect the primary only in SYNC and NEARSYNC modes. SYNC mode may impact the system performance significantly higher than the other synchronization modes. In general, four synch modes are available for DB2 HADR.

o In SYNC mode, the primary database sends the log pages to the standby database after they have been successfully written to the primary database log path. The primary database waits for an acknowledgement from the standby before notifying an application that a transaction was prepared or committed. The standby database sends the acknowledgement after it writes the received log pages to the standby database log path. The performance overhead equals the time needed for writing the log pages on the standby database plus the time needed for sending the messages back to the primary.

o In NEARSYNC mode, the primary database writes and sends log pages in parallel. The primary then waits for an acknowledgement from the standby. The standby database acknowledges as soon as the log pages are received into its memory (standby receive buffer). On a fast network with low latency the overhead to the primary database is minimal. The acknowledgement might have already arrived by the time the primary database finishes the local log write.

o For ASYNC mode, the log write and send are also in parallel; however, in this mode the primary database does not wait for an acknowledgement from the standby. Therefore, network delay is not an issue. Performance overhead is even smaller with ASYNC mode than with NEARSYNC mode. Compared to “standard” DB2 there is still a small gap however as the primary database needs to wait until the log records are passed over to the TCP/IP stack.

o For SUPERASYNC mode, transactions are never blocked or experience elongated response times due to network interruptions or congestion. New transactions can be processed as soon as previously submitted transactions are written to the primary database. Therefore, network delay is not an issue. The elapsed time for the completion of non-forced takeover operations might be longer than in other modes because the log gap between the primary and the standby databases might be relatively larger.

ASYNC or SUPERASYNC modes are not suitable for covering <CLIENT>’s’ high-availability requirements: SYNC mode or NEARSYNC modes should be used for a critical production environment. Only these modes will guarantee data integrity and no loss of committed transactions on the standby in case of a takeover.

Page 59: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 59 of 71

Consider tuning DB2_HADR_SOSNDBUF and DB2_HADR_SORCVBUF registry variables HADR log shipping workload, network bandwidth, and transmission delay are important factors to consider for tuning the TCP socket buffer sizes. In previous DB2 versions the TCP socket buffer size could only be changed on a per system level impacting all TCP connections. Two new registry variables, DB2_HADR_SOSNDBUF and DB2_HADR_SORCVBUF were introduced in Fix Pack 2 and allow the tuning of the TCP socket send and receive buffer size explicitly for HADR connections. These two variables have the value range of 1024 to 4294967295 and default to the socket buffer size of the operating system. The HADR simulator (a command-line tool that generates a simulated HADR workload) can be used to measure network performance and to simulate various network tuning options. The simulator can be downloaded via http://www.ibm.com/developerworks/wikis/display/data/HADR_sim

Consider adjusting the PREC_NUM_WOCB parameter WOCB is a 64-bytes memory control block which is used by redo workers for synchronization purpose. It is allocated by redo master, only for those log records which require some types of synchronization in log replay, e.g. tablespace level synchronization, page level synchronization. The default number of WOCB is 128. If the WOCBs are exhausted, then the redo master has to wait for redo worker to free-up some WOCB. Increasing the number of WOCB decreases the likelihood of blocking of redo master, so it can help in improving redo performance. There is no disadvantage in increasing this parameter except the additional memory consumption.

Identify hot tables and move them into own tablespaces with a large extent size Identify *hot* tables in the table spaces: To avoid table space level blocking on the redo on this tables move the hot tables to separate table spaces with large extent size (change the extent size to 32). Tables which are growing very fast will frequently allocate additional extents. For the log replay on the standby database the allocation will block all further activities in the tablespace. If there are too many tables that grow very fast and are located within one common tablespace this issue might re-occur - even with an extent size of 32 for the tablespace: So not too much hot tables should be located within the same tablespace. The larger extent size will result in larger prefetch sizes too: This may have an impact on overall bufferpool quality, and the PREFETCHSIZE may need to be adjusted.

Identify large indices and move them into own tablespaces with a large page size In case of LOGINDEXBUILD=ON invalid indices are rebuild immediately after a DB2 HADR takeover on the “new” primary database. Placing large indices into own tablespaces with a large page size will optimize this process.

Prepare to handle network congestion For each log write on the primary, the log pages are also sent to the standby: Each write operation is called a flush. The size of the flush is limited to the log buffer size on the primary database (database configuration parameter logbufsz). The exact size of each flush is nondeterministic: A larger log buffer does not necessarily lead to a larger flush size. If the standby database is too slow in replaying log pages, then its log-receiving buffer might fill up. This will prevent the buffer from receiving more log pages. In SYNC and NEARSYNC modes the data will likely be buffered in the network pipeline consisting of the primary machine, the network, and the standby database. Because the standby database does not have free buffer to receive the data, it cannot acknowledge, so the primary database becomes blocked while waiting for the standby database's acknowledgement. In ASYNC mode, the primary database continues to send log pages until the pipeline fills up and it cannot send additional log pages. This condition is called congestion. Congestion is reported by the hadr_connect_status monitor element. For SYNC and NEARSYNC modes,

Page 60: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 60 of 71

the pipeline can usually absorb a single flush and congestion will not occur. However, the primary database remains blocked waiting for an acknowledgement from the standby database on the flush operation. Congestion can occur also if the standby database is replaying log records that take a long time to replay, such as database or table reorganization log records. In SUPERASYNC mode, since the transaction commit operations on the primary database are not affected by the relative slowness of the HADR network or the standby HADR server, the log gap between the primary database and the standby database might continue to increase. It is important to monitor the log gap as it is an indirect measure of the potential number of transactions that might be lost should a true disaster occur on the primary system. In disaster recovery scenarios, any transactions committed during the log gap would not be available to the standby database. Therefore, monitor the log gap by using the hadr_log_gap monitor element; if it occurs that the log gap is not acceptable, investigate the network interruptions or the relative speed of the standby HADR server and take corrective measures to reduce the log gap.

o Recommendations for minimizing network congestion:

The standby database should be powerful enough to replay the logged operations of the database as fast as they are generated on the primary. Consider tuning the size of the standby database log-receiving buffer using the DB2_HADR_BUF_SIZE registry variable. A larger buffer can help to reduce congestion, although it might not remove all of the causes of congestion. By default, the size of the standby database log-receiving buffer is two times the size of the primary database log-writing buffer. The database configuration parameter logbufsz specifies the size of the primary database log-writing buffer. Use the “db2pd –hadr” command to check if the standby's log-receiving buffer is inadequate: If the value for StandByRcvBufUsed, which indicates the percentage of standby log receiving buffer used, gets close to 100%, then the DB2_HADR_BUF_SIZE should be increased. Consider setting the DB2_HADR_PEER_WAIT_LIMIT registry variable, which allows preventing primary database logging from blocking because of a slow or blocked standby database. DB2_HADR_PEER_WAIT_LIMIT was introduced in DB2 Version 9.5 Fix Pack 1. When registry variable DB2_HADR_PEER_WAIT_LIMIT is set, then the HADR primary database will break out of the peer state if logging on the primary database has been blocked for the specified number of seconds because of log replication to the standby. When this limit is reached, the primary database will break the connection to the standby database. If the peer window is disabled, the primary will enter disconnected state and logging resumes. If the peer window is enabled, the primary database will enter disconnected peer state, in which logging continues to be blocked. The primary database leaves disconnected peer state upon re-connection or peer window expiration. Logging resumes once the primary database leaves disconnected peer state. Honouring peer window transition when breaking out of peer state ensures peer window semantics for safe takeover in all cases. If the primary fails during the transition, normal peer window protection still applies (safe takeover from standby as long as it's still in disconnected-peer state).

1.4.4 DB2 Log File considerations

If the primary database fails while in peer state or disconnected peer state and the synchronization mode is synchronous (SYNC) then the standby database will not lose any transaction that was reported as committed to the SAP application before the primary database failed.

Page 61: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 61 of 71

If the primary database fails while in peer state or disconnected peer state and the synchronization mode is near-synchronous (NEARSYNC) then the standby database will not lose any transaction that was reported as committed to the SAP application before the primary database failed as long as the standby wasn’t impacted too during the failure of the primary. The cluster will handle this situation fully automatically and no additional measures required. If the storage of the primary database fails while not in peer state, then the standby database must be activated by a “TAKEOVER BY FORCE” to get it online. In such a situation (“not in peer state”) there would be data loss! Additional measures are required to prevent from such a situation. If the DB2 Online Logs are mirrored across both storage systems (AIX LVM), then the latest information may get available on the secondary node. This will require manual expert actions. (See Figure 16)

Storage 2Storage 1

PowerHA Cluster

LPAR #3 LPAR #4

saphostagent saphostagent

DB2 UDB

(primary)

DB2 UDB

(standby)

takeoverSMDA96

Storage 2Storage 1

DB2 HADR

DB #1: Active Logs

(AIX LVM mirrored) Storage 1 Storage 2

DB #2: Active Logs

(AIX LVM mirrored)Storage 1 Storage 2

DB #1: Database data DB #2: Database data

AIX LVM

AIX LVM

Figure 16 AIX LVM mirror of DB2 online logs – normal situation

If <STORAGE SYSTEM>-1 fails (Figure 17), the standby database needs to be activated as the new primary:

If HADR was in peer state, this is a fully automated action controlled by PowerHA and the PowerHA application server scripts.

If primary DB fails and the standby DB is not in peer state:

o No „immediate“ takeover by force shall be executed

o The application server script needs to be adapted: In the rc.hadr_start script the “takeover by force” action needs to be replaced by “notify” and “exit” commands in the clause handling the start of the primary when not in peer state.

Page 62: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 62 of 71

Storage 2Storage 1

PowerHA Cluster

LPAR #3 LPAR #4

saphostagent saphostagent

DB2 UDB

(primary)

DB2 UDB

(standby)

takeoverSMDA96

Storage 2Storage 1

DB2 HADR

DB #1: Active Logs

(AIX LVM mirrored) Storage 1 Storage 2

DB #2: Active Logs

(AIX LVM mirrored)Storage 1 Storage 2

DB #1: Database data DB #2: Database data

AIX LVM

AIX LVM

Figure 17 Crash of <STORAGE SYSTEM>-1

The manual high-level steps then include the following actions (Figure 18): (1) Manual access „latest“ online logs to LPAR #4

• Adapt LUN masking on <STORAGE SYSTEM>-2 to assign the volumes to LPAR#4 • Eventually adapt SAN zoning (to be able to attach the volumes on LPAR #4) • Import AIX volume group (forced, as only one copyset is left) • Mount the log_directory on a new mountpoint

(2) Manual Transfer of latest logs from DB#1 to log directory of DB #2 (3) Manual Handling of DB Start and recovery

Page 63: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 63 of 71

Storage 2Storage 1

PowerHA Cluster

LPAR #3 LPAR #4

saphostagent saphostagent

DB2 UDB

(primary)

DB2 UDB

(standby)

takeoverSMDA96

Storage 2Storage 1

DB2 HADR

DB #1: Active Logs

(AIX LVM mirrored) Storage 1 Storage 2

DB #2: Active Logs

(AIX LVM mirrored)Storage 1 Storage 2

DB #1: Database data DB #2: Database data

AIX LVM

AIX LVM

1)

2)

3)

Figure 18 Manual activation

1.4.4.1 Description of the high-level recovery procedure

This alert needs to trigger the manual execution of the following high-level procedure by the expert team:

Stop SAP Application

Check the current state of the PowerHA cluster, and stop the PowerHA cluster (“Unmanage resource groups”) on the LPAR of the Standby Database

Deactivate the standby database using the “DEACTIVATE DATABASE” command.

Stop HADR on the standby database using the “STOP HADR” command

Database role changes to standard. Database configuration parameter hadr_db_role is updated to STANDARD. Database remains offline. Database is put into rollforward pending mode.

Make the latest database logs from the primary database available for rollforward recovery of the standby database

o This may include a “forced” import of the remaining disks on the 2nd storage system

(containing one AIX LVM copyset of the logical volume of the primaries’’ log directory) to the standby machine:

o Grant access to this volumes to the LPAR of the standby (LUN masking in the EMC storage)

o Clear any outstanding reservation (which would inhibit access to the standby LPAR) using EMC tools

o Run an AIX configuration manager on the LPAR of the standby (“cfgmgr”). The volumes are recognized as new disk devices in AIX on the standby LPAR

o Check access to the new disk devices using “low level” commands

Page 64: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 64 of 71

(e.g. lquerypv –h /dev/hdiskX ) o Forced importvg of the Volume Group

(importvg –f hdiskX ) o Change the mount point for the log filesystem in the new VG

(as it would be a duplicate of the existing log directory of the standby database) o Mount the log filesystem of “primaries’’ rescued log volume group” on the standby

A “fsck” may be required o Save the content of the log directory of the standby

Copy the “rescued” log files to a log path included in the rollforward recovery process of the standby.

Rollforward the database o ROLLFORWARD DATABASE <DB> to END OF LOGS

Specifies that all committed transactions from all online archive log files listed in the database configuration parameter logpath are to be applied. Inactive log files which were already archived to the backup environment may need to be retrieved from there.

o Check with ROLLFORWARD DATABASE <DB> QUERY STATUS Lists the log files that the database manager has rolled forward, the next archive file required, and the time stamp (in UTC) of the last committed transaction since rollforward processing began.

o Complete Rollforward ROLLFORWARD DATABASE <DB> COMPLETE Stops the rolling forward of log records, and completes the rollforward recovery process by rolling back any incomplete transactions and turning off the rollforward pending state of the database. This allows access to the database or table spaces that are being rolled forward.

Activate and Check the database o The database is started as a “STANDARD” database and doesn’t participate in a

HADR scenario any longer. This needs to be set up once again when the “previous” primary is repaired/ rebuilt. An additional downtime may be required to re-establish the PowerHA cluster, as there were a lot of changes due to this “emergency” procedure.

Perform a full backup of the database This full backup is a new starting point for further database recoveries and also for re-building the standby database. To optimize backup duration a “split-mirror” backup maybe used.

Acquire the virtual IP address (“SAPDBHOST”) on the Standby LPAR Manual activation of the IP alias (ifconfig …) in case it is not already activated on the Standby LPAR

Start SAP

Clean-Up o Unmount “rescued” log directory o Varyoff “rescued volume group” o Export “rescued” volume group o Remove LUN masking for these volumes

Page 65: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 65 of 71

1.4.5 PowerHA Planning Sheet

The following text describes a planning sheet for Cluster Topology and Resources (example of <SID>): Cluster Topology Cluster Name

cl_sap_< SAP Component > respectively cl_db_< SAP Component>

Cluster Nodes Communication path to node

<node 1> <IP address of node 1 (persistent)>

<node 2> <IP address of node 2 (persistent)>

Network Name Network type Netmask Enable IPAT via Alias IP Address offset for

Heartbeating

Access_net <SAP Component>_net True n/a

Serial RS232 Network (Non-IP)

<SAP Component>_rs232 n/a False a/a

net_diskhb_01 <SAP Component>_diskhb1 n/a False n/a

net_diskhb_02 <SAP Component>_diskhb2 n/a False n/a

Name Type Network Nettype Shared IP Address Node

<node 1> Persistent Access_net ether No n/a <node 1>

<node 2> Persistent Access_net ether No n/a <node 2>

<SAP Component>nfs Service Access_net ether Yes n/a n/a

<SAP Component>scs Service Access_net ether Yes n/a n/a

<SAP Component>db Service Access_net ether Yes n/a n/a

Cluster Resources Resource Group Startup

Behavior Failover Behavior Fallback

Behavior Participating Nodes

rg_<SAP Component>_db Home Node Next Priority Node Never Fallback <node 1> <node2>

rg_< SAP Component >_cs Home Node Next Priority Node Never Fallback <node 1> <node2>

rg_< SAP Component >_nfs Home Node Next Priority Node Never Fallback <node 1> <node2>

rg_< SAP Component >_er Home Node Next Priority Node Never Fallback <node 2> <node1>

Resource Group rg_<sid>_db rg_<sid>_cs rg_<sid>_er

Service IP Label n/a

Filesystems (default is All) n/a n/a n/a

Filesystems Consistency Check logredo logredo logredo

Filesystems Recovery Method sequential parallel parallel

Filesystems to Export (NFSv2/3) n/a n/a n/a

Filesystems to Export (NFSv4) n/a /export/sapmnt /export/sapmnt/<SID> /export/usr/sap/trans<SID> /export/share<SID>

n/a

Stable Storage Path (NFSv4) n/a /NFS4_DONT_REMOVE n/a

Filesystems to NFS Mount n/a /sapmnt;/export/sapmnt /sapmnt/<SID>;/export/sapmnt/<SID> /usr/sap/trans<SID>;/export/usr/sap/trans<SID> /share<SID>;/export/share<SID>

n/a

Network for NFS Mount n/a Access_Net n/a

Volume Groups vg<SID>ora vg<SID>log vg<SID>data

vg<SID>sap vg<SID>ers

Concurrent Volume Groups n/a n/a n/a

Raw Disk PVIDs n/a n/a n/a

Tape Resources n/a n/a n/a

Application Servers As_<sid>_db as_<sid>_cs as_<sid>_ers

Primary Workload Manager Class n/a n/a n/a

Secondary Workload Manager Class

n/a n/a n/a

Page 66: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 66 of 71

Resource Group rg_<sid>_db rg_<sid>_cs rg_<sid>_er

Miscellaneous Data n/a n/a n/a

Import Volume Groups false false false

Filesystems Mounted before IP Configured

false True false

Application Server Start-Script Stop-Script

as_<sid>_db /usr/sap/<SID>/PowerHA/start_db /usr/sap/<SID>/PowerHA/stop_db

as_<sid>_cs /usr/sap/<SID>/PowerHA/start_cs /usr/sap/<SID>/PowerHA/stop_cs

as_<sid>_er /usr/sap/<SID>/PowerHA/start_er /usr/sap/<SID>/PowerHA/stop_er

Application Monitor mon_sap_<sid>_ascs

Application Server as_sap_<sid>_ascs

Type Customer

Monitor Method /usr/es/sbin/cluster/sap/cl_sap_monitor ASCS10 <SID>

Monitor Interval 60

Hung Monitor Signal 9

Stabilization Interval 300

Restart Count 0

Restart Interval 0

Action on App Failure Fallover

Notify Method n/a

Cleanup Method n/a

Restart Method n/a

Application Monitor mon_sap_<sid>_aers mon_sap_<sid>_aers2

Application Server as_sap_<sid>_aers as_sap_<sid>_aers

Type Customer Customer

Monitor Method /usr/es/sbin/cluster/sap/cl_ers_monitor ERS11 <SID> STARTUP

/usr/es/sbin/cluster/sap/cl_ers_monitor ERS11 <SID> NODE_JOIN

Monitor Interval 60 60

Hung Monitor Signal 9 9

Stabilization Interval 300 300

Restart Count 2 0

Restart Interval 396 0

Action on App Failure Fallover Fallover

Notify Method n/a n/a

Cleanup Method n/a n/a

Restart Method /usr/es/sbin/cluster/sap/cl_sap_start ERS11 <SID>

Table 27 Example for Cluster Configuration (<SID>)

Page 67: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 67 of 71

1.4.6 Interdependencies on other Infrastructure Components

Infrastructure

Component

PowerHA takeover cluster

Backup / Restore The client components for filesystem backup/ restores of shared volume groups / filesystems and the components for database backup need to be installed “HA cluster aware”. This may include the “cluster aware” installation of the products and the deployment of them/ their configuration files on shared filesystems.

Disaster Recovery Disaster recovery solutions need to be considered as an extension of the High-Availability cluster solution. The HA solution will cover a failure of single components only, but not a multi-system failure.

Monitoring Monitoring of the PowerHA cluster state needs to be integrated into the overall infrastructure monitoring solution. Deployment of additional monitoring agents may be required.

User Management User IDs and group IDs must be identical on both cluster nodes. All operating systems users and their passwords must be kept in synch across both cluster nodes.

If the password of the database connect user is changed with the dscdb6up tool, the password in the file dscdb6.conf and the operating system password on the database host are changed. On the standby database server, the operating system password is not changed: As a result, the SAP systems cannot connect to the standby database server if DB2 is started on this node If the password of connect user sap<sid> or <sid>adm is changed, the operating system password on the standby database server needs to be changed too (manually, or implement a PW synchronization procedure)

Capacity Planning Recommendation is to use a symmetrical setup for the two nodes in the cluster setup: Both nodes should have the same configuration for CPU and memory. If the capacity is increased on the primary server the same capacity increase should be applied to the standby system.

For HADR setup, both servers must have the same amount of memory since buffer pool operations are also replayed by the standby database server.

Use of Test Systems Test systems need to be available for the planning/ execution of changes before they’re applied on the production clusters. (No test in production). This can be assured by the deployment of HA clusters for pre-production systems along the promote-to-production path.

Network If multiple networks are used and need to be implemented in the PowerHA cluster: All networks need to satisfy the subnet requirements for PowerHA.

Change Management Cluster-aware change management: Qualify changes to be eventually differently implemented for clustered and non-clustered systems. Use cluster tests on the pre-production system to successfully test a change there before implementing the change in production.

Incident Management Requirement for Inventory solution (in alignment with SLD) for all SAP systems to be able to identify the correct category

Table 28 Dependencies of HA to other infrastructure components

Page 68: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 68 of 71

1.5 Roles & Responsibilities (setup, operations)

Deployment Unit

Sub-component Setup Operations

PowerHA

cluster

AIX IT Infrastructure team IT Infrastructure team

PowerHA – Topology

IT Infrastructure team IT Infrastructure team

PowerHA – Resources

IT Infrastructure team(Resource Groups)

SAP Basis team (SAP Layout)

IT Infrastructure team

SAP Basis team

PowerHA – Resources – Application Monitoring

IT Infrastructure team/ SAP Basis team, Details to be clarified

IT Infrastructure team/ SAP Basis team, Details to be clarified

SAP Installation SAP Basis team SAP Basis team

Cluster tests IT Infrastructure team/ SAP Basis team

IT Infrastructure team/ SAP Basis team

Table 29 Roles & responsibilities for Infrastructure Component High Availability

In the areas “cluster tests” and the continuous cluster operation (especially after incidents triggering a failover” a strong workflow and clear responsibilities need to be established between the IT infrastructure team and the SAP basis team. The PowerHA cluster controls the availability of the SPOFs for the SAP system: Database, SAP Central Services and NFS filesystems. During a cluster test, the SAP basis team needs to be involved for checking and starting impacted SAP application servers. After an incident causing a failover, the full redundancy and system capacity is restored when the failing cluster node is repaired and re-joined the cluster, and the SAP basis team has restarted the SAP application server on it. Database and SAP Central Services will continue to run on the 2

nd

cluster node. If it is desired to bring back database and SAP Central services to node 1, this needs to be done during an outage/ maintenance window and will involve both SAP basis and IT infrastructure team.

Page 69: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 69 of 71

1.5.1 Considerations for Continuous Availability

With each new generation of hardware and software IBM improves technology and its RAS features. This enables <CLIENT> to move from High Availability (the ability to mask unplanned outages) to Continuous Availability (which also includes the masking of planned outages). Reaching high levels of availability is a journey which requires continual focus independent from the platform. The first step to achieve the availability goal is to implement the solution through the HA architecture and design to ensure there is no single point of failure (application, people, process, and infrastructure components) for all key business functions. The next step is to then take advantage of load balancing/replication, server clustering, data/database mirroring, virtualization, monitoring, and automation technologies to eliminate outages or reduce the impact of outages.

1.5.1.1 POWER Systems

The table below describes the different regular maintenance topics and the methods to minimize/ mitigate a service disruption to the end users for IBM POWER Systems

POWER System

Topic Frequency (Best-Practices)

Description

System Firmware

One time per year In nearly all cases IBM Power Systems firmware update can be done concurrently. In the unlikely event the update cannot be done online, the Live Partition Mobility feature could be used to allow the movement of the affected LPAR non-disruptively to another managed system.

HMC One time per year No impact to the application. Redundant Implementation (“Dual HMC”) allows to perform configuration changes/ tasks while the other one is maintained

Virtual I/O server maintenance

One time per year A redundant implementation of the virtual I/O servers (two per system) allows the VIOS code update without any impact to the application. No VIOS redundancy during the time interval for the change.

Operating System Maintenance

Operating System version: typically once every three – four years Operating System Technology Level: typically one time per year

The activation of the change will require a reboot of the LPAR: The AIX operating system however allows operations to prepare for the change online; so the outage will be a few minutes only from an operating system view. If the application is implemented within a HA cluster, it can be moved to the takeover servers in a planned outage window disruptively (stop/ start). The application can be kept online during the change itself. A symmetrical layout between the two cluster nodes is advised to allow the application running on both of them with equal rights. No fallback will then be required after the change. Virtualization techniques can be used to optimize the resource consumption and distribution.

Operating System Maintenance

Security fixes, fixes for encountered “defect” problems (applied before next “image rollout”)

Reboot required only in some cases: Concurrent AIX kernel updates in AIX 6.1 allow installation of some kernel patches without rebooting the system. This can reduce the number of outages required to maintain a secure, reliable system. For the few cases a reboot will be required, HA cluster allows to relocate the application during a planned step (see above).

PowerHA cluster manager (HACMP)

Version: typically once every three – four years

“Rolling Migration” for PowerHA cluster manager version upgrades; can be combined together with Operating System version upgrades (see above)

Page 70: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 70 of 71

POWER System

Topic Frequency (Best-Practices)

Description

maintenance FixPacks: One time per year

Cluster software PowerHA 6.1 allows to apply FixPacks non disruptively for the managed application (“unmanaged Resource Groups”)

Table 30 Regular maintenance topics

Page 71: High Availability for SAP V1.5

High Availability Solution Design Document for SAP 15-Nov-12 http://www.ibm.com/support/techdocs © Copyright IBM Corporation, 2012 Page 71 of 71

COPYRIGHT LICENSE If this information contains sample application programs in source language, which illustrate programming techniques on various operating platforms, you may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. The information contained in this document has not been submitted to any formal IBM test and is provided "AS IS" with no warranties or guarantees either expressed or implied. All examples cited or described in this document are presented as illustrations of the manner in which some IBM products can be used and the results that may be achieved. Actual environmental costs and performance characteristics will vary depending on individual client configurations and conditions. IBM is not responsible for printing errors in this document that result in pricing or information inaccuracies. A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml A full list of trademarks owned by SAP may be found at: http://www.sap.com/company/legal/copyright/trademark.epx SAP and other SAP products and services mentioned herein, as well as their respective logos, are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. UNIX is a registered trademark in the United States, other countries or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries or both. Microsoft, Windows, Windows NT and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries or both.


Recommended