+ All Categories
Home > Documents > 1 Copyright © 2013, Oracle and/or its affiliates. All...

1 Copyright © 2013, Oracle and/or its affiliates. All...

Date post: 23-Apr-2018
Category:
Upload: lamdien
View: 215 times
Download: 2 times
Share this document with a friend
64
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 1
Transcript

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 1

Cloning of Oracle Homes on Oracle RAC 11.2 Cluster Nodes after Linux Upgrade

Andreas Becker,

Principal Member Technical Staff

Oracle Server Technologies, SAP Dev.

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 5

Program Agenda

The Project

The Plan

The Big Picture

DelNode/AddNode vs. Oracle Home Cloning

Summary

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 7

The following is intended to outline our general product direction. It is intended

for information purposes only, and may not be incorporated into any contract.

It is not a commitment to deliver any material, code, or functionality, and should

not be relied upon in making purchasing decisions. The development, release,

and timing of any features or functionality described for Oracle’s products

remains at the sole discretion of Oracle.

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 9

The Project

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 10

Project Description

SAP System on Oracle RAC

Oracle RAC Cluster with 4 nodes

Oracle Release 11.2.0.3

Oracle Linux Release 6.1 on SAP Application Server

Oracle Linux Release 5.8 on Oracle RAC nodes

ASM + ACFS

Target: Oracle Linux Release 6.1 on all Oracle RAC nodes

Current Situation and Project Target

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 11

Project Description SAP System + Oracle RAC System Overview

SAP

Oracle

RAC

Database

RAC Nodes

RAC Instance RAC Instance RAC Instance RAC Instance S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

RAC Instances Services

SAP User

OL6.1 Node #1 Node #2 Node #3 Node #4

SAP Instances

Database

SAP Instance 2

OL6.1

SAP Instance 1

OL6.1

SAP Instance 3

OL6.1

SAP Instance 4

OL6.1

OL5.8

ASM / ACFS

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 12

Linux OS Upgrade Oracle Linux

In-place upgrading from a major version of Oracle

Linux 5 or earlier is not supported. Although

Anaconda provides an option to perform an upgrade,

fresh installation is strongly recommended.

Oracle® Linux

Release Notes for Release 6.4

http://docs.oracle.com/cd/E37670_01/E39522/html/ol

_sup_64rn.html

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 13

Linux OS Upgrade Red Hat Linux

Red Hat does not support in-place upgrades between

any major versions of Red Hat Enterprise Linux.

Red Hat Enterprise Linux 6 Installation Guide

https://access.redhat.com/site/documentation/en-

US/Red_Hat_Enterprise_Linux/6/html/Installation_Guid

e/ch-upgrade-x86.html

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 14

Project Description

Technical restrictions

– Linux upgrade requires fresh installation of all RAC nodes

SAP System HA requirements

– SAP application must remain available, no longer downtimes

– Downtimes due to SAP application server restarts are acceptable

– 2 of the 4 RAC nodes are sufficient run the SAP system

Restrictions and Requirements

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 15

Project Description

Project goal: Linux OS Upgrade

– No other requirements or changes

– No hardware replacement

– No host name change

– No network configuration change, no IP change

Restrictions and Requirements

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 18

The Plan

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 19

The Plan

2 RAC nodes are sufficient to run the SAP load (for a while)

2-phases

– Phase 1: „OS upgrade“ of RAC nodes 1 and 2

– Phase 2: „OS upgrade“ of RAC nodes 3 and 4

2-Phase-Approach

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 20

The Plan

Between Phase 1 and Phase 2 the system would run with different OS

versions (for a while)

MOS 220970.1 For Oracle Database 11g Release 2, Oracle Clusterware and ASM binaries are combined into a single

ORACLE_HOME called the grid infrastructure home. This home fully supports rolling upgrades for patches, bundles,

patchsets and releases. (If you are upgrading ASM from Oracle Database 10g to 11g Release 2, you will not be able

to upgrade ASM in a rolling fashion.)

The Oracle Clusterware and Oracle Real Application Clusters both support rolling upgrades of the OS software

when the version of the Oracle Database is certified on both releases of the OS (and the OS is the same, no Linux

and Windows or AIX and Solaris, or 32 and 64 bit etc.). This can apply a patch to the operating system, a patchset

(such as EL4u4 to EL4u6) or a release (EL4 to EL5).

2-Phase-Approach

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 22

The Plan

Oracle® Database SAP® Bundle Patch 11.2.0.3.7 - 201309 for Grid Infrastructure

If you restrict installation of GI-PSU 11.2.0.3.7 and this SBP to the grid homes and the non-SAP homes as described above, you can also keep

the SAP databases on the cluster and the SAP applications running on these available. However, due to limitations in Oracle's connection

handling such a RAC-rolling grid home SBP installation is not transparent to the SAP applications connected against the databases of the

cluster. To avoid transaction or other errors in the SAP applications when installing this SBP in a rolling manner, follow the additional

instructions in section 2.4, "Maintaining SAP Applications during a RAC-Rolling Patch Installation".

…To avoid transaction or other errors in the SAP applications when installing GI-PSU 11.2.0.3.7 and the Grid Infrastructure patches of this

SBP in a rolling manner, you must follow the installation instructions below on each of the cluster nodes in turn:

All connections from SAP applications to SAP database instances running on the cluster node must be terminated by the SAP application

before installing GI-PSU 11.2.0.3.7 and the Grid Infrastructure patches of this SBP on the cluster node.

Any SAP application component (for example, the Enqueue Server) which is dependent on the Oracle Clusterware on a cluster node must be

stopped by the SAP application before installing GI-PSU 11.2.0.3.7 and the Grid Infrastructure patches of this SBP on the cluster node.

RAC-Rolling Grid Home Patching

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 23

The Plan

…To avoid transaction or other errors in the SAP applications…

1. All connections from SAP applications to SAP database instances running on

the cluster node must be terminated by the SAP application. …

2. Any SAP application component (for example, the Enqueue Server) which is

dependent on the Oracle Clusterware on a cluster node must be stopped by

the SAP application…

RAC-Rolling Grid Home Patching

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 25

The Big Picture

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 26

The Big Picture Target

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

SAP User

OL6.1 OL6.1 OL6.1 OL6.1 OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

SAP Instance 2

OL6.1

SAP Instance 1

OL6.1

SAP Instance 3

OL6.1

SAP Instance 4

OL6.1

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 28

The Big Picture Starting Point

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

SAP User

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 29

The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

1. Block SAP instances 1 and 2 for logon and

disable batch jobs

2. Inform SAP user to reconnect

3. Inform SAP user about SAP instance restart

4. Relocate database services

5. Restart SAP instance(s) 1 and 2

6. Allow logon on SAP instances 1 and 2

and enable batch jobs again

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 30

The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

1. Block SAP instances 1 and 2 for logon and

disable batch jobs

2. Inform SAP user to reconnect

3. Inform SAP user about SAP instance restart

4. Relocate database services

5. Restart SAP instance(s) 1 and 2

6. Allow logon on SAP instances 1 and 2

and enable batch jobs again

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 31

The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

1. Block SAP instances 1 and 2 for logon and

disable batch jobs

2. Inform SAP user to reconnect

3. Inform SAP user about SAP instance restart

4. Relocate database services

5. Restart SAP instance(s) 1 and 2

6. Allow logon on SAP instances 1 and 2

and enable batch jobs again

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

S1 S2

$ srvctl relocate service… $ srvctl relocate service…

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 32

The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

1. Block SAP instances 1 and 2 for logon and

disable batch jobs

2. Inform SAP user to reconnect

3. Inform SAP user about SAP instance restart

4. Relocate database services

5. Restart SAP instance(s) 1 and 2

6. Allow logon on SAP instances 1 and 2

and enable batch jobs again

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

S1 S2 S3 S4

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 33

The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

1. Block SAP instances 1 and 2 for logon and

disable batch jobs

2. Inform SAP user to reconnect

3. Inform SAP user about SAP instance restart

4. Relocate database services

5. Restart SAP instance(s) 1 and 2

6. Allow logon on SAP instances 1 and 2

and enable batch jobs again

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 34

The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ERS

Services

1. Relocate SAP ERS instance (sapctl)

2. Relocate SAP ASCS instance (sapctl)

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ASCS ERS

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 35

The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Relocate SAP ERS instance (sapctl)

2. Relocate SAP ASCS instance (sapctl)

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ASCS ASCS ERS

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 36

The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Relocate SAP ERS instance (sapctl)

2. Relocate SAP ASCS instance (sapctl)

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ASCS ERS ERS

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 37

The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

SAP preparations are completed.

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ERS ASCS

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 38

The Big Picture Preparations

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Stop RAC instances on nodes 1 and 2

$ srvctl stop instance –i <instance> -d

<db>

2. Stop CRS on nodes 1 and 2 (root)

# crsctl stop crs

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ERS ASCS

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 39

The Big Picture Preparations completed

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

Preparations for phase 1 complete.

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ERS ASCS

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 40

Program Agenda

The Project

The Plan

The Big Picture

DelNode/AddNode vs. Oracle Home Cloning

Summary

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 42

The Oracle Home Cloning Approach

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 43

The Alternatives

1. Backup Grid home, database home(s), oratab, host configuration files,

user home directories, user configuration, /etc/hosts, /etc/passwd, …

2. Stop database instances and CRS

3. Install the new Linux version and configure the node exactly as before

hostname, IP, accounts, user homes, …).

4. Ensure that installation prerequisites are met

5. Restore backup of Grid home and database home(s)

6. Clone Grid home and database home(s)

7. Start CRS and database instances

Oracle home Cloning Approach

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 47

The Big Picture Preparations completed

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

Preparations for phase 1 complete.

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ERS ASCS

Node 1

Node 2

Node 3

Node 4

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 48

The Alternatives Oracle home Cloning Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Backup Grid home and database home on

each node.

2. Backup /etc/oraInst.loc, /etc/oratab

3. Backup OS configuration files like

/etc/hosts, /etc/groups, /etc/passwd

4. ASM: mapping of ASM disks

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ERS ASCS

Node 1

Node 2

Node 3

Node 4

gi_home_node1.tar gi_home_node2.tar

db_home_node1.tar db_home_node2.tar

$ tar cvf … $ tar cvf …

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 49

The Alternatives Oracle home Cloning Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Install new OS version

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ERS ASCS

Node 1

Node 2

Node 3

Node 4

OL6.1 OL6.1

Install Linux OS

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 50

The Alternatives Oracle home Cloning Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Restore & Clone Grid Home

2. Restore & Clone Database Home

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ERS ASCS

Node 1

Node 2

Node 3

Node 4

OL6.1 OL6.1

gi_home_node1.tar gi_home_node2.tar

db_home_node1.tar db_home_node2.tar

$ tar xvf …

$ clone.pl $ tar xvf …

$ clone.pl

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 51

The Alternatives

Cloning the Grid home $ su root -c "$OHGRID/crs/install/rootcrs.pl -unlock“

$ /usr/bin/perl $OHGRID/clone/bin/clone.pl ORACLE_BASE=$ORACLE_BASE ORACLE_HOME=$OHGRID -defaultHomeName

INVENTORY_LOCATION=/oracle/oraInventory -O'"CLUSTER_NODES={<node list>}"' -O'"LOCAL_NODE=<node>"' CRS=false -

O"SHOW_ROOTSH_CONFIRMATION=false“

su root -c "$OHGRID/rdbms/install/rootadd_rdbms.sh"

su root -c "$OHGRID/crs/install/rootcrs.pl -patch"

su root -c "chmod 755 /opt/oracle“

Cloning the Database home $ $ORACLE_HOME/clone/bin/clone.pl ORACLE_HOME=$ORACLE_HOME ORACLE_BASE=$ORACLE_BASE

-defaultHomeName

$ $ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=$ORACLE_HOME "CLUSTER_NODES={<comma

separated list of cluster nodes>}"

# $ORACLE_HOME/root.sh

For details, see My Oracle Support note MOS 1136544.1.

Oracle Home Cloning Approach

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 52

The Alternatives Oracle home Cloning Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4

ERS ASCS

OL5.8 OL5.8 Node #1 Node #2

OL6.1 OL6.1

1. Start CRS on new nodes

2. Start RAC instances on new nodes

Next: Rebalance

RAC Instance RAC Instance

Node 1

Node 2

Node 3

Node 4

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 53

The Alternatives Oracle home Cloning Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance S3 S4

OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4 OL5.8 OL5.8 Node #1

Node #2 OL6.1 OL6.1

1. Rebalance to complete phase 1

Phase 1 complete

Next steps:

1. Start phase 2

2. Rebalance to complete phase 2

RAC Instance RAC Instance

Node 1

Node 2

Node 3

Node 4

S1 S2

ASCS ERS

Zero OCR

updates

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 54

The Alternatives Oracle home Cloning Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance S3 S4

OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4 OL5.8 OL5.8 Node #1

Node #2 OL6.1 OL6.1

RAC Instance RAC Instance

Node 1

Node 2

Node 3

Node 4

S1 S2

ASCS ERS

Zero OCR

updates

Phase 1 completed.

Phase 2 completed.

Oracle RAC is running OL 6.1.

OL6.1 OL6.1

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 55

The DelNode/AddNode Approach

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 56

The Alternatives

1. Remove a RAC node from the cluster

2. Install the new Linux version

3. Add the node to the cluster again.

Repeat steps 1-3 for all nodes that need to be installed. When finished,

backup OCR.

DelNode: <GRIDHOME>/bin/crsctl delete node -n <nodename>

AddNode: <GRIDHOME>/oui/bin/addNode.sh

DelNode/AddNode Approach

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 57

The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

Services

1. Remove nodes

2. Install new OS version

3. Add nodes

OL6.1

OCR

Voting

Node #1 Node #2 Node #3 Node #4

ERS ASCS

Node 1

Node 2

Node 3

Node 4

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 58

The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8 OL5.8

Services

1. Remove nodes

2. Install new OS version

3. Add nodes

OL6.1

OCR

Voting

OL5.8 Node #1

Node #2 Node #3 Node #4

ERS ASCS

Node 2

Node 3

Node 4

$ crsctl delete node node 1

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 59

The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8

Services

1. Remove nodes

2. Install new OS version

3. Add nodes

OL6.1

OCR

Voting

OL5.8 Node #1

OL5.8 Node #2

Node #3 Node #4

ERS ASCS

Node 3

Node 4

$ crsctl delete node node 2

2 OCR

updates

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 60

The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8

Services

1. Remove nodes

2. Install new OS version

3. Add nodes

OL6.1

OCR

Voting

OL5.8 Node #1n

OL5.8 Node #2n

Node #3 Node #4

ERS ASCS

Node 3

Node 4

OL6.1

OL6.1

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

2 OCR

updates

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 61

The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8

Services

1. Remove nodes from RAC cluster

2. Install new OS version

3. Add nodes to RAC cluster again

OL6.1

OCR

Voting

Node #3 Node #4

ERS ASCS

OL5.8 Node #1n OL6.1

Node 1n

Node 3

Node 4

OL5.8 Node #2n OL6.1

$ addNode.sh

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

3 OCR

updates

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 62

The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8

Services

1. Remove nodes from RAC cluster

2. Install new OS version

3. Add nodes to RAC cluster again

OL6.1

OCR

Voting

Node #3 Node #4

ERS ASCS

OL5.8 OL5.8 Node #1n Node #2n

OL6.1 OL6.1

Node 1n

Node 2n

Node 3

Node 4

$ addNode.sh

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

4 OCR

updates

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 63

The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance

S1 S2 S3 S4

OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4

ERS ASCS

OL5.8 OL5.8 Node #1n Node #2n

OL6.1 OL6.1

1. Start CRS on new nodes

2. Start RAC instances on new nodes

RAC Instance RAC Instance

Node 1n

Node 2n

Node 3

Node 4

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 64

The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance S3 S4

OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4 OL5.8 OL5.8 Node #1n

Node #2n OL6.1 OL6.1

1. Rebalance again: move SAP users, SAP

Database Services, and SAP instances

back to RAC instances on nodes 1 and 2

RAC Instance RAC Instance

Node 1n

Node 2n

Node 3

Node 4

S1 S2

ASCS ERS

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 65

The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance S3 S4

OL5.8 OL5.8

Services

OL6.1

OCR

Voting

Node #3 Node #4 OL5.8 OL5.8 Node #1n

Node #2n OL6.1 OL6.1

RAC Instance RAC Instance

Node 1n

Node 2n

Node 3

Node 4

S1 S2

ASCS ERS

Phase 1 completed.

Phase 2: next to start

Oracle RAC is running OL 6.1 and OL 5.8

(mix).

4 OCR

updates

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 66

The Alternatives DelNode/AddNode Approach

SAP

Oracle

RAC

Database

RAC

Nodes

RAC Instance RAC Instance RAC Instance RAC Instance S1 S2 S3 S4

OL5.8 OL5.8 OL5.8 OL5.8

ASCS ERS

Services

OL6.1 OL6.1 OL6.1 OL6.1 OL6.1 Node #1n

Node #2n Node #3n Node #4n

Phase 1 completed.

Phase 2 completed.

Oracle RAC is running OL 6.1.

OCR

Voting

Node 1n

Node 2n

Node 3n

Node 4n

8 OCR

updates

SAP Instance 2 SAP Instance 1 SAP Instance 3 SAP Instance 4

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 67

Program Agenda

The Project

The Plan

The Big Picture

DelNode/AddNode vs. Oracle Home Cloning

Summary

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 68

Summary

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 71

Summary

DelNode/AddNode Oracle Home Cloning

• Nodes are removed from cluster and added to

cluster

• Every ‚node add‘/‘node remove‘

operation changes the OCR and affects

the remaining cluster

• N-node-RAC 2xN OCR configuration

changes

• Zero Addnode/DelNode operations

• OCR configuration before/after is not identical

• OCR backup required when finished

• OCR/Cluster configuration remains

unchanged

Comparison DelNode/AddNode vs. Oracle Home Cloning Approach

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 72

Summary

DelNode/AddNode Oracle Home Cloning

• General approach • More specific approach

Suitable for

• Cluster extensions

• Cluster configuration changes (host names,

hardware, IP address)

• Preparation of new nodes while system is

online

• Hardware exchange

Suitable for

• OS upgrade / OS fresh installations

• Rolling GI Home Patching

Comparison DelNode/AddNode vs. Cloning Approach

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 75

1696869 - Patching of Oracle Homes with Minimal Downtime

http://service.sap.com/sap/support/notes/1696869

Rolling Patching

Oracle® Database SAP® Bundle Patch

11.2.0.3.7 - 201309 for Grid Infrastructure

README

References

SAP Notes

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 76

Oracle Support Document 1136544.1 (Minimal downtime patching via cloning 11gR2 ORACLE_HOME directories) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1136544.1

Oracle Support Document 1262925.1 (How to remove/delete a node from Grid Infrastructure Clusterware when the node has failed) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1262925.1

Oracle Support Document 1332451.1 (How to Add Node/Instance or Remove Node/Instance in 10gR2, 11gR1 and 11gR2 Oracle Clusterware and RAC) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1332451.1

Oracle Support Document 220970.1 (RAC: Frequently Asked Questions) can be found at: https://support.oracle.com/epmos/faces/DocumentDisplay?id=220970.1

References

MOS Notes

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 77

Red Hat Enterprise Linux 6 Product Documentation

https://access.redhat.com/site/documentation/en-

US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/ch-upgrade-x86.html

Oracle Linux Release Notes for Release 6.4

Supported Upgrade Path

http://docs.oracle.com/cd/E37670_01/E39522/html/ol_sup_64rn.html

References

Other

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 78

Questions?

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 80

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. DOAG 2013 81

© 2013 SAP AG. All rights reserved.

© 2013 SAP AG. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

Microsoft, Windows, Excel, Outlook, PowerPoint, Silverlight, and Visual Studio are registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, z10, z/VM, z/OS, OS/390, zEnterprise, PowerVM, Power Architecture, Power Systems, POWER7, POWER6+, POWER6, POWER, PowerHA, pureScale, PowerPC, BladeCenter, System Storage, Storwize, XIV, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, AIX, Intelligent Miner, WebSphere, Tivoli, Informix, and Smarter Planet are trademarks or registered trademarks of IBM Corporation.

Linux is the registered trademark of Linus Torvalds in the United States and other countries.

Adobe, the Adobe logo, Acrobat, PostScript, and Reader are trademarks or registered trademarks of Adobe Systems Incorporated in the United States and other countries.

Oracle and Java are registered trademarks of Oracle and its affiliates.

UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems Inc.

HTML, XML, XHTML, and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

Apple, App Store, iBooks, iPad, iPhone, iPhoto, iPod, iTunes, Multi-Touch, Objective-C, Retina, Safari, Siri, and Xcode are trademarks or registered trademarks of Apple Inc.

IOS is a registered trademark of Cisco Systems Inc.

RIM, BlackBerry, BBM, BlackBerry Curve, BlackBerry Bold, BlackBerry Pearl, BlackBerry Torch, BlackBerry Storm, BlackBerry Storm2, BlackBerry PlayBook, and BlackBerry App World are trademarks or registered trademarks of Research in Motion Limited.

Google App Engine, Google Apps, Google Checkout, Google Data API, Google Maps, Google Mobile Ads, Google Mobile Updater, Google Mobile, Google Store, Google Sync, Google Updater, Google Voice, Google Mail, Gmail, YouTube, Dalvik and Android are trademarks or registered trademarks of Google Inc.

INTERMEC is a registered trademark of Intermec Technologies Corporation.

Wi-Fi is a registered trademark of Wi-Fi Alliance.

Bluetooth is a registered trademark of Bluetooth SIG Inc.

Motorola is a registered trademark of Motorola Trademark Holdings LLC.

Computop is a registered trademark of Computop Wirtschaftsinformatik GmbH.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, SAP HANA, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.

Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company.

Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase Inc. Sybase is an SAP company.

Crossgate, m@gic EDDY, B2B 360°, and B2B 360° Services are registered trademarks of Crossgate AG in Germany and other countries. Crossgate is an SAP company.

All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.


Recommended