Add Node to an existing 10.2.0.4 ASM based RAC ........................................................... 1 References ........................................................................................................................ 1 Start Blackout on Grid Control ........................................................................................ 1 Disable Cron Jobs on the Database Server ...................................................................... 2 Shutdown and take backups of the existing Cluster ........................................................ 2

Stop Database services on all nodes ............................................................................ 2 Stop agent On all nodes ............................................................................................... 2 Stop CRS On all nodes ................................................................................................ 2 Backup all nodes .......................................................................................................... 2 Take a BCV/Snapshot backup of the Database ........................................................... 3

Start CRS and database on All nodes of the Existing Cluster ......................................... 3 Add the new node to the Cluster (HW and OS) ............................................................... 4

Clean up remnants of Previous Oracle install on Node D ........................................... 4 Install Oracle Clusterware (CRS) on the New Node ....................................................... 6

Run addNode.sh as oracle ............................................................................................ 6 Configure VIP service from Node A ......................................................................... 11 Configure ONS service from Node A ........................................................................ 15 Verify Cluster install and health ................................................................................ 16

Adding Nodes at the Oracle RAC Database Layer ........................................................ 18 Extend RDBMS Software to the new Node .............................................................. 18 Extend ASM Software to the new Node .................................................................... 22 Update Listener configuration ................................................................................... 27 Run DBCA from Node A to extend ASM instance ................................................... 33 Run DBCA from Node A to add DB instance ........................................................... 36

Add Node to an existing 10.2.0.4 ASM based RAC

Existing Nodes: node-a, node-b, node-cTo be Added: node-d (Node D)

References

Metalink Doc 269320.1: Removing a Node from a 10g RAC ClusterMetalink Doc 270512.1: Adding a Node to a 10g RAC Clusterhttp://download-west.oracle.com/docs/cd/B19306_01/rac.102/b14197/adddelunix.htm#BEICADHDhttp://www.oracle.com/technology/pub/articles/vallath-nodes.htmlhttp://blogs.oracle.com/AlejandroVargas/gems/RAC10gR2AddNode.pdf

Start Blackout on Grid Control

Disable Cron Jobs on the Database Server

Shutdown and take backups of the existing Cluster

Stop Database services on all nodessrvctl stop database -d ORCLsrvctl stop asm -n node-asrvctl stop asm -n node-bsrvctl stop asm -n node-c

srvctl stop nodeapps -n node-asrvctl stop nodeapps -n node-bsrvctl stop nodeapps -n node-c

Stop agent On all nodes

cd /u01/app/oracle/product/10.2.0/agent10g/bin./emctl stop agent

============ (Commands in Red MUST be executed As root)==============

Stop CRS On all nodes

sudo su -/etc/init.crs stop. ~oracle/.profile$ORA_CRS_HOME/bin/oprocd stop

/usr/sbin/slibclean

Ensure no libraries are loaded in memory

genld -l | grep /u01/app/crs/product/10.2.0/crsgenkld | grep /u01/app/crs/product/10.2.0/crs

Backup all nodes

Backup files From Node A:

mkdir -p /backups/preAddNodeBackupssudo su -cd /backups/preAddNodeBackupsdd if=/dev/oracle_ocr of=ocr_backup.raw bs=8192 count=131072gzip ocr_backup.raw

dd if=/dev/oracle_vote of=vote_backup.raw bs=8192 count=131072gzip vote_backup.raw

cd /u01/app/crs/product/10.2.0/crs/cdatatar cpvf – orcl_prod | gzip -c > /backups/preAddNodeBackups/orcl_prod.tar.gz

cd /backups/preAddNodeBackupstar cpvf crsinit.tar /etc/init.crs /etc/init.crsd /etc/init.cssd /etc/init.evmd /etc/rc.d/rc2.d/K96init.crs /etc/rc.d/rc2.d/S96init.crsgzip crsinit.tar

cp -p /etc/inittab .

tar cpvf etc_oracle.tar /etc/oracle /tmp/.oracle /etc/oratab /etc/oraInst.locgzip etc_oracle.tar

tar cpvf - /u01/app/crs/product/10.2.0/crs | gzip -c > crs_10.2.0.4.tar.gz

tar cpvf - /u01/app/oracle/product/10.2.0/db01 | gzip -c > rdbms_10.2.0.4.tar.gz

tar cpvf - /u01/app/asm/product/10.2.0/asm | gzip -c > asm_10.2.0.4.tar.gz

tar cpvf - /u01/app/oraInventory | gzip -c > oraInventory.tar.gz

From All other Nodes:

mkdir -p /backups/preAddNodeBackupssudo su -cd /backups/preAddNodeBackups

tar cpvf crsinit.tar /etc/init.crs /etc/init.crsd /etc/init.cssd /etc/init.evmd /etc/rc.d/rc2.d/K96init.crs /etc/rc.d/rc2.d/S96init.crsgzip crsinit.tar

cp -p /etc/inittab .

tar cpvf etc_oracle.tar /etc/oracle /tmp/.oracle /etc/oratab /etc/oraInst.locgzip etc_oracle.tar

tar cpvf - /u01/app/crs/product/10.2.0/crs | gzip -c > crs_10.2.0.4.tar.gz

tar cpvf - /u01/app/oracle/product/10.2.0/db01 | gzip -c > rdbms_10.2.0.4.tar.gz

tar cpvf - /u01/app/asm/product/10.2.0/asm | gzip -c > asm_10.2.0.4.tar.gz

tar cpvf - /u01/app/oraInventory | gzip -c > oraInventory.tar.gz

Take a BCV/Snapshot backup of the Database

Start CRS and database on All nodes of the Existing Cluster

sudo su -/etc/init.crs start

As oracle

srvctl start nodeapps -n node-asrvctl start nodeapps -n node-bsrvctl start nodeapps -n node-c

srvctl start asm -n node-asrvctl start asm -n node-bsrvctl start asm -n node-c

srvctl start database -d ORCL

Add the new node to the Cluster (HW and OS)

Clean up remnants of Previous Oracle install on Node D

Node D:

rm -rf /etc/oraclerm -f /etc/oratab /etc/oraInst.locrm -rf /u01/app/crs/product/10.2.0/crs/*rm -rf /u01/app/oracle/product/10.2.0/db01/*rm -rf /u01/app/asm/product/10.2.0/asm/*rm -rf /u01/app/oraInventory/*cd /etc; rm -rf ora_save*rm -rf /tmp/OraInstall*rm -rf /tmp/.oraclerm -rf /u01/app/oracle/product/10.2.0/db01/.patch_storage*

rm /etc/init.cssd rm /etc/init.crs rm /etc/init.crsd rm /etc/init.evmd rm /etc/rc.d/rc2.d/K96init.crsrm /etc/rc.d/rc2.d/S96init.crsrm /etc/inittab.crs

-- Check on fixes and patches-- Verify the hardware requirements

-- Verify same interface names on the new node-- Verify new node on the same subnet as existing nodes

-- Verify the software requirements by comparing the outputs from all nodes

/usr/sbin/instfix -i -k "IY68989 IY68874 IY70031 IY61034 IY62191 IY60759 IY76807 IY76140"/usr/sbin/instfix -i -k "IY65305"lslpp -l bos.adt.base bos.adt.lib bos.adt.libm bos.perf.perfstat bos.perf.libperfstat bos.perf.proctools rsct.basic.rte rsct.compat.clients.rte xlC.aix50.rte xlC.rte bos.adt.prof bos.alt_disk_install.rte bos.cifs_fs.rte

-- Check /etc/hosts and make sure nslookup matches for all nodes (existing and new)-- Ping all private interconnects (existing and new)

-- Check oracle user capabilitieslsuser -a capabilities oracle

The command above should returnoracle capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE

-- Check shared disks are accessible by the new node

ls -l /dev/oracle_ocr /dev/oracle_vote

chown oracle:oinstall /dev/oracle_ocrchown oracle:dba /dev/oracle_votechmod 660 /dev/oracle_ocrchmod 660 /dev/oracle_vote

cd /tmpdd if=/dev/oracle_ocr of=temp.out bs=8192 count=10dd if=/dev/oracle_vote of=temp.out bs=8192 count=10rm temp.out

-- Check on the user equivalence to and from the new nodes using ssh.

-- Generate ssh key pair for oracle on Node D

ssh-keygen -t rsa

Password should be left empty

-- Copy the public key to the existing nodes server

Login to Node Acdcd .sshscp oracle@node-d:/home/oracle/.ssh/id_rsa.pub node-d.pubcat node-d.pub >> authorized_keysscp authorized_keys oracle@node-b:/home/oracle/.ssh/scp authorized_keys oracle@node-c:/home/oracle/.ssh/scp authorized_keys oracle@node-d:/home/oracle/.ssh/

-- Check umask returns 022 over ssh

ssh node-a umaskssh node-b umaskssh node-c umaskssh node-d umask

-- Check all nodes have the same system date

ssh node-ai datessh node-bi datessh node-ci datessh node-di date

-- Check on oracle profile in the new node:Update .profile on the new node to point to the correct ORACLE_HOME etc.

-- Make sure these directories exits and wipe out any remnants of a previous install

mkdir -p /u01/app/oracle/product/10.2.0/db01mkdir -p /u01/app/crs/product/10.2.0/crsmkdir -p /u01/app/asm/product/10.2.0/asm

chown -R oracle:oinstall /u01

ls -ld $ORACLE_HOME $ORACLE_BASE $ORA_CRS_HOME

-- Go through the pre-install checklist

-- Make sure SED is turned off

-- Run CLUVFY from node A:

$ORA_CRS_HOME/bin/cluvfy stage -post hwos -n node-d -verbose

Install Oracle Clusterware (CRS) on the New Node

-- Copy over some pre-install scripts to the new node:

on Node D-------------mkdir -p /u01/app/install_cds/CRS/Disk1/

On Node A--------------cd /backups/install/clusterware/Disk1tar cpvf rootpre.tar rootpre upgradescp rootpre.tar oracle@node-d:/u01/app/install_cds/CRS/Disk1/

On Node D--------------cd /u01/app/install_cds/CRS/Disk1/tar xpvf rootpre.tar

sudo su -umask 022cd /u01/app/install_cds/CRS/Disk1/rootpre./rootpre.sh

Run addNode.sh as oracleOn Node A--------------- Make sure CRS is running- Make sure ORA_CRS_HOME is set- Run as oracle from one of the existing nodes

export DISPLAY=10.2.9.36:0cd $ORA_CRS_HOME/oui/bin./addNode.sh

New node information:Public Node Name: node-dPrivate Node Name: node-diVirtual host name: node-dv

-- Run orainstRoot.sh on D as root:On Node D as root----------------------sudo su -umask 022/u01/app/oraInventory/orainstRoot.sh

-- Run rootaddnode.sh on A as root:On Node A as root----------------------sudo su -vi /u01/app/crs/product/10.2.0/crs/install/rootaddnode.shand change this line (last line of the script) and save:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$SU $CRS_ORACLE_OWNER -c "$CH/bin/cluutil -sourcefile $OCRCONFIG -destfile $CH/srvm/admin/ocr.loc -nodelist $NODES_LIST"

to

$SU - $CRS_ORACLE_OWNER -c "$CH/bin/cluutil -sourcefile $OCRCONFIG -destfile $CH/srvm/admin/ocr.loc -nodelist $NODES_LIST"~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

sudo su -umask 022

export DISPLAY=10.2.9.36:0/u01/app/crs/product/10.2.0/crs/install/rootaddnode.sh

-- Run root.sh on D as root:On Node D as root----------------------

vi /u01/app/crs/product/10.2.0/crs/install/rootconfig

Either comment out this paragraph around line 580

if $CRS_VNDR_CLUSTER;then

$ECHO "Checking to see if any 9i GSD is up" GSDNODE=`$LSDB -g` GSDCHK_STATUS=$?

if [ $GSDCHK_STATUS != 0 ];then $ECHO "9i GSD is running on node '$GSDNODE'. Stop the GSD and rerun root.sh" exit 1 fifi

Or, change:

GSDNODE=`$LSDB -g`

to

GSDNODE=`(LIBPATH=$ORA_CRS_HOME/lib ; export LIBPATH ; $LSDB -g)`

Then as root on Node D:

sudo su -umask 022export DISPLAY=10.2.9.36:0/u01/app/crs/product/10.2.0/crs/root.sh

Configure VIP service from Node A

This step was not needed in 10.2.0.4 as the add Node step above configured the VIPs.This step however can still be executed to verify VIP configuration.

On Node A as root

export DISPLAY=10.2.9.36:0. ~oracle/.profile$ORA_CRS_HOME/bin/vipca

Configure ONS service from Node A

as oracle:cat $ORA_CRS_HOME/opmn/conf/ons.config

-----------------------------localport=6113remoteport=6200loglevel=3useocr=on-----------------------------Note the remote port number from the output above and use that port number in the racgons command below.

as root on Node Asudo su -cd /u01/app/crs/product/10.2.0/crs/bin./racgons add_config node-d:6200

Verify Cluster install and health

$ORA_CRS_HOME/bin/cluvfy stage -post crsinst -n all -verbose

Output below

Performing post-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "node-a" Destination Node Reachable? ------------------------------------ ------------------------ node-c yes node-d yes node-b yes node-a yesResult: Node reachability check passed from node "node-a".

Checking user equivalence...

Check: User equivalence for user "oracle" Node Name Comment ------------------------------------ ------------------------ node-d passed node-c passed node-b passed node-a passedResult: User equivalence check passed for user "oracle".

Checking Cluster manager integrity...

Checking CSS daemon... Node Name Status ------------------------------------ ------------------------ node-d running node-c running node-b running node-a runningResult: Daemon status check passed for "CSS daemon".

Cluster manager integrity check passed.

Checking cluster integrity...

Node Name ------------------------------------ node-a node-b node-c node-d

Cluster integrity check passed

Checking OCR integrity...

Checking the absence of a non-clustered configuration...All nodes free of non-clustered, local-only configurations.

Uniqueness check for OCR device passed.

Checking the version of OCR...OCR of correct Version "2" exists.

Checking data integrity of OCR...Data integrity check for OCR passed.

OCR integrity check passed.

Checking CRS integrity...

Checking daemon liveness...

Check: Liveness for "CRS daemon" Node Name Running ------------------------------------ ------------------------ node-d yes node-c yes node-b yes node-a yesResult: Liveness check passed for "CRS daemon".

Checking daemon liveness...

Check: Liveness for "CSS daemon" Node Name Running ------------------------------------ ------------------------ node-d yes node-c yes node-b yes node-a yesResult: Liveness check passed for "CSS daemon".

Checking daemon liveness...

Check: Liveness for "EVM daemon" Node Name Running ------------------------------------ ------------------------ node-d yes node-c yes node-b yes node-a yesResult: Liveness check passed for "EVM daemon".

Liveness of all the daemons Node Name CRS daemon CSS daemon EVM daemon ------------ ------------------------ ------------------------ ---------- node-d yes yes yes node-c yes yes yes node-b yes yes yes node-a yes yes yes

Checking CRS health...

Check: Health of CRS Node Name CRS OK? ------------------------------------ ------------------------ node-d yes node-c yes node-b yes node-a yesResult: CRS health check passed.

CRS integrity check passed.

Checking node application existence...

Checking existence of VIP node application Node Name Required Status Comment ------------ ------------------------ ------------------------ ---------- node-d yes exists passed node-c yes exists passed node-b yes exists passed node-a yes exists passedResult: Check passed.

Checking existence of ONS node application Node Name Required Status Comment ------------ ------------------------ ------------------------ ---------- node-d no exists passed node-c no exists passed node-b no exists passed node-a no exists passedResult: Check passed.

Checking existence of GSD node application Node Name Required Status Comment ------------ ------------------------ ------------------------ ---------- node-d no exists passed node-c no exists passed node-b no exists passed node-a no exists passedResult: Check passed.

Post-check for cluster services setup was successful.

Adding Nodes at the Oracle RAC Database Layer

Run addNode.sh as oracle from Node A first for RAC and then for ASM HOME

Make sure CRS is up.

Extend RDBMS Software to the new NodeFrom Node A as oracle----------------------------export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db01cd $ORACLE_HOME/oui/binexport DISPLAY=10.2.9.36:0./addNode.sh

-- Run root.shOn Node D as root----------------------sudo su -umask 022export DISPLAY=10.2.9.36:0/u01/app/oracle/product/10.2.0/db01/root.sh

Extend ASM Software to the new NodeFrom Node A as oracle----------------------------export ORACLE_HOME=/u01/app/asm/product/10.2.0/asmcd $ORACLE_HOME/oui/bin./addNode.sh

-- Run root.shOn Node D as root----------------------sudo su -umask 022export DISPLAY=10.2.9.36:0/u01/app/asm/product/10.2.0/asm/root.sh

Update Listener configurationon node D as oracle------------------------export DISPLAY=10.2.9.36:0netca

After netca is complete, make sure there are not extra entries in this file on all nodes (specially Node D)

/u01/app/oracle/product/10.2.0/db01/network/admin/listener.ora

-- Add entry for the 4th node in listener aliases in the tnsnames.ora on all nodes

vi /u01/app/oracle/product/10.2.0/db01/network/admin/tnsnames.ora

The entries should look something like this

LISTENERS_ORCL = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = node-av)(PORT = 8001)) (ADDRESS = (PROTOCOL = TCP)(HOST = node-bv)(PORT = 8001)) (ADDRESS = (PROTOCOL = TCP)(HOST = node-cv)(PORT = 8001)) (ADDRESS = (PROTOCOL = TCP)(HOST = node-dv)(PORT = 8001)) )

LISTENER_ORCL4 = (ADDRESS = (PROTOCOL = TCP)(HOST = node-dv)(PORT = 8001))

Run DBCA from Node A to extend ASM instance

export DISPLAY=10.2.9.36:0export ORACLE_HOME=/u01/app/asm/product/10.2.0/asm$ORACLE_HOME/bin/dbca

After ASM is extended delete ASM1 related files from Node D

On node D as oraclecd /u01/app/asm/product/10.2.0/asm/dbsrm *ASM1*

Run DBCA from Node A to add DB instance

You will need following files for the new instance-- undo tablespace-- Redo Logs-- Standby redo logs (If using Datguard)

Since ASM and Oracle Managed Files are being used, DBCA will create the above per the template used when creating the database.

export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db01$ORACLE_HOME/bin/dbca

The next screen is going to take a long time to pop-up. Be patient…..

alter system set cluster_database_instances=4 scope=spfile sid='*';

Update tnsnames.ora on all the nodes to reflect the new instance

Restart the database

srvctl stop database -d ORCLsrvctl start database -d ORCL