Adding Node To Oracle RAC Database Summary: ============================================================== This presentation will provide detail steps for Oracle DBA and Linux Engineer to add new node to existing 11gR1 (11.1.0.6) database RAC. The most critical steps that need to be followed are: Verify the exists cluster configuration Installation Prerequisite Configuring SSH on New Cluster Member Nodes Pre-install checking Backup the files (OCR, Voting Disk, root.sh, oraInventory) Adding an Oracle Clusterware home to new nodes using OUI in interactive node Configure ONS for the new node. Adding an Oracle ASM home to new nodes using OUI in interactive node (Required If you have separate home directory ) Adding an Oracle RDBMS home to new nodes using OUI in interactive mode Reconfigure listener on new node using NETCA Adding ASM instance to new nodes using DBCA Adding DB Instance to new nodes using DBCA 1. a) Verify the exists cluster configuration [oracle@krac1 ~]$ crs_stat -t -v Name Type R/RA F/FT Target State Host ---------------------------------------------------------------------- ora....L1.inst application 0/5 0/0 ONLINE ONLINE krac1 ora....L2.inst application 0/5 0/0 ONLINE ONLINE krac2 ora.ORCL.db application 0/0 0/1 ONLINE ONLINE krac1 ora....SM1.asm application 0/5 0/0 ONLINE ONLINE krac1 ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE krac1 ora.krac1.gsd application 0/5 0/0 ONLINE ONLINE krac1 ora.krac1.ons application 0/3 0/0 ONLINE ONLINE krac1 ora.krac1.vip application 0/0 0/0 ONLINE ONLINE krac1
Transcript
Adding Node To Oracle RAC Database
Summary:==============================================================This presentation will provide detail steps for Oracle DBA and Linux Engineer to add new node to existing 11gR1 (11.1.0.6) database RAC.
The most critical steps that need to be followed are:
Verify the exists cluster configuration Installation Prerequisite Configuring SSH on New Cluster Member Nodes Pre-install checking Backup the files (OCR, Voting Disk, root.sh, oraInventory) Adding an Oracle Clusterware home to new nodes using OUI in interactive node Configure ONS for the new node. Adding an Oracle ASM home to new nodes using OUI in interactive node (Required
If you have separate home directory ) Adding an Oracle RDBMS home to new nodes using OUI in interactive mode Reconfigure listener on new node using NETCA Adding ASM instance to new nodes using DBCA Adding DB Instance to new nodes using DBCA
Listener Configuration Details[oracle@krac1 ~]$ srvctl config listener -n krac1krac1 LISTENER_KRAC1[oracle@krac1 ~]$ srvctl status nodeapps -n krac1VIP is running on node: krac1GSD is running on node: krac1Listener is running on node: krac1ONS daemon is running on node: krac1[oracle@krac1 ~]$ srvctl status nodeapps -n krac2VIP is running on node: krac2GSD is running on node: krac2Listener is running on node: krac2ONS daemon is running on node: krac2
Nodes and Environment Details[oracle@krac1 ~]$ olsnodes -n -p -ikrac1 1 krac1-priv krac1-vipkrac2 2 krac2-priv krac2-vip=====================================================================Exist Nodes : krac1.dbprod.com,krac2.dbprod.comNew Node : krac3.dbprod.comCRS_HOME : /u01/crs11g -- crs(OS) user is owner ASM_HOME : /u02/app/oracle/product/11.1.0/asm_1 -- oracle(OS) user ownerRAC_HOME : /u02/app/oracle/product/11.1.0/db_1 -- oracle(OS) user ownerNote : In this setup ASM configured in separate home directory for High availability.
eth0 152.168.1.0 global publiceth1 122.168.1.0 global cluster_interconnect
2. Installation Prerequisite
Refer Oracle 11g Release 1 RAC Installation Steps On Linux blog for more details.
3. a. Configuring SSH on New Cluster Member Nodes [crs@krac3 ~]$ su - crs [crs@krac3 ~]$ mkdir ~/.ssh[crs@krac3 ~]$ chmod 700 ~/.ssh[crs@krac3 ~]$ cd ~/.ssh[crs@krac3 .ssh]$ /usr/bin/ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/home/crs/.ssh/id_rsa): <ENTER>Enter passphrase (empty for no passphrase): <ENTER>Enter same passphrase again: <ENTER>Your identification has been saved in /home/crs/.ssh/id_rsa.Your public key has been saved in /home/crs/.ssh/id_rsa.pub.The key fingerprint is:2e:85:e0:06:c8:14:66:65:28:83:47:db:78:71:17:01 [email protected][crs@krac3 .ssh]$ /usr/bin/ssh-keygen -t dsaGenerating public/private dsa key pair.Enter file in which to save the key (/home/crs/.ssh/id_dsa): <ENTER>Enter passphrase (empty for no passphrase): <ENTER>Enter same passphrase again: <ENTER>Your identification has been saved in /home/crs/.ssh/id_dsa.Your public key has been saved in /home/crs/.ssh/id_dsa.pub.The key fingerprint is:60:52:61:65:b7:df:5a:5c:b2:83:79:b4:d2:9b:9d:21 [email protected][crs@krac3 .ssh]$ ls -ltrtotal 16-rw-r--r-- 1 crs oinstall 402 Oct 22 21:50 id_rsa.pub-rw------- 1 crs oinstall 1675 Oct 22 21:50 id_rsa-rw-r--r-- 1 crs oinstall 610 Oct 22 21:50 id_dsa.pub-rw------- 1 crs oinstall 668 Oct 22 21:50 id_dsa
b. SCP the authorized_keys file from exist node (on krac1)su - crs [crs@krac1 .ssh]$ scp ~/.ssh/authorized_keys krac3:$HOME /.ssh/.
c. Login into new node (krac3.dbprod.com) [crs@krac3 .ssh]$ ls -ltrtotal 20-rw-r--r-- 1 crs oinstall 402 Oct 22 21:50 id_rsa.pub-rw------- 1 crs oinstall 1675 Oct 22 21:50 id_rsa-rw-r--r-- 1 crs oinstall 610 Oct 22 21:50 id_dsa.pub-rw------- 1 crs oinstall 668 Oct 22 21:50 id_dsa-rw-r--r-- 1 crs oinstall 2024 Oct 22 21:50 authorized_keys[crs@krac3 .ssh]$ cat id_rsa.pub >> authorized_keys [crs@krac3 .ssh]$ cat id_dsa.pub >> authorized_keys
d. Copy the latest authorized_key file to remaining all nodes in cluster.
e. Enabling SSH User Equivalency on Cluster Member Nodes (Need to done both crs and oracle users)ssh krac1 datessh krac2 datessh krac3 datessh krac1.dbprod.com datessh krac2.dbprod.com datessh krac3.dbprod.com datessh krac1-priv datessh krac2-priv datessh krac3-priv datessh krac1-priv.dbprod.com datessh krac2-priv.dbprod.com datessh krac3-priv.dbprod.com date
f. Repeat the same steps as a oracle user in all new nodes. 3. Pre-install checking
a) Verify cluster healthy on existing nodes (ocrcheck, cluvfy). [oracle@krac1 ~]$ ocrcheckStatus of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 1043916 Used space (kbytes) : 5612 Available space (kbytes) : 1038304 ID : 1347985972 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded
Device/File not configured
Cluster registry integrity check succeeded
b) This theory assumes that the Prerequisites Installation takes have already been performed on the nodes.$cluvfy stage -post hwos -n krac3
c) Check OS version, Kernel parameters, /etc/hosts file and ensure they are identical on all nodes.
[oracle@krac1 ~]$ uname -nrmokrac1.dbprod.com 2.6.18-194.el5 i686 GNU/Linux [oracle@krac1 ~]$ cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.5 (Tikanga)
d) Verify below lines in /etc/sysclt.conf file on new node(krac3) .
e) Confirm a new node(commented in red color) entry in /etc/hosts files on all the nodes. ##################################################### LocalHost##################################################127.0.0.1 localhost.localdomain localhost##################################################### Public IP Address##################################################152.168.1.50 krac1.dbprod.com krac1 152.168.1.51 krac2.dbprod.com krac2152.168.1.52 krac3.dbprod.com krac3##################################################### Private IP Address##################################################192.168.1.50 krac1-priv.dbprod.com krac1-priv192.168.1.51 krac2-priv.dbprod.com krac2-priv192.168.1.52 krac3-priv.dbprod.com krac3-priv##################################################### VIP Address##################################################152.168.1.60 krac1-vip.dbprod.com krac1-vip152.168.1.61 krac2-vip.dbprod.com krac2-vip152.168.1.62 krac3-vip.dbprod.com krac3-vip
Checking node reachability...Node reachability check passed from node "krac1".
Checking user equivalence...User equivalence check passed for user "crs".
Checking administrative privileges...User existence check passed for "crs".Group existence check passed for "oinstall".Membership check for user "crs" in group "oinstall" [as Primary] passed.
Administrative privileges check passed.
Checking node connectivity...
Node connectivity check passed for subnet "152.168.1.0" with node(s) krac3,krac2,krac1.Node connectivity check passed for subnet "192.168.1.0" with node(s) krac3,krac2,krac1.
Interfaces found on subnet "152.168.1.0" that are likely candidates for VIP:krac3 eth0:152.168.1.52krac2 eth0:152.168.1.51 eth0:152.168.1.61krac1 eth0:152.168.1.50 eth0:152.168.1.60
Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP:krac3 eth1:192.168.1.52krac2 eth1:192.168.1.51krac1 eth1:192.168.1.50
WARNING: Could not find a suitable set of interfaces for the private interconnect.
Node connectivity check passed.
Checking system requirements for 'crs'...Total memory check passed.Free disk space check passed.Swap space check passed.System architecture check passed.Kernel version check passed.Package existence check passed for "make-3.81".Package existence check passed for "binutils-2.17.50.0.6".Package existence check passed for "gcc-4.1.1".Package existence check passed for "libaio-0.3.106".Package existence check passed for "libaio-devel-0.3.106".Package existence check passed for "libstdc++-4.1.1".Package existence check passed for "elfutils-libelf-devel-0.125".Package existence check passed for "sysstat-7.0.0".Package existence check passed for "compat-libstdc++-33-3.2.3".Package existence check passed for "libgcc-4.1.1".Package existence check passed for "libstdc++-devel-4.1.1".Package existence check passed for "unixODBC-2.2.11".Package existence check passed for "unixODBC-devel-2.2.11".Package existence check passed for "glibc-2.5-12".Group existence check passed for "dba".Group existence check passed for "oinstall".User existence check passed for "nobody".
System requirement passed for 'crs'
Pre-check for cluster services setup was successful.
[root@krac3 ~]# /u02/app/oracle/oraInventory/orainstRoot.shChanging permissions of /u02/app/oracle/oraInventory to 770.Changing groupname of /u02/app/oracle/oraInventory to oinstall.The execution of the script is complete[root@krac3 ~]# ssh krac1 /u01/crs11g/install/rootaddnode.shroot@krac1's password: clscfg: EXISTING configuration version 4 detected.clscfg: version 4 is 11 Release 1.Attempting to add 1 new nodes to the configurationUsing ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node <nodenumber>: <nodename> <private interconnect name> <hostname>node 3: krac3 krac3-priv krac3Creating OCR keys for user 'root', privgrp 'root'..Operation successful./u01/crs11g/bin/srvctl add nodeapps -n krac3 -A krac3-vip/255.255.255.0/eth0[root@krac3 ~]# /u01/crs11g/root.shChecking to see if Oracle CRS stack is already configured/etc/oracle does not exist. Creating it now.OCR LOCATIONS = /dev/raw/raw1OCR backup directory '/u01/crs11g/cdata/krac_cluster' does not exist. Creating nowSetting the permissions on OCR backup directorySetting up Network socket directoriesOracle Cluster Registry configuration upgraded successfullyclscfg: EXISTING configuration version 4 detected.clscfg: version 4 is 11 Release 1.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node <nodenumber>: <nodename> <private interconnect name> <hostname>node 1: krac1 krac1-priv krac1node 2: krac2 krac2-priv krac2clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.-force is destructive and will destroy any previous clusterconfiguration.Oracle Cluster Registry for cluster has already been initializedStartup will be queued to init within 30 seconds.Adding daemons to inittabExpecting the CRS daemons to be up within 600 seconds.Cluster Synchronization Services is active on these nodes. krac1 krac2 krac3Cluster Synchronization Services is active on all the nodes. Waiting for the Oracle CRSD and EVMD to startOracle CRS stack installed and running under init(1M)
From the first node, Looking at the ons.config file located in <CRS_HOME>/opmn/config directory. You can determine the ONS remote port to be used. You need to use the same port in racgons add_config as show in below to make sure that the first node can communicate with the ONS on new node.
[crs@krac1 ]$ cd $ORA_CRS_HOME/opmn/conf[crs@krac1 ]$ cat ons.config localport=6150useocr=onallowgroup=trueusesharedinstall=true[crs@krac1 conf]$
On new node(krac3) execute below commands. [crs@krac3 ~]$ crs_stat -t -c krac3Name Type Target State Host ------------------------------------------------------------ora.krac3.gsd application ONLINE ONLINE krac3 ora.krac3.ons application ONLINE ONLINE krac3 ora.krac3.vip application ONLINE ONLINE krac3 [oracle@krac3 ]$ su - crs[crs@krac3 ]$cd $ORA_CRS_HOME/bin[crs@krac3 bin]$ ./racgons add_config rac3:6150
Adding an ASM home to new nodes using OUI in interactive node.(Its needed when ASM is installed with separate home directory for high availability)
[crs@krac1 ]$ su - oracle[oracle@krac1 ]$ cd $ORA_ASM_HOME/oui/bin[oracle@krac1 ]$ ./addNode.sh
Action : NothingAfter Installation completes OUI will prompt you to run root.sh as root user. Once root.sh script execution completed follow with below steps.
The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u02/app/oracle/product/11.1.0/asm_1
Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script.
Action : NothingAfter Installation completes OUI will prompt you to run root.sh as root user. Once root.sh script execution completed follow with below steps.
The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u02/app/oracle/product/11.1.0/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]: The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying dbhome to /usr/local/bin ...The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying oraenv to /usr/local/bin ...The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script.Now product-specific root actions will be performed.Finished product-specific root actions.
Action : Click "Next"
Reconfigure listener on new node using NETCA
[crs@krac1 ]$ su - oracle[oracle@krac1 ]$ cd $ORA_ASM_HOME/bin[oracle@krac1 ]$ ./netca
SQL*Plus: Release 11.1.0.6.0 - Production on Mon Oct 29 19:55:42 2012
Copyright (c) 1982, 2007, Oracle. All rights reserved. Connected to:Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - ProductionWith the Partitioning, Real Application Clusters, OLAP, Data Miningand Real Application Testing options
SQL> col host_name for a25SQL> col instance_name for a20SQL> select instance_name,host_name from v$instance;
