Post on 04-Mar-2015
transcript
Step by Step of Adding Node to 11g RAC cluster on Linux:By Bhavin Hingubhavin@oracledba.org
<<HOME>>
Add Node To 11g RAC Cluster:
This document explains the step by step process of adding a Node to 11g RAC Cluster. Inthis process, I am going to add a single node (node2-pub)to single node RAC cluster online without affecting the availability of the Existing RACDatabase.
Existing RAC Architecture:
RAC Nodes:
Node 1: Public: node1-pub Private: node1-prv
Virtual: node1-vip
ORACLE_HOMES (Local on Each Node);
CRS_HOME: /u01/app/crs DB_HOME: /u01/app/oracle/product/11g/db_2 ASM_HOME: /u01/app/asm/product/11gr1
Database / ASM::
DB Name: test DB Instances: test1 on node1-pub ASM Instances: +ASM1 on node1-pub.
Node to be added: node2-pub
Assumptions:
MAXINSTANCES in controlfile is set to atleast 2.Node node2-pub is physically connected to the Cluster.Same Linux OS version as node 1 is installed on the new node
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
1 of 32 8/17/2011 4:07 PM
Tasks List (to be executed in Order):
Provision the New Node (node2-pub)Install CRSInstall 11g RAC DB Software for ASM_HOME on Node2-pubAdd LISTENER on node2-pubAdd DB Software for ORACLE_HOME on node2-pubAdd ASM Instance +ASM2 on node2pub ManuallyAdd DB Instance teste on node2-pub Manually.
Provision the New Node (node2-pub);
Packages:
Oracle 11g R1 on CentOS EL 4 Update 5 requires below extra packages to beinstalled with the version same or higher than listed below.
binutils-2.15.92.0.2-18compat-libstdc++-33.2.3-47.3elfutils-libelf-0.97-5elfutils-libelf-devel-0.97.5glibc-2.3.9.4-2.19glibc-common-2.3.9.4-2.19glibc-devel-2.3.9.4-2.19gcc-3.4.5-2gcc-c++-3.4.5-2libaio-devel-0.3.105-2libaio-0.3.105-2libgcc-3.4.5libstdc++-3.4.5-2
libstdc++-devel-3.4.5-2make-3.80-5sysstat-5.0.5unixODBC-2.2.11unixODBC-devel-2.2.11iscsi-initiator-utils-4.0.3.0-5 <-- For iscsi based SANconfiguration
Configuring Public and Private network for the
Cluster Nodes:
Each New node in the cluster must have 3 network adapter (eth0, eth1andeth2) one for the public, second one for the privatenetwork interface (internode communication, interconnect) and third onefor the Network Storage System (Private).Follow the below steps to configre these networks:
(1) Change the hostname value by executing the below command:
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
2 of 32 8/17/2011 4:07 PM
[root@localhost ~]# hostname node2-pub.hingu.net
(2) Edit the /etc/hosts file as shown below:
[root@node2-pub ~]# cat /etc/hosts# Do not remove the following line, or various programs# that require network functionality will fail.127.0.0.1 localhost.localdomain localhost ## Public Node names 192.168.10.11 node1-pub.hingu.net node1-pub192.168.10.22 node2-pub.hingu.net node2-pub ## Private Network (Interconnect) 192.168.0.11 node1-prv node1-prv192.168.0.22 node2-prv node2-prv ## Private Network (Network storage) 192.168.1.11 node1-nas node1-nas192.168.1.22 node2-nas node2-nas192.168.1.33 nas-server nas-server ## Virtual IPs 192.168.10.111 node1-vip.hingu.net node1-vip192.168.10.222 node2-vip.hingu.net node2-vip
(3) Edit OR create the /etc/sysconfig/network-scripts/ifcfg-eth0 as shownbelow:
[root@node2-pub ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0BOOTPROTO=noneIPADDR=192.168.10.22HWADDR=00:06:5B:AE:AE:7FONBOOT=yesTYPE=Ethernet[root@node2-pub ~]#
(4) Edit OR create the /etc/sysconfig/network-scripts/ifcfg-eth1 as shownbelow: -<-- For Cluster interconnects
[root@node2-pub ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1DEVICE=eth1BOOTPROTO=staticHWADDR=00:13:46:6A:FC:6DONBOOT=yesIPADDR=192.168.0.22NETMASK=255.255.255.0TYPE=Ethernet[root@node2-pub ~]#
(5) Edit OR create the /etc/sysconfig/network-scripts/ifcfg-eth2 on RACNodes as shown below: <-- For iSCSI SAN Storage Network
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
3 of 32 8/17/2011 4:07 PM
[root@node2-pub ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2# ADMtek NC100 Network Everywhere Fast Ethernet 10/100DEVICE=eth2ONBOOT=yesBOOTPROTO=staticIPADDR=192.168.1.22NETMASK=255.255.255.0HWADDR=00:18:F8:0F:0D:C1[root@node2-pub ~]#
(6) Edit the /etc/sysconfig/network file with the below contents:
[root@node2-pub ~]# cat /etc/sysconfig/networkNETWORKING=yesHOSTNAME=node2-pub.hingu.net
(7) Restart the network service OR reboot the nodes:Verify the network interface configurations by running the ifconfig command as shownbelow.
[root@node2-pub ~]# ifconfig -aeth0 Link encap:Ethernet HWaddr 00:06:5B:AE:AE:7F inet addr:192.168.10.22 Bcast:192.168.10.255 Mask:255.255.255.0 inet6 addr: fe80::206:5bff:feae:ae7f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:28768 errors:0 dropped:0 overruns:0 frame:0 TX packets:11884 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1970115 (1.8 MiB) TX bytes:3414698 (3.2 MiB) Interrupt:11 Base address:0xe480 eth1 Link encap:Ethernet HWaddr 00:13:46:6A:FC:6D inet addr:192.168.0.22 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::213:46ff:fe6a:fc6d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:472914 errors:0 dropped:0 overruns:0 frame:0 TX packets:468223 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:93005393 (88.6 MiB) TX bytes:105775559 (100.8MiB) Interrupt:10 Base address:0x6800 eth2 Link encap:Ethernet HWaddr 00:18:F8:0F:0D:C1 inet addr:192.168.1.22 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::218:f8ff:fe0f:dc1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:710843 errors:0 dropped:0 overruns:0 frame:0 TX packets:1107434 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:365638109 (348.6 MiB) TX bytes:211568148 (201.7MiB) Interrupt:9 Base address:0x6c00 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:41797 errors:0 dropped:0 overruns:0 frame:0
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
4 of 32 8/17/2011 4:07 PM
TX packets:41797 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:19150389 (18.2 MiB) TX bytes:19150389 (18.2 MiB) sit0 Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Memory and swap Space:
Oracle 10g RAC requires to have 1GB of RAM available on each node tosucessfully install 11g RAC. Well, I havemanaged to install it with 512 MB RAM. You will get warning during checking ofpre-requise step of installationstep which you can ignore. Please goto Adding an Extra Swapspace if you wantto have an extra swapspace added.
Kernel Parameters:
Oracle recommends that you set shared memory segmentattributes as well as semaphores to the following values.If not set, database instance creation will fail. I added thefollowing lines to /etc/sysctl.conf file. Every OS processneedssemaphore where It waits on for the resources. For more onsemaphore, please read the UNIX os documents.Oracle 10g RAC requires to have 1GB of RAM available oneach node to sucessfully install 11g RAC. Well, I havemanaged to install it with 512 MB RAM. You will get warningduring checking of pre-requise step of installationstep which you can ignore. Please goto Adding an ExtraSwapspace if you want to have an extra swapspace added.
NOTE: If the current value for any parameter is higher thanthe value listed in this table, then do not change the value of that parameter.
To see the current setting in the kernel, please use thebelow command.
/sbin/sysctl -a | grep sem -- for semmsl,semmns, semopm, semmni/sbin/sysctl -a | grep shm -- for shmall,shmmax, shmmni/sbin/sysctl -a | grep file-max/sbin/sysctl -a | grep ip_local_port_range/sbin/sysctl -a | grep rmem_default
Please add/change the appropriate variables value in the/etc/sysctl.conf file as shown below.
[root@node2-pub ~]# cat /etc/sysctl.conf# Kernel sysctl configuration file for Red Hat Linux
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
5 of 32 8/17/2011 4:07 PM
## For binary values, 0 is disabled, 1 is enabled. Seesysctl(8) and# sysctl.conf(5) for more details. # Controls IP packet forwardingnet.ipv4.ip_forward = 0 # Controls source route verificationnet.ipv4.conf.default.rp_filter = 1 # Do not accept source routingnet.ipv4.conf.default.accept_source_route = 0 # Controls the System Request debugging functionality ofthe kernelkernel.sysrq = 0 # Controls whether core dumps will append the PID to thecore filename.# Useful for debugging multi-threaded applications.kernel.core_uses_pid = 1 # Extra parameters For 11g RAC installation kernel.shmmax = 2147483648kernel.shmmni = 4096kernel.shmall = 2097152kernel.sem = 250 32000 100 128fs.file-max = 6553600net.ipv4.ip_local_port_range = 1024 65000net.core.rmem_default = 4194304net.core.wmem_default = 262144net.core.wmem_max= 262144net.core.rmem_max = 4194304 [root@node2-pub ~]#
After adding these lines to /etc/sysctl.conf, please run the below commandas root to make them enabled.
[root@node2-pub ~]# sysctl -p
Creating oracle OS User Account:
groupadd -g 900 dbagroupadd -g 901 oinstalluseradd -u 900 -g oinstall -G dba oraclepasswd oracle
Plesae verify that oracle user has same gid and uid on all the RAC nodes
by executing the this command
[oracle@node2-pub ~]$ iduid=900(oracle) gid=901(oinstall) groups=901(oinstall),900(dba)
[oracle@node1-pub ~]$ iduid=900(oracle) gid=901(oinstall) groups=901(oinstall),900(dba)
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
6 of 32 8/17/2011 4:07 PM
Creating Oracle Software Directories:
Perform the below steps on all the nodes in cluster.
[root@node2-pub ~]# mkdir -p /u01/app/crs[root@node2-pub ~]# mkdir -p /u01/app/asm[root@node2-pub ~]# mkdir -p /u01/app/oracle[root@node2-pub ~]# mkdir -p /u02/ocfs2
[root@node2-pub root]# chown -R oracle:oinstall /u01[root@node2-pub root]# chown -R oracle:oinstall /u02[root@node2-pub root]# chmod -R 775 /u01/app/oracle[root@node2-pub root]# chmod -R 775 /u01
Setting Shell Limits for the Oracle User:
To improve the performance of the software on Linuxsystems, you must increase the following shell limits for the oracle user:
Add the following lines to the /etc/security/limits.conf file:
oracle soft nproc 2047 oracle hard nproc 16384 oracle soft nofile 1024 oracle hard nofile 65536
Add or edit the following line in the /etc/pam.d/login file, if it does notalready exist:
session required /lib/security/pam_limits.so
For the Bourne, Bash, or Korn shell, add the following lines to the/etc/profile:
if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi
fi
For the C shell (csh or tcsh), add the following lines to the /etc/csh.login.
if ( $USER == "oracle" ) then limit maxproc 16384 limit descriptors 65536
endif
Enable SSH oracle user Equivalency on Both the
Cluster Nodes:
To configure SSH user equivalency, you must create RSA and DSA keys on each
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
7 of 32 8/17/2011 4:07 PM
cluster node and copy these keysfrom all the cluster node members into an authorised key file on each node.Follow the below steps to achieve this task.
su - oraclemkdir ~/.sshchmod 700 ~/.ssh
(A) Generate the RSA and DSA keys on Each NODE:
/usr/bin/ssh-keygen -t rsa/usr/bin/ssh-keygen -t dsa
(B) on node1-pub:
Add keys to the authorised key file and then send the same file to every nodes in cluster:
touch ~/.ssh/authorized_keyscd ~/.ssh
(1)
ssh node1-pub cat /home/oracle/.ssh/id_rsa.pub >>authorized_keysssh node1-pub cat /home/oracle/.ssh/id_dsa.pub >>authorized_keysssh node2-pub cat /home/oracle/.ssh/id_rsa.pub >>authorized_keysssh node2-pub cat /home/oracle/.ssh/id_dsa.pub >>authorized_keys
(2)
[oracle@node1-pub .ssh]$ scp authorized_keys node2-pub:/home/oracle/.ssh/
(C) ON Node2-pub:
(1)
ssh node1-pub cat /home/oracle/.ssh/id_rsa.pub >>authorized_keysssh node1-pub cat /home/oracle/.ssh/id_dsa.pub >>authorized_keysssh node2-pub cat /home/oracle/.ssh/id_rsa.pub >>authorized_keysssh node2-pub cat /home/oracle/.ssh/id_dsa.pub >>authorized_keys
(2)
[oracle@node2-pub .ssh]$ scp authorized_keys node1-pub:/home/oracle/.ssh/
(D) Change the Permission of authorized_Keys file (on each node)
[oracle@node2-pub .ssh]$ chmod 600 ~/.ssh/authorized_keys
During executing step B - (1), you may be prompted as show below. Enter "yes" and continue.
[oracle@node2-pub .ssh]$ ssh node1-pub cat /home/oracle/.ssh/id_rsa.pub >> authorized_keysThe authenticity of host 'node1-pub (216.160.37.154)' can't be
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
8 of 32 8/17/2011 4:07 PM
established.RSA key fingerprint is <**********>.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'node1-pub,216.160.37.154' (RSA) tothe list of known hosts.Warning: No xauth data; using fake authentication data for X11forwarding.
Now, try executing the date (or any other command) on remote nodeto make sure that oracle is not asked for the password.Do the same on both the nodes in cluster. You should not recieve anyerror message while you execute these commands onall the nodes. If you get any error, first fix them before you gofurther.
[oracle@node2-pub oracle]$ ssh node1-prv dateThu Sep 20 21:16:56 CDT 2007[oracle@node2-pub oracle]$ ssh node2-prv dateThu Sep 20 21:17:03 CDT 20075[oracle@node2-pub oracle]$ ssh node1-pub dateThu Sep 20 21:17:09 CDT 2007[oracle@node2-pub oracle]$ ssh node2-pub date
Thu Sep 20 21:17:19 CDT 2007
Errors / Warnings during the network configurations:
I got the below warning when I tried below command.
[oracle@node2-pub .ssh]$ ssh node1-pub dateWarning: No xauth data; using fake authentication data for X11forwarding.Thu Sep 20 20:43:39 CDT 2007
To fix the above warning, create the /home/oracle/.ssh/config file (logged in asoracle user) and make the below entryin it. then run the same command again and the above warning would not showup.
[oracle@node2-pub oracle]$ cat .ssh/config Host * Forwardx11 no
You may be promted to enter 'yes' or 'no' for the first time performing sshcommands. Simply ente'yes' and continue.Afterwords, when oracle connect to the remote node, it won't be asked for thepassword.
[oracle@node2-pub oracle]$ ssh node1-prv dateThe authenticity of host 'node1-prv (192.168.203.1)' can't beestablished.RSA key fingerprint is<********************************************>Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'node1-prv,192.168.203.1' (RSA) to thelist of known hosts.Sun Dec 18 20:01:09 CST 2005
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
9 of 32 8/17/2011 4:07 PM
[oracle@node2-pub oracle]$ ssh node1-prv dateSun Dec 18 20:01:13 CST 2005[oracle@node2-pub oracle]$
[oracle@node2-pub oracle]$ ssh node2-prv dateWarning: Permanently added the RSA host key for IP address'192.168.203.2' to the list of known hosts.Sun Dec 18 20:14:16 CST 2005[oracle@node2-pub oracle]$ ssh node2-pub dateSun Dec 18 20:15:05 CST 2005
If you get then below erorr message when try to connect toremote node, please make sure that the firewall is disabled ontheremote node.
[root@node2-pub root]# telnet node1-prvTrying 192.168.203.1...telnet: Unable to connect to remote host: No route to host
Configuring System for Shared Disk Storage Device (iSCSIbased IP SAN):
The below 5 volumes are created under the rac11g volume group. Formore Information on setting up and creating shared volumes, please gotohere.
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
10 of 32 8/17/2011 4:07 PM
Make sure that each volumes allows the shared access to all the nodes inthe cluster. You can do that by clicking the 'Edit' link on the above screenfor each volume name as shown below.
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
11 of 32 8/17/2011 4:07 PM
The below screen shows that both the nodes in the cluster has sharedaccess on the storage. Click on the General tab to add/modify the rac nodesinfo.The network information provided in this table are provate network for theshared storage.
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
12 of 32 8/17/2011 4:07 PM
Discovering the volumes to the new node node2-pub as scsi
devices
NOTE: make sure that SELinux and firewall has been disabled on all theRAC nodes. If not, then disable by "Administraion Sections"--> "system settings" --> "security Level"
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
13 of 32 8/17/2011 4:07 PM
on node2-pub:
Make sure that the iscsi-initiator is installed. If not, then downlaod the specified
RPM and istall them.
[root@node2-pub rpms]# rpm -qa | grep iscsiiscsi-initiator-utils-4.0.3.0-5 [root@node2-pub rpms]# service iscsi start -- if not already running
[root@node2-pub rpms]# service iscsi statusiscsid (pid 3256 3253) is running...
Configure nodes for the Disk Discovery:
Edit the /etc/iscsi.conf and assign the address of the iscsi target
to the below parameter:
# --------------------------# Discovery Address Category# --------------------------# To localize parameters to targets found on a particular discoveryaddress# (i.e., to override the global settings), you need to use the# "DiscoveryAddress" setting. The format for the "DiscoveryAddress"setting is# a.b.c.d, a.b.c.d:e (where e is a TCP port number), or an instancename. An# example of these settings would be:##DiscoveryAddress=10.4.100.0# or#DiscoveryAddress=10.4.100.1:3260# orDiscoveryAddress=192.168.1.33 # <-- This is the address of thenas-server#
(b)
Reboot All the Nodes and run the iscsi-ls command to see if thevolumes have been discovered on the RAC nodes as scsi devices.
(c)
[root@node2-pub rpms]# iscsi-ls*******************************************************************************SFNet iSCSI Driver Version ...4:0.1.11-4(15-Jan-2007)*******************************************************************************TARGET NAME : iqn.2006-01.com.openfiler:rac11g.ocfs-dskTARGET ALIAS :HOST ID : 0BUS ID : 0TARGET ID : 0TARGET ADDRESS : 192.168.1.33:3260,1SESSION STATUS : ESTABLISHED AT Sun Sep 2 18:49:57 CDT 2007SESSION ID : ISID 00023d000001 TSIH 1200*******************************************************************************TARGET NAME : iqn.2006-01.com.openfiler:rac11g.asm-dsk4TARGET ALIAS :
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
14 of 32 8/17/2011 4:07 PM
HOST ID : 1BUS ID : 0TARGET ID : 0TARGET ADDRESS : 192.168.1.33:3260,1SESSION STATUS : ESTABLISHED AT Sun Sep 2 18:49:57 CDT 2007SESSION ID : ISID 00023d000001 TSIH 1000*******************************************************************************TARGET NAME : iqn.2006-01.com.openfiler:rac11g.asm-dsk3TARGET ALIAS :HOST ID : 2BUS ID : 0TARGET ID : 0TARGET ADDRESS : 192.168.1.33:3260,1SESSION STATUS : ESTABLISHED AT Sun Sep 2 18:49:57 CDT 2007SESSION ID : ISID 00023d000001 TSIH 1100*******************************************************************************TARGET NAME : iqn.2006-01.com.openfiler:rac11g.asm-dsk2TARGET ALIAS :HOST ID : 3BUS ID : 0TARGET ID : 0TARGET ADDRESS : 192.168.1.33:3260,1SESSION STATUS : ESTABLISHED AT Sun Sep 2 18:49:57 CDT 2007SESSION ID : ISID 00023d000001 TSIH 1300*******************************************************************************TARGET NAME : iqn.2006-01.com.openfiler:rac11g.asm-dsk1TARGET ALIAS :HOST ID : 4BUS ID : 0TARGET ID : 0TARGET ADDRESS : 192.168.1.33:3260,1SESSION STATUS : ESTABLISHED AT Sun Sep 2 18:49:58 CDT 2007SESSION ID : ISID 00023d000001 TSIH 1400*******************************************************************************[root@node2-pub rpms]#
(4) Map the Volumes on the iscsi-target (nas-server) to the Disksdisceverd on the local RAC nodes.
Host ID Target ID discovered as 0 iqn.2006-01.com.openfiler:rac11g.ocfs-dsk --- 1 iqn.2006-01.com.openfiler:rac11g.asm-dsk4 --- 2 iqn.2006-01.com.openfiler:rac11g.asm-dsk3 --- 3 iqn.2006-01.com.openfiler:rac11g.asm-dsk2 --- 4 iqn.2006-01.com.openfiler:rac11g.asm-dsk1 ---
Now, run the below command to find out the "Attached" devices tothe Host IDs. The scsi Id in thisoutput maps to the Host ID on the "iscsi-ls" output.
[root@node2-pub rpms]# dmesg | grep Attached Attached scsi disk sda at scsi0, channel 0, id 0, lun 0Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0Attached scsi disk sdc at scsi2, channel 0, id 0, lun 0Attached scsi disk sdd at scsi3, channel 0, id 0, lun 0Attached scsi disk sde at scsi4, channel 0, id 0, lun 0
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
15 of 32 8/17/2011 4:07 PM
In first line, scsi0 (Host Id 0) has device "sda" attahced to it. So, By filling the
above table
with this information gives the mapping of discovered Disks at client to its actual
Volums on the iscsi-target.
Host ID Target ID VolumeName discovered as 0 iqn.2006-01.com.openfiler:rac11g.ocfs-dsk ocfs-dsk sda 1 iqn.2006-01.com.openfiler:rac11g.asm-dsk4 asm-dsk4 sdb 2 iqn.2006-01.com.openfiler:rac11g.asm-dsk3 asm-dks3 sdc 3 iqn.2006-01.com.openfiler:rac11g.asm-dsk2 asm-dsk2 sdd 4 iqn.2006-01.com.openfiler:rac11g.asm-dsk1 asm-dsk1 sde
NOTE: You do not partition the Shared Disks on new Node. After successfull discoveryof the Shared Volumes, as shown above, all the available partitions on these volumeswill become visible on node2-pub node.
[root@node2-pub ~]# fdisk -l
Disk /dev/hda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hda1 * 1 13 104391 83 Linux
/dev/hda2 14 19457 156183930 8e Linux LVM
Disk /dev/sda: 80.6 GB, 80664854528 bytes
255 heads, 63 sectors/track, 9806 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 9806 78766663+ 83 Linux
Disk /dev/sdb: 104.8 GB, 104857600000 bytes
255 heads, 63 sectors/track, 12748 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 12748 102398278+ 83 Linux
Disk /dev/sdc: 104.8 GB, 104857600000 bytes
255 heads, 63 sectors/track, 12748 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 12748 102398278+ 83 Linux
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
16 of 32 8/17/2011 4:07 PM
Disk /dev/sdd: 104.8 GB, 104857600000 bytes
255 heads, 63 sectors/track, 12748 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 12748 102398278+ 83 Linux
Disk /dev/sde: 104.8 GB, 104857600000 bytes
255 heads, 63 sectors/track, 12748 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 12748 102398278+ 83 Linux[root@node1-pub ~]#
Make Device name persistent (node2-pub):
DOWNLOAD the latest rpm of devlabel from Dell's website:1.
Install it on all the RAC nodes:2.
[root@node1-pub Desktop]# rpm -ivh devlabel-0.48.01-1.i386.rpmwarning: devlabel-0.48.01-1.i386.rpm: V3 DSA signature: NOKEY,key ID 23b66a9dPreparing... ########################################### [100%] 1:devlabel ########################################### [100%][root@node1-pub Desktop]#
3. Create the SYMLINK using the devlabel (All the RAC nodes):
(A). Get the Host ID and the Devices attached to that ID bybelow command:
[root@node1-pub ~]# dmesg | grep AttachedAttached scsi disk sda at scsi5, channel 0, id 0, lun 0Attached scsi disk sdb at scsi8, channel 0, id 0, lun 0Attached scsi disk sdc at scsi6, channel 0, id 0, lun 0Attached scsi disk sdd at scsi9, channel 0, id 0, lun 0Attached scsi disk sde at scsi7, channel 0, id 0, lun 0
(B). Get the Target Name associated to each Host ID by belowcommand:
[root@node1-pub ~]# iscsi-ls
(C). Create SYMLINK using below command:
[root@node1-pub ~]# devlabel add -d /dev/sdc1 -s/dev/ocfs2SYMLINK: /dev/ocfs2 -> /dev/sde1Added /dev/ocfs2 to /etc/sysconfig/devlabel[root@node1-pub ~]#[root@node1-pub ~]# devlabel add -d /dev/sde1 -s/dev/asmdsk4SYMLINK: /dev/asmdsk4 -> /dev/sde1
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
17 of 32 8/17/2011 4:07 PM
Added /dev/asmdsk4 to /etc/sysconfig/devlabel[root@node1-pub ~]#[root@node1-pub ~]# devlabel add -d /dev/sde1 -s/dev/asmdsk3SYMLINK: /dev/asmdsk3 -> /dev/sde1Added /dev/asmdsk3 to /etc/sysconfig/devlabel[root@node1-pub ~]#[root@node1-pub ~]# devlabel add -d /dev/sdb1 -s/dev/asmdsk2SYMLINK: /dev/asmdsk2 -> /dev/sdb1[root@node1-pub ~]#Added /dev/asmdsk2 to /etc/sysconfig/devlabel[root@node1-pub ~]# devlabel add -d /dev/sdd1 -s/dev/asmdsk1SYMLINK: /dev/asmdsk1 -> /dev/sdd1[root@node1-pub ~]#Added /dev/asmdsk1 to /etc/sysconfig/devlabel[root@node1-pub ~]#
(D). Confirm that the symlinks are associated with the UUID ofeach physical devices.
[root@node1-pub ~]# cat /etc/sysconfig/devlabel# devlabel configuration file## This file should generally not be edited by hand.# Instead, use the /sbin/devlabel program to makechanges.# devlabel by Gary Lerhaupt <gary_lerhaupt@dell.com>## format: <SYMLINK> <DEVICE> <UUID># or format: <RAWDEVICE> <DEVICE> <UUID>
/dev/ocfs2 /dev/sda1S83.1:4f70656e66696c000000000005000000b92700000e000000OpenfileVirtualdisks/dev/asmdsk4 /dev/sdc1S83.1:4f70656e66696c000000000004000000a52700000e000000OpenfileVirtualdisks/dev/asmdsk3 /dev/sde1S83.1:4f70656e66696c000000000003000000912700000e000000OpenfileVirtualdisks/dev/asmdsk2 /dev/sdb1S83.1:4f70656e66696c0000000000020000007d2700000e000000OpenfileVirtualdisks/dev/asmdsk1 /dev/sdd1S83.1:4f70656e66696c000000000001000000672700000e000000OpenfileVirtualdisks[root@node1-pub ~]#
(E). Mount the ocfs2 devices under the appropriate mount pointand update the /etc/fstab (in my case it is /u02/ocfs2):
[root@node1-pub ~]# mount -t ocfs2 -o datavolume,nointr/dev/ocfs2 /u02/ocfs2[root@node1-pub ~]# vi /etc/fstab
[root@node1-pub ~]# cat /etc/fstab# This file is edited by fstab-sync - see 'manfstab-sync' for details/dev/VolGroup00/LogVol00 / ext3 defaults 1 1LABEL=/boot /boot ext3 defaults 1 2none /dev/pts devpts gid=5,mode=620 0 0none /dev/shm tmpfs defaults 0 0none /proc proc defaults 0 0
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
18 of 32 8/17/2011 4:07 PM
none /sys sysfs defaults 0 0/dev/VolGroup00/LogVol01 swap swap defaults 0 0
## OCFS2 mountpoint
/dev/ocfs2 /u02/ocfs2 ocfs2 _netdev,datavolume,nointr 0 0/dev/hdc /media/cdrom auto pamconsole,exec,noauto,managed 0 0/dev/fd0 /media/floppy auto pamconsole,exec,noauto,managed 0 0
(F). Update the /etc/rc.local files
Copy the existing /etc/rc.local file from node1-pub to node2-pub.
[root@node2-pub ~]# cat /etc/rc.local
THe below is the sample output of my /etc/rc.local
# Affirms the mappings of symlinks to storage devices. If a# device cannot be found by its UUID, the existingsymlink will# be deleted. If the device name has changed, it willupdate the# symlink. Both start and restart are synonyms forthis command.
service iscsi restartdevlabel reloadmount -t ocfs2 -o datavolume,nointr /dev/ocfs2/u02/ocfs2
Installing and Configuring OCFS2 (Oracle Cluster File
System):
I have followed the OCFS2 setup guide to install and configure the OCFS2.
Download and Install the required rpms:
Please download the below rpms from Oracle's website and install them on newnode node2-pub as shown below,
ocfs2-2.6.9-55.0.2.EL-1.2.5-6.i686.rpm
ocfs2-tools-1.2.4-1.i386.rpm
ocfs2-tools-debuginfo-1.2.4-1.i386.rpm
ocfs2-tools-devel-1.2.4-1.i386.rpm
[root@node2-pub rpms]# rpm -ivh ocfs2-2.6.9-55.0.2.EL-1.2.5-6.i686.rpm \> ocfs2-tools-1.2.4-1.i386.rpm \> ocfs2console-1.2.4-1.i386.rpm \> ocfs2-tools-debuginfo-1.2.4-1.i386.rpm \> ocfs2-tools-devel-1.2.4-1.i386.rpmPreparing... ###########################################[100%] 1:ocfs2-tools ########################################### [
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
19 of 32 8/17/2011 4:07 PM
20%] 2:ocfs2-2.6.9-55.0.2.EL ########################################### [40%] 3:ocfs2console ########################################### [60%] 4:ocfs2-tools-debuginfo ########################################### [80%] 5:ocfs2-tools-devel ###########################################[100%][root@node2-pub rpms]#
Configure OCFS2:
As we are using OCFS2 filesystem for the OCR and VoteDisk, we need toinstall OCFS2 on the new Node as well so that we can mount the samevolume on the new Node to get access to the OCFS2 filesystem. Configurethe OCFS2 by running ocfs2console tool as shown below.
Running ocfs2console command line utility brings this screen.
You can close this box as you can enable the o2cb service later.
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
20 of 32 8/17/2011 4:07 PM
Click Cluster --> Configure Nodes. Add node names for node2-pubMake sure to add exact same node name as it has been returned by the`hostname` command.The hostname for the new node is node2-pub.hingu.net.
Prapogate the files to all the nodes in cluster:
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
21 of 32 8/17/2011 4:07 PM
So, the above settings of "Name" (node2-nas) in the ocfs2 configuraion willlead to below error when I try to enable o2cb service.
[root@node2-pub rpms]# /etc/init.d/o2cb enableWriting O2CB configuration: OKStarting O2CB cluster ocfs2: FailedCluster ocfs2 createdNode node1-nas addedNode node2-nas addedo2cb_ctl: Configuration error discovered while populating cluster ocfs2. None of its nodes were considered local. A node is considered local when
its node name in the configuration matches this machine's host name.Stopping O2CB cluster ocfs2: OK[root@node2-pub rpms]#
So, stop o2cb service, open the /etc/ocfs2/cluster.conf file and update thehostname value to the one that is returned by `hostname` command. Donot update the IP. start the service and load it again and this time errorshould go away.--=================
[oracle@node2-pub ~]$ cat /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 192.168.0.11
number = 0
name = node1-pub.hingu.net
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
22 of 32 8/17/2011 4:07 PM
cluster = ocfs2
node:
ip_port = 7777
ip_address = 192.168.0.22
number = 1
name = node2-pub.hingu.net
cluster = ocfs2
cluster:
node_count = 2
name = ocfs2
--================
[root@node2-pub rpms]# /etc/init.d/o2cb loadLoading module "configfs": OKCreating directory '/config': OKMounting configfs filesystem at /config: OKLoading module "ocfs2_nodemanager": OKLoading module "ocfs2_dlm": OKLoading module "ocfs2_dlmfs": OKCreating directory '/dlm': OKMounting ocfs2_dlmfs filesystem at /dlm: OK[root@node2-pub rpms]# /etc/init.d/o2cb statusModule "configfs": LoadedFilesystem "configfs": MountedModule "ocfs2_nodemanager": LoadedModule "ocfs2_dlm": LoadedModule "ocfs2_dlmfs": LoadedFilesystem "ocfs2_dlmfs": MountedChecking O2CB cluster ocfs2: Offline[root@node2-pub rpms]#
Configure o2cb to startup at Boot time:
[root@node2-pub rpms]# /etc/init.d/o2cb configureConfiguring the O2CB driver. This will configure the on-boot properties of the O2CB driver.The following questions will determine whether the driver is loaded onboot. The current values will be shown in brackets ('[]'). Hitting<ENTER> without typing an answer will keep that current value. Ctrl-Cwill abort. Load O2CB driver on boot (y/n) [n]: yCluster to start on boot (Enter "none" to clear) [ocfs2]:Specify heartbeat dead threshold (>=7) [7]:Specify network idle timeout in ms (>=5000) [10000]:Specify network keepalive delay in ms (>=1000) [5000]:Specify network reconnect delay in ms (>=2000) [2000]:Writing O2CB configuration: OKStarting O2CB cluster ocfs2: OK[root@node2-pub rpms]#
[root@node2-pub rpms]# chkconfig --add ocfs2[root@node2-pub rpms]# chkconfig --add o2cb[root@node2-pub rpms]# mkdir -p /u02/ocfs2 -- ocfs2 mountpoint
Mount the filesystem
Mount the ocfs2 devices under the appropriate mount point and update the
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
23 of 32 8/17/2011 4:07 PM
/etc/fstab (in my case it is /u02/ocfs2):
[root@node2-pub ~]# mount -t ocfs2 -o datavolume,nointr /dev/ocfs2/u02/ocfs2
[root@node2-pub ~]# cat /etc/fstab# This file is edited by fstab-sync - see 'man fstab-sync' for details/dev/VolGroup00/LogVol00 / ext3 defaults 1 1LABEL=/boot /boot ext3 defaults 1 2none /dev/pts devpts gid=5,mode=620 0 0none /dev/shm tmpfs defaults 0 0none /proc proc defaults 0 0none /sys sysfs defaults 0 0/dev/VolGroup00/LogVol01 swap swap defaults 0 0
## OCFS2 mountpoint
/dev/ocfs2 /u02/ocfs2 ocfs2 _netdev,datavolume,nointr 0 0/dev/hdc /media/cdrom auto pamconsole,exec,noauto,managed 0 0/dev/fd0 /media/floppy auto pamconsole,exec,noauto,managed 0 0
[root@node2-pub ~]# mount -t ocfs2 /dev/sda1 /u02/ocfs2mount.ocfs2: Transport endpoint is not connected while mounting /dev/sda1on /u02/ocfs2. Check 'dmesg' for more information on this error.
Solution: Make sure that SELinux and Firewall has been disabled.If not, Disable them and reboot the nodes,
and it should be ok.
Creating Automatic Storage Management (ASM)
Disks for the Clustered Database:
Download and Install these RPMs on node node2-pub.
[oracle@node2-pub ~]$ uname -aLinux node2-pub.hingu.net 2.6.9-55.0.2.EL #1 Tue Jun 26 14:08:18 EDT2007 i686 i686 i386 GNU/Linux
oracleasm-2.6.9-55.0.2.EL-2.0.3-1.i686.rpm
oracleasmlib-2.0.2-1.i386.rpm
oracleasm-support-2.0.3-1.i386.rpm
[root@node2-pub rpms]# rpm -ivh oracleasm-2.6.9-55.0.2.EL-2.0.3-1.i686.rpm\> oracleasmlib-2.0.2-1.i386.rpm \> oracleasm-support-2.0.3-1.i386.rpmPreparing... ###########################################[100%] 1:oracleasm-support ########################################### [33%]
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
24 of 32 8/17/2011 4:07 PM
2:oracleasm-2.6.9-55.0.2.########################################### [67%] 3:oracleasmlib ###########################################[100%][root@node2-pub rpms]#
Configure ASM on node2-pub
[root@node2-pub ~]# /etc/init.d/oracleasm configureConfiguring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM librarydriver. The following questions will determine whether the driver isloaded on boot and what permissions it will have. The current valueswill be shown in brackets ('[]'). Hitting <ENTER> without typing ananswer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: oracleDefault group to own the driver interface []: dbaStart Oracle ASM library driver on boot (y/n) [n]: yFix permissions of Oracle ASM disks on boot (y/n) [y]:Writing Oracle ASM library driver configuration: [ OK ]Creating /dev/oracleasm mount point: [ OK ]Loading module "oracleasm": [ OK ]Mounting ASMlib driver filesystem: [ OK ]Scanning system for ASM disks: [ OK ][root@node1-pub ~]#
Scan the ASM Disk Device(s) that are currently being used by ASM:
[root@node2-pub ~]# /etc/init.d/oracleasm scandisksScanning system for ASM disks: [ OK ][root@node2-pub ~]# /etc/init.d/oracleasm listdisksDSK1DSK2DSK3DSK4[root@node2-pub ~]#[root@node2-pub ~]# /etc/init.d/oracleasm statusChecking if ASM is loaded: [ OK ]Checking if /dev/oracleasm is mounted: [ OK ]
Install CRS;
Connect to an existing Node as oracle user (node1-pub), and execute$CRS_HOME/oui/bin/addNode.sh script.Make sure to run this scrips using X-terminal.
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
25 of 32 8/17/2011 4:07 PM
Specify the Public, Private and Virtual Name of the New Node..
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
26 of 32 8/17/2011 4:07 PM
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
27 of 32 8/17/2011 4:07 PM
Click Install
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
28 of 32 8/17/2011 4:07 PM
Run the mentioned script one by one in order on the mentioned nodes.
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
29 of 32 8/17/2011 4:07 PM
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
30 of 32 8/17/2011 4:07 PM
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
31 of 32 8/17/2011 4:07 PM
Verify the CRS as well as nodeapps status on the new node node2-pub byrunning crsctl check cluster followed by crs_stat -t command.
Install ASM_HOME
<<HOME>>
REFERENCES:
Oracle Documents
11g RAC -- Add Node http://www.oracledba.org/11g/rac/11g_RAC_Add_Node.html
32 of 32 8/17/2011 4:07 PM