© 2012. Santosh Kangane

- Santosh Kangane

Santosh Kangane – Oracle DBAsantoshkangane.blogspot.inwww.linkedin.com/pub/santosh-kangane/1b/78b/6b

System Requirement

Oracle 11gR2 RAC overview

Storage Option

SAN

NAS

SCAN

OCR, Voting Disk and Time Synchronisation

Oracle ASM

Oracle Enterprise manager for RAC

Highlights

System Requirement

Oracle Certified Operating Systems for RAC : Linux :

Red Hat Enterprise Linux : 4.7, 5.2, 5.5, 6 SUSE Linux Enterprise Server : 10 SP2, 11 Oracle Enterprise Linux : All versions

Oracle Solaris ( Most popular option for production systems )

Windows* All Node must have same OS and Kernel Version

Memory Requirement for Linux platform:

RAM Swap Space Required

Between 1 GB and 2 GB 1.5 times the size of the RAM

Between 2 GB and 16 GB Equal to the size of the RAM

More than 16 GB 16 GB

Shared Memory : Minimum 40% of RAM; Should be grater than size of SGA

Oracle 11g Real Application Cluster

© 2012. Santosh Kangane

LAN

Storage Server (PTS0012)

RAC Node 1 (PTS0009)

RHEL 5.0

RAC Node 2 (PTS0006)

RHEL 5.0

eth1 (private IP)10.88.32.102

eth0 (public IP)10.88.33.23PTS0009-vip10.88.33.28

eth1 (private IP)10.88.32.101

eth0 (public IP)10.88.33.22PTS0006-vip10.88.33.27 DNS

ORARAC-SCAN10.88.33.2410.88.33.2510.88.33.26

3 IP’s with round robin allocation mode

Oracle 11g RAC – PEG Lab setup

Oracle 11g Real Application Cluster Stack

1

2

4

3

5

6

7

* Number shows the startup sequence

Storage Options

NAS and SAN

IP – Based Storage Area Network Configuration

Standard configuration options- Tool provided by Storage vendors- OpenFiler Open source DOS - Manual Setup using iSCSI protocol

Target (Ubuntu 12.04 on PTS0012) Install iscsitarget, iscsitarget-source, iscsitarget-dkms package Format the disk and create the partitions as per your requirements. ( I have 10 GB for

OCR &Voting Disk; 61 GB for oracle storage). edit /etc/default/iscsitarget. Change the default value of ISCSITARGET_ENABLE=false

over to ISCSITARGET_ENABLE=true. Define LUNs in /etc/iet/ietd.conf

Target iqn.2012-08.in.co.persistent:storage.disk0. pts0012LUN 0 Path=/dev/cciss/c0d1p5,Type=fileio,ScsiId=lun0,ScsiSN=lun0

Node :― 0th LUN is mandatory.― Make sure there is no space in LUN Definition above and every parameter isseparated by “,”

Restart the service iscsitarget

IP – Based Storage Area Network

Initiator ( All RAC node RHEL5) Install iscsi-initiator-util package.

(Make sure that you have iscsi version >= iscsi-initiator-utils-6.2.0.868-0.7.el5) Edit /etc/iscsi/iscsid.conf and set node.startup = automatic Then restart the initiator: /etc/init.d/iscsi restart Make sure iscsid service is configured to start at system boot up

$ chkconfig --list iscsidiscsid 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Using CHAP authentication for security : Edit /etc/iscsi/iscsid.confEnable following entries :

node.session.auth.authmethod = CHAPnode.session.auth.username = <<sanusername>>node.session.auth.password = <<sanpassword>>

Discover the target using command $ iscsiadm –m discovery –p <<Target IP>>:3260

View detected nodes$ iscsiadm –m node

IP – Based Storage Area Network

Initiator ( All RAC node RHEL5) …cont Login to the LUNs ( use same sequence to login on all nodes so that the attached

devices names will be identical )

$iscsiadm -m node --targetname “<<inq name of first LUN>>" --portal “<< Target IP >>:3260" --login Verify the sessions

$ iscsiadm -m session -P3 | less Now in $fdisk –l all newly attached iscsi drives will be shown ( make sure every

LUN of storage has got same Device name on all RAC nodes) Format newly added drives from any ONE node and use $partprobe to push

partition table on all other nodes Verify on all nodes if newly attached drives are showing valid portions using

fdisk -l

For more details on Storage Initiator setup and RAC Ethernet tuning refer :(http://www.scribd.com/doc/46183679/Red-Hat-Linux-v5-x-Software-iSCSI-Initiator-Configuration-MPIO-and-Tuning-Guide#outer_page_5)

Oracle RAC 11gR2 – Single Client Access Name (SCAN)

How do I connect to a cluster ? How to configure Load balancing? How to handle Node failure?Who to isolate node addition/removal without affecting client connections and downtime ?

Single client access name (SCAN) is the virtual hostname (of 1-15 Char) to provide for all clients connecting to the cluster and It should be unique across the network domain.

SCAN is registered with DNS (or Grid Naming Service) with at least one and up to three IP addresses from the same subnet as that of Public and VIP address.

These IP addresses should be allocated in round robin fashion

Oracle RAC 11gR2 – Scan Single Client Access Name (SCAN)

1. On each Node there is pair of SCAN Listener & Local Listener

3 SCAN IP address will be mapped as: • 3 Node : Each node will be mapped

with one SCAN IP• More than 3 node : only 3 nodes will

be mapped to SCAN IPs• 2 Node : One node will have double

SCAN IP & SCAN Listener and other node will have single IP and SCAN Listener

2. PMON process of each instance report the current work load to SCAN listener Service (specified in the REMOTE_LISTENER database parameter.)

3. On connect request DNS resolves the orarac-scan and returns list of 3 SCAN IP addresses

4. Client select the first IP from the list and connect to the RAC; if it fails then request again with next IP.

5. Depending on the Load SCAN listener re-direct the request to Local listener of lightly loaded node. ( Note here in case of More than 3 node cluster, this is how request can still be given to lightly loaded node via SCAN listener service, even if it does not have SCAN IP mapped )

6. Now all further communication happens through local listener directly.

PEGLAB1

PEGLAB2

2

3

4

5

6

SQL> connuser/pwd@orarac-scan/peglab

OCR And Voting Disk

Oracle Cluster Registry:The OCR maintains cluster configuration information that is used by each node

of the cluster to determine the state of the cluster.OCR also maintains information about cluster resources:

• Databases• Instance• Services

Each node in the cluster maintains a copy of the OCR in memory for better performance and also responsible for updating the OCR as required in shared storage.

Voting Disk:The Voting Disk Files are used by Oracle Clusterware to determine which nodes

are currently members of the cluster. The voting disk files are also used in concert withother Cluster components such as CRS to maintain the clusters integrity.

Auto Segment Management : Architecture

PEGLAB1

CLUSTERWARE

+ASM1

CLUSTERWARE

+ASM2

PEGLAB

Clusterware

ASM Instances

RDBMS Instances

Database Files

PEGLAB2

ASM Disk Groups, Disks and Database Files

Disk Group 1

Disk 1 Disk 4

Disk Group 2

Disk 2 Disk 5 Disk 6 Disk 7

Disk 3

Disk Group 3

File 3

File 4

File 5

File 2

File 5

File 6

File 1

ASM Failure Groups - External Redundancy

Disk Group

Disk 1 Disk 2 Disk 3

ASM Failure Groups – Normal Redundancy

Failure Group 1 Failure Group 2

Disk Group

Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6

ASM Failure Groups – High Redundancy

Failure Group 1

Disk Group

Disk 1 Disk 2

Failure Group 2

Disk 1 Disk 2

Failure Group 3

Disk 1 Disk 2

Extent Distribution – External Redundancy – 1 Disk

Disk 0

Metadata

0

1

2

3

4

5

6

7

Metadata Data

Extent Distribution – External Redundancy – 2 Disk

Disk 0 Disk 1

Metadata

0

2

4

6

8

10

12

14

1

3

5

7

9

11

13

15

Metadata Data

Extent Distribution – Normal Redundancy – 2 Disk

Disk 0 Disk 1 Disk 2 Disk 3

0

3

0 1

23

1

2

4

7

5

6

4

7

5

6

20

23

Failure Group 1 Failure Group 2

16

19

12

15

0

3

4

7

8

11

18

21

14

17

10

13

1

2

5

6

9

22

19

20

15

16

11

12

0

3

4

7

8

23

21

22

17

18

13

14

1

2

5

6

9

10

Primary SecondaryMetadata

Data Primary Secondary

Extent Distribution – High Redundancy – 1 Disk

Disk 0 Disk 1 Disk 2

0 0

Failure Group 1

0 0

Failure Group 2 Failure Group 3

0

111

2 2 2

3 33

444

5 5 5

0

11 1

22 2

3 3 3

44 4

55 5

6 6 6

77 7

88 8

9 9 9

Primary Secondary TertiaryMetadata

Data Primary Secondary Tertiary

RAC Implementation Step By Step

Document Explains the step by steps executions and seen dumps for- Pre-requisite Linux packages installation - Oracle 11gR2 Clusterware installation- Oracle 11gR2 Database software installation- Creation of ASM Disk group- Creation of Oracle 11g Database on Cluster

References

1. http://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm2. http://www.oracle.com/technetwork/products/clustering/overview/scan-

129069.pdf3. https://support.oracle.com4. http://www.scribd.com/doc/46183679/Red-Hat-Linux-v5-x-Software-iSCSI-

Initiator-Configuration-MPIO-and-Tuning-Guide#outer_page_55. http://www.oracledba.org/11gR2/Pre_Install_11gR2.htm6. http://www.toadworld.com/KNOWLEDGE/KnowledgeXpertforOracle/tabid/6

48/TopicID/RACR2ARC6/Default.aspx7. http://juliandyke.com/Presentations/Presentations.html

25

santoshkangane.blogspot.com