VERITAS Storage Foundation4.0 for Oracle RAC

Oz MelamedE&M Computing

Cluster Basics

Private NetworkLLT

Failover and Switchover

Failover Configurations

Asymmetric

Failover Configurations

Symmetric

What is RAC? • Real Application Clusters (RAC) is a parallel database

environment that takes advantage of the processing power of multiple, interconnected computers.

• A cluster comprises two or more computers, also known as nodes or servers.

• In RAC environments, all nodes concurrently run Oracle instances and execute transactions against the same database.

• RAC coordinates each node’s access to the shared data to provide consistency and integrity.

• Each node adds its processing power to the cluster as a whole and can increase overall throughput or performance.

RAC Architecture• From a high level, RAC is multiple Oracle instances,

accessing a single Oracle database and carrying out simultaneous transactions.

• An Oracle database is the physical data stored in tablespaces on disk.

• An Oracle instance represents the software processes necessary to access and manipulate the database.

• In traditional environments, only one instance accesses a database at a specific time.

• With Oracle RAC, multiple instances communicate to coordinate access to a physical database, greatly enhancing scalability and availability.

Architectu

re

• The various Oracle processes making up an instance (such as DB Writer, Log Writer, Checkpoint, Archiver, and others) read and write data to storage via the I/O stack shown in the diagram.

• Oracle communicates via the Oracle Disk Manager (ODM) interface to the VERITAS Cluster File System (CFS), which in turn accesses the storage via the VERITAS Cluster Volume Manager (CVM).

Data Stack Overview

Data Stack Overview

• RAC instances must communicate to coordinate protection of data blocks in the database.

• ODM processes must communicate to coordinate data file protection and access across the cluster.

• CFS coordinates metadata updates for file systems, and finally CVM must coordinate the status of logical volumes and distribution of volume metadata across the cluster.

• VERITAS Cluster Server (VCS) controls starting and stopping of components in the Storage Foundation for Oracle RAC stack and provides monitoring and notification on failure.

Communications Stack Overview

Communications Stack Overview

• GAB’s second function is to provide reliable cluster communications, used by many Storage Foundation for Oracle RAC modules.

• GAB provides guaranteed delivery of point to point messages and broadcast messages to all nodes.

• Point to point messaging uses a send and acknowledgment. Atomic Broadcast ensures all systems within the cluster receive all messages.

• If a failure occurs while transmitting a broadcast message, GAB’s atomicity ensures that, upon recovery, all systems have the same information.

Cluster Communications

Cluster Communications

• Cluster Volume Manager is an extension of VERITAS Volume Manager (VxVM), the industry standard storage virtualization platform.

• CVM extends the concepts of VxVM across multiple nodes. • Each node sees the same logical volume layout, and more

importantly, the same state of all volume resources. • In a Storage Foundation for Oracle RAC cluster, all storage is

managed with standard VxVM commands from one node in the cluster.

• All other nodes immediately recognize any changes in disk group and volume configuration.

• CVM supports all performance enhancing capabilities, such as striping, mirroring, and mirror break-off (snapshot) for off-host backup.

Cluster Volume Manager (CVM)

• Cluster Volume Manager is designed with a master/slave architecture.

• One node in the cluster acts as the configuration master for logical volume management, and all others are slaves.

• Any node can take over as master if the existing master fails.

• The CVM master is established on a per-cluster basis.

CVM Architecture

• The Oracle Disk Manager (ODM) is a standard API specified by Oracle for doing database I/O.

• For example, when Oracle wishes to write, it calls the odm_iofunction.

• ODM improves both performance and manageability of the file system.

• The VERITAS-provided implementation of ODM improves performance by providing direct access for the database to the underlying storage without passing through the actual file system interface.

• This means the database sees performance that is equivalent to using raw devices.

• The administrator sees the storage as easy-to-manage file systems, including the support of resizing data files while in use.

Oracle Disk Manager

• VCS communicates the status of resources running on each system to all systems in the cluster.

• The High Availability Daemon, or “HAD,” is the main VCS daemon running on each system.

• HAD collects all information about resources running on the local system, forwarding it to all other systems in the cluster.

• It also receives information from all other cluster members to update its own view of the cluster.

VCS Architecture

• Each type of resource supported in a cluster is associated with an agent.

• An agent is an installed program designed to control a particular resource type.

• Each system runs necessary agents to monitor and manage resources configured to run on that node.

• The agents communicate with HAD on the node. • HAD distributes its view of resources on the local

node to other nodes in the cluster using GAB and LLT.

VCS Architecture

VCS Architecture

• Storage Foundation for Oracle RAC uses parallel service groups to support RAC.

• There is one CVM group per server. • This group has the CVM resource and the

necessary resources for support of CFS. • Users can modify this group to contain all

common components needed by Oracle to support RAC.

• This includes a shared ORACLE_HOME directory and the Oracle Net Services process (LISTENER).

Storage Foundation for Oracle RAC Service Groups

Service Groups

• I/O fencing is a feature within a kernel module of Storage Foundation for Oracle RAC designed to guarantee data integrity, even in the case of faulty cluster communications causing a split brain condition.

I/O Fencing

• Split brain is an issue faced by all cluster solutions.

• To provide high availability, the cluster must be capable of taking corrective action when a node fails.

• In Storage Foundation for Oracle RAC, this is carried out by the reconfiguration of CVM, CFS, and RAC to change membership.

• Problems arise when the mechanism used to detect the failure of a node breaks down.

Understanding Split Brain and the Need for I/O Fencing

Split Brain• The symptoms look identical to a failed node. For

example, if a system in a two-node cluster were to fail, it would stop sending heartbeats over the private interconnects and the remaining node would take corrective action.

• However, the failure of the private interconnects would present identical symptoms. In this case, both nodes would determine that their peer has departed and attempt to take corrective action.

• This typically results in data corruption when both nodes attempt to take control of data storage in an uncoordinated manner.

• Storage Foundation for Oracle RAC uses a relatively new enhancement to the SCSI specification, known as SCSI-3 Persistent Reservations, (SCSI-3 PR).

• SCSI-3 PR is designed to resolve the issues of using SCSI reservations in a modern clustered SAN environment.

• SCSI-3 PR supports multiple nodes accessing a device while at the same time blocking access to other nodes.

• SCSI-3 PR reservations are persistent across SCSI bus resets and SCSI-3 PR also supports multiple paths from a host to a disk

SCSI-3 Persistent Reservations

• Coordinator disks are special purpose disks in a Storage Foundation for Oracle RAC environment. Coordinator disks are three (or an odd number greater than three) standard disks, or LUNs, set aside for use by I/O fencing during cluster reconfiguration.

Coordinator Disks

Check Cluster Status[root@tali ~]# hastatus -sum-- SYSTEM STATE-- System State Frozen

A gali RUNNING 0A mali RUNNING 0A tali RUNNING 0

-- GROUP STATE-- Group System Probed AutoDisabled

State

B cvm gali Y N ONLINE B cvm mali Y N ONLINE B cvm tali Y N ONLINE

Check Cluster Interconnect

• root@tali ~]# /sbin/lltstat -nvv|more

LLT node information:Node State Link Status Address

* 0 tali OPEN eth0 UP 00:04:23:B2:0F:EEeth2 UP 00:09:3D:11:A3:84

1 mali OPEN eth0 UP 00:09:3D:11:AA:88eth2 UP 00:09:3D:11:AA:88

2 gali OPEN eth0 UP 00:04:23:AE:E3:14eth2 UP 00:04:23:AE:E3:14

Verifying GAB Port Membership

• galaxy# /sbin/gabconfig -a

GAB Port Memberships=============================================Port a gen 4a1c0001 membership 01Port b gen g8ty0002 membership 01Port d gen 40100001 membership 01Port f gen f1990002 membership 01Port h gen d8850002 membership 01Port o gen f1100002 membership 01Port v gen 1fc60002 membership 01Port w gen 15ba0002 membership 01

GAB• Port Function

a GABb I/O fencingd ODM (Oracle Disk Manager)f CFS (Cluster File System)h VCS (VERITAS Cluster Server: high availability daemon)o VCSMM driverv CVM (Cluster Volume Manager)w vxconfigd (module for CVM)

To verify the I/O Fencing configuration

• # vxfenadm -d

The output of this command should look similar to:I/O Fencing Cluster Information:================================Cluster Members:

* 0 (galaxy)1 (nebula)

RFSM State Information:node 0 in state 8 (running)node 1 in state 8 (running)

To determine whether a node is the CVM master or a CVM slave

• On one node, determine whether it is the master or the slave node by using the command:

• # vxdctl -c modeExpected output:cluster active - MASTERor:cluster active - SLAVE

To enable write access to the volumes in the disk group

• You must enable write access to the shared volumes so you can create databases on them.

• You can use the following commands on the CVM master node:

# vxdg -s import shared_disk_group# vxvol -g shared_disk_group startall# vxdg -g shared_disk_group set activation=sw

Mount CFS

• On all nodes, mount the file system:

mount –t vxfs –o cluster /dev/vx/dsk/ora_dg/od01 /dwh/od01

How I/O Fencing Works in Different Event Scenarios