VRITAS_RAC

VERITAS Database EditionTM /Advanced Cluster for Oracle9i RAC

Installation and Configuration Guide

Solaris

July 2002N08843F

Disclaimer

The information contained in this publication is subject to change without notice.VERITAS Software Corporation makes no warranty of any kind with regard to thismanual, including, but not limited to, the implied warranties of merchantability andfitness for a particular purpose. VERITAS Software Corporation shall not be liable forerrors contained herein or for incidental or consequential damages in connection with thefurnishing, performance, or use of this manual.

Copyright

Copyright © 2002 VERITAS Software Corporation. All rights reserved. VERITAS,VERITAS Software, the VERITAS logo, and all other VERITAS product names and slogansare trademarks of VERITAS Software Corporation in the USA and/or other countries.Other product names and/or slogans mentioned herein may be trademarks or registeredtrademarks or their respective companies.

VERITAS Software Corporation350 Ellis St.Mountain View, CA 94043Phone 650–527–8000Fax 650–527–2908www.veritas.com

www.veritas.com

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

How This Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

VERITAS Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xii

VERITAS Volume Manager Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xii

VERITAS File System Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xii

VERITAS Cluster Server Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

VERITAS Cluster Server Oracle Enterprise Agent . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Man Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Oracle Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

Obtaining License Keys for DBE/AC for Oracle9i RAC . . . . . . . . . . . . . . . . . . . . . . . . . xiv

Using the VERITAS vLicenseTM Web Site to Obtain a License Key . . . . . . . . . . xiv

Faxing the License Key Request Form to Obtain a License Key . . . . . . . . . . . . xiv

Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xv

Technical Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xv

Chapter 1. Overview: DBE/AC for Oracle9i RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

What is RAC? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

RAC Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Oracle Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Operating System Dependant Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Cluster Manager (CM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Cluster Inter-Process Communication (IPC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Lock Management and Cache Fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Shared Disk Subsystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

DBE/AC Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Data Stack Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Communications Stack Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

DBE/AC Communications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Low Latency Transport (LLT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

IPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

iii

Traffic Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Heartbeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Group Membership Services/Atomic Broadcast (GAB) . . . . . . . . . . . . . . . . . . . . . . . 6

Cluster Membership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Cluster Communications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Cluster Volume Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

CVM Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Cluster File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

CFS Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

CFS Usage in DBE/AC for RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Oracle Disk Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

ODM Clustering Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

VERITAS Cluster Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

VCS Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

DBE/AC Service Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

DBE/AC OSD Layer Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Cluster Membership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Inter-Process Communications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

I/O Fencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Understanding Split Brain and the Need for I/O Fencing . . . . . . . . . . . . . . . . . . . . 13

SCSI-3 Persistent Reservations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

I/O Fencing Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Data Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Coordinator Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

I/O Fencing Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Chapter 2. Installing the Component Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Installation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

After Installation View - A Cluster Running Oracl9i RAC . . . . . . . . . . . . . . . . . . . . 17

Phases of the Installation: An Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

iv VERITAS Database Edition/AC for Oracle9i RAC Installation and Configuration Guide

Phase One: Setting up and Configuring the Hardware . . . . . . . . . . . . . . . . . . . . 18

Phase Two: Installing DBE/AC and Configuring its Components . . . . . . . . . . 19

Phase Three: Installing Oracle9i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Phase Four: Creating the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Phase Five: Setting up VCS to Manage RAC Resources . . . . . . . . . . . . . . . . . . . . 20

Installation and Configuration Flowchart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Prerequisites for Installing Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Preparing for Using the installDBAC Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

General Information Requested by Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Web-based ClusterManager Information Requested by Script . . . . . . . . . . . . . . 23

SMTP/SNMP (Optional) Requested by Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Installing DBE/AC Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Setting the PATH Variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Setting the MANPATH Variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Setting Up System-to-System Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Mounting the VERITAS DBE/AC for Oracle9i RAC CD . . . . . . . . . . . . . . . . . . . . . . . 25

Running the VERITAS DBE/AC Install Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Verifying Systems for Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Configuring the Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

Configuring Cluster Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Configuring SMTP Email Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Configuring SNMP Trap Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Installing VCS Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Configuring VCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Starting VCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Installing Base CVM and CFS Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Configuring CVM and CFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Completing Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Verifying All Prerequisite Packages Are Installed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Configuring VxVM Using vxinstall and Rebooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

v

Verifying Storage Supports SCSI-3 Persistent Reservations . . . . . . . . . . . . . . . . . . . . . . 40

For EMC Symmetrix 8000 Series Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

For Hitachi Data Systems 99xx Series Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Verifying Shared Storage Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Using the vxfentsthdw Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Setting Up Coordinator Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Requirements for Coordinator Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Configuring a Disk Group Containing Coordinator Disks . . . . . . . . . . . . . . . . . . . . 44

Starting I/O Fencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

The Contents of the /etc/vxfentab File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Replacing Failed Coordinator Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Running gabconfig -a After Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Example VCS Configuration File After Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Example main.cf file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Chapter 3. Installing Oracle9i Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Oracle9i Installation Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Prerequisites for Installing Oracle9i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Creating Shared Disk Groups and Volumes - General . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Determining Information About a Disk Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Setting the Connectivity Policy on a Shared Disk Group . . . . . . . . . . . . . . . . . . . . . 52

Enabling Write Access to the Volumes in the Disk Group . . . . . . . . . . . . . . . . . . . . 52

Creating Shared Disk Group and Volume for SRVM . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Installing Oracle9i Release 1 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Installing Oracle9i Release 1 on Shared Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Alternate Method - Installing Oracle9i Release 1 Locally . . . . . . . . . . . . . . . . . . . . . 58

Installing Oracle9i Release 1 Patch Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

Installing Oracle9i Release 2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Installing Oracle9i Release 2 on Shared Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Alternate Method - Installing Oracle9i Release 2 Locally . . . . . . . . . . . . . . . . . . . . . 66

vi VERITAS Database Edition/AC for Oracle9i RAC Installation and Configuration Guide

Removing Temporary rsh Access Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Linking VERITAS ODM Libraries to Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Creating Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Chapter 4. Configuring VCS Service Groups for Oracle . . . . . . . . . . . . . . . . . . . . . .71

Checking cluster_database Flag in Oracle9i Parameter File . . . . . . . . . . . . . . . . . . . . . . 71

Configuring CVM and Oracle Groups in main.cf File . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

VCS Service Groups for RAC: Dependencies Chart . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Setting Attributes as Local for the CVM and Oracle Groups . . . . . . . . . . . . . . . . . . 73

Modifying the CVM Service Group in the main.cf File . . . . . . . . . . . . . . . . . . . . . . . 73

Adding Sqlnet, NIC, IP CVMVolDg, and CFSMount Resources . . . . . . . . . . . . 73

Adding Oracle Resource Groups in the main.cf File . . . . . . . . . . . . . . . . . . . . . . . . . 76

Additional RAC Processes Monitored by the VCS Oracle Agent . . . . . . . . . . . . 77

Attributes of CVM and Oracle Groups that Must be Defined as Local . . . . . . . . . . 78

Modifying the VCS Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Location of VCS and Oracle Agent Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Chapter 5. I/O Fencing Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81

I/O Fencing of Shared Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

The Role of Coordinator Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

How I/O Fencing Works in Different Event Scenarios . . . . . . . . . . . . . . . . . . . . . . . 82

The vxfenadm Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Registration Key Formatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

Chapter 6. Uninstalling DBE/AC for Oracle9i RAC . . . . . . . . . . . . . . . . . . . . . . . . . . .87

Offlining the Oracle and Sqlnet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Removing the Oracle Database (optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Uninstalling Oracle9i (optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Offlining the Oracle Service Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Stopping VCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Unmount Other VxFS File Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

vii

Unconfiguring the CFS Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Unconfiguring the I/O Fencing Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Removing Optional CFS and CVM Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Removing Required CFS and CVM Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Remove VxVM Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Uninstalling VCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

Removing License Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Remove Other Configuration Files (Optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

Chapter 7. Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Running Scripts for Engineering Support Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

getdbac . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

getcomms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

hagetcf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

Troubleshooting Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Error When Starting an Oracle Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Missing Dialog Box During Installation of Oracle9i Release 1 . . . . . . . . . . . . . . . . . 98

Missing Dialog Box During Installation of Oracle9i Release 2 . . . . . . . . . . . . . . . . . 99

Instance Numbers Must be Unique (Error Code 205) . . . . . . . . . . . . . . . . . . . . . . . . 99

ORACLE_SID Must be Unique (Error Code 304) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

Oracle Log Files Show Shutdown Called Even When Not Shutdown Manually 100

File System Configured Incorrectly for ODM Shuts Down Oracle . . . . . . . . . . . . 100

VCSIPC Errors in Oracle Trace/Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Shared Disk Group Cannot be Imported . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

vxfentsthdw Fails When SCSI TEST UNIT READY Command Fails . . . . . . . . . . 102

CVMVolDg Does Not Go Online Even Though CVMCluster is Online . . . . . . . . 102

Restoring Communication Between Host and Disks After Cable Disconnection 102

Adding a System to an Existing Two-Node Cluster Causes Error . . . . . . . . . . . . 103

Removing Existing Keys From Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

System Panics to Prevent Potential Data Corruption . . . . . . . . . . . . . . . . . . . . . . . 105

viii VERITAS Database Edition/AC for Oracle9i RAC Installation and Configuration Guide

How vxfen Driver Checks for Pre-existing Split Brain Condition . . . . . . . . . . 105

Case 1: System 2 Up, System 1 Ejected (Actual Potential Split Brain) . . . . . . . 106

Case 2: System 2 Down, System 1 Ejected (Apparent Potential Split Brain) . . 106

Using vxfenclearpre Command to Clear Keys After Split Brain . . . . . . . . . . . . . . . 107

Appendix A. Sample main.cf File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .109

/etc/VRTSvcs/conf/sample_rac/main.cf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Appendix B. CVMCluster, CVMVolDg, and CFSMount Agents . . . . . . . . . . . . . . . .115

CVMCluster Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

CVMCluster Agent Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

CVMCluster Agent Type Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

CVMCluster Agent Sample Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Configuring the CVMVolDg and CFSMount Resources . . . . . . . . . . . . . . . . . . . . . . . . 118

CVMVolDg Agent, Entry Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

CVMVolDg Agent Type, Attribute Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

CVMVolDg Agent Type Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

Sample CVMVolDg Agent Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

CFSMount Agent, Entry Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

CFSMount Agent Type, Attribute Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

CFSMount Agent Type Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Sample CFSMount Agent Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

Appendix C. Tunable Kernel Driver Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . .123

LMX Tunable Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

Example: Configuring LMX Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

vxfen Tunable Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Appendix D. Error Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127

LMX Error Messages, Critical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

LMX Error Messages, Non-Critical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

ix

VxVM Errors Related to I/O Fencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

VXFEN Driver Error Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

VXFEN Driver Informational Message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Informational Messages When Node is Ejected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

Appendix E. Creating Starter Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Using Oracle dbca to Create a Starter Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

Preparing Tablespaces for Use With dbca . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Tablespace Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Creating Shared Raw Volumes for Tablespaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

Creating the Oracle Starter Database Using Oracle9i dbca . . . . . . . . . . . . . . . . . . . 136

Using VERITAS Scripts to Create a Starter Database . . . . . . . . . . . . . . . . . . . . . . . . . . 137

Using VERITAS Scripts to Create Database on Raw Volumes . . . . . . . . . . . . . . . . 137

Using VERITAS Scripts to Create Database on Cluster File System . . . . . . . . . . . 139

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

x VERITAS Database Edition/AC for Oracle9i RAC Installation and Configuration Guide

Preface

The VERITAS Database Edition/Advanced ClusterTM for Oracle9i RAC software is anintegrated set of software products. It enables administrators of Oracle Real ApplicationCluster (RAC) to operate a database in an environment of cluster systems runningVERITAS Cluster ServerTM (VCS) and the cluster features of VERITAS Volume ManagerTM

and VERITAS File SystemTM, also known as CVM and CFS, respectively.

The VERITAS Database Edition/AC for Oracle9i RAC Installation and Configuration Guide isintended for system administrators responsible for configuring and maintaining OracleReal Application Cluster running on a VCS cluster with VxVM disk management.

This guide assumes that the administrator has a:

◆ Basic understanding of system and database administration

◆ Working knowledge of the Solaris operating system

◆ Working knowledge of Oracle databases

How This Guide is OrganizedChapter 1 “Overview: DBE/AC for Oracle9i RAC” describes the components of OracleRAC and the DBE/AC components in support of Oracle Real Application Cluster.

Chapter 2 “Installing the Component Packages” describes the sequence of steps to installthe required software. The “Installation and Configuration Flowchart” on page 20provides a visual summary of the installation activities.

Chapter 3 “Installing Oracle9i Software” describes how to set up shared storage, installOracle9i software. Installation is typically on shared storage, but procedures for installingon local systems is also provided.

Chapter 4 “Configuring VCS Service Groups for Oracle” describes configuring VCSservice groups for Oracle, the database, and the listener process in parallel mode. Itdescribes configuring the agents within the services group.

Chapter 5 “I/O Fencing Scenarios” describes I/O fencing.

Chapter 6 “Uninstalling DBE/AC for Oracle9i RAC” describes how to uninstall andremove the components.

xi

VERITAS Documentation

Chapter 7 “Troubleshooting” discusses typical problems and provides recommendationsfor solving them. Also, you can use the scripts described in this chapter to generateinformation about your systems that VERITAS support personnel can use to assist you.

Appendix A “Sample main.cf File” provides a sample VCS configuration file main.cf,which includes an example configuration of the Oracle and CVM service groups.

Appendix B “CVMCluster, CVMVolDg, and CFSMount Agents” provides details for theCVMCluster agent configured when you run the installDBAC script. It also describesthe CVMVolDg and CFSMount agents that must be configured after installation.

Appendix C “Tunable Kernel Driver Parameters” describes tunable parameters for theLMX and VXFEN drivers.

Appendix D “Error Messages” lists LMX and VXFEN messages that you may encounter.

Appendix E “Creating Starter Databases” contains procedures for creating starterdatabases using dbac or VERITAS scripts on raw shared volumes or in VxFS file systems.

VERITAS DocumentationThe following documents provide important information on the software componentsthat support CVM, VxFS, and VCS. They are provided as PDF files on the VERITASStorage Solutions 3.5 CDs.

VERITAS Volume Manager DocumentationAfter installation, find the PDF versions of these documents in /opt/VRTSvxvm/docs.

VERITAS Volume Manager 3.5 Installation Guide, Solaris

VERITAS Volume Manager 3.5 Storage Administrator Administrator’s Guide

VERITAS Volume Manager Administrator’s Guide

VERITAS Volume Manager 3.5 Troubleshooting Guide

VERITAS Volume Manager 3.5 Hardware Notes, Solaris

VERITAS File System DocumentationAfter installation, find the PDF versions of these documents in /opt/VRTSfsdoc.

VERITAS File System 3.5 Installation Guide

VERITAS File System 3.5 System Administrator’s Guide

xii VERITAS Database Edition/AC for Oracle9i RAC Installation and Configuration Guide

Oracle Documentation

VERITAS Cluster Server DocumentationAfter installation, find the PDF versions of these documents in /opt/VRTSvcsdc.

VERITAS Cluster Server 3.5 Installation Guide

VERITAS Cluster Server 3.5 User’s Guide

VERITAS Cluster Server 3.5 Bundled Agents Reference Guide

VERITAS Cluster Server 3.5 Agent Developer’s Guide

VERITAS Cluster Server Oracle Enterprise AgentOn theVERITAS Storage Solutions 3.5 CD containing enterprise agents, find a PDF versionof this document:

VERITAS Cluster Server Enterprise Agent for Oracle Installation and Configuration Guide

Man PagesSee “Setting the MANPATH Variable” on page 24 to enable access to manual pages.

Oracle DocumentationThe following documents, not shipped with VERITAS Database Edition/AC for Oracle9iRAC, provide necessary related information:

◆ Oracle9i Installation Guide

◆ Oracle9i Parallel Server Setup and Configuration Guide

◆ Notes for the latest Oracle9i Data Server patch set

Preface xiii

Obtaining License Keys for DBE/AC for Oracle9i RAC

Obtaining License Keys for DBE/AC for Oracle9i RACDBE/AC for Oracle RAC is a licensed software product. The installDBAC programprompts you for a license key for each system. You cannot use your VERITAS softwareproduct until you have completed the licensing process. Use either method described inthe following two sections to obtain a valid license key.

Using the VERITAS vLicenseTM Web Site to Obtain a License Key

You can obtain your license key most efficiently using the VERITAS vLicense web site.The License Key Request Form (LKRF) has all the information needed to establish a UserAccount on vLicense and generate your license key. The LKRF is a one-page insertincluded with the CD in your product package. You must have this form to obtain asoftware license key for your VERITAS product.

Note Do not discard the License Key Request Form. If you have lost or do not have the formfor any reason, email [email protected].

The License Key Request Form contains information unique to your VERITAS softwarepurchase. To obtain your software license key, you need the following information shownon the form:

◆ Your VERITAS customer number

◆ Your order number

◆ Your serial number

Follow the appropriate instructions on the vLicense web site to obtain your license keydepending on whether you are a new or previous user of vLicense:

1. Access the web site at http://vlicense.veritas.com.

2. Log in or create a new login, as necessary.

3. Follow the instructions on the pages as they are displayed.

When you receive the generated license key, you can proceed with installation.

Faxing the License Key Request Form to Obtain a License Key

If you do not have Internet access, you can fax the LKRF to VERITAS. Be advised thatfaxing the form generally requires several business days to process in order to provide alicense key.

Before faxing, sign and date the form in the appropriate spaces. Fax it to the numbershown on the form.

xiv VERITAS Database Edition/AC for Oracle9i RAC Installation and Configuration Guide

http://vlicense.veritas.com

Conventions

Conventions

Technical SupportFor assistance with any VERITAS product, contact Technical Support:

U.S. and Canada: call 1-800-342-0652.

Europe, the Middle East, or Asia: visit the Technical Support Web site athttp://support.veritas.com for a list of each country’s contact information.

Software updates, Tech Notes, product alerts, and hardware compatibility lists, are alsoavailable from http://support.veritas.com.

To learn more about VERITAS and its products, visit http://support.veritas.com.

Typeface Usage

courier computer output, files, attribute names, device names, and directories

courier (bold) user input and commands, keywords in grammar syntax

italic new terms, titles, emphasis

italic variables

Symbol Usage

% C shell prompt

$ Bourne/Korn shell prompt

# root user prompt (for all shells)

Preface xv

http://support.veritas.com

http://support.veritas.com

Technical Support

xvi VERITAS Database Edition/AC for Oracle9i RAC Installation and Configuration Guide

Overview: DBE/AC for Oracle9i RAC
1 What is RAC?
Real Application Clusters (RAC) is a parallel database environment that takes advantageof the processing power of multiple, interconnected computers. A cluster comprises twoor more computers, also known as nodes or servers. In RAC environments, all nodesconcurrently run Oracle instances and execute transactions against the same database.RAC coordinates each node’s access to the shared data to provide consistency andintegrity. Each node adds its processing power to the cluster as a whole and can increaseoverall throughput or performance.

RAC serves as an important component of a robust high availability solution. A properlyconfigured Real Application Clusters environment can tolerate failures with minimaldowntime and interruption to users. With clients accessing the same database on multiplenodes, failure of a node does not completely interrupt access, as clients accessing thesurviving nodes continue to operate. Clients attached to the failed node simply reconnectto a surviving node and resume access. Recovery after failure in a RAC environment is farquicker than a failover database, because another instance is already up and running.Recovery is simply a matter of applying outstanding redo log entries from the failed node.

RAC ArchitectureAt the highest level, RAC is multiple Oracle instances, accessing a single Oracle databaseand carrying out simultaneous transactions. An Oracle database is the physical datastored in tablespaces on disk. An Oracle instance represents the software processesnecessary to access and manipulate the database. In traditional environments, only oneinstance accesses a database at a specific time. With Oracle RAC, multiple instancescommunicate to coordinate access to a physical database, greatly enhancing scalabilityand availability.

1

What is RAC?

Oracle Instance

The Oracle instance consists of a set of processes and shared memory space that provideaccess to the physical database. The instance consists of server processes acting on behalfof clients to read data into shared memory as well as make modifications of it, andincludes background processes to write changed data out to disk.

In an Oracle9i RAC environment, multiple instances access the same database. Thisrequires significant coordination between the instances to keep each instance’s view of thedata consistent.

Database

Datafiles &Control(files)

Index and Temp(files)

Online RedoLogs (files)

Archive Logs(files)

ListenerListener

9i RACInstance

ODM

9i RACInstance

ODM

CFSCFS

CVMCVM

VC

S

VC

S

High Speed Interconnect

OCI Client

2 VERITAS Database Edition/AC for Oracle9i RAC Installation and Configuration Guide

What is RAC?

Operating System Dependant LayerOracle9i RAC relies on several support services provided by the VCS. The importantfeatures are cluster membership carried out by the cluster manager and inter-nodecommunication. The actual implementation of these functions is described later in thischapter.

Cluster Manager (CM)

The Cluster Manager provides a global view of the cluster and all nodes in it. CM isresponsible for determining cluster membership and enforcing protection of data bypreventing non-cluster nodes from corrupting stored data.

Cluster Inter-Process Communication (IPC)

RAC relies heavily on an underlying high-speed interprocess communication (IPC)mechanism. IPC defines the protocols and interfaces required for the RAC environment totransfer messages between instances.

Lock Management and Cache Fusion

Lock management coordinates access by multiple instances to the same data to maintaindata consistency and integrity. Cache Fusion provides memory-to-memory transfers ofdatablocks between RAC instances. Cache Fusion transfers are faster than transfers madeby writing to and then reading from disk.

Shared Disk Subsystems

RAC also requires that all nodes have simultaneous access to database storage. This givesmultiple instances concurrent access to the same database.

Chapter 1, Overview: DBE/AC for Oracle9i RAC 3

DBE/AC Overview

DBE/AC OverviewDatabase Edition/Advanced Cluster provides a complete I/O and communications stackto support Oracle9i RAC. It also provides monitoring and management of instancestartup and shutdown. The following section describes the overall data andcommunications flow of the DBE/AC stack.

Data Stack Overview

The diagram above details the overall data flow from an instance running on a server tothe shared storage. The various Oracle processes making up an instance (such as DBWriter, Log Writer, Checkpoint, Archiver, and Server) read and write data to storage viathe I/O stack shown in the diagram. Oracle communicates via the Oracle Disk Manager(ODM) interface to the VERITAS Cluster File System (CFS), which in turn accesses thestorage via the VERITAS Cluster Volume Manager (CVM). Each of these components inthe data stack is described in this chapter.

Oracle9iRACInstance

Oracle9iRACInstance

LGWRARCHCKPTDBWRServer

LGWRARCHCKPTDBWRServer

ODM ODM

CFSCFS

CVM CVM

RedoLogFiles Archive

Storage

DataandControlFiles

Disk I/O Disk I/O


DBE/AC Overview

Communications Stack Overview

The diagram above shows the data stack as well as the communications stack. Each of thecomponents in the data stack requires communications with its peer on other systems toproperly function. RAC instances must communicate to coordinate protection of datablocks in the database. ODM processes must communicate to coordinate data fileprotection and access across the cluster. CFS coordinates metadata updates for filesystems, and finally CVM must coordinate the status of logical volumes and distributionof volume metadata across the cluster. VERITAS Cluster Server (VCS) controls startingand stopping of components in the DBE/AC stack and provides monitoring andnotification on failure. VCS must communicate status of its resources on each node in thecluster. For the entire system to work, each layer must reliably communicate.

The diagram also shows Low Latency Transport (LLT) and the Group MembershipServices/Atomic Broadcast (GAB), which make up the communications package centralto the operation of DBE/AC. During an operational steady state, the only significanttraffic through LLT and GAB is due to Lock Management and Cache Fusion, while thetraffic for the other data is relatively sparse.

RAC

ODM

CFS

CVM

VC

S C

ore

GA

BL

LT

Cluster State

Cache Fusion/Lock Mgmt

Datafile Management

Filesystem MetaData

Volume Management

RAC

ODM

CFS

CVM

VC

S C

ore

GA

BL

LT


DBE/AC Communications

DBE/AC CommunicationsDBE/AC communications consist of LLT and GAB

Low Latency Transport (LLT)LLT provides fast, kernel-to-kernel communications, and monitors network connections.LLT functions as a high performance replacement for the IP stack. LLT runs directly ontop of the Data Link Protocol Interface (DLPI) layer. The use of LLT rather than IPremoves latency and overhead associated with the IP stack. LLT has several majorfunctions.

IPC

RAC Inter-Process Communications (IPC) uses the VCSIPC shared library for interprocesscommunication. In turn, VCSIPC uses LMX, an LLT multiplexer, to provide fast datatransfer between Oracle processes on different cluster nodes. IPC leverages all of LLT’sfeatures.

Traffic Distribution

LLT distributes (load balances) inter-node communication across all available privatenetwork links. All cluster communications are evenly distributed across as many as eightnetwork links for performance and fault resilience. On failure of a link, traffic is redirectedto remaining links.

Heartbeat

LLT is responsible for sending and receiving heartbeat traffic over network links. Thisheartbeat is used by the Group Membership Services function of GAB to determinecluster membership.

Group Membership Services/Atomic Broadcast (GAB)The Group Membership Services /Atomic Broadcast protocol (GAB), is responsible forCluster Membership and Cluster Communications as described below.


DBE/AC Communications

Cluster Membership

A distributed system such as DBE/AC requires all nodes to be aware of which nodes arecurrently participating in the cluster. Nodes can leave or join the cluster because ofshutting down, starting up, rebooting, powering off, or faulting.

Cluster membership is determined using LLT heartbeats. When systems no longer receiveheartbeats from a peer for a predetermined interval, then a protocol is initiated to excludethe peer from the current membership. When systems start receiving heartbeats from apeer that is not part of the membership, then a protocol is initiated for enabling it to jointhe current membership.

The new membership is consistently delivered to all nodes and actions specific to eachmodule are initiated. For example, following a node fault, the Cluster Volume Managermust initiate volume recovery, and Cluster File System must do a fast parallel file systemcheck.

Cluster Communications

GAB’s second function is to provide reliable cluster communications, used by manyDBE/AC modules. GAB provides guaranteed delivery of point to point messages andbroadcast messages to all nodes. Point to point uses a send and acknowledgment. AtomicBroadcast ensures all systems within the cluster receive all messages. If a failure occurswhile transmitting a broadcast message, GAB’s atomicity ensures that, upon recovery, allsystems have the same information.

GAB Messaging- Cluster Membership/State S

erver

Ser

ver

NIC

NIC

NIC

NIC

- Datafile Management- Filesystem Metadata- Volume Management


Cluster Volume Manager

Cluster Volume ManagerCluster Volume Manager is an extension of VERITAS Volume Manager (VxVM), theindustry standard storage virtualization platform. CVM extends the concepts of VxVMacross multiple nodes. Each node sees the same logical volume layout, and moreimportantly, the same state of all volume resources.

In a DBE/AC cluster, all storage is managed with standard VxVM commands from onenode in the cluster. All other nodes immediately recognize any changes in disk group andvolume configuration with no interaction. CVM supports all performance enhancingcapabilities, such as striping, mirroring, and mirror break-off (snapshot) for off hostbackup.

CVM ArchitectureCluster Volume manager is designed with a master/slave architecture. One node in thecluster acts as the configuration master for logical volume management, and all others areslaves. Any node can take over as master if the existing master fails. The CVM master isestablished on a per cluster basis.

Since CVM is an extension of VxVM, it operates in a very similar fashion. The volumemanager configuration daemon, vxconfigd, maintains the configuration of logicalvolumes. Any changes to volumes are handled by vxconfigd, which updates theoperating system at the kernel level when the new volume state is determined. Forexample, if a mirror of a volume fails, it is detached from the volume and the error ispassed off to vxconfigd, which then determines the proper course of action, updates thenew volume layout, and informs the kernel of a new volume layout. CVM extends thisbehavior across multiple nodes in a master-slave architecture. Changes to a volume arepropagated to the master vxconfigd. The vxconfigd process on the master pushes thesechanges out to slave vxconfigd processes, each of which in turn updates the local kernel.

CVM does not impose any write locking between nodes. Each node is free to update anyarea of the storage. All data integrity is the responsibility of the upper application. Froman application perspective, logical volumes are accessed identically on a stand-alonesystem as on a CVM system.

CVM imposes a “Uniform Shared Storage” model. All systems must be connected to thesame disk sets for a given disk group. Any system unable to see the entire set of physicaldisks as other nodes for a given disk group cannot import the disk group. If a node losescontact with specific disk, it is excluded from participating in the use of that disk.

CVM uses GAB and LLT for transport of all its configuration data.


Cluster File System

Cluster File SystemThe VERITAS Cluster File System is an extension of the industry standard VERITAS FileSystem (VxFS). CFS allows the same file system to be simultaneously mounted onmultiple nodes. Unlike other clustered file systems, CFS is a true SAN file system. All I/Otakes place over the storage area network. Coordination between nodes is enabled bymessages sent across the cluster interconnects.

CFS ArchitectureCFS is designed with a master/slave architecture. Though any node can initiate anoperation to create, delete, or resize data, the actual operation is carried out by the masternode. Since CFS is an extension of VxFS, it operates in a very similar fashion. As withVxFS, it caches metadata and data in memory (typically called buffer cache or vnodecache). A distributed locking mechanism, called the Global Lock Manager (CGM) is usedfor metadata and cache coherency across the multiple nodes. GLM provides a way toensure all nodes have a consistent view of the file system. When any node wishes to readdata, it requests a shared lock. If another node wishes to write to the same area of the filesystem, it must request an exclusive lock. The GLM revokes all shared lock beforegranting the exclusive lock and informs the reading nodes that their data is no longervalid. If a node requests a shared lock for data that is exclusively held by another node, inaddition to revoking the exclusive lock, GLM sends the data over the cluster interconnectsto the requesting node.

CFS Usage in DBE/AC for RACCFS is used in DBE/AC to manage a file system in a large database environment. Whenused in DBE/AC for Oracle9i RAC, Oracle accesses data files stored on CFS file systemswith the ODM interface. This essentially bypasses the filesystem buffer and filesystemlocking for data. This means only Oracle buffers data and coordinates writing to files andnot the GLM, which is minimally used with the ODM interface. A single point of lockingand buffering ensures maximum performance.

Oracle Disk ManagerThe Oracle Data Manager (ODM) is a standard API created by Oracle for doing databaseI/O. For example, when Oracle wishes to write, it calls the odm_io function. ODMimproves both performance and manageability of the file system.

ODM improves performance by providing direct access for the database to the underlyingstorage without passing through the actual filesystem interface. This means the databasesees performance equivalent to using raw devices. The administrator sees the storage as


VERITAS Cluster Server

easy-to- manage file systems, including the support of resizing data files while in use.ODM improves manageability by allowing Oracle to directly invoke operations such asresizing data files while the database is online.

ODM Clustering ExtensionsAll Oracle Disk Manager features have been extended to operate in a cluster environment.Nodes communicate before performing any operation that could potentially affectanother node. For example, before creating a new data file with a specific name, ODMchecks with other nodes to see if the file name is already in use.

VERITAS Cluster ServerVCS in the DBE/AC environment performs as a director of operations. It controls startupand shutdown of the component layers of RAC. In the DBE/AC configuration, the RACservice groups run as parallel service groups. VCS does not attempt to migrate a failedservice group, but it can be configured to restart it on failure. VCS also notifies users ofany failures.

DBE/AC provides specific agents for VCS to operate in a DBE/AC environment,including CVM, CFS and Oracle.

VCS ArchitectureVCS communicates the status of resources running on each system to all systems in thecluster. The High Availability Daemon, or “HAD,” is the main VCS daemon running oneach system. HAD collects all information about resources running on the local system,forwarding it to all other systems in the cluster. It also receives information from all othercluster members to update its own view of the cluster.

Each type of resource supported in a cluster is associated with an agent. An agent is aninstalled program designed to control a particular resource type. Each system runsnecessary agents to monitor and manage resources configured to run on that node. Theagents communicate with HAD on the node. HAD distributes its view of resources on thelocal node to other nodes in the cluster using GAB and LLT.


VERITAS Cluster Server

DBE/AC Service GroupsDBE/AC uses parallel service groups to support RAC. There is one CVM/Infrastructuregroup per server. This group has the CVM resource and the necessary resources forsupport of CFS. This group also contains all common components needed by Oracle tosupport RAC. This includes a shared ORACLE_HOME directory and the Oracle NetServices process (LISTENER).

For each Oracle database, there is a service group . This service group includes thedatabase on shared storage and the supporting CVM and CFS resources

Sqlnet

IP

NIC

CFSMount

CFSfsckd

CFSQlogckd

CVMVolDg

CVMCluster

CVM Service GroupParallel Service Group

Oracle

CFSMount

CVMVolDg

CFSMountCFSMount

Oracle DatabaseService Group


DBE/AC OSD Layer Support

DBE/AC OSD Layer SupportDBE/AC OSD layer support includes the VCSMM and IPC components

Cluster MembershipOracle provides an API for providing membership information to RAC. This is known asskgxn (system kernel generic interface node membership). Oracle RAC expects to makespecific skgxn calls and get membership information required. In DBE/AC this isimplemented as a library linked to Oracle when 9i RAC is installed. The skgxn library inturn makes ioctl calls to a kernel module for membership information.

This module is known as VCSMM (VCS membership module). Oracle uses the linked inskgxn library to communicate with VCSMM, which obtains membership informationfrom GAB.

Inter-Process CommunicationsIn order to coordinate access to a single database by multiple instances, Oracle usesextensive communications between nodes and instances. Oracle uses Inter-ProcessCommunications (IPC) for locking traffic and Cache Fusion.

DBE/AC uses LLT to support IPC in a cluster and leverages its high performance andfault resilient capabilities.

Oracle has a defined API for Inter-Process Communication that isolates Oracle from theunderlying transport mechanism. Oracle communicates between processes on instances,and does not have to know how the data is moved between systems. The Oracle API forIPC is referred to as System Kernel Generic Interface Inter-Process Communications(skgxp).

DBE/AC provides a library linked to Oracle at install time to implement the skgxpfunctionality. This module communicates with the LLT Multiplexer (LMX) via ioctl calls.

The LMX module is a kernel module designed to receive communications from the skgxnmodule and pass to the correct process on the correct instance on other nodes. The module“multiplexes” communications between multiple related processes into a single multithreaded LLT port between systems. LMX leverages all features of LLT, including loadbalancing and fault resilience.

I/O FencingI/O fencing is a new feature to DBE/AC, designed to guarantee data integrity, even in thecase of faulty cluster communications causing a split brain condition.


I/O Fencing

Understanding Split Brain and the Need for I/O FencingSplit brain is an issue faced by all cluster solutions. To provide high availability, the clustermust be capable of taking corrective action when a node fails. In DBE/AC, this is carriedout by the reconfiguration of CVM, CFS and RAC to change membership. Problems arisewhen the mechanism used to detect the failure of a node breaks down. The symptomslook identical to a failed node. For example, if a system were to fail, it would stop sendingheartbeats over the private interconnects and the remaining nodes would take correctiveaction. However, the failure of the private interconnects would present identicalsymptoms. In this case, both nodes would determine that their peer has departed andattempt to take corrective action. This typically results in data corruption when bothnodes attempt to take control of data storage in an uncoordinated manner.

In addition to a broken set of private networks, other scenarios can cause this situation. Ifa system were so busy as to appear hung, it would be declared dead. This can also happenon systems where the hardware supports a “break” and “resume” function. Dropping thesystem to prom level with a break and subsequently resuming means the system could bedeclared as dead, the cluster could reform, and when the system returns, it could beginwriting again.

DBE/AC uses a technology called I/O fencing to remove the risk associated with splitbrain. I/O fencing blocks access to storage from specific nodes. This means even if thenode is alive, it cannot cause damage.

SCSI-3 Persistent ReservationsDBE/AC uses a relatively new enhancement to the SCSI specification, known as SCSI-3Persistent Reservations, (SCSI-3 PR). SCSI-3 PR is designed to resolve the issues of usingSCSI reservations in a modern clustered SAN environment. SCSI-3 PR supports multiplenodes accessing a device while at the same time blocking access to other nodes. SCSI-3 PRreservations are persistent across SCSI bus resets and SCSI-3 PR also supports multiplepaths from a host to a disk. By contrast, SCSI-2 reservations can only be used by one host,with one path. This means if there is a need to block access for data integrity concerns,only one host and one path remain active. The requirements for larger clusters, withmultiple nodes reading and writing to storage in a controlled manner have made SCSI-2reservations obsolete.

SCSI-3 PR uses a concept of registration and reservation. Systems wishing to participateregister a “key” with a SCSI-3 device. Each system registers its own key. Multiple systemsregistering keys form a membership. Registered systems can then establish a reservation.This is typically set to “Write Exclusive Registrants Only” (WERO). This means registeredsystems can write, and all others cannot.

With SCSI-3 PR technology, blocking write access is as simple as removing a registrationfrom a device. Only registered members can “eject” the registration of another member. Amember wishing to eject another member issues a “preempt and abort” command that


I/O Fencing

ejects another node from the membership. Nodes not in the membership cannot issue thiscommand. Once a node is ejected, it cannot in turn eject another. This means ejecting isfinal and “atomic”.

In the DBE/AC implementation, a node registers the same key for all paths to the device.This means that a single preempt and abort command ejects a node from all paths to thestorage device.

There are several important concepts to summarize here

◆ Only a registered node can eject another

◆ Since a node registers the same key down each path, ejecting a single key blocks allI/O paths from the node

◆ Once a node is ejected, it has no key registered and it cannot eject others.

The SCSI-3 PR specification simply describes the method to control access to disks withthe registration and reservation mechanism. The method to determine who can registerwith a disk and when a registered member should eject another node is implementationspecific. The following paragraphs describe DBE/AC I/O fencing concepts andimplementation.

I/O Fencing ComponentsI/O Fencing, or simply fencing, allows write access to members of the active cluster andblocks access to non-members. I/O fencing in DBE/AC uses several components. Thephysical components are coordinator disks and data disks. Each has a unique purpose anduses different physical disk devices.

Data Disks

Data disks are standard disk devices used for data storage. These can be physical disks orRAID Logical Units (LUNs). These disks must support SCSI-3 PR. Data disks areincorporated in standard VxVM/CVM disk groups. In operation, CVM is responsible forfencing data disks on a disk group basis. Since VxVM enables I/O fencing, several otherfeatures are provided. Disks added to a disk group are automatically fenced, as are newpaths discovered to a device.

Coordinator Disks

Coordinator disks are special purpose disks in a DBE/AC environment. Coordinatordisks are three standard disks, or LUNs, that are set aside for use by I/O fencing duringcluster reconfiguration.


I/O Fencing

The coordinator disks act as a global lock device during a cluster reconfiguration. Thislock mechanism is used to determine who gets to fence off data drives from other nodes.From a high level, a system must eject a peer from the coordinator disks before it can fencethe peer from the data drives. This concept of racing for control of the coordinator disks togain the capability to fence data disks is key to understanding the split brain preventioncapability of fencing.

Coordinator disks cannot be used for any other purpose in the DBE/AC configuration.The user may not store data on these disks, or include the disks in a disk group used byuser data. The coordinator disks can be any three disks that support SCSI-3 PR. VERITAStypically recommends the smallest possible LUNs for coordinator use.

I/O Fencing OperationI/O fencing provided by the kernel-based fencing module performs identically on nodefailures and communications failures. When the fencing module on a node is informed ofa change in cluster membership by the GAB module, it immediately begins the fencingoperation. The node immediately attempts to eject the key for departed node(s) from thecoordinator disks using the preempt and abort command. When the node has successfullyejected the departed nodes from the coordinator disks, it ejects the departed nodes fromthe data disks. If this were a split brain scenario, both sides of the split would be “racing”for control of the coordinator disks. The side winning the majority of the coordinator diskswins the race and fences the loser. The loser then panics and reboots.


I/O Fencing


Installing the Component Packages
2 Installation Overview
The following few paragraphs introduce what’s involved in installing and configuring aDBE/AC cluster with Oracle9i RAC, and describe a completed installation.

After Installation View - A Cluster Running Oracl9i RACFrom a high level, when installation of VERITAS DBE/AC for Oracle9i RAC and Oracle9iis completed and the starter database is created, the RAC cluster should have thefollowing characteristics:

◆ Two systems are connected by two VCS private network links using Ethernetcontrollers on each system. The links are cross-over Ethernet cables, or independenthubs, for each VCS communication network. Hubs are powered from separatesources. Each system uses two independent network cards to provide redundancy.

◆ Each system is connected to the shared storage devices through a switch.

◆ Each system runs VERITAS Cluster Server software, VERITAS Volume Manager withcluster features (CVM), VERITAS Cluster File System with cluster features (CFS), andthe DBE/AC agents and components, including I/O fencing.

◆ Each system has a local VxVM with a “rootdg.” This could be an encapsulated rootdrive or an internal disk. (VERITAS does not support using the rootdg for storageshared between systems. Oracle 9i RAC software must be placed in another diskgroup.)

◆ The Oracle9i RAC software is installed on the shared storage accessible to bothsystems. It could also have been installed locally, but this document features a sharedinstallation.

◆ The Oracle9i RAC database is configured on the shared storage (cluster file system orraw) available to both systems.

◆ VCS is configured such that agents direct and manage the resources required byOracle9i RAC, which runs in parallel on both systems.

17

Installation Overview

Phases of the Installation: An OverviewThe installation and configuration of a DBE/AC cluster with Oracle9i RAC can be seen toconsist of five phases.

Phase One: Setting up and Configuring the Hardware

Prior to installing the DBE/AC software you need a basic hardware setup in place. Thisdocument does not describe the steps to install this hardware, but you can find thesupported hardware detailed in the VERITAS DBE/AC for Oracle9i RAC Release Notes.

✔ The two systems are up and running the same Solaris operating system andconnected to the public network.

✔ Two to four ethernet links directly link the systems to form a private network that is tohandle direct system-to-system communication. Two switches may also be used forthe private network links.

✔ Shared storage is set up that both systems can access through a switch. Note that thisstorage must support SCSI-3 persistent reservations.

Server BServer A

Disk Arrays

Switch

Private Network

SAN

LANClients

Coordinator Disks


Installation Overview

Phase Two: Installing DBE/AC and Configuring its Components

This guide describes the installation procedures for installation DBE/AC. From a highlevel view, when you install DBE/AC, you are:

✔ Running the installDBAC script to install VCS, CVM, and CFS. Included with thisinstallation is the VCS Oracle enterprise agent for Oracle. The script is interactive.

✔ Running the vxinstall utility to initialize VxVM and encapsulate the local disk in aroot disk group. You do not initialize the shared disks at this time.

✔ Rebooting the systems.

✔ Running a script called vxfentsthdw to verify shared storage can be I/O fenced.

✔ Setting up the coordinator disks used by the I/O fencing feature into a disk group.

✔ Starting I/O fencing.

This chapter describes these procedures.

Phase Three: Installing Oracle9i

When the DBE/AC components are installed and configured, you can install the Oracle9isoftware, either Oracle9i Release 1 or Oracle9i Release 2. In either case, installing Oracle9iincludes:

✔ Creating a disk group and a shared volume or a cluster file system for the Oracle9iSRVM component.

✔ Creating a disk group and a shared volume or a cluster file system for the Oracle9iRAC software.

✔ Installing Oracle on the shared storage by running the Oracle9i installer.

Note Oracle9i Release 1 requires a special procedure prior to installing the software.Carefully follow the procedure “Installing Oracle9i Release 1 Software” on page 54.

✔ Linking the Oracle software to the VERITAS ODM libraries.

Installing Oracle9i is described in “Installing Oracle9i Software” on page 51.

Phase Four: Creating the Database

There are many way to create a database. This guide describes how to create a starterdatabase on raw volumes within a VERITAS VxVM disk group or on a VERITAS ClusterFile System, using either the Oracle9i dbca utility or scripts provided by VERITAS. Theseprocedures are provided in case you don’t have database creation tools available. Refer to“Creating Starter Databases” on page 133.

Chapter 2, Installing the Component Packages 19

Installation and Configuration Flowchart

Phase Five: Setting up VCS to Manage RAC Resources

VCS uses a configuration file, called main.cf, to manage the resources used by thecluster. A basic VCS configuration file is created during installation. After Oracle isinstalled and the database is created, the main.cf file must be modified to reflect the newresources and their configuration. The examples used in Chapter 4 describe:

✔ Editing the CVM service group to define the location Oracle9i binaries, the listenerprocess, and the resources on which they depend.

✔ Creating an Oracle database service group to define the database and the resources itdepends on.

Installation and Configuration FlowchartThe flowchart graphically depicts the steps to install and configure VERITAS DBE/AC forOracle9i RAC:

Set PATH, MANPATH variables

Mount CD and cd /cdrom/database_ac_for_oracle9i

See this chapter

Run installDBAC to install VCS,VxVM, VxFS packages

Run chk_dbac_pkgs to verify installation

Anothersystem?

Yes

No

See next page

Anothersystem?

Yes

No

Reboot

Run vxinstallSee VxVMSystem Administrator’s

Guide


Installation and Configuration Flowchart

Create shared DG’s / Vol’s for OracleSee Chapter 3, and VxVM

See Chapter 4

Create Oracle starter database - Optional

Edit CVM groupSee Chapter 4

& VCS Oracle E.A. ICG

System Administrator’s Guide

Install Oracle9i and Oracle9i patchSee Chapter 3, and Oracle

Installation Guide

See Chapter 3, and OracleInstallation Guide

Create & configure coordinator disk group

Create /etc/vxfendg

Link Oracle to VERITAS ODM librariesSee Chapter 3

Anothersystem?

Yes

No

Verify SCSI-3 PR support for all shareddisks including coordinator disks

From previous page

See this chapter

Edit Oracle group

Start I/O fence driver


Prerequisites for Installing Components

Prerequisites for Installing Components✔ Obtain license key to use VERITAS Database Edition for Advanced Cluster. Refer to

“Obtaining License Keys for DBE/AC for Oracle9i RAC” on page xiv.

✔ Make sure your cluster meets the necessary hardware and software requirements.Refer to the VERITAS DBE/AC for Oracle9i RAC Release Notes.

✔ Each system in the cluster requires the following local disk space:

- For the installation of VERITAS DBE/AC for Oracle9i RAC:

- For the installation of Oracle9i, each local system requires approximately 4.6gigabytes. For the installation of Oracle9i Release 1, a portion of the disk space,approximately 2 GB, is required for copies of the Oracle9i installation disks.

✔ Remove any versions of VCS and unmount the file systems (VxFS) that currently runon the systems where you intend to install VERITAS DBE/AC for Oracle9i RAC. TheVERITAS DBE/AC for Oracle9i RAC product requires the versions of VCS, VxVM, andVxFS contained on the CD. Refer to the installation guide corresponding to eachproduct for uninstallation instructions. If you are running versions of VCS, VxVM, orVxFS, uninstall VCS first and upgrade to release 3.5 using the installDBAC utility.

✔ Obtain and install the Sun SPARCstorage Array (SUNWsan) package on each node.

✔ Obtain and install the Solaris 8 patches required by VxVM one each node; theyinclude 108528-14, 108827-19, 109529-06, 110722-01, and 111413-06.

✔ Obtain and install the Solaris 8 patches required by VxFS on each node: 108528-02 and108901-03.

/ 450 MB

/tmp 100MB for the duration of the installation


Preparing for Using the installDBAC Script

Preparing for Using the installDBAC ScriptThe installDBAC script is interactive. It prompts you for information about the clusterwhere you are installing VERITAS DBE/AC for Oracle9i RAC. Before you start the script,have the following information at hand:

General Information Requested by Script

✔ The name of the cluster (must begin with a letter of the alphabet (a-z, A-Z)

✔ The ID number of the cluster (each cluster within a site requires a unique ID)

✔ The host names of the systems in the cluster

✔ The device names for the two private network links

✔ A valid license key for VERITAS DBE/AC for Oracle9i RAC

Web-based ClusterManager Information Requested by Script

✔ The name of the public NIC for the VCS ClusterManager

✔ The virtual IP address for the NIC used by ClusterManager

✔ The netmask for the virtual IP address

SMTP/SNMP (Optional) Requested by Script

✔ Domain-based address of SMTP server

✔ Email address of SMTP notification recipients

✔ Severity level of events for SMTP notification

✔ System name for the SNMP console.

✔ Severity level of events for SNMP notification

Note The installation of VCS using the installDBAC installation utility is essentially thesame as that described in the VERITAS Cluster Server Installation Guide. The VCSInstallation Guide contains more extensive details about VCS installation.


Installing DBE/AC Packages

Installing DBE/AC PackagesThe installDBAC script installs VERITAS Cluster Server (VCS), VERITAS VolumeManager (VxVM), VERITAS File System (VxFS), on each system. Refer to the preceding“Installation and Configuration Flowchart.”

Setting the PATH VariableThe installation and other commands are located in various directories. On each system,add these directories to your PATH environment variable using the following commands:

If you use the Bourne Shell (sh or ksh):

$ PATH=/sbin:/usr/sbin:/usr/bin:/usr/lib/vxvm/bin:\/usr/lib/fs/vxfs:/opt/VRTSvxfs/sbin:/opt/VRTSvcs/bin:\/opt/VRTSvcs/rac/bin:/opt/VRTSob/bin:$PATH; export PATH

If you use the C Shell (csh or tcsh):

% setenv PATH /sbin:/usr/sbin:/usr/bin:/usr/lib/vxvm/bin:\/usr/lib/fs/vxfs:/opt/VRTSvxfs/sbin:/opt/VRTSvcs/bin:\/opt/VRTSvcs/rac/bin:/opt/VRTSob/bin:$PATH

Note For root user, do not define paths to a cluster file system in theLD_LIBRARY_PATH variable. For example, define ORACLE_HOME/lib inLD_LIBRARY_PATH for user oracle only.

The path defined to /opt/VRTSob/bin is optional, required if you install the optionalpackage, VERITAS Enterprise Administrator.

Setting the MANPATH VariableSet the MANPATH variable to enable viewing manual pages:


$ MANPATH=/usr/share/man:/opt/VRTS/man; export MANPATH


% setenv MANPATH /usr/share/man:/opt/VRTS/man



Setting Up System-to-System CommunicationProvide the capability of each node to have remote rsh access to the other, at least duringinstallation and disk verification. On each system, placing a “+” character in the first lineof the file /.rhosts gives remote access to the system running the install program. Youcan also edit the hosts.equiv file to limit the remote access to specific systems. Refer tothe manual page for hosts.equiv files for more information about these files.

Note Remove the remote rsh access permissions when they are no longer required forinstallation and disk verification. See “Removing Temporary rsh AccessPermissions” on page 69.

Mounting the VERITAS DBE/AC for Oracle9i RAC CDInsert the CD containing the VERITAS DBE/AC for Oracle9i RAC software in a CD-ROMdrive connected to the system. The Solaris volume-management tool automaticallymounts the CD.

With the CD mounted, you are ready to start installing the packages.

Running the VERITAS DBE/AC Install ScriptTo start the installation of VERITAS DBE/AC:

1. Log in as root user on one of the systems for installation.

2. Enter the following commands to start installDBAC:

# cd /cdrom/database_ac_for_oracle9i# ./installDBAC

3. At the prompt, enter the names of the two systems for installation:

Enter the names of the systems on which VCS is to be installedseparated by spaces (example: system1 system2): galaxy nebula

Analyzing . . .



Verifying Systems for Installation

4. The utility begins by checking that the systems are ready for installation. On thesystem where you are running the utility, the utility verifies that:

a. Sufficient disk space exists to install the packages

b. The required Solaris patches and the Solaris SUNWsan package are installed

Note You can ignore warnings about recommended optional patches that are not present

c. User has root permissions

The utility exits with a brief message if a system is not ready for the installation.

5. Before installing any packages, the utility checks whether the VERITAS licensepackage, VRTSvlic, is present on the installation system.

- If the VRTSvlic package is not present, the installation utility installs it.

- If the installed VRTSvlic package is not the current version, the utility promptsyou to upgrade to the current version. The utility exits if you decline to upgrade.

6. The utility repeats the checks performed in step 4 and step 5 on the second system.

7. The utility checks for a VERITAS DBE/AC license key on the first system. If it cannotfind a key, it prompts you to enter one:

You do not have a Database Edition for Advanced Cluster licenseinstalled on galaxy.

Do you want to add a license key for this product on galaxy ?[Y/N](Y) y

Enter your license key : XXXX-XXXX-XXXX-XXXX-XXXX-XXXRegistering license for galaxy.

You do not have a Database Edition for Advanced Cluster licenseinstalled on nebula.

Do you want to add a license key for this product on nebula ?[Y/N](Y) y

Enter your license key : XXXX-XXXX-XXXX-XXXX-XXXX-XXXRegistering license for nebula.



8. When you are prompted to continue VERITAS Cluster Server installation, pressReturn to accept the default (Y) or enter “Y.”

You can skip VERITAS Cluster server installation, if you need toinstall only VERITAS Volume manager and VERITAS Filesystem

Would you like to continue VCS installation. [Y/N](Y)? YStarting VCS installation.

9. For each system, the script checks system to system communications:

Checking for ssh on galaxy ............................ not found Verifying communication with nebula ............. ping successful Attempting rsh with nebula ....................... rsh successful Checking OS version on galaxy. ........................ SunOS 5.8 Creating /tmp subdirectory on galaxy... /tmp subdirectory created Checking OS version on nebula ......................... SunOS 5.8

Using /usr/bin/rsh and /usr/bin/rcp to communicate with galaxy Communication check completed successfully

10. The VCS portion of the installation utility verifies the license status of each node.

VCS licensing verification:

Checking galaxy .............found DBED/AC Node Locked key Do you want to enter a new key for galaxy? (N) n Using the existing DBED/AC Node Locked key located on nebula.

Checking nebula .............found DBED/AC Node Locked key Using the existing DBED/AC Node Locked key located on nebula.

DBED/AC licensing completed successfully

11. When you are ready to start installation of VCS, enter “Y” when prompted:

Are you ready to start the Cluster installation now? (Y) Y

12. For each system, the VCS portion of the utility checks if any of the packages to beinstalled are present. If any are found, the utility continues if there are no conflicts orexits if packages must be manually uninstalled.

13. For each system, the utility checks for the necessary file system space.

14. For each system, the utility check whether any DBE/AC processes are running. Anyrunning processes are stopped.



Configuring the Cluster

15. The script highlights the information required to proceed with installation of VCS:

To configure VCS the following is required:

A unique Cluster name A unique Cluster ID number between 0-255 Two NIC cards on each system used for private network

heartbeat links

Are you ready to configure DBED/AC on these systems? (Y)

Note When the installDBAC installation script shows a single response withinparentheses, it is the default response. Press Return to choose the default response.

Enter the unique Cluster Name: vcs_raclusterEnter the unique Cluster ID number between 0-255: 7

16. After you enter the cluster ID number, the program discovers and lists all NICs on thefirst system:

Discovering NICs on galaxy: ..... discovered hme0 qfe0 qfe1qfe2 qfe3

Enter the NIC for the first private network heartbeat link ongalaxy: (hme0 qfe0 qfe1 qfe2 qfe3) qfe0

Enter the NIC for the second private network heartbeat link onnebula: (hme0 qfe1 qfe2 qfe3) qfe1

Are you using the same NICs for private heartbeat links on all systems? (Y)

If you answer “N,” the program prompts for the NICs of each system.

17. The installation program asks you to verify the cluster information:

Cluster information verification:

Cluster Name: vcs_raclusterCluster ID Number: 7Private Network Heartbeat Links for galaxy: link1=qfe0 link2=qfe1Private Network Heartbeat Links for nebula: link1=qfe1 link2=qfe1

Is this information correct? (Y)

If you enter “N,” the program returns to the screen describing the installationrequirements (step 15).



Configuring Cluster Manager

18. The installation program describes the information required to configure ClusterManager (Web Console):

The following information is required to configure the ClusterManager:

A public NIC used by each system in the clusterA Virtual IP address and netmask for the Cluster Manager

Do you want to configure the Cluster Manager (Web Console) (Y)

Press Return to configure Cluster Manager (Web Console) on the systems. You canenter “N” and skip configuring Cluster Manager and the program advances you tothe screen enabling you to configure SMTP notification.

19. In reply to the prompts that follow, confirm that you want to use the discoveredpublic NIC on the first system. Also indicate whether all systems use the same publicNIC. Press Return to choose the default responses.

Active NIC devices discovered on galaxy: hme0Enter the NIC for the Cluster Manager (Web Console) to use ongalaxy: (hme0)Is hme0 the public NIC used for all systems (Y)

20. Enter the virtual IP address to be used by Cluster Manager:

Enter the Virtual IP address for the Cluster Manager:10.180.88.199

You can confirm the default netmask, or enter another:

Enter the netmask for IP 10.180.88.199: (255.0.0.0)

21. The installation program asks you to verify the Cluster Manager information:

Cluster Manager (Web Console) verification:

NIC: hme0 IP: 10.180.88.199 Netmask: 255.0.0.0


To indicate the information is correct, press Return. Otherwise, enter “n” and theprogram returns to the screen describing the Cluster Manager requirements (step 18).



Configuring SMTP Email Notification

22. The installation program describes the information required to configure the SMTPnotification feature of VCS:

The following information is required to configure SMTPnotification:

The domain based address of the SMTP server The email address of each SMTP recipient A minimum severity level of messages to send to each

recipient

Do you want to configure SMTP notification? (Y) y

You can enter “N” and skip configuring SMTP notification. The program advancesyou the screen enabling you to choose SNMP notification (see step 25).

23. Respond to the prompts and provide information to configure SMTP notification.

Enter the domain based address of the SMTP server (example:smtp.yourcompany.com) smtp.xyzstar.com

Enter the email address of the SMTP recipient [email protected] the minimum severity of events for which mail should besent to [email protected]: [I=Information, W=Warning, E=Error,S=SevereError] w

Would you like to add another SMTP recipient? (N) yEnter the email address of the SMTP recipient [email protected] the minimum severity of events for which mail should besent to [email protected]: [I=Information, W=Warning,E=Error, S=SevereError] e

Would you like to add another SMTP recipient? (N)

24. Verify that the SMTP notification information is correct.

SMTP notification verification:

SMTP Address: smtp.xyzstar.com Recipient: [email protected] receives email for Warning or

higher events Recipient: [email protected] receives email for Error or

higher events


Press Return if the information is correct. If it is incorrect, enter “N.” The programreturns to the screen describing SMTP notification requirements (see step 22). You canbypass configuration of SMTP notification or reenter the information.



Configuring SNMP Trap Notification

25. The installation program describes the information required to configure the SNMPnotification feature of VCS:

The following information is required to configure SNMPnotification:

System names of SNMP console(s) to receive DBED/AC trap messagesSNMP trap daemon port numbers for each consoleA minimum severity level of messages to send to each console

Do you want to configure SNMP notification? (Y)

You can enter “N” and skip configuring SNMP notification. The installation programstarts installation; see step 28.

26. Respond to the prompts and provide information to configure SNMP notification.

Enter the SNMP trap daemon port (162)Please enter the SNMP console system name saturnEnter the minimum severity of events for which SNMP traps shouldbe sent to saturn: [I=Information, W=Warning, E=Error,S=SevereError] e

Would you like to add another SNMP console? (N) yPlease enter the SNMP console system name jupiterEnter the minimum severity of events for which SNMP traps shouldbe sent to jupiter: [I=Information, W=Warning, E=Error,S=SevereError] s

Would you like to add another SNMP console? (N)

27. Verify that the SNMP notification information is correct.

SNMP notification verification:

SNMP Port: 162Console: saturn receives SNMP traps for Error or higher eventsConsole: jupiter receives SNMP traps for SevereError or higher

events


Press Return if the information is correct. If the information is incorrect, enter “N” andthe program returns to the screen describing the SNMP notification requirements (seestep ). You can either bypass configuration of SNMP notification or reenter theinformation.



Installing VCS Packages

28. After you have verified that the information you have entered is correct, theinstallation program begins installing the packages on the first system:

Installing VCS on galaxy:

Installing VRTSvlic package ........................... Done Installing VRTSperl package ........................... Done Installing VRTSllt package ............................ Done Installing VRTSgab package ............................ Done Installing VRTSvcs package ............................ Done Installing VRTSvcsmg package .......................... Done Installing VRTSvcsag package .......................... Done Installing VRTSweb package ............................ Done Installing VRTSvcsw package ........................... Done Installing VRTSvcsdc package .......................... Done Installing VRTSvcsmn package .......................... Done Installing VRTSvcsor package .......................... Done Installing VRTSdbac package ........................... Done

The same packages are installed on each machine in the cluster:

Installing VCS on nebula:

Copying VRTSvlic binaries ............................. Done Installing VRTSvlic package ........................... Done Copying VRTSperl binaries ............................. Done Installing VRTSperl package ........................... Done

.

.

.

Package installation completed successfully

Configuring VCS

29. The installation program continues by creating configuration files and copying themto each system:

Configuring DBED/AC ..................................... Done Copying DBED/AC configuration files to galaxy ........... Done Copying DBED/AC configuration files to nebula ........... Done

Configuration files copied successfully



Starting VCS

30. You can now start VCS and its components on each system:

Do you want to start the cluster components now? (Y)...

Starting DBED/AC on galaxy

Starting LLT ...................................... Started Starting GAB ...................................... Started Starting DBED/AC .................................. Started Starting DBED/AC drivers........................... Started

Starting DBED/AC on nebula

Starting LLT ...................................... Started Starting GAB ...................................... Started Starting DBED/AC .................................. Started Starting DBED/AC drivers........................... Started



31. When the utility completes the installation of VCS, it reports on:

- The creation of backup configuration files (named with the extension“init.name_of_cluster”). For example, the file /etc/llttab in the installationexample has a backup file named /etc/llttab.init.vcs_racluster.

- The URL and the login information for Cluster Manager (Web Console), if youchose to configure it. Typically this resembles:

http://10.180.88.199:8181/vcs

You can access the Web Console using the User Name: “admin” and thePassword: “password”. Use the /opt/VRTSvcs/bin/hauser command to addnew users.

- The location of a VCS installation report, which is named:/var/VRTSvcs/installvcsReport.name_of_cluster

32. When installation is complete, you see the message:

DBED/AC installation completed successfullySuccessful installation of VCS

Installing Base CVM and CFS Packages

33. After the script installs and starts VCS on each cluster system, it prompts you aboutinstalling the base VERITAS DBE/AC packages, which include those for the ClusterVolume Manager (CVM) and Cluster File System (CFS) components.

The installDBAC script installs required packages followed by the optionalpackages that you choose. The base packages are installed first on the first systementered at the start of installation.

Installing on galaxy machine.

Starting installation of VERITAS Database Edition for AdvancedCluster on galaxy

Kernel Config parameters are set.

The following packages need to be installedfor VERITAS Database Edition for Advanced Cluster.

VRTSvxvm is required VRTSvxfs is required VRTSglm is required VRTSgms is required VRTSodm is required



VRTScavf is required VRTSvmman is Optional VRTSvmdoc is Optional VRTSfsdoc is Optional VRTSob is Optional VRTSvmpro is Optional VRTSfspro is Optional VRTSobgui is Optional

Continue? [y,n,?,q] y

34. Installation of the packages listed proceeds when you answer “y.” First, the requiredpackages are installed. After each package is installed, press Return to install the nextrequired package.

Checking existing package installation ... Installing VRTSvxvm. Installation of VRTSvxvm................................Done Installing VRTSvxfs. . . .

35. When prompted about installing the optional packages, you can choose whichpackages you want to install. For example:

. . . Installing VRTSvmman. system VRTSvmman VERITAS Volume Manager, Manual Pages This package is optional. Install? [y,n,?,q] y Installation of VRTSvmman................................Done . . .



36. The script completes installation on the first system:

Completed Installation of VERITAS database Edition for AdvancedCluster.The following packages were installed:

VRTSobgui VRTSfspro VRTSvmpro VRTSob VRTSfsdoc VRTSvmdoc VRTSvmman VRTScavf VRTSodm VRTSgms VRTSglm VRTSvxfs VRTSvxvm

Successfully Installed Foundation products on galaxy

37. Installation continues on the next system:

Installing on nebula machine. Starting installation of VERITAS Database Edition for Advanced Cluster on nebula.

Kernel Config parameters are set.

The following packages need to be installed for VERITAS Database Edition for Advanced Cluster.

VRTSvxvm is required VRTSvxfs is required VRTSglm is required VRTSgms is required VRTSodm is required VRTScavf is required VRTSvmman is Optional VRTSvmdoc is Optional VRTSfsdoc is Optional VRTSob is Optional VRTSvmpro is Optional VRTSfspro is Optional VRTSobgui is Optional



Continue? [y,n,?,q] y

Checking existing package installation ...Copying VRTSvxvm binaries ...

Installing VRTSvxvm. Installation of VRTSvxvm...............................Done

Copying VRTSvxfs binaries ... .

.

Completed Installation of VERITAS Database Edition for AdvancedCluster.

The following packages were installed:

VRTSobgui VRTSfspro VRTSvmpro VRTSob VRTSfsdoc VRTSvmdoc VRTSvmman VRTScavf VRTSodm VRTSgms VRTSglm VRTSvxfs VRTSvxvm

Configuring CVM and CFS

38. The install script prompts you about configuring Cluster Volume Manager andCluster File System:

Successfully installed Foundation products on nebulaWould you like to configure Cluster Volume Manager and ClusterFile System [Y/N](Y) y

The cluster configuration information as read from cluster configuration file is as follows. Cluster : vcs_racluster Nodes : galaxy nebula

You will now be prompted to enter the information pertaining to the cluster and the individual nodes.



39. You can choose a different “timeout” value, if necessary, or press Return to accept thedefault:

Enter the timeout value in seconds. This timeout will beused by CVM in the protocol executed during clusterreconfiguration. Choose the default value, if you do not wishto enter any specific value.

Timeout: [<timeout>,q,?] (default: 200)

40. When additional cluster information is displayed, “gab” is shown as the configuredtransport protocol, meaning GAB is to provide communications between systems inthe cluster. For VERITAS DBE/AC for Oracle9i RAC, GAB is required.

------- Following is the summary of the information: ------ Cluster : vcs_racluster Nodes : galaxy nebula Transport : gab-----------------------------------------------------------Hit RETURN to continue.

Is this correct [y,n,q,?] (default: y) y

41. The configuration of CVM begins. When CVM has been added, you receive a messageresembling:

You will now be prompted to enter whether you want to start cvm on the systems in the cluster. If you choose to do so, then attempt will be made to start the cvm on all the systems in the cluster. It will not be started on the systems which do not have cluster manager running.

Do you want to start cvm ? [y,n] n

======================================================== Cluster File System Configuration is in progress... cfscluster: CFS Cluster Configured Successfully


Verifying All Prerequisite Packages Are Installed

Completing Installation

42. After reporting that the product is successfully installed, the utility tells you that youneed to reboot all systems in the cluster:

You must reboot all the system in the cluster.

Note Do not reboot the systems at this time. You must first run the vxinstall utility oneach system before rebooting.

43. The installDBAC installation utility is done.

Verifying All Prerequisite Packages Are InstalledAfter running installDBAC, verify that all the necessary packages are installed. Use thefollowing command on each system:

# /opt/VRTSvcs/bin/chk_dbac_pkgs

This utility does not check for the documentation packages (VRTSvmdoc, VRTSfsdoc,VRTSvcsdc, VRTSvcsmn, or VRTSvmman) or the graphical user interface packages(VRTSob, VRTSobgui, VRTSfspro, VRTSvmpro, VRTSweb, or VRTSvcsw).

If a requisite package or patch is not present, the utility reports an error, such as:

Checking package - VRTSglmERROR: Information for "VRTSglm" was not found

Package VRTSglm not installed - install it before proceeding


Configuring VxVM Using vxinstall and Rebooting

Configuring VxVM Using vxinstall and RebootingOn each system, you must run the vxinstall utility. The local disk on each system mustbe encapsulated in the root disk group, or “rootdg.” Do not include the shared disks inrootdg. You must reboot both systems after running vxinstall.

# vxinstall

Refer to the VERITAS Volume Manager Installation Guide for instructions on how to use thevxinstall utility to initialize VxVM. You must run vxinstall on each system beforeverifying that shared storage can support SCSI-3 persistent reservations.

Verifying Storage Supports SCSI-3 Persistent ReservationsThe shared storage used with VERITAS DBE/AC for Oracle9i RAC software must supportSCSI-3 persistent reservations. SCSI-3 persistent reservations support enables the use ofI/O fencing to prevent data corruption caused by a split brain. I/O fencing is described inthe chapter on “I/O Fencing Scenarios” on page 81.

To protect the data on shared disks, each system in the cluster must be configured withI/O fencing. You must verify that each disk array you use for shared storage, includingthe coordinator disks, supports SCSI-3 persistent reservations.

While the following sections pertain to all shared storage used with VERITAS DBE/AC forOracle9i RAC, they also pertain to the coordinator disks.


Verifying Storage Supports SCSI-3 Persistent Reservations

For EMC Symmetrix 8000 Series Storage◆ Ask your EMC representative to help you verify that the firmware on the storage unit

is set to enable SCSI-3 persistent reservations. The PR flag must be set on each EMCvolume.

◆ Verify that the same serial number for a LUN is returned on all paths to the LUN.Refer to the vxfenadm(1M) manual page. You can use the command:

vxfenadm -i diskpath

For example:

# vxfenadm -i /dev/rdsk/c2t13d0s2Vendor id : EMCProduct id : SYMMETRIXRevision : 5567Serial Number : 42031000a

For Hitachi Data Systems 99xx Series Storage◆ The SCSI-3 persistent reservations feature requires that the disk array be running the

latest firmware (see VERITAS DBE/AC for Oracle9i RAC Release Notes).

◆ System Mode 186 must be enabled for the SCSI reservations feature. Requestassistance from HDS to verify this.

◆ Verify that the same serial number for a LUN is returned on all paths to the LUN.Refer to the vxfenadm(1M) manual page. You can use the command:

vxfenadm -i diskpath

For example:

# vxfenadm -i /dev/rdsk/c2t0d2s2Vendor id : HITACHIProduct id : OPEN-3 -SUNRevision : 0117Serial Number : 0401EB6F0002



Verifying Shared Storage ArraysTo verify that the shared storage arrays support SCSI-3 persistent reservations and I/OFencing, use the vxfentsthdw utility. Test at least one disk in each shared disk array. Werecommend you test as many disks as possible in each array.

Note Only supported storage devices can be used with I/O fencing of shared storage.Currently, supported devices include: EMC Symmetrix 8000 Series or Hitachi DataSystem Series 99xx. Other disks, even if they pass the following test procedure, arenot supported. Please see the VERITAS DBE/AC for Oracle9i RAC Release Notes foradditional information about supported shared storage.

The utility, which you can run from one system in the cluster, tests the storage by settingSCSI-3 registrations on the disk you specify, verifying the registrations on the disk, andremoving the registrations from the disk. See the chapter on “I/O Fencing Scenarios” onpage 81 for information on I/O fencing and a description of how it works. Refer also tothe vxfenadm(1M) manual page.

Caution The utility vxfentsthdw overwrites and destroys existing data on the disks.

Using the vxfentsthdw Utility

Assume that you need to check the shared device that, in the example, both systems knowas /dev/rdsk/c4t8d0s2. (It’s also possible that each system would use a differentname for the same physical device.)

1. Make sure system-to-system communication is setup. See “Setting UpSystem-to-System Communication” on page 25.

2. On one system, start the utility:

# cd /opt/VRTSvcs/rac/bin# ./vxfentsthdw

The utility begins by providing an overview of its function and behavior. It warns youthat the tests it performs overwrites any data on the disks you check:

******** WARNING!!!!!!!! ********THIS UTILITY WILL DESTROY THE DATA ON THE DISK!!

Do you still want to contnue : [y/n] (default: n)yEnter the first node of the cluster:galaxy



Enter the second node of the cluster:nebula

3. Enter the name of the disk you are checking. For each node, the disk may be knownby the same name, as in our example.

Enter the disk name to be checked for SCSI-3 PGR on node galaxy inthe format: /dev/rdsk/cxtxdxsx/dev/rdsk/c4t8d0s2

Enter the disk name to be checked for SCSI-3 PGR on node nebula inthe format: /dev/rdsk/cxtxdxsxMake sure it’s the same disk as seen by nodes galaxy and nebula/dev/rdsk/c4t8d0s2

Note the disk names, whether or not they are identical, must refer to the samephysical disk, or the testing terminates without success.

4. The utility starts to perform the check and report its activities. For example:

Keys registered for disk /dev/rdsk/c2t8d0s2 on node galaxy

Verifying registrations of keys for disk /dev/rdsk/c2t8d0s2 on node galaxy: passed

Reads from disk /dev/rdsk/c4t8d0s2 successful on node galaxy

Reads from disk /dev/rdsk/c4t8d0s2 successful on node nebula

Writes from disk /dev/rdsk/c4t8d0s2 successful on node galaxy

Writes from disk /dev/rdsk/c4t8d0s3 successful on node nebula . . .

5. For a disk that is ready to be configured for I/O Fencing on both systems, the utilityreports success. For example:

The disk /dev/rdsk/c4t8d0s2 is ready to be configured for I/OFencing on node galaxyThe disk /dev/rdsk/c4t8d0s2 is ready to be configured for I/OFencing on node nebula

6. Run the vxfentsthdw utility for each disk you intend to verify.


Setting Up Coordinator Disks

Setting Up Coordinator DisksI/O Fencing requires coordinator disks configured in a disk group that each of thesystems in the cluster can access. The use of coordinator disks enables the vxfen driver toresolve potential split brain conditions and prevent data corruption. See the chapter “I/OFencing Scenarios” on page 81 for a discussion of I/O fencing and the role of coordinatordisks.

A coordinator disk is not used for data storage, and so may be configured as the smallestpossible LUN on a disk array to avoid wasting space.

Requirements for Coordinator DisksCoordinator disks have the following requirements:

✔ There must be at least three coordinator disks and the total number of disks must bean odd number. This ensures a majority of disks can be achieved during I/O fencing.

✔ Each of the coordinator disks must use a physically separate disk or LUN.

✔ Each of the coordinator disks should be on a different disk array, if possible.

✔ Coordinator disks in a disk array should use hardware-based mirroring.

✔ The coordinator disks must support SCSI-3 persistent reservations. See “VerifyingStorage Supports SCSI-3 Persistent Reservations” on page 40.

Configuring a Disk Group Containing Coordinator Disks

1. From one system, create a disk group named: vxfencoorddg. This group mustcontain an odd number of disks/LUNs and a minimum of three disks. Thedisks/LUNs must support SCSI-3 persistent reservations. It is recommended you usethe smallest size disks/LUNs, so that space for data is not wasted. The disk groupmust be accessible to all cluster systems.

Refer to the VERITAS Volume Manager Administrator’s Guide for instructions oncreating disk groups.

2. Deport the disk group:

# vxdg deport vxfencoorddg

3. Import the disk group with the -t option so that it is not automatically importedwhen the systems are rebooted:

# vxdg -t import vxfencoorddg


Setting Up Coordinator Disks

4. Deport the disk group. The disk group is no longer imported. It is used to prevent thecoordinator disks from being used for other purposes.

# vxdg deport vxfencoorddg

5. On all systems, enter the command:

# echo "vxfencoorddg" > /etc/vxfendg

No spaces should appear between the quotes in the "vxfencoorddg" text.

This command creates the file /etc/vxfendg, which includes the name of thecoordinator disk group. Based on the /etc/vxfendg file, the rc script creates the file/etc/vxfentab for use by the vxfen driver.

Starting I/O FencingTo start the I/O fencing driver, issue the following command on each system:

# /etc/init.d/vxfen start

The Contents of the /etc/vxfentab FileOn each system, the disks used as coordinator disks are listed in the file /etc/vxfentab.The same disks may be listed using different names on each system. An example/etc/vxfentab file on one system resembles:

/dev/rdsk/c1t1d0s2/dev/rdsk/c2t1d0s2/dev/rdsk/c3t1d0s2

The rc startup script automatically creates /etc/vxfentab by reading the diskscontained in the disk group listed in /etc/vxfendg, then invokes the vxfenconfigcommand. This command configures the vxfen driver to start and use the coordinatordisks listed in /etc/vxfentab.

Replacing Failed Coordinator DisksAt the present time, it is not possible to replace a failed coordinator disk withoutrebooting the cluster.


Running gabconfig -a After Installation

Running gabconfig -a After InstallationAfter you have installed the VERITAS DBE/AC for Oracle9i RAC packages, run vxinstall,and rebooted both systems, you can verify the installation by running the command:

# /sbin/gabconfig -a

For example:

galaxy# /sbin/gabconfig -aGAB Port Memberships===============================================================Port a gen 4a1c0001 membership 01Port b gen g8ty0002 membership 01Port d gen 40100001 membership 01Port f gen f1990002 membership 01Port h gen d8850002 membership 01Port o gen f1100002 membership 01Port q gen 28d10002 membership 01Port v gen 1fc60002 membership 01Port w gen 15ba0002 membership 01

The output of the gabconfig -a command displays which cluster systems havemembership with the modules that have been installed and configured thus far in theinstallation.The first line indicates that both systems (0 and 1) have membership with theGAB utility, which uses “Port a.” The ports listed, including port a, are configured for thefollowing functions:

Port Function

b I/O fencing

d ODM (Oracle Disk Manager)

f CFS (Cluster File System)

h VCS (VERITAS Cluster Server: high availability daemon)

o VCSMM driver

q QuickLog daemon

v CVM (Cluster Volume Manager)

w vxconfigd (module for cvm)


Example VCS Configuration File After Installation

Example VCS Configuration File After InstallationTo verify the installation, you can examine the VCS configuration file, main.cf, in thedirectory /etc/VRTSvcs/conf/config. Also, see an example, “Example main.cf file”on page 48. The main.cf file is created during the installation of VERITAS DBE/AC forOracle9i RAC software. Note the following about the VCS configuration:

◆ The “include” statements list types files for VCS (types.cf), CFS (CFSTypes.cf),CVM (CVMTypes.cf), and the Oracle enterprise agent (OracleTypes.cf). Thesefiles are located in /etc/VRTSvcs/conf/config and they define the agents thatcontrol the resources in the cluster.

- The VCS types include all agents bundled with VCS. Refer to the VERITASBundled Agents Reference Guide for a information about all VCS agents.

- The CFS types include the CFSMount, CFSQlogckd, and CFSfsckd.

The CFSMount agent mounts and unmounts the shared volume file systems.

The CFSQlogckd and CFSfsckd are types defined for cluster file systemdaemons and do not require user configuration.

- The CVM types include the CVMCluster and CVMVolDg types.

The CVMCluster resource, which is automatically configured during installation,imports the shared disk groups and defines how systems in the cluster are tocommunicate volume state information. Refer to “CVMCluster, CVMVolDg, andCFSMount Agents” on page 115.

The CVMVolDg resource imports disk groups and sets activation modes. See“Modifying the CVM Service Group in the main.cf File” on page 73.

- The Oracle enterprise agent types file includes definitions for the Oracle agentand the Sqlnet agent. The Oracle agent monitors the resources for an Oracledatabase, and the Sqlnet resource manages the resources for the listener process.

◆ The service group “cvm” is added. It includes definitions for monitoring the CFS andCVM resources. The CVMCluster agent resource definition indicates how the systemsin the cluster are to communicate volume state information. Later, after theinstallation of Oracle, the cvm group must be modified to include Sqlnet, IP, and NICresources required for the listener process.

◆ The cvm group has the parallel attribute set to 1, which means that the resources areset to run in parallel on each system in the system list.



Example main.cf fileinclude "types.cf"include "CFSTypes.cf"include "CVMTypes.cf"include "OracleTypes.cf"

cluster vcs_racluster ( UserNames = { admin = "cDRpdxPmHpzS." } Administrators = { admin } HacliUserLevel = COMMANDROOT CounterInterval = 5 )

system galaxy ( )

system nebula ( )

group cvm ( SystemList = { galaxy = 0, nebula = 1 } AutoFailOver = 0 Parallel = 1 AutoStartList = { galaxy, nebula } )

CFSQlogckd qlogckd ( Critical = 0 )

CFSfsckd vxfsckd ( )

CVMCluster cvm_clus ( Critical = 0 CVMClustName = vcs_racluster CVMNodeId = { galaxy = 0, nebula = 1 } CVMTransport = gab CVMTimeout = 200 )

vxfsckd requires qlogckd

// resource dependency tree // // group cvm



// { // CVMCluster cvm_clus // CFSfsckd vxfsckd // { // CFSQlogckd qlogckd // } // }




Installing Oracle9i Software
3 This chapter describes how to install Oracle9i software. You can install Oracle9i on sharedstorage or locally on each node. However, the shared storage installation is featured in thisguide. The configuration of VCS service groups described in Chapter 4 and the examplemain.cf file shown in Appendix A are based on using shared storage to install Oracle9i.
Oracle9i Installation TasksInstalling Oracle9i in a VERITAS DBE/AC for Oracle9i RAC environment involves thefollowing tasks:

◆ Creating shared disk group and volume for the Oracle9i SRVM component

◆ Installing Oracle9i, Release 1 or Release 2

You can install Oracle9i on either shared disks or a local disk.

◆ Linking VERITAS ODM libraries to Oracle

Prerequisites for Installing Oracle9iBefore installing Oracle and creating a database, do the following:

1. Edit the file /etc/system and set the shared memory parameter. Refer to theOracle9i Installation Guide.

2. Verify that the disk arrays used for shared storage support SCSI-3 persistentreservations and I/O fencing. Refer to “Verifying Storage Supports SCSI-3 PersistentReservations” on page 40.

The shared storage used with VERITAS DBE/AC for Oracle9i RAC must supportSCSI-3 persistent reservations. SCSI-3 persistent reservations support enables the useof I/O fencing to prevent data corruption caused by a split brain. I/O fencing isdescribed in the chapter on “I/O Fencing Scenarios” on page 81.

51

Creating Shared Disk Groups and Volumes - General

Creating Shared Disk Groups and Volumes - GeneralThe following paragraphs highlight general information that applies to creating shareddisk groups. Refer to the VERITAS Volume Manager Administrator’s Guide for fulldocumentation about creating and managing shared disk groups.

Determining Information About a Disk GroupYou can display information about a specific disk group by entering the command:

# vxdg list disk_group

Setting the Connectivity Policy on a Shared Disk GroupTo set the connectivity policy for a disk group as “local,” use the following command fromthe master node:

# vxedit set diskdetpolicy=local shared_disk_group

The command has effect on all systems in the cluster. The local connectivity policy meansthat when a disk fails, the disk is detached only from the system detecting the failure.Subsequently, the disk is detached cluster wide only when all systems report the failure.

Enabling Write Access to the Volumes in the Disk GroupAt the time VCS is installed, the installation utility creates and edits the configuration file/etc/default/vxdg on each system such that the default activation mode of shareddisk groups is set to “off” when the cluster starts. This setting prevents uncontrolledaccess to the shared storage. Later, after you create shared disk groups and volumes forthe Oracle9i binaries and database and configure them in the Oracle or CVM servicegroups, VCS automatically controls access to the volumes using the CVMVolDg agent.

However, so that you can now create databases on the shared volumes, you must enablewrite access to them. You can use the following commands:

1. On the CVM master node, enter:

# vxdg -s import shared_disk_group# vxvol -g shared_disk_group startall# vxdg -g shared_disk_group set activation=sw

2. On the slave node, enter:

# vxdg -g shared_disk_group set activation=sw


Creating Shared Disk Group and Volume for SRVM

Refer to the description of disk group activation modes in the VERITAS Volume ManagerAdministrator’s Guide for more information (see the chapter on “Cluster Functionality”).

Note For VERITAS DBE/AC for Oracle9i RAC, release 3.5, a shared disk group is notautomatically configured for I/O fencing when it is created. A newly created diskgroup must be deported and re-imported to be configured with I/O fencing. Youcan use vxdg commands to accomplish this; refer to VERITAS Volume Manager 3.5User’s Guide - VERITAS Enterprise Administrator.

Creating Shared Disk Group and Volume for SRVMOn the shared storage device, you must create a shared raw volume for use by the OracleSRVM component, srvconfig. Oracle requires the SRVM component when the Oracledatabase is set up on shared storage. This volume requires at least 200 megabytes.

For example, to create the volume srvm_vol of 300 megabytes in a shared disk grouporasrv_dg on disk c2t3d1, do the following:

1. From the master node, create the shared disk group on the shared disk c2t3d1:

# vxdg -s init orasrv_dg c2t3d1

2. Create the volume in the shared disk group:

# vxassist -g orasrv_dg make srvm_vol 300M

Later, when you install the Oracle9i software, you must provide the name of thisvolume. For example, you would enter /dev/vx/rdsk/orasrv_dg/srvm_vol.

3. Set the activation mode to rw:

# vxdg -g orasrv_dg set activation=sw

4. On the other node, enter:

# vxdg -g orasrv_dg set activation=sw

Chapter 3, Installing Oracle9i Software 53

Installing Oracle9i Release 1 Software

Installing Oracle9i Release 1 SoftwareYou can install Oracle9i Release 1 on a shared disk or on each nodes local hard disk.

Installing Oracle9i Release 1 on Shared DiskThe following procedure describes installing Oracle9i binaries on a cluster file system. Ifyou intend to install Oracle9i binaries on each system locally, refer to “Alternate Method -Installing Oracle9i Release 1 Locally” on page 58.

Note If you are installing Oracle9i Release 1 Patch software, you must first install Oracle9iRelease 1 software. Then use the procedure in “Installing Oracle9i Release 1 PatchSoftware” on page 61.

1. Log in as root user.

2. Add the directory path to the jar utility in the PATH environment variable. Typically,this is /usr/bin. Do this on both nodes.

3. On one node, create a shared disk group:

# vxdg -s init orabinvol_dg c2t3d1s2

4. Create the volume in the shared group:

# vxassist -g orabinvol_dg make orabinvol 5000M

For the Oracle9i Release 1 binaries, make the volume 5,000 MB.

5. Set the activation mode on one system:

# vxdg -g orabinvol_dg set activation=sw



7. On one node, create a VxFS file system on the shared volume on which to install theOracle9i binaries. For example, create the file system on orabinvol:

# mkfs -F vxfs /dev/vx/rdsk/orbinvol_dg/orabinvol



8. On both systems, create the mount point for the file system:

# mkdir /oracle

9. On both systems, mount the file system, using the device file for the block device:

# mount -F vxfs -o cluster /dev/vx/dsk/orbinvol_dg/orabinvol/oracle

10. On both systems, create a local group and a local user for Oracle. For example, createthe group dba and the user oracle. Be sure to assign the same user ID and group IDfor the user on each system.

11. Set the home directory for the oracle user as /oracle.

12. On both systems, assign ownership of the Oracle directory to oracle:

# chown oracle:dba /oracle/*

13. On one system, create the directory where you intend to copy the contents of theOracle9i software CDs. A total of 2 gigabytes is required.

# mkdir /install# cd /install

14. Copy all the files from the three Oracle9i CDs. For example:

a. Insert the first of the three Oracle9i disks in the CD-ROM drive.

- If you are running Solaris volume-management software, the softwareautomatically mounts the CD as /cdrom/cdrom0. Type the command:

# cd /cdrom/cdrom0

- If you are not running Solaris volume-management software, you mustmount the CD manually. For example:

# mount -F hsfs -o ro /dev/dsk/c0t6d0s2 /cdrom

In this example, /dev/dsk/c0t6d0s2 is the name for the CD drive.

# cd /cdrom

b. Copy the files to their appropriate locations.

# cp -R Disk1 /install

c. Repeat step a and step b for the remaining Oracle9i software CDs, copying thefiles from the second CD into /install/Disk2 and the files from the third CDinto /install/Disk3.



15. Run the VERITAS DBE/AC for Oracle9i RAC script 9iinstall:

# /opt/VRTSvcs/rac/lib/9iinstall

The script prompts you to specify whether the Oracle you are installing is 32-bit or64-bit.

Enter Oracle bits 32 or 6464

Enter the directory location where you copied the contents of the Oracle9i installationCDs. For this example, you would enter /install:

Please enter Oracle Installation copy location./install

The script modifies some of the copied Oracle9i files to be compatible with VERITASDBE/AC for Oracle9i.

16. Log in as oracle.

17. On one system, and create a directory for the installation of the Oracle9i binaries. Forexample:

$ mkdir VRT$ export ORACLE_HOME=/oracle/VRT

18. On both systems, edit the file .rhosts to provide the other system access to the localsystem during the installation. Place a “+” character in the first line of the file. Notethat you can remove this permission after installation is complete.

19. Set the DISPLAY variable.


$ DISPLAY=host:0.0;export DISPLAY


$ setenv DISPLAY host:0.0

20. Run the Oracle9i utility runInstaller. The utility is located in the directory whereyou copied Oracle9i software from Disk1.

$ /install/Disk1/runInstaller

- As the utility starts up, be sure to select the installation option: Software Only.Refer to the Oracle9i Installation Guide for additional information about runningthe utility.

- When the Node Selection dialog box appears, select only the local node.



Note If the Node Selection dialog box does not appear, refer to “Missing Dialog BoxDuring Installation of Oracle9i Release 1” on page 98.

- As the installer runs, it prompts you to provide the name of the shared volumeyou created previously for the Oracle SRVM component, srvconfig. In thisexample, we would enter /dev/vx/rdsk/orasrv_dg/srvm_vol.See the“Creating Shared Disk Group and Volume for SRVM” on page 53.

21. Oracle requires the global services daemon (gsd) to be running before any serverutilities can be started. When installation completes, run the gsd.sh script in thebackground on both systems as user oracle. For example, where $ORACLE_HOMEequals /oracle/VRT, enter:

$ /oracle/VRT/bin/gsd.sh &

22. Log in as root.

23. Copy /var/opt/oracle from the installation node to the other node:

# rcp -r /var/opt/oracle sysb:/var/opt



Alternate Method - Installing Oracle9i Release 1 LocallyThe following procedure describes installing Oracle9i binaries on the local hard disk oneach system. If you intend to install Oracle9i binaries on a shared disk, refer to “InstallingOracle9i Release 1 on Shared Disk” on page 54.

1. Log in as root user on the first system.

2. Add the directory path to the jar utility in the PATH environment variable. Typically,this is /usr/bin.


# vxdg init or_dg c0d1t1s4


# vxassist -g or_dg make or_vol 5000M


5. Create a VxFS file system on orabinvol to install the Oracle9i binaries. For example:

# mkfs -F vxfs /dev/vx/rdsk/or_dg/or_vol

6. Create the mount point for the file system:

# mkdir /oracle

7. Mount the file system, using the device file for the block device:

# mount -F vxfs /dev/vx/rdsk/or_dg/or_vol /oracle

8. Edit the /etc/vfstab file, list the new file system, and specify “yes” for the “mountat boot” choice. For example:

#device device mount FS fsck mount mount#to mount to fsck point type pass at boot options#../dev/vx/rdsk/or_dg/or_vol /dev/vx/rdsk/or_dg/or_vol /oracle vxfs 1yes -

9. Create a local group and a local user for Oracle. For example, create the group dbaand the user oracle. Be sure to assign the same user ID and group ID for the user oneach system.




11. Assign ownership of the Oracle directory to oracle:


12. Repeat step 1 through step 11 on the other system.

13. On one system, create the directory where you intend to copy the contents of theOracle9i software CDs. A total of 2 gigabytes is required.

# mkdir /install# cd /install

14. From one system, copy all the files from the three Oracle9i CDs. For example:

a. Insert the first of the three Oracle9i disks in the CD-ROM drive.


# cd /cdrom/cdrom0

- If you are not running Solaris volume-management software, you mustmount the CD manually. For example:



# cd /cdrom

b. Copy the files to their appropriate locations.

# cp -R Disk1 /install

c. Repeat step a and step b for the remaining Oracle9i software CDs, copying thefiles from the second CD into /install/Disk2 and the files from the third CDinto /install/Disk3.

15. Run the VERITAS DBE/AC for Oracle9i RAC script 9iinstall:

# /opt/VRTSvcs/rac/lib/9iinstall

The script prompts you to specify the Oracle you are installing as 64-bit or 32-bit.


Enter the directory location where you copied the contents of the Oracle9i installationCDs. For this example, you would enter /install:

Please enter Oracle Installation copy location./install



The script modifies some of the copied Oracle9i files to be compatible with VERITASDBE/AC for Oracle9i.


17. On both systems, as user oracle, create a directory to install the Oracle9i binaries:








20. Run the Oracle9i utility runInstaller. The utility is located in the directory whereyou copied Oracle9i software from Disk1.

$ /install/Disk1/runInstaller

- As the utility starts, select the installation option: Software Only. Refer to theOracle9i Installation Guide for additional information about running the utility.

- When the Node Selection dialog box appears, select both nodes for installation.



21. Oracle requires the global services daemon (gsd) to be running before any serverutilities can be started. When installation completes, run the gsd.sh script in thebackground on both systems as user oracle. For example, where $ORACLE_HOMEequals /oracle/VRT, enter:

$ /oracle/VRT/bin/gsd.sh &


Installing Oracle9i Release 1 Patch Software

Installing Oracle9i Release 1 Patch SoftwareTo install Oracle9i Patch 2 or Patch 3 Software, you must have installed Oracle9i Release 1software. Before you install the Oracle patch, review the Patch Set Notes that accompanyOracle 9i Data Server 9.0.1 Patch Set for Sun SPARC Solaris for additional instructionsinstalling the patch set and on performing the post install actions.

1. Log in as superuser.

2. Add the directory path to the jar utility in the PATH environment variable. Typicallythis is /usr/bin.

3. On one system, create the directory where you intend to copy the Oracle9i Patchsoftware.

# mkdir /install/patch# cd /install/patch

4. Copy all the files included with the downloaded Oracle9i patch software to/install/patch. When you uncompress and untar the downloaded ZIP file, thesoftware is placed in a directory Disk1.

5. Log in as superuser; you may log into another window.

6. Depending on the level of patch you are installing, run the applicable VERITASDBE/AC for Oracle9i RAC script.

For Oracle9i Patch 2, enter:

# /opt/VRTSvcs/rac/lib/9iP2install

For Oracle9i Patch 3, enter:

# /opt/VRTSvcs/rac/lib/9iP3install

7. The script prompts you to specify whether the Oracle you are installing is 32-bit or64-bit.


8. The script prompts you for the location of the copied Oracle9i Patch files. For thisexample, you would enter the directory /install/patch:

Please enter Oracle patch Installation copy location./install/patch

As the script proceeds, it modifies some of the copied Oracle9i Patch files to becompatible with VERITAS DBE/AC for Oracle9i.


Installing Oracle9i Release 1 Patch Software








12. Run the Oracle9i utility runInstaller:

$ $ORACLE_BASE/oui/install/runInstaller

13. As the runInstaller utility starts up, be sure to:

- Select product.jar from the directory /install/patch/Disk1/stage1.

- Select both nodes for installation when the Node Selection dialog box appears.

14. Permit the installation to continue.

15. To perform post install actions, refer to the Patch Set Notes that accompany the patch.



Installing Oracle9i Release 2 SoftwareYou can install Oracle9i Release 2 on a shared disk or on each nodes local hard disk.

Installing Oracle9i Release 2 on Shared DiskThe following procedure describes installing the Oracle9i Release 2 Software in a RACenvironment. The procedure installs the software on shared storage:


2. Add the directory path to the jar utility in the PATH environment variable. Typically,this is /usr/bin. Do this on both nodes.

3. On the master node, create a shared disk group:

# vxdg init orabinvol_dg c2t3d1s2


# vxassist -g orabinvol_dg make orabinvol 5000M

For the Oracle9i Release2 binaries, make the volume 5,000 MB.

5. Deport and import the disk group and set the activation:

# vxdg deport orabinvol_dg# vxdg -s import orabinvol_dg# vxvol -g orabinvol_dg startall# vxdg -g orabinvol_dg set activation=sw



7. On the master node, create a VxFS file system on the shared volume on which toinstall the Oracle9i binaries. For example, create the file system on orabinvol:

# mkfs -F vxfs /dev/vx/rdsk/orbinvol_dg/orabinvol

8. On both systems, create the mount point for the file system:

# mkdir /oracle

9. On both systems, mount the file system, using the device file for the block device:

# mount -F vxfs -o cluster /dev/vx/dsk/orbinvol_dg/orabinvol/oracle



10. On both systems, create a local group and a local user for Oracle. For example, createthe group dba and the user oracle. Be sure to assign the same user ID and group IDfor the user on each system.

11. On both systems, set the home directory for the oracle user as /oracle.

12. On both systems, assign ownership of the Oracle directory to oracle:


13. On one system, enter the following commands, using the appropriate example belowto copy the VERITAS CM library based on the version of Oracle9i:

# cd /opt/ORCLcluster/lib/9iR2

If the version of Oracle9i is 32-bit, enter:

# cp libskgxn2_32.so ../libskgxn2.so



14. On the first system, insert Disk1 of the Oracle9i disks in the CD-ROM drive.


# cd /cdrom/cdrom0

- If you are not running Solaris volume-management software, you must mountthe CD manually. For example:



# cd /cdrom

15. Log in as oracle on both systems.

16. On the first system, edit the file .rhosts to provide the other system access to thelocal system during the installation. Place a “+” character in the first line of the file.Note that you can remove this permission after installation is complete.

17. Repeat step 16 on the other system.

18. On both systems, create a directory for the installation of the Oracle9i binaries:




19. On the first system, set the DISPLAY variable.





20. On the first system, run the Oracle9i utility runInstaller.

$ /cdrom/Disk1/runInstaller

- As the utility starts up, be sure to select the installation option: Software Only.Refer to the Oracle9i Installation Guide for additional information about runningthe utility.




- When you are prompted for other Oracle9i disks, refer to step 14 if necessary. Youmay need to log in as root to manually mount the CDs and log back in as oracleto continue.

21. On each system, copy the relevant VCSIPC library to $ORACLE_HOME/lib.

For 32 bit Oracle:

$ cp /opt/ORCLcluster/lib/9iR2/libskgxp92_32.so$ORACLE_HOME/lib/libskgxp9.so

For 64 bit Oracle:




Alternate Method - Installing Oracle9i Release 2 LocallyUse this procedure to install Oracle9i Release 2 on each system locally in a VERITASDBE/AC for Oracle9i RAC environment.

1. Log in as root user on one system.

2. Add the directory path to the jar utility in the PATH environment variable. Typically,this is /usr/bin.


# vxdg init or_dg c0d1t1s4


# vxassist -g or_dg make or_vol 5000M


5. Create a VxFS file system on orabinvol to install the Oracle9i binaries. For example:

# mkfs -F vxfs /dev/vx/rdsk/or_dg/or_vol

6. Create the mount point for the file system:

# mkdir /oracle

7. Mount the file system, using the device file for the block device:

# mount -F vxfs /dev/vx/rdsk/or_dg/or_vol /oracle

8. Edit the /etc/vfstab file, list the new file system, and specify “yes” for the “mountat boot” choice. For example:

#device device mount FS fsck mount mount#to mount to fsck point type pass at boot options#../dev/vx/rdsk/or_dg/or_vol /dev/vx/rdsk/or_dg/or_vol /oracle vxfs 1yes -

9. Create a local group and a local user for Oracle. For example, create the group dbaand the user oracle. Be sure to assign the same user ID and group ID for the user oneach system.




11. Assign ownership of the Oracle directory to oracle:



13. On one system, enter the following commands, using the appropriate example belowto copy the VERITAS CM library based on the version of Oracle9i:






14. On the first system, insert Disk1 of the Oracle9i disks in the CD-ROM drive.


# cd /cdrom/cdrom0

- If you are not running Solaris volume-management software, you must mountthe CD manually. For example:



# cd /cdrom

15. Log in as oracle on both systems.

16. On the first system, edit the file .rhosts to provide the other system access to thelocal system during the installation. Place a “+” character in the first line of the file.Note that you can remove this permission after installation is complete.

17. Repeat step 16 on the other system.

18. On both systems, create a directory for the installation of the Oracle9i binaries:


19. On the first system, set the DISPLAY variable.







20. On the first system, run the Oracle9i utility runInstaller.

$ /cdrom/Disk1/runInstaller

- As the utility starts up, be sure to select the installation option: Software Only.Refer to the Oracle9i Installation Guide for addition information about running theutility.




- When you are prompted for other Oracle9i disks, refer to step 14 if necessary. Youmay need to log in as root to manually mount the CDs and log back in as oracleto continue.

21. On each system, copy the relevant VCSIPC library to $ORACLE_HOME/lib.

For 32 bit Oracle:


For 64 bit Oracle:



Removing Temporary rsh Access Permissions

Removing Temporary rsh Access PermissionsWhen Oracle9i installation is complete, remove the temporary rsh access permissions youhave set for the systems in the cluster. For example, if you added a “+” character in thefirst line of the .rhosts file, remove that line.

Linking VERITAS ODM Libraries to OracleAfter installing Oracle, create a link to the VERITAS ODM libraries on both systems.

1. Log in as the user oracle.

2. Rename the file libodm9.so:

$ cd $ORACLE_HOME/lib$ mv libodm9.so libodm9.so.old

3. Depending on whether you are using the 32-bit or the 64-bit Oracle version, link thelibraries using one of the following commands:

For 32-bit versions:

$ ln -s /usr/lib/libodm.so libodm9.so

For 64-bit versions:

$ ln -s /usr/lib/sparcv9/libodm.so libodm9.so


Later, after starting Oracle instances, you can confirm that Oracle is using the VERITASODM libraries by examining the Oracle alert file, alert_$ORACLE_SID.log. Look forthe line that reads:

Oracle instance running with ODM: VERITAS 3.5 ODM Library, Version 1.1

Creating DatabasesAt this time, you can an create Oracle database on shared storage. Use you own tools orrefer to “Creating Starter Databases” on page 133 to create a starter database.


Creating Databases


Configuring VCS Service Groups for Oracle
4 This chapter describes how to modify the VCS configuration file, main.cf, to define theservice groups for CVM and for Oracle databases in a VCS configuration within a RACenvironment.
Checking cluster_database Flag in Oracle9i Parameter FileOn each system, confirm that the “cluster_database” flag is set in the Oracle9iparameter file $ORACLE_HOME/dbs/init$ORACLE_SID.ora. This flag enables theOracle instances to run in parallel. Verify the file contains the line:

cluster_database = true

Configuring CVM and Oracle Groups in main.cf FileDuring the installation of DBE/AC for Oracle9i RAC, a rudimentary CVM service group isautomatically defined in the VCS configuration file, main.cf. This CVM service groupincludes the CVMCluster resource, which enables Oracle RAC communications betweenthe nodes of the cluster, and the CFSQlogckd and CFSfsckd daemons that control thetransmission of file system data within the cluster. Refer to the “Example VCSConfiguration File After Installation” on page 47.

After installing Oracle9i and creating a database, you can modify the CVM service groupto include the Oracle listener process, the IP and NIC resources used by the listenerprocess, and the CFSMount and CVMVolDg resources. The NIC and IP resources are usedby the listener process to communicate with clients via the public network. TheCFSMount and CVMVolDg resources are for the Oracle binaries installed on sharedstorage. The modification of the CVM service group is discussed in “Modifying the CVMService Group in the main.cf File” on page 73.

Then you can add the Oracle database service group. This group typically consists of anOracle database created on shared storage, and the CFSMount and CVMVolDg resourcesused by the database. There may be multiple Oracle database service groups, one for eachdatabase. See “Adding Oracle Resource Groups in the main.cf File” on page 76.

71

Configuring CVM and Oracle Groups in main.cf File

VCS Service Groups for RAC: Dependencies ChartThe following illustration shows the dependencies among the resources in a typical RACenvironment. The illustration shows two Oracle service groups and one CVM servicegroup. The dependencies among the resources within a group are shown as well as thedependency that each Oracle group has on the CVM group. This chapter describes how tospecify the resources.

Sqlnet

Oracle

IP

NIC

LISTENER

VRT_db

listener_ip

listener_hme_0

Sqlnet

CVMVolDg

CFSMount

Oracle

rac_db

CVMVolDg

CFSMount

oradb1_grp oradb2_grp

CFSMount

oradb1_mnt

oradb1_voldg

oradb2_mnt

oradb2_voldg

orabin_mnt

orabin_voldg

CVMVolDg

CVMCluster

cvm_clus

CFSfsckd

CFSQlogckd

vxfsckd

qlogckd

cvm



Setting Attributes as Local for the CVM and Oracle GroupsThe CVM and Oracle service groups must be configured as “parallel,” with certainattributes of key resources defined locally, that is, to reflect the specific system on whichthey run. The attributes that must be defined locally are described in a table; see“Attributes of CVM and Oracle Groups that Must be Defined as Local” on page 78.

Modifying the CVM Service Group in the main.cf FileThe cvm service group is created during the installation of DBE/AC for Oracle9i RAC.After installation, the main.cf file resembles the example shown in “Example VCSConfiguration File After Installation” on page 47. Because Oracle9i had not been installed,thecvm service group includes only resources for the CFSQlogckd and CFSfsckddaemons and the CVMCluster resource.

Adding Sqlnet, NIC, IP CVMVolDg, and CFSMount Resources

You must modify the cvm service group to add the Sqlnet, NIC, IP, CVMVolDg, andCFSMount resources to the configuration. You can refer to “Sample main.cf File” onpage 109 to see a complete example of how a cvm group is configured.

When configuring these resources, use the following procedure (key lines of theconfiguration file are shown in bold font):

1. Make sure the cvm group has the group Parallel attribute set to 1. Typically this isalready done during installation.

.group cvm (

SystemList = { galaxy = 0, nebula = 1 } AutoFailOver = 0

Parallel = 1 AutoStartList = { galaxy, nebula } )..

Chapter 4, Configuring VCS Service Groups for Oracle 73


2. Define the NIC and IP resources. The VCS bundled NIC and IP agents are describedin VERITAS Cluster Server Bundled Agents Reference Guide. The device name and the IPaddresses are required by the listener for public network communication. Note thatfor the IP resource, the Address attribute is localized for each node (see “Attributesof CVM and Oracle Groups that Must be Defined as Local” on page 78).

. NIC listener_hme0 (

Device = hme0 NetworkType = ether )

IP listener_ip ( Device = hme0

Address @galaxy = "192.2.40.21" Address @nebula = "192.2.40.22" ) .

3. Define the Sqlnet resource. The Sqlnet listener agent is described in detail in theVERITAS Cluster Server Enterprise Agent for Oracle, Installation and Configuration Guide.Note that the listener attribute is localized for each node.

. Sqlnet LISTENER ( Owner = oracle Home = "/oracle/orahome" TnsAdmin = "/oracle/orahome/network/admin" MonScript = "./bin/Sqlnet/LsnrTest.pl"

Listener @galaxy = LISTENER_a Listener @nebula = LISTENER_b EnvFile = "/opt/VRTSvcs/bin/Sqlnet/envfile" ).

4. You must configure the CVMVolDg and CFSMount resources in the cvm group for theOracle binaries installed on shared storage. Refer to the appendix “CVMCluster,CVMVolDg, and CFSMount Agents” on page 115 for description of CVMVolDg andCFSMount agents.

. CVMVolDg orabin_voldg ( CVMDiskGroup = orabindg CVMVolume = { "orabinvol", "srvmvol" } CVMActivation = sw )



CFSMount orabin_mnt ( Critical = 0 MountPoint = "/oracle" BlockDevice = "/dev/vx/dsk/orabindg/orabinvol" Primary = galaxy )..

5. Define the dependencies of resources in the group. The dependencies are specifiedsuch that the Sqlnet resource requires the IP resource that, in turn, depends on theNIC resource. The Sqlnet resource also requires the CFSMount resource. TheCFSMount resource requires the daemons, vxfsckd and qlogckd, used by the clusterfile system. The CFSMount resource also depends on the CVMVolDg resource, which,in turn, requires the CVMCluster resource for communications within the cluster. The“VCS Service Groups for RAC: Dependencies Chart” on page 72 visually shows thedependencies specified in the following statements:

.

. vxfsckd requires qlogckd orabin_voldg requires cvm_clus orabin_mnt requires vxfsckd orabin_mnt requires orabin_voldg listener_ip requires listener_hme0 LISTENER requires listener_ip LISTENER requires orabin_mnt..



Adding Oracle Resource Groups in the main.cf FileFor a complete description of the VCS Oracle enterprise agent, refer to the document,VERITAS Cluster Server Enterprise Agent for Oracle, Installation and Configuration Guide. Thatdocument includes instructions for configuring the Oracle and Sqlnet agents.

Note The VCS Enterprise Agent for Oracle version 2.0.1 is installed when you runinstallDBAC. When you refer to the VERITAS Cluster Server Enterprise Agent forOracle Installation and Configuration Guide, ignore the steps described in the section“Installing the Agent Software.”

1. Using the “Sample main.cf File” on page 109 as an example, add a service group tocontain the resources for an Oracle database. For example, add the grouporadb1_grp. Make sure you assign the Parallel attribute a value of 1.

.group oradb1_grp (

SystemList = { galaxy = 0, nebula = 1 } AutoFailOver = 1

Parallel = 1 AutoStartList = { galaxy, nebula } )..

2. Create the CVMVolDg and CFSMount resource definitions. See “CVMCluster,CVMVolDg, and CFSMount Agents” on page 115 for a description of these agents andtheir attributes.

.

. CVMVolDg oradb1_voldg (

CVMDiskGroup = oradb1dg CVMVolume = { "oradb1vol" } CVMActivation = sw )

CFSMount oradb1_mnt ( MountPoint = "/oradb1" BlockDevice = "/dev/vx/dsk/oradb1dg/oradb1vol" Primary = galaxy

)..



3. Define the Oracle database resource. Refer to the VERITAS Cluster Server EnterpriseAgent for Oracle, Installation and Configuration Guide for information on the VCSenterprise agent for Oracle. Note that the Oracle attributes Sid, Pfile, and Tableattributes must be set locally, that is, they must be defined for each cluster system.

. Oracle VRT_db (

Sid @galaxy = VRT1 Sid @nebula = VRT2 Owner = oracle Home = "/oracle/orahome"

Pfile @galaxy = "/oracle/orahome/dbs/initVRT1.ora" Pfile @nebula = "/oracle/orahome/dbs/initVRT2.ora" User = scott Pword = tiger

Table @galaxy = vcstable_galaxy Table @nebula = vcstable_nebula MonScript = "./bin/Oracle/SqlTest.pl" AutoEndBkup = 1 EnvFile = "/opt/VRTSvcs/bin/Oracle/envfile" ) .

4. Define the dependencies for the Oracle service group. Note that the Oracle databasegroup is specified to require the cvm group, and that the required dependency isdefined as “online local firm,” meaning that the cvm group must be online andremain online on a system before the Oracle group can come online on the samesystem. Refer to the VERITAS Cluster Server User’s Guide for a description of groupdependencies.

Refer to “VCS Service Groups for RAC: Dependencies Chart” on page 72.

.

. requires group cvm online local firm oradb1_mnt requires oradb1_voldg VRT requires oradb1_mnt..

See the “Sample main.cf File” on page 109 for a complete example. You can also find thecomplete file in /etc/VRTSvcs/conf/sample_rac/main.cf.

Additional RAC Processes Monitored by the VCS Oracle Agent

For shallow monitoring, the VCS Oracle agent monitors the ora_lmon and ora_lmdprocesses in addition to the ora_dbw, ora_smon, ora_pmon, and ora_lgwr processes.



Attributes of CVM and Oracle Groups that Must be Defined asLocal

The following table lists attributes that must be defined as local for the CVM and Oracleservice groups (note that each attribute has string-scalar as the type and dimension).

If other attributes for the Oracle resource differ for various RAC instances, define themlocally as well. These other attributes may include the Oracle resource attributes User,PWord, the CVMVolDg resource attribute CVMActivation, and others.

Resource Attribute Definition

IP Address The virtual IP address (not the base IP address) associated with theinterface. For example:

Address @sysa = "192.2.40.21"

Address @sysb = "192.2.40.22"

Sqlnet Listener The name of the Listener. For example:

Listener @sysa = LISTENER_a

Listener @sysb = LISTENER_b

Oracle Sid The variable $ORACLE_SID represents the Oracle system ID. Forexample, if the SIDs for two systems, sysa and sysb, are VRT1 andVRT2 respectively, their definitions would be:

Sid @sysa = VRT1

Sid @sysb = VRT2

Oracle Pfile The parameter file: $ORACLE_HOME/dbs/init$ORACLE_SID.ora.

For example:

Pfile @sysa = "/oracle/VRT/dbs/initVRT1.ora"

Pfile @sysb = "/oracle/VRT/dbs/initVRT2.ora"

Oracle Table The table used for in-depth monitoring by User/PWord on each clusternode. For example:

Table @sysa = vcstable_sysa

Table @sysb = vcstable_sysb

Using the same table on all Oracle instances is not recommended. Usingthe same table generates IPC traffic and could cause conflicts betweenthe Oracle recovery processes and the agent monitoring processes inaccessing the table.

Note Table is only required if in-depth monitoring is used. If thePWord varies by RAC instance, it must also be defined as local.



Modifying the VCS ConfigurationFor additional information and instructions on modifying the VCS configuration, refer tothe VERITAS Cluster Server User’s Guide.

Location of VCS and Oracle Agent Log FilesOn all cluster nodes, look at the following log files for any errors or status messages:

/var/VRTSvcs/log/engine_A.log/var/VRTSvcs/log/Oracle_A.log/var/VRTSvcs/log/Sqlnet_A.log




I/O Fencing Scenarios
5 I/O Fencing of Shared Storage
When two systems have access to the data on shared storage, the integrity of the datadepends on the systems communicating with each other so that each is aware when theother is writing data. Usually this communication occurs in the form of heartbeatsthrough the private networks between the systems. If the private links are lost, or even ifone of the systems is hung or too busy to send or receive heartbeats, each system could beunaware of the other’s activities with respect to writing data. This is a split brain conditionand can lead to data corruption.

The I/O fencing capability of the DBE/AC, managed by VERITAS Volume Manager,prevents data corruption in the event of a split brain condition by using SCSI-3 persistentreservations for disks. This allows a set of systems to have registrations with the disk anda write-exclusive registrants-only reservation with the disk containing the data. Thismeans that only these systems can read and write to the disk, while any other system canonly read the disk. The I/O fencing feature fences out a system that no longer sendsheartbeats to the other system by preventing it from writing data to the disk.

The VxVM manages all shared storage subject to I/O fencing. It assigns the keys thatsystems use for registrations and reservations for the disks—including all paths—in thespecified disk groups. The vxfen driver is aware of which systems have registrations andreservations with specific disks.

To protect the data on shared disks, each system in the cluster must be configured to useI/O fencing.

The Role of Coordinator DisksThe vxfen driver, which is installed on each system, uses coordinator disks to resolvepotential split brain conditions. If a split brain condition occurs, systems “race” for thecoordinator disks to gain control of them. The winning system remains in the cluster. Thedriver ejects systems that lose the race and fences them off from the shared storage.Because they lose their reservations for any data disks, ejected systems cannot write tothem and corrupt the data. If a system realizes that it has been ejected, it removes itselffrom the cluster. See “Setting Up Coordinator Disks” on page 44.

81

I/O Fencing of Shared Storage

How I/O Fencing Works in Different Event ScenariosThe following table describes how I/O fencing functions to prevent data corruption indifferent failure event scenarios. For each event, corrective operator actions are indicated.

Event Node A: What Happens? Node B: What Happens? Operator Action

Both privatenetworks fail.

Node A races for majority ofcoordinator disks.

If Node A wins race forcoordinator disks, Node Aejects Node B from theshared disks and continues.

Node B races for majority ofcoordinator disks.

If Node B loses the race forthe coordinator disks, NodeB removes itself from thecluster.

When Node B isejected from cluster,repair the privatenetworks beforeattempting to bringNode B back.

Both privatenetworksfunction againafter eventabove.

Node A continues to work. Node B has crashed. Itcannot start the databasesince it is unable to write tothe data disks.

Reboot Node Bafter privatenetworks arerestored.

One privatenetwork fails.

Node A prints messageabout an IOFENCE on theconsole but continues.

Node B prints messageabout an IOFENCE on theconsole but continues.

Repair privatenetwork. Afternetwork is repaired,both nodesautomatically use it.

Node A hangs. Node A is extremely busyfor some reason or is in thekernel debugger.

When Node A is no longerhung or in the kerneldebugger, any queuedwrites to the data disks failbecause Node A is ejected.When Node A receivesmessage from GAB aboutbeing ejected, it removesitself from the cluster.

Node B loses heartbeatswith Node A, and races fora majority of coordinatordisks.

Node B wins race forcoordinator disks and ejectsNode A from shared datadisks.

Verify privatenetworks functionand reboot Node A.



Nodes A and Band privatenetworks losepower.Coordinatorand data disksretain power.

Power returnsto nodes andthey reboot,but privatenetworks stillhave no power.

Node A reboots and I/Ofencing driver (vxfen)detects Node B is registeredwith coordinator disks. Thedriver does not see Node Blisted as member of clusterbecause private networksare down. This causes theI/O fencing device driver toprevent Node A fromjoining the cluster. Node Aconsole displays:

Potentially apreexisting splitbrain. Droppingout of thecluster. Refer tothe userdocumentation forsteps required toclear preexistingsplit brain.

Node B reboots and I/Ofencing driver (vxfen)detects Node A is registeredwith coordinator disks. Thedriver does not see Node Alisted as member of clusterbecause private networksare down. This causes theI/O fencing device driver toprevent Node B fromjoining the cluster. Node Bconsole displays:


Refer to section inTroubleshootingchapter forinstructions onresolvingpreexisting splitbrain condition.


Chapter 5, I/O Fencing Scenarios 83


Node A crasheswhile Node Bis down. NodeB comes upand Node A isstill down. Node A is crashed. Node B reboots and detects

Node A is registered withthe coordinator disks. Thedriver does not see Node Alisted as member of thecluster. The I/O fencingdevice driver printsmessage on console:


Refer to section inTroubleshootingchapter forinstructions onresolvingpreexisting splitbrain condition.

The disk arraycontaining twoof the threecoordinatordisks ispowered off.

Node B leavesthe cluster andthe disk arrayis still poweredoff.

Node A continues tooperate as long as no nodesleave the cluster.

Node A races for a majorityof coordinator disks. NodeA fails because only one ofthree coordinator disks isavailable. Node A removesitself from the cluster.

Node B continues tooperate as long as no nodesleave the cluster.

Node B leaves the cluster.Power on faileddisk array andrestart I/O fencingdriver to enableNode A to registerwith all coordinatordisks.




The vxfenadm UtilityAdministrators can use the vxfenadm command to troubleshoot and test fencingconfigurations. The command’s options for use by administrators are:

-g - read and display keys

-i - read SCSI inquiry information from device

-m - register with disks

-n - make a reservation with disks

-p - remove registrations made by other systems

-r - read reservations

-x - remove registrations

Registration Key Formatting

The key defined by VxVM associated with a disk group consists of seven bytes maximum.This key becomes unique among the systems when the VxVM prefixes it with the ID ofthe system. The key used for I/O fencing, therefore, consists of eight bytes.

The keys currently assigned to disks can be displayed by using the vxfenadm command.For example, from the system with node ID 1, display the key for the disk/dev/rdsk/c2t1d0s2 by entering:

# vxfenadm -g /dev/rdsk/c2t1d0s2Reading SCSI Registration Keys...Device Name: /dev/rdsk/c2t1d0s2Total Number of Keys: 1key[0]:

Key Value [Numeric Format]: 65,80,71,82,48,48,48,48Key Value [Character Format]: APGR0000

The -g option of vxfenadm displays all eight bytes of a key value in two formats. In thenumeric format, the first byte, representing the Node ID, contains the system ID plus 65.The remaining bytes contain the ASCII values of the letters of the key, in this case,“PRG0000.” In the next line, the node ID 0 is expressed as “A;” node ID 1 would be “B.”

NodeID

VxVMDefined

VxVMDefined

VxVMDefined

VxVMDefined

VxVMDefined

VxVMDefined

VxVMDefined

0 7

Chapter 5, I/O Fencing Scenarios 85



Uninstalling DBE/AC for Oracle9i RAC
6 When removing the VERITAS DBE/AC for Oracle9i RAC software from the systems in thecluster, use the sequence shown in the flowchart below. Some of the activities require thatyou refer to other documents, namely, the VERITAS Volume Manager Installation Guide, theVERITAS File System Installation Guide, and the Oracle9i Installation Guide.
Offline Oracle & Sqlnet resources

Next page

Start deinstallation on one system

Oracleon CFS?

No

Yes

Anothersystem?

No

Yes

Anothersystem?

Yes

No

Remove database (optional)

RemoveOracle9i?

Yes

Remove binaries

Remove binaries

No

Offline Oracle service group

Anothersystem?

Yes

No

87

Remove optional packages (in order shown):pkgrm VRTSfspro VRTSvmpro VRTSobguiVRTSob VRTSfsdoc VRTSvmdoc VRTSvmman

Remove required packages (in order shown):pkgrm VRTScavf VRTSodm VRTSgmsVRTSglm VRTSvxfs

Encapsulated

No

Remove VCS: uninstallvcs

Anothersystem?

No

Yes

Done

Remove license & configuration files (optional)

Unmount VxFS file systems

Unconfigure CFS related drivers

root DG?Yes

Remove VxVM: pkgrm VRTSvxvm

Remove VxVM:Refer to VxVMInstall Guide

Anothersystem?

No

Yes

Stop VCS

Unconfigure I/O fencing driver vxfen


Offlining the Oracle and Sqlnet Resources

Offlining the Oracle and Sqlnet ResourcesOffline the Oracle and Sqlnet resources on each node. For example:

# hares -offline VRT -sys galaxy# hares -offline rac -sys galaxy# hares -offline VRT -sys nebula# hares -offline rac -sys nebula# hares -offline LISTENER -sys galaxy# hares -offline LISTENER -sys nebula

These commands stop the Oracle instances running on the specified systems.

Removing the Oracle Database (optional)You can remove the Oracle database after safely relocating the data as necessary.

Uninstalling Oracle9i (optional)To uninstall Oracle9i, use the runInstaller utility. You must run the utility on eachsystem if Oracle9i is installed locally. If Oracle9i is installed on a cluster file system, youneed only run the utility once.

1. On one system, log in as oracle, if necessary.






3. Run the Oracle9i utility runInstaller.

$ cd $ORACLE_BASE/oui/install$ runInstaller

As the utility starts up, be sure to select the option to uninstall the Oracle9i software.Refer to the Oracle9i Installation Guide for addition information about running theutility.

4. If necessary, remove Oracle9i from the other system.

Chapter 6, Uninstalling DBE/AC for Oracle9i RAC 89

Offlining the Oracle Service Groups

Offlining the Oracle Service GroupsOffline the Oracle service groups on each system:

# hagrp -offline oragrp -sys galaxy# hagrp -offline oradb_grp -sys galaxy# hagrp -offline oragrp -sys nebula# hagrp -offline oradb_grp -sys nebula

These commands take the Oracle resource groups, including all their configuredresources, offline.

Stopping VCSOn one node, stop VCS by entering:

# hastop -local

This command removes membership for ports h (VCS), v (CVM), and w (the vxconfigddaemon). Ports b (I/O fencing), d (ODM), f (CFS), q (QIO), and o (VCSMM) still havemembership. Use the /sbin/gabconfig -a command to verify this.

You can stop VCS on the other node later. Refer to the flow diagram at the beginning ofthis chapter.

Unmount Other VxFS File SystemsOther file VxFS file systems not under VCS control must be unmounted.

1. Determine file systems to be unmounted. You can check the /etc/mnttab file. Forexample, enter:

# cat /etc/mnttab | grep -i vxfs

The output shows each line for the file /etc/mnttab that contains an entry for avxfs file system.

2. By specifying its mount point, unmount each of the vxfs file systems listed in theoutput:

# umount mount_point


Unconfiguring the CFS Drivers

Unconfiguring the CFS DriversTo unconfigure the CFS drivers, run the following commands:

# Cd /opt/VRTSvcs/rac# ./uload_drv

The uload_drv script unconfigures the CFS related drivers, which use ports f, q, andd. You can use the /sbin/gabconfig -a command to verify they are no longerconfigured.

Unconfiguring the I/O Fencing DriverOn each system, unconfigure the I/O fencing driver, vxfen, by running the command:

# /etc/init.d/vxfen stop

Use the /sbin/gabconfig -a command to verify port b is no longer configured.

Removing Optional CFS and CVM PackagesFrom each system, remove the optional CFS and CVM packages.

1. Log in as root user on one of the systems.

2. Remove any optional packages from the system; use the order shown in the followingexample:

# pkgrm VRTSfspro VRTSvmpro VRTSobgui VRTSob VRTSfsdocVRTSvmdoc VRTSvmman

3. Repeat step 1 and step 2 on the other system to remove any of the optional packages.


Removing Required CFS and CVM Packages

Removing Required CFS and CVM PackagesFrom each system, remove the required CFS and CVM packages.

1. Log in as root user on one of the systems.

2. Remove the required packages from the system; use the order shown in the followingexample:

# pkgrm VRTScavf VRTSodm VRTSgms VRTSglm VRTSvxfs

3. Repeat step 1 and step 2 on the other system to remove the required packages.

Remove VxVM PackageIf the root file system resides in an encapsulated disk group, refer to the VERITAS VolumeManager Installation Guide for instructions on how to unconfigure VxVM.

Otherwise, you can use the following command on each system:

# pkgrm VRTSvxvm

Uninstalling VCSUse the uninstallvcs utility to remove all packages installed by the installvcsutility, which include VRTSperl, VRTSvcs, VRTSgab, VRTSllt, VRTSvcsag,VRTSvcsmg, VRTSvcsdc, VRTSdbac, and VRTSvcsor. The uninstallvcs utility alsounconfigures the drivers associated with VCS and VERITAS DBE/AC for Oracle9i RAC.

Before removing the packages from any system in the cluster, shutdown applications suchas the Java Console or any VCS enterprise agents that depend on VCS. Theuninstallvcs script supplied with VERITAS DBE/AC for Oracle9i RAC does notuninstall VCS enterprise agents other than the VCS Oracle enterprise agent. See thedocumentation for a specific enterprise agent for instructions on removing it.

Note Do not attempt to uninstall VCS if you are running another VERITAS applicationthat uses the VCS modules GAB or LLT. Refer to the documentation supplied withthat application for instructions on how to shut down the application beforeuninstalling VCS or any of its utilities.


Uninstalling VCS

1. Log in as root user, and at the system prompt, enter the following commands:

galaxy# cd /opt/VRTSvcs/installgalaxy# ./uninstallvcs

2. The script begins by discovering any VCS configuration files present on the systemand reporting cluster information from them:

VCS configuration files exist on this system with the followinginformation:

Cluster Name: vcs_racluster Cluster ID Number: 7 Systems: galaxy nebula Service Groups: ClusterService groupA groupB

Do you want to uninstall VCS from all of the systems in thiscluster?(Y)

The default is “Y.” If you answer “N,” the script prompts you to enter the names ofthe systems in the cluster from which you want to uninstall VCS.

Verifying communication with nebula......... rsh available

Communication check completed successfully

3. The script determines the current version of the VCS packages it finds to remove andindicates whether dependencies for them exist. If a package has dependencies, thedependent packages are listed and you are prompted to remove them.

Checking current installation on galaxy:

Checking product mode ............ VCS version 3.5 installed Checking VRTSvcsor ................. version 2.0.1 installed Checking VRTSvcsor dependencies ....................... none Checking VRTSvcsag ................... version 3.5 installed Checking VRTSvcsag dependencies ....................... none Checking VRTSvcsmg ................... version 3.5 installed Checking VRTSvcsmg dependencies ....................... none Checking VRTSvcs ..................... version 3.5 installed Checking VRTSvcs dependencies .................... VRTSvcsor

Do you want to uninstall VRTSvcsor which is dependent onpackage VRTSvcs? (Y)

Checking VRTSgab ..................... version 3.5 installed Checking VRTSgab dependencies ....... . .


Uninstalling VCS

.Checking current installation on nebula:

Checking product mode ............ VCS version 3.5 installed Checking VRTSvcsor ................... version 3.5 installed Checking VRTSvcsor dependencies ....................... none Checking VRTSvcsag ................... version 3.5 installed Checking VRTSvcsag dependencies ....................... none Checking VRTSvcsmg ................... version 3.5 installed Checking VRTSvcsmg dependencies ....................... none Checking VRTSvcs ..................... version 3.5 installed Checking ..... .

4. The script prompts you to remove existing VCS configuration files and the VCSlicense key. If you decide not to remove the configuration files, they remain in place.You must remove them or rename them before reinstalling VCS.

Do you want to remove the VCS configuration files and licensekey? (N) y

Do you want to remove the VCS logs and configuration filecopies? (N) y

Installation check completed successfully

5. The script proceeds to terminate VCS processes running on the cluster systems.

Terminating VCS processes on galaxy:

Checking VCS .................................... runningChecking ClusterService group ...........ONLINE on galaxyOfflining ClusterService Group on galaxy............ DoneStopping VCS on all systems ........................ DoneChecking CmdServer .............................. runningKilling CmdServer .................................. Done..

Terminating VCS processon nebula:

Checking VCS .................................not runningChecking CmdServer .............................. runningKilling CmdServer .................................. Done...

VCS process termination completed successfully


Removing License Files

6. The script proceeds to remove VCS packages from the cluster systems after checkingdependencies among them.

Uninstalling VCS on galaxy:

Removing VRTSvcsw package .......................... DoneRemoving VRTSvcsmn package ......................... DoneRemoving VRTSvcsmg package ......................... DoneRemoving VRTSvcsdc package ......................... Done...Removing VCS configuration files

Uninstalling VCS on nebula:Removing VRTSvcsw package .......................... Done...

VCS uninstallation completed successfully

Removing License Files

1. To see what license key files you have installed on a system, enter:

# /sbin/vxlicrep

The output lists the license keys and information about their respective products.

2. Go to the directory containing the license key files and list them. Enter:

# cd /etc/vx/licenses/lic# ls -a

3. Using the output from step 1, identify and delete unwanted key files listed in thestep 2.

Remove Other Configuration Files (Optional)You can remove the following configuration files:

/etc/vcsmmtab/etc/vxfentab/etc/vxfendg


Remove Other Configuration Files (Optional)


Troubleshooting
7 Running Scripts for Engineering Support Analysis
You can use a set of three scripts that gather information about the configuration andstatus of your cluster and its various modules. The scripts also identify packageinformation, debugging messages, console messages, and information about disk groupsand volumes. You can forward the output of each of these scripts to VERITAS customersupport who can analyze the information and assist you in solving any problems.

getdbacThis script gathers information about the VERITAS DBE/AC for Oracle9i RAC modules.Enter the following command on each system:

# /opt/VRTSvcs/bin/getdbac -local

The file /tmp/vcsopslog.time_stamp.sys_name.tar.Z contains the script’s output.

getcommsThis script gathers information about the GAB and LLT modules. On each system, enter:

# /opt/VRTSgab/getcomms -local

The file /tmp/commslog.time_stamp.tar contains the script’s output.

hagetcfThis script gathers information about the VCS cluster and the status of resources. To runthis script, enter the following command on each system:

# /opt/VRTSvcs/bin/hagetcf

The output from this script is placed in a tar file, /tmp/vcsconf.sys_name.tar,on eachcluster system.

97

Troubleshooting Topics

Troubleshooting TopicsThe following troubleshooting topics have headings that indicate likely symptoms or thatindicate procedures required for a solution.

Error When Starting an Oracle InstanceIf the VCSMM driver (the membership module) is not configured, an error is displayed onstarting the Oracle instance that resembles:

ORA-29702: error Occurred in Cluster Group Operation

To start the driver, enter the following command:

# /sbin/vcsmmconfig -c

The command included in the /etc/vcsmmtab file enables the VCSMM driver to bestarted at system boot.

Missing Dialog Box During Installation of Oracle9i Release 1During installation of Oracle9i Release 1 using the runInstaller utility, if you choosethe Enterprise Edition or Custom Install (with RAC option), a dialog box prompting youabout the installation nodes should appear.

If the dialog box fails to appear, run the following command to verify that the correctversion of the lsnodes command is installed:

/opt/VRTSvcs/rac/lib/install_files/lsnodes

If it does not display the nodes in the cluster, the RAC installation does not succeed.Oracle9i is installed as single instance as a result. Take the following corrective actions:

1. Make sure that the VCSMM driver is running on both the nodes and start it if is not:


2. Make sure you have started with a clean system. It is possible to have a version of thelsnodes command from another vendor. Delete the contents of directoriesoraInventory and $ORACLE_HOME, and the directory where you copied theOracle9i files from the Oracle installation CDs.

3. Restart the installation.



Missing Dialog Box During Installation of Oracle9i Release 2During installation of Oracle9i Release 2 using the runInstaller utility, if you choosethe Enterprise Edition or Custom Install (with RAC option), a dialog box prompting youabout the installation nodes should appear.

If the dialog box fails to appear, exit the runInstaller utility, and do the following:

1. On the first system, enter the following commands, using the appropriate examplebelow to copy the VERITAS CM library based on the version of Oracle9i:






2. Start the VCSMM driver on both the nodes by entering:


3. Restart the runInstaller utility.

Instance Numbers Must be Unique (Error Code 205)If you encounter error code 205 when the skgxnreg function fails (look in the Oracletrace files to find the error returned), make sure there is a unique instance numberspecified in the $ORACLE_HOME/dbs/init${ORACLE_SID}.ora file on each system.

ORACLE_SID Must be Unique (Error Code 304)If you encounter error code 304 when the skgxnreg function fails (look in the Oracletrace file to find the error returned), make sure that the ORACLE_SID environmentvariable specified during Oracle startup is unique on each system in your cluster. Also,make sure that the SID attribute for the Oracle resource in the main.cf is specifiedlocally and is unique.

Chapter 7, Troubleshooting 99


Oracle Log Files Show Shutdown Called Even When NotShutdown Manually

The Oracle enterprise agent calls shutdown if monitoring of the Oracle/Sqlnet resourcesfails for some reason. On all cluster nodes, look at the following VCS and Oracle agent logfiles for any errors or status:

/var/VRTSvcs/log/engine_A.log/var/VRTSvcs/log/Oracle_A.log

File System Configured Incorrectly for ODM Shuts DownOracle

Linking Oracle9i with the VERITAS ODM libraries provides the best file systemperformance. See “Linking VERITAS ODM Libraries to Oracle” on page 69 forinstructions on creating the link and confirming that Oracle uses the libraries. Shared filesystems in RAC clusters without ODM Libraries linked to Oracle9i may exhibit slowperformance and are not supported.

If ODM cannot find the resources it needs to provide support for cluster file systems, itdoes not allow Oracle to identify cluster files and causes Oracle to fail at startup. Run thefollowing command:

# cat /dev/odm/clustercluster status: enabled

If the status is “enabled,” ODM is supporting cluster files. Any other cluster statusindicates that ODM is not supporting cluster files. Other possible values include:

If /dev/odm is not mounted, no status can be reported.

pending ODM cannot yet communicate with its peers, but anticipates being able toeventually.

failed ODM cluster support has failed to initialize properly. Check console logs.

disabled ODM is not supporting cluster files. If you think it should, check:

- /dev/odm mount options in /etc/vfstab. If the “nocluster” option isbeing used, it can force the “disabled” cluster support state.

- Make sure the VRTSgms (group messaging service) package is installed.Run the following command:

# /opt/VRTSvcs/bin/chk_dbac_pkgs

The utility reports any missing required packages.



VCSIPC Errors in Oracle Trace/Log FilesIf you see any VCSIPC errors in the Oracle trace/log files, check /var/adm/messagesfor any LMX error messages. If you see messages that contain any of the following:

. . . out of buffers

. . . out of ports

. . . no minors available

Refer to “Tunable Kernel Driver Parameters” on page 123.

If you see any VCSIPC warning messages in Oracle trace/log files that resemble:

connection invalid

or,

Reporting communication error with node

check whether the Oracle Real Application Cluster instance on the other system is stillrunning or has been restarted. The warning message indicates that the VCSIPC/LMXconnection is no longer valid.

Shared Disk Group Cannot be ImportedIf you see a message resembling:

vxvm:vxconfigd:ERROR:vold_pgr_register(/dev/vx/rdmp/disk_name):local_node_id<0Please make sure that CVM and vxfen are configured and operating correctly

This message is displayed when CVM cannot retrieve the node ID of the local systemfrom the vxfen driver. This usually happens when port b is not configured. Verify thatthe vxfen driver is configured by checking the GAB ports with the command:

# /sbin/gabconfig -a

Port b must exist on the local system.



vxfentsthdw Fails When SCSI TEST UNIT READY CommandFails

If you see a message resembling:

Issuing SCSI TEST UNIT READY to disk reserved by other node FAILED.Contact the storage provider to have the hardware configurationfixed.

The disk array does not support returning success for a SCSI TEST UNIT READYcommand when another host has the disk reserved using SCSI-3 persistent reservations.This happens with Hitachi Data Systems 99XX arrays if bit 186 of the system mode optionis not enabled.

CVMVolDg Does Not Go Online Even Though CVMCluster isOnline

When the CVMCluster resource goes online, the shared disk groups are automaticallyimported. If the disk group import fails for some reason, the CVMVolDg resources fault.Clearing and offlining the CVMVolDg type resources does not fix the problem.

Workaround:

1. Fix the problem causing the import of the shared disk group to fail.

2. Offline the service group containing the resource of type CVMVolDg as well as theservice group containing the CVMCluster resource type.

3. Bring the service group containing the CVMCluster resource online.

4. Bring the service group containing the CVMVolDg resource online.

Restoring Communication Between Host and Disks AfterCable Disconnection

If a fiber cable is inadvertently disconnected between the host and a disk, you can restorecommunication between the host and the disk without rebooting by doing the following:

1. Reconnect the cable.

2. Use the format command to verify that the host sees the disks. It may take a fewminutes before the host is capable of seeing the disk.



3. Issue the following vxdctl command to force the VxVM configuration daemonvxconfigd to rescan the disks:

# vxdctl enable

Adding a System to an Existing Two-Node Cluster CausesError

If the local system attempts to join an existing two-node cluster, an error is generated thatresembles:

vxfenconfig: ERROR: The maximum cluster size is two nodes.

This error is generated as a result of a check to limit cluster size to two systems. Thefollowing messages are also sent to the console:

<date,time> <system> vxfen: WARNING: Cluster size maximum is two.<date,time> <system> Dropping out of cluster.<date,time> <system><date,time> <system> gab: GAB:20032: Port b closed.

Removing Existing Keys From DisksTo remove the registration and reservation keys created by another node from a disk, usethe following procedure:

1. Create a file to contain the access names of the disks:

# vi /tmp/disklist

For example:

/dev/rdsk/c1t0d11s2

2. Read the existing keys:

# vxfenadm -g all -f /tmp/disklist

The output from this command displays the key:

Device Name: /dev/rdsk/c1t0d11s2Total Number Of Keys: 1key[0]:

Key Value [Numeric Format]: 65,49,45,45,45,45,45,45Key Value [Character Format]: A1------



3. If you know on which node the key was created, log in to that node and enter thefollowing command:

# vxfenadm -x -kA1 -f /tmp/disklist

The key is removed.

4. If you do not know on which node the key was created, follow step 5 through step 7to remove the key.

5. Register a second key “A2” temporarily with the disk:

# vxfenadm -m -kA2 -f /tmp/disklistRegistration completed for disk path /dev/rdsk/c1t0d11s2

6. Remove the first key from the disk by preempting it with the second key:

# vxfenadm -p -kA2 -f /tmp/disklist -vA1key: A2------ prempted the key: A1------ on disk/dev/rdsk/c1t0d11s2

7. Remove the temporary key assigned in step 5.

# vxfenadm -x -kA2 -f /tmp/disklistDeleted the key : [A2------] from device /dev/rdsk/c160d11s2

No registration keys exist for the disk.



System Panics to Prevent Potential Data CorruptionWhen a system experiences a split brain condition and is ejected from the cluster, it panicsand displays the following console message:

VXFEN:vxfen_plat_panic: Local cluster node ejected from cluster toprevent potential data corruption.

How vxfen Driver Checks for Pre-existing Split Brain Condition

The vxfen driver functions to prevent an ejected node from rejoining the cluster after thefailure of the private network links and before the private network links are repaired.

For example, suppose the cluster of system 1 and system 2 is functioning normally whenthe private network links are broken. Also suppose system 1 is the ejected system. Whensystem 1 reboots before the private network links are restored, its membershipconfiguration does not show system 2; however, when it attempts to register with thecoordinator disks, it discovers system 2 is registered with them. Given this conflictinginformation about system 2, system 1 does not join the cluster and returns an error fromvxfenconfig that resembles:

vxfenconfig: ERROR: There exists the potential for a preexistingsplit-brain. The coordinator disks list no nodes which are in thecurrent membership. However, they also list nodes which are notin the current membership.

I/O Fencing Disabled!

Also, the following information is displayed on the console:

<date> <system name> vxfen: WARNING: Potentially a preexisting<date> <system name> split-brain.<date> <system name> Dropping out of cluster.<date> <system name> Refer to user documentation for steps<date> <system name> required to clear preexisting split-brain.<date> <system name><date> <system name> I/O Fencing DISABLED!<date> <system name><date> <system name> gab: GAB:20032: Port b closed

However, the same error can occur when the private network links are working and bothsystems go down, system 1 reboots, and system 2 fails to come back up. From the view ofthe cluster from system 1, system 2 may still have the registrations on the coordinatordisks.



Case 1: System 2 Up, System 1 Ejected (Actual Potential Split Brain)

Determine if system1 is up or not. If it is up and running, shut it down and repair theprivate network links to remove the split brain condition. Reboot system 1.

Case 2: System 2 Down, System 1 Ejected (Apparent Potential SplitBrain)

1. Physically verify that system 2 is down.

2. Verify the systems currently registered with the coordinator disks. Use the followingcommand:

# vxfenadm -g all -f /etc/vxfentab

The output of this command identifies the keys registered with the coordinator disks.

3. Clear the keys on the coordinator disks as well as the data disks using the command/opt/VRTSvcs/rac/bin/vxfenclearpre. See “Using vxfenclearpre Commandto Clear Keys After Split Brain” on page 107.

4. Make any necessary repairs to system 2 and reboot.



Using vxfenclearpre Command to Clear Keys After Split BrainWhen you have encountered a split brain condition, use the vxfenclearpre commandto remove SCSI-3 registrations and reservations on the coordinator disks as well as on thedata disks in all shared disk groups.

1. Shut down all other systems in the cluster that have access to the shared storage. Thisprevents data corruption.

2. Start the script:

# cd /opt/VRTSvcs/rac/bin# ./vxfenclearpre

3. Read the script’s introduction and warning. Then, you can choose to let the script run.

Do you still want to continue: [y/n] (default : n)y

Note Informational messages resembling the following may appear on the console of oneof the nodes in the cluster when a node is ejected from a disk/LUN:

<date> <system name> scsi: WARNING: /sbus@3,0/lpfs@0,0/sd@0,1(sd91):

<date> <system name> Error for Command: <undecoded cmd 0x5f> Error Level:

Informational

<date> <system name> scsi: Requested Block: 0 Error Block 0

<date> <system name> scsi: Vendor: <vendor> Serial Number: 0400759B006E

<date> <system name> scsi: Sense Key: Unit Attention

<date> <system name> scsi: ASC: 0x2a (<vendor unique code 0x2a>), ASCQ: 0x4,

FRU: 0x0

These informational messages may be ignored.

Cleaning up the coordinator disks...

Cleaning up the data disks for all shared disk groups...

Successfully removed SCSI-3 persistent registration andreservations from the coordinator disks as well as the shareddata disks.

Reboot the server to proceed with normal cluster startup...#

4. Reboot all systems in the cluster.




Sample main.cf File
A A sample VCS configuration file, /etc/VRTSvcs/conf/sample_rac/main.cf, isavailable online. It is also shown here for offline reference.
/etc/VRTSvcs/conf/sample_rac/main.cf// %W% %G% %U% - %Q% #//ident "@(#)vcsops:%M% %I%"//// Copyright (c) 2000-2002 VERITAS Software Corporation. All Rights// Reserved.//// THIS SOFTWARE CONTAINS CONFIDENTIAL INFORMATION AND// TRADE SECRETS OF VERITAS SOFTWARE. USE, DISCLOSURE,// OR REPRODUCTION IS PROHIBITED WITHOUT THE PRIOR// EXPRESS WRITTEN PERMISSION OF VERITAS SOFTWARE CORPORATION//// RESTRICTED RIGHTS LEGEND// USE, DUPLICATION OR DISCLOSURE BY THE U.S. GOVERNMENT IS// SUBJECT TO RESTRICTIONS SET FORTH IN// THE VERITAS SOFTWARE CORPORATION LICENSE AGREEMENT AND AS// PROVIDED IN DFARS 227.7202-1(a) and// 227.7202-3(a) (1998), and FAR 12.212, as applicable.// VERITAS Software Corporation// 350 Ellis Street,// Mountain View, CA 94043.// UNPUBLISHED -- RIGHTS RESERVED UNDER THE COPYRIGHT// LAWS OF THE UNITED STATES. USE OF A COPYRIGHT NOTICE// IS PRECAUTIONARY ONLY AND DOES NOT IMPLY PUBLICATION// OR DISCLOSURE.//

include "types.cf"include "CFSTypes.cf"include "CVMTypes.cf"include "OracleTypes.cf"

109

/etc/VRTSvcs/conf/sample_rac/main.cf

cluster vcs

system sysa

system sysb

group cvm ( SystemList = { sysa = 0, sysb = 1 } AutoFailOver = 0 Parallel = 1 AutoStartList = { sysa, sysb } )

CVMCluster cvm_clus ( Critical = 0 CVMClustName = vcs CVMNodeId = { sysa = 0, sysb = 1 } CVMTransport = gab CVMTimeout = 200 )

CFSQlogckd qlogckd ( Critical = 0 )

CFSfsckd vxfsckd ( )

CVMVolDg orabin_voldg ( CVMDiskGroup = orabindg CVMVolume = { "orabinvol", "srvmvol" } CVMActivation = sw )

CFSMount orabin_mnt ( Critical = 0 MountPoint = "/oracle" BlockDevice = "/dev/vx/dsk/orabindg/orabinvol" Primary = sysa )

NIC listener_hme0 ( Device = hme0 NetworkType = ether )



IP listener_ip ( Device = hme0 Address @sysa = "192.2.40.21" Address @sysb = "192.2.40.22" )

Sqlnet LISTENER ( Owner = oracle Home = "/oracle/orahome" TnsAdmin = "/oracle/orahome/network/admin" MonScript = "./bin/Sqlnet/LsnrTest.pl" Listener @sysa = LISTENER_a Listener @sysb = LISTENER_b EnvFile = "/opt/VRTSvcs/bin/Sqlnet/envfile" )

vxfsckd requires qlogckd orabin_voldg requires cvm_clus orabin_mnt requires vxfsckd orabin_mnt requires orabin_voldg listener_ip requires listener_hme0 LISTENER requires listener_ip LISTENER requires orabin_mnt

// resource dependency tree // // group cvm // { // Sqlnet LISTENER // { // CFSMount orabin_mnt // { // CFSfsckd vxfsckd // { // CFSQlogckd qlogckd // } // CVMVolDg orabin_voldg // { // CVMCluster cvm_clus // } // } // IP listener_ip // { // NIC listener_hme0 // } // } // }

Appendix A, Sample main.cf File 111


group oradb1_grp ( SystemList = { sysa = 0, sysb = 1 } AutoFailOver = 1 Parallel = 1 AutoStartList = { sysa, sysb } )

CVMVolDg oradb1_voldg ( CVMDiskGroup = oradb1dg CVMVolume = { "oradb1vol" } CVMActivation = sw )

CFSMount oradb1_mnt ( MountPoint = "/oradb1" BlockDevice = "/dev/vx/dsk/oradb1dg/oradb1vol" )

Oracle VRT_db ( Sid @sysa = VRT1 Sid @sysb = VRT2 Owner = oracle Home = "/oracle/orahome" Pfile @sysa = "/oracle/orahome/dbs/initVRT1.ora" Pfile @sysb = "/oracle/orahome/dbs/initVRT2.ora" User = scott Pword = tiger Table @sysa = vcstable_sysa Table @sysb = vcstable_sysb MonScript = "./bin/Oracle/SqlTest.pl" AutoEndBkup = 1 EnvFile = "/opt/VRTSvcs/bin/Oracle/envfile" )

requires group cvm online local firm oradb1_mnt requires oradb1_voldg VRT requires oradb1_mnt



// resource dependency tree // // group oradb1_grp // { // Oracle VRT // { // CFSMount oradb1_mnt // { // CVMVolDg oradb1_voldg // } // } // }

group oradb2_grp ( SystemList = { sysa = 1, sysb = 0 } AutoFailOver = 1 Parallel = 1 AutoStartList = { sysb, sysa } )

CVMVolDg oradb2_voldg ( CVMDiskGroup = oradbdg2 CVMVolume = { "oradb2vol" } CVMActivation = sw )

CFSMount oradb2_mnt ( MountPoint = "/oradb2" BlockDevice = "/dev/vx/dsk/oradbdg2/oradb2vol" Primary = sysb )

Oracle rac_db ( Sid @sysa = rac1 Sid @sysb = rac2 Owner = oracle Home = "/oracle/orahome" Pfile @sysa = "/oracle/orahome/dbs/initrac1.ora" Pfile @sysb = "/oracle/orahome/dbs/initrac2.ora" User = scott Pword = tiger Table @sysa = vcstable_sysa Table @sysb = vcstable_sysb MonScript = "./bin/Oracle/SqlTest.pl" AutoEndBkup = 1 EnvFile = "/opt/VRTSvcs/bin/Oracle/envfile" )

Appendix A, Sample main.cf File 113


requires group cvm online local firm oradb2_mnt requires oradb2_voldg rac requires oradb2_mnt

// resource dependency tree // // group oradb2_grp // { // Oracle rac // { // CFSMount oradb2_mnt // { // CVMVolDg oradb2_voldg // } // } // }


CVMCluster, CVMVolDg, and CFSMountAgents
B
This Appendix describes the entry points and the attributes of the CVMCluster,CVMVolDG, and CFSMount agents. Use this information to make necessary changes tothe configuration.

CVMCluster AgentThe CVMCluster resource is configured automatically during installation. TheCVMCluster agent controls system membership on the cluster port associated with VxVMin a cluster.

The following table describes the entry points used by the CVMCluster agent.

Entry Point Description

Online Joins a system to the CVM cluster port.

Offline Removes a system from the CVM cluster port.

Monitor Monitors the system’s CVM cluster membership state.

115

CVMCluster Agent

CVMCluster Agent Type

CVMCluster Agent Type DefinitionThe following type definition is included in the file, CVMTypes.cf. Note that theCVMNodeAddr, PortConfigd, and PortKmsgd attributes are not used in a DBE/ACenvironment because GAB, the required cluster communication messaging mechanism,does not use them.

type CVMCluster ( static int NumThreads = 1 static int OnlineRetryLimit = 2 static int OnlineTimeout = 400 static str ArgList[] = { CVMTransport, CVMClustName, CVMNodeAddr, CVMNodeId, PortConfigd, PortKmsgd, CVMTimeout } NameRule = "" str CVMClustName str CVMNodeAddr{} str CVMNodeId{} str CVMTransport int PortConfigd int PortKmsgd int CVMTimeout)

Attribute Dimension Description

CVMClustName string-scalar Name of the cluster.

CVMNodeId{} string-association An associative list. The first part names the system andthe second part contains the system’s LLT ID number.

CVMTransport string-scalar Specifies cluster messaging mechanism.

Default = gab

CVMTimeout integer-scalar Timeout in seconds used for CVM clusterreconfiguration.

Default = 200


CVMCluster Agent

CVMCluster Agent Sample ConfigurationThe following is an example definition for the CVMCluster service group. See alsoAppendix A “Sample main.cf File” on page 109 for a more extensive main.cf examplethat includes the CVMCluster resource.

CVMCluster cvm_clus ( Critical = 0 CVMClustName = RACCluster1 CVMNodeId = { galaxy = 0, nebula = 1 } CVMTransport = gab CVMTimeout = 200)

Appendix B, CVMCluster, CVMVolDg, and CFSMount Agents 117

Configuring the CVMVolDg and CFSMount Resources

Configuring the CVMVolDg and CFSMount ResourcesThe CVMVolDg agent represents and controls CVM disk groups and the CVM volumeswithin the disk groups. Because of the global nature of the CVM disk groups and theCVM volumes, they are imported only once on the CVM master node.

Configure the CVMVolDg agent for each disk group used by an Oracle service group. Adisk group must be configured to only one Oracle service group. If cluster file systems areused for the database, configure the CFSMount agent for each volume in the disk group.

CVMVolDg Agent, Entry PointsThe following table describes the entry points used by the CVMVolDg agent.


Online If the system is the MASTER and the disk group is not imported, itimports the disk group and starts all the volumes in the shared diskgroup. It then sets the disk group activation mode to shared-writeas long as the CVMActivation attribute is set to sw. The activationmode can be set on both slave and master systems.

Offline Sets the disk group activation mode to off so that all the volumes indisk group are invalid.

Monitor Monitors the specified critical volumes in the disk group. The volumesto be monitored are specified by the CVMVolList attribute. At least onevolume in a disk group must be specified.

Clean Sets the disk group activation mode to off so that all the volumes indisk group are invalid (same as the Offline entry point).



CVMVolDg Agent Type, Attribute DescriptionsThe following table describes the user-modifiable attributes of the CVMVolDg resourcetype.

CVMVolDg Agent Type DefinitionThe CVMVolDg type definition is included in the CVMTypes.cf file, installed by theinstallDBAC utility.

type CVMVolDg (static str ArgList[] = { CVMDiskGroup, CVMVolume, CVMActivation }NameRule = ""str CVMDiskGroupkeylist CVMVolume[]str CVMActivation

)

Sample CVMVolDg Agent ConfigurationEach Oracle service group requires a CVMVolDg resource type to be defined. Refer to“/etc/VRTSvcs/conf/sample_rac/main.cf” on page 109 to see CVMVolDg defined in amore extensive example.

CVMVolDg ora_voldg (CVMDiskGroup = oradatadgCVMVolume = { oradata1, oradata2 }CVMActivation = sw)


CVMDiskGroup string-scalar Names the disk group.

CVMVolume string-keylist Lists the critical volumes in the disk group. At least one volumein the disk group must be specified.

CVMActivation string-scalar Sets the activation mode for the disk group.

Default = sw



CFSMount Agent, Entry PointsThe CFSMount agent brings online, takes offline, and monitors a cluster file system mountpoint. The agent executable is /opt/VRTSvcs/bin/CFSMount/CFSMountAgent. TheCFSMount type definition is in the file /etc/VRTSvcs/conf/config/CFSTypes.cf.

CFSMount Agent Type, Attribute DescriptionsThe table lists user-modifiable attributes of the CFSMount Agent resource type.


Online Mounts a block device in cluster mode.

Offline Unmounts the file system, forcing unmount if necessary, and setsprimary to secondary if necessary.

Monitor Determines if the file system is mounted. Checks mount status usingthe fsclustadm command.

Clean A null operation for a cluster file system mount.


MountPoint string-scalar Directory for the mount point.

BlockDevice string-scalar Block device for the mount point.

MountOpt string-scalar Options for the mount command. To create a valid MountOptattribute string:

- Use the VxFS type-specific options only.

- Do not use the -o flag to specify the VxFS-specific options.

- Do not use the -F vxfs file system type option.

- The cluster option is not required.

- Specify options in comma-separated list as in these examples:

ro ro,cluster blkclear,mincache=closesync

Primary string-scalar INFORMATION ONLY. Stores primary node name for a VxFS filesystem. Primary is automatically modified when an unmountedfile system is mounted or another node becomes the primary.



CFSMount Agent Type DefinitionThe CFSMount agent type definition is included in the CFSTypes.cf file, installed by theinstallDBAC utility.

type CFSMount ( static keylist RegList = { MountOpt, Policy, NodeList } static int FaultOnMonitorTimeouts = 1 static int OnlineWaitLimit = 0 static str ArgList[] = { MountPoint, BlockDevice, MountOpt } NameRule = resource.MountPoint str MountPoint str MountType str BlockDevice str MountOpt str Primary keylist NodeList keylist Policy)

Sample CFSMount Agent ConfigurationEach Oracle service group requires a CFSMount resource type to be defined. Refer to“/etc/VRTSvcs/conf/sample_rac/main.cf” on page 109 to see CFSMount defined in amore extensive example.

CFSMount ora_mount ( MountPoint = "/oradata" BlockDevice = "/dev/vx/dsk/oradatadg/oradatavol1" Primary = nebula

)




Tunable Kernel Driver Parameters
C LMX Tunable Parameters
Edit the file /kernel/drv/lmx.conf to change the values of the LMX driver tunableglobal parameters. The following table describes the LMX driver tunable parameters.

LMXParameter

Description DefaultValue

MaximumValue

lltport Specifies the LLT port with which LMX registers andreceives data. The port can be an unused port between 0and 31; dedicated ports such as 0, 7, 31 and others cannotbe used. Use the command /sbin/lltstat -p todetermine the ports in use.

10 N/A

contexts(minors)

Specifies the maximum number of contexts system-wide.Each Oracle process typically has two LMX contexts.

“Contexts” and “minors” are used interchangeably in thedocumentation; “context” is an Oracle-specific term andshould be used to specify the value in the lmx.conf file.

8192 65535

ports Specifies the number of communication endpoints fortransferring messages from the sender to the receiver in auni-directional manner.

4096 65535

buffers Specifies the number of addressable regions in memoryto which LMX data can be copied.

4096 65535

update Enables asynchronous I/O. A value of zero (0) disablesasynchronous I/O.

1 [enable] N/A

msgbuf Specifies the size of the LMX message buffer in kilobytes(KB).

32 64

123

LMX Tunable Parameters

Example: Configuring LMX ParametersIf you see the message “no minors available” on one node, you can edit the file/kernel/drv/lmx.conf and add a configuration parameter increasing the value forthe maximum number of contexts. (While the term “minors” is reported in the errormessage, the term “contexts” must be used in the configuration file.) Be aware thatincreasing the number of contexts on a system has some impact on the resources of thatsystem.

In the following example, configuring contexts=16384 allows a maximum of 8192Oracle processes (8192 * 2 = 16384). Note that double-quotes are not used to specify aninteger value.

## LMX configuration file#name=”lmx” parent=”pseudo” contexts=16384 instance=0;

For the changes to take effect, either reboot the system, or reconfigure the LMX moduleusing the following steps:

1. Shut down all Oracle service groups on the system:

# hagrp -offline oragrp -sys galaxy

2. Stop all Oracle client processes on the system, such as sqlplus and svrmgrl.

3. Unconfigure the LMX module:

# /sbin/lmxconfig -U

4. Determine the LMX module ID:

# /usr/sbin/modinfo | grep -i lmx

The module ID is the number in the first column of the output.

5. Unload the LMX module, using the module ID you determined:

# /usr/sbin/modunload -i module_ID

6. Configure the LMX module:

# /sbin/lmxconfig -c

7. Bring the service groups back online:

# hagrp -online oragrp -sys galaxy


vxfen Tunable Parameters

vxfen Tunable ParametersOn each node, edit the file /kernel/drv/vxfen.conf to change the values of the vxfendriver tunable global parameters. You must reboot the system to put changes into effect.

For example, to change the size of the kernel log buffer to 65536 bytes, edit the file andadd the configuration parameter dbg_log_size=65536:

## vxfen configuration file#name=”vxfen” parent=”pseudo” dbg_log_size=65536 instance=0;

vxfenParameter

Description DefaultValue

MaximumValue

dbg_log_size Size of the kernel log buffer in bytes. The minimumvalue is 32768 bytes.

32768 131072

Appendix C, Tunable Kernel Driver Parameters 125

vxfen Tunable Parameters


Error Messages
D LMX Error Messages, Critical
The following table lists LMX kernel module error messages. These messages reportcritical errors seen when the system runs out of memory, when LMX is unable tocommunicate with LLT, or when you are unable to load or unload LMX. Refer to“Running Scripts for Engineering Support Analysis” on page 97 for information on howto gather information about your systems and configuration that VERITAS supportpersonnel can use to assist you.

Message ID LMX Message

00001 lmxload packet header size incorrect (number)

00002 lmxload invalid lmx_llt_port number

00003 lmxload context memory alloc failed

00004 lmxload port memory alloc failed

00005 lmxload buffer memory alloc failed

00006 lmxload node memory alloc failed

00007 lmxload msgbuf memory alloc failed

00008 lmxload tmp msgbuf memory alloc failed

00009 lmxunload node number conngrp not NULL

00010 lmxopen return, minor non-zero

00011 lmxopen return, no minors available

00012 lmxconnect lmxlltopen(1) err= number

127

LMX Error Messages, Non-Critical

LMX Error Messages, Non-CriticalThe following table contains LMX error messages that may be displayed during runtime.Refer to “Running Scripts for Engineering Support Analysis” on page 97 for informationon how to gather information about your systems and configuration that VERITASsupport personnel can use to assist you.

If you encounter errors while running your Oracle application due to the display of thesemessages, you may use the lmxconfig command to turn off their display. For example,to disable the display of the messages:

# /sbin/lmxconfig -e 0

To re-enable the display of the messages, you can enter:

# /sbin/lmxconfig -e 1

00013 lmxconnect new connection memory alloc failed

00014 lmxconnect kernel request memory alloc failed

00015 lmxconnect mblk memory alloc failed

00016 lmxconnect conn group memory alloc failed

00017 lmxlltfini: LLT unregister failed err = number

00018 lmxload contexts number > number, max contexts = system limit = number

00019 lmxload ports number > number, max ports = system limit = number

00020 lmxload buffers number > number, max buffers = system limit = number

00021 lmxload msgbuf number > number, max msgbuf size = system limit = number



06001 lmxreqlink duplicate kreq= 0xaddress, req= 0xaddress

06002 lmxreqlink duplicate ureq= 0xaddress kr1= 0xaddress, kr2= 0xaddress req type =number


LMX Error Messages, Non-Critical

06003 lmxrequnlink not found kreq= 0xaddress from= number

06004 lmxrequnlink_l not found kreq= 0xaddress from= number

06101 lmxpollreq not in doneq CONN kreq= 0xaddress

06201 lmxnewcontext lltinit fail err= number

06202 lmxnewcontext lltregister fail err= number

06301 lmxrecvport port not found unode= number node= number ctx= number

06302 lmxrecvport port not found (no port) ctx= number

06303 lmxrecvport port not found ugen= number gen= number ctx= number

06304 lmxrecvport dup request detected

06401 lmxinitport out of ports

06501 lmxsendport lltsend node= number err= number

06601 lmxinitbuf out of buffers

06602 lmxinitbuf fail ctx= number ret= number

06701 lmxsendbuf lltsend node= number err= number

06801 lmxconfig insufficient privilege, uid= number

06901 lmxlltnodestat: LLT getnodeinfo failed err= number


Appendix D, Error Messages 129

VxVM Errors Related to I/O Fencing

VxVM Errors Related to I/O Fencing

VXFEN Driver Error Messages

VXFEN Driver Informational Messagedate and time VXFEN:00021:Starting to eject leaving nodes(s) from datadisks.

date and time VXFEN:00022:Completed ejection of leaving node(s) fromdata disks.

These messages are for information only. They show how long it takes the data disks to befenced for nodes that have left the cluster.

Message Explanation

vold_pgr_register(disk_path):failed to open the vxfen device.Please make sure that the vxfendriver is installed and configured

The vxfen driver has not been configured. Follow theinstructions in “Setting Up Coordinator Disks” on page 44to set up coordinator disks and start I/O fencing. Thenclear the faulted resources and online the service groups.

vold_pgr_register(disk_path):Probably incompatible vxfen driver.

Incompatible versions of VxVM and the vxfen driver areinstalled on the system. Install the proper version ofDBE/AC.

Message Explanation

VXFEN: Unable to register withcoordinator disk with serial number:xxxx

This message appears when the vxfen driver is unable toregister with one of the coordinator disks. The serialnumber of the coordinator disk that failed is printed.

VXFEN: Unable to register with amajority of the coordinator disks.Dropping out of cluster.

This message appears when the vxfen driver is unable toregister with a majority of the coordinator disks. Theproblems with the coordinator disks must be cleared beforefencing can be enabled.

This message is preceded with the message “VXFEN:Unable to register with coordinator disk with serial numberxxxx.”


Informational Messages When Node is Ejected

Informational Messages When Node is EjectedInformational messages resembling the following may appear on the console of one of thenodes in the cluster when a node is ejected from a disk/LUN:

<date> <system name> scsi: WARNING: /sbus@3,0/lpfs@0,0/sd@0,1(sd91):<date> <system name> Error for Command: <undecoded cmd 0x5f> ErrorLevel: Informational<date> <system name> scsi: Requested Block: 0 Error Block 0<date> <system name> scsi: Vendor: <vendor> Serial Number:0400759B006E<date> <system name> scsi: Sense Key: Unit Attention<date> <system name> scsi: ASC: 0x2a (<vendor unique code 0x2a>), ASCQ:0x4, FRU: 0x0

These informational messages may be ignored.

Appendix D, Error Messages 131

Informational Messages When Node is Ejected


Creating Starter Databases
E The optional procedures in this appendix describe suggested methods for creating astarter Oracle9i database on shared storage. They are provided in case you are not creatingyour own database using tools you may have available.
You can create the database on raw VxVM volumes or on VxFS shared file systems. Twomethods to create the database are described. You can use either dbca (Oracle9i DatabaseCreation Assistant) or VERITAS provided scripts.

For Oracle9i Release 1 and Release 2, you can use dbca to create a starter database in rawvolumes, or you can use VERITAS scripts to a create database in either raw volumes or aVxFS file system.

Using Oracle dbca to Create a Starter DatabaseIf you intend to use the Oracle9i Database Configuration Assistant (dbca) utility to createa starter database in raw VxVM volumes, you must prepare the shared storage for thedatabase tablespaces.

If you intend to create the database in a VxFS file system, you must use the VERITASscripts instead of dbca. See “Using VERITAS Scripts to Create a Starter Database” onpage 137.

133

Using Oracle dbca to Create a Starter Database

Preparing Tablespaces for Use With dbcaWhen creating an Oracle general purpose starter database, you must prepare aconfiguration file that the database creation utilities can use to create the tablespaces. Thetablespace name and size requirements are described in the following table. Note that thesizes are recommended, not minimum.

Tablespace Requirements

The total recommended size required for these tablespaces is 6.56 GB.

Tablespace Recommended Size

system1 1 gigabyte (GB)

spfile1 10 megabytes (MB)

users1 1 GB

temp1 1 GB

undotbs1 700 MB

undotbs2 700 MB

example1 350 MB

cwmlite1 200 MB

indx1 150 MB

tools1 50 MB

drsys1 200 MB

control1 200 MB

control2 200 MB

redo1_1 200 MB

redo1_2 200 MB

redo2_1 200 MB

redo2_2 200 MB



Creating Shared Raw Volumes for TablespacesThe procedure described in this section applies if you plan to use the Oracle dbca utilityto create a starter database on raw volumes.


2. On the master node, create a shared disk group:

# vxdg init ora_dg c2t3d1s2

3. Create a volume in the shared group for each of the tablespaces listed in the previoustable, “Tablespace Requirements” on page 134:

# vxassist -g ora_dg make VRT_system1 1000M# vxassist -g ora_dg make VRT_spfile1 10M..

4. Enable write access to the shared disk group. On the master node, enter:

# vxdg deport ora_dg# vxdg -s import ora_dg# vxvol -g ora_dg startall# vxdg -g ora_dg set activation=sw


# vxdg -g ora_dg set activation=sw

Refer to the description of disk group activation modes in the VERITAS VolumeManager Administrator’s Guide for more information (see the chapter on “ClusterFunctionality”).

6. Define the access mode and permissions for the volumes storing the Oracle data. Foreach volume listed in $ORACLE_HOME/raw_config, use the vxedit(1M) command:

vxedit -g disk_group set group=group user=user mode=660 volume

For example:

# vxedit -g ora_dg set group=dba user=oracle mode=660 VRT_system1

In this example, VRT_system1 is the name of one of the volumes. Repeat thecommand to define access mode and permissions for each volume in the ora_dg.

You can now create the database. See “Creating the Oracle Starter Database UsingOracle9i dbca” on page 136.

Appendix E, Creating Starter Databases 135


Creating the Oracle Starter Database Using Oracle9i dbcaAs Oracle user, start the Oracle9i Database Configuration Assistant (dbca) utility on themaster node to create a general purpose database. The dbca utility is a graphical userinterface and requires setting of the DISPLAY environment variable to run. Refer to theOracle9i Installation Guide.

Note If you find the dbca utility is unable to create a database, you can create the starterdatabase using the VERITAS scripts described in the next section.


Using VERITAS Scripts to Create a Starter Database

Using VERITAS Scripts to Create a Starter DatabaseYou may use the VERITAS scripts to create a starter database in either raw VxVM volumes(see below) or a VERITAS cluster file system (see “Using VERITAS Scripts to CreateDatabase on Cluster File System” on page 139).

Using VERITAS Scripts to Create Database on Raw VolumesWhen you create a starter database on raw volumes, you can manually create the Oracledatabase using the scripts in the /opt/VRTSvcs/rac/db_scripts/raw directory. AREADME file in that directory guides you through the following procedure, which assumesyou use sh or ksh. If you use another shell, make sure the syntax is appropriate.

1. Create a disk group on shared storage in which to create the database. For example:

# vxdg init raw_db_dg c2t3d1s2

2. Deport and import the disk group, start the volume, and enable write access to theshared disk group. On the master node, enter:

# vxdg deport raw_db_dg# vxdg -s import raw_db_dg# vxvol -g raw_db_dg startall# vxdg -g raw_db_dg set activation=sw


# vxdg -g ora_dg set activation=sw

Refer to the description of disk group activation modes in the VERITAS VolumeManager Administrator’s Guide for more information (see the chapter on “ClusterFunctionality”).

4. Log in as user oracle.

5. Create a directory in which to copy the database creation scripts from the/opt/VRTSvcs/rac/db_scripts/raw directory. Do not edit the original files inthe original directory because creating the database replaces the variables in thescripts. For example:

$ mkdir db_cr_raw$ cp /opt/VRTSvcs/rac/db_scripts/raw/* db_cr_raw

6. Read the file README that is copied with the database creation scripts. This procedureis described in the README.



7. Edit the file conf, and set all variable as they apply to your environment:

$ vi conf


9. As root user, source the conf file:

# . conf

10. Run the script cr_vols. This script creates volumes for tablespaces in the disk groupyou created in step 1.


12. As user oracle, source the conf file:

$ . conf

13. Set the ORACLE_BASE and ORACLE_HOME environment variables.

14. Run the convert command. This command uses the values of the environmentvariables set in the conf file to modify the database creation scripts. Enter:

$ convert

15. Examine the script runall.sh and use comment symbols to prevent the use of anydatabase options you don’t want.The following example shows the runall.sh scriptedited such that demo schemas are not created:

$ vi runall.sh

# This is a sample database creation script.# It is assumed that the script is run from Bourne or K shell.# Comment out those lines which you don’t want to be executed,# e.g., if demo schemas are not required, comment out the# corresponding line.

ORACLE_SID=${INSTANCE_1}export ORACLE_SID

# Create the directory structure for Oracle logsmkdir -p ${ORACLE_BASE}/admin/${DB_NAME}/createmkdir -p ${ORACLE_BASE}/admin/${DB_NAME}/bdumpmkdir -p ${ORACLE_BASE}/admin/${DB_NAME}/cdumpmkdir -p ${ORACLE_BASE}/admin/${DB_NAME}/udump

# Add this entry in the oratab: SEA1:${ORACLE_HOME}:Y${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/CreateDB.sql${ORACLE_HOME}/bin/sqlplus /nolog@${SCRIPT_DIR}/CreateDBFiles.sql



${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/CreateDBCatalog.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/JServer.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/ordinst.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/spatial.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/cwmlite.sql#${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/demoSchemas.sql${ORACLE_HOME}/bin/sqlplus /nolog@${SCRIPT_DIR}/CreateClustDBViews.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/postDBCreation.sqlexit

16. Check the permissions for runall.sh. It requires execute permission for the user.

17. Run the command to create the starter database:

$ runall.sh

Using VERITAS Scripts to Create Database on Cluster FileSystem

When creating a starter database in a cluster file system, manually create the Oracledatabase using the scripts in the directory /opt/VRTSvcs/rac/db_scripts/fs. AREADME file in that directory guides you through the following procedure, which assumesyou use sh or ksh. If you use another shell, make sure the syntax is appropriate.

1. Create a disk group on shared storage in which to create the database. For example:

# vxdg init racfs_db c2d2t1s1


3. Create a directory and copy the contents of /opt/VRTSvcs/rac/db_scripts/fsto it. Don’t edit the original files in the original directory because creating the databasereplaces the variables in the scripts. For example:

$ mkdir db_cr_fs$ cp /opt/VRTSvcs/rac/db_scripts/fs/* db_cr_fs

4. Read the file README that is copied with the database creation scripts. This procedureis described in the README.

5. Edit the file conf, and set all the variables as they apply to your environment:

$ vi conf




7. As root user, source the conf file:

# . conf

8. Run the script prep_fs. This script creates and mounts a file system with therequired options and settings based on the contents of the conf file.


10. As user oracle, source the conf file:

$ . conf

11. Set the ORACLE_BASE and ORACLE_HOME environment variables.

12. Run the convert command. This command uses the values of the environmentvariables set in the previous step to modify the database creation scripts. Enter:

$ convert

13. Examine the script runall.sh and use comment symbols to prevent the use of anydatabase options you don’t want:

$ vi runall.sh

The following example shows the runall.sh script edited such that demo schemasare not created:

# This is a sample database creation script.# It is assumed that the script is run from Bourne or K shell.# Comment out those lines which you don’t want to be executed,# e.g., if demo schemas are not required, comment out the# corresponding line.

ORACLE_SID=${INSTANCE_1}export ORACLE_SID

# Create the directory structure for Oracle logsmkdir -p ${ORACLE_BASE}/admin/${DB_NAME}/createmkdir -p ${ORACLE_BASE}/admin/${DB_NAME}/bdumpmkdir -p ${ORACLE_BASE}/admin/${DB_NAME}/cdumpmkdir -p ${ORACLE_BASE}/admin/${DB_NAME}/udump

# Add this entry in the oratab: SEA1:${ORACLE_HOME}:Y${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/CreateDB.sql${ORACLE_HOME}/bin/sqlplus /nolog@${SCRIPT_DIR}/CreateDBFiles.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/CreateDBCatalog.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/JServer.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/ordinst.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/spatial.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/cwmlite.sql



#${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/demoSchemas.sql${ORACLE_HOME}/bin/sqlplus /nolog@${SCRIPT_DIR}/CreateClustDBViews.sql${ORACLE_HOME}/bin/sqlplus /nolog @${SCRIPT_DIR}/postDBCreation.sqlexit

14. Check the permissions for runall.sh. It requires execute permission for the user.

15. Run the command to create the starter database:

$ runall.sh




Index

Symbols.rhosts

editing to enable rsh 25, 56, 60, 62, 64, 67editing to remove rsh permissions 69

/etc/default/vxdg 52/opt/VRTSvcs/bin/chk_dbac_pkgs 39/sbin/lmxconfig 124/sbin/vcsmmconfig, starting VCSMM 98

Numerics9iinstall script 56, 599iP2install script 619iP3install script 61

AActivation mode, setting 53, 135, 137Agents

CFSMount 120CVMCluster 116CVMVolDg 118Oracle. See Oracle enterprise agent

AttributesCFSMount agent 120CVMCluster agent 116CVMVolDg agent 119Oracle agent, setting as local 73Sqlnet agent, setting as local 73

CCFS (Cluster File System)

extension of VxFS 9used with ODM 9

CFSMount agent 120definition 118sample configuration 121type definition 119

CFSTypes.cf file 120Cluster Manager (CM) 3Cluster membership 3, 12Commands

/sbin/lltstat -p (listing ports in use) 123convert 138format (verify disks) 103lmxconfig -c (configure LMX) 124lmxconfig -U (unconfigure LMX) 124vxassist 53vxdctl enable (scan disks) 103vxdg list (disk group information) 52vxedit (set shared volume mode) 135vxfenclearpre 107

Communicationsdata stack illustration 5LLT and GAB overview 6provided by GAB 7

Configuration filesLMX tunable parameters 123sample main.cf file 109VXFEN tunable parameters 125

Connectivity policy of disk groups 52Contexts

also know as minors 123LMX tunable parameter 123

convert command 138Coordinator disks

concept of 14description 81setting up 44

cr_vols script 138CVM (Cluster Volume Manager)

architecture 8extension of VxVM 8

CVM service groupconfiguring in main.cf 73created after installation 47

CVMCluster agentdescription 115sample configuration 117type definition 116

143

CVMTypes.cf file 116, 119CVMVolDg agent

description 118type definition 119

DData corruption

preventing with I/O fencing 13system panics to prevent 105

Databaseusing dbca to create 133using scripts to create 137

Dependencies among service groups 72Disk group

creating for SRVM 53displaying information about 52general guidelines 52importing 53, 135, 137setting activation mode 53, 135, 137setting the connectivity policy 52

EEjected systems

error messages displayed 131losing access to shared storage 81recovering from ejection 105

EMC Symmetrix 8000 series storagePR flag requirement 41

Enterprise agent for OracleSee Oracle enterprise agent

Environment variablesMANPATH 24PATH variable 24

Error messagesLMX messages 127running vxfenclearpre command 107VXFEN messages 130VxVM, related to I/O fencing 130when node is ejected 131

FFencing. See I/O fencingFiles

.rhosts 25/etc/default/vxdg 52

Flowchart, RAC installation overview 20format command 103

GGAB

part of communications package 5

port memberships 46getcomms (troubleshooting script) 97getdbac (troubleshooting script) 97

Hhagetcf (troubleshooting script) 97Hitachi Data Systems

System Mode 186 must be set 41Hitachi Data Systems storage

99xx Series supported 41

II/O fencing

components 14description 81overview 12scenarios for I/O fencing 82starting 45stopping 91

Importing disk groups 53, 135, 137Inter-Process Communication (IPC) 3

KKernel driver parameters 123Keys

registration keys, formatting of 85removing registration keys 103

LLicense keys

adding during installation 26obtaining xiv

Listener processconfiguring as resource in main.cf 74shown in configuration 11

LLT 6LMX

configuring with lmxconfig -c 124error messages 127tunable driver parameters 123unconfiguring 124

Local, defining attributes as 78Log files

location of Oracle agent log files 79location of VCS log files 79

LUNsusing for coordinator disks 44verifying serial numbers for paths to 41

Mmain.cf


example 109example after installation 48

MANPATH environment variable 24Manual pages, setting MANPATH 24Minors

also known as contexts 123appearing in LMX error messages 101increasing maximum number of 124

OODM (Oracle Disk Manager)

described 9linking Oracle9i to VERITAS libraries 69

Oracle enterprise agentconfiguring for RAC 73documentation 76location of log files 79

Oracle instance 2Oracle service group

configuring in main.cf 76dependencies among resources 72

Oracle9iconfirming parallel flag 71installation tasks 51installing Release 1 locally 58installing Release 1 on shared disk 54installing Release 1 Patch 61installing Release 2 locally 66installing Release 2 on shared disks 63prerequisites for installing 51runInstaller utility 60, 62

PPackages

for CVM and CFS installation 34for VCS installation 32removing 91verifying installation of 39

Parallel attribute, setting for groups 73Parallel flag, confirming in Oracle 71Parameters

LMX tunable driver parameters 123shared memory parameter 51vxfen tunable driver parameters 125

Patchesrequired for installation 22verifying installation of 39

PATH environment variable, setting 24Persistent reservations 41, 81Ports, GAB

displaying 46membership, list of 46removing membership for 90

Ports, tunable kernel parameter 123prep_fs script 140

RRAC

architecture 1defined 1installation flowchart 20

Registration keydisplaying with vxfenadm 85formatting of 85

Registrationsfor I/O fencing 81key formatting 85

Reservationsdescription 81description and use 13SCSI-3 persistent (PR) 18

rsh permissionsfor inter-system communication 25removing 69

runall.sh script 138runInstaller, Oracle 9i utility 60, 62

SScripts

9iinstall 56, 599iP2install, 9iP3install 61cr_vols 138prep_fs 140runall.sh 138using to create Database 137

SCSI-3 persistent reservationsdescribed 13EMC Symmetrix support 41requirement for I/O fencing 40verifying EMC Symmetrix support 41verifying that storage supports 40, 51

Service groupsCVM 72dependencies among 72

Split braindescribed 12removing risks associated with 13

Sqlnet agent, setting local attributes 73Sqlnet resource, configuring in main.cf 74srvconfig (Oracle SRVM) component 53, 57

Index 145

SRVM, creating disk group for 53Storage

setting up shared in cluster 18supported for DBE/AC 41testing for I/O fencing 42

TTablespaces

table of tablespace names and sizes 134Troubleshooting scripts 97

UUninstalling DBE/AC for Oracle9i RAC 92uninstallvcs utility 92

VVCS

architecture 10location of log files 79uninstalling previous version 22

VCS membership module 12VCSIPC

copying library into place 65errors in trace/log files 101

VCSMM module 12vcsmmconfig command 98

VERITAS ODM libraries, linking 69vxassist command 53vxdctl command 103vxfen command 45, 91vxfen driver

kernel tunable parameters 125starting 45unconfiguring 91VXFEN error messages 130

vxfenadm commandoptions for administrators 85using -i to read LUN info 41

vxfenclearpre commanderror messages 107running 107

vxfentab file, created by rc script 45vxfentsthdw utility, using to test disks 42VxFS

removing previous version 22Solaris 8 patches for 22

VxVMCVM is extension of 8removing 92Solaris 8 patches for 22


Date post:	27-Jun-2015
Category:	Documents
Upload:	nilesh-vaity
View:	106 times
Download:	2 times

VRITAS_RAC

Documents