Parallel Database Cluster Model PDC/O2000 for Oracle8 ... · viii Parallel Database Cluster Model...

Compaq Confidential – Need to Know RequiredWriter: Linda Waterhouse Project: Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide Comments:

Part Number: 161397-001 File Name: a-frnt.doc Last Saved On: 11/3/99 10:41 AM

Parallel Database Cluster ModelPDC/O2000 for Oracle8 Release8.0.5 and Oracle8i Release 8.1.5Administrator Guide

First Edition (December 1999)Part Number 161397-001Compaq Computer Corporation



NoticeThe information in this publication is subject to change without notice.

COMPAQ COMPUTER CORPORATION SHALL NOT BE LIABLE FOR TECHNICAL OREDITORIAL ERRORS OR OMISSIONS CONTAINED HEREIN, NOR FOR INCIDENTAL ORCONSEQUENTIAL DAMAGES RESULTING FROM THE FURNISHING, PERFORMANCE, ORUSE OF THIS MATERIAL. THIS INFORMATION IS PROVIDED “AS IS” AND COMPAQCOMPUTER CORPORATION DISCLAIMS ANY WARRANTIES, EXPRESS, IMPLIED ORSTATUTORY AND EXPRESSLY DISCLAIMS THE IMPLIED WARRANTIES OFMERCHANTABILITY, FITNESS FOR PARTICULAR PURPOSE, GOOD TITLE AND AGAINSTINFRINGEMENT.

This publication contains information protected by copyright. No part of this publication may bephotocopied or reproduced in any form without prior written consent from Compaq ComputerCorporation.

© 1999 Compaq Computer Corporation.

All rights reserved. Printed in the U.S.A.

The software described in this guide is furnished under a license agreement or nondisclosure agreement.The software may be used or copied only in accordance with the terms of the agreement.

Compaq, Deskpro, Compaq Insight Manager, ServerNet, StorageWorks, Systempro, Systempro/LT,ProLiant, ROMPaq, QVision, SmartStart, NetFlex, QuickFind, PaqFax, ProSignia, registered UnitedStates Patent and Trademark Office.

Fastart, Netelligent, Systempro/XL, SoftPaq, QuickBlank, QuickLock and Neoserver are trademarksand/or service marks of Compaq Information Technologies Group, L.P. in the U.S. and/or othercountries.

Microsoft, MS-DOS, Windows, and Windows NT are registered trademarks of Microsoft Corporation.

Pentium is a registered trademark and Xeon is a trademark of Intel Corporation.

Oracle is a registered trademark. Oracle8 and Oracle8i are trademarks of Oracle Corporation.

Other product names mentioned herein may be trademarks and/or registered trademarks of theirrespective companies.

Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8 Release 8.0.5 and Oracle8iRelease 8.1.5 Administrator GuideFirst Edition (December 1999)Part Number 161397-001



Contents

About This GuidePurpose .................................................................................................................... xiiiAudience.................................................................................................................. xiiiScope ........................................................................................................................xivReferenced Manuals ..................................................................................................xvSupplemental Documents .........................................................................................xviText Conventions.....................................................................................................xviiSymbols in Text.......................................................................................................xviiSymbols on Equipment.......................................................................................... xviiiRack Stability ...........................................................................................................xixGetting Help .............................................................................................................xix

Compaq Technical Support ...............................................................................xixCompaq Website.................................................................................................xxCompaq Authorized Reseller..............................................................................xx

Chapter 1Clustering Overview

Clusters Defined ...................................................................................................... 1-2Availability .............................................................................................................. 1-3Scalability ................................................................................................................ 1-3Compaq Parallel Database Cluster Overview.......................................................... 1-4

iv Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Chapter 2Architecture

Compaq ProLiant Servers ........................................................................................ 2-2High-Availability Features of ProLiant Servers ............................................... 2-3

Shared Storage Subsystem Components .................................................................. 2-3RA4000 Array................................................................................................... 2-3RA4000 Array Controllers................................................................................ 2-4Storage Hubs..................................................................................................... 2-5Fibre Host Adapters .......................................................................................... 2-5Gigabit Interface Converter-Shortwave ............................................................ 2-6Fibre Channel Cables........................................................................................ 2-6

Shared Storage Subsystem Features......................................................................... 2-6Maximum Distances Between Cluster Nodes and Shared StorageSubsystem Components .................................................................................... 2-7Redundant Fibre Channel Arbitrated Loop....................................................... 2-7Multiple Redundant Fibre Channel Arbitrated Loops....................................... 2-9I/O Data Paths................................................................................................. 2-11SCSI Disks...................................................................................................... 2-13

I/O Path Configuration Guidelines......................................................................... 2-13I/O Path Configuration Rules.......................................................................... 2-16Active/Standby Configuration Examples........................................................ 2-17Active/Active Configuration Examples .......................................................... 2-23Summary of I/O Path Failure and Failover Scenarios..................................... 2-28

Cluster Interconnect Options.................................................................................. 2-33Redundant Ethernet Cluster Interconnect ....................................................... 2-34Redundant ServerNet Cluster Interconnect..................................................... 2-37Local Area Network........................................................................................ 2-39

Chapter 3Cluster Software Components

Overview of the Cluster Software............................................................................ 3-1Microsoft Windows NT Server 4.0 .......................................................................... 3-2Compaq Software..................................................................................................... 3-2

Compaq SmartStart and Support Software ....................................................... 3-2Compaq System Configuration Utility ............................................................. 3-3Compaq Array Configuration Utility................................................................ 3-3Fibre Channel Fault Isolation Utility ................................................................ 3-3Compaq Insight Manager.................................................................................. 3-4Compaq Insight Manager XE ........................................................................... 3-4Compaq Options ROMPaq ............................................................................... 3-4Compaq Redundancy Manager......................................................................... 3-5Compaq Operating System Dependent Modules .............................................. 3-5

Contents v



Cluster Software Componentscontinued

Oracle Software ....................................................................................................... 3-6Oracle8 Server Enterprise Edition Release 8.0.5.............................................. 3-7Oracle8i Server Enterprise Edition Release 8.1.5............................................. 3-9Application Failover and Reconnection Software .......................................... 3-11

Chapter 4Planning

Site Planning............................................................................................................ 4-2Capacity Planning for Cluster Hardware ................................................................. 4-3

Compaq ProLiant Servers................................................................................. 4-3Planning Shared Storage Subsystem Components ........................................... 4-4Planning Cluster Interconnect and Client LAN Components ........................... 4-5Reference Material for Hardware Sizing .......................................................... 4-7

Planning the Cluster Configuration ......................................................................... 4-7Sample Small Configuration for the PDC/O2000 ............................................ 4-8Sample Large Configuration for the PDC/O2000 ............................................ 4-9

RAID Planning ...................................................................................................... 4-10Supported RAID Levels ................................................................................. 4-12Raw Data Storage and Database Size ............................................................. 4-13Selecting RAID Levels................................................................................... 4-14

Planning the Grouping of Physical Disk Storage Space ........................................ 4-15Disk Drive Planning .............................................................................................. 4-16

Nonshared Disk Drives................................................................................... 4-16Shared Disk Drives......................................................................................... 4-16

Network Planning .................................................................................................. 4-17Windows NT Server Hosts Files for the Ethernet Cluster Interconnect ......... 4-17Windows NT Server Hosts Files for the ServerNet Cluster Interconnect....... 4-18Client LAN ..................................................................................................... 4-18

vi Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Chapter 5Installation and Configuration for Oracle8 Release 8.0.5

Installation Overview............................................................................................... 5-2Installing the Hardware............................................................................................ 5-3

Setting Up the Nodes ........................................................................................ 5-3Installing the Fibre Host Adapters .................................................................... 5-4Installing GBIC-SW Modules for the Fibre Host Adapters .............................. 5-4Cabling the Fibre Host Adapters to the Storage Hubs ...................................... 5-4Installing the Ethernet Cluster Interconnect Adapters ...................................... 5-5Installing the Client LAN Adapters .................................................................. 5-6Setting Up the RA4000 Arrays ......................................................................... 5-6Installing GBIC-SW Modules for the RA4000 Array Controllers.................... 5-8Cabling the Storage Hubs to the RA4000 Array Controllers............................ 5-8Installing Additional Redundant FC-ALs ....................................................... 5-12Cabling the Ethernet Cluster Interconnect ...................................................... 5-13Cabling the Client LAN.................................................................................. 5-14

Installing Operating System Software and Configuring the RA4000 Arrays ........ 5-15Guidelines for Clusters ................................................................................... 5-15Automated Installation Using SmartStart ....................................................... 5-16

Installing Compaq Redundancy Manager .............................................................. 5-20Verifying Shared Storage Using Redundancy Manager ................................. 5-21Defining Active Array Controllers ................................................................. 5-22

Installing Oracle Software ..................................................................................... 5-24Verifying Cluster Communications ....................................................................... 5-24Installing the Compaq OSDs.................................................................................. 5-25

OSD Installation Steps.................................................................................... 5-25Running the NodeList Configurator ............................................................... 5-27

Installing Object Link Manager ............................................................................. 5-28Configuring Oracle Software ................................................................................. 5-29

Additional Notes on Configuring Oracle Software......................................... 5-29Verifying the Hardware and Software Installation................................................. 5-31

Cluster Communications................................................................................. 5-31Access to Shared Storage from All Nodes...................................................... 5-32OSDs............................................................................................................... 5-32

Power Distribution and Power Sequencing Guidelines.......................................... 5-32Server Power Distribution............................................................................... 5-33RA4000 Array Power Distribution ................................................................. 5-33Power Sequencing........................................................................................... 5-34

Contents vii



Chapter 6Installation and Configuration for Oracle8i Release 8.1.5

Installation Overview............................................................................................... 6-2Installing the Hardware............................................................................................ 6-3

Setting Up the Nodes........................................................................................ 6-3Installing the Fibre Host Adapters .................................................................... 6-4Installing GBIC-SW Modules for the Fibre Host Adapters.............................. 6-4Cabling the Fibre Host Adapters to the Storage Hubs ...................................... 6-5Installing the Cluster Interconnect Adapters..................................................... 6-6Installing the Client LAN Adapters .................................................................. 6-7Setting Up the RA4000 Arrays......................................................................... 6-7Installing GBIC-SW Modules for the RA4000 Array Controllers ................... 6-9Cabling the Storage Hubs to the RA4000 Array Controllers............................ 6-9Installing Additional Redundant FC-ALs....................................................... 6-13Cabling the Cluster Interconnect .................................................................... 6-14Cabling the Client LAN.................................................................................. 6-17

Installing Operating System Software and Configuring the RA4000 Arrays ........ 6-18Guidelines for Clusters ................................................................................... 6-18Automated Installation Using SmartStart ....................................................... 6-19

Installing Compaq Redundancy Manager.............................................................. 6-23Verifying Shared Storage Using Redundancy Manager ................................. 6-24Defining Active Array Controllers ................................................................. 6-25

Installing Compaq OSDs ....................................................................................... 6-27Verifying Installation of the SNMP Service................................................... 6-28Verifying Cluster Communications ................................................................ 6-29Mounting Remote Drives and Verifying Administrator Privileges ................ 6-30Installing Ethernet OSDs ................................................................................ 6-31Installing ServerNet OSDs, Drivers, and SNMP Agents................................ 6-40Verifying the ServerNet Cluster Interconnect ................................................ 6-52

Installing Oracle Software ..................................................................................... 6-52Configuring Oracle Software................................................................................. 6-53

Additional Notes on Configuring Oracle Software ........................................ 6-53Installing Object Link Manager ............................................................................. 6-54Verifying the Hardware and Software Installation ................................................ 6-55

Cluster Communications ................................................................................ 6-55Access to Shared Storage from All Nodes...................................................... 6-55OSDs .............................................................................................................. 6-55

Power Distribution and Power Sequencing Guidelines ......................................... 6-56Server Power Distribution .............................................................................. 6-56RA4000 Array Power Distribution................................................................. 6-57Power Sequencing .......................................................................................... 6-58

viii Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Chapter 7Cluster Management

Cluster Management Concepts ................................................................................ 7-2Powering Off a Node Without Interrupting Cluster Services ........................... 7-2Managing a Cluster in a Degraded Condition................................................... 7-2Managing Network Clients Connected to a Cluster.......................................... 7-3Cluster Events ................................................................................................... 7-3

Management Applications ....................................................................................... 7-4Monitoring Server and Network Hardware....................................................... 7-4Managing Shared Drives................................................................................... 7-5Monitoring Redundant Fibre Channel Arbitrated Loops .................................. 7-5Monitoring the Database................................................................................... 7-6Remotely Managing a Cluster .......................................................................... 7-7

Software Maintenance for Oracle8 Release 8.0.5 .................................................... 7-7Deinstalling the OSDs....................................................................................... 7-7Upgrading Oracle8 Server ................................................................................ 7-8

Software Maintenance for Oracle8i Release 8.1.5 ................................................... 7-8Deinstalling the OSDs....................................................................................... 7-8Deinstalling a Partial OSD Installation........................................................... 7-11Upgrading Oracle8i Server ............................................................................. 7-11

Managing Changes to Shared Storage ................................................................... 7-12Replacing a Failed Disk .................................................................................. 7-12Adding Disk Drives to Increase Storage Capacity.......................................... 7-12Adding an RA4000 Array............................................................................... 7-13

Replacing a Cluster Node ...................................................................................... 7-15Removing the Node ........................................................................................ 7-15Adding the Replacement Node ....................................................................... 7-16

Adding a Cluster Node........................................................................................... 7-19Preparing the New Node................................................................................. 7-20Preparing the Existing Cluster Nodes ............................................................. 7-21Installing the Cluster Software for Oracle8 Release 8.0.5 .............................. 7-21Installing the Cluster Software for Oracle8i Release 8.1.5 ............................. 7-22

Monitoring Cluster Operation ................................................................................ 7-23Tools Overview............................................................................................... 7-23Using Compaq Redundancy Manager ............................................................ 7-24

Contents ix



Chapter 8Troubleshooting

Basic Troubleshooting Tips ..................................................................................... 8-2Power................................................................................................................ 8-2Physical Connections........................................................................................ 8-2Accessibility of Resources................................................................................ 8-3Software Revisions ........................................................................................... 8-4Firmware Revisions.......................................................................................... 8-5

Troubleshooting Oracle8 Release 8.0.5 and OSD Installation Problems andError Messages ........................................................................................................ 8-6

While Running the NodeList Configurator, a Dialog Box AppearsIndicating Inability to Connect to Remote Nodes............................................. 8-6Error Message: Unable to Start Cluster Manager (CMSRVR.EXE) ................ 8-6Error Message: Unable to Start Oracle Service ................................................ 8-7Error Message: “Dependent Service Has Not Started” Received WhileAttempting to Start the Oracle Service ............................................................. 8-7Error Message: Unable to Configure Oracle Using OPSCONF Utility............ 8-7Error Message: Error In Creating Oracle Instance ........................................... 8-8Error Message: Initialization of the Dynamic Link Library CM.DLLFailed. The Process Is Terminating Abnormally .............................................. 8-8Unable to Start the Database............................................................................. 8-8

Troubleshooting Oracle8i Release 8.1.5 and OSD Installation Problems andError Messages ........................................................................................................ 8-9

Potential Difficulties Installing the OSDs with the Oracle UniversalInstaller ............................................................................................................. 8-9Unable to Start OracleCMService .................................................................. 8-10Unable to Start OracleNMService .................................................................. 8-11Unable to Start the Database........................................................................... 8-11Initialization of the Dynamic Link Library NM.DLL Failed.......................... 8-12

Troubleshooting Node-to-Node Connectivity Problems ....................................... 8-12Nodes Are Unable to Communicate with Each Other .................................... 8-12viping Does Not Complete Successfully ........................................................ 8-14Unable to Ping the Cluster Interconnect or the Client LAN ........................... 8-15Node or Nodes Unable to Rejoin the Cluster.................................................. 8-16Ping <computer_name> Shows Cluster Interconnect IP Address Instead ofClient LAN IP Address................................................................................... 8-17

Troubleshooting Client-to-Cluster Connectivity Problems ................................... 8-18A Network Client Cannot Communicate with the Cluster ............................. 8-18

Troubleshooting Shared Storage Problems............................................................ 8-19Verifying Connectivity to the Redundant Fibre Channel Arbitrated Loop..... 8-19Shared Disks in the RA4000 Arrays Are Not Recognized By One or MoreNodes.............................................................................................................. 8-20Node Cannot Connect to the Shared Drives ................................................... 8-21

x Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Troubleshootingcontinued

Troubleshooting Compaq Redundancy Manager Problems................................... 8-22Event Logging................................................................................................. 8-22Informational Messages .................................................................................. 8-22Warning Message ........................................................................................... 8-25Error Messages ............................................................................................... 8-25Other Redundancy Manager Problems ........................................................... 8-27

Troubleshooting Other Potential Problems ............................................................ 8-28Windows NT Blue Screen With AFD.SYS Failure Displayed ....................... 8-28

Appendix Aviping Utility

Syntax and Option Summary .................................................................................. A-1Example ........................................................................................................... A-2

Error Diagnostics .................................................................................................... A-2

Appendix BDiagnosing and Resolving Shared Disk Problems

Introduction..............................................................................................................B-1Run Object Link Manager On All Nodes.................................................................B-3Restart All Affected Nodes in the Cluster................................................................B-4Rerun and Validate Object Link Manager On All Affected Nodes .........................B-4Run and Validate Compaq Redundancy Manager On All Nodes ............................B-5Run Windows NT Server Disk Administrator On All Nodes ..................................B-6Run and Validate the Array Configuration Utility On All Nodes............................B-6Perform Cluster Software and Firmware Checks.....................................................B-7Perform Cluster Hardware Checks...........................................................................B-7Contact Your Compaq Support Representative .......................................................B-8

Glossary

Index

List of FiguresFigure 1-1. Example of a Compaq Parallel Database

Model PDC/O2000 cluster ................................................................................. 1-2Figure 2-1. Maximum distances between PDC/O2000 cluster nodes and

shared storage subsystem components ............................................................... 2-7Figure 2-2. Two-node PDC/O2000 cluster with one redundant Fibre

Channel Arbitrated Loop.................................................................................... 2-8Figure 2-3. Two-node PDC/O2000 cluster with two redundant Fibre

Channel Arbitrated Loops ................................................................................ 2-10

Contents xi



Figure 2-4. Host adapter-to-Storage Hub data paths............................................. 2-11Figure 2-5. Storage Hub-to-RA4000 Array data paths ......................................... 2-12Figure 2-6. Active/standby configuration with one RA4000 Array...................... 2-18Figure 2-7. Active/standby configuration with two RA4000 Arrays.................... 2-19Figure 2-8. Active/standby configuration with three RA4000 Arrays.................. 2-20Figure 2-9. Active/standby configuration with four RA4000 Arrays ................... 2-21Figure 2-10. Active/standby configuration with five RA4000 Arrays.................. 2-22Figure 2-11. Active/active configuration with two RA4000 Arrays..................... 2-24Figure 2-12. Active/active configuration with three RA4000 Arrays................... 2-25Figure 2-13. Active/active configuration with four RA4000 Arrays .................... 2-26Figure 2-14. Active/active configuration with five RA4000 Arrays .................... 2-27Figure 2-15. Redundant Ethernet cluster interconnect components in a

two-node PDC/O2000 cluster .......................................................................... 2-36Figure 2-16. Redundant ServerNet cluster interconnect components in a

two-node PDC/O2000 cluster .......................................................................... 2-38Figure 2-17. Redundant ServerNet cluster interconnect components in a

four-node PDC/O2000 cluster.......................................................................... 2-39Figure 4-1. Two-node PDC/O2000 cluster with one RA4000 Array...................... 4-8Figure 4-2. Six-node PDC/O2000 cluster with five RA4000 Arrays ..................... 4-9Figure 4-3. RA4000 Array disk grouping for a PDC/O2000 cluster .................... 4-15Figure 5-1. Connecting Fibre Host Adapters to Storage Hubs ............................... 5-5Figure 5-2. RA4000 Arrays connected to clustered servers through one

redundant FC-AL ............................................................................................... 5-7Figure 5-3. Cabling Storage Hubs to RA4000 Array Controllers in an

active/standby configuration .............................................................................. 5-9Figure 5-4. Method 1 cabling in an active/active configuration with two

RA4000 Arrays ................................................................................................ 5-11Figure 5-5. Method 2 cabling in an active/active configuration with two

RA4000 Arrays ................................................................................................ 5-11Figure 5-6. Redundant client LAN and Ethernet cluster interconnect .................. 5-13Figure 5-7. Server power distribution in a three-node cluster............................... 5-33Figure 6-1. Connecting Fibre Host Adapters to Storage Hubs ............................... 6-5Figure 6-2. RA4000 Arrays connected to clustered servers through one

redundant FC-AL ............................................................................................... 6-8Figure 6-3. Cabling Storage Hubs to RA4000 Array Controllers in an

active/standby configuration ............................................................................ 6-10Figure 6-4. Method 1 cabling in an active/active configuration with two

RA4000 Arrays ................................................................................................ 6-12Figure 6-5. Method 2 cabling in an active/active configuration with two

RA4000 Arrays ................................................................................................ 6-12Figure 6-6. Redundant client LAN and Ethernet cluster interconnect .................. 6-15Figure 6-7. Redundant ServerNet cluster interconnect ......................................... 6-16Figure 6-8. Server power distribution in a three-node cluster............................... 6-57Figure B-1. Tasks for diagnosing and resolving shared storage problems .............B-2

xii Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



List of TablesTable 2-1 High-Availability Components of ProLiant Servers................................ 2-3Table 2-2 Features of Active/Standby and Active/Active Configurations............. 2-14Table 2-3 I/O Path Failure and Failover Scenarios for Active/Standby

Configurations With One RA4000 Array......................................................... 2-28Table 2-4 I/O Path Failure and Failover Scenarios for Active/Standby

Configurations With Two or More RA4000 Arrays......................................... 2-29Table 2-5 I/O Path Failure and Failover Scenarios for Active/Active

Configurations With Two or More RA4000 Arrays......................................... 2-31Table 5-1 Active/Active Cabling Methods ............................................................ 5-10Table 5-2 Active Array Controller Locations ........................................................ 5-22Table 6-1 Active/Active Cabling Methods ............................................................ 6-11Table 6-2 Active Array Controller Locations ........................................................ 6-26Table 8-1 Compaq Redundancy Manager Informational Messages ...................... 8-22Table 8-2 Compaq Redundancy Manager Warning Message ................................ 8-25Table 8-3 Compaq Redundancy Manager Error Messages.................................... 8-25Table 8-4 Troubleshooting Other Redundancy Manager Problems....................... 8-27Table A-1 viping Errors .......................................................................................... A-2



About This Guide

PurposeThis administrator guide provides information about the planning, installation,configuration, implementation, management, and troubleshooting of theCompaq Parallel Database Cluster Model PDC/O2000 (PDC/O2000).

AudienceThe expected audience of this guide consists primarily of MIS professionalswhose jobs include designing, installing, configuring, and maintainingCompaq Parallel Database Clusters.

The audience of this guide must have a working knowledge of MicrosoftWindows NT Server and of Oracle databases or have the assistance of adatabase administrator.

This guide contains information for network administrators, databaseadministrators, installation technicians, systems integrators, and othertechnical personnel in the enterprise environment for the purpose of clusterplanning, installation, implementation, and maintenance.

IMPORTANT: This guide contains installation, configuration, and maintenance

information that can be valuable for a variety of users. If you are installing the PDC/O2000

but will not be administering the cluster on a daily basis, please make this guide available

to the person or persons who will be responsible for the clustered servers after you have

completed the installation.

xiv Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



ScopeThis guide offers significant background information about clusters as well asbasic concepts associated with designing clusters. It also contains detailedproduct descriptions and installation steps.

This administrator guide is designed to assist you in the following objectives:

■ Understanding basic concepts of clustering technology

■ Recognizing and using the high-availability features of a PDC/O2000

■ Planning and designing a PDC/O2000 cluster configuration to meet yourbusiness needs

■ Installing and configuring PDC/O2000 hardware and software

■ Managing a PDC/O2000

■ Troubleshooting a PDC/O2000

The following summarizes the contents of this guide:

■ Chapter 1, “Clustering Overview,” provides an introduction toclustering technology features and benefits.

■ Chapter 2, “Architecture,” describes the hardware components of aPDC/O2000 and provides I/O path configuration information.

■ Chapter 3, “Cluster Software Components,” describes softwarecomponents used with a PDC/O2000.

■ Chapter 4, “Planning,” outlines an approach to planning and designingcluster configurations that meet your business needs.

■ Chapter 5, “Installation and Configuration for Oracle8 Release 8.0.5,”outlines the steps you will take to install and configure PDC/O2000hardware and software for Oracle8 Release 8.0.5.

■ Chapter 6, “Installation and Configuration for Oracle8i Release 8.1.5,”outlines the steps you will take to install and configure PDC/O2000hardware and software for Oracle8i Release 8.1.5.

■ Chapter 7, “Cluster Management,” includes techniques for managingand maintaining a PDC/O2000.

■ Chapter 8, “Troubleshooting,” contains troubleshooting information fora PDC/O2000.

■ Appendix A, “viping Utility,” documents the use of the viping utility totest the ServerNet cluster interconnect.

About This Guide xv



■ Appendix B, “Diagnosing and Resolving Shared Disk Problems,”describes procedures to diagnose and resolve shared disk problems.

■ Glossary, contains definitions of many terms used in this guide.

Some clustering topics are mentioned, but not detailed, in this guide. Forexample, this guide does not describe how to install and configure Oracle8 orOracle8i on a cluster. For information about these topics, see the referencedand supplemental documents listed in subsequent sections.

Referenced ManualsFor additional information, refer to documentation related to the specifichardware and software components of the Compaq Parallel Database Cluster.These related manuals include, but are not limited to:

■ Documentation related to the ProLiant servers you are clustering (forexample, guides, posters, and performance and tuning guides)

■ Compaq ServerNet documentation (for clusters using Oracle8i)

� ServerNet PCI Adapter Installation Guide

� ServerNet Switch Installation Guide

■ Compaq StorageWorks documentation

� Compaq StorageWorks RAID Array 4000 User Guide

� Compaq StorageWorks Fibre Channel Host Bus Adapter InstallationGuide

� Compaq StorageWorks Fibre Channel Storage Hub 7 InstallationGuide

� Compaq StorageWorks Fibre Channel Storage Hub 12 InstallationGuide

■ Microsoft Windows NT Server documentation

� Microsoft Windows NT Server Administrator’s Guide

� Microsoft Windows NT Server/Enterprise Edition Administrator’sGuide

xvi Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



■ Oracle8 Release 8.0.5 documentation

� Oracle8 Enterprise Edition Getting Started Release 8.0.5 forWindows NT

� Oracle Parallel Management User’s Guide

� Oracle8 Enterprise Edition CD

■ Oracle8i Release 8.1.5 documentation

� Oracle8i Parallel Server Setup and Configuration Guide Release8.1.5

� Oracle8i Enterprise Edition for Windows NT and Windows 95/98Release Notes, Release 8.1.5

� Oracle8i Enterprise Edition CD

Supplemental DocumentsThe following technical documents contain important supplementalinformation for the Compaq Parallel Database Cluster Model PDC/O2000:

■ Supported Ethernet Interconnects for Compaq Parallel DatabaseClusters Using Oracle Parallel Server (ECG062/0299), at

www.compaq.com/highavailability

■ Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8Parallel Server Release 8.0.5 Certification Matrix, at


■ Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8iParallel Server Release 8.1.5 Certification Matrix, at


■ Configuring Compaq RAID Technology for Database Servers, technote#1842061, at


■ Various technical white papers on Oracle and cluster sizing, which areavailable from Compaq ActiveAnswers website, at

www.compaq.com/activeanswers

About This Guide xvii



Text ConventionsThis document uses the following conventions to distinguish elements of text:

User Input, GUIUser Input, GUIUser Input, GUIUser Input, GUISelectionsSelectionsSelectionsSelections

Text a user types or enters appears in boldface.Items a user selects from a GUI, such as tabs,buttons, or menu items, also appear in boldface.User input and GUI selections can appear inuppercase and lowercase letters.

File Names, CommandNames, DirectoryNames, Drive Names

These elements can appear in uppercase andlowercase letters.

Menu Options, DialogBox Names

These elements appear in initial capital letters.

Type When you are instructed to type information, typethe information withoutwithoutwithoutwithout pressing the EnterEnterEnterEnter key.

Enter When you are instructed to enter information, typethe information and then press the EnterEnterEnterEnter key.

Symbols in TextThese symbols might be found in the text of this guide. They have thefollowing meanings.

WARNING: Text set off in this manner indicates that failure to follow directions

in the warning could result in bodily harm or loss of life.

CAUTION: Text set off in this manner indicates that failure to follow directions

could result in damage to equipment or loss of information.

IMPORTANT: Text set off in this manner presents clarifying information or specific

instructions.

NOTE: Text set off in this manner presents commentary, sidelights, or interesting points

of information.

xviii Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Symbols on EquipmentThese icons may be located on equipment in areas where hazardous conditionsmay exist.

Any surface or area of the equipment marked with these symbols

indicates the presence of electrical shock hazards. Enclosed area

contains no operator serviceable parts.

WARNING: To reduce the risk of injury from electrical shock hazards,

do not open this enclosure.

Any RJ-45 receptacle marked with these symbols indicates a Network

Interface Connection.

WARNING: To reduce the risk of electrical shock, fire, or damage to

the equipment, do not plug telephone or telecommunications

connectors into this receptacle.

Any surface or area of the equipment marked with these symbols

indicates the presence of a hot surface or hot component. If this

surface is contacted, the potential for injury exists.

WARNING: To reduce the risk of injury from a hot component, allow

the surface to cool before touching.

Power Supplies or Systems marked with these symbols

indicate the equipment is supplied by multiple sources of

power.

WARNING: To reduce the risk of injury from electrical shock,

remove all power cords to completely disconnect power from

the system.

About This Guide xix



Rack Stability

WARNING: To reduce the risk of personal injury or damage to the equipment,

be sure that:

■ The leveling jacks are extended to the floor.

■ The full weight of the rack rests on the leveling jacks.

■ The stabilizing feet are attached to the rack if it is a single rack installation.

■ The racks are coupled together in multiple rack installations.

■ A rack can become unstable if more than one component is extended forany reason. Extend only one component at a time.

Getting HelpIf you have a problem and have exhausted the information in this guide, youcan get further information and other help in the following locations.

Compaq Technical Support

In North America, call the Compaq Technical Phone Support Center at1-800-OK-COMPAQ1. This service is available 24 hours a day, 7 days a week.

Outside North America, call the nearest Compaq Technical Support PhoneCenter. Telephone numbers for worldwide Technical Support Centers arelisted on the Compaq website. Access the Compaq website by logging on tothe Internet at

www.compaq.com

1 For continuous quality improvement, calls may be recorded or monitored.

xx Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Be sure to have the following information available before you call Compaq:

■ Technical support registration number (if applicable)

■ Product serial numbers

■ Product model names and numbers

■ Applicable error messages

■ Add-on boards or hardware

■ Third-party hardware or software

■ Operating system type and revision level

■ Detailed, specific questions

Compaq Website

The Compaq website has information on this product as well as the latestdrivers and Flash ROM images. You can access the Compaq website bylogging on to the Internet at

http://www.compaq.com

Compaq Authorized Reseller

For the name of your nearest Compaq authorized reseller:

■ In the United States, call 1-800-345-1518.

■ In Canada, call 1-800-263-5868.

■ Elsewhere, see the Compaq website for locations and telephonenumbers.


Part Number: 161397-001 File Name: b-ch1 Clustering Overview.doc Last Saved On: 10/29/99 11:47 AM

Chapter 1Clustering Overview

For many years, companies have depended on clustered computer systems tofulfill two key requirements: to ensure users can access and processinformation that is critical to the ongoing operation of their business, and toincrease the performance and throughput of their computer systems at minimalcost. These requirements are known as availability and scalability,respectively.

Historically, these requirements have been fulfilled with clustered systemsbuilt on proprietary technology. Over the years, open systems haveprogressively and aggressively moved proprietary technologies intoindustry-standard products. Clustering is no exception. Its primary features,availability and scalability, have been moving into client/server products forthe last few years.

The absorption of clustering technologies into open systems products iscreating less expensive, non-proprietary solutions that deliver levels offunction commonly found in traditional clusters. While some uses of theproprietary solutions will always exist, such as those controlling stockexchange trading floors and aerospace mission controls, many criticalapplications can reach the desired levels of availability and scalability withnon-proprietary client/server-based clustering.

These new clustering solutions use industry-standard hardware and software,thereby providing key clustering features at a lower price than proprietaryclustering systems. Before examining the features and benefits of the CompaqParallel Database Cluster Model PDC/O2000 (PDC/O2000), it is helpful tounderstand the concepts and terminology of clustered systems.

1-2 Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Clusters DefinedA cluster is an integration of software and hardware products that enables a setof loosely coupled servers and shared storage subsystem components topresent a single system image to clients and to operate as a single system. As acluster, the group of servers and shared storage subsystem components offers alevel of availability and scalability far exceeding that obtained if each clusternode operated as a stand-alone server.

The PDC/O2000 uses Oracle8 Parallel Server Release 8.0.5 or Oracle8iParallel Server Release 8.1.5, each of which is a parallel database that candistribute its workload among the cluster nodes.

Figure 1-1 shows an example of a PDC/O2000, including two nodes (ProLiantservers), two Compaq StorageWorks RAID Array 4000s (RA4000 Arrays),two Compaq StorageWorks Fibre Channel Storage Hubs (Storage Hubs), aredundant cluster interconnect, and a client local area network (LAN).

A PDC/O2000 can contain multiple redundant Fibre Channel Arbitrated Loops(FC-ALs). In this example, the clustered nodes are connected to the databaseon the RA4000 Arrays through one redundant FC-AL. Clients access thedatabase through the client LAN, and the cluster nodes communicate acrossthe cluster interconnect.

Fibre HostAdapters (2)

RA4000 Array #1 RA4000 Array #2

StorageHub #1

Node 1 Node 2

StorageHub #2


ClusterInterconnectSwitch/Hub

Client LAN

Figure 1-1. Example of a Compaq Parallel DatabaseModel PDC/O2000 cluster

A PDC/O2000 running Oracle8 Release 8.0.5 supports a redundant Ethernetcluster interconnect. A PDC/O2000 running Oracle8i Release 8.1.5 supports aredundant Ethernet or Compaq ServerNet cluster interconnect.

Clustering Overview 1-3



AvailabilityWhen computer systems experience outages, the amount of time the system isunavailable is referred to as downtime. Downtime has several primary causes:hardware faults, software faults, planned service, operator error, andenvironmental factors. Minimizing downtime is a primary goal of a cluster.

Simply defined, availability is the measure of how well a computer system cancontinuously deliver services to clients.

Availability is a system-wide endeavor. The hardware, operating system, andapplications must be designed for availability. Clustering requires stability inthese components, then couples them in such a way that failure of one itemdoes not render the system unusable. By using redundant components andmechanisms that detect and recover from faults, clusters can greatly increasethe availability of applications critical to business operations.

ScalabilitySimply defined, scalability is a computer system characteristic that enablesimproved performance or throughput when supplementary hardware resourcesare added. Scalable systems allow increased throughput by adding componentsto an existing system without the expense of adding an entire new system.

In a stand-alone server configuration, scalable systems allow increasedthroughput by adding processors or more memory. In a cluster configuration,this result is usually obtained by adding cluster nodes.

Not only must the hardware benefit from additional components, but alsosoftware must be constructed in such a way as to take advantage of theadditional processing power. Oracle8 Parallel Server and Oracle8i ParallelServer distribute the workload among the cluster nodes. As more nodes areadded to the cluster, cluster-aware applications can use the parallel features ofOracle8 or Oracle8i Parallel Server to distribute workload among moreservers, thereby obtaining greater throughput.




Compaq Parallel DatabaseCluster Overview

As traditional clustering technology has moved into the open systems ofclient/server computing, Compaq has provided innovative, customer-focusedsolutions. The PDC/O2000 moves client/server computing one step closer tothe capabilities found in expensive, proprietary cluster solutions, at a fractionof the cost.

The PDC/O2000 combines the popular Microsoft Windows NT Serveroperating system and the industry-leading Oracle8 Parallel Server or Oracle8iParallel Server with award-winning Compaq ProLiant servers and sharedstorage subsystems.

Together, these hardware and software components provide improvedperformance through a truly scalable parallel application and improvedavailability using clustering software that rapidly recovers from detectablefaults. These components also provide improved availability throughconcurrent multinode database access using Oracle8 or Oracle8i ParallelServer.

Compaq Confidential – Need to Know RequiredWriter: John Blackburn Project: Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide Comments:

Part Number: 161397-001 File Name: c-ch2 Architecture.doc Last Saved On: 11/1/99 4:39 PM

Chapter 2Architecture

The Compaq Parallel Database Cluster Model PDC/O2000 (PDC/O2000) is anintegration of a number of different hardware and software products. Thischapter discusses how these products play a role in bringing a completeclustering solution to your computing environment.

The hardware products include:

■ Compaq ProLiant servers

■ Shared storage subsystem components

� Compaq StorageWorks RAID Array 4000 (RA4000 Array)

� Compaq StorageWorks RAID Array 4000 Controller (RA4000Array Controller)

� Compaq StorageWorks Storage Hub (Storage Hub)

� Compaq StorageWorks Fibre Channel Host Bus Adapter (Fibre HostAdapter)

� Gigabit Interface Converter-Shortwave (GBIC-SW) modules

� Fibre Channel cables

■ Cluster interconnect components

� NIC adapters (Ethernet or ServerNet)

� Cables (Ethernet or ServerNet)

� Switches/hubs (Ethernet or Compaq ServerNet)




The software products include:

■ Microsoft Windows NT Server 4.0 with Service Pack 3, 4, or 5

■ Compaq drivers and utilities

■ Oracle8 Enterprise Edition Release 8.0.5 with Oracle Parallel ServerOption Release 8.0.5 or Oracle8i Enterprise Edition Release 8.1.5 withthe Oracle8i Parallel Server Option Release 8.1.5

IMPORTANT: Compaq recommends Service Pack 4 or 5 for Windows NT Server for aredundant Ethernet cluster interconnect or client LAN. Using Service Pack 3 requiresinstalling the approved Microsoft patch (hot fix) article ID Q156655, entitled “MemoryLeak and STOP Screens Using Intermediate NDIS Drivers.”

Refer to Chapter 3, “Cluster Software Components,” for a description of thesoftware products used with the PDC/O2000.

Compaq ProLiant ServersA primary component of any cluster is the server. Each PDC/O2000 consistsof cluster nodes in which each node is a Compaq ProLiant server.

All nodes in a PDC/O2000 cluster must be identical in model. In addition, allcomponents common to all nodes in a cluster, such as memory, number ofCPUs, and the interconnect adapters, must be identical and identicallyconfigured.

NOTE: For an up-to-date list of Compaq Parallel Database Cluster-certified servers for thePDC/O2000 and detailed information about minimum and maximum clusterconfigurations, refer to the Compaq Parallel Database Cluster Model PDC/O2000 forOracle8 Parallel Server Release 8.0.5 Certification Matrix or the Compaq Parallel DatabaseCluster Model PDC/O2000 for Oracle8i Parallel Server Release 8.1.5 Certification Matrix.These documents are available on the Compaq website at


Architecture 2-3



High-Availability Features of ProLiant Servers

In addition to the increased application and data availability enabled byclustering, ProLiant servers include many reliability features that provide asolid foundation for effective clustered server solutions. The PDC/O2000cluster is based on ProLiant servers, most of which offer excellent reliabilitythrough redundant power supplies, redundant cooling fans, and ErrorChecking and Correcting (ECC) memory. The high-availability features ofProLiant servers are a critical foundation of Compaq clustering products.Table 2-1 lists the high-availability features found in many ProLiant servers.

Table 2-1High-Availability Components of ProLiant Servers

Hot-pluggable hard drives Redundant power supplies

Digital Linear Tape (DLT) Array (optional) ECC-protected processor-memory bus

Uninterruptible power supplies (optional) Redundant processor power modules

ECC memory PCI hot plug sots (in some servers)

Offline backup processor Redundant cooling fans

Shared Storage Subsystem ComponentsThe PDC/O2000 is based on a cluster architecture known as shared storageclustering, in which clustered nodes share access to a common set of shareddisk drives.

RA4000 Array

The RA4000 Array is the shared storage solution for the PDC/O2000. Eachredundant Fibre Channel Arbitrated Loop (FC-AL) contains from one to fiveRA4000 Arrays. Each RA4000 Array contains two single-port RA4000 ArrayControllers. Each array controller connects the RA4000 Array to one StorageHub.

The RA4000 Array can hold twelve 1-inch high or eight 1.6-inch highWide-Ultra SCSI drives. The drives must be mounted on Compaqhot-pluggable drive trays. SCSI IDs are assigned automatically according totheir drive location, allowing 1-inch and 1.6-inch drives to be intermixedwithin the same RA4000 Array.

The RA4000 Array comes in either a rack-mountable or a tower model.




For more information about the RA4000 Array, refer to the CompaqStorageWorks RAID Array 4000 User Guide.

RA4000 Array Controllers

Two single-port RA4000 Array Controllers are installed in each RA4000Array. One array controller is configured as the active controller; the other isthe standby controller. Only one array controller can be active at any giventime. To ensure fault tolerance of shared storage on the RA4000 Array, thetwo array controllers must be connected to a different Storage Hub.

From the perspective of the cluster nodes, each RA4000 Array Controller issimply another device connected to one of the cluster’s I/O paths.Consequently, each node sends its I/O requests to the active RA4000 ArrayController just as it would to any SCSI device. The RA4000 Array Controllerreceives the I/O requests from the nodes and directs them to the shared storagedisks to which it has been configured. Because the array controller processesthe I/O requests, the cluster nodes are not burdened with the I/O processingtasks associated with reading and writing data to multiple shared storagedevices.

When an RA4000 Array and the cluster nodes to which it is physicallyconnected are first powered on, the RA4000 Array communicates with thenodes to identify which of its two array controller slots contains the activearray controller. Note that the location of this active slot (top rear or bottomrear in RA4000 Array rack models, right rear or left rear in tower models) isnot always the same. The array controller that is installed in the active slot isautomatically assigned active status by Compaq Redundancy Manager,without the need for any further configuration.

To change the active slot location, you must use Redundancy Manager tomake the controller in the other slot the active array controller. Forinformation about configuring the standby array controller to be active, refer to“Defining Active Array Controllers” in Chapter 5, “Installation andConfiguration for Oracle8 Release 8.0.5,” or Chapter 6, “Installation andConfiguration for Oracle8i Release 8.1.5.”

If the active RA4000 Array Controller in an RA4000 Array fails, RedundancyManager causes the standby controller to become the active array controller.

Architecture 2-5



Access to the same logical disks is provided to both RA4000 Array Controllersto allow for successful failovers. In this configuration, both the active andstandby array controllers are configured to receive and transmit data for thesame logical disks.

For more information about the RA4000 Array Controller, refer to theCompaq StorageWorks RAID Array 4000 User Guide.

Storage Hubs

Storage Hubs are used to connect the Fibre Host Adapters in cluster nodeswith the array controllers in RA4000 Arrays. Two Storage Hubs are used ineach redundant FC-AL of a PDC/O2000, one for each FC-AL path. Using twoStorage Hubs provides fault tolerance and supports the redundant architecturedescribed in “Redundant Fibre Channel Arbitrated Loop” in this chapter.

On each Storage Hub, one port is used by one Fibre Host Adapter in each nodeand one port is used to connect to one of the two array controllers in eachRA4000 Array.

The PDC/O2000 allows the use of either the Storage Hub 7 (with 7 ports) orthe Storage Hub 12 (with 12 ports). Using the Storage Hub 7 limits the size ofthe PDC/O2000 cluster. For example, a cluster with four cluster nodes andfour RA4000 Arrays requires Storage Hubs with at least 8 ports (Storage Hub12s). In your selection of Storage Hubs, you should also consider thelikelihood of cluster growth.

Refer to the Compaq StorageWorks Fibre Channel Storage Hub 7 InstallationGuide and the Compaq StorageWorks Fibre Channel Storage Hub 12Installation Guide for further information about these products.

Fibre Host Adapters

Each cluster node contains two Fibre Host Adapters, one for each FC-AL path.The top (or leftmost) Fibre Host Adapter in every node is connected to thesame Storage Hub. The bottom (or rightmost) Fibre Host Adapter in everynode is connected to the other Storage Hub.

If the cluster contains multiple redundant FC-ALs, each redundant FC-ALmust have its own dedicated pair of Fibre Host Adapters in each cluster node.




Compaq Redundancy Manager software is installed on each cluster node toensure the proper detection of failures on an active FC-AL path and successfulfailover to the standby FC-AL path. For information about installingRedundancy Manager, see “Installing Compaq Redundancy Manager” inChapter 5, “Installation and Configuration for Oracle8 Release 8.0.5,” orChapter 6, “Installation and Configuration for Oracle8i Release 8.1.5.”

For more information about the Fibre Channel Host Adapter, refer to theCompaq StorageWorks Fibre Channel Host Bus Adapter Installation Guide.

Gigabit Interface Converter-Shortwave

A Gigabit Interface Converter-Shortwave (GBIC-SW) module must beinstalled at both ends of a Fibre Channel cable. You insert the module intoeach Fibre Host Adapter, Storage Hubs, and RA4000 Array Controller.

Four GBIC-SW modules are provided with each RA4000 Array (two for eachRA4000 Array Controller) and two are provided with each Fibre HostAdapter.

GBIC-SW modules provide 100 MB/second performance. Fibre Channelcables connected to these modules can be up to 500 meters in length.

Fibre Channel Cables

Shortwave (multi-mode) fibre optic cables are used to connect the nodes,Storage Hubs, and RA4000 Arrays in a PDC/O2000 cluster.

Shared Storage Subsystem FeaturesThis section describes these features of the shared storage subsystem for thePCD/O2000:

■ Maximum distances between cluster nodes and shared storagesubsystem components

■ Redundant Fibre Channel Arbitrated Loops

■ Multiple redundant Fibre Channel Arbitrated Loops

■ I/O data paths

■ SCSI disks

Architecture 2-7



Maximum Distances Between Cluster Nodes andShared Storage Subsystem Components

By using standard short-wave Fibre Channel cables with Gigabit InterfaceConverter-Shortwave (GBIC-SW) modules, each RA4000 Array can be placedup to 500 meters from the Storage Hubs, and the Storage Hubs can be placedup to 500 meters from the Fibre Host Adapters in the cluster nodes. SeeFigure 2-1.



StorageHub #1

Node 1

500 metersmaximum

500 metersmaximum

Node 2

StorageHub #2


Figure 2-1. Maximum distances between PDC/O2000 cluster nodes andshared storage subsystem components

Redundant Fibre Channel Arbitrated Loop

Each redundant FC-AL in a PDC/O2000 cluster provides two independent I/Ocommunication paths between one pair of Fibre Host Adapters in each nodeand the RA4000 Arrays connected to them.




Figure 2-2 shows a two-node PDC/O2000 cluster with one redundant FC-AL.



StorageHub #1

Node 1 Node 2

StorageHub #2


Figure 2-2. Two-node PDC/O2000 cluster with one redundant FibreChannel Arbitrated Loop

The term “redundant” does not mean that there are just two I/O cable runs inthe FC-AL. Instead, it refers to the fact that there are two independent pathsfrom each node to each RA4000 Array. In Figure 2-2, light and dark FibreChannel cable lines differentiate these redundant paths.

Used in conjunction with the I/O path failover capabilities of CompaqRedundancy Manager software, this redundant path configuration gives clusterresources increased availability and fault tolerance.

Architecture 2-9



Multiple Redundant Fibre Channel Arbitrated Loops

The PDC/O2000 supports the use of multiple redundant Fibre ChannelArbitrated Loops within the same cluster. You would install additionalredundant FC-ALs in a PDC/O2000 cluster to:

■ Increase the amount of shared storage space available to the cluster’snodes. Each redundant FC-AL can connect to a maximum of fiveRA4000 Arrays. These RA4000 Arrays are available only to the FibreHost Adapters in that redundant FC-AL.

■ Increase the PDC/O2000 cluster’s I/O performance.

Adding a second redundant FC-AL involves duplicating the hardwarecomponents used in the first redundant FC-AL. Note, however, that anyredundant FC-AL in a cluster does not need to contain the maximum fiveRA4000 Arrays.

Each redundant FC-AL consists of the following hardware:

■ Two Fibre Host Adapters in each node

■ Two Storage Hubs

■ Up to five RA4000 Arrays, each containing two array controllers

■ Fibre Channel cables to connect the Fibre Host Adapters to the StorageHubs and the Storage Hubs to the RA4000 Array Controllers

■ GBIC-SW modules for Fibre Host Adapters, Storage Hubs, and RA4000Array Controllers

The maximum number of redundant FC-ALs allowed in a PDC/O2000 clusteris determined by the maximum number of Fibre Host Adapters you can installin each server model supported for the PDC/O2000. Refer to the Compaqdocumentation for your server for this information.




Figure 2-3 shows a two-node PDC/O2000 cluster with two redundant FC-ALs.Each redundant FC-AL has its own pair of Fibre Host Adapters in each node, apair of Storage Hubs, and one or more RA4000 Arrays (each with twoRA4000 Array Controllers). In Figure 2-3, the hardware components thatconstitute the second redundant FC-AL are shaded.



StorageHub #1

Node 1 Node 2

StorageHub #2


StorageHub #1

RedundantFC-AL #1

RedundantFC-AL #2

StorageHub #2


Figure 2-3. Two-node PDC/O2000 cluster with two redundant Fibre ChannelArbitrated Loops

Architecture 2-11



I/O Data Paths

Each redundant FC-AL in a PDC/O2000 cluster has two distinct data paths:

■ One data path runs from the leftmost (or top) Fibre Host Adapter in eachnode’s adapter pair to one Storage Hub, then on to one array controllerin each RA4000 Array.

■ A second data path runs from the rightmost (or bottom) Fibre HostAdapter in each node’s adapter pair to the second Storage Hub, then onto one array controller in each RA4000 Array.

For further information about I/O hardware failure events and their failoverresponses, refer to tables 2-3, 2-4, and 2-5.

Fibre Host Adapter-to-Storage Hub Data Paths

Figure 2-4 highlights the I/O data paths that run between the Fibre HostAdapters and the Storage Hubs.



StorageHub #1

Node 1 Node 2

StorageHub #2


Figure 2-4. Host adapter-to-Storage Hub data paths

Redundancy Manager monitors the status of the components along the activeand standby FC-AL paths. If Redundancy Manager detects the failure of aFibre Host Adapter, Fibre Channel cable, or Storage Hub along an active path,it automatically transfers all I/O activity on that path to the standby FC-ALpath.

For detailed information about I/O hardware failure events and their failoverresponses, refer to “Summary of I/O Path Failure and Failover Scenarios” inthis chapter.




Storage Hub-to-RA4000 Array Data Paths

Figure 2-5 highlights the I/O data paths that run between the Storage Hubs andthe two RA4000 Array Controllers installed in each RA4000 Array.



StorageHub #1

Node 1 Node 2

StorageHub #2


Figure 2-5. Storage Hub-to-RA4000 Array data paths

If any component along an active path fails, Redundancy Manager detects thefailure and automatically transfers all I/O activity to the appropriatecomponent or components on the standby path, including the standby RA4000Array Controller in each affected RA4000 Array and the Storage Hub andFibre Host Adapter to which the controller is connected.

For detailed information about I/O hardware failure events and their failoverresponses, refer to “Summary of I/O Path Failure and Failover Scenarios” inthis chapter.

Architecture 2-13



SCSI Disks

The RA4000 Array uses standard hot-plug, Wide-Ultra SCSI disk drives,ensuring investment protection for existing SCSI users.

Each RA4000 Array can hold up to twelve 1-inch or eight 1.6-inch Wide-UltraSCSI drives. A number of drive storage capacities are supported. Due to thisvariety of capacities, the choice of which drives to use in the enclosures is leftto the system administrator.

In a shared disk clustering scheme like the PDC/O2000, the SCSI drives mustbe 100 percent compatible with the clustering software. Compaq has workeddiligently to ensure that Compaq SCSI drives meet the stringent needs ofclustering.

I/O Path Configuration GuidelinesYou can use either of two I/O path configurations for every redundant FibreChannel Arbitrated Loop in your PDC/O2000 cluster:

■ Active/standby configuration

■ Active/active configuration

In the active/standby configuration, only one of the two Fibre Host Adapters ineach node is active at any one time; the other Fibre Host Adapter is in thestandby state. The active Fibre Host Adapter is connected to the first StorageHub, which is connected to the active array controller in each RA4000 Array.The standby Fibre Host Adapter is connected to the second Storage Hub,which is connected to the standby array controller in each RA4000 Array. Thestandby Fibre Host Adapter remains in the standby state unless a failover fromthe active I/O path occurs. In the active/standby configuration, the failure ofany component along an active I/O path (Fibre Host Adapter, Storage Hub,active array controller, or Fibre Channel cables) causes Redundancy Managerto implement a complete failover to the components on the standby I/O path.

In the active/active configuration, both Fibre Host Adapters in each node aresimultaneously active. Both are active because each Fibre Host Adapter isconnected to a Storage Hub that, in turn, is connected to an active arraycontroller in each RA4000 Array. Because each Fibre Host Adapter andStorage Hub must connect to at least one active array controller, two or moreRA4000 Arrays must be present in an active/active configuration.




See “Active/Standby Configuration Examples “ in this chapter for a detaileddescription of active/standby configuration examples for a redundant FC-ALwith from one to five RA4000 Arrays. See “Active/Active ConfigurationExamples” for a detailed description of active/active configuration exampleswhen two to five RA4000 Arrays are present.

Table 2-2 identifies the features of the active/standby and active/activeconfigurations.

Table 2-2Features of Active/Standby and Active/Active Configurations

I/O Path Configuration Advantage Disadvantage

Active/standby with one RA4000Array

Active/standby is the only I/Opath configuration you can usein a redundant FC-AL thatcontains just one RA4000 Array.

Active/standby with two or moreRA4000 Arrays

Provides true cabling symmetrybetween Storage Hubs and arraycontrollers; a Storage Hubconnects to the same arraycontroller slot (top or bottom) inevery RA4000 Array.

Load balancing between the twoStorage Hubs is less than idealbecause the connection to theactive array controller in everyRA4000 Array is routed throughthe same Storage Hub. Thesecond Storage Hub provides noactive I/O pathway unless anactive array controller or itscable connection to the firstStorage Hub fails.

continued

Architecture 2-15



Table 2-2Features of Active/Standby and Active/Active Configurations continued

I/O Path Configuration Advantage Disadvantage

Active/active with two or moreRA4000 Arrays

Provides a small but measurableimprovement in I/O performanceover the active/standbyconfiguration because both FibreHost Adapters in each node andboth Storage Hubs aresimultaneously active. Thisimprovement can be meaningfulfor customers with large clusterdatabases or high I/Otransaction requirements.

Provides better load balancingbetween the two Storage Hubsthan the active/standbyconfiguration. Both StorageHubs are connected to the sameor equivalent numbers of bothactive and standby arraycontrollers in the RA4000Arrays.

Does not provide true cablingsymmetry between StorageHubs and array controllers if youconsistently configure the top orrightmost array controller as theactive controller. Each StorageHub is cabled to top (active)array controllers in someRA4000 Arrays and bottom(standby) controllers in others.You can achieve cablingsymmetry if you configure thebottom array controller in someRA4000 Arrays as active.However, this requires usingRedundancy Manager toconfigure the lower arraycontroller as active if it is instandby mode.




I/O Path Configuration Rules

The following rules must be observed in I/O path configurations forPDC/O2000 clusters:

■ Each active/standby or active/active configuration is confined to oneredundant FC-AL.

■ For each redundant FC-AL, two Fibre Host Adapters are installed ineach cluster node.

■ For each redundant FC-AL, two Storage Hubs (Storage Hub #1 andStorage Hub #2) are installed between the nodes and the RA4000Arrays.

■ From one to five RA4000 Arrays can be installed in one redundantFC-AL.

■ A minimum of one RA4000 Array is required for the active/standbyconfiguration.

■ A minimum of two RA4000 Arrays is required for the active/activeconfiguration.

■ Each RA4000 Array must contain two array controllers.

■ Only one of the two array controllers in an RA4000 Array can be activeat a given time. The other array controller is the standby controller.

■ I/O path hardware components must be connected using Fibre Channelcables and GBIC-SW modules.

Architecture 2-17



Active/Standby Configuration Examples

This section describes examples of active/standby configurations when one,two, three, four, and five RA4000 Arrays are present in one redundant FC-ALof a four-node PDC/O2000 cluster. These examples represent one method forconfiguring active/standby configurations. They are presented here to providea relatively simple and consistent method for building active/standbyconfigurations.

IMPORTANT: Figures 2-6 through 2-10 show active/standby configurations for afour-node cluster. Active/standby configurations for clusters with two, three, five, or morenodes are not described here. However, the illustrated active/standby configurationexamples provided should supply sufficient information for building an active/standbyconfiguration in any PDC/O2000 cluster.

The active/standby configuration examples shown in Figures 2-6 through 2-10follow these configuration guidelines:

■ For every Fibre Host Adapter pair, the top or leftmost Fibre HostAdapter in each node is connected to the odd-numbered Storage Hub(Storage Hub #1). This is the active Fibre Host Adapter in the pair.

■ For every Fibre Host Adapter pair, the bottom or rightmost Fibre HostAdapter in each node is connected to the even-numbered Storage Hub(Storage Hub #2). This is the standby Fibre Host Adapter in the pair.

■ In each RA4000 Array, the top (rack model) or right rear (tower model)array controller is always the active controller.

■ In each RA4000 Array, the bottom (rack model) or left rear (towermodel) array controller is always the standby controller.

■ The odd-numbered Storage Hub (Storage Hub #1) is connected to theactive array controller in each RA4000 Array.

■ The even-numbered Storage Hub (Storage Hub #2) is connected to thestandby array controller in each RA4000 Array.

NOTE: The following active/standby configurations are examples only. You are notrequired to follow these configurations.




For more information about installing active/standby configurations, refer tothese sections in Chapter 5, “Installation and Configuration for Oracle8Release 8.0.5” or Chapter 6, “Installation and Configuration for Oracle8iRelease 8.1.5”:

■ Cabling the Fibre Host Adapters to the Storage Hubs

■ Cabling the Storage Hubs to the RA4000 Array Controllers

In Figures 2-6 through 2-10, active I/O path components have been shaded todistinguish them from standby (inactive) components. Black Fibre Channelcables identify connections between active components; gray cables identifyconnections between standby components.

Active/Standby Configuration with One RA4000Array

Figure 2-6 shows an active/standby I/O path configuration for a four-nodecluster with one RA4000 Array.

Fibre HostAdapter(Active)

Fibre HostAdapter

(Standby)

RA4000 Array #1Standby Array Controller

Active Array Controller

StorageHub #1

Node 1 Node 2 Node 3 Node 4

StorageHub #2

ProLiantServers (4)

Figure 2-6. Active/standby configuration with one RA4000 Array

Architecture 2-19



Active/Standby Configuration with Two RA4000Arrays

Figure 2-7 shows an active/standby I/O path configuration for a four-nodecluster with two RA4000 Arrays.


Fibre HostAdapter

(Standby)

RA4000Array #1

RA4000Array #2

StandbyArray Controller

ActiveArray Controller

StorageHub #1


StorageHub #2

ProLiantServers (4)

Figure 2-7. Active/standby configuration with two RA4000 Arrays




Active/Standby Configuration with ThreeRA4000 Arrays

Figure 2-8 shows an active/standby I/O path configuration for a four-nodecluster with three RA4000 Arrays.


Fibre HostAdapter

(Standby)

RA4000Array #1

RA4000Array #3

RA4000Array #2



StorageHub #1


StorageHub #2

ProLiantServers (4)

Figure 2-8. Active/standby configuration with three RA4000 Arrays

Architecture 2-21



Active/Standby Configuration with Four RA4000Arrays

Figure 2-9 shows an active/standby I/O path configuration for a four-nodecluster with four RA4000 Arrays.

RA4000Array #1

RA4000Array #3

RA4000Array #4

RA4000Array #2



StorageHub #1


StorageHub #2

ProLiantServers (4)


Fibre HostAdapter

(Standby)

Figure 2-9. Active/standby configuration with four RA4000 Arrays




Active/Standby Configuration with Five RA4000Arrays

Figure 2-10 shows an active/standby I/O path configuration for a four-nodecluster with five RA4000 Arrays.

RA4000Array #1

RA4000Array #3

RA4000Array #5

RA4000Array #4

RA4000Array #2



StorageHub #1


StorageHub #2

ProLiantServers (4)


Fibre HostAdapter

(Standby)

Figure 2-10. Active/standby configuration with five RA4000 Arrays

Architecture 2-23



Active/Active Configuration Examples

This section describes examples of active/active configurations when two,three, four, and five RA4000 Arrays are present in one redundant FC-AL of afour-node PDC/O2000 cluster. These examples represent one method forconfiguring active/active configurations. They are presented here to provide arelatively simple and consistent method for building active/activeconfigurations.

IMPORTANT: Figures 2-11 through 2-14 show active/active configurations for afour-node cluster. Active/active configurations for clusters with two, three, five, or morenodes are not described here. However, the illustrated active/active configurationexamples provided should supply sufficient information for building an active/standbyconfiguration in any PDC/O2000 cluster.

The active/active configuration examples shown in Figures 2-11 through 2-14follow these configuration guidelines:

■ For every Fibre Host Adapter pair, the top or leftmost Fibre HostAdapter in each node is connected to the odd-numbered Storage Hub(Storage Hub #1). This is an active Fibre Host Adapter.

■ For every Fibre Host Adapter pair, the bottom or rightmost Fibre HostAdapter in each node is connected to the even-numbered Storage Hub(Storage Hub #2). This is also an active Fibre Host Adapter.

■ In each RA4000 Array, the top (rack model) or right rear (tower model)array controller is always the active controller.

■ In each RA4000 Array, the bottom (rack model) or left rear (towermodel) array controller is always the standby controller.

■ The odd-numbered Storage Hub (Storage Hub #1) is connected to theactive array controller in each odd-numbered RA4000 Array(1, 3, and 5).

■ The odd-numbered Storage Hub (Storage Hub #2) is connected to thestandby array controller in each even-numbered RA4000 Array(2 and 4).

■ The even-numbered Storage Hub (Storage Hub #2) is connected to theactive array controller in each even-numbered RA4000 Array (2 and 4).

■ The even-numbered Storage Hub is connected to the standby arraycontroller in each odd-numbered RA4000 Array (1, 3, and 5).

NOTE: The following active/active configurations are examples only. You are not requiredto follow these configurations.




For more information about installing active/active configurations, refer tothese sections in Chapter 5, “Installation and Configuration for Oracle8Release 8.0.5” or Chapter 6, “Installation and Configuration for Oracle8iRelease 8.1.5”:

■ Cabling the Fibre Host Adapters to the Storage Hubs

■ Cabling the Storage Hubs to the RA4000 Array Controllers

In Figures 2-11 through 2-14, active I/O path components have been shaded todistinguish them from standby (inactive) components. Black Fibre Channelcables identify connections between active components; gray cables identifyconnections between standby components.

Active/Active Configuration with Two RA4000Arrays

Figure 2-11 shows an active/active I/O path configuration for a four-nodecluster with two RA4000 Arrays.


RA4000Array #1

RA4000Array #2



StorageHub #1


StorageHub #2

ProLiantServers (4)

Figure 2-11. Active/active configuration with two RA4000 Arrays

Architecture 2-25



Active/Active Configuration with Three RA4000Arrays

Figure 2-12 shows an active/active I/O path configuration for a four-nodecluster with three RA4000 Arrays.

RA4000Array #1

RA4000Array #3

RA4000Array #2



StorageHub #1


StorageHub #2

ProLiantServers (4)


Fibre HostAdapter

(Standby)

Figure 2-12. Active/active configuration with three RA4000 Arrays




Active/Active Configuration with Four RA4000Arrays

Figure 2-13 shows an active/active I/O path configuration for a four-nodecluster with four RA4000 Arrays.

RA4000Array #1

RA4000Array #3

RA4000Array #4

RA4000Array #2



StorageHub #1


StorageHub #2

ProLiantServers (4)


Figure 2-13. Active/active configuration with four RA4000 Arrays

Architecture 2-27



Active/Active Configuration with Five RA4000Arrays

Figure 2-14 shows an active/active I/O path configuration for a four-nodecluster with five RA4000 Arrays.

RA4000Array #1

RA4000Array #3

RA4000Array #5

RA4000Array #4

RA4000Array #2



StorageHub #1


StorageHub #2

ProLiantServers (4)


Fibre HostAdapter

(Standby)

Figure 2-14. Active/active configuration with five RA4000 Arrays




Summary of I/O Path Failure and FailoverScenarios

Table 2-3 identifies possible I/O path failure events for active/standbyconfigurations with one RA4000 Array and the failover response, if any,implemented by Redundancy Manager for each failure.

Table 2-3I/O Path Failure and Failover Scenarios for Active/Standby Configurations With

One RA4000 Array

Description of Failure Failover Response

The active array controller in theRA4000 Array fails.

Redundancy Manager forces a complete failover to the standby I/Opath components, including the standby Fibre Host Adapter in eachnode, the other Storage Hub, the standby array controller in theRA4000 Array, and Fibre Channel cables.

The standby array controller in theRA4000 Array fails.

None

The Fibre Channel cableconnection between the activearray controller and its StorageHub is broken.


The Fibre Channel cableconnection between the standbyarray controller and its StorageHub is broken.

None

The Storage Hub connected to theactive array controller fails.


The Storage Hub connected to thestandby array controller fails.

None

The Fibre Channel cableconnection between a Fibre HostAdapter and the Storage Hubconnected to the active arraycontroller is broken.


continued

Architecture 2-29



Table 2-3I/O Path Failure and Failover Scenarios for Active/Standby Configurations With One RA4000Array continued


The Fibre Channel cableconnection between a Fibre HostAdapter and the Storage Hubconnected to the standby arraycontroller is broken.

None

A Fibre Host Adapter connected tothe Storage Hub that connects tothe active array controller fails.


A Fibre Host Adapter connected tothe Storage Hub that connects tothe standby array controller fails.

None

Table 2-4 identifies possible I/O path failure events for active/standbyconfigurations with two or more RA4000 Arrays and the failover response, ifany, implemented by Redundancy Manager for each failure.

Table 2-4I/O Path Failure and Failover Scenarios for Active/Standby Configurations With

Two or More RA4000 Arrays


The active array controller in oneRA4000 Array fails.

Redundancy Manager forces a complete failover to the standby I/Opath components, including the standby Fibre Host Adapter in eachnode, the other Storage Hub, the standby array controller in eachRA4000 Array, and Fibre Channel cables.

The standby array controller in oneRA4000 Array fails.

None

The Fibre Channel cableconnection between the activearray controller in one RA4000Array and its Storage Hub isbroken.


continued




Table 2-4I/O Path Failure and Failover Scenarios for Active/Standby Configurations With Two or MoreRA4000 Arrays continued


The Fibre Channel cableconnection between the standbyarray controller in one RA4000Array and its Storage Hub isbroken.

None

The Storage Hub that is connectedto the active array controller ineach RA4000 Array fails.


The Storage Hub that is connectedto the standby array controller ineach RA4000 Array fails.

None

The Fibre Channel cableconnection between the StorageHub that is connected to the activearray controllers and a Fibre HostAdapter is broken.

Redundancy Manager forces a complete failover to the standby I/Opath components, including the standby Fibre Host Adapter in eachnode, the other Storage Hub, the standby array controller in theeach RA4000 Array, and Fibre Channel cables.

The Fibre Channel cableconnection between the StorageHub that is connected to thestandby array controllers and aFibre Host Adapter is broken.

None

A Fibre Host Adapter connected tothe Storage Hub that is connectedto the active array controllers fails.


A Fibre Host Adapter connected tothe Storage Hub that is connectedto the standby array controllersfails.

None

Architecture 2-31



Table 2-5 identifies possible I/O path failure events for active/activeconfigurations with two or more RA4000 Arrays and the failover response, ifany, implemented by Redundancy Manager for each failure.

Table 2-5I/O Path Failure and Failover Scenarios for Active/Active Configurations With Two

or More RA4000 Arrays


The active array controller in oneRA4000 Array fails.

Redundancy Manager makes the standby array controller in theRA4000 Array active and reroutes I/O activity to that arraycontroller. The Storage Hub that is connected to the new activearray controller becomes the active I/O path to this RA4000 Array.In each node, I/O activity between the Fibre Host Adapter that isconnected to the failed array controller is rerouted to the secondFibre Host Adapter in the pair, but only along the I/O path to theaffected RA4000 Array. The first Fibre Host Adapter in each nodecontinues to be the active I/O path for active array controllers inother RA4000 Arrays to which it is connected.

The standby array controller in oneRA4000 Array fails.

None

The Fibre Channel cableconnection between the activearray controller in one RA4000Array and its Storage Hub isbroken.

Redundancy Manager makes the active array controller to whichthe failed cable is connected inactive. The standby array controllerin the RA4000 Array becomes active. I/O activity is routed throughthe Fibre Channel cable installed to the new active array controllerand the other Storage Hub. In each node, I/O activity between theFibre Host Adapter that is connected to the failed array controller isrerouted to the second Fibre Host Adapter in the pair, but only alongthe I/O path to the affected RA4000 Array. The first Fibre HostAdapter in each node continues to be the active I/O path for activearray controllers in other RA4000 Arrays to which it is connected.

continued




Table 2-5I/O Path Failure and Failover Scenarios for Active/Active Configurations With Two or MoreRA4000 Arrays continued


The Fibre Channel cableconnection between the standbyarray controller in one RA4000Array and its Storage Hub isbroken.

None

A Storage Hub fails. Redundancy Manager makes each active array controller to whichthe failed Storage Hub is connected inactive. The standby arraycontroller in each affected RA4000 Array becomes the active arraycontroller. The Storage Hub that is connected to the new activearray controllers becomes the active I/O path for these RA4000Arrays. In each node, the Fibre Host Adapter that is connected tothe failed Storage Hub becomes inactive, and all I/O activity isrerouted through the other Fibre Host Adapter in the pair and theremaining active Storage Hub.

The Fibre Channel cableconnection between a StorageHub and a Fibre Host Adapter inone node is broken.

Redundancy Manager makes all I/O path connections between theaffected Fibre Host Adapter and active array controllers inactive.The standby array controller in every affected RA4000 Arraybecomes the active array controller. The Storage Hub connected tothe newly active array controllers becomes the active I/O path forthe entire FC-AL. The Storage Hub connected to the failed FibreChannel cable becomes inactive. The second Fibre Host Adapter ineach node, which is connected to the only active Storage Hub,becomes the only active Fibre Host Adapter in the node’s pair.

A Fibre Host Adapter in a nodefails.

Redundancy Manager makes all I/O path connections between theaffected Fibre Host Adapter and active array controllers inactive.The standby array controller in every affected RA4000 Arraybecomes the active array controller. The Storage Hub connected tothe newly active array controllers becomes the active I/O path forthe entire FC-AL. The Storage Hub connected to the failed FibreHost Adapter becomes inactive. The second Fibre Host Adapter ineach node, which is connected to the only active Storage Hub,becomes the only active Fibre Host Adapter in the node’s pair.

Architecture 2-33



Cluster Interconnect OptionsThe cluster interconnect is the data path over which all of the nodes in aPDC/O2000 cluster communicate. The nodes use the cluster interconnect datapath to:

■ Communicate individual resource and overall cluster status

■ Send and receive heartbeat signals

■ Coordinate database locks through the Oracle Integrated DistributedLock Manager

NOTE: Several terms for cluster interconnect are used throughout the industry. Theseinclude private LAN, private interconnect, system area network, and private network.Throughout this guide, the term cluster interconnect is used.

The PDC/O2000 uses these types of cluster interconnects:

■ Clusters using Oracle8 Parallel Server Release 8.0.5 must use aredundant Ethernet cluster interconnect.

■ Clusters using Oracle8i Parallel Server Release 8.1.5 can use aredundant Ethernet cluster interconnect or a redundant ServerNet clusterinterconnect.

IMPORTANT: A redundant cluster interconnect uses redundant hardware to provide faulttolerance along the entire cluster interconnect path. In keeping with the redundant designof the shared storage subsystem, a redundant cluster interconnect is required for thePDC/O2000.




Redundant Ethernet Cluster Interconnect

NOTE: Refer to the technical white paper Supported Ethernet Interconnects for CompaqParallel Database Clusters Using Oracle Parallel Server for detailed information aboutconfiguring redundant Ethernet cluster interconnects. This document is available at


A redundant Ethernet cluster interconnect uses the following component sets:

■ Two Ethernet adapters in each cluster node

■ Two 100 Mbit/sec Ethernet switches or hubs for two-node clusters

■ Two 100 Mbit/sec Ethernet switches for clusters with three or morenodes

■ One Ethernet crossover cable installed between the two Ethernetswitches or hubs

IMPORTANT: Compaq recommends Service Pack 4 or 5 for Windows NT Server for aredundant Ethernet cluster interconnect or client LAN. Using Service Pack 3 requiresinstalling the approved Microsoft patch (hot fix) article ID Q156655, entitled “ MemoryLeak and STOP Screens Using Intermediate NDIS Drivers.”

Ethernet Cluster Interconnect Adapters

NOTE: For recommended dual-port and single-port Ethernet adapters for your cluster,see the Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8 Parallel ServerRelease 8.0.5 Certification Matrix or the Compaq Parallel Database Cluster ModelPDC/O2000 for Oracle8i Parallel Server Release 8.1.5 Certification Matrix. Thesedocuments are available at


To implement the Ethernet cluster interconnect, each cluster node must beequipped with Ethernet adapters capable of 100 Mbit/sec transfer rates. Someadapters may be capable of 10 Mbit/sec and 100 Mbit/sec; however, adaptersused for the Ethernet cluster interconnect must run at 100 Mbit/sec.

The Ethernet adapters must have passed Windows NT Server 4.0 hardwarecompatibility test (HCT) certification.

Architecture 2-35



If you are using dual-port Ethernet adapters, you can use one port for theEthernet cluster interconnect and the second port for the client LAN. Refer tothe technical white paper Supported Ethernet Interconnects for CompaqParallel Database Clusters Using Oracle Parallel Server at


For detailed information about installing a redundant Ethernet clusterinterconnect, see Chapter 5, “Installation and Configuration for Oracle8Release 8.0.5,” or Chapter 6, “Installation and Configuration for Oracle8iRelease 8.1.5.”

Ethernet Switch or Hub

IMPORTANT: The Ethernet switches or hubs used for the Ethernet cluster interconnectmust be dedicated to the cluster interconnect. They cannot be connected to the client LANor to cluster nodes that are not part of the PDC/O2000 cluster.

In a two-node PDC/O2000 cluster, two 100 Mbit/sec Ethernet switches orhubs are connected by cables to the two Ethernet adapters in each node.

In PDC/O2000 clusters with three or more nodes, two 100 Mbit/sec Ethernetswitches are required for the cluster interconnect path; hubs cannot be used.The 100 Mbit/sec Ethernet switch handles higher network loads, which isessential to the uninterrupted operation of the cluster. Ethernet hubs cannot beused in clusters with three or more nodes.




Figure 2-15 shows the redundant Ethernet cluster interconnect componentsused in a two-node PDC/O2000 cluster.

Node 1 Node 2

Client LANHub/Switch #1


Ethernet Switch/Hub #2for Cluster Interconnect


CrossoverCable

Dual-portEthernet

Adapters (2)

Dual-portEthernet

Adapters (2)

CrossoverCable

Figure 2-15. Redundant Ethernet cluster interconnect components in atwo-node PDC/O2000 cluster

In the configuration example shown in Figure 2-15, the top port on eachdual-port Ethernet adapter connects by Ethernet cable to one of the twoEthernet switches or hubs provided for the cluster interconnect. A crossovercable is installed between the two Ethernet switches or hubs used for thecluster interconnect. The bottom port on each adapter connects by Ethernetcable to the client LAN for the PDC/O2000 cluster.

A crossover cable should be installed between client LAN hubs or switches toprovide fault tolerance in the client LAN path should a failure occur on one ofthe hubs or switches.

Architecture 2-37



Redundant ServerNet Cluster Interconnect

If a Compaq ServerNet cluster interconnect is installed in a PDC/O2000cluster, it must be a redundant ServerNet cluster interconnect. This providesfault tolerance along the cluster interconnect.

A redundant ServerNet cluster interconnect uses these components:

■ One ServerNet PCI Adapter installed in each cluster node in the cluster

■ Two ServerNet Switches for clusters with three or more cluster nodes

■ Two ServerNet cables installed between each ServerNet PCI Adapter ina two-node cluster or to the two ServerNet Switches in clusters withthree or more cluster nodes

In addition, the ServerNet driver software must be installed on each node inthe cluster by the Oracle Universal Installer (OUI) for Oracle8i Release 8.1.5.

ServerNet PCI Adapters

The Compaq ServerNet PCI Adapter is a bi-directional, high-bandwidth,low-latency, redundant path PCI adapter that uses a low-overhead,message-passing software interconnect. Each ServerNet PCI Adapterconnection provides a redundant data path to and from the cluster node inwhich it is installed.

One dual-port ServerNet PCI Adapter is installed in a PCI slot in each clusternode. In a two-node cluster, these adapters can be directly connected by twoServerNet cables. In clusters with three or more nodes, the ServerNet cablesfrom each ServerNet PCI Adapter are connected to two ServerNet Switches.

IMPORTANT: To provide optimal performance, the ServerNet PCI Adapter should beinstalled on the correct PCI bus on each server in the cluster. For detailed placementinformation about each Compaq server supported by the PDC/O2000, refer to the CompaqParallel Database Cluster Model PDC/O2000 for Oracle8 Parallel Server Release 8.0.5Certification Matrix or the Compaq Parallel Database Cluster Model PDC/O2000 forOracle8i Parallel Server Release 8.1.5 Certification Matrix. These documents are availableat





Figure 2-16 shows the redundant ServerNet cluster interconnect componentsfor a two-node PDC/O2000 cluster.

ServerNet Cablefor the Y Path

ServerNetPCI Adapter


Node 1 Node 2

ServerNet Cablefor the X Path

YX

Y

X

Figure 2-16. Redundant ServerNet cluster interconnect components in atwo-node PDC/O2000 cluster

As Figure 2-16 shows, the ServerNet PCI Adapters in the two nodes areconnected by two ServerNet cables at adapter ports X and Y. The two portsprovide connections to the ServerNet System Area Network X path and Ypath.

Refer to the Compaq ServerNet PCI Adapter Installation Guide for detailedinformation about the ServerNet PCI Adapter.

ServerNet Switch

The ServerNet Switch is a point-to-point networking device that connects thecluster nodes in the cluster interconnect to the ServerNet system area network.The ServerNet Switch routes ServerNet packets from originating nodes todestination nodes.

Two ServerNet Switches are required for a PDC/O2000 cluster that containsthree or more nodes and are optional for a cluster with two nodes. With twoServerNet Switches (X and Y) installed in the ServerNet cluster interconnect,the backup ServerNet system area network path takes control if the primarypath fails.

The ServerNet Switch uses destination-based routing to deliver ServerNetpackets between the cluster nodes. The six-port crossbar of the ServerNetSwitch allows input from one of the six input ports to attach to any of the sixoutput ports.

Architecture 2-39



Refer to the Compaq ServerNet Switch Installation Guide for detailedinformation about the ServerNet Switch.

Figure 2-17 shows the redundant ServerNet cluster interconnect componentsused in a four-node PDC/O2000 cluster.

ServerNet Switch(Y Path)

ServerNet Switch(X Path)





Node 2

X X X XY Y Y Y

Node 1 Node 3 Node 4

Figure 2-17. Redundant ServerNet cluster interconnect components in afour-node PDC/O2000 cluster

From each node, the ServerNet cable from the X-port on the ServerNet PCIAdapter is connected to one ServerNet Switch. The cable from the Y-port isconnected to the second ServerNet Switch.

Local Area Network

IMPORTANT: For the PDC/O2000, the client LAN and the cluster interconnect areseparate networks. Do not use either network to handle the other network’s traffic.

Every client/server application requires a local area network (LAN) overwhich client machines and servers communicate. In the case of a cluster, thehardware components of the client LAN are no different than in a stand-aloneserver configuration.

The software components used by network clients should have the ability todetect node failures and automatically reconnect the client to another clusternode. For example, Net8, Oracle Call Interface (OCI) and Transaction ProcessMonitors can be used to address this issue.


Part Number: 161397-001 File Name: d-ch3 Cluster Software Components.doc Last Saved On: 11/1/99 4:41 PM

Chapter 3Cluster Software Components

Overview of the Cluster SoftwareThe Compaq Parallel Database Cluster Model PDC/O2000 (PDC/O2000)combines software from several leading computer vendors. The integration ofthese components creates a stable cluster management environment in whichthe Oracle database can operate.

For the PDC/O2000, the only supported operating system is MicrosoftWindows NT Server. The cluster management software is a combination ofCompaq operating system dependent modules (OSDs) and Oracle software:

■ Oracle8 Enterprise Edition Release 8.0.5 with the Oracle8 ParallelServer Option Release 8.0.5 or

■ Oracle8i Enterprise Edition Release 8.1.5 with the Oracle8i ParallelServer Option Release 8.1.5

NOTE: For information about currently-supported software revisions for the PDC/O2000,refer to the Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8 ParallelServer Release 8.0.5 Certification Matrix or the Compaq Parallel Database Cluster ModelPDC/O2000 for Oracle8i Parallel Server Release 8.1.5 and Certification Matrix at





Microsoft Windows NT Server 4.0The PDC/O2000 supports only version 4.0 of Microsoft Windows NT Server.Both Windows NT Server 4.0 Standard Edition and Windows NT Server 4.0Enterprise Edition are supported. Windows NT Server 4.0 with Service Pack3, 4, or 5 is required.

IMPORTANT: Windows NT Server 4.0 with Service Pack 4 or 5 is required for properoperation of the redundant Ethernet cluster interconnect. Because certain applicationsmight not work with Service Pack 4 or 5, you might need to use Service Pack 3 with anapproved Microsoft fix to support a redundant Ethernet cluster interconnect. Consult withyour software expert to confirm that your applications can run with Service Pack 4 or 5.

NOTE: The PDC/O2000 does not work in conjunction with Microsoft Cluster Server. Donot install Microsoft Cluster Server on any of the cluster nodes.

Compaq SoftwareCompaq offers an extensive set of features and optional tools to supporteffective configuration and management of your PDC/O2000:

■ Compaq SmartStart and Support Software

■ Compaq System Configuration Utility

■ Compaq Array Configuration Utility

■ Fibre Channel Fault Isolation Utility

■ Compaq Insight Manager

■ Compaq Insight Manager XE

■ Compaq Options ROMPaq

■ Compaq Redundancy Manager

■ Compaq operating system dependent modules (OSDs)

Compaq SmartStart and Support Software

SmartStart, which is located on the SmartStart and Support Software CD, isthe best way to configure Windows NT Server on a PDC/O2000 cluster.SmartStart uses an automated step-by-step process to configure the operatingsystem and load the system software.

Cluster Software Components 3-3



The Compaq SmartStart and Support Software CD also contains device driversand utilities that enable you to take advantage of specific capabilities offeredon Compaq products. These drivers are provided for use with Compaqhardware only.

The PDC/O2000 requires version 4.3 or later of the SmartStart and SupportSoftware CD. For information about SmartStart, refer to the Compaq ServerSetup and Management pack.

Compaq System Configuration Utility

The SmartStart and Support Software CD also contains the Compaq SystemConfiguration Utility. This utility is the primary means to configure hardwaredevices within your servers, such as I/O addresses, boot order of diskcontrollers, and so on.

For information about the System Configuration Utility, see the CompaqServer Setup and Management pack.

Compaq Array Configuration Utility

The Compaq Array Configuration Utility, found on the Compaq SmartStartand Support Software CD, is used to configure the hardware aspects of anydisk drives attached to an array controller, including the non-shared drives inthe servers and the shared drives in the Compaq StorageWorks RAID Array4000s (RA4000 Arrays).

The Array Configuration Utility also allows you to configure RAID levels andto add disk drives or RA4000 Arrays to an existing configuration.

For information about the Array Configuration Utility, see the CompaqStorageWorks RAID Array 4000 User Guide.

Fibre Channel Fault Isolation Utility

The SmartStart and Support Software CD also contains the Fibre ChannelFault Isolation Utility (FFIU). The FFIU verifies the integrity of a new orexisting Fibre Channel Arbitrated Loop (FC-AL) installation. This utilityprovides fault detection and help in locating a failing device on an FC-AL.

For more information about the FFIU, see the Compaq SmartStart and SupportSoftware CD.




Compaq Insight Manager

Compaq Insight Manager, loaded from the Compaq Management CD, is asoftware utility used to collect information about the servers in the cluster.Compaq Insight Manager performs these functions:

■ Monitors server fault conditions and status

■ Forwards server alert fault conditions

■ Remotely controls servers

The Integrated Management Log is used to collect and feed data to CompaqInsight Manager. This log is used with the Compaq Integrated ManagementDisplay (IMD), the optional Remote Insight controller, and SmartStart.

In Compaq servers, each hardware subsystem, such as non-shared diskstorage, system memory, and system processor, has a robust set ofmanagement capabilities. Compaq Full Spectrum Fault Management notifiesthe end user of impending fault conditions.

For information about Compaq Insight Manager, refer to the documentationyou received with your Compaq ProLiant server.

Compaq Insight Manager XE

Compaq Insight Manager XE is a Web-based management system. It can beused in conjunction with Compaq Insight Manager agents as well as its ownWeb-enabled agents. This browser-based utility provides increased flexibilityand efficiency for the administrator.

Compaq Insight Manager XE is an optional CD available upon request fromthe Compaq System Management website at

www.compaq.com/sysmanage

Compaq Options ROMPaq

The Compaq Options ROMPaq diskettes allow a user to upgrade the ROMFirmware images for Compaq System product options, such as arraycontrollers, disk drives, and tape drives used for non-shared storage.




Compaq Redundancy Manager

Compaq Redundancy Manager works in conjunction with the Windows NTfile system (NTFS). Redundancy Manager increases the availability ofclustered systems that use the RA4000 Arrays and Compaq ProLiant servers.

Redundancy Manager can detect failures in redundant FC-AL components,including Compaq StorageWorks Fibre Channel Host Bus Adapters (FibreHost Adapters), Fibre Channel cables, Compaq StorageWorks Fibre ChannelStorage Hubs (Storage Hubs), and Compaq StorageWorks RAID Array 4000Array Controllers (RA4000 Array Controllers) in Compaq StorageWorksRAID Array 4000s (RA4000 Arrays).

When a failure occurs on an active FC-AL path, I/O processing is reroutedthrough a redundant path, allowing applications to continue processing. Thisrerouting is transparent to NTFS. Redundancy Manager, in combination withredundant hardware components, is the basis for the enhanced high availabilityfeatures of the PDC/O2000.

Compaq Redundancy Manager (Fibre Channel) CD is included in your clusterkit for the PDC/O2000.

Compaq Operating System Dependent Modules

Compaq supplies low-level services, called operating system dependentmodules (OSDs), that are required by Oracle Parallel Server. The OSD layermonitors critical clustering hardware components, constantly relaying clusterstate information to Oracle Parallel Server. Oracle Parallel Server monitorsthis information and takes pertinent action as needed.

For example, the OSD layer is responsible for monitoring the performance ofeach node in the cluster. The OSD layer determines if one of the nodes is nolonger responding to the cluster heartbeat. If the node still does not respond,the OSD layer determines it is unavailable, and communicates this informationto Oracle Parallel Server. Oracle Parallel Server then evicts the node from thecluster, recovers the part of the database affected by that node, andreconfigures the cluster with the remaining nodes.

A PDC/O2000 can run either Oracle8 Parallel Server Release 8.0.5 or Oracle8iParallel Server Release 8.1.5.




OSDs for Oracle8 Parallel Server Release 8.0.5

For a detailed description of how the OSD layer interacts with Oracle8 ParallelServer, refer to the Oracle8 Enterprise Edition Getting Started Release 8.0.5for Windows NT manual.

The Compaq OSD software for Oracle8 is installed using a setup program andconfigured using the NodeList Configurator. These programs and the OSDsare on the Compaq Parallel Database Cluster for Oracle8 Release 8.0.5Ethernet Clustering Software CD.

OSDs for Oracle8i Parallel Server Release 8.1.5

For a detailed description of how the OSD layer interacts with Oracle8iParallel Server, refer to the Oracle8i Parallel Server Setup and ConfigurationGuide Release 8.1.5.

The Compaq OSD software for Oracle8i is an installable package of OracleUniversal Installer (OUI). OUI is the Java-based installer from Oracle that hasbecome the standard installer for all Oracle8i software. The OUI installs theOSDs and, if appropriate, the cluster interconnect drivers from one node to allother nodes in the cluster.

The OSD software is found on the Compaq Parallel Database Cluster forOracle8i Release 8.1.5 Ethernet Clustering Software CD or Compaq ParallelDatabase Cluster for Oracle8i Release 8.1.5 ServerNet Clustering SoftwareCD.

Oracle SoftwareThe PDC/O2000 supports Oracle8 Release 8.0.5 or Oracle8i Release 8.1.5software. Support for future releases on the 8.0 and 8.1 product lines isanticipated. However, if you are using a release other than 8.0.5 or 8.1.5,confirm that the release has been certified for the PDC/O2000 on the Compaqwebsite at





Oracle8 Server Enterprise Edition Release 8.0.5

The Oracle8 Server Enterprise Edition Release 8.0.5 provides the following:

■ Oracle8 Server Release 8.0.5

■ Oracle8 Parallel Server Option Release 8.0.5

■ Oracle8 Enterprise Manager Release 8.0.5

Oracle8 Server Release 8.0.5

Oracle8 Server Release 8.0.5 is the database application software and must beinstalled on each node in the PDC/O2000 cluster.

Refer to the documentation for Oracle8 Server Release 8.0.5 for additionalinformation.

Oracle8 Parallel Server Option Release 8.0.5

Oracle8 Parallel Server Option Release 8.0.5 is the key component in theOracle8 clustering architecture. Oracle8 Parallel Server allows the databaseserver to divide its workload among the physical cluster nodes. This isaccomplished by running a distinct instance of Oracle8 Server on each node inthe PDC/O2000 cluster.

Oracle8 Parallel Server manages the interaction between these instances.Through its Integrated Distributed Lock Manager, Oracle8 Parallel Servermanages the ownership of database records that are requested by multipleinstances.

At a lower level, Oracle8 Parallel Server monitors cluster membership. Itinteracts with the OSDs, exchanging information about the state of eachcluster node.

For additional information, refer to:

■ Oracle8 Enterprise Edition Getting Started Release 8.0.5 for Windows NT

■ Other Oracle documentation for Oracle8 Server Release 8.0.5 andOracle8 Parallel Server Release 8.0.5




Oracle8 Enterprise Manager Release 8.0.5

Oracle8 Enterprise Manager Release 8.0.5 is responsible for monitoring thestate of both the database entities and the cluster members. It primarilymanages the software components of the cluster. Hardware components aremanaged with Compaq Insight Manager.

Do not install Oracle8 Enterprise Manager on any of the PDC/O2000 clusternodes. It must be installed on a separate server that is running Oracle8 Release8.0.5 and has network access to the cluster nodes. Before installing Oracle8Enterprise Manager, read its documentation to ensure it is installed andconfigured correctly for an Oracle8 Parallel Server environment.

Oracle8 Certification

To ensure that Oracle8 Parallel Server Release 8.0.5 is used in a compatiblehardware environment, Oracle has established a certification process, which isa series of test scripts designed to stress an Oracle8 Parallel Serverimplementation and verify stability and function.

All hardware providers who choose to deliver platforms for use with Oracle8Parallel Server Release 8.0.5 must demonstrate the successful completion ofthe Oracle8 Parallel Server for Windows NT Certification. Neither Oracle norCompaq will support any implementation of Oracle8 Parallel Server that doesnot strictly conform to the configurations certified with this process.

For a complete list of certified Compaq servers, see the Compaq ParallelDatabase Cluster Model PDC/O2000 for Oracle8 Parallel Server Release 8.0.5Certification Matrix at





Oracle8i Server Enterprise Edition Release 8.1.5

The Oracle8i Server Enterprise Edition Release 8.1.5 provides the following:

■ Oracle8i Server Release 8.1.5

■ Oracle8i Parallel Server Option Release 8.1.5

■ Oracle8i Enterprise Manager Release 8.1.5

Oracle8i Server Release 8.1.5

Oracle8i Server Release 8.1.5 is the database application software and must beinstalled on each node in the PDC/O2000 cluster.

Refer to the documentation for Oracle8i Server Release 8.1.5 for additionalinformation.

Oracle8i Parallel Server Option Release 8.1.5

Oracle8i Parallel Server Option Release 8.1.5 is the key component in theOracle8i clustering architecture. Oracle8i Parallel Server allows the databaseserver to divide its workload among the physical cluster nodes. This isaccomplished by running a distinct instance of Oracle8i Server on each nodein the PDC/O2000 cluster.

Oracle8i Parallel Server manages the interaction between these instances.Through its Integrated Distributed Lock Manager, Oracle8i Parallel Servermanages the ownership of database records that are requested by multipleinstances.

At a lower level, Oracle8i Parallel Server monitors cluster membership. Itinteracts with the OSDs, exchanging information about the state of eachcluster node.

For additional information, refer to:

■ Oracle8i Parallel Server Setup and Configuration Guide Release 8.1.5

■ Other Oracle documentation for Oracle8i Server Release 8.1.5 andOracle 8i Parallel Server Release 8.1.5




Oracle8i Enterprise Manager Release 8.1.5

Oracle8i Enterprise Manager Release 8.1.5 is responsible for monitoring thestate of both the database entities and the cluster members. It primarilymanages the software components of the cluster. Hardware components aremanaged with Compaq Insight Manager.

Do not install Oracle8i Enterprise Manager on any of the PDC/O2000 clusternodes. It must be installed on a separate server that is running Oracle8iRelease 8.1.5 and has network access to the cluster nodes. Before installingOracle8i Enterprise Manager, read its documentation to ensure it is installedand configured correctly for an Oracle8i Parallel Server environment.

Oracle8i Certification

To ensure that Oracle8i Parallel Server Release 8.1.5 is used in a compatiblehardware environment, Oracle has established a certification process, which isa series of test scripts designed to stress an Oracle8i Parallel Serverimplementation and verify stability and full functionality.

All hardware providers who choose to deliver platforms for use with Oracle8iParallel Server Release 8.1.5 must demonstrate the successful completion ofthe Oracle8i Parallel Server for Windows NT Certification. Neither Oracle norCompaq will support any implementation of Oracle8i Parallel Server that doesnot strictly conform to the configurations certified with this process. For acomplete list of certified Compaq servers, see the Compaq Parallel DatabaseCluster Model PDC/O2000 for Oracle8i Parallel Server Release 8.1.5Certification Matrix at





Application Failover and Reconnection Software

When a network client computer operates in a clustered environment, it mustbe more resilient than when operating with a stand-alone server. Because aclient can access the database through any of the cluster nodes, the failure ofthe connection to a node does not have to prevent the client from reattaching tothe cluster and continuing its work.

Oracle clustering software provides the capability to allow the automaticreconnection of a client and application failover in the event of a node failure.To implement this application and connection failover, a software interfacebetween the Oracle software and the client must be written.

Such a software interface would be responsible for detecting when the client’scluster node is no longer available and then connecting the client to one of theremaining, operational cluster nodes.

NOTE: For complete information on how to ensure client auto-reconnect in an OracleParallel Server environment, contact your Oracle representative.


Part Number: 161397-001 File Name: e-ch4 Planning.doc Last Saved On: 10/29/99 4:54 PM

Chapter 4Planning

Before connecting any cables or powering on any hardware on your CompaqParallel Database Cluster Model PDC/O2000 (PDC/O2000), it is importantthat you understand how all the various cluster components fit together to meetyour operational requirements. The major topics discussed in this chapter are:

■ Site planning

■ Capacity planning for cluster hardware

■ Planning the cluster configuration

■ RAID planning

■ Planning the grouping of physical disk storage space

■ Disk drive planning

■ Network planning




Site PlanningYou must carefully select and prepare the site to ensure a smooth installationand a safe and efficient work environment. To select and prepare a location foryour cluster, consider the following:

■ The path from the receiving dock to the installation area

■ Availability of appropriate equipment and qualified personnel

■ Space for unpacking, installing, and servicing the computer equipment

■ Sufficient floor strength for the computer equipment

■ Cabling requirements, including the placement of network and FibreChannel cables within one room (under the subfloor, on the floor, oroverhead) and possibly between rooms

■ Client LAN resource planning, including the number of hubs orswitches and cables to connect to the cluster nodes

■ Environmental conditions, including temperature, humidity, and airquality

■ Power, including voltage, current, grounding, noise, outlet type, andequipment proximity

IMPORTANT: Carefully review the power requirements for your cluster components toidentify special electrical supply needs in advance.

Planning 4-3



Capacity Planning for Cluster HardwareCapacity planning determines how much computer hardware is needed tosupport the applications and data on your clustered servers. Given the size ofyour database and the performance you expect, you must decide how manyservers and shared storage arrays the cluster needs.

Compaq ProLiant Servers

The number of servers you install in a PDC/O2000 cluster should take intoaccount the levels of availability and scalability your site requires. Start byplanning your cluster so that the failure of a single node will not adverselyimpact cluster operations. For example, when running a two-node cluster, thefailure of one node leaves the one remaining node to service all clients. Thiscould result in an unacceptable level of performance.

Within each server, the appropriate number and speed of the CPUs andmemory size are all determined by several factors. These include the types ofdatabase applications being used and the number of clients connecting to theservers.

IMPORTANT: All Compaq servers in a PDC/O2000 cluster must be the same model type.All the servers must be identically configured, including adapter slot placement, theamount of memory, the number of CPUs, and so on.

NOTE: For an up-to-date list of Compaq Parallel Database Cluster-certified servers for thePDC/O2000 and detailed information about minimum and maximum cluster nodeconfigurations, refer to the Compaq Parallel Database Cluster Model PDC/O2000 forOracle8 Parallel Server Release 8.0.5 Certification Matrix or the Compaq Parallel DatabaseCluster Model PDC/O2000 for Oracle8i Parallel Server Release 8.1.5 Certification Matrix.These documents are available on the Compaq website at





Planning Shared Storage Subsystem Components

Several key components make up the shared storage subsystem for thePDC/O2000. Each redundant Fibre Channel Arbitrated Loop (FC-AL) in aPDC/O2000 cluster provides redundant I/O paths from each node to eachCompaq StorageWorks RAID Array 4000 (RA4000 Array). Together, theseredundant I/O paths use the following hardware components:

■ Two Compaq StorageWorks Fibre Channel Host Bus Adapters (FibreHost Adapters) in each server (one Fibre Host Adapter for each path)

■ Two Compaq StorageWorks Storage Hubs (Storage Hubs), each ofwhich connects all of the cluster nodes to one FC-AL path

■ From one to five RA4000 Arrays. Each RA4000 Array can hold up toeight 1.6-inch disk drives or twelve 1-inch disk drives.

■ Two single-port Compaq StorageWorks RAID Array 4000 Controllers(RA4000 Array Controllers) installed in each RA4000 Array. Only onearray controller in the RA4000 Array is active at any one time; the otheris the standby array controller.

NOTE: For more information about redundant Fibre Channel Arbitrated Loops (FC-ALs) ina PDC/O2000 cluster, see Chapter 2, “Architecture.”

The Storage Hubs are available in two models:

■ Storage Hub 7 (7 ports)

■ Storage Hub 12 (12 ports)

To determine which Storage Hub model is appropriate for a redundant FC-ALin your cluster, identify the total number of nodes and RA4000 Arrays present.If the combined number of nodes and RA4000 Arrays exceeds seven, youmust use Storage Hub 12s. Also consider the possibility of future clustergrowth when you select your Storage Hubs.

The number of RA4000 Arrays and shared disk drives used in a PDC/O2000depends on the amount of shared storage space required by the database, thehardware RAID levels used on the shared storage disks, and the number andstorage capacity of disk drives installed in the enclosures. Refer to “Raw DataStorage and Database Size” in this chapter for more details.

NOTE: For improved I/O performance and cluster integrity, as you increase the number ofnodes in a PDC/O2000 cluster, you should also increase the aggregate bandwidth of theshared storage subsystem by adding more or higher-capacity disk drives.

Planning 4-5



Planning Cluster Interconnect and Client LANComponents

The PDC/O2000 uses these types of cluster interconnects:

■ Clusters using Oracle8 Parallel Server Release 8.0.5 must use aredundant Ethernet cluster interconnect.

■ Clusters using Oracle8i Parallel Server Release 8.1.5 can use aredundant Ethernet cluster interconnect or a redundant CompaqServerNet cluster interconnect.

IMPORTANT: A redundant cluster interconnect uses redundant hardware to provide faulttolerance along the entire cluster interconnect path. In keeping with the redundant designof the shared storage subsystem, a redundant cluster interconnect is required for thePDC/O2000.

Planning a Redundant Ethernet ClusterInterconnect

IMPORTANT: Compaq recommends Service Pack 4 or 5 of Windows NT Server for aredundant Ethernet cluster interconnect or client LAN. Using Service Pack 3 requiresinstalling the approved Microsoft patch (hot fix) article ID Q156655, entitled “MemoryLeak and STOP Screens Using Intermediate NDIS Drivers.”

NOTE: Refer to the technical white paper Supported Ethernet Interconnects for CompaqParallel Database Clusters Using Oracle Parallel Server for detailed information aboutconfiguring redundant Ethernet cluster interconnects. This document is available at


If you will be installing a redundant Ethernet cluster interconnect in aPDC/O2000 cluster, review these planning considerations:

■ Whether to use two Ethernet switches or two Ethernet hubs for thecluster interconnect. If your cluster will contain or grow to three or morenodes, you must use two Ethernet switches.

■ Whether to use two dual-port Ethernet adapters in each node that willconnect to both the cluster interconnect and the client LAN or to useseparate single-port adapters for the Ethernet cluster interconnect andthe client LAN.




NOTE: For information about recommended dual-port and single-port Ethernet adaptersfor your redundant Ethernet cluster interconnect, refer to the Compaq Parallel DatabaseCluster Model PDC/O2000 for Oracle8 Parallel Server Release 8.0.5 Certification Matrix orthe Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8i Parallel ServerRelease 8.1.5 Certification Matrix These documents are available at


Planning a Redundant ServerNet ClusterInterconnect

If you will be installing a redundant ServerNet cluster interconnect in aPDC/O2000 cluster, review these planning considerations:

■ Identify the PCI bus slot on each cluster node into which you will installthe ServerNet PCI Adapter. For information about optimum PCI slotplacement of the ServerNet PCI Adapter in ProLiant servers, refer to theCompaq Parallel Database Cluster Model PDC/O2000 for OracleParallel Server Release 8.0.5 Certification Matrix or the CompaqParallel Database Cluster Model PDC/O2000 for Oracle8i ParallelServer Release 8.1.5 Certification Matrix

■ If the cluster will initially contain two nodes, consider future expansionwhen you select your Compaq ServerNet cluster interconnect hardwarecomponents. For example, you may want to install two ServerNetSwitches between the two nodes instead of ServerNet cables only; twoServerNet Switches are required in clusters with three or more nodes.

■ If you will be using ServerNet Switches, consider whether to installthem in a rack cabinet or as stand-alone units.

Planning the Client LAN

Every client/server application requires a local area network (LAN) overwhich client machines and servers communicate. In the case of a cluster, thehardware components of the client LAN are no different than in a stand-aloneserver configuration.

In keeping with the redundant architecture of the cluster interconnect and theshared storage subsystem, you may choose to install a redundant client LAN,with redundant Ethernet adapters and redundant Ethernet switches or hubs.

Planning 4-7



Reference Material for Hardware Sizing

For information about hardware sizing for online transaction processing(OLTP) databases, see the guidelines in the Oracle white paper Sizing CompaqProLiant Servers for Oracle OLTP Applications, May 1997. You can find thiswhite paper at

www.compaq.com

Planning the Cluster ConfigurationOnce you have investigated your requirements with respect to particular partsof the cluster (ProLiant servers, shared storage subsystem components, clusterinterconnect components, client LAN components), you need to plan theconfiguration of the entire PDC/O2000. This section describes sampleconfigurations for small and large clusters.

IMPORTANT: Use the Oracle documentation identified in the front matter of this guide toobtain detailed information about planning for the Oracle software. Once the required levelof performance, the size of the database, and the type of database have been determined,use this Oracle documentation to continue the planning of the cluster’s physicalcomponents.




Sample Small Configuration for the PDC/O2000

Figure 4-1 shows an example of a small PDC/O2000 cluster: a two-nodecluster with one RA4000 Array.


RA4000 Array #1

StorageHub #1

Node 1 Node 2

StorageHub #2


Figure 4-1. Two-node PDC/O2000 cluster with one RA4000 Array

The sample small configuration shown in Figure 4-1 contains these key clustercomponents:

■ Two ProLiant servers (nodes)

■ Two Fibre Host Adapters in each server


■ One RA4000 Array

■ Two single-port array controllers in the RA4000 Array

■ Cluster interconnect hardware (not shown)

� Redundant Ethernet NIC adapters, cables, and Ethernet switches orhubs for the Ethernet cluster interconnect or

� Redundant ServerNet PCI Adapters, ServerNet cables, andServerNet Switches for the ServerNet cluster interconnect

■ Ethernet NIC adapters, switches or hubs (not shown), and cables for theclient LAN

Planning 4-9



Sample Large Configuration for the PDC/O2000

Figure 4-2 shows an example of a large PDC/O2000 cluster: a six-node clusterwith five RA4000 Arrays.

RA4000Array #1

RA4000Array #3

RA4000Array #5

RA4000Array #4

RA4000Array #2

StorageHub #1


StorageHub #2


Node 5 Node 6

Figure 4-2. Six-node PDC/O2000 cluster with five RA4000 Arrays




The sample large configuration shown in Figure 4-2 contains these key clustercomponents:

■ Six ProLiant servers (nodes)

■ Two Fibre Host Adapters in each server node

■ Two Storage Hub 12s

■ Five RA4000 Arrays. Each RA4000 Array contains up to eight 1.6-inchdisk drives or twelve 1-inch disk drives, providing a total of between40 and 60 shared disk drives.

■ Two single-port array controllers installed in each RA4000 Array.

■ Cluster interconnect hardware (not shown)

� Redundant Ethernet NIC adapters, cables, and Ethernet switches forthe Ethernet cluster interconnect or

� Redundant ServerNet PCI Adapters, ServerNet cables, andServerNet Switches for the ServerNet cluster interconnect

■ Ethernet NIC adapters, switches or hubs (not shown), and cables for theclient LAN

This sample large configuration could be made even larger by adding a secondor third redundant FC-AL to the PDC/O2000. Each redundant FC-AL canprovide from one to five RA4000 Arrays without the need for adding newnodes to the cluster.

RAID PlanningShared storage subsystem performance is one of the most important aspects oftuning database cluster servers for optimal performance. Efforts to plan,configure, and tune a PDC/O2000 cluster should focus on getting the most outof each shared disk drive and having an appropriate number of shared drives inthe cluster. When properly configured, the shared storage subsystem shouldnot be the limiting factor in overall cluster performance.

RAID technology provides cluster servers with more consistent performance,higher levels of fault tolerance, and easier fault recovery than non-RAIDsystems. RAID uses redundant information stored on different disks to ensurethat the cluster can survive the loss of any disk in the array without affectingthe availability of data to users.

RAID also uses the technique of striping, which involves partitioning eachdrive’s storage space into units ranging from a sector (512 bytes) up to severalmegabytes. The stripes of all the disks are interleaved and addressed in order.

Planning 4-11



In a PDC/O2000 cluster, each node is connected to shared storage disk driveshoused in RA4000 Arrays. When planning the amount of shared storage foryour cluster, you must consider the following:

■ The maximum allowable number of shared storage arrays in one cluster.This maximum depends on several factors, including the physicallimitation of five RA4000 Arrays for each redundant FC-AL and thenumber of redundant FC-ALs you plan to install in the cluster.

■ The number of redundant FC-ALs allowed in a cluster, in turn, dependsupon the maximum number of Fibre Host Adapters that can be installedin the ProLiant server model you will be using. Refer to the serverdocumentation for this information.

■ The appropriate number of shared storage arrays in a cluster isdetermined by the performance requirements of your cluster.

Refer to “Planning Shared Storage Subsystem Components” in this chapter formore information.

The PDC/O2000 implements RAID at the hardware level, which is faster thansoftware RAID. When you implement RAID on shared storage arrays, you usethe hardware RAID to perform such functions as making copies of the data orcalculating checksums. Use the Compaq Array Configuration Utility toimplement RAID on your logical disks.

NOTE: Do not use the software RAID offered by the operating system to configure yourshared storage disks.




Supported RAID Levels

RAID provides several fault-tolerant options to protect your cluster’s shareddata. However, each RAID level offers a different mix of performance,reliability, and cost.

The RA4000 Array supports these RAID levels:

■ RAID 0

■ RAID 0+1

■ RAID 1

■ RAID 4

■ RAID 5

NOTE: RAID 0 does not provide the fault tolerance feature of other RAID levels.

For RAID level definitions and information about configuring hardwareRAID, refer to the following:

■ Refer to the information about RAID configuration contained in theCompaq StorageWorks RAID Array 4000 User Guide.

■ Refer to the Compaq white paper “Configuring Compaq RAIDTechnology for Database Servers,” #ECG 011/0598, available at theCompaq website at

www.compaq.com

■ Refer to the various white papers on Oracle8 and Oracle8i, which areavailable at the Compaq ActiveAnswers website at

www.compaq/activeanswers

Planning 4-13



Raw Data Storage and Database Size

Raw data storage is the amount of storage available before any RAID levelshave been configured. It is called raw data storage because RAID volumesrequire some overhead. The maximum size of a database stored in a RAIDsystem will always be less than the amount of raw data storage available.

To calculate the amount of raw data storage in a PDC/O2000 cluster,determine the total amount of shared storage space available to the cluster. Todo this, you need to know the following:

■ The number of RA4000 Arrays in the cluster

■ The number and sizes of disk drives contained in each RA4000 Array(for example, from one to twelve 1-inch high, 9-GB drives or from oneto eight 1.6-inch high, 18-GB drives)

Add together the planned storage capacity of all RA4000 Arrays to calculatethe total amount of raw data storage in the PDC/O2000 cluster. The maximumamount of raw data storage in an RA4000 Array depends on what type of diskdrives you install in the RA4000 Arrays. For example, using 1-inch high,9-GB drives provides a maximum storage capacity of 108 GB per RA4000Array (twelve 9-GB drives). Using the 1.6-inch high, 18-GB drives provides amaximum storage capacity of 144 GB per RA4000 Array (eight 18-GBdrives).

The amount of shared disk space required for a given database size is affectedby the RAID levels you select and the overhead required for indexes, I/Obuffers, and logs. Consult with your Oracle representative for further details.

NOTE: To plan for future expansion, you are advised to define from 10 to 30 percentmore extended partitions than you currently require.




Selecting RAID Levels

Many factors affect which RAID levels you select for your cluster database.These include the specific availability, performance, reliability, and recoverycapabilities required from the database. Each cluster must be evaluatedindividually by qualified personnel.

The following general guidelines apply to RAID selection for a cluster withRA4000 Arrays using Oracle8 Parallel Server Release 8.0.5 or Oracle8iParallel Server Release 8.1.5:

■ Oracle recommends that some form of disk fault tolerance beimplemented in the cluster.

■ In order to ease the difficulty of managing dynamic space allocation inan Oracle Parallel Server raw volume environment, Oracle recommendsthe creation of “spare” raw volumes that can be used to dynamicallyextend tablespaces when the existing datafiles approach capacity. Thenumber of these spare raw volumes should represent from 10 to30 percent of the total database size. To allow for effective loadbalancing, the spares should be spread across a number of disks andcontrollers. The database administrator should decide, on a case by casebasis, which spare volume to use based on which volume would havethe least impact on scalability (for both speedup and scaleup).

Planning 4-15



Planning the Grouping of Physical DiskStorage Space

Figure 4-3 shows how the physical storage space in one RA4000 Array thatcontains eight physical disk drives might be grouped for an Oracle8 ParallelServer database or Oracle8i Parallel Server database.

RA4000 Array

DiskDrive

DiskDrive

DiskDrive

DiskDrive

DiskDrive

DiskDrive

DiskDrive

DiskDrive

Create extendedpartitions withNT Disk Administrator

ExtendedPartition

ExtendedPartition

ExtendedPartition

RAID 5Disk Array

Create logical drivearrays withCompaq ArrayConfiguration Utility

RAID 1Disk Array

RAID 1Disk Array

Create logicalpartitions withNT Disk Administrator / D / E / F / G /H / I / J / / K / L / / M / N /

Figure 4-3. RA4000 Array disk grouping for a PDC/O2000 cluster

Using the Compaq Array Configuration Utility, group the RA4000 Array diskdrives into RAID disk arrays at specific RAID levels. This example showsfour disk drives grouped into one RAID disk array at RAID level 5. It alsoshows two RAID level 1 disk arrays containing two disk drives each.

A logical drive is what you see labeled as Disk 1, Disk 2, and so on, fromWindows NT Server Disk Administrator. For information about RAID diskarrays and logical drives, refer to the information on drive arrays in theCompaq StorageWorks RAID Array 4000 User Guide.




Use Disk Administrator to define one extended partition per RAID logicaldrive. Also using Disk Administrator, divide the extended partitions intological partitions, each having its own drive letter. (Windows NT Serverlogical partitions are called “logical drives” in the Oracle documentation.)

NOTE: To plan for future expansion, you are advised to define from 10 to 30 percentmore extended partitions than you currently require.

Disk Drive Planning

Nonshared Disk Drives

Nonshared disk drives, or local storage, operate the same way in a cluster asthey do in a single-server environment. These drives can be in the server drivebays or in an external storage enclosure. As long as they are not accessible bymultiple servers, they are considered nonshared.

Treat nonshared drives in a clustered environment as you would in anon-clustered environment. In most cases, some form of RAID is used toprotect the drives and aid in restoration of a failed drive. Since the OracleParallel Server application files are stored on these drives, it is recommendedthat you use hardware RAID.

Hardware RAID is the recommended solution for RAID configuration becauseof its superior performance. For the PDC/O2000, hardware RAID fornonshared drives can be implemented with a Compaq SMART-2 controller orby using dedicated RA4000 Arrays for nonshared storage.

Shared Disk Drives

The shared disk drives contained in the RA4000 Arrays are accessible to eachnode in a cluster. You can use hardware RAID levels 0, 0+1, 1, 4, or 5 on theshared disk drives contained in RA4000 Arrays.

If a logical drive is configured with a RAID level that does not support faulttolerance (for example, RAID 0), then the failure of the shared disk drives inthat logical drive will disrupt service to all Oracle databases that are dependenton that disk drive. See “Selecting RAID Levels” earlier in this chapter.

As with other types of failures, Compaq Insight Manager monitors the statusof shared disk drives and will mark a failed drive as “Failed.”

Planning 4-17



Network Planning

Windows NT Server Hosts Files for the EthernetCluster Interconnect

When a redundant Ethernet cluster interconnect is installed between clusternodes, the Compaq operating system dependent modules (OSDs) require aunique entry in the hosts and lmhosts files located at%SystemRoot%\system32\drivers\etc for each network port on each node.

Each node needs to be identified by the IP address assigned to the Ethernetadapter port used by the Ethernet cluster interconnect and by the IP addressassigned to the Ethernet adapter port used by the client LAN. The suffix“_san” stands for system area network.

The following list identifies the format of the hosts and lmhosts files for afour-node PDC/O2000 cluster with an Ethernet cluster interconnect:

IP address node1

IP address node1_san

IP address node2


IP address node3


IP address node4





Windows NT Server Hosts Files for the ServerNetCluster Interconnect

When a redundant ServerNet cluster interconnect is installed between clusternodes, the Compaq operating system dependent modules (OSDs) require aunique entry in the hosts and lmhosts files located at%SystemRoot%\system32\drivers\etc for each addressable network port oneach node.

With a ServerNet cluster interconnect present, TCP/IP is used for client LANaddresses, but not for cluster interconnect addresses. Therefore, each nodeneeds to be identified only by the IP address assigned to the Ethernet adapteror adapter port used by the client LAN.

The following list shows the format of the hosts and lmhosts files for afour-node PDC/O2000 cluster with a ServerNet cluster interconnect:

IP address node1

IP address node2

IP address node3

IP address node4

Client LAN

Physically, the structure of the client network is no different than that used fora nonclustered configuration.

To ensure continued access to the database when a cluster node is evicted fromthe cluster, each network client should have physical network access to all ofthe cluster nodes.

Software used by the client to communicate to the database must be able toreconnect to another cluster node in the event of a node eviction. For example,clients connected to cluster node1 need the ability to automatically reconnectto another cluster if cluster node1 fails.


Part Number: 161397-001 File Name: f-ch5 Installation and Configuration.doc Last Saved On: 11/1/99 4:43 PM

Chapter 5Installation and Configuration for

Oracle8 Release 8.0.5

This chapter provides instructions for installing and configuring the CompaqParallel Database Cluster Model PDC/O2000 (PDC/O2000) for use withOracle8 Release 8.0.5 software.

A PDC/O2000 is a combination of several individually available products. Asyou set up your cluster, have the following materials available duringinstallation. You will find references to them throughout this chapter.

■ User guides for the clustered Compaq ProLiant servers

■ Installation posters for the clustered ProLiant servers

■ Installation guides for the cluster interconnect and client LANinterconnect adapters

■ Compaq StorageWorks RAID Array 4000 User Guide

■ Compaq StorageWorks Fibre Channel Host Bus Adapter InstallationGuide

■ Compaq StorageWorks Storage Hub 7 Installation Guide


■ Compaq SmartStart Installation Poster

■ Compaq SmartStart and Support Software CD

■ Microsoft Windows NT Server Administrator’s Guide




■ Microsoft Windows NT Server Standard or Enterprise Edition 4.0CD/Service Pack 3, 4, or 5

■ Compaq Redundancy Manager (Fibre Channel) CD

■ Compaq Parallel Database Cluster for Oracle8 Release 8.0.5 EthernetClustering Software CD

■ Oracle8 Enterprise Edition Getting Started Release 8.0.5 forWindows NT

■ Oracle8 Enterprise Edition CD

Installation OverviewThe following summarizes the installation and setup of your PDC/O2000:

■ Installing the hardware, including:

� Proliant servers

� Compaq StorageWorks Fibre Channel Host Bus Adapters (FibreHost Adapters)


� Compaq StorageWorks Fibre Channel Storage Hubs (Storage Hubs)

� Compaq StorageWorks RAID Array 4000s (RA4000 Arrays)

� Cluster interconnect and client LAN adapters

� Ethernet hubs or switches

■ Installing and configuring operating system software, including:

� SmartStart 4.3 or later

� Windows NT Server 4.0 Standard or Enterprise Edition and ServicePack 3 , 4, or 5

■ Configuring the RA4000 arrays

■ Installing Compaq Redundancy Manager

■ Installing Oracle software, including:

� Oracle8 Enterprise Edition with Oracle8 Parallel Server Option

■ Verifying cluster communications

■ Installing and configuring the Compaq operating system dependentmodules (OSDs)

Installation and Configuration for Oracle8 Release 8.0.5 5-3



■ Installing Object Link Manager

■ Configuring Oracle software

■ Verifying the hardware and software installation, including:

� Cluster communications

� Access to shared storage from all nodes

� Client access to the Oracle8 database

■ Power distribution and power sequencing guidelines

Installing the Hardware

Setting Up the Nodes

Physically preparing the nodes (servers) for a cluster is not very different thanpreparing them for individual use. You will install all necessary adapters andinsert all internal hard disks. You will attach network cables and plug in SCSIand Fibre Channel cables. The primary difference is in setting up the sharedstorage subsystem.

Set up the hardware on one node completely, then set up the rest of the nodesidentically to the first one. Do not load any software on any cluster node untilall the hardware has been installed in all cluster nodes. Before loadingsoftware, read “Installing the Operating System Software and Configuring theRA4000 Arrays” in this chapter to understand the idiosyncrasies ofconfiguring a cluster.

IMPORTANT: The servers in the cluster must be set up identically. The clustercomponents common to all nodes in the cluster must be identical, for example, theProLiant server model, cluster interconnect adapters, amount of memory, cache, andnumber of CPUs must be the same for each cluster node. It also means the Fibre hostadapters must be installed into the same PCI slots in each server.

While setting up the physical hardware, follow the installation instructions inyour Compaq ProLiant Server Setup and Installation Guide and in yourCompaq ProLiant Server Installation Poster. When you are ready to install theFibre Host Adapters and your cluster interconnect adapters, refer to theinstructions in the pages that follow.




Installing the Fibre Host Adapters

Each redundant Fibre Channel Arbitrated Loop (FC-AL) requires two FibreHost Adapters in each cluster node. Install these devices as you would anyother PCI adapter.

Install two Fibre Host Adapters on the same PCI bus in each server and intothe same PCI slots in each server. If you need specific instructions, see theCompaq StorageWorks Fibre Channel Host Bus Adapter Installation Guide.

Installing GBIC-SW Modules for the Fibre HostAdapters

Each Fibre Host Adapter ships with two GBIC-SW modules. Insert onemodule into the Fibre Host Adapter and the other module into a Storage Hub.Each end of the Fibre Channel cable connecting a Fibre Host Adapter to aStorage Hub plugs into a GBIC-SW module.

To install GBIC-SW modules:

1. Insert a GBIC-SW module into each Fibre Host Adapter in a server.

2. Insert a GBIC-SW module into a port on each Storage Hub.

3. Repeat steps 1 and 2 for all other Fibre Host Adapters in the redundantFC-AL.

Cabling the Fibre Host Adapters to the StorageHubs

Each redundant FC-AL requires two Storage Hubs. The cabling from FibreHost Adapters to the Storage Hubs is the same for active/standby (one activeand one standby Fibre Host Adapter in each server) and active/active (twoactive Fibre Host Adapters in each server) configurations.

To cable the Fibre Host Adapters to the Storage Hubs:

1. Using Fibre Channel cables, connect Fibre Host Adapter #1 in eachserver to Storage Hub #1.





Figure 5-1 shows the Fibre Host Adapters in two servers connected to twoStorage Hubs.


StorageHub #1

Node 1 Node 2

StorageHub #2


Figure 5-1. Connecting Fibre Host Adapters to Storage Hubs

For more information about the Storage Hubs, see the Compaq StorageWorksStorage Hub 7 Installation Guide and the Compaq StorageWorks Storage Hub12 Installation Guide.

Installing the Ethernet Cluster InterconnectAdapters

A PDC/O2000 cluster requires a redundant Ethernet cluster interconnect.

IMPORTANT: Compaq recommends Service Pack 4 or 5 of Windows NT Server for aredundant Ethernet cluster interconnect or client LAN. Using Service Pack 3 requiresinstalling the approved Microsoft patch (hot fix) article ID Q156655, entitled “MemoryLeak and STOP Screens Using intermediate NDIS Drivers.”

Install one dual-port Ethernet adapter or two single-port Ethernet adapters intoeach cluster node. For recommended dual-port and single-port Ethernetadapters, see the Compaq Parallel Database Cluster Model PDC/O2000 forOracle8 Parallel Server Release 8.0.5 Certification Matrix at


If you need specific instructions on how to install an Ethernet adapter, refer tothe documentation of the Ethernet adapter you are installing or refer to the userguide of the ProLiant server you are using.




Installing the Client LAN Adapters

Unlike other clustering solutions, the PDC/O2000 does not allow transmissionof intra-cluster communication across the client LAN. All suchcommunication must be sent over the cluster interconnect.

Install a NIC into each cluster node for the client LAN. Configuration of theclient LAN is defined by site requirements. To avoid a single point of failurein the cluster, install a redundant client LAN.

If you need specific instructions on how to install an adapter, refer to thedocumentation of the adapter you are installing or refer to the user guide of theProLiant server you are using.

Setting Up the RA4000 Arrays

Unless otherwise indicated in this guide, follow the instructions in the CompaqStorageWorks RAID Array 4000 User Guide to set up shared storagesubsystem components. For example, the Compaq StorageWorks RAID Array4000 User Guide shows you how to install shared storage subsystemcomponents for a single server, however; a PDC/O2000 contains multipleservers connected to one or more RA4000 Arrays through redundant storagepaths.

IMPORTANT: When installing an RA4000 Array, do not mount the Fibre Channel cableson cable management arms. Support the Fibre Channel cable so that a bend radius at thecable connector is not less than 3 inches.

Figure 5-2 shows two RA4000 Arrays connected to two clustered serversthrough one redundant FC-AL. The FC-AL is redundant because there are twopaths from each node to each RA4000 Array.






StorageHub #1

Node 1 Node 2

StorageHub #2


Figure 5-2. RA4000 Arrays connected to clustered servers through oneredundant FC-AL

IMPORTANT: Although you can configure the RA4000 Array with a single drive installed,it is strongly recommended for cluster configuration that all shared drives be in theRA4000 Array before running the Compaq Array Configuration Utility.


The Array Configuration Utility is used to set up the hardware aspects of anydrives attached to an array controller, including the drives in the sharedRA4000 Arrays. The Array Configuration Utility stores the driveconfiguration information on the drives themselves; therefore, after you haveconfigured the drives from one of the cluster nodes, it is not necessary toconfigure the drives from the other cluster node.

Before you run the Array Configuration to set up your drive arrays during theSmartStart installation, review the instructions in the “Installing the OperatingSystem Software” section of this chapter. These instructions include clusteringinformation that is not included in the Compaq StorageWorks RAID Array4000 User Guide.

For detailed information about configuring the drives using the ArrayConfiguration Utility, see the Compaq StorageWorks RAID Array 4000 UserGuide.

For information about configuring your shared storage subsystem with RAID,see Chapter 4, “Planning.”




Installing GBIC-SW Modules for the RA4000 ArrayControllers

Each RA4000 Array contains two Compaq StorageWorks RAID Array 4000Controllers (RA4000 Array Controllers) and ships with four GBIC-SWmodules. Insert one module into an RA4000 Array Controller and the othermodule into a Storage Hub. Each end of the Fibre Channel cable connecting anRA4000 Array Controller to a Storage Hub plugs into a GBIC-SW module.


1. Insert a GBIC-SW module into each RA4000 Array Controller in anRA4000 Array.

2. Insert a GBIC-SW module into each of two ports on each Storage Hub(one module for each RA4000 Array controller).

3. Repeat steps 1 and 2 for all other RA4000 Arrays in the redundantFC-AL.

Cabling the Storage Hubs to the RA4000 ArrayControllers

You can configure a redundant FC-AL as an active/standby configuration oran active/active configuration. These types of configurations indicate the stateof the Fibre Host Adapters in each node.

In an active/standby configuration, one Fibre Host Adapter in each clusternode is connected to an active RA4000 Array Controller and the other FibreHost Adapter in each cluster node is connected to a standby RA4000 ArrayController. In an active/active configuration, both Fibre Host Adapters in eachnode are connected to an active array controller. The number of arraycontrollers each Fibre Host Adapter is connected to depends on the number ofRA4000 Arrays in the redundant FC-AL. For more information, see Chapter 2,“Architecture.”

In a PDC/O2000, each RA4000 Array has two RA4000 Array Controllers, oneactive and one standby. In some RA4000 Arrays, the top RA4000 ArrayController defaults to be the active array controller and in other cases thebottom RA4000 Array Controller defaults to be the active array controller.This default can be changed using the Compaq Redundancy Manager.

You verify the definition of the active array controller in each RA4000 Arrayafter installing Compaq Redundancy Manager. See “Defining Active ArrayControllers” later in this chapter.




NOTE: RA4000 Arrays are available in rack and tower models. RA4000 Array Controllersin rack models are located in top rear and bottom rear slots; in tower models, the arraycontrollers are located in right rear and left rear slots. The right rear and left rear slots intower models correspond to the top and bottom slots, respectively, in rack models. Theexamples in this guide show the array controller locations in rack models.

Cabling an Active/Standby Configuration

In an active/standby configuration, Storage Hub #1 connects to active RA4000Array Controllers and Storage Hub #2 connects to standby RA4000 ArrayControllers. The cabling instructions are the same for any number of RA4000Arrays.

NOTE: When defining the active array controllers as instructed later in this chapter, makethe top array controller in each RA4000 Array the active controller.

To cable the Storage Hubs to the RA4000 Array Controllers in anactive/standby configuration:

1. Using Fibre Channel cables, connect Storage Hub #1 to the top (active)RA4000 Array Controller in each RA4000 Array.

2. Using Fibre Channel cables, connect Storage Hub #2 to the bottom(standby) RA4000 Array Controller in each RA4000 Array.

Figure 5-3 shows two Storage Hubs connected to the RA4000 ArrayControllers in two RA4000 Arrays. Active components are shaded.

RA4000Array #1

RA4000Array #2



StorageHub #1

StorageHub #2

Figure 5-3. Cabling Storage Hubs to RA4000 Array Controllers in anactive/standby configuration




Cabling an Active/Active Configuration

In an active/active configuration, each Storage Hub is connected to active andstandby RA4000 Array Controllers.

You can cable the Storage Hubs to the RA4000 Array Controllers using one oftwo methods. The differences between the two methods are the location of theactive array controllers in the RA4000 Arrays and the configuration of theFibre Channel cables that connect the Storage Hubs to the array controllers.

Table 5-1 summarizes the active/active cabling methods. Select the methodthat is best for your site.

Table 5-1Active/Active Cabling Methods

MethodActive Array

Controller Location Advantage Disadvantage

1 Top array controllerslot.

Consistency in active arraycontroller definition. All activearray controllers are defined tobe in the top array controller slotin each RA4000 Array.

Cabling is more complexthan for method 2. Somecables from each StorageHub connect to the top arraycontroller in each RA4000Array, and some cablesfrom each Storage Hubconnect to the bottom arraycontroller in each RA4000Array.

2 Top array controllerslot in odd-numberedRA4000 Arrays.

Bottom arraycontroller slot ineven-numberedRA4000 Arrays.

Consistency in cabling. All cablesfrom Storage Hub #1 connect tothe top array controller in eachRA4000 Array. All cables fromStorage Hub #2 connect to thebottom array controller in eachRA4000 Array.

Defining active arraycontrollers is more complexthan for method 1. Theactive array controller mustbe defined as the top arraycontroller slot in someRA4000 Arrays and as thebottom array controller slotin other RA4000 Arrays.




Figure 5-4 shows an example of using method 1 cabling to connect twoStorage Hubs to RA4000 Array Controllers in two RA4000 Arrays.

RA4000Array #1

RA4000Array #2



StorageHub #1

StorageHub #2

Figure 5-4. Method 1 cabling in an active/active configuration with twoRA4000 Arrays


RA4000Array #1

RA4000Array #2



StorageHub #1

StorageHub #2





NOTE: When defining the active array controllers as instructed later in this chapter, formethod 1 cabling make the top array controller in each RA4000 Array the activecontroller. For method 2 cabling, make the top array controller in odd-numbered RA4000Arrays the active controller; make the bottom array controller in even-numbered RA4000Arrays the active controller.

To cable the Storage Hubs to the RA4000 Array Controllers in an active/activeconfiguration:

1. Using Fibre Channel cables, connect Storage Hub #1 to the activeRA4000 Array Controller in odd-numbered RA4000 Arrays (RA4000Array #1, RA4000 Array #3, and RA4000 Array #5).

2. Using Fibre Channel cables, connect Storage Hub #1 to the standbyRA4000 Array Controller in even-numbered RA4000 Arrays (RA4000Array #2 and RA4000 Array #4).

3. Using Fibre Channel cables, connect Storage Hub #2 to the activeRA4000 Array Controller in even-numbered RA4000 Arrays.

4. Using Fibre Channel cables, connect Storage Hub #2 to the standbyRA4000 Array Controller in odd-numbered RA4000 Arrays.

Installing Additional Redundant FC-ALs

At this point, you have installed the hardware for one redundant FC-AL. Toadd a redundant FC-AL, install another set of the hardware required for oneredundant FC-AL, including:



■ Fibre Channel cables connecting the Fibre Host Adapters to the StorageHubs

■ One to five RA4000 Arrays

■ Fibre Channel cables connecting the Storage Hubs to the RA4000Arrays





Cabling the Ethernet Cluster Interconnect

For clusters of three or more nodes, Compaq requires that you use twoEthernet 100-Mbit/sec switches to maintain good network performance acrossthe cluster. If there are only two nodes in a cluster, you can use either Ethernethubs or switches and standard Ethernet cables to connect the nodes.

NOTE: If the current cluster contains two nodes but you anticipate adding nodes in thefuture, consider installing switches now.

Redundant crossover cables are not supported and therefore cannot be used ina PDC/O2000 cluster.

To install Ethernet hubs or switches:

1. Insert the ends of two Ethernet cables into two Ethernet adapter portsdesignated for the cluster interconnect.

2. Connect the other end of one Ethernet cable to an Ethernet hub orswitch. Connect the other end of the second Ethernet cable to the secondEthernet hub or switch.

3. Repeat steps 1 and 2 for all nodes in the cluster.

4. Install one crossover cable between the Ethernet hubs or switches.

Figure 5-6 shows an example of a redundant client LAN and a redundantEthernet cluster interconnect.

Node 1 Node 2





CrossoverCable

Dual-portEthernet

Adapters (2)

Dual-portEthernet

Adapters (2)

CrossoverCable

Figure 5-6. Redundant client LAN and Ethernet cluster interconnect




For more information on configuring Ethernet connections in a redundantcluster interconnect, including enabling failover from one Ethernet path toanother, see Supported Ethernet Interconnects for Compaq Parallel DatabaseClusters Using Oracle Parallel Server at


Cabling the Client LAN

You can use any TCP/IP network to connect to a client LAN. The followingprocedure contains instructions for cabling an Ethernet client LAN.

To cable an Ethernet client LAN:

1. Insert one end of an Ethernet cable into an Ethernet adapter portdesignated for the client LAN in a cluster node.

If you are using a recommended dual-port Ethernet adapter for thecluster interconnect, connect the client LAN to the empty port.

If you are using a recommended single-port adapter for the clusterinterconnect, connect the client LAN to the port on the embeddedadapter or to another single-port Ethernet adapter.

2. Connect the node to the client LAN by inserting the other end of theclient LAN Ethernet cable to a port in the Ethernet hub or switch.

3. Repeat steps 1 and 2 for all other cluster nodes.

Redundant Client LAN

If you elect to install an Ethernet client LAN, a redundant client LAN requirestwo single-port Ethernet adapters or one dual-port Ethernet adapter in eachcluster node. It also requires two Ethernet hubs or switches, and one Ethernetcrossover cable must be installed between the Ethernet hubs or switches.Installing redundant crossover cables directly between the nodes is notsupported.

For information on configuring Ethernet connections in a redundant clientLAN, including enabling failover from one Ethernet path to another, seeSupported Ethernet Interconnects for Compaq Parallel Database ClustersUsing Oracle Parallel Server at





IMPORTANT: Compaq recommends Service Pack 4 or 5 of Windows NT Server for aredundant Ethernet cluster interconnect or client LAN. Using Service Pack 3 requiresinstalling the approved Microsoft patch (hot fix) article ID Q156655, entitled “ MemoryLeak and STOP Screens Using intermediate NDIS Drivers.”

Installing Operating System Software andConfiguring the RA4000 Arrays

You will follow an automated procedure using Compaq SmartStart to installthe operating system software and configure the shared storage on the RA4000Arrays.

Guidelines for Clusters

Installing clustering software requires several specific steps and guidelines thatmight not be necessary when installing software on a single server. Be sure toread and understand the following items beforebeforebeforebefore proceeding with the specificsoftware installation steps in “Automated Installation Steps.”

■ Because a PDC/O2000 contains multiple servers, have sufficientsoftware licensing rights to install Windows NT Server softwareapplications on each server.

■ Be sure your servers, adapters, hubs, and switches are installed andcabled before you install the software.

■ Power on the cluster as instructed later in this chapter in “PowerDistribution and Power Sequencing Guidelines.”

■ SmartStart runs the Compaq Array Configuration Utility, which is usedto configure the drives in the RA4000 Arrays. The Array ConfigurationUtility stores the drive configuration information on the drivesthemselves. After you have configured the shared drives from one of thecluster nodes, it is not necessary to configure the drives from the othercluster nodes.

When the Array Configuration Utility runs on the first cluster node,configure the shared drives in the RA4000 Array. When SmartStart runsthe utility on the other cluster nodes, you will be presented theinformation on the shared drives that was entered when the ArrayConfiguration Utility was run on the first node. Accept the informationas presented and continue.

NOTE: Local drives on each cluster node still need to be configured.




■ When you set up an Ethernet cluster interconnect, be sure to selectTCP/IP as the network protocol. The Ethernet cluster interconnectshould be on its own subnet.

IMPORTANT: The IP addresses of the Ethernet cluster interconnect must be static, notdynamically assigned by DHCP.

■ Be sure to set up unique IP addresses and node names for each node inthe hosts and lmhosts files at %SystemRoot%\system32\drivers\etc.

� For a redundant Ethernet cluster interconnect, one IP address andnode name is for the cluster interconnect, and the other IP addressand node name is for the client LAN. Both entries are required foreach node in the cluster.

� After setting up these file entries, be sure to restart the node so itpicks up the correct IP addresses.

■ Run Windows NT Server Disk Administrator on each node to verify youcan see the same shared storage subsystem resources on all RA4000Arrays and select the Commit Changes option in Disk Administrator.Restart all nodes.

Automated Installation Using SmartStart

CAUTION: Automated installation using SmartStart assumes that it is beinginstalled on new servers. If there is any existing data on the servers, it will bedestroyed.

You will need the following during SmartStart installation:

■ SmartStart and Support Software CD 4.3 or later (some server modelsmight require a later version)

■ Microsoft Windows NT Server Standard or Enterprise Edition 4.0 andService Pack 3, 4, or 5





■ SmartStart Installation Poster

■ Server Profile Diskette

Cluster-Specific SmartStart Installation

The SmartStart Installation Poster describes the general flow of configuringand installing software on a single server. The installation for a PDC/O2000will be very similar.

The one difference is that through the Array Configuration Utility, SmartStartgives you the opportunity to configure the shared drives on all servers. For thePDC/O2000, configure the drives on the first server, then accept the samesettings for the shared drives when given the option on the other servers.

Automated Installation Steps

You will perform the following automated installation steps to installoperating system software on every node in the cluster.

1. Power up the following cluster components in this order: RA4000Arrays, Storage Hubs, and Ethernet hubs/switches.

2. Power up a cluster node and put the SmartStart and Support SoftwareCD into the CD-ROM drive.

3. Select the Assisted IntegrationAssisted IntegrationAssisted IntegrationAssisted Integration installation path.

4. When prompted, insert the Server Profile Diskette into the floppy diskdrive.

5. Select Windows NT Server Standard Edition or Windows NT ServerEnterprise Edition as the operating system.

6. Continue with the Assisted Integration installation. Windows NT Serveris installed as part of this process.

NOTE: For clustered servers, take the default for Automatic Server Recovery (ASR) andselect standalone as the server type.

NOTE: When configuring the network as part of Windows NT Server, you will beprompted for the IP addresses to be associated with the network ports. Assign the initialport with the IP address to be associated with the client LAN and assign the other portwith the IP Address to be associated with the cluster interconnect.




NOTE: When prompted to use the Array Configuration Utility, it is only necessary toconfigure the shared drives during node1 setup. When configuring the other nodes, theutility shows the results of the shared drives configured during node1 setup.

7. When prompted, install Windows NT Server Service Pack 3, 4, or 5. Ifyou are installing Windows NT Server/Enterprise Edition after theService Pack, the Enterprise Edition Installer loads automatically.

NOTE: Do not install Microsoft Cluster Server.

8. Select the ProtocolsProtocolsProtocolsProtocols tab in the Network applet, and double-click TCP/IPTCP/IPTCP/IPTCP/IP.The TCP/IP Properties dialog box appears. The IP addresses that wereentered during the installation are shown. Note which IP addresses areassociated with which ports.

9. Enter unique IP addresses and node names for each node in the hostsand lmhosts files located at %SystemRoot%\system32\drivers\etc.Record this information.

For the redundant Ethernet cluster interconnect, one IP address and nodename is for the redundant cluster interconnect, and the other IP addressand node name is for the client LAN. For example, node1node1node1node1 for the clientLAN and node1_sannode1_sannode1_sannode1_san for the cluster interconnect. (The “_san” stands forsystem area network.)

10. Restart the node so it picks up the IP addresses.

Due to the complexity of Windows NT Server and multiple-NICservers, you need to verify that the correct IP addresses are assigned tothe correct ports/NICs and that the Ethernet cables are connected to thecorrect ports. If IP addresses are not assigned to the correct port, Oraclesoftware and external programs cannot communicate over the propernetwork link. The next step describes how to perform this verification.

11. Verify that the IP addresses for the client LAN and cluster interconnectare correctly assigned by pinging the machine host name. (Find thisname by selecting the IdentificationIdentificationIdentificationIdentification tab in the Network control panel.)The IP address returned by the ping utility is one of the IP addresses youspecified; it is the IP address that Windows NT Server assigned to theclient LAN.

12. If the ping command does not return the IP address you specified in theTCP/IP Properties dialog box for the client LAN port and you are usingService Pack 3:3:3:3:

a. Swap the IP addresses specified for the client LAN port and clusterinterconnect port.

b. Click OKOKOKOK and restart the system if prompted.




c. Now that you know which Ethernet port Windows NT Serverassigned to the client LAN and cluster interconnect, verify that theEthernet cables are connected to the appropriate ports. You do notneed to modify the hosts or lmhosts files.

13. If the ping command does not return the IP address you specified in theTCP/IP Properties dialog box for the client LAN port and you are usingService Pack 4 or 5:4 or 5:4 or 5:4 or 5:

a. Double-click the NetworkNetworkNetworkNetwork icon in the Control Panel and select theBindingsBindingsBindingsBindings tab.

b. Select all protocols in the Show Bindings for scroll box.

c. Click ++++ (plus sign) next to the TCP/IP protocol. A list of all installedEthernet NICs appears, including the slot number and port numberof each. Windows NT Server uses the NIC at the top of the list forthe client LAN.

d. Change the binding order of the NICs to put the NIC you specifiedfor the client LAN at the top of the list. Find the client LAN NIC inthe list and select it.

e. With the client LAN NIC selected, click the Move UpMove UpMove UpMove Up button toposition this NIC to the top of the list.

f. Click CloseCloseCloseClose on the dialog box and restart the node when prompted.

IMPORTANT: Record the cluster interconnect node name, the client LAN node name, andthe IP addresses assigned to them. You will need this information later when installingCompaq OSDs.

14. If you are installing node1, open Disk Administrator to create extendeddisk partitions and logical partitions within the extended partitions on allRA4000 Arrays. (If you are installing a node other than node1, skip toStep 15.)

a. Within each extended partition, create a small (10 megabytes)logical partition as the first logical partition. Format it with NTFSand label it. This label will display in both the Redundancy ManagerGUI and the Disk Administrator window, allowing you to identifythe drive in either utility.

NOTE: Do not unassign the drive letter automatically assigned to each partition by DiskAdministrator. Redundancy Manager cannot see the volume label if there is no drive letterassigned.

b. Create all disk partitions on the RA4000 Arrays from node1, selectCommit Changes NowCommit Changes NowCommit Changes NowCommit Changes Now from the File menu, and restart the node.

For more information on creating partitions, see the Oracle8 EnterpriseEdition Getting Started Release 8.0.5 for Windows NT.




15. If you are installing a node other than node1, open Disk Administratorto verify that the same shared disk resources are seen from this node asthey are seen from other installed nodes in the cluster. If they are not,restart the node and review the shared disk resources again.

16. Repeat steps 2 through 15 on all other cluster nodes.

17. After configuring all nodes in the cluster, verify the client LANconnections by pinging all nodes in the cluster from each cluster node.Use the client LAN node name, for example, node1node1node1node1 with the pingcommand. Verify the Ethernet cluster interconnect connections by usingthe Ethernet cluster interconnect node name, for example, node1_san,node1_san,node1_san,node1_san,with the ping command.

Installing Compaq Redundancy ManagerIn clusters using RA4000 Arrays, the Compaq Redundancy Manager detectsfailures in redundant FC-AL components, such as Fibre Host Adapters, FibreChannel cables, Storage Hubs, and RA4000 Array Controllers. WhenRedundancy Manager detects a failure of an FC-AL component on an activepath, it reroutes I/O through another FC-AL path.

You must install Redundancy Manager on all cluster nodes.

To install Redundancy Manager:

1. Put the Compaq Redundancy Manager (Fibre Channel) CD into theCD-ROM drive of a cluster node. The Install program is automaticallyloaded.

2. Follow the instructions on the Redundancy Manager screens.

3. Remove the Compaq Redundancy Manager (Fibre Channel) CD fromthe CD-ROM drive.

4. Restart the node.

5. Repeat steps 1 through 4 for all nodes in the cluster.




Verifying Shared Storage Using RedundancyManager

Make sure you can see the same shared disk resources from each node usingthe Redundancy Manager as you can using the Disk Administrator.

To verify the shared storage resources from Redundancy Manager:

1. On a cluster node, click Start,Start,Start,Start, Programs,Programs,Programs,Programs, and Compaq RedundancyCompaq RedundancyCompaq RedundancyCompaq RedundancyManager.Manager.Manager.Manager. The Compaq Redundancy Manager (Fibre Channel) screen isdisplayed.

Logical Dr iveLogical Dr iveLogical Dr iveLogical Dr iveTop Array ControllerTop Array ControllerTop Array ControllerTop Array ControllerBottom Array ControllerBottom Array ControllerBottom Array ControllerBottom Array Controller

This screen shows five RA4000 Arrays, each named by its serialnumber and each containing one logical drive. The logical drives havevolume labels Disk 1, Disk 2, and so on. Each logical drive also has adrive letter assigned to it. The top and bottom array controllers of eachRA4000 Array are shown below the logical drives.




The array controllers on the active FC-AL paths are in bold. In thisexample, the bottom array controllers are active. The screen also showsthat the Fibre Host Adapter to which the active array controller isconnected is located in server PCI slot #6.

2. Verify that the Fibre Host Adapter to which you intended to connect theactive array controller is located in the server PCI slot indicated, in thisexample, in PCI slot #6.

3. If the shared disk resources do not look the same as they do in DiskAdministrator, restart the node and verify again.

4. Exit the Redundancy Manager.

IMPORTANT: Redundancy Manager should be run from only one cluster node at any onetime; otherwise, multiple Redundancy Manager processes will try to control the sameRA4000 Arrays.

5. Repeat steps 1 through 4 for all cluster nodes.

Defining Active Array Controllers

In some RA4000 Arrays, the RA4000 Array Controller in the top arraycontroller slot is the active array controller, and in other cases the RA4000Array Controller in the bottom array controller slot is the active arraycontroller.

Define the active array controllers according to which type of configurationyou are using (active/standby or active/active). If you are using anactive/active configuration, the active array controller location also depends onthe cabling method, as discussed previously in “Cabling an Active/ActiveConfiguration.”

Table 5-2Active Array Controller Locations

Configuration Active Array Controller Location

Active/Standby Top array controller slot

Active/Active: Cabling Method 1 Top array controller slot

Active/Active: Cabling Method 2 Top array controller slot in odd-numbered RA4000Arrays

Bottom array controller slot in even-numberedRA4000 Arrays




NOTE: If Redundancy Manager indicates that the active array controllers are alreadylocated in the proper slots, you do not need to redefine their locations by performing thefollowing procedure.

Changing the location of the active array controller changes the entire FC-ALpath, which includes the Fibre Host Adapter, the Storage Hub, and theRA4000 Array Controller. So, when you change the active array controller,you change the entire active FC-AL path.

To change the active FC-AL path:

1. On any cluster node, click Start,Start,Start,Start, Programs,Programs,Programs,Programs, and Compaq RedundancyCompaq RedundancyCompaq RedundancyCompaq RedundancyManager.Manager.Manager.Manager. The Compaq Redundancy Manager (Fibre Channel) screen isdisplayed.


2. Right-click the standby FC-AL path you want to make active. A pop-upmenu appears.

3. Select Set As ActiveSet As ActiveSet As ActiveSet As Active and confirm your selection when prompted. All thestandby FC-AL paths from that RA4000 Array to the indicated FibreHost Adapter are now active (in bold). All the formerly active FC-ALpaths from that RA4000 Array are now indicated as standby FC-ALpaths (not in bold).

IMPORTANT: Wait at least 10 to 30 seconds after changing an FC-AL path in oneRA4000 Array before changing an FC-AL path in another RA4000 Array.

4. Repeat steps 1 through 3 for every RA4000 Array in which you need tochange the active FC-AL path.

NOTE: You can change a path from standby to active, as in this example, or you canchange the path from active to standby.




Installing Oracle SoftwareUsing the Oracle8 Enterprise Edition Getting Started Release 8.0.5 forWindows NT, follow the steps to install Oracle Enterprise Edition Release8.0.5 software on all cluster nodes, including:

■ Oracle8 Server Release 8.0.5

■ Oracle8 Parallel Server Option Release 8.0.5

■ Oracle8 Parallel Server Manager Release 8.0.5

■ Oracle8 Enterprise Manager Release 8.0.5

After installing Oracle software, the Oracle manual instructs you to installvendor-supplied operating system dependent modules (OSDs). Return to thisdocument to install the Compaq OSDs.

Verifying Cluster CommunicationsUse the ping utility to verify that each node in the cluster can communicatewith every other node.

Run the ping utility from each node in the cluster. Ping the client LAN nameof every node in the cluster. The client LAN name and client LAN IP addresswere assigned to each cluster node while using SmartStart to perform theautomated installation steps. When using the client LAN name with the pingcommand, the IP address displayed by the ping utility should be the clientLAN IP address.

A cluster interconnect name and IP address were assigned to each clusternode. Ping the Ethernet cluster interconnect name of every node in the cluster.When using an Ethernet cluster interconnect name, the IP address displayed bythe ping utility should be an Ethernet cluster interconnect address.

If the ping utility does not return the expected IP addresses, check the entriesin the hosts and lmhosts files at %SystemRoot%\system32\drivers\etc.




For example, the command ping node1ping node1ping node1ping node1 should return the following type ofinformation.

Pinging node1.loc1.yoursite.com [127.0.0.1] with 32 bytes of data:

Reply from 127.0.0.1: bytes=32 time<10ms TTL=128




Installing the Compaq OSDsYou must install the OSDs on each node in the cluster. However, you need torun the NodeList Configurator only once (on one node) to configure all thenodes in the cluster because the NodeList Configurator copies information tothe other nodes in the cluster, provided that:

■ The servers are communicating through a local area network using aTCP/IP protocol.

■ The user running the NodeList Configurator has administrative rightsand a common login on every node.

To install the Compaq operating system dependent modules (OSDs) forOracle8 Parallel Server, use the setup program on the Compaq ParallelDatabase Cluster for Oracle8 Release 8.0.5 Ethernet Clustering Software CD.

OSD Installation Steps

The setup program performs the following tasks:

■ Copies the OSDs to the internal hard drive of the server.

■ Sets up registry entries for file location and standard registry parametersfor the OSDs.

■ Creates a Start menu icon for the NodeList Configurator programprovided with the OSDs.

■ Prompts you to run the NodeList Configurator, a program that allowsyou to configure the nodes (servers) in the cluster. This program alsoenters the Oracle instance ID into the registry of each node.




To install the OSDs:

1. Log in as a user who has administrator privileges on all nodes in thecluster. If all servers in the cluster are in the same Windows NT Serverdomain, log in as a user who is a member of the Domain Administratorsgroup. If the servers are in different domains, create a local user on eachserver with the same Windows NT Server user name and password oneach node. Then make that local user a member of the localAdministrators group on each node before installing the OSDs.

2. Insert the Compaq Parallel Database Cluster for Oracle8 Release 8.0.5Ethernet Clustering Software CD into the CD-ROM drive.

3. Start the OSD installation by clicking the StartStartStartStart button and select Run.Run.Run.Run.The Run dialog box appears.

4. Type d:\setupd:\setupd:\setupd:\setup and click OK.OK.OK.OK. Substitute the letter assigned to the CDdrive for d.d.d.d. The setup program describes the license agreement anddisplays the readme file. It also prompts you for information includingyour name, company, and where you want to install the OSD software.

5. Click NextNextNextNext on the Setup Type dialog box to install the Compaq OSDcomponents.

6. When the setup program completes, it prompts you to run the NodeListConfigurator. Leave the checkbox unselected and click NextNextNextNext and thenFinishFinishFinishFinish to exit.

NOTE: Run the NodeList Configurator only once, after the OSDs have been installed on allnodes in the cluster. When prompted to run the NodeList Configurator on the last node,select the checkbox and then run the NodeList Configurator.

7. Install the OSDs on the remaining nodes in the cluster by repeating steps1 through 6.




Running the NodeList Configurator

To run the NodeList Configurator:

1. Type the following information in the NodeList Configurator dialog boxfor each node in the cluster. It is required that the host names youindicate for the client LAN and cluster interconnect be the same as thoseyou entered in the hosts and lmhosts files at%SystemRoot%\system32\drivers\etc.

Computer Name Name of this computer in the Microsoft network

Client LAN Name of this computer in the client LAN. Can be the sameas the Computer Name.

Cluster Interconnect Name of this computer in the cluster interconnect. Mustnot be the same as the Computer Name.

Oracle Home Directory Location where Oracle is installed

ORACLE_SID System ID identifying the Oracle instance.

2. Type the name of the database if it is different from the default of OPS.

3. When you have completed the information in the NodeList Configuratordialog box, click OKOKOKOK. The information you entered is copied to the othernodes, provided they are all up and running and are communicatingthrough an Ethernet local area network.

4. When the program indicates the setup process is complete, click FinishFinishFinishFinishto exit the program.

The following graphic shows an example of a complete NodeListConfigurator screen.




Installing Object Link Manager

IMPORTANT: After installing Oracle software, you must install Object Link Manager on allcluster nodes. This is required to maintain the symbolic links between disk partitions andOracle data files. Failure to install Object Link Manager can result in the Oracle databasenot starting or, if it starts, in not finding the correct data.

Windows NT Server assigns disk numbers to drives in the shared storagesubsystems based on the order in which the shared storage subsystems arepowered on. Through the symbolic links created by SETLINKS, Oracle8Server uses the order in which disks are brought online to determine thelocation of its database files.




Therefore, the order in which disks are brought online directly affects the disknumbers assigned to the disk, which in turn affects the ability of Oracle8Server to find Oracle data files. If shared storage subsystems are powered offand then powered on in a different order than they were when the cluster wasinitially configured, the order and numbering of the shared disk drives willchange for all nodes in the cluster.

To avoid this problem, install Object Link Manager. Object Link Managersimplifies the creation and maintenance of symbolic links between diskpartitions and Oracle data files by placing the symbolic link names directlyinto the disk partitions, thereby tracking the symbolic links dynamically. Thismeans that if the order in which disk drives are brought online changes,Oracle8 Server can find the correct data files and can properly start the Oracledatabase.

You must install Object Link Manager before configuringbefore configuringbefore configuringbefore configuring Oracle software.

For instructions on installing and using Object Link Manager, see the readmefile on the Compaq Parallel Database Cluster for Oracle8 Release 8.0.5Ethernet Clustering Software CD at:

\Object Link Manager\readme.txt

Configuring Oracle SoftwareUse Oracle8 Enterprise Edition Getting Started Release 8.0.5 for Windows NTto configure the Oracle software. Configuring Oracle software includesconfiguring Oracle Parallel Server, Oracle Parallel Server Manager, OracleEnterprise Manager, and administering multiple instances.

Additional Notes on Configuring Oracle Software

In addition to the steps outlined in the Oracle documents, you must performseveral additional steps as described in this section.

Set OraclePGMSService to Start in Manual Mode

Before configuring Oracle software, verify that the OraclePGMSService is setto start in manual, not automatic, mode. It defaults to start in automatic mode.




If Your Database Is Not Called OPS

The database name defaults to OPS, but you can change it. You enter thedatabase name in the OPSCONF utility and the NodeList Configuratorprogram. However, to run the OPSCONF utility successfully with a databasename other than OPS, you must first:

■ Run the NodeList Configurator program. The NodeList Configuratorupdates the appropriate entries in the registry that OPSCONF dependson.

■ Edit the database name in the ORACLE_DIR\database\init.ora,ORAHOME\database\init_com.ora, and ORACLE_DIR\ops\ops.sqlfiles on each node.

Create a Symbolic Link for OPS_CMDISK

The OPS_CMDISK file contains status information for all nodes in the cluster.Using the Object Link Manager, you must create a symbolic link from theOPS_CMDISK file to a disk partition.

Do Not Configure the OSD Layer or Add anORACLE_SID Entry

When configuring Oracle, do not perform the steps for configuring the OSDlayer and adding an ORACLE_SID entry to the registry. You completed thesetasks by configuring the nodes in the cluster using the setup utility for theOSDs and the NodeList Configurator.

Create Services

There might be an omission in the Oracle8 Enterprise Edition Getting StartedRelease 8.0.5 for Windows NT manual concerning using the CRTSRV batchfile. When you enter the crtsrv crtsrv crtsrv crtsrv SIDSIDSIDSID command, you must follow it with apassword. This same password is required when you connect to an instanceusing the server manager, SVRMGR30.

The Oracle8 Enterprise Edition Getting Started Release 8.0.5 for Windows NTmanual indicates that some vendors might require changing the password forthe Oracle SID and the OracleTNSListener to the administrator’s password forthe node. Do not change the password as instructed.




Configure the Network

There might be an omission in the Oracle manual regarding the step of testingthe configuration. You should restart the Oracle service before startingSVRMGR30.

Start the Database in Parallel Mode

IMPORTANT: Before starting the database in parallel mode, restart all nodes in thecluster.

There might be an omission in the Oracle8 Enterprise Edition Getting StartedRelease 8.0.5 for Windows NT manual regarding starting the database usingSVRMGR30. Before starting the database using SVRMGR, log on as userINTERNAL.

Verifying the Hardware and SoftwareInstallation

Cluster Communications

Use the ping utility to verify that each node in the cluster can communicatewith every other node.

Run the ping utility from the installing cluster node. Verify the installing nodecan communicate with all cluster nodes by pinging the client LAN name andcluster interconnect name of every node in the cluster.

When using the client LAN name, the IP address displayed by the ping utilityshould be the client LAN IP address. When using an Ethernet clusterinterconnect name, the IP address displayed by the ping utility should be theEthernet cluster interconnect address. If this is not the case, check the entriesin the hosts and lmhosts files at %SystemRoot%\system32\drivers\etc.




Access to Shared Storage from All Nodes

Open Disk Administrator to verify that the same shared disk resources are seenfrom this every node in the cluster.

Make sure you can see the same shared disk resources using the RedundancyManager as you can using the Disk Administrator. See “Verifying SharedStorage Using Redundancy Manager” earlier in this chapter.

OSDs

After verifying all the nodes have access to the shared storage, start an Oracleinstance. For example, at a C: command prompt start an instance by entering:

net start oracleservice<SID>

If the OSDs are installed correctly, a message will appear indicating that theOracle service has started successfully.

If the Oracle service did not start successfully, make sure that the cmsrvrprocess is started and the SID is in the Oracle registry. There are two logs thatcan provide information about why the Oracle service did not start. These logsare:

■ \Compaq\OPS\error.log

■ %SystemRoot%\system32\pgms.log

Power Distribution and PowerSequencing Guidelines

It is recommended you connect most cluster components to a Compaq powerdistribution unit (PDU). PDUs connect to an uninterruptible power supply(UPS) or building power. The PDUs and UPSs are the only clustercomponents connected to a building power source. If there is no UPS, thePDU plugs into building power.

It is also recommended that you connect cluster components, such as servers,storage subsystems, switches, and hubs, to two PDUs and UPSs so the clustercomponents continue to operate if one PDU or UPS fails.




Server Power Distribution

Figure 5-7 shows an example of server power distribution for a three-nodecluster. Power supply #1 in each server is connected to PDU #1. Powersupply #2 in each server is connected to PDU #2.

PDU #1 is connected to UPS #1, and PDU #2 is connected to UPS #2. EachUPS is connected to building power.

BuildingPower

UPSPDUProliant Servers

PowerSupplies

Figure 5-7. Server power distribution in a three-node cluster

Having two PDUs and UPSs in the cluster provides two paths from the serversto building power and means that the cluster stays up and running if a PDU orUPS fails in one of the paths.

RA4000 Array Power Distribution

An RA4000 Array can have one or two power supplies. If each array in thecluster has two power supplies, the power distribution (with respect toconnecting them to PDUs and UPSs) can be configured similarly to theexample shown for server power distribution. By substituting RA4000 Arraysfor ProLiant servers in Figure 5-7, you can provide two redundant paths fromthe RA4000 Arrays to building power.

If each RA4000 Array has one power supply, connect an RA4000 Array to aPDU, connect the PDU to a UPS, and then connect the UPS to building power.If there is no UPS, connect the PDU directly to building power.




Power Sequencing

Be sure to power up the cluster components in the following order:

1. RA4000 Arrays

2. Storage Hubs (Power is applied when the AC power cord is plugged in.)

3. Ethernet hubs/switches

4. ProLiant servers

Be sure to power down the cluster components in the following order:

1. ProLiant servers

2. Ethernet hubs/switches


4. RA4000 Arrays

Shutting down and powering off the servers first allows them to perform taskssuch as flushing queued database write transactions to disk and properlyterminating running processes.


Part Number: 161397-001 File Name: g-ch6 Installation and Configuration.doc Last Saved On: 11/3/99 10:59 AM

Chapter 6Installation and Configuration for

Oracle8i Release 8.1.5

This chapter provides instructions for installing and configuring the CompaqParallel Database Cluster Model PDC/O2000 (PDC/O2000) for use withOracle8i Release 8.1.5 software.

A PDC/O2000 is a combination of several individually available products. Asyou set up your cluster, have the following materials available duringinstallation. You will find references to them throughout this chapter.

■ User guides for the clustered Compaq ProLiant servers

■ Installation posters for the clustered ProLiant servers

■ Installation guides for the cluster interconnect and client LANinterconnect adapters

■ Compaq StorageWorks RAID Array 4000 User Guide

■ Compaq StorageWorks Fibre Channel Host Bus Adapter InstallationGuide



■ Compaq SmartStart Installation Poster

■ Compaq SmartStart and Support Software CD

■ Microsoft Windows NT Server Administrator’s Guide




■ Microsoft Windows NT Server Standard or Enterprise Edition 4.0CD/Service Pack 3, 4, or 5

■ Compaq Redundancy Manager (Fibre Channel) CD

■ Compaq Parallel Database Cluster for Oracle8i Release 8.1.5 EthernetClustering Software CD or Compaq Parallel Database Cluster forOracle8i Release 8.1.5 ServerNet Clustering Software CD

■ Oracle8i Parallel Server Setup and Configuration Guide Release 8.1.5

■ Oracle8i Enterprise Edition for Windows NT and Windows 95/98Release Notes, Release 8.1.5

■ Oracle8i Enterprise Edition CD

Installation OverviewThe following summarizes the installation and setup of your PDC/O2000:

■ Installing the hardware, including:

� Proliant servers

� Compaq StorageWorks Fibre Channel Host Bus Adapters (FibreHost Adapters)


� Compaq StorageWorks Fibre Channel Storage Hubs (Storage Hubs)

� Compaq StorageWorks RAID Array 4000s (RA4000 Arrays)

� Cluster interconnect and client LAN adapters

� Ethernet hubs or switches or Compaq ServerNet switches

■ Installing and configuring operating system software, including:

� SmartStart 4.3 or later

� Windows NT Server 4.0 Standard or Enterprise Edition and ServicePack 3, 4, or 5

■ Configuring the storage arrays

■ Installing Compaq Redundancy Manager

Installation and Configuration for Oracle8i Release 8.1.5 6-3



■ Installing and configuring the Compaq operating system dependentmodules (OSDs), including:

� Verifying installation of the SNMP Service, verifying clustercommunications, mounting remote drives, and verifying systemadministrator privileges

� Using Oracle Universal Installer to install OSDs for an Ethernet orCompaq ServerNet cluster interconnect

■ Installing and configuring Oracle software, including:

� Oracle8i Enterprise Edition with Oracle8i Parallel Server Option

■ Installing Object Link Manager

■ Verifying the hardware and software installation, including:

� Cluster communications

� Access to shared storage from all nodes

� Client access to the Oracle8i database

■ Power distribution and power sequencing guidelines

Installing the Hardware

Setting Up the Nodes

Physically preparing the nodes (servers) for a cluster is not very different thanpreparing them for individual use. You will install all necessary adapters andinsert all internal hard disks. You will attach network cables and plug in SCSIand Fibre Channel cables. The primary difference is in setting up the sharedstorage subsystem.

Set up the hardware on one node completely, then set up the rest of the nodesidentically to the first one. Do not load any software on any cluster node untilall the hardware has been installed in all cluster nodes. Before loadingsoftware, read “Installing the Operating System Software and Configuring theRA4000 Arrays” in this chapter to understand the idiosyncrasies ofconfiguring a cluster.




IMPORTANT: The servers in the cluster must be set up identically. The clustercomponents common to all nodes in the cluster must be identical, for example, theProLiant server model, cluster interconnect adapters, amount of memory, cache, andnumber of CPUs must be the same for each cluster node. It also means the Fibre hostadapters must be installed into the same PCI slots in each server.

While setting up the physical hardware, follow the installation instructions inyour Compaq ProLiant Server Setup and Installation Guide and in yourCompaq ProLiant Server Installation Poster. When you are ready to install theFibre Host Adapters and your cluster interconnect adapters, refer to theinstructions in the pages that follow.

Installing the Fibre Host Adapters

Each redundant Fibre Channel Arbitrated Loop (FC-AL) requires two FibreHost Adapters in each cluster node. Install these devices as you would anyother PCI adapter.

Install two Fibre Host Adapters on the same PCI bus in each server and intothe same two PCI slots in each server. If you need specific instructions, see theCompaq StorageWorks Fibre Channel Host Bus Adapter Installation Guide.

Installing GBIC-SW Modules for the Fibre HostAdapters

Each Fibre Host Adapter ships with two GBIC-SW modules. Insert onemodule into the Fibre Host Adapter and the other module into a Storage Hub.Each end of the Fibre Channel cable connecting a Fibre Host Adapter to aStorage Hub plugs into a GBIC-SW module.


1. Insert a GBIC-SW module into each Fibre Host Adapter in a server.

2. Insert a GBIC-SW module into a port on each Storage Hub.

3. Repeat steps 1 and 2 for all other Fibre Host Adapters in the redundantFC-AL.




Cabling the Fibre Host Adapters to the StorageHubs

Each redundant FC-AL requires two Storage Hubs. The cabling from FibreHost Adapters to the Storage Hubs is the same for active/standby (one activeand one standby Fibre Host Adapter in each server) and active/active (twoactive Fibre Host Adapters in each server) configurations.

To cable the Fibre Host Adapters to the Storage Hubs:



Figure 6-1 shows the Fibre Host Adapters in two servers connected to twoStorage Hubs.


StorageHub #1

Node 1 Node 2

StorageHub #2


Figure 6-1. Connecting Fibre Host Adapters to Storage Hubs

For more information about the Storage Hubs, see the Compaq StorageWorksStorage Hub 7 Installation Guide and the Compaq StorageWorks Storage Hub12 Installation Guide.




Installing the Cluster Interconnect Adapters

A PDC/O2000 cluster requires a redundant cluster interconnect. You canimplement the cluster interconnect using Ethernet adapters or CompaqServerNet PCI Adapters.

IMPORTANT: Compaq recommends Service Pack 4 or 5 of Windows NT Server for aredundant Ethernet cluster interconnect or client LAN. Using Service Pack 3 requiresinstalling the approved Microsoft patch (hot fix) article ID Q156655, entitled “MemoryLeak and STOP Screens Using intermediate NDIS Drivers.”

Ethernet Cluster Interconnect

Install one dual-port Ethernet adapter or two single-port Ethernet adapters intoeach cluster node. For recommended dual-port and single-port Ethernetadapters, see the Compaq Parallel Database Cluster Model PDC/O2000 forOracle8i Parallel Server Release 8.1.5 Certification Matrix at


If you need specific instructions on how to install an Ethernet adapter, refer tothe documentation for the Ethernet adapter you are installing or refer to theuser guide for the ProLiant server you are using.

ServerNet Cluster Interconnect

Install one ServerNet PCI Adapter into each cluster node. ServerNet PCIAdapters are dual-ported, each having an X port and a Y port to provide twoindependent data paths. For specific instructions on installing a ServerNet PCIAdapter, see the ServerNet PCI Adapter Installation Guide.

Depending on the server model, the location of the ServerNet PCI Adapter canaffect its performance. For information on locating the ServerNet PCI Adapterfor optimal performance, see the Compaq Parallel Database Cluster ModelPDC/O2000 for Oracle8i Parallel Server Release 8.1.5 Certification Matrix at


IMPORTANT: The ServerNet PCI Adapter Installation Guide instructs you to installServerNet drivers provided on the CD in the kit with the ServerNet PCI Adapter. DO NOTinstall these drivers. You will be instructed later to install ServerNet drivers using theCompaq Parallel Database Cluster for Oracle8i Release 8.1.5 ServerNet ClusteringSoftware CD.




Installing the Client LAN Adapters

Unlike other clustering solutions, the PDC/O2000 does not allow transmissionof intra-cluster communication across the client LAN. All suchcommunication must be sent over the cluster interconnect.

Install a NIC into each cluster node for the client LAN. Configuration of theclient LAN is defined by site requirements. To avoid a single point of failurein the cluster, install a redundant client LAN.

If you need specific instructions on how to install an adapter, refer to thedocumentation of the adapter you are installing or refer to the user guide of theProLiant server you are using.

Setting Up the RA4000 Arrays

Unless otherwise indicated in this guide, follow the instructions in the CompaqStorageWorks RAID Array 4000 User Guide to set up shared storagesubsystem components. For example, the Compaq StorageWorks RAID Array4000 User Guide shows you how to install shared storage subsystemcomponents for a single server, however; a PDC/O2000 contains multipleservers connected to one or more RA4000 Arrays through redundant storagepaths.

IMPORTANT: When installing an RA4000 Array, do not mount the Fibre Channel cableson cable management arms. Support the Fibre Channel cable so that a bend radius at thecable connector is not less than 3 inches.

Figure 6-2 shows two RA4000 Arrays connected to two clustered serversthrough one redundant FC-AL. The FC-AL is redundant because there aretwo paths from each node to each RA4000 Array.






StorageHub #1

Node 1 Node 2

StorageHub #2


Figure 6-2. RA4000 Arrays connected to clustered servers through oneredundant FC-AL

IMPORTANT: Although you can configure the RA4000 Array with a single drive installed,it is strongly recommended for cluster configuration that all shared drives be in theRA4000 Array before running the Compaq Array Configuration Utility.


The Array Configuration Utility is used to set up the hardware aspects of anydrives attached to an array controller, including the drives in the sharedRA4000 Arrays. The Array Configuration Utility stores the driveconfiguration information on the drives themselves; therefore, after you haveconfigured the drives from one of the cluster nodes, it is not necessary toconfigure the drives from the other cluster node.

Before you run the Array Configuration to set up your drive arrays during theSmartStart installation, review the instructions in the “Installing the OperatingSystem Software” section of this chapter. These instructions include clusteringinformation that is not included in the Compaq StorageWorks RAID Array4000 User Guide.

For detailed information about configuring the drives using the ArrayConfiguration Utility, see the Compaq StorageWorks RAID Array 4000 UserGuide.

For information about configuring your shared storage subsystem with RAID,see Chapter 4, “Planning.”




Installing GBIC-SW Modules for the RA4000 ArrayControllers

Each RA4000 Array contains two Compaq StorageWorks RAID Array 4000Controllers (RA4000 Array Controllers) and ships with four GBIC-SWmodules. Insert one module into an RA4000 Array Controller and the othermodule into a Storage Hub. Each end of the Fibre Channel cable connecting anRA4000 Array Controller to a Storage Hub plugs into a GBIC-SW module.


1. Insert a GBIC-SW module into each RA4000 Array Controller in anRA4000 Array.

2. Insert a GBIC-SW module into each of two ports on each Storage Hub(one module for each RA4000 Array controller).

3. Repeat steps 1 and 2 for all other RA4000 Arrays in the redundantFC-AL.

Cabling the Storage Hubs to the RA4000 ArrayControllers

You can configure a redundant FC-AL as an active/standby configuration oran active/active configuration. These types of configurations indicate the stateof the Fibre Host Adapters in each node.

In an active/standby configuration, one Fibre Host Adapter in each clusternode is connected to an active RA4000 Array Controller and the other FibreHost Adapter in each cluster node is connected to a standby RA4000 ArrayController. In an active/active configuration, both Fibre Host Adapters in eachnode are connected to an active array controller. The number of arraycontrollers each Fibre Host Adapter is connected to depends on the number ofRA4000 Arrays in the redundant FC-AL. For more information, see Chapter 2,“Architecture.”

In a PDC/O2000, each RA4000 Array has two RA4000 Array Controllers, oneactive and one standby. In some RA4000 Arrays, the top RA4000 ArrayController defaults to be the active array controller and in other cases thebottom RA4000 Array Controller defaults to be the active array controller.This default can be changed using the Compaq Redundancy Manager.

You verify the definition of the active array controller in each RA4000 Arrayafter installing Compaq Redundancy Manager. See “Defining Active ArrayControllers” later in this chapter.




NOTE: RA4000 Arrays are available in rack and tower models. RA4000 Array Controllersin rack models are located in top rear and bottom rear slots; in tower models, the arraycontrollers are located in right rear and left rear slots. The right rear and left rear slots intower models correspond to the top and bottom slots, respectively, in rack models. Theexamples in this guide show the array controller locations in rack models.

Cabling an Active/Standby Configuration

In an active/standby configuration, Storage Hub #1 connects to active RA4000Array Controllers and Storage Hub #2 connects to standby RA4000 ArrayControllers. The cabling instructions are the same for any number of RA4000Arrays.

NOTE: When defining the active array controllers as instructed later in this chapter, makethe top array controller in each RA4000 Array the active controller.

To cable the Storage Hubs to the RA4000 Array Controllers in anactive/standby configuration:

1. Using Fibre Channel cables, connect Storage Hub #1 to the top (active)RA4000 Array Controller in each RA4000 Array.

2. Using Fibre Channel cables, connect Storage Hub #2 to the bottom(standby) RA4000 Array Controller in each RA4000 Array.

Figure 6-3 shows two Storage Hubs connected to the RA4000 ArrayControllers in two RA4000 Arrays.

RA4000Array #1

RA4000Array #2



StorageHub #1

StorageHub #2

Figure 6-3. Cabling Storage Hubs to RA4000 Array Controllers in anactive/standby configuration




Cabling an Active/Active Configuration

In an active/active configuration, each Storage Hub is connected to active andstandby RA4000 Array Controllers.

You can cable the Storage Hubs to the RA4000 Array Controllers using one oftwo methods. The differences between the two methods are the location of theactive array controllers in the RA4000 Arrays and the configuration of theFibre Channel cables that connect the Storage Hubs to the array controllers.

Table 5-1 summarizes the active/active cabling methods. Select the methodthat is best for your site.

Table 6-1Active/Active Cabling Methods

MethodActive Array

Controller Location Advantage Disadvantage

1 Top array controllerslot.

Consistency in active arraycontroller definition. All activearray controllers are defined tobe in the top array controller slotin each RA4000 Array.

Cabling is more complexthan for method 2. Somecables from each StorageHub connect to the top arraycontroller in each RA4000Array, and some cablesfrom each Storage Hubconnect to the bottom arraycontroller in each RA4000Array.

2 Top array controllerslot in odd-numberedRA4000 Arrays.

Bottom arraycontroller slot ineven-numberedRA4000 Arrays.

Consistency in cabling. All cablesfrom Storage Hub #1 connect tothe top array controller in eachRA4000 Array. All cables fromStorage Hub #2 connect to thebottom array controller in eachRA4000 Array.

Defining active arraycontrollers is more complexthan for method 1. Theactive array controller mustbe defined as the top arraycontroller slot in someRA4000 Arrays and as thebottom array controller slotin other RA4000 Arrays.





RA4000Array #1

RA4000Array #2



StorageHub #1

StorageHub #2



RA4000Array #1

RA4000Array #2



StorageHub #1

StorageHub #2





NOTE: When defining the active array controllers as instructed later in this chapter, formethod 1 cabling make the top array controller in each RA4000 Array the activecontroller. For method 2 cabling, make the top array controller in odd-numbered RA4000Arrays the active controller; make the bottom array controller in even-numbered RA4000Arrays the active controller.

To cable the Storage Hubs to the RA4000 Array Controllers in an active/activeconfiguration:

1. Using Fibre Channel cables, connect Storage Hub #1 to the activeRA4000 Array Controller in odd-numbered RA4000 Arrays (RA4000Array #1, RA4000 Array #3, and RA4000 Array #5).

2. Using Fibre Channel cables, connect Storage Hub #1 to the standbyRA4000 Array Controller in even-numbered RA4000 Arrays (RA4000Array #2 and RA4000 Array #4).

3. Using Fibre Channel cables, connect Storage Hub #2 to the activeRA4000 Array Controller in even-numbered RA4000 Arrays.

4. Using Fibre Channel cables, connect Storage Hub #2 to the standbyRA4000 Array Controller in odd-numbered RA4000 Arrays.

Installing Additional Redundant FC-ALs

At this point, you have installed the hardware for one redundant FC-AL. Toadd a redundant FC-AL, install another set of the hardware required for oneredundant FC-AL, including:



■ Fibre Channel cables connecting the Fibre Host Adapters to the StorageHubs

■ One to five RA4000 Arrays

■ Fibre Channel cables connecting the Storage Hubs to the RA4000Arrays





Cabling the Cluster Interconnect

If the cluster uses an Ethernet cluster interconnect, two Ethernet switches arerequired for clusters with three or more nodes. If there are only two nodes in acluster, you can use Ethernet hubs or switches and standard Ethernet cables toconnect the nodes. Redundant crossover cables installed directly betweennodes are not supported and therefore cannot be used in a PDC/O2000 cluster.

If ServerNet is the cluster interconnect, two Compaq ServerNet Switches arerequired for clusters with three or more nodes. When two nodes are connectedby ServerNet, they can be connected directly by using ServerNet cables or byusing two ServerNet Switches.

NOTE: If the current cluster contains two nodes but you anticipate adding nodes in thefuture, consider installing switches now.

Ethernet Cluster Interconnect

If there are only two nodes in a cluster, you can cable the Ethernet clusterinterconnect using standard Ethernet cables and Ethernet hubs or switches.

To maintain good network performance across the cluster interconnect in acluster containing more three or more nodes, Compaq requires using two100 Mbit/sec Ethernet switches.

To install Ethernet hubs or switches:

1. Insert the ends of two Ethernet cables into two Ethernet adapter portsdesignated for the cluster interconnect.

2. Connect the other end of one Ethernet cable to an Ethernet hub orswitch. Connect the other end of the second Ethernet cable to the secondEthernet hub or switch.


4. Install one crossover cable between the Ethernet hubs or switches.




Figure 6-6 shows an example of a redundant client LAN and a redundantEthernet cluster interconnect.

Node 1 Node 2





CrossoverCable

Dual-portEthernet

Adapters (2)

Dual-portEthernet

Adapters (2)

CrossoverCable

Figure 6-6. Redundant client LAN and Ethernet cluster interconnect

For more information on configuring Ethernet connections in a redundantcluster interconnect, including enabling failover from one Ethernet path toanother, see Supported Ethernet Interconnects for Compaq Parallel DatabaseClusters Using Oracle Parallel Server at


ServerNet Cluster Interconnect

If there are only two nodes in the cluster, ServerNet PCI Adapters can beconnected directly by ServerNet cables between the nodes or through twoServerNet Switches. If the cluster contains three or more nodes, ServerNet PCIAdapters must be connected through two ServerNet Switches.

To directly connect two nodes using ServerNet cables:

1. Connect the X port of the ServerNet PCI Adapter in the first node to theX port of the ServerNet PCI Adapter in the second node with oneServerNet cable.

2. Connect the Y port of the ServerNet PCI Adapter in the first node to theY port of the ServerNet PCI Adapter in the second node with a secondServerNet cable.




To connect two or more nodes using two ServerNet Switches:

1. Using a ServerNet cable, Connect the X port of the ServerNet PCIAdapter in a cluster node to a port on the ServerNet Switch dedicated tothe X path.

2. using a ServerNet cable, connect the Y port of the ServerNet PCIAdapter in the same cluster node to a port on the ServerNet Switchdedicated to the Y path.

3. Repeat steps 1 and 2 for the ServerNet PCI Adapter in each cluster node.

IMPORTANT: The ServerNet Switch ports that each node is connected to must be thesame on each ServerNet Switch. For example, if node1 is connected to port 0 on the Xpath switch, it must also be connected to port 0 on the Y path switch.

Figure 6-7 shows a redundant ServerNet cluster interconnect in a four-nodecluster.

ServerNet Switch(Y Path)

ServerNet Switch(X Path)





Node 2

X X X XY Y Y Y

Node 1 Node 3 Node 4

Figure 6-7. Redundant ServerNet cluster interconnect

See the ServerNet Switch Installation Guide for instructions on how to install aServerNet Switch and how to connect ServerNet PCI Adapters to it usingServerNet cables.




Cabling the Client LAN

You can use any TCP/IP network to connect to a client LAN. The followingprocedure contains instructions for cabling an Ethernet client LAN.

To cable an Ethernet client LAN:

1. Insert one end of an Ethernet cable into an Ethernet adapter portdesignated for the client LAN in a cluster node.

If you are using a recommended dual-port Ethernet adapter for thecluster interconnect, connect the client LAN to the empty port.

If you are using a recommended single-port adapter for the clusterinterconnect, connect the client LAN to the port on the embeddedadapter or to another single-port Ethernet adapter.

2. Connect the node to the client LAN by inserting the other end of theclient LAN Ethernet cable to a port in the Ethernet hub or switch.

3. Repeat steps 1 and 2 for all other cluster nodes.

Redundant Client LAN

If you elect to install an Ethernet client LAN, a redundant client LAN requirestwo single-port Ethernet adapters or one dual-port Ethernet adapter in eachcluster node. It also requires two Ethernet hubs or switches, and one Ethernetcrossover cable must be installed between the Ethernet hubs or switches.Installing redundant crossover cables directly between the nodes is notsupported.

For information on configuring Ethernet connections in a redundant clientLAN, including enabling failover from one Ethernet path to another, seeSupported Ethernet Interconnects for Compaq Parallel Database ClustersUsing Oracle Parallel Server at






Installing Operating System Software andConfiguring the RA4000 Arrays

You will follow an automated procedure using Compaq SmartStart to installthe operating system software and configure the shared storage disks on theRA4000 Arrays.

Guidelines for Clusters

Installing clustering software requires several specific steps and guidelines thatmight not be necessary when installing software on a single server. Be sure toread and understand the following items beforebeforebeforebefore proceeding with the specificsoftware installation steps in “Automated Installation Steps.”

■ Because a PDC/O2000 contains multiple servers, have sufficientsoftware licensing rights to install Windows NT Server softwareapplications on each server.

■ Be sure your servers, adapters, hubs, and switches are installed andcabled before you install the software.

■ Power on the cluster as instructed later in this chapter in “PowerDistribution and Power Sequencing Guidelines.”

■ SmartStart runs the Compaq Array Configuration Utility, which is usedto configure the drives in the RA4000 Arrays. The Array ConfigurationUtility stores the drive configuration information on the drivesthemselves. After you have configured the shared drives from one of thecluster nodes, it is not necessary to configure the drives from the othercluster nodes.

When the Array Configuration Utility runs on the first cluster node,configure the shared drives in the RA4000 Array. When SmartStart runsthe utility on the other cluster nodes, you will be presented theinformation on the shared drives that was entered when the ArrayConfiguration Utility was run on the first node. Accept the informationas presented and continue.

NOTE: Local drives on each cluster node still need to be configured.

■ When you set up an Ethernet cluster interconnect, be sure to selectTCP/IP as the network protocol. The Ethernet cluster interconnectshould be on its own subnet.




IMPORTANT: The IP addresses of the Ethernet cluster interconnect must be static, notdynamically assigned by DHCP.

■ Be sure to set up unique IP addresses and node names for each node inthe hosts and lmhosts files at %SystemRoot%\system32\drivers\etc.

� For a redundant Ethernet cluster interconnect, one IP address andnode name is for the cluster interconnect, and the other IP addressand node name is for the client LAN. Both entries are required foreach node in the cluster.

� For a ServerNet cluster interconnect, an IP address and node name isrequired only for the client LAN. This entry is required for eachnode in the cluster.

� After setting up these file entries, be sure to restart the node so itpicks up the correct IP addresses.

■ Run Windows NT Server Disk Administrator on each node to verify youcan see the shared storage subsystem resources on all RA4000 Arraysand select the Commit Changes option in Disk Administrator. Restart allnodes.

Automated Installation Using SmartStart

CAUTION: Automated installation using SmartStart assumes that it is beinginstalled on new servers. If there is any existing data on the servers, it will bedestroyed.

You will need the following during SmartStart installation:

■ SmartStart and Support Software CD 4.3 or later (some server modelsmight require a later version)

■ Microsoft Windows NT Server Standard or Enterprise Edition 4.0 andService Pack 3, 4, or 5


■ SmartStart Installation Poster

■ Server Profile Diskette




Cluster-Specific SmartStart Installation

The SmartStart Installation Poster describes the general flow of configuringand installing software on a single server. The installation for a PDC/O2000will be very similar.

The one difference is that through the Array Configuration Utility, SmartStartgives you the opportunity to configure the shared drives on all servers. Forcluster configuration, you should configure the drives on the first server, thenaccept the same settings for the shared drives when given the option on theother servers.

Automated Installation Steps

You will perform the following automated installation steps to installoperating system software on every node in the cluster.

1. Power up the following cluster components in this order: RA4000Arrays, Storage Hubs, and Ethernet hubs/switches or ServerNetswitches.

2. Power up a cluster node and put the SmartStart and Support SoftwareCD into the CD-ROM drive.

3. Select the Assisted IntegrationAssisted IntegrationAssisted IntegrationAssisted Integration installation path.

4. When prompted, insert the Server Profile Diskette into the floppy diskdrive.

5. Select Windows NT Server Standard Edition or Windows NT ServerEnterprise Edition as the operating system.

6. Continue with the Assisted Integration installation. Windows NT Serveris installed as part of this process.

NOTE: For clustered servers, take the default for Automatic Server Recovery (ASR) andselect standalone as the server type.

NOTE: When configuring the network as part of Windows NT Server, you will beprompted for the IP addresses to be associated with the network ports. Assign the initialport with the IP address to be associated with the client LAN and assign the other portwith the IP Address to be associated with the cluster interconnect.

NOTE: When prompted to use the Array Configuration Utility, it is only necessary toconfigure the shared drives during node1 setup. When configuring the other nodes, theutility shows the results of the shared drives configured during node1 setup.




7. During the Windows NT Server installation process, you are asked ifyou want to install additional services. If you intend to install ServerNetSNMP agents later, click AddAddAddAdd and select SNMP Service.SNMP Service.SNMP Service.SNMP Service.

8. When prompted, install Windows NT Server Service Pack 3, 4, or 5. Ifyou are installing Windows NT Server/Enterprise Edition after theService Pack, the Enterprise Edition Installer loads automatically.

NOTE: Do not install Microsoft Cluster Server.

9. Select the ProtocolsProtocolsProtocolsProtocols tab in the Network applet, and double-click TCP/IP.TCP/IP.TCP/IP.TCP/IP.The TCP/IP Properties dialog box appears. The IP addresses that wereentered during the installation are shown. Note which IP addresses areassociated with which ports.

10. Enter unique IP addresses and node names for each node in the hostsand lmhosts files located at %SystemRoot%\system32\drivers\etc.Record this information.

For a redundant Ethernet cluster interconnect, one IP address and nodename is for the redundant cluster interconnect, and the other IP addressand node name is for the client LAN. For example, node1node1node1node1 for the clientLAN and node1_sannode1_sannode1_sannode1_san for the cluster interconnect. (The “_san” stands forsystem area network.)

For a ServerNet cluster interconnect, an IP address and node name arerequired only for the client LAN.

11. Restart the node so it picks up the IP addresses. If you are using aServerNet cluster interconnect, skip to Step 16.

Due to the complexity with Windows NT Server and multiple-NICservers, you need to verify that the correct IP addresses are assigned tothe correct ports/NICs and that the Ethernet cables are connected to thecorrect ports. If IP addresses are not assigned to the correct port, Oraclesoftware and external programs cannot communicate over the propernetwork link. The next step describes how to perform this verification.

12. Verify that the IP addresses for the client LAN and cluster interconnectare correctly assigned by pinging the machine host name. (Find thisname by selecting the IdentificationIdentificationIdentificationIdentification tab in the Network control panel.)The IP address returned by the ping utility is one of the IP addresses youspecified; it is the IP address that Windows NT Server assigned to theclient LAN.




13. If the ping command does not return the IP address you specified in theTCP/IP Properties dialog box for the client LAN port and you are usingService Pack 3:3:3:3:

a. Swap the IP addresses specified for the client LAN port and clusterinterconnect port.

b. Click OKOKOKOK and restart the system if prompted.

c. Now that you know which Ethernet port Windows NT Serverassigned to the client LAN and cluster interconnect, verify that theEthernet cables are connected to the appropriate ports. You do notneed to modify the hosts or lmhosts files.

14. If the ping command does not return the IP address you specified in theTCP/IP Properties dialog box for the client LAN port and you are usingService Pack 4 or 5:4 or 5:4 or 5:4 or 5:

a. Double-click the NetworkNetworkNetworkNetwork icon in the Control Panel and select theBindingsBindingsBindingsBindings tab.

b. Select all protocols in the Show Bindings for scroll box.

c. Click ++++ (plus sign) next to the TCP/IP protocol. A list of all installedEthernet NICs appears, including the slot number and port numberof each. Windows NT Server uses the NIC at the top of the list forthe client LAN.

d. Change the binding order of the NICs to put the NIC you specifiedfor the client LAN at the top of the list. Find the client LAN NIC inthe list and select it.

e. With the client LAN NIC selected, click the Move UpMove UpMove UpMove Up button toposition this NIC to the top of the list.

f. Click CloseCloseCloseClose on the dialog box and restart the node when prompted.

IMPORTANT: Record the cluster interconnect node name, the client LAN node name, andthe IP addresses assigned to them. You will need this information later when installingCompaq OSDs.

15. If you are installing node1, open Windows NT Server DiskAdministrator to create extended disk partitions and logical partitionswithin the extended partitions on all RA4000 Arrays. (If you areinstalling a node other than node1, skip to step 16.)

a. Within each extended partition, create a small (10 megabytes)logical partition as the first logical partition. Format it with NTFSand label it. This label will display in both the Redundancy ManagerGUI and the Disk Administrator window, allowing you to identifythe drive from either utility.




NOTE: Do not unassign the drive letter automatically assigned to each partition by DiskAdministrator. Redundancy Manager cannot see the volume label if there is no drive letterassigned.

b. Create all disk partitions on the RA4000 Arrays from node1, selectCommit Changes NowCommit Changes NowCommit Changes NowCommit Changes Now from the File menu, and restart the node.

For more information on creating partitions, see the Oracle8i ParallelServer Setup and Configuration Guide Release 8.1.5.

16. If you are installing a node other than node1, open Disk Administratorto verify that the same shared disk resources are seen from this node asthey are seen from other installed nodes in the cluster. If they are not,restart the node and review the shared disk resources again.


18. After configuring all nodes in the cluster, verify the client LANconnections by pinging all nodes in the cluster from each cluster node.Use the client LAN node name, for example, node1node1node1node1 with the pingcommand. If you are using a redundant Ethernet cluster interconnect,verify the cluster interconnect connections by using the Ethernet clusterinterconnect node name, for example, node1_san,node1_san,node1_san,node1_san, with the pingcommand.

Installing Compaq Redundancy ManagerIn clusters using RA4000 Arrays, the Compaq Redundancy Manager detectsfailures in redundant FC-AL components, such as Fibre Host Adapters, FibreChannel cables, Storage Hubs, and RA4000 Array Controllers. WhenRedundancy Manager detects a failure of an FC-AL component on an activepath, it reroutes I/O through another FC-AL path.

You must install Redundancy Manager on all cluster nodes.

To install Redundancy Manager :

1. Put the Compaq Redundancy Manager (Fibre Channel) CD into theCD-ROM drive of a cluster node. The Install program is automaticallyloaded.

2. Follow the instructions on the Redundancy Manager screens.

3. Remove the Compaq Redundancy Manager (Fibre Channel) CD fromthe CD-ROM drive.






Verifying Shared Storage Using RedundancyManager

Make sure you can see the same shared disk resources from each node usingthe Redundancy Manager as you can using Disk Administrator.

To verify the shared storage resources from Redundancy Manager:

1. On a cluster node, click StartStartStartStart, ProgramsProgramsProgramsPrograms, and Compaq RedundancyCompaq RedundancyCompaq RedundancyCompaq RedundancyManagerManagerManagerManager. The Compaq Redundancy Manager (Fibre Channel) screen isdisplayed.





This screen shows five RA4000 Arrays, each named by its serialnumber and each containing one logical drive. The logical drives havevolume labels Disk 1, Disk 2, and so on. Each logical drive also has adrive letter assigned to it. The top and bottom array controllers of eachRA4000 Array are shown below the logical drives.

The array controllers on the active FC-AL paths are in bold. In thisexample, the bottom array controllers are active. The screen also showsthat the Fibre Host Adapter to which the active array controller isconnected is located in server PCI slot #6.

2. Verify that the Fibre Host Adapter to which you intended to connect theactive array controller is located in the server PCI slot indicated (in thisexample, in PCI slot #6.

3. If the shared disk resources do not look the same as they do in DiskAdministrator, restart the node and verify again.

4. Exit the Redundancy Manager.


5. Repeat steps 1 through 4 for all cluster nodes.

Defining Active Array Controllers

In some RA4000 Arrays, the RA4000 Array Controller in the top arraycontroller slot is the active array controller and in other cases the RA4000Array Controller in the bottom array controller slot is the active arraycontroller.

Define the active array controllers according to which type of configurationyou are using (active/standby or active/active). If you are using anactive/active configuration, the active array controller location also depends onthe cabling method, as discussed previously in “Cabling an Active/ActiveConfiguration.”




Table 6-2Active Array Controller Locations

Configuration Active Array Controller Location

Active/Standby Top array controller slot

Active/Active: Cabling Method 1 Top array controller slot

Active/Active: Cabling Method 2 Top array controller slot in odd-numbered RA4000Arrays

Bottom array controller slot in even-numberedRA4000 Arrays

NOTE: If Redundancy Manager indicates that the active array controllers are alreadylocated in the proper slots, you do not need to redefine their locations by performing thefollowing procedure.

Changing the location of the active array controller changes the entire FC-ALpath, which includes the Fibre Host Adapter, the Storage Hub, and theRA4000 Array Controller. So, when you change the active array controller,you change the entire active FC-AL path.


1. On any cluster node, click Start,Start,Start,Start, Programs,Programs,Programs,Programs, and Compaq RedundancyCompaq RedundancyCompaq RedundancyCompaq RedundancyManager.Manager.Manager.Manager. The Compaq Redundancy Manager (Fibre Channel) screen isdisplayed.




IMPORTANT: Wait at least 10to 30 seconds after changing an FC-AL path in one RA4000Array before changing an FC-AL path in another RA4000 Array.






Installing Compaq OSDsUse the Oracle Universal Installer (OUI) program to install the Compaqoperating system dependent modules (OSDs) for Oracle8i Parallel Server.Compaq supplies two software packages with the OUI. One installs CompaqOSDs for an Ethernet cluster interconnect. Another installs Compaq OSDs,ServerNet device drivers, and (optionally) ServerNet SNMP agents for aServerNet cluster interconnect.

NOTE: If you elect to install SNMP agents, the Microsoft SNMP Service must be installedbefore installing the agents with the OUI.

The OUI runs from one node in the cluster and performs the following tasks:

■ Copies the OSDs to the internal hard drive of all cluster nodes.

■ Sets up registry entries for file location and standard registry parametersfor the OSDs on all cluster nodes.

■ If the cluster interconnect is ServerNet, the OUI installs and configuresServerNet device drivers and (optionally) ServerNet SNMP agents onall cluster nodes.

From one node in the cluster, the OUI installs and configures OSDs, ServerNetdevice drivers, and (optionally) ServerNet SNMP agents on the other nodes inthe cluster, provided that:

■ The servers are communicating through a local area network using aTCP/IP protocol.

■ The user running the OUI program has administrator privileges on everynode.




Verifying Installation of the SNMP Service

If you decide to install the SNMP agent for the ServerNet PCI Adapter or theServerNet Switch, or both, the Microsoft SNMP Service must be installedbefore installing any SNMP agents with the OUI.

Verify this service is installed on each node by going to the Services applet ofthe Control Panel. If there is an entry for SNMP Service, it is installed.

If SNMP Service is not listed, do the following to install it:

1. Insert the Windows NT Server Standard or Enterprise Edition 4.0 CDinto the CD-ROM drive.

2. Go to the NetworkNetworkNetworkNetwork applet of the Control Panel and select the ServicesServicesServicesServicestab.

3. Click AddAddAddAdd on the Network Services dialog box. The Select NetworkService dialog box appears.

4. Select SNMP ServiceSNMP ServiceSNMP ServiceSNMP Service from the Network Service list and click OK.OK.OK.OK. TheSetup Copy dialog box appears.

5. Verify that the drive letter displayed in the dialog box refers to theCD-ROM drive and click Continue.Continue.Continue.Continue. The Microsoft SNMP Propertiesdialog box appears.

6. Type the Optional ContactOptional ContactOptional ContactOptional Contact and LocationsLocationsLocationsLocations information and review theServiceServiceServiceService information. The default setting for the Service information isusually fine.

7. Select the TrapsTrapsTrapsTraps tab on the Microsoft Properties dialog box.

8. Type a Community Name,Community Name,Community Name,Community Name, for example, public,public,public,public, and click OK.OK.OK.OK. TheCommunity Name is required to allow Compaq Insight Manager tocommunicate with the system.

9. Choose the Network applet to complete the installation. The appletprompts you to restart the server.

10. Click YesYesYesYes and wait for the server to restart.

11. Reinstall the Service Pack and restart the node when prompted. Makesure you don’t overwrite any drivers installed since the last time theService Pack was applied.

12. Repeat steps 1 through 11 for all cluster nodes on which the SNMPService is not listed.




Verifying Cluster Communications

IMPORTANT: The successful completion of the OUI program depends on TCP/IPconnectivity being set up correctly. It also depends on the person running the OUI havingadministrator access to every node in the cluster. Before running the OUI, verify clustercommunications and the administrator privileges of the person performing the installation.

If the OSDs do not install successfully, the OUI will not offer the option to install Oracle8iParallel Server.

The Compaq operating system dependent modules (OSDs) are installed fromone node in the cluster (the installing node). The node on which you run theOUI program must be able to communicate with every other node in thecluster in order to copy information to all other cluster nodes.

Use the ping utility to verify that each node in the cluster can communicatewith every other node.

Run the ping utility from each node in the cluster. Ping the client LAN nameof every node in the cluster. The client LAN name and client LAN IP addresswere assigned to each cluster node while you were using SmartStart to installthe operating system software.

If you are using an Ethernet cluster interconnect, a cluster interconnect nameand IP address were assigned to each cluster node. Ping the Ethernet clusterinterconnect name of every node in the cluster.

When using the client LAN name with the ping command, the IP addressdisplayed by the ping utility should be the client LAN IP address. When usingan Ethernet cluster interconnect name, the IP address displayed by the pingutility should be an Ethernet cluster interconnect address. If the ping utilitydoes not return the expected IP addresses, check the entries in the hosts andlmhosts files at %SystemRoot%\system32\drivers\etc.

For example, the command ping node1ping node1ping node1ping node1 should return the following type ofinformation.

Pinging node1.loc1.yoursite.com [127.0.0.1] with 32 bytes of data:








Mounting Remote Drives and VerifyingAdministrator Privileges

Because the OUI installs and configures OSDs, ServerNet device drivers, and(optionally) ServerNet SNMP agents on the other nodes in the cluster from theinstalling machine, you must first mount a drive on all remote nodes in thecluster. Remote nodes are those nodes other than the node on which the OUI isrunning (the installing machine).

The drive you mount should be the drive the OUI will write to, and it must bethe same drive on all cluster nodes. There should be at least five megabytes ofspace on the remote drives.

The net use command can be used to mount drives on remote nodes. Theperson running the net use command must have administrator privileges.Therefore, running the net use command not only mounts drives on remotenodes but also verifies the administrator privileges of the installer.

Administrator privileges are required for most installation tasks, such asinstalling device drivers and changing registry entries. To install the OSDs,you must have administrator privileges on each node from the installingmachine.

To mount drives on remote nodes, enter a net use command from a C: prompton the installing machine. Enter one instance of the command for each remotecluster node. The following example mounts the C: drive on a remote node.

C: net use \\machine_name\C$ * /User: <admin_login >

The “*” results in the system prompting you for a password, and the passwordis not echoed. (The Map Network Drive option in Windows NT Explorer is analternative to the net use command.)

If successful, the net use command returns:

“ The command completed successfully.”

If the command is not successful, make sure the administrative shares onremote node drives are present.




Installing Ethernet OSDs

To install the OSDs for an Ethernet cluster interconnect:

1. From the installing node, log in as a user who has administratorprivileges on all nodes in the cluster. If all servers in the cluster are inthe same Windows NT Server domain, log in as a user who is a memberof the Domain Administrators group. If the servers are in differentdomains, create a local user on each server with the same Windows NTServer user name and password on each node. Then make that local usera member of the local Administrators group on each node beforeinstalling the OSDs.

2. Insert the Compaq Parallel Database Cluster for Oracle8i Release 8.1.5Ethernet Clustering Software CD into the CD-ROM drive.

3. If autorun is enabled, the Oracle Universal Installer Welcome screenappears.

If autorun is disabled, enter d:\setupd:\setupd:\setupd:\setup at a C: prompt. Substitute the letterassigned to the CD drive for d.d.d.d. The Oracle Universal Installer Welcomescreen appears.

4. Click NextNextNextNext to continue. The License screen appears.

5. Click AcceptAcceptAcceptAccept to accept the terms of the license agreement. The FileLocations screen appears.




6. Click NextNextNextNext to continue. The Installation Types screen appears.

NOTE: The OUI screens might indicate later revision levels for the OSDs.




7. Select TypicalTypicalTypicalTypical to install the Ethernet OSDs into the default location,C:\Compaq\OPS. Skip to Step 14.

8. Select CustomCustomCustomCustom to install the Ethernet OSDs into a location of yourchoice. The Component Locations screen appears.

9. Select Compaq Ethernet OSDs 4.0.Compaq Ethernet OSDs 4.0.Compaq Ethernet OSDs 4.0.Compaq Ethernet OSDs 4.0. The Component Locations screen isrefreshed to indicate the default location.




10. If you want to accept the default location, skip to Step 14.

11. If you want to specify an alternate location and you know the exactdirectory path, specify it by typing over the default location and skip toStep 14.

12. If you want to specify an alternate location but do not know the exactdirectory path, click Change Location.Change Location.Change Location.Change Location. The Choose Directory screenappears.




13. Browse to the location where you want to install the OSDs. Click OK.OK.OK.OK.The Component Location screen appears indicating the alternatelocation you specified.




14. Click NextNextNextNext to continue. The Cluster Members screen appears.




15. The Cluster Members screen indicates the host name of the installingmachine. Define the node names for the client LAN by adding the nodenames you specified in the hosts and lmhosts files for the client LAN forall other cluster nodes. Do not use special characters. Use a space toseparate the node names.

16. Click NextNextNextNext to continue. The Cluster Member screen reappears.

17. The Cluster Members screen now shows the node names you specifiedon the first Cluster Members screen with “_san” appended to each name,indicating it as a cluster interconnect node name.

If necessary, retype the cluster interconnect node names to match thenames you entered into the hosts and lmhosts files for the Ethernetcluster interconnect. Do not use special characters. Separate the nodenames with a space.

18. Click NextNextNextNext to continue. The OSD Summary screen appears.




19. If the information on the OSD Summary screen is not correct, clickPreviousPreviousPreviousPrevious to modify the previous screens.

When the information is correct, click NextNextNextNext to continue. The OracleSummary screen appears.




20. Click InstallInstallInstallInstall to install the components listed in the summary. At thispoint, the OUI verifies that the installing node can communicate withother cluster nodes through both the client LAN and cluster interconnectnode names. If the OUI cannot communicate with all nodes, this processcan take a while.

21. An Install screen appears that shows a percentage completion bar as theinstallation progresses.

22. When the installation completes, the End of Installation screen appears.




23. Click ExitExitExitExit to quit the OUI, Next InstallNext InstallNext InstallNext Install to install another product.

24. Start the cluster manager, OracleCMService,OracleCMService,OracleCMService,OracleCMService, on all nodes in the cluster.This service must be started before installing Oracle software. See theOracle documentation for information on starting the cluster manager.

Installing ServerNet OSDs, Drivers, and SNMPAgents

To install the OSDs, drivers, and SNMP agents for a ServerNet clusterinterconnect:

1. If you plan to install SNMP agents, verify the SNMP Service is installedand stop it if it is running. For information on installing the SNMPService, see “Verifying Installation of the SNMP Service” earlier in thischapter.

2. From the installing node, log in as a user who has administratorprivileges on all nodes in the cluster. If all servers in the cluster are inthe same Windows NT Server domain, log in as a user who is a memberof the Domain Administrators group. If the servers are in differentdomains, create a local user on each server with the same Windows NTServer user name and password on each node. Then make that local usera member of the local Administrators group on each node beforeinstalling the OSDs.




3. Insert the Compaq Parallel Database Cluster for Oracle8i Release 8.1.5ServerNet Clustering Software CD into the CD-ROM drive.


If autorun is disabled, enter d:\setupd:\setupd:\setupd:\setup at a C: prompt. Substitute the letterassigned to the CD drive for d.d.d.d. The Oracle Universal Installer Welcomescreen appears.

5. Click NextNextNextNext to continue. The License screen appears.

6. Click AcceptAcceptAcceptAccept to accept the terms of the license agreement. The FileLocations screen appears.




7. Click NextNextNextNext to continue. The Installation Types screen appears.

NOTE: The OUI screens might indicate later revision levels for the OSDs, drivers, andutilities.




8. A typical installation does not install ServerNet SNMP agents; for thatyou must select a custom installation. Select TypicalTypicalTypicalTypical to install theServerNet OSDs and drivers into the default location, C:\Compaq\. Skipto Step 18.

9. Select CustomCustomCustomCustom if you want to install the ServerNet OSDs and driversinto a location of your choice, or if you want to install ServerNet SNMPagents. The Available Product Components screen appears.




10. The Available Product Components screen shows all ServerNet OSDs,drivers, and SNMP agents selected. Select the products you want toinstall (at least one) and click Next.Next.Next.Next.

You must install the OSDs and drivers for a ServerNet clusterinterconnect to work. Install the SNMP agents only if you have alreadyinstalled the SNMP Service.

ServerNet PCI Adapter Driver is the device driver for the ServerNet PCIAdapter. Sanman is the device driver that configures the ServerNet IDs.ServerNet-I VI Protocol is the virtual interface (VI) architectureemulation driver. SNMP SPA agent is for the ServerNet PCI Adapter.SNMP Switch agent is for the ServerNet Switch.

11. The Component Locations screen appears.




12. Select a product, for example, SNMP Switch Agent 1.1.1.SNMP Switch Agent 1.1.1.SNMP Switch Agent 1.1.1.SNMP Switch Agent 1.1.1. TheComponent Locations screen is refreshed to indicate the default locationfor the SNMP Switch Agent.

13. If you want to accept the default location, skip to Step 18.




NOTE: For ServerNet drivers, the default or alternate location is a working directorywhere the drivers are stored temporarily. Accept the default location unless there is notenough space in the default directory.

14. If you want to specify an alternate location and you know the exactdirectory path, specify it by typing over the default location and skip toStep 18.

15. If you want to specify an alternate location but do not know the exactdirectory path, click Change Location.Change Location.Change Location.Change Location. The Choose Directory screenappears.

16. Browse to the location where you want to install the SNMP SwitchAgent. Click OK.OK.OK.OK. The Component Location screen appears indicatingthe alternate location you specified.




17. Repeat steps 11 through 15 for each product to be installed.

18. Click NextNextNextNext to continue. The Cluster Members screen appears.




19. The Cluster Members screen indicates the host name of the installingmachine. Define the node names for the client LAN by adding the nodenames you specified in the hosts and lmhosts files for the client LAN forall other cluster nodes. Do not use special characters. Use a space toseparate the node names.

20. Click NextNextNextNext to continue. The OSD Summary screen appears.

21. If the information on the OSD Summary screen is not correct, clickPreviousPreviousPreviousPrevious to modify information in the previous screens.

When the information is correct, click NextNextNextNext to continue. The NodesConnected to ServerNet Switch screen appears.




22. If the cluster interconnect configuration does not use a ServerNetSwitch, accept the default values on this screen and skip to Step 24.

23. The Nodes Connected to ServerNet Switch screen maps cluster nodes toports on a ServerNet Switch. ServerNet Switch ports range from port 0to port 5. The screen shows the installing node mapped to port 0. If theinstalling node is not connected to port 0, type the correct port number.

Type the node names and port each is connected to on the ServerNetSwitch, for every cluster node. Use a space to separate node names andport numbers.

The ServerNet Switch ports each node is connected to must be the sameon each switch. For example, if node1 is connected to port 0 on the Xpath switch, it must also be connected to port 0 on the Y path switch.

24. Click Next.Next.Next.Next. The Oracle Summary screen appears.




25. Click InstallInstallInstallInstall to install the components listed in the summary. At thispoint, the OUI verifies that the installing node can communicate withother cluster nodes through both the client LAN and cluster interconnectnode names. If the OUI cannot communicate with all nodes, this processcan take a while.

26. An Install screen appears that shows a percentage completion bar as theinstallation progresses.

27. When the installation completes, the End of Installation screen appears.




28. If you installed any or all of the ServerNet drivers, restart every node inthe cluster.

29. Start the cluster manager, OracleCMService,OracleCMService,OracleCMService,OracleCMService, on all nodes in the cluster.This service must be started before installing Oracle software. See theOracle documentation for information on starting the cluster manager.




Verifying the ServerNet Cluster Interconnect

The viping utility verifies that cluster nodes can communicate with each otherusing the ServerNet cluster interconnect. Use this utility now, afterafterafterafter installingServerNet hardware and OSDs but beforebeforebeforebefore installing Oracle software, to isolateand correct any ServerNet connectivity problems.

Run the viping utility from each node in the cluster to verify connectivity withitself and every other node. For example, if there are four nodes in the cluster,you will run viping four times from each cluster node to test connectivity toitself and the other three nodes.

You can use the node name (machine host name) or the node’s ServerNet IDas the operand for viping. The following example shows a viping commandentered at node1 to test ServerNet connectivity with node2. It also showssample output indicating that the command was successful. (0xF0080 is anexample of a ServerNet ID.)

C: \viping node2

Pinging node2 [0xF0080] with 12 bytes of data:Reply from 0xF0080: bytes = 12 time = 12ms

If the viping command does not return successfully, make sure that theServerNet PCI Adapters are seated firmly and that all ServerNet cables aresecurely attached. Verify that ServerNet drivers are installed and runningusing the Network Control Panel. Also see Chapter 7, “Troubleshooting.”

For information about viping options and error diagnostics, see Appendix A,“viping Utility.”

Installing Oracle SoftwareUsing the Oracle8i Parallel Server Setup and Configuration Guide Release8.1.5, follow the steps to install Oracle8i Enterprise Edition release 8.1.5software on all cluster nodes, including:

■ Oracle8i Server Release 8.1.5

■ Oracle8i Parallel Server Option Release 8.1.5

■ Oracle8i Parallel Server Manager Release 8.1.5

■ Oracle8i Enterprise Manager Release 8.1.5

NOTE: If the Oracle Universal Installer does not offer the option to install Oracle8i ParallelServer, it could mean that the OSDs did not install successfully or that theOracleCMService is not running.




After installing Oracle software, the Oracle manual instructs you to installvendor-supplied operating system dependent modules (OSDs). You havealready completed this step.

Configuring Oracle SoftwareConfiguring Oracle software includes configuring Oracle8i Parallel Server,Oracle8i Parallel Server Manager, Oracle8i Enterprise Manager, andadministering multiple instances. Use the Oracle8i Parallel Server Setup andConfiguration Guide Release 8.1.5 and the Oracle8i Enterprise Edition forWindows NT and Windows 95/98 Release Notes, Release 8.1.5 to configurethe Oracle software.

Additional Notes on Configuring Oracle Software

In addition to the steps outlined in the Oracle documents, you must create asymbolic link for OPS_CMDISK. Verifying access to shared storage from allnodes is recommended.

Creating a Symbolic Link for OPS_CMDISK

The OPS_CMDISK file contains status information for all nodes in the cluster.Using the SETLINKS utility, you must create a symbolic link from theOPS_CMDISK file to a disk partition. Do this by editing the ORALINKn.TBLfiles on the primary node and then run the SETLINKS utility on each node inthe cluster.

Verifying Access to Shared Storage from AllNodes

IMPORTANT: Verify access to shared storage from all nodes in the cluster beforestarting any Oracle instance.

Open Disk Administrator to verify that the same shared disk resources are seenfrom this node as they are seen from other installed nodes in the cluster.Restart all running nodes.




Installing Object Link Manager

IMPORTANT: After installing and configuring Oracle software, you must install ObjectLink Manager on all cluster nodes. This is required to maintain the symbolic links betweendisk partitions and Oracle data files. Failure to install Object Link Manager can result inthe Oracle database not starting or, if it starts, in not finding the correct data.

Windows NT Server assigns disk numbers to drives in the shared storagesubsystems based on the order in which the shared storage subsystems arepowered on. Through the symbolic links created by SETLINKS, Oracle8iServer uses the order in which disks are brought online to determine thelocation of its database files.

Therefore, the order in which disks are brought online directly affects the disknumbers assigned to the disk, which in turn affects the ability of Oracle8iServer to find Oracle data files. If shared storage subsystems are powered offand then powered on in a different order than they were when the cluster wasinitially configured, the order and numbering of the shared disk drives willchange for all nodes in the cluster.

To avoid this problem, install Object Link Manager. Object Link Managersimplifies the creation and maintenance of symbolic links between diskpartitions and Oracle data files by placing the symbolic link names directlyinto the disk partitions, thereby tracking the symbolic links dynamically. Thismeans that if the order in which disk drives are brought online changes,Oracle8i Server can find the correct data files and can properly start the Oracledatabase.

You must install Object Link Manager afterafterafterafter performing the initial Oracleinstallation and configuration tasks using SETLINKS, as instructed previouslyin this chapter.

For instructions on installing and using Object Link Manager, see the readmefile on the Compaq Parallel Database Cluster for Oracle8i Release 8.1.5Ethernet Clustering Software CD or Compaq Parallel Database Cluster forOracle8i Release 8.1.5 ServerNet Clustering Software CD at:





Verifying the Hardware and SoftwareInstallation

Cluster Communications

Use the ping utility to verify that each node in the cluster can communicatewith every other node over the client LAN. Run the ping utility from theinstalling cluster node. Verify the installing node can communicate with allcluster nodes by pinging the client LAN name. The IP address displayed bythe ping utility should be the client LAN IP address.

If you are using an Ethernet cluster interconnect, ping the Ethernet clusterinterconnect name. The IP address displayed by the ping utility should be theEthernet cluster interconnect address. If this is not the case, check the entriesin the hosts and lmhosts files at %SystemRoot%\system32\drivers\etc.

If you are using a ServerNet cluster interconnect, use the viping utility toverify ServerNet connectivity. See Appendix A, “viping Utility.”

Access to Shared Storage from All Nodes

Open Disk Administrator to verify that the same shared disk resources are seenfrom this node as they are from other installed nodes in the cluster.

Make sure you can see the same shared disk resources using the RedundancyManager as you can using the Disk Administrator. See “Verifying SharedStorage Using Redundancy Manager” earlier in this chapter.

OSDs

After verifying all the nodes have access to the shared storage, start an Oracleinstance. For example, at a C: command prompt start an instance by entering:

net start oracleservice<SID>

If the OSDs are installed correctly, a message will appear indicating that theOracle service has started successfully.




If the Oracle service did not start successfully, make sure that theOracleCMService is started and the SID is in the Oracle registry. There aretwo logs that can provide information about why the Oracle service did notstart. These logs are:

■ \Compaq\OPS\cmsrvr.log

■ \Compaq\OPS\nmsrvr.log

Power Distribution and PowerSequencing Guidelines

It is recommended you connect most cluster components to a Compaq powerdistribution unit (PDU). PDUs connect to an uninterruptible power supply(UPS) or building power. The PDUs and UPSs are the only clustercomponents connected to a building power source. If there is no UPS, thePDU plugs into building power.

It is also recommended that you connect cluster components, such as servers,storage subsystems, switches, and hubs, to two PDUs and UPSs so the clustercomponents continue to operate if one PDU or UPS fails.

Server Power Distribution

Figure 6-8 shows an example of server power distribution for a three-nodecluster. Power supply #1 of each server is connected to PDU #1. Power supply#2 of each server is connected to PDU #2.

PDU #1 is connected to UPS #1, and PDU #2 is connected to UPS #2. EachUPS is connected to building power.




BuildingPower

UPSPDUProliant Servers

PowerSupplies

Figure 6-8. Server power distribution in a three-node cluster

Having two PDUs and UPSs in the cluster provides two paths from the serversto building power and means that the cluster stays up and running if a PDU orUPS fails in one of the paths.

RA4000 Array Power Distribution

An RA4000 Array can have one or two power supplies. If each RA4000 Arrayin the cluster has two power supplies, the power distribution (with respect toconnecting them to PDUs and UPSs) can be configured similarly to theexample shown for server power distribution. By substituting RA4000 Arraysfor ProLiant servers in Figure 6-8, you can provide two redundant paths fromthe RA4000 Arrays to building power.

If each RA4000 Array has one power supply, connect the RA4000 Array to aPDU, connect the PDU to a UPS, and then connect the UPS to building power.If there is no UPS, connect the PDU directly to building power.




Power Sequencing

Be sure to power up the cluster components in the following order:

1. RA4000 Arrays


3. Ethernet hubs/switches or ServerNet switches

4. ProLiant servers

Be sure to power down the cluster components in the following order:

1. ProLiant servers

2. Ethernet hubs/switches or ServerNet switches


4. RA4000 Arrays

Shutting down and powering off the servers first allows them to perform taskssuch as flushing queued database write transactions to disk and properlyterminating running processes.


Part Number: 161397-001 File Name: h-ch7 Cluster Management.doc Last Saved On: 10/29/99 5:31 PM

Chapter 7Cluster Management

Throughout the life of your cluster, you might need to improve itsperformance, upgrade hardware components, upgrade software, increasestorage capacity, and monitor ongoing activities. This chapter describes thesemanagement activities for the Compaq Parallel Database Cluster ModelPDC/O2000 (PDC/O2000). The topics addressed in this chapter include:

■ Cluster management concepts

■ Management applications

■ Software maintenance

■ Managing changes to shared storage

■ Replacing a cluster node

■ Adding a cluster node

■ Managing cluster operation

NOTE: The procedures in this chapter contain high-level instructions. The instructionssummarize what has either already been discussed in this guide, Oracle documentation,or storage subsystem documentation.




Cluster Management Concepts

Powering Off a Node Without InterruptingCluster Services

At some time during the life of your cluster you will need to perform anoperation on a cluster node that will require it to be powered off. Physicallymoving the cluster node, removing a hardware device, and adding a hardwaredevice commonly require the node to be powered off.

It is a good practice to gracefully shut down the node before powering it off.The process of gracefully shutting down a node in a PDC/O2000 can besummarized as:

1. Make sure that clients connecting to the database through the node canreconnect to the database through one of the other nodes.

2. Properly shutdown the Oracle instance running on the node.

3. Shut down Windows NT Server on the node.

The node can now be powered off safely. The database will remain accessiblesince the remaining cluster nodes are operating normally.

Managing a Cluster in a Degraded Condition

Due to the high availability benefits of clustering, applications and networkclients remain operational even while some cluster components do not. Whenthe cluster enters a degraded condition, it is helpful to follow thistroubleshooting process:

1. Determine what caused the degradation.

2. Determine whether the condition affects one cluster node, multiplecluster nodes, or all cluster nodes.

If one or more nodes is unaffected by the condition, and if enoughperformance can be obtained from the unaffected nodes, continueoperating the database.

3. Determine whether the condition will continue to worsen.

Cluster Management 7-3



4. Determine how critical it is to repair the problem.

a. If the problem is not considered to be critical, wait until a non-peaktime to service the problem.

b. If the problem is critical but does not affect all cluster nodes, shutdown the Oracle instances on the affected nodes and wait until anon-peak time to service the problem.

c. If the problem is critical and affects all the cluster nodes, shut downthe database instances on all cluster nodes and correct the problem.

Managing Network Clients Connected to a Cluster

An important aspect of managing network clients is informing users that theirapplications are now running on a cluster. As the cluster is initially broughtinto production, it can be helpful to describe in a memorandum the effects acluster will have on user ability to access the database.

Since users will experience some disruption of service and possibly aperformance degradation during a node eviction, they might becomeconcerned about the availability and stability of their applications.

When a node eviction or node integration occurs, the users will notice that fora brief moment they cannot access their database. When users have beenproperly forewarned of the effects of operating in a clustered environment,they will more readily recognize when such an event is occurring. Most userswill then know to wait several seconds before attempting to reaccess theirdatabase.

Cluster Events

The majority of cluster events can be viewed either within the Windows NTServer Event Log or within specific error log files. The Oracle8 Server orOracle8i Server events are sent to the Application Event Log and can beviewed from the Windows NT Server Event Viewer.

The lower-level software components, such as the cluster manager, sendevents to error log files.

NOTE: If the OSDs are not installed in the default directory, the log files will be in thesame directory as the OSDs.




Log Files for Oracle8 Server Release 8.0.5

■ The Group Membership Service (PGMS) logs messages to%SystemRoot%\System32\pgms.log

■ By default, the cmserver process sends messages toC:\compaq\ops\error.log

Log Files for Oracle8i Server Release 8.1.5

■ The Node Manager (NM), provided by Compaq, logs messages toC:\compaq\ops\nmsrvr.log

■ By default the OracleCMService sends messages toC:\compaq\ops\cmsrvr.log

Management Applications

Monitoring Server and Network Hardware

Compaq Insight Manager is used to manage the hardware components of yourcluster. Compaq Insight Manager, loaded from the Compaq Management CD,is an easy-to-use software utility for collecting server, storage, and networkinformation. Compaq Insight Manager performs the following functions:

■ Monitors fault conditions and system status

■ Monitors shared storage and interconnect adapters

■ Forwards server alert fault conditions

■ Remotely controls servers

In Compaq servers, each hardware subsystem, such as network adapters,system processors, and system memory, has a robust set of managementcapabilities. Compaq Full Spectrum Fault Management notifies of impendingfault conditions.

Several steps are required to configure the system for Insight Manager. Forexample you must:

■ Load Insight agents on each cluster node.

■ Install the Compaq Insight Manager Console.

■ Make sure that proper user rights are granted.




For additional information concerning Compaq Insight Manager, see theCompaq Server Setup and Management pack.

Compaq Insight Manager XE is a web-based management system that can alsobe used to monitor cluster hardware components. Compaq Insight ManagerXE is an optional CD available upon request from the Compaq SystemManagement website at

www.compaq.com/sysmanage

Managing Shared Drives

There are two levels of management with shared drives. The physical disksand the drive arrays created from them are managed with the ArrayConfiguration Utility (ACU). The logical drives are created and managed withthe Windows NT Disk Administrator.

The ACU is first run during initial configuration of the shared storagesubsystem and can be run any time afterward to view existing drive arrays,modify existing drive arrays, or add new drive arrays. The ACU is availableon the SmartStart and Support Software CD, which is found in the CompaqServer Setup and Management pack. For additional information about theACU see the Compaq StorageWorks RAID Array 4000 User Guide.

Disk Administrator is first run after the drive arrays have been created by theACU. Disk Administrator is used to partition the drive arrays, create logicaldrives, and assign volume labels. For additional information about DiskAdministrator see your Windows NT Server documentation.

Monitoring Redundant Fibre Channel ArbitratedLoops

Fibre Channel Arbitrated Loops (FC-ALs) contain Compaq StorageWorksFibre Channel Host Bus Adapters (Fibre Host Adapters), Fibre Channelcables, Compaq StorageWorks Fibre Channel Storage Hubs (Storage Hubs),and Compaq StorageWorks RAID Array 4000 Array Controllers (RA4000Array Controllers) in Compaq StorageWorks RAID Array 4000s (RA4000Arrays).

The Fibre Channel Fault Isolation Utility (FFIU) can verify the integrity of anewly installed or existing FC-AL installation. This utility provides faultdetection and help in locating a failing device on an FC-AL. FFIU is on theSmartStart and Support Software CD.




Compaq Redundancy Manager can detect failures in redundant FC-ALcomponents. When such a failure occurs on an active FC-AL path,Redundancy Manager reroutes I/O through a redundant path, allowingapplications to continue processing.

Compaq Redundancy Manager (Fibre Channel) CD is included in the CompaqParallel Database Model PDC/O2000 kit.

Monitoring the Database

Oracle8 Enterprise Manager and Oracle8i Enterprise Manager use agents tocontinuously monitor database activities and offer a graphical consoleallowing administrators to manage the database. From the Enterprise ManagerConsole, the administrator can administer, diagnose, and tune multipledatabases. Jobs can also be scheduled, software can be distributed, and objectsand events can be monitored from the console.

Several steps are required to properly configure the system for Oracle8Enterprise Manager or Oracle8i Enterprise Manager. Additional steps arerequired to integrate the management of parallel databases. For example, youmust do the following:

■ Create an Enterprise Manager Repository on the console node.

■ Obtain sufficient rights on the console machine as well as on themanaged cluster nodes.

■ Install Oracle agents on each managed node.

Be sure to read through the Oracle documentation. For Oracle8 EnterpriseManager, see:

■ Oracle8 Enterprise Manager Administrator’s Guide

■ Oracle8 Parallel Server Management User’s Guide

Oracle8 Enterprise Manager and its corresponding documentation are shippedas part of Oracle8 Enterprise Edition.

For Oracle8i Enterprise Manager, see:

■ Oracle8i Enterprise Manager Administrator’s Guide

■ Oracle8i Parallel Server Management User’s Guide

Oracle8i Enterprise Manager and its corresponding documentation are shippedas part of Oracle8i Enterprise Edition.




Remotely Managing a Cluster

Oracle8 Enterprise Manager, Oracle8i Enterprise Manager, Compaq InsightManager, and Compaq Insight Manager XE can be run from any networkclient machine that has network access to the cluster nodes. They can operateon Windows NT Server or Windows 95.

Before installing Compaq Insight Manager, Compaq Insight Manager XE,Oracle8 Enterprise Manager, or Oracle8i Enterprise Manager, it isrecommended that you read through the corresponding documentation todetermine how to set up and configure each of these programs to run remotely.

Software Maintenance for Oracle8Release 8.0.5

Deinstalling the OSDs

At some point, you might need to deinstall the operating system dependentmodules (OSDs) from a node. Some example situations are:

■ When a node is being permanently removed from the cluster

■ When a node is being replaced

■ When a node is being added to the cluster

■ When the OSDs are being upgraded to a newer revision level

To deinstall the OSDs:

1. Shutdown the Oracle instance on a cluster node.

2. Stop the Oracle services running on the cluster node.

3. Remove Compaq OSD Modules for Oracle Parallel ServerCompaq OSD Modules for Oracle Parallel ServerCompaq OSD Modules for Oracle Parallel ServerCompaq OSD Modules for Oracle Parallel Server from thecluster node using the Add/Remove Programs control panel applet.

4. Repeat steps 1 through 3 on all remaining cluster nodes.




Upgrading Oracle8 Server

The design of the PDC/O2000 is tightly integrated with Oracle8 Server.Significant changes in Oracle8 Server will likely affect the operation of thecluster.

Before upgrading to any new release of Oracle8 Server, consult your Compaqand Oracle service representatives. Make sure that the new release is certifiedand supported on the PDC/O2000, and get assistance to determine whatprocedure you should follow to perform the upgrade properly.

Software Maintenance for Oracle8iRelease 8.1.5

Deinstalling the OSDs

At some point, you might want to deinstall the operating system dependentmodules (OSDs). Some example situations are:

■ When a node is being permanently removed from the cluster

■ When a node is being replaced

■ When a node is being added to the cluster

■ When the OSDs are being upgraded to a newer revision level

Use Oracle Universal Installer to deinstall the OSDs and device drivers.

IMPORTANT: The Oracle Universal Installer removes the OSDs from all cluster nodes. Toproperly maintain the cluster, the Oracle Universal Installer cannot remove the OSDs fromindividual nodes. After the OSDs are deinstalled, the cluster is disbanded and cannot rununtil the OSDs are reinstalled on all the nodes.

Be sure to shut down the Oracle instance on all nodes and stop the Oracleservices running on all nodes prior to removing the OSDs.




To deinstall the OSDs:

1. Log in as a user who has administrator privileges on all nodes in thecluster.

2. Insert the Compaq Parallel Database Cluster for Oracle8i Parallel ServerRelease 8.1.5 Ethernet Clustering Software CD or the Compaq ParallelDatabase Cluster for Oracle8i Parallel Server Release 8.1.5 ServerNetClustering Software CD into the CD-ROM drive of the primary node.This is the node where the OSDs were installed originally.

IMPORTANT: To deinstall the OSDs, you must run the Oracle Universal Installer from thesame node where the OSDs were installed.


4. If autorun is disabled, enter d:\setupd:\setupd:\setupd:\setup at a C: prompt. Substitute the letterassigned to the CD drive for d.d.d.d. The Oracle Universal Installer Welcomescreen appears.




5. Click Deinstall Products.Deinstall Products.Deinstall Products.Deinstall Products. Next, a window containing an inventory of theinstalled components appears. The following shows the Inventory screenif ServerNet is used for the interconnect. A different Inventory screendisplays if Ethernet is used for the interconnect.

6. Checkmark the items you want to remove and click Remove.

NOTE: This screen might indicate later revision levels for the drivers.

7. Follow the Oracle Universal Installer directions to complete thedeinstallation.

IMPORTANT: Although not noted in the OUI, you must reboot all cluster nodes tocomplete the deinstallation process.




Deinstalling a Partial OSD Installation

While not a common occurrence, the OUI might not be able to complete theinstallation of the Compaq Ethernet or ServerNet OSDs. An incomplete OSDinstallation could result for various reasons, for example:

■ A cable is no longer seated tightly into its connector.

■ The node from which you are running the OUI cannot communicatewith all other nodes in the cluster.

■ The user running the OUI does not have administrator privileges on allcluster nodes.

An incomplete installation can result in a partial installation of the OSDs onthe cluster. In this case, you must clean up after the partial installation (byremoving it) before you can successfully run the OUI again.

To remove a partial OSD installation:

1. Insert the Compaq Parallel Database Cluster for Oracle8i Parallel ServerRelease 8.1.5 Ethernet Clustering Software CD or the Compaq ParallelDatabase Cluster for Oracle8i Parallel Server Release 8.1.5 ServerNetClustering Software CD into the CD-ROM drive of a cluster node.

2. From a command prompt, navigate to the CD-ROM drive.

3. Connect to the \Utilities\Utilities\Utilities\Utilities directory.

4. Run the uninstall utility by entering:

uninstallOSDuninstallOSDuninstallOSDuninstallOSD



Upgrading Oracle8i Server

The design of the PDC/O2000 is tightly integrated with Oracle8i Server.Significant changes in Oracle8i Server will likely affect the operation of thecluster.

Before upgrading to any new release of Oracle8i Server, consult your Compaqand Oracle service representatives. Make sure that the new release is certifiedand supported on the PDC/O2000, and get assistance to determine whichprocedure you should follow to perform the upgrade properly.




Managing Changes to Shared Storage

Replacing a Failed Disk

At some point you might need to replace a failed drive in an RA4000 Array. Itis assumed you are employing RAID levels 0, 1, 0+1, 4, or 5 for all devices inthe storage array.

IMPORTANT: If the failed drive is not part of a fault-tolerant drive array you can losesome or all of the data on the failed drive.

With RAID employed, the effects of replacing a drive are felt only by thestorage subsystem. The operating system and Oracle are unaware of theactivity.

The removal of a drive in an RA4000 Array must follow certain rules are thatinterpreted by reading the LEDs on the disk drive. Before proceeding, locatethe information in the Compaq StorageWorks RAID Array 4000 User Guidethat describes the conditions under which a drive can be removed. Once youhave determined it is safe to remove the drive, open both latches on the drivetray and remove the drive from the drive bay. Put the replacement drive intothe drive bay and snap the latches in place.

CAUTION: Failure to follow the rules described in the Compaq StorageWorksRAID Array 4000 User Guide can result in loss of data.

Adding Disk Drives to Increase Storage Capacity

During the life of your PDC/O2000 cluster you might need to expand thecapacity of an RA4000 Array. The following steps describe how to add drivesto an RA4000 Array and how to allocate the added disk capacity to Oracle.

The drive can be added and allocated to Windows NT Server while thedatabase is online. The following summarizes the procedure.

1. Physically add the drives to the RA4000 Array.

2. On the primary node, run the Array Configuration Utility to create drivearrays, configured with RAID, and then create logical drives.

3. Using Disk Administrator, create an extended partition on each of thelogical drives.

4. Using Disk Administrator, create logical partitions within each extendedpartition.




IMPORTANT: Do not format the drives. Oracle Parallel Server uses raw partitions, whichrequires that the drive not be formatted with any file system.

5. Within each extended partition, create a small logical partition as thefirst logical partition. Format it with NTFS and label it. (This partition isused to identify the extended partition, not to store Oracle data.)

6. Verify that the same shared disk resources are seen from every node inthe cluster by running Disk Administrator on each node.

7. Make sure you can see the same shared disk resources from each nodeusing Compaq Redundancy Manager as you can using the DiskAdministrator.

8. Perform the necessary Oracle commands to associate the new data filesto the database.

If you are running Oracle8 software, see the Oracle8 Enterprise EditionGetting Started Release 8.0.5 for Windows NT manual for more information.If you are running Oracle8i software, see the Oracle8i Parallel Server Setupand Configuration Guide Release 8.1.5 for instructions.

Adding an RA4000 Array

You must shut down the cluster to add an RA4000 Array. The following stepssummarize the procedure.

1. Shutdown the Oracle instance on each cluster node.

2. Stop the Oracle services running on each node.

3. Shut down Windows NT Server on each node and power off all thecluster nodes.

4. Power off all of the RA4000 Arrays.

5. Insert drives into the added RA4000 Array.

6. Using the Fibre Channel cables, physically attach the RA4000 ArrayControllers to the Storage Hubs.

7. Connect the RA4000 Arrays to a power source and restart them.

8. Restart the primary cluster node.

9. On the primary node, run the Array Configuration Utility to create drivearrays, configured with RAID, and then create logical drives.




10. Using Disk Administrator, create an extended partition on each of thelogical drives.

11. Using Disk Administrator, create logical partitions within each extendedpartition.

IMPORTANT: Do not format the drives. Oracle Parallel Server uses raw partitions, whichrequires that the drive not be formatted with any file system.

12. Within each extended partition, crate a small logical partition as the firstlogical partition. Format it with NTFS and label it. (This partition isused to identify the extended partition, not to store Oracle data.)

13. Power on the other cluster nodes.

14. Verify that the same shared disk resources are seen from every node inthe cluster by running Disk Administrator on each node.

15. Make sure you can see the same shared disk resources from each nodeusing Compaq Redundancy Manager as you can using the DiskAdministrator.

16. Define the active array controller in the newly-added RA4000 Array.

17. Perform the necessary Oracle commands to associate the new data filesto the database.

If you are running Oracle8 software, see the Oracle8 Enterprise EditionGetting Started Release 8.0.5 for Windows NT manual for more information.If you are running Oracle8i software, see the Oracle8i Parallel Server Setupand Configuration Guide Release 8.1.5 for instructions.




Replacing a Cluster NodeBefore replacing a node, verify that the new configuration is supported by thePDC/O2000. To determine if your new server is supported, see the CompaqParallel Database Cluster Model PDC/O2000 for Oracle8 Parallel ServerRelease 8.0.5 Certification Matrix or the Compaq Parallel Database ClusterModel PDC/O2000 for Oracle8i Parallel Server Release 8.1.5 CertificationMatrix at


IMPORTANT: The servers in the cluster must be set up identically. For example, thismeans using one model server for all nodes. The cluster components common to allnodes in the cluster must be identical; for example, the cluster interconnect adapters,amount of memory, cache, and number of CPUs must be the same in each cluster node.

Replacing a node consists of removing it logically from the cluster, removingit physically from the cluster, then adding a new server to take its place.

Removing the Node 1. Record pertinent information that will be needed when configuring thenew node. Such information includes:

� The IP address and network name of the client LAN adapters.

� If using an Ethernet cluster interconnect, the IP address and networkname of the Ethernet cluster interconnect adapters.

� If using a ServerNet cluster interconnect, the ports in each ServerNetSwitch to which each ServerNet PCI Adapter is connected.

2. If the new node will use applications or data that is stored locally on thenode being replaced, back up that data to a removable storage mediumor to a remote file server that can be restored onto the replacement node.

3. Stop the Oracle instance on the node to be removed.

4. Stop all Oracle services on the node to be removed.

5. Shutdown Windows NT Server on that node and power it off.

6. Disconnect the Fibre Channel cables between the evicted node and theStorage Hubs.

7. Disconnect the cluster interconnect cables from the evicted node andlabel them as the cluster interconnect cables.




8. Disconnect the client LAN cables from the evicted node and label themas the client LAN cables.

9. If adapters from the evicted node will be used in the new node, move theadapters to the new node.

Adding the Replacement Node

Adding the replacement node involves the following activities:

■ Preparing the replacement node

■ Installing the cluster software

These subsections contain a high-level outline of the steps to follow. If youneed detailed instructions you should read through the appropriate chapter:

■ Chapter 5, “Installation and Configuration for Oracle8 Release 8.0.5,” or

■ Chapter 6, “Installation and Configuration for Oracle8i Release 8.1.5”

Preparing the Replacement Node

Several steps should be performed on the new server before it is integratedinto the cluster.

Prior to integrating the server into the cluster, perform the following steps:

1. Connect the server to the Storage Hubs, cluster interconnect, and to theclient LAN. Power on the new node. Although physically connected tothe cluster, the server is not yet integrated into the cluster.

2. Start the replacement node with the Compaq SmartStart and SupportSoftware CD in the CD-ROM drive.

3. Use the Compaq System Configuration Utility to configure the hardwaresettings of the node and its adapters.

4. Use the Compaq Array Configuration Utility to configure the server’slocal non-shared disks with RAID.

5. Install and configure Windows NT Server 4.0 with Service Pack 3, 4, or5, whichever is installed on the other cluster nodes.

6. Install necessary Compaq drivers and utilities from the CompaqSmartStart and Support Software CD.

7. Enter unique IP addresses and node names for each node in the hostsand lmhosts files located at %SystemRoot%\system32\drivers\etc.




8. Install Compaq Redundancy Manager.

9. Restart the replacement node.

10. Verify TCP/IP connectivity between the replacement node and theexisting cluster nodes. Run the ping utility from the replacement node,pinging the client LAN adapters on all other nodes. If Ethernet is usedas the cluster interconnect, also ping the cluster interconnect adapters ofall other nodes.

11. Verify the replacement node can access the shared storage. From thereplacement node, start Disk Administrator to verify that the sameshared disk resources are seen from this node as they are seen fromother installed nodes in the cluster. Also make sure you can see the sameshared disk resources from the replacement node using CompaqRedundancy Manager as you can using Disk Administrator.

Installing the Cluster Software for Oracle8Release 8.0.5

Now that the added node is configured and physically connected to the cluster,the next procedure is to integrate the node into the cluster. This involvesinstalling the low-level cluster management software and the OSDs. It alsoinvolves installing the application-level cluster software, including Oracle8Server with the Oracle Parallel Server Option.

To install the cluster software, perform the following steps:

1. Install Oracle8 Server with the Oracle8 Parallel Server Option on thereplacement node.

2. Install the OSDs on the replacement node.

3. Run the NodeList Configurator on one of the existing cluster nodes, notthe replacement node. Use the same computer name, client LAN nameand IP address, and cluster Interconnect name and IP address for thereplacement node as you did for the original node.

4. Install Object Link Manager on the replacement node.

5. Configure the Oracle software. See the Oracle8 Enterprise EditionGetting Started Release 8.0.5 for Windows NT manual for informationon configuring Oracle software.

6. Start the Oracle services and Oracle instance on the replacement node.




Installing the Cluster Software for Oracle8iRelease 8.1.5

Now that the added node is configured and physically connected to the cluster,the next procedure is to integrate the node into the cluster. This involvesinstalling the low-level cluster management software and the OSDs. It alsoinvolves installing the application-level cluster software, including Oracle8iServer with the Oracle Parallel Server Option.

The cluster must be brought offline to install the cluster software. Because thedatabase will be unavailable while configuring the Oracle software, it isrecommended that this procedure be performed during non-peak work hours.


1. To add a node to the cluster, you must disassemble the existing clusterand then reform it to include the replacement node. Do the following onall original cluster nodes (not the replacement node):

a. Shut down the Oracle instance.

b. Stop all Oracle services.

c. Deinstall the OSDs.

d. Restart the node to complete the deinstallation process.

2. Reinstall the OSDs on each cluster node, including the original nodesand the replacement node.

3. Restart each cluster node to complete the OSD installation process.

4. If ServerNet is used as the cluster interconnect, verify connectivitybetween the cluster nodes by running the viping utility from thereplacement node and pinging the cluster interconnect names of theother nodes.

5. Install Oracle8i Server with the Oracle8i Parallel Server Option on thereplacement node.

6. Restart the replacement node which, upon restart, is now fully integratedinto the cluster.

7. Configure the Oracle software. See the Oracle8i Parallel Server Setupand Configuration Guide Release 8.1.5 manual for information onconfiguring Oracle software.

8. Install Object Link Manager on the replacement node.

9. Start the Oracle services and Oracle instance on each cluster node.




Adding a Cluster NodeBefore adding a node, verify the new configuration is supported by thePDC/O2000. To determine if your new server is supported, see the CompaqParallel Database Cluster Model PDC/O2000 for Oracle8 Parallel ServerRelease 8.0.5 Certification Matrix or the Compaq Parallel Database ClusterModel PDC/O2000 for Oracle8i Parallel Server Release 8.1.5 CertificationMatrix at


IMPORTANT: The servers in the cluster must be set up identically. For example, thismeans using one model server for all nodes. The cluster components common to allnodes in the cluster must be identical, for example, the cluster interconnect adapters,amount of memory, cache, and number of CPUs must be the same in each cluster node.

During the life of the PDC/O2000, you might need to add a new node to thecluster. A desire to increase performance or throughput can drive this decision.Adding another node also can increase overall availability of the system.

This can be a time-consuming and complex procedure. Be sure to read throughall of the following directions as well as those in the Oracle documentationbefore starting.

Adding a new node involves the following activities:

■ Preparing the new node

■ Preparing the existing cluster nodes

■ Installing the cluster software

These subsections contain a high-level outline of the steps to follow. If youneed detailed instructions you should read through the appropriate chapter:

■ Chapter 5, “Installation and Configuration for Oracle8 Release 8.0.5,” or

■ Chapter 6, “Installation and Configuration for Oracle8i Release 8.1.5”




Preparing the New Node

To minimize downtime of the cluster, several preparation steps should beperformed on the new server before it is integrated into the cluster.

Prior to integrating the server into the cluster, perform the following steps:

1. Connect the server to the Storage Hubs, cluster interconnect, and to theclient LAN. Power on the new node. Although physically connected tothe cluster, the server is not yet integrated into the cluster.

2. Start the new node with the Compaq SmartStart and Support SoftwareCD in the CD-ROM drive.

3. Use the Compaq System Configuration Utility to configure the hardwaresettings of the node and its adapters.

4. Use the Compaq Array Configuration Utility to configure the server’slocal non-shared disks with RAID.

5. Install and configure Windows NT Server 4.0 with Service Pack 3, 4, or5, whichever is installed on the other cluster nodes.

6. Install necessary Compaq drivers and utilities from the CompaqSmartStart and Support Software CD.

7. Enter unique IP addresses and node names for each node in the hostsand lmhosts files located at %SystemRoot%\system32\drivers\etc.

8. Install Compaq Redundancy Manager on the new node.

9. Restart the new node.

10. Verify TCP/IP connectivity between the replacement node and theexisting cluster nodes. Run the ping utility from the replacement node,pinging the client LAN adapters on all other nodes. If Ethernet is usedas the cluster interconnect, also ping the cluster interconnect adapters ofall other nodes.

11. Verify the replacement node can access the shared storage. From thereplacement node, start Disk Administrator to verify that the sameshared disk resources are seen from this node as they are seen fromother installed nodes in the cluster. Also make sure you can see the sameshared disk resources from the replacement node using CompaqRedundancy Manager as you can using Disk Administrator.




Preparing the Existing Cluster Nodes

To prepare the existing cluster nodes for adding a new node:

1. Add the unique IP addresses and node names for the new node in thehosts and lmhosts files of each existing cluster node. These files arelocated at %SystemRoot%\system32\drivers\etc.

2. Verify TCP/IP connectivity between the replacement node and theexisting cluster nodes. Run the ping utility from the replacement node,pinging the client LAN adapters on all other nodes. If Ethernet is usedas the cluster interconnect, ping the cluster interconnect adapters of allother nodes.

Installing the Cluster Software for Oracle8 Release8.0.5

Now that the added node is configured and physically connected to the cluster,the next procedure is to integrate the node into the cluster. This involvesinstalling the low-level cluster management software and the OSDs. It alsoinvolves installing the application-level cluster software, including Oracle8Server or Oracle8i Server with the Oracle Parallel Server Option.



1. To add a node to the cluster, you must disassemble the existing clusterand then reform it to include the replacement node. Do the following onall original cluster nodes (not the new node):



2. Install Oracle8 Server with the Oracle8 Parallel Server Option on thenew node.

3. Install the OSDs on the new node.

4. Run the NodeList Configurator on one of the existing cluster nodes, notthe new node.

5. Install Object Link Manager on the new node.




6. Configure the Oracle software. See the Oracle8 Enterprise EditionGetting Started Release 8.0.5 for Windows NT manual for informationon configuring Oracle software.

NOTE: The addition of an Oracle instance that runs on the new node requires the creationof two additional database log files and additional rollback segments to the database.

7. Start the Oracle services and Oracle instance on all cluster nodes.

Installing the Cluster Software for Oracle8i Release8.1.5

Now that the added node is configured and physically connected to the cluster,the next procedure is to integrate the node into the cluster. This involvesinstalling the low-level cluster management software and the OSDs. It alsoinvolves installing the application-level cluster software, including Oracle8iServer with the Oracle Parallel Server Option.



1. To add a node to the cluster, you must disassemble the existing clusterand then reform it to include the replacement node. Do the following onall original cluster nodes (not the replacement node):



c. Deinstall the OSDs.

d. Restart the node to complete the deinstallation process.

2. Reinstall the OSDs on each cluster node, including the original nodesand the new node.

3. Restart each cluster node to complete the OSD installation process.

4. If ServerNet is used as the cluster interconnect, verify connectivitybetween the cluster nodes by running the viping utility from thereplacement node and pinging the cluster interconnect names of theother nodes.

5. Install Oracle8i Server with the Oracle8i Parallel Server Option on thenew node.




6. Restart the new node which, upon restart, is now fully integrated intothe cluster.

7. Configure the Oracle software. See the Oracle8i Parallel Server Setupand Configuration Guide Release 8.1.5 manual for information onconfiguring Oracle software.

NOTE: The addition of an Oracle instance that runs on the new node requires the creationof two additional database log files and additional rollback segments to the database.

8. Install Object Link Manager on the new node.

9. Start the Oracle services and Oracle instance on each cluster node.

Monitoring Cluster Operation

Tools Overview

Several tools exist to monitor the operation of your cluster. Use the tools toregularly monitor the operation of each node and each Oracle instance. Thegathered information might identify a disparity in the workload beingperformed by each node, which signifies a need to adjust the distribution of theworkload so that the overall cluster performance is maximized.

■ Compaq Insight Manager and Compaq Insight Manager XE can be usedto gather low-level, in-depth hardware statistics for the server andnetworking hardware components used in the cluster.

■ Windows NT Performance Monitor gathers performance characteristicsof many objects. Examples of monitored objects are memory, processor,hard drive performance, network software, and application software.

■ Oracle8 Performance Manager and Oracle8i Performance Managercapture, compute, and present performance data about the Oracledatabase. Specific data about Oracle Parallel Server metrics aremonitored and viewed with this application.

■ Compaq Redundancy Manager monitors the hardware components onthe redundant FC-AL paths. If a failure occurs on an active FC-AL path,Redundancy Manager reroutes I/O through a redundant FC-AL path.




Using Compaq Redundancy Manager

Compaq Redundancy Manager is a GUI-based monitoring tool for redundantFC-AL paths. It is not a real-time management tool; the Refresh and Rescanfunctions need to be executed to update the Redundancy Manager GUI toreflect FC-AL path changes after a path failover and to detect new host busadapters and array controllers.

You can use Redundancy Manager to view and to make redundant FC-ALpath changes. For example, if Redundancy Manager detects an FC-ALcomponent failure on an active path, it fails over to the standby path in thatredundant FC-AL. The standby path then becomes the active path. Afterreplacing the failed component, you might want to change the paths in thatredundant FC-AL so that the original active path is once again the active path.

IMPORTANT: If there has been an FC-AL component failure on an active path andsubsequent failover to a standby path, verify the component has been replaced and isfunctioning normally before returning the FC-AL paths to their original configuration.

Changing FC-AL Paths


1. On any cluster node, click StartStartStartStart, Programs,Programs,Programs,Programs, and Compaq RedundancyCompaq RedundancyCompaq RedundancyCompaq RedundancyManager.Manager.Manager.Manager. The Compaq Redundancy Manager (Fibre Channel) screen isdisplayed.




IMPORTANT: Wait at least 10 to 30 seconds after changing an FC-AL path in oneRA4000 Array before changing an FC-AL path in another RA4000 Array.




The following screen shows the bottom array controller in each RA4000Array as the controller on the active FC-AL path.




Using the Refresh Function

The Refresh function updates information on the Redundancy Manager GUIscreen, checks for path failures and path changes, and displays the currentconfiguration. The GUI will not update automatically. Use the Refreshfunction to update the Redundancy Manager screen to see the currentconfiguration or to see if a failure has occurred on an FC-AL path.

Select the Refresh function from the Feature menu or press F5.




The following screen shows an FC-AL path in italicized text on Disk 1 (F:) ofRA4000 Array D829HUQ1007. This means there was a failure on that FC-ALpath.

Using the Rescan Function

The Rescan function discovers hardware changes in the FC-AL paths. Rescanchecks for new or removed host bus adapters, array controllers, and logicaldrives. Use Rescan after every hot-swap of FC-AL path hardware.

Select the Rescan function from the Feature menu.

Compaq Confidential – Need to Know RequiredWriter: John Blackburn Project: Parallel Database Cluster Model PDC/02000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide Comments:

Part Number: 161397-001 File Name: i-ch8 Troubleshooting.doc Last Saved On: 11/1/99 4:46 PM

Chapter 8Troubleshooting

This chapter addresses potential problems you might encounter as you install,configure, test, and operate your Compaq Parallel Database Cluster ModelPDC/O2000 (PDC/O2000). This chapter is organized as follows:

■ Basic Troubleshooting Tips

■ Troubleshooting Oracle8 Release 8.0.5 and operating system dependentmodule (OSD) installation problems and error messages

■ Troubleshooting Oracle8i Release 8.1.5 and operating system dependentmodule (OSD) installation problems and error messages

■ Troubleshooting node-to-node connectivity problems

■ Troubleshooting client-to-cluster connectivity problems

■ Troubleshooting shared storage problems

■ Troubleshooting Compaq Redundancy Manager problems

■ Troubleshooting other potential problems

8-2 Parallel Database Cluster Model PDC/02000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Basic Troubleshooting Tips

Power

Verify that the cluster nodes, StorageWorks RAID Array4000s (RA4000 Arrays), Compaq StorageWorks StorageHubs (Storage Hubs), and the switches or hubs for thecluster interconnect are all powered on.

The correct power-on sequence is:

■ RA4000 Arrays

■ Storage Hubs (Power is applied when the AC cord is plugged in.)

■ Ethernet switches/hubs or Compaq ServerNet Switches

■ ProLiant servers

NOTE: The Storage Hub and some network hub or switch devices do not have on/offbuttons; they receive power when the power cord is plugged in.

Physical Connections

Verify that all cable connections between the following components areproperly made. Where redundant connections exist, be sure that the redundantpaths are cabled correctly between:

■ Cluster nodes and the cluster interconnect switches or hubs

■ Cluster nodes and the client LAN switches or hubs

■ Cluster nodes and the Storage Hubs

■ Storage Hubs and the Compaq StorageWorks RAID Array 4000Controllers (RA4000 Controllers) installed in each RA4000 Array

Troubleshooting 8-3



Accessibility of Resources

Once the physical connections have been checked, ensure that communicationis occurring between all the cluster components.

■ To test the redundant Ethernet cluster interconnect and client LANconnections, use the ping utility. From each cluster node, ping theEthernet cluster interconnect adapter or port located in the other nodes.Similarly, from each cluster node, ping the name, then the address, ofthe client LAN Ethernet adapter or port located in the other nodes. Acorrect ping response from each node confirms that the Ethernetconnections are correct. For information about using the ping utility,refer to “Verifying Cluster Communications” in Chapter 5, “Installationand Configuration for Oracle8 Release 8.0.5,” or Chapter 6, “Installationand Configuration for Oracle8i Release 8.1.5.”

■ To test the redundant ServerNet cluster interconnect, use the vipingutility. From each cluster node, ping the ServerNet PCI Adapter locatedin every other node. For information about using the viping utility to testServerNet cluster interconnect connections, see “Verifying theServerNet Cluster Interconnect” in Chapter 6, “Installation andConfiguration for Oracle8i Release 8.1.5.”

■ To confirm that the shared storage is accessible to each cluster node, runRedundancy Manager or Windows NT Server Disk Administrator fromeach node. Confirm that the same RA4000 Array disk volumes can beseen from each node. If the shared storage resources are not consistentacross all nodes, refer to Appendix B, “Diagnosing and ResolvingShared Disk Problems” for more detailed guidelines.




Software Revisions

The following software components must have the same revision level on eachof the cluster nodes:

■ Oracle8 Server Release 8.0.5 with Oracle8 Parallel Server OptionRelease 8.0.5

■ Oracle8i Server Release 8.1.5 with Oracle8i Parallel Server OptionRelease 8.1.5

■ RA4000 Array Controller drivers

■ Compaq Insight Manager

■ Compaq Array Configuration Utility

■ Compaq Redundancy Manager

■ OSDs

■ Cluster interconnect drivers

■ Microsoft Windows NT Server 4.0, including Service Pack 3, 4, or 5

IMPORTANT: Compaq recommends Service Pack 4 or 5 of Windows NT Server for aredundant Ethernet cluster interconnect or client LAN. Using Service Pack 3 requiresinstalling the approved Microsoft patch (hot fix) article ID Q156655, entitled “MemoryLeak and STOP Screens Using Intermediate NDIS Drivers.”

For information about currently supported software revisions for thePDC/O2000, refer to the Compaq Parallel Database Cluster ModelPDC/O2000 for Oracle8 Parallel Server Release 8.0.5 Certification Matrix orthe Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8iParallel Server Release 8.1.5 Certification Matrix. These documents areavailable at


Depending on your specific configuration, other software (for example,customer applications such as packet utilities) might exist that should be of thesame revision level. If the same software is used on all the cluster nodes andeither assists with the cluster’s operation or takes advantage of clustering, therevision level should be the same.

Troubleshooting 8-5



Firmware Revisions

The firmware level must be identical for each of the following clustercomponents:

■ The system ROM on each ProLiant server

■ Firmware for the Compaq StorageWorks Fiber Channel Host BusAdapter (Fibre Host Adapter)

■ Firmware for the RA4000 Array Controller

For information about currently supported firmware revisions for thePDC/O2000, refer to the Compaq Parallel Database Cluster ModelPDC/O2000 for Oracle8 Parallel Server Release 8.0.5 Certification Matrix orthe Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8iParallel Server Release 8.1.5 Certification Matrix. These documents areavailable at


Depending on your specific configuration, other firmware might exist thatmust be of the same revision level. A determining factor is whether or not thehardware device is part of a path that is accessible by more than one clusternode, more than one RA4000 Array, or more than one client computer. If so, itis required that the firmware level of the hardware devices be identical.




Troubleshooting Oracle8 Release 8.0.5and OSD Installation Problems and ErrorMessages

While Running the NodeList Configurator, a DialogBox Appears Indicating Inability to Connect toRemote Nodes

■ Verify that you logged on as a user with all required administrativerights.

■ Verify that all the cluster nodes are powered on.

■ Verify that the client LAN connections are assigned properly by usingthe ping command. See Chapter 5, “Installation and Configuration forOracle8 Release 8.0.5,” for detailed instructions about using the pingcommand.

■ Rerun the NodeList Configurator utility and confirm that all of theentries are correct.

Error Message: Unable to Start Cluster Manager(CMSRVR.EXE)

■ Verify that the database links are properly set up on all nodes and thatcorrect entries exist for all datafiles, including OPS_CMDISK. Refer tothe Oracle8 Enterprise Edition Getting Started Release 8.0.5 forWindows NT manual for further details.

■ Verify that the shared storage is properly configured, powered on, andaccessible to each node. Refer to Appendix B, “Diagnosing andResolving Shared Disk Problems” for more details.

■ Verify that the redundant Ethernet cluster interconnect is properlyconfigured.

■ Verify that the OSDs are properly configured.

Troubleshooting 8-7



Error Message: Unable to Start Oracle Service■ Verify that you have run the NodeList Configurator and that no errors

were reported.

■ Verify redundant Ethernet cluster interconnect connectivity by pingingcluster interconnect Ethernet names.

■ Verify the ability to see the shared disks using Disk Administrator.

■ Verify that the Oracle PGMS Service has properly started.

■ Verify that the Cluster Manager (CMSRVR.EXE) has properly startedby checking the error logs. The default location for this error log isc:\compaq\ops\error.log.

Error Message: “Dependent Service Has NotStarted” Received While Attempting to Start theOracle Service

■ Look in the PGMS error log for activity.

■ Verify that the redundant Ethernet cluster interconnect connections areassigned properly by using the ping command. See Chapter 5,“Installation and Configuration for Oracle8 Release 8.0.5,” for detailedinstructions about using the ping command.

■ Use Disk Administrator or Redundancy Manager to ensure that the sameshared storage information is seen from each node.

■ Verify that the Cluster Manager (CMSRVR.EXE) has properly startedby checking the error logs. The default location for these error logs isc:\compaq\ops\error.log.

Error Message: Unable to Configure Oracle UsingOPSCONF Utility

■ Verify that you have run the NodeList Configurator and that no errorswere reported.

■ Verify that the database name specified in the NodeList Configuratormatches the name specified in OPSCONF.




Error Message: Error In Creating Oracle Instance■ Verify that you have correctly installed and configured any necessary

vendor OSD software components required for PDC/O2000. SeeChapter 5, “Installation and Configuration for Oracle8 Release 8.0.5,”for detailed instructions on how to install the OSD software.

■ The cluster control disk might not be assigned correctly. Verify thatSetlinks has been run and that the cluster control disk (OPS_CMDISK)is associated with a partition on the shared storage.

■ The CM_HOSTNAME in the registry is incorrect. Re-execute theNodeList Configurator and verify that the correct node names areentered.

Error Message: Initialization of the Dynamic LinkLibrary CM.DLL Failed. The Process Is TerminatingAbnormally

An error message is displayed if initialization of the DLL file cm.dll(c:\compaq\ops\cm.dll) fails. Stop the currently running PGMS Service andCluster Manager (CMSRVR.EXE), then restart them.

Unable to Start the Database■ From each node, use Disk Administrator or Redundancy Manager to

verify the node’s ability to accurately see the shared storage resources inthe RA4000 Arrays. If the shared storage resources are not seenconsistently across all nodes, refer to Appendix B, “Diagnosing andResolving Shared Disk Problems” for further guidelines.

■ Verify that all files created by OPSCONF were transferred properly toeach respective node.

■ Verify redundant Ethernet cluster interconnect connectivity by pingingcluster interconnect Ethernet names.

■ Verify the connections and accessibility of the cluster components.Refer to the “Basic Troubleshooting Tips” section earlier in this chapter.

■ Verify that the database links are properly set up on all nodes and thatcorrect entries exist for all datafiles, including OPS_CMDISK. Refer tothe Oracle8 Enterprise Edition Getting Started Release 8.0.5 forWindows NT manual for further details.

Troubleshooting 8-9



■ Verify that the OracleService is running. Run the Service applet locatedin the Windows NT Control Panel. Scan the list of services and check tosee if the OracleService is started.

Troubleshooting Oracle8i Release 8.1.5and OSD Installation Problems and ErrorMessages

Potential Difficulties Installing the OSDs with theOracle Universal Installer

■ The Oracle Universal Installer (OUI) uses an inventory file to trackwhich components of the OSDs have been installed on each of thecluster nodes. A single inventory file is maintained on the node fromwhich the OUI was initially run. All subsequent uses of the OUI must bemade from that same installing node.

■ Installing onto a cluster node that already contains some or all of theOSD components will likely result in an incomplete installation. Ifdifficulties persist, contact your support representative for assistance.

■ The OUI uses the network connections to transfer files from the primarynode to the other cluster nodes. Use the ping utility to ensure that propernode-to-node connectivity exists.

■ You cannot use the OUI to install the OSDs while any Oracle services(for example, OracleCMService or OracleNMService) are running.Oracle services must be stopped before you install the OSDs.

■ The SNMP service should be installed but not running when you use theOUI to install the OSDs. If the service is running, you must stop it untilthe installation is finished.

■ The OUI requires administrator rights on all cluster nodes. To verifyconnectivity and access rights for each node, issue the following fromthe command line on the primary node:

C: net use \\machine_name\C$ * /User: <admin_login>

in which the “*” results in the system prompting you for a password,and the password is not echoed.

If successful, the net use command returns:

“The command completed successfully.”




■ Determine whether or not you need to deinstall a partial OSDinstallation. An incomplete installation can result in a partial installationof the OSDs on the cluster. If this occurs, you must clean up after thepartial installation (by removing it) before you can successfully run theOUI again. An incomplete OSD installation could result for variousreasons, for example:

� A cable is no longer seated tightly into its connector

� The node from which you are running the OUI cannot communicatewith all other nodes in the cluster

� The user running the OUI does not have administrator privileges onall cluster nodes

Refer to “Deinstalling a Partial OSD Installation” in Chapter 7, “ClusterManagement,” for information about how to remove a partial OSDinstallation.

Unable to Start OracleCMService■ Verify that the cluster interconnect is properly configured.

■ Make sure the OSDs have been properly installed on the nodes byrunning the OUI on the cluster’s primary node. Click InstalledProducts and verify that an entry for the OSDs exists.

■ Verify that the OracleCMService has properly started by checking itserror log. The default location for the error log isc:\compaq\ops\cmsvr.log. If not found in the default location, look in thesame directory in which the OSD modules were installed.

■ Verify that the Node Manager Service, OracleNMService, is running. Ifnot, see the error logs in C:\Compaq\ops\nmsrvr.log. IfOracleNMService is running, check the CM Server logs inC:\Compaq\ops\cmsrvr.log. Attempt to start the OracleCMServiceagain.

■ Check the Windows NT Server event log for errors. Be sure to checkboth the system and application logs.

Troubleshooting 8-11



Unable to Start OracleNMService■ Verify that the OracleNMService has properly started by checking the

error log. The default location for the error log isc:\compaq\ops\nmsrvr.log. If not found in the default location, look inthe same directory in which the OSD modules were installed.

■ Verify that Oracle8i Release 8.1.5 software is correctly installed on allthe nodes. Refer to Chapter 6, “Installation and Configuration forOracle8i Release 8.1.5” for installation instructions.

■ Verify the connections and accessibility of the cluster components.Refer to the “Basic Troubleshooting Tips” section in this chapter.

■ Check the Windows NT Server event log for errors. Be sure to checkboth the system and application logs.

Unable to Start the Database■ Verify that Object Link Manager is properly installed and running on

each node. Also, verify that Object Link Manager accurately sees all ofthe Oracle partitions and their symbolic link associations. Refer toAppendix B, “Diagnosing and Resolving Shared Disk Problems” fordetailed guidelines.

■ Verify that all files created by Net8 Config Assistant were transferredproperly to each respective node.


■ Verify that the OracleService is running. Run the Service applet locatedin the Windows NT Control Panel. Scan the list of services and check tosee if the OracleService is started.

■ Verify that the database links are properly set up on all nodes and thatcorrect entries exist for all datafiles, including OPS_CMDISK. Refer tothe Oracle8i Parallel Server Setup and Configuration Guide Release8.1.5 for further details.




Initialization of the Dynamic Link LibraryNM.DLL Failed

If this failure occurs, stop the currently running OracleCMService andOracleNMService. Restart the OracleNMService, then the OracleCMService.Do this by running the Service applet located in the Windows NT ControlPanel. Locate each service in the displayed list, highlight it, then stop it.Finally, highlight the service and restart it.

Troubleshooting Node-to-NodeConnectivity Problems

Nodes Are Unable to Communicate with Each Other■ Check the Windows NT Server event logs for errors.

■ Check the OracleCMService error log for information. The defaultlocation for the error log is c:\compaq\ops\cmsrvr.log. If not found inthe default location, look in the same directory in which the OSDs wereinstalled.


■ The cluster interconnect must be on a separate subnet than the clientLAN. Check this by viewing the IP addresses assigned to allinterconnect adapters and to all client LAN adapters.

■ If Compaq Insight Manager is loaded and configured, use it to check theoperation statistics and possible error messages for the interconnectadapter installed in each cluster node.

When Using the Redundant Ethernet ClusterInterconnect

■ Make sure that the redundant Ethernet cluster interconnect components(adapters, cables, switches or hubs) are properly installed as specified inChapter 5, “Installation and Configuration for Oracle8 Release 8.0.5,” orChapter 6, “Installation and Configuration for Oracle8i Release 8.1.5.”

■ Make sure that the Ethernet cluster interconnect and the client LAN areconnected to different Ethernet switches or hubs.




■ Use the ping utility to test the redundant Ethernet cluster interconnectconnections. For information about the ping utility, refer to “VerifyingCluster Communications” in Chapter 5, “Installation and Configurationfor Oracle8 Release 8.0.5,” or Chapter 6, “Installation and Configurationfor Oracle8i Release 8.1.5.”

■ Make sure that all Ethernet traffic not related to the Oracle data is usingthe client LAN, not the cluster interconnect.

When Using the Redundant ServerNet ClusterInterconnect

■ Make sure that the ServerNet cluster interconnect components (adapters,cables, and switches) are properly installed as specified in Chapter 6,“Installation and Configuration for Oracle8i Release 8.1.5.” Verify thatthe components are installed to the correct ServerNet path (X or Y).

■ Use the viping utility to verify that all nodes can communicate with eachother over the redundant ServerNet cluster interconnect. For informationabout using the viping utility to test ServerNet cluster interconnectconnections, see “Verifying the ServerNet Cluster Interconnect” inChapter 6, “Installation and Configuration for Oracle8i Release 8.1.5.”

■ Verify that the ServerNet PCI Adapter is installed in the “optimum” PCIslot in each node. Depending on the server model, the location of theServerNet PCI Adapter can affect its performance. For information onlocating the ServerNet PCI Adapter for optimal performance, see theCompaq Parallel Database Cluster Model PDC/O2000 for Oracle8Parallel Server Release 8.0.5 Certification Matrix or the CompaqParallel Database Cluster Model PDC/O2000 for Oracle8i ParallelServer Release 8.1.5 Certification Matrix. These documents areavailable at





viping Does Not Complete Successfully

Sometimes, when using the viping utility to ping the node name (machine hostname), the utility is not successful in communicating with the node specified.Before concluding that cluster nodes cannot communicate through a CompaqServerNet cluster interconnect, try pinging the ServerNet ID of the node inquestion.

Get the ServerNet ID of a node by entering a viping command from that nodeusing the local node name as the operand for viping. For example, if themachine host name of a node is node1, enter viping node1viping node1viping node1viping node1 to get the ServerNetID of node1.

If on node1 the command viping node2viping node2viping node2viping node2 does not return successfully:

1. Get the ServerNet ID of node2 by entering viping node2viping node2viping node2viping node2 at node2. Theviping utility responds in the following way. (0xF0080 is an example ofa ServerNet ID.)

Pinging node2 [0xF0080] with 12 bytes of data:

Reply from 0xF0080: bytes = 12 time = 12ms

2. Test node1 to node2 ServerNet connectivity using the ServerNet ID byentering viping 0xF0080viping 0xF0080viping 0xF0080viping 0xF0080 at node1. The viping utility responds in thefollowing way.

Pinging 0xF0080 with 12 bytes of data:

Reply from 0xF0080: bytes = 12 time = 12ms

If the viping utility does not complete successfully using either the ServerNetID or node name, verify that the ServerNet cluster interconnect hardware(ServerNet PCI Adapters, ServerNet cables, ServerNet Switches) is properlyinstalled and connected. If the connections are good, you might need todeinstall and reinstall the ServerNet OSDs. See Chapter 7, “ClusterManagement.”

For information about viping options and error diagnostics, see Appendix A,“viping Utility.”




Unable to Ping the Cluster Interconnect or theClient LAN

If you are using the recommended method of using static IP addresses, ensurethat the hosts file and lmhosts files are properly set up.

When Using the Redundant Ethernet ClusterInterconnect

■ Make sure that the Ethernet cables are properly connected and that theEthernet switches or hubs are powered on. Verify that all Ethernetadapters used as the primary cluster interconnect path are connected tothe same switch and that all adapters used as the redundant clusterinterconnect path are connected to the redundant switch.

■ Make sure that the Ethernet drivers are properly installed and configuredon the cluster nodes. Verify the Ethernet drivers for the interconnectadapters are the same revision level on all cluster nodes.

■ If you are using two dual-port Ethernet adapters in each node to connectto both the client LAN and the Ethernet cluster interconnect, make surethat the cables are installed between the correct adapter ports andswitches.

When Using the Redundant ServerNet ClusterInterconnect

■ In two-node clusters, ServerNet cables can be installed directly betweenthe two ServerNet PCI Adapters. Make sure that the ServerNet cablesare properly connected between the X ports and Y ports on bothServerNet PCI Adapters.

■ In clusters with three or more nodes, two ServerNet Switches must beused to create the X-path and Y-path for ServerNet system area networkcommunication. Make sure that the ServerNet cables are properlyconnected and the ServerNet Switches are powered on. The ServerNetcables connected to the X-port on the ServerNet PCI Adapters must allbe connected to the same X-path ServerNet Switch. Likewise, theServerNet cables connected to the Y-port on the ServerNet PCIAdapters must all be connected to the same Y-path ServerNet Switch.

■ Make sure that the ServerNet drivers are properly installed andconfigured on the cluster nodes. Verify the ServerNet drivers for theServerNet cluster interconnect are the same revision level on all clusternodes.




Node or Nodes Unable to Rejoin the Cluster■ If you have a redundant Ethernet cluster interconnect, use ping to ensure

that the rejoined node or nodes can communicate with other nodes in thecluster. For information about using the ping utility to test redundantEthernet cluster interconnect connections, see “Verifying ClusterCommunications” in Chapter 5, “Installation and Configuration forOracle8 Release 8.0.5,” or Chapter 6, “Installation and Configuration forOracle8i Release 8.1.5.”

■ If you have a redundant ServerNet cluster interconnect, use the vipingutility to ensure that the rejoined node or nodes can communicate withthe other nodes in the cluster. For information about using the vipingutility to test ServerNet cluster interconnect connections, see “Verifyingthe ServerNet Cluster Interconnect” in Chapter 6, “Installation andConfiguration for Oracle8i Release 8.1.5.”

■ Make sure the OSDs have been properly installed on the affected nodeor nodes by running the OUI on the cluster’s primary node. ClickInstalled ProductsInstalled ProductsInstalled ProductsInstalled Products and verify an entry for the OSDs exists.

■ Make sure the OracleCMService and all other Oracle services havestarted without error. Run the Services applet in the Windows NTControl Panel and check to see if the services have started. Run theWindows NT Event Log and check for error messages about the Oracleservices.

■ Run Object Link Manager from each node to verify that the Oraclepartitions and symbolic link associations are accurately displayed.

■ Verify that the node or nodes have access to the shared storage. RunRedundancy Manager or Disk Administrator and make sure that all theshared disk resources appear as expected. If the view is not consistentacross all cluster nodes, refer to Appendix B, “Diagnosing andResolving Shared Disk Problems.”




Ping <computer_name> Shows ClusterInterconnect IP Address Instead of Client LAN IPAddress

This issue is commonly caused by a problem in Windows NT Server ServicePack 4 provides a fix. For information about this fix, refer to the Compaqwhite paper Supported Ethernet Interconnect for Compaq Database Clustersusing Oracle Parallel Server (Doc # ECG062/0299) at


If you do not have Windows NT Service Pack 4 installed, the followingsuggestions might alleviate the issue:

■ Due to the complexity of Windows NT Server and multiple-NIC serversunder Windows NT Server, the manner by which you assign a clusterinterconnect or client LAN to a specific network card might not occur asyou expected. Use the ping utility to verify IP and computer nameaddresses. You may have to accept the address that Windows NT Serverhas assigned to a given NIC port or subport.

■ Check the %SystemRoot%\system32\drivers\etc\hosts file to verify thatthe redundant Ethernet cluster interconnect IP address and computernames are correctly associated and that the client LAN IP addresses andcomputer names are correctly associated. Perform ping testing asdescribed in “Verifying Cluster Communications” in Chapter 5,“Installation and Configuration for Oracle8 Release 8.0.5,” or Chapter 6,“Installation and Configuration for Oracle8i Release 8.1.5.”

■ Install Windows NT Server Service Pack 4 or 5.




Troubleshooting Client-to-ClusterConnectivity Problems

A Network Client Cannot Communicate with theCluster

■ Verify that the user ID has sufficient rights to perform the desiredaction.

■ Make sure proper network connectivity exists by verifying that theconnections from the network client to the cluster are cabled correctly.Additionally, verify that the network routers/switches/hubs used in theclient LAN are powered on and properly cabled.

■ Make sure that pinging an IP address over the client LAN from a clientto the cluster issues a correct reply. Perform ping testing as described inthe “Verifying Cluster Communications” section in Chapter 5,“Installation and Configuration for Oracle8 Release 8.0.5,” or Chapter 6,“Installation and Configuration for Oracle8i Release 8.1.5.”

■ Make sure that the TNSNames.ora file is properly configured on clientand cluster nodes as defined by the Oracle Net8 Config Assistant utility.

■ Make sure that the Oracle TNSListener is running on the cluster nodes.Run the Service applet in the Windows NT Control Panel to verify thatit is started.

■ Make sure that the Oracle Listener files are set up on the cluster nodesby using the Oracle TNSPing utility.

■ Run Object Link Manager from each node to verify correct Oraclepartitions and symbolic link associations. Refer to Appendix B,“Diagnosing and Resolving Shared Disk Problems,” for detailedguidelines.




Troubleshooting Shared StorageProblems

This section addresses potential problems arising from the use of RA4000Arrays as the shared storage devices for the PDC/O2000. This section does notaddress RA4000 Array problems that are specific to the product itself or to theuse of the RA4000 Array in a stand-alone server configuration. For thoseissues, refer to the Compaq StorageWorks RAID Array 4000 User Guide.

Verifying Connectivity to the Redundant FibreChannel Arbitrated Loop

To ensure that all I/O path connections in the redundant FC-AL are intact andfunctioning, verify the following:

■ Two Fibre Host Adapters have been installed in the same two PCI slotsacross all servers in the cluster.

■ Verify that Fibre Channel cables are installed in the correctconfiguration between the Fibre Host Adapter pair installed in everyserver and their two Storage Hubs. Refer to “Cabling the Fibre HostAdapters to the Storage Hubs” in Chapter 5, “Installation andConfiguration for Oracle8 Release 8.0.5,” or Chapter 6, “Installation andConfiguration for Oracle8i Release 8.1.5,” for cabling guidelines.

■ Verify that the two array controllers in each RA4000 Array areconnected by Fibre Channel cable to different Storage Hubs. Refer to“Cabling the Storage Hubs to the RA4000 Arrays” in Chapter 5,“Installation and Configuration for Oracle8 Release 8.0.5” or Chapter 6,“Installation and Configuration for Oracle8i Release 8.1.5,” for cablingguidelines.




Shared Disks in the RA4000 Arrays Are NotRecognized By One or More Nodes

If it appears that the shared disks in the RA4000 Arrays are not properlyrecognized by one or more nodes, do the following:

■ You may not have waited long enough to view the most up-to-dateinformation. Wait a minute, then retry.

■ Restart the affected node or nodes.

■ Make sure that the Compaq StorageWorks Fibre Channel Host Adapter(Fibre Host Adapter) driver for Windows NT Server is installed andrunning on all cluster nodes. If it is not installed, refer to the CompaqStorageWorks Fibre Channel Host Adapter Installation Guide forinstallation instructions.

■ Determine whether or not hardware errors or communications transportfailures are causing the problem. Use Windows NT Event Viewer toverify that no hardware errors or transport problems exist. Check theevent log for disk I/O error messages or indications of problems with thecommunications transport.

■ Determine whether or not the Compaq servers support the physicalSCSI disks installed in the RA4000 Arrays. Check the event log for diskI/O error messages or indications of problems with the communicationstransport.

■ If the drives are still not recognized, it is recommended that you startinvestigating the problem from the lowest level of drive configuration.

� Run the Compaq Array Configuration Utility. You can run thisutility online if at least one logical drive is configured andrecognized. The Array Configuration Utility is used to see if any ofthe drive arrays are offline (degraded or failed). The online utility isinstalled as part of the Compaq SmartStart and Support SoftwareCD.

� If all drives are not recognized by the Array Configuration Utility, itis probably a physical connection problem.

� Verify that the lights on the Fibre Host Adapters, Storage Hubs, andRA4000 Array Controller are all green.

� Verify that the shortwave Gigabit Interface Converters (GBICs) areproperly seated in the Fibre Host Adapters, Storage Hubs, andRA4000 Array Controllers.




� Verify that all Fibre Channel cables are properly connected to theirGigabit Interface Converter-Shortwave (GBIC-SW) modules. Fordetails on how to connect the GBIC-SW modules and the FibreChannel cables, refer to the Compaq StorageWorks RAID Array4000 User Guide.

■ If all drives are recognized by the Array Configuration Utility and areconfigured correctly, start Windows NT Server and run DiskAdministrator. Verify that all drive volumes can be accurately viewedfrom Disk Administrator on each node.

■ If you continue to have problems, refer to Appendix B, “Diagnosing andResolving Shared Disk Problems,” for further information. Also refer tothe Compaq Fibre Channel Troubleshooting Guide for further correctiveactions.

Node Cannot Connect to the Shared Drives■ Verify that a physical connection exists from each node to the Storage

Hub.

� Verify the shortwave Gigabit Interface Converters (GBICs) areproperly seated in the affected Fibre Host Adapter and its StorageHub.

� Verify that all Fibre Channel cables are properly connected to theirGBICs.

■ Verify all nodes can access the shared storage in the RA4000 Arrays.On each node, start Disk Administrator to verify that the same shareddisk resources are seen from every node in the cluster.

■ The shared storage components must be powered on before the servers.If any cluster node was powered on before the Storage Hubs or theRA4000 Arrays, shut down the node and restart.

■ If the node is still unable to connect to the RA4000 Arrays, refer toAppendix B, “Diagnosing and Resolving Shared Disk Problems.”




Troubleshooting Compaq RedundancyManager Problems

The following sections address events, warning messages, and error messagesyou may receive from Redundancy Manager.

Event Logging

Redundancy Manager displays messages warning of possible changes andevents to the system. When a message is displayed, click on Help to receivemore details about that particular message.

The Event Viewer for Microsoft Windows NT Server displays additionalinformation. For more information on using the Windows NT Event Viewer,refer to your Microsoft Windows NT Server documentation.

Informational Messages

Table 8-1 identifies the informational messages and actions to take usingRedundancy Manager.

Table 8-1Compaq Redundancy Manager Informational Messages

Message Description Action

Another instance has locked theloop. This instance is running inRead Only mode.

Another application has issued alock management command toan array controller. The lockmanagement command onlyallows viewing of the data.

No action needed to view thedata.

The lock could not be set. An array controller board is in anunknown state that has causeda failed connection to the arraycontroller.


continued




Table 8-1Compaq Redundancy Manager Informational Messages continued


The loop has been locked byanother application.



Another program has a lock onthe array controller(s). Thisprogram is either still running orhas exited improperly.

It is recommended thatRedundancy Manager not be runwhile another program has alock on the array controller(s).To stop this instance fromstarting, select the Cancelbutton. To start this instanceselect the Start button.

Another program has issued alock management command toan array controller. The lockmanagement command onlyallows viewing of the data.


This RA4000 is inActive/Standby mode. All logicaldisks in this array must have thesame path assignments. Are yousure you want to continue?

A request has been made tochange the status of a path inan Active/Standby configuration.

Proceed or click Cancel.

continued




Table 8-1Compaq Redundancy Manager Informational Messages continued


You have not selected all thePaths to the following logicaldisk(s):

The logical disk or drives shownare claimed but not all possiblepaths were assigned.

Click OK for the RedundancyManager to automatically assignthe paths for this logical disk;

Or, click Cancel to assign thepaths manually.

You have not selected an ActivePath to <xxxxxx>.

A logical disk is claimed but noActive Path is selected.

Click OK for the RedundancyManager to automatically assignthe path for this logical disk;

Or, click Cancel to assign thepath manually.

You have not selected any Pathsto <xxxxxx>.

A logical disk is claimed but nopaths are assigned.

Click OK for the RedundancyManager to automatically assignthe paths for this logical disk;

Or, click Cancel to assign thepaths manually.

You have not selected a StandbyPath to <xxxxxx>.

A logical disk is claimed but noStandby Path is assigned.

Click OK for the RedundancyManager to automatically assignthe path for this logical disk;

Or, click Cancel to assign thepath manually.




Warning Message

Table 8-2 identifies a warning message and actions to take usingRedundancy Manager.

Table 8-2Compaq Redundancy Manager Warning Message


The previous lock by anotherapplication has expired. Do youwant to override the lock andtake control of the loop?

The previous lock on an arraycontroller has expired.

No action needed to view thedata;

Or, override and take control ofthe array controller.

Error Messages

Table 8-3 identifies the error messages and actions to take usingRedundancy Manager.

Table 8-3Compaq Redundancy Manager Error Messages


Another instance has locked theloop. This instance is running inRead Only mode.



To configure the data, close theother application to unlock thearray controller. If the lock hasexpired, Redundancy Managerallows overriding the lock andtaking control of the arraycontroller.

Could not load resource DLL. The file containing theconfiguration utility resourcescould not be loaded.

Reload Redundancy Managerfrom the CD.

continued




Table 8-3Compaq Redundancy Manager Error Messages continued


Logical disk could not bereached in the followingRA4000: <xxxxxx>.

Redundancy Manager found anRA4000Array with no drives init. This could be caused by adamaged array or a badconnection.

Check the array and allconnections.

The lock could not be set. An array controller board is in anunknown state that caused afailed connection to the arraycontroller.


To configure the data, checkboth array controllers and allphysical connections.

The loop has been locked byanother application. RedundancyManager is running in Read Onlymode.

Another application has issued alock management command toan array controller. The lockmanagement command onlyallows viewing the data.


To configure the data, close theother application to unlock thearray controller. If the lock hasexpired, Redundancy Managerallows overriding the lock andtaking control of the arraycontroller.

There was an error attemptingto shut down Windows NT.Please close all applications andshut down now.

An error occurred while shuttingdown Windows NT.

Close all applications and shutdown the computerimmediately.

A version of the Compaq FibreChannel Host Adapter SCSIMiniport Driver (cpqfcalm.sys) isbeing used that does notsupport redundancy. Theminimum version forredundancy support is VX.X. Thecurrent version is VX.Z.Redundancy has been disabled.

This message informs you thatthe version of cpqfcalm.sysdoesn’t support redundancy.

Make sure you have the correctversion of cpqfcalm.sys.




Other Redundancy Manager Problems

Table 8-4 identifies the text messages that Redundancy Manager displays towarn of possible changes and events to the system. When a message isdisplayed, click on HelpHelpHelpHelp to receive more details about that particular message.

Table 8-4Troubleshooting Other Redundancy Manager Problems


Could not find the resource DLLfile.

The resource DLL file has beencorrupted or is missing.

Reload the RedundancyManager software from the CD.

Intercontroller Link Failure. This indicates a hardwareproblem.

Replace the RA4000 backplaneor add a new array controller.

Illegal Drives. You are using theMicropolis 4/9-gigabyte drives.The firmware cannot resolvetwo array controllers talking tothe drive.

Replace the drives.

Array controller firmwareversions don’t match.

The array controllers havedifferent firmware versions.

Run the Options ROMPaq toupdate the firmware to the mostcurrent version.

Array controller hardware is notthe same.

The array controller models aredifferent.

Make both array controllers thesame model.

Both array controllers don’t seeall the drives.

The backplane in the RA4000may be faulty.

Repair or replace the backplane.

Expand operation in is progress. The cache is not mirrored duringthis operation.

Wait until the expand operationis compete.




Troubleshooting Other Potential Problems

Windows NT Blue Screen With AFD.SYS FailureDisplayed

This blue screen can occur as a result of a significant amount of traffic goingover the Microsoft Windows NT NDIS driver.

■ Restart the node.

■ If the problem persists, contact Microsoft and request the AFD.SYS fix.This is a Windows NT Service Pack 3 issue.


Part Number: 161397-001 File Name: j-appa Viping Utility.doc Last Saved On: 10/29/99 5:37 PM

Appendix Aviping Utility

The viping utility verifies that cluster nodes can communicate with each otherusing the ServerNet cluster interconnect. The following shows the syntax andoption summary of the viping command.

Syntax and Option Summaryviping < node |snid > [-t] [-b size] [-v] [-?]

<node |snid> Node name or ServerNet ID. The node name is the machinehost name, or local node name. 0xF0080 is an example of aServerNet ID. A node can determine its own ServerNet IDby using its machine host name as the operand for viping.

-t Retry the viping command until interrupted. This option isuseful if, for example, you are changing ServerNet cablingand you want viping to continue trying to establishcommunication while making cable changes.

-b size Size of the viping buffer. The default size is 12 bytes. Themaximum size is 256K bytes, which is useful whendiagnosing problems with large data transfers.

-v Enable verbose logging output. This option providesextended tracing information, which can be useful indebugging situations.

-? Display viping command syntax and option summary.

A-2 Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Example

To test ServerNet connectivity from one node to another in which the machinehost names are node1 and node2, enter the following command at node1:

C: \viping node2

If successful, viping returns the following sample output. (0xF0040 is anexample of a ServerNet ID.)

Pinging node2 [0xF0080] with 12 bytes of data:Reply from 0xF0080: bytes = 12 time = 12ms

Error DiagnosticsThe following table shows the errors viping can produce, possible causes, andwhat action to take to correct the problem.

Table A-1viping Errors

Error Possible Cause/Corrective Action

Generic system resourceerror.

The system might be running low on memory or systemresources. Try the operation again. If the problem persists,restart the system.

Generic vi internal error. Check ServerNet hardware connections. If the problempersists, restart the system or uninstall and reinstallServerNet OSDs.

Data integrity check failed. Potential data corruption has been detected. CheckServerNet hardware connections. If the problem persists,restart the system or uninstall and reinstall ServerNetOSDs.

continued

viping Utility A-3



Table A-1viping Errors continued

Error Possible Cause/Corrective Action

Connection to remote nodetimed out.

A communications error with another node has occurred.Check ServerNet hardware connections and verify theServerNet ID for the remote node. If the problem persists,restart the system or uninstall and reinstall ServerNetOSDs.

Request timed out. A communications error with another node has occurred.Check ServerNet hardware connections. If the problempersists, restart the system or uninstall and reinstallServerNet OSDs.

Unrecognized parameter -a. The format of viping does not conform to normal usage.Correct the command syntax and try again.

Multiple node namesdefined node and node.

More than one node name has been specified. Correct thecommand syntax and try again.

Unable to determine SNIDfor remote node node.

viping could not communicate with the remote node usingthe node name. Try again using the ServerNet ID as theoperand for viping.


Part Number: 161397-001 File Name: k-appb Verifying Shared Storage Visibility.doc Last Saved On: 10/29/99 5:40 PM

Appendix BDiagnosing and Resolving Shared Disk

Problems

IntroductionWhen one or more nodes in a Compaq Parallel Database Cluster ModelPDC/O2000 (PDC/O2000) cannot properly join the cluster, you need toconsider two possible causes:

■ A problem related to the shared disk storage in the cluster’s CompaqStorageWorks RAID Array 4000s (RA4000 Arrays) is preventing one ormore nodes from accurately viewing or successfully accessing theavailable shared storage resources.

■ A connectivity problem exists either within the cluster interconnect orthe client LAN, preventing one or more nodes from communicating withthe other nodes or preventing a client from communicating with thecluster.

This appendix describes the procedures you should follow to diagnose andresolve suspected shared disk problems in a PDC/O2000.

Figure B-1 identifies, in a flowchart format, the sequence of tasks you shouldperform to diagnose, isolate, and resolve shared disk problems in aPDC/O2000. A detailed description of each task is provided in the followingsections of this appendix.

B-2 Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Entry Point: Potential SharedDisk-Related Problem Detected

Run Object Link ManagerOn All Nodes

Restart All Affected Nodesin the Cluster

Rerun and Validate Object LinkManager On All Affected Nodes

Run and ValidateCompaq Redundancy Manager

On All Nodes

Run Windows NT ServerDisk Administrator On All Nodes

Run and Validate theArray Configuration Utility

On All Nodes

Perform Cluster Softwareand Firmware Checks

Perform Cluster Hardware Checks

Contact YourCompaq Support Representative

Problem Persists

Proceed To

Problem Persists

Problem Persists

Problem Persists

Problem Persists

ProblemPersists

Problem Persists

Problem Persists

OK

OK

OK

OK

OKOK

OK

OK

Exit. ProblemFixed

or No FurtherDisk

Problems

Rerun andValidate

Object LinkManager

On AllAffectedNodes

Exit. ProblemFixed

or No FurtherDisk

Problems

Figure B-1. Tasks for diagnosing and resolving shared storage problems

Diagnosing and Resolving Shared Disk Problems B-3



Run Object Link Manager On All NodesObject Link Manager simplifies the creation and maintenance of symboliclinks between disk partitions and Oracle data files by placing the symbolic linknames directly into the disk partitions, thereby tracking the symbolic linksdynamically. This means that if the order in which disk drives are broughtonline changes, Oracle8 Server or Oracle8i Server can find the correct datafiles and can properly start the Oracle database.

Object Link Manager also provides a detailed view of the Oracle partitions andtheir symbolic link associations in the cluster’s shared storage.

To view the shared storage information from Object Link Manager, performthese tasks:

1. From each node, run Object Link Manager.

2. Verify that it runs properly and displays no errors. If necessary, use theonline help facility to assist you in your navigation of Object LinkManager.

3. Verify that Object Link Manager can see all Oracle partitions and theirsymbolic link associations and that this view is consistent across allnodes.

4. Repeat steps 1 through 3 on all other nodes in the cluster.

5. If Object Link Manager does not run properly or displays incorrectOracle partition and symbolic link information on one or more nodes,proceed to “Restart All Affected Nodes In the Cluster” in this appendixfor further instructions.

For instructions on installing and using Object Link Manager, see the readmefile:


This file is located on these cluster kit CDs for the PDC/O2000:

■ Compaq Parallel Database Cluster for Oracle8 Parallel Server Release8.0.5 Ethernet Clustering Software CD

■ Compaq Parallel Database Cluster for Oracle8i Parallel Server Release8.1.5 Ethernet Clustering Software CD

■ Compaq Parallel Database Cluster for Oracle8i Parallel Server Release8.1.5 ServerNet Clustering Software CD




Restart All Affected Nodes in the ClusterIf Object Link Manager does not run properly on one or more nodes or doesnot provide consistent information about Oracle partitions and symbolic linksfrom all nodes, perform these steps:

1. Identify the node or nodes on which Object Link Manager either doesnot run properly or does not present the correct information aboutOracle partitions and symbolic links.

2. Restart each affected node.

3. Proceed to “Rerun and Validate Object Link Manager On All AffectedNodes” in this appendix for further instructions.

Rerun and Validate Object Link ManagerOn All Affected Nodes

Perform these steps from every node on which Object Link Manager does notrun properly:

1. Verify that the Object Manager service is properly installed on the node.For instructions on installing and using Object Link Manager, refer tothis readme file contained on each of the three cluster kit CDs:


2. Verify that the correct symbolic link names have been used for theshared disk partitions.

3. Restart the Object Manager service.





Run and Validate Compaq RedundancyManager On All Nodes

The Redundancy Manager screens provide information about the sharedstorage resources in the cluster, including RA4000 Arrays, array controllerconfigurations, drive lettering, and volume labels.

To run and validate Redundancy Manager, perform these steps from eachnode:

IMPORTANT: Redundancy Manager should be run from only one node at a time;otherwise, multiple Redundancy Manager processes will attempt to control the sameRA4000 Arrays.

1. Use the Redundancy Manager screens to verify that every RA4000Array is visible and that all logical disks and Compaq StorageWorksRAID Array 4000 Controllers (RA4000 Array Controllers) are correctlyand consistently defined across all nodes. Also confirm that the driveletters and volume labels associated with the shared disks are consistentacross all nodes. Refer to “Verifying Shared Storage Using RedundancyManager” in Chapter 5, “Installation and Configuration for Oracle8Release 8.0.5” or Chapter 6, “Installation and Configuration forOracle8i Release 8.1.5” for descriptions of the Redundancy Managerscreens.

2. Verify that the version number for Redundancy Manager is correct.Refer to the Compaq Parallel Database Cluster Model PDC/O2000 forOracle8 Parallel Server Release 8.0.5 Certification Matrix or theCompaq Parallel Database Cluster Model PDC/O2000 for Oracle8iParallel Server Release 8.1.5 Certification Matrix for versioninformation.

3. Verify that Redundancy Manager has been properly installed on thenode. Refer to “Installing Redundancy Manager” in Chapter 5,“Installation and Configuration for Oracle8 Release 8.0.5” or Chapter 6,“Installation and Configuration for Oracle8i Release 8.1.5” for detailedinstallation instructions.





Run Windows NT Server DiskAdministrator On All Nodes

Perform these steps from each node:

1. From Windows NT Server Disk Administrator, verify that each RA4000Array is visible.

2. Verify that all shared drive information (drive letters, volume labels,number of partitions, partition sizes, etc.) is consistent across all nodes.


Run and Validate the Array ConfigurationUtility On All Nodes

The Array Configuration Utility is used to configure the hardware aspects ofdisk drives connected to the RA4000 Array Controllers. The ArrayConfiguration Utility allows you to configure RAID levels and to add diskdrives or RA4000 Arrays to an existing configuration. Like DiskAdministrator and Redundancy Manager, it also provides each node with adetailed view of the shared storage resources in the cluster.

To run and validate the Array Configuration Utility from each node, performthese steps:

1. From the Array Configuration Utility, verify that all RA4000 Arrays andtheir logical disks are visible and consistent across all nodes.

2. Verify that each RA4000 Array’s serial number for all shared disks isconsistent across all nodes.

3. Verify that the RAID configurations of RA4000 Array shared disks areconsistent across all nodes.





Perform Cluster Software and FirmwareChecks

Use the Compaq SmartStart and Support Software CD to perform these checkson the cluster software and firmware:

1. Verify that all necessary drivers have been installed. For example, verifythat the Compaq StorageWorks Fibre Channel Host Adapter (Fibre HostAdapter) driver for Windows NT Server is installed and running on allcluster nodes. Refer to Chapter 5, “Installation and Configuration forOracle8 Release 8.0.5” or Chapter 6, “Installation and Configuration forOracle8i Release 8.1.5,” for driver installation guidelines.

2. Verify that that the version levels for all drivers, including those forFibre Host Adapters and RA4000 Array Controllers, are correct andconsistent across all nodes.

3. Verify that all firmware versions, including those for the system ROMon each server, the Fibre Host Adapter firmware, and the RA4000 ArrayController firmware, are correct and consistent across all nodes.

NOTE: Refer to the Compaq Parallel Database Cluster Model PDC/O2000 for Oracle8Parallel Server Release 8.0.5 Certification Matrix or the Compaq Parallel Database ClusterModel PDC/O2000 for Oracle8i Parallel Server Release 8.1.5 Certification Matrix forinformation about supported software and firmware versions.

Perform Cluster Hardware ChecksValidate the cluster hardware by performing these steps:

1. Determine whether or not hardware errors or communications transportfailures might be causing the problem. Use Windows NT Event Viewerto verify that no hardware errors or transport problems exist. Check theevent log for disk I/O error messages or indications of problems with thecommunications transport.

2. Determine whether or not the Compaq servers support the physicalSCSI disks that are installed in the RA4000 Arrays. Check the event logfor disk I/O error messages or indications of problems with thecommunications transport.

3. Verify that the lights on the Compaq StorageWorks Fibre Channel HostAdapters (Fibre Host Adapters), Compaq StorageWorks Storage Hubs(Storage Hubs), and RA4000 Array Controllers are all green.




4. Verify that the Gigabit Interface Converter-Shortwave (GBIC-SW)modules are all the correct brand or model type and are properly seatedin the Fibre Host Adapters, Storage Hubs, and RA4000 ArrayControllers.

5. Inspect all Fibre Channel cables to verify that they are in good conditionand have no breaks or excessive bends.

6. Verify that the Fibre Channel cables are properly connected to theGBIC-SW modules in the Fibre Host Adapters, Storage Hubs, andRA4000 Array Controllers. For details on how to connect the GBIC-SWmodules and the Fibre Channel cables, refer to the CompaqStorageWorks RAID Array 4000 User Guide.

7. Verify that all Fibre Host Adapters, RA4000 Array Controllers, andRA4000 shared disk modules are properly installed and seated in theirreceptacles.

Contact Your Compaq SupportRepresentative

If you are unable to resolve the problem, contact your Compaq supportrepresentative for assistance.

In North America, call the Compaq Technical Support Phone Center at1-800-OK-COMPAQ. Outside North America, call the nearest CompaqTechnical Support Phone Center. Telephone numbers for worldwide TechnicalSupport Centers are listed on the Compaq website at

www.compaq.com


Part Number: 161397-001 File Name: l-glossary.doc Last Saved On: 10/29/99 3:46 PM

Glossary

Availability A measure of how well a computer system can deliver services to its

clients continuously. Availability is typically expressed as a

percentage, with 100 percent being the best possible rating.

Client LAN A local area network dedicated to servicing requests between

network client machines and between the cluster nodes and the

client machines. This is different than a cluster interconnect, which

is dedicated to servicing information requests to and from only

cluster nodes.

Cluster An integration of software and hardware technologies that enables a

set of loosely coupled servers and storage to present a single

system image to clients and to operate as a single system.

Clustering is a way to improve availability and scalability.

Downtime The period of time that a computer system is unable to service

clients’ requests.

Ethernet A standard network protocol that operates mostly on a physical

level, using network interface cards and cabling to transmit data

between computers. Transfer rates are normally 10 or 100 megabits

per second.

Fault tolerance A characteristic that allows continued operation of a computer entity

when a fault (or failure) is encountered. Fault tolerance applies to a

computer component, an individual computer, or a cluster of

computers, primarily through designing redundant elements into the

entity.

2 Parallel Database Cluster Model PDC/O2000 for Oracle Releases 8.0.5 and 8.1.5 Administrator Guide



Fibre Channel An IEEE standard for providing high-speed data transfer among

workstations, mainframes, supercomputers, desktop computers,

storage devices, and display devices.

Fibre ChannelArbitrated Loop(FC-AL)

A data path from the Fibre Host Adapters in the cluster nodes to the

array controllers in the RA4000 Arrays. A PDC/O2000 requires one

or more redundant FC-ALs. Each FC-AL is redundant because there

are two paths from each node to each RA4000 Array.

GBIC-SW module Gigabit Interface Converter-Shortwave module. A module installed at

both ends of a Fibre Channel cable.

Heartbeat A signal transmitted between cluster nodes to indicate whether the

nodes are operating.

High availability A term used to identify a computer system that can continuously

deliver services to its clients 99.9 percent of the time (no more than

8.5 hours of downtime per year).

Hot-pluggable A computer component that can be removed and replaced without

powering down the system.

Hot spare A computer component that is powered on, not actively processing

data, and awaiting a request to take over operation of a failed

component.

Interconnect A physical connection between cluster nodes that transmits

intra-cluster communication, e.g. the cluster heartbeat. The Compaq

Parallel Database Cluster interconnect is dedicated to servicing

requests only between cluster nodes.

IntegratedDistributed LockManager

An Oracle software component that coordinates database locks

among cluster nodes. Sometimes abbreviated as IDLM.

IP Address Internet Protocol Address. An address assigned to a network

interface card, which computer entities use to locate and

communicate with each other. IP addresses can be statically or

dynamically assigned.

LAN Local Area Network.

Load balancing To balance the amount of work (processing) undertaken by each

node in a cluster.

Glossary 3



NIC Network Interface Controller. A board that allows a computer to be

connected with a network, and works with the network operating

system to control the flow of information over the network.

Node An individual server in a cluster.

NodeListConfigurator

A program that configures the Compaq OSDs on the nodes in a

PDC/O2000 cluster running Oracle8 Release 8.0.5 software.

NTFS New Technology File System. A file organization system by which

data is stored and accessed in a Microsoft Windows NT operating

system.

OperatingSystemDependent (OSD)layer

A collection of low-level software modules that monitors critical

clustering hardware components and constantly transmit/receive

cluster state information to and from Oracle8 Parallel Server Release

8.0.5 or Oracle8i Parallel Server Release 8.1.5.

Oracle ParallelServer Option

A software component that ships as an option of Oracle8 Enterprise

Edition Release 8.0.5 and Oracle8i Enterprise Edition Release 8.1.5.

The component is responsible for enabling parallel database

support. An additional license is required for Oracle Parallel Server.

Oracle UniversalInstaller

An installation application used to install and de-install both Oracle8i

software and the hardware vendor’s Operating System Dependent

modules for Oracle8i.

POST Power-On Self-Test. A set of operations executed every time a

system is turned on to verify that components are present and

operating.

Raw partition A disk partition that does not contain file system formatting

information. An unformatted partition. Oracle Parallel Server requires

its database files to be stored on raw partitions

RAID Redundant Array of Inexpensive Disks. A method of using hard disk

drives in an array to provide data redundancy to increase system

reliability and performance. RAID is categorized into several levels.

RedundancyManager

The Compaq software that provides cluster nodes with continuous

data access to RA4000 Arrays. Redundancy Manager monitors the

components on each FC-AL path and automatically reroutes I/O

through another path when a component on an active data path

fails.




Reliability The continuous integrity of a system (server, storage, network, or

cluster).

Resource A software or hardware entity upon which a client/server application

or service is dependent. As it pertains to the PDC/O2000, a cluster

resource must have the ability to be managed by the cluster and

must reside on one of the cluster nodes.

Scalability The ability of a system (server, storage, network, or cluster) to grow

or expand in performance, throughput, or capacity.

ServerNet A Compaq communications architecture that supports high-speed

cluster interconnect communications using ServerNet System Area

Network hardware (adapters, cables, switches), firmware, and

software drivers.

Service A data set or operation set exported by application servers to their

clients.

Shared resource A type of cluster organization in which some resources are

accessible to all systems in the cluster.

Shared storage A storage system that can be simultaneously accessed by multiple

nodes in a PDC/O2000 cluster. The RA4000 Arrays house the shared

storage for the PDC/O2000.

System A complete computer system capable of operating independently. A

system can be either a single processor system or an SMP system.

Example: a server.

UPS Uninterruptible power supply.


Part Number: 161397-000 File Name: m-index.doc Last Saved On: 10/29/99 5:48 PM

Index

Aactive/active configurations

cabling for Oracle8 5-10cabling for Oracle8i 6-11definition 2-13example with five RA4000

Arrays 2-27example with four RA4000

Arrays 2-26example with three RA4000

Arrays 2-25example with two RA4000

Arrays 2-24examples overview 2-23

active/standby configurationscabling for Oracle8 5-9cabling for Oracle8i 6-10definition 2-13example with five RA4000

Arrays 2-22example with four RA4000

Arrays 2-21example with one RA4000

Array 2-18example with three RA4000

Arrays 2-20example with two RA4000

Arrays 2-19examples overview 2-17

ACU See Compaq ArrayConfiguration Utility

addingdisk drives in RA4000

Arrays 7-12new nodes 7-19RA4000 Arrays 7-13replacement nodes 7-16

administrator privilegeswhen installing ODSs for

Oracle8i 6-27when installing OSDs for

Oracle8i 6-30when running the NodeList

Configurator 5-25application failover software 3-11Array Configuration Utility See

Compaq Array ConfigurationUtility

automated installationfor Oracle8 5-16for Oracle8i 6-19

availabilitydefinition 1-3features of Compaq ProLiant

Servers 2-3

Ccabling

active/active configuration forOracle8 5-10

active/active configuration forOracle8i 6-11




active/standby configurationfor Oracle8 5-9

active/standby configurationfor Oracle8i 6-10

client LAN for Oracle8 5-14client LAN for Oracle8i 6-17Ethernet cluster interconnect

for Oracle8 5-13Ethernet cluster interconnect

for Oracle8i 6-14Fibre Host Adapters for

Oracle8 5-4Fibre Host Adapters for

Oracle8i 6-5RA4000 Array Controllers for

Oracle8 5-8RA4000 Array Controllers for

Oracle8i 6-9ServerNet cluster

interconnect 6-15ServerNet Switches 6-16

capacity planningclient LAN 4-6cluster interconnect

components 4-5Ethernet cluster

interconnect 4-5overview 4-3ProLiant servers 4-3reference material 4-7ServerNet cluster

interconnect 4-6shared storage

components 4-4certification

Oracle8 software 3-8Oracle8i software 3-10

CIM See Compaq Insight Managerclient LAN

adapters installation forOracle8 5-6

adapters installation forOracle8i 6-7

cabling for Oracle8 5-14cabling for Oracle8i 6-17overview 2-39

planning 4-6, 4-18redundant for Oracle8 5-6,

5-13, 5-14redundant for Oracle8i 6-7,

6-15, 6-17transmissions allowed for

Oracle8 5-6transmissions allowed for

Oracle8i 6-7verifying communications for

Oracle8 5-24verifying communications for

Oracle8i 6-29verifying IP addresses for

Oracle8 5-18verifying IP addresses for

Oracle8i 6-21client reconnection software 3-11cluster communications

ServerNet clusterinterconnect 6-52

verifying before installingOSDs for Oracle8 5-24

verifying before installingOSDs for Oracle8i 6-29

verifying over client LAN forOracle8 5-31

verifying over client LAN forOracle8i 6-55

verifying over Ethernet clusterinterconnect forOracle8 5-31

verifying over Ethernet clusterinterconnect forOracle8i 6-55

verifying over ServerNetcluster interconnect 6-55

cluster configuration planningsample large PDC/O2000

cluster 4-9sample small PDC/O2000

cluster 4-8cluster connectivity,

troubleshooting 8-15cluster degradation,

troubleshooting 7-2

Index 3



cluster interconnectadapters installation for

Oracle8 5-5adapters installation for

Oracle8i 6-6defined 2-33distributed lock manager 2-33Ethernet cluster

interconnect 2-34options 2-33overview 2-33ServerNet cluster

interconnect 2-37troubleshooting 8-12types supported 1-2

cluster managementoverview 7-1remote 7-7

cluster manager, starting 6-40,6-51

cluster nodesadding a new node 7-19connected to ServerNet Switch

ports 6-16, 6-49connectivity failure 8-21eviction 7-3installing cluster software on a

new node for Oracle8 7-21installing cluster software on a

new node forOracle8i 7-22

installing cluster software on areplacement node forOracle8 7-17

installing cluster software on areplacement node forOracle8i 7-18

installing operating systemsoftware for Oracle8 5-15

installing operating systemsoftware for Oracle8i 6-18

integration 7-3IP addresses and node names

for Oracle8 5-18IP addresses and node names

for Oracle8i 6-21

monitoring operation of 7-23replacing a node 7-15shutting down cleanly 7-2supported servers 2-2

cluster operation, monitoring 7-23cluster planning

capacity planning 4-3client LAN 4-18cluster configuration 4-8disk drive planning 4-16grouping physical disk storage

space 4-15network planning 4-17non-shared disk drives 4-16overview 4-1RAID planning 4-10raw data storage and database

size 4-13selecting RAID levels 4-14shared disk drives 4-16site planning 4-2supported RAID levels 4-12

cluster softwareCompaq software

overview 3-2Oracle software 3-7, 3-9overview 3-1

cluster troubleshootingAFD.SYS failure 8-28basic tips 8-2client to cluster connectivity

problems 8-18cluster cannot communicate

with client 8-18cluster connectivity 8-15CM.DLL initialization failure

for Oracle8 8-8error starting Oracle service for

Oracle8 8-7Ethernet cluster

interconnect 8-12FC-AL connectivity

guidelines 8-19NM.DLL 8-12node cannot connect to shared

drives 8-21




NodeList Configuratorerrors 8-6

nodes cannotcommunicate 8-12

nodes unable to rejoincluster 8-16

Oracle instance creation errorfor Oracle8 8-8

Oracle8 installation 8-6Oracle8i installation 8-9OracleCMService 8-10OracleNMService 8-11other Redundancy Manager

problems 8-27pinging shows wrong IP

address 8-17RA4000 Arrays not

recognized 8-20Redundancy Manager error

messages 8-25Redundancy Manager

informationalmessages 8-22

Redundancy Manager warningmessages 8-25


summary of potentialproblems 8-1

unable to configure OPSCONFfor Oracle8 8-7

unable to start database forOracle8 8-8

unable to start database forOracle8i 8-11

unable to start Oracle servicefor Oracle8 8-7

unable to start Oracle8 ClusterManager 8-6

viping utility 8-14clusters

availability in 1-3defined 1-2overview 1-1PDC/O2000 example 1-2

role of Oracle8 Parallel Serverand Oracle8i Parallel Serverin 1-2

scalability in 1-3Compaq Array Configuration

Utilityconfiguring RA4000 Arrays for

Oracle8 5-15configuring RA4000 Arrays for

Oracle8i 6-18managing shared drives

with 7-5overview 3-3running to check for shared

disk problems B-6troubleshooting shared storage

with 8-20using to set up RA4000 Arrays

for Oracle8 5-7using to set up RA4000 Arrays

for Oracle8i 6-8Compaq authorized reseller xxCompaq Full Spectrum Fault

Management 3-4Compaq Insight Manager

monitoring cluster operationwith 7-23

monitoring server and networkhardware with 7-4

overview 3-4using Compaq Full Spectrum

Fault Managementwith 3-4

using Integrated ManagementLog with 3-4

Compaq Insight Manager XEmonitoring server and network

hardware with 7-5overview 3-4

Compaq operating systemdependent modulesadministrator privileges to

install for Oracle8i 6-27,6-30

Index 5



deinstalling a partialinstallation forOracle8i 7-11

deinstalling for Oracle8 7-7deinstalling for Oracle8i 7-8installation troubleshooting for

Oracle8 8-6installation troubleshooting for

Oracle8i 8-9installation verification for

Oracle8 5-32installation verification for

Oracle8i 6-55installing for Oracle8 5-25installing for Oracle8i 6-27mounting remote drives 6-30overview 3-5overview for Oracle8 3-6overview for Oracle8i 3-6

Compaq Options ROMPaq,overview 3-4

Compaq Parallel Database Cluster,overview 1-4

Compaq Redundancy Managerinstalling for Oracle8 5-20installing for Oracle8i 6-23monitoring cluster operation

with 7-23overview 3-5Refresh function 7-25Rescan function 7-26running to check for shared

disk problems B-5troubleshooting error

messages 8-25troubleshooting informational

messages 8-22troubleshooting other

problems 8-27troubleshooting warning

messages 8-25verifying shared storage for

Oracle8 5-21verifying shared storage for

Oracle8i 6-24

Compaq SmartStart and Supportsoftware 3-2

Compaq SmartStart and SupportSoftware CD 3-3

Compaq software overviewCompaq Array Configuration

Utility 3-3Compaq Insight Manager 3-4Compaq Insight Manager

XE 3-4Compaq operating system

dependent modules 3-5Compaq Options

ROMPaq 3-4Compaq Redundancy

Manager 3-5Compaq SmartStart and

Support software 3-2Compaq SmartStart and

Support Software CD 3-3Compaq System Configuration

Utility 3-3Fibre Channel Fault Isolation

Utility 3-3Compaq StorageWorks Fibre

Channel Host Bus Adapterscabling for Oracle8 5-4cabling for Oracle8i 6-5description 2-5installing for Oracle8 5-4installing for Oracle8i 6-4PCI slot placement for

Oracle8 5-4PCI slot placement for

Oracle8i 6-4Compaq StorageWorks Fibre

Channel Storage Hubsdescription 2-5setting up for Oracle8 5-6setting up for Oracle8i 6-7

Compaq StorageWorks RAIDArray 4000 Array Controllersactive and standby 2-4cabling for Oracle8 5-8cabling for Oracle8i 6-9




defining active controllers forOracle8 5-22

defining active controllers forOracle8i 6-25

description 2-4Compaq StorageWorks RAID

Array 4000sadding to a cluster 7-13configuring for Oracle8 5-15configuring for Oracle8i 6-18description 2-3power distribution for

Oracle8 5-33power distribution for

Oracle8i 6-57setting up for Oracle8 5-6setting up for Oracle8i 6-7

Compaq System ConfigurationUtility 3-3

Compaq website xxconfiguring

Oracle8 software 5-29Oracle8i software 6-53RA4000 Arrays for

Oracle8 5-15RA4000 Arrays for

Oracle8i 6-18creating partitions

using Disk Administrator forOracle8 5-19

using Disk Administrator forOracle8i 6-22

crossover cablesbetween Ethernet hubs or

switches for Oracle8 5-13between Ethernet hubs or

switches for Oracle8i 6-14redundant not supported for

Oracle8 5-13redundant not supported for

Oracle8i 6-14

Ddeinstallation

OSDs for Oracle8 7-7OSDs for Oracle8i 7-8partial OSD installation for

Oracle8i 7-11Disk Administrator See Windows

NT Server Disk Administratordisk drives

adding to increase storagecapacity 7-12

planning 4-16replacing in RA4000

Arrays 7-12shared 4-16

downtime 1-3

Eerror logs

for Oracle8 7-4for Oracle8i 7-4

Ethernet adaptersdescription 2-34planning 4-5

Ethernet client LANcabling for Oracle8 5-14cabling for Oracle8i 6-17

Ethernet cluster interconnectadapter installation for

Oracle8 5-5adapter installation for

Oracle8i 6-6adapters shared by cluster

interconnect and clientLAN 2-36

cabling for Oracle8 5-13cabling for Oracle8i 6-14Ethernet switch 2-35lmhosts and hosts files entries

for Oracle8 5-16lmhosts and hosts files entries

for Oracle8i 6-19overview 2-34

Index 7



recommended networkingadapters 2-34

redundant for Oracle8 5-5redundant for Oracle8i 6-6verifying communications for

Oracle8 5-24verifying communications for

Oracle8i 6-29verifying IP addresses for

Oracle8 5-18verifying IP addresses for

Oracle8i 6-21Ethernet hubs

installing for Oracle8 5-13installing for Oracle8i 6-14

Ethernet OSDsinstallation types for

Oracle8i 6-33installing for Oracle8 5-25installing for Oracle8i 6-31

Ethernet switchesdescription 2-35installing for Oracle8 5-13installing for Oracle8i 6-14planning 4-5

FFC-AL See Fibre Channel

Arbitrated LoopsFFIU See Fibre Channel Fault

Isolation UtilityFibre Channel Arbitrated Loops

installing additional loops forOracle8 5-12

installing additional loops forOracle8i 6-13

multiple loops 2-9multiple loops example 2-10redundant paths in 2-8

Fibre Channel cables, maximumdistances 2-7

Fibre Channel Fault IsolationUtilitymonitoring Fibre Channel

Arbitrated Loops with 7-5overview 3-3

Fibre Host Adapters See CompaqStorageWorks Fibre ChannelHost Bus Adapters

firmware revisions 8-5

GGBICs See Gigabit Interface

Converter-Shortwave modulesgetting help xixGigabit Interface Converter-

Shortwave modulesdescription 2-6installing for Oracle8 5-4, 5-8installing for Oracle8i 6-4, 6-9

grouping physical disk storagespace 4-15

Hhardware

installation verification forOracle8 5-31

installation verification forOracle8i 6-55

helpadditional sources xixCompaq authorized resellers,

telephone numbers xxCompaq website xxtechnical support telephone

numbers xixhosts files

for Ethernet clusterinterconnect 4-17

for Oracle8 5-16, 5-18, 5-24for Oracle8i 6-19, 6-21, 6-29for ServerNet cluster

interconnect 4-18http://www.compaq.com xx




II/O data paths

Fibre Host Adapters to StorageHubs 2-11

overview 2-11Storage Hubs to RA4000

Arrays 2-12I/O path configuration

active/active examples 2-23active/standby examples 2-17failure and failover

scenarios 2-28guidelines 2-13rules 2-16summary of features 2-14

I/O path failure and failoverscenariosfor active/active with two or

more RA4000 Arrays 2-31for active/standby with one

RA4000 Array 2-28for active/standby with two or

more RA4000 Arrays 2-29icons, symbols on equipment xviiiinstallation

additional Fibre ChannelArbitrated Loops forOracle8 5-12

additional Fibre ChannelArbitrated Loops forOracle8i 6-13

automated using SmartStart forOracle8 5-16

automated using SmartStart forOracle8i 6-19

client LAN adapters forOracle8 5-6

client LAN adapters forOracle8i 6-7

cluster interconnect adaptersfor Oracle8 5-5

cluster interconnect adaptersfor Oracle8i 6-6

Ethernet cluster interconnectfor Oracle8 5-5

Ethernet cluster interconnectfor Oracle8i 6-6

Ethernet hubs forOracle8 5-13

Ethernet hubs forOracle8i 6-14

Ethernet OSDs forOracle8 5-25

Ethernet OSDs forOracle8i 6-31

Ethernet switches forOracle8 5-13

Ethernet switches forOracle8i 6-14

Fibre Host Adapters forOracle8 5-4

Fibre Host Adapters forOracle8i 6-4

GBIC-SW modules forOracle8 5-4, 5-8

GBIC-SW modules forOracle8i 6-4, 6-9

Object Link Manager forOracle8 5-28

Object Link Manager forOracle8i 6-54

operating system software forOracle8 5-15

operating system software forOracle8i 6-18

Oracle8 software 5-24Oracle8i software 6-52OSDs for Oracle8 5-25OSDs for Oracle8i 6-27overview for Oracle8 5-2overview for Oracle8i 6-2Redundancy Manager for

Oracle8 5-20Redundancy Manager for

Oracle8i 6-23ServerNet cluster

interconnect 6-6ServerNet OSDs 6-40Service Packs for Windows NT

Server for Oracle8 5-18

Index 9



Service Packs for Windows NTServer for Oracle8i 6-21

SNMP Service 6-21, 6-28Integrated Management Log, using

with Compaq InsightManager 3-4

Llmhosts files

for Ethernet clusterinterconnect 4-17

for Oracle8 5-16, 5-18, 5-24for Oracle8i 6-19, 6-21, 6-29for ServerNet cluster

interconnect 4-18local area network (LAN),

overview 2-39log files

for Oracle8 5-32, 7-4for Oracle8i 6-56, 7-4

Mmanaging network clients 7-3Microsoft Windows NT Server

installing for Oracle8 5-16installing for Oracle8i 6-19overview 3-2versions supported 3-2

multiple redundant Fibre ChannelArbitrated Loopsexample 2-10overview 2-9

Nnet use command 6-30network clients

database access 7-3troubleshooting 8-7

network protocolsServerNet 2-37TCP/IP for Oracle8 5-16TCP/IP for Oracle8i 6-18

NodeList Configuratoradministrator privileges to

install 5-25error troubleshooting 8-6running to install OSDs 5-27

nonshared disk drives 4-16

OObject Link Manager

installing for Oracle8 5-28installing for Oracle8i 6-54rerunning and validating on

affected nodes B-4running to check for shared

disk problems B-3operating system dependent

modules See Compaqoperating system dependentmodules

operating system softwareinstalling for Oracle8 5-15installing for Oracle8i 6-18versions supported 3-2

OPS_CMDISK files 5-30, 6-53Oracle database

maintaining symbolic links forOracle8 5-28

maintaining symbolic links forOracle8i 6-54

Oracle softwarecertification for Oracle8 3-8certification for Oracle8i 3-10configuring for Oracle8 5-29configuring for Oracle8i 6-53installation troubleshooting for

Oracle8 8-6installation troubleshooting for

Oracle8i 8-9installing for Oracle8 5-24installing for Oracle8i 6-52Oracle8 Enterprise

Manager 3-8Oracle8 Parallel Server 3-7Oracle8 Server 3-7




Oracle8 Server EnterpriseEdition 3-7

Oracle8i EnterpriseManager 3-10

Oracle8i Parallel Server 3-9Oracle8i Server 3-9Oracle8i Server Enterprise

Edition 3-9overview 3-6troubleshooting the Oracle

Universal Installer 8-9Oracle Universal Installer

installing OSDs forOracle8i 6-27

overview 3-6Oracle8 Enterprise Edition,

overview 3-7Oracle8 Enterprise Manager

monitoring the databasewith 7-6

overview 3-8Oracle8 Parallel Server and

Oracle8i Parallel Server, role ina cluster 1-2

Oracle8 Parallel Server,overview 3-7

Oracle8 Performance Manager andOracle8i Performance Manager,monitoring cluster operationwith 7-23

Oracle8 Serveroverview 3-7upgrading 7-8

Oracle8i Enterprise Edition,overview 3-9

Oracle8i Enterprise Managermonitoring the database

with 7-6overview 3-10

Oracle8i Parallel Server,overview 3-9

Oracle8i Serveroverview 3-9upgrading 7-11

OracleCMServicestarting 6-40, 6-51troubleshooting 8-10

OracleNMService,troubleshooting 8-11

OSDs See Compaq operatingsystem dependent modules

OUI See Oracle Universal Installeroverview

cluster management 7-1cluster software 3-1Compaq Parallel Database

Cluster 1-4Compaq software 3-2installation for Oracle8 5-2installation for Oracle8i 6-2Oracle software 3-6Oracle8 Server Enterprise

Edition 3-7Oracle8i Server Enterprise

Edition 3-9shared disk problems B-1

PPCI slot configuration

Fibre Host Adapters forOracle8 5-4

Fibre Host Adapters forOracle8i 6-4

ServerNet PCI Adapters 6-6PDC/O2000 sample cluster 1-2PDUs See power distribution unitsping utility


planningcapacity planning 4-3client LAN 4-18cluster configuration 4-8disk drives 4-16grouping physical disk storage

space 4-15network planning 4-17nonshared disk drives 4-16overview 4-1

Index 11



RAID planning 4-10raw data storage and database

size 4-13selecting RAID levels 4-14shared disk drives 4-16shared storage

components 4-4site planning 4-2supported RAID levels 4-12

power distributionfor Oracle8 5-32for Oracle8i 6-56

power distribution unitsfor Oracle8 5-32for Oracle8i 6-56

power sequencingfor Oracle8 5-34for Oracle8i 6-58

ProLiant servershigh-availability features 2-3power distribution for

Oracle8 5-33power distribution for

Oracle8i 6-56supported servers 2-2

RRA4000 Array Controllers See

Compaq StorageWorks RAIDArray 4000 Array Controllers

RA4000 Arrays See CompaqStorageWorks RAID Array4000s

RAIDnonshared disk drives 4-16planning 4-10selecting RAID levels 4-14supported RAID levels 4-12

RAID planningoverview 4-10raw data storage and database

size 4-13selecting RAID levels 4-14supported RAID levels 4-12

Redundancy Manager SeeCompaq Redundancy Manager

redundant client LANfor Oracle8 5-6, 5-13, 5-14for Oracle8i 6-7, 6-15, 6-17

redundant Ethernet client LANusing Service Pack 3 for

Oracle8 5-14using Service Pack 3 for

Oracle8i 6-17redundant Ethernet cluster

interconnectadapter installation for

Oracle8 5-5adapter installation for

Oracle8i 6-6cabling for Oracle8 5-13cabling for Oracle8i 6-14using Service Pack 3 for

Oracle8 5-14using Service Pack 3 for

Oracle8i 6-15redundant Fibre Channel Arbitrated

LoopsI/O data paths 2-11installing additional loops for

Oracle8 5-12installing additional loops for

Oracle8i 6-13multiple loops 2-9overview 2-8

redundant ServerNet clusterinterconnect 6-16

replacingcluster nodes 7-15disk drives in RA4000

Arrays 7-12RJ-45 receptacle xviiiROMPaq See Compaq Options

Rompaq




Ssample large PDC/O2000

cluster 4-9sample small PDC/O2000

cluster 4-8Sanman device driver 6-44scalability 1-3SCSI drives 2-13SCU See Compaq System

Configuration UtilityServerNet cluster interconnect

connecting nodesdirectly 6-15

lmhosts and hosts files entriesfor Oracle8i 6-19

overview 2-37redundant 6-16, 8-15ServerNet PCI Adapters

description 2-37ServerNet PCI Adapters

installation 6-6ServerNet Switches 2-38,

6-16verifying communication

in 6-52ServerNet device drivers 6-44ServerNet IDs

configuring 6-44using in viping utility A-1

ServerNet OSDsinstallation types 6-43installing 6-40

ServerNet PCI Adapter driver 6-44ServerNet PCI Adapters

connecting directly 6-15description 2-37dual-port feature 2-37installing 6-6PCI slot configuration 6-6planning 4-6X and Y ports 6-16

ServerNet Switchescabling 6-16description 2-38dual-path topology 2-38

planning 4-6ports connected to cluster

nodes 6-16, 6-49ServerNet-I VI Protocol 6-44Service Pack 3

and redundant Ethernet clientLAN for Oracle8 5-14

and redundant Ethernet clientLAN for Oracle8i 6-17

and redundant Ethernet clusterinterconnect forOracle8 5-5

and redundant Ethernet clusterinterconnect forOracle8i 6-6

Service Packs for Windows NTServerinstalling for Oracle8 5-18installing for Oracle8i 6-21supported for Oracle8 5-16supported for Oracle8i 6-19versions supported 3-2

SETLINKS utility 6-53for Oracle8 5-28for Oracle8i 6-54

shared disk problemscontacting your Compaq

support representative B-8introduction B-1performing cluster hardware

checks B-7performing cluster software

and firmware checks B-7rerunning and validating Object

Link Manager B-4restarting all affected

nodes B-4running and validating Array

Configuration Utility B-6running and validating

Redundancy Manager B-5running Disk

Administrator B-6running Object Link

Manager B-3shutting down a cluster node 7-2

Index 13



single points of failure in non-shared disk drives 4-16

site planning 4-2SmartStart See Compaq

SmartStart and SupportSoftware

SNMP Serviceinstallation 6-21, 6-28verifying installation 6-28

SNMP SPA agent 6-44SNMP Switch agent 6-44software

application failover and clientreconnection 3-11

installation verification forOracle8 5-31

installation verification forOracle8i 6-55

installing for Oracle8 5-15installing for Oracle8i 6-18revisions 8-4

Storage Hubs See CompaqStorageWorks Fibre ChannelStorage Hubs

symbolic linksmaintaining in Oracle8

database 5-28maintaining in Oracle8i

database 6-54symbols in text xviisymbols on equipment xviiiSystem Configuration Utility See

Compaq System ConfigurationUtility

TTCP/IP

configuring for Oracle8 5-18configuring for Oracle8i 6-21network protocol for

Oracle8 5-16network protocol for

Oracle8i 6-18

required for OSDs forOracle8 5-25

required for OSDs forOracle8i 6-27

telephone numbers xxtext conventions xviitroubleshooting

AFD.SYS failure 8-28basic tips 8-2client to cluster connectivity

problems 8-18cluster cannot communicate

with client 8-18cluster connectivity 8-15cluster degradation 7-2cluster interconnect 8-12CM.DLL initialization failure

for Oracle8 8-8error starting Oracle service for

Oracle8 8-7Ethernet cluster

interconnect 8-12FC-AL connectivity

guidelines 8-19NM.DLL 8-12node cannot connect to shared

drives 8-21NodeList Configurator

errors 8-6nodes cannot

communicate 8-12nodes unable to rejoin

cluster 8-16Oracle instance creation error

for Oracle8 8-8Oracle8 installation 8-6Oracle8i installation 8-9OracleCMService 8-10OracleNMService 8-11other Redundancy Manager

problems 8-27pinging shows wrong IP

address 8-17RA4000 Arrays not

recognized 8-20




Redundancy Manager errormessages 8-25

Redundancy Managerinformationalmessages 8-22

Redundancy Manager warningmessages 8-25


summary of potentialproblems 8-1

unable to configure OPSCONFfor Oracle8 8-7

unable to start database forOracle8 8-8

unable to start database forOracle8i 8-11

unable to start Oracle servicefor Oracle8 8-7

unable to start Oracle8 ClusterManager 8-6

viping utility 8-14troubleshooting tips

accessibility of resources 8-3firmware revisions 8-5physical connections 8-2power 8-2software revisions 8-4

Uuninterruptible power supplies


upgradingOracle8 Server 7-8Oracle8i Server 7-11

UPSs See uninterruptible powersupplies

Vverifying IP addresses


viping utility 6-52error diagnostics A-2example A-2syntax and option

summary A-1

Wwarnings

electrical shock xviiirack stability xix

Windows NT Performance Monitor,monitoring cluster operationwith 7-23

Windows NT Server SeeMicrosoft Windows NT Server

Windows NT Server DiskAdministratorcreating partitions for

Oracle8 5-19creating partitions for

Oracle8i 6-22running to check for shared

disk problems B-6verifying shared storage for

Oracle8 5-20verifying shared storage for

Oracle8i 6-23www.compaq.com xix, 3-4

Date post:	15-Mar-2020
Category:	Documents
Upload:	others
View:	13 times
Download:	0 times