EMC SRDF/Cluster Enabler for MSCS · EMC SRDF/Cluster Enabler for MSCS ... Uninstalling the Cluster...

EMC SRDF /Cluster Enablerfor MSCS

Version 2.2

Product GuideP/N 300-001-286

REV A03

®®

EMC CorporationCorporate Headquarters:

Hopkinton, MA 01748-9103

1-508-435-1000www.EMC.com

2

Copyright © 2001–2006 EMC Corporation. All rights reserved.

Published November, 2006

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All other trademarks used herein are the property of their respective owners.

EMC SRDF/Cluster Enabler for MSCS Product Guide

Contents

Preface............................................................................................................................ 15

Chapter 1 IntroductionSRDF/CE overview.......................................................................... 20

SRDF/CE components.............................................................. 21What is SRDF?............................................................................ 21

Main features in SRDF/CE.............................................................. 24SRDF/CE Configuration Utility GUI ..................................... 24SRDF/CE Installation Wizard ................................................. 25SRDF/CE Configuration Wizard ............................................ 26Majority Node Set support....................................................... 26Majority Node Set with file share witness ............................. 26Forced Failover........................................................................... 27SRDF/Asynchronous compatibility ....................................... 27SRDF/CE SRDF swap support................................................ 28SRDF/CE configuration with multiple remote adapters .... 29Pushing or pulling a configuration to a node........................ 30Replacing the remote storage array ........................................ 30Autoconfiguration ..................................................................... 30Delay failback............................................................................. 31Logging enhancements ............................................................. 33Mount point support................................................................. 33

SRDF/CE: A geographic cluster system ....................................... 35SRDF/CE modes of operation ................................................. 37Pre-SRDF/CE clustering considerations................................ 40

Microsoft Cluster Service overview ............................................... 42MSCS cluster concepts .............................................................. 44MSCS modes of operation ........................................................ 45

Application software in a cluster environment ............................ 47

EMC SRDF/Cluster Enabler for MSCS Product Guide 3

Contents

Chapter 2 SRDF/CE and Cluster BehaviorSRDF/CE failover operation........................................................... 50

SRDF/CE failover and recovery behavior with MSCS........ 52SRDF/CE unique behavior ...................................................... 53Complete site failure and recovery......................................... 54

Response to complete site failure ................................................... 56Failure behavior when using MNS with file share witness........ 59

Chapter 3 Automatic Installation and ConfigurationSystem requirements ........................................................................ 62

Hardware requirements ........................................................... 62Software requirements.............................................................. 63SRDF/CE installation and operating requirements .............. 66

Install considerations ....................................................................... 67Prerequisites ............................................................................... 67Before you begin ........................................................................ 68Planning requirements.............................................................. 72

SRDF/CE Installation Wizard ........................................................ 73SRDF/CE Configuration Wizard ................................................... 80

Configuring SRDF/CE and MSCS for the first time ............ 81Converting an MSCS cluster.................................................. 125Adding an SRDF/CE node to an existing cluster............... 132

Configuring a two-node MNS cluster with file share witness. 135Downloading the Microsoft hotfix........................................ 135Creating the file share witness............................................... 135Configuring a new SRDF/CE MNS cluster......................... 136Configuring an existing SRDF/CE MNS cluster ................ 137

Uninstalling SRDF/CE .................................................................. 138

Chapter 4 Upgrading SRDF/CEGetting started................................................................................. 140

System requirements............................................................... 140Prerequisites ............................................................................. 140PC emulation software ........................................................... 140Verifying your actions ............................................................ 140

Upgrading GeoSpan clusters to SRDF/CE................................. 142Upgrading SRDF/CE V2.0.x to V2.2............................................ 146Upgrading SRDF/CE V2.1.x to V2.2............................................ 148

EMC SRDF/Cluster Enabler for MSCS Product Guide4

Contents

Chapter 5 SRDF/CE Configuration Utility GUIConfiguration Utility introduction ............................................... 150SRDF/CE for MSCS icon options ................................................. 151

Change User Credentials ........................................................ 151Configuration Wizard ............................................................. 152Discover..................................................................................... 152

Control component ......................................................................... 154Quorum Control....................................................................... 155Site Mode Operations.............................................................. 161Advanced Controls .................................................................. 163Logging Control ....................................................................... 166Logging enhancements ........................................................... 169Forced Failover......................................................................... 175

Groups component ......................................................................... 179Groups right-click menu options........................................... 182Device group right-click menu options ................................ 188

Storage component ......................................................................... 191Nodes component ........................................................................... 202

Add Cluster Member Node.................................................... 203Update Lateral/Peer Designations........................................ 205

Chapter 6 AdministrationOverview .......................................................................................... 208

SRDF/CE Configuration Utility ............................................ 209Using Cluster Administrator and Configuration Utility ... 209Multicluster configurations .................................................... 209

SRDF/CE unique cluster behavior ............................................... 210SRDF/CE administration............................................................... 211

Stopping the Cluster Service .................................................. 211Uninstalling the Cluster Service on a Windows Server 2003 cluster......................................................................................... 211Manually starting the Cluster Server Services..................... 212Modifying cluster groups ....................................................... 213Renaming groups..................................................................... 213Configuring additional disk devices..................................... 214

Restore/recovery operations......................................................... 216Restoring from site failure ...................................................... 216Recovery from site failure....................................................... 218Recovery from total IP communications failure.................. 219Recovery from HBA or SCSI channel failure....................... 220Recovery from an SRDF link failure ..................................... 220Recovery from a corrupt quorum log ................................... 222

5EMC SRDF/Cluster Enabler for MSCS Product Guide

Contents

Symmetrix replacement from the R1 side............................ 222Upgrade procedures....................................................................... 223

Upgrading Windows 2000 to Windows Server 2003 ......... 223Upgrading a quorum disk...................................................... 226

Configuration troubleshooting ..................................................... 228Overview .................................................................................. 228SRDF/CE group in wrong state (not ready/invalid/suspended).................................................... 229Unable to start cluster service................................................ 230GeoSpan_res.dll prevents MSCS group failover ................ 231GeoSpan_res.dll stays offline after failback of the quorum group ......................................................................................... 231External SYMAPI locks........................................................... 231Unable to see a newly added disk ........................................ 232

Errors and warnings....................................................................... 234GeoSpan_res.dll errors ........................................................... 234SRDF warning.......................................................................... 235IP communication warning.................................................... 235Error while attempting to connect with MSCS ................... 236MSCS error 170 ........................................................................ 236

Best practices ................................................................................... 237How to delay loading of MSCS service (change service dependencies) .......................................................................... 237Enabling groups to come online while SyncInProg ........... 239Changing the quorum log size .............................................. 239

URLs to additional information ................................................... 241

Chapter 7 Manual Installation and ConfigurationSystem requirements ...................................................................... 244Install considerations ..................................................................... 244

Prerequisites ............................................................................. 245PC emulation software ........................................................... 245Rebooting.................................................................................. 245Verifying your actions ............................................................ 245

SRDF/CE manual installation procedure ................................... 247Overview .................................................................................. 247Preinstallation .......................................................................... 247Manually installing SRDF/CE............................................... 248

Uninstalling SRDF/CE .................................................................. 272


Contents

Appendix A SRDF/CE Alerts and ErrorsSRDF/CE alerts .............................................................................. 274SRDF/CE errors and events ......................................................... 276

Interpreting Windows event log information...................... 276Event messages ............................................................................... 277

System event log ...................................................................... 277Application event log messaging .......................................... 278

Appendix B Preinstallation WorksheetPreinstallation worksheet .............................................................. 284

Glossary........................................................................................................................ 287

Index .............................................................................................................................. 301


Contents


Title Page

Figures

1 Basic SRDF configuration .............................................................................. 232 SRDF/CE COM server interfaces with SYMAPI and MMC snap-ins .... 253 Lateral and peer nodes in SRDF backup operations ................................. 324 A geographically distributed two-node SRDF/CE cluster ...................... 355 A geographically distributed four-node SRDF/CE cluster ..................... 366 Multiple SRDF/CE clusters supported on an SRDF link ......................... 377 Two-node two-cluster SRDF/CE configuration ........................................ 398 Recommended cabling configuration ......................................................... 409 A typical two-node MSCS cluster ................................................................ 4310 A typical four-node MSCS cluster ............................................................... 4411 SRDF/CE failover operation ........................................................................ 5112 Types of complete site failure ....................................................................... 5413 SRDF/CE installation and Configuration Wizard process flowchart .... 7414 Welcome window .......................................................................................... 7515 License Agreement window ......................................................................... 7616 Destination Folder window .......................................................................... 7617 Upgrade Type window ................................................................................. 7718 Ready to Install the Program window ........................................................ 7819 InstallShield Wizard Completed window .................................................. 7820 Reboot dialog box ........................................................................................... 7921 Opening the Configuration Wizard from the Configuration Utility ...... 8122 SRDF/CE Configuration Wizard welcome screen .................................... 8223 Overview of phase 1 configuration steps ................................................... 8524 Selecting the configuration mode ................................................................ 8625 Selecting the node configuration ................................................................. 8726 Designating the first node and entering cluster number ......................... 8827 Selecting local Symmetrix array ................................................................... 8928 Selecting remote Symmetrix array ............................................................... 8929 Before adding a device to the device group ............................................... 9130 After adding a device to the device group ................................................. 91


Figures

31 SRDF/Asynchronous group creation warning ......................................... 9232 Write-enabling devices .................................................................................. 9333 Selecting local IP address .............................................................................. 9434 Running Disk Manager ................................................................................. 9535 Disk Management utility .............................................................................. 9636 Write Signature and Upgrade Disk Wizard ............................................... 9737 Selecting the hostname where first node is located .................................. 9838 Retrieving groups from the first node and write-enabling devices ........ 9939 RDF State of Devices message ...................................................................... 9940 Selecting local IP address for an intermediate or last node ................... 10041 Running Disk Manager on an intermediate or last node ....................... 10142 Overview of phase 2 configuration steps for first node ......................... 10443 Selecting the configuration mode in phase 2 ........................................... 10544 Adding a cluster member node ................................................................. 10645 Selecting the cluster type ............................................................................ 10746 Assigning quorum control settings ........................................................... 10947 Assigning site failure mode and advanced control settings .................. 11048 End of phase 2 for node 1 ............................................................................ 11249 Phase 2 configuration steps for intermediate or last node ..................... 11350 Selecting the configuration mode in phase 2 ........................................... 11451 Retrieving phase 2 options from node 1 ................................................... 11552 Viewing cluster type .................................................................................... 11653 Viewing quorum settings ............................................................................ 11754 Identifying if node is last node ................................................................... 11855 Overview of phase 3 configuration steps ................................................. 11956 Installing MSCS ............................................................................................ 12057 Post-MSCS configuration activities ........................................................... 12258 End of phase 3 configuration for the last SRDF/CE node ..................... 12359 Applying MSCS information to SRDF/CE groups ................................. 12960 Assigning site mode failure and advanced control settings .................. 13061 Retrieving group and cluster information from the first node ............. 13462 Creating a checkpoint in the GeoSpan Configuration Utility ............... 14363 Selecting discover in the SRDF/CE Configuration Utility .................... 14464 Restoring the checkpoint in the SRDF/CE Configuration Utility ........ 14465 SRDF/CE Configuration Utility GUI ........................................................ 15066 SRDF/CE for MSCS icon right-click menu .............................................. 15167 Changing user credentials .......................................................................... 15268 Issuing a discover through the Configuration Utility GUI .................... 15369 Issuing a manual SYMCLI discover command ....................................... 15370 Expanded control view in the SRDF/CE Configuration Utility ........... 15471 Control right-click menu ............................................................................. 15572 Quorum Control expanded view ............................................................... 15673 Quorum Control expanded view (continued) ......................................... 156


Figures

74 Selecting edit Quorum Control settings .................................................... 15875 Changing quorum properties ..................................................................... 15976 Pushing a new quorum group definition to MSCS ................................. 16077 Quorum Control right-click menu ............................................................. 16178 Selecting the quorum type .......................................................................... 16179 Expanded site mode operations view and right-click menu ................. 16280 Changing site mode operations parameters ............................................. 16381 Expanded advanced control view and right-click menu ........................ 16382 Changing Advanced Control parameters ................................................. 16583 Expanded SRDF/CE service logging view ............................................... 16684 Editing log level parameters ....................................................................... 16785 Editing log file path ...................................................................................... 16786 Editing the log file path ............................................................................... 16887 Viewing the latest log file ............................................................................ 16888 SRDF/CE Log Viewer .................................................................................. 16989 Expanded Forced Failover view and right-click menu ........................... 17790 Enabling the Forced Failover option ......................................................... 17891 Expanded groups view with the device groups list ................................ 17992 Expanded groups view with the device groups list (continued) .......... 17993 Device groups detail view ........................................................................... 18094 Device groups detail view (continued) ..................................................... 18095 Groups right-click menu .............................................................................. 18296 Creating a new device group ...................................................................... 18397 Adding a device to the device group ......................................................... 18498 Creating an SRDF/Asynchronous-enabled device group ..................... 18599 SRDF/Asynchronous group creation warning ........................................ 185100 Async devices in consistent state warning ............................................... 186101 Pulling groups from SYMCLI ..................................................................... 187102 Listing device groups ................................................................................... 187103 Device group right-click menu ................................................................... 188104 Release Symmetrix locks warning ............................................................. 188105 Array Operations dialog box ...................................................................... 189106 Expanded storage view ............................................................................... 191107 Symmetrix device detail view ..................................................................... 192108 Symmetrix device detail view (continued) ............................................... 192109 Selecting columns to view ........................................................................... 194110 Modifying R1/R2 device columns ............................................................. 195111 Adding and removing R1/R2 device rows .............................................. 195112 Drag and drop a device into a device group ............................................ 196113 Deleting a device from a device group ...................................................... 197114 Storage right-click menu .............................................................................. 198115 Listing Symmetrix information .................................................................. 198116 Unlocking Symmetrix arrays ...................................................................... 199


Figures

117 Selecting the remote Symmetrix array ...................................................... 200118 Configuring the remote storage array ....................................................... 200119 Completing the replacement of the remote storage array ..................... 200120 Expanded node view ................................................................................... 202121 Nodes right-click menu ............................................................................... 203122 Adding a host node ..................................................................................... 203123 Individual peer node right-click menu ..................................................... 204124 Selecting Update Lateral/Peer Designations ........................................... 205125 Windows Server 2003 Enterprise Edition Setup dialog box .................. 224A-1 SRDF/CE alerts display .............................................................................. 274


Title Page

Tables

1 Cluster mount point example ........................................................................342 Quorum Control column headings ............................................................1573 Groups column headings .............................................................................1814 R1/R2 device description column headings .............................................1935 Manual installation: Two-node installation procedure ...........................2496 System event log descriptions .....................................................................2777 Application event log descriptions .............................................................279


Tables


Preface

As part of an effort to improve and enhance the performance and capabilities of its product line, EMC from time to time releases revisions of its hardware and software. Therefore, some functions described in this document may not be supported by all revisions of the software or hardware currently in use. For the most up-to-date information on product features, refer to your product release notes.

If a product does not function properly or does not function as described in this document, please contact your EMC representative.

Audience This guide is part of the EMC SRDF/CE for MSCS (referred to herein as SRDF/CE) documentation set and is intended for use by system administrators during installation, system setup, and routine operations.

System administrators working with SRDF/CE must be proficient in the use of the following products:

◆ Microsoft products:

• Windows Server 2003 Enterprise and Datacenter Editions

• Microsoft Cluster Services (MSCS)

◆ EMC products:

• EMC Solutions Enabler (SYMCLI/SYMAPI)

• EMC ControlCenter Symmetrix Remote Data Facility (SRDF) Manager, if installed

• EMC PowerPath, if installed


16

Preface

Relateddocumentation

The following documentation from EMC Corporation contains information that may be helpful in an SRDF/CE clustered environment:

EMC ControlCenter

◆ EMC ControlCenter Installation Planning Booklet ◆ EMC ControlCenter Guidebook ◆ EMC ControlCenter Installation Guide ◆ Symmetrix SRDF Host Component Product Guide

EMC Solutions Enabler

◆ EMC Solutions Enabler Symmetrix Array Management CLI Product Guide

◆ EMC Solutions Enabler Symmetrix SRDF Family CLI Product Guide◆ EMC Solutions Enabler Symmetrix TimeFinder Family CLI Product

Guide◆ EMC Solutions Enabler Installation Guide

EMC PowerPath

◆ EMC PowerPath Version Product Guide◆ EMC PowerPath Version Product Overview white paper

Fibre Channel

◆ Symmetrix Fibre Channel Product Guide

Related third-partydocumentation

The following Microsoft documentation contains information about or related to the products discussed in this guide:

◆ Microsoft’s Cluster Server documentation set, especially the Microsoft Cluster Server Administrator’s Guide, Document No. X0327902

◆ Step-by-Step Guide to Installing Cluster Service, Microsoft Technical Library

◆ Introducing Windows 2000 Clustering Technologies, Microsoft Technical Library


Preface

Typographicalconventions

The following type style conventions are used in this guide:

Where to get help EMC support, product, and licensing information can be obtained as follows.

Product information — For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to the EMC Powerlink website (registration required) at:

http://Powerlink.EMC.com

Technical support — For technical support, go to EMC WebSupport on Powerlink. To open a case on EMC WebSupport, you must be a WebSupport customer. Information about your site configuration and the circumstances under which the problem occurred is required.

Your commentsYour suggestions will help us continue to improve the accuracy, organization, and overall quality of the user publications. Please send your opinion of this guide to:

[email protected]

bold • User actions (what the user clicks, presses, or selects)• Interface elements (button names, dialog box names)• Names of keys, commands, programs, scripts, applications,

utilities, processes, notifications, system calls, services, applications, and utilities in text

italic • Book titles• New terms in text• Emphasis in text

Courier • Prompts • System output • Filenames • URLs • Syntax when shown in command line or other examples

Courier, bold • User entry• Options in command-line syntax

Courier italic • Arguments in examples of command-line syntax• Variables in examples of screen or file output• Variables in pathnames


http://powerlink.emc.com

18

Preface


1Invisible Body Tag

This chapter provides a high-level overview of clustering, Microsoft Cluster Services (MSCS), Symmetrix Remote Data Facility (SRDF), and explains how EMC SRDF/CE V2.2 for MSCS provides disaster recovery protection in geographically distributed clusters.

Important: EMC recommends reading this chapter in its entirety before installing and configuring SRDF/CE V2.2 for MSCS.

◆ SRDF/CE overview ........................................................................... 20◆ Main features in SRDF/CE............................................................... 24◆ SRDF/CE: A geographic cluster system......................................... 35◆ Microsoft Cluster Service overview ................................................ 42◆ Application software in a cluster environment ............................. 47

Introduction

Introduction 19

20

Introduction

SRDF/CE overviewEMC® SRDF®/Cluster Enabler (CE) for Microsoft Cluster Service (MSCS) is a software extension (MMC snap-in) for MSCS. EMC SRDF/CE V2.2 for MSCS (referred to herein as SRDF/CE) allows Windows Server 2003 Enterprise and Datacenter editions running MSCS to operate across a single pair of SRDF1-connected Symmetrix® arrays in geographically distributed clusters. These clusters are referred to as SRDF/CE clusters. SRDF/CE software can extend the Symmetrix enterprise system to support up to 64 MSCS clusters per Symmetrix pair.

Using an SRDF link, SRDF/CE expands the range of cluster storage and management capabilities while ensuring full business continuance protection. A SCSI or Fibre Channel connection from each cluster node is made to its own Symmetrix array. Two Symmetrix arrays are connected through SRDF to provide automatic failover of SRDF-mirrored volumes during MSCS node failover. This connection effectively extends the distance between cluster nodes up to 60 kilometers and forms a geographically distributed cluster with disaster-tolerant capabilities.

SRDF/CE protects data from the following types of failures, 24x7x365 days per year:

◆ Storage failures◆ System failures◆ Site failures

1. Symmetrix Remote Data Facility (SRDF). For more information on SRDF, refer to “What is SRDF?” on page 21.


Introduction

SRDF/CE componentsSRDF/CE integrates MSCS software with EMC Symmetrix hardware and SYMAPI software, allowing the seamless use of Symmetrix disks to function as a single SCSI disk. SRDF/CE achieves this using several components:

◆ SRDF/CE Configuration Utility — An MMC-based (Microsoft Management Console) user interface that allows you to:

• Configure operational parameters• Monitor the SRDF/CE Service• Modify logging settings

◆ COM Server — Component that handles all direct interfaces to the SYMAPI from the MMC snap-in and all intra-SRDF communication.

◆ SRDF/CE Service — The main control service mechanism.

◆ SRDF/CE Resource.dll — A .dll used by MSCS to perform group failover/failback operations for all group resources.

◆ Quorum Filter Driver — Component that performs arbitration or ownership protocol for the MSCS database, the quorum.

What is SRDF? SRDF is a Symmetrix-based business continuance and disaster recovery solution sold as a separate license by EMC Corporation. In basic terms, SRDF is a configuration of multiple Symmetrix arrays whose purpose is to maintain multiple, realtime copies of logical volume data in more than one location.

SRDF duplicates production (source) site data to a recovery (target) site transparently to users, applications, databases, and host processors. If the primary site is not able to continue processing, data at the secondary site is current up to the last I/O transaction.

SRDF can be used in several key areas including, but not limited to:

◆ Disaster Recovery◆ Remote Backup◆ Data Center Migration◆ SDMS—Symmetrix Data Migration Service◆ Data Center Decision Solutions

SRDF/CE overview 21

22

Introduction

When primary (source) systems are down, SRDF enables fast switchover to the recovery (target) copy of the data, allowing critical information to become available in minutes. Business operations and related applications may resume full functionality with minimal interruption.

Protecting against data loss allows the operations and applications to resume at the secondary site. SRDF can be used:

◆ By itself, and data processing can be resumed by powering up a standby system and manually restarting.

◆ In combination with more sophisticated software to automatically resume operations.

Figure 1 on page 23 illustrates a basic SRDF configuration.

SRDF/CE combines MSCS and SRDF to provide a more sophisticated solution. SRDF/CE provides both a manual configuration utility (refer to Chapter 5, ”SRDF/CE Configuration Utility GUI”) and an automated configuration wizard (refer to Chapter 3, ”Automatic Installation and Configuration”) to be used in conjunction with the MSCS Cluster Administrator to administer the SRDF-enabled cluster.

Note: For greater detail on SRDF, consult the SRDF documentation set (refer to “Related documentation”).


Introduction

Figure 1 Basic SRDF configuration

Site A(production)

Site B(recovery)

LocalHost

RemoteHost

SRDF Links

SymmetrixA

SymmetrixB

RecoveryPath

Active HostPath

Local volume Source volume Target volume

SRDF/CE overview 23

24

Introduction

Main features in SRDF/CEThe following are the main features of SRDF/CE:

◆ “SRDF/CE Configuration Utility GUI”◆ “SRDF/CE Installation Wizard” on page 25◆ “SRDF/CE Configuration Wizard” on page 26◆ “Majority Node Set support” on page 26◆ “Forced Failover” on page 27◆ “SRDF/Asynchronous compatibility” on page 27◆ “SRDF/CE SRDF swap support” on page 28◆ “SRDF/CE configuration with multiple remote adapters” on

page 29◆ “Pushing or pulling a configuration to a node” on page 30◆ “Replacing the remote storage array” on page 30◆ “Autoconfiguration” on page 30◆ “Delay failback” on page 31◆ “Logging enhancements” on page 33

SRDF/CE Configuration Utility GUIThe SRDF/CE Configuration Utility GUI is an MMC snap-in. This MMC snap-in GUI consists of two main components: the snap-in itself and a COM Server.

The SRDF/CE COM Server functions as the interface between SYMAPI and the MMC snap-ins for information and control requests (both local and remote). The COM Server also makes SRDF/CE configuration changes to the Registry. The COM Server performs all other control operations of the Symmetrix array via SYMAPI calls.

Figure 2 on page 25 shows how both the local and remote SRDF/CE MMC snap-ins and the SRDF/CE Installation Wizard interface to the SRDF/CE COM Server to access information about the Symmetrix array.


Introduction

Figure 2 SRDF/CE COM server interfaces with SYMAPI and MMC snap-ins

The MMC snap-in is the presentation component of the SRDF/CE Configuration Utility GUI, and the COM Server is the control and status mechanism for the MMC client. The SRDF/CE COM Server handles actual configuration changes, status information, and control of Symmetrix devices.

Note: For more information on the Configuration Utility, refer to Chapter 5, ”SRDF/CE Configuration Utility GUI.”

SRDF/CE Installation WizardThe SRDF/CE Installation Wizard is an InstallShield-based application that copies the SRDF/CE code in a user-designated location and provides a shortcut for the SRDF/CE MMC snap-in. No actual processing or authentication occurs during this process.

The SRDF/CE Installation Wizard should be invoked when:

◆ Installing SRDF/CEor

◆ Upgrading/removing SRDF/CE

SRDF/CE COMServer

SRDF/CE SYMAPI DB file

MMC Console

SRDF/CE MMCSnap-in

Registry

SRDF/CE Configuration

Wizard

MMC Console

Local Host

Remote SRDF/CECOM Server

Remote SRDF/CECOM Server

1 to 7 Remote Nodes

Main features in SRDF/CE 25

26

Introduction

After running the Installation Wizard, executing the SRDF/CE MMC snap-in will load the SRDF/CE Configuration Utility GUI.

Note: For more information on the Installation Wizard, refer to Chapter 3, ”Automatic Installation and Configuration.”

SRDF/CE Configuration WizardThe SRDF/CE Configuration Wizard simplifies the SRDF/CE configuration process by automating the configuration of SRDF/CE and facilitating the installation of MSCS on all nodes in your system. Configuration settings must be applied on a node-by-node basis.

Note: For more information on the SRDF/CE Configuration Wizard, refer to Chapter 3, ”Automatic Installation and Configuration.”

Majority Node Set supportMicrosoft Majority Node Set (MNS) provides an alternative quorum resource for clusters with nodes in separate geographic sites. It is useful for clusters with physically separate storage subsystems that implement mirroring to replicate data. MNS enables you to create a server cluster without any shared disks for the quorum resource. SRDF/CE allows you to configure an MNS cluster on Windows Server 2003 Enterprise and Datacenter Editions.

If you are configuring your cluster on Windows Server 2003, the Quorum Control right-click menu in the SRDF/CE Configuration Utility allows you to choose the cluster quorum type you want to configure: a quorum disk-based cluster (Shared Quorum) or a Microsoft Majority Node Set (MNS) cluster.

Majority Node Set with file share witnessIn failover cluster configurations, you can use Microsoft Windows Cluster service and its new type of cluster model known as Majority Node Set (MNS) with file share witness. This model allows you to use a file share that is external to the cluster as an additional vote to determine the status of the cluster in a two-node MNS cluster deployment. That is, the witness is used by the two cluster nodes as a note-board to track which node is in control of the cluster.


Introduction

Forced FailoverThe Forced Failover option is a configurable parameter found under the Control component in the SRDF/CE Configuration Utility. This option provides an automatic way to keep the cluster running on a particular Symmetrix array or arrays in the event of a total site disaster.

The Forced Failover option performs differently depending on whether the MSCS cluster is defined as a quorum disk-based cluster or an MNS-based cluster. For quorum disk-based clusters, the node that owns the quorum decides what nodes will stay up. For MNS-based clusters, you can configure a list of nodes that will be forced up in a site disaster.

Note: MNS is only supported on Windows Server 2003 Enterprise and Datacenter Editions. For more information on the Forced Failover option, refer to “Forced Failover” on page 175.

SRDF/Asynchronous compatibilitySRDF/CE is compatible with EMC SRDF/Asynchronous (SRDF/A). SRDF/A is a high-performance, extended-distance asynchronous replication that uses a delta set architecture for reduced bandwidth requirements and no host performance impact.

Asynchronous mode provides a point-in-time image on the target (R2) device that is only slightly behind the source (R1) device. SRDF/A session data is transferred to the remote Symmetrix system in delta sets, eliminating the redundancy of same-track changes being transferred over the link, thereby reducing the required bandwidth. SRDF/A only needs enough bandwidth to support the average production workload versus peak workloads, provided there is enough Symmetrix cache to support the peak workloads.

SRDF/A is intended for users who require no host application impact while maintaining a consistent, restartable image of their data on the R2 side at all times.

Note: Beginning with version 2.2, SRDF/CE for MSCS always enables consistency on SRDF/A groups. SRDF/A consistency ensures that applications have a consistent copy on the remote side when they failover.


Introduction

SRDF/CE supports Enginuity™ (microcode) releases as outlined in the EMC Support Matrix. At the 5x70 Enginuity level, you can specify a single SYMCLI group whose device members have been previously defined as SRDF/A enabled. Once configured, SRDF/CE automatically fails over this group to the target side as necessary. After the failover, when you want to fail back, SRDF/CE automatically establishes a synchronous connection, waits until the data is propagated back, and then changes the SRDF mode back to asynchronous.

Note: SRDF/CE does not support clusters where the target (R2) side is larger than the source (R1) side. When the system fails over to the R2 side, it can never fail back since the R2 cannot resynchronize all its data back to the R1 side.

SRDF/CE SRDF swap supportAn R1/R2 personality swap (or R1/R2 swap) refers to swapping the RDF personality of the RDF device designations of a specified device group, so that source R1 device(s) become target R2 device(s) and target R2 device(s) become source R1 device(s).

R1/R2 RDF swaps are available with Enginuity Version 5567 or higher. There are two types of R1/R2 swaps: FastSwap and Dynamic Swap. A FastSwap occurs immediately after failover if the group is fully synchronized. A Dynamic Swap takes longer because after failover, the tracks are checked to determine if they are synchronized, and then the swap occurs. If you enable an R1/R2 swap for a group, SRDF/CE automatically checks during a failover to determine whether FastSwap is available. If FastSwap is available, SRDF/CE will use it. If FastSwap is not supported, SRDF/CE will automatically use Dynamic Swap.

Using the R1/R2 swapThis section describes several scenarios in which it is beneficial to execute an R1/R2 swap.

Symmetrix load balancingIn today's rapidly changing computing environments, it is often necessary to deploy applications and storage on a different Symmetrix array without having to lose disaster protection. R1/R2 swap can enable this redeployment with minimal disruption, while

28 EMC SRDF/Cluster Enabler for MSCS Product Guide

Introduction

offering the benefit of load balancing across two Symmetrix storage arrays.

For example, if you want to reconfigure an SRDF/CE environment after having decided where the R1 and R2 devices will sit, this procedure will allow you to go from an active/passive configuration to active/active.

Primary data center relocationSometimes a primary data center needs to be relocated to accommodate business practices. For example, several financial institutions in New York City routinely relocate their primary data center across the Hudson River to New Jersey as part of their disaster drills. R1/R2 swaps allow these customers to run their primary applications in their New Jersey data centers. The Manhattan data centers then acts as the disaster protection site.

Post-failover temporary protection measureYou can regain a measure of protection after failing over to the remote site. If the hosts on the source side are down for maintenance, R1/R2 swap permits the relocation of production computing to the target site without giving up the security of remote data protection. When all problems are solved on the local Symmetrix array, fail over again and swap the personality of the devices to return to the original configuration.

SRDF/CE configuration with multiple remote adaptersSRDF/CE can be configured with multiple RDF links and remote adapter (RA) groups. SRDF/CE not only allows multiple RAs, but periodically tests them to ensure they are functioning. Multiple RA groups are also allowed, and these RA groups do not have to be symmetrical across all RDF links; any one RA group can be allocated over a subset of the defined RDF links.

If a situation occurs where an RDF link goes down, an Event Log message is posted and an entry is placed in the SRDF/CE log. In the case where an RA group can no longer be failed over because there are no RDF links up to support that operation, an SRDF/CE pop-up event in the System Tray is reported as well as an entry in the SRDF/CE log.


30

Introduction

Pushing or pulling a configuration to a nodeSRDF/CE allows new or reconfigured nodes to have a valid configuration pushed (that is, propagated) or pulled (that is, retrieved) to that node. This process completely deletes the entire existing SRDF/CE configuration on that target node and replaces it with the configuration the user specifies as valid. This process does not affect the operation of other nodes in the SRDF/CE cluster.

Replacing the remote storage arraySRDF/CE has the automated ability to change the remote storage array in the case of a remote site disaster, lease swap, or other event where the remote storage array must be replaced. This change is done without disruption to the running local cluster nodes. Once the local cluster nodes are reconfigured using this feature, the remote nodes must each have a configuration pushed or pulled to them for them to recognize the new storage array.

These configuration changes require that the source/target Symmetrix device remapping is changed prior to invoking this function. In addition, the node from which the Replace Storage Array function is invoked will be used as the validation node. The validation node is used to validate each local lateral node device configuration and source/target mapping.

AutoconfigurationThe SRDF/CE COM Server allows the SRDF/CE configuration information to be propagated among cluster peers through the Autoconfiguration utility. Proper time synchronization for all host nodes is critical to ensure the proper functioning of Autoconfiguration. EMC recommends all servers in an SRDF/CE cluster be based on GMT and time servers be employed.

When you open the SRDF/CE snap-in, the timestamps of all nodes are validated. If the timestamp of the nodes is older than the current configuration, you will be prompted to update the configuration. You must decide whether the node you are on has the correct configuration. The Autoconfiguration utility then overloads the local configuration parameters with the newer information.


Introduction

The Autoconfiguration utility specifies the following configuration information about SRDF/CE:

◆ Quorum Control — Quorum Group Name, Quorum Drive Letter (the physical drive is included, but not used by AutoConfig)

◆ Site Mode — Site Mode setting for node

◆ Advanced Information — Cluster Number, Delay Failback, AutoDel Release, and SRDF/A Failover Method settings

◆ Device Groups — Group names and devices

Delay failbackThe Delay Failback feature is an option found in the Advanced Control menu and is propagated as part of the Autoconfiguration functionality. This feature modifies the Preferred Owner list for each MSCS cluster group so a failover will occur to a lateral node first, and if the lateral node is unavailable, to a peer node. Lateral nodes are defined as nodes connected to the same Symmetrix array. Peer nodes are defined as nodes connected to different Symmetrix arrays located across the SRDF link from each other (refer to Figure 3 on page 32).

SRDF/CE manipulates the MSCS Preferred Owners list whenever a group is brought online. SRDF/CE then examines the group Preferred Owners list and determines which node is the lateral node. It can then modify the MSCS Preferred Owner list so the current node and its lateral partner are the first two in the list.

Therefore, no matter which side (R1 or R2) a group is moved to, the Preferred Owner list is modified to allow a group to fail over to a lateral node, and not fail back or fail over across the SRDF link as a first option. MSCS only moves a group across the SRDF link as a last resort. This prevents MSCS from arbitrarily performing what amounts to a failback/failover across the SRDF link in an automatic fashion. This feature delays the actual failback of a group to an R1 node, and is therefore termed delay failback.


32

Introduction

Figure 3 Lateral and peer nodes in SRDF backup operations

The Delay Failback feature runs simultaneously on all nodes. Therefore, when a group comes online on any node, the Preferred Owner list is updated, regardless of whether it is an R1 or R2 device. For example, the sequence for Group x on Node 1 will be the following:

1. The Delay Failback feature first determines if it knows the other nodes in the four-node cluster. This information is gathered by the SRDF/CE COM Server during normal operations. If not, the feature is bypassed because it is unable to differentiate between a lateral node, peer node, etc.

2. If the Delay Failback feature knows the other nodes, then it determines if Group x has come online on Node 1.

3. If Group x has come online on Node 1, then the MSCS Preferred Owner list is modified so that Node 1 is the first Preferred Owner, followed by the lateral node and then the peer nodes.

SRDF Link

Symmetrix BSymmetrix A

Site BSite A

Lateral Nodes Lateral Nodes

Peer Nodes

Peer Nodes

Primary (R1) Side Secondary (R2) Side

Node 1 Node 2

Node 4Node 3


Introduction

Logging enhancementsBeginning with SRDF/CE for MSCS Version 2.1, new logging features provide more detailed logging. If there is a problem with SRDF/CE for MSCS, detailed logs provide EMC Customer Support with the technical information necessary to help diagnose the problem and help SRDF/CE engineers with debugging.

The logging facility captures detailed information, and the size and number of log files being created has increased compared to prior versions of SRDF/CE for MSCS.

To help minimize disk capacity utilized by log files, log compression and log retention are two new features that address these issues.

Refer to “Logging enhancements” on page 169 for more details.

Mount point supportBeginning with SRDF/CE for MSCS Version 2.2, , SRDF/CE includes support for mount points. One of its benefits is that mount points overcome the limitation on drive letters, making it possible for a cluster to support more than 26 volumes.

For mount points to work correctly, all related disks must belong to the same MSCS group and the same RA group. If related disks are spread across multiple MSCS groups, volumes cannot be brought online because MSCS groups can be online on different nodes. To avoid this scenario, SRDF/CE first groups all related disks by identifying the mount points on a given disk and any disks upon which the given disk is mounted. SRDF/CE then creates a parent/child relationship between disks.

When a user chooses a disk to create a group (or adds a disk to an existing group), SRDF/CE finds all related disks by traversing its parent/child relationships and adding every related disk to the group. It then adds appropriate dependencies between the disks so that the resources can be brought online in an orderly fashion.

Note: Because the volume and its mount points are not mounted if the underlying disk is read-only, you can create groups or add/delete devices to/from a group only on the side that has disks write-enabled.


34

Introduction

Table 1 illustrates a cluster example consisting of drive letters and mount points for six volumes. Using this configuration, you can see various parent/child relationships among the disks.

For example, the user chooses E:\MNT1. Therefore:

◆ E:\MNT1 is a mount point with E:\ as its parent.

◆ E:\ is a child of C:\. Thus, disk C:\will be included in the group.

◆ C:\ has additional children C:\MNT2 and C:\MNT2\MNT3. Thus, the group will include these disks too.

The result of these parent/child relationships is that the group will include volumes 0BCE, 0BCF, 0BD0, 0BD1, and 0BD2. Each disk is dependent on its parent to come online. In this example, 0BCF is dependent on 0BCE, and 0BD0 is dependent on 0BCE, and so forth.

Of course, each group is also dependent on the SRDF/CE resource.

Table 1 Cluster mount point example

When you delete a device, SRDF/CE finds all related disks and deletes them too. For example, if the current mount points are C: and C:\MNT2 and C:\MNT2\MNT3, and if the device that corresponds to C:\MNT2 is deleted from the SRDF/CE group, all three devices corresponding to C:\, C:\MNT2, and C:\MNT2\MNT3 are deleted.

However, if you were to first delete mount point C:\MNT2 from the operating system and then delete its corresponding device from the group, SRDF/CE would delete only the devices that correspond to C:\MNT2 and C:\MNT2\MNT3. The device corresponding to C:\ would be left in the group because, after the mount point deletion, it is no longer related to C:\MNT2.

Drive letter and/or mount point Symmetrix volume ID

C:\ 0BCE

C:\MNT1, E:\ 0BCF

C:\MNT2 0BD0

C:\MNT2\MNT3 0BD1

D:\ 0BCD

E:\MNT1 0BD2


Introduction

SRDF/CE: A geographic cluster systemSRDF/CE provides disaster-tolerant capabilities that enable the cluster servers to be geographically separated by distances of up to 60 kilometers, or where the network connection provides a guaranteed maximum round-trip latency between nodes of no more than 300 milliseconds. Since many servers can connect to one Symmetrix array, it is possible to implement many clusters across this distance.

Figure 4 illustrates the hardware configuration of a two-node SRDF/CE cluster solution.

Figure 4 A geographically distributed two-node SRDF/CE cluster

Private Interconnect (Heartbeat Connector)

Enterprise LAN/WAN

SymmetrixArray 1

SymmetrixArray 2

Clients

R1 R2

R1R2

BidirectionalSRDF Interconnect

Host ANode 1

Host BNode 2

Ultra Wide SCSIor

Fibre Channel

Ultra Wide SCSIor

Fibre Channel

(Distance of up to 60 km)

SRDF/CE: A geographic cluster system 35

36

Introduction

Figure 5 illustrates the hardware configuration of a four-node SRDF/CE cluster solution.

Figure 5 A geographically distributed four-node SRDF/CE cluster

SRDF/CE provides disaster-tolerant capabilities by exploiting SRDF capabilities. SRDF allows two Symmetrix arrays to be attached using direct-connect fiber at distances up to 60 kilometers. The SRDF group when built is established in synchronous or asynchronous mode. Once attached, mirrored pairs of disks are established on the devices, with R1 (read/write) disks on one Symmetrix array and R2 (read-only) mirrors on the other arrays. Additionally, four-way mirroring can be implemented, which locally mirrors each member of the R1/R2 mirror pair.

Symmetrix A Symmetrix B

SRDF Link

Fibre ChannelHub/Switch

Node 1

Node 4Node 3

Node 2Fibre Channel

Hub/Switch

VLAN Switch VLAN Switch

Site APrimary

Site BSecondary

Private Interconnect

Enterprise LAN/WAN

Clients


Introduction


A Symmetrix array can support multiple MSCS clusters on a single Symmetrix array. Likewise, multiple SRDF/CE clusters can be supported on an SRDF link (refer to Figure 6).

Figure 6 Multiple SRDF/CE clusters supported on an SRDF link

SRDF/CE modes of operationSRDF/CE supports 8 node clusters Windows Server 2003 Enterprise and Datacenter Editions.

Different cluster designs support different modes of operation and data sharing mechanisms. The configuration for an SRDF/CE two-node or multinode cluster in a geographically distributed cluster environment is either active/passive or active/active. EMC defines active/passive and active/active configurations as follows:

Site A Site B

LAN

SRDF Link

Cluster 1

Cluster 2Cluster 3

Cluster 2

Cluster 3

Cluster 1

Cluster 1Cluster 2

Cluster 3


38

Introduction

◆ Active/Passive: A cluster of two nodes or more where all processing is done on one node during normal operation and the work is picked up by a remaining passive node (or nodes) only when a failure occurs on the active node. In a two-node configuration, half of the hardware is normally idle. When failover occurs, the application restarts with full performance.

Note: Active/passive multinode clustering provides greater flexibility than the standard active/passive MSCS two-node cluster by providing more options in resolving failures and load distribution after server failures. For example, in a multinode cluster, your configuration may include one or more passive (idle) servers to take over the load from other servers during a site failure, or you may distribute the load among the surviving active nodes.

◆ Active/Active: A cluster of two nodes or more where all nodes are running application software during normal operation. When a failure occurs on a node (or nodes), the work is transferred to a remaining node (or nodes) and restarted. The node(s) that picks up the work must then handle the processing load of both systems, and performance is usually degraded. However, all the computer hardware is used during normal operation.

Note: The terms active/active and active/passive apply to the cluster and to the applications running on the cluster. Both the cluster software and the application software must be designed for active/active operation.

Figure 7 presents a typical SRDF/CE two-node two-cluster configuration.


Introduction

Figure 7 Two-node two-cluster SRDF/CE configuration

Symmetrix A Symmetrix B

SRDF/Synchronous Link



VLAN Switch VLAN SwitchExtended IP Subnet

Cluster 1/Host 2

Cluster 2/Host 1Cluster 2/Host 2

Cluster 1/Host 1


40

Introduction

Pre-SRDF/CE clustering considerationsTo ensure disaster recovery protection in an SRDF/CE-enabled cluster, consider the following prior to its installation and configuration:

◆ Cabling

◆ Booting

◆ SRDF coexistence

Cabling Avoid routing all cables through the same path, both in buildings and between sites. To provide an installation with no single point of failure, use a configuration similar to Figure 8.

Figure 8 Recommended cabling configuration

Symmetrix

Clients

Symmetrix

Path 1

Node 1 Node 2

LAN ALAN B

Path 2


Introduction

Booting Currently, MSCS can only boot from separate private disks. (MSCS cannot boot off the same bus.) Therefore, SRDF/CE nodes must contain an internal disk for booting or be attached to a nonclustered Symmetrix disk.

SRDF coexistence Multiple SRDF/CE clusters can share the same SRDF pair. SRDF/CE software can extend the Symmetrix enterprise system to support up to 64 MSCS clusters per Symmetrix pair.


42

Introduction

Microsoft Cluster Service overviewMicrosoft Cluster Service (MSCS) is the clustering extension to Windows Server 2003 Enterprise and Datacenter editions.

MSCS enables up to eight servers running the same Windows operating system in the same domain to be connected to a shared storage system. Typically, this is a RAID array on a shared SCSI storage bus. With SCSI connections, all servers must be within 40 meters of one other (each less than 20 meters from the storage). With Fibre Channel connections, you can increase the distance between the two servers.

Note: Consult with EMC Customer Support if you need more detail.

MSCS protects against failure of production server hardware or network connections. For data protection, MSCS uses a protected storage subsystem. The standard MSCS cluster relies on a RAID 1 or RAID 5 array storage to guarantee data protection.

In a typical MSCS cluster containing one to eight nodes, server nodes share the application workload. Typically, in a node cluster environment with n nodes, each node serves one-nth of the total number of disks and clients connected by a common SCSI bus. If one server node fails, one or several of the remaining nodes take ownership of all the disks and assume all the application workload.


Introduction

Figure 9 presents a typical two-node MSCS cluster on Windows Server 2003 Enterprise and Datacenter editions.

Figure 9 A typical two-node MSCS cluster


Enterprise LAN/WAN

Symmetrix

Clients

Ultra Wide SCSIor

Fibre Channel

Ultra Wide SCSIor

Fibre Channel

Host ANode 1 Host B

Node 2

Microsoft Cluster Service overview 43

44

Introduction

Figure 10 presents a typical four-node Windows Server 2003 cluster.

Figure 10 A typical four-node MSCS cluster

MSCS cluster conceptsMSCS is a loosely coupled cluster system. It is not a fault-tolerant, closely coupled system. The concept of a cluster is to take two or more, off-the-shelf, independent computers and set them up to work together to provide higher availability and scalability than what you can obtain using a single system. When failure occurs in the cluster, control of a disk, or resource, moves to another cluster node. This process is called a failover. Failovers can be initiated by a number of events, including the following:

◆ Manual failover — The moving of resources from one server to another. Done for system load balancing or for server maintenance.

◆ Failover due to hardware failure — The surviving node takes over when a server, SCSI or Fibre Channel host bus adapter (HBA) card, or network interface card (NIC) fails.

◆ Failover due to application failure — The failure of a virtual server or IP resource can initiate the failover.

VLAN Switch VLAN SwitchHost ANode 1

Host BNode 2 Host D

Node 4

Host CNode 3

Fibre Switch Fibre Switch

Ultra Wide SCSI orFibre Channel

Ultra Wide SCSI orFibre Channel

Symmetrix


Introduction

By contrast, a fault-tolerant system uses special-purpose hardware to run multiple computers in lock step, which provides nonstop computing with no data loss when a component failure occurs. Therefore, fault-tolerant systems are more expensive.

There are benefits and limitations to using a cluster architecture.

Benefits Clustering provides:

◆ Improved availability by continuing to provide a service even during hardware or software failure.

◆ Increased scalability by allowing new components to be added as system load increases.

◆ Simplified management of groups of systems and their applications by enabling multiple applications on multiple servers to be managed as a single system.

Limitations Clustering cannot protect against:

◆ Software corruption◆ Human-induced failures

Note: Protection of user data through backup, EMC business continuance volumes (BCVs), or other forms of offline data redundancy, remain vitally important to the reliable operation of mission-critical applications.

MSCS modes of operationMSCS supports 8 node clusters Windows Server 2003 Enterprise and Datacenter Editions.

Similar to the modes of operation generally discussed for SRDF/CE (refer to “SRDF/CE modes of operation” on page 37), the configuration for an MSCS multinode cluster in a geographically distributed cluster environment is either active/passive or active/active.

Availability MSCS allows active/active application operation. During normal operation, software applications can be running on both nodes. If either node fails, the applications are restarted on the remaining cluster node. This provides high availability by minimizing application downtime. Usually, it takes one to 10 minutes to fail over and restart an application on MSCS. Restart time is highly application dependent.

Microsoft Cluster Service overview 45

46

Introduction

Scalability In addition to availability protection, cluster technology is scalable. You can add new components to the system and run the same application (accessing the same database) on multiple nodes of a cluster to deliver increased processing power. To provide scalability, data sharing is needed.

For additional information on different modes of operation and clustering concepts, refer to the following Microsoft URL:

http://www.microsoft.com/windowsserver2003/technologies/clustering/default.mspx


http://www.microsoft.com/windowsserver2003/technologies/clustering/default.mspx

Introduction

Application software in a cluster environmentSoftware running on a cluster may, or may not, be cluster aware. When software is cluster aware, it provides a restart mechanism invoked whenever the application resource is moved to another node in the cluster.

Application failover requires a restart of the application whenever failover occurs. Restart is not instantaneous. Unlike a fault-tolerant computer, a distributed cluster does not provide nonstop computing. The time that restart takes, and the completeness of the recovery, is application dependent:

◆ For a transaction-oriented application (such as SQL or Exchange that contain both a database and transaction log files), the application provides a restart mechanism to recover work in progress. Usually a transaction log is used to record all work in progress. When a node fails, the information in host memory is lost, but the work can be reconstructed by applying the transaction log to the database to restart. This mechanism recovers all transactions completed before the failure. Transactions partially complete are lost and must be reentered.

◆ Applications such as Microsoft Word or Microsoft Excel provide a checkpoint capability. If the application experiences a failover, all work since the last disk checkpoint is lost.

◆ If an application has neither a database nor checkpoint capability, and also retains no information (or state) between client requests (such as a Web Browser or an MS-Outlook client), then it can fail over by reissuing the outstanding request. In this scenario, no work is lost, and no restart is needed on the server.

◆ If the application has neither a checkpoint nor restart capability, and it retains the state between client requests to the server, then it must be rerun from the beginning when the node it is running on fails.

Application software in a cluster environment 47

48

Introduction


2Invisible Body Tag

This chapter describes SRDF/CE behavior in various operation modes. Unless otherwise noted, SRDF/CE behavior is described for a standard two-node cluster.

◆ SRDF/CE failover operation ............................................................ 50◆ Response to complete site failure .................................................... 56◆ Failure behavior when using MNS with file share witness ......... 59

SRDF/CE and ClusterBehavior

SRDF/CE and Cluster Behavior 49

50

SRDF/CE and Cluster Behavior

SRDF/CE failover operationClusters are designed to overcome failures. There are several possible failure modes in a cluster configuration. SRDF/CE protects against more failure scenarios than local clusters can. Failure of an individual client affects only one user and is not discussed in this chapter. In an SRDF/CE cluster, eight types of cluster elements (Figure 11 on page 51) can fail (singly or in combination).

This section describes the following:

◆ “SRDF/CE failover and recovery behavior with MSCS” on page 52

◆ “SRDF/CE unique behavior” on page 53

◆ “Complete site failure and recovery” on page 54



Figure 11 SRDF/CE failover operation

The section that follows discusses how a two-node cluster responds to various combinations of element failures. Cluster response during failure modes is similar in a three- or four-node system, but a standard two-node system is used in this section for discussion purposes. The starting condition for each of these failure scenarios is:

◆ Both nodes are operational◆ Node 1 (N1) owns the quorum disk◆ Both the public link (internode LAN link) and the private link

(heartbeat link) are configured in MSCS as enabled for all network access

Symmetrix Symmetrix

Clients Clients

R1 R2

R1R2

Ultra Wide SCSIor

Fibre Channel

Host ANode 1

Host BNode 2

3

8

4

6

7

1

2

SW

5HW

Enterprise LAN/WAN


BidirectionalSRDF Interconnect

Ultra Wide SCSIor

Fibre Channel

SRDF/CE failover operation 51

52


Simply stated, the failover and recovery operations SRDF/CE provides can be divided into situations where:

◆ The behavior of SRDF/CE is the same as MSCS local clusters.◆ The geographic separation and disaster tolerance of SRDF/CE

causes unique behavior and provides recovery alternatives.

SRDF/CE failover and recovery behavior with MSCSThe following sections introduce SRDF/CE failover and recovery behavior common with MSCS (refer to Figure 11 on page 51).

LAN linkfailure (1)

If the LAN connection between nodes fails, both servers are still available and can communicate over the heartbeat link. No failover occurs, current processing continues, and client requests from clients connected to the LAN locally continue to be serviced. Client traffic from clients connected through the LAN link fail.

Heartbeat linkfailure (2)

If the heartbeat link fails, MSCS routes heartbeat messages across the public LAN. Operation of the cluster continues with no failover of resources.

Host NICfailure (4)

The host is cut off from all clients. Processing continues uninterrupted on the other host. On the failed host, client input to that host fails, but current processing activities continue. MSCS detects the NIC has failed. The isolated node takes resources offline to halt processing. The other node brings the failed resources online so application failover can occur.

Server failure (5) If the host node hardware fails, or the operating system crashes, all heartbeat messages to the remaining node cease. The remaining node then uses the quorum disk to discover the first host has failed. The remaining node then brings the resources of the failed node online and starts the applications recovery procedures.

Application softwarefailure (6)

If an application module fails, MSCS initiates failover to the remaining node. The SRDF/CE resource monitor is directed to make the storage resource for the failed application available on the other node to allow application failover.

Host bus adapterfailure (7)

An HBA failure is a resource failure that triggers an MSCS failover operation. If both Symmetrix arrays are still running, the failover operation completes normally.



SRDF/CE unique behaviorThe following sections introduce SRDF/CE unique behavior which is different from MSCS behavior (refer to Figure 11 on page 51).

Symmetrix arrayfailure (8)

When a mirrored disk fails in a Symmetrix array, it is not visible to the host because normal operations continue with the mirror, and the failed drive is hot replaced without disturbing the host. However, if an entire Symmetrix array fails, it appears to its attached server as a resource failure indistinguishable from an HBA failure. MSCS on that server triggers a failover operation. However, because the Symmetrix array itself has failed, the remaining Symmetrix devices recognize that SRDF communication is lost and prevent failover from completing unless overrides are set as described in “Complete site failure and recovery” on page 54.

SRDF link failure (3) If the SRDF link between a Symmetrix array fails, the EMC ControlCenter® Symmetrix Manager application notices the condition and reports an error.

The MSCS service does not notice the change (because access to existing disk resources is not disturbed) unless it tries to bring a new resource online. If it does try to activate a disk, the local Symmetrix array does not allow the disk to be brought online. The local Symmetrix array also returns a negative reply to Is Alive? queries from MSCS.

All such SRDF link failures are noted in the Event Log, in the SRDF/CE log, and in the System tray icon pop-up bubble. If MSCS or a user attempts to fail over or fail back a group, and there is no SRDF link available to perform that operation, the operation is not allowed. However, if there are multiple active lateral nodes and the group(s) in question are on that lateral side, lateral-to-lateral failover is permitted.


54


Complete site failure and recovery

Local MSCS Cluster In a local MSCS cluster, if an entire site fails (such as from a flood, fire, and so forth) the entire cluster fails. By contrast, with an SRDF/CE cluster, each site contains only one of the two nodes in the cluster (or only one of the n nodes in a multinode cluster).

SRDF/CE Cluster In SRDF/CE, a complete site failure can be caused by either a site failure or a total communication failure. Figure 12 illustrates the two types of complete site failure.

Figure 12 Types of complete site failure

Site (server and Symmetrix) failures (5+8)Site failure occurs when the host and the Symmetrix array both fail (such as from a natural disaster or human error).

Total communication failure (1+2+3)A total communication failure can occur while the host and Symmetrix array remain operational (such as a backhoe digs up the cable conduit where all communications cables leave a building).

Symmetrix Symmetrix

R1 R2

R1R2

Host A Host B

SRDF Link

Heartbeat

Enterprise LAN/WAN

Clients

Symmetrix Symmetrix

R1 R2

R1R2

Host A Host B

SRDF Link

Clients

Site Failure Total Communication Failure

Heartbeat

Enterprise LAN/WAN



A total communication failure, while both nodes remain operational, is referred to as a split-brain condition and is a potential cause of logical data corruption. For example, if both sides assume the other is dead and begin processing new transactions against their copy of the data, two separate and unreconcilable copies of the data can be created.

Both nodes are isolated from each other, but not from local clients. It is impossible to determine if the other node is alive. No remote client processing is possible, but running processes continue.

Note: There is no way for the surviving node to determine which of these two types of failures caused the site failure.


56


Response to complete site failureIn SRDF/CE, site failure modes determine the behavior of a cluster when a failure occurs separating the two Symmetrix arrays and suspends remote data mirroring protection.

If a complete site failure occurs, MSCS on the surviving node first notices that heartbeat messages are no longer being received. MSCS attempts to communicate with the other node using the LAN communication path, to see if communication is still possible.

If both communication links are unresponsive, MSCS queries the quorum disk to attempt to discover whether it should fail over the resources. The query to the quorum disk fails because Node 1 is no longer in contact with the Symmetrix array on Node 2. MSCS then concludes the entire site is down.

MSCS then queries the status of the disk resource and decides whether to bring the disk resources on the local node online or to set them offline. The commands to perform this query from MSCS to SRDF/CE are:

◆ Is Alive? — Determines whether a currently online resource is still healthy and can continue to be used, or whether it and all dependent cluster resources must be taken offline.

◆ Online Request — Changes the state of an offline resource to online for a failover.

Site failure mode settings determine how SRDF/CE responds to queries from Cluster Service. This setting must be manually configured to select the desired failover and recovery behavior.

The SRDF/CE site failure mode settings are:

◆ No New Onlines — Both nodes are allowed to continue processing with the resources currently online. In addition, both nodes are allowed to take resources offline, but are not permitted to bring currently offline resources online. In this mode, MSCS can cause all resources on one or more nodes to enter the offline state.

Note: No New Onlines is the recommended (default) mode.

Important: If MSCS cannot write to the quorum disk when it wants to, it terminates.



Inappropriate user actions that cause groups to bounce back act differently. If you attempt to move the quorum group when the SRDF link is down and No New Onlines is configured, the MSCS destination node terminates, and the group bounces back. Active/active configurations are obviously affected because any applications on the destination node now move. This behavior is a result of the preceding behavior.

◆ Local Override — In an SRDF link failure, this setting will only attempt to move disks laterally. If the SRDF link is up, this setting has no impact.

◆ Failstop — On both nodes, SRDF/CE returns a negative status to an Is Alive? inquiry by Cluster Service. This causes MSCS to take all SRDF/CE-dependent storage resources offline. Shutdown of all processing activity on both nodes occurs. It must be decided then whether to continue with the failover operation at the site that is still active.

Important: The mode selected must be set the same for both nodes in the cluster.

Whenever a failure occurs such that mirrored data protection between sites is lost (for example, the SRDF link is down or a Symmetrix array is down), SRDF/CE responds to the failure in the following ways:

◆ No new disk groups can be brought online until communication with the other node is reestablished (unless the SRDF Override feature is set).

◆ If you have set the No New Onlines mode, then the disks currently online will return a positive Is Alive status to MSCS so the disks can remain online.

◆ If you have set the Failstop mode, then all the disks will return a negative Is Alive status to MSCS so the disks are taken offline by MSCS.

Behavior override In addition to the site failure mode settings, SRDF/CE provides the ability to override the mode behavior and bring resources back online under user direction through the SRDF Override feature. This enables you to decide where processing is allowed to continue.

Response to complete site failure 57

58


If you determine that one site is actually down, and the other site remains operational, you can use the SRDF Override to:

◆ Override the failure mode.

◆ Allow disk resources to be brought online, even though SRDF is not operating and there is no mirror protection of data.

CAUTION!Use the SRDF Override feature with great care. EMC does not recommend using the SRDF Override feature during normal nondisaster operations.



Failure behavior when using MNS with file share witness

Disable ForcedFailover

The SRDF/CE Forced Failover option must not be used in a two-node MNS cluster with file share witness.

Enable Local Override SRDF/CE Local Override should be enabled on both cluster nodes. In the event of RDF link failure, this allows SRDF/CE cluster groups to be taken offline or online normally (but not failover). Local Override is a per-node option and should be enabled on each node with the SRDF/CE Configuration Utility Control\Site Mode Operations.

Failure Behavior andRecovery

In general, SRDF/CE behaves similarly to a two-node cluster using a quorum disk.

For example, the R1 node NodeA is at site A. The R2 node NodeB is at site B and the file share node is at a third site. The cluster is configured with all the settings described above. In the following examples, groups are cluster groups that contain one or more SRDF/CE managed physical disk resources.

Symmetrix failure at site A

◆ Groups on NodeB remain online but cannot failover.

◆ Groups on NodeA move to NodeB but stay offline and must be brought online manually by enabling SRDF Override.

WARNING

Data Loss is possible for any group from NodeA that is brought online with SRDF Override if outstanding writes were not mirrored to the R2 site.

SRDF Link Failure


◆ Groups on NodeA remain online but cannot failover.

◆ To move a group to a different node, enable SRDF Override on the destination node.

WARNING

Data Loss is possible for any group that is moved with SRDF Override if outstanding writes were not mirrored.

Failure behavior when using MNS with file share witness 59

60


Site failure (server and Symmetrix) at site A


◆ Groups on NodeA move to NodeB but stay offline and must be brought online manually by enabling SRDF Override.

WARNING


Total communication failure

◆ If both nodes have connectivity to the file share witness, the cluster will take one of the nodes offline.

◆ If only one node has connectivity to the file share witness, the cluster will take the other node offline.

◆ If neither node has connectivity to the file share witness, the entire cluster will go offline. (See Microsoft procedures for forcing an MNS cluster node online.)

◆ If NodeB is the surviving node:

• Groups on NodeB remain online but cannot failover.

• Groups on NodeA move to NodeB but stay offline and must be brought online manually by enabling SRDF Override.

WARNING



3Invisible Body Tag S

This chapter provides instructions on and lists the requirements for automatically installing and configuring the SRDF/CE for MSCS software using the SRDF/CE Installation Wizard and the SRDF/CE Configuration Wizard. It also describes how to uninstall SRDF/CE. In the case where a manual installation is required (that is, if you are configuring a dynamic cluster), refer to Chapter 7, ”Manual Installation and Configuration.”

◆ System requirements ......................................................................... 62◆ Install considerations......................................................................... 67◆ SRDF/CE Installation Wizard.......................................................... 73◆ SRDF/CE Configuration Wizard..................................................... 80◆ Configuring a two-node MNS cluster with file share witness .. 135◆ Uninstalling SRDF/CE.................................................................... 138

Automatic Installationand Configuration

Automatic Installation and Configuration 61

62

Automatic Installation and Configuration

System requirementsThis section lists the hardware, software, and EMC Enginuity requirements for an SRDF/CE cluster.

Hardware requirements

You need the following hardware to create a geographically distributed MSCS cluster using SRDF/CE software:

Symmetrix ◆ A currently supported Symmetrix system with Enginuity 5x69 or later, as documented in the EMC Support Matrix.

◆ Two Symmetrix arrays with Remote Link Director (RLD) cards installed and RDF links established in synchronous or asynchronous mode with auto recovery enabled. The Symmetrix arrays must have RDF-type disks for the device groups.

Note: Each Symmetrix array pair supports up to 64 MSCS clusters using SRDF/CE software.

◆ SRDF must be configured for bidirectional SRDF/Synchronous or SRDF/Asynchronous operation. Dual unidirectional RDF links will work; however, SRDF/CE will not report on down links with a dual unidirectional configuration.

◆ A minimum of one HBA is required for connectivity to the Symmetrix array (either QLogix or Emulex). If EMC PowerPath® software is used, an additional HBA is required. For HBA driver recommendations, refer to “HBA driver firmware and software” on page 64. For clustering reliabiity, it is recommended to have at least two paths per device.

◆ Two network interface cards; one for each public and private cluster connections between hosts.

◆ Gatekeepers — Each Symmetrix array must be configured with six or more gatekeeper devices per node, per side for SRDF/CE exclusive use. Gatekeepers should be mirrored with signatures. This prevents MSCS from hiding them. In addition, for performance reasons, an additional gatekeeper should be associated with each SRDF/CE device group.

Note: EMC recommends for each SRDF/CE group created allocating one additional gatekeeper on each host and associated per group for performance. It is also recommended the gatekeepers be mirrored.



◆ All disks need to be Read Write visible to the intended hosts at least once to allow Windows to fully populate its internal data stores.

Servers Two or more x86 or x64 servers with separate external SCSI or Fibre Channel buses and at least two external LAN connections configured for all communications (public and private). All servers in a cluster must be of the same processor architecture.

Note: MSCS configuration refers to different intracluster LAN interconnects as public and private. SRDF/CE requires all intracluster LAN interconnects be public for the nodes to communicate.

Disks SRDF/CE requires all disks have signatures. You can assign disk signatures using Windows device management initialization or other third-party tools.

Mandates EMC mandates that all hardware configurations (servers, adapter cards, cluster system, and so forth) be supported in the SRDF/CE cluster environment, as documented in the E-Lab Interoperability Navigator at:

http://elabnavigator.EMC.com

Note: Compaq AlphaServer is not supported by SRDF/CE.

Software requirementsTo install and configure SRDF/CE, the following Microsoft software and EMC software must be installed on all nodes that compose the SRDF/CE cluster.

Microsoft software The software requirements for Microsoft products are:

◆ Windows Server 2003 Enterprise or Datacenter Editions, with the appropriate service pack and current hot fixes.

Note: The Windows Server 2003 Enterprise and Datacenter Editions support clusters of up to eight nodes.

Note: Before installing the required services packs and/or hot fixes, refer to the EMC SRDF/CE for MSCS Version 2.2 Release Notes.

System requirements 63

64


◆ Microsoft Internet Explorer 6.0 and higher (only required for online help). Currently, the latest release is 6.0 (refer to http://www.microsoft.com/windows/ie/default.mspx).

◆ Microsoft Cluster Server (MSCS) for Windows Server 2003 SP1. Depending on the installation type, it can be preinstalled if the MSCS cluster is fully functional and a conversion to SRDF/CE occurs, or it can be installed as part of the SRDF/CE installation/configuration process.

◆ Be aware of known problems dealing with adding new devices in a clustered environment:

Refer to Microsoft Knowledge Base article 834911 - Volume label is blank after you mount a volume in Windows Server 2003 at: http://support.microsoft.com/kb/q834911.

Refer to Microsoft Knowledge Base article 82088 - Computer stops responding (hangs) when it tries to mount a NTFS volume after you restart the computer at: http://support.microsoft.com/kb/q820888.

HBA driver firmware and software

CAUTION!SRDF/CE supports either the SCSIport or the STORport driver; however, there are issues to consider for both drivers.Only Emulux adapters are supported.

It is recommended that two Symmetrix arrays used for the RDF relationship have serial numbers with differing last two digits. If they are the same, the SRDF/CE groups must be built manually using SYMCLI, and then imported into SRDF/CE. SRDF/CE will then populate these groups into MSCS.

SCSIport driver The vendor-dependent data returned by a SCSI Inquiry command and stored in the Windows Registry after a boot or Windows device scan only returns the last two digits of the Symmetrix serial number.

In addition, the latest EMC HBA firmware is required.

STORport driverCurrent versions of the STORport driver do not recognize devices created on Symmetrix arrays with a Symmetrix internal device LUN value greater than 255 decimal. This means devices presented to a


http://www.microsoft.com/windows/ie/default.mspx

http://support.microsoft.com/kb/q834911


http://support.microsoft.com/?id=820888



host for use by MSCS and SRDF/CE must have a Symmetrix internal device LUN less than 255 decimal to be visible to MSCS and SRDF/CE. Microsoft is addressing this problem.

Microsoft Hot Fixes MSKB817789 and MSKB823728 and the latest HBA hardware vendor driver are required.

EMC software Any SRDF-capable Symmetrix array can be used in a geographically distributed MSCS cluster.

Note: For details on the required versions of Symmetrix Enginuity (microcode), refer to the EMC SRDF/CE for MSCS Version 2.2 Release Notes, or contact your EMC representative.

Additional EMC software is required to run an SRDF/CE cluster:

◆ The full runtime kit of EMC Solutions Enabler V6.2.1 or higher.

Note: Installations of SRDF/CE V2.2 for MSCS on Windows x64 platforms where Solutions Enabler V6.2.1 isn't already installed requires a minimum Solutions Enabler V6.3 for Windows x64 platforms.

◆ Solutions Enabler SRDF component license.

◆ EMC Solutions Enabler Kit Version 6.2.1 or higher, with licenses for the base component, SRDF, and SRDF/CE features is required. SRDF/CE requires the installation of the full runtime kit. SYMAPI must be installed in its default installation folder at C:\Program Files\EMC\SYMCLI\shlib.

EMC Solutions Enabler Version 6.2.1 is obtained through EMC Customer Service or the EMC Powerlink® website at http://Powerlink.EMC.com.

IMPORTANT: If you choose to install the base daemon, storapid, with EMC Solutions Enabler 6.2.1 you MUST issue the following command which will load the base daemon automatically before SRDF/CE service loads:

stordaemon install storapid –autostart

Note: These products must be installed prior to SRDF/CE installation.

◆ EMC PowerPath is an option, but only required if more than one HBA is attached per host and you want to load balance between the HBAs.

System requirements 65

66


SRDF/CE installation and operating requirementsSRDF/CE requires a Windows installation and runtime user ID. Either a local or a domain-based user ID can be used to install or configure SRDF/CE. The user ID must be the same in all cases across all cluster member nodes, have local admin rights, be a member of the administrator’s group, and have the following permissions:

◆ Act as part of the operating system◆ Log on as a service◆ Log on as a batch job

Important: You must assign the previous three permissions directly to the SRDF/CE user ID, not to the group to which the SRDF/CE user ID belongs.

SRDF/CE requires a minimum of 16 MB of disk space, not including runtime logging storage.

SRDF/CE uses a runtime licensing methodology. To license this product, the correct SRDF/CE runtime license must be entered into the EMC Solutions Enabler License Management Facility using the symlmf command. This process is documented in the EMC Solutions Enabler Installation Guide.

Note: After installation, to change the SRDF/CE user ID or the password, use the Change User Credentials option in the SRDF/CE Configuration Utility (refer to “SRDF/CE for MSCS icon options” on page 151) or uninstall/reinstall SRDF/CE.



Install considerationsThe automatic installation procedure describes how to use the SRDF/CE Installation Wizard (refer to “SRDF/CE Installation Wizard” on page 73).

Note: The automatic configuration procedure describes how use the SRDF/CE Configuration Wizard and is described in “Configuring SRDF/CE and MSCS for the first time” on page 81.

The SRDF/CE Installation Wizard provides a user-friendly alternative to using a manual installation procedure, which is more difficult to use. Only use a manual installation procedure if installing a dynamic cluster with VERITAS Volume Manager on your system. If this is the case, refer to Chapter 7, ”Manual Installation and Configuration.”

Contact EMC Customer Support for assistance if any of the following issues are applicable:

◆ You have applications layered on with dependencies.◆ You need other devices online.◆ You are not confident about installing and configuring new

software within the context of Windows Server 2003, MSCS, and Symmetrix with SRDF.

Prerequisites The automatic installation procedure assumes:

◆ Anyone installing SRDF/CE has a working knowledge of the following:

• MSCS setup and configuration • Windows Server 2003 concepts and administration • Symmetrix operations• SRDF family of software solutions• Cluster-aware applications• Disaster-tolerant/disaster-recovery concepts

◆ Windows Server 2003 SP1 (Enterprise or Datacenter Editions), is installed as a member server of an existing domain on each node that makes up the cluster, and each node is rebooted.

Install considerations 67

68


◆ A supported Internet browser is installed to view online help.

◆ A copy of the appropriate service packs and/or hot fixes as outlined in the SRDF/CE release notes and the EMC Support Matrix.

Before you begin Before installing SRDF/CE, read through all installation procedures and related information for an overall understanding of the installation process. The configuration guidelines provide references for properly installing hardware and software in an SRDF/CE cluster environment.

Devices A device must be write enabled on each cluster member node at least once before it can be used by a MSCS cluster and SRDF/CE for MSCS. MS Windows will not update the WMI (Windows Management Instrumentation) data structures until the disk resource is write enabled.

The following steps explain how to make them write enabled on both R1 and R2 sides.

Assume, for this example, node A is on the R1 side and node B is on the R2 side.

4. After adding devices to node A on the R1 side, use the EMC Solutions Enabler command line interface to create a temporary device group (temp) containing those devices.

symdg create -type r1 tempsymld -g temp add dev 1EF, 1F0, 1F1

5. After adding devices to node B on the R2 side, use the EMC Solutions Enabler command line to create a group (temp) containing those devices.

symdg create -type r2 tempsymld -g temp add dev 1F7, 1F8, 1F9



6. On node A, fail back the temp group to R1 side. If devices are not in a valid RDF state to be failed over, split the devices and then do a full establish.

7. Reboot node A.


70


8. After node A is up, perform a Symmetrix discover, and then check the RDF state of the devices. Wait until all newly added devices are in Synchronized state.

Once the disks are in the Synchronized state go to the Computer Management Console, Device Manager and scan for hardware changes. When complete, open Disk Manager and initialize the newly added disks as MBR.



9. On node B, failover the temp group to the R2 side.

10. Reboot node B.


72


11. After node B is up, perform a Symmetrix discover.

12. Remove the temp group on both nodes A and B, as follows:

symdg delete temp -force

The newly added devices are ready for use in an MSCS environment.

Planning requirements

EMC recommends completing the preinstallation worksheet (refer to Appendix B) and your EMC Customer Service Engineer (CSE) reviewing it prior to installing SRDF/CE. Using the worksheet, you and the EMC CSE can gather all the required information (node IDs, device IDs, RA groups, Symmetrix IDs, and so on) to successfully install and configure SRDF/CE. Having this information readily available makes the installation and configuration process smoother.



SRDF/CE Installation Wizard The SRDF/CE Installation Wizard is used to automatically install SRDF/CE. The Installation Wizard is also used to install SRDF/CE in certain upgrade procedures, refer to Chapter 4, ”Upgrading SRDF/CE.”

You must install SRDF/CE using the Installation Wizard on each node in your cluster. After installing SRDF/CE on the first node in your cluster, configure the node with SRDF/CE using the Configuration Wizard (refer to “SRDF/CE Configuration Wizard” on page 80). After installing SRDF/CE and completing the Configuration Wizard Phase 1 and Phase 2 activities on the first node, repeat this process on each of the remaining nodes in the cluster (that is, install SRDF/CE using the Installation Wizard on the second node, configure it using the Configuration Wizard; then install SRDF/CE on the third node and configure it, and so on).

Figure 13 on page 74 presents a process flowchart showing the sequence of steps to correctly install and configure SRDF/CE on the individual nodes in your cluster using the Installation and Configuration Wizards.

The step-by-step process the Configuration Wizard leads you through when initially configuring SRDF/CE and MSCS is further explained in “Configuring SRDF/CE and MSCS for the first time” on page 81.

SRDF/CE Installation Wizard 73

74


Figure 13 SRDF/CE installation and Configuration Wizard process flowchart

Co

nfi

gu

rati

on

Wiz

ard

Ph

ase

1

Sel

ect c

onfig

urat

ion

mod

eS

elec

t nod

e co

nfig

urat

ion

setu

pId

entif

y as

the

first

nod

e an

d en

ter

clus

ter

#S

elec

t loc

al a

nd r

emot

e S

ymm

etrix

arr

ays

S

RD

F/C

E S

ervi

ce s

tart

sC

reat

e te

mpo

rary

gro

ups

Writ

e en

able

dev

ices

Fa

il ov

er d

evic

es (

if an

y)S

elec

t loc

al IP

add

ress

(es)

Run

Dis

k M

anag

er to

ass

ign

driv

e le

tters

/vo

lum

e la

bels

Configure Node 2

Configure Next Node

Co

nfi

gu

rati

on

Wiz

ard

Ph

ase

1C

on

fig

ura

tio

n W

izar

d P

has

e 1

Fir

st N

od

e in

Clu

ster

No

de

N (

Inte

rmed

iate

No

des

)L

ast

No

de

in C

lust

er

MS

CS

Inst

alla

tio

n

En

d o

f M

SC

S In

stal

lati

on

on

Las

t N

od

e in

Clu

ster

Co

nfi

gu

rati

on

Wiz

ard

Ph

ase

3U

pdat

e M

SC

S p

rope

rtie

sA

dd S

RD

F/C

E r

esou

rce

type

Cre

ate

quor

um g

roup

Con

figur

e qu

orum

arb

itrat

iont

tmeo

uts

Reb

oot n

ode

Install MSCS on Node N

Install MSCS on Node 1 Install MSCS on Node 1

MS

CS

Inst

alla

tio

n

Co

nfi

gu

rati

on

Wiz

ard

Ph

ase

3Jo

in th

e no

de in

the

exis

ting

clus

ter

En

d o

f M

SC

S In

stal

lati

on

on

Fir

st N

od

e in

Clu

ster

SR

DF

/CE

for

MS

CS

MS

CS

SR

DF

/CE

for

MS

CS

En

d o

f M

SC

S In

stal

lati

on

o

n In

term

edia

te N

od

es

MS

CS

Inst

alla

tio

n

C

on

fig

ura

tio

n W

izar

d P

has

e 3

Join

the

node

in th

e ex

istin

g cl

uste

r

Use

r in

pu

tA

uto

mat

ed

Co

nfi

gu

rati

on

Wiz

ard

Ph

ase

2S

elec

t con

figur

atio

n m

ode

Add

clu

ster

mem

ber

node

(s)

Sel

ect c

lust

er ty

peA

ssig

n qu

orum

con

trol

set

tings

Ass

ign

site

failu

re m

ode

and

ad

vanc

ed c

ontr

ol s

ettin

gs

Co

nfi

gu

rati

on

Wiz

ard

Ph

ase

2C

on

fig

ura

tio

n W

izar

d P

has

e 2

Sel

ect c

onfig

urat

ion

mod

eR

etrie

ve P

hase

2 s

ettin

gs fr

om N

ode

1V

iew

quo

rum

set

tings

Iden

tify

if la

st n

ode

in c

lust

er

Sel

ect c

onfig

urat

ion

mod

eR

etrie

ve P

hase

2 s

ettin

gs fr

om N

ode

1V

iew

quo

rum

set

tings

Iden

tify

if la

st n

ode

in c

lust

er

Sel

ect c

onfig

urat

ion

mod

eS

elec

t nod

e co

nfig

urat

ion

setu

pId

entif

y as

not

firs

t nod

e in

clu

ster

S

elec

t loc

al a

nd r

emot

e S

ymm

etrix

arr

ays

S

RD

F/C

E S

ervi

ce s

tart

sS

elec

t the

hos

t nam

e of

firs

t nod

eR

etrie

ve P

hase

1 s

ettin

gs fr

om fi

rst n

ode

and

writ

e en

able

dev

ices

R

etrie

ve te

mpo

rary

gro

ups

from

firs

t nod

e

Fai

l ove

r de

vice

s (if

any

)S

elec

t loc

al IP

add

ress

(es)

R

un D

isk

Man

ager

Sel

ect c

onfig

urat

ion

mod

eS

elec

t nod

e co

nfig

urat

ion

setu

pId

entif

y as

not

firs

t nod

e in

clu

ster

S

elec

t loc

al a

nd r

emot

e S

ymm

etrix

arr

ays

S

RD

F/C

E S

ervi

ce s

tart

sS

elec

t the

hos

t nam

e of

firs

t nod

eR

etrie

ve P

hase

1 s

ettin

gs fr

om fi

rst n

ode

and

writ

e en

able

dev

ices

R

etrie

ve te

mpo

rary

gro

ups

from

firs

t nod

e

Fai

l ove

r de

vice

s (if

any

)S

elec

t loc

al IP

add

ress

(es)

R

un D

isk

Man

ager

Del

ete

tem

pora

ry g

roup

s

Inst

all S

RD

F/C

E v

ia C

D a

nd In

stal

l Wiz

ard

S

RD

F/C

E S

ervi

ce is

reg

iste

red

(

but n

ot s

tart

ed)

Inst

alla

tio

n P

has

eIn

stal

l SR

DF

/CE

via

CD

and

Inst

all W

izar

d

SR

DF

/CE

Ser

vice

is r

egis

tere

d

(bu

t not

sta

rted

)

Inst

alla

tio

n P

has

e

Inst

all S

RD

F/C

E v

ia C

D a

nd In

stal

l Wiz

ard

S

RD

F/C

E S

ervi

ce is

reg

iste

red

(

but n

ot s

tart

ed)

Inst

alla

tio

n P

has

e



To install SRDF/CE, insert the CD into an attached and mounted CD drive. If Autorun is enabled on the host, the installation starts automatically. If Autorun is not enabled, use Windows Explorer to double-click the setup.exe application found at the root of the CD. This launches the SRDF/CE Installation Wizard. The Installation Wizard Welcome window appears as shown in Figure 14.

Figure 14 Welcome window


76


Click Next and the License Agreement window appears as shown in Figure 15.

Figure 15 License Agreement window

After accepting the license conditions, click Next and the Destination Folder window appears as shown in Figure 16.

Figure 16 Destination Folder window



Click Next. If a previous version of SRDF/CE or GeoSpan is detected, the Upgrade Type window appears as shown in Figure 17. See Chapter 4 for details of the upgrade procedure.

Figure 17 Upgrade Type window

Click Next and the Ready to Install the Program window appears as shown in Figure 18.


78


Figure 18 Ready to Install the Program window

Click Install. After installing SRDF/CE, the InstallShield Wizard Completed window appears, as shown in Figure 19.

Figure 19 InstallShield Wizard Completed window

Click Finish and you will be prompted to reboot, as shown in Figure 20 on page 79.



Figure 20 Reboot dialog box

You must reboot the node to complete the installation of SRDF/CE.

After installing SRDF/CE on the node you are on and rebooting your computer, you can invoke the SRDF/CE snap-in based Configuration Utility by double-clicking the SRDF/CE Configuration Utility desktop icon. The Configuration Utility is the base display for SRDF/CE.

Note: For more information on the Configuration Utility, refer to Chapter 5, ”SRDF/CE Configuration Utility GUI.”

The first time you successfully log in to the Configuration Utility, several setup functions are performed on the node you are on, including the setup of the SRDF/CE Services. To change the SRDF/CE user ID or password after this, use the Change User Credentials option in the Configuration Utility (refer to “SRDF/CE for MSCS icon options” on page 151) or uninstall/reinstall SRDF/CE.

Once you have successfully installed SRDF/CE on the node you are on, configure the node with SRDF/CE using the Configuration Wizard. The Configuration Wizard is accessed by right-clicking the SRDF/CE for MSCS icon in the left pane of the Configuration Utility and selecting the Config Wizard option (refer to “Configuring SRDF/CE and MSCS for the first time” on page 81).


80


SRDF/CE Configuration WizardThe SRDF/CE Configuration Wizard enables you to set up cluster configuration parameters automatically in a step-by-step fashion. You can use the SRDF/CE Configuration Wizard for the following three node configuration options:

◆ Configuring both SRDF/CE and MSCS for the first time. In this case, after you install SRDF/CE on the first node in the cluster, use the Configuration Wizard to configure SRDF/CE on this node. Repeat this process for each of the remaining nodes in the cluster (that is, installing and configuring SRDF/CE on each individual node). When all nodes in the cluster are configured with SRDF/CE, use the Configuration Wizard to install and configure MSCS on each node in the cluster starting with the last node on which you configured SRDF/CE. Refer to “Configuring SRDF/CE and MSCS for the first time” on page 81. (Most of this chapter describes this configuration option.)

◆ Converting your existing MSCS cluster to an SRDF/CE cluster. Refer to “Converting an MSCS cluster” on page 125.

◆ Adding an SRDF/CE node to an existing, running SRDF/CE cluster. Refer to “Adding an SRDF/CE node to an existing cluster” on page 132.

Access the Configuration Wizard by right-clicking the SRDF/CE for MSCS icon in the Configuration Utility (refer to “SRDF/CE for MSCS icon options” on page 151 in the “SRDF/CE Configuration Utility GUI” chapter).



Configuring SRDF/CE and MSCS for the first timeOpen the Configuration Wizard by right-clicking the SRDF/CE for MSCS icon in the left pane of the SRDF/CE Configuration Utility and selecting the Config Wizard option (Figure 21).

Figure 21 Opening the Configuration Wizard from the Configuration Utility

To configure both SRDF/CE and MSCS for the first time, the SRDF/CE Configuration Wizard consists of three phases. The Configuration Wizard allows you to perform the configuration steps associated with each of the three phases. The first two configuration phases (Phase 1 and 2) involve SRDF/CE configuration activities. The third phase (Phase 3) involves the installation of MSCS.

The first screen that appears when you launch the Configuration Wizard is shown in Figure 22 on page 82.

SRDF/CE Configuration Wizard 81

82


Figure 22 SRDF/CE Configuration Wizard welcome screen

Phase 1 and Phase 2 Configuration activities apply to the node you are on; you must apply settings on a node-by-node basis (there is no remote configuration). In other words, you must first performPhase 1 and Phase 2 steps on Node 1, then on Node 2, and so on until Phase 1 and Phase 2 steps are performed on all nodes. Once all your nodes are configured with SRDF/CE, proceed to Phase 3 to install MSCS on all nodes in the cluster, starting with the last node you configured with SRDF/CE.

The Phase 3 Configuration steps consist of installing MSCS and performing post-MSCS activities on a node-by-node basis on all the nodes in the cluster, starting with the last node on which you performed Phase 1 and 2 Configuration activities. The Phase 3 Configuration steps are performed on each node in the reverse node order from which you configured the nodes in Phases 1 and 2. For example, if you configured Nodes 1, 2, 3, and 4 in that order for Phases 1 and 2, then for Phase 3, you will perform Phase 3 Configuration activities on Nodes 4, 3, 2, and 1, in that order.

If you cancel out of any phase during the Configuration Wizard, all settings and temporary groups created in previous completed phases are preserved, but settings are not saved for any uncompleted phase. Therefore, if you cancel out of Phase 1 after creating temporary



groups or restart the Configuration Wizard during Phase 1, manually remove any unnecessary temporary groups using SYMCLI.

If you are configuring an SRDF/CE cluster on Windows Server 2003, it is possible some storage devices will not be enabled correctly. For example, devices may appear with incorrect labels, and there may be inaccessible or unavailable disks. In this case, rebooting the host is recommended. If you reboot during a Configuration Wizard phase, settings for that phase will not be saved, and you may have to manually remove any temporary groups created in that phase.

Figure 13 on page 74 presents a process flowchart showing the sequence of steps that the SRDF/CE Configuration Wizard leads you through when initially configuring both SRDF/CE and MSCS.

Phase 1 configuration overviewThe Phase 1 Configuration activities are slightly different depending on whether you are on the first node.

Phase 1 configuration steps for the first nodeFor the first node, the Phase 1 Configuration activities consist of the following nine steps. These steps must be performed in exact order.

1. “Step 1. Select configuration mode” on page 86.

2. “Step 2. Select node configuration” on page 87.

3. “Step 3. Designating the first node and enter cluster number” on page 88.

4. “Step 4. Select local Symmetrix array” on page 89.

5. “Step 5. Select remote Symmetrix array” on page 89.

6. “Step 6. Create temporary device groups” on page 90.

7. “Step 7. Write-enable devices” on page 93.

8. “Step 8. Select local IP addresses” on page 94.

9. “Step 9. Run Disk Manager” on page 95.

Phase 1 configuration steps for intermediate or last nodeFor any node other than the first node (that is, intermediate nodes or the last node), the Phase 1 Configuration activities consist of the same nine steps just listed, except the sixth and seventh steps are different. The steps are as follows:



84



3. “Step 3. Designating the first node and enter cluster number” on page 88.

4. “Step 4. Select local Symmetrix array” on page 89.

5. “Step 5. Select remote Symmetrix array” on page 89.

6. “Step 6. Select the hostname where first node is located” on page 98.

7. “Step 7. Retrieving groups and write-enabling devices” on page 98.

8. “Step 8. Select local IP address(es)” on page 100.

9. “Step 9. Run Disk Manager” on page 101.



Phase 1 configuration steps for the first nodeThe first dialog box to appear in the Phase 1 Configuration lists the nine steps you must complete in Phase 1 for the first node (Figure 23).

Figure 23 Overview of phase 1 configuration steps

Click Next to start the Phase 1 configuration.


86


Step 1. Select configuration modeAs shown in Figure 24, you can select the Automatic or Manual mode. Selecting Automatic mode enables you to perform the configuration steps associated with the three phases through the SRDF/CE Configuration Wizard GUI. Selecting Manual mode enables you to perform the configuration steps through the SRDF/CE MMC snap-in GUI.

Important: EMC recommends using the Automatic mode to configure your cluster. Use Manual mode only to install a dynamic cluster that uses VERITAS software product Volume Manager for Windows (previously called LDM Pro). You can create and manage dynamic and basic disks on Windows Server 2003 using VERITAS Volume Manager for Windows. VERITAS Volume Manager for Windows enables you to logically group disks under disk groups.

Figure 24 Selecting the configuration mode

Select the configuration mode and click Next to continue.



Step 2. Select node configurationSelect the SRDF/CE for MSCS option as your node configuration setup, since you are installing and configuring both SRDF/CE and MSCS for the first time. As shown in Figure 25, you must select the type of node to configure from one of the following three options:

◆ SRDF/CE for MSCS — Select this option if you are installing and configuring both SRDF/CE and MSCS for the first time. In this case, you install SRDF/CE first and then MSCS. Most of this chapter pertains to this option.

◆ Convert MSCS Cluster to SRDF/CE — Select this option if you want to convert your existing MSCS cluster to an SRDF/CE cluster. “Converting an MSCS cluster” on page 125 describes how this option differs from the first option.

◆ Add a new SRDF/CE Cluster Node to an Existing, Running SRDF/CE Cluster — Select this option if you want to add an SRDF/CE node to an existing, running SRDF/CE cluster. “Adding an SRDF/CE node to an existing cluster” on page 132 describes how this option differs from the first option.

Figure 25 Selecting the node configuration

Select the type of node configuration and click Next to continue.


88


Step 3. Designating the first node and enter cluster numberAs shown in Figure 26, you must identify whether the node is the first node in the cluster. A node designated as the first node requires slightly different configuration steps than the other nodes in the cluster. The other cluster node(s) can be intermediate nodes or the last node.

In this step, you can also choose the cluster number. The cluster number can be between 0 and 63. If you are setting up only one cluster and there are no other SRDF/CE clusters defined in your network, you can leave the default cluster number equal to 0.

CAUTION!This feature only works if multicast is active. If it is not active, the cluster numbers must be manually assigned and tracked.

Figure 26 Designating the first node and entering cluster number

After designating the node as the first node in the cluster, and entering the cluster number, click Next. After clicking Next, a



Symmetrix Discover is completed, the cluster number is validated, and the SRDF/CE Service starts.

Note: The length of time to complete a Symmetrix Discover depends on the number of disks in the storage subsystem connected to the current node.

If you have only one local Symmetrix array and one remote Symmetrix array, the Configuration Wizard will skip Steps 4 and 5 and proceed directly to “Step 6. Create temporary device groups” on page 90.

Step 4. Select local Symmetrix arraySelect a Symmetrix array from the list. The selected Symmetrix array will be configured as the local Symmetrix array (Figure 27).

Figure 27 Selecting local Symmetrix array

Note: If you have only one local Symmetrix array and one remote Symmetrix array, the Configuration Wizard will skip Steps 4 and 5 and proceed directly to “Step 6. Create temporary device groups”.

Step 5. Select remote Symmetrix arraySelect a Symmetrix array from the list. The selected Symmetrix array will be configured as the remote Symmetrix array (Figure 28).

Figure 28 Selecting remote Symmetrix array

Note: If you have only one local Symmetrix array and only one remote Symmetrix array, the Configuration Wizard will skip Steps 4 and 5 and proceed directly to Step 6.


90


Step 6. Create temporary device groupsIn the SRDF/CE configuration process, you must create temporary device groups so that operations (such as write-enabling) can be performed on the disks. These temporary device groups are then deleted later in the configuration process at the end of Phase 1 for the last node.

Use the Configuration Wizard to create a temporary group to test its suitability and usability across the cluster.

You can create an R1 or R2 type device group that is synchronous or asynchronous. In Figure 29 on page 91, the user has selected an RDF2 device type but has not yet added a device to the device group.

You must add at least one device to the group. You can choose any device, regardless of whether it has a drive letter. In Figure 30 on page 91, the user has added device DD5. Although this device displays related mount points, they are ignored for the purpose of this procedure. Even if they are mounted to each other, a device added to a temporary group here is not grouped together with related devices because MSCS is not involved. Therefore, only device DD5 is added.

Note: Device group information automatically propagates to the other nodes in the cluster.

If you cancel out of any phase during the Configuration Wizard, all settings and temporary groups created in previous completed phases are preserved, but settings are not saved for any uncompleted phase. Therefore, if you cancel out of Phase 1 after creating temporary groups or restart the Configuration Wizard during Phase 1, you must manually remove any unnecessary temporary groups using SYMCLI.



Figure 29 Before adding a device to the device group

Figure 30 After adding a device to the device group


92


If you are creating a temporary device group, and one of the devices is not in the Synchronized or Failed Over state, you will receive an error message regarding the RDF state of the devices.

If you try to create an asynchronous device group in the Failed Over state, the warning shown in Figure 31 appears.

Figure 31 SRDF/Asynchronous group creation warning



Step 7. Write-enable devicesThe temporary device groups must be write-enabled on this node so that disk letters and volume labels can be assigned to the temporary device groups (Figure 32). In this step, all devices are write-enabled on this node, and if devices are not resident on this node, they will fail over to this node and be write-enabled.

Note: If you are configuring an SRDF/CE cluster on Windows Server 2003, it is possible that some storage devices will not be enabled correctly. For example, devices may display with incorrect labels, and there may be inaccessible or unavailable disks. In this case, rebooting the host is recommended. If you reboot during a Configuration Wizard phase, settings for that phase will not be saved, and you may have to manually remove any temporary groups created in that phase.

Figure 32 Write-enabling devices

Click Next to continue.


94


Step 8. Select local IP addressesSelect a maximum of two available IP addresses from the list to designate as your local IP address (Figure 33). The IP addresses selected must be able to communicate with the other SRDF/CE hosts in the cluster because SRDF/CE will use them for intraprocess communication. These IP addresses can be the same or different as those used by MSCS.

Figure 33 Selecting local IP address



Step 9. Run Disk ManagerClick Next to assign volume labels and disk letters to the devices in the temporary groups through the Disk Manager utility (Figure 34).

Figure 34 Running Disk Manager

CAUTION!Be aware of known problems dealing with adding new devices in a clustered environment:

◆ Refer to Microsoft Knowledge Base article 834911 - Volume label is blank after you mount a volume in Windows Server 2003 at http://support.microsoft.com/kb/q834911.

◆ Refer to Microsoft Knowledge Base article 82088 - Computer stops responding (hangs) when it tries to mount a NTFS volume after you restart the computer at http://support.microsoft.com/kb/q820888.




96


When you run Disk Manager, the Disk Management utility appears, as shown in Figure 35. The Write Signature and Upgrade Disk Wizard also appears when you run Disk Manager. Use this wizard to assign new disk signatures and partition and label disks (Figure 36 on page 97).

CAUTION!Do not upgrade any disks to dynamic disks. SRDF/CE can only use basic disks with Microsoft Cluster Server.

Figure 35 Disk Management utility



Figure 36 Write Signature and Upgrade Disk Wizard

This is the last Phase 1 Configuration step for the first node. Now you must perform Phase 2 configuration activities on the first node. Proceed to Phase 2 Configuration “Step 1. Select configuration mode” on page 105.

Phase 1 Configuration steps for intermediate or last nodePhase 1 Configuration activities for any node other than the first node (that is, intermediate nodes or the last node) consist of the same nine Phase 1 steps listed in the previous section for the first node, except the sixth and seventh steps are different. The Phase 1 steps for an intermediate or last node are discussed next.

Step 1. Select configuration modeRefer to “Step 1. Select configuration mode” on page 86.

Step 2. Select node configurationRefer to “Step 2. Select node configuration” on page 87.

Step 3. Designate that the node is not the first node in clusterIdentify whether the node is the first node in the cluster. Select No in the Is this the first node in the cluster? field (refer to Figure 26 on page 88).

Step 4. Select local Symmetrix arrayRefer to “Step 4. Select local Symmetrix array” on page 89.


98


Step 5. Select remote Symmetrix arrayRefer to “Step 5. Select remote Symmetrix array” on page 89.

Step 6. Select the hostname where first node is locatedThis step is different from Step 6 of Phase 1 for the first node. For an intermediate or last node, perform this step and enter the hostname where the first node is located (Figure 37). The first node refers to the node from which you want to pull the configuration. You can enter the hostname in the box or select it by clicking Browse (...).

Figure 37 Selecting the hostname where first node is located

Step 7. Retrieving groups and write-enabling devicesThis step is different from Step 7 of Phase 1 for the first node. For an intermediate or last node, this step retrieves the groups from the first node (that is, the node from which you want to pull the configuration) and applies the Phase 1 settings (that is, all the group information) from the first node to the current node. Then, the devices on the current node are write-enabled (Figure 38 on page 99).



Note: If you are configuring an SRDF/CE cluster on Windows Server 2003, it is possible that some storage devices will not be enabled correctly. For example, devices may appear with incorrect labels, and there may be inaccessible or unavailable disks. In this case, rebooting the host is recommended. If you reboot during a Configuration Wizard phase, settings for that phase will not be saved, and you may have to manually remove any temporary groups created in that phase.

Figure 38 Retrieving groups from the first node and write-enabling devices

If devices in the device group cannot be write-enabled, the RDF State of Devices message box appears, as shown in Figure 39.

Figure 39 RDF State of Devices message


100


Step 8. Select local IP address(es)Select a maximum of two available IP addresses from the list to designate as your local IP address (Figure 40). The IP addresses selected must communicate with the other SRDF/CE hosts in the cluster because SRDF/CE will use them for intraprocess communication. These IP addresses can be the same or different as those used by MSCS.

Figure 40 Selecting local IP address for an intermediate or last node



Step 9. Run Disk ManagerClick Next if you want to assign volume labels and disk letters to the devices in the temporary groups through the Disk Manager utility (Figure 41).

Figure 41 Running Disk Manager on an intermediate or last node

CAUTION!Do not upgrade any disks to dynamic disks. SRDF/CE can only use basic disks with Microsoft Cluster Server.

Note: You may not see some of the devices in the temporary groups on the intermediate or last nodes. If this is the case, reboot the host. If you reboot in the middle of a Configuration Wizard phase, settings will not be saved, and you may have to manually remove any temporary groups created in that phase.

Note: If this is the last node, the temporary groups will be deleted.


102


This is the last Phase 1 step for the intermediate or last node you are on. Now proceed to Phase 2 Configuration “Step 1. Select configuration mode” on page 114.

Phase 2 configuration overviewThe Phase 2 Configuration activities are different depending on whether you are on the first node.

Phase 2 configuration steps for the first nodeFor the first node, the Phase 2 Configuration activities consist of the following six steps. These steps must be performed in exact order.


2. “Step 2. Add cluster member node(s)” on page 106.

3. “Step 3. Select cluster type” on page 107. This step only displays if you are configuring a cluster on Windows Server 2003 Enterprise or Datacenter Editions. On the Windows Server 2003 platform, you have the option of configuring a quorum disk-based cluster or an MNS cluster. On any other platform, the cluster can only be configured as a quorum disk-based cluster.

• If you select Shared Quorum, the next step is Step 4, assigning quorum control settings.

• If you select Majority Node Set, the next step is Step 5, assigning site failure mode and advanced control settings.

4. “Step 4. Assign quorum control settings” on page 108.

5. “Step 5. Assign site failure mode and advanced control settings” on page 110.

Important: After performing the above Phase 2 Configuration steps on the first node, you do not proceed directly to the Phase 3 Configuration. First you must do the following on each of the remaining nodes in the cluster:

a. Install SRDF/CE.

b. Perform the Phase 1 and 2 Configuration steps.



Phase 2 configuration steps for the intermediate or last nodeFor any node other than the first node (that is, intermediate nodes or the last node), the Phase 2 Configuration activities consist of four steps:


2. “Step 2. Retrieve phase 2 settings” on page 115.

3. “Step 3. Viewing cluster type” on page 116. This step is for informational purposes only. You cannot change the cluster type because this setting is pulled from the first node.

4. “Step 4. Viewing quorum settings” on page 117. This step is for informational purposes only. You cannot change these settings since they were pulled from the configuration node.

5. “Step 5. Designate if node is the last node” on page 118.

After performing these four steps, if you are on an intermediate node, you must do the following on each of the remaining intermediate nodes in the cluster:

a. Install SRDF/CE.

b. Perform the Phase 1 and 2 Configuration steps.

If this node is the last node in the cluster, proceed directly to “Phase 3 configuration” on page 119.


104


Phase 2 configuration steps for the first nodeFigure 42 shows the first dialog box to appear in the Phase 2 Configuration when you are on the first node.

Figure 42 Overview of phase 2 configuration steps for first node

Click Next to begin the Phase 2 configuration.




Important: EMC recommends using the Automatic mode to configure your cluster. Use Manual mode only to install a dynamic cluster that usesVERITAS software product Volume Manager for Windows (previously called LDM Pro). You can create and manage dynamic and basic disks on Windows Server 2003 using VERITAS Volume Manager for Windows. VERITAS Volume Manager for Windows enables you to logically group disks under disk groups.

Figure 43 Selecting the configuration mode in phase 2



106


Step 2. Add cluster member node(s)A minimum of one cluster member node is required in the Phase 2 Configuration. As shown in Figure 44, select a cluster member node by clicking Browse (...) or enter more than one hostname in the box by separating the hostnames with a comma.

Figure 44 Adding a cluster member node




Step 3. Select cluster typeIf you are configuring a cluster on Windows Server 2003 Enterprise or Datacenter Editions, you have the option of configuring a quorum disk-based cluster or an MNS1 cluster (Figure 45). On any other platform, the cluster can only be configured as a quorum disk-based cluster, and this step will not appear.

Select Shared Quorum if your cluster quorum data will reside on a shared disk in the cluster. Select Majority Node Set (MNS) if your cluster quorum will reside on a local disk on each node and be managed by MSCS.

Note: MNS is only supported on Windows Server 2003 Enterprise and Datacenter Editions.

Figure 45 Selecting the cluster type

1. MNS enables you to create a server cluster without any shared disk for the quorum resource. MNS provides an alternative quorum resource for clusters with nodes in separate geographic sites.


108


If you select Shared Quorum, the next step is assigning quorum control settings (refer to “Step 4. Assign quorum control settings” on page 108).

If you select Majority Node Set, the next step in the Configuration Wizard is assigning site failure mode and advanced control settings (refer to “Step 5. Assign site failure mode and advanced control settings” on page 110).

Important: Once you select and configure a cluster based on the type of quorum management, this value cannot be changed. If you decide later you want to use a different cluster quorum type, you must reinstall SRDF/CE.

Step 4. Assign quorum control settingsIn this step, you select the quorum disk and place it in a group. A quorum disk contains the configuration information of the cluster. You may edit the quorum disk group name and the quorum mount point (Figure 46 on page 109). The physical drive number can be between zero and 999 and the group name must conform to the following restrictions:

◆ be alphanumeric◆ "-" and "_" are allowed◆ no spaces or special characters are allowed



Figure 46 Assigning quorum control settings

The group to which the quorum disk is assigned cannot be defined as an SRDF/A group. If the group is defined as an SRDF/A group, then you are prompted to either cancel the requested action or remove the SRDF/A designation.

Important: The quorum disk you specify in SRDF/CE must be the same as the quorum disk that you specify in MSCS.

Note: You only need to set the Quorum Control settings on one node because the settings will automatically be propagated to the other nodes in the cluster.

Click Next to continue and proceed.


110


Step 5. Assign site failure mode and advanced control settings As shown in Figure 47, in this step you must select site failure mode options for Local Override (Yes/No), SRDF Override (Yes/No), and Site Mode (No New OnLines or FailStop). You must also assign Advanced Control Settings for the Delay Failback feature (Yes/No).

Figure 47 Assigning site failure mode and advanced control settings

Site Failure Mode Settings

Local Override — Selecting Yes means when a site failure occurs (that is, loss of SRDF link status), SRDF/CE will not attempt to automatically try the SRDF link during a failover process. Instead, it will only attempt lateral failovers. The default is No.

SRDF Override — Selecting Yes enables you to move or bring new groups online even though the SRDF link is down and there is no mirror protection of data. The default is No.

CAUTION!EMC does not recommend using the SRDF Override mode during normal nondisaster operations. Use this feature with care.



Site Mode — No New OnLines is the recommended (default) mode. The SRDF/CE site failure mode settings are:

◆ No New OnLines — Both nodes are allowed to continue processing with the resources currently online. In addition, both nodes are allowed to take resources offline, but are not permitted to bring currently offline resources online. In this mode, MSCS can cause all resources on one or more nodes to enter the offline state.

◆ Failstop — This setting causes groups to go offline when there is a server, site, or SRDF failure. On both nodes, SRDF/CE returns a negative status to an Is Alive? inquiry by Cluster Service. This causes MSCS to take all SRDF/CE-dependent storage resources offline. Shutdown of all processing activity on both nodes occurs. Then it must be decided whether to continue with the failover operation at the active site.

Important: The mode selected must be set the same for all nodes in the cluster.

Advanced control settings

Delay Failback — The default for the Delay Failback feature is Yes. The Delay Failback feature is automatically propagated to the other nodes in the cluster.

The Delay Failback feature modifies the Preferred Owner list for each MSCS cluster group such that a failover will occur to a lateral node first, and if the lateral node is unavailable, to a peer node. Lateral nodes are defined as nodes connected to the same Symmetrix array. Peer nodes are defined as nodes connected to different Symmetrix arrays located across the SRDF link from each other.

Note: Since the Delay Failback feature makes use of the MSCS Preferred Owner list any list that preexists or is modified by the administrator will be overwritten.

SRDF/CE manipulates the MSCS Preferred Owners list whenever a group is brought online. SRDF/CE then examines the group Preferred Owners list and determines what node is the lateral node. It can then modify the MSCS Preferred Owner list so the current node and its lateral partner are the first two in the list. Therefore, no matter which side (R1 or R2) a group is moved to, the Preferred Owner list is modified to allow a group to fail over to a lateral node and not fail back or over across the SRDF link as a first option. MSCS only moves a group across the SRDF link as a last resort. This prevents MSCS from arbitrarily performing what amounts to a failback/failover


112


across the SRDF link in an automatic fashion. This feature delays the actual failback of a group to an R1 node, and is therefore termed delay failback.

The Delay Failback feature runs simultaneously on all nodes. Therefore, when a group comes online on any node, the Preferred Owner list is updated, regardless of whether it is an R1 or R2 device.

Important: The Delay Failback feature is enabled by default. You must not change the Preferred Owner list; doing so will result in indeterminate failover behavior.

Note: You only need to change the Site Mode and Delay Failback settings on one node because the settings automatically propagate to the other nodes in the cluster.

After assigning the site failure mode and advanced control settings for the first node, the screen shown in Figure 48 appears.

Figure 48 End of phase 2 for node 1



Important: This is the last step in the Phase 2 Configuration for the first node. Do not proceed directly to Phase 3. You must first do the following on each of the remaining nodes in the cluster:

1. Install SRDF/CE.

2. Perform the Phase 1 and Phase 2 Configuration steps.

When you complete these steps for all the intermediate nodes and the last node in your cluster, you can proceed to the Phase 3 Configuration (refer to “Phase 3 configuration” on page 119). Phase 3 involves the installation of MSCS on all the nodes in the cluster (starting with the last node you configured with SRDF/CE).

Phase 2 configuration steps for the intermediate or last nodeFigure 49 shows the first dialog box to appear in the Phase 2 Configuration.

Figure 49 Phase 2 configuration steps for intermediate or last node

Click Next to begin the Phase 2 configuration.


114



Important: EMC recommends using the Automatic mode to configure your cluster. Use Manual mode only to install a dynamic cluster that uses VERITAS software product Volume Manager for Windows (previously called LDM Pro). You can create and manage dynamic and basic disks on Windows Server 2003 using VERITAS Volume Manager for Windows. VERITAS Volume Manager for Windows enables you to logically group disks under disk groups.

Figure 50 Selecting the configuration mode in phase 2




Step 2. Retrieve phase 2 settingsIf you are on an intermediate node or the last node, this step retrieves all the Phase 2 settings from the first node and applies them to the current node. The Phase 2 settings retrieved and applied to the current node include all the cluster member node, site failure mode, advanced control, multicast, and cluster type settings (Figure 51).

Figure 51 Retrieving phase 2 options from node 1



116


Step 3. Viewing cluster typeThis step is informational since the cluster type is predetermined by Node 1. You can view the cluster type in the dialog box shown in Figure 52.

Figure 52 Viewing cluster type




Step 4. Viewing quorum settingsThis step is informational since the quorum disk is predetermined by Node 1. You can view the quorum control settings in the dialog box shown in Figure 53. For information relating to what each field describes, refer to “Step 4. Assign quorum control settings” on page 108.

Figure 53 Viewing quorum settings



118


Step 5. Designate if node is the last node

You must identify if the node you are on is the last node in the cluster (Figure 54).

Figure 54 Identifying if node is last node

If you are on an intermediate node, perform the Phase 1 and Phase 2 configuration steps on each of the remaining intermediate nodes in the cluster.

If you are on the last node, you can proceed to the Phase 3 Configuration (refer to “Phase 3 configuration” on page 119).

Note: If this is the last node, it is critical you indicate this. During last node processing, several critical internal steps are performed. If you do not identify the last node as such, you will have to reconfigure the node from the beginning of Phase 1.



Phase 3 configurationThe Phase 3 configuration steps consist of installing MSCS and performing post-MSCS activities on a node-by-node basis on all the nodes in the cluster, starting with the last node on which you performed Phase 1 and 2 configuration activities. The Phase 3 configuration steps are performed on each node in the reverse node order from which you configured the nodes in Phases 1 and 2. For example, if you configured Nodes 1, 2, 3, and 4 in that order for Phases 1 and 2, then for Phase 3, you will perform Phase 3 configuration activities on Nodes 4, 3, 2, and 1, in that order.

The Phase 3 configuration activities consist of the following two steps. These steps must be performed in exact order.

1. “Step 1. MSCS installation” on page 120.

2. “Step 2. Post-MSCS configuration activities” on page 121.

Figure 55 shows the first dialog box to appear in the Phase 3 configuration.

Figure 55 Overview of phase 3 configuration steps


120


Step 1. MSCS installationThe first step in the Phase 3 Configuration is the installation of MSCS on the last SRDF/CE node (Figure 56). The actual installation of MSCS is outside the scope of this document. Refer to the appropriate Microsoft MSCS installation guides.

Figure 56 Installing MSCS

After you install MSCS, click Next. (Next is disabled until MSCS installation is complete.)

CAUTION!When you initially install MSCS on the last SRDF/CE node, all disks are resident on this last node. Now, you should only select the quorum disk during the MSCS install for MSCS to manage. Do not include all disks during the initial MSCS install (including all disks is the MSCS default option).If you did not install MSCS first on only the quorum disk (that is, you included all disks during the initial MSCS install), MSCS will create groups for all disks. SRDF/CE will not recognize these



groups, and thus cannot manage them. Therefore, in this case, you must delete the disks and groups from MSCS and create the necessary groups within SRDF/CE.

Important: When you install MSCS on Windows Server 2003, click Advanced and choose the minimal configuration option. (You must do this now for the last SRDF/CE node and later on for all subsequent nodes.) On the next tab, make sure to click Quorum and select the quorum disk selected during the SRDF/CE configuration.

Note: On Windows Server 2003, if you do not click Advanced and choose the minimal configuration option, all subsequent nodes will fail the MSCS configuration process.

Once the installation of MSCS on this node is completed, the SRDF/CE Configuration Wizard automatically detects the MSCS installation. Click Next to proceed.

Note: MSCS reads disk configurations during MSCS installation, loads that information into its Registry, and reads its in-memory copy. Therefore, any disk not present during the MSCS install may not be recognized by MSCS.

Step 2. Post-MSCS configuration activitiesThis is the last step of the Phase 3 Configuration for this node. As shown in Figure 57 on page 122, in this step, four post-MSCS installation activities are completed:

1. The MSCS service is set to Manual start mode.

2. SRDF/CE is added as a resource type under MSCS.

3. The quorum group is created within MSCS.

Note: If the quorum disk in SRDF/CE and the quorum disk in MSCS are not the same, the following warning message appears.


122


4. Quorum arbitration timeouts are automatically set.

Figure 57 Post-MSCS configuration activities



You have successfully completed the SRDF/CE configuration for the last SRDF/CE node in the cluster, and the dialog box appears as shown in Figure 58. You must reboot the last SRDF/CE node after completing the Phase 3 configuration (that is, MSCS installation and post-MSCS configuration activities). The MSCS quorum arbitration occurs prior to the SRDF/CE drivers being put into place and a reboot of the last SRDF/CE node activates the SRDF/CE arbitration code.

CAUTION!If you ignore the reboot screen, SRDF/CE will work, but the quorum arbitration will not start and the quorum failovers will fail.

Figure 58 End of phase 3 configuration for the last SRDF/CE node

After rebooting the last SRDF/CE node, complete the Phase 3 Configuration activities for the remaining nodes in the cluster in the reverse node order from which you configured the nodes in Phases 1 and 2.


124


CAUTION!To have a fully functional and supported SRDF/CE cluster, you must have installed it as per the installation documentation and completed all phases. Failure to do so may result in an either a nonfunctional or partly functional SRDF/CE cluster configuration where failover results may vary.

Post phase 3 activitiesOnce SRDF/CE is fully installed and configured on each node and Phase 3 is completed on all nodes, configure disk groups that represent the applications you want SRDF/CE and MSCS to protect. To do this, refer to “Groups component” on page 179. It is recommended for each individual application, or group of federated applications, a separate SRDF/CE group be built to ensure disk consistency. When an SRDF/CE group is built, it will be automatically added to MSCS.

After you define new SRDF/CE groups, it is highly recommended the application(s) and the disk group(s) be validated by failing them over to each node and ensuring the application and its attendant resources come up correctly in each case.



Converting an MSCS clusterThis section outlines the procedure to convert a standard operational MSCS cluster to an SRDF/CE cluster. To convert the cluster, you must have a valid MSCS cluster configuration and comply with the hardware and software requirements discussed in “System requirements” on page 62.

Pre-conversion considerationsConsider the following requirements and warnings before proceeding with converting a standard MSCS cluster to an SRDF/CE cluster:

◆ Be familiar with SRDF/CE and the process to install a new SRDF/CE cluster as outlined in “Install considerations” on page 67 and “SRDF/CE Installation Wizard” on page 73 before attempting to do the conversion procedure detailed in this section.

◆ All applicable parts of a clean SRDF/CE install must be fulfilled in addition to the conversion procedure detailed in this section (for example, complete the SRDF/CE configuration worksheet found in Appendix B, “Preinstallation Worksheet.”)

◆ This conversion procedure is more complex than a simple install, and therefore should not be attempted by unfamiliar personnel. Inappropriate or unknowledgeable use can result in data loss.

◆ Ensure a recent, full backup is available before attempting this procedure.

This procedure assumes your MSCS cluster is presently operational and functional, all groups can successfully fail over, all storage for the MSCS cluster is contained on the same, single Symmetrix, and an RDF relationship is established between the source Symmetrix and the alternate Symmetrix array. This procedure also assumes all your MSCS disks are presently running on SRDF devices as either an R1 or an R2. If not, you must make these changes to the devices before attempting this procedure.

This procedure does not address designing or configuring the EMC Symmetrix array or SRDF aspects. The procedure assumes the existing cluster data has been moved to a Symmetrix array with properly configured and tested SRDF devices, and the cluster and applications are verified on the SRDF device hardware.


126


The steps you perform when converting an MSCS cluster to an SRDF/CE cluster are different depending on whether you are on the first node. The first node is the node with the quorum disk online.

To ensure you convert the MSCS cluster to an SRDF/CE cluster correctly, follow these steps:

1. Check each node to see if the proper service pack and hot fixes are installed. If not, install them now.

2. Install Solutions Enabler V6.2.1 on both nodes if necessary.

3. Add the SRDF/CE runtime license to the EMC Solutions Enabler license file.

4. Using SYMCLI commands, ensure you can see all devices (local and remote) and also ping the remote Symmetrix array. Ensure the SRDF is set for Synchronous or Asynchronous and Bidirection.

5. Write down everything about the existing disk configuration (you need physical drive numbers, RA groups, and the Symmetrix device numbers of Node 1) and the MSCS group names. Note this information in the SRDF/CE Configuration Worksheet in Appendix B, “Preinstallation Worksheet.” Ensure you know the physical drive number and the Symmetrix device number for each disk in the cluster, and to what group each disk belongs.

Determine the physical drive number of the quorum disk on each node. This is recommended because MSCS removes the physical drive for the disk where the quorum in not online since MSCS is installed and using the quorum disk.

6. In Cluster Administrator, you must rename any and all MSCS groups that contain spaces or characters other than A-Z, a-z, 0-9. MSCS group names must conform to the following restrictions:

• length less than 31 characters• only A-Z, a-z, 0-9 are allowed ("-" and "_" are not allowed)• no spaces are allowed

The SYMAPI database cannot accept special characters. Create a separate group for the quorum disk and move the quorum to this group so it is isolated from the other resources.



In addition, all MSCS resources with the same name as the MSCS group they are in must be renamed. As a reminder, during the installation process a new group is built to contain the quorum disk. Any resources in the preexisting quorum disk group (typically the Cluster Group) must not be named the same as the user defined group name.

7. Verify all SRDF communication is enabled and you have documented all the cluster settings.

8. Install SRDF/CE on each MSCS node of the cluster. Prior to converting an existing MSCS cluster, SRDF/CE must be installed on each MSCS node. When SRDF/CE detects MSCS installed during the SRDF/CE configuration process, SRDF/CE attempts to register its resources with MSCS. For this process to be successful, MSCS must find the SRDF/CE code installed on each node.

9. On each node, open the SRDF/CE Configuration Utility. Double-click the SRDF/CE for MSCS icon and enter the correct login information.

10. On each node, right-click the SRDF/CE for MSCS icon and select Discover. The SRDF/CE Services must be configured and started by logging in to the SRDF/CE snap-in, and then issuing an SRDF/CE Discover.

After completing Discover on the node that does not own the quorum, if the following message displays, ignore it and continue.

11. Start the SRDF/CE Service on all nodes.

12. On Node 1, right-click the SRDF/CE for MSCS icon and select Configuration Wizard.


128


You are now ready to proceed with the conversion steps on the first node. Refer to “MSCS to SRDF/CE cluster conversion steps for the first node” on page 128.

MSCS to SRDF/CE cluster conversion steps for the first nodeFor the first node, the Phase 1 Configuration activities consist of the following nine steps, which must be performed in exact order. The sequence of steps for converting an MSCS cluster to an SRDF/CE cluster on the first node are the following:



3. “Step 3. Designating the first node and enter cluster number” on page 88. The MSCS group names are checked at this point to verify they comply with SYMAPI naming standards (for example, spaces and special characters are not allowed). If they do not comply, an error message appears and the Config Wizard exits.

4. Select local Symmetrix array(s). If your configuration only has two Symmetrix arrays, one local and one remote, this step is skipped (refer to “Step 4. Select local Symmetrix array” on page 89).

5. Select remote Symmetrix array(s). If your configuration only has two Symmetrix arrays, one local and one remote, this step is skipped (refer to “Step 5. Select remote Symmetrix array” on page 89).

6. Select the local IP address(es) (refer to “Step 8. Select local IP addresses” on page 94).

7. Retrieve MSCS information to apply to SRDF/CE. Retrieving MSCS information includes retrieving the cluster member nodes and group names from MSCS (Figure 59 on page 129). The geospan_res dependency is then added as a resource type to MSCS.



Figure 59 Applying MSCS information to SRDF/CE groups

A warning message appears indicating the groups online on this node will be brought offline briefly to add the geospan_res dependency to each physical disk resource in each group configured by SRDF/CE. Then the MSCS groups are imported and created in SRDF/CE.

8. Assign the Quorum Control settings (refer to “Step 4. Assign quorum control settings” on page 108).

9. Assign site mode failure and delay failback settings (refer to Figure 60).


130


Figure 60 Assigning site mode failure and advanced control settings

10. At the end of the SRDF/CE for MSCS configuration page of the first node, you will be prompted to not restart the machine. It is critical to not reboot this specific host at this step. Otherwise disk resources will fail to move to the other cluster member nodes and the cluster could end up in a failed state.

MSCS to SRDF/CE cluster conversion steps for an intermediate or last nodeFor the intermediate or last node, the Phase 1 Configuration activities consist of the following 10 steps, which are only slightly different than those for the first node. Note the first six steps are the same as those for the first node (refer to “MSCS to SRDF/CE cluster conversion steps for the first node” on page 128), and then steps 7 through 10 are different. These steps must be performed in exact order. The sequence of steps for converting an MSCS cluster to an SRDF/CE cluster on the intermediate or last node are the following:

1. Select the configuration mode (refer to “Step 1. Select configuration mode” on page 86).



2. Select the Convert MSCS Cluster to SRDF/CE option for your node configuration setup (refer to “Step 2. Select node configuration” on page 87).

3. Designate the node as not the first node in the cluster (refer to “Step 3. Designating the first node and enter cluster number” on page 88). The Configuration Wizard then performs a storage array discover.



6. Select the local IP addresses (refer to “Step 8. Select local IP addresses” on page 94).

7. Select the hostname where the first node is located. The first node refers to the node from which you want to pull the configuration. (refer to “Step 6. Select the hostname where first node is located” on page 98).

8. Pull configuration from node. Enter the node from which you want to retrieve the cluster configuration. This step will retrieve all the Phase 2 settings from the first node and apply them to the current node. The Phase 2 settings retrieved and applied to the current node include all the cluster member node, quorum control, site mode failure, and delay failback settings (refer to “Step 2. Retrieve phase 2 settings” on page 115).

9. Retrieve MSCS information to apply to SRDF/CE. Retrieving MSCS information includes retrieving the cluster member nodes and group names from MSCS. The MSCS group names are then checked to verify they comply with SYMAPI naming standards (for example, spaces and special characters are not allowed). The geospan_res dependency is then added as a resource type to MSCS (refer to Figure 59 on page 129).


132


10. Assign/view quorum control settings. Since the quorum disk is on Node 1, MSCS makes it invisible to the other nodes. Therefore, you must supply the physical drive number for the quorum disk. All other information previously retrieved from the first node cannot be changed (refer to “Step 4. Assign quorum control settings” on page 108).

If the following screen displays, ignore it and continue:

11. At the end of the SRDF/CE for MSCS configuration page for the intermediate or last node, you are prompted to restart the machine. Select Yes, I want to restart the computer, and click Finish.

12. After restarting the last node, go back to the first node (the node that owns the quorum) and restart the first node.

Adding an SRDF/CE node to an existing cluster The procedure for adding an SRDF/CE node to an existing SRDF/CE cluster consists of two phases: pre-MSCS activities and post-MSCS activities.



Pre-MSCS activitiesTo complete the pre-MSCS activities, do the following:

Note: Install EMC Solutions Enabler V6.2.1, if necessary.

1. Select the configuration mode (refer to “Step 1. Select configuration mode” on page 86).

2. Select Add a new SRDF/CE cluster node to an existing SRDF/CE cluster as your node configuration setup (refer to “Step 2. Select node configuration” on page 87).



5. Select the local IP addresses (refer to “Step 8. Select local IP addresses” on page 94).

6. Select the hostname where the first node is located (the first node refers to the node from which you want to pull the configuration). (refer to “Step 6. Select the hostname where first node is located” on page 98).

7. Pull configuration (refer to Figure 61 on page 134). In this step, group and cluster member node information are retrieved from the first node (that is, the node from which you want to pull the configuration) and applied to the current node (that is, the SRDF/CE node you want to add to the SRDF/CE cluster). Then, the latest cluster member node list is propagated to all nodes in the SRDF/CE cluster.


134


.

Figure 61 Retrieving group and cluster information from the first node

8. View quorum control settings. This step is informational since the quorum disk is predetermined by Node 1. You can view the quorum control settings in the dialog box shown in Figure 53 on page 117. For descriptions of each field, refer to “Step 4. Assign quorum control settings” on page 108.

Post-MSCS activitiesTo complete the post-MSCS activities do the following:

1. “Step 1. MSCS installation” on page 120.

2. “Step 2. Post-MSCS configuration activities” on page 121.



Configuring a two-node MNS cluster with file share witnessThe file share witness feature is an improvement to the current Majority Node Set (MNS) quorum model. This feature allows you to use a file share that is external to the cluster as an additional "vote" to determine the status of the cluster in a two-node MNS quorum cluster deployment. SRDF/CE supports this cluster model with modified procedures and settings.

Downloading the Microsoft hotfixMicrosoft has released a hotfix that enables a two-node MNS cluster with a file share witness. The procedures below require you to apply this hotfix to both cluster nodes. For details about the hotfix and a link to the download, refer to the Microsoft Knowledge Base article 921181 at:

http://support.microsoft.com/kb/921181

Creating the file share witnessEach cluster must have a separate file share for use as a file share witness. The node hosting the file share must be joined to the same domain as the cluster. Ideally, the file share witness should be located at a separate site so that the loss of either the R1 or R2 site will not affect the file share.

1. Create a directory on the file share node. For example, if the cluster name is CLUS1:

mkdir E:\FSW_CLUS1

2. Share the directory and grant full access permissions to the cluster service account.

net share FSW_CLUS1=E:\FSW_CLUS1 /GRANT:Domain\ClusterAccount,FULL

3. Set security on the shared directory to grant full access to the local Administrators group and the cluster service account:

cacls E:\FSW_CLUS1 /G BUILTIN\Administrators:F Domain\ClusterAccount:F

The following sections describe how to configure either a new or an existing SRDF/CE MNS cluster.

Configuring a two-node MNS cluster with file share witness 135

136


Configuring a new SRDF/CE MNS clusterPerform the following configuration steps for a new MNS cluster:

1. Create the file share witness as described previously.

2. Apply the Microsoft KB921181 hotfix (described previously) to both cluster nodes.

3. Configure SRDF/CE as described earlier in this chapter, but with the following exceptions:

• Node 1:

In Figure 45 on page 107 insure that you choose MNS Cluster.

In Figure 47 on page 110 set Local Override to Yes, SRDF Override to No (the default), Site Mode to No New OnLines (the default), and Delay Failback to No (the default).

• Node 2:

In Figure 58 on page 123 after the Post-MSCS Configuration Activities, do NOT reboot Node 2 when prompted (choose No).

4. Set the file share witness property on Node 2 using the following command:

cluster.exe resource "Majority Node Set" /priv MNSFileShare=\\server_name\share_name

5. On Node 2: In the SRDF/CE Configuration Utility Control\Site Mode Operations make sure that Local Override to is set to Yes.

6. Reboot Node 2 and wait for the cluster to come back online.

7. Complete Phase 3 of configuring SRDF/CE on Node 1 as described previously in this chapter.

8. Insure that the cluster can survive the loss of one node and all resources properly failover.



Configuring an existing SRDF/CE MNS clusterPerform the following configuration steps for an existing MNS cluster:

1. Create the File Share Witness as described previously.

2. Apply the Microsoft KB921181 hotfix (described previously) to the nodes in the cluster that will be used for the two-node MNS cluster with file share witness.

3. If the cluster has more than two nodes, you must evict the extra nodes with Microsoft Cluster Administrator. Be sure to leave an R1 and an R2 node.

4. Set the file share witness property using the following command:

cluster.exe resource "Majority Node Set" /priv MNSFileShare=\\server_name\share_name

5. Move the cluster group containing the Majority Node Set resource to the other node using the following command:

cluster.exe group "Cluster Group" /move

6. (Optional) Move the cluster group containing the Majority Node Set resource back to the original node using the following command:

cluster.exe group "Cluster Group" /move

7. On each cluster node, open SRDF/CE Configuration Utility and in Control\Site Mode Operations set Local Override to Yes, SRDF Override to No (the default), Site Mode to No New OnLines (the default). In Control\Advanced Control, set Delay Failback to No (the default).

8. Insure that the cluster can survive the loss of one node and all resources properly failover.

Configuring a two-node MNS cluster with file share witness 137

138


Uninstalling SRDF/CEThere are two supported scenarios for uninstalling SRDF/CE from a configured cluster.

1. To remove some cluster nodes and leave SRDF/CE on the remaining cluster nodes.

a. Open Microsoft Cluster Administrator.

b. Ensure no cluster resource groups are owned by the node(s) you will remove. Move any owned resource groups to a different node.

c. Right-click the node(s) to remove and choose Stop Cluster Service. Wait for the cluster service to stop on the node(s) as indicated by a red X.

d. Right-click the node(s) you want to remove and choose Evict. Evicting a node will uninstall the cluster service on that node and remove that node from the cluster.

e. Uninstall SRDF/CE from the evicted node(s). Use the Add/Remove Programs utility in the Control Panel to remove SRDF/CE for MSCS. Reboot when prompted to complete the uninstall.

2. Uninstalling SRDF/CE from all nodes of the cluster. The cluster will be deleted.

a. Follow the above procedure for evicting all but one node in the cluster.

b. Right-click the last node and choose Evict. Evicting the last node will uninstall the cluster service on that node and permanently delete this MSCS cluster.

c. Uninstall SRDF/CE from the last node. Use the Add or Remove Programs utility in the Control Panel to remove SRDF/CE for MSCS. Reboot when prompted to complete the uninstall.


4Invisible Body Tag

This chapter describes the procedures for upgrading a GeoSpan cluster to an SRDF/CE V2.2 cluster and for upgrading an SRDF/CE V2.0 or V2.1 cluster to an SRDF/CE V2.2 cluster.

◆ Getting started .................................................................................. 140◆ Upgrading GeoSpan clusters to SRDF/CE .................................. 142◆ Upgrading SRDF/CE V2.0.x to V2.2............................................. 146◆ Upgrading SRDF/CE V2.1.x to V2.2............................................. 148

Upgrading SRDF/CE

Upgrading SRDF/CE 139

140

Upgrading SRDF/CE

Getting started

System requirementsRefer to “System requirements” on page 62 for a list of the hardware, software, and EMC firmware requirements for an SRDF/CE clusters.

PrerequisitesRefer to the following sections before attempting a GeoSpan to SRDF/CE V2.2 upgrade procedure or when upgrading from SRDF/CE V2.0.x or V2.1.x to SRDF/CE V2.2:

◆ “Prerequisites” on page 67

◆ “Before you begin” on page 68

◆ “Planning requirements” on page 72

PC emulation softwareAlthough not required, it is useful to have PC emulation software installed on both cluster nodes, particularly if they are in different physical locations. The GeoSpan to SRDF/CE V2.2 and the SRDF/CE V2.0.x or V2.1.x to SRDF/CE V2.2 upgrade procedures require that you physically move back and forth between nodes. Using remote access software to control nodes from a remote location enables you to perform all steps on one machine.

Verifying your actionsAfter completing your SRDF/CE upgrade procedure, EMC recommends testing the system to ensure that your actions are accurately carried out. Significant steps include the following:

◆ Moving or copying files

◆ Creating or deleting device groups

◆ Adding or removing devices within groups

◆ Changing the status of SRDF devices (from synchronized to failed over, for example)


Upgrading SRDF/CE

Use any of the following methods to verify your actions:

◆ Look in the folders to which you moved or copied files.

◆ Use MSCS Cluster Administrator or the SRDF/CE Configuration Utility GUI to view node or device configuration information.

◆ Use Windows Server 2003 to view details on disk drives, such as a drive’s current status.

CAUTION!Do not perform configuration changes unless you are familiar with the procedure and are aware of its possible impact on subsequent configuration steps.

Getting started 141

142

Upgrading SRDF/CE

Upgrading GeoSpan clusters to SRDF/CEThe following procedure describes how to upgrade an EMC GeoSpan for MSCS V1.2.3 cluster to an SRDF/CE cluster.

Note: This upgrade can only be performed on version 1.2.3 of GeoSpan MSCS. In addition, this upgrade requires Solutions Enabler Version 6.2.1.

1. SRDF/CE requires a Windows installation and runtime user ID. Either a local or a domain-based user ID can be used to install or configure SRDF/CE. The user ID must be the same in all cases across all cluster member nodes, have local admin rights, be a member of the administrator’s group, and it must have the following permissions:

• Act as part of the operating system• Log on as a service• Log on as a batch job


2. The following additional privileges must be assigned to the SRDF/CE user ID during the upgrade:

• Backup files and directories

• Restore files and directories

You can set these privileges from Local Security Settings. After setting these privileges, reboot the machine for them to take effect.

These privileges are only required for the checkpoint restore during the upgrade. Once the upgrade is complete, these privileges can be removed.

3. Bring all groups online on Node 1. This step is essential because the upgrade process must work from the last node back to the first.

4. Perform each of the following steps on each cluster member node in reverse order (that is, starting from the last node going back to the first node). For example, in a four-node cluster, do Step 4a first on Node 4, followed by Node 3, Node 2, and Node 1, and then proceed to Step 4b.


Upgrading SRDF/CE

a. Open the GeoSpan Configuration Utility and create a checkpoint as shown in Figure 62. Creating the checkpoint is mandatory for the upgrade process to successfully complete. Once the checkpoint process is started, select Yes when asked to overwrite the backup group and save the configuration. After making the checkpoint file, close the Configuration Utility. It is advisable to backup the checkpoint file.

Figure 62 Creating a checkpoint in the GeoSpan Configuration Utility

b. Change the Cluster Service to manual, stop the Cluster Service, and then stop the GeoSpan Service, in that order, on each cluster member node in reverse chronological order.

c. Install SRDF/CE V2.2 on each cluster member node in reverse order and select Upgrade when asked how to upgrade from SRDF/CE 2.x. When the installation completes, reboot when prompted.

Note: Prior to upgrading a GeoSpan V1.2.3 cluster, SRDF/CE must be installed on each node. When SRDF/CE detects MSCS installed during the SRDF/CE configuration process, SRDF/CE attempts to register its resources with MSCS. For this process to be successful, MSCS must find the SRDF/CE code installed on each node.

d. When the system comes back up, open the SRDF/CE Configuration Utility on each cluster member node in reverse order. After logging in to the SRDF/CE Configuration Utility with the user ID set up as described in Step 1, right-click the SRDF/CE for MSCS icon and select Discover as shown in Figure 63.

Upgrading GeoSpan clusters to SRDF/CE 143

144

Upgrading SRDF/CE

Note: The Pull Configuration From Peer dialog box appears when you log in to the SRDF/CE Configuration Utility on the remaining cluster member nodes after Node 1. When this dialog box appears, click Cancel.

Figure 63 Selecting discover in the SRDF/CE Configuration Utility

e. On each cluster member node in reverse order, restore the checkpoint. To do so, right-click the Control component, and then select SRDF/CE Config Options, Restore as shown in Figure 64.

Figure 64 Restoring the checkpoint in the SRDF/CE Configuration Utility


Upgrading SRDF/CE

5. On Node 1, start the SRDF/CE driver by issuing the command net start SRDFCE at the command prompt. After that completes, start the SRDF/CE Service, and then start the Cluster Service, in that order.

6. Repeat Step 5 on each of the remaining cluster member nodes in chronological order (for example, on Node 2, Node 3, Node 4 and so on).

7. Once the upgrade is complete, return to each node and check the service settings for the Cluster Service. It is recommended this is set to Automatic startup.

Upgrading GeoSpan clusters to SRDF/CE 145

146

Upgrading SRDF/CE

Upgrading SRDF/CE V2.0.x to V2.2The following procedure describes how to upgrade an SRDF/CE V2.0.x cluster to an SRDF/CE V2.2 cluster.

Note: This upgrade requires EMC Solutions Enabler V6.2.1.

1. SRDF/CE requires a Windows installation and runtime user ID. Either a local or a domain-based user ID can be used to install or configure SRDF/CE. The user ID must be the same in all cases across all cluster member nodes, have local admin rights, be a member of the administrator’s group, and have the following permissions:



2. The following additional privileges must be assigned to the SRDF/CE user ID during the upgrade:

• Backup files and directories

• Restore files and directories

You can set these privileges from Local Security Settings. After setting these privileges, reboot the machine for them to take effect.

These privileges are only required for the checkpoint restore during the upgrade. Once the upgrade is complete, these privileges can be removed.

3. Bring all groups online on Node 1. This step is essential because the upgrade process must work from the last node back to the first.

4. On the last node of the cluster do the following:

a. Open the Configuration Utility and create a checkpoint as shown in Figure 64 on page 144. Creating the checkpoint is mandatory for the upgrade process to successfully complete. Once the checkpoint process is started, select Yes when asked to


Upgrading SRDF/CE

overwrite the backup group and save the configuration. After making the checkpoint file, close the Configuration Utility. It is advisable to backup the checkpoint file.

b. Change the Cluster Service to manual, stop the Cluster Service, and then stop the SRDF/CE Service, in that order.

c. Install SRDF/CE V2.2 and select Yes when asked to upgrade. When the installation is complete, select Reboot if prompted, and then restart the system.

d. When the system comes back up, open the SRDF/CE Configuration Utility. After logging in to the SRDF/CE Configuration Utility with the user ID set up as described in Step 1, right-click the SRDF/CE for MSCS icon and select Discover as shown in Figure 63 on page 144.

Note: The Pull Configuration From Peer dialog box displays when you log into the SRDF/CE Configuration Utility on the remaining cluster member nodes after Node 1. When this dialog box appears, click Cancel.

e. Restore the checkpoint by right-clicking the Control component, and then select SRDF/CE Config Options, Restore as shown in Figure 64 on page 144.

f. Start the SRDF/CE Service, and then start the Cluster Service in that order.

g. Move all groups from Node 1 to this node in MSCS.

5. Repeat Steps 4a through 4f on each cluster member nodes in reverse order. For example, if you began your installation on the last node (for example Node 4), then the next node to be configured should be Node 3, followed by Node 2, and so on.

Upgrading SRDF/CE V2.0.x to V2.2 147

148

Upgrading SRDF/CE

Upgrading SRDF/CE V2.1.x to V2.2To upgrade a configured SRDF/CE V2.1.x cluster to V2.2, complete the following tasks:

On one cluster node1. Set the following cluster properties by running these commands:

cluster.exe /prop QuorumArbitrationTimeMax=150cluster.exe /prop QuorumArbitrationTimeMin=30

2. Remove Cluster IP Address resource dependency on SRDF/CE Quorum device group resource and Quorum Physical Disk resource. This can be done using the Cluster Administrator.

On every node in the cluster1. Move all cluster groups owned by this node to a different node.

2. Start the SRDF/CE Configuration Utility and create a checkpoint file to save registry settings.

3. Close the SRDF/CE Configuration Utility.

4. Set the cluster service to manual start.

5. Stop the cluster service and SRDF/CE service.

6. Install SRDF/CE V2.2 and select Yes when when asked if you want to upgrade.

7. Complete the InstallShield wizard and reboot.

8. Start the SRDF/CE Configuration Utility and restore the checkpoint file to restore registry settings.

9. Start the cluster service.

10. Confirm that cluster groups can be moved to this node.

11. Set the cluster service to automatic start.


5Invisible Body Tag

This chapter describes the functionality of the SRDF/CE Configuration Utility graphical user interface (GUI).

◆ Configuration Utility introduction................................................ 150◆ SRDF/CE for MSCS icon options .................................................. 151◆ Control component .......................................................................... 154◆ Groups component .......................................................................... 179◆ Storage component .......................................................................... 191◆ Nodes component ............................................................................ 202

SRDF/CE ConfigurationUtility GUI

SRDF/CE Configuration Utility GUI 149

150

SRDF/CE Configuration Utility GUI

Configuration Utility introductionThe SRDF/CE Configuration Utility GUI configures SRDF/CE and controls SRDF disk configurations and MSCS disk resource definitions. The SRDF/CE Configuration Utility GUI allows you to set up and configure Symmetrix-based disk resources and configure these resources automatically within MSCS. This allows MSCS to automatically move geographically dispersed disk resource groups back and forth. MSCS resource groups ensure that application dependencies within resource groups are observed, and the application software is coordinated with the movement of data resources. Figure 65 is an example of how the SRDF/CE Configuration Utility GUI appears after configuration.

Figure 65 SRDF/CE Configuration Utility GUI

As shown in Figure 65, the SRDF/CE Configuration Utility GUI comprises four components under the SRDF/CE for MSCS icon:

◆ Control◆ Groups◆ Storage◆ Nodes

These components are described in the following sections.



SRDF/CE for MSCS icon options The SRDF/CE for MSCS icon is enabled after logging in to SRDF/CE for the first time. By clicking the SRDF/CE for MSCS icon, you can access the right-click menu options to change user credentials, access the Configuration Wizard, or to discover Symmetrix arrays (Figure 66).

Figure 66 SRDF/CE for MSCS icon right-click menu

Change User CredentialsThe Change User Credentials option allows you to change the login credentials (Figure 67 on page 152). To change the user ID, do the following:

◆ Coordinate this action between all nodes due to DCOM use.

◆ Ensure you are logged in as the new user ID before making the change.

◆ Ensure the Debug Programs privilege is assigned to the new user ID, or ensure the COM server is not running as the old user ID before making the change.

SRDF/CE for MSCS icon options 151

152


Figure 67 Changing user credentials

Configuration WizardThe Config Wizard option launches the SRDF/CE Configuration Wizard. The Configuration Wizard enables you to set up cluster configuration parameters automatically in a step-by-step fashion. The Configuration Wizard guides you through any of the following configuration tasks:

◆ Configuring SRDF/CE and MSCS for the first time◆ Converting an MSCS cluster to an SRDF/CE cluster◆ Adding an SRDF/CE node to an existing SRDF/CE cluster

Note: For more information on the SRDF/CE Configuration Wizard, refer to “SRDF/CE Configuration Wizard” on page 80.

DiscoverThe Discover option scans the storage subsystem connected to the current node and gathers device information. Use Discover when configuring SRDF/CE, making Symmetrix configuration changes, updating the SYMAPI database, or following group failover/ failback operations.

Important: SRDF/CE uses a dedicated SYMCLI/SYMAPI database. If you make storage configuration changes, update the SRDF/CE database by either issuing a discover through the SRDF/CE Configuration GUI (Figure 68 on page 153), or manually issuing

set symcli_db_file=fully-qualified db file locationand then the manual SYMCLI discover command (Figure 69 on page 153).



Figure 68 Issuing a discover through the Configuration Utility GUI

Figure 69 Issuing a manual SYMCLI discover command

SRDF/CE for MSCS icon options 153

154


Control componentAs shown in Figure 70, the Control component of the SRDF/CE Configuration Utility GUI contains five configurable parameters of SRDF/CE. The following list provides the parameters and references the page numbers for more details:

◆ “Quorum Control” on page 155◆ “Site Mode Operations” on page 161◆ “Advanced Controls” on page 163◆ “Logging Control” on page 166◆ “Forced Failover” on page 175

Figure 70 Expanded control view in the SRDF/CE Configuration Utility

The Control component enables you to manage the SRDF/CE configuration locally and on all peer nodes. The Control right-click menu has functionality for additional configuration synchronization options with cluster member nodes (Figure 71 on page 155). The right-click menu item SRDF/CE Config Options on the Control display node enables you to perform the following options:

◆ Checkpoint — Saves to file the current SRDF/CE Registry entries, SYMCLI device group information, and all control parameters and options. This data can subsequently be used to reconfigure the SRDF/CE-specific data back onto a clean installation with the Restore feature.



Important: EMC recommends a configuration checkpoint file be generated and saved after the configuration is complete. Using checkpoint helps to ensure the procedure can be quickly and reliably performed should you need to replace SRDF/CE and/or perform a recovery into an existing SRDF/CE cluster.

◆ Restore — Loads a checkpoint file to restore saved SRDF/CE control information and device group definitions.

Restore does not restore the MSCS configuration. It only restores what is specific to SRDF/CE.

Figure 71 Control right-click menu

Quorum ControlAs shown in Figure 72 and Figure 73 on page 156, the Quorum Control option lists the following parameters in the right pane:

◆ Quorum Type◆ Symmetrix Device◆ Group◆ Owning Node◆ Owner SymDevID◆ Physical Device◆ Mount Point◆ R1/R2◆ Device Status◆ Capacity (MB)◆ RA Group

Control component 155

156


Figure 72 Quorum Control expanded view

Figure 73 Quorum Control expanded view (continued)



The Quorum Control parameters are described in Table 2.

Table 2 Quorum Control column headings

Column heading Description

Capacity (MB) Shows the device capacity in megabytes.

Device Status Indicates the R1/R2 device status. The possible device status states are Ready, Not Ready, and Write-Disabled.

Group Indicates the SRDF/CE group name of the quorum.

Owner SymDevID Indicates the SymDevID on the owning node (that is, the MSCS owning node) corresponding to the local SymDevID. For example, in Figure 72 on page 156, Symmetrix device 0BC (which is part of the quorum group) is mapped to 0BC on the remote Symmetrix array. Since this device is managed by MSCS and is online on API1190 (which is connected to the remote Symmetrix array), the Owner SymDevId shows 0BC (as seen from API1190).

Mount Point Indicates the mount point of the physical drive on the owning node. If the owning node field is blank, this column shows the local mount point (if it exists).

Owning Node Shows the MSCS node name that owns the quorum. This information is obtained directly from MSCS. If the quorum is not part of MSCS, or if MSCS is not installed on the local node, the owning node field will be blank.

Physical Device Shows the physical device number as seen on the local node for a particular Symmetrix device. If that device is managed by MSCS, the physical drive number will be that of the owning node.For example, in Figure 72 on page 156, Symmetrix device 0BC is part of MSCS and is online on API1190. On API1190 this device is mapped to \\.\PHYSICALDRIVE21.

Quorum Type Indicates whether the quorum is a quorum disk-based cluster (Shared Quorum) or a Microsoft Majority Node Set (MNS) cluster. MNS enables you to create a server cluster that does not have any shared disk for the quorum resource.

R1/R2 Shows the R1 or R2 device type.

RA Group Indicates the RA group to which the quorum belongs.

Symmetrix Device Shows the Symmetrix device ID that is mapped to the host.


158


Edit Quorum Controlparameters

To modify the Quorum Control parameters, right-click any of the parameters displayed in the right pane and select the Edit Quorum Control Parameters option (Figure 74). The Quorum Properties dialog box appears (Figure 75 on page 159).

Figure 74 Selecting edit Quorum Control settings



Figure 75 Changing quorum properties

Two options are available to change the quorum disk properties. The default option allows you to select from a pull-down list the currently mounted disks from which to choose the quorum disk. When you select an available disk using the default option, the selected disk will be force mounted to the defined mount point. If any disk is already at that drive letter, that disk will be dismounted. This is the normal option most users should choose.

There are situations where you might want to force a specific disk to be defined as the SRDF/CE quorum disk and you do not want any other action other than the SRDF/CE configuration definition to be changed. Or, you might want to change the SRDF/CE quorum designation when the quorum disk is not online on the current node. In these situations, select the Edit Quorum values using direct text entry option.


160


Note: Only disks with formatted file systems will be visible since the MSCS quorum requires a file system.

Push to MSCS By right-clicking any of the Quorum Control parameters in the right pane, you can also select Push to MSCS. This option allows you to push a new quorum group definition to MSCS. When you select Push to MSCS, the dialog box in Figure 76 appears.

Figure 76 Pushing a new quorum group definition to MSCS

Set quorum type This option is only available for Windows Server 2003. If you are configuring your cluster on Windows Server 2003, the Quorum Control right-click menu allows you to choose the cluster quorum type you want to configure: a quorum disk-based cluster (Shared Quorum) or a Microsoft Majority Node Set (MNS) cluster. To select the quorum type, select Set Quorum Type from the Quorum Control right-click menu (Figure 77 on page 161). The Quorum Type dialog box shown in Figure 78 on page 161 appears. MNS enables you to create a server cluster without any shared disk for the quorum resource.




Figure 77 Quorum Control right-click menu.

Figure 78 Selecting the quorum type

Site Mode OperationsThe Site Mode Operations option controls how SRDF/CE behaves when a failure occurs. There are three parameters associated with this option: Local Override, SRDF Override, and Site Mode.

◆ Local Override — In an SRDF link failure, this setting will only attempt to move disks laterally. If the SRDF link is up, this setting has no impact. The default is No.

◆ SRDF Override — You can select either Yes or No. Yes enables you to move or bring new groups online if the SRDF link is down. The default is No.


162


◆ Site Mode — The site mode you set must be the same on all nodes of the cluster. You can select either No New Onlines or FailStop. The default is No New Onlines.

• FailStop — On all nodes, SRDF/CE returns a negative status to an Is Alive? inquiry by Cluster Service. This causes MSCS to take all storage resources offline. Shutdown of all processing activity on all nodes occurs. Therefore, you must decide whether to continue with the failover operation at the site still active.

• No New Onlines — All nodes are allowed to continue processing with the resources currently online. In addition, all nodes are allowed to take resources offline, but are not permitted to bring currently offline resources online. In this mode, it is possible that MSCS will cause all resources on all nodes to enter the offline state.

To modify these parameters, right-click Site Mode Operations in the left pane or any of the parameters in the right pane, and select Edit Site Mode Parameters (Figure 79). The Site Mode Operations dialog box appears (Figure 80 on page 163).

Figure 79 Expanded site mode operations view and right-click menu



Figure 80 Changing site mode operations parameters

Advanced ControlsAdvanced Control settings are for advanced debugging purposes only and should only be used with EMC recommendation. As shown in Figure 81, Advanced Control lists the Cluster Number, Delay Failback, SRDF/A Failover Method, Auto Del Release, Multicast IP, and Multicast Port parameters.

Figure 81 Expanded advanced control view and right-click menu

◆ Cluster Number — A unique identifier associated with the nodes within an SRDF/CE cluster. The cluster number equals a value between 0 and 63 and is propagated through the Autoconfiguration function.


164


The Autoconfiguration function attempts to automatically determine if other SRDF/CE clusters are running and what cluster number they are using. This functionality is limited to SRDF/CE clusters that are up and network-accessible to the new cluster. EMC GeoSpan for MSCS clusters do not support or respond to this automatic discovery feature. If there are other GeoSpan or SRDF/CE clusters not network accessible, ensure the cluster number you choose does not conflict with a preexisting cluster.

◆ Delay Failback — Modifies the Preferred Owner list for each MSCS cluster group such that a failover will occur to the lateral nodes first, and if the lateral nodes are unavailable, then to the peer nodes. If the Preferred Owner list is not coded for an MSCS Group, a list is inserted with the lateral nodes followed by the peer nodes. This setting gets updated each time the MSCS Group comes online. If Delay Failback is not on, MSCS controls the node failover.

Important: The Delay Failback feature is enabled by default. Do not change the Preferred Owner list because this results in indeterminate failover behavior.

Note: Delay Failback uses a UDP-based multicast to differentiate between local and remote nodes.

◆ SRDF/Asynchronous Failover Method — If you have SRDF/A-enabled groups, this option allows you to determine how the groups are split after failover (Normal or Immediate). The Normal option splits the groups at the end of the current bundle cycle, ensuring a consistent copy. Normal is the default. The Immediate option splits the groups immediately after failover and therefore the copy is indeterminate (that is, the data copy may or may not be consistent). Only select the Immediate option if the consistency of the data is not important. The Immediate option deletes the pending tracks and immediately performs the failover, with a potential loss of data.

◆ Auto Del Release — In some host environments, long-running SRDF operations may have left Symmetrix Lock 9 locked. The Auto Del Release function, when enabled during a failover or failback operation, checks if the lock value is held. If the lock remains locked for 180 seconds with the Auto Del Release option set, the lock is automatically removed and the failover/failback operation is allowed to proceed.



◆ Multicast IP — The multicast IP address is used as part of the configuration of the Delay Failback operation. The multicast IP address is part of a thread that periodically keeps track of which hosts are running and connected to the Symmetrix array. This information is fed into the Delay Failback operation to ensure a current list of hosts, their state, and what Symmetrix array they are connected to is maintained.

◆ Multicast Port — The multicast port address is used as part of the configuration of the Delay Failback operation in the same way as that described above for the multicast IP address.

To change any Advanced Control parameters, select the Edit Advanced Control Parameters option on the Advanced Control right-click menu in the left pane, or right-click any of these parameters in the right pane (Figure 81 on page 163). When you select the Edit Advanced Control Parameters option, the dialog box appears as shown in Figure 82.

Note: The default for the Status Display is Off for improved performance.

Figure 82 Changing Advanced Control parameters


166


The Status Display option at the bottom of the Advanced Control Parameters dialog box allows you to determine whether you want to display the status bubble in the bottom-right corner of the SRDF/CE Configuration Utility. The status bubble provides information on Configuration Utility performance. The default is Off.

Logging ControlLogging Control sets the SRDF/CE logging levels and paths. As shown in Figure 83, Logging Control lists the SRDF/CE Log Level, the SRDF/CE Log Path, and the SRDF/CE Log File Viewer parameters.

◆ SRDF/CE Log Level — Sets the SRDF/CE log level. Log level can be set from 1–Normal to 4–Trace high. 1 allows only error, warning, and informational messages to be displayed in the log file; 2 enables minimum trace information; 3 enables additional trace information; 4 enables maximum trace information. 1–Normal is the default.

◆ SRDF/CE Log Path — Sets the directory location for the SRDF/CE log file. This is a trace file where all logging information is placed.

◆ SRDF/CE Log File Viewer — Enables you to view any log file.

Figure 83 Expanded SRDF/CE service logging view



To edit the log level parameters, right-click SRDF/CE Log Level in the right pane and select Edit Log Level Parameters (Figure 84).

Figure 84 Editing log level parameters

To edit the log file path, right-click SRDF/CE Log Path in the right pane and select Edit Log File Path (Figure 85). The Select the SRDF/CE Log Path dialog box opens (Figure 86 on page 168).

Figure 85 Editing log file path


168


Figure 86 Editing the log file path

To view the latest log file, as well as other log files, right-click SRDF/CE Log File Viewer in the right pane and select Open Latest Log (Figure 87). You can also view log files by double-clicking SRDF/CE Log File Viewer in the right pane. The SRDF/CE Log Viewer dialog box then appears (Figure 88 on page 169).

Figure 87 Viewing the latest log file

To open the log files in the SRDF/CE Log Viewer, do one of the following:

◆ Select File, Open or File, Open With Other Editor.◆ Double-click the file.



To choose which columns you want to display in the SRDF/CE Log Viewer, do one of the following:

◆ Select View, Add/Remove Columns (refer to Figure 88).◆ Right-click column headings and select Select or Hide to sort

columns.◆ Drag and drop column headings to sort columns.

You can also search for text within the log files displayed in the SRDF/CE Log Viewer by selecting View, Find.

Figure 88 SRDF/CE Log Viewer

Logging enhancementsBeginning with SRDF/CE for MSCS Version 2.1, new logging features provide more detailed logging. If there is a problem with SRDF/CE for MSCS, detailed logs provide EMC Customer Support with the technical information necessary to help diagnose the problem and help SRDF/CE engineers with debugging.


17


The logging facility captures detailed information, and the size and number of log files being created has increased compared to prior versions of SRDF/CE for MSCS.

To help minimize disk capacity utilized by log files, log compression and log retention are two new features that address these issues.

Log compression Log compression uses Microsoft NTFS compression for the SRDF/CE logs. When logging is enabled the Log Compression feature is enabled, and it is not configurable by the user. The minimum individual log file size for SRDF/CE logs is 20 megabytes. Once this threshold has been reached, the logging facility closes the log file, compresses it, and starts SRDF/CE logging to a new log file.

The naming convention used for SRDF/CE log files is srdfce_ followed by a date and timestamp with a .log suffix. For example, a typical log file name would be srdfce_20051012085417.log. When this log file reaches the maximum individual size threshold, the log file will be compressed and a new log file created with a different date and timestamp.

0 EMC SRDF/Cluster Enabler for MSCS Product Guide


To obtain the actual log file size for an NTFS compressed file from within Windows Explorer, right click on the file and select Properties. The Size on disk field is the amount of disk space the compressed log file is utilizing. The Size field is the uncompressed size of the file.

Note: If log files are moved to another directory and the directory is not a NTFS compressed directory the log file(s) will be decompressed to their actual size.

Opening log files using SRDF/CE log viewer or other standard editing tools will not require decompressing the log files.

Log retention Beginning with version 2.1, SRDF/CE for MSCS generates more logging information and a greater number of log files than previous versions of the product. Log retention is a feature that customizes the amount of time log files will exist, or the maximum disk space the log files utilize.


172


By selecting the Control node in the SRDF/CE user interface and then selecting Logging Control and double-click on the Log Retention element the following dialog box will be presented:

There are three user options: Disable log retention; Number of days; and Allocate disk space for SRDF/CE logs.

Disable log retention The Disable log retention option is mutually exclusive with the Number of days and Allocate disk space for SRDF/CE logs options and causes SRDF/CE for MSCS to retain all log files.

Number of daysWhen the Number of days option is selected, the Log Retention value text box is enabled. The value entered in this text box determines how many days from the current date SRDF/CE for MSCS will keep SRDF/CE log files in the configured log file directory.



For example, if you enter 10 in the text box, any log file less than or equal to the current date minus 10 days is deleted.

Allocate disk space for SRDF/CE logsTo allocate disk space for SRDF/CE logs, select Controls, Logging control, and then SRDF/CE Log Retention. The following dialog will appear.

The value located in the log retention value textbox determines how much physical disk space will be allocated for SRDF/CE logs. The value is calculated by adding the size of all log files. If the sum of log


174


files is greater than the value specified for the log retention value, log files are deleted from oldest to newest until the size of all log files is less than or equal to the log retention value.

Immediate log cullingThe algorithm to determine which log files to delete is only invoked when a new log file is being created. If either the time or size log retention options are enabled you can also take immediate action by right clicking the SRDF/CE Log Retention element and selecting Delete log files. This feature is only enabled when you select a log retention feature and is disabled if log retention is disabled.

Once you select this feature a confirmation dialog appears asking if you are sure you want to delete log files using existing log retention values.



Forced FailoverThe Forced Failover option provides an automatic way to keep the cluster running on a particular Symmetrix array or arrays in the event of a total site disaster.

The Forced Failover option performs differently depending on whether the MSCS cluster is defined as a quorum disk-based cluster or an MNS-based cluster. For quorum disk-based clusters, the node that owns the quorum decides what nodes will stay up. For MNS-based clusters, you can configure a list of nodes to be forced up in a site disaster.


Quorum disk-based clustersFor quorum disk-based clusters, the side that remains up with respect to a Symmetrix array is based on what node owns the quorum resource. In a site disaster, MSCS keeps all nodes up on the side owning the quorum. All resources owned by the other side are moved to the surviving side.

In the quorum disk case, SRDF/CE monitors all the nodes. If tracks (that is, data) are not owed to the surviving side, then the move proceeds smoothly. If tracks are owed to the surviving side, then the SRDF Override option is required to make the move successful. Therefore, if SRDF/CE detects a split-brain1 condition during MSCS' normal group failover processing, the Forced Failover option automatically invokes the SRDF Override function so the groups from the failing site successfully transition to the new site.

MNS-based clustersAn MNS cluster is different from the quorum disk-based cluster because MSCS automatically shuts down the entire cluster when it detects the majority of nodes are not running. Under normal MSCS operations, you can evaluate the situation and restart the cluster in

1. A total communication failure, while both nodes remain operational, is referred to as a split-brain condition and is a potential cause of logical data corruption. For example, if both sides assume the other is dead and begin processing new transactions against their copy of the data, two separate and unreconcilable copies of the data can be created.


176


the Service Control Manager with the ForceQuorum option. The SRDF/CE Forced Failover option automates this process.

When the entire cluster shuts down due to a loss of the majority of nodes, MSCS then automatically attempts a node restart. On restart, a list of nodes (preconfigured by the user during the initial SRDF/CE configuration) is supplied to MSCS. These nodes now constitute the new cluster majority. If the nodes you were running on are included in the preconfigured list of nodes, the node restart will occur with the ForceQuorum option. As groups from the failed site are brought online, the SRDF Override function is automatically invoked to allow that movement.

Important: All nodes in the preconfigured forced failover list need to be running and communicating with each other or the cluster will not be forced up successfully.

To rejoin nodes into a cluster once the cluster is forced up successfully, perform the following steps on the nodes you want to rejoin:

1. Stop the Cluster Service on the nodes with the ForceQuorum option.

2. Remove the ForceQuorum key.

3. Restart the Cluster Service on these nodes.

Note: The Forced Failover option is no longer enabled after you add the nodes not running to the cluster and the cluster resumes normal MNS operation.

The Forced Failover option therefore enables you to configure a node or list of nodes on the same side of the Symmetrix array to restart in the event the majority of nodes are lost.



As shown in Figure 89, Forced Failover displays whether this feature is enabled, and if it is an MNS cluster, it displays the configured list of nodes forced up in a site disaster.

Figure 89 Expanded Forced Failover view and right-click menu

To enable or disable the Forced Failover option, do one of the following:

◆ Double-click Forced Failover in the left pane.◆ Right-click Forced Failover in the left or right pane and select

Edit Forced Failover (refer to Figure 89).

The dialog box shown in Figure 90 on page 178 appears. If you are running an MNS cluster, you can select which nodes you want to force up in a total site disaster. In Figure 90, the user has selected node L82AP216 as the node to be forced up in a site disaster. If you are not running an MNS cluster, the MNS Node Selection group is disabled and grayed out. When you click OK, SRDF/CE verifies the nodes you selected are lateral nodes with respect to each other. You cannot select peer nodes to be forced up since each node would be connected to a different Symmetrix array. In addition, the following warning message appears:

You should only enable this feature when trying to automatically fail resources over following the loss of a site. It is not recommended you enable this feature because there is the potential for data loss. Do you want to continue?


178


Figure 90 Enabling the Forced Failover option



Groups componentThe Groups component lists the currently configured SRDF/CE device groups. As shown in Figure 91 and Figure 92, clicking Groups gives you device group information in the right pane, such as the owning node (the MSCS node name that owns the device), the RDF configuration, the RDF state, the specific devices, whether R1/R2 Swap is enabled, the RA group to which the device belongs, whether the SRDF/Asynchronous feature is enabled, the SRDF/Asynchronous failover method, the number of R1/R2 invalid tracks (if any), and the SRDF/A lag time. Clicking each device group in the left pane displays more detailed information about the specific group, such as its Symmetrix device, owning node, owner SymDevId, physical device, mount point, device status, and so on. Table 3 on page 181 provides a description of the Groups column headings.

Figure 91 Expanded groups view with the device groups list

Figure 92 Expanded groups view with the device groups list (continued)

Groups component 179

180


The device group detail is shown in Figure 93 and Figure 94.

Figure 93 Device groups detail view

Figure 94 Device groups detail view (continued)



Table 3 Groups column headings



Devices Shows all R1/R2 Symmetrix device IDs mapped to the host.


Group Indicates the SRDF/CE group name to which the device belongs.

Owner SymDevID Indicates the SymDevID on the owning node (that is, the MSCS owning node) corresponding to the local SymDevID.


Owning Node Shows the MSCS node name that owns this particular disk.This information is obtained directly from MSCS. If the disk is not part of MSCS, or if MSCS is not installed on the local node, then the owning node field will be blank.

Physical Device Shows the physical device number as seen on the local node for a particular Symmetrix device. If that device is managed by MSCS, then the physical drive number will be that of the owning node.


R1/R2 Invalid Tracks Indicates the number of invalid R1 and R2 tracks (if any).

R1R2 Swap Shows whether R1/R2 swap is enabled. Refer to “SRDF/CE SRDF swap support” on page 28 for more information on this feature.

RA Group Indicates the RA group to which the group belongs.

RDF Config Shows the RDF configuration of the group (that is, the R1 or R2 device type).

RDF State Indicates the RDF state of the group. For a listing of the possible RDF states, refer to EMC Solutions Enabler SRDF Family CLI Product Guide.

SRDF/Async Lag Time Indicates the lag time between the target (R2) device and the source (R1) device in an SRDF/Asynchronous environment.

Symmetrix Device Shows the Symmetrix device ID mapped to the host.

SRDF/Asynchronous Shows whether the SRDF/Asynchronous feature is enabled.

SRDF/Asynchronous Failover Method

Shows the SRDF/Asynchronous failover method. It can be normal or immediate. Refer to “Advanced Controls” on page 163 for more information.


182


Groups right-click menu optionsYou can create new device groups, pull groups from SYMCLI, or list device groups by right-clicking Groups, selecting SRDF/CE Group Options (Figure 95), and selecting one of the following options:

◆ “Create New Device Group” on page 183

◆ “Pull groups from SYMCLI” on page 187

◆ “List” on page 187.

Figure 95 Groups right-click menu



Create New DeviceGroup

The Create New Device Group option enables you to create new device groups by physical device or Symmetrix device. When you select Create New Device Group, the Select and Group Devices dialog box appears (Figure 96). The Device Display Type list also allows you to choose SRDF/A-capable devices.

Figure 96 Creating a new device groupFigure 97 shows this same screen after the user has entered a device group name and added a device to the new group.


184


Figure 97 illustrates adding a device to the device group named Test. If you select one of the mounted devices, all devices mounted to each other will be selected and added to the new group.

In this case, the user selected device DD5. Device DD5 is mounted on DD4 (mount point path E:\Mount F), and DD6 is mounted on DD5 (mount point path F:\Mount G). Therefore, devices DD4 and DD6 are also added to the group.

Refer to “Mount point support” on page 33 for more details about the SRDF/CE implementation of mount points.

Figure 97 Adding a device to the device group



If you select SRDF/A devices as a group, the device listings are replaced with Symmetrix RDF device groups that consist of SRDF/A-capable devices (Figure 98). If you try to create an asynchronous device group in the Failed Over state, the warning shown in Figure 99 appears.

Figure 98 Creating an SRDF/Asynchronous-enabled device group

Figure 99 SRDF/Asynchronous group creation warning


186


You can also create an SRDF/Synchronous group with SRDF/A- enabled devices in a consistent state. In this case, a warning displays indicating in order to create an SRDF/Synchronous group, the asynchronous devices in the same RA group will be changed to synchronous (Figure 100).

Figure 100 Async devices in consistent state warning

The quorum group, by Microsoft requirement, cannot be designated as SRDF/A-capable. Therefore, if you select a device/group SRDF/A-capable and later choose this as the quorum device, then the SRDF/A option will be deactivated. You cannot use any device from a group that is SRDF/Asynchronous to be the quorum disk. Microsoft requires the quorum disk to be synchronous.

Note: SRDF/CE automatically determines what Enginuity level is running on the local and remote Symmetrix arrays. If the Enginuity level is earlier than 5x70, the asynchronous options in the Device Display Type box will not be available. If the pair of Symmetrix arrays is running on Enginuity level 5x70, you will be allowed to create only one SRDF/A group and the R1-R2 swap option will be dimmed. Enginuity levels 5x71 and higher supports multiple SRDF/A groups and R1-R2 swap.



Pull groups fromSYMCLI

The Pull Groups from SYMCLI option displays all the groups defined in SYMCLI that are not SRDF/CE groups, and then leads you through the process of importing these groups into SRDF/CE and MSCS (Figure 101).

Figure 101 Pulling groups from SYMCLI

List The List option lists the device groups as shown in the example in Figure 102.

Figure 102 Listing device groups


188


Device group right-click menu optionsWhen you right-click the specific device group in the left pane, the SRDF/CE Group Operations right-click menu option enables you to instruct MSCS to move cluster groups or make them come online or offline, to fail over and fail back device groups, to set the R1-R2 Swap option, and to designate a group as SRDF/Synchronous or SRDF/Asynchronous via the Array Operations option (Figure 103).

Figure 103 Device group right-click menu

◆ MSCS Move — Instructs MSCS to initiate a cluster group move.

◆ MSCS Online — Brings the selected cluster group online.

◆ MSCS Offline—Takes the selected cluster group offline.

◆ Unlock — The Unlock option unlocks any device group locks currently in place. When you select Unlock, the warning message shown in Figure 104 appears.

Figure 104 Release Symmetrix locks warning

CAUTION!Use Unlock with caution and only with a thorough understanding of this function.



◆ Array Operations — When you click Array Operations from the SRDF/CE Group Operations right-click menu, the Array Operations dialog box opens (Figure 105). The Storage Group Name list shows the single corresponding SYMCLI group that is the same as the SRDF/CE group. There is only one SYMCLI group per SRDF/CE group.

Figure 105 Array Operations dialog box

Current Status shows the current group RDF state, whether the SRDF/Asynchronous failover method is normal or immediate, whether the group is SRDF/Synchronous or SRDF/Asynchronous, and if R1-R2 Swap is enabled.

In the Group Actions area you can Synchronize Disks or set the R1-R2 Swap. The R1-R2 Swap option enables you to automatically swap the RDF device designations of a specified device group so that source R1 device(s) become target R2 device(s) and target R2 device(s) become source R1 device(s). If an existing group is being enabled for dynamic swap and this group is currently live on the R2 side, the change will not take


190


effect until either the group is failed back, or the group is taken offline, and then placed online. If the group is not dynamic swap-capable, this option will be grayed out.

The SRDF/Asynchronous Failover Method option allows you to determine how the groups (if you have SRDF/A-enabled groups) are split after failover (Normal or Immediate). The Normal option splits the groups at the end of the current bundle cycle, ensuring a consistent copy. Normal is the default. The Immediate option splits the groups immediately after failover and therefore the copy is indeterminate (that is, the data copy may or may not be consistent). Only select the Immediate option if the consistency of the data is not important. The Immediate option deletes the pending tracks and immediately performs the failover, with a potential loss of data.

The Set SRDF Protocol Type area also allows you to designate the group as SRDF/Synchronous or SRDF/Asynchronous. If the group is not SRDF/Asynchronous-capable, this option will be dimmed.

The Group Movement area includes the SRDF Failover option and SRDF Failback option. SRDF Failover performs an SRDF Symmetrix group failover of the selected cluster group. A failover is defined as the process of taking one or more resources offline on one cluster member and bringing them online on another cluster member. The SRDF Failback option performs an SRDF Symmetrix group failback of the selected cluster group. A failback is defined as the action of moving a resource back to the cluster member designated to be the resource's preferred owner. By default, resources are owned by their preferred owners, so a failback would only occur if the resource is moved from its preferred owner. This is likely the result of a failover.

CAUTION!Use the SRDF Failover and Failback options with caution; these options do not interact with MSCS, and therefore could result in data corruption/loss as well as cluster loss.



Storage componentThe Storage component lists the local and remote storage arrays connected to the node (Figure 106). For each EMC Symmetrix array, there are expanded lists of the R1 and R2 devices. For example, if you click the R1 Devices folder icon for a Symmetrix array, all the R1 devices associated with that Symmetrix array are listed, as shown in Figure 107 and Figure 108 on page 192. R1 device information that appears in the right pane is described in Table 4 on page 193.


SRDF/CE uses a dedicated SYMCLI/SYMAPI database. If you make storage configuration changes, you must update the SRDF/CE database by either issuing a Symmetrix SRDF/CE Discover, or manually issuing the set symcli_db_file=<fully-qualified db file location> command and then the manual SYMCLI discover command, symcfg disc.

Note: A major change in the SRDF/CE configuration is the physical device information is now based on the disk signature instead of the SYMAPI database. This allows you to view disks on any node, regardless of where the disks are online.

Figure 106 Expanded storage view

Storage component 191

192


Figure 107 Symmetrix device detail view

Figure 108 Symmetrix device detail view (continued)



Table 4 R1/R2 device description column headings




Dynamic Swap Capable Indicates whether the device is enabled for dynamic swap. Refer to “SRDF/CE SRDF swap support” on page 28 for more information on dynamic swap.

Group Name Indicates the SRDF/CE group name to which the device belongs.


Owning Node Shows the MSCS node name that owns this particular disk. This information is obtained directly from MSCS. If the disk is not part of MSCS, or if MSCS is not installed on the local node, then the owning node field will be blank.

Owner SymDevID Indicates the SymDevID on the owning node (that is, the MSCS owning node) corresponding to the local SymDevID. For example, in Figure 107 on page 192, Symmetrix device 0B4 (which is part of the quorum group) is mapped to 0B4 on the remote Symmetrix array. Since this device is managed by MSCS and is online on API1190 (which is connected to the remote Symmetrix array), the Owner SymDevId shows 0B4 (as seen from API1190).

Physical Device Shows the physical device number as seen on the local node for a particular Symmetrix device. If that device is managed by MSCS, then the physical drive number will be that of the owning node.For example, in Figure 107 on page 192, Symmetrix device 0B5 is part of MSCS and is online on API1191. On API1190 this device is mapped to \\.\PHYSICALDRIVE9.


RA Group Indicates the RA group to which the device belongs.

Signature Indicates whether a disk signature was found for a particular device. If a disk signature cannot be found (or is not present), the disk will be unusable from the point of view of SRDF/CE, and you cannot add this disk to a group until a valid signature is written.

SRDF/Async Capable Shows whether the device can be used in an SRDF/Asynchronous environment.

Symmetrix Device Shows all the R1/R2 Symmetrix device IDs that are mapped to the host.


194


In summary, the columns Symmetrix Device, Group Name, R1/R2, Device Status, Capacity (MB), and RA Group pertain to the local node. The Owning Node column lists the node where the device is online in MSCS. The Owner SymDevID column pertains to the owning node. The Physical Device and Mount Point columns pertain to the owning node if the device is managed by MSCS. If the device is not managed by MSCS, these columns pertain to the local node. The Dynamic Swap Capable, SRDF/A Capable, and Signature columns pertain to the device itself.

Select which columns to view in the right pane by selecting View, Add/Remove Columns in Windows Server 2003 (Figure 109).

Figure 109 Selecting columns to view



The Modify Columns dialog box appears (Figure 110).

Figure 110 Modifying R1/R2 device columns

Select which rows to view in the right pane by selecting View, Add/Remove Rows (Figure 109 on page 194). The Add/Remove Rows dialog box appears, as shown in Figure 111. This option can be used to remove individual grouped devices from the display list in the right pane.

Figure 111 Adding and removing R1/R2 device rows


196


A feature of the SRDF/CE MMC snap-in is ungrouped devices can be dragged and dropped into an existing SRDF/CE device group. As shown in Figure 112, device-type checking is performed and you are prompted to confirm the move. Note that when you select one device with a mount point, all devices mounted to each other are selected. If you choose to continue, another message will confirm that all devices have been added. Refer to “Mount point support” on page 33 for more details about the SRDF/CE implementation of mount points.

Figure 112 Drag and drop a device into a device group

If the device being added is found to be a raw device (that is, a device without a NTFS file system installed on it and/or without a drive letter assigned to it), that device will be added to the SRDF/CE device group, but not added to MSCS. To do this, manually create the MSCS resource. Raw device and MSCS interaction has proven to be unpredictable.

Note: Prior to dragging and dropping a disk into a group, verify that the disk is write-enabled on the side where you want to perform the drag-and-drop and that the disk has a mount point assigned.



Devices in an SRDF/CE group must be defined as either Ready or Write Disabled. Not Ready devices can cause MSCS to hang. As with the case of raw devices, Not Ready devices will be added to the SRDF/CE group but not added to the corresponding MSCS group.

Note: MSCS operates on the principle all storage is local in nature. Because of this limitation, MSCS must see all the devices on all nodes in the cluster. To accomplish this, when you set up the Symmetrix devices to be used by the SRDF/CE cluster, do not define these devices as Not Ready.

As shown in Figure 113, you can delete a device from a device group. However, when you select one device with a mount point, all devices mounted to each other are selected. If you choose to continue, another message will confirm that all devices have been deleted.

Figure 113 Deleting a device from a device group

You can list information about a Symmetrix array, unlock features on a Symmetrix array, or replace a remote Symmetrix array with another, by right-clicking on the Storage icon and selecting Symmetrix Options (Figure 114 on page 198). The Symmetrix options include List, Unlock, and Replace Remote Storage Array.


198


Figure 114 Storage right-click menu

The List option displays statistics about a particular Symmetrix array, as shown in Figure 115.

Figure 115 Listing Symmetrix information



The Unlock option unlocks any Symmetrix external locks currently in place. The Unlock option allows you to release the arbitration cluster address, arbitration lock, arbitration reserve lock, and the RDF lock (Figure 116).

CAUTION!Use Unlock with caution and only with a thorough understanding of this function.

Figure 116 Unlocking Symmetrix arrays


200


The Replace Remote Storage Array option enables you to replace the remote (and only the remote) Symmetrix array. This function can be used to replace a Symmetrix array in the case of a physical site outage, a lease change, or other event where the remote Symmetrix array must be replaced. Replacing the remote array does not impact the running lateral nodes, but it does presume the peer nodes on the remote side of the SRDF link are down. If you select this option, the dialog boxes shown in Figures 117 through 119 will appear.

Figure 117 Selecting the remote Symmetrix array

Figure 118 Configuring the remote storage array

Figure 119 Completing the replacement of the remote storage array



If you choose to replace the remote storage array, complete the following steps:

1. Ensure the remote nodes are down.

2. Recable/rezone the new remote Symmetrix array to the remote nodes.

3. Reconfigure the new remote Symmetrix array to be the SRDF pair for the existing local Symmetrix array and all necessary devices.

4. Verify on the local side the necessary R1/R2 mappings have occurred.

5. Run Replace Remote Storage Array. This will validate the R1/R2 mappings on all local nodes. Ensure the local Symmetrix array is connected and configured for all devices to the remote Symmetrix array. This will update all local nodes with the new remote Symmetrix array information.

Once you have successfully completed the Replace Remote Storage Array function, perform a Pull Configuration operation for each remote peer node to synchronize the remote nodes back into the SRDF/CE cluster.

CAUTION!Use Replace Remote Storage Array with extreme caution and only with a thorough understanding of this function.


202


Nodes componentThe Nodes component lists the current set of SRDF/CE cluster member nodes (not necessarily the MSCS cluster member nodes). For each node listed as part of the SRDF/CE cluster, the status of the SRDF/CE Service and SRDF/CE COM Server are listed (either running/stopped). The IP address and configuration timestamp for each SRDF/CE cluster member node are also listed (Figure 120).

Figure 120 Expanded node view



Add Cluster Member NodeTo add a new cluster member node, select Add Cluster Member Node from the Nodes right-click menu (Figure 121). If SRDF/CE cannot find a specified host, you are presented with the option to do a Force Add of the host. The Add Host Node dialog box appears as shown in Figure 122.

Figure 121 Nodes right-click menu

Figure 122 Adding a host node

Nodes component 203

204


As shown in Figure 123, by right-clicking on a specific cluster member node name, you can:

◆ Start and stop a cluster member node’s SRDF/CE Service.◆ Perform an IP discover.◆ Edit IP settings.◆ Push or pull configuration to/from peer nodes. Pushing the

configuration to all nodes unconditionally forwards the configuration from the local node to all other peer nodes and resets their configuration timestamps to equal that of the local node.

Note: Pushing or pulling configurations from one node to another overwrites the entire configuration of the target node.

Figure 123 Individual peer node right-click menu



Update Lateral/Peer DesignationsIn the event of a manual configuration change, GeoSpan to SRDF/CE upgrade, or other events it is sometimes necessary to update the SRDF/CE configuration to note what SRDF/CE nodes are lateral and which are peer to each other. This option causes all SRDF/CE nodes to update their view of this relationship. To do this, select Update Lateral/Peer Designations from the Nodes right-click menu as shown in Figure 124.

Figure 124 Selecting Update Lateral/Peer Designations

SRDF/CE warnings, or alerts, appear in the bottom-right corner of your desktop. For more information on SRDF/CE alerts, refer to Appendix A, “SRDF/CE Alerts and Errors.”

Nodes component 205

206



6Invisible Body Tag

This chapter describes administrative procedures for routine operations, maintenance, and troubleshooting of SRDF/CE. SRDF/CE, once installed and configured, requires minimal maintenance.

For those occasions requiring administrative intervention, refer to this chapter, which walks you through the procedures to ensure SRDF/CE operates properly in a complex environment.

◆ Overview........................................................................................... 208◆ SRDF/CE unique cluster behavior................................................ 210◆ SRDF/CE administration................................................................ 211◆ Restore/recovery operations.......................................................... 216◆ Upgrade procedures ........................................................................ 223◆ Configuration troubleshooting ...................................................... 228◆ Errors and warnings ........................................................................ 234◆ Best practices..................................................................................... 237◆ URLs to additional information..................................................... 241

Administration

Administration 207

208

Administration

OverviewSRDF/CE is an EMC application that operates in a complex environment and works with other products such as:

◆ Microsoft — Microsoft Windows Server 2003 Enterprise and Datacenter Editions and Microsoft Cluster Service (MSCS) clusters

◆ EMC Corporation — EMC SRDF, EMC TimeFinder®, Device Masking, PowerPath, and others

◆ User applications — SQL Server, Exchange, Lotus Notes, and others

Basic function The basic function of SRDF/CE is to relocate data volumes from one system to another, using an SRDF connection. When a volume is set offline on one server and made online on another server, some applications could potentially be impacted by this configuration change.

Likewise, if a server is taken offline to add new equipment, just recently purchased, and then brought back online in an SRDF/CE cluster, the potential exists for impact on MSCS and SRDF/CE operations.

Basic understanding Therefore, to perform administrative functions properly in this complex environment, you must fully understand:

◆ What the functions are doing.

◆ How they may interact with the operations of SRDF/CE.

Note: Chapter 1, ”Introduction” addresses the concepts pertaining to SRDF/CE operation.

That level of knowledge must be combined with an understanding of how other products work together in the complex environment. An example of what administrators of an SRDF/CE cluster should know or recognize is you cannot perform an EMC SRDF commit together with failover/failback procedures.

This chapter is the first step in helping to address those issues.


Administration

SRDF/CE Configuration UtilityAs discussed in Chapter 5, use the SRDF/CE Configuration Utility (refer to Figure 65 on page 150) to set up and configure SRDF/CE and to control SRDF disk configurations and MSCS resource definitions.

Note: Chapter 5, ”SRDF/CE Configuration Utility GUI,” describes the components of the SRDF/CE Configuration Utility. Refer to this chapter prior to making any configuration changes. This information is also documented in the Help utility.

Using Cluster Administrator and Configuration UtilityCluster Administrator Use MSCS Cluster Administrator for all normal cluster operations.

SRDF/CE ConfigurationUtility

Initially, the SRDF/CE Configuration Utility is used to set up and configure SRDF/CE during installation. Thereafter, use the SRDF/CE Configuration Utility to reconfigure resources. This utility can also be useful in troubleshooting certain error conditions.

Multicluster configurationsMultiple SRDF/CE cluster environments can be supported by a single pair of Symmetrix systems. Anyone who administers the cluster must be aware if several nodes fail simultaneously, the SRDF/CE resource groups fail over one at a time. This is normal SRDF/CE behavior.

Note: The failover of resources from different clusters can be intermixed.

Overview 209

210

Administration

SRDF/CE unique cluster behaviorPhysical differences between local and SRDF/CE clusters introduce the possibility for failure and operating modes impossible with the other configuration. For example:

◆ With local clusters use shared physical disks, destruction of physical storage while connected to one node eliminates the possibility of the other node picking up the device and operating successfully.

◆ With SRDF/CE clusters, the remote node can resume operation using the copy of the destroyed device data.

SRDF/CEconsiderations

When a site failure occurs, determine:

◆ Whether the other node is actually dead (as can be determined by network and SRDF links).

◆ Whether you want this second copy of the data to be modified.

Example In the event that both nodes in a two-node configuration are still operational but cannot communicate, two distinct copies of data— each modified at a different host by a different application—become possible. This situation is not possible under normal clusters. To work around this fragile situation, proceed carefully.

For planning purposes, decide:

◆ How well the sites and their interconnects are protected.

◆ How important is it to stay online and operational versus the cost of having multiple modified copies of data.

Additional application-specific resources provide additional dependencies. SRDF/CE operates using MSCS defined rules. For example, in the special case of a site failure, care must be taken to determine which type of site failure occurred (refer to “SRDF/CE unique behavior” on page 53) before proceeding with the recovery actions.


Administration

SRDF/CE administrationThis section describes the following administrative procedures:

◆ “Stopping the Cluster Service” on page 211

◆ “Uninstalling the Cluster Service on a Windows Server 2003 cluster” on page 211

◆ “Manually starting the Cluster Server Services” on page 212

◆ “Modifying cluster groups” on page 213

◆ “Renaming groups” on page 213

◆ “Configuring additional disk devices” on page 214

Stopping the Cluster Service To stop the Cluster Service:

1. Take all groups except the Quorum group offline.

2. Exit Cluster Administrator.

3. Stop the Cluster Server Service on the node without the Quorum group.

4. Stop the Cluster Server Service on the remaining node.

Note: If the MSCS service fails to stop, go into the task manager and stop the clussvc.exe process and any resrcmon.exe processes.

Stopping the SRDF/CE Service is optional anytime after the Cluster Service is stopped on a node.

Uninstalling the Cluster Service on a Windows Server 2003 clusterIn Windows Server 2003, the Cluster Service is installed along with the operating system. You cannot remove the Cluster Service, but you can return it to an unconfigured state. For more information, refer to the Microsoft Knowledge Base article at:


SRDF/CE administration 211


212

Administration

In summary, the steps to uninstall the Cluster Service on a Windows Server 2003 cluster are:

1. Start Cluster Administrator (CluAdmin.exe).

2. Right-click the node, and select Stop the Cluster service.

Note: Do not perform this next step if this server is the last node in the cluster.

3. Right-click the node, and select Evict Node. This step returns the cluster to its original unconfigured state. You can add it later to the same or to a different cluster.

If you cannot start the Cluster Service, or if you have trouble removing the node, you can manually deconfigure the Cluster Service by doing the following:

1. Run the Cmd.exe program to open a command prompt.

2. At the command prompt, type cluster node nodename /forcecleanup, and then press Enter.

Note: If the Cluster Service does not exist in the Registry, the command does not respond. To create a place holder, type sc create clussvc at the command line, and then press Enter.

Manually starting the Cluster Server ServicesTo manually start the Cluster Server Services:

1. Boot all nodes

2. Ensure the SRDF/CE Service started on all of the nodes. If you stopped the SRDF/CE Service on any nodes, start the SRDF/CE Service on all nodes.

3. Manually start the Cluster Server Service on the node with the Quorum last (the last node shut down in the previous procedure).

4. Wait for the MSCS Service startup to complete, and then open Cluster Administrator to connect to the cluster. Verify the cluster is operational.

5. If not already started, start the SRDF/CE service on those particular nodes remaining in the cluster. Then start the Cluster Server Service on those nodes.


Administration

6. Use Cluster Administrator to move the groups back to their R1 sides. This happens quickly because no SRDF operation is involved.

7. Bring all groups online.

Modifying cluster groupsIt may be necessary to move devices from one group to another occasionally. Within MSCS, right-click a resource and select the Change Group option to move the disk to another group. When SRDF/CE is installed into the cluster, the move operation is complicated by introducing SRDF and SYMAPI groups. To move a disk from one group to another without updating the SYMAPI database, the moved device would not fail over when commanded to do so.

If you want to remove or delete one disk from a group that contains many disks, do the following:

1. Within the SRDF/CE Configuration Utility, open the Groups component and click the group containing the device in question. In the right pane listing all disks, right-click the disk to remove, and select Delete. This will remove the disk and the corresponding GeoSpan_res resource from this group as well as from MSCS.

2. Now add this device to another group.

Renaming groupsRenaming a cluster group name affects the functionality of the SRDF/CE product. If you rename a MSCS cluster group, you can failover/back without issue — but only if the new group name adheres to the SRDF/CE naming standards. The only time you will experience problems is if you modify the disk group in any way. If you deconfigure the group or add a disk to this group you will experience problems if the cluster group name and the SRDF/CE group name do not match. If you need to permanently rename a group, it is best to deconfigure the group, and then re-create it with the new name.


214

Administration

If you do rename a group within an MSCS cluster and later forget the original name , look at the name of the GeoSpan_res resource. Its name will not change unless you deconfigure it through the SRDF/CE Configuration Utility.

Configuring additional disk devicesThis procedure assumes you added the disk to the SCSI bus. If you have not, please do so before continuing.

This procedure describes configuring additional disk devices for a two-node system. If you have more than two nodes, perform the steps described for Node 2 on all the remaining nodes in the cluster.

1. Disable the Cluster Service and reboot all nodes after the storage is added to all nodes. The reason this reboot is necessary is that MSCS does not read a node’s device information except at boot time and we need to add a new disk definition without MSCS involved.

2. After the reboot, use the syminq command on all nodes to verify you can see the new storage.

3. Open the SRDF/CE Configuration Utility on all nodes and right-click EMC and select Discover. This updates the SRDF/CE SYMAPI database to see the newly added device.

4. Within the SRDF/CE Configuration Utility on Node 1, create a temporary group for the newly added disk. Use this temporary group to move the disk between nodes to get the disk signature and drive letter assignments synchronized on both nodes. Move the disk group to the Node 1 side. Once this is done, open Disk Administrator and create a new partition for the device. Format, label, and assign a drive letter. You may need to delete/modify the disk key to make this disk appear normally. Do not proceed to the next step until the disk is formatted and labeled.

5. Once you are done on Node 1, use SRDF/CE to move the disk group to the Node 2 side. Once complete, open Disk Administrator on Node 2 and assign the proper drive letter to the new device. You may need to delete/modify the disk key to make this disk appear normally. If you must delete the disk key, ensure you reassign drive letters to all devices. Do not proceed to the next step until you can see the disk with the proper drive letter on each side.


Administration

6. Once the temporary group is transitioned to all nodes, open the SRDF/CE Configuration Utility on Node 1 and delete the temporary group created for the device.

7. On all nodes, reconfigure the Cluster Service to Automatic startup and reboot.

8. To create a new SRDF/CE group or add a disk to an existing group, use the SRDF/CE Configuration Utility.

9. Test failing over the new group between nodes to ensure the configuration is correct.

10. Once you add a new device, update your Checkpoint file within the SRDF/CE Configuration Utility. Do this by right-clicking Control and selecting SRDF/CE Config Options, Check Point.


216

Administration

Restore/recovery operationsThis section describes the following restore and recovery operations:

◆ “Restoring from site failure”

◆ “Recovery from site failure” on page 218

◆ “Recovery from total IP communications failure” on page 219

◆ “Recovery from HBA or SCSI channel failure” on page 220

◆ “Recovery from an SRDF link failure” on page 220

◆ “Recovery from a corrupt quorum log” on page 222

◆ “Symmetrix replacement from the R1 side” on page 222

Restoring from site failureThe following procedure describes how to restore your storage system after a site failure occurs with all links lost (that is, with one side down).

1. Restore the SRDF link.

2. Open the SRDF/CE Configuration Utility and verify the groups moved are in a Split state. All other groups should be Synchronized or Failed over.

3. Go to a command prompt and enter:

set symcli_db_file=SRDF/CE home directory\SRDFCESymapi.db

4. At the command prompt, enter:

symdg list

This should list the SRDF/CE groups.

5. Choose one of the groups that are in a Split status and enter the following:

symrdf –g GroupName query

This should report the status of the disks within this group.

CAUTION!Contact EMC Customer Support before proceeding to the next step.


Administration

6. Before you establish/restore the disks, take the disk group offline within Microsoft Cluster Administrator.

7. To get the disks out of the Split state, decide which side of the R1/R2 pair has the most recent data. Using the wrong command here causes data loss. To synchronize the RDF devices, use one of the following SYMCLI commands:

• If the R1 owns the good data, enter: symrdf –g GroupName establish

• If the R2 owns the good data, enter: symrdf -g GroupName restore

Note: This process could take significant time, as this is a full establish/restore.

8. After you establish/restore the disk, it takes time to resynchronize the drives. To check the status of the establish/restore, enter:

symrdf –g GroupName query

9. When the devices are done with the establish/restore, the devices are left in a synchronized state. This means the device is read/write-enabled on the R1 side and write-disabled on the R2 side. Any devices you did a restore operation on have moved to the R1 side. Use the SRDF/CE Configuration Utility to perform an SRDF failover operation on these device groups to bring them back to the R2 side. Access the SRDF Failover option in the Configuration Utility by right-clicking any group under the Groups component and selecting SRDF/CE Group Operations, Array Operations, SRDF Failover.

10. Before starting the Node 1 services, ensure all the disks are located on the surviving node within SRDF/CE. For example, if the surviving node is the SRDF/CE slave node, then all R1 devices should read Synchronized and all R2 groups will be in a Failed over status.

11. Restore the previously dead node.

12. Start the SRDF/CE Service and then the Cluster Server Service.

13. Use MSCS to move the groups back to their original locations.

Restore/recovery operations 217

218

Administration

Recovery from site failureThe following procedure describes how to recover from a site failure with all links lost (that is, with one side down).

1. Open the MSCS Administrator on the surviving node and verify which groups have failed. If the quorum group failed, it must come online first before the failover of the other groups can happen.

2. Decide which groups should be brought online on the surviving node. All groups do not have to be brought online. If you do not want to bring any groups online simply restore the SRDF connection and recover from there. Otherwise, use the following process to move groups to the surviving node:

a. Using SRDF/CE, set the SRDF Override to Yes.

b. Use MSCS to bring the quorum group online. The GeoSpan_res may fail to come online. This is acceptable behavior and you should just bring this resource online. Do not proceed until both the quorum disk and the GeoSpan_res are online. On Windows 2000, if you lost the node that owned the quorum disk, you must attempt to restart the Cluster Service to bring the groups online. If this does not work, you must do the following from a command line:

– To set SYMCLI to see the SRDF/CE API database, enter:set symcli_db_file=SRDF/CE home directory\SRDFCESymapi.db

– Use symdg list to verify you can now see the SRDF/CE disk groups

– For each device group, enter the following command:symrdf -g GroupName rw_enable R1|R2

Select R1 or R2 depending on which device is local to this machine.

– Once all devices are read/write enabled on this node, start the Cluster Service

c. Once the quorum is online, bring each group online one at a time. Wait for each resource within a group to completely come online before proceeding to the next group.


Administration

d. Once all groups are online on the surviving node, use SRDF/CE to set the SRDF Override to No.

Note: If Site Mode is set to FailStop, change the Site Mode to No New Onlines or leave the SRDF Override on. Otherwise, the groups go offline when the override is turned off.

Recovery from total IP communications failureWhen all IP communications between nodes are lost, all Cluster groups fail over to the node that owns the quorum device. Additionally, in Windows 2000 only, if the node that owned the quorum no longer has access to any IP communications (NIC failure/switch failure) all IP resources within the cluster fail and any resources that depend on the IP (network names and so on) go offline.

To recover from this scenario:

1. Stop the Cluster Service on the node(s) that does (do) not own the groups.

2. Restore IP communications.

3. Bring all groups online.

4. Start Cluster Service on the node(s) without the groups.

To change group ownership while the IP links are down, do the following:

1. Stop the Cluster Service on the node(s) that does (do) not own the groups.

2. Stop the Cluster Service on the other node(s).

3. Start the Cluster Service on the node(s) you want to bring all groups online.

You cannot have groups active on more than one node during total IP communications failure.


220

Administration

Recovery from HBA or SCSI channel failureWhen one node loses connection to the storage, whether it is cable, HBA, or some other type of channel failure, all groups on the node with the channel failure will fail over to the surviving node. Also, the SRDF/CE Configuration Utility will display an SRDF warning on the node with the failure.

To recover from this type of failure:

1. Stop the Cluster Service on the node with the channel failure.

2. Restore connection to the storage.

3. If applicable, reopen the paths to the storage within PowerPath.

4. Once the connection is reestablished to the storage, the SRDF warning within SRDF/CE should go away.

5. Start the Cluster Service.

Note: If the disks are not visible, issue a bus rescan from within the disk scan utility, then issue a Windows dcomcnfg remove command and the manual SYMCLI discover command, symcfg disc. If the disks are still not visible, perform a bus rescan again. It might be necessary to reboot before starting the Cluster Service.

Recovery from an SRDF link failureIf the SRDF link goes down, one of the following happens depending on site mode settings:

◆ Site Mode set to No New Onlines

• SRDF/CE detects the SRDF failure and the SRDF Warning displays within SRDF/CE.

• All MSCS groups remain online. If Local Override is enabled, the groups may be taken offline or online normally, and moved to lateral nodes.

• If an MSCS group attempts to fail over or move to a different site, SRDF/CE does not allow it and the group goes offline. If you must fail this group over, use the procedure from “Recovery from site failure” on page 218 to move the group using SRDF Override. The group is in a split state if it is moved.


Administration

CAUTION!If you use SRDF Override, the device will be read/write enabled on both sites. If Local Override is enabled, MSCS will allow a group to be moved between sites even though the RDF Link is down and no data is being replicated. To prevent this situation, you should write-disable the device on the site from which you moved the group. To do so, go to a command prompt and enter the following SYMCLI commands:

set symcli_db_file=SRDF/CE HomeDirectory\SRDFCESymapi.dbsymrdf -g GroupName write_disable R1|R2

Select R1 or R2 depending on which site originally had the group. For example, if you used SRDF Override on the R2 site, you should enter the above commands on the R1 site and select R1.

◆ Site Mode set to FailStop

• SRDF/CE detects the SRDF failure and the SRDF Warning displays within SRDF/CE.

• All MSCS groups go offline.

If you want to bring any groups online, set the SRDF Override to Yes and bring the groups online within MSCS. Bring the quorum group online first!

With the override set, you can fail over a group and this will cause the groups to be in a split state when the link is restored.

To recover from this failure:

1. Restore the SRDF link.

2. Once the SRDF link is restored, SRDF/CE should no longer display an SRDF warning.

If you moved any groups between sites while the SRDF link went down, use the same recovery procedure described in “Restoring from site failure” on page 216 to get groups out of a Split state.

If you did not move a group between sites but the devices are in the Suspended state, go to a command prompt and enter the following SYMCLI commands:

set symcli_db_file=SRDF/CE HomeDirectory\SRDFCESymapi.dbsymrdf -g GroupName resume


222

Administration

Recovery from a corrupt quorum log The complete Microsoft article can be found on Microsoft Knowledge Base Article 172951 at the following URL:


Symmetrix replacement from the R1 sideThe following process assumes the Symmetrix replacement is being done on the R1 side and there are no Dynamic RDF groups configured on either Symmetrix array.

1. Change the MSCS service start up to manual on all cluster nodes.

2. Ensure all groups are online on the R2 side, and shut down all cluster nodes on R1 and R2.

3. Replace the R1 Symmetrix and perform the appropriate configuration change to the R2 side as well.

4. Once the configuration change is made, and the new Symmetrix array is in place, bring the RDF link up.

5. After performing the appropriate zoning and device masking operations, bring up the R1 and R2 nodes. Open the snap-in and perform a Symmetrix discover (symcfg discover) on each node. Start the SRDF/CE service, starting with the surviving node. It is important the discover operation is performed prior to starting the service; failure to do so may result in MSCS malfunction.

6. On the R2 node (the surviving node), right click on the storage icon and do symreplace. Verify and select the new Symmetrix serial number, then click OK. Follow the next step the system prompts, to pull from the surviving node.

7. Using the SYMCLI, perform a full RDF restore for each group.

8. Once all groups are in synchronized state, start MSCS on the R1 node. Verify all groups came online. Then start MSCS on the R2 node.



Administration

Upgrade proceduresThis section describes the following upgrade procedures:

◆ Upgrading a node in an SRDF/CE cluster from Windows 2000 to Windows Server 2003 (refer to the next section)

◆ Upgrading a quorum disk (refer to page 226)

Upgrading Windows 2000 to Windows Server 2003The following procedure describes how to upgrade a node in an SRDF/CE cluster from Windows 2000 to Windows Server 2003.

This procedure describes an upgrade for a two-node system. If you have more than two nodes, perform the steps described for Node 2 on all remaining nodes in the cluster.

1. Verify you have moved all the available groups to Node 1 (the primary node).

Note: Verify the SRDF/CE and Cluster Services are set to Manual mode before you start upgrading the system.

2. On Node 2 (the secondary node):

a. Select Start, Programs, Administrative Tools, Services and stop the Cluster Service and SRDF/CE Service, respectively.

b. Close the Cluster Administrator and the SRDF/CE Configuration Utility.

c. Insert the Windows Server 2003 CD in the CD drive.

d. Select Install Windows Server 2003 Enterprise Edition.

e. Select Upgrade (Recommended) in the Welcome to Setup Windows dialog box, and click Next.

f. Select I accept the agreement in the License Agreement dialog box, and click Next.

g. Enter the product key provided and click Next.

Upgrade procedures 223

224

Administration

h. The Windows Server 2003, Enterprise Edition Setup dialog box appears as shown in Figure 125. In the Report System Compatibility window, take notice of any compatibility issues listed. If you do not want to correct any of the listed issues, click Next to continue.

Figure 125 Windows Server 2003 Enterprise Edition Setup dialog box

i. Once you have finished updating from Windows 2000 to Windows Server 2003, reboot the system and come up in Safe mode.

j. Once up in Safe mode, change the Cluster Service from Automatic start to Disabled.

k. Reboot normally.

l. You will not be able to see any attached devices. Change the HBA driver from the Windows 2000 version to the Windows Server 2003 version. Once you have changed the driver, reboot normally.

m. Once you validate all devices can be seen change the Cluster Service startup to Automatic and start the Cluster Service.

3. On Node 2:

After the node boots up, follow the procedure detailed below to edit the Volume Shadow Copy Service Access Control Registry Entry to grant access.


Administration

Important: Before attempting this procedure to modify the Registry, EMC recommends reading article number 256986 in the Microsoft Knowledge Base, which can be accessed at the following URL:


This article provides information on how to back up, restore, and edit the Registry. Before you modify the Registry, back it up and understand how to restore the Registry if a problem occurs.

CAUTION!Using the Registry Editor incorrectly can result in serious problems requiring you to reinstall your operating system. Microsoft cannot guarantee you will solve problems resulting from using the Registry Editor incorrectly. Use Registry Editor at your own risk.

a. Click Start, Run, type regedit, and click OK.

b. Locate the following Registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl

Note: If the VssAccessControl subkey does not exist, add it. To do this, right-click VSS, Point to New, and then click Key. In the New Key #1 dialog box, enter VssAccessControl, and press Enter.

c. Right-click VssAccessControl, select New, and click DWORD Value.

d. In the New Value #1 dialog box, enter example.com\cluster account (where example.com is the name of the domain where the cluster account is located, and cluster account is the name of the cluster service account), and press Enter.

e. Right-click the new Registry value you added, and click Modify. In the Value data box, enter 1, and click OK.

f. Quit Registry Editor.


226

Administration

4. On Node 2:

a. Select Start, Programs, Administrative Tools, Services and first start the SRDF/CE Service, and then start the Cluster Service.

Note: You can change the SRDF/CE Service and the Cluster Service to Automatic. If you do this, verify the Cluster Service is dependent on the SRDF/CE Service.

b. Open the SRDF/CE Configuration Utility by selecting Start, Programs, SRDF/CE Configuration Utility.

c. Move all groups present on Node 1 to Node 2.

Note: Verify all groups have been moved successfully to Node 2.

5. On Node 1:

a. Perform the same procedures from Steps 2–4.

b. Both nodes should be up and running SRDF/CE and Cluster Administrator, and you should be able to move the groups from one node to another.

Upgrading a quorum diskThe following procedure describes how to upgrade a quorum disk with SRDF/CE:

1. Create a temporary SRDF/CE device group that contains only the new quorum disk.

2. Perform an MSCS Move operation to each of the nodes. This will validate the new quorum disk can be successfully brought online to each node.

Note: At each node note the Window’s physical drive number.

3. Perform an MSCS Move operation on the temporary quorum disk group back to the starting node.

4. Delete the temporary new quorum disk from within SRDF/CE.

5. Add the new quorum disk to the current quorum disk group from within SRDF/CE.


Administration

6. Change the quorum disk from within MSCS, using the published Microsoft MSCS procedures for changing the quorum disk. It is critical you follow this procedure:

a. It is recommended you perform the procedures on all the nodes on one side of the SRDF link, and then move to the other side of the SRDF link.

b. When you start working on the nodes on the remote side of the SRDF link, you will have to perform an SRDF failover of the quorum group.

7. When you have fully configured MSCS with the new quorum disk, go to each SRDF/CE node in the cluster, and use the SRDF/CE Configuration Utility to do the following:

a. Open the Control component.

b. Click Quorum Control.

c. In the right pane, right-click any of the displayed Quorum Control parameters, and select Edit Quorum Control Parameters.

d. Change the quorum disk parameters to match exactly what was specified to MSCS.

CAUTION!This is a critical step. If it is not followed your cluster may not be functional.

8. Reboot all nodes in the cluster.


228

Administration

Configuration troubleshooting

Overview This section troubleshoots the following configuration issues you may encounter while using SRDF/CE:

◆ “SRDF/CE group in wrong state (not ready/invalid/suspended)” on page 229

◆ “Unable to start cluster service” on page 230

◆ “GeoSpan_res.dll prevents MSCS group failover” on page 231

◆ “GeoSpan_res.dll stays offline after failback of the quorum group” on page 231

◆ “External SYMAPI locks” on page 231

◆ “Unable to see a newly added disk” on page 232

EMC technicalsupport

For technical support relating to SRDF/CE, contact EMC Customer Support.

Microsoft technicalsupport

The first step you should take in troubleshooting a Microsoft product is to reference the Microsoft Knowledge Base for possible solutions. Access the knowledge base online at:

http://support.microsoft.com

Another excellent resource for troubleshooting MSCS problems is the Microsoft Cluster Server Troubleshooting and Maintenance Guide. This guide is published at:

ftp://ftp.microsoft.com/bussys/winnt/winnt-docs/papers (download the mscs.exe file)


http://support.microsoft.com

ftp://ftp.microsoft.com/bussys/winnt/winnt-docs/papers/mscs.exe

Administration

SRDF/CE group in wrong state (not ready/invalid/suspended)When you create groups, if the status of the group is not synchronized, failed over, or R1 updated, the state is wrong. This is usually caused by an incomplete SRDF configuration. To fix this:

1. Set the SYMAPI database to point to the SRDF/CE database.

2. From a command prompt, enter:

SET SYMCLI_DB_FILE=SRDF-CE home directory\SRDFCESymapi.db

3. Verify the above command worked by typing SYMDG LIST. This should list the groups you created in SRDF/CE.

4. View the status of the groups. For example, to check the status of the group R1_Group (this is case-sensitive) enter:

symrdf –g R1_Group query

This command displays information about the disks within this group. To the far right, it displays the status of the devices. If the status here is anything other than synchronized, failed over, or R1 updated, issue the following command: symrdf –g R1_Group –force resume

5. Once you issue the command, check the group status by typing:

symrdf –g R1_Group query

You should see the devices begin to synchronize.

6. Once the group is completely synchronized or failed over, repeat Steps 3–5 for each device group in an improper status.

Configuration troubleshooting 229

230

Administration

Unable to start cluster serviceThere are several reasons why the MSCS service may not start:

◆ Attempting to start the Cluster Service on a node without access to the quorum.

◆ The quorum disk or the quorum log is corrupt.

◆ The cluster is in split-brain1 status.

◆ SRDF/CE Service must be started before MSCS. If it is not started, MSCS may not start.

To start Cluster Service:

1. Shut down the services on one node.

2. Power down that node.

3. On the surviving node (s), stop the MSCS and SRDF/CE Services if started.

4. Open the SRDF/CE Configuration Utility and move all disks to this side.

5. Start the SRDF/CE Service, and then start the Cluster Service. The services should come online for this node. If not, try rebooting this node.

6. Once the services have started, power on the other node and start its services. SRDF/CE must be started before MSCS.

7. If the above does not work, you may need to reinstall MSCS on all nodes. If you do reinstall, ensure you format the quorum disk prior to installation to remove all references to the previous cluster.

1. A total communication failure, while both nodes remain operational, is referred to as a split-brain condition and is a potential cause of logical data corruption. For example, if both sides assume the other is dead and begin processing new transactions against their copy of the data, two separate and unreconcilable copies of the data can be created.


Administration

GeoSpan_res.dll prevents MSCS group failoverWhen you complete the SRDF/CE installation, move a group within Cluster Administrator. When the Geospan_res attempts to come online, it tries to restart according to the threshold set, and then fails back to its original node. If the MSCS group does not fail over, an error occurred with the srdfce_svc service account.

Check the setting of the srdfce_svc service and ensure all nodes are using the same service account. Ensure that this account is a member of the domain administrator’s group as well as the local machine administrator’s group. Also check to see if this account has log on as a service rights.

GeoSpan_res.dll stays offline after failback of the quorum groupAfter SRDF/CE and MSCS are installed and properly configured, you attempt to move the quorum group between the cluster nodes. When you move the group, the quorum disk comes online, but the SRDF/CE resource remains offline. This is caused by a timeout value set within Microsoft cluster we have no control over for failover of the quorum disk. This is resolved by using a smaller physical quorum disk, or by simply deleting this resource from the quorum group. SRDF/CE uses the information contained within the Configuration Utility under Quorum Control first when failing over the quorum resource. The SRDF/CE resource is only used if the information placed here is incorrect. This resource can be deleted without having any affects on product functionality.

External SYMAPI locksUnder certain circumstances, the SYMAPI may leave the Symmetrix system in a locked state. The symptoms indicating you are in a locked state may include the following: unable to fail over devices, failure to move groups, or failure to start MSCS service. To check if you are in a locked state, go to a command line and enter the following:

symcfg -lock list

This command should echo the local and remote Symmetrix systems and their lock status (locked/unlocked). If you are in a locked state, identify the lock numbers to release these locks.


232

Administration

To locate the lock number, do the following:

1. In the SRDF/CE Configuration Utility, right-click Symmetrix and select Discover.

2. Once a successful Discover is made, right-click Symmetrix and select List. After a few moments, this displays a long list of many things within the Symmetrix array. Scroll through the document until you reach the Lock section (near the top). The Lock section runs through each bus and specifies lock information about each one. The typical report states No lock for 0 SYMAPI_C_SYM_LOCK_NOTFOUND and a lock displays lock information along with the length of time for the lock. Find which device number has the lock and record this number (typically 0 or 15).

To release the locks, enter the following from a command line:

symcfg -lockn LockNum release

Note: You can also unlock the Symmetrix array using the SRDF/CE Configuration Utility by right-clicking Symmetrix and selecting the Unlock option.

If you are unsuccessful, add the -force flag to force the release. After unlocking the Symmetrix array, verify the lock is released by typing symcfg -lock list. Once both Symmetrix arrays are unlocked, your issue should be resolved.

Unable to see a newly added disk Microsoft Cluster may not allow you to see the newly added device. This causes problems when you attempt to assign drive letters or manipulate the disk signature. To get cluster to release its grasp on the disk, do one of two things:

◆ Uninstall MSCS on one node, or

◆ Stop the Cluster Disk service on the Control Panel, Devices list. You can stop this by setting the startup value to manual and rebooting.

If you do not have the disk listed when you are adding the disk resource to your group, do the following:

1. Evict one node from the cluster.


Administration

2. Uninstall MSCS from this node.

3. Configure the disk on this node.

4. Reinstall MSCS software to join the cluster.

5. Reapply service packs and hot fixes.

6. Add the disk resource to your group.

7. Add the dependency to this disk resource.

Note: MSCS operates on the principle all storage is local in nature. Because of this limitation, MSCS must see all devices on all nodes in the cluster. To accomplish this, when you set up the Symmetrix devices to be used by the SRDF/CE cluster, do not define these devices as Not Ready.


234

Administration

Errors and warningsSRDF/CE is a service that depends on MSCS and the EMC Symmetrix API (SYMAPI) to function. There are few events caused and/or generated by the SRDF/CE Service. Virtually all SRDF/CE errors can be traced to problems with MSCS or the SYMAPI. Therefore, this section is not intended to be a troubleshooting guide or a definitive reference for the Windows event and application logs. This section explains the following errors/warnings you may encounter:

◆ “GeoSpan_res.dll errors”

◆ “SRDF warning” on page 235

◆ “IP communication warning” on page 235

◆ “Error while attempting to connect with MSCS” on page 236

◆ “MSCS error 170” on page 236

For more information on the alerts, event messages, and errors you may encounter while using the SRDF/CE Configuration Utility graphical user interface, refer to Appendix A, “SRDF/CE Alerts and Errors.”

GeoSpan_res.dll errorsWhen you install the GeoSpan_res.dll, you may receive an error within MSCS. To determine the severity of this error:

1. In MSCS, click the Resource Types folder and locate the geospan_res resource.

2. If the resource has a red circle around it, uninstall the resource and try reinstalling it.

3. If the resource fails the second time you install it, uninstall it again and attempt to install it from the opposite node. This may fix the problem.

Note: If the above did not work, check the path to the SRDF-CE home directory. These paths must be exactly the same for this resource to work properly. If you installed one operating system on the C:\ drive and the other on the D:\ drive, this will not work. The %systemroot% path also needs to be exactly the same on all nodes before you can continue with the SRDF/CE installation.


Administration

Important: If you get an error and the geospan_res appears to be operational, make a note of the error and continue with the installation. It is possible to experience false errors at this point; MSCS gets confused.

SRDF warning If the SRDF warning appears, the configuration is not correct. Check the following:

1. Ensure the SRDF/CE Service is started on both sides. This error will appear if the service is not started.

2. Check the cables. Check the SRDF connection as well as the connection to the HBA.

3. From each host, issue a symrdf ping command. If you can ping from both sides, you should have a valid connection.

4. If this does not resolve the problem, contact the EMC Customer Support for further assistance.

IP communication warningMany things can cause an IP communication warning error. Check the following:

1. Check the SRDF/CE and the Cluster Service are running. If not, this error appears.

2. Ping all adapters from each node.

3. If the IP communications warning is displayed on one side only, ping all adapters on the opposite node to verify connectivity.

If you can ping, this message is in error. This is a known issue and can only be cleared by a series of reboots.

Errors and warnings 235

236

Administration

Error while attempting to connect with MSCSMSCS uses a remote procedural call (RPC) to connect to the cluster. You see this error if the RPC service has not fully started. If you receive this error, do one of three things:

◆ Wait a few minutes and try again.

◆ Try connecting to the IP address of the cluster instead of the NetBIOS name.

◆ Enter a period (.) to connect using LPC to the local node as opposed to the cluster name/address.

Note: This must be resolved before MSCS will operate properly.

MSCS error 170This error is issued when MSCS attempts to arbitrate for a device the opposite node already owns. In an SRDF/CE configuration, this can be caused by improper setup of the SRDF/CE arbitration numbers. Each cluster needs to use a different set of arbitration numbers. Each node within the cluster must use the same arbitration number.

If the cluster member nodes do not use the same numbers, you may not fail groups over to the opposite node. If MSCS logging is enabled, you will see the Error 170 message throughout the cluster log.

If two separate clusters attempt to use the same arbitration numbers, the first cluster to attempt to come online should do so, but the second cluster will fail to start the Cluster Service. If MSCS logging is enabled, you will see the Error 170 message throughout the log.

To correct this problem, ensure each cluster uses a unique set of arbitration numbers. Ensure each node within a cluster uses the same arbitration numbers.

Note: This error can also be caused by two or more nodes attempting to gain control of the device at the same time. This could happen if two or more nodes reboot at the same time. This is described in Microsoft KB article 249194, at: http://support.microsoft.com/kb/q249194.


Administration

Best practicesThis section describes the following best practices that pertain to both MSCS and SRDF/CE:

◆ “How to delay loading of MSCS service (change service dependencies)”

◆ “Enabling groups to come online while SyncInProg” on page 239

◆ “Changing the quorum log size” on page 239

How to delay loading of MSCS service (change service dependencies)Please read the following article before attempting to implement this procedure:


1. Open regedt32.exe and find the following Registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ClusSvc

2. Select the subkey and in the right-hand side, double-click the DependOnService field.

3. When the Data dialog box appears, enter the name or names of the services you prefer to start before this service with one entry for each line. Enter srdfce_svc, and click OK. Ensure you do not delete the current values.

4. Close the Registry Editor.

Note: The use of any type of fault-tolerant adapter or teaming for the heartbeat is not recommended. If you require redundancy for your heartbeat connection, use multiple network adapters set to Internal Communication Only and define their network priority in the Cluster configuration. Microsoft also does not recommend using a multiported network adapter. Please refer to your NIC manufacturer for information as to its compatibility on a Server Cluster.

5. Click Internet Protocol (TCP/IP), and select Properties.

Best practices 237

238

Administration

6. On the General tab, verify you have selected a static IP address not on the same subnet or network as another one of the public network adapters. An example of good IP addresses to use for the private adapters is 10.10.10.10 on Node 1 and 10.10.10.11 on Node 2 with a subnet mask of 255.255.255.0.

7. Ensure there is no value set in the Default Gateway box.

8. Verify there are no values defined in the Use the following DNS server addresses box.

9. Click Advanced.

10. On the DNS tab, verify there are no values defined. Ensure the Register this connection's addresses in DNS and Use this connection's DNS suffix in DNS registration checkboxes are cleared.

11. On the WINS tab, verify there are no values defined. Click Disable NetBIOS over TCP/IP.

12. When you close the dialog box, you may receive the following prompt. If you receive this prompt, click Yes:

This connection has an empty primary WINS address. Do you want to continue?

13. If you are using a crossover cable for your private heartbeat interconnect, disable the TCP/IP stack destruction feature of Media Sense.

Add the following Registry value to each node:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\ParametersValue Name: DisableDHCPMediaSenseData Type: REG_DWORDData: 1

For additional information about this, refer to Microsoft Knowledge Base article 254651 Cluster Network Role Changes Automatically at the following URL:


14. Complete the previous steps on all other nodes in the cluster.

15. Start Cluster Administrator.

16. Click the cluster name at the root of Administrator. On the File menu, select Properties.


Administration

17. On the Network Priority tab, verify the private network is listed at the top. If it is not, use the Move Up button to increase its priority.

18. Click the private network, and then click Properties.

19. Select the Enable this network for cluster use checkbox.

20. Click Internal cluster communications only (private network).

Enabling groups to come online while SyncInProgIf you are experiencing long failover times and want to speed up the failback process, manipulate the Registry to allow the cluster groups to come online while the SRDF devices are still being synchronized.

To do so, open up the Registry Editor and modify the following key:

HKEY_LOCAL_MACHINE\SOFTWARE\EMC\SRDF/CE for MSCS\Config\WaitTargetState

The Registry item WaitTargetState controls whether to allow devices to come online while in the SyncInProg state. Default value 0 means do not come online unless the devices are synchronized. Setting this value to 1 allows the devices to come online before the devices are fully synchronized. Restart the SRDF/CE Service for this to take effect.

CAUTION!Not waiting for synchronization can be very dangerous and is not recommended. If you lose the RDF links, a host, or a site during the resynchronization process and you attempt to fail over while the SyncInProg is still in progress, data loss can occur.

Changing the quorum log sizeThe default quorum log size of any cluster is 64 kilobytes (KB). If you increase the number of shares or transactions significantly, the quorum log size may be too small.

You might need to adjust the quorum log size when you use a large number of file shares. For example, an administrator adds new file shares that work properly, and then a failover occurs. The file shares are no longer displayed in a cluster.

Best practices 239

240

Administration

You may receive an event message indicating the quorum log size is too small.

To change the maximum quorum log size:

1. Open Cluster Administrator.

2. In the console tree, click the cluster. On the File menu, select Properties.

3. Click the Quorum tab.

4. In Reset quorum log at, enter the maximum size (in kilobytes) that the quorum log file can be before the cluster resets it.

Note: You do not need to restart the computer or service unless you are running a third-party tool, such as a quota or virus protection program, which might prevent the change from occurring.

Note: The cluster records all changes to the cluster database in the quorum log file. When the quorum log attains the specified size, the cluster saves the database and resets the log file.


Administration

URLs to additional informationThe following URLs provide additional information.

◆ Microsoft’s Cluster website:

http://www.microsoft.com/ntserver/ProductInfo/Enterprise/default.asp

◆ Microsoft Cluster Server Troubleshooting Guide (download):

ftp://ftp.microsoft.com/bussys/winnt/winnt-docs/papers/mscs.exe

◆ MSCS Installation Resources:


◆ Recommended WINS Configuration for Microsoft Cluster Server:


◆ How to Enable MSCS Cluster Logging:


◆ How to Change Service Dependencies (Delay Loading of MSCS Service):


◆ How to Add Disks to the Shared SCSI Bus:


◆ How to Enable File Auditing in MSCS:


◆ Replace Cluster Disks:


◆ Installing Exchange 5.5 into MSCS:

http://www.microsoft.com/exchange/en/55/help/default.htm

◆ Installing SQL into MSCS


◆ Installing IIS into MSCS:


◆ Configuring a File Share for MSCS:


URLs to additional information 241

http://www.microsoft.com/ntserver/ProductInfo/ Enterprise/default.asp

ftp://ftp.microsoft.com/bussys/winnt/winnt-docs/ papers/mscs.exe

http://www.microsoft.com/exchange/en/55/help/default.htm


242

Administration

◆ Configuring a Printer Queue for MSCS:


◆ Cluster Server Cannot Use Disk Beyond Device Number 25:


◆ How to Recover From a Corrupt Quorum Log:



7Invisible Body Tag

This chapter provides detailed instructions on and lists the requirements for manually installing and configuring SRDF/CE software. Only use a manual installation procedure if you are configuring a dynamic cluster. This chapter also describes how to uninstall SRDF/CE.

◆ System requirements ....................................................................... 244◆ Install considerations....................................................................... 244◆ SRDF/CE manual installation procedure .................................... 247◆ Uninstalling SRDF/CE.................................................................... 272

Manual Installationand Configuration

Manual Installation and Configuration 243

244

Manual Installation and Configuration

System requirementsRefer to “System requirements” on page 62 for a list of the hardware, software, and EMC firmware requirements for an SRDF/CE cluster.

Install considerationsThe manual installation procedure describes how to use the SRDF/CE Configuration Utility GUI to configure SRDF/CE. The SRDF/CE Configuration Utility GUI semiautomates the configuration process and provides an alternative to using Symmetrix utilities and SYMCLI commands.

Important: You should perform the manual installation and configuration procedures outlined in this chapter only if you are installing dynamic cluster with VxVM on your system. Table 5 on page 249 describes the manual installation/configuration procedure for a two-node system. Changes in the procedures for a multinode system are noted in Table 5.

If you are not installing a dynamic cluster, use the SRDF/CE Installation Wizard and the SRDF/CE Configuration Wizard, which are easier to use (refer to Chapter 3, ”Automatic Installation and Configuration”).

The manual installation procedure assumes MSCS is not currently installed on your system. Do not install MSCS before installing SRDF/CE. If you do have an existing MSCS installation, use the Configuration Wizard.

Contacting EMC Customer Support is recommended for assistance if any of the following issues are applicable:

◆ You have applications layered on with dependencies.◆ You need other devices online.◆ You are not confident about installing and configuring new

software within the context of Windows Server 2003, MSCS, and Symmetrix with SRDF.



Prerequisites Refer to the following sections before manually installing and configuring SRDF/CE:

◆ “Prerequisites” on page 67

◆ “Before you begin” on page 68

◆ “Planning requirements” on page 72

PC emulation softwareAlthough not required, it is useful to have PC emulation software installed on both cluster nodes, particularly if they are in different physical locations. The manual installation procedure requires physically moving back and forth between the two nodes. Using remote access software to control both nodes from a remote location enables you to perform all the steps on one machine.

Rebooting At various points in the manual installation procedure, you will be instructed to restart (reboot) your system.

Important: Failure to reboot where the SRDF/CE installation procedure explicitly instructs you to do so will result in the SRDF/CE driver failing to load.

Verifying your actionsAfter completing significant steps in the manual installation procedure, EMC recommends testing the system to ensure your actions are accurately carried out. Significant steps include the following:

◆ Moving or copying files

◆ Creating or deleting device groups

◆ Adding or removing devices within groups

◆ Changing the status of SRDF devices (from synchronized to failed over, for example)

Use any of the following methods to verify your actions:

◆ Look in the folders to which you moved or copied files.

◆ Use MSCS Cluster Administrator or the SRDF/CE Configuration Utility GUI to view node or device configuration information.


246


◆ Use Windows Explorer to view details on disk drives, such as a drive’s current status.

CAUTION!Do not perform configuration changes unless you are familiar with the procedure and are aware of its possible impact on subsequent configuration steps.



SRDF/CE manual installation procedureSRDF/CE runs in the Windows Server 2003 (Enterprise or Datacenter Editions) environment.

Overview The installation process comprises the following phases:

◆ Preinstall required software and EMC firmware products◆ Create an MSCS Administrative Account for SRDF/CE◆ Install SRDF/CE◆ Initially configure SRDF/CE◆ Build temporary groups for SRDF devices◆ Remove the temporary groups◆ Install MSCS and start all nodes◆ Configure SRDF/CE resources◆ Verify the installation

For clarity, these procedures refer to the cluster nodes as Node 1 and Node 2. This is an arbitrary designation. However, to follow along, decide which one of your servers will act as Node 1 and Node 2. EMC recommends having the server:

◆ at your primary data center act as Node 1.◆ at your secondary (failover) site act as Node 2.

Important: To have SRDF/CE operate as documented on both nodes, explicitly follow the installation and configuration procedures for the appropriate operating system. Note that checkpoints are provided at the completion of major steps to help ensure the procedure is progressing correctly.

Preinstallation The required software and EMC firmware products (refer to “Software requirements” on page 63) must be preinstalled on the nodes that will comprise the SRDF/CE cluster. (Refer to the EMC SRDF/CE for MSCS Version 2.2 Release Notes for the required versions of these products.)

Note: If you are unfamiliar with these product installation procedures, contact your EMC Customer Support Engineer or refer to the following EMC publications: EMC ControlCenter Installation Guide and the EMC Solutions Enabler Installation Guide.

SRDF/CE manual installation procedure 247

248


Manually installing SRDF/CETo ensure SRDF/CE installs correctly and operates as intended, explicitly follow the steps described in Table 5 on page 249.

Once SRDF/CE is installed, use the SRDF/CE Configuration Utility GUI to complete the initial setup.



Table 5 Manual installation: Two-node installation procedure

Step On node Do the following

Step 1: Preinstallation

1 1. Check all hardware used in the SRDF/CE cluster environment is listed in the EMC Support Matrix. The EMC Support Matrix can be found at http://Powerlink.EMC.com.

2. Ensure applications (such as Microsoft Exchange, SQL Server, Lotus Notes, and so forth) used in the SRDF/CE cluster are cluster-aware.

Note: Important: Any application-specific dependencies that apply to MSCS also apply to SRDF/CE.

3. Have MSCS software readily available.

4. Check that the current operating service packs and patches are installed.

5. Check that the appropriate supported browser is installed.

6. Check that the following EMC software is installed as documented in their corresponding installation guides, and the products are fully functional:

• EMC Solutions Enabler, which includes EMC SYMAPI • Symmetrix Remote Data Facility (SRDF)

Note: Refer to the EMC SRDF/CE for MSCS Version 2.2 Release Notes for the required software versions.

7. Do the following to ensure SRDF operates appropriately in an SRDF/CE cluster:

• Test SRDF without being part of a cluster.• Ensure SRDF is in synchronous or asynchronous mode and not

domino mode.• Get drive assignments.

For additional information, refer to the EMC SRDF documentation.

Node 1

Node 2


http://powerlink.emc.com

250


1

(cont.)

8. Ensure:

• Symmetrix devices do not have locks on them.• Gatekeepers are configured and have signatures written on them

(refer to “Hardware requirements” on page 62).

Note: Verify there are a minimum of six gatekeepers per node. It is recommended gatekeepers be mirrored and for performance one gatekeeper be assigned per group.

• The volumes to be shared as clustered devices are basic. If they are not, convert these to basic. (Refer to MSCS documentation.)

• The R1 and R2 devices and gatekeepers are visible to each node and the disks are defined as either Read/Write (RW) or Write-Disabled (WD) depending on what side is active, and not Not-Ready (NR). You should also have a minimum of six gatekeepers assigned to each server that participates in the cluster.

9. Have the preinstallation worksheet (refer to Appendix A) readily accessible.

Table 5 Manual installation: Two-node installation procedure (continued)


Node 1

Node 2



Step 2: Create an MSCS administrative account for SRDF/CE (node 1)

2 On your domain controller, create a new domain administrative account with the appropriate privileges for administering MSCS and SRDF/CE, if one does not previously exist. The easiest way to do this is:

1. Manually create the domain administrator’s account.

2. Enter a password for this account.

Note: This password is required in Step 5.

3. Give the new account a descriptive name, such as srdfce-admin. You will use this account to install, configure, and administer both MSCS and SRDF/CE. In addition, this account will be used as the running account for SRDF/CE Services. This account must have the following privileges:


Note: To change the user credentials for SRDF/CE (not for MSCS), right-click the SRDF/CE for MSCS icon within the SRDF/CE Configuration Utility and select the Change User Credentials option. This option allows you to change the SRDF/CE logon credentials (refer to “SRDF/CE for MSCS icon options” on page 151 for more information).



DomainController


252


Steps 3-6: Begin Installing SRDF/CE (both nodes)

3 On all cluster nodes apply operating system maintenance:

For Windows 2003, apply at least Service Pack 1 on both nodes and reboot both nodes.

4 On all cluster nodes install SRDF/CE.

1. As outlined in the EMC Solutions Enabler Installation Guide, add the SRDF/CE runtime license key to the EMC Solutions Enabler licensing database using the symlmf command.

2. Run the SRDF/CE setup program from the installation CD.

3. When the Welcome dialog box appears, click Next.

4. Read the Software License Agreement and click Yes.

5. When the Start Copying Files dialog box displays, click Next to proceed with the installation.

6. In the Setup Complete dialog box, select Yes in response to the prompt I want to restart my computer now.

7. Click Finish.

5 On all cluster nodes, test whether SRDF/CE installed correctly with the correct privileges. Do this by:

1. Double-clicking the EMC SRDF-CE Configuration Utility desktop icon to invoke the SRDF/CE snap-in based Configuration Utility.

2. Double-clicking the SRDF/CE for MSCS icon and entering the Administrative ID information.

If this process succeeds, SRDF/CE is installed correctly. Do not exit the Configuration Utility at this point. Successive steps will use it.

Note: This step cannot be skipped because it validates the installation, registers the necessary services, and sets various default values needed later in the manual process.



Node 1

Node 2

Node 1

Node 2

Node 1

Node 2



6 On all cluster nodes, ensure Time Service is available for all members participating in the cluster by issuing the command net time \\node-1 /set /y for all servers that will join the cluster. If Windows Time service in the Services Control panel is not listed, the cluster members have to point to a common server to achieve time synchronization. The best way to do this is to point the servers to a stable domain controller or an NTP server. Ensure all servers are GMT-based.

At this point:

• There should be an Administrative ID in the domain for Cluster Administration. This ID should have the correct security permissions defined.

• The SRDF/CE software should be installed and licensed.• All nodes in the SRDF/CE cluster are validated.• The prospective nodes of the SRDF/CE cluster should be time-synchronized.



Node 1

Node 2


254


Step 7: Create a SYMAPI database and specify Symmetrix IDs

7 On all cluster nodes, obtain local and remote node storage configuration information.

1. If you are not in the SRDF/CE Configuration Utility from previous steps, select Start, Programs, SRDF/CE Configuration Utility.

Note: To use the SRDF/CE Configuration Utility, expand and collapse the tree view by single-clicking the plus (+) or minus (-) icon, and then right-clicking an item to display its menu selection.

2. Double-click the SRDF/CE for MSCS icon. This will bring up the SRDF/CE for MSCS login panel.

Log in to SRDF/CE. Once the login is successful, SRDF/CE will automatically register the SRDF/CE Services.

3. Right-click the SRDF/CE for MSCS icon and select Discover. Discover creates a local SRDF/CE SYMAPI database and populates it with discovered storage array information.

Depending on the storage configuration, this step could take time.

In the case where there are only two Symmetrix arrays, one local and one remote, the Discover process also populates the corresponding local and remote arrays in the Configuration Utility.

Note: In the case where multiple Symmetrix arrays were discovered, a panel will pop up requesting you first designate the local Symmetrix array and then designate the remote Symmetrix array.

Steps 8-10: Add peer nodes, set advanced control and quorum settings

8 On Node 1:

Right-click Nodes, and then select Add Cluster Member Node. This will bring up a dialog box to add nodes to the SRDF/CE configuration. These nodes and the network must be accessible.



Node 1

Node 2

Node 1



9 On Node 1:

1. Expand Control.2. Select Advanced Control.3. Right-click Advanced Control and select Edit Advanced Control

Parameters.The only time the default values in Advanced Control should be changed is when another cluster is installed on a Symmetrix system running another SRDF/CE cluster.

Note: If you are setting up only one cluster, leave the default cluster ID at 0. If your configuration includes more than 64 clusters, contact EMC Customer Support.

Adjust Cluster Number to a value from 0 to 63. Once you choose a value and click OK, SRDF/CE will attempt to validate that no other cluster using either of the two Symmetrix arrays defined above is using that lock value. The results of this validation will be reported to you once you make your selection. If the Cluster Number discovery process indicates that the Cluster Number is in use, select another number.



Node 1


256


10 On Node 1, specify the quorum settings:

1. Expand Control.

2. Select the type of cluster to configure by selecting Set Quorum Type from the Quorum Control right-click menu. If configuring a cluster on Windows Server 2003 Enterprise or Datacenter Editions, you have the option of configuring a quorum disk-based cluster or an MNS cluster.

Note: MNS enables you to create a server cluster without any shared disk for the quorum resource. It provides an alternative quorum resource for clusters with nodes in separate geographic sites. MNS is only supported on Windows Server 2003 Enterprise and Datacenter Editions.

Select Shared Quorum if your cluster quorum data will reside on a shared disk in the cluster. Select Majority Node Set if your cluster quorum will reside on a local disk on each node and be managed by MSCS.

Note: You may safely ignore the warning that indicates that SRDF/CE is unable to verify whether your selection matches MSCS. This is because MSCS is not yet installed. You must ensure that it matches the quorum type you intend to use when you install MSCS.

If you select Shared Quorum, proceed to Step 3, which follows. If you select Majority Node Set, proceed to Step 4 on page 257.

3. Right-click any of the parameters displayed in the right pane and select Edit Quorum Control Parameters and enter:

• Quorum Group Name. Enter the device group name for the quorum device. This device group name must be the same across all nodes in the cluster.

• Quorum Disk. Select the desired quorum disk from devices present on this node.



Node 1



10 (cont.)Note: When you install MSCS, the quorum device group name you specify in SRDF/CE must be the same as the quorum group name later specified during MSCS installation / configuration. The quorum name is case-sensitive.

4. Start the SRDF/CE Service. This service must be started after the quorum configuration steps are completed.

At this point, the following is specified on both nodes:

• Local and remote Symmetrix IDs• Nodes to become members of the SRDF/CE cluster• The cluster number• The quorum disk information

Step 11: Configuring node 2

11 On Node 2:

1. Select Nodes.

2. Right-click the node name representing the current local node, and then select Peer Configurations, then Get Configuration from Node. A node selection box appears.

3. In the Get Configuration from Node node selection box, specify the node you want to pull the configuration from—usually Node 1. If Node 2 is not specified as a valid node during the Node 1 configuration process, the operation will fail because Node 1 has not authorized Node 2 to have access to the configuration.

Note: This function will pull the configuration from Node 1 and completely overlay the configuration of Node 2. Any changes done locally on Node 2 will be lost.

4. Expand Control.

5. Select Quorum Control and verify that Quorum settings match those in Step 10.

At this point, Node 2 is a fully participating member of the SRDF/CE cluster. Any further changes to either Node 1 or Node 2 will be autoconfigured throughout the SRDF/CE cluster.



Node 2


258


Steps 12-16: Build temporary groups for the SRDF devices

12 On Node 1:

To validate a given group can be moved among all of the SRDF/CE nodes, you must define temporary device groups. Once these groups are defined, move the groups from node to node. This verifies that all devices within a group are in the same state, all devices can be failed over between nodes, and Windows builds all the correct device definitions.

To change the SRDF state of the device, first put the device into a disk group.

1. From the SRDF/CE Configuration Utility, right-click Groups and select SRDF/CE Group Options, Create New Device Group. The Create New Device Groups dialog box appears.

2. Specify whether you want to retrieve information about R1 or R2 devices:

a. To request information about R1 devices, select RDF1 from Device Display Type. To request information about R2 devices, select RDF2 from Device Display Type.

b. Provide a name for the device group in the Device Group Name field.

c. Select the devices to make accessible.

d. Move devices to the lower pane. Put R1 devices in one group. Put R2 devices in another group. Separate devices in groups by RA group number.

e. It is recommended for cluster failover performance all groups be enabled for R1-R2 Swap. This feature is a Symmetrix device configuration option enabled within the Symmetrix bin. Once this change is done, select either at group creation time or later to enable R1-R2 swap. Click OK.

Note: You may safely ignore the error that indicates that the MSCS create for this group failed. This is because MSCS is not yet installed. These are temporary groups that will be deconfigured and reinstalled in later steps, at which time they will be successfully created in MSCS.

This group is now autoconfigured on all nodes in the SRDF/CE cluster.



Node 1



12 (cont.)

3. Repeat Step 12 (1 and 2) for other groups, as necessary.

Note: EMC recommends the Symmetrix array and the R1-R2 devices be configured to use Dynamic Swap where available. This will speed failover times.

Note: If dynamic Rx devices are used, establish an initial R1/R2 pairing. EMC recommends all R1/R2 devices be set to the Synchronized state prior to building device groups. You can do this using SYMCLI commands.

13 Since a group can contain multiple devices, and since each of these devices can be online on a different node, EMC recommends that each group, once configured, should be write-enabled with respect to the local node. This ensures group consistency and that each node in the cluster has the correct Windows device mappings prior to the installation of MSCS.

On each node in the SRDF/CE cluster, perform the following steps from the SRDF/CE Configuration Utility:

1. From Groups, right-click Refresh or click F5.

2. For each group (including the quorum group), determine the current read/write state. Select Groups, and in the right-hand panel, observe each group's RDF Config and RDF State. For an R1 group, the state must show Synchronized, and for an R2group, the state must show Failed Over.

3. For each group in (2) that is not in the correct state, you must move the devices to the appropriate storage array side by performing an SRDF FailOver or SRDF FailBack operation, so the devices become accessible and read/write from the current node. Right-click the appropriate group name, select SRDF/CE Group Operations, then Array Operations. From the Array Operations panel, under Group Movement, select SRDF Failover (for R2 group) or SRDF Failback (for R1 group) as required.

4. For each group (including the quorum group), move the group to the desired state. Expand Groups, select a group, right-click, and then select SRDF FailOver or SRDF FailBack.



Node 1

Node 1


260


13

(cont.)

5. Move all devices to the appropriate storage array side so the devices become accessible and read/write from the current node.

6. Repeat the process for each group.

Once all device groups are visible to a given node, run a scan for hardware changes on that node to ensure these devices are registered within Windows:

1. Select Start, Programs, Administrative Tools, Computer Management.

2. Select Device Manager.

3. Right-click in the results pane on Disk Devices and do a Scan for hardware changes.

Now that the disks are added to the Windows Registry, map each disk to a drive letter using Disk Management. This will be the drive letter ultimately used by MSCS.

1. Select Start, Programs, Administrative Tools, Computer Management.

2. Select Disk Management.

Note: In the Initialize and Convert Disks Wizard, when presented with the Select Disks to Initialize panel, you MUST change the partition style from the default to MBR (Master Boot Record). SRDF/CE does not support GPT (GUID Partition Table) disks.

3. Right-click in the results pane on the various physical devices and format, label, and mount on a specific drive letter as necessary. Close Disk Management when done.

Note: The mounted drive letter for each physical disk must be the same across all nodes in the SRDF/CE cluster. SRDF/CE only supports drive letters. Windows Server 2003 supports more than drive letters (that is, mount points), but SRDF/CE does not support anything other than drive letters.

Note: Failure to close Disk Management when completed with a node could corrupt that node’s Registry.



Node 1



13

(cont.)

Note: To avoid corrupting the volume labels, do not open Disk Manager on the nodes where the disks are not write-enabled.

4. In the case of the quorum group, once it is read/write on each node, the physical device information is available. This information must be added to each Node’s SRDF/CE configuration:

a. Expand Control.

b. Select Quorum Control.

c. Right-click Shared Quorum in the right hand pane and then select Edit Quorum Control Parameters.

d. In the Quorum Disk field, select the desired quorum disk from those devices present on this node.

14 Repeat Step 13 for Node 2.



Node 1

! CAUTION

Node 2


262


15 On the last node in the SRDF/CE cluster, verify that all the disks configured in the previous steps are now visible on this last node.

1. Invoke Disk Management.

2. Verify all disks configured in Step 13 are now visible on this last node.

Note: Drive letters on all nodes should match those assigned by Disk Management.

3. After verification, close Disk Management. The system regenerates the Registry key. The Registry keys for all nodes should now reflect a parallel disk structure for those disks and devices participating in the cluster.

Note: Failure to close Disk Management when completed with a node could corrupt that node’s Registry.

16 1. On the Node 2, deconfigure (remove) the temporary disk groups you previously created.

a. Invoke the SRDF/CE Configuration Utility.

b. Expand Groups.

c. Select each group in turn except the Quorum group, right-click, and then select Delete. This will delete the temporary group from all nodes in the SRDF/CE cluster.

At this point:

• The SRDF/CE installation and device mapping is complete on all nodes.• The temporary groups for validating moving disk devices between nodes are created

and then removed.• All nodes display the same set of drive letters and labels as when Node 1 was

read/write enabled. • All nodes display the shared disks with identical drive letters.

Note: These disks will only be read/write enabled on the last node.



Node 2

Node 2



Steps 17 -18: Complete SRDF/CE and MSCS installation on the last node

17 On Node 2, verify quorum disk configuration information:

1. Invoke the SRDF/CE Configuration Utility.

2. Expand Quorum Control and verify the following:

• Group name

• Drive Number

• Mount Point

• Cluster Type

3. Verify the Advanced values for the following:

• Cluster Number

• Delay Failback

• SRDF/Asynchronous Failover Method

Note: The only time these default values should change is when you want to install another cluster on a Symmetrix array already running another SRDF/CE cluster. If this is the case, the preferred action is to contact EMC Customer Support.

4. Start the SRDF/CE Service. This service must be started after the quorum configuration steps are complete.



Node 2


264


18 On Node 2, install and configure MSCS.

1. Install MSCS. (Form a new cluster and select the Advanced (minimum) Configuration from the Select Computer panel of the Create New Cluster Wizard. Configure the network following Microsoft’s recommendations. During the installation, run the Cluster Service from the SRDF/CE administrative account.

2. Configure and verify MSCS dependencies, startup mode, and recovery settings.

a. Select Start, Programs, Administration Tools, Computer Management.

b. Expand Services and Applications and select Services.

c. Right-click in the results pane on Cluster Service:

– In the default General tab, verify that Startup Type is Automatic.– In the Log On tab, verify your account is set to the domain

administrative account that you created in Step 2.– In the Recovery tab, verify the selections are set to the following:

First failure: Restart the serviceSecond failure: Restart the serviceSubsequent failures: Take no actionReset fail count after: 7 daysRestart service after: 3 minutes

3. You must add dependencies on the SRDF/CE Service and Windows Management Instrumentation Driver Extensions to the Cluster Service: From a command prompt, issue the command sc qc clussvc. Note the DEPENDENCIES list returned in the command.

4. Issue the command sc config clussvc depend= SRDFCE_svc/Wmi/<serviceslist>

where <services list> is the list of services noted in step 3, delimited by "/".Example: "sc config clussvc depend= SRDFCE_svc/Wmi/ClusNet/RpcSs/W32Time/NetMan. Important: One space after the equal sign (=) and no spaces between service names. Also note that service names are case sensitive. For further help, consult Microsoft documentation.

(Continued on next page)



Node 2



18 (cont.) 5. To verify step 4, from Computer Management, Services and Applications, Services, right-click on Cluster Service, and select the Dependencies tab. Verify that EMC SRDF/CE for MSCS Service and Windows Management Instrumentation Driver Extensions are now included in the dependencies list.

19 On Node 2, install the SRDF/CE resource .dll file. The file, geospan_res.dll, resides in the SRDF/CE installation folder. Do not change the filename. This file creates a resource type named geospan_res in MSCS.

1. Open a command window.

2. Enter the appropriate command line for the operating system, typing continuously, with no line breaks. Type the following command:

cluster restype "SRDF/CE for MSCS" /create /dllname:geospan_res.dll /type:geospan_res

If you incorrectly typed the command line "geospan_res", to recover from the error:

a. Type cluster restype xxxxx /delete /type

Where xxxxx is used as an example of an incorrect entry for "geospan_res". You must type in your incorrectly spelled entry for "geospan _res" exactly as you previously typed it.

b. Retype the original command line(s).

3. Invoke the MSCS Cluster Administrator and verify the resource type in step 2 has been added.

4. For consistency and clarity, it is recommended that you rename the Cluster Groupto the same name as the Quorum group you defined in step 10.

5. Reboot cluster node Node 2.

At this point, .After Node 2 restarts, the SRDF/CE and Cluster Server services are started on Node 2. MSCS Cluster Administrator:

• Should be aware of the single-node cluster.



Node 2


266


Step 20: Complete SRDF/CE and MSCS installation on node 1

20 On Node 1, install MSCS, have Node 1 join the cluster previously created on Node 2, and set the startup mode.1. Start the SRDF/CE Service on Node 1 and verify the groups configured

on Node 2 are propagated to Node 1.2. Configure and verify MSCS dependencies, startup mode, and recovery

settings.

a. Select Start, Programs, Administrative Tools, Computer Management.

b. Expand Services and Applications.

c. Select Services.

d. Right-click in the results pane on Cluster Service.

1. In the General tab, verify that Startup Type is Automatic.2. In the Log On tab verify that your account is set to the domain

administrative account that you created in Step 2. 3. In the Recovery tab and verify the selections are set to the

following: First failure: Restart the service Second failure: Restart the service Subsequent failures: Take no action Reset fail count after: 7 days Restart service after: 3 minutes

4. You must add dependencies on the SRDF/CE Service and Windows Management Instrumentation Driver Extensions to the Cluster Service: From a command prompt, issue the command: sc qc clussvc. Note the DEPENDENCIES list returned in the command.



Node 1



20 (cont.) 5. Issue the command sc config clussvc depend= SRDFCE_svc/Wmi/ServicesList

where ServicesList is the list of services noted in step 4, delimited by "/". Example: "sc config clussvc depend= SRDFCE_svc/Wmi/ClusNet/RpcSs/W32Time/NetMan" Important: One space after the equal sign ( = ) and no spaces between service names. Also note that service names are case sensitive. For further help, Extensions are now included in the dependencies list.

6. To verify step 5, from Computer Management->Services and Applications->Services, right-click on Cluster Service, select the Dependencies tab and verify that EMC SRDF/CE for MSCS Service and Windows Management Instrumentation Driver Extensions are now included in the dependencies list.

7. From a command prompt, issue the following two commands: cluster /prop QuorumArbitrationTimeMax=1800 cluster /prop QuorumArbitrationTimeMin=15

8. Reboot cluster node Node 1.

At this point, .After Node 1 restarts, the SRDF/CE and Cluster Server services are started on Node 1. The MSCS Cluster Administrator:

• Should be aware of the two-node cluster.

This completes the installation procedure. Now configure the remaining SRDF/CE Configuration Utility components.




268


21 SRDF/CE uses its own SYMAPI database (SRDFCESymapi.db); this reduces interference with SYMAPI databases used by other applications.

Note: Configuring devices for control by more than one SYMAPI database can interfere with normal operation of software that relies on those devices.

1. Before configuring a device for SRDF/CE control, determine whether that device is configured for control by another SYMAPI database on your system or on some other system. Operations (such as failover) that SRDF/CE or some other software initiates on such devices might not be reflected in all corresponding SYMAPI databases.

2. Invoke the SRDF/CE Configuration Utility.

a. Right-click the SRDF/CE for MSCS icon and select Discover to rebuild the local Symmetrix database.

b. Right-click the Groups component and select SRDF/CE Group Options, Create New Device Group. The Select and Group Devices dialog box appears.

c. Create device groups and assign devices to each group:

– Create the new group, assign its devices, and click OK. d. From Groups, right-click Refresh to view the changes for Node 2.

3. Invoke the MSCS Cluster Administrator and verify the resource type and disks are correct for each group.



Node 2



Step 22: Configure SRDF/CE resources

22 On either one of the two nodes, from the SRDF/CE Configuration Utility:

1. Expand Site Mode Operations to set the SRDF communications:

a. Right-click Site Mode and select the mode that controls the behavior of how you want the nodes to respond to site failure events:

– Site Mode = No New Onlines. This is the default.– Site Mode = FailStop. This setting causes groups to go offline

when there is a server, site, or SRDF failure.

Note: Change the Site Mode settings on one node because the settings will circulate to the other node through Autoconfiguration.

On both nodes, from the SRDF/CE Configuration Utility:

2. Expand Logging Control to enable or disable logging activity, if required.

a. Select SRDF/CE Log Level to turn the SRDF/CE log level on or off. When turned on, logging detail can be set from 1-Low to 4-Mega. 1 enables minimal debugging output; 3 enables a more lengthy debugging output. 0-Off is the default.

b. Select SRDF/CE Log Path to set the directory location for the SRDF/CE log file. This is a trace file where all the logging information is placed.

c. Select SRDF/CE Log File Viewer to view any log file.



Node 1

Node 2


270


Step 23: Verify the Installation

23 Verify the installation to ensure MSCS sees nodes and disk resources.

1. Invoke MSCS Cluster Administrator.

2. Verify:

• Both nodes are visible and online.

• All disk resources configured are available.

• The Physical Disk dependencies you created for each group are established:

– Select the Physical Disk resource, right-click, and then select Properties.

– Click the Dependencies tab.

The Physical Disk resource must be dependent on the SRDF/CE resource.

Note: All disks must be dependent on the SRDF/CE resource, except the quorum device. All disks (except the quorum) must be dependent on an SRDF/CE resource, a physical disk resource, a cluster name resource, and a cluster IP resource.

3. Move the groups back and forth between the cluster nodes. For each group:

a. Select the group, right-click, and select Move Group.

b. Select the same group, right-click, and reselect Move Group.

Note: The group should successfully be brought online by the other node in the cluster.



Node 1

Node 2



At this point:

◆ A new MSCS cluster group is created. ◆ An SRDF/CE resource is added to the new cluster group.◆ The devices you specified are moved into the new cluster group. The devices are

dependent on the resource; the resource acquires new dependencies, one for each device.◆ A new SYMAPI database group is created on each node—the database name is the same

on both nodes—and the appropriate Symmetrix devices are moved into each group.___________________________________________

You can rename the MSCS cluster group, the SRDF/CE resource, and the two SYMAPI groups (one on each node). However, the SRDF/CE resource and both SYMAPI groups must always have the same name. Therefore, to change the name of the SRDF/CE resource or either SYMAPI group, you must make all three the same. Some applications, such as SAP, will change the MSCS resource name. If this occurs, reconfigure so all three names are the same.___________________________________________

◆ The quorum control name matches the quorum group name enabling SRDF/CE to see the quorum in the Registry.

Step 24: Create a checkpoint file

24 The checkpoint file is used to recover the SRDF/CE API database if it becomes corrupt. To create the checkpoint file, do the following:

1. On each node, invoke the SRDF/CE Configuration Utility.

2. Right-click Control and select SRDF/CE Config Options, Check Point.

3. A message will pop up asking if you want to proceed. Click Yes.

4. A Checkpoint complete message appears stating the local Registry and group information was written to the RegList.txt file in the SRDF/CE base installation directory. Click OK.

This completes the creation of the checkpoint file. The checkpoint file should be updated anytime changes are made to the group information in the SRDF/CE management utility.

You can now start working with SRDF/CE



Node 1

Node 2


272


Uninstalling SRDF/CETo uninstall SRDF/CE, you must stop the cluster on all nodes and use the Add/Remove Programs utility (in the Control Panel) to remove SRDF/CE.

Important: Without SRDF/CE, MSCS cannot be failed over to a cluster member node. Therefore, remove and reinstall MSCS Cluster Service to make the cluster operational again in a local disk configuration.


AInvisible Body Tag

This appendix describes the alerts, event messages, and errors you may encounter while using the SRDF/CE Configuration Utility GUI.

◆ SRDF/CE alerts ................................................................................ 274◆ SRDF/CE errors and events ........................................................... 276◆ Event messages................................................................................. 277

SRDF/CE Alerts andErrors

SRDF/CE Alerts and Errors 273

274

SRDF/CE Alerts and Errors

SRDF/CE alertsSRDF/CE warnings, or alerts, appear in the bottom-right corner of the SRDF/CE Configuration Utility (Figure A-1). The possible alerts that could display are the following:

◆ SRDF/CE Service ◆ RDF Local Lock◆ RDF Remote Lock ◆ SRDF/CE Driver ◆ IP Communications◆ SRDF Communications◆ Quorum Reserve

Figure A-1 SRDF/CE alerts display

SRDF/CE Service — Indicates the SRDF/CE Service is not started on the local node. Once the service is started, this warning will not appear.

RDF Local Lock — Indicates an application has exclusive access to the SYMAPI database on the local Symmetrix array designated within the SRDF/CE Configuration Utility. This warning briefly displays when a failover to the remote Symmetrix array occurs. If this warning persists, you may need to manually release this lock using SYMCLI commands.

RDF Remote Lock — Indicates an application has exclusive access to the SYMAPI database on the remote Symmetrix array designated within the SRDF/CE Configuration Utility. This warning briefly displays when a failover to the remote Symmetrix array occurs. If this warning persists, you may need to manually release this lock using SYMCLI commands.



SRDF/CE Driver — Indicates the SRDF/CE Service is unable to communicate with the SRDF/CE device driver. Quorum failover will not work properly if this warning appears. If this warning appears after SRDF/CE is installed and configured, you may need to restore the communication links.

IP Communications — Indicates the SRDF/CE Services on the cluster nodes cannot communicate. If this warning appears, failover/failback operations may be affected and MSCS may initiate group movement. If this warning appears after SRDF/CE is installed and configured, you may need to restore the communication links.

SRDF Communications — Reflects the inability of the SRDF/CE Service to ping from its local Symmetrix array to the remote Symmetrix array through the SRDF link. If the SRDF Communications warning appears, failover/failback operations may not work properly. Call EMC Customer Support for resolution.

Quorum Reserve — Indicates the cluster is up and functioning properly and that there is a reserve on the quorum. This warning appears if no reserve exists on the quorum.

SRDF/CE alerts 275

276


SRDF/CE errors and eventsSRDF/CE relies on MSCS and the EMC Symmetrix API (SYMAPI) to function. There are a few events caused and/or generated by the SRDF/CE service. Virtually all SRDF/CE errors can be traced to problems with MSCS or the SYMAPI. Therefore, the sections that follow are not intended to be a troubleshooting guide or a definitive reference for the Windows event and application logs.

Interpreting Windows event log information

Users involved in diagnosing problems in an SRDF/CE cluster must have a thorough understanding of the following logs:

◆ Windows event logs

◆ SRDF/CE application logs

◆ MSCS logs

◆ SYMAPI logs

◆ Logs of major applications deployed on the cluster

Event log entries occur in the order in which they are processed by the system. This can cause confusion as to the actual cause of an error because related errors from different sources may be interspersed by other, nonrelated entries. This is particularly true in the system and application logs. Additionally, since related errors may be entered into different logs, correlating the relevant entries temporarily is necessary.

Example A resource assigned to an SQL virtual server becomes unavailable to MSCS due to a cluster error. This problem will generate errors in the Windows application and system event logs, the MSCS log, the SYMAPI log and the SRDF/CE log. Additionally, SQL Server, SRDF/CE, MSCS and SYMAPI generate entries in the Windows event logs. Depending on where the diagnosis starts, the problem could appear to have originated in any of the applications. Therefore, concise analysis is required to ascertain the actual error path.



Event messagesCurrently, SRDF/CE system-level messaging comprises the following categories:

◆ Device driver messages recorded in the System Event Log

◆ Service error messages recorded in the Application Event Log

System event log The following are System Event logs:

Table 6 System event log descriptions

Event Description Action

geospan_device_loadedSRDFCD: Driver loaded successfully.

Load success. No action necessary.

geospan_device_attach_failedSRDFCD: Failed to attach to <device name>.

The SRDF/CE failed to set up properly because driver startup failed.

This could be due to environment problems with MSCS. Check the quorum device name and Symmetrix array mapping. Fix problems and restart.

geospan_device_disk_number_invalidSRDFCD: Invalid quorum disk number <disknumber>, registered by application.

SRDF/CE failed to start properly because the quorum disk number is invalid.

Check the MSCS configuration and SYMAPI database.

geospan_device_disk_number_missingSRDFCD: Signature matching diskNumber <disknumber> was not found in <hostname> registry.

SRDF/CE failed to start properly because the quorum disk number could not be read.

Check the MSCS configuration and SYMAPI database.

geospan_device_load_failed SRDF/CE failed to start properly because the quorum filter driver failed to load. This could be due to environment problems with MSCS.

Check the installation.

geospan_device_not_a_signature Quorum signature invalid. Check the MSCS configuration and SYMAPI database.

Event messages 277

278


Application event log messagingInformational: After the SRDF/CE installation completes, the machine is rebooted

and you logon for the first time, you will see an application popup indicating a service or driver failed to start, as follows:

Application popup: Service control Manager : At least one service or driver failed during system startup. Use Event Viewer to examine the event log for details.

In addition, you will see the following event log message:

The following boot-start or system-start driver(s) failed to load: SRDFCE

For more information see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

No action is required. The problem will be gone after MSCS is configured.

The following entries will occur during a normal startup of the SRDF/CE Service:

service start main for GeoSpan service.service_main for WNTServiceClass.AppInit.

The following entries will occur during a normal shutdown of the SRDF/CE Service:

WNTServiceClass::TskSvcEvent=serviceStopEvent.ServiceControlHandle:Exiting. Cleaning up and leaving.cleanup service_main for WNTServiceClass.StartServiceCtrlDispatcher return false.

geospan_device_signature_failed Quorum signature mismatch. Check the MSCS environment and Symmetrix configuration.

geospan_device_signature_missing Quorum signature missing. Check the MSCS environment and Symmetrix configuration.

geospan_device_unable_to_read_signature Cannot read quorum signature. Check the MSCS configuration and SYMAPI database.

Table 6 System event log descriptions (continued)

Event Description Action



Table 7 Application event log descriptions

Error Description Action

E_INVALID_PDEV_INFOThe physical device information from the Symmetrix does not match the physical device information from WMI. This can happen if Windows renumbers the physical device ID's.

Under certain conditions Windows can renumber the physical device identifiers. One of these conditions is on reboot. If this happens the physical device identifiers extracted from WMI do not match the physical device identifiers from the Symmetrix array. In this case we use the physical device identifiers from WMI.

A Symmetrix discover is required.

E_R1_R2_SWAP_IGNOREDR1/R2 personality swap is enabled, but will be ignored because the SRDF/CE Override option or the Forced Failover option is also enabled.

R1/R2 personality swap can be enabled for each group, but will be ignored during failover if the SRDF/CE override option or the Forced Failover option is also enabled.

No action required.

E_GATEKEEPER_LOCKEDOne or more Gate Keeper devices are locked.

Each Symmetrix call requires a gatekeeper device to complete the call successfully. This error indicated that no gatekeeper devices were available to service the request. As a result the Symmetrix call failed.

Add more gatekeeper devices and retry the action.

E_SYMINIT_FAILEDUnable to initialize a connection to the Symmetrix. Call to SymInit failed with a status of [<symapierrno>]. SYMAPI error text for [<symapierrno>] is [<symapi description of the error>]. The application can not continue.

An error occured while trying to make a connection to the Symmetrix array.

SRDF/CE can not continue. You must reboot the machine and restart the application.

GROUP_MOVE_FAILED:A failure occurred trying to move group [<groupname>].

Group Move across cluster nodes failed. Check the SRDF/CE logs for more information.

SRDFCE_SRDF_SET_MODE_ASYNCHRONOUS_ERROR:CClusSymInfo::SetSRDFMode: Problem encountered trying to Set Mode [<0/1>] for group [<groupname>]. Retry count [<retries>]. RC = [<symapierrno>] - [<symapi description of the error>].

RDF mode 0 means synchronous, 1 means asynchronous. SRDFCE failed to set RDF mode of a group to Synchronous/Asynchronous.

The SYMAPI description should give you enough information regarding the nature of the problem. Check the RDF configuration and the RA link state. Fix any problems you see and try the command again.

Event messages 279

280


SRDFCE_SRDF_SET_MODE_ASYNCHRONOUS_ERROR:CClusSymInfo::SwapGroup: SymDgRdfControl for group [<groupname>] for FAST SWAP failed after [<retries>] retries. RC = [<symapierrno>] - [<symapi description of the error>]

Swapping the group failed after n number of retries.

The SYMAPI error description should give you enough information regarding the nature of the problem.

SRDFCE_CANNOT_COMMUNICATE_WITH_DRIVER:Cannot open device to send commands to driver\r\nEither SRDF/CE Filter Driver is not Running or not responding.

The filter driver creates \\.\SRDFCEDevice0 control device for SRDFCE application to communite with driver. This device is not present since the driver is not loaded successfully.

Make sure the driver c:\windows\system32\drivers\srdfce.sys is present on the system. If not, run the install wizard to repair the installation. Also make sure that the driver is not disabled.

SRDFCE_SRDF_SET_MODE_ASYNCHRONOUS_WARN:CClusSymInfo::SetSRDFMode: SymDgRdfSet for Set Mode failed for Group Name [<groupname>]. RC = [<symapierrno>] - [<symapi description of the error>]. Retries: [<retry count>].

Setting the RDF mode to sync/async failed. No action is required until all retries failed.

SRDFCE_RES_STATE_INVALID:The RDF pair state for group [<groupname>] was found to be [SYMAPI_RDF_STATE_MIXED] during a MSCS Is / LooksAlive check interval.

One or more devices in the RDF group is different from rest of the members.

Device state must be either Failover/Synchronized/Consistent state depending on RDF mode and the failover state of the group. Any other RDF state must be investigated and fixed as soon as possible.

SRDFCE_SRDF_FAST_SWAP_WARNING: CClusSymInfo::SwapGroup: SymDgRdfControl for group [<groupname>] for FAST SWAP failed. RC = [<symapierrno>] - [<symapi description of the error>]. Retries done: [<number of retries>]. Retries left: [<number of retries left>]."

Swapping group as part of group failover failed.

No action is necessary until all retries failed.

Table 7 Application event log descriptions (continued)




Unable to handle to node MSCS is not running. Start Cluster Service or attempt to diagnose.

Cannot open device to send commands to driver The SRDF/CE driver is unavailable. Check installation. In addition, this could be due to environment problems with MSCS. Check the quorum device name and Symmetrix array mapping. Fix problems and restart.

Table 7 Application event log descriptions (continued)


Event messages 281

282



BInvisible Body Tag

This appendix contains a sample of the preinstallation worksheet to use with an EMC technical representative before incorporating SRDF/CE for MSCS into your clustering environment.

◆ Preinstallation worksheet ............................................................... 284

PreinstallationWorksheet

Preinstallation Worksheet 283

284

Preinstallation Worksheet

Preinstallation worksheetBefore installing SRDF/CE for MSCS, meet with your EMC representative to complete the preinstallation worksheet presented on the next page.

In the worksheet:

◆ The Group Name is the same as the MSCS device group name.

◆ The Pdev value is the numeric designation in the host's physical drive number (for example, the value 5 in \\.\physicaldrive5).

◆ The Label value specifies the name of the disk.

◆ The Drive value is the drive letter designation to be assigned to a given Windows device (disk) when the cluster is operational.

◆ The Symdev value is the Symmetrix device name that is a four-hex number assigned to the given device by the Symmetrix array.

◆ The RA Group value is the Symmetrix remote adapter or RDF group number.

Note: Clustered devices have the same group name and drive letter on both nodes.

Gather all information before installing any software (including the operating system). You must have all the information by the time you verify the configured drive letters on both nodes and before you install MSCS on either node.



Note: The gray area above represents an example only.

Synchronous SRDF Link

Bidirectional

Windows Server

Windows Server

Symmetrix Client(s)

IP Address ___.___.___.__

___.___.___.__

____________

Private IP Address

NETBIOS Name

Windows Server 1

IP Address ___.___.___.__

___.___.___.__

____________

Private IP Address

NETBIOS Name

Windows Server 2

Name of ClusterNETBIOS Name ___.___.___.__

___.___.___.__

____________

MSCSConfiguration

Cluster IP Address

Cluster Subnet Mask

Client(s)

Symmetrix

SRDF/CE for MSCS Configuration Worksheet

Symmetrix 112-Digit Symmetrix ID _ _ _ _ _ _ _ _ _ _ _ _

Symmetrix InformationGroup Name Pdev Drive Symdev

QuorumGrp Quorum-O Q 0006(R1)

Label RA Group

Symmetrix 112-Digit Symmetrix ID _ _ _ _ _ _ _ _ _ _ _ _

Symmetrix InformationGroup Name Pdev Drive Symdev

QuorumGrpphysicaldrive5 Quorum-O Q 0006(R1)

Label RA Group

1 1

physicaldrive5

Preinstallation worksheet 285

286


On the reverse side of the worksheet, answer the following questions.

Description

How many of these gatekeepers per host are dedicated to SRDF/CE?a

SRDF/CE requires all drives formatted NTFS. If No, these drives need to be converted.

What is the size of the drive that will contain the quorum? A quorum size of at least 500 MB is recommended.

Are all data drives for the cluster and quorum configured as RDF devices?

What is the name of the MSCS administrative account?

What is the name of the SRDF/CE Service Account?

a. SRDF/CE requires a minimum of six dedicated gatekeepers per host and six per group, per node.


Glossary

This glossary contains terms related to the EMC SRDF/CE software.

Aagent An installed program designed to control a particular resource type.

Each type of resource supported in a cluster is associated with an agent.

Application ProgramInterface (API)

A language and message format used by an application program to communicate with another program that provides services for it. APIs are usually implemented by writing function calls. Examples of APIs are the calls made by an application program to such programs as an operating system, messaging system, or database management system.

See also SYMAPI.

asynchronous mode See SRDF Asynchronous (SRDF/A).

autoconfiguration The utility the SRDF/CE COM Server uses to propagate the SRDF/CE configuration among cluster peers. Proper time synchronization for all host nodes is critical to ensure the proper functioning of Autoconfiguration. EMC recommends all servers in an SRDF/CE cluster be based on GMT and time servers be employed.

availability The ability to continue to provide a service even during hardware or software failure.


288

Glossary

BBCV device A Symmetrix business continuance volume (BCV) that functions as a

mirrored media to a standard device for a protected storage environment.

BCV mirror BCV device upon establishing or reestablishing a BCV pair.

BCV pair A standard device and a BCV device that provide a protected storage environment.

business continuance An SRDF function that ensures business applications continue running despite possible disk failures.

business continuancevolume

See BCV device.

Ccache Random access electronic storage used to retain frequently used data

between the CPU and either a hard disk or slower RAM. It speeds up general data flow because a cache can be accessed quickly.

channel director The component in the Symmetrix array that interfaces between the host channels and data storage. It transfers data between the channel and cache.

client A computer using services or resources provided by a remote machine, called a server. Often, communications software will have a separate version for the client or guest, and the server or host.

Clients create a TCP/IP session with a service in the cluster using a known IP address. This address appears to the cluster software as a resource in the same group as the application providing the service. In a failure, the Cluster Service will move the entire group to another system.

client failover The response of a client machine after resource failure on the server for the client caused a resource failover. A client will detect a failure in the session and reconnect in exactly the same manner as the original connection. The IP address is now available on another machine and the connection is quickly reestablished. In this simple case, all information related to the original session not committed to disk is lost. This provides higher availability, but no fault tolerance for the service. Applications can use transactions to guarantee the


Glossary

client request is committed to the server database to gain fault-tolerant semantics.

cluster A group of two or more independent computers addressed and used as a single system.

cluster aware software Software that provides a restart mechanism invoked whenever the application resource is moved to another node in the cluster.

Cluster Service The collection of software on each node that manages all cluster-specific activity.

Ddata center

migrationsAn SRDF function that reduces application outage to minutes instead of hours.

delay failback An SRDF/CE feature modifying the Preferred Owner list for each MSCS cluster group so that a failover will occur to a lateral node first, and if the lateral node is unavailable, to a peer node.

See also lateral node and peer node.

dependency The requirement of one resource needing another resource to function properly. The SRDF/CE resource becomes a dependency for physical disk resources in the cluster. Therefore, any operations performed on the disk resource cannot be completed until the SRDF/CE resource has been invoked.

device A uniquely addressable part of the Symmetrix array consisting of a set of access arms, the associated disk surfaces, and the electronic circuitry required to locate, read and write data.

device group A grouping of several devices established to provide configuration, status, and performance data on the collective devices within the group.

director The component in the Symmetrix array that allows the Symmetrix array to transfer data between the host channels and disk devices.

See also channel director and disk director.

disaster recovery An SRDF function that recovers data at the disaster recovery site in minutes rather than days.


290

Glossary

discover A discover action performed in the SRDF/CE Configuration Utility or Configuration Wizard scans the Symmetrix array connected to the current node and gathers device information.

disk director The component in the Symmetrix array that interfaces between cache and the disk devices.

EEMC Solutions Enabler Also known as SYMCLI, an application written using the Symmetrix

Application Programming Interface (SYMAPI) that retrieves data from a Symmetrix array using special low-level SCSI commands.

The EMC Solutions Enabler allows you to run commands on the host to obtain configuration, status, and performance data from the Symmetrix arrays attached to hosts that are running in an open systems environment.

SYMCLI SRDF and TimeFinder components allow you to perform control operations on RDF and BCV devices.

establish A BCV process that assigns a BCV device as the next available mirror of a standard device.

established The BCV pair condition where the BCV device and standard device are synchronized and functioning as a Symmetrix mirror. A BCV pair is established by the BCV commands establish, reestablish, restore, or incremental restore.

Ffailback The action of moving a resource back to the cluster member

designated to be the resource's preferred owner. By default, resources are owned by their preferred owners, so a failback would only occur if the resource moved from its preferred owner. This is likely the result of a failover.

failover The process of taking one or more resources offline on one cluster member and bringing them online on another cluster member.

fault-tolerant Continuous operation in case of failure. A fault-tolerant system can be created using two or more computers that duplicate all processing, or having one system stand by if the other fails. It can also be built with redundant processors, control units, and peripherals. Fault-tolerant operation requires backup power in a power failure. It


Glossary

may also imply duplication of systems in disparate locations in the event of natural catastrophe or vandalism.

FDDl An acroynym for Fiber Distributed Data Interface.

Fibre Channel A high-speed serial interface capable of data transfer rates of up to 100 MB/s.

Fibre Channel Director The Fibre Channel adapter (FA) in the Symmetrix subsystem that interfaces between the host Fibre Channel interface and data storage. It transfers data between the channel and cache.

forced failover An SRDF/CE feature allowing you to automatically keep a cluster up on a particular Symmetrix array or arrays in a total site disaster.

forced quorum Software functionality allowing the cluster to be forced up in the event that total communication is lost between nodes and MSCS wants to shut down the cluster to avoid a split-brain condition.

See also split-brain condition.

Ggraphical userinterface (GUI)

A method that allows users to interact with the computer and its special applications based on graphics instead of text. GUIs use icons, pictures, and menus and use a mouse as well as a keyboard to accept input.

group A collection of resources to be managed as a single unit. Usually, a group contains all elements needed to run a specific application and for client systems to connect to the service provided by the application. Groups allow an administrator to combine resources into larger logical units and manage them as a unit. Operations performed on a group affect all resources contained within that group.

HHBA See also host bus adapter (HBA).

heartbeat A polling communication mechanism used by the cluster processes to determine whether the other member(s) of the cluster are alive and working or have failed. If the heartbeat is not functioning, a failover is initiated, and another node in the cluster will take over the services.


292

Glossary

high availability The characteristic of a computer system/computing environment that allows it to continue to provide applications and access to data if a single component or resource fails. Service is interrupted for only a brief time, and may or may not be apparent to the end users.

host bus adapter(HBA)

A device circuit board that provides an interface between the SCSI bus and the computer I/O bus (for example, PCI, EISA, microchannel).

hyper-volume The term used by the Symmetrix array to make a physical disk appear as multiple physical disks. Each hypervolume has its own unique SCSI address.

II/O Input/output.

identifier (ID) A sequence of bits or characters that identifies a program, device, controller, or system.

incremental establish A TimeFinder BCV or SRDF control operation.

For BCV control operations, an incremental establish causes the BCV device to be incrementally synchronized and functioning as a Symmetrix mirrored device. (The devices must have been previously paired.) This is the same as an establish operation except an incremental establish is much faster: It copies only the differences or new storage data from the standard device to the BCV device. Any changed tracks on the BCV device are overwritten by the data on the corresponding tracks from the standard device.

For SRDF control operations, an incremental establish causes the target (R2) device to be incrementally synchronized and established as a Symmetrix mirrored device. (The devices must have been previously paired.) This is the same as an establish operation except that an incremental establish is much faster: It copies only the differences or new storage data from the source (R1) device to the target (R2) device. Any changed tracks on the device are overwritten by the data on the corresponding tracks from the source (R1) device.

incremental restore A TimeFinder BCV or SRDF control operation.

In BCV control operations, an incremental restore is a control operation that reassigns a BCV device as the next available mirror of the standard device in the pair. However, the standard devices are


Glossary

updated with only the data written to the BCV device during the time of the original pair split. The data written to the standard device during the split is overwritten with data from the BCV mirror.

In SRDF control operations, an incremental restore is a control operation that reassigns a target (R2) device as the next available mirror of the source (R1) device in the pair. The source (R1) devices are updated with only the data written to the target (R2) device during the time of the original pair split. The data written to the source (R1) device during the split is overwritten with data from the target (R2) mirror.

Llateral node Nodes connected to the same Symmetrix array.

MMajority Node Set

(MNS)A quorum-capable resource based on replicating data to local disks associated with a majority of cluster nodes. MNS enables you to create a server cluster without shared disk for the quorum resource. SRDF/CE allows you to configure an MNS cluster on Windows Server 2003 Enterprise and Datacenter Editions.

MicrosoftManagement

Console (MMC)

A Microsoft user interface (UI) framework for use in administrating different components of the Microsoft Windows operating platform. This framework is used to host specific UI/control extensions called snap-ins. Use snap-ins to administer both local and remote computers. Third-party snap-ins can be written for use with MMC, which is the case for the SRDF/CE Configuration Utility.

mirrored pair A device comprising two hypervolumes with all data recorded twice—once on each disk drive.

mirroring A device comprising two hypervolumes with all data recorded twice—once on each disk drive. The Symmetrix array maintains two or more identical copies of a set of data on separate disks. Each copy automatically updates during a write operation. If one disk device fails, the Symmetrix array automatically uses one of the other copies from another disk drive.

MMC See Microsoft Management Console (MMC).


294

Glossary

MSCS Microsoft Cluster Service. A shared-nothing cluster solution for Windows Server Family.

Nnetwork interface

card (NIC)A device that provides network communication capabilities to and from a computer system.

nodes Members of a cluster. Also referred to as systems. A node contains a CPU, disk, and network resource.

Ooffline The state of a resource or group that classifies it as unavailable. When

used in context with a cluster member, offline implies the cluster member may not be booted, or the cluster service on the node in question may not be functioning properly.

online The state of a resource or group that classifies it as available. When used in context with a cluster member, online implies the other cluster members are receiving heartbeats from the cluster member in question. See also resource.

Ppeer node Nodes connected to different Symmetrix arrays located across the

SRDF link from each other.

Qquery A command reporting the state of all the BCV devices in the system,

as well as the status of SRDF states.

quorum disk An ordinary disk volume used as a special communication mechanism between server systems. In an MSCS cluster, a small amount of cluster system data (a few megabytes) is stored on this volume. The SCSI-3 Reserve and Reset commands are used to move quorum-disk ownership back and forth between nodes. If the heartbeat mechanism fails, the quorum disk is used for each node to verify whether the other node is still functioning. Because not all disk products implement these multihost SCSI-3 commands, not all disk products will work in an MSCS environment. Thus, Microsoft is very rigorous in providing the Cluster/RAID category of tests to qualify


Glossary

disks (refer to Microsoft’s Hardware Compatibility List) capable of running with MSCS software ).

RR1 device See source (R1) device.

R2 device See target (R2) device.

RDF1/RDF2 A type of SRDF device group. Only RDF devices can be assigned to an RDF group. An RDF1 group type contains source (R1) devices and an RDF2 group type contains target (R2) devices.

RDF State The SRDF/CE Configuration Utility displays the RDF state information in a column in the right pane when you select:

• Control, Quorum Control

• A specific group under Groups

• R1/R2 devices under Storage

The possible RDF states are the following:

Invalid — The device and link are in an unrecognized combination.

SyncInProg — Synchronizing in progress.

Synchronized — The source and target have identical data.

Split — The source is split from the target and the target is enabled.

Suspended — The link is suspended.

Failed Over — The target is write-enabled, the source is write-disabled and the link is suspended.

Partitioned — The communication link to the remote Symmetrix array is down and the device is write-enabled.

R1 Updated — The target is write-enabled, the source is write-disabled and the link is up.

R1 UpdInProg — The target is write-enabled, the source is write-disabled, the link is up, but there are invalid tracks between the target and the source.

Mixed — This state is only set for a SymDgShow() call when the RDF states of the devices in the group are different from each other, thereby making the RDF state of the group mixed.


296

Glossary

N/A — Not applicable.

Consistent — R2 data is consistent.

RA Remote adapter. An RA provides the link connection and fiber optic protocol support between the local and remote Symmetrix arrays. The RA cable connection is ESCON fibre (ESCON protocol).

RAID Redundant array of independent disks. Data is stored on multiple magnetic or optical disk drives to increase output performance and storage capacities and to provide varying degrees of redundancy and fault tolerance. Instead of storing valuable data on a single hard disk that could fail at any time, RAID ensures a backup copy of all information always exists by spreading data among multiple hard disks.

reestablish A business continuance process that reassigns a BCV device as the next available mirror of the standard device with which it was previously paired. The BCV mirror is updated with the data written to the standard device during the period the BCV pair was split. The data written to the BCV device during the split is overwritten by data from the standard device.

Remote Link Director(RLD)

RLDs create the data link paths between two data storage units. Each Symmetrix array requires a minimum of two, up to a maximum of eight RLDs, depending on the Symmetrix model in use. Each RLD manages two ESCON fibre link connections. Each RLD can perform a single I/O at a time to its paired RLD in the remote Symmetrix array.

RLDs have either an RA1 or RA2 designation. RA1s reside in the source Symmetrix array. RA2s reside in the target Symmetrix array. These RLDs can also be assigned to an RA group. Up to four RA groups can exist in an SRDF configuration.

See also RA.

resource An object managed by the Cluster Service that sees all resources as identical opaque objects. Resources may include physical hardware devices such as disk drives and network cards, or logical items such as disk partitions, TCP/IP addresses, entire applications, and databases. A resource is said to be online on a node when it is providing its service on that specific node.

resource failback The movement of resources back to their preferred location in the cluster. This is usually done under manual user control to avoid a


Glossary

situation where a resource is failed back, and then immediately fails over again because of an unresolved node problem. MSCS also allows automatic failback and provides a timing window to try to avoid repeated failovers.

resource failover The process where control of a resource moves to another node of a cluster. Failover can be initiated automatically or manually. When initiated automatically, the cluster management software detects a failure of server node hardware or an application. When manually initiated, the cluster administrator uses the MSCS Cluster Administrator software application.

resource group A collection of resources to be managed as a single unit. Usually a group contains all elements needed to run a specific application, and for client systems to connect to the service provided by the application. Groups allow an administrator to combine resources into larger logical units and manage them together. Operations performed on a group affect all resources contained within that group.

restore A TimeFinder BCV or SRDF control operation.

In BCV control operations, a restore copies a full BCV mirror back to the standard device in the pair and reassigns the BCV device as the next available mirror to the standard device.

In SRDF control operations, a restore copies the full target (R2) device back to the source (R1) device in the pair and reassigns the target (R2) device as the next available mirror to the source (R1) device.

See also incremental restore.

RF A remote adapter that provides the link connection and fiber optic protocol support between the local and remote Symmetrix arrays. The RF cable connection is SCSI fibre (SCSI protocol). An RF differs from an RA only in the type of connection; an RA uses an ESCON fibre connection, and an RF uses a SCSI fibre connection.

See also RA.

Sscalability The ability to add new components to a storage system as system

load increases.


298

Glossary

SCSI Small Computer System Interface. SCSI is a high-speed parallel interface used to connect microcomputers to SCSI peripheral devices, such as disks, printers, and other computers and local area networks.

semisynchronousmode

An SRDF mode of operation that provides an asynchronous mode of operation. Applications are notified an I/O (or I/O chain) is complete once the data is in the cache of the local RA1 Symmetrix array. Any new data is then written to cache in the remote RA2 Symmetrix array. The remote Symmetrix array acknowledges receipt of the data once it is secure in its cache. If source tracks are pending transfer to a target (R2) device, and a second write is attempted to the source (R1) device, the Symmetrix array disconnects (nonimmediate retry request), and waits for the pending track to transfer to the remote Symmetrix array.

snap-in See Microsoft Management Console (MMC).

Solutions Enabler See EMC Solutions Enabler.

source (R1) device A Symmetrix source (R1) device that is participating in SRDF operations with a target (R2) device. All writes to this device are mirrored to a target (R2) device in a remote Symmetrix array. An R1 device must be assigned to an RDF1 group type.

See also RDF1/RDF2.

source unit In an SRDF configuration, it is the primary data storage subsystem. It initiates many of the SRDF synchronization activities. An SRDF configuration must have at least one source unit and one target unit. See also target unit.

split A business continuance process that removes the BCV mirror from the existing BCV pair and assigns the BCV mirror back to its original device address. The BCV device then holds an instant copy of the data from the standard device.

split-brain condition A total communication failure while both nodes remain operational. A split-brain condition is a potential cause of logical data corruption. For example, if both sides assume the other is dead and begin processing new transactions against their copy of the data, two separate and unreconcilable copies of the data can be created.

SRDF Symmetrix Remote Data Facility. SRDF consists of the microcode and hardware required to support Symmetrix remote mirroring.


Glossary

SRDF Asynchronous(SRDF/A)

A high-performance, extended-distance asynchronous replication using a delta set architecture for reduced bandwidth requirements and no host performance impact.

Asynchronous mode provides a point-in-time image on the target (R2) device only slightly behind the source (R1) device. SRDF/A session data is transferred to the remote Symmetrix system in delta sets, eliminating the redundancy of same-track changes being transferred over the link, reducing the required bandwidth. SRDF/A only needs enough bandwidth to support the average production workload versus peak workloads.

SRDF/A is intended for users who require no host application impact while maintaining a consistent, restartable image of their data on the R2 side at all times.

SRDF link Fiber optic connections and channels between two Symmetrix arrays. A minimum of two to a maximum of eight links can exist between the two units.

SYMAPI Symmetrix Application Program Interface. See Application Program Interface (API).

SYMCLI See EMC Solutions Enabler.

synchronous mode An SRDF mode of operation that ensures 100 percent synchronized mirroring between the two Symmetrix arrays. This is a synchronous mode of operation. Applications are notified that an I/O (or I/O chain) is complete when the RA2 Symmetrix array acknowledges that the data has been secured in cache.

systems See nodes.

Ttarget (R2) device A Symmetrix target (R2) device participating in SRDF operations

with a source (R1) device. It resides in the remote, or target, Symmetrix array. It is paired with a source (R1) device in the local Symmetrix array and receives all write data from its mirrored pair. This device is not accessed by user applications during normal I/O operations. An R2 device must be assigned to an RDF2 group type. See also RDF1/RDF2.

target unit In an SRDF configuration, this subsystem maintains synchronization with the devices it is paired with in the source unit. It can serve as a


300

Glossary

source unit during disaster recovery. An SRDF configuration must have at least one source unit and one target unit. See also source unit.

Vvirtual servers See nodes.

Wworkload migrations In SRDF, similar to data center migrations; especially useful for

minimizing outages during preventative maintenance of hardware or software.


Index

Aadministrative procedures 207advanced control settings 163application

event log messaging 278software failure 52-specific resources 210

array operations 189Asynchronous SRDF

failover method 164, 190asynchronous SRDF

compatibility 27failover method 263group 90, 109, 185, 188

auto del release 164autoconfiguration 30

Bbehavior

common with MSCS 52override 57SRDF/CE failover operation 50

best practices 237booting 41build temporary group for SRDF devices 258

Ccabling 40changing

quorum log size 239user credentials 151

checkpoint 154

clusteradministrator 209architecture 45-aware software 47benefits 45concepts 44limitations 45modes of operation 45modifying groups 213multiple SRDF/CE for MSCS 36number 163pre-SRDF/CE considerations 40SRDF/CE 20

cluster servicemanually starting 212stopping 211unable to start 230uninstalling on Windows 2003 211

clustering 40Configuration Utility GUI 24, 150

SRDF/CE for MSCS icon 151Configuration Wizard 26, 80, 152configuration, troubleshooting 228configurations

basic SRDF 22multicluster 209

configuring additional disk devices 214control settings 154

advanced controlauto del release 164cluster number 163delay failback 164SRDF/A failover method 164, 190

forced failover 175


302

Index

logging control 166log file viewer 166log level 166, 269log path 166

quorum control 155site mode operations 161

converting MSCS cluster to SRDF/CE 125create

a resource type 265a SYMAPI database 254an administrative account 251new device group 183

credentials 151

Ddata protection 20delay

failback 31, 164loading of MSCS Service 237

device group 183list 187unlock 188

discover 152disk devices 214documentation 16dynamic swap 28, 189, 193

Eedit log parameters 167EMC

business continuance volumes (BCVs) 45documentation 16software requirements 63, 65technical support 228

error messages 234event messages 277

application event log 278geospan_device_attach_failed 277geospan_device_disk_number_missing 277geospan_device_load_failed 277geospan_device_loaded 277geospan_device_not_a_signature 277geospan_device_signature_failed 278geospan_device_signature_missing 278geospan_device_unable_to_read_signature

278

system event log 277external SYMAPI locks 231

Ffailover

due to application failure 44due to hardware failure 44forced 27, 175manual 44operation 50

failstop 111, 162failure

modes 56types 50

FastSwap 28fault-tolerant system 45forced failover 27, 175

GGeoSpan v1.2.3 to SRDF/CE upgrade procedure

142GeoSpan_res.dll

errors 234prevents MSCS group failover 231stays offline after quorum failback 231

groupenabling 239renaming 213

groups settings 179array operations 189create new device group 183list 187MSCS move 188MSCS offline 188MSCS online 188pull groups from SYMCLI 187set R1R2 swap 189unlock 188

Hhardware

NIC 52requirements 62SRDF/CE architecture 35

heartbeat


Index

link failure 52host bus adapter

driver software requirements 64failure 52, 220

IInstallation

Wizard 25, 75installation

prerequisites 67requirements 62, 244

IPcommunication warning 235communications failure 219

LLAN link failure 52lateral node 31list

device groups 187Symmetrix information 198

localmirrors 36override 161

locks 199, 231log

edit parameters 167file viewer 166interpreting 276level 166, 269path 166

logging control 166

Mmain SRDF/CE features 24majority node set 26, 107, 256manual failover 44Microsoft

required software 63technical support 228

mirroring 36MNS 26, 107, 256modes of operation

active/active 38active/passive 38

modifying cluster groups 213MSCS

Cluster Administrator 209converting to an SRDF/CE cluster 125error 170 236error while attempting to connect 236move 188offline 188online 188service 237

multicluster configurations 209multiple

remote adapters 29SRDF/CE clusters 209

Nnew device group 183newly added disk, unable to see 232no new onlines 111, 162node settings 202

Ooffline

data redundancy 45MSCS 188

overridelocal 57, 110, 161SRDF 57, 110, 161, 175, 218, 221

overviewadministrative procedures 208Configuration Utility GUI 150Configuration Wizard 80error messages 234Installation Wizard 67MSCS 42SRDF 21SRDF/CE 20SRDF/CE and Cluster Behavior 49

PPC emulation software 140, 245peer node 31

settings 202preinstallation worksheet 72, 283prerequisites


304

Index

for automatic SRDF/CE installation 67for manual SRDF/CE installation 140, 245for upgrading SRDF/CE 140

pull groups from SYMCLI 187push/pull configuration to a node 30, 204

Qquorum

control 155log size 239upgrading 226

RR1/R2 swap 28, 181, 189, 193, 258rebooting 123recovery

from corrupt quorum log 222from HBA or SCSI channel failure 220from site failure 218from SRDF link failure 220from total IP communications failure 219

remoteadapters 29storage array 200

renaminggroups 213groups in MSCS 126

replace remote storage array 30, 200requirements

converting MSCS cluster to SRDF/CE 125hardware 62HBA driver software 64installing and configuring SRDF/CE 62, 244Microsoft software 63pre-SRDF/CE installation planning 72SRDF/CE installation and operating 66upgrading SRDF/CE 140

restore 155from site failure 216

SSCSI channel failure 220SCSIport driver 64select and group devices 183service dependencies 237

site failuremodes 56recovery 218restoring 216

site mode 111operations 161setting 162

software requirements 63SRDF 21, 209

asynchronouscompatibility 27failover method 164, 190, 263group 90, 109, 185, 188

coexistence 41link failure 220override 57, 110, 161, 175, 218, 221swap 28synchronous 90, 188warning 235

SRDF/Acompatibility 27failover method 164, 190, 263group 90, 109, 185, 188

SRDF/CEadministration 211automatic installation prerequisites 67behavior 49behavior override 57Configuration Utility GUI 24, 150configuration with multiple RAs 29Configuration Wizard 26converting from MSCS cluster 125failover operation 50group in wrong state 229hardware architecture 35icon 151installation requirements 66Installation Wizard 25log file viewer 166log level 166, 269log path 166main features 24manual installation prerequisites 140, 245overview 20push/pull configuration to a node 30replacing the remote storage array 30response to complete site failure 56


Index

SRDF swap 28uninstalling 138, 272unique behavior 53unique cluster behavior 210

SRDF/CE Configuration Utility GUIcontrol settings 154

advanced control 163checkpoint 154forced failover 175logging control 166quorum control 155restore 155site mode operations 161

groups settings 179nodes settings 202storage settings 191

storage array, remote 200STORport driver 64SYMAPI

database 268locks 231

SYMCLI pull groups 187Symmetrix

list 198locks 199replace remote storage array 200settings 191unlock 199

system event log 277

Ttechnical support

EMC 228Microsoft 228

troubleshootingconfiguration 228EMC technical support 228Microsoft technical support 228

Uunable

to see newly added disk 232to start cluster service 230

uninstallingcluster service on Windows 2003 211SRDF/CE 138, 272

uniquecluster behavior 210

unlockdevice groups 188Symmetrix 199

upgrade procedureGeoSpan v1.2.3 to SRDF/CE 142quorum disk 226requirements 140Windows 2000 to Windows 2003 223

user credentials 151

Wwarning

IP communication 235SRDF 235

Windows 2000 to 2003 upgrade procedure 223


306

Index


Date post:	24-Jun-2018
Category:	Documents
Upload:	hathuan
View:	236 times
Download:	2 times

EMC SRDF/Cluster Enabler for MSCS · EMC SRDF/Cluster Enabler for MSCS ... Uninstalling the Cluster...

Documents