VERITAS Cluster Serverfor SolarisConfiguring VCS Response to Resource Faults

VCS_3.5_Solaris_R3.5_20020915

10-2

How VCS Responds to Faults

Systemavailable inSystemList?

Critical online resource

in path?

Offline all resources in path

Call resfault (if present)

Offline entire service group

Start service group elsewhere

Keep group partially online

Keep service group offline

Run NoFailover trigger

Y N

N

Y

VCS_3.5_Solaris_R3.5_20020915

10-3

Failover Policies

• The AutoFailOver attribute indicates whether automatic failover is enabled for the service group.Default value is 1, enabled.

• The FailOverPolicy attribute specifies how a target system is selected:

– Priority—System with the lowest priority number in the list is selected (default).

– RoundRobin—System with the least number of active service groups is selected.

– Load—System with greatest available capacity is selected.

• Example configuration:hagrp –modify group AutoFailOver 0hagrp –modify group FailOverPolicy Load

VCS_3.5_Solaris_R3.5_20020915

10-4

Priority Failover Policy

DB

Svr2

Svr1AP1

SystemList = {Svr1 = 0, Svr2 = 1}

SystemList = {Svr2 = 0, Svr1 = 1}

Svr3

SystemList = {Svr3=0, Svr1=1, Svr2=2}AP2

Lowest numbered system in SystemList selected

VCS_3.5_Solaris_R3.5_20020915

10-5

Round Robin Failover Policy

Svr3

Svr4Svr2

Svr1

System with fewest running service groups selected

VCS_3.5_Solaris_R3.5_20020915

10-6

Load Failover Policy1. Define system Capacity based on server capability.

- Load

= Available

Capacity 300300 300 150

We decide each of these servers has a Capacity of 300.

This server has a Capacity of 150.

VCS_3.5_Solaris_R3.5_20020915

10-7

Determining Load

iPlanet requires 100 units of Load

iPlanet100

Load

Sybase requires 125

Sybase125

Load

Oracle 8i requires 150

NFS shares require 75 each

Oracle 8i150

LoadNFS1 75

Load

NFS2 75

LoadNFS3 75

Load

1. Define system Capacity based on server capability.

2. Define group Load based on application requirements.

- Load

= Available

Capacity 300

150 +75

300

100

200

300

125

175

150

75+75

075

VCS_3.5_Solaris_R3.5_20020915

10-8

Determining the Failover Target

iPlanet100

Load

Sybase125

Load

Oracle 8i150

LoadNFS1 75

Load

NFS275

LoadNFS3 75

Load

Oracle 8i150

Load

- Load

= Available

Capacity 300

150 +75

300

100

200

300

125

175

150

75+75

075

Oracle 8i FAILS.1. VCS brings Oracle 8i online on the server with 200

AvailableCapacity.2. VCS recalculates AvailableCapacity based on new Load.

VCS_3.5_Solaris_R3.5_20020915

10-9

Tracing a Server Failure

- Load

= Available

Capacity 300

iPlanet100

Load

Sybase125

Load

75

NFS175

Load

300

250

50

300

125

175

150

75+75

0225

NFS2 75

LoadNFS3 75

Load

Oracle 8i150

Load

The NFS server FAILS.1. VCS brings NFS2 online on the server with 225

AvailableCapacity.2. VCS recalculates AvailableCapacity based on new Load.

NFS2 75

Load

VCS_3.5_Solaris_R3.5_20020915

10-10

Completing Fail Over

- Load

= Available

Capacity

iPlanet100

Load

Sybase125

Load NFS1 75

Load

300

250

50

300

125

175

150

75+75

0

300

150

150

NFS3 75

Load

Oracle 8i150

Load

1. VCS brings NFS2 online on the server with 175 AvailableCapacity.

2. VCS recalculates AvailableCapacity based on new Load.

NFS2 75

LoadNFS3

75 Load

VCS_3.5_Solaris_R3.5_20020915

10-11

Setting Load and Capacity

• The Load and Capacity attributes are user-defined values.

• Set attributes using the hagrp and hasys commands.

• Examples:hasys –modify LgSrv1 Capacity 300

hagrp –modify OracleSG Load 150

• AvailableCapacity calculated by VCS:

Capacity minus Load equals AvailableCapacity

VCS_3.5_Solaris_R3.5_20020915

10-12

Dynamic Load Balancing

iPlanet100

Load

Sybase125

Load

Oracle 8i150

LoadNFS1 75

Load

Process

40 Load

Process

40 Load

External software monitors CPU utilization (30, 40, 75, and 80 percent utilization for systems shown below).

The software sets DynamicLoad attribute according to system Capacity value using hasys –load system value.

For example, if CPU utilization is 30% and Capacity is set to 300, set Dynamic load to 90 (30% of 300).

- DynLoad

= Available

Capacity 300

225

300

90

210

300

120

175

100

80

2075

30% 40% 75% 80%

VCS_3.5_Solaris_R3.5_20020915

10-13

The LoadWarning Trigger

Sybase125

Load

Oracle 8i150

LoadNFS1 75

Load

Process

40 Load

Process

40 Load

Runs when system has been running at a specified percent of Capacity level for a specified period of time.

Configured by placing loadwarning script in /opt/VRTSvcs/bin/triggers and setting system attributes.

This example configuration causes VCS to run the trigger if system Srv4 runs at 90 percent of capacity for ten minutes.

- DynLoad

= Available

Capacity 300

225

300

90

210

300

120

175

100

80

2075

80%System Svr4 ( Capacity=100 LoadWarningLevel=90 LoadTimeThreshold=600 )

System Svr4 ( Capacity=100 LoadWarningLevel=90 LoadTimeThreshold=600 )

Srv4

main.cfmain.cf

VCS_3.5_Solaris_R3.5_20020915

10-14

System Limits1. Define system Limits based on the server properties:

Limits = {Processors-4, Mem=512}

- Prereq

= Current

Limits 1,128

Each of these servers has: Processors=4 Mem=512

This server has: Processors=1 Mem=128

4,5124,5124,512

VCS_3.5_Solaris_R3.5_20020915

10-15

Service Group Prerequisites

iPlanet:1, 184

Sybase requires

1 Processor212 Mb RAMSybase:

1,212

Oracle requires 2 Processors256 Mb RAM

NFS requires 1 Proc

48 Mb RAM

Oracle 8i2,256

NFS1 1,48

NFS2 1,48

NFS3 1,48

1. Define system Limits based on the server properties.

2. Define service group Prerequisites based on application requirements.

- Prereq

= Current

Limits

iPlanet requires

1 Processor184 Mb RAM

4,512 1,128

3,3041,184 1,212 1,96

1,2081,328 1,300 1,32

4,5124,512

VCS_3.5_Solaris_R3.5_20020915

10-16

Combining Capacity and Limits

When used together, VCS determines the failover target as follows:• Limits and Prerequisites are used to determine a subset of

potential failover targets.

• Of this subset, the system with the highest value for AvailableCapacity is selected.

• If multiple systems have the same AvailableCapacity, the first system in SystemList is selected.

• Limits are hard values—if a system does not meet the Prerequisites, the service group cannot be started on that system.

• Capacity is a soft limit —the system with the lowest AvailableCapacity is selected, even if AvailableCapacity results in a negative number.

VCS_3.5_Solaris_R3.5_20020915

10-17

Failover Zones

Web

Preferred failoverzone for Web service group

Preferred failover zone for database service group

sysc sysd

syse sysf

The SystemList for both service groups includes all systems in the cluster.

Database

sysa sysb

VCS_3.5_Solaris_R3.5_20020915

10-18

SystemZones Attribute• Used to define the preferred failover zones for each service

group.• If the service group is online in a system zone, it fails to other

systems in the same zone based on the FailOverPolicy, until there are no systems available in that zone.

• When there are no other systems for failover in the same zone, VCS chooses a system in a new zone from the SystemList based on the FailOverPolicy.

• To define SystemZones:– Syntax:hagrp –modify group_name SystemZones \ sys1 zone# sys2 zone# sys zone# …

– Example:hagrp –modify OracleSG SystemZones sysa \ 0 sysb 0 sysc 1 sysd 1 syse 1 sysf 1

VCS_3.5_Solaris_R3.5_20020915

10-19

Controlling Failover Behavior with Resource Type Attributes• RestartLimit

– Affects how the agent responds to a resource fault

– Default: 0

• ConfInterval– Determines the amount of time that a tolerance or restart

counter can be incremented

– Default: 600 seconds

• ToleranceLimit– Enables the monitor entry point to return OFFLINE several

times before the resource is declared FAULTED

– Default: 0

VCS_3.5_Solaris_R3.5_20020915

10-20

Restart Example

• RestartLimit=1

Resource to be restarted one time withinthe ConfInterval timeframe

• ConfInterval=180

Resource can be restarted once within a three minute interval.

• MonitorInterval=60 seconds (default value)

Resource is monitored every 60 seconds.

MonitorIntervalRestart Faulted

Online OfflineOnlineOfflineOnlineConfInterval

VCS_3.5_Solaris_R3.5_20020915

10-21

Adjusting Monitoring

• MonitorInterval:– Default value is 60 seconds for most resource

types.– Consider reducing to 10 or 20 seconds for testing.– Use caution when changing this value:

• Load is increased on cluster systems.• Resources can fault if they cannot respond in the

interval specified.

• OfflineMonitorInterval:– Default is 300 seconds for most resource types.– Consider reducing to 60 seconds for testing.

VCS_3.5_Solaris_R3.5_20020915

10-22

Modifying Resource Type Attributes

• Can be used to optimize agents

• Applied to all resources of the specified type

• Command line example:hatype –modify FileOnOff MonitorInterval 5

VCS_3.5_Solaris_R3.5_20020915

10-23

Controlling Clean behaviour on resource faults

The ManageFaults attribute specifies whether VCS calls the Clean entry point when a resource faults. ManageFaults is a service group attribute;

• If the ManageFaults attribute is set to ALL, VCS calls the Clean

entry point when a resource faults.

• If the ManageFaults attribute is set to NONE, VCS takes no action on a resource fault; it “hangs the service group until administrative action can be taken. VCS marks the resource state as ADMIN_WAIT and does not fail over the service group until the resource fault is removed and the ADMIN_WAIT state is cleared.

VCS_3.5_Solaris_R3.5_20020915

10-24

Clearing resources in ADMIN_WAIT state

To clear a resource• 1 Take the necessary actions outside VCS to bring all resources

into the required state.

• 2 Verify that resources are in the required state by issuing the command:

# hagrp -clearadminwait group -sys system

This command clears the ADMIN_WAIT state for all resources. If VCS continues to detect resources that are not in the required state, it resets the resources to the ADMIN_WAIT state.

• 3 If resources continue in the ADMIN_WAIT state, repeat step 1 and step 2, or issue the following command to stop VCS from setting the resource to the ADMIN_WAIT state:

• # hagrp -clearadminwait -fault group -sys system

VCS_3.5_Solaris_R3.5_20020915

10-25

Controlling fault propagation

• The FaultPropagation attribute defines whether a resource fault is propagated up the resource dependency tree. It also defines whether a resource fault causes a service group failover.

• If the FaultPropagation attribute is set to 1 (default), a resource fault is propagated up the dependency tree. If a resource in the path is critical, the service group is taken offline and failed over, provided the AutoFailOver attribute is set to 1.

• If the FaultPropagation is set to 0, resource faults are contained at the resource level. VCS does not take the dependency tree offline, thus preventing failover. If the resources in the service group remain online, the service group remains in the PARTIAL|FAULTED state. If all resources are offline or faulted, the service group remains in the OFFLINE| FAULTED state.

VCS_3.5_Solaris_R3.5_20020915

10-26

Preventing Failover

• Frozen service group does not fail over when a critical resource faults.

• Service group must be unfrozen to enable fail over.

• To freeze a service group:hagrp -freeze service_group [-persistent]

• To unfreeze a service group:hagrp -unfreeze service_group [-persistent]

• A persistent freeze:

– Requires the cluster configuration to be open

– Remains in effect even if VCS stopped and restarted throughout the cluster

VCS_3.5_Solaris_R3.5_20020915

10-27

Clearing Faults

• Verify that the faulted resource is offline.

• Fix the problem that caused the fault and clean up any residual effects.

• To clear a fault, type:hares -clear resource_name [-sys system_name]

• To clear all faults in a service group, type:hagrp -clear group_name [-sys system_name]

• Persistent resources are cleared by probing:hares -probe resource_name -sys system_name

VCS_3.5_Solaris_R3.5_20020915

10-28

Probing Resources

• Causes VCS to immediately monitor the resource

• To probe a resource, type:

hares –probe resource_name –sys system_name

• You can clear a persistent resource by probing it after the underlying problem has been fixed.

VCS_3.5_Solaris_R3.5_20020915

10-29

Flushing Service Groups

• All online/offline agent processes are stopped.

• All resources in transitional states waiting to go online are taken offline.

• Propagation of the offline operation is stopped, but resources waiting to go offline remain in the transitional state.

• You must verify the physical or software resources are stopped at the operating system level after flushing to avoid creating a concurrency violation.

• To flush a service group, type:hagrp –flush group_name –sys system_name

VCS_3.5_Solaris_R3.5_20020915

10-30

Testing Failover

• Use test resources, such as FileOnOff, when applicable.

• Set lower values for MonitorInterval, OfflineMonitorInterval, and ConfInterval to detect faults more quickly.

• Manually online, offline, and switch the service group among all systems.

• Simulate failure of each resource in the service group.

• Simulate failover of the entire system.

VCS_3.5_Solaris_R3.5_20020915

10-31

Testing Examples

• Force a resource to fault.

• Reboot a system.

• Halt and reboot a system.

• Remove power from a system.