VERITAS Cluster Serverfor SolarisEvent Notification

VCS_3.5_Solaris_R3.5_20020915

9-2

Objectives

After completing this lesson, you will be able to:

• Describe the VCS notifier component.

• Configure the notifier to signal changes in cluster status.

• Describe SNMP configuration.

• Describe event triggers.

• Configure triggers to provide notification.

VCS_3.5_Solaris_R3.5_20020915

9-3

Notification

How VCS performs notification: The had daemon sends a message to the notifier daemon

when an event occurs. The notifier daemon formats the event message and sends

an SNMP trap or e-mail message (or both) to designated recipients.

had

notifier

SMTPSNMP

had

VCS_3.5_Solaris_R3.5_20020915

9-4

Message Severity Levels

had had

notifier

SMTP

SNMP

SNMP

SNMPError

SevereError

Warning

Information

Concurrency violation

Service group is online.

Resource has faulted.

Agent has faulted

VCS_3.5_Solaris_R3.5_20020915

9-5

Message Queues The had daemon stores a message in a queue when an event

is detected. The message is sent over the private cluster network to all

other had daemons to replicate the message queue. The notifier daemon can be started on another system in

case of failure without loss of messages.

had

notifier

SMTP

SNMP

had

notifier

SMTPSNMP

Replicated Queue

VCS_3.5_Solaris_R3.5_20020915

9-6

Configuring Notifier

• The notifier daemon can be started and monitored by the NotifierMngr resource.

• Attributes define recipients and severity levels. For example:

SmtpServer = "smtp.acme.com"

SmtpRecipients = { "admin@acme.com" = Warning }

notifier

NIC

NotifierMngr

NIC

NotifierMngr

VCS_3.5_Solaris_R3.5_20020915

9-7

The NotifierMngr Agent

• Functions:

Starts, stops, and monitors the notifier daemon

• Required attribute:

PathName Full path of the notifier daemon

• Required attributes for SMTP e-mail notification:

SmtpServer Host name of the SMTP e-mail server

SmtpRecipients E-mail address and message severity level for each recipient

• Required attribute for SNMP notification:

SnmpConsole Name of the SNMP manager and message severity level

VCS_3.5_Solaris_R3.5_20020915

9-8

The NotifierMngr Resource• Optional attributes:

MessagesQueue Number of messages in queue; default = 30

NotifierListeningPort TCP/IP port number; default =14144

SnmpdTrapPort TCP/IP port to which SNMP traps are sent; default=162

SnmpCommunity Community ID for the SNMP manager; default = "public“

• Example resource configuration:

NotifierMngr Notify_Ntfr (

PathName = "/opt/VRTSvcs/bin/notifier"

SnmpConsoles = { snmpserv = Information }

SmtpServer = "smtp.your_company.com"

SmtpRecipients = { "vcsadmin@your_company.com" = SevereError }

)

VCS_3.5_Solaris_R3.5_20020915

9-9

SNMP Configuration

• Load MIB for VCS traps into SNMP console.• For HP OpenView Network Node Manager, merge

events:xnmevents -merge vcs_trapd

• VCS SNMP configuration files:– /etc/VRTSvcs/snmp/vcs.mib– /etc/VRTSvcs/snmp/vcs_trapd

VCS_3.5_Solaris_R3.5_20020915

9-10

The ResourceOwner Attribute

• VCS sends an e-mail message to the account specified in the ResourceOwner attribute when notification is configured and the attribute is defined for a resource.

• An entry is also created in the log file:TAG_E 2001/12/03 11:23:48 VCS:10304:Resource file1

(Owner=daniel, Group testgroup) is offline on machine1

• These events cause VCS to send notification:ResourceStateUnknown, ResourceMonitorTimeout, ResourceNotGoingOffline, ResourceRestartingByAgent, ResourceWentOnlineByItself, ResourceFaulted

• To set the ResourceOwner attribute:hares –modify res_name ResourceOwner daniel

VCS_3.5_Solaris_R3.5_20020915

9-11

Event Triggers

How VCS performs notification:1. VCS determines if notification is enabled.

– If disabled, no action is taken.– If enabled, VCS runs hatrigger with event-specific

parameters.

• The hatrigger script invokes the event-specific trigger script with parameters passed by VCS.

• The event trigger script performs the notification tasks.

VCS_3.5_Solaris_R3.5_20020915

9-12

Types of Triggers

postofflineService group went offlinePostOffline

resnotoffResource not offlineResNotOff

loadwarningSystem is overloadedLoadWarning

resstatechangeResource changed stateResStateChange

nofailoverService group cannot failoverNoFailover

unable_to_restart_hadHAD cannot be restartedUnableToRestartHad

postonlineService group went onlinePostOnline

preonlineGroup about to come onlinePreOnline

violationResource online on more than one system

Violation

injeopardyCluster in jeopardyInJeopardy

sysofflineSystem went offlineSysOffline

resfaultResource faultedResFault

Script NameDescriptionTrigger

VCS_3.5_Solaris_R3.5_20020915

9-13

Configuring Triggers

• Triggers enabled by presence of script file:– ResFault– ResNotOff– SysOffline– InJeopardy– Violation– NoFailover– PostOffline– PostOnline– LoadWarning– UnableToRestartHAD

• Triggers configured by service group attributes:

– PreOnline

– ResStateChange

• Triggers configured by default:– Violation

VCS_3.5_Solaris_R3.5_20020915

9-14

Sample Triggers

• Sample trigger scripts include example code to send an e-mail message.

• Mail must be configured on the system invoking trigger to use sample e-mail code.# Here is a sample code to notify a bunch of users.# @recipients=("username@servername.com");# $msgfile="/tmp/resnotoff$2";# `echo system = $ARGV[0], resource = $ARGV[1] >

$msgfile`;## foreach $recipient (@recipients) {# # Must have elm setup to run this.# `elm -s resnotoff $recipient < $msgfile`;# }#`rm $msgfile`;

VCS_3.5_Solaris_R3.5_20020915

9-15

ResFault Trigger

• Provides notification that a resource has faulted• Arguments to resfault:

– system: Name of the system where the resource faulted– resource: Name of the faulted resource

VCS_3.5_Solaris_R3.5_20020915

9-16

ResNotOff Trigger

• Provides notification that a resource has not been taken offline

• If a resource is not offline on one system, the service group cannot be brought online on another.

• VCS cannot fail over the service group in the event of a fault, because the resource will not come offline.

• Arguments to resnotoff:– system: Name of the system where the resource is

not offline– resource: Name of the resource that is not offline

VCS_3.5_Solaris_R3.5_20020915

9-17

ResStateChange Trigger

• Provides notification that a resource has changed state

• Set at the service group level by the ResStateChange attribute:– hagrp serv_grp -modify TriggerResStateChange

• Arguments to resstatechange:– system: Name of the system where the resource faulted– resource: Name of the faulted resource– previous_state: State of the resource before change– new_state: State of the resource after change

VCS_3.5_Solaris_R3.5_20020915

9-18

SysOffline Trigger

• Provides notification that a system has gone offline

• Executed on another system when no heartbeat is detected

• Arguments to sysoffline:– system: Name of the system that went offline– systemstate: Value of the SysState attribute for the offline

system

VCS_3.5_Solaris_R3.5_20020915

9-19

NoFailover Trigger

• Run when VCS determines that a service group cannot fail over

• Executed on the lowest numbered system in a running state when the condition is detected

• Arguments to nofailover:– systemlastonline: Name of the last system where the

service group is online or partially online– service_group: Name of the service group that cannot fail

over

VCS_3.5_Solaris_R3.5_20020915

9-20

Summary

You should now be able to:• Describe the VCS notifier component.

• Configure the notifier to signal changes in cluster status.

• Describe SNMP configuration.

• Describe event triggers.

• Configure triggers to provide notification.

VCS_3.5_Solaris_R3.5_20020915

9-21

Lab 8: Event Notification

Triggersresfault

nofailoversysoffline

Student Red Student Blue

resfaultnofailoversysoffline

RedNFSSG

ClusterService

BlueNFSSG

webip

webnic

notifier