+ All Categories
Home > Documents > How to Cluster a Netbackup 7

How to Cluster a Netbackup 7

Date post: 01-Jun-2018
Category:
Upload: -
View: 323 times
Download: 9 times
Share this document with a friend
35
8/9/2019 How to Cluster a Netbackup 7 http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 1/35  S Y M A N T EC S T O R A G E F O U N D A T I O N H I G H A V A I L A B I L I T Y How To Cluster a NetBackup 7.x Master Server using Veritas Cluster Server 6.x for Unix/Linux Robert Budini Principal Support Engineer 8 th  April 2014
Transcript
Page 1: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 1/35

 

SYM

ANT

EC

STORAGE

FOUNDATION

HIGH

AVAILA

BILITY

How To Cluster a NetBackup 7.x

Master Server using Veritas Cluster

Server 6.x for Unix/Linux

Robert Budini

Principal Support Engineer

8th

 April 2014

Page 2: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 2/35

2

Table of ContentsIntroduction .................................................................................................................................................. 3 

About NetBackup Clustering ......................................................................................................................... 4 

Cluster Configuration Details ........................................................................................................................ 5 

NetBackup Cluster Configuration Details ..................................................................................................... 5 

Configuring Storage For Netbackup. ............................................................................................................. 7 

Installing and Configuring NetBackup ........................................................................................................... 8 

Reviewing and Understanding the Configuration ....................................................................................... 22 

How the Netbackup Agent for Veritas Cluster Works ................................................................................ 24 

Troubleshooting .......................................................................................................................................... 26 

How to Manually Start NetBackup Outside of Cluster Control .................................................................. 27 

Page 3: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 3/35

Page 4: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 4/35

4

 About NetBackup Clustering

A cluster provides high availability of applications and data to end users. In a cluster, 2 or more servers

(also known as nodes) are linked together to provide a highly available service to users. The servers runthe clustering software that allows each server in cluster to access shared disks. If a node in the cluster

fails the services that were running on that node are migrated to any other node that is available in the

cluster.

When you then install NetBackup in the cluster, it is installed as a virtual server application. The server

is assigned a virtual name (eg nbumaster), an IP address and shared disks. This server can then be failed

over (or switched) to any node in the cluster.

In a cluster configuration the NetBackup binaries are installed in/usr/openv locally on each node of

the cluster.

The shared storage /opt/VRTSnbu contains the following:

  Cluster configuration

  Image Catalog

  Volmgr files

  EMM database

For more detailed information please refer to the following guides:

Installation prerequisites for NetBackup on VCS on UNIX/Linux 

 About preinstallation checklist for a NetBackup failover server installation on VCS on Unix\Linux 

NetBackup installation information for all VCS configurations 

NetBackup installation information for specific VCS configurations 

Installing a failover NetBackup server on VCS on UNIX/Linux 

Configuring a failover NetBackup server on VCS on UNIX/Linux 

 About adding a node to existing cluster  

Installing or upgrading NetBackup database agents and options on VCS on UNIX/Linux  

Upgrading a NetBackup failover server on VCS on Unix/Linux  

Page 5: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 5/35

5

Cluster Configuration Details

2 x Red Hat Enterprise Linux Server release 5.5 (sym1 and sym2)

Netbackup 7.5 For Redhat Linux

Storage Foundation HA 6.0 for Linux (SFHA6.0)

Share Storage

First private heartbeat link: eth4 

Second private heartbeat link: eth5 

NIC for Virtual IP of the Cluster: eth0 

Cluster name: nbu 

Virtual IP address: 10.10.2.51 

Netmask: 255.255.255.0 

NetBackup Cluster Configuration Details

Disk Group Resource: nbudg 

Start Volumes: 1 

Stop Volumes: 1 

Volume Resource: nbuvol 

Mount Resource:

Mount Point: /opt/VRTSnbu 

Block Device: /dev/vx/dsk/nbudg/nbuvol 

File System Type: vxfs 

Fsck Option: -y 

NetBackup Virtual Name: nbumaster 

Virtual IP for NetBackup: 10.10.2.100

Page 6: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 6/35

6

Installing Storage Foundation HA using the CPI

Change directory where the install media is located and run the installer script:

Begin the installation and configuration of SFHA 6.0 on sym1 and sym2 using the following information:

 – Product: Veritas Storage Foundation and High Availability (SFHA)

 – Install all rpms

 – RHEL5 system names: sym1 and sym2

 – Enable keyless licensing and complete system licensing later

 – Product mode to license: SF Enterprise HA  

 – Disable replication

 – Disable the Global Cluster Option

- Do not configure I/O fencing at this stage

 – Cluster name: nbu 

- First private heartbeat link: eth4 

- Second private heartbeat link: eth5 

- No additional low-priority heartbeat link

- Use the same NICs for private heartbeat on all systems

 – Accept the unique cluster ID

 – NIC for Virtual IP of the Cluster to use on: eth0 

 – Virtual IP address: 10.10.2.51 

 – Net mask for 10.10.2.51: 255.255.255.0 

 – Do not configure the VCS cluster in secure mode.

 – Accept the default cluster credentials for the admin account.

 – Do not add another user.

 – Do not configure SMTP or SNMP notifications.

Page 7: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 7/35

7

Configuring Storage For Netbackup.

In this example I will be using 3 disks and creating a 6g volume.

# /etc/vx/bin/vxdisksetup -i emc0_dd1 (and repeat for other disks)

# /usr/sbin/vxdg init nbudg nbudg01=emc0_dd1 nbudg02=emc0_dd2

nbudg03=emc0_dd3

# /usr/sbin/vxassist -g nbudg make nbuvol 12359680

# /sbin/mkfs -t vxfs /dev/vx/rdsk/nbudg/nbuvol

Page 8: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 8/35

Page 9: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 9/35

9

On sym1:

Change directory where the NBU install media is located and run the install script.

Do you wish to continue? [y,n] (y) 

Participate in the NetBackup Product Improvement Program? [y,n] (y) n

The NetBackup and Media Manager software is built for use onLINUX_RH_X86 hardware.

Do you want to install NetBackup and Media Manager files? [y,n] (y)

NetBackup and Media Manager are normally installed in /usr/openv.

Is it OK to install in /usr/openv? [y,n] (y) 

Saving server binaries for LINUX_RH_X86.

A NetBackup Server or Enterprise Server license key is needed

for installation to continue.

Enter license key: ########################

All additional keys should be added at this time.

Do you want to add additional license keys now? [y,n] (y) n 

Would you like to use "sym1.example.com" as the configured

Page 10: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 10/35

10

NetBackup server name of this machine? [y,n] (y) n

Enter the name of this NetBackup server: nbumaster

Is nbumaster the master server? [y,n] (y)

Do you want to create a NetBackup cluster? [y,n] (y)

**********************************************************************

NetBackup will run on the following nodes: sym1

IP address of the node sym1 = 10.10.2.11

Is this correct? [y,n]: y

Enter the VCS resource name for the NIC (nbu_nic):

Enter the network device for NetBackup: eth0 

**********************************************************************

You entered the following for Network configuration

NetBackup NIC VCS resource name = nbu_nic

The NIC to use with NetBackup on sym1 = eth0

Is this correct? [y/n]: y

Enter the VCS group name for the NetBackup group (nbu_group): 

Enter the VCS resource name for NetBackup (nbu_server):

**********************************************************************

Page 11: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 11/35

11

VCS group name for the NetBackup group is :nbu_group

VCS resource name for NetBackup is : nbu_server

Is this correct? [y,n]: y 

Enter the VCS resource name for the NetBackup server IP (nbu_ip):

Enter the IP address for nbumaster: 10.10.2.100 

Enter the netmask for the NetBackup service (255.255.255.0):

**********************************************************************

You entered the following for Virtual IP configuration

NetBackup IP Resource name = nbu_ip

NetBackup virtual name = nbumaster

The IP address to use with NetBackup = 10.10.2.100

The IP address type( IPv4 or IPv6 ) = IPv4

The netmask to use with NetBackup = 255.255.255.0

Is this correct? [y/n]: y

Select your VCS disk configuration. 

1. Disk Group

2. None (We will just configure a mount for NetBackup)

Enter the disk configuration: 1

Page 12: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 12/35

12

Disk Group Configuration.

Enter the VCS disk group resource name for NetBackup (nbu_dg):

Enter the disk group for NetBackup: nbudg

Enter the Start Volumes for NetBackup [0,1]: 1 

Enter the Stop Volumes for NetBackup [0,1]: 1 

Do you have volume resources to configure? [y,n]: y 

Enter the VCS Resource name for your volume (nbu_vol):

Enter the name of your volume: nbuvol 

**********************************************************************

You have entered the following:

Disk Group Resource Name = nbu_dg

Disk Group = nbudg

Start Volumes = 1

Stop Volumes = 1

Volume Resource Name = nbu_vol

Volume = nbuvol

Is this correct? [y,n]: y

Enter the VCS resource name for the NetBackup shared disk mount point

(nbu_mount):

Enter the mount point for the NetBackup disk (/opt/VRTSnbu):

Enter the Block Device for the NetBackup disk:

/dev/vx/dsk/nbudg/nbuvol

Enter the FS Type for the NetBackup file system: vxfs

Enter any mount options you would like passed to the NetBackup disk

(Enter for none):

Page 13: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 13/35

13

Enter any fsck options you would like passed to the NetBackup disk

(Default is -y. This must be included if options are added):

**********************************************************************

VCS NetBackup Mount Point resource name = nbu_mount

NetBackup Mount Point = /opt/VRTSnbu

Block Device = /dev/vx/dsk/nbudg/nbuvol

File System type = vxfs

fsck options = -y

Is this correct? [y,n]: y

**********************************************************************

Creating NetBackup Group....

Setting up NetBackup limitations on sym1

Adding NetBackup type to VCS...

Adding nbu_group

Adding nbu_nic to nbu_group

Adding nbu_ip to nbu_group

Adding nbu_dg to nbu_group

Adding nbu_vol to nbu_group

Adding nbu_mount to nbu_group

Linking nbu_mount to nbu_vol

Linking nbu_vol to nbu_dg

Linking nbu_ip to nbu_nic

Linking nbu_server to nbu_ip

Linking nbu_server to nbu_mount

Probing resources on sym1.

Page 14: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 14/35

14

Waiting for NIC resource to finish probing

**********************************************************************

Do you want to add any media servers now? [y,n] (n)

Checking network connections.

DNS Lookup failed for host nbumaster error:-2!

bp.conf: IP_ADDRESS_FAMILY = AF_INET: default value, no update needed

TELEMETRY_UPLOAD value NO saved.

Sending SIGHUP to xinetd process.

Reloading configuration: [ OK ]

NetBackup maintains a centralized catalog (separate from the image

catalog) for data related to media and device configuration, device

management, storage units, hosts and host aliases, media server

status,

NDMP credentials, and other information. This is managed by the

Enterprise Media Manager server.

Enter the Enterprise Media Manager server (default: nbumaster):

Enter the OpsCenter server (default: NONE):

NetBackup daemons will be shut down to complete the

NetBackup cluster configuration.

Updating /usr/openv/volmgr/vm.conf

We have detected that this is an EMM server.

Page 15: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 15/35

15

Adding nbemm, nbrb & NB_dbsrv to the list of monitored daemons...

Starting the NetBackup agent on sym1

Waiting 240 seconds for nbu_group to come online...

Waiting 210 seconds for nbu_group to come online...

Waiting 180 seconds for nbu_group to come online...

Waiting 150 seconds for nbu_group to come online...

Waiting 120 seconds for nbu_group to come online...

Waiting 90 seconds for nbu_group to come online...

Waiting 60 seconds for nbu_group to come online...

Waiting 30 seconds for nbu_group to come online...

Failed to start the NetBackup group. Check your configuration

NetBackup server installation complete.

File /usr/openv/tmp/install_trace.13991 contains a trace of this

install.

That file can be deleted after you are sure the install was

successful.

Page 16: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 16/35

16

We can confirm that NBU is up by running /opt/VRTSvcs/bin/hastatus -sum 

We can see that the nbu_group is running on sym1. Adding sym2 to the cluster occurs later on.

# /opt/VRTSvcs/bin/hastatus -sum 

-- SYSTEM STATE

-- System State Frozen

A sym1 RUNNING 0

A sym2 RUNNING 0

-- GROUP STATE

-- Group System Probed AutoDisabled State

B ClusterService sym1 Y N ONLINE

B ClusterService sym2 Y N OFFLINE

B nbu_group sym1 Y N ONLINE

Page 17: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 17/35

17

We can check what NetBackup processes have started by running the following:

# /usr/openv/netbackup/bin/bpps -x

NB Processes

------------

root 20533 1 0 17:23 ? 00:00:00

/usr/openv/netbackup/bin/vnetd -standalone

root 20536 1 0 17:23 ? 00:00:00 /usr/openv/netbackup/bin/bpcd

-standalone

root 20624 1 1 17:23 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv

@/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn

7

root 20655 1 0 17:23 ? 00:00:00

/usr/openv/netbackup/bin/nbevtmgr

root 20673 1 0 17:23 ? 00:00:00

/usr/openv/netbackup/bin/nbaudit

root 21015 1 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/nbemm

root 21072 1 0 17:24 ? 00:00:00 /usr/openv/netbackup/bin/nbrb

root 21146 1 0 17:24 ? 00:00:00 /usr/openv/netbackup/bin/bprd

root 21160 1 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/bpcompatd

root 21163 1 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/bpdbm

root 21166 1 0 17:24 ? 00:00:00 /usr/openv/netbackup/bin/nbjm

root 21199 1 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/nbpem

root 21213 1 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/nbstserv

root 21237 1 0 17:24 ? 00:00:00 /usr/openv/netbackup/bin/nbim

root 21240 21163 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/bpjobd

root 21272 1 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/nbrmms

root 21365 1 0 17:24 ? 00:00:00 /usr/openv/netbackup/bin/nbsl

Page 18: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 18/35

18

root 21395 1 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/nbars

root 21494 1 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/nbvault

root 21499 1 0 17:24 ? 00:00:00/usr/openv/netbackup/bin/nbcssc -a NetBackup

root 21771 1 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/admincmd/bpstsinfo -UPDATE

root 21791 21199 0 17:24 ? 00:00:00

/usr/openv/netbackup/bin/nbproxy dblib nbpem

MM Processes

------------

root 21089 1 0 17:24 ? 00:00:00 /usr/openv/volmgr/bin/ltid

root 21566 1 0 17:24 ? 00:00:00 vmd

Shared Symantec Processes

-------------------------

root 6789 1 0 16:22 ? 00:00:00 /opt/VRTSpbx/bin/pbx_exchange

At this point we are now ready to go to sym2 and install NBU.

Page 19: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 19/35

19

On sym2 change directory where the NBU install media is located and run the install script.

Do you wish to continue? [y,n] (y)

Participate in the NetBackup Product Improvement Program? [y,n] (y) n 

The NetBackup and Media Manager software is built for use on

LINUX_RH_X86 hardware.

Do you want to install NetBackup and Media Manager files? [y,n] (y) 

NetBackup and Media Manager are normally installed in /usr/openv.

Is it OK to install in /usr/openv? [y,n] (y)

Would you like to use "sym2.example.com" as the configured

NetBackup server name of this machine? [y,n] (y) n 

Enter the name of this NetBackup server: nbumaster 

Is nbumaster the master server? [y,n] (y) y 

This is the list of existing NetBackup cluster groups:

nbu_group

Do you wish to join this existing NetBackup cluster group? [y,n] (y) y

Page 20: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 20/35

20

Adding node sym2 to NetBackup Group nbu_group VCS

Enter the network device for NetBackup on sym2: eth0 

You entered the following for Network configuration

The NIC to use with NetBackup on sym2 = eth0

The IP address for sym2 is : 10.10.2.12

Is this correct? [y/n]: y

Do you want to add any media servers now? [y,n] (n) 

Enter the OpsCenter server (default: NONE):

NetBackup server installation complete.

Page 21: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 21/35

21

We can confirm that sym2 has been added the Veritas Cluster Configuration for the nbu_group and is in

an offline state on sym2.

# /opt/VRTSvcs/bin/hastatus -sum

-- SYSTEM STATE

-- System State Frozen

A sym1 RUNNING 0

A sym2 RUNNING 0

-- GROUP STATE

-- Group System Probed AutoDisabled State

B ClusterService sym1 Y N ONLINE

B ClusterService sym2 Y N OFFLINE

B nbu_group sym1 Y N ONLINE

B nbu_group sym2 Y N OFFLINE

Page 22: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 22/35

22

Reviewing and Understanding the Configuration

The /etc/VRTSvcs/conf/config/main.cf shows the configuration for Netbackup service

group.

group nbu_group (

SystemList = { sym1 = 0, sym2 = 1 }

AutoStartList = { sym1, sym2 }

)

DiskGroup nbu_dg (

DiskGroup = nbudg

)

IP nbu_ip (

Device @sym1 = eth0

Device @sym2 = eth0

Address = "10.10.2.100"

NetMask = "255.255.255.0"

)

Mount nbu_mount (

MountPoint = "/opt/VRTSnbu"

BlockDevice = "/dev/vx/dsk/nbudg/nbuvol"

FSType = vxfs

FsckOpt = "-y"

)

NIC nbu_nic (

Page 23: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 23/35

23

Device @sym1 = eth0

Device @sym2 = eth0

NetworkHosts = { "10.10.2.11", "10.10.2.12" }

)

NetBackup nbu_server (

Critical = 0

ResourceOwner = unknown

ServerName = NBU_Server

ServerType = NBUMaster

MonScript = NONE

RSPFile = "/usr/openv/netbackup/bin/cluster/NBU_RSP"

GroupName = nbu_group

Volume nbu_vol (

DiskGroup = nbudg

Volume = nbuvol

)

nbu_ip requires nbu_nic

nbu_mnt_stu requires nbu_vol

nbu_mount requires nbu_vol

nbu_server requires nbu_ip

nbu_server requires nbu_mnt_stu

nbu_server requires nbu_mount

nbu_vol requires nbu_dg

We can see that Veritas Cluster Server has populated the main.cf with all the information provided

during the installation of Netbackup.

Page 24: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 24/35

24

How the Netbackup Agent for Veritas Cluster Works

The NetBackup Agent for Veritas Cluster brings online, takes offline and monitors NetBackup processes.

Agent Functions:

Online: Starts NetBackup using

/usr/openv/netbackup/bin/cluster/util/online\

/usr/openv/netbackup/bin/cluster/NBU_RSP

The Online will start the processes listed in the START_PROCS from the NBU_RSP file 

Offline: Stops NetBackup using

/usr/openv/netbackup/bin/cluster/util/offline\

/usr/openv/netbackup/bin/cluster/NBU_RSP

Monitor: probes the processes listed in the PROBE_PROCS from the file

/usr/openv/netbackup/bin/cluster/NBU_RSP

Also runs a /usr/openv/netbackup/bin/bpdbm –alive -local

The Offline will stop the processes listed in the START_PROCS from the NBU_RSP file 

We need to understand what the "/usr/openv/netbackup/bin/cluster/NBU_RSP" is used

for. This file contains some of the following required to start NetBackup in a cluster.

  Service Group Name

  The Shared Storage

  Nodes that are clustered for Netbackup

  The Virtual name of the Netbackup Master Server

  IP and Subnet

  Cluster type

  What Netbackup Process to Start

  What Netbackup Process to Monitor

Page 25: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 25/35

25

The format of the file is as follows:

#DO NOT DELETE OR EDIT THIS FILE!!!

NBU_GROUP=nbu_group

SHARED_DISK=/opt/VRTSnbu

NODES=sym1 sym2

VNAME=nbumaster

VIRTUAL_IP=10.10.2.100

SUBNET=255.255.255.0

CLUTYPE=VCS

START_PROCS=NB_dbsrv nbevtmgr nbemm nbrb ltid vmd bpcompatd nbjm nbpem

nbstserv nbrmms nbsl nbvault nbsvcmon bpdbm bprd bptm bpbrmds bpsched bpcd

bpversion bpjobd nbproxy vltcore acsd tl8cd odld tldcd tl4d tlmd tshd rsmd

tlhcd pbx_exchange nbkms nbaudit nbatd nbazd nbim

PRODUCT_CODE=NBU

DIR=netbackup mkdir

DIR=netbackup/db mv

DIR=var mkdir

DIR=var/global mv

DIR=volmgr/mkdir

DIR=volmgr/misc mkdir

DIR=volmgr/misc/robotic_db mv

DIR=kms mv

DIR=netbackup/vault mkdir

DIR=netbackup/vault/sessions mv

LINK=volmgr/misc/robotic_db

LINK=netbackup/db

LINK=netbackup/vault/sessions

LINK=var/global

Page 26: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 26/35

26

PROBE_PROCS=nbevtmgr nbstserv vmd bprd bpdbm nbpem nbjm nbaudit nbsl nbrmms

nbemm nbrb NB_dbsrv 

Troubleshooting

If NetBackup fails to start on a node in the cluster we can review the following logs file to determine

why.

The NetBackup Agent log is here:

/usr/openv/netbackup/bin/cluster/AGENT_DEBUG.log.

This file will show if NetBackup started successfully or not. An example of a successful online:

Mon Mar 24 21:29:05 2014 vcs/Offine.pl: Calling:

/usr/openv/netbackup/bin/cluster/util/offline/usr/openv/netbackup/bin/cluster/NBU_RSP

An example of a failed online:

Thu Apr 19 17:02:24 2012 vcs/Online.pl: Calling: /online/usr/openv/netbackup/bin/cluster/NBU_RSPThu Apr 19 17:02:24 2012 vcs/online.pl:/usr/openv/netbackup/bin/cluster/util/online/usr/openv/netbackup/bin/cluster/NBU_RSP exited with 99Thu Apr 19 17:02:24 2012 vcs/Online.pl: Calling:

/usr/openv/netbackup/bin/cluster/util/online

/usr/openv/netbackup/bin/cluster/NBU_RSP

The NetBackup Cluster log is here:

/usr/openv/netbackup/logs/cluster/log.xxx 

as an example we can see from the logs why NetBackup did not start.

17:02:42.446 [14307] <4> NBClusterApp::createLinks: Creating link from

/usr/openv/volmgr/misc/robotic_db to /nbu/volmgr/misc/robotic_db 17:02:42.600

[14307] <4> NBClusterApp::createLinks: Local file stat: -1, Shared disk file

stat: -1 17:02:42.600 [14307] <4> NBClusterApp::createLinks: Couldn't create

link to shared disk. Application can not start. 

From this example output we can see that it failed to start because it could not create a link to shared

disks.

Page 27: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 27/35

27

How to Manually Start NetBackup Outside of Cluster Control

Is it important NOT to run “/usr/openv/netbackup/bin/goodies/netbackup start” to

start NetBackup manually outside of the cluster.

The reason is Netbackup expects to find certain files in /usr/openv/netbackup. When VCS starts

NetBackup it first creates symbolic links that are required and THEN starts NetBackup.

The /usr/openv/netbackup/logs/cluster/log.xxx  shows exactly what the NetBackup

Cluster Server Agent does.

Step 1 - it sources the NBU_RSP file

18:45:54.603 [29616] <4> Online::main: Initializing NBCluster using

/usr/openv/netbackup/bin/cluster/NBU_RSP file

Step 2 - it then is creates the symbolic links here:

18:45:54.642 [29616] <4> NBClusterApp::createLinks: Creating link from

/usr/openv/volmgr/misc/robotic_db to /opt/VRTSnbu/volmgr/misc/robotic_db

18:45:54.642 [29616] <4> NBClusterApp::createLinks: Local file stat: -1,

Shared disk file stat: 0

18:45:54.642 [29616] <4> NBClusterApp::createLinks: Creating link from

/usr/openv/netbackup/db to /opt/VRTSnbu/netbackup/db

18:45:54.642 [29616] <4> NBClusterApp::createLinks: Local file stat: -1,

Shared disk file stat: 0

18:45:54.642 [29616] <4> NBClusterApp::createLinks: Creating link from

/usr/openv/netbackup/vault/sessions to /opt/VRTSnbu/netbackup/vault/sessions

18:45:54.643 [29616] <4> NBClusterApp::createLinks: Local file stat: -1,

Shared disk file stat: 0

18:45:54.643 [29616] <4> NBClusterApp::createLinks: Creating link from

/usr/openv/var/global to /opt/VRTSnbu/var/global

18:45:54.643 [29616] <4> NBClusterApp::createLinks: Local file stat: -1,Shared disk file stat: 0

Step 3 - after the symbolic links are created, the agent then starts NetBackup as follows:

18:45:54.643 [29616] <4> NBClusterApp::runCommand: Entering, with

command:/usr/openv/netbackup/bin/goodies/netbackup start

Page 28: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 28/35

28

After NetBackup starts you can check the links:

# ls -l /usr/openv/volmgr/misc/robotic_db

lrwxrwxrwx 1 root root 35 Mar 25 18:52 /usr/openv/volmgr/misc/robotic_db ->

/opt/VRTSnbu/volmgr/misc/robotic_db

# ls -l /usr/openv/netbackup/vault/sessions

lrwxrwxrwx 1 root root 37 Mar 25 18:52 /usr/openv/netbackup/vault/sessions ->

/opt/VRTSnbu/netbackup/vault/sessions

# ls -l /usr/openv/var/global

lrwxrwxrwx 1 root root 23 Mar 25 18:52 /usr/openv/var/global ->

/opt/VRTSnbu/var/global

If you have to manually start NetBackup outside of Cluster Server you will need to create the above links

before you run “/usr/openv/netbackup/bin/goodies/netbackup start” 

Likewise if you now need to stop NetBackup you need to remember to remove the links after NetBackup

has stopped.

Page 29: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 29/35

29

This is what will happen if you don’t create the symbolic links before you start the NetBackup.

Firstly lets confirm that the service group is offline.

# hagrp -state nbu_group

#Group Attribute System Value

nbu_group State sym1 |OFFLINE|

nbu_group State sym2 |OFFLINE|

Lets now check and make sure that there are no NetBackup processes running:

# /usr/openv/netbackup/bin/bpps -x

NB Processes

------------

root 13472 1 0 15:57 ? 00:00:00

/usr/openv/netbackup/bin/vnetd -standalone

root 13475 1 0 15:57 ? 00:00:00 /usr/openv/netbackup/bin/bpcd

- standalone

MM Processes

------------

Shared Symantec Processes

-----------

root 6759 1 0 15:50 ? 00:00:00 /opt/VRTSpbx/bin/pbx_exchange

Using the JAVA GUI or command line, online all resources EXPECT for the nbu_server

From the /var/VRTSvcs/log/engine_A.log we can confirm that our resources are up.

2014/04/07 16:06:29 VCS INFO V-16-1-10298 Resource nbu_dg (Owner:

Unspecified, Group: nbu_group) is online on sym2 (VCS initiated)

2014/04/07 16:06:45 VCS INFO V-16-1-10298 Resource nbu_vol (Owner:

Unspecified, Group: nbu_group) is online on sym2 (VCS initiated)

2014/04/07 16:07:05 VCS INFO V-16-1-10298 Resource nbu_ip (Owner:

Unspecified, Group: nbu_group) is online on sym2 (VCS initiated)

Page 30: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 30/35

30

2014/04/07 16:07:05 VCS NOTICE V-16-1-10301 Initiating Online of Resource

nbu_mount (Owner: Unspecified, Group: nbu_group) on System sym2

2014/04/07 16:07:07 VCS INFO V-16-1-10298 Resource nbu_mount (Owner:

Unspecified, Group: nbu_group) is online on sym2 (VCS initiated)

Confirm the symbolic links are not there.

# ls -l /usr/openv/volmgr/misc/robotic_db

ls: /usr/openv/volmgr/misc/robotic_db: No such file or directory

# ls -l /usr/openv/netbackup/vault/sessions

ls: /usr/openv/netbackup/vault/sessions: No such file or directory

# ls -l /usr/openv/var/global

ls: /usr/openv/var/global: No such file or directory

Lets now start Netbackup without creating the symbolic links.

# /usr/openv/netbackup/bin/goodies/netbackup start

NetBackup Authentication daemon started.

NetBackup network daemon started.

NetBackup client daemon started.

NetBackup SAN Client Fibre Transport daemon started.

NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running

NetBackup Authorization daemon started.

NetBackup Event Manager started.

NetBackup Audit Manager started.

NetBackup Deduplication Manager started.

NetBackup Deduplication Engine started.

NetBackup Enterprise Media Manager started.

NetBackup Resource Broker started.

Rebuilding device nodes.

Page 31: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 31/35

31

Media Manager daemons started.

NetBackup request daemon started.

NetBackup compatibility daemon started.

NetBackup Job Manager started.

NetBackup Policy Execution Manager started.

NetBackup Storage Lifecycle Manager started.

NetBackup Indexing Manager started.

NetBackup Remote Monitoring Management System started.

NetBackup Key Management daemon started.

NetBackup Service Layer started.

NetBackup Agent Request Server started.

NetBackup Bare Metal Restore daemon started.

NetBackup Vault daemon started.

NetBackup CloudStore Service Container started.

VCS

NetBackup Service Monitor started.

NetBackup Bare Metal Restore Boot Server daemon started.

Let us once again confirm that the links are not there:

# ls -l /usr/openv/volmgr/misc/robotic_db

ls: /usr/openv/volmgr/misc/robotic_db: No such file or directory

# ls -l /usr/openv/netbackup/vault/sessions

ls: /usr/openv/netbackup/vault/sessions: No such file or directory

# ls -l /usr/openv/var/global

ls: /usr/openv/var/global: No such file or directory

Page 32: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 32/35

Page 33: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 33/35

33

We can see that several processes are missing. For example theNB_dbsrv  is missing. The NB_dbsrv

is the NetBackup Database. Even though the NetBackup Database NB_dbsrv is not running, the

NetBackup Agent reports it as online. We can confirm by running:

# /usr/openv/netbackup/bin/cluster/util/monitor

/usr/openv/netbackup/bin/cluster/NBU_RSP

Some Processes are DOWN while others are UP

Following Process are found DOWN: nbstserv bprd bpdbm nbpem nbaudit nbsl

nbemm nbrb NB_dbsrv

Following Process are found UP: nbevtmgr vmd nbjm nbrmms

# echo $?

110

The value 110 means the nbu_server is ONLINE, even though the processesnbstserv bprd bpdbm

nbpem nbaudit nbsl nbemm nbrb NB_dbsrv are not running.

Since the monitor is reporting 110 the cluster thinks the service group is up. 

# hagrp -state nbu_group

#Group Attribute System Value

nbu_group State sym1 |OFFLINE|

nbu_group State sym2 |ONLINE|

The NetBackup agent should be reporting the value 100, which is an OFFLINE. Etracke348052 has

been created to address this issue.

To resolve this you can do the following:

# /usr/openv/netbackup/bin/bp.kill_all

# /usr/openv/netbackup/bin/cluster/util/online\

/usr/openv/netbackup/bin/cluster/NBU_RSP

Now we can see that symbolic links have been created.

# ls -l /usr/openv/volmgr/misc/robotic_db

lrwxrwxrwx 1 root root 35 Apr 7 16:58 /usr/openv/volmgr/misc/robotic_db ->

/opt/VRTSnbu/volmgr/misc/robotic_db

Page 34: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 34/35

34

# ls -l /usr/openv/netbackup/vault/sessions

lrwxrwxrwx 1 root root 37 Apr 7 16:58 /usr/openv/netbackup/vault/sessions ->

/opt/VRTSnbu/netbackup/vault/sessions

# ls -l /usr/openv/var/global

lrwxrwxrwx 1 root root 23 Apr 7 16:58 /usr/openv/var/global ->

/opt/VRTSnbu/var/global

# ls -l /usr/openv/netbackup/db

lrwxrwxrwx 1 root root 25 Apr 7 16:58 /usr/openv/netbackup/db ->

/opt/VRTSnbu/netbackup/db

Page 35: How to Cluster a Netbackup 7

8/9/2019 How to Cluster a Netbackup 7

http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 35/35


Recommended