8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 1/35
SYM
ANT
EC
STORAGE
FOUNDATION
HIGH
AVAILA
BILITY
How To Cluster a NetBackup 7.x
Master Server using Veritas Cluster
Server 6.x for Unix/Linux
Robert Budini
Principal Support Engineer
8th
April 2014
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 2/35
2
Table of ContentsIntroduction .................................................................................................................................................. 3
About NetBackup Clustering ......................................................................................................................... 4
Cluster Configuration Details ........................................................................................................................ 5
NetBackup Cluster Configuration Details ..................................................................................................... 5
Configuring Storage For Netbackup. ............................................................................................................. 7
Installing and Configuring NetBackup ........................................................................................................... 8
Reviewing and Understanding the Configuration ....................................................................................... 22
How the Netbackup Agent for Veritas Cluster Works ................................................................................ 24
Troubleshooting .......................................................................................................................................... 26
How to Manually Start NetBackup Outside of Cluster Control .................................................................. 27
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 3/35
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 4/35
4
About NetBackup Clustering
A cluster provides high availability of applications and data to end users. In a cluster, 2 or more servers
(also known as nodes) are linked together to provide a highly available service to users. The servers runthe clustering software that allows each server in cluster to access shared disks. If a node in the cluster
fails the services that were running on that node are migrated to any other node that is available in the
cluster.
When you then install NetBackup in the cluster, it is installed as a virtual server application. The server
is assigned a virtual name (eg nbumaster), an IP address and shared disks. This server can then be failed
over (or switched) to any node in the cluster.
In a cluster configuration the NetBackup binaries are installed in/usr/openv locally on each node of
the cluster.
The shared storage /opt/VRTSnbu contains the following:
Cluster configuration
Image Catalog
Volmgr files
EMM database
For more detailed information please refer to the following guides:
Installation prerequisites for NetBackup on VCS on UNIX/Linux
About preinstallation checklist for a NetBackup failover server installation on VCS on Unix\Linux
NetBackup installation information for all VCS configurations
NetBackup installation information for specific VCS configurations
Installing a failover NetBackup server on VCS on UNIX/Linux
Configuring a failover NetBackup server on VCS on UNIX/Linux
About adding a node to existing cluster
Installing or upgrading NetBackup database agents and options on VCS on UNIX/Linux
Upgrading a NetBackup failover server on VCS on Unix/Linux
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 5/35
5
Cluster Configuration Details
2 x Red Hat Enterprise Linux Server release 5.5 (sym1 and sym2)
Netbackup 7.5 For Redhat Linux
Storage Foundation HA 6.0 for Linux (SFHA6.0)
Share Storage
First private heartbeat link: eth4
Second private heartbeat link: eth5
NIC for Virtual IP of the Cluster: eth0
Cluster name: nbu
Virtual IP address: 10.10.2.51
Netmask: 255.255.255.0
NetBackup Cluster Configuration Details
Disk Group Resource: nbudg
Start Volumes: 1
Stop Volumes: 1
Volume Resource: nbuvol
Mount Resource:
Mount Point: /opt/VRTSnbu
Block Device: /dev/vx/dsk/nbudg/nbuvol
File System Type: vxfs
Fsck Option: -y
NetBackup Virtual Name: nbumaster
Virtual IP for NetBackup: 10.10.2.100
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 6/35
6
Installing Storage Foundation HA using the CPI
Change directory where the install media is located and run the installer script:
Begin the installation and configuration of SFHA 6.0 on sym1 and sym2 using the following information:
– Product: Veritas Storage Foundation and High Availability (SFHA)
– Install all rpms
– RHEL5 system names: sym1 and sym2
– Enable keyless licensing and complete system licensing later
– Product mode to license: SF Enterprise HA
– Disable replication
– Disable the Global Cluster Option
- Do not configure I/O fencing at this stage
– Cluster name: nbu
- First private heartbeat link: eth4
- Second private heartbeat link: eth5
- No additional low-priority heartbeat link
- Use the same NICs for private heartbeat on all systems
– Accept the unique cluster ID
– NIC for Virtual IP of the Cluster to use on: eth0
– Virtual IP address: 10.10.2.51
– Net mask for 10.10.2.51: 255.255.255.0
– Do not configure the VCS cluster in secure mode.
– Accept the default cluster credentials for the admin account.
– Do not add another user.
– Do not configure SMTP or SNMP notifications.
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 7/35
7
Configuring Storage For Netbackup.
In this example I will be using 3 disks and creating a 6g volume.
# /etc/vx/bin/vxdisksetup -i emc0_dd1 (and repeat for other disks)
# /usr/sbin/vxdg init nbudg nbudg01=emc0_dd1 nbudg02=emc0_dd2
nbudg03=emc0_dd3
# /usr/sbin/vxassist -g nbudg make nbuvol 12359680
# /sbin/mkfs -t vxfs /dev/vx/rdsk/nbudg/nbuvol
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 8/35
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 9/35
9
On sym1:
Change directory where the NBU install media is located and run the install script.
Do you wish to continue? [y,n] (y)
Participate in the NetBackup Product Improvement Program? [y,n] (y) n
The NetBackup and Media Manager software is built for use onLINUX_RH_X86 hardware.
Do you want to install NetBackup and Media Manager files? [y,n] (y)
NetBackup and Media Manager are normally installed in /usr/openv.
Is it OK to install in /usr/openv? [y,n] (y)
Saving server binaries for LINUX_RH_X86.
A NetBackup Server or Enterprise Server license key is needed
for installation to continue.
Enter license key: ########################
All additional keys should be added at this time.
Do you want to add additional license keys now? [y,n] (y) n
Would you like to use "sym1.example.com" as the configured
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 10/35
10
NetBackup server name of this machine? [y,n] (y) n
Enter the name of this NetBackup server: nbumaster
Is nbumaster the master server? [y,n] (y)
Do you want to create a NetBackup cluster? [y,n] (y)
**********************************************************************
NetBackup will run on the following nodes: sym1
IP address of the node sym1 = 10.10.2.11
Is this correct? [y,n]: y
Enter the VCS resource name for the NIC (nbu_nic):
Enter the network device for NetBackup: eth0
**********************************************************************
You entered the following for Network configuration
NetBackup NIC VCS resource name = nbu_nic
The NIC to use with NetBackup on sym1 = eth0
Is this correct? [y/n]: y
Enter the VCS group name for the NetBackup group (nbu_group):
Enter the VCS resource name for NetBackup (nbu_server):
**********************************************************************
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 11/35
11
VCS group name for the NetBackup group is :nbu_group
VCS resource name for NetBackup is : nbu_server
Is this correct? [y,n]: y
Enter the VCS resource name for the NetBackup server IP (nbu_ip):
Enter the IP address for nbumaster: 10.10.2.100
Enter the netmask for the NetBackup service (255.255.255.0):
**********************************************************************
You entered the following for Virtual IP configuration
NetBackup IP Resource name = nbu_ip
NetBackup virtual name = nbumaster
The IP address to use with NetBackup = 10.10.2.100
The IP address type( IPv4 or IPv6 ) = IPv4
The netmask to use with NetBackup = 255.255.255.0
Is this correct? [y/n]: y
Select your VCS disk configuration.
1. Disk Group
2. None (We will just configure a mount for NetBackup)
Enter the disk configuration: 1
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 12/35
12
Disk Group Configuration.
Enter the VCS disk group resource name for NetBackup (nbu_dg):
Enter the disk group for NetBackup: nbudg
Enter the Start Volumes for NetBackup [0,1]: 1
Enter the Stop Volumes for NetBackup [0,1]: 1
Do you have volume resources to configure? [y,n]: y
Enter the VCS Resource name for your volume (nbu_vol):
Enter the name of your volume: nbuvol
**********************************************************************
You have entered the following:
Disk Group Resource Name = nbu_dg
Disk Group = nbudg
Start Volumes = 1
Stop Volumes = 1
Volume Resource Name = nbu_vol
Volume = nbuvol
Is this correct? [y,n]: y
Enter the VCS resource name for the NetBackup shared disk mount point
(nbu_mount):
Enter the mount point for the NetBackup disk (/opt/VRTSnbu):
Enter the Block Device for the NetBackup disk:
/dev/vx/dsk/nbudg/nbuvol
Enter the FS Type for the NetBackup file system: vxfs
Enter any mount options you would like passed to the NetBackup disk
(Enter for none):
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 13/35
13
Enter any fsck options you would like passed to the NetBackup disk
(Default is -y. This must be included if options are added):
**********************************************************************
VCS NetBackup Mount Point resource name = nbu_mount
NetBackup Mount Point = /opt/VRTSnbu
Block Device = /dev/vx/dsk/nbudg/nbuvol
File System type = vxfs
fsck options = -y
Is this correct? [y,n]: y
**********************************************************************
Creating NetBackup Group....
Setting up NetBackup limitations on sym1
Adding NetBackup type to VCS...
Adding nbu_group
Adding nbu_nic to nbu_group
Adding nbu_ip to nbu_group
Adding nbu_dg to nbu_group
Adding nbu_vol to nbu_group
Adding nbu_mount to nbu_group
Linking nbu_mount to nbu_vol
Linking nbu_vol to nbu_dg
Linking nbu_ip to nbu_nic
Linking nbu_server to nbu_ip
Linking nbu_server to nbu_mount
Probing resources on sym1.
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 14/35
14
Waiting for NIC resource to finish probing
**********************************************************************
Do you want to add any media servers now? [y,n] (n)
Checking network connections.
DNS Lookup failed for host nbumaster error:-2!
bp.conf: IP_ADDRESS_FAMILY = AF_INET: default value, no update needed
TELEMETRY_UPLOAD value NO saved.
Sending SIGHUP to xinetd process.
Reloading configuration: [ OK ]
NetBackup maintains a centralized catalog (separate from the image
catalog) for data related to media and device configuration, device
management, storage units, hosts and host aliases, media server
status,
NDMP credentials, and other information. This is managed by the
Enterprise Media Manager server.
Enter the Enterprise Media Manager server (default: nbumaster):
Enter the OpsCenter server (default: NONE):
NetBackup daemons will be shut down to complete the
NetBackup cluster configuration.
Updating /usr/openv/volmgr/vm.conf
We have detected that this is an EMM server.
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 15/35
15
Adding nbemm, nbrb & NB_dbsrv to the list of monitored daemons...
Starting the NetBackup agent on sym1
Waiting 240 seconds for nbu_group to come online...
Waiting 210 seconds for nbu_group to come online...
Waiting 180 seconds for nbu_group to come online...
Waiting 150 seconds for nbu_group to come online...
Waiting 120 seconds for nbu_group to come online...
Waiting 90 seconds for nbu_group to come online...
Waiting 60 seconds for nbu_group to come online...
Waiting 30 seconds for nbu_group to come online...
Failed to start the NetBackup group. Check your configuration
NetBackup server installation complete.
File /usr/openv/tmp/install_trace.13991 contains a trace of this
install.
That file can be deleted after you are sure the install was
successful.
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 16/35
16
We can confirm that NBU is up by running /opt/VRTSvcs/bin/hastatus -sum
We can see that the nbu_group is running on sym1. Adding sym2 to the cluster occurs later on.
# /opt/VRTSvcs/bin/hastatus -sum
-- SYSTEM STATE
-- System State Frozen
A sym1 RUNNING 0
A sym2 RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B ClusterService sym1 Y N ONLINE
B ClusterService sym2 Y N OFFLINE
B nbu_group sym1 Y N ONLINE
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 17/35
17
We can check what NetBackup processes have started by running the following:
# /usr/openv/netbackup/bin/bpps -x
NB Processes
------------
root 20533 1 0 17:23 ? 00:00:00
/usr/openv/netbackup/bin/vnetd -standalone
root 20536 1 0 17:23 ? 00:00:00 /usr/openv/netbackup/bin/bpcd
-standalone
root 20624 1 1 17:23 ? 00:00:00 /usr/openv/db//bin/NB_dbsrv
@/usr/openv/var/global/server.conf @/usr/openv/var/global/databases.conf -hn
7
root 20655 1 0 17:23 ? 00:00:00
/usr/openv/netbackup/bin/nbevtmgr
root 20673 1 0 17:23 ? 00:00:00
/usr/openv/netbackup/bin/nbaudit
root 21015 1 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/nbemm
root 21072 1 0 17:24 ? 00:00:00 /usr/openv/netbackup/bin/nbrb
root 21146 1 0 17:24 ? 00:00:00 /usr/openv/netbackup/bin/bprd
root 21160 1 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/bpcompatd
root 21163 1 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/bpdbm
root 21166 1 0 17:24 ? 00:00:00 /usr/openv/netbackup/bin/nbjm
root 21199 1 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/nbpem
root 21213 1 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/nbstserv
root 21237 1 0 17:24 ? 00:00:00 /usr/openv/netbackup/bin/nbim
root 21240 21163 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/bpjobd
root 21272 1 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/nbrmms
root 21365 1 0 17:24 ? 00:00:00 /usr/openv/netbackup/bin/nbsl
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 18/35
18
root 21395 1 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/nbars
root 21494 1 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/nbvault
root 21499 1 0 17:24 ? 00:00:00/usr/openv/netbackup/bin/nbcssc -a NetBackup
root 21771 1 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/admincmd/bpstsinfo -UPDATE
root 21791 21199 0 17:24 ? 00:00:00
/usr/openv/netbackup/bin/nbproxy dblib nbpem
MM Processes
------------
root 21089 1 0 17:24 ? 00:00:00 /usr/openv/volmgr/bin/ltid
root 21566 1 0 17:24 ? 00:00:00 vmd
Shared Symantec Processes
-------------------------
root 6789 1 0 16:22 ? 00:00:00 /opt/VRTSpbx/bin/pbx_exchange
At this point we are now ready to go to sym2 and install NBU.
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 19/35
19
On sym2 change directory where the NBU install media is located and run the install script.
Do you wish to continue? [y,n] (y)
Participate in the NetBackup Product Improvement Program? [y,n] (y) n
The NetBackup and Media Manager software is built for use on
LINUX_RH_X86 hardware.
Do you want to install NetBackup and Media Manager files? [y,n] (y)
NetBackup and Media Manager are normally installed in /usr/openv.
Is it OK to install in /usr/openv? [y,n] (y)
Would you like to use "sym2.example.com" as the configured
NetBackup server name of this machine? [y,n] (y) n
Enter the name of this NetBackup server: nbumaster
Is nbumaster the master server? [y,n] (y) y
This is the list of existing NetBackup cluster groups:
nbu_group
Do you wish to join this existing NetBackup cluster group? [y,n] (y) y
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 20/35
20
Adding node sym2 to NetBackup Group nbu_group VCS
Enter the network device for NetBackup on sym2: eth0
You entered the following for Network configuration
The NIC to use with NetBackup on sym2 = eth0
The IP address for sym2 is : 10.10.2.12
Is this correct? [y/n]: y
Do you want to add any media servers now? [y,n] (n)
Enter the OpsCenter server (default: NONE):
NetBackup server installation complete.
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 21/35
21
We can confirm that sym2 has been added the Veritas Cluster Configuration for the nbu_group and is in
an offline state on sym2.
# /opt/VRTSvcs/bin/hastatus -sum
-- SYSTEM STATE
-- System State Frozen
A sym1 RUNNING 0
A sym2 RUNNING 0
-- GROUP STATE
-- Group System Probed AutoDisabled State
B ClusterService sym1 Y N ONLINE
B ClusterService sym2 Y N OFFLINE
B nbu_group sym1 Y N ONLINE
B nbu_group sym2 Y N OFFLINE
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 22/35
22
Reviewing and Understanding the Configuration
The /etc/VRTSvcs/conf/config/main.cf shows the configuration for Netbackup service
group.
group nbu_group (
SystemList = { sym1 = 0, sym2 = 1 }
AutoStartList = { sym1, sym2 }
)
DiskGroup nbu_dg (
DiskGroup = nbudg
)
IP nbu_ip (
Device @sym1 = eth0
Device @sym2 = eth0
Address = "10.10.2.100"
NetMask = "255.255.255.0"
)
Mount nbu_mount (
MountPoint = "/opt/VRTSnbu"
BlockDevice = "/dev/vx/dsk/nbudg/nbuvol"
FSType = vxfs
FsckOpt = "-y"
)
NIC nbu_nic (
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 23/35
23
Device @sym1 = eth0
Device @sym2 = eth0
NetworkHosts = { "10.10.2.11", "10.10.2.12" }
)
NetBackup nbu_server (
Critical = 0
ResourceOwner = unknown
ServerName = NBU_Server
ServerType = NBUMaster
MonScript = NONE
RSPFile = "/usr/openv/netbackup/bin/cluster/NBU_RSP"
GroupName = nbu_group
Volume nbu_vol (
DiskGroup = nbudg
Volume = nbuvol
)
nbu_ip requires nbu_nic
nbu_mnt_stu requires nbu_vol
nbu_mount requires nbu_vol
nbu_server requires nbu_ip
nbu_server requires nbu_mnt_stu
nbu_server requires nbu_mount
nbu_vol requires nbu_dg
We can see that Veritas Cluster Server has populated the main.cf with all the information provided
during the installation of Netbackup.
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 24/35
24
How the Netbackup Agent for Veritas Cluster Works
The NetBackup Agent for Veritas Cluster brings online, takes offline and monitors NetBackup processes.
Agent Functions:
Online: Starts NetBackup using
/usr/openv/netbackup/bin/cluster/util/online\
/usr/openv/netbackup/bin/cluster/NBU_RSP
The Online will start the processes listed in the START_PROCS from the NBU_RSP file
Offline: Stops NetBackup using
/usr/openv/netbackup/bin/cluster/util/offline\
/usr/openv/netbackup/bin/cluster/NBU_RSP
Monitor: probes the processes listed in the PROBE_PROCS from the file
/usr/openv/netbackup/bin/cluster/NBU_RSP
Also runs a /usr/openv/netbackup/bin/bpdbm –alive -local
The Offline will stop the processes listed in the START_PROCS from the NBU_RSP file
We need to understand what the "/usr/openv/netbackup/bin/cluster/NBU_RSP" is used
for. This file contains some of the following required to start NetBackup in a cluster.
Service Group Name
The Shared Storage
Nodes that are clustered for Netbackup
The Virtual name of the Netbackup Master Server
IP and Subnet
Cluster type
What Netbackup Process to Start
What Netbackup Process to Monitor
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 25/35
25
The format of the file is as follows:
#DO NOT DELETE OR EDIT THIS FILE!!!
NBU_GROUP=nbu_group
SHARED_DISK=/opt/VRTSnbu
NODES=sym1 sym2
VNAME=nbumaster
VIRTUAL_IP=10.10.2.100
SUBNET=255.255.255.0
CLUTYPE=VCS
START_PROCS=NB_dbsrv nbevtmgr nbemm nbrb ltid vmd bpcompatd nbjm nbpem
nbstserv nbrmms nbsl nbvault nbsvcmon bpdbm bprd bptm bpbrmds bpsched bpcd
bpversion bpjobd nbproxy vltcore acsd tl8cd odld tldcd tl4d tlmd tshd rsmd
tlhcd pbx_exchange nbkms nbaudit nbatd nbazd nbim
PRODUCT_CODE=NBU
DIR=netbackup mkdir
DIR=netbackup/db mv
DIR=var mkdir
DIR=var/global mv
DIR=volmgr/mkdir
DIR=volmgr/misc mkdir
DIR=volmgr/misc/robotic_db mv
DIR=kms mv
DIR=netbackup/vault mkdir
DIR=netbackup/vault/sessions mv
LINK=volmgr/misc/robotic_db
LINK=netbackup/db
LINK=netbackup/vault/sessions
LINK=var/global
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 26/35
26
PROBE_PROCS=nbevtmgr nbstserv vmd bprd bpdbm nbpem nbjm nbaudit nbsl nbrmms
nbemm nbrb NB_dbsrv
Troubleshooting
If NetBackup fails to start on a node in the cluster we can review the following logs file to determine
why.
The NetBackup Agent log is here:
/usr/openv/netbackup/bin/cluster/AGENT_DEBUG.log.
This file will show if NetBackup started successfully or not. An example of a successful online:
Mon Mar 24 21:29:05 2014 vcs/Offine.pl: Calling:
/usr/openv/netbackup/bin/cluster/util/offline/usr/openv/netbackup/bin/cluster/NBU_RSP
An example of a failed online:
Thu Apr 19 17:02:24 2012 vcs/Online.pl: Calling: /online/usr/openv/netbackup/bin/cluster/NBU_RSPThu Apr 19 17:02:24 2012 vcs/online.pl:/usr/openv/netbackup/bin/cluster/util/online/usr/openv/netbackup/bin/cluster/NBU_RSP exited with 99Thu Apr 19 17:02:24 2012 vcs/Online.pl: Calling:
/usr/openv/netbackup/bin/cluster/util/online
/usr/openv/netbackup/bin/cluster/NBU_RSP
The NetBackup Cluster log is here:
/usr/openv/netbackup/logs/cluster/log.xxx
as an example we can see from the logs why NetBackup did not start.
17:02:42.446 [14307] <4> NBClusterApp::createLinks: Creating link from
/usr/openv/volmgr/misc/robotic_db to /nbu/volmgr/misc/robotic_db 17:02:42.600
[14307] <4> NBClusterApp::createLinks: Local file stat: -1, Shared disk file
stat: -1 17:02:42.600 [14307] <4> NBClusterApp::createLinks: Couldn't create
link to shared disk. Application can not start.
From this example output we can see that it failed to start because it could not create a link to shared
disks.
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 27/35
27
How to Manually Start NetBackup Outside of Cluster Control
Is it important NOT to run “/usr/openv/netbackup/bin/goodies/netbackup start” to
start NetBackup manually outside of the cluster.
The reason is Netbackup expects to find certain files in /usr/openv/netbackup. When VCS starts
NetBackup it first creates symbolic links that are required and THEN starts NetBackup.
The /usr/openv/netbackup/logs/cluster/log.xxx shows exactly what the NetBackup
Cluster Server Agent does.
Step 1 - it sources the NBU_RSP file
18:45:54.603 [29616] <4> Online::main: Initializing NBCluster using
/usr/openv/netbackup/bin/cluster/NBU_RSP file
Step 2 - it then is creates the symbolic links here:
18:45:54.642 [29616] <4> NBClusterApp::createLinks: Creating link from
/usr/openv/volmgr/misc/robotic_db to /opt/VRTSnbu/volmgr/misc/robotic_db
18:45:54.642 [29616] <4> NBClusterApp::createLinks: Local file stat: -1,
Shared disk file stat: 0
18:45:54.642 [29616] <4> NBClusterApp::createLinks: Creating link from
/usr/openv/netbackup/db to /opt/VRTSnbu/netbackup/db
18:45:54.642 [29616] <4> NBClusterApp::createLinks: Local file stat: -1,
Shared disk file stat: 0
18:45:54.642 [29616] <4> NBClusterApp::createLinks: Creating link from
/usr/openv/netbackup/vault/sessions to /opt/VRTSnbu/netbackup/vault/sessions
18:45:54.643 [29616] <4> NBClusterApp::createLinks: Local file stat: -1,
Shared disk file stat: 0
18:45:54.643 [29616] <4> NBClusterApp::createLinks: Creating link from
/usr/openv/var/global to /opt/VRTSnbu/var/global
18:45:54.643 [29616] <4> NBClusterApp::createLinks: Local file stat: -1,Shared disk file stat: 0
Step 3 - after the symbolic links are created, the agent then starts NetBackup as follows:
18:45:54.643 [29616] <4> NBClusterApp::runCommand: Entering, with
command:/usr/openv/netbackup/bin/goodies/netbackup start
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 28/35
28
After NetBackup starts you can check the links:
# ls -l /usr/openv/volmgr/misc/robotic_db
lrwxrwxrwx 1 root root 35 Mar 25 18:52 /usr/openv/volmgr/misc/robotic_db ->
/opt/VRTSnbu/volmgr/misc/robotic_db
# ls -l /usr/openv/netbackup/vault/sessions
lrwxrwxrwx 1 root root 37 Mar 25 18:52 /usr/openv/netbackup/vault/sessions ->
/opt/VRTSnbu/netbackup/vault/sessions
# ls -l /usr/openv/var/global
lrwxrwxrwx 1 root root 23 Mar 25 18:52 /usr/openv/var/global ->
/opt/VRTSnbu/var/global
If you have to manually start NetBackup outside of Cluster Server you will need to create the above links
before you run “/usr/openv/netbackup/bin/goodies/netbackup start”
Likewise if you now need to stop NetBackup you need to remember to remove the links after NetBackup
has stopped.
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 29/35
29
This is what will happen if you don’t create the symbolic links before you start the NetBackup.
Firstly lets confirm that the service group is offline.
# hagrp -state nbu_group
#Group Attribute System Value
nbu_group State sym1 |OFFLINE|
nbu_group State sym2 |OFFLINE|
Lets now check and make sure that there are no NetBackup processes running:
# /usr/openv/netbackup/bin/bpps -x
NB Processes
------------
root 13472 1 0 15:57 ? 00:00:00
/usr/openv/netbackup/bin/vnetd -standalone
root 13475 1 0 15:57 ? 00:00:00 /usr/openv/netbackup/bin/bpcd
- standalone
MM Processes
------------
Shared Symantec Processes
-----------
root 6759 1 0 15:50 ? 00:00:00 /opt/VRTSpbx/bin/pbx_exchange
Using the JAVA GUI or command line, online all resources EXPECT for the nbu_server
From the /var/VRTSvcs/log/engine_A.log we can confirm that our resources are up.
2014/04/07 16:06:29 VCS INFO V-16-1-10298 Resource nbu_dg (Owner:
Unspecified, Group: nbu_group) is online on sym2 (VCS initiated)
2014/04/07 16:06:45 VCS INFO V-16-1-10298 Resource nbu_vol (Owner:
Unspecified, Group: nbu_group) is online on sym2 (VCS initiated)
2014/04/07 16:07:05 VCS INFO V-16-1-10298 Resource nbu_ip (Owner:
Unspecified, Group: nbu_group) is online on sym2 (VCS initiated)
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 30/35
30
2014/04/07 16:07:05 VCS NOTICE V-16-1-10301 Initiating Online of Resource
nbu_mount (Owner: Unspecified, Group: nbu_group) on System sym2
2014/04/07 16:07:07 VCS INFO V-16-1-10298 Resource nbu_mount (Owner:
Unspecified, Group: nbu_group) is online on sym2 (VCS initiated)
Confirm the symbolic links are not there.
# ls -l /usr/openv/volmgr/misc/robotic_db
ls: /usr/openv/volmgr/misc/robotic_db: No such file or directory
# ls -l /usr/openv/netbackup/vault/sessions
ls: /usr/openv/netbackup/vault/sessions: No such file or directory
# ls -l /usr/openv/var/global
ls: /usr/openv/var/global: No such file or directory
Lets now start Netbackup without creating the symbolic links.
# /usr/openv/netbackup/bin/goodies/netbackup start
NetBackup Authentication daemon started.
NetBackup network daemon started.
NetBackup client daemon started.
NetBackup SAN Client Fibre Transport daemon started.
NetBackup will not run without /usr/openv/db/bin/NB_dbsrv running
NetBackup Authorization daemon started.
NetBackup Event Manager started.
NetBackup Audit Manager started.
NetBackup Deduplication Manager started.
NetBackup Deduplication Engine started.
NetBackup Enterprise Media Manager started.
NetBackup Resource Broker started.
Rebuilding device nodes.
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 31/35
31
Media Manager daemons started.
NetBackup request daemon started.
NetBackup compatibility daemon started.
NetBackup Job Manager started.
NetBackup Policy Execution Manager started.
NetBackup Storage Lifecycle Manager started.
NetBackup Indexing Manager started.
NetBackup Remote Monitoring Management System started.
NetBackup Key Management daemon started.
NetBackup Service Layer started.
NetBackup Agent Request Server started.
NetBackup Bare Metal Restore daemon started.
NetBackup Vault daemon started.
NetBackup CloudStore Service Container started.
VCS
NetBackup Service Monitor started.
NetBackup Bare Metal Restore Boot Server daemon started.
Let us once again confirm that the links are not there:
# ls -l /usr/openv/volmgr/misc/robotic_db
ls: /usr/openv/volmgr/misc/robotic_db: No such file or directory
# ls -l /usr/openv/netbackup/vault/sessions
ls: /usr/openv/netbackup/vault/sessions: No such file or directory
# ls -l /usr/openv/var/global
ls: /usr/openv/var/global: No such file or directory
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 32/35
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 33/35
33
We can see that several processes are missing. For example theNB_dbsrv is missing. The NB_dbsrv
is the NetBackup Database. Even though the NetBackup Database NB_dbsrv is not running, the
NetBackup Agent reports it as online. We can confirm by running:
# /usr/openv/netbackup/bin/cluster/util/monitor
/usr/openv/netbackup/bin/cluster/NBU_RSP
Some Processes are DOWN while others are UP
Following Process are found DOWN: nbstserv bprd bpdbm nbpem nbaudit nbsl
nbemm nbrb NB_dbsrv
Following Process are found UP: nbevtmgr vmd nbjm nbrmms
# echo $?
110
The value 110 means the nbu_server is ONLINE, even though the processesnbstserv bprd bpdbm
nbpem nbaudit nbsl nbemm nbrb NB_dbsrv are not running.
Since the monitor is reporting 110 the cluster thinks the service group is up.
# hagrp -state nbu_group
#Group Attribute System Value
nbu_group State sym1 |OFFLINE|
nbu_group State sym2 |ONLINE|
The NetBackup agent should be reporting the value 100, which is an OFFLINE. Etracke348052 has
been created to address this issue.
To resolve this you can do the following:
# /usr/openv/netbackup/bin/bp.kill_all
# /usr/openv/netbackup/bin/cluster/util/online\
/usr/openv/netbackup/bin/cluster/NBU_RSP
Now we can see that symbolic links have been created.
# ls -l /usr/openv/volmgr/misc/robotic_db
lrwxrwxrwx 1 root root 35 Apr 7 16:58 /usr/openv/volmgr/misc/robotic_db ->
/opt/VRTSnbu/volmgr/misc/robotic_db
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 34/35
34
# ls -l /usr/openv/netbackup/vault/sessions
lrwxrwxrwx 1 root root 37 Apr 7 16:58 /usr/openv/netbackup/vault/sessions ->
/opt/VRTSnbu/netbackup/vault/sessions
# ls -l /usr/openv/var/global
lrwxrwxrwx 1 root root 23 Apr 7 16:58 /usr/openv/var/global ->
/opt/VRTSnbu/var/global
# ls -l /usr/openv/netbackup/db
lrwxrwxrwx 1 root root 25 Apr 7 16:58 /usr/openv/netbackup/db ->
/opt/VRTSnbu/netbackup/db
8/9/2019 How to Cluster a Netbackup 7
http://slidepdf.com/reader/full/how-to-cluster-a-netbackup-7 35/35