The functional structure of NetBackup and Media Manager. Objectives: The aim of this module is to describe, in detail, the architecture and functionality of NetBackup and Media Manager, and how these two software products interact with each other: • What is a master server, a media server and a client ? • What is the difference between a daemon and a process ? • What are the NetBackup daemons and processes, and what do they do ? • What are the Media Manager daemons and processes, and what do they do ? • What is the function and location of the NetBackup and Media Manager catalogs ? • What are the relationships between the catalogs ?
Transcript
The functional structure of NetBackup and Media Manager.
Objectives:
The aim of this module is to describe, in detail, the architecture and functionality of NetBackup and Media Manager, and how these two software products interact with each other:
• What is a master server, a media server and a client ?• What is the difference between a daemon and a process ?• What are the NetBackup daemons and processes, and what do they do ? • What are the Media Manager daemons and processes, and
what do they do ?• What is the function and location of the NetBackup and Media Manager catalogs ?• What are the relationships between the catalogs ?• What is the bp.conf file
A detailed and thorough understanding of the components of
NetBackup and Media Manager is vital if you are to:
• Understand the design of existing environments.• Architect or modify your own environment.• Troubleshoot failures of all kinds.
What is a master server, a media server and a client?
NetBackup consists of three primary components, the master server, media server(s) and client(s).
• These machines are virtual machines. • A master server, media server and client might be
physically different machines.
However, it is possible that:• The master server is also the media server. • The media server is also the client.
Always remember that whether you are installing a master
server or a media server:
• The actual software that you are installing is identical. • It is each server's NetBackup configuration file that 'SERVER
=' entries identify whether a system is a master server or a media server.
• If you are installing a server of either type, the client software is automatically installed as well (the servers have to back themselves up!).
If you are installing a client:
• A client will have only the NetBackup client software installed.
Let us now consider, in detail, what a master server, media
server and client actually are.
Master server
• The master server is the overall controlling server for the entire NetBackup environment.
• It is the location of the scheduler, which determines when automatic backup jobs are run.
• The master server is also the home of most of the NetBackup and Media Manager catalogs.
• The most significant of these are the NetBackup image catalog, and Media Manager volume catalog.
Media server
• The media server is a NetBackup server that has some sort of storage resource physically attached to it.
• If a master server has attached tape drives which are configured and usable by NetBackup, then it is referred to as a master/media server.
• It is only one physical machine, but from a NetBackup perspective it should be thought of as two virtual machines which happen to share the same host name.
Media server (cont)
• Traditionally, media servers would be placed at strategic points around the network in order to minimise network bottlenecks when backing up multiple machines across IP.
• In more modern implementations, and particularly in Storage Area Networks (SANs), it is often the case that media servers are also clients, and back up their own data to local tape drives.
• These local drives are normally dynamically shared between two or more media servers using Shared Storage Option (SSO).
Client
• A client is defined as any machine whose files are backed up or restored by NetBackup.
What is the difference between a daemon and a process?
It is important to understand the difference between a NetBackup or Media Manager daemon, and a process.
Daemons
• Daemons are started by the NetBackup startup script, or by other daemons, and run in the background all time, irrespective of whether there are jobs running or not.
• Daemons may spawn copies of themselves. Examples are bprd, the NetBackup master server request daemon, and vmd, the Media Manager volume catalog daemon.
Processes
• Processes are started on demand, usually by a system call or a socket handover.
• Examples are bpcd, the NetBackup connection daemon, which will be seen on all three types of NetBackup machines, and bptm, the NetBackup tape manager.
What are the NetBackup daemons and processes and
what do they do?
The daemons and processes that run on the NetBackup master server, media servers and clients differ, depending upon which type of machine it is. We will now look at the different binaries that run on each type of machine.
Master server
bprd - this is the request daemon, the overall controlling daemon on the master server.
• It is started by the NetBackup startup script, and runs in the background all time.
• Its purpose is to listen for incoming requests for user directed backups, archives or restores (in other words, any user directed operation).
• As it starts, it reads the contents of the configuration catalog, caching this data in shared memory.
• Depending upon the wakeup interval that is set (default is 10 minutes), bprd will call bpsched, the scheduler process.
bpsched - this is the scheduler process.
• It is started at regular intervals, determined by the wakeup interval that is set in the global configuration.
• At startup, it makes a system call to bpdbm to read through the class catalog which contains the policy definitions and schedule settings.
• Through bpdbm, the scheduler identifies any jobs that need to be run. If there are no jobs to be run, bpsched exits.
• If a job is found to be run, bpdbm compiles a work list which is then passed back to bpsched.
• If bpsched receives a work list, it connects to the media server and passes the work list on for execution.
• When the backup is running, bpsched receives the file list metadata from the media server and passes this back to bpdbm which uses this information to compile the appropriate image catalog entry.
bpdbm - this is the master server database manager. • It maintains all of the NetBackup catalogs on the master
server, with the exception of the jobs catalog.• It takes instructions from bpsched to read the class catalog
and identify jobs to be run.• As part of its maintenance function, it will also
intermittently search for images that have expired and need to be deleted from the image catalog.
• It manages the flow of data to and from bpjobd, the jobs catalog daemon.
bpjobd - this is the job catalog daemon.
• Works under the control of bpdbm.• Manages the flow of data into and out of the job catalog. • The job catalog daemon feeds information to the activity
monitor in the GUI.
Master Server
Media Server Client
Media server
inetd - this daemon is not part of the NetBackup installation, but is actually a UNIX operating system daemon that is used by NetBackup.
• It listens for incoming IP connections for specific processes on specific ports.
• As part of the NetBackup installation, an entry is inserted in the inetd configuration file, /etc/inetd.conf. This entry instructs inetd to listen for incoming calls on IP port 13782:
• On Windows systems, there is no inetd daemon. When installing a Windows server or client, an extra daemon is added as part of the installation. This is called bpinetd and performs the same function as inetd on UNIX.
• When a connection request is received by the media server from bpsched on the master server, it is captured by inetd which calls bpcd, the connection daemon.
bpcd - the connection daemon.
• This is actually a process rather than a daemon, since it is only invoked on demand.
• It is called by inetd as a result of all incoming connection requests from the master server, and authenticates the name and IP address of the master server against the name that is configured as the first 'SERVER =' entry in bp.conf.
• If the authentication fails, the connection is refused and the job will fail with return codes 54, 57 or 58.
• If the authentication succeeds, bpcd does a socket handover to bpbrm, passing over the work list that it has received from the master server, and immediately exits.
bpbrm - the backup and restore manager.
• This process has overall control of the media server.• Once it is called by bpcd, it receives and processes the
work list, determining which storage unit it is required to use.
• If a Media Manager storage unit is required, bpbrm will invoke bptm, the tape manager.
• If a disk storage unit is required, then it will invoke bpdm, the disk manager.
• bpbrm will then attempt to connect to the client in order to start the backup.
• Once the backup has been initiated, bpbrm receives the image catalog metadata from bpbkar on the client and transmits it back to bpdbm on the master server. This transmission takes place in intermittent bursts.
bptm - the tape manager.
• The most important NetBackup process, from the perspectives of troubleshooting and performance tuning.
• Manages the size and number of shared memory buffers that are used when writing data to tape.
• The instance of bptm that is initiated by bpbrm is referred to as the 'parent' bptm.
• The parent reads the contents of the local media catalog on the media server, manages the writing of data to tape, and acts is the interface between NetBackup and Media Manager.
bptm - the tape manager.
• If the backup is being performed over IP, then the parent bptm will spawn child processes, whose purpose is to read the data that is being transmitted to the media server from the client. The parent bptm writes the data to the tape. For this reason, in the case of a backup, the parent bptm is sometimes called the 'writer' and the child bptm is sometimes called the 'reader'.
• If the media server is backing up itself (i.e. it is also the client), then no bptm reader is created. Instead, bpbkar (the client backup and archive process) writes the client data directly into bptm's shared memory buffers.
Master Server
Media Server Client
bpdm - the disk manager.
• This process is the equivalent of bptm, but is invoked when the backup is being sent to a disk storage unit rather than the Media Manager storage unit.
• It does not interface with Media Manager, since for disk based backups Media Manager is not involved.
• Like bptm, there will be parent processes that are initiated by bpbrm which manage the writing of data to disk, and, if backing up over IP, child bpdm processes which read the client data from the network.
Master Server
Media Server Client
Client
inetd.conf - the UNIX networking daemon.
• Functions exactly as it does on the media server.
bpcd - the connection daemon.
• Functions much like bpcd on the media server, but where bpcd on the media server authenticates the name and IP of the master server only; on the client it authenticates the master server and the media server.
bpbkar - the backup and archive daemon (BPBKAR32.EXE on
Windows).
• Reads the client files during a backup or archive.• Transmits the files to bptm or bpdm to be written to tape
or disk.• If the backup is taking place over IP, then bpbkar sends
data over the network to the bptm reader.• If the backup is local, bpbkar writes the files directly into
the shared memory buffers that are owned by the bptm (or bpdm) writer.
• Generates a stream of metadata which is sent to bpbrm, then back to bpdbm on the master server, to update the image catalog.
tar - A version of tar that is installed with the NetBackup client binaries (tar32.exe on Windows).
• A modified version of GNUtar, which is used by NetBackup when performing restores.
• Behaves in the opposite way to bpbkar (i.e. it writes the client files during a restore)
Master Server
Media Server Client
Metadata
What are the Media Manager daemons and processes,
and what do they do?
• Media Manager daemons and processes run only on the master servers and media servers, there is no Media Manager activity on clients.
• There are three Media Manager daemons that will be found running automatically in the background, no matter what type of Media Manager storage devices are configured. These daemons manage the assignment and scanning of devices and, in the case of vmd, also manage the volume and pool catalogs on the master server.
vmd - the Media Manager volume daemon.
• One instance of vmd runs on the master server and on each media server.
• The instance of vmd on the master server manages the volume catalog and pool catalog.
• From a Media Manager point of view, the master server is referred to as the volume database host because it is the location of the volume catalog.
• On the media server, vmd handles tape location requests that are passed to vmd on the master server.
vmd - the Media Manager volume daemon.
• When vmd receives information on the requested assigned volume from ltid, it searches the volume catalog and returns details of that tape's physical location (i.e. which slot in the robot it is located in) to ltid.
• Tracks the number of mounts and the last mount time for each tape.
• Starting ltid will automatically start vmd. However, stopping ltid will not stop vmd. This is because it may be servicing requests from another NetBackup server and because it is used for remote device configurations and management.
ltid - the library and tape interface daemon, sometimes called
the Media Manager device daemon.
• ltid runs on media servers that have storage devices configured within Media Manager.
• On startup, ltid will read the contents of the device catalog to identify what storage devices are configured.
• Starting ltid will also start vmd and avrd, and the appropriate daemons for any robot defined drives.
ltid - the library and tape interface daemon, sometimes called
the Media Manager device daemon.
• If there are no storage devices configured (e.g. if the server is a standalone master server) then ltid will immediately shut down since it has no devices to manage
• ltid handles media mount requests and will translate those requests into commands which control the robotic arms and tape drives
• ltid is the daemon whose activities can be observed through the Device Monitor interface and the command line with vmoprcmd
avrd - the automatic volume recognition daemon.
• avrd is started by Itid and stops when Itid stops.• Continually polls each of the configured tape drives to
determine if they have recently mounted a tape.• Every tape in the NetBackup environment has a unique
label, written to it by NetBackup. This label is known as the recorded volume serial number or rvsn.
• avrd reads this label to ensure that it matches the barcode label of the tape that has been mounted. This ensures that data will been written to the correct tape
The remaining Media Manager daemons are the robotic daemons and robotic control daemons (if the latter are applicable) for each robot type. Which of these specific daemons are started depends upon the type of library and tape drives that you are using.
tlxd - tape library robotic daemon.
• This daemon manages robotic drive requests.• The value of x is determined by the method used to
communicate with the library and is named after the type of drive that the library most commonly uses.
• tlxd receives requests to mount and unmount tapes, and passes these requests onto the robotic control daemon tlxcd.
tlxd - tape library robotic daemon.
• Three examples of this type of daemon and the hardware that is appropriate to them is as follows:o tldd - library controlled by SCSI commands (with DLT
drives)o tlhd - IBM library controlled by dedicated control server
over IP (with half-inch drives)o tl8d - library controlled by SCSI commands (with 8mm
drives)o tlmd - ADIC library controlled by dedicated control
server over IP (with multimedia drives)• When Media Manager is running on a media server, one
instance of tlxd will run in the background. This will spawn copies of itself to perform actions as directed by ltid.
tlxcd - tape library robotic control daemon.
• Receives and handles requests to mount and unmount tapes that are passed to it from tlxd.
• Three examples of this type of daemon are:o tldcdo tlhcdo tl8cd
Master Server
Media Server
Client
These are the most common Media Manager robotic and
robotic control daemons. Here are some of the others that we may come across:
• acsd - specific to StorageTek ACS/LS libraries.• lmfd and lmfcd - Library Management Facility.• odld - Optical Disk Library.• tl4d - tape library 4mm daemon.• ts8d and tsdd - tape stacker 8mm daemon and tape
stacker DLT daemon respectively.
Master Server
Media Server Client
Metadata
How does it all work together?
What is the function and location of the NetBackup and
Media Manager catalogs?
The NetBackup catalogs on both the master server and media server are located here:
/usr/openv/netbackup/db
The Media Manager catalogs on the master server and media server are located here:
/usr/openv/volmgr/database
The NetBackup catalogs on the master server are as follows:
image, config, class, jobs, error and vault.
Each media server has a NetBackup catalog called the media catalog.
The Media Manager catalogs on the master server are the volume, pool and global device database catalogs. Each media device catalog.
First, let’s look at the NetBackup catalogs:
Image catalog
• The image catalog is located at /usr/openv/netbackup/db/images on the master server. This is a directory, and below it are subdirectories named after the individual client names. Note that they are not grouped together by policy name.
• The image catalog contains data on every file that has been backed up and holds this information for the duration of the retention period.
• Each image entry comprises two files, a header file and a ‘.f’ file.
• The header file contains metadata about how and when the image was made and has a name in the format ‘POLICY_TIMESTAMP_SCHEDULE’. This file is quite small and is always in ASCII format.
Image catalog
• The ‘.f’ file holds the data about the files that make up the image. By default it is a binary file. The format of the name of this file is ‘POLICY_TIMESTAMP_SCHEDULE.f’
• If the file has been compressed, then it will have a name of the form ‘POLICY_TIMESTAMP_SCHEDULE.f.z’
• If fragmentation has been used, then you will see files with a name like ‘POLICY_TIMESTAMP_SCHEDULE_F1’ and ‘POLICY_TIMESTAMP_SCHEDULE.f_F1’
• If there are multiple copies if the same image, then you will see files with a name like ‘POLICY_TIMESTAMP_SCHEDULE_C1’ and ‘POLICY_TIMESTAMP_SCHEDULE.f_C1’
Image catalog
For versions of NetBackup prior to 4.5, the image catalog ‘.f’ files were in ASCII format which:
• Could be read using a text editor.• Could result in a very large image catalog that was difficult
to manage and slow to access.
From version 4.5 onwards, NetBackup has used a binary format for the ‘.f’ files.
• The software is still capable of reading entries in the old ASCII format.
• It is highly recommended that the default binary format is used.
• Individual image catalog entries can be converted from binary to ASCII and back again using the cat_convert utility.
Image catalog
The image catalog uses an indexing system to help speed up the process of browsing for restores.
During the installation you will be asked if you wish to index the catalog, and which index level you would like to use.
Indexing is recommended and the default index level is 9.
Class catalog
• Located in /usr/openv/netbackup/db/class.• Contains information about the policy configurations and
schedules.
Config catalog
• Located in /usr/openv/netbackup/db/config.• Holds the global configuration parameters, storage unit
configuration and catalog backup details.• This catalog is read by bprd when it starts, and its contents
are cached in shared memory.
Client catalog
• Located in /usr/openv/netbackup/db/client.• Hold client specific information. This instructs particular
clients to behave differently from the standard client-type rule sets held in the class catalog.
• Usually empty, entries for particular clients can be added and modified using the bpclient command.
Altnames catalog
• Located at /usr/openv/netbackup/db/altnames.• Files in this catalog are manually created by the user and
hold alternative client names and rules for performing alternative client restores.
Error catalog
• Located at /usr/openv/netbackup/db/error.• Holds data collected from all NetBackup processes.• This catalog can be interrogated at the command line with the
bperror command, and can also be accessed from the GUI with the Problems and All Log Entries reports.
Jobs catalog
• Located at /usr/openv/netbackup/db/jobs.• This catalog is the source of real time data for the activity
monitor. • It is managed by the bpjobd daemon, under the control of
bpdbm.
Media catalog
• There is a media catalog on every media server.• Located at /usr/openv/netbackup/db/media.• Managed by bptm on each media server.• Hold records of every tape assigned to the media server
and the NetBackup media states of the tapes (e.g. ACTIVE, FROZEN or SUSPENDED).
Now let’s look at the Media Manager catalogs:
Volume catalog
The volume catalog exists only on the volume database host, which is almost always the master server. It holds data on all of the media in the NetBackup environment, including their physical location, and records media-related information such as volume pools and bar code rules.
It is located at /usr/openv/volmgr/database and is made up of the following components, each of which can be thought of as a catalog in its own right:
volDB - the volume database.
• Holds data on the physical location of tapes (i.e. robot number, slot number, and therefore the volume group that they are in).
• This is the catalog that is updated by performing a robotic inventory and update.
poolDB - the pool database.
• Holds information on the volume pools, such as their names, numbers and any host residency.
• Note that the definition of the SCRATCH pool is not held here, but in the Media Manager configuration file, /usr/openv/volmgr/vm.conf.
ruleDB - the rule database.
• Holds the barcode rule information.
globDB - the global device database.
• Holds information on all of the devices across all of the media servers.
• Used by the device configuration wizards.
Device catalog
• There is a device catalog on each media server.• It is located in /usr/openv/volmgr/database.• Comprised of two specific text files.• ltidevs contains data on the drives, such as type (TLD etc),
the operating system device file (e.g. /dev/rmt/0cbn), location if they are robotic, and status (UP or DOWN etc).
• robotic_def holds information on robotic devices, such as the type and the location of the appropriate SCSI pass-thru driver.
What are the relationships between the catalogs?
• The image catalog records details of every file that has been backed up, grouped together by image number.
• The image number is a unique reference that is given to every backup that NetBackup makes.
• The image catalog also records the media server that actually performed the backup. On the media server the media catalog records the media ID that each image was backed up on.
• Finally, the volume catalog on the master server records the physical location of all removable media.
What are the relationships between the catalogs?
• This diagram illustrates the fact that the NetBackup/Media Manager catalogs are essentially a distributed relational database.
• Although they are in different locations on different servers, and perform different functions, they must be protected as if they were a single entity.
• It is for this reason that a totally separate backup and recovery utility, which operates using a separate scheduler and outside of normal NetBackup functionality, is provided.
What are the relationships between the catalogs?
Master Server
NBU /file /media serverimage #
Media Server
NBU image # media ID
locationrobot?slot?
MM
Master Server
media ID
What is the bp.conf file?
bp.conf is the NetBackup configuration file. There is a bp.conf file on every UNIX NetBackup machine; master server, media server and client.
It is located at /usr/openv/netbackup/bp.conf.
On every machine, the first entry in bp.conf must be:SERVER = <master_server_name>
This is followed by ‘SERVER = <media_server_name> entries for each of the media servers.
Each machine must have its own client name represented in an entry like this:
CLIENT_NAME = <local_machine_name>
Other entries to perform different functions can be added to bp.conf, such as:
BPTM_QUERY_TIMEOUT = 2100ALLOW_MEDIA_OVERWRITE = TAR
and many others.
For any Windows NetBackup machines, the functionality provided
by the bp.conf file is located in the system registry at:
These entries are usually set at installation time or through the ‘Host Properties’ window in the Administrator GUI. If neither of these methods are possible, the regedt32 utility can be used to change them manually – this is sometimes required to change the list of SERVER entries if a client needs to be moved to a new master server.
NOTE: the manual manipulation of the registry is not officially supported by either Veritas or Microsoft.
Here is an example of a master server bp.conf:
And a media server:
And a client:
Summary
You should now have the skills necessary to:
• Understand the design of existing environments.• Architect or modify your own environment.• Troubleshoot failures of all kinds.
