2
ATO is a utility providing tape-out capabilities from an Avamar system. It
relies on a re-hydrated approach recovering data to a disk staging area.
The staging area is subsequently backed up to tape using a standard tape
Backup application.
The tape out process involves three steps;
1. client/backup selection2. confirmation view to see which backups were selected3. staging selected backup data and perform a tape backup
ATO Batch Manager fully automates these three phases.
ATOAvamar Tape Out
3
ATO – Data Flow
Client Grp-1
Drv-Z
Data-Flow
Data-Flow Client Grp-2
Client Grp-3
Tape Policy - 7 Year Retention
Tape Policy - 1 Year Retention
Tape Policy - 3 Year Retention
Drv-X
Drv-X
Drv-X
Drv-Z
Drv-Z
ATO Environment File Configuration CriteriaATO Client Configuration File Criteria
Drv-Y Drv-Y
Drv-Y
Data-Flow
Data-Flow
Data-Flow
Data-Flow
Data-Flow
Source or TargetSingle or Multi Node Systems Networker De-Dupe Nodes
4
ATO - Features
Single Point of Management & Control
• MENU Driven User Interface
• Multiple Concurrent Tape Out Sessions (1/Staging-Server)
• Failure Checkpoint Recovery
• Non-Inc & Incremental Tape Out
• Staging Recoveries Visible in Activity Monitor
• File Level Recovery from Tape to Original or Alternate Client
• Comprehensive Backup Selection Filters
• Structured Event and Batch Logs
• Audit Mechanism Tracks Critical Path TO Components
• Automated Generation of Required Tape Script
• Email Notifcations
5
ATO - Support Matrix
• Staging Platforms Windows, Red Hat, SCO, FreeBSD, Solaris, AIX, HPUX
• Avamar Plug-InLotus Notes, SQL, Exchange 2003, NDMP, VMimage, (Oracle)
• Tape ApplicationsNetworker, NetBackup, Backup Exec, Arcserv, HP Data Protector, TSM, Commvault
• Avamar Node TypesSingle & GRID, Source or Target, AVE, Networker De-Dup
6
User Interface
• Fully Interactive
• All Functionality Available
• Color Coded Displays
• Access to Logs
• Online Help
• Automatic Upgrade Process
ATO Rocks!
7
3-Step Operational Cycle
8
Selection Paramaters
Client Selection– By client group –gid group-id– By individual client within a group –gid group-id –client client-id– Client must be in a group
• Plug-In Types FS (default) SQL –sql Exchange –exchdb or -exchvss NDMP –ndmp Oracle –oracle Sharepoint –shpt Lotus Notes –lotus
Backup Reduction Filters– First –first last –last or –first_F or –last_F– Day of the week –week sun or day instance of the month –week sun_1– Avamar policy group name –gname policy-group-name– Retention tag type –rtype daily | weekly | monthly | yearly– Backup-ID number –buid “#’s– Backup Type –but mod | cod | nah ... “
9
Selection Date Range
• Default Search Range– 1st day of current month to your current date– Expand default range: -nday +# expand start date going back in time the specified #
of days
Relative Search Range– By Month: -rdate # where # is a relative month count going back in time from your
current month. I.E. –rdate 1 will go back one month in time– By Day: -rday # where # is a relative date of the month– By Day Range: -nday # where # is the number of days back in time from your current
date
• Fixed Date Range– Start date: -sdate yyyy-mm-dd – End date: -edate yyyy-mm-dd– Refer to Flds-2 & 3 in client configuration file, used for a more permanent override– Day Count: -nday # adjust # to number of days from your current date
10
Sample Select Process
–last filter used limiting selected backups to most recent only
clients scanned are those defined in group linux only
final selected backup count is 1 with 3 being excluded
used default date range
2011-07-01 to 2011-07-05
used ATO environment file 1
Note: By default the selection process acts on scheduled backups only– refer to –butype option to select non-scheduled backups
11
Sample View Process
Selected Backup View– backup metrics per client– total combined backup size– combined size allocated per staging FS or folder location– To View: ato -v or ato –view
12
Tape-Out Process
A Tape-Out Session Initiated with –tapeout Includes 2 Steps– Staging or redirected recovery of the selected backups to a staging location– Tape-Backup refers to initiating the tape backup automaticaly of the staged data
Step-1 - Staging– destination determined by Environment & Client configuration files– concurrent staging supported only with multiple staging servers available– perform stage only using –s option (no tape backup will be initiated)– Incremental using –inc provides significant performance gains– Inclusion and exclusion filters for folders or files
–data “C:/Program Files | My Documents” –xdata “C:/Program Files | My Documents”
Step-2 - Tape Backup– Perfromed by default or optionally using –t (tapeonly no stgaing will be performed)– appropriate tape script generated and initiated automatically
user defined scripts can be specified– tape backup policy referenced is assumed to exist within tape application– tape incremental backups can be leveraged only when incremental staging is used
13
Sample Tape-Out Process
14
ATO Environment File
15
Used to define physical infastructure of Tape Out environment– x1 Windows & x1 UNIX staging server and x1 tape backup server name per
environment file– Tape backup application – Path to binaries of Avamar & Tape backup agents– Event log, client configuration & temp file locations, pseudo client– Various control variables - key variables established automatically
Max of 20 environment files supported– each environment functions independently– ATO sessions can use any defined environment file– lock mechanism to prevent concurrent use of a given environment
Environment Files
16
Environment File Contents
Site specific parameters will require modification
– tape backup server parameters– Staging server paramaters– email notification address list
Operational parameters– adjust as required– refer to documentation for details
ATO Control Files– Caution: modification of control file
names not recommended– TMP_PATH, CFG & LOG files, Pseudo
Client name etc.
17
Establishing & Accessing Environment Files
Establishing Environment Files– ato [#] –env where # represents the desired environment number, default of 1 is assumed
creates a environment config file /usr/local/avamar/etc/atoenv.cfg[#] establishes temp file location /tmp/atocfg[#] establishes event log file /usr/local/avamar/var/atoevent.log[#] references client configuration file /usr/local/avamar/etc/atoclient.cfg creates the Avamar pseudo client /clients/tapeoutato[#] establish various control variables with default values file names remain consistent with the exception of an appended numeric value I.E. /usr/local/avamar/etc/atoenv,cfg2, /tmp/atocfg2, /usr/local/avamar/var/atoevent.log2 etc..
Accessing Environment Files – Display ato [#] -env - displays environment file contents– Update: ato [#] -env update - opens a vi session using appropriate file name– Verify: ato [#] –env parse - verifies file contents and report on any anomalies
– Note: environment # when required must be the first argument specified to ATO
18
ATO Client Configuration File
19
Client Configuration File
CSV file with 18 fields per client entry– client names and how they are grouped– destination staging server type UNIX or Windows– destination staging FS or folder, can vary between clients within a group– References the tape policy name to use– Determines whether to use an auto generated or user defined tape script– various backup filters applied on a per client basis
Enables Simplified Tape-Out Operations– client groups enable more precise selection and allows segmented work flow
better to define several smaller groups as opposed to one large one
– clients from different domains are permitted
20
Client Configuration File View
Establishing Client Configuration– CSV formatted file– base file established automatically– a single client configuration file is shared between all environment #’s– update interactively or manually– automatic Avamar domain and client discovery process ensures accuracy– Primarily a one time task at install time with infrequent updates afterwards
21
Client Configuration File Contents
Viewing Configuration File Contents– To Display: ato –cfg (all groups) or ato –cfg <grp-id> (specific group only)– Content Verification: ato [#] –cfg parse – perform checks of file contents– Color Codes: Blue=predefined sample lines Green=user comments Cyan=important field highlight
Red=field separatorConfiguration File Display
22
Client Configuration File Usage
Key ATO Process’s Rely on This File– All tape out clients must first be defined in this file
Client Groups– group names cannot contain spaces or special characters– a client can be defined once per group but may exist in any number of groups– all clients in a group must share a common tape policy name & staging server type
An Organized Group Naming Convention is Beneficial – I.E. A marketing dept has 100 clients where 25 clients is the desired workload per session. Group
names might be mkt1, mkt2, mkt3, mkt4 maintaining a logical connection between them while facilitating easy work load segmentation and automation from CRON or 3 rd party Scheduler using ATO’s Batch Manager described later
Several Options Available on Both Command Line & in Client File– If specified on command line will pertain to all clients in the group– If client requirments vary within a given group, define these in the client file– date ranges, incdel include/exclude folders, Avamar policy group name, retention tag and
destination path
23
Client Configuration File Update-1
Manual Update: ato –cfg update Update Method-1 (bulk – useful at install time only)
– Automated Update: ato –cfg add or ato –cfg add_v (initiate discovery & update process)– 1st blue highlighted line contains a generic potential line entry– DOMAINX & CLIENTX are keywords replaced automatically, do not modify these– green highlighted line shows a user modified line with fields Fld-1=training, Fld-10=Drive T being changed– first non configured client displayed was optionally rejected– second non configured client displayed and optionally accepted & added to configuration file– subsequent non configured clients are displayed and added or rejected as required– existing client entries requiring modification can be edited manually or using interactive update (Method-2)
►
►
►
►
►
Selection Phase
Method-1 Update Session
24
Configuration Manager Domain & Group View
Update Method-2 (interactive & preferred method)– Via Interactive Update: ato –cfg manager– Clients viewed by Avamar Domain or ATO Client Group– Drill down into a Avamar domain or ATO group provides options to
add, delete, disable, enable, modify & view individual client entries
Selection Phase
Method-1 Update Session
25
Configuration Manager Cont...
Interactive Method-2 cont....– Client status view realtive to ATO showing all available clients – Select client by number to add, disable, enable, modify, remove & view individual
client entries
26
Client Enable/Disable
Interactive Method-2 cont..– Sample process to disable/enable a configured client to ATO– Client status display, Blue=Not configured, Green=Enabled, Red=Disabled to ATO
enter “d” to disable a client
enter the client-# to disable
confirm the displayed line is okay to disable
client#2 disabled confirmed by its color change
27
Client Entry Update-2
Method-2 cont..– Sample process to modify an existing client entry– Process to add a new client (not-shown) is similar
enter “m” to modify a client entry
enter the client-# to modify
verification screen can be looped through as often as necessary modifying field contents as required. When ready to accept the change enter “c” to commit the change
enter the group name for client entry to be modified
28
Locate & Track Data on Tape System
All Staged Data Belongs to the Staging Server(s) Clients Used– tape backup effectively owned by staging server client
Data Staged to Structured Directory– <user-defined>/<ATO-defined>/Orig-Client-Name/<client-data>
Tape Solution Reports Relative to Staging Client Name Only
Combine Tape System Reports with ATO Audit Report– tape media involved– relative folder location where to find TO client in staging client backup– staging and tape backup dates– staging server name used– tape backup server used
29
Audit Log
• Used to Track Critical Path Component Name Changes Impacting Recovery Procedure From Tape
Critical Path Components Tracked– Tape Backup Server name– Tape Policy Name– Staging Server Name– Staging Server Destination Folder
Audit Process Is Automatic Requires No User Intervention
– Critical path component changes generates a environment audit record– Audit records readily available interactively basedd on client name from
Configuration Manager option T=Audit
– Used to determine where a given clients data resides on tape system
30
Audit Record Updates
Environment Change Detected in Critical Path– will occur relatively infrequently over time– displayed in audit summary or detail report
Non Critical Path records– audit record established every time a client is staged– displayed in audit detail report
Audit Log File– any filters or script can be run against audit log contents to format your own custom or
alternative report formats– available through ATO interface for display purposes
31
Sample Audit Summary Report
initiate audit report
initiate audit reportinitiate audit reportselect client for report
initiate audit report
select client group
selected report type defaults to summary or enter D = detail
• Summary Report Displays Critical Path Change Records Only
32
Sample Audit Detail Report
• Detail Report Displays Critical and Non Critical Audit Records
selected report d=detail
33
ATOTape Backup Script
Staging Server SpecsDestination Path
34
ATO - Tape Scripts
Automated Tape Script Generation– criteria taken from environment and client configuration files to generate a suitable script
autotapeout.bat or autotapeout.sh located in the environments temp directory– transferred to staging server <AvamarHomePath>/etc/scripts and executed as a pre script– supports incremental and non-incremental – non-incremental script will remove staged data after a confirmed successful tape backup– Supported tape applications
Arcserv – Brightstore, HPDP, Networker, Backupexec NetBackup, TSM & Commvault Networker support for client alias name, I.E. tape out data owned by original client name (disabled)
– any tape solution with a suitable CLI could be accommodated
User Defined Scripts– must contain necessary logic to enable ATO to capture its completion status– refer to ATO doc for example or use an auto generated script as a reference– must be kept on the Utility node, suggested location /atoadmin/userscripts
Tape Backup Status– tape status is logged on staging server in <Avamar-Home-Path>/etc/scripts/autotapeout.stat – The contents of this log are recovered back to ATO which parses and displays results– detailed failure information logged to event and batch logs– any non-zero tape RC is considered a failure
35
Staging Server & Destination Path Specs
• Staging Server Specs• can be physical or virtual• Must support required Avamar and Tape App file system agents• Decent CPU capacity with the equivelant of at leat 6-8GHz
• Structured Non-Incremental Destination Path:•<user-defined-path>/BYDATE/<client-name>/<date-time-buid#>/<backup-data>
• Sturctured Incremental Destination Path:•<user-defined-path>/INCREMENTAL/<client-name>/<backup-data>
1. Path definition shown in RED is not user modifiable2. Optional path suffix inserted prior to </backup-data> using the -path option or from Fld-16 in client configuration file
<user-defined-path>: taken from client configuration file
BYDATE or INCREMENTAL: inserted by ATO
<client-name>: inserted by ATO
<date-time>: inserted by ATO
<backup-data>: recovered backup data
36
Sample Networker Save Set Definition
Sample Networker client resource
– BYDATE– INCREMENTAL
Define multiple client resources as required
All staged clients to a given location will be included in the Tape Backup
User portion of staging path defined in client cfg. file
Each client resource may have its own media pool, group assignment and schedule etc.
37
Networker Recover View Of Tape-Out Data
Recovery View on Staging Server of Incremental TO
Recovery View on Staging Server of non-Incremental TO referred to as BYDATE
38
ATO Incremental Advantage
39
Incremental Tape-Out
Incremental Concept– incremental staging is relative to previously staged data, 1st stage is always a full– requires staged data remain on the staging server between ATO sessions
refer to –incdel option for a method to resynchronize staged data with backup client– beneficial for FS & NDMP not for SQL, Exchange, Oracle or Lotus Notes– total disk space required equivalent to size of one copy of source data involved– staging disk can be any disk type, Arrays, JBOD’s, USB stand alone etc.– disk I/O performance less critical when using incremental– makes it feasible to leverage an incremental tape-backup
Performance Gains– typical effective throughput gains versus non-incremental ~5 to 8 times– expect ~600-900 GB’s/Hr for staging dependent on % incremental change– If using incremental tape policy
expect equivalent tape backup throughput gains expect reductions in tape media usage of ~70-80%
– use of concurrent staging servers will improve overall aggregate throughput
How to Enable Incremental– Add to the -tapeout action, –inc option
40
Incremental AdvantageExample-1 Environment: single node
Initial Stage 4.7 GB’s 2.45 min’s
Initial Tape Backup 4.7GB’s 4.5 min’s
41
Incremental AdvantageExample-1 Environment: single node
Post Initial Stage 4.7 GB’s 15 sec’s
Post Initial Tape Backup 4.7 GB’s 1 min
42
Incremental Advantage Example-2 Environment: single node
Initial Stage 10.5 GB’s 11.45 min’s
Initial Tape Backup 10.5 GB’s 8 min’sSession continued...
43
Incremental AdvantageExample-2 Environment: single node
Post Initial Stage 10.5 GB’s 30 sec’s
Post Initial Tape Backup 10.5 GB’s 1.5 min’s
44
Incremental Advantage
Example-1 Data SizeRun Time
Staging / TapeEffective Throughput
Staging / Tape
Initial Stage 4.7 GB’s 2.75 / 4.5 min’s ~110GB/Hr / ~65GB/hr
Post Initial Stage 4.7 GB’s 0.25 / 1.00 min’s ~1.1TB/Hr / ~288 GB/hr
Performance Increase n/a x11 / x4.5 x10 / x4.5
Example-2 Data SizeRun Time
Staging / TapeEffective Throughput
Staging / Tape
Initial Stage 10.5 GB’s 11.75 / 7.5 min’s ~55GB/Hr / ~84 GB/Hr
Post Initial Stage 10.5 GB’s 0.5 / 1.5 min’s ~1.2TB/Hr / ~420 GB/hr
Performance Increase n/a x23 / x5 x21 / x5
Single Node Incremental Comparison Results
• Factors Influencing Test Results•Networker Server is using Windows•Networker configured to use a DL3D VTL•no changed files were involved
45
Incremental Advantage Example-3 Environment: multi node(4)
Initial Stage 44.5 GB’s 12 min’s
Initial Tape Backup 44.5 GB’s 23 min’s
46
Incremental AdvantageExample-3 Environment: multi node(4)
Post Initial Stage 44.5 GB’s 45 sec’s
Post Initial Tape Backup 44.5 GB’s 30 sec’s
47
ATO – Incremental Advantage
Example-3 Data SizeRun Time
Staging / TapeEffective Throughput
Staging / Tape
Initial Stage 44.5 GB’s 12.0 / 23.5 min’s ~223GB/Hr / ~114 GB/Hr
Post Initial Stage 44.5 GB’s 0.75 / 0.5 min’s ~3.6TB/Hr / ~5.2 TB/Hr
Performance Increase n/a x16 / x46 x16 / x46
Multi Node Incremental Comparison Results
•Factors Impacting Test Results•Networker Server & Staging server were the same server, an Avamar spare node•Networker configured using adv_file type device B2D•no changed files were involved•Staging configured to use all-nodes, provided ~10-15% increase in throughput
48
ATO Event Log
49
ATO - Event Log
All –select and –tapeout actions logged
Consolidates informational & Error Messages– Readily accessible to user for success / failure confirmation or problem diagnosis
To access Event Log: ato –l or ato -log– most recent event displayed initially– specify default browse direction by entering P=Previous or N=Next then press enter
to browse all events in the direction specified– to jump to a specific event#, enter desired event number & press enter– to search event log enter “s” then “h” for search editor syntax, similar to vi commands
Event messages are color coded– informational messages displayed in your default terminal color– errors messages displayed in red– info returned from tape application displayed in white
Each Environment# maintains its own Event Log
Email alerts contain corresponding Event Log details
50
ATO - Event Log Sample – Successful Stage & Tape Status
Avamar recovery log name located on SS and available within Activity Monitor
details returned from tape backup command issued by tape script
tape backup command syntax
issued by auto generated tape script
optional debug flags used to provide additional command trace information
staging process completed okay
51
ATO - Event Log Sample – Staging Failure
staging failure information, client, buid, error code
establish retry checkpoint containing only failed buid’s
if there’s no successfully staged items, tape backup back-up initiation will be skipped
Avamar log file name available in Activity Monitor or on the staging server
staging error count > 0
52
ATO - Event Log Sample - Tape Backup Failure
tape backup failed due to a mis-configured tape policy. The path specified to be backed up is not configured for tape backup.
staging completed okay
suggested rerun syntax to perform only the tape portion of this TO session no staging is required.
53
ATO Batch Manager
54
Batch Manager (BM)
Use to configure & monitor ATO batch sessions
Btach functionality includes– Create batch profiles to be initiated interactively, from command line or from CRON– Monitor status & progress of any number of active or completed ATO batch jobs– Review status of any given batch session as a summary or in detail– Maintains associated batch logs automatically– Use –D (upper case) debug flags to increase verbosity in batch logs
Color coded provides at a glance status confirmation of all batch sessions
– White=Active, Green=Completed Okay, Red=Failed
Views, dashboard and configuration– Dashboard provides log access and status view of all batch jobs– any number of batch profiles can be configured and managed
Access Batch Manager: ato –batch – Run a batch profile: ato –batch <profile-name>
55
Batch Manager Screen Shots
• Sample monitor mode screen showing summary view of job #7
• Sample configuration screen showing available profiles
view completion status of batch session#7
56
Batch Manager Execution Screen Shots
• Sample BM configuration screen showing batch profile #1 being initiated interactively
• followed by (2) monitor screen shots showing it as active (3) the final completion status
Batch job is now shown as being active in log monitor mode indicated by its date/time being displayed in White
Batch job completed okay indicated by date/time being displayed in Green
to review the entire log enter “r” to open the log file
initiation confirmation display1
2
3
57
ATO – Batch Manager Profile Screen Shots
• Sample Batch Profile• A profile contains the arguments required for the –select, -view and –tapeout process
• BM builds a suitable batch script at execution time utilizing a profiles contents
• the batch script maintains a log of the same name with .log extension in /usr/local/avamar/var
• Batch profiles reside in /usr/local/avamar/etc
• Batch profiles and log names always begin with “atobatch”
• add debug flags –D flags to capture additional detail to batch log
• add debug flags –d flags to capture additional detail to Event Log
58
Batch Manager CRON Screen Shots
Batch Manager CRON interface– Color coding of profile dates identifies if profile is configured, not configured or disabled in the CRON– Allows selection of CRON run times, day-of-week, month etc.– Supports any valid basic or complex CRON syntax for each field
59
ATO Recovery Manager
60
ATO – Recovery Manager RM
Recovery Manager (RM) “Checkpoints” Failed ATO Sessions
Two Failure Scenarios – Retry = individual staging failures during an ATO session– Rerun = staging items never initiated the result of a premature end of an ATO session
Failed Sessions Automatically Tracked Containing Only Failed or Outstanding BU-ID’s
– recovery sessions relative to the environment# used
Variable in Environment File Determines Quantity of Recovery Sessions to Maintain
– default is 10– round robin basis with oldest being rolled off the list and deleted
Failed Sessions Viewable Using RM Interface ato -recover– determine contents and size– similar to normal -view display
Failed Sessions can be Re-Executed or Deleted as Required
Subsequent Failures During a Re-Execute Result in a New Checkpoint Session Providing an Incremental Reduction During the Retry/Rerun Process
61
ATO – Recovery Manager - Retry View
RM View Similar to Standard View with Title Highlighted in Red
Top Half of Display shows Contents of Selected Retry Session
Bottom Half Lists the Available Retry Sessions– Lists color coded for quick at a glance confirmation. – Blue=Never Re-Executed, Green= Successfully Re-Executed, Red=Failed Re-Execution
normal view of the contents of retry session-#1
session-#1 has been retried identified by green date which is the date/time of the failure
syntax required to retry this session
this is a RETRY view
enter a session-# to execute,view or delete
62
ATO – Recovery Manager - Rerun View
this is a RERUN viewsession-#4 has never been rerun identified by blue date which is the date/time of the failure
syntax required to rerun this session
normal view of the contents of retry session-#4
Rerun Mode entered by Using M=Mode or ato –recover rerun
Top Half of Display shows Contents of Selected Rerun Session
Bottom Half Lists the Available Rerun Sessions– Lists color coded for quick at a glance confirmation. – Blue=Never Re-Executed, Green= Successfully Re-Executed, Red=Failed Re-Execution
enter a session-# to execute,view or delete
63
ATO Troubleshooting
64
ATO - Troubleshooting
Troubleshooting Staging Recovery Failures– troubleshoot a failed staging activity as you would any recovery starting with the Activity Monitor
– staging progress can be monitored using the Activity Monitor or ATO Batch Manager
– tape backup is initiated first as a recovery to transfer the tape script to the staging server then executed as a post script. This session will remain active for the duration of the tape backup phase with little data movement displayed in Avamar
– staging server logs in <HOMEPATH>/avamar/var will be backed up to Avamar automatically when a staging recovery fails making them available in the event the Activity Monitor is cleared
– log-id# of each stage activity is listed with the batch log staging information message and present in ATO’s event log when a single –d flag is used
ATO Batch and Event Logs– first place to look to determine overall status– useful for identifying most problem situations especially tape related– remember, event logs are maintained on a per environment basis and batch logs relative to the
job regarless of environment number being used–
65
ATO - Troubleshooting
Debug flags increase verbosity of messages displayed & logged to STDOUT & Event Log
– impact of debug flags are cumulative, the more you specify the more you get
– debug flags are primarily valid on –select & -tapeout sessions only
– Upper case –D logs to stdout only, useful for batch runs as they’re acptured in batch logs
– -d provides additional ATO specific informational messages including capturing the staging information message for each client and successful tape application results. Tape info always logged when tape errors detected
– -d –d echo avtar and mccli cmd syntax
– -d –d –d removes quiet mode from avtar and mccli cmds
– -d –d –d –d retains ATO session temp files to be used for analysis including the auto generated tape scripts
– -d –d –d –d –d –d full trace of the ATO session
66
ATO - Troubleshooting
Be careful with tape policy configuration– ensure required staging folders are configured matching those defined within the
client group– with a Networker server initiated backups use savgrp command, ATO will not be able
to detect a mis-configured save-set as savgrp acts on whatever is defined to it– with staging server (client initiated) backups incorrect save-set definitions are
detectable and these details logged to the event and batch logs
Auto tape scripts– located on staging server in <Avamar home path>/etc/scripts, – run manually for troubleshooting purposes autotapeout.bat or .sh as applicable– Auto Tape script run status captured in autotapeout.stat – custom scripts must contain logic to update & capture the stat file to enable ATO to
accurately report the tape backup status– required logic for custom scripts documented in ATO documentation, alternatively
review an auotgenerated script for example logic
Ensure correct Environment# is being used– make sure your looking at correct event log# – make sure the ATO action calls are using the intended environment#
67
Troubleshooting
Capture ATO trace and Grab’s for debug purposes– reduce selection criteria to the minimum possible in order to reproduce the problem– run the failing session as follows:
ato <your-ato-args> –d –d –d –d –d –d > /tmp/atotrace.txt 2>&1– run ATO grab diagnostic
ato -grab– send grab file to ATO support: [email protected]– A grab file is the single most useful debug tool, ALWAYS request or provide one
Staging Failures Often Related with Incorrect Values in Environment or Client Configuration Files
– always run a parse check against each file, ato –env parse or ato –cfg parse– is there sufficient space on the staging server mount point(s) being used?– is the specifed mount point in the client configuration file correct?– what does the ATO Event or Btach logs show?– can you stage the failing BUID manually using the GUI?– Is Avamar GC or Checkpoint active, if so do not run ATO?– run ATO health check, ato –health– refer to ATO menu Opt-12 ATO Administration