1

IBM® DB2® Universal Database™ Version 8 and VERITAS Storage Foundation for DB2

Enabling Disaster Recovery for DB2 UDB Using VERITAS Storage Foundation’s Volume Replicator

April 4, 2005 VERITAS Software Corporation IBM Toronto Laboratory, IBM Canada Ltd.

2

This document contains proprietary information of VERITAS and IBM. It is provided under a license agreement and is protected by copyright law. The information contained in this publication does not include any product warranties, and any statements provided in this document should not be interpreted as such. © Copyright IBM Corporation and VERITAS Software Corporation, 2005. All rights reserved.

3

TRADEMARKS The following terms are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both: AIX DB2 DB2 Universal Database IBM iSeries zSeries The following terms are trademarks or registered trademarks of VERITAS Software Corporation or Hitachi Data Systems in the United States, or other countries, or both: VERITAS, the VERITAS Logo, VERITAS Storage Foundation UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others. The furnishing of this document does not imply giving license to any IBM or VERITAS patents. References in this document to IBM products, programs, or services do not imply that IBM intends to make these available in all countries in which IBM operates.

4

ABOUT THE AUTHORS

VERITAS Software Corporation Ulrich Maurus Server and Storage Management Group Ulrich Maurus has been with VERITAS Software for more than ten years and was recently named Senior Technical Product Manager – VERITAS Storage Foundation for DB2. As such, he is responsible for technical information exchange between VERITAS engineering and its sales force. He is also the VERITAS liaison to the IBM Toronto Laboratory. Before assuming his latest role in product management, Ulrich was Senior Product Specialist for clustering and replication, Product Manager for SAP Edition, Engineer for VERITAS Cluster Server development, and Senior Consultant for OpenVision and VERITAS Germany.

IBM Toronto Laboratory

Nuzio Ruffolo DB2 UDB System Test,

Toronto Laboratory, IBM Canada

Nuzio Ruffolo has been with IBM for over ten years as part of the DB2 Universal Database System Test Organization (DB2 UDB SVT). Nuzio is responsible for testing DB2 UDB in a complex test environment that includes the Linux ™, UNIX®, and Windows® platforms and the iSeries™ and zSeries® servers. DB2 UDB testing is performed in this environment based on expected production environment requirements – millions of transactions and thousands of operational tasks repeatedly executed during the test cycle. On this team, Nuzio has held various team lead and management positions. Nuzio represents the entire DB2 UDB test organization on the IBM test council and has presented various testing related topics at the annual Centre for Advanced Studies Conference (http://cas.ibm.com). Nuzio is currently responsible for testing DB2 UDB in high-availability environments.

5

Table of Contents Abstract...............................................................................................................................................................6 Overview of the Tested Software Components ..................................................................................................8

DB2 Universal Database V8.1 for Linux, UNIX, and Windows ......................................................................8 VERITAS Storage Foundation for DB2 ..........................................................................................................8

VERITAS Volume Manager .......................................................................................................................8 VERITAS Volume Replicator .....................................................................................................................9

DB2 UDB and Hardware Configuration ............................................................................................................12 Storage Configuration ..................................................................................................................................12 DB2 Layout...................................................................................................................................................13

Replication Test Runs.......................................................................................................................................15 Switch of Replication Direction for Disaster Recovery/Promoting the Secondary Site ...........................17 Volume Snapshot and Database Activation on the Secondary as a Fire Drill.........................................17 Volume Snapshot on the Secondary with a Quiescent Primary Database..............................................18

Database State After Snapshots ..................................................................................................................19 Summary ......................................................................................................................................................20

Appendix A: Scripts to Administer Snapshots ..................................................................................................21 Take A Snapshot..........................................................................................................................................21 Description File for Snap ..............................................................................................................................22 Command File to Enable Snap Mount by Database Administrator..............................................................22 Utility to Enable Umount by Database Administrator ...................................................................................24 Script to Remove a Complete Snapshot for Another Update ......................................................................25

6

ABSTRACT This paper describes the implementation and design of a replication solution for the IBM® DB2® Universal Database™ Version 8.1.5 (FixPak 5) product (DB2 UDB), running on the Solaris 9 platform. The replication solution uses VERITAS Storage Foundation™ for DB2 and VERITAS Volume Replicator, an option for Storage Foundation for DB2. Guidance and best practices are provided for replication and off-host processing using DB2 UDB. Practical considerations regarding design and implementation with VERITAS Volume Replicator 4.0 are discussed later in the document. This paper will reveal not only how to use VERITAS Volume Replicator to quickly activate the most recent DB2 UDB copy when faced with a disastrous event at a Primary site (DR case), but also how to activate a consistent point-in-time copy of the database at the Secondary site while the replication continues. Additional point-in-time copies can be used for various off-host tasks such as fast recovery of logical corruption on the Primary site, data mining, or full backup to tapes without any impact to performance or availability on the production database. An understanding of the business drivers and terminology of high-availability computing systems is required for the reader to obtain maximum benefit from the information contained in this paper. In addition, familiarity with the process of installing and configuring DB2 UDB, along with a working knowledge of VERITAS Volume Manager, VERITAS File System, VERITAS Volume Replicator, and TCP/IP networking protocols, is assumed. This paper will not explore the latency and performance impact of data replication for DB2 database environments. Nevertheless, the decision regarding whether to set up VERITAS Volume Replicator to transmit data in synchronous or asynchronous mode will be discussed so that the reader obtains an idea about the associated impact on I/O performance. Any attempt to extrapolate results obtained inside any test environment to a real wide-area replication configuration is extremely complex because of differences in available network bandwidth, data change rate, and distance between sites. Although the DB2 server configuration was located in a single machine room, with a 100 Base-T Ethernet network as transport medium for the replication link, the implemented procedures for replication and data snapshots are similar to real-world wide-area data replication scenarios across Internet Protocol (IP) networks. Since most wide-area networks are implemented by dedicated, external hardware and no longer hosted by the servers themselves, the ports and protocols are the same for wide-area and local-area data communication. Unlike hardware-based replication offered by disk array vendors, VERITAS Volume Replicator does not require any proprietary data transmission protocols or hardware, and can actually share existing IP links on database servers. Of course, sharing a single network port for various types of data, including replication traffic, may pose a computing bottleneck. For higher availability of the replication scenario, a cluster product such as VERITAS Cluster Server in combination with redundant network hardware can be implemented. VERITAS Volume Replicator transmits data continuously across an IP link, rather than database log shipment replication, as provided, for example, by DB2 UDB Version 8.1. The original limitation associated with continuous replication has been the inability to access the replicated storage at the receiving site. With the 4.0 release of VERITAS Storage Foundation for DB2, a new volume management feature called Instant Space-Optimized Volume Snapshot was introduced. As a result, multiple, single point-in-time copies of the replicated data can be created with only a small amount of additional disk space. Combining VERITAS Volume Replicator replication with volume snapshots is like merging the concept of continuous storage replication with periodic log shipment and database roll-forward technologies.

7

Even though VERITAS Volume Replicator can be configured to replicate only parts of the DB2 environment, such as the transactional log files, we decided to implement a complete database storage replication scenario. There is associated overhead of replication traffic because the log file updates as well as the database container updates have to be transmitted between the Primary and Secondary sites. This overhead is counterbalanced by simplified administration. In particular, all database configuration changes and even non-logged database loads, as well as application files outside the database, can be replicated using VERITAS Volume Replicator.

8

OVERVIEW OF THE TESTED SOFTWARE COMPONENTS This section provides a short introduction to IBM DB2 Universal Database (DB2 UDB) design and the relevant VERITAS software components. For more in-depth product information, refer to the specific product’s documentation. DB2 UNIVERSAL DATABASE V8.1 FOR LINUX, UNIX, AND WINDOWS DB2 UDB marks the next stage in the evolution of the relational database, by providing enhancements in the areas of performance, availability, scalability, and manageability. DB2 UDB is the database of choice for business-critical solutions such as e-business, business intelligence (BI), content management, enterprise resource planning (ERP), and customer relationship management (CRM). In today’s global business environment, the ability to access data at any time is critical. The built-in capabilities of DB2 UDB to handle planned and unplanned outages ensure that business applications are available whenever needed. Whether switching to a standby database server if an unexpected database failure occurs, or carrying out online maintenance such as the ability to perform an online reorg, DB2 UDB makes sure all business applications remain available. Online utilities, such as index rebuild, index create, and table load, as well as configuration parameters that can be changed without stopping the database, deliver improved performance and high availability. For more information and a complete list of DB2 UDB features, refer to the product documentation, available online at: http://www.ibm.com/software/data/pubs/. VERITAS STORAGE FOUNDATION FOR DB2 VERITAS Storage Foundation for DB2 is an integrated suite of data, storage, and system management technologies that optimize performance, availability, and manageability of DB2 UDB databases. In addition to DB2 UDB-specific utilities for simplified administration, Storage Foundation for DB2 leverages the core VERITAS technologies, which include Volume Manager, File System — with Quick I/OTM and Cached Quick I/O, VERITAS Cluster Server as well as the VERITAS Volume Replicator option. This paper only covers technology areas relevant to replication solutions. Further details about these solutions are available from the VERITAS Web site (www.veritas.com) or discussed in the associated VERITAS documentation available as part of the software distribution. VERITAS Volume Manager VERITAS Volume Manager, a key component of VERITAS Storage Foundation for DB2, is a storage virtualization tool that supports Solaris, HP-UX, Linux, AIX®, and Windows, and allows the management of physical disks as logical devices, called volumes. A volume is a logical device that appears to data management systems as a physical disk partition device. By allowing volumes to span multiple disks, VERITAS Volume Manager enables the management of virtual storage pools rather than actual physical disks. By using VERITAS Volume Manager as an abstraction layer that virtualizes storage and makes that storage easier to manage, users as well as operating systems and applications overcome the physical restrictions imposed by hardware disk devices. Through support of RAID, VERITAS Volume Manager protects against disk and hardware failure. Additionally, Volume Manager provides features that enable fault tolerance and fast recovery from disk failure. VERITAS Volume Manager also provides easy-to-use online disk storage management for tasks such as dynamically configuring disk storage while the system is active, and ensuring that data remains available.

9

VERITAS Storage Foundation for DB2 4.0 introduced a new Volume Manager capability called Instant Space-Optimized Snapshot, which greatly improves the ability to perform off-host application processing. While traditional third-mirror split/snapshot techniques place constraints on storage space and time — a complete copy of the data set must be synchronized before the snapshot can take place — the VERITAS solution supports the creation of almost instantaneous multiple snapshots. Space-optimized instant snapshots save disk space because they only retain copies of the data blocks that have changed from the original data set. Unchanged data blocks are not retained. With space-optimized instant snapshot, the space requirements for the changed blocks of all data volume snapshots can be consolidated in a single data volume called a shared cache object. A single shared cache object can cover multiple snapshots of the same set of volumes, from different points in time. This new feature is especially useful in replication environments, where volumes used to store the replicated copy of the data are not accessible. With space-optimized instant snapshots, multiple read/write copies of the data can be created without a large investment in additional storage. VERITAS Volume Replicator For the tests documented in this paper, the standard Volume Manager capabilities were expanded by installing VERITAS Volume Replicator, an option available with VERITAS Storage Foundation for DB2. Volume Manager can only access storage attached to a host-based disk controller — such as SCSI or fibre channel. Adding the Volume Replicator option allows write requests from an application to be sent to a remote location using Internet Protocol-based networks.

Figure 1: VERITAS Volume Replicator – asynchronous replication

Figure 1 illustrates the basic VERITAS Volume Replicator process. For a detailed discussion of the architecture, read the VERITAS Volume Replicator Administrator's Guide and Configuration Notes available at: http://support.veritas.com/menu_ddProduct_VR.htm. The figure shows the data flow when using VERITAS Volume Replicator in asynchronous replication mode. Any data written by an application — such as a DB2 UDB database using storage under VERITAS Volume Manager control — will pass through the Volume Manager I/O daemon (vxvm).

10

If replication is enabled for specific data volumes — this database table space can be either Database Managed Space or System Managed Space (DMS or SMS) — all write requests are first stored in the Volume Replicator Storage Replicator Log (SRL). The SRL is a standard data volume accessed in write-only mode during regular operations. Because the SRL must reside on persistent storage, the vxvm daemon can commit I/O requests to the application as soon as the write to the SRL has been completed. If the Primary server fails, data volumes will be returned to a consistent state, reflecting the most recent writes to database data, during the reboot process. System recovery accesses the SRL, during a reboot, to ensure data consistency. The SRL maintains write order fidelity, storing write I/Os in the same sequence as they were executed by the database. The SRL maintains information about the size of each I/O request, in addition to the raw data blocks, to ensure the completeness and consistency of replication recovery at a disaster recovery (DR) site. After successfully writing to the SRL, the vxvm daemon initiates the I/O against the local data volume, and then sends a copy to the Secondary server using the IP protocol link. Up to 31 Secondary systems are supported by VERITAS Volume Replicator. The network connection between each Secondary location and the Primary location is controlled by a pair of VERITAS Volume Replicator-specific Volume Manager objects called replication links (RLINK). The RLINK uses a two-way communication protocol (TCP and UDP are supported) and includes a heartbeat utility to monitor the state of the link even if no data is transferred. After the replicated data has been received by the corresponding Volume Manager I/O daemon at the Secondary, and stored in the buffer cache, a network acknowledgement is sent back to the primary server.

Figure 2: VERITAS Volume Replicator – synchronous replication

In asynchronous replication mode, the network acknowledgement has no impact on the data flow. But, as Figure 2 illustrates, in synchronous replication mode, the network acknowledgement triggers the write-commit between the vxvm daemon and the application on the Primary server. The only difference between VERITAS Volume Replicator asynchronous and synchronous replication is the time required for the write request to be acknowledged by the application.

11

Asynchronous replication is clearly faster than synchronous replication, but in a system crash scenario, Secondary data volumes might not contain the latest updates written to the Primary data volumes. Although data on the Secondary volumes is consistent, because of the write order fidelity and write request size guarantees of the SRL, some of the most recent database transactions may have been acknowledged by the database server at the Primary location but not received at the Secondary location. However, assuming that the complete database environment is replicated as part of a single replicated volume group, the standard DB2 UDB crash recovery process will get the database into a transactional consistent state.1 In synchronous replication mode, the Primary and Secondary data volumes are always identical, but there can be a significant impact on write performance to the Primary volumes. The latency of the network link, which varies depending on network bandwidth and distance between the replication sites, adds to the elapsed time between the application write I/O and the acknowledgement from the vxvm daemon to the database writer process. The replication mode is governed by attributes of the RLINK. The mode can be dynamically altered, through the VERITAS Volume Replicator command line interface (CLI) or the VERITAS Enterprise Administrator (VEA) graphical user interface (GUI). Altering the replication mode does not impact applications. If the network link between Primary and Secondary hosts is disconnected for an indeterminate amount of time, synchronous replication can be automatically switched to asynchronous, avoiding blocked application writes that are waiting for acknowledgement from the Secondary server. The VERITAS Volume Replicator Administrator’s Guide discusses a variety of fault scenarios, including the possibility of an overflow of the SRL in a situation where replication cannot continue for an extended period of time. In this scenario, the length of time before the SRL overflows is dependent on the size of the log and the application data change rate. The SRL is written in round-robin fashion, with space in the log freed as soon as a data acknowledgement message is received from the Secondary host — item 7 in Figure 1, and item 6 in Figure 2. During standard operations, with adequately sized replication hosts, network bandwidth, and SRL, there will always be ample space available in the log. Note that VERITAS Volume Replicator does not require all volumes at the Primary location to be replicated to the Secondary location. The VERITAS Volume Manager Replicated Volume Group (RVG) is maintained by the administrator and contains a list of volumes to be replicated. Volumes can be attached to, or detached from, the RVG dynamically without interrupting the replication process or applications that access the storage. Initial synchronization steps are necessary, however, when beginning replication to a new Secondary volume. VERITAS Volume Replicator is also able to replicate between volumes on heterogeneous storage hardware. Because replication occurs at the logical level, the only prerequisite is that Primary and Secondary volumes be sized with the same number of bytes. Volume Manager 4.0 also enables replication to occur between different server operating systems, although it must be remembered that application binaries, including those for DB2 UDB, and data containers might not be interchangeable across operating environments. When replicating between a Primary and Secondary location, it may become important to synchronize events that affect the data. VERITAS Volume Replicator can use the In-Band Control Messaging (IBC) utility to synchronize actions between local and remote sites. An IBC message does not interact with the data and is only used to trigger a command sequence at the Secondary location. The IBC utility must be registered with both the remote and local copies of VERITAS Volume Replicator, and execution related to a message may impact replication and server performance: the RLINK is frozen while the Secondary location executes the commands associated with an IBC message.

1 Details about the DB2 UDB crash recovery process can be found in the standard documentation set or Chapter 2 of High Availability Guide to DB2 by Chris Eaton and Enzo Cialini, Prentice Hall, 2004.

12

DB2 UDB AND HARDWARE CONFIGURATION The tests discussed below were executed on two Sun Enterprise 450 servers. The replication network link was implemented using crossover Ethernet cable connected to separate 100 BaseT ports. The configuration was useful for separating the replication IP traffic from that of DB2 clients accessing the database. Moreover the network protocol (TCP/IP) for the local link is the same as for wide-area replication. In many real-world replication configurations, Ethernet network interface cards are used as access points on servers instead of direct point-to-point wide-area protocol adapters. For these configurations, a hardware converter like a bridge routes the Ethernet frames across long distant network links. As shown in Figure 3, one server was attached to an IBM Enterprise Storage Server (Shark) utilizing 100 GB of disk space, separated into 100 logical units (LUNs) of 1 GB each. This was the original VERITAS Volume Replicator Primary. The other server was connected to a Sun StoreEdge A5200 array, configured with 22 x 36 GB drives – and configured as Secondary at the start of the test.

Figure 3: Hardware Setup

STORAGE CONFIGURATION Each array’s disks were added to a local diskgroup named db2repl. Because Volume Manager 4.0 no longer requires a diskgroup named rootdg, the db2repl was the sole diskgroup on each server. When using VERITAS Volume Replicator, it is highly advisable to ensure that all Volume Manager objects have the same name on both hosts. Otherwise, additional command-line arguments must be provided to configure and maintain the replicated volume group, thus increasing the complexity of administration. It is good practice to use the same diskgroup alignment value on both servers — in this test, a value of 8192 bytes was used. For the database container and control files, 16 volumes were created on each server inside the db2repl diskgroup. Again, the same names were chosen for matching volumes, to ease replication administration. Because VERITAS Volume Replicator assumes that the replication direction can be switched, it is essential that matching volumes, such as the file system of the database log files, be assigned exactly the same number of disk blocks. All volumes at the Primary site were then initialized with VERITAS File System. Quick I/O files were created for the database container space, and these were accessed by the DB2 engine as database managed space

13

(DMS). For details about Quick I/O, consult the VERITAS Storage Foundation for DB2 Database Administrator’s Guide. Two more volumes were created inside each diskgroup (one for the replication and the other for snapshots). To enable instant, multiple snapshots, a shared data cache volume was created as a striped RAID-0 device. Good write performance is important for the cache, and separating the data storage from the data volumes is also a good practice. Because the snapshots were used only temporarily on the Secondary replication server, redundancy was not needed. It was, therefore, unnecessary to mirror the shared cache volume of the db2repl diskgroup located in the A5200 disk array. The VERITAS Volume Replicator SRL is accessed during the reboot of the primary server following a system crash. The loss of SRL data can cause inconsistency in the database data, especially in asynchronous mode replication, so guaranteeing local availability of SRL data is essential. The Shark array controller enables RAID at the hardware layer, which makes it unnecessary to configure software RAID using VERITAS Volume Manager. Only a single LUN was needed for the SRL. On the A5200, the SRL was implemented on a volume configured for RAID-1 with a complete disk for each sub-mirror. As with the shared cache volume, performance of the SRL is critical and allocating a complete spindle for these volumes is justified. Note: Combining application data and the SRL on a single physical device will adversely impact performance and should be avoided. The database control and log files reside on a single Volume Manager volume and VERITAS File System sized to fit for a 24-hour run with a DB2 UDB workload. The storage for the database container was placed on separate volumes, with corresponding VERITAS File Systems and pre-allocated Quick I/O files on each volume. Using a total of 14 separate volumes for a single database allowed us to test the scalability of the replication and snapshot environment. Maintaining consistent data across multiple snapshot volumes is harder than when placing all data on a single volume and file system – the latter may be the better choice for smaller environments, where more straightforward administration is an important consideration. But the finer granularity of this test storage configuration is more appropriate for large enterprise DB2 database implementations, where performance improvements down to the disk level are a requirement. DB2 LAYOUT A database layout was created to support a simulated online transaction processing (OLTP) workload. This design is commonly used as part of the DB2 UDB system test cycle. Four buffer pools of 16 MB each were created, for a total database buffer pool space of 64 MB. Page sizes of 4KB, 8KB, 16KB and 32KB were used, with one unique page size for each of the four buffer pools. The database layout consisted of the following DMS table spaces, using device type containers of various page sizes.

• Four temporary table spaces of 4KB, 8KB, 16KB and 32KB page sizes totaling 20 GB • Four regular table spaces of 4KB, 8KB, 16KB and 32KB page sizes to store data totaling 20 GB • Four regular table spaces of 4KB, 8KB, 16KB and 32KB page sizes to store index data totaling 8

GB • One long/large table space of 4KB page size to store long data columns totaling 40 GB

Other database objects — tables, views, and indexes — were created on these table spaces. The database is made up of 1,286 tables. Each table has a corresponding alias. The database also has 24 views, and 51 indexes.

14

The width of the tables used in this workload ranged from a single column to 203 columns, averaging 10 columns per table. Once the objects were created, a random data generator seeded the tables with data creating a database with an initial size of 15 GB for use in testing.

15

REPLICATION TEST RUNS For our test runs, we first installed the DB2 distribution packages as well as identical DB2 instance environments on both servers on local, non-replicated storage. It is important to use the same numerical user ID and group ID for the instance owner on both servers, because file ownership and access permissions are being replicated by VERITAS Volume Replicator as well. Separate diskgroups and data volumes for the test database were then created on each host. Again, it is a good practice to use the same naming scheme and volumes sizes on each machine. The creation of file systems, container files, and database tables were only executed on the dedicated DB2 Primary server. The replication seeding process we used will transfer the complete data from Primary to Secondary server, thus rendering the creation of the volume objects superfluous on the standby system. The seeding process includes file system structural data as well as file system log information when using VERITAS File System. Instead of creating a new database environment, an initial setup can start with an existing database environment. In this case, the server used as the replication Secondary at the start has to be updated so that the DB2 revision and storage configuration exactly match the existing DB2 server. The duplication of an existing Volume Manager environment is not an automated process. It is a system administrator task executed through the command line or GUI. After configuring and loading the DB2 test database on the Primary server, we initialized the data volumes for the snapshots. Creating the associated data change objects (DCO) prior to implementing replication reduces the complexity of the configuration process. The version 20 DCO bitmap for snapshots supersedes the data change map used by prior versions of VERITAS Volume Replicator. The preparation of storage for volume snapshots can be achieved using the Volume Manager CLI — see example command syntax below — or the VERITAS Enterprise Administrator GUI. Each volume used for the DB2 database must be initialized once: a single DCO can be used by multiple snapshots concurrently. To support a switch of the Primary and Secondary site, we prepared all database volumes on both servers at the start of the test cycle. PRIMARY_root# vxsnap -g db2repl prepare <volume_name> SECONDARY_root# vxsnap -g db2repl prepare <volume_name> To support multiple, space-optimized snapshots a shared cache object was created in each Volume Manager diskgroup: PRIMARY_root # vxassist -g db2repl make db2cach-vol <size_of_disk> PRIMARY_root # vxcache -g db2repl att db2cach-vol db2cache SECONDARY_root # vxassist -g db2repl make db2cach-vol <size_of_one_disk> SECONDARY_root # vxcache -g db2repl att db2cach-vol db2cache The auto-grow feature is assigned by default, and was not changed for the tests, but no automatic volume expansion was observed. After the snapshot initialization for the two disk groups on each server, the Volume Manager objects for complete database replication were created. There are multiple methods of performing the initial setup, such as using atomic Volume Manager commands or menu selections, as part of the VEA GUI. Alternatively, the vradmin utility can be used from the command line.

16

If the vradmin command is used, the special daemon, vradmind, must be started on both servers. For Solaris implementations, a vradmin startup script is installed with Volume Manager, and the process is automatically started during the system boot. The SRL must be created as a standard data volume before creating the RVG object. As with the cache storage space, it is good practice to assign a complete disk spindle to the SRL to keep it separate from other data storage. Note: Because the SRL is used for data volume recovery on a VERITAS Volume Replicator Primary after a system crash, additional protection of the SRL volume using hardware RAID or a Volume Manager mirror is highly recommended. In this test, both options were chosen — hardware RAID-5 on the sun-ha3 and IBM ESE and a Volume Manager mirror across two disks on the Secondary (A5200) disk array combination. PRIMARY_root # vxassist -g db2repl make db2srl <size_of_one_LUN> SECONDARY_root # vxassist -g db2repl make db2srl <size_of_one_disk> The volumes for the database container and control files for replication were created on each server, with exactly the same sizes, in disk blocks, and the same names. The two replication network links were already initialized, using 192.168.1.1 for the initial Primary and 192.168.2.1 for the Secondary. In addition to using the same names for all replicated Volume Manager objects (such as disk group and data volumes), another security option must be enabled to support the remote creation of Secondary objects based on these Primary objects. The vradmin command uses the hidden file /etc/vx/vras/.rdg to determine if the Primary is authorized to create VERITAS Volume Replicator objects on the local machine. Although the + sign is supported as a wildcard, the more secure fully qualified diskgroup ID.name — obtained using the vxdg list command on the Primary server — was used for this configuration. PRIMARY_root# vxdg list NAME STATE ID db2repl enabled,cds 1076972548.1330.<Primary_hostname> SECONDARY_root# echo "1076972548.1330.<Primary_hostname>" > /etc/vx/vras/.rdg Now the initialization of all replication-related Volume Manager objects can be executed only on the initial Primary server: PRIMARY_root# vradmin -g db2repl createpri db2_rvg <list of data volumes> db2srl PRIMARY_root# vradmin -g db2repl addsec db2_rvg 192.168.1.1 192.168.1.2 After all replication objects are created, the data volumes on both servers must be synchronized to seed the replication. The VERITAS Volume Replicator Administrator's Guide describes several alternative methods of synchronizing a Secondary server. For this test, automatic synchronization was chosen since the database had already been installed on the Primary and the transfer of 15 GB of the database could be accommodated by the available network bandwidth. PRIMARY_root# vradmin -g db2repl -a -c <checkpoint_name> startrep \ db2_rvg sun-ha4vvr The advantage of using the vradmin -a startrep command is the ability to run the application on the Primary server during Secondary synchronization. It is important to note, however, that the data on the Secondary is inconsistent while the automatic synchronization takes place. Only after all data blocks have been copied by the Primary to the Secondary is the data on the Secondary usable either for snapshots or as the active server. If network bandwidth is insufficient to perform automatic synchronization within a suitable timeframe, VERITAS Volume Replicator also supports initial synchronization from backup images. However, it’s

17

important to note that backup images may be slightly outdated when installed at the Secondary site; therefore, a differential synchronization of the data blocks must be performed across the network. In this test, differential synchronization was used to verify the contents of the data volumes on both sites. In test environments, where changes to the Volume Manager configuration may be applied independently at either server, the vradmin syncrvg allows administrators to synchronize the configuration while minimizing network traffic. Differential synchronization uses a checksum, computed on both sites; therefore, only data that differs from the checksum is transferred across the network. Switch of Replication Direction for Disaster Recovery/Promoting the Secondary Site After the initial synchronization was completed, consistency of the data on the Secondary server was verified by switching the Primary and Secondary sites. The vradmin command used to perform this switch of direction can only be executed if the database has been stopped and the associated file systems have been unmounted. To switch direction of the replication, the following commands must be executed on the Primary: PRIMARY_DB2Admin> db2stop [force] PRIMARY_root# umount <db2 control + container file systems> PRIMARY_root# vradmin migrate db2_rvg sun-ha3vvr The vradmin migrate command reverses the direction of the replication and now the new Primary (formerly Secondary) server can be activated. Mig_PRIMARY_root# mount <db2 control + container file systems> Mig_PRIMARY_DB2Admin> db2start In a real disaster in which the Primary is unable to issue a vradmin migrate command, the procedure executed on the Secondary is very similar. The Secondary must be promoted using the vradmin takeover command prior to the mount. Volumes used to apply replicated data are in a state that prevents any read/write I/O. If the failed former VERITAS Volume Replicator Primary is repaired or rebooted, and the network link connecting the two servers becomes active, the failed Primary server will automatically be configured as the Secondary. In this scenario, VERITAS Volume Replicator can determine what data changed since the takeover occurred by comparing the SRL on the new Primary to that on the new Secondary. The VERITAS Volume Replicator documentation contains more details on how this is accomplished. Volume Snapshot and Database Activation on the Secondary as a Fire Drill Replication for off-host processing using a full-volume snapshot was previously supported by VERITAS. However, this replication method proved to be very storage-intensive. It required a second, complete copy of the database volumes, in addition to those used for replication. In addition, all data blocks on replicated and Secondary copy volumes had to be synchronized before the Secondary could be split off, thereby taking up a considerable amount of time. The tests detailed here used the new space-optimized instant snapshot feature of Volume Manager Release 4.0, which allows the creation and use of a complete copy of data volumes within seconds. While all Volume Manager commands can only be executed by the root user — UNIX account with user ID 0 — additional scripts were created to administer and activate snapshots by the database administrator without root privileges (see Appendix A: Scripts to Administer Snapshots). The easiest way to create an instant snapshot is to do so without interfering with the active database. This method is essentially like a fire drill for disaster recovery, where the point-in-time of a disaster is unknown.

18

During testing, the volumes created by the snapshot command were always consistent. As part of the activation process, a file system replay was necessary; only VERITAS file systems were used to host database control, log, and container files. After the replay, the standard db2start command, which includes a crash recovery process, was used to activate access to the database. No other actions by the system administrator or database administrator were necessary. SECONDARY_DB2Admin> snap00 SECONDARY_DB2Admin_SEC> snap00mnt SECONDARY_DB2Admin_SEC> db2start Volume Snapshot on the Secondary with a Quiescent Primary Database To obtain a synchronized, more consistent snapshot, the snap command can be issued — via an rsh interface from the Primary server — after quiescing the database. PRIMARY_DB2Admin> db2 connect to <db_name> PRIMARY_DB2Admin> db2 set write suspend for database PRIMARY_DB2Admin> sync; sync PRIMARY_DB2Admin> rsh <SECONDARY> <Path_to_snap_scripts>/snap00 PRIMARY_DB2Admin> db2 set write resume for database If the steps needed to activate remote or secure shell command execution are not available, another method, using In-Band Control (IBC) Messaging, can be used. In this case, a series of system administrator and database administrator commands must be executed in the correct order. For sites with separate individuals or groups responsible for the administration of the operating system and database, executing these commands can become complex. Executable scripts can be implemented with set userid. However, as discussed in Appendix A, Solaris 9 has restrictions on the inheritance of the effective user ID, which prevents the execution of the vradmin command. The vradmin command actually executes atomic Volume Manager commands — such as vxvol and vxedit — which do not inherit the root ID from the parent process. A special set of scripts was developed for this series of tests, but individual commands were also issued by personnel assuming the roles of both system administrator and database administrator. As with most elements of UNIX, these restrictions can be circumvented with some effort. However, this could result in an unacceptably open and insecure system: tolerable in a test environment, perhaps, but not where sensitive production data is involved. In preparation for the IBCs using the vradmin command, three files must be created on the Secondary server: SECONDARY_root# cd /etc/vx/vvr/ibc_scripts SECONDARY_root# mkdir db2_snap SECONDARY_root# cd db2_snap SECONDARY_root# echo "exit 0" > prefreeze SECONDARY_root# echo "exit 0" > postfreeze SECONDARY_root# ln -s <Path_to_snap_scripts>/snap00 onfreeze Because these tests involve a single instance of DB2 UDB, configured with only one database, minimal versions of these scripts were used. For larger database environments, the argument list parsed by the vradmin daemon can be used to create more complex snapshots. The argument list contains diskgroup, RVG, taskname — the name of the subdirectory created in step 2 of the above script — and replication link name.

19

For a space-optimized instant snapshot, no preparation was necessary prior to the snapshot, and no additional steps were necessary afterwards to make the data usable. The mount and the start database commands can be automated, using the pre- and post-freeze scripts. However, this would run the snapshot under the control of the system administrator, possibly colliding with requirements of the database administrator. For sites where system and database administration are executed by the same group, the vradmin interface supports further automation. Two more scripts, quiesce and unquiesce, can be implemented to automate the db2 set write [ suspend | resume ] database functionality. Once the preliminary steps have been completed, the snapshot can be issued from the Primary server. PRIMARY_DB2Admin> db2 connect to <db_name> PRIMARY_DB2Admin> db2 set write suspend for database PRIMARY_root# sync; sync; vradmin -g db2repl ibc db2rvg db2_snap <Secondary> PRIMARY_DB2Admin> db2 set write resume for database PRIMARY_DB2Admin> db2 terminate After the snapshot has been created on the Secondary server, using our snap00 script, the DB2 database can be activated. SECONDARY_DB2Admin> snap00mnt SECONDARY_DB2Admin> db2start DATABASE STATE AFTER SNAPSHOTS Each time the DB2 database was started, we scanned the log files for error messages regarding data inconsistencies. For the arbitrary point-in-time snapshots, the database was always in a roll-forward recoverable state, similar to the state after a system crash. During our numerous tests, we didn't have to manually correct database objects. The standard DB2 crash recovery procedures always opened the database in a consistent state for immediate user access. Besides the fact that these tests show the very robust DB2 recovery design, they confirm that the VERITAS Volume Replicator and Volume Snapshot mechanism do not result in incomplete I/O writes. Moreover, even for multiple volumes, all snapshots were taken at exactly the same single point-in-time, not causing a mismatch between different database containers. Because the complete update I/O stream from the database writer processes has to pass the vxvm daemon, a temporary freeze of I/O activity allows even a multiple volume or file system snapshot. Moreover, during the timeframe when I/O activity is frozen, operation internal Volume Manager buffers are flushed, thereby guaranteeing consistency of the information on disk. The fact that Volume Manager does not break write requests into smaller units – such as 512 or 1024 byte blocks in case of a disk driver – prevents a log write request by DB2 of 4 KB (by default) to be partially written. On the other hand, the freeze or thaw time period of VERITAS Volume Manager was below the I/O timeout recognized by the file system or the DB2 writer process for our 14-volume snapshot configuration. As such, our tests show that a quiescence of the DB2 database is only necessary if a snapshot is taken at a very specific point-in-time: for example, at the end of a day. In this case, the DB2 standard feature write suspend / resume, in combination with a VERITAS Volume Replicator IBC, can do the trick.

20

SUMMARY This document shows that the VERITAS Volume Replicator option for VERITAS Storage Foundation for DB2 - a tool used to create a remote copy of a DB2 database environment - is easy to configure and to maintain. By replicating the complete environment – the database container as well as control and log files – in a single data stream, the state of the remote copy is always recoverable using the standard DB2 crash recovery processing. Using a space-optimized snapshot of the replicated volumes, a frozen point-in-time image of the database can be created at the remote site with minimal additional storage requirements. Since the snapshot represents a full readable and writeable copy of the replicated database, it can be used for various types of off-host processing like database backup or data mining. This technology, therefore, allows enterprises to leverage their investment in host-based replication to be used for other business processes without jeopardizing their disaster protection plans.

21

APPENDIX A: SCRIPTS TO ADMINISTER SNAPSHOTS Although we used a single snapshot for our tests, Volume Manager supports multiple snapshots for volumes at any given point in time. Volume Manager supports even full data snapshots and hierarchical snapshots of snapshots. For multiple snapshots, an administration interface will be very desirable, because the number of volumes and output of standard commands becomes cluttered because of the number of objects for each snapshot. Nevertheless, the following samples show a possible hierarchy for a multiple snapshot approach. For our purpose, a set of four scripts for each snapshot was implemented in separate sub-directories. This was necessary because the setuid/setgid permission, set to make the Perl scripts executable by the DB2 database administrator, prohibited parsing of information as part of the execution command line. All arguments used for privileged external Perl "system" calls have to be written into the script and, unlike input from files or other commands, cannot be obtained externally.2 In addition to scripts for creating and mounting a snapshot (snap_db and snap_mnt), two more scripts were implemented. The first script unmounts the file systems — snap_umnt — and the second removes the snapshot objects for a possible update — snap_rm. Since the snapshot had to be created for all volumes at the same time, it was not possible to use separate vxsnap commands for each volume inside a scripted loop. Instead, the sparsely documented -d <description_file> made the creation of a fifth file necessary. This file contains the triplets for vxsnap make. The volume snapshot information can be parsed to vxsnap as part of the command line, but for configurations with many volumes, or complex volume names, the UNIX operating system’s maximum character limit for a single command line will likely be exceeded. In addition, the separation between the volume information and vxsnap makes the script easier to read. TAKE A SNAPSHOT #!/opt/VRTSperl/bin/perl # # $Source: /project/db2ed_src/vxdba/repl/snap_db.pl,v $ # $Id: create_mnt_table.pl,v 1.0 2004/02/29 11:11:11 uli Exp $ # # START COPYRIGHT-NOTICE: 2002-2004 # # Copyright 2002-2004 VERITAS Software Corporation. # All rights reserved. # # VERITAS, the VERITAS Logo and all other VERITAS product names and # slogans are trademarks or registered trademarks of VERITAS Software # Corporation. VERITAS and the VERITAS Logo Reg. U.S. Pat. & Tm. Off. # Other product names and/or slogans mentioned herein may be trademarks # or registered trademarks of their respective companies. # # The Licensed Software and Documentation are deemed to be commercial # computer software and commercial computer software documentation as # defined in FAR Sections 12.212 and DFARS Section 227.7202. # # END COPYRIGHT-NOTICE. # This script creates an instant, space-optimized snapshot for volumes # defined by description file $SnapDesc.

2 For more information, see the chapter on security in the Perl documentation at http://www.perl.org/.

22

# Secure environment $ENV{PATH} = "/usr/bin:/usr/sbin"; $ENV{ENV} = ""; $ENV{LANG} = "C"; $ENV{LD_LIBRARY_PATH} = "/usr/lib"; delete @ENV{qw(IFS CDPATH ENV BASH_ENV)}; my $VxSnap = "/usr/sbin/vxsnap"; my $SnapID = "SNAP00"; my $DGroup = "db2repl"; my $RVGName = "db2_rvg"; my $SnapDesc = "/export/home/adcv0vv9/bin/.setuids/$SnapID/snap.desc"; $RetCode = system("$VxSnap -g $DGroup -d $SnapDesc make"); if ($RetCode != 0) { printf("Create of snapshot for volume %s in diskgroup %s returned error code %d\n",$Vol,$DGroup,$RetCode); exit 1; } printf("Created new snapshot %s for diskgroup %s\n", $SnapID, $DGroup); exit 0; DESCRIPTION FILE FOR SNAP source=db03/new=SNAP00-db03/cache=db2cache source=db03log/new=SNAP00-db03log/cache=db2cache source=long4k/new=SNAP00-long4k/cache=db2cache source=data4k/new=SNAP00-data4k/cache=db2cache source=data8k/new=SNAP00-data8k/cache=db2cache source=data16k/new=SNAP00-data16k/cache=db2cache source=data32k/new=SNAP00-data32k/cache=db2cache source=index4k/new=SNAP00-index4k/cache=db2cache source=index8k/new=SNAP00-index8k/cache=db2cache source=index16k/new=SNAP00-index16k/cache=db2cache source=index32k/new=SNAP00-index32k/cache=db2cache source=temp4k/new=SNAP00-temp4k/cache=db2cache source=temp8k/new=SNAP00-temp8k/cache=db2cache source=temp16k/new=SNAP00-temp16k/cache=db2cache source=temp32k/new=SNAP00-temp32k/cache=db2cache COMMAND FILE TO ENABLE SNAP MOUNT BY DATABASE ADMINISTRATOR #!/opt/VRTSperl/bin/perl # # $Source: /project/db2ed_src/vxdba/repl/snap_mnt.pl,v $ # $Id: create_mnt_table.pl,v 1.0 2004/02/29 11:11:11 uli Exp $ #

23

# START COPYRIGHT-NOTICE: 2002-2004 # # Copyright 2002-2004 VERITAS Software Corporation. # All rights reserved. # # VERITAS, the VERITAS Logo and all other VERITAS product names and # slogans are trademarks or registered trademarks of VERITAS Software # Corporation. VERITAS and the VERITAS Logo Reg. U.S. Pat. & Tm. Off. # Other product names and/or slogans mentioned herein may be trademarks # or registered trademarks of their respective companies. # # The Licensed Software and Documentation are deemed to be commercial # computer software and commercial computer software documentation as # defined in FAR Sections 12.212 and DFARS Section 227.7202. # # END COPYRIGHT-NOTICE. # # Secure environment $ENV{PATH} = "/usr/bin:/usr/sbin"; $ENV{LD_LIBRARY_PATH} = "/usr/lib"; $ENV{ENV} = ""; $ENV{"LANG"} = "C"; $ENV{LD_LIBRARY_PATH} = "/usr/lib"; delete @ENV{qw(IFS CDPATH ENV BASH_ENV)}; my $MountFS = "/usr/sbin/mount"; my $Fsck = "/usr/sbin/fsck"; my $SnapID = "SNAP00"; my $DGroup = "db2repl"; @MntTable = (); push(@MntTable,'db03 /db03 vxfs rw,suid,delaylog,largefiles,qio,ioerror= mwdisable'); push(@MntTable,'data16k /db03/data16k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); push(@MntTable,'data32k /db03/data32k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); push(@MntTable,'data4k /db03/data4k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); push(@MntTable,'data8k /db03/data8k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); push(@MntTable,'index16k /db03/index16k vxfs rw,suid,delaylog,largefi les,qio,ioerror=mwdisable'); push(@MntTable,'index32k /db03/index32k vxfs rw,suid,delaylog,largefi les,qio,ioerror=mwdisable'); push(@MntTable,'index4k /db03/index4k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); push(@MntTable,'index8k /db03/index8k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); push(@MntTable,'long4k /db03/long4k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); push(@MntTable,'db03log /db03/svtdbm/NODE0000/SQL00001/SQLOGDIR vxfs rw,suid,

24

delaylog,largefiles,qio,ioerror=mwdisable'); push(@MntTable,'temp16k /db03/temp16k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); push(@MntTable,'temp32k /db03/temp32k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); push(@MntTable,'temp4k /db03/temp4k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); push(@MntTable,'temp8k /db03/temp8k vxfs rw,suid,delaylog,largefiles,qio, ioerror=mwdisable'); for (@MntTable) { chomp; ($Volume,$MntDir,$FSType,$MntOpts) = split; system("$Fsck -F $FSType -y /dev/vx/rdsk/$DGroup/$SnapID-$Volume"); system("$MountFS -F $FSType -o $MntOpts /dev/vx/dsk/$DGroup/$SnapID-$Volume $MntDir"); } exit 0; UTILITY TO ENABLE UMOUNT BY DATABASE ADMINISTRATOR #!/opt/VRTSperl/bin/perl # # $Source: /project/db2ed_src/vxdba/repl/snap_umnt.pl,v $ # $Id: create_mnt_table.pl,v 1.0 2004/02/29 11:11:11 uli Exp $ # # START COPYRIGHT-NOTICE: 2002-2004 # # Copyright 2002-2004 VERITAS Software Corporation. # All rights reserved. # # VERITAS, the VERITAS Logo and all other VERITAS product names and # slogans are trademarks or registered trademarks of VERITAS Software # Corporation. VERITAS and the VERITAS Logo Reg. U.S. Pat. & Tm. Off. # Other product names and/or slogans mentioned herein may be trademarks # or registered trademarks of their respective companies. # # The Licensed Software and Documentation are deemed to be commercial # computer software and commercial computer software documentation as # defined in FAR Sections 12.212 and DFARS Section 227.7202. # # END COPYRIGHT-NOTICE. # # Secure environment $ENV{PATH} = "/usr/bin:/usr/sbin"; $ENV{LD_LIBRARY_PATH} = "/usr/lib"; $ENV{ENV} = ""; $ENV{"LANG"} = "C"; $ENV{LD_LIBRARY_PATH} = "/usr/lib"; delete @ENV{qw(IFS CDPATH ENV BASH_ENV)}; my $UMountFS = "/usr/sbin/umount";

25

my $SnapID = "SNAP00"; my $DGroup = "db2repl"; @MntTable = (); push(@MntTable,'temp8k'); push(@MntTable,'temp4k'); push(@MntTable,'temp32k'); push(@MntTable,'temp16k'); push(@MntTable,'long4k'); push(@MntTable,'index8k'); push(@MntTable,'index4k'); push(@MntTable,'index32k'); push(@MntTable,'index16k'); push(@MntTable,'data8k'); push(@MntTable,'data4k'); push(@MntTable,'data32k'); push(@MntTable,'data16k'); push(@MntTable,'db03log'); push(@MntTable,'db03'); for (@MntTable) { chomp; ($Volume) = split; system("$UMountFS /dev/vx/dsk/$DGroup/$SnapID-$Volume"); $RetCode = $? >> 8; if ($RetCode) { printf("Unmount for volume %s-%s failed with error code %d.\n", $SnapID, $Volume, $RetCode); next; } } exit 0; SCRIPT TO REMOVE A COMPLETE SNAPSHOT FOR ANOTHER UPDATE #!/opt/VRTSperl/bin/perl # # $Source: /project/db2ed_src/vxdba/repl/snap_rm.pl,v $ # $Id: create_mnt_table.pl,v 1.0 2004/02/29 11:11:11 uli Exp $ # # START COPYRIGHT-NOTICE: 2002-2004 # # Copyright 2002-2004 VERITAS Software Corporation. # All rights reserved. # # VERITAS, the VERITAS Logo and all other VERITAS product names and # slogans are trademarks or registered trademarks of VERITAS Software # Corporation. VERITAS and the VERITAS Logo Reg. U.S. Pat. & Tm. Off. # Other product names and/or slogans mentioned herein may be trademarks # or registered trademarks of their respective companies. # # The Licensed Software and Documentation are deemed to be commercial

26

# computer software and commercial computer software documentation as # defined in FAR Sections 12.212 and DFARS Section 227.7202. # # END COPYRIGHT-NOTICE. # # Secure environment $ENV{PATH} = "/usr/bin:/usr/sbin"; $ENV{LD_LIBRARY_PATH} = "/usr/lib"; $ENV{ENV} = ""; $ENV{"LANG"} = "C"; $ENV{LD_LIBRARY_PATH} = "/usr/lib"; delete @ENV{qw(IFS CDPATH ENV BASH_ENV)}; my $VxSnap = "/usr/sbin/vxsnap"; my $VxEdit = "/usr/sbin/vxedit"; $DiskFree = "/usr/bin/df -n"; my $SnapID = "SNAP00"; my $DGroup = "db2repl"; my @RepVols = (); push(@RepVols,'temp8k'); push(@RepVols,'temp4k'); push(@RepVols,'temp32k'); push(@RepVols,'temp16k'); push(@RepVols,'long4k'); push(@RepVols,'index8k'); push(@RepVols,'index4k'); push(@RepVols,'index32k'); push(@RepVols,'index16k'); push(@RepVols,'data8k'); push(@RepVols,'data4k'); push(@RepVols,'data32k'); push(@RepVols,'data16k'); push(@RepVols,'db03log'); push(@RepVols,'db03'); for (@RepVols) { chomp; my ($Volume) = split; $RetCode = system("$DiskFree /dev/vx/dsk/$DGroup/$SnapID-$Volume 1>/dev/null 2>&1"); unless ( $RetCode ) { printf("Volune %s-%s is still mounted - skipping removal of volume.\n", $SnapID, $Volume); next; } system("$VxSnap -g $DGroup dis $SnapID-$Volume"); system("$VxEdit -g $DGroup -f -r rm $SnapID-$Volume"); } exit 0;