How to do a Painless QA DB Refresh

Session 911A

Who we are…..

• Tracy Ferrell – Mindworks Consulting. Senior Basis Administrator assigned to ChevronTexaco for 2.5 years. Originally with IBM and then Accenture- about 11 years with SAP R/3 basis

• Rich Bernat – ChevronTexaco. Senior Basis Administrator. Crack DBA and RMAN expert.

What we do…..

ChevronTexaco Basis Team

• Manage 6 - SAP Production Systems (R/3 4.6B and 4.6C, BW 2.1)• 100GB – 700GB Database sizes• HP-UX, Oracle 8.1.7.4• FI/CO, SD, Purchasing, HR, BW, IS-Oil, MM/PP and PS

Why do we care about DB Refreshes?

• Customers expect us to deliver environments in a narrow time window and they have to be functional .• ChevronTexaco has a fairly strict regression test requirement for all production changes

Why do we care about DB Refreshes?

• All QA environments are refreshed monthly • Project environments are refreshed on demand• Projects create, copy, refresh, restore many times for different project phases• All QA refreshes occur on the same day …6+Tb of data moved over a weekend!!!

What will we talk about ???

DB refresh strategies Assumptions that have been made What can be “saved” from the old QA

database What are the various restore strategies

that can be used What are the tools that can be used What can be automated and how to get

to that point

Data Refresh Strategies

Client Export/Import – works well with small clients

SAP DB Migration tool Automated Tools (Loadrunner, Autotester,

CATT, etc.) – requires maintenance to scripts when changes are made

ALE – works well for master data not so well for transaction data

Full database restore – requires pre and post work, but is complete and is the strategy used by this presentation

Database Restore Strategies

Full DB Restore Strategy is further defined by the backup/restore tools that are being utilized

Full off-line backupsThe easiest method, however requires production downtime (brbackup, rcp, Legato, etc)

On-line backupsNot as straight forward – but can be managed

Third mirror systems

Database Restore Strategies (cont.)

Full DB Restore Strategy is further defined by the backup/restore tools that are being utilized

Oracle RMANThe easiest to manage, however, requires extensive RMAN knowledge and will not be covered in this presentation – catch Rich Bernat’s presentation handouts - 910A – “Reducing Costs of Environment Copies Using Oracle - Recovery Manager” which were presented on Tuesday at 2pm

This QA DB Refresh Strategy

This presentation will cover the strategy of using an off-line backup of the source database.

Slight changes can be made to this process to use an on-line backup

ChevronTexaco had been using this strategy for about 7 years.

Assumptions

QA database (disk) is sized sufficiently to take a full copy of the production/source DB

The existing QA database is up and running. Oracle is installed and at the same

version as production SAP executables exist and are

functioning QA database is not co-located with

production database

Assumptions (cont.)

If refreshing the development database, much more work is required to “preserve” versioning information

User has root authority oraNEW, oraOLD, NEWadm and OLDadm

accounts all have access to target database server

These hints and tips work well for SAP versions 4.0- 4.6 and Oracle 8.1.x

Where are we going????

Gotta have a plan !!

Prep Work DB copy Post WorkPost DB

copywork

Database Refresh Flow

General flow of refresh

Preparation of target server Export of tables that can be preserved Shutdown of target R/3 system Removal of all target system Oracle

datafiles Restore of source datafiles to target server Rename of source database to target

database Import of items that were preserved Clean-up

Where are we??

Gotta have a plan !!

Prep Work DB copy Post WorkPost DB

copy work

Preparation for Refresh

Export items that can be preserved and imported back into the QA database to save manual post configuration steps

SM59 tables RZ12/SMLG tables RZ04 tables User masters and activity groups R/3 Profiles (RZ10) Transport status tables (next transport

number) Other custom tables (ChevronTexaco pulls

custom table that control interfaces)

Preparation for Refresh

Things that should be done before DB restore

Validate that the size of the sapdatas on the target server can accept the sapdatas from the source system

Create control file from source database Modify control file for later use Ensure that an initOLD.ora file exits on

target server Create a new initNEW.ora file or leave the

original one from the target database if sufficient (beware of rollback segments)

Preparation for Refresh (cont.)

Shutdown the target DB Remove all sapdatas, archive files, online

redo log files from the target database Create a link from the “OLD” SID to the

“NEW” SID (for restore purposes and to bring up the “OLD” database to run the create control file script – a key point!)

Where are we??

Gotta have a plan !!

Prep WorkDB

restorePost Work

Post DB copywork

Database Restore

Completely quiesce and shutdown source database

shutdown startup restrict (to ensure all activity is gone) switch log files – with the current option shutdown

Backup and then restore all the datafiles from the source database to the target database with the tool of choice – brrestore, Omniback, ADSM, rcp, etc.

Remember: This method assumes an offline backup of the source database

Where are we??

Gotta have a plan !!

Prep Work DB copy Post WorkPost DB

copywork

Database Rename Work

This is the toughest step!!!

• Startup “OLD” database on new server• Execute create control file from sql• Rename datafiles• Rename logfiles• Execute other SQL based changes – ChevronTexaco at this point manages the printers (server assign and disable), batch jobs, transport system • Backup new control file• Shutdown “New” instance

Database Rename Work (How?)

Startup the “OLD” database on the new server in mount mode

SQL> connect as sysdba; SQL> startup nomount; SQL> @create_control_file.sql SQL> alter database open resetlogs;

Should have new database name at this point

Note: This NEW database has the same DBID as the OLD database.

Altered control file SQL script

Database Rename Work

Shutdown – just for fun and testing Restart to make sure it really worked Backup control file of the NEW database Create new OPS$sidadm account Reset oracle system, sys, sapr3 passwords

if required. Execute other SQL scripts as required

Where are we??

Gotta have a plan !!

Prep Work DB copy Post WorkPost DB

copywork

Post Refresh Work

Startup R/3 Import security

tp import SIDK9nnnnn SID u1 Sync user buffers (from se38 RSUSR405)

Import items that can were preserved SM59 tables RZ12/SMLG tables RZ04 tables R/3 Profiles (RZ10)Transport status tables (next transport

number) Other custom tables

Post Refresh Work (cont)

Remove all batch jobs except housekeeping Fix print server assignment Clean up

New Saplicense with saplicense –install –p <ifile>

Userid access to Unix Permissions on oracle files Remove extraneous Oracle ID’s

(OPS$OLDadm, etc.) ALE re-setup TMS configuration for NEW SID (run SE06

with DB copy option)

Where are we??

DONE!!!!!!!!!

Prep Work DB copy Post WorkPost DB

copywork

Automation Opportunities

ChevronTexaco has automated the entire process

But how can you????

I can’t give you all our scripts, but here are some tips

1. Start small and work you way up2. Start by creating scripts without error

recovery and little automation, i.e., query the person executing the script for the old SID, new SID, new print server name, etc.

Automation Opportunities

4. Modularize, Modularize, modularize – did I say modularize?

5. Group SQL commands in a .sql script that are generic per SID or can be easily modified with a vi command

6. Make a checklist and follow it religiously for each refresh!!!!!

Modified Control file (SID refers to the target)

Hints….

CREATE CONTROLFILE REUSE REUSE DATABASE “NEWSID" RESETLOGS ARCHIVELOG MAXLOGFILES 255 MAXLOGMEMBERS 3 MAXDATAFILES 1000 MAXINSTANCES 50 MAXLOGHISTORY 9302LOGFILE GROUP 1 ( '/oracle/SID/sapdata1/log_1/loga.dbf', '/oracle/SID/sapdata3/log_1/logb.dbf' ) SIZE 20M, GROUP 2 ( '/oracle/SID/sapdata2/log_2/loga.dbf', '/oracle/SID/sapdata4/log_2/logb.dbf' ) SIZE 20M,DATAFILE '/oracle/SID/sapdata6/system_1/system.data1', '/oracle/SID/sapdata21/roll_1/roll.data1', '/oracle/SID/sapdata24/temp_1/temp.data1', '/….etc '/….etc '/….etc '/oracle/SID/sapdata190/btabd_20/btabd.data20', '/oracle/SID/sapdata190/zza01ai_29/zza01ai.data29', '/oracle/SID/sapdata191/glpcad_31/glpcad.data31'; Key semi-colon

Hints…… (cont.)

Table exports from R/3:

We use R3trans on most of our table exports:

1. Create an R3trans control file2. As the sidadm, execute “R3trans –w logfile

export_r3trans_control_file”

3.Some of the tables are managed with SQL, others with tp (like security)…..this is mostly just historical and most could managed with R3trans.

Hints…… (cont.)

R3trans export control file format:

ExportClient=nnnFile=‘/usr/sap/trans/tmp/export_db_refresh.data’select * from RFCDES select * from RZLLITAB select * from QSENDDEST select * from TVARV select * from TPFET select * from TPFHT select * from TPFID select * from BTCOMSET select * from CCC_CHECK select * from CCCFLOW select * from DBCON

Hints…… (cont.)

R3trans import control file format:

importclient=nnnfile=‘/usr/sap/trans/tmp/export_db_refresh.data’

R3trans command format:

hostname:sidadm 3> cd /usr/sap/trans/tmphostname:sidadm 3> R3trans –w …/log/import_data_nnn.log

import_control_file_name

Hints…… (cont.)

How to find tables that could/should be exported:

Decide what areas need to be manually adjusted after each refresh…From the transaction in R/3, press F1, then F9. This will display the table that contains the field that you have to export/import.Add that table to the R3trans export list.TestAnd more Testing (there are probably more tables)

Note: This is probably not supported by SAP

Hints…… (cont.)

Here are some script samples (all in ksh!):

Prompting for parameters:##################################################################

echo "\n Enter SID of source system: \c" read OLD echo "\n Enter SID of target system: \c" read NEW echo "\nDo you want to delete all scheduled jobs? [y] \c " read answer delete_scheduled_jobs=${answer:-y}

Hints…… (cont.)

Deleting scheduled batch jobs: (put this in a SQL script - it is generic!!!)

##################################################################-- TBTCO: job status overview tabledelete from sapr3.tbtcowhere jobname not like 'SAP_%‘ and jobname not like 'DBA%';-- TBTCS: batch schedule tabledelete from sapr3.tbtcswhere jobname not like 'SAP_%‘ and jobname not like 'DBA%';-- TBTCP: batch log tabledelete from sapr3.tbtcpwhere jobname not like 'SAP%’ and jobname not like 'DBA%';

Hints…… (cont.)

Making sure there is enough space in the sapdatas before attempting the restore

################################################################### Create temporary file that contains the source systems list of sapdatasremsh $source -l ora$old -n "bdf -l | grep OLD | grep sapdata" > /tmp/OLD_sapdatas

2> /dev/null

# Compare with target sapdatasfor i in /oracle/SID/sapdata* do new_sapdata=`basename $i` free_space=`bdf -l | grep ${i}$ | awk '{print $2}'` needed_space=`grep ${new_sapdata}$ /tmp/OLD_sapdatas | awk '{print $3}'` if [[ $needed_space -gt $free_space ]] then echo sapdata $i needs $needed_space more space fidone

Hints…… (cont.)

Creating a SQL on the fly that can be used to rename the datafiles:

###########################################################################spool &3/chglogs.sql

select 1 orderit, 'alter database rename file ' from dual unionselect 2 orderit, ''''||member||''''||',' from v$logfile unionselect 3 orderit, 'to' from dual unionselect 4 orderit, ''''||rtrim(replace(member,upper('&&1'),upper('&&2')))||''''||',' from

v$logfile order by 1;

spool off;

Note: This chglogs.sql must be modified to make it executable……see next slide…..

Hints…… (cont.)

Sample SQL to rename log files:

###########################################################################alter database rename file'/oracle/oldSID/origlogA/log_g11m1.dbf','/oracle/oldSID/origlogA/log_g13m1.dbf','/oracle/oldSID/origlogB/log_g12m1.dbf','/oracle/oldSID/origlogB/log_g14m1.dbf',to'/oracle/newSID/origlogA/log_g11m1.dbf','/oracle/newSID/origlogA/log_g13m1.dbf','/oracle/newSID/origlogB/log_g12m1.dbf','/oracle/newSID/origlogB/log_g14m1.dbf‘;

Hints…… (cont.)

Renaming log files:

dbshutdbstart mountremsh localhost -l ora$new -n "\$ORACLE_HOME/bin/svrmgrl <<EOF connect internal; --- change log file references @${NEW}/sql/chglogs.sql --- change file references @${NEW}/sql/chgfiles.sql alter database open; --- reset transport number @${NEW}/sql/tadir.sql – updates header record in tadir exitEOF" >> /dev/null 1>$SQLERR_LOG_FILE 2>&1

Hints…… (cont.)

Creating a SQL on the fly that reflects the new SID and then can then be executed:

###########################################################################echo "\n Generating the change_printer_profiles.sql script ...\n“

echo "connect internal;" > ${NEW}/sql/change_printer.sqlecho "update sapr3.tsp03c set pamsserver='${new_print_server}' where pamsserver='$

{old_print_server};" >> ${NEW}/sql/change_printer.sqlecho "update sapr3.tsp03d set pamsserver='${new_print_server}’ where pamsserver='$

{old_print_server}';" >> ${NEW}/sql/change_printer_profiles.sqlecho "commit;" >> ${NEW}/sql/change_printer.sql

Hints…… (cont.)

This the script can then be executed as :

##################################################################remsh localhost -l ora$new -n "\$ORACLE_HOME/bin/svrmgrl <<EOF connect internal; --- change printer profiles @${NEW}/sql/change_printer_profiles.sql --- backup controlfile alter database backup controlfile to trace;EOF" >> /dev/null 1>$SQLERR_LOG_FILE 2>&1

Created on previous slide dynamically

Hints…… (cont.)

Interesting OSS Notes :

Note #549828 – Database Copy with an Online BackupNote #179149 – Inst 4.6B R/3 Inst. Homogeneous System Copy – install new system with DBload from another systemNote #147243 – R3Copy under NT Oracle – install new system with DB copy – talks about the control file work that needs to be done.

QUESTIONS ???????

Thank you for attending!Please remember to complete and return your evaluation form following this session.

Session Code: 911A