2559464 Database Shutdown Issue

Issue .......................................................................................................................................................... 2 SQL to check database before Shutdown database .................................................................................. 2 Oracle.Metalink.com document ............................................................................................................... 2

1076161.6:-Shutdown Normal or Shutdown Immediate Hangs. SMON disabling TX Recovery ...................................................................................................................................................... 3 375935.1:- What To Do and Not To Do When 'shutdown immediate' Hangs ............................. 5

Applies to: ............................................................................................................................... 6 Goal .......................................................................................................................................... 6 Solution .................................................................................................................................... 6 References ................................................................................................................................ 8 Errors ........................................................................................................................................ 8

Bug:5057695: Shutdown Immediate Very Slow To Close Database. .......................................... 8 Applies to: ............................................................................................................................... 8 Symptoms ................................................................................................................................ 8 Changes .................................................................................................................................... 8 Cause ........................................................................................................................................ 9 Solution .................................................................................................................................... 9

61997.1: SMON - Temporary Segment Cleanup and Free Space Coalescing ........................... 10 100054.1: Transaction Rollback after a failed operation or during Database Shutdown .......... 15

Conversation between me and Oracle.Metalink.com document ........................................................... 16 More information .................................................................................................................................... 45

1

IssueTO SHUTDOWN DATABASE IT IS TAKING LONG TIME, SOMETIME MORE THAN 2 HOURS. Frequently we are facing this problem. Some time to take shutdown it takes 10 min .One days we have got one error that is " ORA-01013: user requested cancel of current operation" time of shutdown database

SQL to check database before Shutdown database

To make database shutdown faster please run SQL before shutdown database.and see the status of database

1. All active session======================If Active session is exist then, try to find out what is doing in the database by this session. Active session make shutdown slower

select sid, serial#, username, status, schemaname, logon_time from v$session where status='ACTIVE' and username is not null

2.Temporary Tablespace Recover================================Sea any session used Temporary tablespace for sorting . For this also may shutdown slower

Select f.R "Recovered", u.nr "Need Recovered" from (select count(block#) R , 1 ch from sys.fet$ ) f,(select count(block#) NR, 1 ch from sys.uet$) u where f.ch=u.ch

3. Long Running Query=======================select * from v$session_longops where time_remaining>0 order by username

4. Large Transactions========================select sum(used_ublk) from v$transaction

5. Progress of the transactions that Oracle is recovering.========================================================select * from v$fast_start_transactions

6. Parallel transaction recovery==================================select * from v$fast_start_servers

Oracle.Metalink.com document 1 1076161.6:-Shutdown Normal or Shutdown Immediate Hangs. SMON disabling

TX Recovery

2 Note 375935.1 - What To Do and Not To Do When 'shutdown immediate' Hangs

3 Note 428688.1 Bug 5057695: Shutdown Immediate Very Slow To Close Database.

2

https://metalink.oracle.com/metalink/plsql/showdoc?db=Bug&id=5057695

https://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=428688.1&blackframe=1


4 Note 61997.1 SMON Temporary Segment Cleanup and Free Space Coalescing

5 100054.1 Subject: Transaction Rollback after a failed operation or during Database Shutdown

1076161.6:-Shutdown Normal or Shutdown Immediate Hangs. SMON disabling TX Recovery

Subject: Shutdown Normal or Shutdown Immediate Hangs. SMON disabling TX Recovery

Doc ID: Note:1076161.6 Type: BULLETIN

Last Revision Date: 21-DEC-2007 Status: PUBLISHED

Description===========SHUTDOWN NORMAL or SHUTDOWN IMMEDIATE hangs. In the alert.log, you see only the following:

Shutting down instance (immediate)License high water mark = 12Thu Dec 8 18:43:16 1994alter database close normalThu Dec 8 18:43:17 1994SMON: disabling tx recoverySMON: disabling cache recovery

or waiting for smon to disable tx recovery

There are no ORA errors or trace files.

Scope & Application===================Informational

During a SHUTDOWN IMMEDIATE and SHUTDOWN NORMAL, SMON is cleaning up extents which are no longer needed and marking them as freed.

Either wait for SMON to clean up the free extents in the database as it shuts down or perform a SHUTDOWN ABORT to shutdown the instance. A SHUTDOWN ABORT will not perform a clean shutdown.

Verify that temporary segments are decreasingTo verify that the temporary segments are decreasing have an active session available in Server Manager or SQLPLUS during the SHUTDOWN IMMEDIATE. Issue

3

https://metalink.oracle.com/help/usaeng/Search/search.html#file


the following query to ensure the database is not hanging, but is actually perform extent cleanup:

SVRMGR/SQL> select count(block#) from fet$; COUNT(BLOC 7

SVRMGR/SQL> select count(block#) from uet$; COUNT(BLOC 402

After some time has elapsed, reissue the query and see that the values for fet$ have increased while the values or uet$ have decreased:

SVRMGR/SQL> select count(block#) from fet$; COUNT(BLOC 10

SVRMGR/SQL> select count(block#) from uet$; COUNT(BLOC 399

select f.R "Recovered", u.nr "Need Recovered" from (select count(block#) R , 1 ch from fet$ ) f,(select count(block#) NR, 1 ch from uet$) u where f.ch=u.ch

During shutdown the SMON process is cleaning up extents and updating the data dictionary tables with the marked free extents. As the extents are marked as freed, they are removed from the table for used extents, UET$ and placed on the table for free extents, FET$.

How to Avoid creating many Temporary ExtentsOnce the database has shutdown cleanly, to avoid creating many temporary extents change the initial and next extent sizes on temporary tablespaces to a more appropriate size:

ALTER TABLESPACE <temp> DEFAULT STORAGE (INITIAL <size>M/K NEXT <size>M/K); Note: If the temporary tablespace is of type TEMPORARY, then this change will only affect temporary segments created after issuing the above command. Any existing temporary segments already in the TEMPORARY tablespace will not be affected till the instance is restarted. On shutdown, existing temporary segments are dropped. If the TEMPORARY TABLESPACE is of type PERMANENT, then cleanup is performed by SMON after completion of the process using it.

4

Increasing the initial and next extent size will decrease the number of extents that are allocated to temporary segments. Since there are fewer extents to deallocate, the database should shutdown more speedily.

Take the following scenario:

A database was subject to large sorts with the following sort parameter in the "init.ora" file: sort_area_size=1000000 The temporary tablespaces for this database were all created with initial and next extents sized at 50k and the total database size was about 300mb. Database sorts will utilize memory as much as possible based on the "init.ora" parameter "sort_area_size". Once this memorybased sort area is filled, the database will utilize the temporary table space associated with the database user to complete the sort operation. During a shutdown normal, the database will attempt to clean up the temporary tablespaces. If a small extent size is used, then a large number of extents will be created for a large sort. The cleanup of the temporary tablespace takes much longer with a large number of extents.

Note:=====You have to do a shutdown abort and then bring the database back up to run the suggested queries.

For other reasons for slow/hung shutdown see also these notes:

Note 375935.1 What To Do and Not To Do When 'shutdown immediate' HangsNote 428688.1 Bug 5057695: Shutdown Immediate Very Slow To Close Database. References:===========Note 61997.1 SMON Temporary Segment Cleanup and Free Space Coalescing

Search Words:=============hangingshutdown

375935.1:- What To Do and Not To Do When 'shutdown immediate' Hangs

Subject: What To Do and Not To Do When 'shutdown immediate' Hangs Doc ID: Note:375935.1 Type: HOWTO

5






Last Revision

Date: 02-JUN-2007

Status: MODERATED

This document is being delivered to you via Oracle Support's Rapid Visibility (RaV) Rapid Visibility (RaV) process, and therefore has not been subject to an independent technical review.

Applies to: Oracle Server - Enterprise Edition - Version: 8.1.7 to 10.2Information in this document applies to any platform.

GoalWhat to do when shutdown immediate appears to hang. Sometimes, the message 'Waiting for smon to disable tx recovery' is posted in the alert log. This note only addresses situations when the apparent hang occurs when the database is going from OPEN to MOUNT, which is actually the most common situation. If the apparent hang occurs at a different step, then this note does not apply.

SolutionThe big problem in these situations is that it is noticed only after the shutdown immediate has been issued.This kind of situation is mostly caused by 2 things:1. a large query running at the shutdown moment.2. a large transaction running at the shutdown moment.

Both have to complete in order for the database to be brought down when shutdown immediate is issued. Actually, the files cannot be closed consistently because of one of the 2 possibilities above and, as such, the transition from OPEN to MOUNT is postponed until the files are closed, which means that either the large query completes or the large transaction is rolled back. This is not a hang, it is the expected behavior.

So, before issuing the shutdown immediate, it would be recommended to check the following views, especially when the database needs to be brought down for a very short period of time:1. for large queries:select count(*) from v$session_longops where time_remaining>0;

2. for large transactions:select sum(used_ublk) from v$transaction;

A result greater than 0 for the first query and a large value returned for the second one would mean a relatively long time to wait until the shutdown immediate completes.For the second situation, please also check step 9 in Note 117316.1 to "guestimate" the time to rollback the transactions.

-----------------------------------------------------------Step 9 from Note 117316.1 Check for rollback:

6



https://metalink.oracle.com/metalink/plsql/f?p=130:14:2251996494132700064::::p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,375935.1,1,0,1,helvetica

select used_ublk from v$transaction where ADDR=<value from TADDR in v$session>;

If there is a value there, this is the number of undo blocks used by the transaction. Wait one minute and again select "used_ublk" from "v$transaction" where ADDR=<value from TADDR in v$session>; . Note the value. If it is decreasing, a rollback is occuring and based on the difference between these values, you can "guesstimate" the time required to complete the rollback. For example, if the first query returns a value of 80000 and the second one returns 70000, it took 1 minute to rollback 10000 blocks. Based on this number,you can guestimate the time to complete the rollback. In this case, it would be 7 minutes.--------------------------------------------------------------------------

1. For the large queries situation, when the shutdown immediate is hanging, you can just bring down the database using: shutdown abort, as the database could be easily brought to a consistent state by:startup restrict followed by shutdown immediate. One should take the backup and/or do whatever else need to be done after the shutdown immediate.2. For the second situation, the workaround cannot be applied, especially when it's needed to take a cold backup. The database must be closed in a consistent state in order to do this and the consistent state cannot be achieved until all the transactions have completed one way or another (commit/rollback).

As such, it's up to the local personnel to decide what to do, depending on the local needs.It is very important to realize that: BY SHUTTING DOWN A DATABASE YOU DO NOT SOLVE A PERFORMANCE PROBLEM CAUSED BY A LARGE TRANSACTION. You are only making things worse.There are situations when the database is brought down even when a large transaction/large recovery is taking place. Then it's brought up again and a new shutodwn is tried. Again, the shutdown immediate is hanging, for a very simple reason - the large recovery is still going on.At this moment, the v$transaction view is not displaying anything.Hoever, it is still possible to check the recovery operation by checking the:select * from v$fast_start_transactions;and/orselect * from v$fast_start_servers;

views. They are the ones that display the recovery status.As such, when a large transaction is taking place, do not try successive shutdown aborts, startups and shutdown immediate. The hang will reoccur. The database must be consistent when the database is dismounted - performing successive shutdowns/startups is not helping at all, it's only making the recovery even more lengthy.

You should prevent these situations by notifying the users a shutdown will be done and no large operations should be started.If a large operation has already started at the moment when you want to shutdown immedate,

7

assess what would be faster - rollback the current situation or allow it to complete.

ReferencesNote 117316.1 - ORA-0054: When Dropping or Truncating Table, When Creating or Rebuilding Index

ErrorsORA-54 "resource busy and acquire with NOWAIT specified"

Bug:5057695: Shutdown Immediate Very Slow To Close Database.

Subject: Bug:5057695: Shutdown Immediate Very Slow To Close Database.

Doc ID: Note:428688.1 Type: PROBLEM Last Revision Date: 24-DEC-2007 Status: PUBLISHED

Applies to: Oracle Server - Enterprise Edition - Version: 9.2.0.8 to 10.2.0.3 This problem can occur on any platform. Oracle Database Server

Symptoms

Recently upgraded the database to versions between 9.2.0.7 to 10.2.0.3

Database with 150 or so connections take a very long time to shutdown, compared to the same system when using Oracle 9i (9.2.0.6 and prior) where shutdown was very fast.

Given the fact there is no activity on the connections in question, so rollbacks aren't an issue.

There are no trace files and no indication of a problem in the alert log.

Changes

Recently upgraded the database to versions between 9.2.0.7 to 10.2.0.3

8


https://metalink.oracle.com/metalink/plsql/ml2_gui.handleSearchRequest?p_search=Submit&p_text=ORA-54


Cause

SHUTDOWN IMMEDIATE can be slower in 9.2.0.8 and 10.2.0.1/10.2.0.2/10.2.0.3 than in earlier releases due to a change in how processes are terminated introduced in these releases.

Previously the shutdown immediate, the ksukia function would kill each individual user session and check in intervals of 5 seconds that it had died before progressing to the next user.

With fix for bug 5080775 (Unpublished) this time was reduced so that instead of waiting for 5 seconds it now waited 0.05 seconds and this would be repeated 40 times at intervals of 0.05 seconds then it would increase to checking every 5 seconds.

With large numbers of users this could involve a lengthy wait for shutdown that needed to be reduced.

Note this code is relatively new. It was introduced in 9.2.0.7 and 10.2. It was a design change where previously it issues a kill (skgpkill) and then proceeds onto the next process.

Solution

Workaround : Wait for the shutdown to finish.

Solution :

Apply Patch 5057695 available on Metalink on top of 9.2.0.8, 10.2.0.2 or 10.2.0.3

Fix included with 10.2.0.4 PatchsetOne-off patches are available for most platforms, If a patch is not available for your platform, please contact Oracle Support.

9

61997.1: SMON - Temporary Segment Cleanup and Free Space Coalescing

Subject: SMON - Temporary Segment Cleanup and Free Space Coalescing

Doc ID: Note:61997.1 Type: FAQ Last Revision Date: 22-APR-2004 Status: PUBLISHED

SMON Temporary Segment Cleanup and Free Space Coalescing in Oracle 7.3 and Higher

PURPOSE~~~~~~~

Since the introduction of the unlimited extents feature in Oracle 7.3, it is possible for SMON to have to either clean up a large number of temporary extents, or to coalesce a large number of free extents. This can manifest itself by SMON appearing to spin, consuming a high percentage of CPU for long periods. This article explains what is happening, and what (if anything) can be done.

The discussion concentrates mainly on nonTEMPORARY type tablespaces. There is however a section at the end of the article which discusses possible issues with tablespaces of type TEMPORARY.

SCOPE & APPLICATION~~~~~~~~~~~~~~~~~~~ This article is intended to assist DBAs encountering SMON appearing to spin and consume high percentages of CPU by providing an understanding of the issues.

RELATED DOCUMENTS~~~~~~~~~~~~~~~~~ Note 35513.1 Removing `stray` TEMPORARY Segments Note 50592.1 Extent Sizes for Sort, Direct Load and Parallel Operations Note 65973.1 Temporary Tablespaces and the Sort Extent Pool Note 68836.1 How to efficiently drop a table with many extents Note 47400.1 EVENT: DROP_SEGMENTS Forcing cleanup of TEMPORARY segmentsWhat to look for~~~~~~~~~~~~~~~~ The most common indicator is the SMON process consuming large amounts of CPU for a long period. UNIX O/S utilities sar or vmstat will show how busy CPU(s) are; ps will show which process is using the CPU.

What is SMON doing~~~~~~~~~~~~~~~~~~ Once you have identified that SMON is using lots of CPU, you need to identify whether it is performing temporary segment (extent) cleanup, or free space coalescing.

10







Free space coalescing~~~~~~~~~~~~~~~~~~~~~ When does SMON coalesce?

o. SMON wakes itself every 5 minutes and checks for tablespaces with default pctincrease != 0.

How to identify whether SMON is coalescing

o. Check whether there are a large number of free extents that might be being coalesced by running the following query a few times:

SELECT COUNT(*) FROM DBA_FREE_SPACE;

If the count returned is dropping while SMON is working, it is likely that SMON is coalescing free space.

What are the effects on the database?

o. Because SMON acquires the Space Transaction (ST) enqueue in exclusive mode, other processes requiring the enqueue will be blocked. This is typically manifested by multiple <oerr:ORA1575> errors. o. SMON sits in a very tight loop while coalescing, and consumes close to 100% CPU. If the system is CPUbound, the run queue will increase as other processes try to get onto CPU.

Can anything be done to stop SMON grabbing CPU?

o. If there is no CPU contention, and no processes being blocked because of failure to acquire the ST enqueue, DO NOT DO ANYTHING. Leave SMON to complete the coalescing.

THE DATABASE CAN BE SHUTDOWN CLEANLY WITH UNCOALESCED EXTENTS. If SMON is performing the coalesce, a shutdown will NOT undo the work completed so far.

o. Use the 'alter tablespace <tbs name> coalesce' command. This is quicker than SMON, and the work is performed in in fewer space transactions, and therefore makes fewer enqeueue acquisitions. HOWEVER, IF THE COMMAND IS INTERRUPTED, ALL ITS COALESCING WORK WILL BE LOST.

o. It is possible to force a user session to coalesce free extents. See Note 35513.1 "Removing `stray` TEMPORARY Segments" for details. Again, quicker than SMON. HOWEVER, IF THIS OPERATION IS INTERRUPTED, ALL IT'S COALESCING WORK WILL BE LOST.

o. Offlining the tablespace/datafiles containing the extents to be coalesced has NO effect.

Temporary segment cleanup

11


~~~~~~~~~~~~~~~~~~~~~~~~~

When does SMON cleanup temporary segments?

o. Typically a user process allocates a temporary segment (multiple extents) and then dies before cleaning them up, or the user process receives an error causing the statement to fail. SMON is posted to do the cleanup. SMON also might get tied up cleaning uncommitted transactions though, and be too busy to process requests to grow an existing sort segment. Starting with Oracle 8i, playing around with fast_start_parallel_rollback might workaround that. In addition, actions like CREATE INDEX create a temporary segment for the index, and only convert it to permanent once the index has been created. Also, DROP <object> converts the segment to temporary and then cleans up the temporary segment.

o. During normal operations, user processes that create temporary segments are responsible for cleanup.

How to identify whether SMON is cleaning up temporary extents

o. Check whether there are a large number of temporary extents that might be being cleaned up by running the following query a few times:

SELECT COUNT(*) FROM DBA_EXTENTS WHERE SEGMENT_TYPE='TEMPORARY';

If the count returned by the above query is dropping while SMON is working, it is likely that SMON is performing temp segment cleanup. See section 'Tablespaces of type TEMPORARY' for more details on this.

What are the effects on the database?

o. Again, SMON will continually acquire and then release the ST enqueue in exclusive mode. This can cause contention with other processes and lead to <oerr:ORA1575> errors.

o. CPU utilization is not exceptionally high. During tests, SMON consumed between 10% and 20% CPU during cleanup, and so this operation has less impact than coalescing, as far as SMON is concerned. Furthermore, SMON performed the cleanup in 'chunks', cleaning up a subset of the extents at a time.

Can anything be done to stop SMON grabbing CPU?

o. Not a great deal. As with coalescing, if there is no CPU contention, and no processes being blocked because of failure to acquire the ST enqueue, DO NOT DO ANYTHING. However because SMON does not work as hard cleaning up temporary extents, it should not be a big issue. Note: If you are using TEMPORARY type temporary tablespaces then SMONs cleanup of the segment can be a problem as it will not

12

service sort segment requests while performing cleanup. See below (TEMPORARY tablespaces) for more information.

It should be noted that a normal/immediate shutdown will not complete until all temporary segments have been cleaned up. Shutdown will 'kick' SMON to complete cleanup.

o. Offlining the tablespace/datafiles in which the extents reside has NO effect.

o. DROP_SEGMENTS event could be set set to force the cleanup of temporary segments, see Note 47400.1 "EVENT: DROP_SEGMENTS Forcing cleanup of TEMPORARY segments".

Avoidance~~~~~~~~~

With a little forethought and care, the above situations can be avoided:

o. Do not create temporary tablespaces with small initial and next default storage parameters. Also beware of unlimited maxextents on temporary tablespaces.

Note, TEMPORARY type tablespaces set maxextents unlimited automatically. Furthermore, the NEXT AND INITIAL extent sizes are determined from the default NEXT size (default INITIAL is ignored). For more details on temporary extent sizes, see Note 50592.1 "Extent Sizes for Sort, Direct Load and Parallel Operations (PCTAS & PDML)".

o. Use tablespaces of type TEMPORARY. Sort segments in these tablespaces are not cleaned up. This reduces contention on the ST enqueue and also reduces CPU usage by SMON **UNLESS** the database is shutdown and restarted. If TEMPORARY type tablespaces are in use then SMON will clean up its segments after startup following a shutdown. In this case large numbers of extents can be a severe problem as SMON will not service user "sort segment requests" until the cleanup is complete. If the cleanup is to take a long time users will not be able to perform sort operations. In this scenario you can point users at a PERMANENT temporary tablespace while SMON cleans up the TEMPORARY temporary tablespace. This is likely to cause ST enqueue contention but will allow users sessions to sort on disk when necessary rather then them just blocking. Eg:

If SMON is busy cleaning up a TEMP segment containing a lot of extents it cannot service 'sort segment requests' from other sessions. Pointing the users at a PERMANENT tablespace as their temporary tablespace can help keep the system running

until SMON is free again:

CREATE TABLESPACE NEWTEMP .... (your own specification here)

13



(DO NOT CREATE IT AS TYPE TEMPORARY)

Move the users over to this:

select username from dba_users where temporary_tablespace='TEMP';

For each user in this list:

alter user XXXXX temporary tablespace NEWTEMP;

Once SMON has cleaned up the extents reset the storage clause on each tablespace to sensible values and you can then point users back at a TEMPORARY temp tablespace.

Starting with Oracle8i, rather than reverting back to a PERMANENT tablespace if SMON is cleaning up an old sort segment at startup, you can potentially drop and recreate the tempfiles of the existing TEMPORARY tablespace. The cleanup should be faster anyway since by rule a TEMPORARY tablespace made of tempfiles need to be LOCALLY MANAGED. You can remove tempfiles from TEMPORARY tablespaces and keep the logical structure empty.

o. Beware of creating large objects with inappropriate (small) extents. If the creation of the object fails, SMON cleans up. Also, dropping such an object will create a lot of cleanup work for the user process.

Oracle8 ONLY. Make use of the tablespace MINIMUM EXTENT size to help minimise the risk of mistakes in scripts causing small extent sizes. This parameter ensures that every used and/or free extent size in a tablespace is at least as large as, and is a multiple of, this value.

Oracle8i ONLY: It is worth considering the use of a locally managed temporary tablespace. This has the benefit of faster temporary segment cleanup after the instance has been aborted. Note, locally managed temporary tablespaces must be created using tempfile(s). Any attempt to create a locally managed temporary tablespace using a datafile will result in the error: ORA25144: invalid option for CREATE TABLESPACE with TEMPORARY contents

Tablespaces of type TEMPORARY~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

TEMPORARYtype tablespaces were introduced in Oracle 7.3 (see Note 65973.1 "Temporary Tablespace, the Sort Extent Pool, and OPS"). In summary:

o. The first disk sort (after instance startup) creates a sort segment in the TEMPORARY tablespace.

o. Free extents in the sort segment are reused as required by sessions.

14


o. The sort segment grows to a steadystate.

o. Sort extents are not deallocated whilst the instance is running.

o. Permanent objects cannot be created in TEMPORARY tablespaces.

o. There is a maximum of one sort segment per TEMPORARY tablespace.

Thus, contention on the ST enqueue is reduced as user sessions are allocating and deallocating fewer extents. Even if a user session dies, SMON will not deallocate extents that the session was using for sorting.

SMON actually deallocates the sort segment after the instance has been started and the database has been opened. Thus, after the database has been opened, SMON may be seen to consume large amounts of CPU as it first deallocates the (extents from the) temporary segment, and then when it is requested to perform free space coalescing of the free extents created by the temporary segment cleanup. Again, this behaviour will be exaggerated if the TEMPORARY tablespace in which the sort segment resides has inappropriate (small) default NEXT storage parameters (see 'Avoidance' above).

Permanent object cleanup~~~~~~~~~~~~~~~~~~~~~~~~

If a permanent object is made up of many extents, and the object is to be dropped, the user process dropping the object will consume large amounts of CPU in the same way as SMON does when cleaning up a temporary segment. Please see Note 68836.1 "How To Efficiently Drop A Table With Many Extents" for a discussion of this.

References~~~~~~~~~~

100054.1: Transaction Rollback after a failed operation or during Database Shutdown

Subject: Transaction Rollback after a failed operation or during Database Shutdown

Doc ID: Note:100054.1 Type: BULLETIN

Last Revision

Date: 17-JAN-2006

Status: PUBLISHED

Problem Description

To explain what happens to uncommitted transactions after a failed or abortedoperation (i.e. long running transaction fails) or caused by a Database Shutdowncommand being issued.

15



Problem Explanation If a long running transaction (indeed any transaction) is cancelled or fails, then Oracle needs to cleanup the uncommitted work that was done by this transactionso that other transactions can progress. This cleanup involves rolling backthe uncommitted work.

If a database shutdown is issued with outstanding transactions to be rolledback, then the database will not be shutdown until these transactions have completed or have been rolled back.

For example on a shutdown immediate we send a message to all the shadow processesasking them to complete this rollback and cleanup. We will not shutdown until all of them have completed this cleanup. This means that because we have shutdowncleanly no redo needs to be applied on startup.

If the database is aborted with outstanding transactions then these transactionsmust be rolled back after the database has been restarted. Post 7.3 this transaction recovery is applied by SMON after the database has been opened. Ifanother transaction needs to access blocks that were modified by the transactionthat is rolling back then the new session will rollback the blocks that it requires itself. This can cause these transactions to perform slower than normal.The activity of SMON has been designed so as to minimize the impact on otherprocesses by only rolling back a small proportion of rollback entries at a time.

References

Bug 517917 SHUTDOWN IMMEDIATE HANGS SMON DOING TX RECOVERYNote 61997.1 SMON Temporary Segment Cleanup and Free Space Coalescing

Conversation between me and Oracle.Metalink.com document

16



SR Number 6779932.992 Open Date 19-FEB-08 04:30:01Support Identifier 15272898 Name mamun serajiSeverity 3 Last Update 17-APR-08 07:02:57Product Oracle Server - Enterprise

Edition Product Version 9.2.0.8.0

Platform AIX5L Based Systems (64-bit) Detailed Status Soft CloseSR Reference n/a BUG Reference n/aAbstractTO SHUTDOWN DATABASE IT IS TAKING LONG TIME, SOMETIME MORE THAN 2 HOURS. Resolution History

19-FEB-08 04:30:01 GMT

This SR was saved as a Draft SR on: 18-FEB-2008 20:19:00

### Introduction to the Database Performance Issues iTAR Questions ###N/A

### Is your system completely hung or is just performing slow? ###Slow

### Please describe any recent change in your environment? ###During night so many batch job running in our database. That time we have checked alert log file and found "ORA-01555: causedby SQL statement below (Query Duration=8000)". Then we have changed undo_retention parameter value. The old value was 6500 and the new value is 8000.

### When was the last time you performed a tuning process to your database? ###Frequently we are facing this problem. some time to take shutdown it take 10 min.

### When did you start to notice slow performance? ###From beginning we are facing this problem.

### Can you consistently reproduce the problem? ###No

### Is the problem related to heavy load hours only? ###No

### Are you getting any kind of error messages? ###One day we have got one error that is " ORA-01013: user requested cancel of current operation"

17

### Please list all files that you plan to upload: ###if need we will send

Can you easily recover from, bypass or work around the problem? Yes

Does your system or application continue normally after the problem occurs? Yes

Are the standard features of the system or application still available; is the loss of service minor?No

Error : ORA-01013Contact me via : MetaLink

19-FEB-08 09:47:01 GMT

UPDATE======Hi Mamun,

Thank you for using MetaLink. We are currently reviewing/researching the situation and will update the Service Request (SR) or call youas soon as we have relevant information. Thank you for your patience.

Best Regards,BharathGlobal Customer Services

STATUS=======

@WIP -- Work In Progress

19-FEB-08 09:51:14 GMT

.UPDATE=======Hi Mamun,

18

Can you please upload the Alert_log file which contains the last 5-6 shutdown information logged

1) Were there any transactions that were cancelled by doing shutdown immediate.

Regards,Bharath Sunku

STATUS=======

@CUS -- Waiting for requested information

19-FEB-08 09:51:26 GMT

Email Update button has been pressed: Sending email to [email protected].

25-FEB-08 05:29:02 GMT

The customer : [email protected] : has uploaded the following file via MetaLink:Alert_log.txt

25-FEB-08 05:34:01 GMT

New info : [email protected] Mr Bharath SunkuSorry for delay. i was out of office. There was no transaction canceled from back-end at the time of shutdown.

Thanks and regardsMd Ruhul Amin

25-FEB-08 12:18:38 GMT

DATA COLLECTED

19

============

Fri Feb 15 01:50:39 2008Shutting down instance (immediate)License high water mark = 1327Fri Feb 15 01:55:04 2008ALTER DATABASE CLOSE NORMALFri Feb 15 01:55:04 2008SMON: disabling tx recoverySMON: disabling cache recovery

Sun Feb 17 02:06:04 2008Shutting down instance (immediate)License high water mark = 858Sun Feb 17 02:07:39 2008ALTER DATABASE CLOSE NORMALSun Feb 17 02:07:39 2008SMON: disabling tx recoverySMON: disabling cache recoverySun Feb 17 02:07:39 2008Shutting down archive processes

Mon Feb 18 00:56:29 2008Shutting down instance: further logons disabledShutting down instance (immediate)License high water mark = 1762Mon Feb 18 02:52:10 2008ALTER DATABASE CLOSE NORMALMon Feb 18 02:52:10 2008SMON: disabling tx recoverySMON: disabling cache recoveryMon Feb 18 02:52:13 2008Shutting down archive processes

Shutting down instance (immediate)License high water mark = 2430Tue Feb 19 01:53:47 2008ALTER DATABASE CLOSE NORMALTue Feb 19 01:53:48 2008SMON: disabling tx recoverySMON: disabling cache recoveryTue Feb 19 01:53:52 2008Shutting down archive processesArchiving is disabledTue Feb 19 01:53:52 2008

20

Wed Feb 20 00:33:51 2008Shutting down instance (immediate)License high water mark = 1905Wed Feb 20 00:59:31 2008ALTER DATABASE CLOSE NORMALWed Feb 20 00:59:31 2008SMON: disabling tx recoverySMON: disabling cache recoveryWed Feb 20 00:59:33 2008Shutting down archive processesArchiving is disabled

Thu Feb 21 01:04:52 2008Shutting down instance (immediate)License high water mark = 2062Thu Feb 21 01:07:22 2008ALTER DATABASE CLOSE NORMALThu Feb 21 01:07:22 2008SMON: disabling tx recoverySMON: disabling cache recoveryThu Feb 21 01:07:24 2008Shutting down archive processesArchiving is disabled

Thu Feb 21 17:45:07 2008Shutting down instance: further logons disabledShutting down instance (immediate)License high water mark = 376Thu Feb 21 17:46:18 2008ALTER DATABASE CLOSE NORMALThu Feb 21 17:46:18 2008SMON: disabling tx recoverySMON: disabling cache recoveryThu Feb 21 17:46:18 2008Shutting down archive processes

Completed checkpoint up to RBA [0x6aea.2.10], SCN: 0x0002.831b5524Sun Feb 24 00:44:21 2008Shutting down instance (immediate)License high water mark = 618Sun Feb 24 00:46:37 2008ALTER DATABASE CLOSE NORMALSun Feb 24 00:46:37 2008SMON: disabling tx recoverySMON: disabling cache recovery

21

Sun Feb 24 00:46:37 2008Shutting down archive processesArchiving is disabled

.UPDATE========Hi Mamun,

Thanks for the UpdateWe have checked the Alert_log file and found the below

1) Shutdown immediate is not taking 2 hours everytime its executed2) In all the instances of shutdown slow, only Transaction recovery is counsuming 99% of the time.

As you have also specified that there are some Batch Jobs which are running in the Night and we suspect they might have got cancelled due to "Shut immediate" which is consuminglong time for transaction recovery

Please let me know if you need any other clarifications on this issue

RegardsBharath Sunku

25-FEB-08 12:18:49 GMT


25-FEB-08 12:56:02 GMT

New info : [email protected] Bharath Sunku

Thank you for fast replyI need two clarification

1) How you came to know that Transaction recovery is consuming 99% of the time ?2) is there any way to shutdown database at short time ?

Regards

22

Md Ruhul Amin

25-FEB-08 13:11:55 GMT

.UPDATE=======Hi Ruhul,

Thanks for the Update

1) Please see the below~~~~~~~~~~~~~~~~~~~~~~~Mon Feb 18 00:56:29 2008Shutting down instance: further logons disabledShutting down instance (immediate)License high water mark = 1762Mon Feb 18 02:52:10 2008ALTER DATABASE CLOSE NORMALMon Feb 18 02:52:10 2008SMON: disabling tx recovery~~~~~~~~~~~~~~~~~~~~~~~~

From the Above we see that the Shutdown was performed at 00:56 and 02:52 tx recovery was completed..

Please check my Data Collected Tag.. you will find that the 99% of the time is consumed at this step and after that its hardly taking 3-5 minutes to shutdown the database

2) Is the Shutdown completed in short time?

~~~~~~~Thu Feb 21 17:45:07 2008Shutting down instance: further logons disabledShutting down instance (immediate)License high water mark = 376Thu Feb 21 17:46:18 2008ALTER DATABASE CLOSE NORMALThu Feb 21 17:46:18 2008SMON: disabling tx recoverySMON: disabling cache recoveryThu Feb 21 17:46:18 2008

23

Shutting down archive processes~~~~~~~~~~~~Shutdown was executed at 17:45 and 17:46 is the tx recovery completed and immediatly database got shutdown



STATUS=======


25-FEB-08 13:12:06 GMT


25-FEB-08 13:33:06 GMT

New info : [email protected] Bharath SunkuThank you for your nice reply. i am very much happy for this

can u please tell me how can i came to know that today shutdown will take more time. Please arrange sometime to talk over chat.

regardsMd Ruhul Amin

25-FEB-08 14:23:21 GMT

.

24

UPDATE=======Hi Ruhul,

Time to shutdown depends on the operations being performed on the database before you do a ShutdownIf there are any transaction happening on the database and if the SHUT IMMEDIATE is performed which is going to terminate the transactions and then do the Rollback, in this case depending on the time of transaction recovery your shoudown will be depended


My Shift time is from 1:30 AM GMT to 10:00 AM GMTPlease let me at what time you want me to start an Web-Conference.. If you want this issue to be addressed after my Shift time.. Pleasecall your Local HUB from the below Link so that This SR will be rassaigned to an available engineer and he will help you with the issue

http://www.oracle.com/corporate/contact/index.html


STATUS=======


25-FEB-08 14:23:27 GMT


25-FEB-08 14:26:06 GMT


25-FEB-08 14:58:04 GMT

New info : [email protected] Bharath Sunkui will be available from 4.30 am GMT to 10.30 am GMT

25

Regardsmd Ruhul Amin

25-FEB-08 15:04:12 GMT


Will update the SR / Call you so that we can further progress the issue

Thanks,Bharath

STATUS=======


25-FEB-08 15:04:18 GMT


25-FEB-08 16:06:18 GMT

New info : [email protected] Bharath SunkuAny un-commited transaction will be rollback during shutdown database. if there is huge un-commited transaction exists than shut down will take much time. so my concern is there any way to find how many transaction is un-commited now, which need to be roll back at the time of shutdown database.

Thanks and regardsmd Ruhul Amin

26-FEB-08 05:19:40 GMT

26


Please read the below note to understand more on this issue

Article-ID: Note 1076161.6Title: Shutdown Normal or Shutdown Immediate Hangs. SMON disabling TX Recovery

Thanks,Bharath

STATUS=======


26-FEB-08 05:19:51 GMT


26-FEB-08 07:41:05 GMT

New info : [email protected] Mr Bharath SunkuThank you. Please read this document and give me advise

Subject: Bug:5057695: Shutdown Immediate Very Slow To Close Database.Doc ID: Note:428688.1

ThanksMd Ruhul Amin

26-FEB-08 10:06:45 GMT

.ISSUE CLARIFICATION====================During night so many batch job running in our database. That time we have checked alert log

27

fileand found "ORA-01555: caused by SQL statement below (Query Duration=8000)". Then we have changedundo_retention parameter value. The old value was 6500 and the new value is 8000.

.ISSUE VERIFICATION===================Verified by seeing the Alert_log file

.CAUSE DETERMINATION====================Tx recovery is taking lot of time

CAUSE JUSTIFICATION====================

Bug:5057695: Shutdown Immediate Very Slow To Close Database.

.POTENTIAL SOLUTION(S)======================Apply the patch

POTENTIAL SOLUTION JUSTIFICATION(S)====================================

same as above

.SOLUTION / ACTION PLAN=======================

Hi Mamun,

Thanks for the UpdateYes.. this also could be an issue in your case..What the BUG says is that there is an change in the Functionality in which the shutdown

28

happened before 9.2.0.8

If there are any active sessions in your database which has to be killed while "shutdown immediate" of the database, then oracle is waiting for 5 seconds on each session and this is consuming more time when there are many databases.. With the BUG FIX applied on the database.. Oracle is going to wait only for .05 seconds so that the Killing Active sessions will be completed farstly..

Since this is an Generic BUG and implies to all the 9.2.0.8 databases on all the platforms.. we recommand you to apply the patch andsee whether you are still having the same issue



STATUS=======


26-FEB-08 10:06:51 GMT


26-FEB-08 10:25:03 GMT

New info : [email protected] Mr Bharath SunkuThank you for advisePatch 5057695 is no available for IBM AIX 5.3 and database is 9.2.0.8.0

one clarification is need how to find whether this patch is applied in our database or not

please check Regardsmd Ruhul Amin

29

26-FEB-08 10:29:21 GMT


TO find whether the patch is applied or not..

1) Go to Oracle_Home

opatch lsinventory -details

++ This contains the information about the patches applied on the database

With regarding to the one-off patch, please answer the below question so that we can goahead and request for a one-offpatch~~~~~~~~~~1) Platform and the database versionuname -a

++ Uplaod the below file

Unix:

$ORACLE_HOME/inventory/ContentsXML

Windows:$ORACLE_HOME\inventory\ContentsXML

SQL:> select * from V$version;

2) List of one-off patches applied since applying the current patch set.opatch lsinventory -detail

3) Release Status of Customer system affected4) Does this affect a major Milestone?If yes, Milestone Date 5) Milestone Name6) Date by which the fix must be installed on customers system 7) Technical Impact of this bug8) Business Impact:

Please let me know if you need any further clarifications~~~~~~~~~~~~~~~

30


STATUS=======


26-FEB-08 10:29:27 GMT


26-FEB-08 13:41:02 GMT

The customer : [email protected] : has uploaded the following file via MetaLink:Config.zip

26-FEB-08 13:42:06 GMT

New info : [email protected] Mr Bharath SunkuHere is the date

[][DB][FINBRAC]/oracle/app/product/9.2.0/inventory/ContentsXML> uname -aAIX DBServer 3 5 00C6F01D4C00

SQL> select * from V$version;BANNER----------------------------------------------------------------Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit ProductionPL/SQL Release 9.2.0.8.0 - ProductionCORE 9.2.0.8.0 ProductionTNS for IBM/AIX RISC System/6000: Version 9.2.0.8.0 - ProductionNLSRTL Version 9.2.0.8.0 - Production

RegardsMd Ruhul Amin

31

27-FEB-08 04:33:03 GMT

working on the issue

27-FEB-08 04:33:14 GMT


27-FEB-08 04:35:23 GMT


Can you please answer the rest of the questions so that we can goahead and raise the backport requestAnswers to all teh questions are mandatory since based on that the Backport request priority will be decided


STATUS=======


27-FEB-08 04:35:29 GMT


27-FEB-08 07:25:02 GMT

New info : [email protected] Bharath SunkuDaily night we are taking cold backup. when shutdown taking more time, it is not possible take cold backup.

if it is possible need to solve this issue by today

32

Regardsmd Ruhul Amin

27-FEB-08 09:17:22 GMT


Database should be shutdown fully to take cold backup


STATUS=======


27-FEB-08 09:17:28 GMT


27-FEB-08 12:35:03 GMT

New info : [email protected] Bharath SunkuThank you. Please guide me how to solve this issue. Regardsmd Ruhul Amin

27-FEB-08 12:41:02 GMT


33

Since the problem is intermitant so you can goahead and shutdown the database to take the cold backups.Meanwhile we will raise an backport request for the BUG and will try to get the one-off patch as soon as possible so that the issue can be resolved..

For this please provide me the Answers for rest of the questions so that we can goahead with raising the backport request

3) Release Status of Customer system affected4) Does this affect a major Milestone?If yes, Milestone Date 5) Milestone Name6) Date by which the fix must be installed on customers system 7) Technical Impact of this bug8) Business Impact:


STATUS=======


27-FEB-08 12:41:11 GMT


02-MAR-08 05:45:02 GMT

New info : [email protected] Bharath Sunkui need some additional information for answer your following questions

3 q. Release Status of Customer system affected========================================What are you trying to mean by the "Release Status of Customer system affected"? Is it the circular or support request from the user who are currently operating the core system?

please describe in details

34

4 q. Does this affect a major Milestone?========================================What are you trying to mean by the "major Milestone"? please describe in details


03-MAR-08 02:53:56 GMT


"Major Milestone" means is there any specific Activity on your database ( Ex migration , Upgradation.. etc ) which is getting blocked because of this BUG..

"Release status" means is it an production environmet or DR ( Disaster Recovery Site )


STATUS=======


03-MAR-08 02:54:18 GMT


05-MAR-08 04:04:00 GMT

.FOLLOW UP==========Hi Mamun,

Please update the SR as soon as possible with the current status and/or requested information so we can continue working the i

35

ssue.

Thank You,Bharath

STATUS=======

@CUS -- Waiting for Customer Update

05-MAR-08 04:04:05 GMT


05-MAR-08 09:04:04 GMT

New info : [email protected] Mr Bharath

3) Release Status of Customer system affected==================================================It is affecting our Production environment

4 and 5 Does this affect a major Milestone?==========================================Yes. And it is effecting our database backup

6) Date by which the fix must be installed on customers system==============================================================as soon as possible

7) Technical Impact of this bug================================no

8) Business Impact:=====================It is a every day process and there is a time limit for it. we have to open our business activities before 8.00 am.

Regards,Md Ruhul Amin

36

05-MAR-08 09:37:07 GMT


One off patch ( Patch Patch 5057695 ) for your BUG is ready and can be downloaded from metalink..Please download the patch and apply on the database abnd let us know if the issue is resolved or not


STATUS=======


05-MAR-08 09:37:14 GMT


05-MAR-08 11:04:05 GMT

New info : [email protected] Mr BharathThank you for your reply. one clarification i need about patch. from 2 weeks we are not facing problem about database shutdown issue, it is now taking 4-5 minutes. Before one hour of shutdown of database we are checking and killing sessions which sessions running report related Long SQL.

we do not want to kill session manually. Is your patch will make shutdown faster if we are not following our manual process( kill session)?.


37

06-MAR-08 02:26:14 GMT


Firstly the Patch that we have requeste dyou to download is for an identified BUG in 9.2.0.8.0 which is causing problem while Shutdown Immediate of the database

As specified in the BUG, it is also closing the Active sessions bit more faster than before patchset and version releases and so this patch should solve your problem..

We resommand you to apply the patch and monitor the database for a week so that we can progress the issue further


STATUS=======


06-MAR-08 02:26:56 GMT


17-MAR-08 11:01:34 GMT

TAR has been in CUS status for 11 days -- Sending First Close Alert email.

18-MAR-08 02:27:29 GMT


Please update the SR as soon as possible with the current status and/or requested information so we can continue working the i

38

ssue.

Thank You,Preetham

STATUS=======


18-MAR-08 02:27:35 GMT


19-MAR-08 04:32:00 GMT


Please update the SR as soon as possible with the current status and/or requested information so we can continue working the issue.

Thank You,Preetham

STATUS=======


19-MAR-08 04:32:05 GMT


30-MAR-08 11:01:46 GMT

TAR has been in CUS status for 11 days -- Sending First Close Alert email.

02-APR-08 11:02:06 GMT

TAR has been in CUS status for 14 days -- Setting status to ACL -- Sending Second Close Alert email.

39

09-APR-08 17:59:25 GMT


Please update the SR as soon as possible with the current status and/or requested information so we can continue working the issue.

Thank You,Preetham

STATUS=======


09-APR-08 17:59:35 GMT


15-APR-08 04:17:06 GMT

CALL SUMMARY=============Called Rahul and spoke about the issueRahul told that he will start working on this issue from tommarowRahul asked me steps for applying the patch.. Told that Readme file has all the instructions and asked him to follow and if problem asked him to update the SR

15-APR-08 04:17:14 GMT


16-APR-08 06:32:13 GMT

Call Summary==========

40

Called Rahul and spoke about the issueExplained Rahul on how to apply the patchFound that Opatch is not installed in the Ho,e..instruncted on how to download and install the opatch and asked him to apply the patch

Told Rahul to update the SR if he needs any clarifications on how to apply the patch

16-APR-08 08:28:04 GMT

New info : [email protected] have installed OPatch 10.2.0.1.6. Install the Opatch:---------------------------I have unzip file p4898608_10204_GENERIC.zip, in ORACLE home directory.

Again i heve tried to apply patch, but again error found. here is error information

==================================================================[FINBRAC]/home/oracle/5057695> ls -larttotal 24drwxr-xr-x 3 oracle orainv 256 Jun 16 2007 filesdrwxr-xr-x 4 oracle orainv 256 Jun 16 2007 etc-rw-rw-rw- 1 oracle orainv 4968 Jun 16 2007 README.txtdrwxr-xr-x 4 oracle orainv 256 Jun 16 2007 .drwxr-xr-x 8 oracle orainv 4096 Apr 16 14:00 ..[FINBRAC]/home/oracle/5057695> /oracle/oraInventory/OPatch/opatch apply Invoking OPatch 10.2.0.1.6

Oracle Interim Patch Installer version 10.2.0.1.6Copyright (c) 2007, Oracle Corporation. All rights reserved.

Oracle Home : /oracle/oraInventoryCentral Inventory : /oraclefrom : /etc/oraInst.locOPatch version : 10.2.0.1.6OUI version : 10.1.0.5.0OUI location : /oracle/oraInventory/ouiLog file location : /oracle/oraInventory/cfgtoollogs/opatch/opatch2008-04-16_14-18-43PM.log

41

ApplySession failed:--------------------------------------------------------------------------------The Oracle Home does not meet OUI version requirement.This OPatch (version 10.2.0.1.6) detects OUI version 10.1.0.5.0 in the home.It requires OUI version 10.2 or above.

System intact, OPatch will not attempt to restore the system

OPatch failed with error code 73[fin7013][FINBRAC]/home/oracle/5057695>

===============================================

The log file with attachment.

regardsMd Ruhul Amin

16-APR-08 08:30:02 GMT

The customer : [email protected] : has uploaded the following file via MetaLink:C:\Documents_and_Settings\User\Desktop\opatch2008-04-16_14-16-17PM.log.txt

16-APR-08 09:18:39 GMT

.UPDATE=======Hi Rahul,

Looks like there is a problem installing opatchWill create a spin off SR with INSTALL team on your behalf to progress the issue further


STATUS

42

=======


16-APR-08 12:36:05 GMT

The customer : [email protected] : has uploaded the following file via MetaLink:C:\Documents_and_Settings\User\Desktop\opatch2008-04-16_14-16-17PM.log.txt

17-APR-08 02:30:12 GMT


We have created a Spin-off ( SR:18780616.6 ) with t h einstall team to resolve opatch errors that you are facing.Please upload the latest log files that you have uploaded in this SR to the New SR so that the Install Analyst can progress the issue further


17-APR-08 02:30:18 GMT


17-APR-08 05:44:01 GMT

New info : [email protected] Bharathi have apply patch through ($opatch apply) command. Thanks a lot to help me to apply patch. is there any way to see patch install is correct or not.

regardsmd Ruhul Amin

43

17-APR-08 06:10:28 GMT


+ Run the below command

opatch lsinventory -details

++ The above command will provide the list of patches installed on your database after the patchset installed on your database

Please let us know whether we can keep the SR in the Soft Closure ( Monitoring ) status for 2 weeks so that you can monitor the database for few days to check whether the same issue is reoccurring or not


STATUS=======


17-APR-08 06:10:41 GMT


17-APR-08 06:30:05 GMT

New info : [email protected] BharathThan you for your help.Please keep the SR in the Soft Closure status for 2 weeks to check whether the same issue is reoccurring or not

Regards,

44

Ruhul

17-APR-08 06:32:24 GMT


Thanks for the UpdateAs per your update we shall keep the SR in the SCL status for next 2 weeks

This means that the TAR will remain in inactive mode(extended sleep mode) for another 14 days, during which periodyou can directly update the TAR via Metalink to re-activate it and I will be glad to assist you further. Otherwise, no update is necessary and the TAR will automatically close after two weeks.For further new issues/errors, if any, please open a new SR so that it gets best handled by appropriate Team


STATUS=======


17-APR-08 07:02:57 GMT


More informationTo get more information, login to this web sidewww.dbasolve.com

45

http://www.dbasolve.com/

Date post:	13-Oct-2014
Category:	Documents
Upload:	oluwasegun-ati
View:	54 times
Download:	2 times