Oow2011 Data Pump

7/28/2019 Oow2011 Data Pump

1/47


2/47

Mastering Oracle Data PumpDean Gagne (Oracle)

Udi Karni (Kaiser Permanente)


3/47

Program Agenda

Deeper understanding of how DataPump works

What is in Data Pumps Master Table

How to get useful information out of the Ma

Hear a large health maintenance organizatioperformance experience with Data Pump in large quantities of data

get real-life performance measurements andhow to make Data Pump run faster.


4/47

What is Oracle Data Pump?

New feature starting in Oracle Database 10g Re

Enables very fast bulk data and metadata move

between Oracle databases

High-speed, parallel Export and Import utilities

impdp) as well as a Web-based Oracle Enterprisinterface


5/47

How Data Pump works

Estimate phase what is it used for

Get Table Data Objects

Phases of Data Pump Export

Unload meta data (single worker process unloads all metadat

Unload data (parallel data unload)(multiple workers or multip

Phases of Data Pump Import Load meta data (serially)

Build indexes (in parallel using pq slaves)

Load package bodies (in parallel using multiple workers)

Load data (in parallel using multiple workers or multiple pq sl


6/47

What was my expdp command

> impdp system/manager directory=dpump_dir

dumpfile=mydmp.dmp master_only=y

SQL> Select name, value_t from SYSTEM.EXPO

where process_order = -59 and name =

CLIENT_COMMAND;

-------------------------------------

CLIENT_COMMAND

system/******** tables=scott.emp directory

dumpfile=ss.dmp reuse_dumpfiles=y


7/47

MASTER_ONLY

Indicates whether to import just the table and then stop the job so that th

contents of the master table can be

examined.

MASTER_ONLY=[YES | NO]


8/47

How many objects have

been exported> expdp system/manager full=y directory=dpump_d

dumpfile=full.dmp metrics=yesProcessing object type DATABASE_EXPORT/SCHEMA/R

Completed 89 ROLE_GRANT objects in 1 secon

Processing object type DATABASE_EXPORT/SCHEMA/D

Completed 2 DEFAULT_ROLE objects in 0 secoProcessing object type DATABASE_EXPORT/RESOURCE

Completed 1 RESOURCE_COST objects in 0 sec

Processing object type DATABASE_EXPORT/SCHEMA/D

Completed 15 DB_LINK objects in 0 seconds


9/47

METRICS

Indicates whether additional informaabout the job should be reported to t

Pump log file.

METRICS=[YES | NO]

When METRICS=YES is used, the num

objects and the elapsed time are reco

the Data Pump log file.


10/47

Whats in my impdp job> impdp system/manager DIRECTORY=dpump_dir1

DUMPFILE=expdat.dmp SCHEMAS=hr ABORT_STSQL> select object_type, object_schema, object_name f

SYSTEM.IMP_SCHEMA where process_order > 0 and dupland processing_status=C and processing_state =

OBJECT_TYPE OBJ OBJECT_NAME

----------------- --- -------------------------

PROCEDURE HR ADD_JOB_HISTORY

ALTER_PROCEDURE HR ADD_JOB_HISTORY

INDEX HR REG_ID_PK

INDEX HR LOC_ID_PK

INDEX HR DEPT_ID_PK


11/47

ABORT_STEP

ABORT_STEP=[n | -1] Values correspond to a proc

number in the master table.

n - If the value is greater than 0, then the job is st

the job is aborted at the object that is stored in th

table with the corresponding process order numb

-1 - If the value is negative one (-1) then abort the

setting it up, but before exporting or importing a

NOTE: Job is restartable after abort_step is used.


12/47

Did my job run parallel

2 Types of parallel

1. multiple workers

2. Parallel execution slaves

> expdp system/manager parallel=directory=dpump_dirdumpfile=scott.dmp keep_mast


13/47

Data unload in parallelSQL> select m.object_schema, m.object_name

(select count(*) from system.export_ta

where t.process_order = m.process_ord

t.duplicate!=0) pq_count

from system.sys_export_table_01 m

where m.process_order > 0 and m.object_t

'TABLE_DATASCHEMA NAME PQ_COUNT

------ ---- --------

SCOTT EMP 2


14/47

How many workers started

select count(*) from

hr.sys_import_table_01

where process_order = -42;

COUNT(*)--------

1


15/47

KEEP_MASTER

Indicates whether the master table sdeleted or retained at the end of a Da

job that completes successfully. The m

table is automatically retained for job

do not complete successfully.

KEEP_MASTER=[YES | NO]


16/47

ACCESS_METHOD

Instructs Export to use a particular method to

data.

ACCESS_METHOD=[AUTOMATIC | DIRECT_PAT

EXTERNAL_TABLE]

Provided so that you can try an alternative medefault method does not work for some reaso

recommends that you use the default option

(AUTOMATIC).

Ch i h


17/47

Choosing the wrong

access_method> impdp system/manager tables=scott.foo_lo

directory=dpump_dir dumpfile=s.dmp

access_method=external_table

Processing object type

TABLE_EXPORT/TABLE/TABLE_DATAORA-31696: unable to export/import

TABLE_DATA:SCOTT"."FOO_LONG" usin

client specified EXTERNAL_TABLE me


18/47

Documented Parameters

Diagnostic Parameters

Access_method

Keep_master

Metrics

Abort_step

Master_only


19/47

What is the Master Table

Oracle table that is used to store information abo

Pump job

Export/import parameters

Current status

Object information

Can access the master table in SQLPLUS Not deleted if job is stopped

Deleted after job is complete or killed

M T bl C


20/47

Master Table Contents

Some Interesting columns

Process_order (+/- numbers)

Object_type

Object_schema

Object_name Processing_state

Processing_status

d


21/47

Interesting Process Orders

Positive process orders describe objects that have

exported.

Negative process orders describe the Data Pump

-1/-2 Job state row contains job status

-5/-6 completion rows status for each object type

-41/-42 worker status rows -51/-52 data filter rows

-53/-54 metadata filter rows

-57/-58 metadata transform rows

-59/-60 job parameter rows

h b l f


22/47

What object types are leftSQL> select unique object_type_seqno, objec

from system.sys_import_full_01

where process_order > 0 AND processing_stat

and processing_status = C;

OBJECT_PATH_SEQNO OBJECT_TYPE

----------------- -------------------------103 PROCEDURE

119 ALTER_PROCEDURE

137 VIEW

Wh l f f h


23/47

Whats left for the current

objectSQL> select object_schema, object_namefrom system.sys_import_full_01

where process_order > 0 and processing_sta

and processing_status = 'C' and

object_path_seqno = 103;

OBJECT_SCHEMA OBJECT_NAME-------------------- --------------------

HR ADD_JOB_HISTORY

HR SECURE_DML


24/47

Business unit in National Pharmacy Programs andKaiser Permanente.

Kaiser Permanente is a not-for-profit health planregions across the United States.

Provides affordable, high-quality health care servimprove the health of our members and the commwe serve.

8.7 million members. 15,000 physicians and 165,000 employees. 35 hospitals and 454 medical offices. A large integrated electronic health record.


25/47

Chronic disease management for 1.1 million mem Member outreach with letters, telephony, secure

messaging. Proactive management of member encounters. Measurement of clinical strategic goals. Forecasting, purchasing, inventory, distribution,

prescribing, dispensing and monitoring of drug t

Intranet portal with over 100 applications and 30users. Oracle databases since 1998 with 8.0.4, 8.1.7.3,

and now 11.2.0.2.


26/47

10 DBAs, 80 large databases (0.5T 240T).

100% of large databases on Oracle. 95% data warehouse, datamart and business

intelligence. Oracle versions

10.2.0.3 20% 11.1.0.7 4% 11.2 76%

Host platforms Windows 2003 Server 20% Oracle Enterprise Linux 80%


27/47

Commodity server approach High performance, low cost Multiple environments Scale horizontally Replace server and storage every 3-4 years R&D to identify next building block Current building block

HP Proliant DL580G5 16 cores or 24 cores w/256 GB RA 60 terabytes of direct attached storage 1 gigabit network connection Oracle Enterprise Linux 5.5 and Oracle Database 11.2.0


28/47

Data Staging DataWarehouse

On-line Database

Production

Training

Test

Development

On-line Database

EHRIntegr

Telep

Busin

Intelli

Pharm

Integr


29/47

Existing 11 DB servers Hardware

HP x64 16 cores

HP Itanium 8 cores HP AMD x64 32 cores

Storage

10 - 30 terabytes OS

Windows Server

Oracle

10.2, 11.1

New 13 DB se Hardware

HP x64 24 co

HP x64 16 co

Storage

60 terabytes OS

OEL 5.5

Oracle

11.2.0.2


30/47

Solution Advantages Disadvantages EstimateParallel databaseenvironment

Current databaseprocesses not affected

Prolonged data loads Maintenance of scripts

in two environments Extensive regression

testing

4-6 weeexperien

Database backup andrestore

Traditionalmethodology

Disaster recovery

process

Write back up file tofile server.

Read back up file from

file server.

110 hou= 20 terper seco

Database link Know process Multiple simultaneous

links

Single threaded Separate migration for

metadata

55 hour= 20 terper seco

Data Pump in Network LinkMode plus 10 gigabit

Move objects andmetadata

Direct source to targetmove

Unknown technology Unproven technology

14 hour= 20 ter

gigabits


31/47

Oracle Data Pump in Network Link Mode

10 gigabit


32/47


33/47

Source TargetServer HP DL580G5 16 core HP DL580G

Operating system Windows 2003 Server OEL 5.5

Oracle database v11.1 v11.2

Data read 286 MB/secCPU sending 82%

Network receiving 187 MB/sec

Data written 278 MB/sec

CPU receivin 51%


34/47

. . imported "LKUSER"."KPHC_KEPT_APPT_OLD" 368,451,699 rows

. . imported "LKUSER"."KPHC_KEPT_APPT" 376,117,378 rows

. . imported "LKUSER"."OUTPAT_ENCOUNTERS_OLD" 488,651,654 rows

. . imported "LKUSER"."SUPERDAILY_CS_MONTH" 159,244,195 rows

. . imported "LKUSER"."OUTPAT_ENCOUNTERS" 575,578,889 rows

. . imported "LKUSER"."KRMS_CMT" 1,192,264,533 rows (bi

. . imported "LKUSER"."KRMS_CMT_OLD" 1,192,264,533 rows (bi

. . imported "LKUSER"."CS_LAB_HIST_OLD" 1,216,560,021 rows (bi

. . imported "LKUSER"."LAB_HIST_CN" 1,031,083,214 rows (bi

. . imported "LKUSER"."CS_LAB_HIST" 1,225,545,758 rows (bil

. . imported "LKUSER"."LAB_HIST_CN_OLD" 1,023,007,500 rows (bi

. . imported "LKUSER"."KPHC_OUTPAT_DX_OLD" 674,288,611 rows

. . imported "LKUSER"."OUTPAT_DX_OLD_O" 1,159,060,603 rows (bi

. . imported "LKUSER"."MEMHIST" 1,009,593,929 rows (bi

. . imported "BTUSER"."PT_ECS_FULL_INIT_WK_CN" 646,657,576 rows

. . imported "BTUSER"."KPHC_ROC_COV_MEMBERSHIP_OLD" 893,791,915 rows

. . imported "LKUSER"."KPHC_OUTPAT_DX" 689,554,580 rows

. . imported "LKUSER"."ABSTRACT_ENCOUNTERS_DX" 138,161,570 rows

. . imported "BTUSER"."VT_10A_PIMS_EXTR_99Q11_TEST" 499,758,332 rows


35/47

Job: SYS_IMPORT_SCHEMA_01

Operation: IMPORT

Mode: SCHEMA

State: EXECUTING

Bytes Processed: 0

Current Parallelism: 16

Job Error Count: 0

Worker 1 Status:

Process Name: DW00

State: EXECUTING

Object Schema: BTUSER

Object Name: KC_MRR_10B_NW_ABC1_88Q44

Object Type: SCHEMA_EXPORT/TABLE/TABLE_DATA

Completed Objects: 15

Total Objects: 23,926

Worker Parallelism: 1

Worker 2 Status:

Process Name: DW01

State: EXECUTING

Object Schema: BTUSER

Object Name: KC_DIR_10A_PARTD1_9

Object Type: SCHEMA_EXPORT/TABLE




Worker 3 Status:

Process Name: DW02

State: EXECUTING

Object Schema: LKUSER

Object Name: JUTILSUMM_HIST

Object Type: SCHEMA_EXPORT/TABLE





36/47

Check log for successful transfer of me

objects and packages. Update tnsnames.ora.

Create database links.

Re-activate users and passwords.

Regression test processes and check re

Roll back to old server, if needed.


37/47


38/47


39/47


40/47


41/47

Parallel data pump workers

Compression of network transfer Already compressed table

Append hint

Partition syntax and locking

Statistics Platform agnostic

Windows limit on 10 gigabit


42/47

Transform parameter


43/47

Network link data pump works

Important patches to consider Append (9721663) Partition wise syntax (11677757) Partition truncate (8692663 for v11.1)

Oracle transfer on10 gigabit network Windows 2003 @ 3 Gb/sec Windows 2008 @ 5 Gb/sec

OEL 5.5 @ 7 Gb/sec

Demogrounds


44/47

Demogrounds

Come see us in the Demogrounds at

Moscone South 7460

Demos of whats in a Master Table


45/47

Q&A


46/47


47/47

Date post:	03-Apr-2018
Category:	Documents
Upload:	mohammed-omar-jaber
View:	217 times
Download:	0 times

Oow2011 Data Pump

Documents