8/2/2019 10g Data Pump
1/47
Oracle 10g Data Pump
Say good bye to exp and imp (or not)!2006/02/23
Simon Pane
8/2/2019 10g Data Pump
2/47
2
Agenda
Data Pump Overview
Using Data Pump
Demonstration
Data Pump Test Cases
8/2/2019 10g Data Pump
3/47
3
Data Pump Overview
8/2/2019 10g Data Pump
4/47
4
What is Data Pump?
A replacement of the traditional export/import utilities?
The evolution of the traditional export/import utilities?
A completely new 10g utility serving a similar yetslightly different purpose?
8/2/2019 10g Data Pump
5/47
5
Other Options for Moving Data
Traditional Export and Import Pros
Easy to use most DBAs have years of experience using theseutilities
Versatile various options available; can specify what to include
Platform independent
Serial output
Cons
Comparatively slow
Can be network intensive
Non-interruptible / resumable
Limited filtering options (for example, can exclude just VIEWS)
Limited remapping options (i.e. from one tablespace to another)
8/2/2019 10g Data Pump
6/47
6
Other Options for Moving Data
Transportable Tablespaces Pros
Undoubtedly the fastest way to move data
Can use the traditional exp/imp or Data Pump to move meta-data
Cross-platform support if the platform byte-order is the same
Cons
Tablespaces must be made read-only
Not selective (must move the entire tablespace)
Flashback is not possible (tablespace is read only when copied)
No physical reorganization is performed
Datafile sizes remain constant
Must use RMAN to convert the datafile if migrating to a platform witha different byte-order (checkV$TRANSPORTABLE_PLATFORM)
8/2/2019 10g Data Pump
7/477
Other Options Used Less Frequently
Extraction to a flat file and loading using SQL Loader Direct copy using database links (SQL Plus COPY
command)
Oracle Streams
3rd Party data ETL or reorg tools
8/2/2019 10g Data Pump
8/478
Top 10 Reasons to Love DataPump
10. Similar look and feel to the old exp/imp
9. Can filter on the full range of object types
8. Can re-map datafiles and or tablespaces on import
7. Estimates the export file size (space needed)
6. Parallelizable
5. Significantly faster than the traditional exp/imp
4. PL/SQL Interface programmable
3. A file is not actually required - can import through anetwork link
2. Track in v$session_longops
1. Resumable (interruptible and restartable)
8/2/2019 10g Data Pump
9/47
9
Top 10 Reasons Not to Love Data Pump
10. Still generates redo (unlike direct path inserts)
9. Aggregation of exported data is not possible (sort only)
8. Performance on the server
7. Harder to tell what its doing at any given time
6. No equivalent to the STATISTICS option5. Cannot be used with sequential media such as tapes and
pipes (not read/written serially)
4. Only accesses files on the server, never the client
3. Oracle directories are required in the DB to access the files
2. Does not support COMMIT on imp or CONSISTENT on exp
1. If constraints are violated on import, the load is discontinued
8/2/2019 10g Data Pump
10/47
10
Operation Fundamentals
Export/Import These utilities would basically connect to the Oracle database
via Oracle NET and run queries or DDL/DML
Processing of returned results and I/O operations were done onthe client
Data Pump
The executables call PL/SQL APIs
Therefore processing is done on the database server
This can be an advantageor a disadvantagedepending on thesituation
Self-Tuning: no longer need to use BUFFERor RECORDSET
8/2/2019 10g Data Pump
11/47
11
Export Operation
Network
Export
File(s)
exp.exe
OracleDatabase
8/2/2019 10g Data Pump
12/47
12
Data Pump Export Operation
Network
ExportFile(s)
expdp.exe
OracleDatabase
8/2/2019 10g Data Pump
13/47
13
Key Differences
Dump and log files are on the server, not the client Must have a DIRECTORY created in the Oracle
database for I/O
Permissions for the useridconnecting to the instance, not theschemas being exported or imported
Canceling the client process does not stop the job
Doesnt automatically overwrite dump file if it alreadyexists returns an error instead
Parameters (command line) are reported in the log file
Exported objects order based on table size(descending) instead of alphabetically
8/2/2019 10g Data Pump
14/47
14
Multiple Interfaces
1. Command line utilitiesexpdb
andimpdb
Similar to the familiar exp and imp in usage
Use HELP=Y for a list of commands
Oracle documentation provides a comparison table to exp/imp
2. Enterprise Manager3. PL/SQL
Can be used independently but is difficult
All of these call the DBMS_DATAPUMP API
Uses Oracle Advanced Queuing
Uses DBMS_METADATA
8/2/2019 10g Data Pump
15/47
15
Unload Mechanisms
Data Pump automatically chooses to unload data eitherusing:
Direct path
External Tables (new driver called ORACLE_DATAPUMP)
Same External Tables mechanism that was introduced inOracle9i
When will it use External tables:
When parallelism can be used
When the table contains a complex data type or structure thatprevents direct path unloads
A lot of tables fall under this situation see Oracle documentation for acomplete list
It doesnt really matter to us which method is used
8/2/2019 10g Data Pump
16/47
16
Multiple Processes
Master Control Process Spawns worker processes
Populates the master control table and log file
The master control table can be queried to track the jobsprocess
At the end of an export, the master control table is written to thedump file and dropped from the database
Worker Processes
Performs the loading/unloading
Number of processes depends on the degree of parallelism(the PARALLEL option)
8/2/2019 10g Data Pump
17/47
17
Detaching and Re-Attaching
Issuing Ctrl-C
from the data pump import will
detach
Import is running on the server so it will continue
Brings you into interactive-command mode
To re-attach, run impdp with theATTACH= option Example: impdp userid=system/oracle attach=JOB_01
Brings you back into interactive-command mode
8/2/2019 10g Data Pump
18/47
18
New Views
DBA_DATAPUMP_JOBSand
USER_DATABASE_JOBS Identify all jobs regardless of their state
Identify any master tables not associated with an active job
DBA_DATAPUMP_SESSIONS Identify user sessions that are attached to a job
Data pump sessions populate v$session_longops
Documentation says that it is 100% accurate for imports buttesting proves otherwise!!!
8/2/2019 10g Data Pump
19/47
19
Security Considerations
Still uses theEXP_FULL_DATABASE
andIMP_FULL_DATABASE
A privileged user will have these two roles
A privileged user can:
Export/import objects owned by other schemas
Export non-schema objects (metadata)
Attach to, monitor, and control jobs initiated by others
Perform schema, datafile, and tablespace remapping
Similar to the traditional export/import
Supports label security
If exporting user has the EXEMPT ACCESS POLICY role
8/2/2019 10g Data Pump
20/47
20
Object Statistics
From Oracle documentation regarding data pumpexports:
A parameter comparable to STATISTICS is not needed.Statistics are always saved for tables.
From Oracle documentation regarding data pumpimports:
A parameter comparable to STATISTICS is not needed. If thesource table has statistics, they are imported.
8/2/2019 10g Data Pump
21/47
21
Other Random Points
Can still use a parameter file and thePARFILE
command line option
Fully supports Automatic Storage Management (ASM)
Can still flashback to a specified time or SCN
Can still extract (or backup) DDL (meta data) Using the SQLFILE option instead of the traditionalINDEXFILE or SHOWoptions
Full support of LOBS
8/2/2019 10g Data Pump
22/47
22
Using Data Pump
8/2/2019 10g Data Pump
23/47
23
Oracle Directory Objects
Must first create an Oracle directory object and give theuser who will be performing the Data Pump activitiespermission to use it (or rely on defaults):
SQL> create or replace directory dpump_demo as 'C:\temp';
Directory created.
SQL> grant read,write on directory dpump_demo to simon;
Grant succeeded.
SQL>
8/2/2019 10g Data Pump
24/47
24
Key Data Pump Export Parameters
CONTENT={ALL | DATA_ONLY | METADATA_ONLY}
DIRECTORY=directory_object (default=DATA_PUMP_DIR)
DUMPFILE=[directory_object:]file_name[,]
ESTIMATE={BLOCKS | STATISTICS}
ESTIMATE_ONLY={Y | N}
EXCLUDE=object_type[:name_clause] [,]
FILESIZE=integer[B | K | M |G] (default=unlimited)
FLASHBACK_SCN=scn_value
FLASHBACK_TIME=TO_TIMESTAMP(time_value) FULL={Y | N}
INCLUDE=object_type[:name_clause] [,]
JOBNAME=jobname_string
8/2/2019 10g Data Pump
25/47
25
Key Data Pump Export Parameters
LOGFILE=[directory_object:]file_name
NOLOGFILE={Y | N}
PARALLEL=integer(default=1)
QUERY=[schema.][table_name:]query_clause
SCHEMAS=schema_name[,]
TABLES=[schema_name.]table_name[:partition_name] [,]
TABLESPACES=tablespace_name[,]
8/2/2019 10g Data Pump
26/47
26
Data Pump Export Parameter Samples
Multiple dump files using a substitution variable (%U):
DUMPFILE=DP_DIR1:SCOTT_20060223_%U.dmp
Excluding indexes that start with EMP:
EXCLUDE=INDEX:LIKE EMP%
Excluding the SCOTT schema from a FULL export:
EXCLUDE=SCHEMA:=SCOTT
Mimicking the traditional CONSITENT parameter:
FLASHBACK_TIME=TO_TIMESTAMP
Exporting only TABLES, FUNCTIONS and VIEWS: INCLUDE=TABLE,FUNCTION,VIEW
Using a query clause
QUERY=emp:WHERE salary > 100000
8/2/2019 10g Data Pump
27/47
27
Key Data Pump Import Parameters
CONTENT={ALL | DATA_ONLY | METADATA_ONLY}
DIRECTORY=directory_object (default=DATA_PUMP_DIR)
DUMPFILE=[directory_object:]file_name[,]
EXCLUDE=object_type[:name_clause] [,]
FULL={Y | N}
INCLUDE=object_type[:name_clause] [,]
JOBNAME=jobname_string
LOGFILE=[directory_object:]file_name
NOLOGFILE={Y | N} PARALLEL=integer(default=1)
QUERY=[schema.][table_name:]query_clause
8/2/2019 10g Data Pump
28/47
28
Key Data Pump Import Parameters
REMAP_DATAFILE=source_datafile:target_datafile
REMAP_SCHEMA=source_schema:target_schema
REMAP_TABLESPACE=source_tablespace:target_tablespace
REUSE_DATAFILES={Y | N}
SCHEMAS=schema_name[,]
SKIP_UNUSABLE_INDEXES={Y | N}
SQLFILE=[directory_object:]file_name
TABLE_EXISTS_ACTION={SKIP|APPEND|TRUNCATE|REPLACE}
TABLES=[schema_name.]table_name[:partition_name] [,] TABLESPACES=tablespace_name[,]
8/2/2019 10g Data Pump
29/47
29
Interactive Mode Commands
ADD_FILE=[directory_object:]file_name[,]
CONTINUE_CLIENT
EXIT_CLIENT
FILESIZE=number
KILL_JOB
PARALLEL=integer
START_JOB
STATUS
STOP_JOB
8/2/2019 10g Data Pump
30/47
30
Demonstration
S S
8/2/2019 10g Data Pump
31/47
31
Exporting and Importing Sample Schemas
expdp system/oracle@ORA1020dumpfile=scott.dmp schemas=scott
impdp system/oracle@ORA1020
dumpfile=scott.dmp schemas=SCOTTremap_schema=SCOTT:LARRY
expdb system/oracle@ORA1020
dumpfile=larry.dmp schemas=larry
SELECT * FROM DBA_DATAPUMP_JOBS;
U i I t ti M d
8/2/2019 10g Data Pump
32/47
32
Using Interactive Mode
Ctrl-C to detach from the current export
Export> status
Export> stop_job
expdp system/oracle@ORA1020attach=SYS_EXPORT_SCHEMA_01
Export> start_job
Export> exit_client
8/2/2019 10g Data Pump
33/47
33
Data Pump Test Cases
T t S i #1
8/2/2019 10g Data Pump
34/47
34
Test Scenario #1
Generated sample SIEBEL data
Brushed off the dust on some old SIEBEL data populationscripts (circa 07/2000)
Designed for SIEBEL 6 on Oracle 8.1.6
Actual data is not important
Schema objects
Tables: 218 (many empty tables)
Indexes: 1180 (SIEBEL is a heavily indexed application)
Schema size (from DBA_SEGMENTS) Tables: 1255MB
Indexes: 148MB
E t P f T t C it i
8/2/2019 10g Data Pump
35/47
35
Export Performance Test Criteria
All work performed on laptop C681
SGA remains constant
SGA_TARGET=0
SGA_MAX_SIZE=256MB
BUFFER CACHE=152MB SHARED_POOL=60MB
Mostly default parameters
Not monitoring CPU utilization
Performed 4 runs
Disregarded results from 1st run and averaged the other 3
E t S i t
8/2/2019 10g Data Pump
36/47
36
Export Scripts
EXP
sqlplus -s system/oracle @timestamp.sql > exp.log
exp.exe userid=system/oracle@ORA1020 file=SIEBEL.dmplog=SIEBEL.log owner='SIEBEL'
sqlplus -s system/oracle @timestamp.sql >> exp.log
EXPDP
erase SIEBEL.dmp
sqlplus -s system/oracle @timestamp.sql > expdp.log
expdp.exe userid=system/oracle@ORA1020dumpfile=SIEBEL.dmp logfile=SIEBEL.logschemas='SIEBEL' directory=test_dir
sqlplus -s system/oracle @timestamp.sql >> expdp.log
E t P f T t R lt
8/2/2019 10g Data Pump
37/47
37
Export Performance Test Results
exp expdp
Average ExportTime
6:02 3:18
Estimated File Size N/A 1.201 GB
Actual File Size 965 MB 621MB
I t S i t
8/2/2019 10g Data Pump
38/47
38
Import Scripts
IMP
sqlplus -s system/oracle @timestamp.sql > exp.log
imp.exe userid=system/oracle@ORA1020 file=SIEBEL.dmplog=SIEBEL.log fromuser='SIEBEL' touser='SCOTT'commit=y
sqlplus -s system/oracle @timestamp.sql >> exp.log
IMPDP
sqlplus -s system/oracle @timestamp.sql > expdp.log
impdp.exe userid=system/oracle@ORA1020dumpfile=SIEBEL.dmp logfile=SIEBEL.logschemas='SIEBEL' directory=test_dirremap_schema=SIEBEL:SCOTTremap_tablespace=TOOLS:SCOTT_DATA
sqlplus -s system/oracle @timestamp.sql >> expdp.log
I t P f T t R lt
8/2/2019 10g Data Pump
39/47
39
Import Performance Test Results
imp impdb
Average Import Time 27:07
Average Import Time (no indexes) 25:19
Average Import Time (no rows) 27:27
Database was inARCHIVELOG mode
Destination tablespace and archived log destinationwere both on ASM drives
Machine performance was degraded much more byimpdb import
No import tuning performed (only COMMIT=Y)
Test Scenario #2
8/2/2019 10g Data Pump
40/47
40
Test Scenario #2
Data taken from an actual CGI internal application
Schema objects
Tables: 22
Indexes: 26
Schema size (from DBA_SEGMENTS) Tables: 300MB
Indexes: 101MB
Export Scripts
8/2/2019 10g Data Pump
41/47
41
Export Scripts
EXP
sqlplus -s system/oracle @timestamp.sql > exp.log
exp.exe userid=system/oracle@ORA1020 file=SCOTT.dmplog=SCOTT.log owner='SCOTT'
sqlplus -s system/oracle @timestamp.sql >> exp.log
EXPDP
erase SIEBEL.dmp
sqlplus -s system/oracle @timestamp.sql > expdp.log
expdp.exe userid=system/oracle@ORA1020dumpfile=SCOTT.dmp logfile=SCOTT.log schemas='SCOTT'directory=test_dir
sqlplus -s system/oracle @timestamp.sql >> expdp.log
Export Performance Test Results
8/2/2019 10g Data Pump
42/47
42
Export Performance Test Results
exp expdp
Average ExportTime
1:24 1:32
Estimated File Size N/A 290MB
Actual File Size 261MB 233MB
Import Scripts
8/2/2019 10g Data Pump
43/47
43
Import Scripts
IMP
sqlplus -s system/oracle @timestamp.sql > exp.log
imp.exe userid=system/oracle@ORA1020 file=SCOTT.dmplog=SCOTT.log fromuser='SCOTT' touser='LARRY' commit=y
sqlplus -s system/oracle @timestamp.sql >> exp.log
IMPDP
sqlplus -s system/oracle @timestamp.sql > expdp.log
impdp.exe userid=system/oracle@ORA1020
dumpfile=SCOTT.dmp logfile=SCOTT.log schemas='SCOTT'directory=test_dir remap_schema=SCOTT:LARRYremap_tablespace=SCOTT_DATA:LARRY_DATA
sqlplus -s system/oracle @timestamp.sql >> expdp.log
Import Performance Test Results
8/2/2019 10g Data Pump
44/47
44
Import Performance Test Results
imp impdb
Average Import Time 5:48 2:26
Database was inNOARCHIVELOG mode
Destination tablespace and archived log destinationwere both on ASM drives
No import tuning performed (only COMMIT=Y)
8/2/2019 10g Data Pump
45/47
45
Conclusions
Conclusions
8/2/2019 10g Data Pump
46/47
46
Conclusions
Data Pump is an exciting new Oracle 10g tool thatprovides many benefits over the traditional export andimport utilities
Whether to use Data Pump, Transportable Tablespaces,or even the traditional exp/imp will depend on the situation
Since the command line interface is easy to use and sosimilar to the traditional exp/imp, DBAs and developersshould spend the time to learn how to use it
Final thought: since Data Pump dump files and traditionalexport dump files are not compatible/interchangeable,should a new file extension be used??? (.dmp vs .dpd)
The End
8/2/2019 10g Data Pump
47/47
The End
Comments, Questions ???