Capture

Innovative Consulting, Inc., 2001intelligent solutions for your businessintelligent solutions for your business

Oracle9i Change Data CaptureOracle9i Change Data CaptureOracle9i Change Data CaptureOracle9i Change Data Capture

Dan WhiteManagerTechnology Solutions GroupInnovative Consulting Inc.

Innovative Consulting, Inc., 2001

ICI Fast FactsICI Fast FactsICI Fast FactsICI Fast Facts

Serving Clients Headquartered in the Greater Philadelphia, New Jersey, and Delaware Area

Headquartered in Malvern, PA

• American Institute for CPCU • Aventis• Campbell Soup• CDNOW Online• Centocor• Cigna• Colorcon• Comcast• Delaware Valley Financial Services• Fleet• Genesis Health Ventures

• GlaxoSmithKline• IKEA• IMS Health • Intracorp • MeDecision• Primavera Systems• Reliance Standard Life• Siemens (Health Services)• Towers Perrin• US Interactive• Syngenta

Industries served

Finance & Insurance17%

Manufacturing, Retail19%

Technology11%

Pharmaceutical, Life Sciences53%

Partnerships and Alliances

Representative Clients

arcplan


ICI At OracleWorldICI At OracleWorldICI At OracleWorldICI At OracleWorld

• Dan White - Oracle 9i Change data Capture- A technical Overview– Monday Nov 11, 2002 1:00pm – 2:00pm

• Dwight DeVera - Delivering Complete Business Intelligence With Oracle9iAS Portal– Wed Nov 13, 2002 1:00pm –2:00pm

• Dwight DeVera - Utilizing Oracle9iAS Portal to deploy Internet Facing Applications– Thurs Nov 14, 2002 4:00pm –5:00pm


AgendaAgendaAgendaAgenda

• What is Change Data Capture?• Traditional Data Capture Methods.• CDC Architecture. • Functional Overview of the DBMS_CDC packages.

• CDC Publish Methods.• CDC Subscribe Methods.• Conclusions.


What is Change Data CaptureWhat is Change Data CaptureWhat is Change Data CaptureWhat is Change Data Capture


What is Change Data What is Change Data What is Change Data What is Change Data Capture?Capture?Capture?Capture?

• Way of extracting delta changes from a source system

• Can be used to optimize the extraction process

• Not only in data warehousing


Traditional Data CaptureTraditional Data CaptureTraditional Data CaptureTraditional Data Capture


Traditional Data Capture Traditional Data Capture Traditional Data Capture Traditional Data Capture MethodsMethodsMethodsMethods• Extract all data to flat files

– Cumbersome– Prone to formatting errors– Large OS files

•Triggers–Introduce a point of failure–Have to maintain multiple objects on “other” systems

•Triggers•Capture Tables•Database Links.•Custom code


Traditional Data Capture Traditional Data Capture Traditional Data Capture Traditional Data Capture MethodsMethodsMethodsMethods• Merge Purge Method

– Extract entire table – See if record exists

• If yes, did it change?• If no insert it!!

– You have to extract very large amounts of data every time.

– Very complex compare code


CDC ArchitectureCDC ArchitectureCDC ArchitectureCDC Architecture


CDC ArchitectureCDC ArchitectureCDC ArchitectureCDC Architecture

Source Databases Redo Logs PDML triggers Staging

Change Tables

DW

Subscription Views

Source System Data Warehouse


Change Data Capture ViewsChange Data Capture ViewsChange Data Capture ViewsChange Data Capture Views

• Change_sources

• Change_sets

• Change_tables

• Dba_source_tables

• Dba_published_columns

• Dba_subscriptions


Functional OverviewFunctional OverviewFunctional OverviewFunctional Overview


Functional Flow: PublishFunctional Flow: PublishFunctional Flow: PublishFunctional Flow: Publish

CREATECHANGETABLE

Does the change table need to bealtered?

Alter changetable

dropsubscriber

view

Yes dropsubscription

Drop changetable

No


Functional Flow: SubscriptionFunctional Flow: SubscriptionFunctional Flow: SubscriptionFunctional Flow: Subscription


DBADBADBADBA

• Typically uses the DBMS_CDC_PUBLISH package

• Creates change tables• Maintains change tables

– Using alter change table**Note** The change table is a physical object so it must be maintained by the DBA or development staff


ETL developersETL developersETL developersETL developers

• Uses DBMS_CDC_SUBSCRIBE

• Sets up and maintains subscriptions

• Sets up and maintains subscription views

• Maintains the subscription windows


DBMS_CDC_PUBLISHDBMS_CDC_PUBLISHDBMS_CDC_PUBLISHDBMS_CDC_PUBLISH

Methods


DBMS_CDC_PUBLISHDBMS_CDC_PUBLISHDBMS_CDC_PUBLISHDBMS_CDC_PUBLISH

• Create_change_table• Alter_change_table• Drop_change_table• Drop_subscriber_view• Drop_Subscription• Purge


Creating a change tableCreating a change tableCreating a change tableCreating a change table

• A change table is a database object that contains change data that resulted from DML statements made on the source table.

• A change table is a database table that maintains data in two types of columns.– Source columns identify data from the source

table to capture.– Control columns contain special metadata.

• The Publisher can control the change table’s physical properties.

• Do not attempt to control a change table’s partitioning properties. CDC Manages partitions.


Create_change_TableCreate_change_TableCreate_change_TableCreate_change_TableDBMS_CDC_PUBLISH.CREATE_CHANGE_TABLE (OWNER => ‘CDC’, CHANGE_TABLE_NAME => ‘SALES_CDC', CHANGE_SET_NAME => 'SYNC_SET', SOURCE_SCHEMA => ‘SRC1', SOURCE_TABLE => ‘SALES’,COLUMN_TYPE_LIST =. ‘prod_id NUMBER(6), cust_id NUMBER, time_id DATE, channel_id CHAR(1), promo_id NUMBER(6), quantity_sold NUNBER(3),amount_sold NUMBER(10,2) ', CAPTURE_VALUES => ‘BOTH', RS_ID => ‘N' ,ROW_ID => ‘N', USER_ID => ‘Y', TIMESTAMP => ‘Y', OBJECT_ID => ‘N', SOURCE_COLMAP => ‘N', TARGET_COLMAP => ‘N', OPTIONS_STRING => ‘TABLESPACE DW STORAGE INITIAL 50M NEXT 12M’);


Change tablesChange tablesChange tablesChange tablesSQL> desc SALES_CDCName Null? Type----------------------------------------- -------- ------------OPERATION$ CHAR(2)CSCN$ NUMBERCOMMIT_TIMESTAMP$ DATEUSERNAME$ VARCHAR2(30)TIMESTAMP$ DATESOURCE_COLMAP$ RAW(128)TARGET_COLMAP$ RAW(128)PROD_ID NUMBER(6)CUST_ID NUMBERTIME_ID DATECHANNEL_ID CHAR(1)PROMO_ID NUMBER(6)QUANTITY_SOLD NUMBER(3)AMOUNT_SOLD NUMBER(10,2)

SQL>


Alter_change_tableAlter_change_tableAlter_change_tableAlter_change_table

• Allows you to add or drop change columns• Allows you to add or drop metadata columns

• You can not add and drop user columns in the same call, these schema changes require separate calls

• Do not specify the name of the control columns in the user column lists


Alter Change TableAlter Change TableAlter Change TableAlter Change TableDBMS_CDC_PUBLISH.ALTER_CHANGE_TABLE ( Owner=>’CDC’, change_table_name =>’SALES_CDC’ ,operation => ‘DROP’,column_list => ‘channel_id CHAR(1)’,rs_id =>’N’row_id =>’N’user_id =>’N’Timestamp =>’N’object_id =>’N’source_colmap =>’N’target_colmap =>’N’);


Drop Change TableDrop Change TableDrop Change TableDrop Change Table

• Used To drop a change tableDBMS_CDC_PUBLISH.DROP_CHANGE_TABLE(Owner=>‘CDC',

Change_table_name =>'SALES_CDC',Force=>‘N');

*** NOTE ***

Force flag = ‘Y’ will drop a change table even if there are subscriptions referencing it


Drop Subscriber viewDrop Subscriber viewDrop Subscriber viewDrop Subscriber view

• This procedure allows a publisher to drop a subscriber view in the subscriber's schema.

• Same as DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIBER_VIEW

DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIBER_VIEW( DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIBER_VIEW( DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIBER_VIEW( DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIBER_VIEW( SUBSCRIPTION_HANDLE =>:subhandle, SUBSCRIPTION_HANDLE =>:subhandle, SUBSCRIPTION_HANDLE =>:subhandle, SUBSCRIPTION_HANDLE =>:subhandle, SOURCE_SCHEMA =>’SRC1’, SOURCE_SCHEMA =>’SRC1’, SOURCE_SCHEMA =>’SRC1’, SOURCE_SCHEMA =>’SRC1’,

SOURCE_TABLE => ‘SALES’);SOURCE_TABLE => ‘SALES’);SOURCE_TABLE => ‘SALES’);SOURCE_TABLE => ‘SALES’);


Drop subscriptionDrop subscriptionDrop subscriptionDrop subscription• This procedure will drop a subscription• Same as DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIPTION

• Why Can we do this from the publish?

DBMS_CDC_PUBLISH.DROP_SUBSCRIPTION ( DBMS_CDC_PUBLISH.DROP_SUBSCRIPTION ( DBMS_CDC_PUBLISH.DROP_SUBSCRIPTION ( DBMS_CDC_PUBLISH.DROP_SUBSCRIPTION ( SUBSCRIPTION_HANDLE => :subhandle);SUBSCRIPTION_HANDLE => :subhandle);SUBSCRIPTION_HANDLE => :subhandle);SUBSCRIPTION_HANDLE => :subhandle);


PurgePurgePurgePurge

• This procedure will purge data from change tables• Automatically monitors subscriptions• You can run this procedure in 2 ways

– Manual at command line– Run in a script to proactively maintain change tables

• Again do not attempt to control a change table’s

partition properties

DBMS_CDC_PUBLISH.PURGE


WAKE UP !!!!!!!!!!!WAKE UP !!!!!!!!!!!WAKE UP !!!!!!!!!!!WAKE UP !!!!!!!!!!!

Question for putty !Who is this?(and don’t say Dan at the bar after the presentation)


Question for PuttyQuestion for PuttyQuestion for PuttyQuestion for Putty

Q: Can YOU manage a change tables partitions?

A: NO, The CDC system manages partitions by it self!


DBMS_CDC_SUBSCRIBEDBMS_CDC_SUBSCRIBEDBMS_CDC_SUBSCRIBEDBMS_CDC_SUBSCRIBE

Methods


DBMS_CDC_SUBSCRIBEDBMS_CDC_SUBSCRIBEDBMS_CDC_SUBSCRIBEDBMS_CDC_SUBSCRIBE

• GET_SUBSCRIPTION_HANDLE • SUBSCRIBE • ACTIVATE_SUBSCRIPTION • EXTEND_WINDOW • PREPARE_SUBSCRIBER_VIEW • DROP_SUBSCRIBER_VIEW • PURGE_WINDOW • DROP_SUBSCRIPTION


GET_SUBSCRIPTION_HANDLEGET_SUBSCRIPTION_HANDLEGET_SUBSCRIPTION_HANDLEGET_SUBSCRIPTION_HANDLE

• Creates a subscription handle• This is the first step in obtaining a subscription

• Subscription handles:– Never get reused– Are tracked from the time of creation– Not shared by subscribers– Validated against the users login id– Can be used to access multiple change tables


GET_SUBSCRIPTION_HANDLEGET_SUBSCRIPTION_HANDLEGET_SUBSCRIPTION_HANDLEGET_SUBSCRIPTION_HANDLE

DBMS_CDC_SUBSCRIBE.GET_SUBSCRIPTION_HANDLE( CHANGE_SET=>'SYNC_SET',DESCRIPTION=>'Change data for SALES', SUBSCRIPTION_HANDLE=>:subhandle);


SUBSCRIBESUBSCRIBESUBSCRIBESUBSCRIBE

• Specifies source table for change data• Specifies source columns for change data• Subscribe procedure is overloaded

– Version1: If you know the publication_id– Version2: If you do not know the publication_id

• Allows you to subscribe to one or more source tables and select columns in each table

• To see all published columns and tables query the ALL_PUBLISHED_COLUMNS view


SubscribeSubscribeSubscribeSubscribe

SQL> desc ALL_PUBLISHED_COLUMNS

Name Null? Type

----------------------------------------- -------- --------------

CHANGE_SET_NAME NOT NULL VARCHAR2(30)

SOURCE_SCHEMA_NAME NOT NULL VARCHAR2(30)

SOURCE_TABLE_NAME NOT NULL VARCHAR2(30)

PUB_ID NOT NULL NUMBER

COLUMN_NAME NOT NULL VARCHAR2(30)

DATA_TYPE VARCHAR2(106)

DATA_LENGTH NOT NULL NUMBER

DATA_PRECISION NUMBER

DATA_SCALE NUMBER

NULLABLE VARCHAR2(1)

SQL>


SubscribeSubscribeSubscribeSubscribe

• Must be created before before you need the data

• You can subscribe to published columns only


SUBSCRIBESUBSCRIBESUBSCRIBESUBSCRIBEDBMS_CDC_SUBSCRIBE.SUBSCRIBE(DBMS_CDC_SUBSCRIBE.SUBSCRIBE(DBMS_CDC_SUBSCRIBE.SUBSCRIBE(DBMS_CDC_SUBSCRIBE.SUBSCRIBE(SUBSCRIPTION_HANDLE=>:subhandle,SUBSCRIPTION_HANDLE=>:subhandle,SUBSCRIPTION_HANDLE=>:subhandle,SUBSCRIPTION_HANDLE=>:subhandle,SOURCE_SCHEMA=>‘SRC1’,SOURCE_SCHEMA=>‘SRC1’,SOURCE_SCHEMA=>‘SRC1’,SOURCE_SCHEMA=>‘SRC1’,SOURCE_TABLE=>’SALES’, SOURCE_TABLE=>’SALES’, SOURCE_TABLE=>’SALES’, SOURCE_TABLE=>’SALES’, COLUMN_LIST=>COLUMN_LIST=>COLUMN_LIST=>COLUMN_LIST=>‘prod_id ,cust_id, time_id, channel_id

,promo_id ,quantity_sold’););););

DBMS_CDC_SUBSCRIBE.SUBSCRIBE ( subscription_handle => :subhandle,publication_id =>31310,COLUMN_LIST=>‘prod_id ,cust_id, time_id,channel_id ,promo_id ,quantity_sold’);


ACTIVATE_SUBSCRIPTIONACTIVATE_SUBSCRIPTIONACTIVATE_SUBSCRIPTIONACTIVATE_SUBSCRIPTION

• Will signify that we are ready to access data

DBMS_CDC_SUBSCRIBE.ACTIVATE_SUBSCRIPTION(SUBSCRIPTION_HANDLE=>:subhandle);


EXTEND_WINDOWEXTEND_WINDOWEXTEND_WINDOWEXTEND_WINDOW

• Sets subscription window boundaries• First call will establish boundaries• Subsequent call will extend High Water Mark

DBMS_CDC_SUBSCRIBE.EXTEND_WINDOW(subscription_handle=>:subhandle);


PREPARE_SUBSCRIBER_VIEW• Creates a subscriber view• You can now access data defined within your window

DBMS_CDC_SUBSCRIBE.PREPARE_SUBSCRIBER_VIEW(DBMS_CDC_SUBSCRIBE.PREPARE_SUBSCRIBER_VIEW(DBMS_CDC_SUBSCRIBE.PREPARE_SUBSCRIBER_VIEW(DBMS_CDC_SUBSCRIBE.PREPARE_SUBSCRIBER_VIEW(SUBSCRIPTION_HANDLE =>:subhandle, SUBSCRIPTION_HANDLE =>:subhandle, SUBSCRIPTION_HANDLE =>:subhandle, SUBSCRIPTION_HANDLE =>:subhandle, SOURCE_SCHEMA =>‘SRC1', SOURCE_SCHEMA =>‘SRC1', SOURCE_SCHEMA =>‘SRC1', SOURCE_SCHEMA =>‘SRC1', SOURCE_TABLE => ‘SALES’, SOURCE_TABLE => ‘SALES’, SOURCE_TABLE => ‘SALES’, SOURCE_TABLE => ‘SALES’, VIEW_NAME => :viewname); VIEW_NAME => :viewname); VIEW_NAME => :viewname); VIEW_NAME => :viewname);


PURGE_WINDOW

• Used to purge the subscription window• Sets low watermark to zero

DBMS_CDC_SUBSCRIBE.PURGE_WINDOW(SUBSCRIPTION_HANDLE=>:subhandle);


DROP_SUBSCRIBER_VIEW• Same as

DBMS_CDC_PUBLISH.DROP_SUBSCRIBER_VIEW

• Subscription must have been created by prepare_subscriber_view

• Must drop the subscriber view before dropping subscription

DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIBER_VIEW(SUBSCRIPTION_HANDLE =>:subhandle, SOURCE_SCHEMA =>‘SRC1', SOURCE_TABLE => ‘SALES');


DROP_SUBSCRIPTION• Drops a subscription • Must use the drop subscriber view first• Same as DBMS_CDC_PUBLISH.DROP_SUBSCRIPTION

DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIPTION(DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIPTION(DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIPTION(DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIPTION(SUBSCRIPTION_HANDLE => :subhandle);


Question for PuttyQuestion for PuttyQuestion for PuttyQuestion for Putty

Q:Can you drop a change table with active subscriptions?

A: YES! if the force flag is set to ‘Y’Q:Is it a good idea?

A:Not if you want to stay in the development teams good graces


ConclusionsConclusionsConclusionsConclusions


Thank youThank youThank youThank you

[email protected]

Date post:	27-Oct-2015
Category:	Documents
Upload:	ranusofi
View:	7 times
Download:	0 times

Capture

Documents