Oracle Streams: Step by Step
● What is Oracle Streams?● What is Advanced Queues (AQ)?● What is Change Data Capture (CDC)?● What is a Logical Change Record (LCR)?● How Streams Works● How do you configure Streams, AQ and CDC?● How do you use Streams in your programs?
– Adding records automatically– Adding record programmatically– Pulling records off the queue
Who am I
● Lewis R Cunningham● Oracle ACE● Certified PL/SQL Developer● Author● Blogger● Database Geek● Member of SOUG, IOUG, ODTUG, ACM
What is Oracle Streams?
● Streams enables data and event movement● Movement may be within or between
databases● Can implement replication● Data and events can be captured
– explicitly: programatically– Implicitly: redo log
● Performant– Log mining reduces overhead– Downstreams mining reduces it even more
What is Oracle Streams?, cont'd
● Streams Features– Distinct capture/propagation/apply services– Message Queueing (via AQ)– Publish Subscribe– Rules based Capture/Transform/Apply– Database integration– Easy configuration– Flexible– Inline transformations
What is Oracle AQ?
● Oracle's messaging solution● Queue based
– Persistent queues protect data– Buffered, non-persistent queues offer maximum
performance● Oracle provides all of the maintenance code● Secure – Queue Level Access Control● Messages Types
– Raw– XML– Object– Anydata
What is Oracle AQ?, cont'd
● Publish/Subscribe– Multi-consumer– Multi-message types
● RAC integration● API support
– OCI (C), OCCI (C++)– Java JMS– PL/SQL– OO40 (COM)– Internet Data Access (IDAP)– ODBC via Heterogeneous Gateways
What is Change Data Capture?
● CDC efficiently identifies changed data and make it available for downstrean use
● CDC enables efficient warehouse loading– No flat files– Direct data movement
● Changes are queued● Multiple subscribers are allowed● Subscribers can receive changed data● Data may be transformed● Changes can include DDL and DML
● LCR = Logical Change Record● Committed records generate an LCR to the
redo log● An LCR shows the difference between an
existing record and the new record● For DDL, the record is a database object● For DML, the record is table data
What is an LCR?
What is an LCR?
10
How it works
Source Database Target Database
Table
11
Data Changes
Source Database Target Database
Table
LogWriter
Data Manipulation
12
DB Writes to OS File
Source Database Target Database
Table
LogWriter
REDO LOG
Writes To
13
Streams Takes Over
Source Database Target Database
Table
LogWriter
REDO LOG
StreamsCapture
Capture Process Reads Redo and
Extracts the Logical Change Record
(LCR)
14
Streams Queues The LCR
Source Database Target Database
Table
LogWriter
REDO LOG
StreamsCapture
Streams Queue
Streams Capture Process writes the LCR to a local OR REMOTE Streams Queue
Table
15
LCR Propagates
Source Database Target Database
Table
LogWriter
REDO LOG
StreamsCapture
Streams Queue
Streams Capture Process writes the LCR to a local OR REMOTE Streams Queue
Table
Streams Queue
16
Changes Are Applied
Source Database Target Database
Table
LogWriter
REDO LOG
StreamsCapture
Streams Queue
Streams Capture Process writes the LCR to a local OR REMOTE Streams Queue
Table
Streams Queue
StreamsApply
Streams Apply can also transform for one record to
many, add columns, etc.
17Oracle Streams Overview - US IFS
Target Database
Source Database Target Database
Table
LogWriter
REDO LOG
StreamsCapture
Streams Queue
Streams Capture Process writes the LCR to a local OR REMOTE Streams Queue
Table
Streams Queue
StreamsApply
Table
Table
Table
Configuring Streams
● The Scenario– Capture changes to the Employee table in 10g
ORCL instance)– Send changes to Employee_audit in 9i
(SECOND instance)– We will start with capture and apply in 10g– Once the capture and apply is working, we will
modify the stream to send the data to 9i
Configuring Streams
● We need to create the Admin user and streams tablespace– sqlplus / as sysdba– create tablespace streams_tbs datafile
'c:\temp\stream_tbs.dbf' size 25M reuse autoextend on maxsize unlimited;
– create user strmadmin identified by strmadmindefault tablespace streams_tbsquota unlimited on streams_tbs;
Configuring Streams● We need to grant the appropriate permissions
– grant dba to strmadmin;– BEGIN
DBMS_STREAMS_AUTH.GRANT_ADMIN_PRIVILEGE( grantee => 'strmadmin', grant_privileges => true);END;/
– conn hr/hr– grant all on hr.employees to strmadmin;
Configuring Streams● Create the Audit Table
– CREATE TABLE employee_audit( employee_id NUMBER(6), first_name VARCHAR2(20), last_name VARCHAR2(25), email VARCHAR2(25), phone_number VARCHAR2(20), hire_date DATE, job_id VARCHAR2(10), salary NUMBER(8,2), commission_pct NUMBER(2,2), manager_id NUMBER(6), department_id NUMBER(4), upd_date DATE, user_name VARCHAR2(30), action VARCHAR2(30));
Configuring Streams● Grant read on the audit table to the admin user
– grant all on hr.employee_audit to strmadmin;● Connect as the streams admin user and create a
monitor table– conn strmadmin/strmadmin– CREATE TABLE streams_monitor (
date_and_time TIMESTAMP(6) DEFAULT systimestamp,
txt_msg CLOB );
Configuring Streams● Create the CDC queue
– BEGIN DBMS_STREAMS_ADM.SET_UP_QUEUE( queue_table => 'strmadmin.streams_queue_table', queue_name => 'strmadmin.streams_queue');END;/
Configuring Streams● Create capture rules
– BEGIN DBMS_STREAMS_ADM.ADD_TABLE_RULES( table_name => 'hr.employees', streams_type => 'capture', streams_name => 'capture_emp', queue_name => 'strmadmin.streams_queue', include_dml => true, include_ddl => false, inclusion_rule => true);END;/
Configuring Streams● Add extra attributes
– BEGIN DBMS_CAPTURE_ADM.INCLUDE_EXTRA_ATTRIBUTE( capture_name => 'capture_emp', attribute_name => 'username', include => true);END;/
Configuring Streams● Mark redo entry as starting point (SCN)
– DECLARE iscn NUMBER; BEGIN iscn := DBMS_FLASHBACK.GET_SYSTEM_CHANGE_NUMBER(); DBMS_APPLY_ADM.SET_TABLE_INSTANTIATION_SCN( source_object_name => 'hr.employees', source_database_name => 'ORCL', instantiation_scn => iscn);END;/
Configuring Streams● Create DML Handler Proc
– CREATE OR REPLACE PROCEDURE emp_dml_handler(in_any IN ANYDATA) IS lcr SYS.LCR$_ROW_RECORD; rc PLS_INTEGER; command VARCHAR2(30); old_values SYS.LCR$_ROW_LIST;BEGIN -- Access the LCR rc := in_any.GETOBJECT(lcr); -- Get the object command type command := lcr.GET_COMMAND_TYPE();
–– -- I am inserting the XML equivalent of the LCR into the monitoring table. insert into streams_monitor (txt_msg) values (command || DBMS_STREAMS.CONVERT_LCR_TO_XML(in_any) );
Configuring Streams● Create DML Handler (cont'd)
– -- Set the command_type in the row LCR to INSERT lcr.SET_COMMAND_TYPE('INSERT'); -- Set the object_name in the row LCR to EMP_DEL lcr.SET_OBJECT_NAME('EMPLOYEE_AUDIT');
–
– -- Set the new values to the old values for update and delete IF command IN ('DELETE', 'UPDATE') THEN -- Get the old values in the row LCR old_values := lcr.GET_VALUES('old'); -- Set the old values in the row LCR to the new values in the row LCR lcr.SET_VALUES('new', old_values); -- Set the old values in the row LCR to NULL lcr.SET_VALUES('old', NULL); END IF;
Configuring Streams● Create DML Handler (cont'd)● -- Add a SYSDATE for upd_date
lcr.ADD_COLUMN('new', 'UPD_DATE', ANYDATA.ConvertDate(SYSDATE)); -- Add a user column lcr.ADD_COLUMN('new', 'user_name', lcr.GET_EXTRA_ATTRIBUTE('USERNAME') ); -- Add an action column lcr.ADD_COLUMN('new', 'ACTION', ANYDATA.ConvertVarChar2(command));
●
● -- Make the changes lcr.EXECUTE(true); commit;END;/
Configuring Streams● Register DML Handler
– BEGIN DBMS_APPLY_ADM.SET_DML_HANDLER( object_name => 'hr.employees', object_type => 'TABLE', operation_name => 'INSERT', error_handler => false, user_procedure => 'strmadmin.emp_dml_handler', apply_database_link => NULL, apply_name => NULL);END;/
● Do the exact same for update and delete.
Configuring Streams● Create the apply rule● DECLARE
emp_rule_name_dml VARCHAR2(30); emp_rule_name_ddl VARCHAR2(30);BEGIN DBMS_STREAMS_ADM.ADD_TABLE_RULES( table_name => 'hr.employees', streams_type => 'apply', streams_name => 'apply_emp', queue_name => 'strmadmin.streams_queue', include_dml => true, include_ddl => false, source_database => 'ORCL', dml_rule_name => emp_rule_name_dml, ddl_rule_name => emp_rule_name_ddl); DBMS_APPLY_ADM.SET_ENQUEUE_DESTINATION( rule_name => emp_rule_name_dml, destination_queue_name => 'strmadmin.streams_queue');END;/
Configuring Streams● Turn off disable on error, start the apply and
capture processes● BEGIN
DBMS_APPLY_ADM.SET_PARAMETER( apply_name => 'apply_emp', parameter => 'disable_on_error', value => 'n');END;/
● BEGIN DBMS_APPLY_ADM.START_APPLY( apply_name => 'apply_emp');END;/
● BEGIN DBMS_CAPTURE_ADM.START_CAPTURE( capture_name => 'capture_emp');END;/
Configuring Streams● Test
– Insert a record into hr.employees– Select from hr.employee_audit to see audit
records– Select from streams_monitor to see LCR
information
Configuring Streams● Configure 9i instance (SECOND)
– sqlplus / as sysdba– create tablespace streams_second_tbs datafile 'c:\temp\stream_2_tbs.dbf' size 25M reuse autoextend on maxsize unlimited;
– create user strmadmin identified by strmadmin default tablespace streams_second_tbs quota unlimited on streams_second_tbs;
– grant dba to strmadmin;
Configuring Streams● Create the Audit Table in the 9i instance
– CREATE TABLE employee_audit( employee_id NUMBER(6), first_name VARCHAR2(20), last_name VARCHAR2(25), email VARCHAR2(25), phone_number VARCHAR2(20), hire_date DATE, job_id VARCHAR2(10), salary NUMBER(8,2), commission_pct NUMBER(2,2), manager_id NUMBER(6), department_id NUMBER(4), upd_date DATE, user_name VARCHAR2(30), action VARCHAR2(30));
Configuring Streams● Create the AQ queue in SECOND● BEGIN
DBMS_AQADM.CREATE_QUEUE_TABLE( queue_table => 'lrc_emp_t', queue_payload_type => 'sys.anydata', multiple_consumers => TRUE, compatible => '8.1');
●
●
DBMS_AQADM.CREATE_QUEUE( queue_name => 'lrc_emp_q', queue_table => 'lrc_emp_t');
●
●
DBMS_AQADM.START_QUEUE ( queue_name => 'lrc_emp_q');END;/
Configuring Streams● Create a link from SECOND to ORCL
– CREATE DATABASE LINK orcl.world CONNECT TO strmadmin IDENTIFIED BY strmadmin USING 'orcl.world';
● Create a link from ORCL to SECOND– CREATE DATABASE LINK second.world CONNECT TO strmadmin IDENTIFIED BY strmadmin USING 'second.world';
Configuring Streams● In 10g (ORCL), as strmadmin, create a propagation
schedule for the captured data● BEGIN DBMS_STREAMS_ADM.ADD_TABLE_PROPAGATION_RULES( table_name => 'hr.employees', streams_name => 'orcl_2_second', source_queue_name => 'strmadmin.lrc_emp_q', destination_queue_name => '[email protected]', include_dml => true, include_ddl => FALSE, source_database => 'orcl.world');END;/
Configuring Streams● Modify the DML Handler Proc (add this just before
the lcr.execute procedure).● DECLARE
enqueue_options DBMS_AQ.enqueue_options_t; message_properties DBMS_AQ.message_properties_t; message_handle RAW(16); recipients DBMS_AQ.aq$_recipient_list_t; BEGIN recipients(1) := sys.aq$_agent( 'anydata_subscriber', '[email protected]', NULL); message_properties.recipient_list := recipients;
Configuring Streams● Modify the DML Handler Proc (in 10g)
– DBMS_AQ.ENQUEUE( queue_name => 'strmadmin.lrc_emp_q', enqueue_options => enqueue_options, message_properties => message_properties, payload => anydata.convertObject(lcr), msgid => message_handle); EXCEPTION WHEN OTHERS THEN insert into streams_monitor (txt_msg) values ('Anydata: ' || DBMS_UTILITY.FORMAT_ERROR_STACK ); END;
Configuring Streams● Create a DEQUEUE proc in 9i
– CREATE OR REPLACE PROCEDURE emp_dq (consumer IN VARCHAR2) AS
– msg ANYDATA;– row_lcr SYS.LCR$_ROW_RECORD;– num_var pls_integer;– more_messages BOOLEAN := true;– navigation VARCHAR2(30);– BEGIN– navigation := 'FIRST MESSAGE';– WHILE (more_messages) LOOP– DBMS_OUTPUT.PUT_LINE('Looping thru messages');– BEGIN– DBMS_STREAMS_MESSAGING.DEQUEUE(– queue_name => 'strmadmin.streams_queue',– streams_name => consumer,– payload => msg,– navigation => navigation,– wait => DBMS_STREAMS_MESSAGING.NO_WAIT);
Configuring Streams● Create a DEQUEUE proc in 9i
– IF msg.GETTYPENAME() = 'SYS.LCR$_ROW_RECORD' THEN– num_var := msg.GetObject(row_lcr); – DBMS_OUTPUT.PUT_LINE(row_lcr.GET_COMMAND_TYPE || – ' row LCR dequeued');– END IF;– navigation := 'NEXT MESSAGE';– COMMIT;– EXCEPTION – WHEN SYS.DBMS_STREAMS_MESSAGING.ENDOFCURTRANS THEN– navigation := 'NEXT TRANSACTION';– WHEN DBMS_STREAMS_MESSAGING.NOMOREMSGS THEN– more_messages := false;– DBMS_OUTPUT.PUT_LINE('No more messages.');– WHEN OTHERS THEN– RAISE; – END;– END LOOP;– END;– /
Configuring Streams● Test
– Insert a record into hr.employees@ORCL– Run emp_dq@second– Select from hr.employee_audit@second to see
audit records
Summary
● Oracle Streams: As easy as it gets!● Flexible, performant, easy● Follow items are
– AnyData– XMLType Queues
● Read my blog– An Expert's Guide to Oracle Technology– http://blogs.ittoolbox.com/oracle/guide