PL/SQL Bulk CollectionsPL/SQL Bulk Collectionsin Oracle 9i and 10gin Oracle 9i and 10g

Kent CrottyKent Crotty

Burleson ConsultingBurleson Consulting

October 13, 2006October 13, 2006

AGENDAAGENDA

• Performance gains with Bulk Processing• Array processing with BULK COLLECT and

FORALL• Oracle 10g FORALL improvements.• Error handling.

Top Books on PL/SQLTop Books on PL/SQL

Dr. Tim Hall Oracle ACEAnd Oracle’s ACE of the Year

John GarmanyA fantastic beginner book on PL/SQL

Bulk ProcessingBulk Processing

• Supercharge your PL/SQL code with BULK COLLECT and FORALL

• Working at a table-level instead of the row-level

• Simple and easy to use

PL/SQL CodePL/SQL Code

• Consists of two types of statementsProcedural (declare, begin, if, while, for …)SQL (select, insert, update, delete)

• Oracle has two engines to process that informationPL/SQL Engine SQL Engine

• A Content Switch occurs each time the PL/SQL engine needs to execute a SQL statement

• Switches are fast but large loops can cause performance delays

Context SwitchesContext Switches

Session PL/SQL Block

PL/SQL Block

PL/SQL Engine

Oracle Server

SQL Engine

Procedural Statement Executor

SQL Statement Executor

DataSQL

PL/SQLBlock

PL/SQL CodePL/SQL CodeConsider this procedure code

CREATE OR REPLACE PROCEDURE update_price ( product_type_in IN product.product_type%TYPE, multiplier_in IN number(2,2) )IS CURSOR products_cur IS SELECT product_id, product_price FROM products WHERE product_type = product_type_in;BEGIN FOR prod_rec IN products_cur LOOP UPDATE products SET product_price = product_price * multiplier_in WHERE product_id = prod_rec.product_id; END LOOP;END update_price;

PL/SQL CodePL/SQL Code

• For each iteration of this loop, there is going to be a conventional bind and a context switch!

• Overhead for these statements can be large

• But there is a solution – Bulk Collections

FOR prod_rec IN products_cur LOOP UPDATE products SET product_price = product_price * multiplier_in WHERE product_id = prod_rec.product_id; END LOOP;

Bulk Collection CategoriesBulk Collection Categories

• SELECT or FETCH statements

BULK COLLECT INTO

• Out-Bind binding

RETURNING clause

• In-Bind binding

FORALL – INSERT, UPDATE, DELETE

SELECT / FETCH statementsSELECT / FETCH statements

Data may be Bulk Collected/Fetched into:

Table.column%TYPE

Record of arrays

Table%ROWTYPE

Cursor%ROWTYPE

Array of records

Nested tables

Products TableProducts TableSQL> create table products ( 2 product_id number, 3 product_name varchar2(15), 4 effective_date date );

Table created.

SQL> begin -- inserting 100000 records into the products table 1 for i in 1 .. 100000 loop 2 insert into products values (i, 'PROD'||to_char(i),sysdate-1); 3 end loop; 4 end; 5 /

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

BULK COLLECT clauseBULK COLLECT clause

• Used in a SELECT statementUsed in a SELECT statement• Binds the result set of the query to a Binds the result set of the query to a

collectioncollection• Much less communication between the Much less communication between the

PL/SQL and SQL enginesPL/SQL and SQL engines• All variables in the INTO clause must be a All variables in the INTO clause must be a

collectioncollection

BULK COLLECTBULK COLLECTSET SERVEROUTPUT ONDECLARE

TYPE prod_tab IS TABLE OF products%ROWTYPE;products_tab prod_tab := prod_tab();start_time number;end_time number;

BEGINstart_time := DBMS_UTILITY.get_time;FOR prod_rec in (SELECT * FROM products WHERE effective_date BETWEEN sysdate - 2 AND TRUNC(sysdate))LOOP

products_tab.extend; products_tab(products_tab.last) := prod_rec;

END LOOP;end_time := DBMS_UTILITY.get_time;DBMS_OUTPUT.PUT_LINE(‘Conventional (‘||products_tab.count||’): ’||to_char(end_time-

start_time));

Start_time := DBMS_UTILITY.get_time;SELECT * BULK COLLECT INTO products_tabFROM productsWHERE effective_date BETWEEN sysdate - 2 AND TRUNC(sysdate); end_time := DBMS_UTILITY.get_time;DBMS_OUTPUT.PUT_LINE(‘Bulk Collect (‘||products_tab.count||’): ’||to_char(end_time-

start_time)); END;

BULK COLLECTBULK COLLECTSQL> /Conventional (100000): 40Bulk Collect (100000): 27

PL/SQL procedure successfully completed.

Bulk Collect is quite a bit faster!

BULK COLLECT – Explicit Cursor- FetchBULK COLLECT – Explicit Cursor- FetchDECLARE

TYPE prod_tab IS TABLE OF products%ROWTYPE;products_tab prod_tab := prod_tab();start_time number;end_time number;CURSOR products_data IS SELECT * FROM products;

BEGINstart_time := DBMS_UTILITY.get_time;OPEN products_data;LOOP

products_tab.extend; FETCH products_data INTO products_tab(products_tab.last);

IF products_data%NOTFOUND THEN products_tab.delete(products_tab.last);

EXIT; END IF;

END LOOP;CLOSE products_data;end_time := DBMS_UTILITY.get_time;DBMS_OUTPUT.PUT_LINE(‘Conventional (‘||products_tab.count||’): ’||to_char(end_time-

start_time));

Start_time := DBMS_UTILITY.get_time;OPEN products_data;FETCH products_data BULK COLLECT INTO products_tab;CLOSE products_data; end_time := DBMS_UTILITY.get_time;DBMS_OUTPUT.PUT_LINE(‘Bulk Collect (‘||products_tab.count||’): ’||to_char(end_time-

start_time)); END;

BULK COLLECTBULK COLLECTSQL> /Conventional (100000): 117Bulk Collect (100000): 14

PL/SQL procedure successfully completed.

Bulk Collect is significantly faster – over 8 times!

BULK COLLECT - LIMITBULK COLLECT - LIMIT

• Collections are arrays held in memory – massive collections can eat up all the memory

• By using the LIMIT clause, we can now process the result set in chunks

• Explicit cursors must be used with the LIMIT clause

BULK COLLECT – Explicit Cursor- LIMITBULK COLLECT – Explicit Cursor- LIMITDECLARETYPE prod_tab IS TABLE OF products%ROWTYPE;products_tab prod_tab := prod_tab();start_time number;end_time number;CURSOR products_data IS SELECT * FROM products;BEGINStart_time := DBMS_UTILITY.get_time;OPEN products_data;LOOPFETCH products_data BULK COLLECT INTO products_tab LIMIT 10000;EXIT WHEN products_data%NOTFOUND;DBMS_OUTPUT.PUT_LINE('Processed '||to_char(products_tab.count)||' rows');END LOOP;CLOSE products_data;end_time := DBMS_UTILITY.get_time;DBMS_OUTPUT.PUT_LINE('Bulk Collect: ‘||to_char(end_time-start_time));end;

Result Set is limited to only 10000 rows – more memory efficient!

Will the processing be slower because of the LIMIT?

BULK COLLECT – Explicit Cursor- LIMITBULK COLLECT – Explicit Cursor- LIMITSQL> /Processed 10000 rowsProcessed 10000 rowsProcessed 10000 rowsProcessed 10000 rowsProcessed 10000 rowsProcessed 10000 rowsProcessed 10000 rowsProcessed 10000 rowsProcessed 10000 rowsProcessed 10000 rowsBulk Collect: 15

PL/SQL procedure successfully completed.

Yes, but only slightly. Still over 8 times better than conventional!

BULK COLLECT - RETURNINGBULK COLLECT - RETURNING• The RETURNING clause can be used to return specific columns after a DML statement

• This is referred to as OUT-BINDING

DECLARETYPE prod_tab IS TABLE OF products.product_id%TYPE;products_tab prod_tab := prod_tab();BEGIN DELETE FROM products WHERE product_id > 20000 RETURNING product_id BULK COLLECT INTO products_tab; DBMS_OUTPUT.PUT_LINE('Deleted Product Ids: '|| products_tab.count||' rows');end;SQL> /Deleted Product Ids: 80000 rows

PL/SQL procedure successfully completed.

BULK COLLECT Use ConsiderationsBULK COLLECT Use Considerations

• Use the LIMIT clause to manage memory requirements

• NO_DATA_FOUND will not be raised if no records are returned – check contents to make sure records are retrieved

BULK COLLECT - FORALLBULK COLLECT - FORALL

• We have seen BULK COLLECT with the SELECT statements

• For the INSERT, UPDATE and DELETE statements there is the FORALL statement

• This is referred to as IN-BINDING

BULK COLLECT – INSERT - FORALLBULK COLLECT – INSERT - FORALLDECLARETYPE prod_tab IS TABLE OF products%ROWTYPE;products_tab prod_tab := prod_tab();start_time number; end_time number;BEGIN-- Populate a collection - 100000 rowsSELECT * BULK COLLECT INTO products_tab FROM products;

EXECUTE IMMEDIATE 'TRUNCATE TABLE products';Start_time := DBMS_UTILITY.get_time;FOR i in products_tab.first .. products_tab.last LOOP INSERT INTO products (product_id, product_name, effective_date) VALUES (products_tab(i).product_id, products_tab(i).product_name, products_tab(i).effective_date);END LOOP;end_time := DBMS_UTILITY.get_time;DBMS_OUTPUT.PUT_LINE(‘Conventional Insert: ’||to_char(end_time-start_time));

EXECUTE IMMEDIATE 'TRUNCATE TABLE products';Start_time := DBMS_UTILITY.get_time;FORALL i in products_tab.first .. products_tab.last INSERT INTO products VALUES products_tab(i);end_time := DBMS_UTILITY.get_time;DBMS_OUTPUT.PUT_LINE(‘Bulk Insert: ’||to_char(end_time-start_time));COMMIT;END;

BULK COLLECT – INSERT - FORALLBULK COLLECT – INSERT - FORALLSQL> /Conventional Insert: 686Bulk Insert: 22

PL/SQL procedure successfully completed.

The Bulk Operation is considerably faster!

BULK COLLECT – UPDATE - FORALLBULK COLLECT – UPDATE - FORALLDECLARETYPE prod_id_tab is TABLE OF products.product_id%TYPE;TYPE prod_tab IS TABLE OF products%ROWTYPE;products_id_tab prod_id_tab := prod_id_tab();products_tab prod_tab := prod_tab();start_time number;end_time number;BEGIN-- Populate a collection - 10000 rowsFOR i in 1 .. 10000 LOOP products_id_tab.extend; products_id_tab(products_id_tab.last) := i+10;

products_tab.extend; products_tab(products_tab.last).product_id := i;END LOOP;

Start_time := DBMS_UTILITY.get_time;FOR i in products_tab.first .. products_tab.last LOOP UPDATE products SET ROW = products_tab(i) -- ROW available in 9.2 WHERE product_id = products_tab(i).product_id;END LOOP;end_time := DBMS_UTILITY.get_time;DBMS_OUTPUT.PUT_LINE('Conventional Update: '||to_char(end_time-start_time));

Start_time := DBMS_UTILITY.get_time;FORALL i in products_tab.first .. products_tab.last UPDATE products SET ROW = products_tab(i) – ROW available in 9.2 WHERE product_id = products_id_tab(i);end_time := DBMS_UTILITY.get_time;DBMS_OUTPUT.PUT_LINE('Bulk Update: '||to_char(end_time-start_time));END;

BULK COLLECT – UPDATE - FORALLBULK COLLECT – UPDATE - FORALLSQL> /Conventional Update: 301Bulk Update: 116

PL/SQL procedure successfully completed.

The Bulk Operation is again considerably faster!

FORALL UseFORALL Use

• Only a single DML statement is allowed per FORALL

• In 9i, the binding array must be sequentially filled

• Use SAVE EXCEPTIONS to continue past errors

• SQL%BULK_ROWCOUNT returns the number of affected rows

The Finer PointsThe Finer Points

• Use bulk bind techniques for recurring SQL statements in a PL/SQL loop.

• Bulk bind rules:• Can be used with any type of collection• Collection subscripts cannot be expressions• Collections should be densely filled• If error, statement is rolled back. Prior

successful DML statements are not rolled back.• Bulk Collects

• Can be used with implicit or explicit cursors• Collection is always filled sequentially starting

with 1

New in 10g – FORALL ImprovementsNew in 10g – FORALL Improvements

• FORALL driving array no longer needs to be processed in sequential order

• The INDICES OF clause is used to reference the row numbers defined in another array

• The VALUES OF clause is used to reference the values defined in another array

New in 10g – FORALL - INDICESNew in 10g – FORALL - INDICES

DECLARETYPE prod_id_tab is TABLE OF BOOLEAN INDEX BY PLS_INTEGER;TYPE prod_tab IS TABLE OF products%ROWTYPE;products_id_tab prod_id_tab := prod_id_tab();products_tab prod_tab := prod_tab();BEGINproducts_tab(10).effective_date := sysdate;products_tab(100).effective_date := sysdate + 10;products_tab(1000).effective_date := sysdate + 100;

products_id_tab(10) := TRUE;products_id_tab(100) := TRUE;products_id_tab(1000) := TRUE;

FORALL i IN INDICES OF products_id_tab UPDATE products SET ROW = products_tab(i) WHERE product_id = products_id_tab(i);END;

New in 10g – FORALL - VALUESNew in 10g – FORALL - VALUES

DECLARETYPE prod_id_tab is TABLE OF BOOLEAN INDEX BY PLS_INTEGER;TYPE prod_tab IS TABLE OF products%ROWTYPE;products_id_tab prod_id_tab := prod_id_tab();products_tab prod_tab := prod_tab();BEGINproducts_tab(10).effective_date := sysdate;products_tab(100).effective_date := sysdate + 10;products_tab(1000).effective_date := sysdate + 100;

products_id_tab(100) := 10;products_id_tab(200) := 100;products_id_tab(300) := 1000;

FORALL i IN VALUES OF products_id_tab UPDATE products SET ROW = products_tab(i) WHERE product_id = products_id_tab(i);END;

FORALL – Error HandlingFORALL – Error Handling

• No more FULL rollback in case of an EXCEPTION• SAVE EXCEPTIONS clause• SQL%BULK_EXCEPTIONS – Collection of records• SQL%BULK_EXCEPTIONS(i).ERROR_INDEX –

stores iteration “i” when exception is raised• SQL%BULK_EXCEPTIONS(i).ERROR_CODE –

stores the Oracle error code• SQL%BULK_EXCEPTIONS.count – returns the

count of the exceptions

Context Switches – Conventional BindsContext Switches – Conventional Binds

Session PL/SQL Block

PL/SQL Block

PL/SQL Engine

Oracle Server

SQL Engine

Procedural Statement Executor

SQL Statement Executor

DataSQL

PL/SQLBlock

Context Switches – Bulk BindsContext Switches – Bulk Binds

Session PL/SQL Block

PL/SQL Block

PL/SQL Engine

Oracle Server

SQL Engine

Procedural Statement Executor

SQL Statement Executor

DataSQL

PL/SQLBlock