Patrick Barel , AMIS, The Netherlands

Wednesday, June 27, 2012

ODTUG KScope 12

San Antonio, Texas, USA

Developers Toolbox - Coding Can collections speed up your PL/SQL?

Agenda

Records

Collections

Bulk Processing

Table Functions

Record

Records

Table

View

Cursor

User Defined

} %rowtype

Records

Record type definition

Field definition

TYPE record_type IS RECORD field_definition (

,

) ;

field datatype

NOT NULL :=

DEFAULT

expression

Collections

Three types available

PL/SQL Tables

Index by Tables

Associative Arrays

Collections

Three types available

Associative Arrays

Nested Tables

Varrays

Collections

Three types available

Associative Arrays

Nested Tables

Varrays

PL/SQL Only

SQL and PL/SQL

SQL and PL/SQL

PL/SQL Collections (3 Types)

Associative Arrays

Varrays

Nested Tables

Number of elements is unbounded, practically speaking.

Valid row numbers range from -231+1 to 231-1. (i.e. -2,147,483,647 to 2,147,483,647)

This range allows you to employ the row number as an intelligent key, such as the primary key or unique index value, because…

AAs also:

Can be sparse.

Data does not have to be stored in consecutive rows, as is required in traditional 3GL arrays and VARRAYs.

Can have index values of integers or strings (Oracle9i R2 and above).

assoc_array_example.sql

collection_of_records.sql

count_vs_rowcount.sql

About Associative Arrays

Name Changes:

7: PL/SQL Tables

8i: Index By Tables

9i: Associative Arrays

Single dimensioned, unbounded, sparse collection of

homogeneous elements

PL/SQL Only value key

Associative Arrays

First

Last

Next (n)

Prior (n)

Count

Exists (n)

Delete [(n[,m])]

idx := emp_t.first;

idx := emp_t.last;

idx := emp_t.next (idx);

idx := emp_t.prior (idx);

idx := emp_t.count;

if emp_t.exists (idx)

then

...

end if;

emp_t.delete (1);

emp_t.delete (2, 4);

emp_t.delete;

Methods

declare

type num_tbl is table of number

index by binary_integer;

l_num_tbl num_tbl;

l_idx integer;

begin

...

value key

Associative Array

declare

type num_tbl is table of number

index by binary_integer;

l_num_tbl num_tbl;

l_idx integer;

begin

l_num_tbl(1):= 12;

l_num_tbl(54):= 5;

...

value

12

5

key

1

54

Associative Array

declare

type num_tbl is table of number

index by binary_integer;

l_num_tbl num_tbl;

l_idx integer;

begin

l_num_tbl(1):= 12;

l_num_tbl(54):= 5;

l_num_tbl(3):= 98;

l_num_tbl(5):= l_num_tbl.count;

l_idx:= l_num_tbl.first;

...

end;

value

12

98

3

key

1

3

5

5 54

Associative Array

value

12

5

key

1

54

2

10

declare

type num_tbl is table of number

index by binary_integer;

l_num_tbl num_tbl;

l_idx integer;

begin

l_num_tbl(1):= 12;

l_num_tbl(54):= 5;

l_num_tbl(3):= 98;

l_num_tbl(5):= l_num_tbl.count;

l_idx:= l_num_tbl.first;

loop

dbms_output.put_line(l_num_tbl(l_idx));

l_idx:= l_num_tbl.next(l_idx);

exit when l_idx is null;

end loop;

l_num_tbl.delete(2,10);

dbms_output.put_line(l_num_tbl.count);

end;

value

12

98

3

key

1

3

5

5 54

Associative Array

declare

type str_tbl is table of varchar2(40)

index by varchar2(40);

l_str_tbl str_tbl;

l_idx varchar2(40);

begin

...

value key

Associative Array VARCHAR2 Key

declare

type str_tbl is table of varchar2(40)

index by varchar2(40);

l_str_tbl str_tbl;

l_idx varchar2(40);

begin

l_str_tbl('one'):= 'een';

l_str_tbl('two'):= 'twee';

...

value

een

twee

key

one

two

Associative Array VARCHAR2 Key

declare

type str_tbl is table of varchar2(40)

index by varchar2(40);

l_str_tbl str_tbl;

l_idx varchar2(40);

begin

l_str_tbl('one'):= 'een';

l_str_tbl('two'):= 'twee';

l_str_tbl('three'):= 'drie';

l_str_tbl('four'):= 'vier';

l_idx:= l_str_tbl.first;

...

end;

value

vier

een

drie

key

four

one

three

twee two

Associative Array VARCHAR2 Key

declare

type str_tbl is table of varchar2(40)

index by varchar2(40);

l_str_tbl str_tbl;

l_idx varchar2(40);

begin

l_str_tbl('one'):= ‘een';

l_str_tbl('two'):= ‘twee';

l_str_tbl('three'):= 'drie';

l_str_tbl('four'):= 'vier';

l_idx:= l_str_tbl.first;

loop

dbms_output.put_line(l_str_tbl(l_idx));

l_idx:= l_str_tbl.next(l_idx);

exit when l_idx is null;

end loop;

l_str_tbl.delete('o', 'tr');

dbms_output.put_line(l_str_tbl.count);

end;

o

tr

value

vier

een

drie

key

four

one

three

twee two

value

vier

twee

key

four

two

Associative Array VARCHAR2 Key

for idx in emp_t.first .. emp_t.last

loop

...

end loop;

for idx in 1 .. emp_t.count

loop

...

end loop;

idx := emp_t.first;

while idx is not null

loop

...

idx := emp_t.next (idx);

end loop;

• Dense

• Count > 0

• Dense

• …

Retrieval

PL/SQL Collections (3 Types)

Associative Arrays

Varrays

Nested Tables

varray_example.sql

About Varrays

Has a maximum size, associated with its type.

Can adjust the size in Oracle10g R2.

Part of object model, requiring initialization.

Is always dense; you can only remove elements

from the end of a varray.

Can be defined as a schema level type and used

as a relational table column type.

Single dimensioned, always bounded, never

sparse collection of homogeneous elements

SQL and PL/SQL

Can be used as column datatype in a table

Stored “in-line” in same table

Retains its ordering

Needs to be initialized and extended

value key

Varray

First

Last

Next (n)

Prior (n)

Count

Exists (n)

Delete

Limit

Extend [(n[,m])]

Trim [(n)]

idx := ename_t.limit;

ename_t.extend (1);

ename_t.trim (1);

ename_t.trim;

Methods

declare

type ename_vt is varray (10) of varchar2(10);

ename_t ename_vt;

begin

ename_t := ename_vt();

ename_t.extend (1);

ename_t(1) := 'Spencer';

...

end;

Initialization

declare

type ename_vt is varray (10) of varchar2(10);

ename_t ename_vt := ename_vt ('Davis');

begin

...

end; Initialization and

Extending

Using Varray

Pre 10gR2: VARRAY needed to be recreated.

10gr2 and up: ALTER TYPE MODIFY LIMIT

Only to increase the limit

varraylimit.sql

How Variable is the Varray?

PL/SQL Collections (3 Types)

Associative Arrays

Varrays

Nested Tables

Name reflects fact that this collection can be

"nested" inside relational table as a column.

Type can be defined at schema level.

No practical, pre-defined limit on a nested table.

Valid row numbers range from 1 to 231-1. (i.e. 1 to 2,147,483,647)

Part of object model, requiring initialization.

Is always dense initially, but can become sparse

after deletes.

nested_table_example.sql

About Nested Tables

Single dimensioned, unbounded, sparse collection

of homogeneous elements

SQL and PL/SQL

Can be used as column datatype in a table

Stored “out-of-line” in a separate table

Initially dense, can be sparse

value key

Nested Tables

First

Last

Next (n)

Prior (n)

Count

Exists (n)

Delete [(n[,m])]

Extend [(n[,m])]

Trim

(Limit)

Methods

declare

type ename_nt is table of varchar2(10);

ename_t ename_nt;

begin

ename_t := ename_nt();

ename_t.extend (1);

ename_t(1) := 'Spencer';

...

end;

Initialization

declare

type ename_nt is table of varchar2(10);

ename_t ename_nt := ename_nt ('Davis');

begin

...

end; Initialization and

Extending

Using Nested Tables

Differences

Feature Associative Array Nested Table VArray

SQL – PL/SQL PL/SQL only SQL and PL/SQL SQL and PL/SQL

Dense - Sparse Sparse Initially Dense

Can become sparse

Dense

Size ‘Unlimited’ ‘Unlimited’ Limited

Order Unordered Unordered Ordered

Usage Any set of data Any set of data Small sets of data

Use in Table No Yes Yes

FORALL Use with inserts, updates and deletes. Move data from collections to tables.

BULK COLLECT Use with implicit and explicit queries. Move data from tables into collections.

In both cases, the "back back" end processing in the SQL engine is unchanged. Same transaction and rollback segment management Same number of individual SQL statements will be

executed. But BEFORE and AFTER statement-level triggers only

fire once per FORALL INSERT statements.

statement_trigger_and_forall.sql

Bulk Processing in PL/SQL

Fetch one or more rows into a collection.

Collection is always filled sequentially from index value 1.

Query does not raise NO_DATA_FOUND if no rows are

fetched.

Instead, the collection is empty.

Use FETCH with LIMIT to manage memory.

SELECT * BULK COLLECT INTO collection FROM table;

FETCH cur BULK COLLECT INTO collection;

BULK COLLECT for multi-row querying

Fetch one or more rows into a collection.

Collection is always filled sequentially from index value 1.

Query does not raise NO_DATA_FOUND if no rows are

fetched.

Instead, the collection is empty.

Use FETCH with LIMIT to manage memory.

SELECT * BULK COLLECT INTO collection FROM table;

FETCH cur BULK COLLECT INTO collection;

BULK COLLECT for multi-row querying

If you are certain that your table will never have

more than N rows, use a VARRAY (N) to hold the

fetched data.

If that limit is exceeded, Oracle will raise an

error.

If you do not know in advance how many rows

you might retrieve, you should:

Declare an explicit cursor.

Fetch BULK COLLECT with the LIMIT clause.

Limiting retrieval with BULK COLLECT

The limit value can be a literal or a variable.

Use a variable for the limit to give you

maximum flexibility.

With very large volumes of data and small

numbers of batch processes, however, a larger

LIMIT could help.

Details on that LIMIT clause

You will need to break the habit of checking

%NOTFOUND right after the fetch.

You might skip processing some of your data.

Instead, do one of the following:

At the end of the loop, check %NOTFOUND.

Right after fetch, exit when collection.COUNT = 0.

At end of loop, exit when collection.COUNT < limit.

LOOP

FETCH my_cursor BULK COLLECT INTO l_collection LIMIT 100;

EXIT WHEN my_cursor%NOTFOUND; BAD IDEA

bulklimit_stop.sql

Terminating loops containing BULK COLLECT

Prior to Oracle10g, you should convert all multiple row

fetch logic, including cursor for loops, to BULK

COLLECTs.

For Oracle10g and above, leave your cursor for loops in

place if they...

contain no DML operations.

seem to be running fast enough.

Explicit BULK COLLECTs will usually run faster than

cursor for loops optimized to Bulk Collect.

When to convert to BULK COLLECT

Convert loops that contain inserts, updates or deletes to FORALL statements.

Header looks identical to a numeric FOR loop. Implicitly declared integer iterator At least one "bind array" that uses this iterator as its

index value.

PROCEDURE upd_for_dept (...) IS

BEGIN

FORALL indx IN low_value .. high_value

UPDATE employee

SET salary = newsal_in

WHERE employee_id = list_of_emps (indx);

END; Binding array

Use FORALL for multi-row DML operations

Use any type of collection with FORALL. One DML statement is allowed per FORALL. Each FORALL is its own "extended" DML statement.

The collection must be indexed by integer. The binding array must be sequentially filled. Unless you use the INDICES OF or VALUES OF clause.

SQL%ROWCOUNT returns total number of rows modified by entire FORALL. Unreliable when used with LOG ERRORS.

Use the SQL%BULK_ROWCOUNT cursor attribute to determine how many rows are modified by each statement.

bulktiming.sql

bulk_rowcount.sql

More on FORALL

Prior to 11g, you cannot reference a field of a record in

FORALL.

You must instead break data into separate collections, or...

You can also perform record-level inserts and updates.

In 11g, this restriction is lifted (but it is an undocumented

feature). http://technology.amis.nl/blog/2367/implementation-restricted-relaxed-in-oracle-11g

11g_field_of_record.sql

FORALL and collections of records

Prior to Oracle10g R2, the binding arrays in a FORALL

statement must be sequentially filled.

Now, however, you can bind sparse collections by using

INDICES OF and VALUES OF in the FORALL header.

10g_indices_of*.sql

10g_values_of*.sql

PROCEDURE upd_for_dept (...) IS

BEGIN

FORALL indx IN INDICES OF list_of_emps

UPDATE employee

SET salary = newsal_in

WHERE employee_id = list_of_emps (indx);

INDICES OF and VALUES OF

When an exception occurs in a DML statement....

That statement is rolled back and the FORALL stops.

All (previous) successful statements are not rolled

back.

Use the SAVE EXCEPTIONS clause to tell Oracle to

continue past exceptions, and save the error information

for later.

Then check the contents of the pseudo-collection of

records, SQL%BULK_EXCEPTIONS.

Two fields: ERROR_INDEX and ERROR_CODE

Exception handling and FORALL

slow-by-slow

Change from integrated, row-by-row approach to

a phased approach.

Phase 1: get the data with BULK COLLECT.

Filling those collections

Phase 2: massage collections so they are ready

for DML operations.

Phase 3: push the data to the database with

FORALL.

cfl_to_bulk_0.sql

cfl_to_bulk_5.sql

10g_indices_of.sql

10g_values_of.sql

Converting old-fashioned code to bulk

Most important performance tuning feature in PL/SQL. Almost always the fastest way to execute multi-row

SQL operations in PL/SQL. You trade off increased complexity of code for

dramatically faster execution. But in Oracle Database 10g and above, the compiler

will automatically optimize cursor FOR loops to BULK COLLECT efficiency.

No need to convert unless the loop contains DML or you want to maximally optimize your code.

Watch out for the impact on PGA memory!

Bulk Processing Conclusions

Oracle server

PL/SQL Runtime Engine SQL Engine

PL/SQL block Procedural

statement

executor SQL

statement

executor

OPEN cur; FETCH cur INTO rec; WHILE cur%found LOOP <<Do Stuff>> FETCH cur INTO rec; END LOOP; CLOSE cur;

Performance penalty

for many “context

switches”

Row by row processing of data in PL/SQL

© Steven Feuerstein/Patrick Barel

Oracle server

PL/SQL Runtime Engine SQL Engine

PL/SQL block Procedural

statement

executor SQL

statement

executor

OPEN cur; FETCH cur BULK COLLECT INTO col; FOR indx IN col.first .. col.last LOOP <<Do Stuff>> END LOOP; CLOSE cur;

Fewer context switches,

same SQL behavior

Row

Row

Row

Row

Row

Row

Row

Row

Row

Row

Row

Row

Bulk processing with BULK COLLECT

© Steven Feuerstein/Patrick Barel

Oracle server

PL/SQL Runtime Engine SQL Engine

PL/SQL block Procedural

statement

executor SQL

statement

executor

FOR rec IN emp_cur LOOP UPDATE employee SET salary = ... WHERE employee_id = rec.employee_id; END LOOP;

Performance penalty

for many “context

switches”

Row by row processing of DML in PL/SQL

© Steven Feuerstein

Oracle server

PL/SQL Runtime Engine SQL Engine

PL/SQL block Procedural

statement

executor SQL

statement

executor

FORALL indx IN list_of_emps.FIRST.. list_of_emps.LAST UPDATE employee SET salary = ... WHERE employee_id = list_of_emps(indx);

Fewer context switches,

same SQL behavior

Update...

Update...

Update...

Update...

Update...

Update...

Update...

Update...

Update...

Update...

Update...

Update...

Bulk processing with FORALL

© Steven Feuerstein

Table Functions

Table functions are functions that produce

a collection of rows (either a nested table or a varray)

that can be queried like a physical database table.

You use a table function like the name of a

database table, in the FROM clause of a query.

Table Functions

Table functions are based on collections

Must be available in the SQL layer

Nested tables and Varray

Table Functions

Create a function in PL/SQL

Make sure it returns a collection

Query it using the TABLE() operator

Table functions can be pipelined (return results as they are produced)

Table functions can be paralellized

Table Functions

You can use Table Functions when

Calculations cannot (easily) be done in SQL

You want to take advantage of PL/SQL e.g.

caching or package variables

You want to leverage the power of PL/SQL in

SQL

Make your views more dynamic

Online

tahiti.oracle.com

For all documentation online

www.allthingsoracle.com o http://allthingsoracle.com/collections-in-oracle-pt-1/

o http://allthingsoracle.com/collections-in-oracle-part-2/

o http://allthingsoracle.com/bulk-processing-in-oracle-part-1/

o http://allthingsoracle.com/bulk-processing-in-oracle-part-2/

Books

Oracle PL/SQL Programming

Chapter 12 (collections) and

chapter 21 (bulk processing)

Oracle PL/SQL for DBAs

Chapter 1 (collections and bulk processing)

Resources