2008 Laoug2 Dynamic SQL

8/17/2019 2008 Laoug2 Dynamic SQL

1/56


2/56

2 of 56

The Problem

It is difficult to build a system that actually meets the

stated requirements.

Is the industry-wide 75% system failure rate still true?

It is very difficult to build a system that doesn’t require

massive changes within a short period of time. How many mistakes were made by previous contractors?

It is impossible to build a system that will not be

obsolete sooner or later. Did you ever meet anyone who is omniscient?


3/56

3 of 56

The Truth… Let’s be fair…

I’ve never seen:

100% perfect analysis 100% complete set of requirements

100% adequate hardware

100% competent users

100% knowledgeable tech support


4/56

4 of 56

Not My Job

Educate users

Improve customer support


5/56

5 of 56

My Job

Impossible Possible

To do perfect analysis To identify directions

To have completespecifications To define the most volatileareas

To have the best possible

hardware

To build flexible

performance tuning

mechanisms


6/56

6 of 56

Unknowns

What elements are involved?

Example: The monthly summary table may not exist.

What to do with the elements you have?

Example: Do you need quarterly or yearly totals?

How you should proceed?

Example: Is a hash join hint the best option?

Whether or not you can proceed?

DDL is still prevented in PL/SQL, isn’t it?


7/56

7 of 56

The Hero

Dynamic SQL:

Makes it possible to build and process complete SQLand PL/SQL statements as strings at runtime.


8/56

8 of 56

Dynamic SQL & PL/SQL

What does "dynamic" mean?

Build a text string on the fly and execute it Very powerful technique

Useful in many contexts

What techniques exist?

EXECUTE IMMEDIATE

Dynamic cursors

DBMS_SQL


9/56

9 of 56

Back to Basics:

What are we talking about?


10/56

10 of 56

Dynamic SQL Core

About 90% of dynamic SQL is covered by a singlecommand (with variations):

decl ar ev_var i abl e_t x var char 2( 32000) ;

begi n

v_var i abl e_t x: =' whatever_you_want';EXECUTE IMMEDIATE v_var i abl e_t x;end;

OR

begi nEXECUTE IMMEDIATE ' whatever_you_want';

end;


11/56

11 of 56

Dynamic Cursors

Syntaxdecl ar e

v_cur SYS_REFCURSOR;

v_sql _t x var char 2( 32000) : =. . .

v_r ec . . . %r owt ype; - - or r ecor d t ype

begi n

open v_cur for v_sql_tx;

f et ch v_cur i nt o v_r ec;

cl ose v_cur ;end;

Most common use:

Processing large datasets with unknown structure


12/56

12 of 56

DBMS_SQL package

Predecessor of native dynamic SQL

Pros:

Goes above 32K in all versions

Separates PARSE and EXECUTE

The same query can be reused with different bind variables.

Works with unknown number/type ofINPUT/OUTPUT values

Cons: Usually slower (up to 5 times)

No user-defined datatypes or output to the record.

More difficult to use


13/56

13 of 56

Things to know (1)

The code can be passed as a variable/string.

The variable/string cannot exceed 32K, except In 11g – passing CLOB as input

If the database is using fonts where 1 byte=1 char

–up to 64K when concatenate a number of strings:

execut e i mmedi at e

v1_t xt | | v2_t xt | | v3_t xt | | …; In 8i and below – no limit on concatenation

(undocumented feature - not supported by Oracle!)


14/56

14 of 56

Things to know (2)

The variable/string can contain bind variables

Start with a colon (:)

Placeholders for values that will be supplied at

runtime (USING and RETURNING clauses)

No validation when compiled No check for datatypes

No check for passing enough values

No direct limit on the number of bind variables.

Like any other variables, they may be IN (default),

OUT, or IN/OUT


15/56

15 of 56

Things to know (3)

Bind variables are limited: CAN only be used to supply values to be passed to the code

CANNOT be used to define the structural elements of queriesor PL/SQL blocks.

Special cases: You cannot pass NULL as a literal.

If statement has a RETURNING clause, it should be also usedin dynamic SQL.

Bind variables may or may not be reusable

NOT reusable in dynamic SQL – the number of variables isequal to the number of parameters.

ARE reusable in dynamic PL/SQL – the number of UNIQUEvariables is equal to the number of parameters.


16/56

16 of 56

!!!REMEMBER!!!

1. Use bind variables

2. Use bind variables3. Use bind variables



© Tom KyteP


17/56

17 of 56

Nightmare of SOX auditor (1)

f unct i on F_GET_col _TX( i _t abl e_t x, i _showcol _t x, i _pkcol _t x, i _pkVal ue_nr )

r et ur n var char 2 i sv_out _t x var char 2( 4000) ;

v_sql_tx varchar2(32000);Begi n

v_sql _t x: =

' sel ect t o_char ( ' | | i _showcol _t x| | ' ) ' | |' f r om ' | | i _t abl e_t x| |' wher e ' | | i _pkcol _t x| | ' =:v01' ;

EXECUTE I MMEDI ATE v_sql _t x I NTO v_out _t xUSING i_pkValue_nr;

r et ur n v_out _t x;

end;

M


18/56

18 of 56

Why do you still need DBMS_SQL?


19/56

19 of 56

Use Case

The problem:

Existing (ancient) UI generates query on the fly.

No way to intercept generation No way to have a limited number of possible cases

Front-end is restricted to view only 500 rows.

Need to generate complete result

The solution:

Generic query processor with a CLOB output (tab-delimited format)


20/56

20 of 56

Process Query (1) Parse query and get column list:

FUNCTI ON f _get Quer y_cl ( i n_sql _t x, i _l i mi t _nr number : =64000)RETURN CLOB I S

v_cur integer := dbms_sql.open_cursor;

v_cols_nr number := 0;v_cols_tt dbms_sql.desc_tab;

...

pr agma aut onomous_t r ansact i on;BEGI NDBMS_SQL.parse

(v_cur, in_sql_tx, DBMS_SQL.native);

DBMS_SQL.describe_cols(v_cur, v_cols_nr, v_cols_tt);

FOR i I N 1 . . v_col s_nr LOOPv_col Li st _t x : =v_col Li st _t x | | ' | | chr ( 9) | | ' ' " ' ' | | v_quer y_t t ( i ) . ' | |

v_cols_tt (i).col_name| | ' | | ' ' " ' ' ' ;v_header : = v_header | | chr ( 9) | | ' " ' | |

v_cols_tt (i).col_name| | ' " ' ;END LOOP;


21/56

21 of 56

Process Query (2)

Define a cursor in the dynamic block . . .v_pr ocess_t x : =' DECLARE

v_out _cl CLOB;v_r ow_t x VARCHAR2( 32000) ;

cursor c1 is '||in_sql_tx||';

TYPE DynamicTable_tt IS TABLE OF c1%rowtype

I NDEX BY BI NARY_I NTEGER;v_quer y_t t Dynami cTabl e_t t ;PROCEDURE p_addl i ne ( pi _t x VARCHAR2) I SBEGI N

DBMS_LOB. wr i t eappend( v_out _cl , l engt h( pi _t x) , pi _t x) ;END;

BEGI NDBMS_LOB. cr eat et empor ar y

( v_out _cl , TRUE, DBMS_LOB. CALL) ;p_addl i ne( : 1) ;. . .


22/56

22 of 56

Process Query (3)

Execute the query. . .open c1;

fetch c1 bulk collect

into v_query_tt limit :2;cl ose c1;FOR i I N 1 . . v_quer y_t t . COUNT LOOP

v_row_tx:='||v_colList_tx ||';

p_addl i ne( v_r ow_t x) ;END LOOP;: 3 : = v_out _cl ;

END; ' ;

EXECUTE IMMEDIATE v_process_tx

USING IN v_header, IN i_limit_nr,

OUT v_out_cl;

r et ur n v_out _cl ;END;


23/56

23 of 56

11g Additions

Conversion between REF_CURSOR and

DBMS_SQL cursor

DBMS_SQL.TO_REFCURSOR

DBMS_SQL.TO_CURSOR_NUMBER

P Q (1)


24/56

24 of 56

Process Query (1)

Parse query and get column list:FUNCTI ON f _get Quer y_cl ( i n_sql _t x, i _l i mi t _nr )RETURN CLOB I Sv_cur i nt eger : = dbms_sql . open_cur sor ;

v_col s_nr number : = 0;v_col s_t t dbms_sql . desc_t ab;v_execute_nr integer;

v_ref_cur SYS_REFCURSOR;

pr agma aut onomous_t r ansact i on;BEGI NDBMS_SQL. par se

( v_cur , i n_sql _t x, DBMS_SQL. nat i ve) ;

DBMS_SQL. descr i be_col s( v_cur , v_col s_nr , v_col s_t t ) ;

v_exec_nr:=dbms_sql.execute(v_cur);

v_ref_cur:=dbm_sql.to_refcursor(v_cur);


25/56

25 of 56

Process Query (2)

Define a cursor in the dynamic block . . .v_pr ocess_t x : =' DECLARE

v_out _cl CLOB;v_r ow_t x VARCHAR2( 32000) ;

cursor c1 is '||in_sql_tx||';

TYPE DynamicTable_tt IS TABLE OF c1%rowtype

I NDEX BY BI NARY_I NTEGER;v_quer y_t t Dynami cTabl e_t t ;PROCEDURE p_addl i ne ( pi _t x VARCHAR2) I SBEGI N

DBMS_LOB. wr i t eappend( v_out _cl , l engt h( pi _t x) , pi _t x) ;END;

BEGI NDBMS_LOB. cr eat et empor ar y

( v_out _cl , TRUE, DBMS_LOB. CALL) ;p_addl i ne( : 1) ;. . .

Q ( )


26/56

26 of 56

Process Query (3)

Execute the queryfetch :2 bulk collectinto v_query_tt limit :3;

FOR i I N 1 . . v_quer y_t t . COUNT LOOP

v_row_tx:='||v_colList_tx ||';

p_addl i ne( v_r ow_t x) ;

END LOOP;

: 3 : = v_out _cl ;END; ' ;

EXECUTE IMMEDIATE v_process_tx

USING IN v_header ,

IN v_ref_cursor,

IN i_limit_nr, OUT v_out_cl;

cl ose v_r ef _cur sor ;

r et ur n v_out _cl ;END;


27/56

27 of 56

Security

S it C


28/56

28 of 56

Security Concerns

What privileges do you need to use dynamic

SQL?

How can you guard against misuse of dynamic

SQL?

G ti P i il (1)


29/56

29 of 56

Granting Privileges (1)

All privileges have to be granted explicitly

(Not via ROLES):

System privileges

cr eat e or r epl ace pr ocedur e p_makeTabl e( i _name_t x var char 2) i s

begi n

execut e i mmedi at e ' cr eat e t abl e '| | i _name_t x| | ' ( a_t x var char 2( 256) ) ' ;

end;

GRANT DBA to SCOTT … wrong!!!

GRANT CREATE TABLE to SCOTT – correct!

G ti P i il (2)


30/56

30 of 56

Granting Privileges (2)

Object privileges

f unct i on f _get Count _nr ( i _user _t x var char 2,

i _t abl e_t x var char 2) r et ur n number i sbegi n

execut e i mmedi at e ' sel ect count ( *) f r om '

| | i _user _t x| | ' . ' | | i _t abl e_t x;end;

GRANT DBA to SCOTT … wrong!!!

GRANT SELECT on EMPLOYEE to SCOTT – correct!

Fighting Code Injections


31/56

31 of 56

Fighting Code Injections

DBA protection

End users should not see administration tools

UI protection

User input should always be passed via bind variables

(no concatenation!) Bind variables cannot affect the structure of the query.

All structural selections should be made from the

limited list of options (repository) Power users/developers populate the repository.

End users only access whatever is already in the repository.

Use Case


32/56

32 of 56

Use Case

The problem:

Large number of requests from the user interface

Take some number of parameters Return something (search, extra info, status, etc.)

The whole set of requests is not clear and may

change each time.

The solution:

Universal wrapper Process all requests dynamically

Storage (1)


33/56

33 of 56

Storage (1)

Single table for both UI and server:cr eat e t abl e t _ext r a_ui( i d_nr number pr i mar y key,

di spl ayName_t x var char 2( 256) ,f unct i on_t x var char 2( 50) ,v1_l abel _t x var char 2( 100) ,v1_t ype_t x var char 2( 50) ,

v1_r equi r ed_yn var char 2( 1) ,v1_l ov_t x var char 2( 50) ,v1_conver t _t x var char 2( 50) ,

. . . )Example:i nser t i nt o t _ext r a_ui ( …)

val ues ( 1, ' Fi l t er Empl oyees' , ' f _get Emp_cl ' ,' J ob' , ' TEXT' , ' N' , nul l , nul l )

Storage (2)


34/56

34 of 56

Storage (2) Published function

ID_NR – unique ID of the function DisplayName_tx – header of the screen

ID and display are shown to users as LOV

Function_tx – real function to be called

Parameters (never needed more than 10) Vx_Label_tx – label for the parameter

if null – parameter is disabled

Vx_Type_tx – helps UI to build the screen – can be: LOV – value list

TEXT – free text

DATE – attached calendar is needed Vx_Required_yn – helps UI enforce needed parameters

Vx_LOV_tx –name of the corresponding value list

Vx_Convert_Tx – any expression with one input Example - 'to_date(:1, ''YYYYMMDD'')' – transformation to the real date

Should always use bind variable with correct ID

The Registered Function


35/56

35 of 56

The Registered Function

f unct i on f_getEmp_CL ( i _j ob_t x var char 2) r et ur n CLOB i s

v_out_cl CLOB;pr ocedur e p_add( pi _t x var char 2) i sbegi n

dbms_l ob. wr i t eappend( v_out _cl , l engt h( pi _t x) , pi _t x) ;

end;begi n

dbms_l ob. cr eat et empor ar y( v_out _cl , t r ue, dbms_l ob. cal l ) ;

p_add('');f or c i n ( sel ect ' ' | | ' ' | | empno| | ' ' | |

' ' | | ename| | ' ' | |' ' r ow_t x

f r om emp wher e j ob = i _j ob_t x)l oop

p_add( c. r ow_t x) ;end l oop;

p_add('');r et ur n v_out _cl ;end;

The Wrapper (1)


36/56

36 of 56

The Wrapper (1)

f unct i on f_wrapper_cl ( i _i d_nr ,v1_tx var char 2: =nul l , . . . , v5_tx var char 2: =nul l )r et ur n CLOB i s

v_out _cl CLOB;

v_sql _t x var char 2( 2000) ;v_r ec t _ext r a_ui %r owt ype;

begi nsel ect * i nt o v_r ec f r om t_extra_ui wher e i d_nr =i _i d_nr ;

i f v_rec.v1_label_tx is not null t henv_sql _t x: =nvl ( v_rec.v1_convert_tx, ' :1' ) ;

end i f ;. . .i f v_rec.v5_label_tx is not null t hen

v_sql _t x: =v_sql _t x| | ' , ' | |

nvl ( v_rec.v5_convert_tx, ' :5' ) ;end i f ;

The Wrapper (2)


37/56

37 of 56

The Wrapper (2)v_sql _t x: =' begi n :out: =' | | v_rec.function_tx| |

' ( ' | | v_sql_tx| | ' ) ; end; ' ;

i f v5_tx i s not nul l t henexecut e i mmedi at e v_sql _t xusing out v_out_cl, v1_tx,…,v5_tx;

. . .el si f v1_tx i s not nul l t henexecut e i mmedi at e v_sql _t xusing out v_out_cl, v1_tx;

el seexecut e i mmedi at e v_sql _t x using out v_out_cl;end i f ;

r et ur n v_out _cl ;end;

11g – Critical Addition


38/56

38 of 56

11g Critical Addition

DBMS_ASSERT – validating strings: SQL_OBJECT_NAME (string) – checks whether or

not string is a valid object SIMPLE _SQL_NAME – checks whether or not string

is a valid SQL name

SCHEMA_NAME – validates that passed string is avalid schema

ENQUOTE_NAME – adds a second quote to everyinstance in the name (and double quotes around)

ENQOUTE_LITERAL – adds single quotes

Nightmare of SOX auditor (2)


39/56

39 of 56

g ( )

f unct i on F_GET_col _TX

( i _t abl e_t x, i _showcol _t x, i _pk_t x, i _pkVal ue_nr )r et ur n var char 2 i s

v_out _t x var char 2( 4000) ;

v_sql _t x var char 2( 32000) ;Begi n

v_sql _t x: =

' sel ect t o_char ( ' | | i _showcol _t x| | ' ) f r om ' | |

dbms_assert.simple_sql_name(i_table_tx)||' wher e ' | |

dbms_assert.simple_sql_name(i_pk_tx)||

| | ' =: v01' ;EXECUTE I MMEDI ATE v_sql _t x I NTO v_out _t x

USI NG i _pkVal ue_nr ;

r et ur n v_out _t x;

end;


40/56

40 of 56

Object Dependencies

Critical Point


41/56

41 of 56

Critical Point

Dynamic SQL is executed at runtime, therefore:Bad things

No dependencies to follow up No way to determine exactly what will be executed

Good things

You can reference objects that may not be in the

database (yet).

You have a “back door” to resolve logical dead loopsor hide existing dependencies.

How can you find dependencies?


42/56

42 of 56

How can you find dependencies?

Use repositories:

It is simple to compare your data to Oracle data

dictionaries!Use samplers:

Generate all possible (or as many as possible) permutations of the code to be executed and create

PL/SQL modules with that code.

Record all dependencies and keep a simple module thatreferences the same set of objects.

If sampler becomes invalid, this means that you have

problems.

Why do you need a back door?


43/56

43 of 56

y do you eed a bac doo

Code generators:

Wrap call of generated modules into dynamic SQL

Then you can refresh them without invalidating all

dependencies

Logical loops: Sometimes the simplest way out of the dependency

loop is to convert one of the calls to dynamic SQL


44/56

44 of 56

Bulk Operations

Supported Features


45/56

45 of 56

pp

Native Dynamic SQL supports object collections

and all kinds of operations on object collections:

FORALL (Currently only in USING clause)

BULK COLLECT

DBMS_SQL supports only arrays as bulk list of bind variables

Fixed in 11g!

FORALL


46/56

46 of 56

FORALL – with limitations: In USING clause:

Right:

forall i in dept.first..dept.last

execut e i mmedi at e

' del et e f r om emp wher e dept no=: 1' using dept(i);

Wrong:

forall i in dept.first..dept.last

execut e i mmedi at e ' dr op t abl e t _' | | dept(i); The whole object only (fixed in 11g!)

Right - dept(i)

Wront - dept(i).deptno

BULK COLLECT


47/56

47 of 56

Syntax:

EXECUTE IMMEDIATE . . .

BULK COLLECT INTO v_col l ect _t t ;Major advantages:

Although dynamic SQL does not support any PL/SQLdatatypes, you can use RECORD as an output of a

dynamic query.

Dynamic SQL does support all user-defined SQL

datatypes.

And everybody forgets…


48/56

48 of 56

y y g

Constructor should be used INSIDE of dynamic SQLtype lov_oty i s obj ect ( i d_nr …, di sp_t x…) ;type lov_nt as t abl e of l ov_ot y;

f unct i on f _get Lov_nt( i _t abl e_t x, i _i d_t x, i _di sp_t x)

r et ur n l ov_nt i s

v_out_nt lov_nt := lov_nt();begi n

execut e i mmedi at e ' sel ect lov_oty( '

| | i _i d_t x| | ' , ' | | i _di sp_t x| | ' ) ' | |' f r om ' | | i _t abl e_t x bulk collect into v_out_nt;r et ur n v_out _nt ;

end;


49/56

49 of 56

Performance

andResource Utilization

Keep in mind (Very Important!)


50/56

50 of 56

EXECUTE IMMEDIATE and dynamic cursors:

If you use bind variables – 1 hard + N soft parses

If you don’t use bind variables – N hard parses

DBMS_SQL – the same plus: Extra option – only 1 parse

Use Case


51/56

51 of 56

The problem:

Users upload CSV files

Name of file defines type Column headers map directly to table columns

1 row of file = 1 logical group (1..N real rows)

Group-level validation

The solution:

Universal CSV Loader Build all inserts on the fly

Build Insertsl


52/56

52 of 56

Decl ar e

t ype i nt eger _t t i s t abl e of i nt eger ;v_cur_tt integer_tt;Begi nf or r i n v_gr oupRow_t t . f i r st . . v_gr oupRow_t t . l ast l oop

v_cur_tt(r):=DBMS_SQL.OPEN_CURSOR;f or c i n c_col s( v_mapRows_t t ( r ) ) l oopf or i i n v_header _t t . f i r st . . v_header _t t . l ast l oop

i f v_header _t t ( i ) . t ext =c. name_t x t henv_col _ t t ( i ) : =c;v_col _t x: =v_col _t x| | ' , ' | | v_col _t t ( i ) . vi ewcol _t x;v_val _t x: =v_val _t x| | ' , : ' | | v_col _t t ( i ) . vi ewcol _t x;

end i f ;end l oop;

end l oop;v_sql_tx:='insert into '||v_map_rec.view_tx||

'('||v_col_tx||') values('||v_value_tx||')';

DBMS_SQL.PARSE(v_cur_tt(r),v_sql_tx,DBMS_SQL.NATIVE);

end l oop;

Process Dataf i i 2 t t t


53/56

53 of 56

f or i i n 2. . v_r ow_t t . count

l oopf or r i nv_gr oupRow_t t . f i r st . . v_gr oupRow_t t . l astl oop

f or c i n v_col _t t . f i r st . . v_col _t t . l astl oop

i f v_col _t t ( c) . i d = v_mapRows_t t ( r ) t hen

DBMS_SQL.BIND_VARIABLE(v_cur_tt(r),':'||v_col_tt(c).viewcol_tx,

v_data_tt(c).text);

end i f ;end l oop;v_nr:=dbms_sql.execute(v_cur_tt(r));

end l oop;end l oop;

Best Practices


54/56

54 of 56

Whenever possible:

Use BULK operations

Minimize parsing

Use bind variables

Options to consider: Build a code repository

Use generated code

Summary


55/56

55 of 56

Dynamic SQL does:

Significantly extend the list of available options for

resolving many problems.

Provide extra maneuvering room in production

environments.Dynamic SQL should NOT:

be considered a substitute for good analysis. be used where “regular” solutions are valid.

Contact Information


56/56

56 of 56

Dr. Paul Dorsey – [email protected] Michael Rosenblum – [email protected]

Dulcian website - www.dulcian.com

Developer AdvancedForms & ReportsDeveloper AdvancedForms & Reports Designer

Handbook

Designer

Handbook

Available now!

Oracle PL/SQL for Dummies

Design Using UMLObject ModelingDesign Using UMLObject Modeling

Date post:	06-Jul-2018
Category:	Documents
Upload:	sbabuind
View:	219 times
Download:	0 times

2008 Laoug2 Dynamic SQL

Documents