Date post: | 27-Feb-2018 |
Category: |
Documents |
Upload: | praveen42517304 |
View: | 226 times |
Download: | 0 times |
of 24
7/25/2019 Incremental Load qlikview
1/24
7/25/2019 Incremental Load qlikview
2/24
Incremental Loadusing qvd files
7/25/2019 Incremental Load qlikview
3/24
Incremental Load
Is sometimes called
Incremental Load
Differential LoadDelta Load
7/25/2019 Incremental Load qlikview
4/24
Incremental Load
Goal:
Load only the new or the changed
records from the database. The restshould already be available, one way or
another.
7/25/2019 Incremental Load qlikview
5/24
omments on !uffer Load
" !uffer #Incremental$ Load
is a solution only for Log files #te%t files$,
but not for D!s.
" !uffer #&tale after ' days$ &elect
is not a good solution. It ma(es a full
Load after ' days. )nd nothing in
between
7/25/2019 Incremental Load qlikview
6/24
Incremental Load
" Load new datafromDatabase table #slow,but few records$
" Load old datafrom*+D file #manyrecords, but fast$
" reate new *+D file
" rocedure must bere-eated for eachtable
7/25/2019 Incremental Load qlikview
7/24
Different D!changes
If source allows
/$ )--end only. #Logfiles$
0$ Insert only. #1o 2-date or Delete$3$ Insert and 2-date. #1o Delete$
4$ Insert, 2-date and Delete.
7/25/2019 Incremental Load qlikview
8/24
/$ )--end only
" 5ust be Log file
" Loads records
added in the end of
the file
7/25/2019 Incremental Load qlikview
9/24
/$ )--end only
Buffer (Incremental)
Load* FromLogFile.txt
(ansi, txt, delimiter is '\t', embedded labels);
Done6!ut it should be renamed to
7!uffer #)--end$ Load8
7/25/2019 Incremental Load qlikview
10/24
0$ Insert only
" an be any D!
" Loads I1&9Ted
records
" 1eeds the field
ModificationDate
7/25/2019 Incremental Load qlikview
11/24
0$ Insert only
QV_Table:
SQLSELECTPrimaryKey, X, Y FROMD!"#L$
WHERE%odi&iation"ime #$(LastExecTime)#;
7/25/2019 Incremental Load qlikview
12/24
0$ Insert only
QV_Table:
SQLSELECTPrimaryKey, X, Y FROMD!"#L$
WHERE%odi&iation"ime #$(LastExecTime)#;
Concatenate
LOADPrimaryKey, X, Y FROMFile.+D;
7/25/2019 Incremental Load qlikview
13/24
0$ Insert only
QV_Table:
SQLSELECTPrimaryKey, X, Y FROMD!"#L$
WHERE%odi&iation"ime #$(LastExecTime)#;
Concatenate
LOADPrimaryKey, X, Y FROMFile.+D;
STORE+!"ableITOFile.+D;
)lmost done!ut there is a small chance
that a record gets loaded twice
7/25/2019 Incremental Load qlikview
14/24
0$ Insert only
QV_Table:
SQLSELECTPrimaryKey, X, Y FROMD!"#L$
WHERE%odi&iation"ime #$(LastExecTime)# AD%odi&iation"ime #$(BeginningThisExecTime)#;
Concatenate
LOADPrimaryKey, X, Y FROMFile.+D;
STORE+!"ableITOFile.+D;
Done6
7/25/2019 Incremental Load qlikview
15/24
3$ Insert and 2-date
" an be any D!
" Loads I1&9Ted and
2D)T9d records
" 1eeds the fields
ModificationDateand
PrimaryKey
7/25/2019 Incremental Load qlikview
16/24
3$ Insert and 2-date
QV_Table:
SQL SELECTPrimaryKey, X, Y FROMD!"#L$
WHERE%odi&iation"ime #$(LastExecTime)#;
Concatenate
LOADPrimaryKey, X, Y FROMFile.+D
WHEREOT$xists(PrimaryKey);
STORE+!"ableITOFile.+D;
Done6
7/25/2019 Incremental Load qlikview
17/24
4$ Insert, 2-date and Delete
" an be any D!
" Loads I1&9Ted and
2D)T9d records
" emoves D9L9T9d
records
" 1eeds the fields
ModificationDateandPrimaryKey
" Tric(y to im-lement
7/25/2019 Incremental Load qlikview
18/24
4$ Insert, 2-date and Delete
QV_Table:
SQL SELECTPrimaryKey, X, Y FROMD!"#L$
WHERE%odi&iation"ime #$(LastExecTime)#;
Concatenate
LOADPrimaryKey, X, Y FROMFile.+D
WHEREOT$X-"(PrimaryKey);
Inner!o"n
SQLSELECTPrimaryKey FROMD!"#L$;
STORE+!"ableITOFile.+D;;
7/25/2019 Incremental Load qlikview
19/24
4$ Insert, 2-date and DeleteListOfDeletedEntries:
SQLSELECTPrimaryKey ASDeleted FROMD!"#L$
WHEREDeletionFlag /
and %odi&iation"ime 0$(LastExecTime)0;
QV_Table:
SQL SELECTPrimaryKey, X, Y FROMD!"#L$WHERE%odi&iation"ime #$(LastExecTime)#;
Concatenate
LOADPrimaryKey, X, Y FROMFile.+D
WHEREOT$xists(PrimaryKey)ADOT$xists(Deleted,PrimaryKey);
Dro#Ta$leListOfDeletedEntries;
STORE+!"ableITOFile.+D;
;
7/25/2019 Incremental Load qlikview
20/24
Last9%ecutionTime > 9rror handling
Let"1is$xe"ime 2o3();
{ Load sequence
Ifri4t$rror5o6nt 7 t%enLetLast$xe"ime "1is$xe"ime;
EndIf
7/25/2019 Incremental Load qlikview
21/24
=inal &cri-t
Let"1is$xe"ime 2o3();
QV_Table:
SQLSELECTPrimaryKey, X, Y FROMD!"#L$
WHERE%odi&iation"ime #$(LastExecTime)#
AD%odi&iation"ime #$(ThisExecTime)#;
ConcatenateLOADPrimaryKey, X, Y FROMFile.+D
WHEREOT$X-"(PrimaryKey);
Inner!o"n SQLSELECTPrimaryKey FROMD!"#L$;
Ifri4t$rror5o6nt 7 t%en
STORE+!"ableITOFile.+D;
LetLast$xe"ime "1is$xe"ime;
EndIf
7/25/2019 Incremental Load qlikview
22/24
&ummary /
" Incremental Load -ossible for
" )--end only. #Logfiles$ ?es6
" Insert only. #1o 2-date or Delete$ ?es6
" Insert and 2-date. #1o Delete$ ?es6
" Insert, 2-date and Delete. &low,
or demands
Deletion=lag
7/25/2019 Incremental Load qlikview
23/24
&ummary 0
" 7Incremental Load8 normally not
equivalent to
!uffer #Incremental$ Load
7/25/2019 Incremental Load qlikview
24/24