Home >Documents >Incremental Load qlikview

Incremental Load qlikview

Date post:27-Feb-2018
Category:
View:220 times
Download:0 times
Share this document with a friend
Transcript:
  • 7/25/2019 Incremental Load qlikview

    1/24

  • 7/25/2019 Incremental Load qlikview

    2/24

    Incremental Loadusing qvd files

  • 7/25/2019 Incremental Load qlikview

    3/24

    Incremental Load

    Is sometimes called

    Incremental Load

    Differential LoadDelta Load

  • 7/25/2019 Incremental Load qlikview

    4/24

    Incremental Load

    Goal:

    Load only the new or the changed

    records from the database. The restshould already be available, one way or

    another.

  • 7/25/2019 Incremental Load qlikview

    5/24

    omments on !uffer Load

    " !uffer #Incremental$ Load

    is a solution only for Log files #te%t files$,

    but not for D!s.

    " !uffer #&tale after ' days$ &elect

    is not a good solution. It ma(es a full

    Load after ' days. )nd nothing in

    between

  • 7/25/2019 Incremental Load qlikview

    6/24

    Incremental Load

    " Load new datafromDatabase table #slow,but few records$

    " Load old datafrom*+D file #manyrecords, but fast$

    " reate new *+D file

    " rocedure must bere-eated for eachtable

  • 7/25/2019 Incremental Load qlikview

    7/24

    Different D!changes

    If source allows

    /$ )--end only. #Logfiles$

    0$ Insert only. #1o 2-date or Delete$3$ Insert and 2-date. #1o Delete$

    4$ Insert, 2-date and Delete.

  • 7/25/2019 Incremental Load qlikview

    8/24

    /$ )--end only

    " 5ust be Log file

    " Loads records

    added in the end of

    the file

  • 7/25/2019 Incremental Load qlikview

    9/24

    /$ )--end only

    Buffer (Incremental)

    Load* FromLogFile.txt

    (ansi, txt, delimiter is '\t', embedded labels);

    Done6!ut it should be renamed to

    7!uffer #)--end$ Load8

  • 7/25/2019 Incremental Load qlikview

    10/24

    0$ Insert only

    " an be any D!

    " Loads I1&9Ted

    records

    " 1eeds the field

    ModificationDate

  • 7/25/2019 Incremental Load qlikview

    11/24

    0$ Insert only

    QV_Table:

    SQLSELECTPrimaryKey, X, Y FROMD!"#L$

    WHERE%odi&iation"ime #$(LastExecTime)#;

  • 7/25/2019 Incremental Load qlikview

    12/24

    0$ Insert only

    QV_Table:

    SQLSELECTPrimaryKey, X, Y FROMD!"#L$

    WHERE%odi&iation"ime #$(LastExecTime)#;

    Concatenate

    LOADPrimaryKey, X, Y FROMFile.+D;

  • 7/25/2019 Incremental Load qlikview

    13/24

    0$ Insert only

    QV_Table:

    SQLSELECTPrimaryKey, X, Y FROMD!"#L$

    WHERE%odi&iation"ime #$(LastExecTime)#;

    Concatenate

    LOADPrimaryKey, X, Y FROMFile.+D;

    STORE+!"ableITOFile.+D;

    )lmost done!ut there is a small chance

    that a record gets loaded twice

  • 7/25/2019 Incremental Load qlikview

    14/24

    0$ Insert only

    QV_Table:

    SQLSELECTPrimaryKey, X, Y FROMD!"#L$

    WHERE%odi&iation"ime #$(LastExecTime)# AD%odi&iation"ime #$(BeginningThisExecTime)#;

    Concatenate

    LOADPrimaryKey, X, Y FROMFile.+D;

    STORE+!"ableITOFile.+D;

    Done6

  • 7/25/2019 Incremental Load qlikview

    15/24

    3$ Insert and 2-date

    " an be any D!

    " Loads I1&9Ted and

    2D)T9d records

    " 1eeds the fields

    ModificationDateand

    PrimaryKey

  • 7/25/2019 Incremental Load qlikview

    16/24

    3$ Insert and 2-date

    QV_Table:

    SQL SELECTPrimaryKey, X, Y FROMD!"#L$

    WHERE%odi&iation"ime #$(LastExecTime)#;

    Concatenate

    LOADPrimaryKey, X, Y FROMFile.+D

    WHEREOT$xists(PrimaryKey);

    STORE+!"ableITOFile.+D;

    Done6

  • 7/25/2019 Incremental Load qlikview

    17/24

    4$ Insert, 2-date and Delete

    " an be any D!

    " Loads I1&9Ted and

    2D)T9d records

    " emoves D9L9T9d

    records

    " 1eeds the fields

    ModificationDateandPrimaryKey

    " Tric(y to im-lement

  • 7/25/2019 Incremental Load qlikview

    18/24

    4$ Insert, 2-date and Delete

    QV_Table:

    SQL SELECTPrimaryKey, X, Y FROMD!"#L$

    WHERE%odi&iation"ime #$(LastExecTime)#;

    Concatenate

    LOADPrimaryKey, X, Y FROMFile.+D

    WHEREOT$X-"(PrimaryKey);

    Inner!o"n

    SQLSELECTPrimaryKey FROMD!"#L$;

    STORE+!"ableITOFile.+D;;

  • 7/25/2019 Incremental Load qlikview

    19/24

    4$ Insert, 2-date and DeleteListOfDeletedEntries:

    SQLSELECTPrimaryKey ASDeleted FROMD!"#L$

    WHEREDeletionFlag /

    and %odi&iation"ime 0$(LastExecTime)0;

    QV_Table:

    SQL SELECTPrimaryKey, X, Y FROMD!"#L$WHERE%odi&iation"ime #$(LastExecTime)#;

    Concatenate

    LOADPrimaryKey, X, Y FROMFile.+D

    WHEREOT$xists(PrimaryKey)ADOT$xists(Deleted,PrimaryKey);

    Dro#Ta$leListOfDeletedEntries;

    STORE+!"ableITOFile.+D;

    ;

  • 7/25/2019 Incremental Load qlikview

    20/24

    Last9%ecutionTime > 9rror handling

    Let"1is$xe"ime 2o3();

    { Load sequence

    Ifri4t$rror5o6nt 7 t%enLetLast$xe"ime "1is$xe"ime;

    EndIf

  • 7/25/2019 Incremental Load qlikview

    21/24

    =inal &cri-t

    Let"1is$xe"ime 2o3();

    QV_Table:

    SQLSELECTPrimaryKey, X, Y FROMD!"#L$

    WHERE%odi&iation"ime #$(LastExecTime)#

    AD%odi&iation"ime #$(ThisExecTime)#;

    ConcatenateLOADPrimaryKey, X, Y FROMFile.+D

    WHEREOT$X-"(PrimaryKey);

    Inner!o"n SQLSELECTPrimaryKey FROMD!"#L$;

    Ifri4t$rror5o6nt 7 t%en

    STORE+!"ableITOFile.+D;

    LetLast$xe"ime "1is$xe"ime;

    EndIf

  • 7/25/2019 Incremental Load qlikview

    22/24

    &ummary /

    " Incremental Load -ossible for

    " )--end only. #Logfiles$ ?es6

    " Insert only. #1o 2-date or Delete$ ?es6

    " Insert and 2-date. #1o Delete$ ?es6

    " Insert, 2-date and Delete. &low,

    or demands

    Deletion=lag

  • 7/25/2019 Incremental Load qlikview

    23/24

    &ummary 0

    " 7Incremental Load8 normally not

    equivalent to

    !uffer #Incremental$ Load

  • 7/25/2019 Incremental Load qlikview

    24/24

Embed Size (px)
Recommended