8/3/2019 LO Data Flow Complete Picture
1/18
LO-COCKPIT extraction
The first and the most important extraction method. LO-COCKPIT
Positioning of LO-COCKPIT:
What is Logistics Cockpit?
We can say that its a new technique to extract logistics information and consists of aseries of a standard extract structures (that is, from a more BW perspective, standard
datasources), delivered in the business content.
Mind you that, the navigation part will no be discussed here but only the theory partwhich is required for understanding LO extraction process. The navigation steps will be
mentioned in brief.
Here we start...........
Data Extraction is the process of loading data from OLTP to OLAP (BW/BI). Here is anillustration...
I have a company, in which daily thousands of transactions happen all over the world. So
to analyze my business on yearly or monthly basis, i am moving to SAP BW/BI, so that ican generate reports and take business decisions.
Tomorrow i am going to load all the data which was captured till yesterday, from SAPR/3 to BW/BI. I do a full load for this. After completing this task, i need to load the
transactions that will happen from tomorrow to BW. This can be done either daily or
weekly or monthy based on the volume of transactions.If there are 1000s of transactions per day, i can use daily load, 10000s - weekly, if in
http://1.bp.blogspot.com/__Lo6_c_tbBw/SJKnK7wpY0I/AAAAAAAAEXk/PLE4Ik8W-Nk/s1600-h/poistion+of+LO+cockpit.JPG8/3/2019 LO Data Flow Complete Picture
2/18
lakhs- monthly.
So, in precise, data has to be extracted in two modes:
1. Full load - Entire data which is available at source is loaded to BW/BI
2. Delta load - Only the new/changed/deleted data is loaded.
Full Load Data FLow:
Let us see, how the data is loaded to BW in Full mode.
http://2.bp.blogspot.com/__Lo6_c_tbBw/SJGjWb9Vx2I/AAAAAAAAEV0/JV0Bw8xfcuo/s1600-h/data+flow+full+load.JPG8/3/2019 LO Data Flow Complete Picture
3/18
Here we need to understand few basic things which happen on R/3 side.
Document Posting means creating a transaction, writing into the application/transactiontables.
So whenever sales order is created ( document posted), it transaction is written into the
database tables/application tables/transaction tables (Ex. EKPO, EKKO, VBAK, VBAP)
Whenever you are doing a full load, setup tables are used.
setup tables:
Access to application tables are not permitted, hence setup tables are there to collect the
required data from the application tables.
When a load fails, you can re-run the load to pull the data from setup tables. Data will bethere in setup tables. Setup tables are used to Initialize delta loads and for full load. Its
part of LO Extraction scenario.
With this option, you avoid pulling from R/3 directly as we need to bring field values
from multiple tables. You can see the data in the setup tables.Setup table table name wiil
be extract structure name followed by SETUP. Set up table names starts with 'MC'followed by application component '01'/'02' etc and then last digits of the datasource
name and then followed by SETUP
Also we can say the communication structure (R/3 side,you can check it in LBWE also)
name followed by 'setup'
example: MC13VD0HDRSETUP
If you want to check data in set up tables you better look at the transaction NPRT
here you can see the table name from which data is picking. Setup tables are cluster tables and are used to extract the data from R/3 Tables.
(LO Extractors)
Basically, for entire application like SD-Billing we have got it's own setupTables...so while filling the set-up tables, we usually fill for the entire application.
8/3/2019 LO Data Flow Complete Picture
4/18
Ex: OLI7BW is for filling setup Tables for SD application.
OLI9BW T-code is for Billing Application,
When u fill the setup Tables, the data from different tables..VBAK, VBAP,VBRK, VBRP...etc will come through communication Structures and saved in
SetupTables...
The main advantage of having setup Tables is, we can read the data in different
levels..Header level as well as Item level.
when we run init load or Full load in BW, the data will be read from Setup Tables
for the first time( Entire data will be read).... and the delta records will be updated
to Delta Queue once the v3 job runs... and we can extract the delta records from
Delta Queue. once we succefully run the init, we can delete setup Tables.
Filling up the set up tables depends on the datasource.
There are different T-codes for the respective extract structures
OLIIBW transaction PM data
OLI1BW INVCO Stat. Setup: Material Movemts
OLI2BW INVCO Stat. Setup: Stor. Loc. StocksOLI3BW Reorg.PURCHIS BW Extract Structures
OLI4BW Reorg. PPIS Extract Structures
OLI7BW Reorg. of VIS Extr. Struct.: OrderOLI8BW Reorg. VIS Extr. Str.: Delivery
OLI9BW Reorg. VIS Extr. Str.: Invoices
OLIABW Setup: BW agency business
OLIFBW Reorg. Rep. Manuf. Extr. StructsOLIIBW Reorg. of PM Info System for BW
OLIQBW QM Infosystem Reorganization for BW
OLISBW Reorg. of CS Info System for BWOLIZBW INVCO Setup: Invoice Verification
OLI7BW is the tcode for Sales Order.
Steps for full load:
*************** This will be updated later - the navigationpart*********************
1.
Delta Load:
As a prerequisite we need to discuss various update methods for delta load.
8/3/2019 LO Data Flow Complete Picture
5/18
1. Serialized V3
2. Queued Delta
3. Direct Delta4. Unserialized V3
Before that we need to understand V1, V2, V3 updates. These are different workprocesses on the application server that makes the update LUW from the running
program and execute it. These is separated to optimize the transaction processingcapabilities.
These are different work processes on the application server that makes the update LUWfrom the running program and execute it. These is separated to optimize the transaction
processing capabilities.
For Example :
If you create/change a purchase order (me21n/me22n), when you press 'SAVE' and see a
success message (PO.... changed..), the update to underlying tables EKKO/EKPO hashappened (before you saw the message). This update was executed in the V1 workprocess.
There are some statistics collecting tables in the system which can capture data forreporting. For example, LIS table S012 stores purchasing data (it is the same data as
EKKO/EKPO stored redundantly, but in a different structure to optimize reporting).
Now, these tables are updated with the transaction you just posted, in a V2 process.Depending on system load, this may happen a few seconds later (after you saw the
success message). You can see V1/V2/V3 queues in SM12 or SM13.
Statistical tables are for reporting on R/3 while update tables are for BW extraction. Andis data stored redundantly in these two (three if you include application tables) sets of
table.
Difference is the fact that update tables are temporary, V3 jobs continually refreshesthese tables (as I understand). This is different from statistics tables which continue to
add all the data. Update tables can be thought of as a staging place on R/3 from where
data is consolidated into packages and sent to the delta queue (by the V3 job).
Update tables can be bypassed (if you use 'direct' or 'queued' delta instead of V3) to send
the updates (data) directly to the BW queue (delta queue). V3 is however better forperformance and so it is an option along with others and it uses update tables.
Statistical table existed since pre BW era (for analytical reporting) and have continued
and are in use when customers want their reporting on R/3.
The structure of statistical table might be different from the update table/BW queue, so,
even though it is based on same data, these might be different subsets of the samesuperset.
V3 collective update means that the updates are going to be processed only when the V3
8/3/2019 LO Data Flow Complete Picture
6/18
job has run.
At the time of oltp transaction, the update entry is made to the update table. Once youhave posted the txn, it is available in the update table and is waiting for the V3 job to run.
When V3 job runs, it picks up these entries from update table and pushes into delta queue
from where BW extraction job extracts it.
Synchronous update (V1 update): Statistics update is carried out at the same time(synchronous) as the document update (in the application tables).
Asynchronous update (V2 update): Document update and the statistics update
take place in different tasks.
So, V1 and V2 updates dont require any scheduling activity.
Collective update (V3 update):As for the previous point (V2), document update is
managed in a separate moment from the statistics update one, but, unlike the V2update, the V3 collective update must be scheduled as a job (via LBWE).
Remember that the V3 update only processes the update data that is successfully
processed with the V2 update.
-------------------------------------------------------------------------------------Serialized V3:
Take an example of the same PO item changing many times in quick succession.
V1 (with enqueue mechanism) ensures that the OLTP tables are updated consistently.
Update table gets these update records which may or may not end up in correct sequence(as there is no locking) when it reaches BW. 'Serialized V3' was to ensure this correct
sequence of update records going from update tables to delta queue (and then to BW).
Since update table records have the timestamp, when the V3 job runs, it can sequence
these records correctly and thus achieve 'serialization'.
The problems in Serialized V3 update are:
Several changes in one second: For technical reasons, collective run updates that
are generated in the same second cannot be serialized. That is, the serialized V3
update can only guarantee the correct sequence of extraction data in a document ifthe document did not change twice in one second.
Different instances and times synchronization: I think its easy to verify how
much it is probable that in a landscape in which there are several applicationservers for the same environment different times can be displayed.The time used
for the sort order in our BW extractions is taken from the R/3 kernel which uses
the operating system clock as a time stamp. But, as experience teaches, in general,
8/3/2019 LO Data Flow Complete Picture
7/18
the clocks on different machines differ and are not exactly synchronized.The
conclusion is that the serialized V3 update can only ensure the correct sequence in
the extraction of a document if the times have been synchronized exactly on allsystem instances, so that the time of the update record (determined from the locale
time of the application server) is the same in sorting the update data.
The V2 update dependence: Not to be pitiless, but the serialized V3 update havealso the fault of depending from the V2 processing successful conclusion.Our
method can actually only ensure that the extraction data of a document is in the
correct sequence (serialized) if no error occurs beforehand in the V2 update, since
the V3 update only processes update data for which the V2 update is successfullyprocessed.Independently of the serialization, its clear that update errors occurred
in the V2 update of a transaction and which cannot be reposted, cause that the V3
updates for the transaction that are still open can never be processed.This could
thus lead to serious inconsistencies in the data in the BW system.
Example:
Take a case where the first update (based on earliest timestamp) to be processed is in
http://3.bp.blogspot.com/__Lo6_c_tbBw/SJKSU4oDCeI/AAAAAAAAEXE/ujs6c9eLAIc/s1600-h/v1v2v3.JPG8/3/2019 LO Data Flow Complete Picture
8/18
language EN (for same PO item). V3 job is then going to process all the update records of
language EN in chronological sequence before going to next language records. If another
language update (for same PO item) happened in between two EN language updates, thisis going to be processed later after all EN updates are processed and thus become out of
sequence.
In the above figure, all the documents in red color (EN language) will be processed first
and later blue colored (IT language), which is an inconsistancy in sequence.
Direct Delta ( 2nd delta update method in our list)
With this update mode,
Each document posting is directly transferred into the BW delta queue
Each document posting with delta extraction leads to exactly one LUW in therespective BW delta queues
Just to remember that LUW stands for Logical Unit of Work and it can be considered as
an inseparable sequence of database operations that ends with a database commit (or a
roll-back if an error occurs).
http://3.bp.blogspot.com/__Lo6_c_tbBw/SJKM3K5mJXI/AAAAAAAAEW8/qLeqdsyAMNY/s1600-h/language+problem+in+v3.JPG8/3/2019 LO Data Flow Complete Picture
9/18
Benifits:
Theres no need to schedule a job at regular intervals (through LBWE Jobcontrol) in order to transfer the data to the BW delta queues; thus, additional
monitoring of update data or extraction queue is not required. Logically, restrictions and problems described in relation to the "Serialized V3
update" and its collective run do not apply to this method: by writing in the deltaqueue within the V1 update process, the serialization of documents is ensured by
using the enqueue concept for applications and, above all, extraction is
independent of V2 update result.
Limits:
The number of LUWs per datasource in the BW delta queues increases
significantly because different document changes are not summarized into one
LUW in the BW delta queues (as was previously for V3 update).Therefore thisupdate method is recommended only for customers with a low occurrence of
documents (a maximum of 10000 document changes - creating, changing or
deleting - between two delta extractions) for the relevant application. Otherwise, a
larger number of LUWs can cause dumps during extraction process.
No documents can be posted during delta initialization procedure from the start of
the recompilation run in R/3 (setup tables filling job) until all records have been
successfully updated in BW: every document posted in the meantime is
irrecoverably lost.
V1 update would be too much heavily burdened by this process.
(Remember that stopping the posting of documents always applies to the entire client).
8/3/2019 LO Data Flow Complete Picture
10/18
------------------------------------------------------------------------------------------
Queued Delta ( the third update method)
With queued delta update mode, the extraction data (for the relevant application) is
written in an extraction queue (instead of in the update data as in V3) and can be
transferred to the BW delta queues by an update collective run, as previously executed
during the V3 update.
After activating this method, up to 10000 document delta/changes to one LUW arecumulated per datasource in the BW delta queues.
If you use this method, it will be necessary to schedule a job to regularly transfer the data
to the BW delta queues
As always, the simplest way to perform scheduling is via the "Job control" function in
LBWE.
SAP recommends to schedule this job hourly during normal operation after successfuldelta initialization, but there is no fixed rule: it depends from peculiarity of every specific
situation (business volume, reporting needs and so on).
Benifits:
When you need to perform a delta initialization in the OLTP, thanks to the logic
of this method, the document postings (relevant for the involved application) can
be opened again as soon as the execution of the recompilation run (or runs, if
several and running in parallel) ends, that is when setup tables are filled, and adelta init request is posted in BW, because the system is able to collect new
http://2.bp.blogspot.com/__Lo6_c_tbBw/SJKbFoGTgnI/AAAAAAAAEXM/1q_1QDFqteo/s1600-h/direct+delta.JPG8/3/2019 LO Data Flow Complete Picture
11/18
document data during the delta init uploading too (with a deeply felt
recommendation: remember to avoid update collective run before all delta init
requests have been successfully updated in your BW!).
By writing in the extraction queue within the V1 update process (that is more
burdened than by using V3), the serialization is ensured by using the enqueueconcept, but collective run clearly performs better than the serialized V3 and
especially slowing-down due to documents posted in multiple languages does notapply in this method.
On the contrary of direct delta, this process is especially recommended for
customers with a high occurrence of documents (more than 10,000 document
changes - creation, change or deletion - performed each day for the application inquestion.
In contrast to the V3 collective run (see OSS Note 409239 Automatically trigger
BW loads upon end of V3 updates in which this scenario is described), an eventhandling is possible here, because a definite end for the collective run is
identifiable: in fact, when the collective run for an application ends, an event
(&MCEX_nn, where nn is the number of the application) is automatically
triggered and, thus, it can be used to start a subsequent job.
Extraction is independent of V2 update.
Limits:
V1 is more heavily burdened compared to V3.
Administrative overhead of extraction queue.
Note:
1. if you want to take a look to the data of all extract structures queues in LogisticCockpit, use transaction LBWQ or "Log queue overview" function in LBWE (but
here you can see only the queues currently containing extraction data).
2. In the posting-free phase before a new init run in OLTP, you should always
execute (as with the old V3) the update collective run once to make sure to emptythe extraction queue from any old delta records (especially if you are already
using the extractor) that, otherwise, can cause serious inconsistencies in your data.3. Then, if you want to do some change (through LBWE or RSA6) to the extractstructures of an application (for which you selected this update method), you have
to be absolutely sure that no data is in the extraction queue before executing these
changes in the affected systems (and especially before importing these changes inproduction environment !).
To perform a check when the V3 update is already in use, you can run in the
target system the RMCSBWCC check report.
8/3/2019 LO Data Flow Complete Picture
12/18
--------------------------------------------------------------------------------------
Unserialized V3 : (The last one)
With this update mode, that we can consider as the serializers brother, the extraction datacontinues to be written to the update tables using a V3 update module and then is read
and processed by a collective update run (through LBWE).But, as the name of this method suggests, the V3 unserialized delta disowns the main
characteristic of his brother: data is read in the update collective run without taking thesequence into account and then transferred to the BW delta queues.
Issues:
Only suitable for data target design for which correct sequence of changes is notimportant e.g. Material Movements
V2 update has to be successful
When this method can be used ?
Only if its irrelevant whether or not the extraction data is transferred to BW in exactly
the same sequence (serialization) in which the data was generated in R/3 (thanks to a
specific design of data targets in BW and/or because functional data flow doesnt requirea correct temporal sequence).
http://2.bp.blogspot.com/__Lo6_c_tbBw/SJKgIPMZSZI/AAAAAAAAEXU/_bCaaeMm4bs/s1600-h/queued+delta.JPG8/3/2019 LO Data Flow Complete Picture
13/18
Here ends the update methods.
************************************************************
Some important points :
The setup tables are the base tables for the Datasource used for Full upload.So if
you are going for only full uploadfull update is possible in LO extractors.
Full update is possible in LO extractors.In the full update whatever data is present
in the setup tables(from the last done init) is sent to BW.
But setup tables do not receive the delta data from the deltas done after the init.So
if ur full update should get ALL data from the source system,u will need to deleteand re-fill setup tables.
**************************************************************
Some Questions:
Question:
The serialized V3 update can guarantee the correct sequence for the extraction data of a
document only if there were no errors in the V2 update. This is because the V3 updateonly processes update data for which a V2 update has be carried out successfully.
Why is V3 dependent on V2, what is V2 and V1 update?
http://2.bp.blogspot.com/__Lo6_c_tbBw/SJKjj9pGAZI/AAAAAAAAEXc/K9IAAh7PaQ8/s1600-h/unserialized+v3.JPG8/3/2019 LO Data Flow Complete Picture
14/18
Answer:
V1 - Synchronous update
V2 - Asynchronous update
V3 - Batch asynchronous update
These are different work processes on the application server that takes the update LUW
(which may have various DB manipulation SQLs) from the running program and executeit. These are separated to optimize transaction processing capabilities.
Taking an example -
If you create/change a purchase order (me21n/me22n), when you press 'SAVE' and see asuccess message (PO.... changed..), the update to underlying tables EKKO/EKPO has
happened (before you saw the message). This update was executed in the V1 work
process.
There are some statistics collecting tables in the system which can capture data for
reporting. For example, LIS table S012 stores purchasing data (it is the same data asEKKO/EKPO stored redundantly, but in a different structure to optimize reporting).
Now, these tables are updated with the txn you just posted, in a V2 process. Depending
on system load, this may happen a few seconds later (after you saw the success message).
You can see V1/V2/V3 queues in SM12 or SM13.
V3 is specifically for BW extraction. The update LUW for these is sent to V3 but is not
executed immediately. You have to schedule a job (eg in LBWE definitions) to processthese. This is again to optimize performance.
V2 and V3 are separated from V1 as these are not as realtime critical (updating statisticaldata). If all these updates were put together in one LUW, system performance
(concurrency, locking etc) would be impacted.
Serialized V3 update is called after V2 has happened (this is how the code running these
updates is written) so if you have both V2 and V3 updates from a txn, if V2 fails or is
waiting, V3 will not happen yet.
BTW, 'serialized' V3 is discontinued now, in later releases of PI you will have only
unserialized V3.
--------------------------------------------------------------------Question:
There are following tables1. Application tables (R/3 tables)
2. Statistical tables (for reporting purpose)
3. update tables
4. BW queue
8/3/2019 LO Data Flow Complete Picture
15/18
For Application tables its V1 update, statistical tables its V2 update and is it that the same
information is again redundantly stored in update tables?
How are statistical tables different from update tables? I mean i understood what
statistical tables are, my question is "Is the same information again redundantly stored inupdate tables for Collective V3 update to pull the records to BW Queue".
I mean is V3 collective update same as Synchronous V3 update? How does the recordsget saved in update tables?
Answer:
Statistical tables are for reporting on R/3 while update tables are for BW extraction. Is
data stored redundantly in these two (three if you include application tables) sets of
table?, yes it is.
Difference is the fact that update tables are temporary, V3 jobs continually refresh thesetables (as I understand). This is different from statistics tables which continue to add all
the data. Update tables can be thought of as a staging place on R/3 from where data isconsolidated into packages and sent to the delta queue (by the V3 job).
Update tables can be bypassed (if you use 'direct' or 'queued' delta instead of V3) to send
the updates (data) directly to the BW queue (delta queue). V3 is however better forperformance and so it is an option alongwith others and it uses update tables.
Statistical table existed since pre BW era (for analytical reporting) and have continuedand are in use when customers want their reporting on R/3.
The structure of statistical table might be different from the update table/BW queue, so,even though it is based on same data, these might be different subsets of the same
superset.
V3 collective update means that the updates are going to be processed only when the V3
job has run. I am not sure about 'synchronous V3'. Do you mean serialized V3?
At the time of oltp transaction, the update entry is made to the update table. Once youhave posted the txn, it is available in the update table and is waiting for the V3 job to run.
When V3 job runs, it picks up these entries from update table and pushes into delta queue
from where BW extraction job extracts it.-----------------------------------------------------------------------------------
Question:
what do you mean by serialization?is it the serialization beween sequence of records in
update tables to the sequence in BW Queue?
and
Can you explain little more about the Collective run performance with different
8/3/2019 LO Data Flow Complete Picture
16/18
languages.
Answer:
The requirement in 'delta' capturing on R/3 side is to be able to capture the delta 'exactly
once in order'.
Take an example of the same PO item changing many times in quick succession.
V1 (with enqueue mechanism) ensures that the OLTP tables are updated consistently.
Update table gets these update records which may or may not end up in correct sequence
(as there is no locking) when it reaches BW. 'Serialized V3' was to ensure this correct
sequence of update records going from update tables to delta queue (and then to BW).
Since update table records have the timestamp, when the V3 job runs, it can sequence
these records correctly and thus achieve 'serialization'. However, there is a technical
problem with this. The timestamp recorded in update record is sent by the applicationserver (where user executed the txn) and if there are multiple app servers there might be
some inconsistency in their system time which can cause incorrect serialization.
Another problem is in the fundamental design of the V3 process. V3 Job sequences the
updates on timestamp, and then processes the update records from update table (to send it
to delta queue), but it does so for one language at a time (update record also has userlogon language stored). Why this is done is not clear to me, but it is a basic design feature
and can not be subverted.
This causes a potential issue if multiple logon languages are used by users. Serialization
may not happen correctly in such a case. Take a case where the first update (based on
earliest timestamp) to be processed is in language EN (for same PO item). V3 job is thengoing to process all the update records of language EN in chronological sequence before
going to next language records. If another language update (for same PO item) happened
in between two EN language updates, this is going to be processed later after all ENupdates are processed and thus become out of sequence. The weblog mentions this
scenario.
These two constraints remain for 'serialized V3' where 'serialization' couldn't be trulyachieved. Hence newer PIs have discarded 'serialized V3' altogether and now you do not
have this option (if you are using a newer PI).
If you use 'serialized V3', you have to be clear that the 'serialization' may not always
work in the above two scenarios (multi language environment, and multiple app servers
or updates to same records in same second(as timestamp has granularity upto secondlevel only)).
***********************************************************************
****
8/3/2019 LO Data Flow Complete Picture
17/18
Now we will discuss about the functions of LO-COCKPIT:
Maintain Extract Strucutres: Here you can add additional fields from thecommunication structures available to the extract structure.
Maintain Data Sources: In the Data source maintenance screen, you can
customize the data source by using the following fields: field name, short text,
selection, hide field, inversion or cancellation field or reverse posting, and fieldonly known in customer exit.
Activating update: By Setting as active, data is written into extract structures both
online as well as during completion of setup tables or restructure table or LO
initialization tables. Depending on the update mode a job has to be scheduled withwhich the updated data is transferred in the background into the central delta
management (Delta Queue).
http://4.bp.blogspot.com/__Lo6_c_tbBw/SJKw0swFFBI/AAAAAAAAEYE/b31BFAc5ftM/s1600-h/maintain+data+source.JPGhttp://3.bp.blogspot.com/__Lo6_c_tbBw/SJKwZCQOpoI/AAAAAAAAEX8/4eOlIGfW5YI/s1600-h/maintain+extract+strucutre.JPG8/3/2019 LO Data Flow Complete Picture
18/18
Controlling update:This talks about the delta update mode you are using and how
do you control the data load based on the volume of data. LO Cockpit supports 4
types of update modes ( delta modes, which we have already discussed):SerializedV3 update,Direct Delta,Queued Delta,Unserialized V3 update.