Module 7: A MultiLoad Application

After completing this module, you will be able to:

Describe the tables involved in a MultiLoad job.

Set error limits as a time value or as a percentage of loaded rows.

Specify a checkpoint interval.

Redefine input record layout.

New Accounts Application – Description

Transaction(s)• Applied to Balance• Added to History

NewAccountInformation

Customer details• New• Existing

MULTILOAD

CustomerC #UPI

Accounts A #UPI

Trans_HistA # DATE

NUPI

Account_CustomerA # C #

NUSIUPI

• Each New Account requires an INSERT into the Accounts table and an INSERT into Account_Customer.

• An Account can be opened for new or pre-existing customer(s) – UPSERT to the Customer table.

• Each New Account will probably require an opening transaction – UPDATE to Account (balance) and INSERT to Trans_History.

New Accounts Application Script (1 of 3)

.LOGTABLE Newacct_logtable_mld;

.LOGON tdpid/username, password;

.BEGIN IMPORT MLOAD TABLES Accounts, Account_Customer, Customer, Trans_Hist ;

.LAYOUT New_Acc_Layout ;

.FILLER in_Field_Indicator 1 CHAR(1) ;

.FIELD in_Account_Number 2 INTEGER ;

.FIELD in_Number * INTEGER ;

.FIELD in_Street * CHAR(25) ;

.FIELD in_City * CHAR(20) ;

.FIELD in_State * CHAR(2) ;

.FIELD in_Zip_Code * INTEGER ;

.FIELD in_Balance_Forward * INTEGER ;

.FIELD in_Balance_Current * INTEGER ;

.FIELD in_Customer_Number 2 INTEGER ;

.FIELD in_Last_Name * CHAR(25) ;

.FIELD in_First_Name * CHAR(20) ;

.FIELD in_Social_Security * INTEGER ;

.FIELD in_AC_Account_Number 2 INTEGER ;

.FIELD in_AC_Customer_Number * INTEGER ;

.FIELD in_Trans_Number 2 INTEGER ;

.FIELD in_Trans_Account_Number * INTEGER ;

.FIELD in_Trans_ID * INTEGER ;

.FIELD in_Trans_Amount * INTEGER ;

New Accounts Application Script (2 of 3)

.DML LABEL Label_A ; INSERT INTO Accounts VALUES ( :in_Account_Number, :in_Number, :in_Street, :in_City ,:in_State, :in_Zip_Code,

:in_Balance_Forward, :in_Balance_Current ) ;

.DML LABEL Label_B ; INSERT INTO Account_Customer VALUES ( :in_AC_Account_Number, :in_AC_Customer_Number);

.DML LABEL Label_C MARK MISSING UPDATE ROWS DO INSERT FOR MISSING UPDATE ROWS ; UPDATE Customer SET Last_Name = :in_Last_Name WHERE Customer_Number = :in_Customer_Number ; INSERT INTO Customer VALUES ( :in_Customer_Number, :in_Last_Name, :in_First_Name, :in_Social_Security ) ;

.DML LABEL Label_D ; INSERT INTO Trans_Hist VALUES ( :in_Trans_Number, DATE, :in_Trans_Account_Number, :in_Trans_ID, :in_Trans_Amount ) ; UPDATE Accounts

SET Balance_Current =Balance_Current + :in_Trans_Amount WHERE Account_Number =:in_Trans_Account_Number ;

.IMPORT INFILE datain4 LAYOUT New_Acc_LayoutAPPLY Label_A WHERE (in_Field_Indicator = 'A' )APPLY Label_B WHERE (in_Field_Indicator = 'B' )APPLY Label_C WHERE (in_Field_Indicator = 'C' )APPLY Label_D WHERE (in_Field_Indicator = 'D' ) ;

.END MLOAD ;

New Accounts Application Script (3 of 3)

.IF (&SYSINSCNT1 > 0 OR &SYSUPDCNT1 > 0) THEN;COLLECT STATISTICS ON Accounts;

.ENDIF;

.IF (&SYSINSCNT2 > 0) THEN;COLLECT STATISTICS ON Account_Customer;

.ENDIF;

.IF (&SYSINSCNT3 > 0 OR &SYSUPDCNT3 > 0) THEN;COLLECT STATISTICS ON Customer;

.ENDIF;

.IF (&SYSINSCNT4 > 0) THEN;COLLECT STATISTICS ON Trans_Hist;

.ENDIF;

.LOGOFF ;

Note:.BEGIN IMPORT MLOAD TABLES Accounts, Account_Customer, Customer, Trans_Hist ;

1st table 2nd table 3rd table 4th table

MultiLoad environment variables can be checked to optionally COLLECT STATISTICS as part of the job.

&SYSDELCNT_ where _ is 1 to 5&SYSINSCNT_ "&SYSUPDCNT_ "&SYSETCNT_ "&SYSUVCNT_ "

.BEGIN IMPORT Task Commands

• Specifies the tables and optionally the work and error tables used in this MultiLoad job.

• Also used to specify miscellaneous MultiLoad options such as checkpoint, sessions, etc.

.BEGIN [IMPORT] MLOAD

TABLES tname1, tname2, ...

WORKTABLES wt_table1, wt_table2, …

ERRORTABLES et_ table1 uv_table1, et_ table2 uv_table2, ...

ERRLIMIT errcount [errpercent]

CHECKPOINT rate (Default – 15 min.)

SESSIONS limit (Default – 1 per AMP + 2)

TENACITY hours (Default – 4 hours)

SLEEP minutes (Default – 6 minutes)

AMPCHECK NONE | APPLY | ALL

NOTIFY OFF | LOW | MEDIUM | HIGH . . .

;

.END MLOAD ;

Work Tables

IMPORT Task: WORKTABLES wt_table1, wt_table2, …

DELETE Task: WORKTABLES wt_table1

• Default is in user’s default database and the work table is named WT_TableName.

• Alternative may be specified as DataBaseName.WorkTableName.

• There must be one work table defined for each data table.

.BEGIN [IMPORT] MLOAD

TABLES Employee, PayCheck

WORKTABLES util_db.WT_Emp

,util_db.WT_Pay

ERRORTABLES util_db.ET_Emp util_db.UV_Emp

,util_db.ET_Pay util_db.UV_Pay

. . . ;

.BEGINparameters

.BEGINparameters

The Error Tables are described on the next page.

Example:

Error Tables

ERRORTABLES et_tab1 uv_tab1, et_tab2 uv_tab2, ...

• Error table 1 (et)– default is the user’s database and the table is named ET_Tablename. – contains any errors that occur in the Acquisition Phase. – contains primary index overflow errors that occur in the Application phase

• Error table 2 (uv)– default is the user’s database and the table is named UV_Tablename. – contains Application Phase errors.

• Uniqueness violations• Constraint errors• Overflow errors on columns other than primary index

.BEGIN [IMPORT] MLOAD

TABLES Employee, PayCheck

WORKTABLES util_db.WT_Emp

,util_db.WT_Pay

ERRORTABLES util_db.ET_Emp util_db.UV_Emp

,util_db.ET_Pay util_db.UV_Pay

. . . ;

.BEGINparameters

.BEGINparameters

Example:

ERRLIMIT

ERRLIMIT ErrCount

Without ERRPERCENT:

• Specifies approximate number of data errors permitted during Acquisition.

• Does not count Uniqueness violations.

ERRLIMIT ErrCount ErrPercent

With ERRPERCENT:

• Specifies a percentage of data errors after an approximate number of records has been transmitted.

.BEGINparameters

.BEGINparameters

ERRLIMIT 10000 5Example:

In this example, after processing 10,000 input records, the system looks for an error rate of 5%.

CHECKPOINT

• Rate may be specified in the Acquisition Phase of a complex IMPORT task as:

– A number of incoming records (exact count; not less than 60)

– A time interval in minutes (approximate; less than 60)

• If no CHECKPOINT value is specified MultiLoad will checkpoint every 15 minutes, and at the end of each Phase. The default is 15 minutes.

.BEGINparameters

.BEGINparameters

CHECKPOINT 30Example 1: In this example, a 30-minute time interval is specified.

CHECKPOINT 100000Example 2: In this example, a checkpoint after 100,000 input records is specified.

More .BEGIN Parameters

SESSIONS

• Used to specify the maximum, and optionally, minimum sessions generated by MultiLoad.

TENACITY

• Number of hours MultiLoad will try to establish a connection to the system.

• The default is 4 hours.

SLEEP

• Number of minutes MultiLoad waits before retrying a logon; must be greater than 0.

• The default is 6 minutes.

NOTIFY

• NOTIFY LOW• NOTIFY MEDIUM for the most significant events.• NOTIFY HIGH for every MultiLoad event that involves an operational decision point.• NOTIFY OFF suppresses the notify option.

.BEGINparameters

.BEGINparametersSESSIONS 64 48

TENACITY 10

SLEEP 3

NOTIFY OFF Note: The MultiLoad manual specifies in detail which events are associated with each level.

More .BEGIN Parameters: AMPCHECK

AMPCHECK NONE | APPLY | ALL

• NONE

– MultiLoad will not perform an AMPCHECK.

– It will proceed if AMPs are offline, provided all target tables are FALLBACK.

• APPLY

– MultiLoad will continue in all phases except the Application phase with AMPs offline, provided all target tables are FALLBACK.

– This is the default.

• ALL

– MultiLoad will not proceed with down AMPs, regardless of the protection-type of the target tables.

– Most conservative option.

.BEGINparameters

.BEGINparameters

DELETE Task Commands

• Specifies the table and optionally the work and error tables used in this MultiLoad Delete task.

• Also used to specify miscellaneous MultiLoad options such as tenacity, sleep, etc.

.BEGIN DELETE MLOAD

TABLES tname1

WORKTABLES wt_table1

ERRORTABLES et_ table1

TENACITY hours (Default – 4 hours)

SLEEP minutes (Default – 6 minutes)

NOTIFY OFF | LOW | MEDIUM | HIGH . . .

;

.END MLOAD

.LAYOUT and .TABLE

.LAYOUT

– Must appear between the .BEGIN IMPORT MLOAD command and the applicable .IMPORT command.

– Must be immediately followed by .TABLE or .FIELD commands.

.TABLE

– The input fields are defined with the same name, data type, and order of an existing table.

.LAYOUT layoutname

CONTINUEIF position = variable ;

INDICATORS

.TABLE tableref ;

Format:

.LAYOUT Parameters — CONTINUEIF

CONTINUEIF

– Record 1 and the immediately following Record 2 each represent a part of the input record.

– Continuation Character Field is defined beginning in character position 1.

.LAYOUT Record_Layout CONTINUEIF 1= 'Y';

Y F1 F2 F3 Record 1

1 2 10 45 80

N FILLER F5 Record 2

1 2 35 80

Physical Input Records

F1 F2 F3 FILLER F5

1 9 44 79 80 112 113 158

Logical Input Transaction

MultiLoad .LAYOUTDefinition

Note: F1 begins in character position 1 and Filler immediately follows F3.

.LAYOUT Parameters — INDICATORS

INDICATORS

– The INDICATORS contain bits that, when equal to 1, represent a null data field.

.LAYOUT Record_Layout INDICATORS;

Indicator Byte(s) F1 F2 F3Physical Input Records

MultiLoad .LAYOUTDefinition

.FIELD and .FILLER

.FIELD fieldname { startpos datadesc } || fieldexp

[ NULLIF nullexpr ]

[ DROP {LEADING / TRAILING } { BLANKS / NULLS }

[ [ AND ] {TRAILING / LEADING } { NULLS / BLANKS } ] ] ;

.FILLER [ fieldname ] startpos datadesc ;

.FIELD

• Input fields supporting redefinition and concatenation.

Startpos identifies the start of a field relative to 1.

Fieldexpr specifies a concatenation of fields in the format:

fieldname1 || fieldname2 [ || fieldname3 …]

The option DROP LEADING / TRAILING BLANKS / NULLS is applicable only to character datatypes, and is sent as a VARCHAR with a 2-byte length field.

.FILLER

• Identifies data NOT to be sent to the Teradata database.

.LAYOUT Command — Examples

.LAYOUT transrecordCONTINUEIF 14 = 'ABC'INDICATORS;

.FIELD field1 1 (char(5)) NULLIF field1 = 'AAAAA' DROP LEADING BLANKS ;

.FILLER field2 * (char(1)) ;

.FIELD field3 * (char(3)) ;

.FIELD field4 field2 | | '&' | | field3 ;

Example 1:

.LAYOUT emp_record;

.TABLE Employee;

Example 2:

Note: Employee is an existing table whose column names and data descriptions are assigned, in the order defined, to fields of the input data records.

Redefining the Input — Example

• A bank loads daily transactions sequentially on a tape for batch processing by MultiLoad.

• An input data record might be an Add, Update or Delete, each of which has a different length and contains different fields, as illustrated:

Add New Account A PI F1 F2 F3

Update Existing Account U PI F4 F5

Delete Inactive Account D PI

.LAYOUT Record_Layout ;

.FILLER trans_type 1 CHAR(1) ;

.FIELD PI 2 INTEGER ;

.FIELD F1 * INTEGER ;

.FIELD F2 * CHAR(25) ;

.FIELD F3 * CHAR(20) ;

.FIELD F4 6 CHAR(2) ;

.FIELD F5 * INTEGER ;

Note: FILLER data is not placed into ML work tables.

The .DML Command Options

Defines Labels along with Error Treatment conditions for one or more following INSERTs, UPDATEs or DELETEs to be applied under various conditions:

.DML LABEL Labelname

MARK | IGNORE DUPLICATE INSERT | UPDATE MARK | IGNORE MISSING UPDATE | DELETE DO INSERT FOR [MISSING UPDATE] ROWS ;

[

[ ][

ROWS

Operation: Default:

INSERT (Duplicate violation) Marked in UV_tablenameUPDATE (Duplicate violation) Marked in UV_tablenameUPDATE (Fails - missing row) Marked in UV_tablenameDELETE (Fails - missing row) Marked in UV_tablename

UPSERT (If successful) IgnoredUPSERT (Fails) Mark failure of INSERT in UV_tablename

Example of UPSERT failure: 1. PI value doesn’t exist, so UPDATE can’t occur. 2. INSERT fails because of check violation - e.g., can’t put character data in a numeric field.

MARK or IGNORE Whether or not to record duplicate or missingINSERT, UPDATE, OR DELETE rows into the UV_error_table and continue processing.

The .DML Command Options (cont.)

.DML LABEL Labelname

MARK | IGNORE DUPLICATE INSERT | UPDATE MARK | IGNORE MISSING UPDATE | DELETE DO INSERT FOR [MISSING UPDATE] ROWS ;

[

[ ][

ROWS

DO INSERT FOR MISSING UPDATE Key statement that indicates an UPSERT. An SQL UPDATE followed by an SQL INSERT is required.

.DML LABEL Action1 DO INSERT FOR MISSING UPDATE ROWS;

The default for an UPSERT operation is to not mark missing update rows.

Example 1:

.DML LABEL Action2 MARK MISSING UPDATE ROWS DO INSERT FOR MISSING UPDATE ROWS;

Example 1:

When the MARK MISSING UPDATE ROWS is used with an UPSERT, this will place (in the UV_table) data rows that can’t be updated (no PI). If the insert also fails, the insert record is also marked in the UV_table.

MultiLoad Statistics

At the end of the application phase, MultiLoad provides statistical information.

****06:06:41 UTY1803 Import Processing Statistics Total

Import 1 Thus farCandidate Records considered . . . . 200000 200000Apply conditions satisfied . . . . . 200000 200000

****06:18:34 UTY0818 Statistics for table ACCOUNTS:Inserts: 50000Updates: 50000Deletes: 0

****06:18:35 UTY0818 Statistics for table ACCOUNT_CUSTOMER:Inserts: 50000Updates: 0Deletes: 0

****06:18:35 UTY0818 Statistics for table CUSTOMER:Inserts: 25000Updates: 25000Deletes: 0

****06:18:35 UTY0818 Statistics for table TRANS_HIST:Inserts: 50000Updates: 0Deletes: 0

INMODs

INMOD to MultiLoad return codes: After a READ call.

0Valid data record in Buffer — EOF not reached. Length field reflects correct length of output record. If an input record was supplied to the INMOD from MLOAD and is to be skipped, the length field should be set to zero. If no input record was supplied, setting the length to zero indicates EOF.

Non 0 INMOD indicates end-of-file condition.

INMOD to MultiLoad return codes: After a non-READ call.

0Indicates a successful operation.

1 Indicates a processing error; MLOAD will terminate

MultiLoad to INMODadditional return codes.

0Calling for the first time. INMOD should open files to read data. MLOAD expects a record.

1MultiLoad expects a record.

2MLOAD has been restarted. INMOD should position itself to the last checkpoint. There may be re-positioning information in the data buffer.

3Request for INMOD to take a Checkpoint. The INMOD should return any information (up to 100 bytes) needed to reposition itself. MLOAD saves this in the Restart Log Table.

4Request for INMOD to reposition itself at the last checkpoint. Repositioning information may be in the data buffer.

5Request for INMOD to wrap up at termination.

6Request for INMOD to initialize and receive record.

7Request for INMOD to receive next record.

MultiLoad INMODDataTeradata

Note:

MultiLoad can also use the FastLoad INMOD return codes.

Summary

• On the .BEGIN statement, optionally, the names of work and error tables can be specified.

• You can:

– Specify error limits and checkpoints.

– Limit sessions.

– Designate time allowed for connection.

– Specify retry time limit.

– Designate the level of notification you prefer.

– Designate how MultiLoad will proceed if AMPs are offline.

• .LAYOUT parameters define the data format.

• .DML commands define Labels and Error Treatment conditions for one or more operations.

• You can use FastLoad or MultiLoad INMODs.

Review Questions

1. Complete the BEGIN statement to accomplish the following:

– Specify an error limit count of 200,000 and an error percentage of 5%.

– Specify a checkpoint at 50,000 records.

– Limit the sessions to 4.

– Set the number of hours to try to establish connection as 6.

.LOGTABLE RestartLog_mld;

.LOGON ________________;

.BEGIN [IMPORT] MLOAD TABLES Trans_Hist

;

.END MLOAD ;

ERRLIMIT 200000 5

CHECKPOINT 50000

SESSIONS 4

TENACITY 6

Review Questions

1. Complete the BEGIN statement to accomplish the following:

– Specify an error limit count of 200,000 and an error percentage of 5%.

– Specify a checkpoint at 50,000 records.

– Limit the sessions to 4.

– Set the number of hours to try to establish connection as 6.

.LOGTABLE RestartLog_mld;

.LOGON ________________;

.BEGIN [IMPORT] MLOAD TABLES Trans_Hist

;

.END MLOAD ;

ERRLIMIT 200000 5

CHECKPOINT 50000

SESSIONS 4

TENACITY 6

Lab Exercises

Lab Exercise 7-1

PurposeIn this lab, you will use MultiLoad to insert the data rows you deleted in a lab from Module 6. The Accounts, Trans, and Customer tables each should have 100 rows in them (from Lab 6-1).

What you need Populated tables and macro AU.Lab7_1.

Tasks1. Export data to a file called data7_1 by executing the macro AU.Lab7_1. Submit the following

commands in BTEQ:.EXPORT DATA FILE = data7_1;EXEC AU.LAB7_1;.EXPORT RESET;

Note: The Trans_Date is exported with an ANSI Date Format of 'YYYY-MM-DD' and a data type of CHAR(10).

2. Prepare a MultiLoad script which inserts rows into one of three different tables using the redefinition feature of MultiLoad. The input record contains a mixture of records which are to be inserted into the Accounts table if the first record byte is an “A”, the Customer table if the first byte is a “C” and the Trans table if the first byte is a “T.”

3. Check your tables for 200 rows per table. Your MultiLoad job should have a final return code of zero and should evidence 100 rows inserted into each of the three tables.

Lab Exercises

Lab Exercise 7-2

PurposeIn this lab, you will prepare and execute a MultiLoad script which performs an 'UPSERT' operation (INSERT MISSING UPDATE) on your Accounts table as a single operation.

What you need Populated tables and macro AU.Lab7_2.

Tasks1. Delete all rows from the Accounts Table and use the following INSERT/SELECT to create 100 rows of

data in your table.

INSERT INTO Accounts SELECT * FROM AU.Accounts WHERE Account_Number LT 20024101 ;

2. Export data to a file data7_2 using the macro AU.lab7_2.

.EXPORT DATA FILE = data7_2 ;EXEC AU.Lab7_2 ;.EXPORT RESET ;

3. Prepare and execute a MultiLoad script which performs an 'UPSERT' operation (INSERT MISSING UPDATE) on your Accounts table as a single operation. Use the data from data7_2 as input to the MultiLoad 'UPSERT' script. If the row exists, UPDATE the Balance_Current with the appropriate incoming value. If not, INSERT a row into the Accounts table.

4. Validate your results. MultiLoad should have performed 100 UPDATES and 100 INSERTS with a final return code of zero.

Lab Solutions for Lab 7-1

cat lab713.mld.LOGTABLE Restartlog713_mld ;.LOGON u4455/tljc30,tljc30 ;.BEGIN MLOAD TABLES Accounts, Customer, Trans ;.LAYOUT Record_Layout _713;.FILLER in_code 1 CHAR(1);.FIELD in_accountno 2 INTEGER;.FIELD in_number * INTEGER;.FIELD in_street * CHAR(25);.FIELD in_city * CHAR(20);.FIELD in_state * CHAR(2);.FIELD in_zip_code * INTEGER;.FIELD in_balancefor * DECIMAL (10,2);.FIELD in_balancecur * DECIMAL (10,2);.FIELD in_custno 2 INTEGER;.FIELD in_lastname * CHAR(30);.FIELD in_firstname * CHAR(20);.FIELD in_socsec * INTEGER;.FIELD in_transno 2 INTEGER;.FIELD in_transdate * CHAR(10);.FIELD in_accid * INTEGER;.FIELD in_transid * CHAR(4);.FIELD in_transamount * DECIMAL(10,2);

(continued).DML LABEL Insert_Acct ; INSERT INTO Accounts VALUES (:in_accountno, :in_number, :in_street,

:in_city, :in_state, :in_zip_code, :in_balancefor,:in_balancecur) ;

.DML LABEL Insert_Cust ; INSERT INTO Customer VALUES ( :in_custno, :in_lastname,

:in_firstname, :in_socsec );

.DML LABEL Insert_Trans ; INSERT INTO Trans VALUES ( :in_transno, :in_transdate, :in_accid,

:in_transid, :in_transamount);

.IMPORT INFILE data7_1 LAYOUT Record_Layout_713 APPLY Insert_Acct WHERE in_code = 'A' APPLY Insert_Cust WHERE in_code = 'C' APPLY Insert_Trans WHERE in_code = 'T';

.END MLOAD ;

.LOGOFF ;

mload < lab713.mld > lab713.out

Lab Solutions for Lab 7-2

cat lab723.mld.LOGTABLE Restartlog723_mld ;.LOGON u4455/tljc30,tljc30 ;.BEGIN IMPORT MLOAD TABLES Accounts ;.LAYOUT Record_Layout_723;

.FIELD in_accountno 1 INTEGER;

.FIELD in_number * INTEGER;

.FIELD in_street * CHAR(25);

.FIELD in_city * CHAR(20);

.FIELD in_state * CHAR(2);

.FIELD in_zip_code * INTEGER;

.FIELD in_balancefor * DECIMAL (10,2);

.FIELD in_balancecur * DECIMAL (10,2);

.DML LABEL Fix_Account DO INSERT FOR MISSING UPDATE ROWS ;

UPDATE Accounts SET Balance_Current = :in_balancecurWHERE Account_Number = :in_accountno ;

INSERT INTO Accounts VALUES (:in_accountno, :in_number, :in_street, :in_city, :in_state, :in_zip_code, :in_balancefor, :in_balancecur);

.IMPORT INFILE data7_2 LAYOUT Record_Layout_723 APPLY Fix_Account;

.END MLOAD;

.LOGOFF;

mload < lab723.mld > lab723.out