8/12/2019 Defining Components
1/53
Defining components
Composition view of data
Double-click group object in Type tree view to define and view group components in the
order they appear in data in group view
Defining components
Identify the data types that make up a group
Describe the order that data appears in a group
8/12/2019 Defining Components
2/53
Component lists
Component name in group view refers to the type in the type tree using the full name
Always drag and drop types into component lists
8/12/2019 Defining Components
3/53
Component lists
Component ranges
Component range indicates the number of consecutive occurrences of component allowed
Specified as minimum_occurrences:maximum_occurrences
The default is one and only one, designated by 1:1
The range of 1:1is assumed if no value is displayed
If the component is optional,
minimum occurrences is
zero, designated by 0:1
If component can appear an
unknown number of times;
the maximum_occurrences is S
8/12/2019 Defining Components
4/53
Displaying all range values
Setting the range
Type tree analyzer
Logical analysis verifies integrity of defined relationships
Detects undefined components
Identifies distinguishability problems
Structure analysis verifies integrity of physical type tree file
8/12/2019 Defining Components
5/53
Does not compare definitions to actual data
Type tree must analyze error-free before it can be used in map
All errors must be resolved
Warnings indicate an inconsistency that should be resolved
Type tree analysis results
Messages include:
Error number
L for error in logical analysis
S for error in structural analysis
Text message describing error
(error) or (warning)
Interpreting type tree analysis error messages
1. Identify type in error
2. Read error message relative to type in error
3.
Double click error message in Analyzer Results view to find and select type in type tree
4. Identify component in error by component number
8/12/2019 Defining Components
6/53
Displaying component numbers
8/12/2019 Defining Components
7/53
Validating data definitions
Using Map Designer
Create maps to specify logic necessary to transform data:
Identify data sources and targets
Identify type trees that describe format and structure of data sources and targets
Provide map rules that describe transformation requirements and business rules
Map Designer user interface
Map Designer basics
Fromwindow
Data sources (inputs)
Shows format of input data
Towindow
Data targets (outputs)
Shows format of output data
Cards represent data objects
One input card for each data source
One output card for each data target
Rule bar used to create and view complete map rules
Rule column displays portion of map rules
8/12/2019 Defining Components
8/53
Map preferences
Maps and map source files
Mapis a definition of how to generate outputs from given inputs
Map source file(.mms file) stores one or more map definitions
Composition view
Shows map source file and map definitions they contain
Shows map relationships
Arranged in compositional hierarchy
Functional maps listed beneath executable map that calls it
New maps only appear when at least one card is defined
8/12/2019 Defining Components
9/53
Outline view
Shows map definitions in alphabetical
order
In expanded view
Shows input cards and outputcards for the map definition
Provides access to map
Organizer views
New maps appear immediately in
Outlineview
Navigator map icons
Color and type of icon in the map
navigator provides general information
regarding the map purpose and content
Main, executable map definition
Functional map definition
Map Solution Diagram
Provides graphical view of map
relationships
1. Right-click map in map Outline
or Composition view2.
Select Map Solution Diagram
8/12/2019 Defining Components
10/53
Defining an input card
At a minimum:
1.
Enter CardNamethat describes input
2.
Select TypeTreethat
contains input data definition3. Select Typein type tree
that defines input data
4. Specify Sourceresource adapter
5.
Enter adapter specific
settings
How card definition relates to card view
CardNameappears in card header and a first object in tree structure
Typeappears in parentheses next to card name in card header and determines top level
object for mapping
8/12/2019 Defining Components
11/53
Defining an output card
At a minimum:
1.
Enter a descriptive CardName
2.
Select the type tree
that defines the output
data format
3.
Select Typefrom
the type tree that
defines output data
4. Select Targetadapter
5.
Specify adapterspecific settings
Map Designer card hierarchy
Executable map
Map that is built and run
Top level map that operates on entire files, tables and messages
Minimally must include data description (type tree) and physical location of data (source or
target settings) on cards
8/12/2019 Defining Components
12/53
Creating an executable map
1. Open the Map Designer
2. Create a new map source file
or
Open an existing map source file
3. Create a new map definition (Map > New)
4.
Add an input card for each data source (Card > New)
5.
Add an output card for each data target (Card > New)
6. Enter map rules
7.
Build (compile) the map
How the Transformation Extender engine works
.Validates input data against type tree
starting with Input card #1
.Applies map rules starting with Output card #1
8/12/2019 Defining Components
13/53
Data validation
Parses data from deepest level out:
1.
Validates each field (level 2) in a
record against type properties,
restrictions and component rules2. Validates record (level 1) against
group properties and
component rules
3.
Repeat steps 1 and 2 for each
record until end of file (level 0)
4. Validates file (level 0) against
group properties and
component rules
Validation map: method #1
Map contains input and output cards
Same TypeTreeand Typein both cards
Map creates an output file that can be compared to input file
If input file exactly matches output file, type tree correctly defines data
Validation map: method #1
8/12/2019 Defining Components
14/53
1. Contract tree in input card and output card to only show top level object
2. Map top level object in input to top level object in output to force validation of input data
against type tree definitions and create output file
Validation map: method #2
Map contains input card only
No output file created
Eliminates rules processing
Preferred method when validating large data files
Modifying the type tree
Always open the type tree from the map to ensure that that the type tree is the same one
referenced in the map
8/12/2019 Defining Components
15/53
After making changes to trees
Map uses the saved version of type tree
When the type tree is changed, follow this procedure:
Option to always save after analysis in Transformation Extender Preferences
Troubleshooting aids
Command Server confirmation window
Run Results views
Trace file
Map debugger
Audit logs
Data audit log
Execution audit log
Command Server confirmation window
Displays information that may be of value for debugging
Return message
Input and output object counts
Run time
8/12/2019 Defining Components
16/53
Run Results views
1.
Select Map > View Run Resultsor click to verify that map:
Read correct input
Generated correct output
2.
Select inputs and outputs
to display
Trace file contents
Summary trace to determine card that failed
Trace on input (InputContentTrace)
Data objects found
Why data is found to be not valid
Sizes and counts of data objects
Position of data objects in data stream
Trace on output (RulesTrace)
Data objects built
Data objects that evaluated to NONE
When to use trace
Generate an input trace when map returns a data related error during validation
Generate an output trace when a step-by-step account of data object generation is required
8/12/2019 Defining Components
17/53
Generating the trace file
1. Select Map
Settingsfrom
Mapmenu
2. Expand
MapTrace
options
3.
Set Switchto ON
4. Specify trace
options
5. Build map
6.
Run map
Viewing the trace file
Reading the trace file
Trace file is text file in the format mapname.mtr where mapnameis the name of the
executable map
Use Edit >Find/Replaceto search within trace information to locate error
8/12/2019 Defining Components
18/53
Search on words such as Not, Failed, Wrong, or Invalid
Look immediately above or below error in trace file for additional information
Summary trace
1.
Generate a summary trace to identify the card in error
2.
If an error occurred on input during validation, run an input content trace on the card
If an error occurred on output, run map debugger
8/12/2019 Defining Components
19/53
Interpreting input content trace results
Interpreting trace results
Reasons for validation failure
Input data did not match a type property in type tree
Data object has wrong Initiator or Terminator
Group is missing a required component
Delimiter does not match definition
Data object failed a component rule
Data object not in restriction list
8/12/2019 Defining Components
20/53
Example: Input data did not match type property
Missing required component
8/12/2019 Defining Components
21/53
Input Valid, Unknown Data Found
Transformation Extender found enough data to meet definition but found additional data
Usual causes
Source object defined as having a single occurrence
Source object defined as having zero or more objects and Transformation Extender
has found at least one, but it is followed by additional data
Functional map being built and run
Undefined data at end of file
Undefined data at end of file
Incorrectly defined data
8/12/2019 Defining Components
22/53
Additional validation
During basic validation, data is isolated and compared to type definitions
Additional validation can be added to prevent processing of invalid data
Additional validation can be added to check if data, once isolated, matches specific datavalue or set of values
Use a restriction listto limit item to particular value or set of values
Use a component ruleto specify condition that must be met for a particular
component to be valid
Restriction lists
Limit valid values of item to a specific set
After changing restriction list, type tree must be analyzed and map re-built
Restriction list is case sensitive unless the Ignore caseitem property is set to Yes
Disabling restriction validation
Restrictions can be ignored when running a map
Set ValidationMap Setting to Custom
Set RestrictionErrorto ignore
8/12/2019 Defining Components
23/53
Restriction settings
Include or exclude
Value restrictions
Range restrictions
Character restrictions
Include or exclude value restrictions
Included values are considered to be valid
Excluded values are considered to be not valid
Include or exclude values supported for:
Characters
Numbers
Dates
8/12/2019 Defining Components
24/53
Identifying restriction values
Use when object data is limited to a set of static values
To add restrictions:
Double-click item type in Type tree view or
right-click and select Open
Enter values in restriction columns
Restriction view column content varies based on restriction property settings
Include character restrictions
Include First
Character list of valid first character Include After
Character list of valid characters that
may follow first character
To specify common range of characters,
enter reserved word in restriction list:
8/12/2019 Defining Components
25/53
Exclude character restrictions
Exclude is list of character
substrings to be excluded
Reference Stringis character string that replaces excluded character substring on output
On input, any data that contains any substring in the Excludelist would be marked as invalid
On output, a character text item would be built by using the Reference Stringif the content
contained any of the Excludecharacter substrings
Include or exclude value ranges
8/12/2019 Defining Components
26/53
Trace file with restriction error
Component rules
Used for data validation
Statement about component that determines if data is valid
Evaluate business rule
Evaluate to TRUE or FALSE
Rule can refer to:
Component it applies to
Any component earlier in same component list
8/12/2019 Defining Components
27/53
Shorthand notation for rules
Use $as shorthand to represent current component in rule
Creating component rules
Created in rule bar
Must include full component name in rule
Press and hold Ctrlto drag and drop component name into rule bar
Always press Enter to save changes
8/12/2019 Defining Components
28/53
Component rule references
Rule can refer to:
Component it applies to
Nested components of any component that can be referred to
Using functions in component rules
Component rules can use most Transformation Extender prebuilt functions
8/12/2019 Defining Components
29/53
What is a function?
Expression that produces a result based on a certain operation
Can be nested with other functions
Can be entered in component rules and map rules
Not all Transformation Extender functions are available for use in component rules
Example: COUNT function
Returns number of occurrences of an object
Syntax: COUNT (series-object-expression)
Meaning: COUNT (objects_to_count)
Returns: Result is an integer equal to the number of objects_to_count.If input
argument evaluates to NONE, COUNT returns zero.
8/12/2019 Defining Components
30/53
Functions view
Lists all Transformation Extender functions
Includes description of function syntax
Drag-and-drop
function from
Functions view
into component rule
Component rules
Component rules always evaluated even if an optional component is not present in data
Use WHEN(PRESENT ) if component rule should only be evaluated when data is present
Sample trace message
8/12/2019 Defining Components
31/53
failed COMPONENT RULE test appears in trace message when component rule evaluates to
False
Comments in component rules
Comment can be added to component and map rules
-
Begin with /* and end with */
-
Appear anywhere in a rule
- Cannot be placed within object name
8/12/2019 Defining Components
32/53
When to use what
For data validation use:
Restriction list
When an item is limited to a particular set of valid values
Component rule
To specify a condition that must be met for a particular component to be valid
For assigning a value to an output group or item, use a mapping rule
8/12/2019 Defining Components
33/53
Map Designer methodology
Map rules
Describe how to build an output data object
Can be any combination of the following:
Hard-coded string enclosed in double quotes
Hard-coded numeric values
Map functions
Object names
Mathematical functions
Concatenation of multiple objects using a plus (+) character
Functional map references
Always start with an equal sign (=) character
Are required for every output object
Enter =None for objects that do not require output
8/12/2019 Defining Components
34/53
Map rule examples
8/12/2019 Defining Components
35/53
Functions
Text
BCDTOTEXT
COUNTSTRING
DATETOTEXTFIND
FILLLEFT
FILLRIGHT
HEXTEXTTOSTREAM
ISALPHA
ISLOWER
ISNUMBER
ISUPPER
LEAVEALPHA
LEAVEALPHANUM
LEAVENUM
LEAVEPRINT
LEFT
LOWERCASE
MAX
MID
MIN
NUMBERTOTEXT
OFFSET
PACKAGE
PROPERCASE
REVERSEBYTE
RIGHT
SERIESTOTEXT
SIZE
SQUEEZE
SUBSTITUTE
TEXT
TEXTTOBCD
TEXTTODATE
TEXTTONUMBER
TEXTTOTIME
TIMETOTEXT
TODATETIME
TONUMBER
TRIMLEFT
TRIMRIGHT
UPPERCASE
WORD
Conversion
BCDTOHEX
BCDTOINT
BCDTOTEXTCONVERT
DATETONUMBER
DATETOTEXT
FROMBASETEN
FROMDATETIME
FROMNUMBER
HEXTEXTTOSTREAM
INT
NUMBERTODATE
NUMBERTOTEXT
PACKPACKAGE
REFORMAT
SERIESTOTEXT
SYMBOL
TEXTTOBCD
TEXTTODATE
TEXTTONUMBER
TEXTTOTIME
TIMETOTEXT
TOBASETEN
TODATETIME
TONUMBER
UNPACK
UNZONE
ZONE
External Interface
DDEQUERY
DBQUERY
DBLOOKUP
ECHOIN
EXIT
HANDLEIN
GET
PUT
RUN
Lookup and Reference
CHOOSE
DBLOOKUP
DBQUERYDDEQUERY
EXTRACT
GETANDSET
GETDIRECTORY
GETFILENAME
GETPARTIONNAME
GETRESOURCENAME
INDEX
INDEXABS
LASTERRORCODE
LASTERRORMSGLOOKUP
MEMBER
SEARCHDOWN
SEARCHUP
SORTDOWN
SORTUP
UNIQUE
Bit Manipulation and
Testing
OFFSET
SETOFF
SETON
TESTOFF
TESTON
Error Handling
CONTAINSERRORS
FAIL
ISERROR
LASTERRORCODE
LASTERRORMSG
ONERROR
REJECT
VALID
Date and Time
ADDDAYS
CURRENTDATE
CURRENTTIMECURRENTDATETIME
DATETONUMBER
DATETOTEXT
FROMDATETIME
MAX
MIN
NUMBERTODATE
TEXTTODATE
TEXTTOTIME
TIMETOTEXT
TODATETIME
Math and Statistics
ABS
COUNT
FROMBASETEN
INT
MAX
MIN
MOD
ROUND
SQRT
SUM
TOBASETEN
TRUNCATE
Logical
ALL
OR
NOT
IF
EITHER
WHEN
8/12/2019 Defining Components
36/53
What can map rules do?
Map input object to output object
Extract input records based on specific criteria
Count a conditional number of records
Map input records based on specific qualifier to specific output record
Convert an object in input data from one value to another value
Perform conditional logic
Run another map to produce an output
Entering map rules
Map rules entered in rule bar
Colors distinguish functions from arguments, object names and other map rule components
Key words can be configured with different colors
Invalid rule can be highlighted with background color
Always press ENTERto accept changes
Always drag-and-drop or copy and paste object names from cards
Getting ready for map rules
8/12/2019 Defining Components
37/53
New card default is collapsed view
Expand group objects to lowest level
Grayed rule cells or objects occurring multiple times indicate possible functional
map use
Must enter a rule in every rule cell that is not gray
Mapping items
If output object is an item type, enter map rule at item level
Mapping groups occurring once
If output object is a group that occurs exactly once, enter map rule at group oritem level
(notboth)
Mapping groups in a series
If a group or an item can occur more than once, decide how many occurrences are to be
built
If there is a known number of outputs, use indexing
8/12/2019 Defining Components
38/53
If the number of outputs depend upon a variable number of inputs, use afunctional
map
Indexing an output
Used when you want to:
Map a specific, known number of outputs one object at a time
Create test data
Indexing an output
8/12/2019 Defining Components
39/53
Managing indexes
Reference a specific index by enclosing instance number in square brackets
Example: Record [2] to reference the second record number
Add an index
1. Right-click object
2. Select Add Index
Delete an index
1.
Right-click object
2.
Select Delete Index
Rearrange indexes
1. Select index object
2.
Press Ctrl+Shiftand drag to desired location
3.
Moves rules associated with selected object to new location
8/12/2019 Defining Components
40/53
Functional maps
What is a functional map?
Like a subroutine
Maps portion of data at a time
Number of output objects are created produced based on some variable number of
corresponding input objects
Is not built and run separately from main map
Does not contain references to data sources and targets
Defining a functional map
1.
Determine need for a functional map
2.
Determine input arguments to functional map
3. Create functional map call
4. Create the functional map
Manually by creating a new map and input and output cards
Automatically using the Functional Map Wizard
5.
Enter map rules in functional map output card
Determining the need
Create a functional map when:
Output group object can occur more than once (component range is greater than 1)
and
Some unknown number of objects are created in output (component range is s)
and
Output group type is different from input group type
Determining the arguments
Functional map syntax is used to:
Name the functional map
8/12/2019 Defining Components
41/53
Specify data to pass to the functional map
Functional maps only have access to data passed to them through function arguments
Identify which objects are required to produce output
Data objects must be passed from calling map as an argument in the functional map
call
One argument is thought of as a trigger
I want one output object for each ...
Creating the functional map call
Created on a component that occurs an unknown number of times (range of s)
At least one argument should be component that occurs an unknown number of times
Recommended to precede functional map name with F_to easily distinguish functional
maps
Creating the functional map call example
Functional map example
8/12/2019 Defining Components
42/53
One input card created for each argument in functional map call
Order of input cards matches order of arguments
One output card always created
Creating functional map manually
1. Create a new map (Map > New)
2. Enter new map name exactly as in functional map reference
3.
Add one input card for each argument in the functional map call (Card > New)
4.
Add one output card (Card > New)
5. Enter map rules in the functional map
Creating a functional map using the Functional Map Wizard
Automatically creates functional maps based on map rules
Creates input and output cards
Creates multiple maps when a rule contains calls to more than one functional map
8/12/2019 Defining Components
43/53
Functional Map Wizard results
Functional map is created in the current map source file (.mms) using the parameters
specified in the Functional Map Wizard dialog
8/12/2019 Defining Components
44/53
Functional Map Wizard icons
Meaning Action to Take
Card is undefined (card name, type
tree file, or type name is not valid)
In the Functional Map Wizard dialog, specify
a valid card name, type tree file, or type
name.
Invalid input argument in a
map rule.
Exit from the Functional Map Wizard and
change the invalid input argument in the
rule.
Everything has been properly
specified for given card or map
No action necessary. Functional map can be
created.
Modifying card attributes
8/12/2019 Defining Components
45/53
Functional Map Wizard restrictions
Does not automatically create input cards for arguments defined as expressions (such as
literals) that evaluate to text or number items
Does not run if an argument is invalid
Does not dynamically update if changes are made to the functional map call
Main and functional map input cards
Functional map output cards
8/12/2019 Defining Components
46/53
From arguments to cards
Functional map input cards
One input card must be created for each argument passed from calling map
Input cards must be created in same order as arguments
8/12/2019 Defining Components
47/53
Functional map output card
Only one output card per functional map
Type matches type of output in calling rule
Uses same type tree as output card in calling map
Argument must match input card
8/12/2019 Defining Components
48/53
Card must match output type
Evaluating functional maps
Evaluated once for each combination of occurrences of each argument passed
To conditionally control input to a functional map or output of an object, use functions and
expressions within a functional map call
8/12/2019 Defining Components
49/53
Example
=F_MakeNewOrderRecord (EXTRACT(LegacyRecord:LegacyFile, PARTITION ( ShipToCode
Field:.:LegacyFile, West)), XrefFile )
If any argument evaluates to NONE, the functional map will not be called for thatcombination of arguments
Functional maps build output; functions do not
Tracking rules
Rules are saved with output name, not position
If output objects change sequence, rules are maintained
If object disappears or the name changes, rules become unresolved
What causes unresolved rules
- Name of output type changes in type tree
- Output card is changed to point to a different object
- An intermediate level is added between object in output
Displaying Unresolved Rules view
Automatically displayed when unresolved rules are present
Can be selected from Show Viewmenu
8/12/2019 Defining Components
50/53
Resolving unresolved rules
Drag rules from unresolved rules into place
Use Modifyto find and replace changed type names
Must delete all unresolved rules before building map
8/12/2019 Defining Components
51/53
Map Debug option
Debug complex rules step by step in Map Designer
Set breakpoints on any rule, in executable or functional maps
Visually step into every rule in every map
Inspect data while it is changing
Step In, Step Out, Step Over, and Run commands
Debug maps remotely by referencing map source locally
Map Debugger features
Enable the Debug Mapiconto switch Map Designer into debug mode
When enabled, allows setting of break points on output objects with rules
Building maps with debug mode enabled produces separate map debug file
No changes to compiled map (.mmc) structure, so maps built with debug are same as other
compiled maps
Map Debugger usage
8/12/2019 Defining Components
52/53
Map Debugger operation
Enable debug and build map
Set breakpoints
Run map
Debug mode causes Debug view window to appear
When breakpoint is reached, engine pauses and the rule and its composite functions
and objects appear in Debug view window
Use buttons in Debug view window to continue processing
Adding breakpoints (1 of 2)
8/12/2019 Defining Components
53/53
Debugger view window
Map Debug options
Step Into see what data is held by objects referenced in rule and data returned by functions
and functional maps
Steps into local, non-referenced functional maps and RUN maps contained in same
map source file (.mms)
Literals and operators are not shown
Step Thrugoes to end of current rule
Does not show resultant object values or function returns
Step Overignores current iteration of a rule and goes to beginning of the next rule to be run
Step Thru/Overprocesses rules in functional maps or RUN maps called from the current
rule, unless a breakpoint is set