InformaticaBest Practices

Velocity v8 Best Practices

Best Practices

l B2B Data Exchange

m B2B Data Transformation Installation (for Unix)

m B2B Data Transformation Installation (for Windows)

m Deployment of B2B Data Transformation Services

m Establishing a B2B Data Transformation Development Architecture

m Testing B2B Data Transformation Services

l Configuration Management and Securitym Configuring Security

m Data Analyzer Security

m Database Sizing

m Deployment Groups

m Migration Procedures - PowerCenter

m Migration Procedures - PowerExchange

m Running Sessions in Recovery Mode

m Using PowerCenter Labels

l Data Analyzer Configurationm Deploying Data Analyzer Objects

m Installing Data Analyzer

l Data Connectivitym Data Connectivity using PowerCenter Connect for BW Integration

Server

m Data Connectivity using PowerExchange for WebSphere MQ

m Data Connectivity using PowerExchange for SAP NetWeaver

m Data Connectivity using PowerExchange for Web Services

l Data Migrationm Data Migration Principles

INFORMATICA CONFIDENTIAL BEST PRACTICES 2 of 954

m Data Migration Project Challenges

m Data Migration Velocity Approach

l Data Quality and Profilingm Build Data Audit/Balancing Processes

m Continuing Nature of Data Quality

m Data Cleansing

m Data Profiling

m Data Quality Mapping Rules

m Data Quality Project Estimation and Scheduling Factors

m Developing the Data Quality Business Case

m Effective Data Matching Techniques

m Effective Data Standardizing Techniques

m Integrating Data Quality Plans with PowerCenter

m Managing Internal and External Reference Data

m Real-Time Matching Using PowerCenter

m Testing Data Quality Plans

m Tuning Data Quality Plans

m Using Data Explorer for Data Discovery and Analysis

m Working with Pre-Built Plans in Data Cleanse and Match

l Development Techniquesm Designing Data Integration Architectures

m Development FAQs

m Event Based Scheduling

m Key Management in Data Warehousing Solutions

m Mapping Auto-Generation

m Mapping Design

m Mapping SDK

m Mapping Templates

m Naming Conventions

m Naming Conventions - B2B Data Transformation


m Naming Conventions - Data Quality

m Performing Incremental Loads

m Real-Time Integration with PowerCenter

m Session and Data Partitioning

m Using Parameters, Variables and Parameter Files

m Using PowerCenter with UDB

m Using Shortcut Keys in PowerCenter Designer

m Working with JAVA Transformation Object

l Error Handlingm Error Handling Process

m Error Handling Strategies - Data Warehousing

m Error Handling Strategies - General

m Error Handling Techniques - PowerCenter Mappings

m Error Handling Techniques - PowerCenter Workflows and Data Analyzer

l Integration Competency Centers and Enterprise Architecturem Business Case Development

m Canonical Data Modeling

m Chargeback Accounting

m Engagement Services Management

m Information Architecture

m People Resource Management

m Planning the ICC Implementation

m Proposal Writing

m Selecting the Right ICC Model

l Metadata and Object Managementm Creating Inventories of Reusable Objects & Mappings

m Metadata Reporting and Sharing

m Repository Tables & Metadata Management

m Using Metadata Extensions

m Using PowerCenter Metadata Manager and Metadata Exchange Views


for Quality Assurance

l Metadata Manager Configurationm Configuring Standard Metadata Resources

m Custom XConnect Implementation

m Customizing the Metadata Manager Interface

m Estimating Metadata Manager Volume Requirements

m Metadata Manager Business Glossary

m Metadata Manager Load Validation

m Metadata Manager Migration Procedures

m Metadata Manager Repository Administration

m Upgrading Metadata Manager

l Operationsm Daily Operations

m Data Integration Load Traceability

m Disaster Recovery Planning with PowerCenter HA Option

m High Availability

m Load Validation

m Repository Administration

m Third Party Scheduler

m Updating Repository Statistics

l Performance and Tuningm Determining Bottlenecks

m Performance Tuning Databases (Oracle)

m Performance Tuning Databases (SQL Server)

m Performance Tuning Databases (Teradata)

m Performance Tuning in a Real-Time Environment

m Performance Tuning UNIX Systems

m Performance Tuning Windows 2000/2003 Systems

m Recommended Performance Tuning Procedures

m Tuning and Configuring Data Analyzer and Data Analyzer Reports


m Tuning Mappings for Better Performance

m Tuning Sessions for Better Performance

m Tuning SQL Overrides and Environment for Better Performance

m Using Metadata Manager Console to Tune the XConnects

l PowerCenter Configurationm Advanced Client Configuration Options

m Advanced Server Configuration Options

m Causes and Analysis of UNIX Core Files

m Domain Configuration

m Managing Repository Size

m Organizing and Maintaining Parameter Files & Variables

m Platform Sizing

m PowerCenter Admin Console

m PowerCenter Enterprise Grid Option

m Understanding and Setting UNIX Resources for PowerCenter Installations

l PowerExchange Configurationm PowerExchange for Oracle CDC

m PowerExchange for SQL Server CDC

m PowerExchange Installation (for AS/400)

m PowerExchange Installation (for Mainframe)

l Project Managementm Assessing the Business Case

m Defining and Prioritizing Requirements

m Developing a Work Breakdown Structure (WBS)

m Developing and Maintaining the Project Plan

m Developing the Business Case

m Managing the Project Lifecycle

m Using Interviews to Determine Corporate Data Integration Requirements

l Upgrades


m Upgrading Data Analyzer

m Upgrading PowerCenter

m Upgrading PowerExchange


B2B Data Transformation Installation (for Unix)Challenge

Install and configure B2B Data Transformation on new or existing hardware, either in conjunction with PowerCenter or co-existing with other host applications on the same application server.

Note: B2B Data Transformation (B2BDT) was formerly called Complex Data Exchange (CDE). All references to CDE in this document are now referred to as B2BDT.

Description

Consider the following questions when determining what type of hardware to use for B2BDT:

If the hardware already exists:

1. Is the processor, operating system supported by B2BDT? 2. Are the necessary operating system and patches applied? 3. How many CPUs does the machine currently have? Can the CPU capacity be expanded? 4. How much memory does the machine have? How much is available to the B2BDT application? 5. Will B2BDT share the machine with other applications? If yes, what are the CPU and memory requirements of the other

applications?

If the hardware does not already exist:

1. Has the organization standardized on hardware or operating system vendor? 2. What type of operating system is preferred and supported?

Regardless of the hardware vendor chosen, the hardware must be configured and sized appropriately to support the complex data transformation requirements for B2BDT.

The hardware requirements for the B2BDT environment depend upon the data volumes, number of concurrent users, application server and operating system used, among other factors. For exact sizing recommendations, contact Informatica Professional Services for a B2BDT Sizing and Baseline Architecture engagement.

Planning for B2BDT Installation

There are several variations on the hosting environment from which B2BDT services will be called. This has implications on how B2BDT is installed and configured.

Host Software Environment

The most common configurations are:

l B2BDT to be used in conjunction with PowerCenter l B2BDT as a stand alone configuration l B2BDT in conjunction with non-PowerCenter integration using an adapter for other middleware software such as

WebMethods

In addition, B2BDT 4.4 included a mechanism for exposing B2BDT services through web services so that they could be called from applications capable of calling web services.

Depending on what host options are chosen, installation may vary.

Installation of B2BDT for a PowerCenter Host EnvironmentINFORMATICA CONFIDENTIAL BEST PRACTICES 8 of 954

Be sure to have the necessary Licenses and the additional plug-in to make PowerCenter work. Refer to the appropriate installation guide or contact Informatica support for details on installing B2BDT in PowerCenter environments.

Installation of B2BDT for a Standalone Environment

When using B2BDT services in a standalone environment, it is expected that one of the invocation methods (e.g., web services, .Net, Java APIs, command line or CGI) will be used to invoke B2BDT services. Consult accompanying B2BDT documentation for use in these environments.

Non-PowerCenter Middleware Platform Integration

Be sure to plan for additional agents to be installed. Refer to the appropriate installation guide or contact Informatica support for details for installing B2BDT in environments other than PowerCenter.

Other Decision Points

Where will the B2BDT service repository be located?

The choices for the location of the service repository are i) a path on the local file system or ii) use of a shared network drive. The justification for using a shared network drive is typically to simplify service deployment if two separate B2BDT servers want to share the same repository. While the use of a shared repository is convenient for a multi-server production environment it is not advisable for development as there could be a danger of multiple development teams potentially overwriting the same project files.

When a repository is shared between multiple machines, if a service is deployed via the B2BDT Studio, the Service Refresh Interval setting controls how fast other installations of B2BDT that are currently running detect the deployment of a service.

What are multi-user considerations?

If multiple users share a machine (but not at same time) the environment variable IFConfigLocation4 can be used to set the location of the configuration file to point to a different configuration file for each user.

Security Considerations

As the B2BDT repository, workspace and logging locations are directory-based all directories to be used should be granted read and write permissions for the user identity under which the B2BDT service will run.

The identity associated with the caller of the B2BDT services will also need to have permissions to execute the files installed in B2BDT binary directory.

Special considerations should be given to environments such as web services where the user identify under which the B2BDT service runs is set to be different for the interactive user or the user associated with the calling application.

Log File and Tracing Locations

Log files and tracing options should be configured for appropriate recycling policies. The calling application must have permissions to read, write and delete files to the path that is set for storing these files.

B2BDT Pre-install Checklist

B2BDT has client and server components. Only the server (or engine) component is installed on UNIX platforms. The client or development studio is only supported on the Windows platform. Reviewing the environment and recording the information in a detailed checklist facilitates the B2BDT install.

Minimum System Requirements

Verify that the minimum requirements for Operating System, Disk Space, Processor Speed and RAM are met and record them in


the checklist.

Verify the following:

l B2BDT requires a Sun Java 2 Runtime Environment (version 1.5.X or above). B2BDT bundles with the appropriate JRE version. The installer can be pointed to an existing JRE or a JRE can be downloaded from Sun.

m If the server platform is AIX, Solaris or Linux JRE version 1.5 or higher is installed and configured. m If the server platform is HP-UX, JRE version 1.5 or higher and the Java -AA add-on is installed and configured.

l A login account and directory have been created for the installation l Confirm that the profile file is not write-protected. The setup program needs to update the profile.

m ~/.profile if you use the sh, ksh, or bash shell m ~/.cshrc or ~/.tcshrc if you use the csh or tcsh shell

l 500Mb or more of temporary workspace is available. l Data and Stack Size

m If the server platform is Linux, the data and the stack size are not limited m If the server platform is AIX, the data size is not limited.

PowerCenter Integration Requirements

Complete a separate checklist for integration if you plan to integrate B2BDT with PowerCenter.

For an existing PowerCenter installation, the B2BDT client will need to be installed on at least one PC in which the PowerCenter client resides. Also, B2BDT components will need to be installed on the PowerCenter server. If utilizing an existing PowerCenter installation ensure the following:

l Which version of PowerCenter is being used (8.x required)? l Is the PowerCenter version 32 bit or 64 bit? l Are the PowerCenter client tools installed on the client PC? l Is the PowerCenter server installed on the server?

For new PowerCenter installations, the PowerCenter Pre-Install Checklist needs to be completed. Keep in mind that the same hardware will be utilized for both PowerCenter and B2BDT.

Non-PowerCenter Integration requirements

In addition to general B2BDT requirements, non-PowerCenter agents require that additional components are installed.

B2BDT Agent for BizTalk - requires that Microsoft BizTalk Server (version 2004 or 2006) is installed on the same computer as B2BDT. If B2BDT Studio is installed on the same computer as BizTalk Server 2004, the Microsoft SP2 service pack for BizTalk Server must be installed.

B2BDT Translator for Oracle BPEL - requires that BPEL 10.1.2 or above is installed.

B2BDT Agent for WebMethods - requires that WebMethods 6.5 or above is installed.

B2BDT Agent for WebSphere Business Integration Message Broker requires that WBIMB 5.0 with CSD06 (or WBIMB 6.0) is installed. Also ensure that the platform supports both the B2BDT Engine and WBIMB.

A valid license key is needed to run a B2BDT project and must be installed before B2BDT services will run on the computer. Contact Informatica support to obtain a B2BDT license file (B2BDTLicense.cfg). B2BDT Studio can be used without installing a license file.


B2BDT Installation and Configuration

The B2BDT installation process involves two main components - the B2BDT development workbench (Studio) and the B2BDT Server, which is an application deployed on a server. The installation tips apply to UNIX environments. This section should be used as a supplement to the B2B Data Transformation Installation Guide.

Before installing B2BDT, complete the following steps:

l Verify that the hardware meets the minimum system requirements for B2BDT. Ensure that the combination of hardware and operating system are supported by B2BDT. Ensure that sufficient space has been allocated to the B2BDT serviceDB.

l Apply all necessary patches to the operating system. l Ensure that the B2BDT license file has been obtained from technical support. l Be sure to have administrative privileges for the installation user id. For *nix systems ensure that read, write and

executive privileges have been given for the installation directory.

Adhere to following sequence of steps to successfully install B2BDT.

1. Complete the B2BDT pre-install checklist and obtain valid license keys. 2. Install B2BDT development workbench (studio) on the windows platform. 3. Install the B2BDT server on a server machine. When used in conjunction with PowerCenter, the server component must

be installed on the same physical machine where PowerCenter resides. 4. Install necessary client agents when used in conjunction with Websphere, WebMethods and Biztalk

B2BDT Install Components

l B2B Data Transformation Studio l B2B Data Transformation Engine l Processors l Optional agents l Optional libraries

The table below provides descriptions of each component:

Component Applicable Platform Description

Engine Both UNIX and Windows The runtime module that executes B2BDT data transformations. This module is required in all B2BDT installations.

Studio Windows only The design and configuration environment for creating and deploying data transformations. B2BDT Studio is hosted within Eclipse on Windows platforms. The Eclipse setup is included in the B2BDT installation package.

Document Processors

Both UNIX and Windows A set of components that perform global processing operations on documents, such as transforming their file formats. All the document processors run on Windows platforms, and most of them run on UNIX-type platforms.

Libraries Windows only (see description)

Libraries of predefined B2BDT data transformations, which can be used with industry messaging standards such as EDI, ACORD, HL7, HIPAA, and SWIFT. Each library contains parsers, serializers, and XSD schemas for the appropriate messaging standard. The libraries can be installed on Windows platforms. B2BDT Studio can be used to import the library components to projects, and deploy the projects to Windows or UNIX-type platforms.


Documentation Windows only An online help library, containing all the B2BDT documentation. PDF version of documentation available for UNIX platform

Install the B2BDT Engine

Step 1:

Run the UNIX installation file from the software folder on the installation CD and follow the prompts. Follow the wizard to complete the install.

TIP During the installation a language must be selected. If there are plans to change the language at a later point in time in the Configuration Editor, Informatica recommends that a non-English language is chosen for the initial setup. If English is selected and then later changed to another language some of the services that are required for other languages might not be installed.

B2BDT supports all of the major UNIX-type systems (e.g., Sun Solaris, IBM AIX, Linux and HP-UX). On UNIX-type operating systems, the installed components are the B2BDT Engine and the document processors.

Note: On UNIX-type operating systems, do not limit the data size and the stack size. To determine whether there is currently a limitation, run the following command:

For AIX, HP, and Solaris:

ulimit a

For Linux:

limit

If very large documents are processed using B2BDT, try adjusting system parameters such as the memory size and the file size. There are two install modes possible under UNIX -- Graphical Interface and Console Mode.

The default installation path is /opt/Informatica/ComplexDataExchange.

The default Service Repository Path is /ServiceDB. This is the storage location for data transformations that are


deployed as B2BDT services.

The default Log path is /CMReports. Log Path page is the location where the B2BDT Engine should store its log files. The log path is also known as the reports path.

The repository location, JRE path and Log path can be changed subsequent to the installation using environment variables.

Step 2:

Install the license file. Verify the validity of the license file with the following command:

CM_console v

The system displays information such as the location and validity of the license file (sample output shown below):

$ ./bin/CM_console v Version: 4.4.0(Build:186) Syntax version:4.00.10 Components: Engine Processors Configuration file: /websrvr/informatica/ComplexDataExchange/CMConfig.xml Package identifier: IF_AIX_OS64_pSeries_C64 License information: License-file path: /websrvr/informatica/ComplexDataExchange/CDELicense.cfg Expiration date: 21/02/08 (dd/mm/yyyy) Maximum CPUs: 1 Maximum services: 1 Licensed components: Excel,Pdf,Word,Afp,Ppt

Step 3:

Load the Environment Variables. When the setup is complete, configure the system to load the B2BDT environment variables.

The B2BDT setup assigns several environment variables that point to the installation directory and to other locations that the system needs.

On UNIX-type platforms, the system must be configured to load the environment variables. B2BDT cannot run until this is done. B2BDT setup creates an environment variables file. This can be in either of the following ways:

Manually from the command line

In lieu of loading environment variables automatically, they can be loaded manually from the command line. This must be done upon each log in before using B2BDT.

For sh, ksh, or bash shell, the command is:

.//setEnv.sh

For csh or tcsh shell, the command is:

source //setEnv.csh

Substitute the installation path for as necessary.

Automatically by inserting the appropriate command in the profile or in a script file

To configure the system to load the environment variables file automatically upon log in:


For the sh, ksh, or bash shell, insert the following line in the profile file.

. //setEnv.sh

For csh or tcsh shell, insert the following line in the login file.

source //setEnv.csh

On UNIX-type platforms, B2BDT uses the following environment variables.

Environment Variable Required/Optional Purpose of the Variable

PATH

Required The environment variables file adds /bin to the paths.

Note: In rare instances, the B2BDT Java document processors require that the JRE be added to the path.

On AIX: LIBPATH

On Solaris and Linux: LD_LIBRARY_PATH

On HP-UX: SHLIB_PATH and LD_LIBRARY_PATH

Required The environment variables file adds the installation directory () to the library path.

It also adds the JVM directory of the JRE and its parent directory to the path, for example, /jre1.4/lib/sparc/server and /jre1.4/lib/sparc. This value can be edited to use another compatible JRE.

CLASSPATH

Required The environment variables file adds /api/lib/CM_JavaAPI.jar to the Java class path.

IFCONTENTMASTER_HOME

Required The environment variables file creates this variable, which points to the B2BDT installation directory ().

IFConfigLocation4

Optional The path of the B2BDT configuration file. This for multiple configurations.

The following is an example of an environment variables file (setEnv.csh) on an AIX system. The variable names and values differ slightly on other UNIX-type operating systems.

## B2B Data Transformation Environment settings setenv IFCMPath /opt/Informatica/ComplexDataExchange setenv CMJAVA_PATH /opt/Informatica/ComplexDataExchange/jre1.4/jre/bin/classic: /opt/Informatica/ComplexDataExchange/jre1.4/jre/bin

# Prepend B2B Data Transformation to the PATH if ( ! $?PATH ) then setenv PATH "" endif setenv PATH "${IFCMPath}/bin:${PATH}"

# Add CM & java path to LIBPATH if ( ! $?LIBPATH ) then


setenv LIBPATH "" endif setenv LIBPATH "${IFCMPath}/bin:${CMJAVA_PATH}:${LIBPATH}"

# Update IFCONTENTMASTER_HOME. setenv IFCONTENTMASTER_HOME "${IFCMPath}"

# Prepend CM path CLASSPATH if ( ! $?CLASSPATH ) then setenv CLASSPATH "" endif setenv CLASSPATH "${IFCMPath}/api/lib/CM_JavaAPI.jar:.:${CLASSPATH}"

Step 4:

Configuration settings

Directory Locations

During the B2BDT setup, prompts were completed for the directory locations of the B2BDT repository, log files and JRE. If necessary, alter these locations by editing the following parameters:

Parameter Explanation

CM Configuration/ Directory services/ File system/Base Path

The B2BDT repository location, where B2BDT services are stored.

CM Configuration/ CM Engine/ JVM Location

On UNIX: This parameter is not available in the Configuration Editor on UNIX-type platforms. For more information about setting the JRE on UNIX, see UNIX Environment Variable Reference.

CM Configuration/ General/ Reports directory

The log path, also called the reports path, where B2BDT saves event logs and certain other types of reports.

CM Configuration/ CM Engine/ Invocation

CM Configuration/ CM Engine/ CM Server

These settings control whether B2BDT Engine runs in-process or out-of-process.

B2BDT has a Configuration Editor, for editing the parameters of a B2BDT installation. To open the Configuration Editor on UNIX in graphical mode: Enter the following command:

/CMConfig

Note: The Configuration Editor is not supported in a UNIX console mode.

Some of the Configuration Editor settings are available for all B2BDT installations. Some additional settings vary depending on the B2BDT version and on the optional components that have been installed. The Configuration Editor saves the configuration in an XML file. By default, the file is .

Note: Before editing the configuration save a backup copy of CMConfig.xml. In the event of a problem the backup can be restored. The file /CMConfig.bak is a backup of the original which the setup program


created when B2BDT was installed. Restoring CMConfig.bak reverts B2BDT to its original configuration.

OS environment variables are used to set aspects of the system such as the Java classpath to be used, location of the configuration file for a specific user, home location for the installed B2BDT instance to be used, library paths, etc.

The following table lists some typical configuration items and where they are set:

Type of configuration item Where configured

Memory for Studio B2BDT Configuration application

JVM / JRE usage B2BDT Configuration application

Tuning parameters threads, timeouts etc B2BDT Configuration application

User specific settings Use environment variable to point to different configuration file

Memory for runtime B2BDT Configuration application

Workspace location B2BDT Configuration application (B2BDT 4.3), B2BDT Studio (B2BDT 4.4)

Event generation Set in project properties

Repository location B2BDT Configuration application

In-Process or Out-of-Process Invocation

Out-of-process invocation requires the use of the B2BDT Server application (which is already installed by the install process). The distinction is that running under server mode causes transformations to potentially run slower, but errors will be isolated from the calling process.

For Web Services, sometime the use of Server mode is recommended as the lifetime of the host process then becomes independent of the life time of the process space allocated to run the web service. For example IIS can run web services in a mode where a process dies or is recycled after a call to web services. For B2BDT the first call after a process startup can take up to 3 seconds (subsequent calls are usually milliseconds) hence it is not optimal to start host process on each invocation. Running in server mode keeps process lifetimes independent.

TIP B2BDT Studio or the CM_console command always runs data transformations in-process.

Running out-of-process has the following advantages:

l Allows 64-bit processes to activate 32-bit versions of B2BDT Engine l An Engine failure is less likely to disrupt the calling application l Helps prevent binary collisions with other modules that run in the process of the calling application.

In-process invocation has the following advantage:

l Faster performance than out-of-process.

Thread pool settings


The thread pool controls the maximum number of Engine threads that can run client requests concurrently per process. If the number of client requests exceeds the number of available threads, the Server queues the requests until a thread is available. The default setting is 4.

Some recommendations are summarized in the table below. Actual needs vary depending upon requirements. Best practices and additional recommendations are part of Jumpstart and Base Line Architecture engagements. Contact an Informatica representative for additional information.

Step 5:

Configure ODBC connectivity.

Note: This step is only needed if the ODBC database support features of B2BDT will be used. In such case, an ODBC driver may need to be configured.

Step 6:

Test the installation to confirm that B2BDT operates properly.

Note: Tests are available to test the engine and document processor installation. Refer the directory \setupTests for B2BDT test projects testCME and testCMDP. Sample output would be similar to following:

cd $IFCONTENTMASTER_HOME cp -R setupTests/testCME ServiceDB/ CM_console testCME Test Succeeded

B2BDT Integration with PowerCenter

B2BDT does support using the runtime as a server process to be invoked from PowerCenter on the same physical machine (in addition to offering the ability to invoke the B2BDT runtime engine in-process with the calling environment). While this does constitute a server process in developer terminology, it does not provide full server administration or monitoring capabilities that are typical of enterprise server products. It is the responsibility of the calling environment to provide these features. Part of the overall solution architecture is to define how these facilities are mapped to a specific B2BDT implementation.


Installation of B2BDT for PowerCenter is a straightforward process. All required plugins needed to develop and run B2BDT transformation are installed as part of a PowerCenter installation. However, certain B2BDT versions and B2BDT plugins need to be registered after the install process. Refer to the PowerCenter Installation Guide for details.

Note: A PowerCenter UDO Transformation can only be created if the UDO plug-in is successfully registered in the repository. If the UDO Option is installed correctly then UDO Transformations can be created in the PowerCenter Designer.

Note: ODBC drivers provided for PowerCenter are not automatically usable from B2BDT as licensing terms prohibit this in some cases. Contact an Informatica support representative for further details.


Additional Details for PowerCenter Integration

l B2BDT is a Custom Transformation object within PowerCenter l INFA passes data via memory buffers to the B2BDT engine and retrieves that output via buffers. l The B2BDT engine runs IN-PROCESS with the PowerCenter engine. l The Custom Transformation object for Informatica can be dragged and dropped inside a PowerCenter mapping.

l

When using B2BDT transformation, PowerCenter does NOT process the input files directly, but instead takes a path and filename (from a text file). Then the engine processes the data through the B2BDT parser defined within the mapping. After this the data is returned to the PowerCenter B2BDT transformation for processing by other Informatica transformation objects.

TIP Verify that the Source filename is the name of the text file where both the file path and the file name are present. It can not be the actual file being parsed by PowerCenter and B2BDT. This is the direct versus indirect sourcing of the file.

Useful Tips and Tricks

Version Compatibility?

l Ensure that version of B2BDT is compatible with PowerCenter. Otherwise many issues can manifest in different forms. l In general B2BDT 4.4 is compatible with 8.5 and with 8.1.1 SP4 (and SP4 only), B2BDT 4.0.6 is compatible with 8.1.1,

and B2BDT 3.2 is compatible with PC 7.x. l For more information refer to the Product Availability Matrix.

Service Deployment?

l Ensure that services are deployed on a remote machine where PowerCenter installed. l Services deployed from studio show up in PowerCenter designer B2BDT transformation as a dropdown list (see

screenshot below).


Note: These services are only ones that are deployed on a local machine. If any services are deployed on remote machines the designer will not display them. As it is easy to mistake them for remote services manually ensure that the services for local and remote machines are in sync.

l After making certain that the services are deployed in the remote server that also has PowerCenter installed, the B2BDT transformation can be specified from the UDO (8.1.1) or B2BDT (8.5) transformation on the Metadata Extensions tab.

Last updated: 02-Jun-08 16:23


B2B Data Transformation Installation (for Windows)Challenge

Installing and configuring B2B Data Transformation (B2BDT) on new or existing hardware, either in conjunction with PowerCenter or co-existing with other host applications on the same server.

Note: B2B Data Transformation was formerly called Complex Data Exchange (CDE). Any references to PowerExchange Complex Data Exchange in this document are now referred to as B2B Data Transformation (B2BDT).

Description

Consider the following questions when determining what type of hardware to use for B2BDT:

If the hardware already exists:

1. Is the processor, operating system supported by B2BDT? 2. Are the necessary operating system and patches applied? 3. How many CPUs does the machine currently have? Can the CPU capacity be expanded? 4. How much memory does the machine have? How much is available to the B2BDT application? 5. Will B2BDT share the machine with other applications? If yes, what are the CPU and memory requirements

of other applications?

If the hardware does not already exist:

1. Has the organization standardized on hardware or operating system vendor? 2. What type of operating system is preferred and supported?

Regardless of the hardware vendor chosen, the hardware must be configured and sized appropriately to support the complex data transformation requirements for B2BDT.

Among other factors, the hardware requirements for the B2BDT environment depend upon the data volumes, the number of concurrent users and the application server and operating system used. For exact sizing recommendations, contact Informatica Professional Services for a B2BDT Sizing and Baseline Architecture engagement.

Planning for the B2BDT Installation

There are several variations of the hosting environment from which B2BDT services will be invoked. This has implications on how B2BDT is installed and configured.

Host Software Environment

The most common configurations are:

1. B2BDT to be used in conjunction with PowerCenter 2. B2BDT as a stand alone configuration 3. B2BDT in conjunction with a non-PowerCenter integration using an adapter for other middleware software

such as WebMethods or Oracle BPEL.


B2BDT 4.4 includes a mechanism for exposing B2BDT services through web services so that they can be called from applications capable of calling web services.

Depending on what host options are chosen, installation options may vary.

Installation of B2BDT for a PowerCenter Host Environment

Be sure to have the necessary licenses and the additional plug-in to make PowerCenter work. Refer to the appropriate installation guide or contact Informatica support for details on the installation of B2BDT in PowerCenter environments.

Installation of B2BDT for a Standalone Environment

When using B2BDT services in a standalone environment, it is expected that one of the invocation methods (e.g., Web Services, .Net, Java APIs, Command Line or CGI) will be used to invoke B2BDT services. Consult accompanying B2BDT documentation for use in these environments.

Non-PowerCenter Middleware Platform Integration

Be sure to plan for additional agents to be installed. Refer to the appropriate installation guide or contact Informatica support for details for installing B2BDT in environments other than PowerCenter.

Other Decision Points

Where will the B2BDT service repository be located?

The choices for the location of the service repository are i) a path on the local file system or ii) use of a shared network drive. The justification for using a shared network drive is typically to simplify service deployment if two separate B2BDT servers want to share the same repository. While the use of a shared repository is convenient for a multi-server production environment it is not advisable for development as there could be a danger of multiple development teams potentially overwriting the same project files.

When a repository is shared between multiple machines, if a service is deployed via the B2BDT Studio, the Service Refresh Interval setting controls how fast other installations of B2BDT that are currently running detect the deployment of a service.

What are multi-user considerations?

If multiple users share a machine (but not at same time) the environment variable IFConfigLocation4 can be used to set the location of the configuration file to point to a different configuration file for each user.

Security Considerations

As the B2BDT repository, workspace and logging locations are directory-based all directories to be used should be granted read and write permissions for the user identity under which the B2BDT service will run.

The identity associated with the caller of the B2BDT services will also need to have permissions to execute the files installed in B2BDT binary directory.

Special considerations should be given to environments such as web services where the user identify under which the B2BDT service runs is set to be different for the interactive user or the user associated with the calling


application.

Log File and Tracing Locations

Log files and tracing options should be configured for appropriate recycling policies. The calling application must have permissions to read, write and delete files to the path that is set for storing these files.

B2BDT Pre-install Checklist

It is best to review the environment and record the information in a detailed checklist to facilitate the B2BDT install.

Minimum System Requirements

Verify that the minimum requirements for the Operating System, Disk Space, Processor Speed and RAM are met and record them the checklist.

l B2BDT Studio requires Microsoft .NET Framework, version 2.0. l If this version is not already installed, the installer will prompt for and install the framework automatically. l B2BDT requires a Sun Java 2 Runtime Environment, version 1.5.X or above. l B2BDT bundles with the appropriate JRE version. The installer can be pointed to an existing JRE or a JRE

can be downloaded from Sun. l To install the optional B2BDT libraries, reserve additional space (refer to documentation for additional

information).

PowerCenter Integration Requirements

Complete the checklist for integration if B2BDT will be integrated with PowerCenter.

For an existing PowerCenter installation, the B2BDT client needs to be installed on at least one PC on which the PowerCenter client resides. B2BDT components also need to be installed on the PowerCenter server. If utilizing an existing PowerCenter installation ensure the following:

l Which version of PowerCenter is being used (8.x required)? l Is the PowerCenter version 32 bit or 64 bit? l Are the PowerCenter client tools installed on the client PC? l Is the PowerCenter server installed on the server?

For new PowerCenter installations, the PowerCenter Pre-Install Checklist should be completed. Keep in mind that the same hardware will be utilized for PowerCenter and B2BDT.

For windows Server, verify the following:

l The login account used for the installation has local administrator rights. l 500Mb or more of temporary workspace is available. l The Java 2 Runtime Environment version 1.5 or higher is installed and configured. l Microsoft .NET Framework, version 2.0 is installed.

Non-PowerCenter Integration Requirements


In addition to the general B2BDT requirements, non-PowerCenter agents require that additional components are installed.

B2BDT Agent for BizTalk - requires that Microsoft BizTalk Server (version 2004 or 2006) is installed on the same computer as B2BDT. If B2BDT Studio is installed on the same computer as BizTalk Server 2004, the Microsoft SP2 service pack for BizTalk Server must be installed.

B2BDT Translator for Oracle BPEL - requires that BPEL 10.1.2 or above is installed.

B2BDT Agent for WebMethods - requires that WebMethods 6.5 or above is installed.

B2BDT Agent for WebSphere Business Integration Message Broker requires that WBIMB 5.0 with CSD06 (or WBIMB 6.0) are installed. Also ensure that the Windows platform supports both the B2BDT Engine and WBIMB.

A valid license key is needed to run a B2BDT project and must be installed before B2BDT services will run on the computer. Contact Informatica support to obtain a B2BDT license file (B2BDTLicense.cfg). B2BDT Studio can be used without installing a license file.

B2BDT Installation and Configuration

The B2BDT installation consists of two main components - the B2BDT development workbench (Studio) and the B2BDT Server (which is an application deployed on a server). The installation tips apply to Windows environments. This section should be used as a supplement to the B2B Data Transformation Installation Guide.

Before installing B2BDT complete the following steps:

l Verify that the hardware meets the minimum system requirements for B2BDT. l Ensure that the combination of hardware and operating system are supported by B2BDT. l Ensure that sufficient space has been allocated for the B2BDT serviceDB. l Ensure that all necessary patches have been applied to the operating system. l Ensure that the B2BDT license file has been obtained from technical support. l Be sure to have administrative privileges for the installation user id.

Adhere to the following sequence of steps to successfully install B2BDT.

1. Complete the B2BDT pre-install checklist and obtain valid license keys. 2. Install B2BDT development workbench (studio) on the windows platform. 3. Install the B2BDT server on a server machine. When used in conjunction with PowerCenter, the server

component must be installed on the same physical machine where PowerCenter resides. 4. Install necessary client agents when used in conjunction with WebSphere, WebMethods and BizTalk.

In addition to the standard B2BDT components that are installed by default additional libraries can be installed. Refer to the B2BDT documentation for detailed information on these library components.

B2BDT Install Components

The install package includes the following components.

l B2B Data Transformation Studio


l B2B Data Transformation Engine l Document Processors l Documentation l Optional agents l Optional libraries

The table below provides descriptions of each component:

Component Description

Engine The runtime module that executes B2BDT data transformations. This module is required in all B2BDT installations.

Studio The design and configuration environment for creating and deploying data transformations. B2BDT Studio is hosted within Eclipse on Windows platforms. The Eclipse setup is included in the B2BDT installation package.

Document Processors

A set of components that perform global processing operations on documents, such as transforming their file formats. All the document processors run on Windows platforms, and most of them run on UNIX-type platforms.

Optional Libraries Libraries of predefined B2BDT data transformations, which can be used with industry messaging standards such as EDI, ACORD, HL7, HIPAA, and SWIFT. Each library contains parsers, serializers, and XSD schemas for the appropriate messaging standard. The libraries can be installed on Windows platforms. B2BDT Studio can be used to import the library components to projects in order to deploy the projects to Windows or UNIX-type platforms.

Documentation An online help library, containing all the B2BDT documentation.

Install the B2BDT Studio and Engine

Step 1:

Run the Windows installation file from the software folder on the installation CD and follow the prompts. Follow the wizard to complete the install.


TIP During the installation a language must be selected. If there are plans to change the language at a later point in time in the Configuration Editor, Informatica recommends that a non-English language is chosen for the initial setup. If English is selected and then later changed to another language some of the services that are required for other languages might not be installed.

l The default installation path is C:\Informatica\ComplexDataExchange. l The default Service Repository Path is /ServiceDB. This is the storage location for data

transformations that are deployed as B2BDT services. l The default Log path is /CMReports. The Log Path is the location where the B2BDT

Engine stores its log files. The log path is also known as the reports path.


The repository location, JRE path and Log path can be changed subsequent to the installation using environment variables.


Step 2:

Install the license file. Verify the validity of the license file with the following command:

CM_console v

The system displays information such as the location and validity of the license file.

Step 3:

Configure the Environment Variables.

The B2BDT setup assigns several environment variables which point to the installation directory and to other locations that the system needs.

On Windows, the B2BDT setup creates or modifies the following environment variables:

Environment Variable Required/Optional Purpose of the Variable

PATH

Required The environment variables file adds /bin to the paths. Note: In rare instances, the B2BDT Java document processors require that the JRE be added to the path.


CLASSPATH

Required The setup adds \api\lib\CM_JavaAPI.jar to the path.

CLASSPATH

Required The environment variables file adds /api/lib/CM_JavaAPI.jar to the Java class path.

IFCONTENTMASTER_HOME

Required The setup creates this environment variable, which points to the B2BDT installation directory ().

IFConfigLocation4

Optional The path of the B2BDT configuration file.

Step 4:

Configuration settings.

The configuration application allows for the setting of properties such as JVM parameters, thread pool settings, memory available to the studio environment and many others. Consult the administrators guide for a full list of settings and their effects.

Properties set using the B2BDT configuration application affect both the operation of the standalone B2BDT runtime environment and the behavior of the B2BDT studio environment.

To open the Configuration Editor in Windows, from the Start menu choose

Informatica > B2BDT > Configuration


Some of the Configuration Editor settings are available for all B2BDT installations. Additional settings vary depending on the B2BDT version and the optional components installed. The Configuration Editor saves the configuration in an XML file. By default, the file is .

The B2BDT studio environment should be installed on each developers machine or environment. While advances in virtualization technologies and technologies such as Windows remote desktop connections theoretically allow for multiple users to share the same B2BDT installation, the B2BDT studio environment does not implement mechanisms such as file locking during the authoring of transformations that are needed to secure multiple users from overwriting each others work.

An environment variable can be defined called IFConfigLocation4. The value of the variable must be the path for a valid configuration file (i.e., c:\MyIFConfigLocation4\CMConfig1.xml).

For example, if two users want to run B2BDT Engine with different configurations on the same platform, store their respective configuration files in their home directories. Both files must have the name CMConfig.xml.

Alternately store a CMConfig.xml file in the home directory for one of the users. The other user will use the default configuration file (e.g., /CMConfig.xml).

TIP Always save a backup copy of CMConfig.xml prior to editing. In the event of a problem the last known backup can be restored. The file /CMConfig.bak is a backup of the original which the setup program created when B2BDT was installed. Restoring CMConfig.bak reverts B2BDT to its original configuration.

OS environment variables are used to set aspects of the system such as the Java classpath to be used, location of the configuration file for a specific user, home location for the installed B2BDT instance to be used, library paths, etc.


The following table lists some typical configuration items and where they are set:

Type of configuration item Where configured

Memory for Studio B2BDT Configuration application

JVM / JRE usage B2BDT Configuration application

Tuning parameters threads, timeouts etc B2BDT Configuration application

User specific settings Use environment variable to point to different configuration file

Memory for runtime B2BDT Configuration application

Workspace location B2BDT Configuration application (B2BDT 4.3), B2BDT Studio (B2BDT 4.4)

Event generation Set in project properties

Repository location B2BDT Configuration application

In-Process or Out-of-Process Invocation

Out-of-process invocation requires the use of the B2BDT Server application (which is already installed by the install process). The distinction is that running under server mode causes transformations to potentially run slower, but errors will be isolated from the calling process.

For Web Services, sometime the use of Server mode is recommended as the lifetime of the host process then becomes independent of the life time of the process space allocated to run the web service. For example IIS can run web services in a mode where a process dies or is recycled after a call to web services. For B2BDT the first call after a process startup can take up to 3 seconds (subsequent calls are usually milliseconds) hence it is not optimal to start host process on each invocation. Running in server mode keeps process lifetimes independent.

TIP B2BDT Studio or the CM_console command, always runs data transformations in-process.

Running out-of-process has the following advantages:

l Allows 64-bit processes to activate 32-bit versions of B2BDT Engine l An Engine failure is less likely to disrupt the calling application l Help prevent binary collisions with other modules that run in the process of the calling application.

In-process invocation has the following advantage:


l Faster performance than out-of-process.

Thread pool settings

The thread pool controls the maximum number of Engine threads that can run client requests concurrently per process. If the number of client requests exceeds the number of available threads, the Server queues the requests until a thread is available. The default setting is 4.

Some recommendations are summarized in the table below. Actual needs vary depending upon requirements. Best practices and additional recommendations are part of Jumpstart and Base Line Architecture engagements. Contact an Informatica representative for additional information.

Key Settings Parameters Suggestions

Eclipse settings Memory available to studio By default Eclipse allocates up to 256MB to Java VM

Set to vmargs Xmx512M to allocate 512mb

Log file locations Location security needs to match identity of B2BDT engine

ServiceDB

Need to have read permissions for service db locations

Preprocessor buffer sizes Change if running out of memory during source file processing

Service Refresh Interval


Step 5:

Configure ODBC connectivity.

Note: this step is only needed if the ODBC database support features of B2BDT will be used. In such case, an ODBC driver may need to be configured.

Step 6:

Test the installation to confirm that B2BDT operates properly

Note: Tests are available to test the engine and document processor installation. Refer the directory \setupTests for B2BDT test projects testCME and testCMDP.


B2BDT Integration With PowerCenter

B2BDT does support using the runtime as a server process to be invoked from PowerCenter on the same physical machine (in addition to offering the ability to invoke the B2BDT runtime engine in-process with the calling environment). While this does constitute a server process in developer terminology, it does not provide full server administration or monitoring capabilities that are typical of enterprise server products. It is the responsibility of the calling environment to provide these features. Part of the overall solution architecture is to define how these facilities are mapped to a specific B2BDT implementation.


Installation of B2BDT for PowerCenter is a straightforward process. All required plugins needed to develop and run B2BDT transformation are installed as part of a PowerCenter installation. However, certain B2BDT versions and B2BDT plugins need to be registered after the install process. Refer to the PowerCenter Installation Guide for details.

The repository option copies the B2BDT plug-ins to the Plugin directory. Register the B2BDT plug-ins in the PowerCenter repository.

PowerCenter 7.1.x

Register the UDT.xml plug-in in the PowerCenter Repository Server installation Plugin directory. The B2BDT plug-in will appear under the repository in the Repository Server Administration Console.

PowerCenter 8.1.x

Register the pmudt.xml plug-in in the Plugin directory of the PowerCenter Services installation. When the B2BDT plug-in is successfully registered in PowerCenter 8.1 it will appear in the Administration Console as follows:


Note:A PowerCenter UDO Transformation can only be created if the UDO plug-in is successfully registered in the repository. If the UDO Option is installed correctly then UDO Transformations can be created in the PowerCenter Designer.

Note: ODBC drivers provided for PowerCenter are not automatically usable from B2BDT as licensing terms prohibit this in some cases. Contact an Informatica support representative for further details.

Additional Details for PowerCenter Integration

l B2BDT is a Custom Transformation object within PowerCenter

m INFA passes data via memory buffers to the B2BDT engine and retrieves that output via buffers. m The B2BDT engine runs IN-PROCESS with the PowerCenter engine. m The Custom Transformation object for Informatica can be dragged and dropped inside a PowerCenter

mapping


l

When using B2BDT transformation, PowerCenter does NOT process the input files directly, but instead takes a path and filename (from a text file). Then the engine processes the data through the B2BDT parser defined within the mapping. After this the data is returned to the PowerCenter B2BDT transformation for processing by other Informatica transformation objects.

TIP Verify that the Source filename is the name of the text file where both the file path and the file name are present. It can not be the actual file being parsed by PowerCenter and B2BDT. This is the direct versus indirect sourcing of the file.


Useful Tips and Tricks

Can I use an existing Eclipse install with B2BDT?

l Yes. But make sure it is compatible with the version of B2BDT installation. Check with product compatibility matrix for additional information.

l B2BDT can be made to work with a different version of Eclipse however it is not guaranteed.

Is there a silent install available for B2BDT on Windows?

l As of B2BDT 4.4 there is no silent install mode. But there is likely be a future release.

Version Compatibility?

l Ensure that version of B2BDT is compatible with PowerCenter. Otherwise many issues can manifest in different forms.

l In general B2BDT 4.4 is compatible with 8.5 and with 8.1.1 SP4 (and SP4 only), B2BDT 4.0.6 is compatible with 8.1.1, and B2BDT 3.2 is compatible with PC 7.x.

l For more information refer to the Product Availability Matrix.

Service Deployment?

l Ensure that services are deployed on a remote machine where PowerCenter installed. l Services deployed from studio show up in PowerCenter designer B2BDT transformation as a dropdown list

(see screenshot below).

Note: These services are only ones that are deployed on a local machine. If any services are deployed on remote machines the designer will not display them. As it is easy to mistake them for remote services manually ensure that the services for local and remote machines are in sync.

l After making certain that the services are deployed in the remote server that also has PowerCenter installed, the B2BDT transformation can be specified from the UDO (8.1.1) or B2BDT (8.5) transformation on the Metadata Extensions tab.


Common Installation Troubleshooting Tips

Problem

Problem Description

The following error occurs when opening B2BDT studio:

There was a problem running ContentMaster studio, Please make sure /CMConfig.XML is a valid configuration file (Error code=2)

Solution

To resolve this issue, do the following:

Edit the CMConfig.xml file

Add the below section of code after and before in the file:

C:/Program Files/Itemfield/ContentMaster4/eclipse C:\Documents and Settings\kjatin.INFORMATICA\My Documents\Itemfield\ContentMaster\4.0\workspace

Note: Modify the path names as necessary to match the installation settings.

Problem

Problem Description

The Content Master studio fails to open with the following error:

Failed to Initialize CM engine! CM license is limited to 1 CPU, and is not compatible with this machine's hardware. Please contact support.

Cause

The Content Master license is licensed for a fewer number of CPUs then what is on the machine. While registering incorrect information was entered for number of CPUs and so the license provided is for machine with lesser number of CPUs.

Solution

To resolve the issue do the registration again, enter the right number of CPU and send the new registration.txt to Informatica Support to get the new license. When the new license is received replace it over the existing one in the Content Master Installation directory.


Problem

Problem Description

When launching the Designer after installing the Unstructured Data Option (UDO) option, the following error is displayed:

Failed to load DLL: pmudtclient.dll for Plug-in: PC_UDT

Cause

This error occurs when Content Master has not been installed along with PowerCenter UDO.

Solution

To resolve this issue, install Content Master

Last updated: 31-May-08 19:00


Deployment of B2B Data Transformation Services

Challenge

Outline the steps and strategies for deploying B2B Data Transformation services.

Description

Deployment is a process wherein a data transformation is made available as a service that is accessible to the B2B Data Transformation runtime engine. When a project is published to a specific transformation service, a directory name is created that corresponds to the published service name in the B2B Data Transformation Service DB which forms a runtime repository of services. A CMW file corresponding to the service name will be created in the same directory. The deployed service is stored in the Data Transformation service repository.

On Windows platforms, the default repository location is:

c:\Program Files\Informatica\ComplexDataExchange\ServiceDB

On UNIX platforms, the default location is:

/opt/Informatica/ComplexDataExchange/ServiceDB

Basics of B2B Data Transformation Service Deployment

When running in the B2B Data Transformation studio environment, developers can test the service directly without deployment. However, in order to test integration with the host environment, platform agents or external code, it is necessary to deploy the service.

Deploying the transformation service copies the service with its current settings to the B2B data transformation service repository (also known as the Service DB folder). Deploying a service also sets the entry point for the transformation service.

Note: The location of the service repository is set using the B2B Transformation configuration utility

If changes are made to the project options or to the starting point of the service, it is necessary to redeploy the service in order for the changes to take effect.

When the service is deployed, all service script files, schemas, sample data and other project artifacts will be deployed to the service repository as specified by the B2B Data Transformation configuration options in effect in the studio environment from which the service is being deployed.

A transformation service can be deployed multiple times under different service names with the same or different options for each deployed service. While Informatica recommends only deploying one service from each B2B data transformation project for production, it is useful to deploy a transformation service under different names when testing different option combinations.

Deployment for Test Purposes

It is important to finish configuration and testing of data transformations before deploying it as a B2B Data Transformation service. Deploying the service allows the B2B Data Transformation runtime engine to access and run the project. When running in the B2B Data Transformation studio environment, developers can test the service


directly without deployment. However, in order to test integration with the host environment, platform agents or external code, it is necessary to deploy the service.

Initial Production Deployment of B2B Data Transformation Services

Deploying services in the production environment allows applications to run the transformation services on live data. B2B Data Transformation services can be deployed from the B2B Data Transformation Studio environment computer to a remote computer such as a production server. The remote computer can be a Windows or UNIX-type platform, where B2B Data Transformation Engine is installed.

A service can be deployed to a remote computer by either a) directly deploying it to the remote computer or b) deploying the service locally and then copying the service to a remote computer.

To deploy a service to a remote computer:

1. Deploy the service on the development computer. 2. Copy the deployed project directory from the B2B Data Transformation repository on the development

computer to the repository on the remote computer 3. If you have added any custom components or files to the B2B Data Transformation autoInclude\user directory,

you must copy them to the autoInclude\user directory on the remote computer.

Alternatively, if the development computer can access the remote file system, you can change the B2B Data Transformation repository to the remote location and deploy directly to the remote computer.

Deployment of Production Updates to B2B Data Transformation Services

B2B Data Transformation Studio cannot open a deployed project that is located in the repository. If you need to edit the data transformation, modify the original project and redeploy it.

To edit and redeploy a project:

1. Open the development copy of the project in B2B Data Transformation Studio. Edit and test it as required. 2. Redeploy the service to the same location, under the same service name. You are prompted to overwrite the

previously deployed version.

Redeploying overwrites the complete service folder, including any output files or other files that you have stored in it. There is no versioning available in B2B Data Transformation. If previous versions of the deployed services are required, make a copy of the current service in a separate location, if desired (not in the service DB directory) or utilize a commercial or open source backup solution.

Renaming the service folder is also possible. The project name has to be renamed as well. This is not a recommended practice for backing up of services or deploying a service multiple times. It is preferred to use the Studio environment to deploy a service multiple times as behaviors may change in future versions. For backup, there are many commercial and open source back up solutions available, and in order to quickly retain a copy of the service, the service should be copied to a directory outside of the Service DB folder.

Important: There can be no more than one deployed service with the same service and project name. Project files contain configuration properties and indicate the transformation startup component. Having multiple services with identical project file names, even if the service names are different, will cause the service execution to fail.

Simple Service Deployment

There are two ways to deploy a service. One way is to deploy it directly as service within Data Transformation Studio while the other is to deploy the service locally and copy the service folder to the appropriate ServiceDB.


Single Service Deployment from Within B2B Data Transformation Studio Environment

1. In the B2B Data Transformation Explorer, select the project to be deployed.

2. On the B2B Data Transformation menu, click Project > Deploy.


3. The Deploy Service window displays the service details. Edit the information as required. Click the Deploy button.


4. Click OK.

5. At the lower right of the B2B Data Transformation Studio window, display the Repository view. The view lists the service that you have deployed, along with any other B2B Data Transformation services that have been deployed on the computer.


Single Service Deployment Via File Movement

Alternatively, the service folder can be copied directly into the ServiceDB folder.

On Windows

To check if the service deployed is valid, run CM_Console in the command line.


Alternatively, the cAPITest.exe can be used test the deployed service.

The B2B Data Transformation Engine determines whether any services have been revised by periodically examining the timestamp of a file called update.txt. By default, the timestamp is examined every thirty seconds. The update.txt file exists in the repository root directory which is, by default, the ServiceDB directory. The content of the file can be empty. If this is the first time a service is deployed to the remote repository, update.txt might not exist. If the file is missing, copy it from the local repository. If update.txt exists, update its timestamp as follows.

l On Windows: Open update.txt in Notepad and save it l On UNIX: Open a command prompt, change to the repository directory, and enter the following command.

touch update.txt

You can change the interval used to check for service updates by modifying the Service refresh interval in the B2B Data Transformation configuration editor.


Multi-Service Deployment

When a solution involves the use of multiple services, these may be authored as multiple independent B2B Data Transformation projects or as a single B2B Data Transformation project with multiple entry points to be deployed as multiple services under different names.

For complex solutions, we recommend the use of multiple separate projects for independent services, reserving the use of multiple runnable components within the same project for test utilities, and trouble shooting items.

While it is possible to deploy the set of services that make up a multi-service solution into production from the Studio environment, we recommend deploying these services to a test environment where the solution can be verified before deploying into production. In this way, mismatch between different versions of solution transformation services can be avoided.

In particular when dependencies occur between services due to the use of B2B Data Transformation features such as TransformByService, or due to interdependencies in the calling system, it is necessary to avoid deploying mismatching versions of transformation services and to deploy services into production as a group.

Simple batch files or shell scripts can be created to deploy the services as a group from the test environment to the production environment, and commercial enterprise system administration and deployment software will usually allow creation of a deployment package to facilitate scheduled unattended deployment and monitoring of deployment operations.

As a best practice, creating a dependency matrix for each project to be deployed allows developers to identify the required services by each project to be deployed and which are commonly accessed by majority of the projects. This allows for better deployment strategies and helps to keep track of impacted services should there be changes made to them.

Deploying for Full Uptime Systems

B2B Data Transformation has the ability to integrate into various applications allowing it to become a full uptime system. An integration component, called the B2B Data Transformation Agent, runs a B2B Data Transformation service that performs the data transformation. Integration systems capabilities are enhanced by supporting the


conversion of many document formats it do not natively support.

Deploying services for full uptime systems follows the same process as that of standalone B2B Data Transformation services. However, it is important to make sure that the user accounts used for the calling application have the necessary permissions to execute the B2B Data Transformation service and write to configuration to store error logs.

After deploying the service, it may be necessary to stop and restart the work flow invoking the service. Make sure that the update.txt timestamp is updated. B2B Data Transformation Engine determines whether any services have been revised by periodically examining the timestamp update.txt. By default, the timestamp is examined every thirty seconds.

Multiple Server Deployment

For enhanced performance, you can install B2B Data Transformation on multiple Windows or UNIX servers. The following discussion assumes that you use a load balancing module to connect to multiple, identically configured servers. The servers should share the same B2B Data Transformation services.

There are two ways to implement a multiple server deployment.

l Shared file system Store a single copy of the B2B Data Transformation repository on a shared disk. Configure all the servers to access the shared repository.

l Replicated file system Configure each server with its own B2B Data Transformation repository. Use an automatic file deployment tool to mirror the B2B Data Transformation repository from a source location to the individual servers.

If the second approach is adopted, it is a must to replicate or touch the file update.txt, which exists in the repository directory. The timestamp of this file notifies B2B Data Transformation Engine when the last service update was performed.

Designing B2B Data Transformation Services for Deployment

Identifying Versions Currently Deployed

Whenever a service is deployed through B2B Data Transformation Studio, the user is prompted to set the options shown in the table below.

Option Description


Service Name The name of the service. By default, this is the project name.

To ensure cross-platform compatibility, the name must contain only English letters (A-Z, a-z), numerals (0-9), spaces, and the following symbols:

% & + - = @ _ { }

B2B Data Transformation creates a folder having the service name, in the repository location.

Label A version identifier. The default value is a time stamp indicating when the service was deployed.

Startup Component The runnable component that the service should start.

Author The person who developed the project.

Description A description of the service.

Although version tracking is not available in the current version of B2B Data Transformation, deployment does take into account the service deployment timestamps. The deployment options are stored in a log file called deploy.log. It keeps a history of all deployments options made through the B2B Data Transformation Studio. The option settings entered in the Deploy Service window are appended to the log file.

Deploying services to different servers through file copying or FTP will not update the deployment log file. It has to be manually updated if added information is required.

Security and User Permissions

User permissions are required by users who install and use B2B Data Transformation Studio and Engine. Depending on the B2B Data Transformation application the organization runs, and the host environment used to invoke the services, additional permissions might be required.

To configure data transformations in B2B Data Transformation Studio, users must have the following permissions:

l Read and write permission for the Eclipse workspace location


l Read and execute permission for the B2B Data Transformation installation directory and for all its subdirectories

l Read and write permission for the B2B Data Transformation repository, where the services are deployed l Read and write permissions for the log application

For applications running B2B Data Transformation Engine, a user account with the following permissions is required.

l Read and execute permission for the B2B Data Transformation installation directory and for its subdirectories l Read for the B2B Data Transformation repository l Read and write permission for the B2B Data Transformation log path, or for any other location where B2B

Data Transformation applications are configured to store error logs

Aside from user permissions, it is important to identify the user types that would be assigned work with B2B Data Transformation. In Windows setup, an administrator or limited user can be registered in the Windows Control Panel.

Windows users who have administrative privileges can perform all B2B Data Transformation operations.

However, limited users have the following restrictions do not have write permissions for the B2B Data Transformation program directory and are NOT allowed to perform the following:

l Install or uninstall the B2B Data Transformation software l Install a B2B Data Transformation license file l Deploy services to the default B2B Data Transformation repository l Add custom components such as document processors or transformers l Change the setting values in the Configuration Editor

Backup Requirements

It is necessary to make regular backups of several B2B Data Transformation directories and files.

In production environment where B2B Data Transformation runs, it is important to backup three locations the Configuration File, Service Repository, and AutoInclude\User directory.

For development environment, we recommend using a commercial or open source-source control system such as Subversion to manage backup and versioning of the B2B Data Transformation Studio workspaces of the developers in the organization. In addition, backup the same locations listed above for production environment.

If you use identical configurations on multiple servers, back up only a single copy of these items.

In the event of a server failure, B2B Data Transformation can be re-installed in the same location as on the failed server and restore the backup.

Failure Handling

If a B2B Data Transformation service fails to execute successfully, it returns a failure status to the calling application. It is the responsibility of the calling application to handle the error.

For example, the application can transmit failed input data to a failure queue. The application can package related inputs in a transaction to ensure that important data is not lost.


In the event of a failure, the B2B Data Transformation Engine will generate an event log if event logging has been enabled for the project. To view the contents of the event file, drag the *.cme file into the events pane in the B2B Data Transformation Studio.

The method used to invoke a B2B Transformation service will affect how and if events are generated. The follow table compares the effect of each invocation method on the generation of events:

API / invocation method Event generation

CM_Console Service deployed with events will produce events. Service deployed without events will not produce events

Java API Service runs without events. In case of error, service is rerun with events

C# / .Net Same as Java

Agents No events unless error irrespective of how service was deployed.

Same behavior is used for PowerCenter

While the events log provides a simple mechanism for error handling, it also has a high cost in resources such as memory and disk space for storing the event logs. For anything other than the simplest of projects, it is recommended to design an error handling mechanism into your transformations and calling logic to handle errors and the appropriate alerting needed when errors occur. In many production scenarios, the event log will need to be switched off for optimal performance and resource usage.

Updating Deployed Services

B2B Data Transformation Studio cannot directly update a deployed project in the transformation service repository. To perform updates on the data transformation, the modifications must be made to the original transformation project and the project then needs to be redeployed.

Note: A different project can be used which may be deployed under the existing service name, so technically it does not have to be exactly the original project.

If it is required to track all deployed versions of the data transformation, make a copy of the current service in a separate location, or alternatively, consider the use of a source control system such as Subversion. Redeploying overwrites the complete service folder, including any output files or other files that you have stored in it.

It is important to test the deployed service following any modifications. While the Studio environment will catch some errors and block deployment if the transformation is invalid, some types of runtime errors cannot be caught by the studio environment prior to deployment.

Upgrading B2B Data Transformation Software (Studio and Runtime Environment)

When upgrading from a previous B2B Data Transformation release, existing projects and deployed services can also be upgraded to the current release. The upgrade of projects from B2B Data Transformation version 3.1 or higher is


automatic. Individual projects can be opened or imported in the B2B Data Transformation Studio with the developer prompted to upgrade the project, if necessary. Test the project and confirm that it runs correctly once upgrade is completed. Deploy the service to production environment.

Another way to upgrade the services is through the syntax conversion tool that comes with B2B Data Transformation. It allows upgrade of multiple projects and services quickly, in an automated operation. It is also used to upgrade global TGP script files, which are stored in the B2B Data Transformation autoInclude\user directory. Syntax conversion tool supports upgrade of project or service from 3.1 and higher on Windows while release 4 on UNIX-type platforms.

Before the upgrade, the tool creates an automatic backup of your existing projects and files. It creates a log file and reports any upgrade errors that it detects. In case of an error, restore the backup, correct the problem, and run the tool again.

It is necessary to organize the projects before running the tool. The tool operates on projects or services that are stored in a single parent directory. It can operate on:

l A B2B Data Transformation Studio version 4 workspace l A B2B Data Transformation repository l Any other directory that contains B2B Data Transformation Studio projects or services

Within the parent directory, the projects must be at the top level of nesting, for example:

If the projects are not currently stored in a single parent directory, re-organize them before running the tool. Alternatively, the tool can be run separately on the individual parent directories.

To run the syntax conversion tool in Windows, go the B2B Data Transformation folder from the Start menu then click Syntax Conversion Tool. The tool is a window with several tabs, where the upgrade settings can be configured.


After the service upgrade is complete, change the repository location to the new location using the Configuration Editor. Test the projects and services to confirm that they work correctly and that their behavior has not changed.

On UNIX platforms, run the command /bin/CM_DBConverter.sh. Only 4.x is supported.

Optionally, you can run the syntax conversion tool from the command line, without displaying the graphical user interface. In an open console, change to the B2B Data Transformation bin directory and run the following command:

l On Windows: CM_DBConverter.bat l On UNIX: CM_DBConverter.sh console

Following each switch, leave a space and type the value. If a path contains spaces, you must enclose it in quotation marks. The are listed in the following table.

Switch Required/Optional Description

-v Required Version from which you are upgrading (3 or 4). On UNIX, only 4 is supported.

-s Required Path of the source directory, containing projects or services.


-d Optional Path of the target directory. If you omit this switch, the tool overwrites the existing directory.

-si Optional Path of the source autoInclude\user directory. If you omit this switch, the tool does not upgrade global TGP files.

-di Optional Path of the target autoInclude\user directory. If you omit this switch, the tool overwrites the existing directory.

-l Optional Path of the upgrade log file. The default is \SyntaxConversionLog.txt.

-b Optional Path of the backup directory, where the tool backs up the original projects or services prior to the upgrade. The default is the value of the -s switch concatenated with the suffix _OLD_Backup.

-e Optional Path of the error directory, where the tool stores any projects or services that it cannot upgrade due to an error. The default is the value of the -s switch concatenated with the suffix _OLD_Failure.

Last updated: 29-May-08 16:47


Establishing a B2B Data Transformation Development Architecture

Challenge

Establish a development architecture that ensures support for team development of B2B Data Transformation solutions; establishes strategies for common development tasks such as error handling and the styles of B2B Data Transformation service authoring; and plans for the subsequent clean migration of solutions between development, test, quality assurance (QA) and production environments that can scale to handle additional users and applications as the business and development needs evolve.

Description

In this Best Practice the term development architecture means establishing a development environment and establishing strategies for error handling, version control, naming conventions, mechanisms for integration with the host environment and other aspects of developing B2B Data Transformation services not specific to a particular solution.

Planning for th

Date post:	01-Dec-2015
Category:	Documents
Upload:	jon-glickman
View:	677 times
Download:	0 times