Using the DataOrchestrator ODS™sgps.wau.edu/Datatel/UsingTheDataOrchestrator.pdf · Using the...

Using the DataOrchestrator ODS™Reissued Manual as of June 18, 2010

This is a new edition of the Using the DataOrchestrator ODS™ manual. This edition replaces the previous edition dated October 20, 2009, and incorporates the changes delivered with software update SU50297.04-485.

The Primary Changes Made

Section Pages Changes Made

Using the DataOrch Source Transform (DOST) Form

Defining Filter Criteria

Using the DataOrch Target Transform (DOTT) Form

83, 87 89, 91 101, 104, 105

An Incremental Date Field is no longer required to perform an incremental refresh of a transform. The As of Date, together with the Incremental Filter Criteria, now allows a transform to be run as an incremental.


86 Individual fields that are a part of a multi-part key are now available on the DataOrch Source Transform (DOST) form. Each field is included in the Include Computed Columns list for a multi-part key file and can be used to extract the individual piece of data from a multi-part key.

Defining Target Transform Columns

110 A timestamp column can be added to each transform on the Transform Columns (DOTC) form to record the time at which the refresh was run.

Defining SQL Views 145 A new field, Material View, was added to the DataOrch Target Views (DOTV) form. This field displays “Yes” for each SQL view that is defined as a materialized view on the DataOrch View Specs (DOVS) form.

Defining the SQL Select Statement for an SQL View

147 - 152 The ability to create a table from an SQL select statement (materialized view) has been added to the DataOrch View Spec (DOVS) form.

Defining Settings for Refreshing Data

172 A new option, Run Stored Procedure, has been added to the DataOrch Refresh (DORE) form to automatically run a stored procedure on the ODS target database at the completion of a refresh.

Viewing the History of a Refresh

204 Information for materialized views is displayed on the DataOrch Refresh History (DORH) form.

Maintaining SQL Views 220 Information was added to explain what now happens when you copy an SQL view to a target and an SQL view with the same name already exists on that target.

Using the DataOrchestrator ODS™

Release 18

June 18, 2010

Datatel Colleague®

For last-minute updates and additional information about this manual, see AnswerNet page 5696.


© 2010 Datatel, Inc. All Rights Reserved

The information in this document is confidential and proprietary to and considered a trade secret of Datatel, Inc., and shall not be reproduced in whole or in part without the written authorization of Datatel, Inc. The information in this document is subject to change without notice.

Colleague and ActiveCampus are registered trademarks of Datatel, Inc. ActiveAlumni and ActiveAdmissions are trademarks of Datatel, Inc. Other brand and product names are trademarks or registered trademarks of their respective holders.

Datatel, Inc.4375 Fair Lakes CourtFairfax, VA 22033(703) 968-9000(800) DATATELwww.datatel.com

Table of Contents

11 Introduction

13 About This Manual13 In This Chapter14 Who Should Read This Manual14 Terms Used in This Manual16 What This Manual Covers17 How This Manual is Organized19 Where to Find More Information

21 About the DataOrchestrator ODS21 In This Chapter22 Understanding the DataOrchestrator ODS24 What Operational Data Stores Cannot Do24 Facts about Operational Data Stores25 Envision Data Mapping on an ODS Target Database26 Architecture of the DataOrchestrator ODS29 Hardware and Software Configuration for the DataOrchestrator

ODS

31 Overview and Installation

33 Overview33 In This Chapter34 Installation Overview for the DataOrchestrator ODS35 Prerequisites for the DataOrchestrator ODS36 Where to Build the Operational Data Stores36 SQL Server37 Oracle38 Preparing the ODS Target Database40 SQL Server Script for Creating the ODS Target Database42 Adding the IDX Filegroup42 Procedure for Adding the IDX Filegroup

Using the DataOrchestrator ODS™, June 18, 2010 5© 2010 Datatel, Inc.

Table of Contents

45 Installing the DataOrchestrator ODS45 In This Chapter46 Before You Begin46 Adding the Optional Module47 Setting Up the ODS Target Database Server and Installing

Software Updates51 Installing the Oracle JDBC Driver51 Procedure for Installing the Oracle JDBC Driver

53 Setting Up DataOrchestrator ODS Parameters

55 Overview of Using the DataOrchestrator ODS55 In This Chapter56 Workflow for Using the DataOrchestrator ODS

59 Defining DataOrchestrator ODS Parameters59 In This Chapter59 Form Used60 Defining Parameters61 Noteworthy Fields on the DOPA Form64 Procedure for Defining DataOrchestrator ODS Parameters

65 Setting Up Targets

67 Defining a Target67 In This Part68 Forms Used70 Defining a Target71 Overview of the DataOrch Target (DOTA) Form72 Noteworthy Fields on the DOTA Form

77 Defining Additional Target Parameters77 In This Chapter77 Form Used78 Defining Additional Target Parameters79 Noteworthy Fields on the DOTP Form80 Procedure for Defining Additional Target Parameters

6 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.

Table of Contents

81 Defining Source Transforms81 In This Chapter81 Forms Used82 Defining Source Transforms83 Using the DataOrch Source Transform (DOST) Form85 Noteworthy Fields on the DOST Form89 Defining Filter Criteria91 Noteworthy Fields on the DOFC Form93 Viewing Generated Target Transforms94 Noteworthy Fields on the DOGT Form95 Procedure for Defining Source Transforms

97 Defining Target Transforms97 In This Chapter98 Forms Used

100 Defining Target Transforms101 Using the DataOrch Target Transform (DOTT) Form103 Noteworthy Fields on the DOTT Form107 Defining Target Transform Columns109 Noteworthy Fields on the DOTC Form113 Defining Properties for a Target Transform Column114 Noteworthy Fields on the DOPR Form116 Defining Operations for Transform Columns117 The Field Extract (DOFE) Form118 Noteworthy Fields on the DOFE Form120 The Multivalue Operation (DOMV) Form121 Noteworthy Fields on the DOMV Form123 The Pointer Reference (DOFR) Form124 Noteworthy Fields on the DOFR Form126 The Null Value Replacement (DONV) Form127 Noteworthy Fields on the DONV Form129 The String Concatenation (DOCA) Form130 Noteworthy Fields on the DOCA Form132 The Substring Selection (DOSS) Form133 Noteworthy Fields on the DOSS Form135 The Validation Code LookUp (DOVC) Form136 Noteworthy Fields on the DOVC Form138 The Expression Entry (DOEE) Form139 Noteworthy Fields on the DOEE Form141 Procedure for Defining Target Transforms


Table of Contents

143 Defining and Creating SQL Views143 In This Chapter144 Forms Used145 Defining SQL Views146 Noteworthy Fields on the DOTV Form147 Defining the SQL Select Statement for an SQL View149 Noteworthy Fields on the DOVS Form152 Procedure for Defining SQL Views153 Creating SQL Views154 Procedure for Creating SQL Views Using the DOTV Form

157 Procedure for Defining a Target157 In This Chapter158 Procedure for Defining a Target

163 Refreshing ODS Data

165 Defining and Running an ODS Refresh165 In This Chapter165 Forms Used166 Defining Settings for Refreshing Data169 Noteworthy Fields on the DORE Form174 Defining Additional Parameters for the Refresh175 Noteworthy Fields on the DORP Form177 Procedure for Defining a Refresh

181 Calculating Stored Computed Columns181 In This Chapter182 Forms Used183 Calculating Stored Computed Columns184 Procedure for Activating Stored Computed Columns186 Procedure for Calculating Stored Computed Columns188 Procedure to Update Flags in the STUDENT.TERMS.CC File


Table of Contents

191 Viewing Reports and Refresh History

193 Viewing Errors for a Refresh193 In This Chapter193 Form Used194 Viewing the Error Analysis Report for a Refresh195 Noteworthy Fields on the DOEA Form197 Procedure for Running the Error Analysis Report

199 Viewing Target Transform Data for a Target199 In This Chapter199 Form Used200 Viewing the Transform Summary Report for a Target201 Noteworthy Fields on the DOTS Form202 Procedure for Running the Transform Summary Report

203 Viewing the History of a Refresh203 In This Chapter203 Form Used204 Viewing the History of a Refresh206 Procedure for Viewing the History of a Refresh

207 Maintaining the DataOrchestrator ODS

209 Maintaining Transforms209 In This Chapter209 Form Used210 Maintaining DataOrchestrator ODS Transforms211 Noteworthy Fields on the DOMA Form213 Procedure for Deleting Transforms214 Procedure for Copying Transforms215 Procedure for Renaming Transforms216 Procedure for Copying and Renaming Transforms217 Procedure for Calculating Foreign Keys218 Procedure for Regenerating Source Transforms


Table of Contents

219 Maintaining SQL Views219 In This Chapter219 Form Used220 Maintaining SQL Views221 Noteworthy Fields on the DOVM Form223 Procedure for Deleting SQL Views224 Procedure for Copying SQL Views225 Procedure for Renaming SQL Views226 Procedure for Copying and Renaming SQL Views

227 Copying Targets to Another Target227 In This Chapter227 Form Used228 Copying One or More Targets to Another Target229 Noteworthy Fields on the DOTY Form230 Procedure for Copying One or More Targets to Another Target

231 Deleting DataOrchestrator ODS Objects231 In This Chapter231 Form Used232 Deleting DataOrchestrator ODS Objects233 Noteworthy Fields on the DOOD Form233 Procedure for Deleting DataOrchestrator ODS Objects

235 Appendices

237 Checklist for Setting Up the DataOrchestrator ODS

237 In This Appendix238 Checklist for Setting Up the DataOrchestrator ODS

241 Frequently Asked Questions241 In This Appendix242 Frequently Asked Questions about the DataOrchestrator ODS

247 Troubleshooting the DataOrchestrator ODS247 In This Appendix248 DataOrchestrator ODS Issues

259 Index



Introduction

Introduction

About This Manual

In This ChapterThis chapter includes information about the users who should read this manual, terms used in this manual, the product that is covered in this manual, the organization of the topics in this manual, and additional resources for more information.

Table 1 lists the topics in this chapter.

Table 1: Topics in This Chapter

Topic Page

Who Should Read This Manual 14

Terms Used in This Manual 14

What This Manual Covers 16

How This Manual is Organized 17

Where to Find More Information 19


Introduction: About This Manual

Who Should Read This ManualThis manual is for system administrators and system programmers who set up and support reporting for your institution. No specific knowledge of the structure of source Envision files is assumed.

Terms Used in This ManualThe following are the terms and descriptions that are important to understanding and using the concepts presented in this manual. These terms are used throughout this manual:

Operational Data Store (ODS). An ODS is a repository of data that can deal with a specific area of knowledge and provides easy access to the data for reporting and analysis. It is a snapshot of data captured at a point in time.

Source Database. This refers to the Colleague environment from which you want to create an ODS.

Target Database. This refers to the database where you want to place an ODS.

Target. A target contains all of the necessary information about an ODS target database including:

Connection information.

The set of source and target transforms that should be run to create and update the tables in the ODS.

Current update status information of the target.

Source Transform. A source transform allows you to specify the data to move to an ODS target database by selecting a set of fields and/or computed columns from a single Colleague file. The selected data will be structured on the target database in one or more tables in the same way as the standard SQL representation for that data in a Colleague environment. Source transforms do not allow you to make any data transformations when the data is populated.

Target Transform. A target transform allows you to specify the format of the data in a table on the ODS target database, the Colleague source data elements that will populate each column, as well as any data transformation operations to be run when the data is populated. Target transforms are a powerful tool for the creation of ODS tables that are optimized for reporting purposes, and provide greater flexibility in defining how the data is structured on the ODS.


Terms Used in This Manual

Refresh. The refresh is the specification for an executable process that refreshes a subset of the transforms associated with a target.

Full Refresh. A full refresh means that all records of the source file for the transform you choose will be processed through the transform and updated on the ODS target database.

Incremental Refresh. An incremental refresh means that only those records of the source file that changed since the last refresh will be processed through the transform.

Reporting Data Access Server (RDAS). A DMI Listener role defined for use with the DataOrchestrator ODS. This DMI Listener role includes a set of transactions for bulk data management.

SQL View. An SQL view combines data from various physical tables where it is stored so that the result appears to be data from a single table. As a result, a report or query is easier to build because the user creating the report does not need to know the tables where data elements are stored or their join relationships.



What This Manual CoversThis manual discusses the DataOrchestrator ODS™, which provides the ability to create and maintain operational data stores, which are individual data stores comprised of a defined subset of information from Colleague.

The information in the operational data stores can be kept up-to-date through periodic refreshes of current data from the Colleague source database. The operational data stores can be used as a source for reporting against this Colleague data and can also be used as a source for data extracts for propagation into data warehouse systems.

The chapters in this manual contain the information and procedures for installing the software for the DataOrchestrator ODS, as well as implementing and using the DataOrchestrator ODS.

In this manual you will find high-level installation and setup information, and references to more detailed installation documentation. This documentation provides information on how to perform the following tasks for implementing the DataOrchestrator ODS and operational data stores.

Install the following software components:• DataOrchestrator ODS, Envision, and DMI software updates.• An additional DMI Listener for the DataOrchestrator ODS.• JDBC driver for an Oracle target database.

Define the settings on which the DataOrchestrator ODS depends, including the set of source and target transforms for an ODS.

Export and refresh data to the operational data stores.


How This Manual is Organized

How This Manual is OrganizedTable 2 shows how this manual is organized.

Table 2: Organization of This Manual

Part Chapter Title Summary

Part 1

Introduction

About This Manual Includes information about who should read this manual, terms used in the manual, what this manual covers, how it is organized, and where to find information on related topics.

About the DataOrchestrator ODS

Discusses what the DataOrchestrator ODS is and how the DataOrchestrator ODS can enhance your institution’s reporting.

Part 2

Overview and Installation

Overview Provides an overview of the installation process and the prerequisites for implementing the DataOrchestrator ODS. Also included is information on preparing the database where you want to build operational data stores.

Installing the DataOrchestrator ODS

Provides information about adding the optional module for the DataOrchestrator ODS, retrieving the Datatel software updates, performing the installation steps, and installing the Oracle JDBC driver.

Part 3

Setting Up DataOrchestrator ODS Parameters

Overview of Using the DataOrchestrator ODS

Provides a high-level overview of the suggested workflow for using the DataOrchestrator ODS.

Defining DataOrchestrator ODS Parameters

Provides information on how to set up the parameters for the DataOrchestrator ODS.

Part 4

Setting Up Targets

Defining a Target Provides an overview on setting up and modifying a target to export and transform Colleague source data to an operational data store using the DataOrchestrator ODS.

Defining Additional Target Parameters

Describes how to define the additional parameters for a target.

Defining Source Transforms

Provides an explanation of how to define source transforms for a target.

Defining Target Transforms Provides an explanation of how to define target transforms.

Defining and Creating SQL Views

Describes how to define, create, and maintain SQL views for a target.

Procedure for Defining a Target

Provides the procedure for defining a target.



Part 5

Refreshing ODS Data

Defining and Running an ODS Refresh

Provides information on how to set up a refresh for exporting Colleague source data using the DataOrchestrator ODS, and how to refresh data.

Calculating Stored Computed Columns

Includes information on how to activate, update, and export stored computed columns that are used as source data elements for operational data stores.

Part 6

Viewing Reports and Refresh History

Viewing Errors for a Refresh

Discusses how to view the error analysis report for a previously run refresh.

Viewing Target Transform Data for a Target

Discusses how to view information for target transforms for a specified target.


Discusses how to view the settings and the export statistics for a previously run refresh.

Part 7

Maintaining the DataOrchestrator ODS

Maintaining Transforms Provides information on how to copy existing target transforms, delete transforms, and rename transforms. Also details how to regenerate source transforms, and create foreign keys for transforms.

Maintaining SQL Views Provides information on how to copy existing SQL views, delete views, and rename SQL views.

Copying Targets to Another Target

Gives information on how to copy all transforms and SQL views from one or more targets to a single destination target.

Deleting DataOrchestrator ODS Objects

Includes information on how to delete existing refreshes, targets, or refresh history records from the source Colleague database.

Appendices Checklist for Setting Up the DataOrchestrator ODS

Provides a quick checklist of the activities to perform to set up the DataOrchestrator ODS.

Frequently Asked Questions

Provides answers to frequently asked questions when using the DataOrchestrator ODS and the data models and views for reporting.

Troubleshooting the DataOrchestrator ODS

Provides suggestions for items to check or steps to take if you encounter issues in using the DataOrchestrator ODS.

Table 2: Organization of This Manual (cont’d)

Part Chapter Title Summary


Where to Find More Information

Where to Find More InformationTable 3 lists additional resources for finding more information.

Table 3: Additional Resources

Type of Information Resource

Information on using the DataOrchestrator ODS data models for reporting.

Reporting from the DataOrchestrator ODS Data Models

Information on the mapping of Colleague data to an ODS target database when using DataOrchestrator source transforms. (See page 24.)

Mapping Envision Files for SQL Server and Oracle

Installing a newly licensed optional module and retrieving software updates. (See page 46.)

Updating Colleague Software

Creating a new DMI_DAS. (See page 47.)

Installation Procedures for Colleague Release 18.0

Setting up batch processes to run at scheduled intervals. (See page 183.)

Envision Runtime Administration

How to activate and calculate stored computed columns. (See page 183.)

Stored Computed Columns

Assistance with implementing SQL-based reporting solutions

Datatel Consulting Services. Contact Services Scheduling:

1-800-DATATEL (328-2835)

Technical Support The Datatel Solution Center. Contact the Tools and Technology Team:

1-800-DATATEL (328-2835)




Introduction

About the DataOrchestrator ODS

In This ChapterThis chapter provides information that you should understand about the DataOrchestrator ODS before you use the product to create and use operational data stores for your institution. It addresses the following questions:

What is the DataOrchestrator ODS, and what value does it bring to reporting?

What are the limitations of the operational data stores?

How can you maximize your use of the operational data stores?

Also included is information about Envision data mapping on an ODS target database. Table 4 lists the topics in this chapter.


Topic Page

Understanding the DataOrchestrator ODS 22

Facts about Operational Data Stores 24

Envision Data Mapping on an ODS Target Database 25

Architecture of the DataOrchestrator ODS 26

Hardware and Software Configuration for the DataOrchestrator ODS 29


Introduction: About the DataOrchestrator ODS

Understanding the DataOrchestrator ODS

The DataOrchestrator ODS was developed to help provide increased reporting capabilities to your institution’s staff, managers, and executives to support their institutional effectiveness strategies and long-range planning efforts.

The DataOrchestrator ODS is a powerful tool that allows you to export and transform data from Colleague applications to create operational data stores that encompass specific areas of knowledge.

Ease of access and usability, as well as fast processing of data, are some of the advantages of operational data stores. Also, operational data stores are easy to use and convenient because they are not restricted to the same server or location as the source database. Data can be exported to a target SQL-based (SQL Server or Oracle) database.

The following are some of the benefits derived from the DataOrchestrator ODS:

Operational data stores, as discussed in this manual, enhance your reporting capability. By utilizing an operational data store strategy, Datatel provides a better means for you to report across Colleague applications while using a variety of reporting tools appropriate for your institution.

When extracting data from Colleague, you can transform the data using flexible functions so that the data is available in a form that best suits your reporting purposes.

Specific subsets of Colleague data can be created in the ODS target database to optimize reporting. This means that you no longer have to search through thousands of elements in Colleague to select information for reporting. Instead, you are able to simply utilize the DataOrchestrator ODS to select your own specific files and fields for each operational data store for your reporting purposes.

For the decision makers at your institution, who need consistent and accurate data on a daily basis, operational data stores provide a snapshot of information, or operational picture, from a particular point in time. For example, all the reports created from an operational data store today will

Note: The DataOrchestrator ODS does not support exporting from a Colleague SQL Server platform to an Oracle database as this configuration is not used by Datatel clients.



have consistent information from one report to another, irrespective of who created the report.

An operational data store can be used to combine data from multiple sources for reporting purposes. You can incorporate data from your non-Colleague systems into an operational data store created using the DataOrchestrator ODS, producing an integrated reporting platform.

You have complete flexibility in determining how often operational data stores should be refreshed. A decision may be reached between the technical staff and the various professional staff at your institution to determine how fresh the data needs to be for reporting from each operational data store.

You have the ability to do full or incremental refreshes of data, and you can also set up the process handler to automatically refresh this data.

Operational data stores improve performance of the transactional database by reducing the load on that database because information for reporting is derived from snapshots of data on another database server.

Your institution may have specific needs that might necessitate adding or removing files or data elements from an operational data store. The DataOrchestrator ODS allows you to modify the set of files and fields that are exported to an operational data store and how this data is transformed, resulting in a customized reporting solution for your institution.



What Operational Data Stores Cannot Do

In addition to understanding what an operational data store can do for your institution, it is also important to understand what it is not designed to do.

An operational data store is not a data warehouse; it is an independent database containing subsets of application data. The information in a data warehouse is highly optimized and aggregated for analytic reporting, and contains a complete historical record of institutional data over time. However, an operational data store retains the level of detail of the data structures in the transactional database and does not contain historical data beyond what is available in the transactional database. In addition, the data in an operational data store is not intended to be used for transactional purposes, such as up-to-the-minute reporting. Operational data store information is taken at a point in time, and the information may not be the most current.

Facts about Operational Data Stores

To further understand the nature of operational data stores and to maximize their utility, consider the following facts:

An operational data store includes static data, not dynamic or live data. The data is captured at the time the data is exported to the operational data store. Thus, it is a snapshot of data at a point in time. The operational data store is updated with new data each time the refresh is run. Therefore, the resulting reports will consistently reflect the data existing at the time of the operational data store’s update.

The latest transactional data will be selected from the database whenever you export data to the operational data store.

You can set up the Process Handler to export data to an operational data store automatically on a preset schedule.

Note: Datatel recommends that any operational data store that contains stored computed columns be set up to automatically run the following processes: – Update GPA SCC Flags (UGSF) if using the STUDENT.TERMS.CC file – Update Stored Computed Column (USCC) This ensures that the stored computed columns will be updated when the ODS target database is updated. Refer to Envision Runtime Administration for further information on how to set a batch process to run at scheduled intervals.



Envision Data Mapping on an ODS Target Database

Data mapping on an ODS target database is different for source transforms and target transforms. For source transforms, the mapping of Colleague data on an ODS target database is identical to the mapping of Colleague data on an R18 SQL Server or Oracle environment. For further information on this mapping, see Mapping Envision Files for SQL Server and Oracle, available on the Datatel website. However, the DataOrchestrator ODS will create on the target database only those tables and columns that are necessary to store the Envision data elements selected for inclusion in an operational data store.

In contrast, target transforms allow a much greater flexibility in the structure of the data on the ODS target database. The number of columns of each target table, as well as their names and data types, can be defined in the way that best fits your institution’s reporting requirements.



Architecture of the DataOrchestrator ODS

A high-level overview of the system architecture of the DataOrchestrator ODS is shown in Figure 1 on page 27. The main elements of this architecture are:

Colleague Application Environment. The top of Figure 1 displays the Colleague application environment. This environment is the source of data for an operational data store and controls refreshes of that data to the operational data store. A key element in this environment is the Envision batch process displayed on the right, which controls the execution of target transforms.

DMI_DAS. The middle of Figure 1 displays the DMI communication layer. This is used by the Colleague application environment to communicate with both the source (environment) database and the target database where the operational data store is created. The three DMI transactions shown in this layer are specifically designed to support bulk extracting and loading of data for the DataOrchestrator ODS.

Source and Target Database Servers. The source server (on the bottom left) contains the Colleague environment database. The target server (on the bottom right) contains the ODS target database.


Architecture of the DataOrchestrator ODS

Figure 1: DataOrchestrator ODS Architecture

The directional arrows in Figure 1 show how control and data flows between these elements during the refresh of a target transform. The process for executing a transform is as follows:

In the Colleague application environment, the batch process creates and sends a DAS Bulk Export transaction from the information in the target transform. Using the information specified in this transaction, the DMI_DAS creates a query statement to extract the data necessary for the transaction from the source database. This query statement is in the native query language of the source database (UniQuery for UniData, and SQL for Oracle or Microsoft SQL Server). The query statement runs, using the bulk extract function of the environment database. The results are stored to a delimited flat file.

In the Colleague application environment, the batch process then uses a DAS Streaming Data Transfer transaction to move the delimited flat file from the source server to the target server.

In the Colleague application environment, the batch process then uses a DAS Bulk Load transaction that utilizes the target database’s native bulk load function (BULK INSERT statement for SQL Server, and SQL*Loader for Oracle) to load the data in the delimited flat file to the appropriate table on the target database.



This process is the core of the design of the DataOrchestrator ODS. This leverages the bulk extract and load functions of your native database platform, while maintaining the Colleague database-independent architecture where the DMI_DAS layer provides all database platform-specific processing.


Hardware and Software Configuration for the DataOrchestrator ODS

Hardware and Software Configuration for the DataOrchestrator ODS

A high-level overview of the configuration of the hardware and software elements required in a DataOrchestrator ODS installation is shown in Figure 2. This hardware and software infrastructure supports the architecture described in Architecture of the DataOrchestrator ODS on page 26.

As Figure 2 shows, there must be a DMI_DAS Listener located on the source database server, and there must also be a DMI_DAS Listener with the RDAS role located on the target database server. For more information on the DMI RDAS Listener, see Setting Up the ODS Target Database Server and Installing Software Updates beginning on page 47.

Figure 2: DataOrchestrator ODS Hardware and Software Configuration







Overview

In This ChapterThis chapter provides you with an overview of the setup requirements and installation steps for the DataOrchestrator ODS. In addition, you will find the prerequisites for the DataOrchestrator ODS, as well as considerations for building your resulting operational data stores and for preparing the target database.



Topic Page

Installation Overview for the DataOrchestrator ODS 34

Prerequisites for the DataOrchestrator ODS 35

Where to Build the Operational Data Stores 36

Preparing the ODS Target Database 38


Overview and Installation: Overview

Installation Overview for the DataOrchestrator ODS

This section provides a high-level overview of the setup requirements and installation procedures for the DataOrchestrator ODS.

Consult Table 6 to be sure that your software environment is ready for working with the DataOrchestrator ODS.

Table 6: Setup Requirements and Installation Procedures

Setup Requirement/ ReferenceInstallation Procedure

Check that your institution has the prerequisites needed for the DataOrchestrator ODS.

Prerequisites for the DataOrchestrator ODS on page 35.

Decide where to build the operational data stores.

Where to Build the Operational Data Stores beginning on page 36.

Prepare the target database where the operational data stores will be built.

Preparing the ODS Target Database on page 38.

For SQL Server, you can use a script to create your ODS target database.

SQL Server Script for Creating the ODS Target Database on page 40

If your institution is using a SQL Server target database, you must be sure that the filegroup named “IDX” is present in your target database.

Adding the IDX Filegroup on page 42

Add the optional module for the DataOrchestrator ODS.

Adding the Optional Module beginning on page 46.

Retrieve the required software updates. Setting Up the ODS Target Database Server and Installing Software Updates beginning on page 47.

Install and configure the target database DMI_DAS, update DAS roles, and install software updates.

Setting Up the ODS Target Database Server and Installing Software Updates on page 47.

If the target database is Oracle, install the Oracle JDBC driver on the ODS target database. If the source environment is UniData and the target database is Oracle, also install the Oracle JDBC driver on the source environment.

Installing the Oracle JDBC Driver on page 51.


Prerequisites for the DataOrchestrator ODS

Prerequisites for the DataOrchestrator ODS

Table 7 lists the prerequisites that your system must meet before you install the software updates for the DataOrchestrator ODS.

Table 7: Prerequisites for the DataOrchestrator ODS

Prerequisite Comment

Colleague R18 Make sure Colleague is current on all software updates. If you need help, contact the Datatel Solution Center.

UniData 7.1 For UniData, you must have the source Colleague database on UniData version 7.1 or later. For information about installing UniData 7.1, see the installation procedures provided by IBM.

SQL Server 2005 or SQL Server 2008 You must have SQL Server 2005 or SQL Server 2008. Other releases are not supported.

Oracle Database 11g or 10g Release 2 You must have Oracle Database 11g or 10g Release 2. Other releases are not supported.

Datatel Messaging Interface (DMI) with Data Access Server (DAS)

Use the DMI_DAS installed in the Colleague application environment on the source (Colleague) database server.

Ports used by the DMI Listeners must be open between the Colleague and ODS database servers.

Open ports are required so that the Colleague database server DMI_DAS can communicate with the ODS database server DMI RDAS.

The port used by the Datatel daemon on the ODS target database server must be open to the client PC using SA Valet in order to install software updates.

The port for the daemon must be open in order to manage and update the DMI RDAS through SA Valet.



Where to Build the Operational Data Stores

The most important thing to think about is how you will use the operational data stores you create. How do you intend to report from them? Will you use Crystal Reports, Safari ReportWriter, Excel, SQL Server, or a data warehouse solution? The following information is intended to help you think through these questions before beginning. Knowing which reporting tool you intend to use can save you time and effort by eliminating the need to do setup work more than once.

SQL Server

Consider the following points for the target database:

Database considerations:• You must have SQL Server 2005 or SQL Server 2008.• You will need to create the SQL Server database in which you want to

build the operational data stores. The process that exports data will create the tables, but cannot create the database itself.

Connection considerations:• Ports used by the DMI Listeners must be open between the Colleague and

ODS servers.• The port used by the Datatel daemon on the ODS server must be open to

the client PC using SA Valet in order to install software updates.

Third-party tool considerations:• None

Note: Do not use an existing Colleague database as a target database for an operational data store. If you try to do this, you will receive an error message when you run the refresh.

Note: This section assumes a high level of general technical knowledge; no detailed explanations of the technology are provided. This section is intended as a quick look at what will be required.


Where to Build the Operational Data Stores

Oracle

Consider the following points for the target database:

Database considerations:• You must have Oracle Database 10g Release 2 or later. • You will need to create the Oracle instance in which you want to build the

operational data stores. The process that exports data will create the tables, but cannot create the instance itself.

Connection considerations:• Ports used by the DMI Listeners must be open between the Colleague and

ODS servers.• The port used by the Datatel daemon on the ODS server must be open to

the client PC using SA Valet in order to install software updates.

Third-party tool considerations:• None

Note: The DataOrchestrator ODS does not support exporting from a Colleague SQL Server platform to an Oracle database as this configuration is not used by Datatel clients.



Preparing the ODS Target Database

Select the database where you want to build the operational data stores. The operational data stores can be stored in any existing or new target SQL-based (SQL Server or Oracle) database.

Make sure a valid target database exists and note the name of that target database.

Make sure that a database user exists for the DataOrchestrator ODS to use for accessing the target database. This user must have full administrative privileges to create database tables and to create and edit data in the target database.

If you are using SQL Server as the target database, this user must have been created with SQL Server Authentication (not Windows Authentication). In addition, you must set the authentication of your SQL Server to “SQL Server and Windows.”

To do this, access Microsoft SQL Server Management Studio. Select the SQL Server and Windows Authentication mode option under Server authentication on the Security tab of the Server Properties window of your SQL Server instance, as shown in Figure 3 on page 39.

ALERT! You should ensure that the target database to which you are exporting Colleague data is adequately secured using the native security controls for your database platform. Some files and fields may contain data that might be considered sensitive in your institution. Access to the ODS target database containing this data should be restricted accordingly. Datatel also highly recommends that you secure access to the DataOrch Target (DOTA) and DataOrch Refresh (DORE) forms. These forms provide a user with the ability to export any data in Colleague to an external database.

Note: You do not have to create any tables. The DataOrchestrator ODS software creates the tables in the target database as part of the export.

Note: If you are using the iStrategy data warehouse solution, you must use a SQL Server target database. You must set the default collation sequence of your database to “SQL_Latin1_General_CP1_CI_AS”.



Figure 3: Server Properties Window in Microsoft SQL Server Management Studio



SQL Server Script for Creating the ODS Target Database

For SQL Server, you can use a script to create your ODS target database. Figure 4 on page 41 shows a sample script for using the CREATE DATABASE statement to create a local ODS target database in SQL Server. Note the following:

This script uses directory paths D:\sqlsrvr\data and E:\sqlsrvr\txlog, and assumes a target database name of testods. You must replace these paths with directory structures that exist on your SQL Server file system and replace testods with your preferred database name.

The script places the database files on a separate drive (D:) from the transaction log (E:). This is expected to provide performance benefits, because the SQL server will write to the transaction log at the same time it writes to the database.

The COLLATE statement defines alphanumeric sorting. Use Latin1_General_BIN, as shown in the example (unless you are using the iStrategy data warehouse solution). This sorts data in the same order as your Colleague database.

ALERT! If you are using the iStrategy data warehouse solution, you must use the “SQL_Latin1_General_CP1_CI_AS” collation sequence for the ODS database in this script.

Technical Tip: If you copy the script in Figure 4 on page 41 from the PDF of this manual, you may need to first paste it into a text editor (such as Windows Notepad) to strip out any hidden characters in the PDF text, and then copy it from the text editor to the SQL application. If you copy directly from the PDF to the SQL application, the script may not run properly. However, if you cut and paste as suggested, you may remove valid carriage returns. After pasting the text in the query window, compare your script to Figure 4 on page 41 and enter carriage returns, if necessary.



Figure 4: Sample CREATE DATABASE Script for Creating an ODS Target Database in SQL Server

CREATE DATABASE testods

ON PRIMARY

( NAME = testods_dat01,

FILENAME = 'D:\sqlsvr\data\testods_dat01.mdf',

SIZE = 500MB,

MAXSIZE = UNLIMITED,

FILEGROWTH = 10% ),

FILEGROUP IDX

( NAME = testods_idx01,

FILENAME = 'D:\sqlsvr\data\testods_idx01.ndf',

SIZE = 250MB,


FILEGROWTH = 10% )

LOG ON

( NAME = 'testods_log01',

FILENAME = 'E:\sqlsvr\txlog\testods_log01.ldf',

SIZE = 500MB,


FILEGROWTH = 10% )

COLLATE Latin1_General_BIN

GO



Adding the IDX Filegroup

If your institution is using a SQL Server target database, you must be sure that the filegroup named “IDX” is present in your target database (in addition to the default/primary filegroup that you use). If the IDX filegroup does not exist on your target database, create a filegroup with this name and assign a file to it. The IDX filegroup is used for the indices.

Procedure for Adding the IDX Filegroup

To add the IDX filegroup and assign a file to it, complete the following steps.

Step 1. Right-click on the name of the ODS target database in Management Studio, and select Properties to display the Properties page.

Step 2. Click the Filegroups page. Click Add and enter IDX in the Name field of the first available open line as shown in Figure 5 on page 43, and then click OK to close the Properties page.

Note: If you used the script described in SQL Server Script for Creating the ODS Target Database on page 40, you can skip the rest of this chapter. The IDX filegroup was created in your target database.



Figure 5: Filegroups Page

Step 3. Open the Properties page again for the database. Click the Files page.

Step 4. Click Add, and then enter the logical name for the new data file in the Logical Name field of the first available line.



Step 5. Click the value shown in the Filegroup field. A dropdown list of available Filegroups is displayed. Select IDX as shown in Figure 6.

Figure 6: Files Page

Step 6. Enter an initial space allocation for the new file in the Initial Size (MB) column.

Step 7. Check the Autogrowth column and adjust it as needed, as the default of 1 MB is probably not sufficient.

Step 8. Check the Path column to confirm that this is the path where you want the index file created.

Step 9. Click OK to save your updates.



Installing the DataOrchestrator ODS

In This ChapterThis chapter provides information about adding the optional module, setting up the ODS target database server, and performing additional installation steps.



Topic Page

Adding the Optional Module 46

Setting Up the ODS Target Database Server and Installing Software Updates

47

Installing the Oracle JDBC Driver 51


Overview and Installation: Installing the DataOrchestrator ODS

Before You BeginTable 9 lists the tasks that must be complete before you can continue with the procedures in this chapter.

Adding the Optional ModuleTo add the DODS optional module for the DataOrchestrator ODS, see Updating Colleague Software, available from the Documentation section of the Datatel website, for information on installing newly licensed optional modules.

Table 9: Before You Begin

Task Reference

License the DODS optional module from Datatel.

Datatel client sales.

Make sure you have the prerequisites for the DataOrchestrator ODS.


Know where to build your operational data stores.

Where to Build the Operational Data Stores on page 36.

Have the target database prepared. Preparing the ODS Target Database on page 38.




To install the DataOrchestrator ODS, perform the following steps to set up the ODS target database and retrieve and install software updates.

Step 1. Install the standard Datatel daemon on the ODS target database server.

Step 2. Use SA Valet to create a new DMI Listener on the ODS target database server. See Installation Procedures for Colleague Release 18.0, available from the Datatel website. See “Procedure for Installing a New DMI Listener.”

When creating the new DMI Listener, in the DMI Listener Roles Selection window, click DBAS and RDAS.

Step 3. Use SA Valet to retrieve the necessary software updates to your local product repository. See Updating Colleague Software (available for downloading from the Documentation section of the Datatel website) for information on retrieving and installing software updates.

Step 4. Install the DMI software update in your environment. This will update all existing DMI Listeners in your Colleague environment and your Colleague local product repository.

Step 5. Install the Envision software updates in your Colleague environment.

Technical Tip: When you create the new Listener, it will inherit the original DAS values of the existing Listener. Be sure to review the Java memory, so that the performance parameters are acceptable for your servers.



Step 6. Create the ODS target database.

a. For SQL Server, you can use a script to create your ODS target database. For more information, see SQL Server Script for Creating the ODS Target Database beginning on page 40. If you do not use this script, you must be sure that the filegroup named “IDX” is present in your target database. For more information, see Adding the IDX Filegroup beginning on page 42.

b. For Oracle, use the Database Creation Assistant (DBCA) to create your ODS target database. Set the database character set as follows:

• Default database character set: WE8ISO8859P1 • Default national character set: AL16UTF16 - Unicode UTF-16 Universal

character set

Step 7. Create a subdirectory to be used by the DMI RDAS Listener on the target database server to store temporary files for the Bulk Load transaction. The following permissions are needed for this directory:

UNIX Oracle target database. The user who starts up the target DMI RDAS Listener must have read/write permissions.

SQL Server target database. • The Windows SYSTEM user must have read/write permissions.

• The DMI RDAS Listener logs in with SQL Server authentication, so the SQL Server process account must have read/write permissions for the BULK INSERT command.

This directory must have sufficient space allocated for these temporary files. The space needed for temporary files can be significant depending on how you define the target. Datatel recommends that you monitor the space usage in this directory as your refresh runs.

Step 8. If you are using a UniData or Oracle source environment, skip to Step 9. For SQL Server only, ensure that the user who will be running the DataOrchestrator ODS refresh has sufficient permissions on the source database to create SQL views.

If this is a SQL server source environment, you will skip Step 9 through Step 13. You are done with installation.

Note: If the Listener is ever started manually (instead of via SA Valet as a Windows service), then the user who starts the Listener must also have read/write permissions.



Step 9. Create a subdirectory on the source database server for use by the DMI_DAS Listener. This subdirectory will be used to store temporary files from the Bulk Extract transaction. The following permissions are needed for this directory:

UNIX UniData source.• The user logged in to Colleague must have read/write permissions.• The user who starts the source DMI_DAS must have read/write

permissions.

Windows UniData source. • The user logged in to Colleague must have read/write permissions.• The SYSTEM user must have read/write permissions.

UNIX Oracle source.• Oracle needs read/write permissions.• The user who starts the source DMI_DAS must have read/write

permissions.


Step 10. Log in to Colleague as the DMI administrative user (for example, dmiadmin).

Step 11. In the UT application, access the DMI Pre-Authenticated Server (DMCC) form. Enter the DMI Admin password.




Step 12. On the DMCC form, enter information for the application server and the Colleague database server.

a. In the Server Name field, enter the name the server is known by in the DMI Registry environment. This is usually the DNS name.

b. In the Enabled field, enter Yes for enabled.

c. In the Domain Name field, enter the fully qualified domain name for this server.

d. In the IP Address field, enter the IP address of the server. (Do not reuse the DNS name.)

Step 13. Save and exit from the DMCC form.

Note: If your institution's Colleague database is UniData, the application server and the Colleague database server are the same server, so you need to enter only one row of information.


Installing the Oracle JDBC Driver

Installing the Oracle JDBC Driver

Procedure for Installing the Oracle JDBC Driver

Step 1. Download the JDBC driver from the Oracle website. See AnswerNet document 4527 for download instructions and the current supported version of the driver.

Step 2. Save the file to your server.

Step 3. Use FTP to transfer the file to the machine where DMI is installed and place it in the lib folder under your home DMI folder. For example, if the DMI folder is /datatel/coll18/production/ods_das, then place the file under the /datatel/coll18/production/ods_das/lib folder.

Step 4. Stop and restart the DMI Listener.

Note: You must be a registered user of the Oracle website to download the Oracle JDBC driver. If your target database is Oracle, you need to install the Oracle JDBC driver on the ODS target database. In addition, if you have a UniData source environment and your target database is Oracle, then you also must install the Oracle JDBC driver on your source environment.







Overview of Using the DataOrchestrator ODS

In This ChapterThis chapter provides a high-level overview of the suggested workflow for using the DataOrchestrator ODS. The purpose of the overview is to give you a an understanding of the steps involved in setting up and maintaining the DataOrchestrator ODS.

Table 10 lists the topic covered in this chapter.

Table 10: Topic in This Chapter

Topic Page

Workflow for Using the DataOrchestrator ODS 56


Setting Up DataOrchestrator ODS Parameters: Overview of Using the DataOrchestrator ODS

Workflow for Using the DataOrchestrator ODS

Table 11 shows the steps to set up, refresh, and maintain the DataOrchestrator ODS and operational data stores.

Table 11: Steps to Set Up, Refresh, and Maintain the DataOrchestrator ODS and Operational Data Stores

Step Form Reference

1. Define parameters for the DataOrchestrator ODS.

DataOrch Parameters (DOPA)

Defining DataOrchestrator ODS Parameters on page 59.

2. Create a target, which defines the configuration of an operational data store on a target database. This includes defining source and target transforms to refresh on the target database. (Optional) Define and create SQL views as shown.

DataOrch Target (DOTA)

Use detail forms as needed.

Defining a Target beginning on page 67 (all chapters in Part 4).

3. Create a Refresh ID with an associated target. Define parameters to control how the DataOrchestrator ODS refreshes a target database with data from Colleague source files. (Optional) Create SQL views.

DataOrch Refresh (DORE)

Defining and Running an ODS Refresh on page 165.

4. Note: If your institution does not use stored computed columns, skip this step.

Activate and calculate stored computed columns for any ODS target database that contains stored computed columns. If your ODS target database uses the STUDENT.TERMS.CC file, update the flags as shown.

Define Stored Computed Column (DSCC)

Update Stored Computed Column (USCC)

Update GPA SCC Flags (UGSF)

Calculating Stored Computed Columns on page 181.


Workflow for Using the DataOrchestrator ODS

5. Run an error analysis report for any errors encountered when exporting data from a Colleague source file to an ODS target database.

DataOrch Error Analysis (DOEA)

Viewing Errors for a Refresh on page 193.

6. (Optional) Run the Transform Summary report to view information for target transforms associated with a target.

DataOrch Transform Summary (DOTS)

Viewing Target Transform Data for a Target on page 199.

7. (Optional) View historical information for previously run refreshes for a specific target.

DataOrch Refresh History (DORH)

Viewing the History of a Refresh on page 203.

8. (Optional) Copy existing transforms from one target to another, delete transforms, rename target transforms, and copy and rename transforms. Also, regenerate source transforms, and calculate foreign keys for transforms.

DataOrch Transform Main (DOMA)

Maintaining Transforms on page 209.

9. (Optional) Copy existing SQL views from one target to another, delete SQL views, rename SQL views, and copy and rename SQL views.

DataOrch View Maintenance (DOVM)

Maintaining SQL Views on page 219.

10.(Optional) Copy all transforms and SQL views from one or more targets to a single destination target.

DataOrch Target Copy (DOTY)

Copying Targets to Another Target on page 227

11.(Optional) Delete existing refreshes, targets, or refresh history records from the source Colleague database.

DataOrch Object Delete (DOOD)

Deleting DataOrchestrator ODS Objects on page 231.

Table 11: Steps to Set Up, Refresh, and Maintain the DataOrchestrator ODS and Operational Data Stores (cont’d)

Step Form Reference


Setting Up DataOrchestrator ODS Parameters: Overview of Using the DataOrchestrator ODS



Defining DataOrchestrator ODS Parameters

In This ChapterThis chapter describes how to set up the parameters for the DataOrchestrator ODS.

Table 12 lists the topics covered in this chapter.

Form UsedTable 13 lists and describes the form used in this chapter.


Topic Page

Form Used 59

Defining Parameters 60

Procedure for Defining DataOrchestrator ODS Parameters 64

Table 13: Form Used in This Chapter

Form Purpose

DataOrch Parameters (DOPA) Enter setup information for the DataOrchestrator ODS.


Setting Up DataOrchestrator ODS Parameters: Defining DataOrchestrator ODS Parameters

Defining ParametersUse the DataOrch Parameters (DOPA) form to define parameters that the DataOrchestrator ODS will use when performing refreshes.

Figure 7: DataOrch Parameters (DOPA) Form


Defining Parameters

Noteworthy Fields on the DOPA Form

The fields described in this section are important for setting up parameters for the DataOrchestrator ODS.

Source DMI_DAS Listener Name

Use the Source DMI_DAS Listener Name field to enter the name of the DMI Listener on the source database server.

Source DMI Listener Host Name

The Source DMI Listener Host Name field displays the TCP/IP address or the DNS alias name of the computer where the DMI Listener is installed.

Source DMI Listener Port

The Source DMI Listener Port field displays the port number associated with the DMI Listener.

Termination Error Threshold

Use the Termination Error Threshold field to specify the default termination error threshold for all transforms. When this threshold is reached, data load will stop to the ODS target database table for a specific transform.

Note: If you are using a SQL Server Colleague environment, leave this field blank.

Note: If you are using a SQL Server Colleague environment, this field will be blank.

Note: If you are using a SQL Server Colleague environment, this field will be blank.



You can use the error analysis report generated by the DataOrch Error Analysis (DOEA) form to identify the source of errors and correct the source data or modify the transform. For more information on this report, see Viewing the Error Analysis Report for a Refresh on page 194.

Depending on your target database type, this setting defines the following:

SQL Server. This value specifies the number of errors allowed per batch commit of a transform.

Oracle. This value specifies the number of errors allowed for the entire data load of a transform.

Batch Commit Size

Use the Batch Commit Size field to enter the batch commit size for the data load into the ODS target database. This is the number of rows to write and commit to the ODS target database for each batch load. Datatel recommends that you leave the default set at 1000.

Max Errors Saved

Use the Max Errors Saved field to enter the maximum number of error rows to return and store for a transform. An error row is a database row that could not be inserted into the target database, but which also doesn't stop the database from loading subsequent rows. There may be more errors than this number encountered by the refresh, but this is the maximum number stored by the DataOrchestrator ODS for later analysis.

Source Drive Name

For a Windows source environment, use the Source Drive Name field to enter the default source drive name of the temporary file path. If you are running on any other source environment, leave this field blank. This value specifies which drive on the source Colleague database server will store the temporary files created by the refresh.

Note: A refresh exports data to the ODS target database in batches. This setting allows you to control the size of these batches.


Defining Parameters

Source Temporary File Path

Use the Source Temporary File Path field to enter the default directory path on the source Colleague database server where the temporary files, created by the refresh, will be stored. Enter each element in the path on a separate line and omit any slashes (/).


The following permissions are needed for this directory:

UNIX UniData. For the directory where the files are being written: • The user logged in to Colleague must have read/write permissions. • The user who starts the source DMI_DAS must have read/write

permissions.

Windows UniData. For the directory where the files are being written: • The user logged in to Colleague must have read/write permissions. • The SYSTEM user must have read/write permissions.

UNIX Oracle. For the directory where the files are being written: • Oracle needs read/write permissions. • The user who starts the source DMI_DAS must have read/write

permissions.

The temporary files in this directory may contain data that might be considered sensitive at your institution. Access to the directory should be restricted accordingly.

Note: If you are using a SQL Server source environment, leave this field blank.




Procedure for Defining DataOrchestrator ODS Parameters

Step 1. From the Envision Runtime (UT) application, access the DataOrch Parameters (DOPA) form.

Step 2. Enter the parameters for the DataOrchestrator ODS. Refer to online help for more information.

Step 3. Save and exit from the DOPA form.



Setting Up Targets

Setting Up Targets

Defining a Target

In This PartThis Part describes how to create and modify a target, which defines the configuration of an operational data store on a target database.

Table 14 lists the topics covered in this Part.

Table 14: Topics in This Part

Topic Page

Forms Used 68

Defining a Target 70

Defining Additional Target Parameters 77

Defining Source Transforms 81

Defining Target Transforms 97

Defining Target Transform Columns 107

Defining Operations for Transform Columns 116

Defining and Creating SQL Views 143

Procedure for Defining a Target 157


Setting Up Targets: Defining a Target

Forms UsedTable 15 shows the forms used in this Part.

Table 15: Forms Used to Define a Target

Form Purpose

DataOrch Target (DOTA) Create and maintain a target, including the source and target transforms to refresh on a target database. Also view data on the status of this target.

Additional Target Parameters (DOTP)

Enter additional parameters for a target.

DataOrch Filter Criteria (DOFC) Enter a MIOSEL expression to limit the records of a file processed through a transform. Also, enter any file suite instances to be included when the transform is refreshed.

DataOrch Target Views (DOTV) Define SQL views for a target and view information about their statuses.

DataOrch View Spec (DOVS) Define the select statement, column documentation, and materialized view parameters for an SQL view.

For Source Transforms only

DataOrch Source Transform (DOST) Define the fields and computed columns of a file to refresh on the target database.

Generated Target Transforms (DOGT)

View the target transforms that were generated to implement a source transform.

For Target Transforms only

DataOrch Target Transform (DOTT) Define a target transform that extracts data from Colleague and creates a table containing that data on the target database.

Transform Columns (DOTC) Specify the output columns for a target transform. Each target column is defined by an input field and an optional transform operation that produces the value to store in the corresponding target database column.

Column Properties (DOPR) Define properties for a transform column.


Forms Used

For Target Transform Columns only - Optional Operations

Field Extract (DOFE) Define a field extract operation. This returns subfields of the data in the source field.

Multivalue Operation (DOMV) Define a multivalue operation. Either the Multivalue Position or the Multivalue Aggregation operation can be performed on a source field.

Pointer Reference (DOFR) Define a pointer reference operation. This references a field in a file, which is pointed to by a value in the source field.

Null Value Replacement (DONV) Define a null value replacement operation. Either the Null Replacement or the Null Test operation can be performed on a source field.

String Concatenation (DOCA) Define specifications for a concatenation operation for a transform column.

Substring Selection (DOSS) Define a substring selection operation. Set up a starting position of a substring for the source field and the length of the substring.

Validation Code LookUp (DOVC) Define a validation code lookup operation. Enter a validation code table name that corresponds to the value in the source field and choose which field to return from the validation code.

Expression Entry (DOEE) Define an expression to use to return a value for a transform column

Table 15: Forms Used to Define a Target (cont’d)

Form Purpose



Defining a TargetUse the DataOrch Target (DOTA) form as the starting point to create or modify a target. Depending on what you want to include in the target, you will also need to access additional forms. Figure 8 shows the additional forms that are available when defining a target.

Figure 8: Forms Available for Defining a Target

DOTA Define Source Transform

Define Target Transform

DOTT (define a transform that extracts data from Colleague)

DOST (select fields and computed columns of a file to refresh)

DOFC (limit records from the file) (optional) DOGT (view target transforms generated from the source transform)

DOFC (limit records from the file) (optional) DOTC (select output columns for a transform)

DOFE (field extract operation) DOMV (multivalue operation) DOFR (pointer reference operation) DONV (null value replacement operation) DOCA (string concatenation operation) DOSS (substring selection operation) DOVC (validation code lookup operation) DOEE (expression entry operation)

Define Operation (optional)

DOPR (define properties for a column)

Define Additional Target Parameters

DOTP (define additional parameters for target)

DOTV (define SQL views)

Define SQL Views (Optional)

DOVS (define select statement, column documentation, and materialized view parameters for an SQL view)


Defining a Target

Overview of the DataOrch Target (DOTA) Form

The DataOrch Target (DOTA) form is used to define and maintain targets, and to view information about the status of these targets. On the DOTA form, you can enter source and target transforms that you want to refresh to a target database.

Source transforms allow you to specify the data to move to a target ODS database by selecting a set of fields and/or computed columns from a single Colleague file. The selected data will be structured on the target database in the same way as the standard SQL representation for that data in a Colleague environment. Source transforms do not allow you to make any data transformations when the data is moved. Detail to the DataOrch Source Transform (DOST) form to include fields and computed columns that will be sent for a transform.

Target transforms allow you to specify the format of the data on the target ODS database, as well as any data transformation operations to be run when the data is moved. Target transforms are a powerful tool for the creation of ODS tables that are optimized for reporting purposes, and provide greater flexibility in defining how the data is structured on the ODS. Detail to the DataOrch Target Transform (DOTT) form to specify the input fields and computed columns for a transform, and to define how these data elements will be transformed when written to the target ODS database.

You must also detail from the DOTA form to the Additional Target Parameters (DOTP) form to define the Target Drive Name and Target Temporary File Path for the temporary files created by this process.

In addition, you can detail from the DOTA form to the DataOrch Target Views (DOTV) form to define the SQL views associated to this target.

After data has been refreshed to this target database, you can view statistics for the ODS that show its overall status.



Figure 9: DataOrch Target (DOTA) Form

Noteworthy Fields on the DOTA Form

The fields described in this section are particularly important for understanding as well as defining and modifying a target.

Status

The Status field displays the status of the target after the most recent refresh.

Oldest Transform Refresh and Latest Transform Refresh

These fields tell you the refresh dates and times for both the oldest and the most recently updated transforms associated with the operational data store. This allows you to see easily the date range of the transforms and determine whether you should take any action based on these dates. For example, if the


Defining a Target

Oldest Transform Refresh field has an older date than expected, this may alert you that one or more of your transforms is not being updated on a regular basis.

Description

Use the Description field to enter the text description to be displayed for this target.

Database Type

Use the Database Type field to enter the type of database where the operational data store will reside. Select the database type from the drop-down list. You can select SQL Server or Oracle.

Database Host Name

Use the Database Host Name field to enter the IP address of the computer on which the target database resides. You may have an existing DNS alias name created for the computer where the target database resides. Datatel recommends that you use that DNS alias name in the Database Host Name field for ease of use.

Database Port

Use the Database Port field to enter the port number of the server on which the target database resides.

Leave this field blank if your Oracle or Microsoft SQL Server database listener service is using the default port number; otherwise, you must enter a port number in this field.

The database port number is the database listener service port number.

For Oracle the default is either 1521 or 1526.

For Microsoft SQL Server the default is 1433.

Note: This is not the Datatel DMI Listener port number.



Database Name/TNS

Use the Database Name/TNS field to enter the target database name for SQL Server or the Transparent Network Substrate (TNS) name for Oracle.

This is the name of the target database to which you want to connect.

For a Microsoft SQL Server target database, enter the database name.

For an Oracle target database, enter the TNS.

Schema Name

Use the Schema Name field to enter the schema name for an Oracle target database.

The schema name is an “owner” string that is prefixed to every target table to uniquely identify the table in a database instance. By default, all tables created on the target database will be created with the schema name of the user who is logged in. To create tables in another schema on the target database, specify the name of the schema in this field.

For a Microsoft SQL Server database, the only schema name that Datatel supports is “dbo”. Therefore, if you selected “SQL Server” in the Database Type field, the Schema Name field is populated with this value and is inquiry only.

Database User ID

Use the Database User ID field to enter a valid user ID for accessing the target database. This ID must have full access to the target database.

Password

Use the Password field to enter the password for accessing the target database.

Enter a valid password for accessing the target database, corresponding to the user ID entered in the Database User ID field. When you enter the password, the value is displayed as asterisks.


Defining a Target

Temporary File Path

Detail on the Temporary File Path field to access the Additional Target Parameters (DOTP) form to enter the name of the directory that is used for temporary files created on the ODS target database during a refresh. For more information, see Defining Additional Target Parameters beginning on page 77.

SQL Views

To view and define SQL views, detail on the SQL Views field to access the DataOrch Target Views (DOTV) form.

From the DOTV form, you can detail to the DataOrch View Spec (DOVS) form to define the select statement, column documentation and materialized view parameters for an SQL view. For more information, see Defining and Creating SQL Views beginning on page 143.

Source Transforms

Use the Source Transforms table to enter a file name to be included in the ODS target database. File names entered here can be refreshed to an ODS target database by using the DataOrch Refresh (DORE) form. You cannot enter application-hierarchy files here (for example, ST.VALCODES, CORE.VALCODES, etc.). Instead, specify the file suite template file name (for example, VALCODES, etc.).

If you want to select fields to be included, detail on the specific source transform to access the DataOrch Source Transform (DOST) form.

If you specify a file containing stored computed columns in this field, you must run the Update Stored Computed Column (USCC) process to ensure that the stored computed columns are refreshed before the DORE process is run. For more information, see Defining Source Transforms beginning on page 81.

Note: The file entered cannot be a logical view. The file must be a physical file in Colleague.

Note: Deleting a file from the Source Transforms table does not automatically delete the corresponding table on the target database. This must be done manually by the database administrator.



Target Transforms

Use the Target Transforms table to enter a name for a target transform to be included in the target database.

The target transform name entered here will be used as the name of the corresponding target table when this transform is refreshed using the DataOrch Refresh (DORE) form.

To define the input fields and column transform operations for the transform, detail on the specific target transform to access the DataOrch Target Transform (DOTT) form. For more information, see Defining Target Transforms beginning on page 97.

Refresh Date/Time

The Refresh Date/Time columns display the date and time when the most recent refresh started for this target for source and/or target transforms.

Status

The Status column displays the status of the most recent refresh for this target for source and/or target transforms.

Columns

For source transforms, the Columns column displays the number of columns included in the source transform. This number is a sum of the columns from the generated target transforms for this source transform (not including the primary key, which is inferred).

For target transforms, the Columns field displays the number of columns included in the target transform, including the primary key.

Note: Deleting a name from the Target Transforms table does not automatically delete the corresponding table on the target database. This must be done manually by the database administrator.


Setting Up Targets


In This ChapterThis chapter describes how to set up the additional target parameters for the DataOrch Target (DOTA) form.


Form Used Table 17 lists and describes the form used in this chapter.


Topic Page

Form Used 77

Defining Additional Target Parameters 78


Form Purpose

Additional Target Parameters (DOTP) Enter additional parameters for a target.


Setting Up Targets: Defining Additional Target Parameters

Defining Additional Target ParametersDetail on the Temporary File Path field from the DataOrch (DOTA) form. This allows you to access the Additional Target Parameters (DOTP) form to define the drive name and temporary file path for a target.

Figure 10: Additional Target Parameters (DOTP) Form



Noteworthy Fields on the DOTP Form

The fields described in this section are particularly important for defining parameters for a target.

Target Drive Name

Use the Target Drive Name field to enter the target drive name of the temporary file path for a Windows server. If you are running on any other source environment, this field should be left blank.

This field specifies which drive on the ODS target database server will store the temporary files created by the refresh.

Target Temporary File Path

Use the Target Temporary File Path to enter the directory path on the ODS target database server where the temporary files created by the refresh will be stored. Enter each element in the path on a separate line and omit any slashes (/).



UNIX Oracle. For the directory where the files are transferred to and will be bulk loaded from:• The user who starts up the source DMI_DAS must have read/write

permissions.

Windows SQL Server. For the directory where the files are created (via the Microsoft SQL Server bcp utility) and will be bulk loaded from: • The SYSTEM user must have read/write permissions.

• The DMI_DAS logs in with SQL Server authentication, so the SQL Server process account must have read/write permissions for the BULK INSERT command.



Setting Up Targets: Defining Additional Target Parameters


Procedure for Defining Additional Target Parameters

Step 1. See Procedure for Defining a Target beginning on page 157.


Setting Up Targets


In This ChapterThis chapter describes how to set up the source transforms for the DataOrch Target (DOTA) form.


Forms Used Table 19 lists and describes the forms used in this chapter.


Topic Page

Forms Used 81

Defining Source Transforms 82

Using the DataOrch Source Transform (DOST) Form 83

Defining Filter Criteria 89

Viewing Generated Target Transforms 93

Table 19: Forms Used in This Chapter

Form Purpose

DataOrch Source Transform (DOST) Define the fields and computed columns of a file to refresh on the target database.


Generated Target Transforms (DOGT) View the target transforms that were generated to implement a source transform.


Setting Up Targets: Defining Source Transforms

Defining Source TransformsTo define a source transform for a target, use the following forms:

DataOrch Source Transform (DOST)

Filter Criteria (DOFC)

To view the target transforms generated from a source transform, detail to the Generated Target Transforms (DOGT) form from the DOST form.




Detail on a Source Transform on the DataOrch Target (DOTA) form to access the DOST form. Use the DOST form to select the fields and computed columns of a file that will be refreshed to the target database when you use the DataOrch Refresh (DORE) form. You can also detail to the DataOrch Filter Criteria (DOFC) form to specify filters, including file suite instances, to limit the records of the input file of the transform. In addition, you can view data about when the source transform was last refreshed.

You also have the option to:

Manipulate the As of Date field for an incremental update to control the content of the update. The As of Date field must have a value to perform an incremental update, whether or not the transform has an Incremental Date Field.

Set the refresh to truncate input fields for string-type columns to the length specified in the transform.

Specify an error threshold for the transform at which the data export will stop.

The source transform will be run by generating a set of target transforms to create and populate data for each target database table needed to hold the specified input fields of the source file. To view the target transforms generated from this source transform, detail to the Generated Target Transforms (DOGT) form.

When target transforms are generated, their output fields are built with the data types and sizes of the referenced Colleague fields. However, if data types or sizes of any referenced Colleague fields change, the target transforms need to be recreated. To do this, you must use the Regenerate Source Transforms operation on the DataOrch Transform Maint (DOMA) form. For more information, see Maintaining Transforms beginning on page 209.



Figure 11: DataOrch Source Transform (DOST) Form



Noteworthy Fields on the DOST Form

The fields described in this section are particularly important when selecting the fields and computed columns of a file that will be refreshed to the target database.

Select All Fields (Y/N)

Use the Select all Fields field to indicate whether you want to select all available fields for the source transform. Enter Yes to select all available fields (which clears any previous entries in the Include Fields table). Enter No to choose specific fields in the Include Fields table or to select no fields.

Include Fields

Use the Include Fields list to select which fields will be refreshed for the file you chose. The fewer fields you select, the better the refresh performance will be and the less chance errors will be encountered.

If you modify the Include Fields list, then target tables will be recreated during the next refresh. To confirm that you want this to occur, you see the following alert:

Field list change will cause the target table to be recreated. Proceed (Y/N).

Click Yes to proceed or No to return to the Include Fields list and leave the list unchanged. If you click Yes, the As of Date will be deleted to force a full refresh.

Technical Tip: If you use a wildcard LookUp for your fields, you can select multiple elements from the LookUp resolution form.



Include Computed Columns

Use the Include Computed Columns list to select which computed columns will be refreshed for the file you chose. Computed columns are recalculated during the refresh, so the fewer computed columns you select, the better the refresh performance will be and the less chance errors will be encountered.

If you modify the Include Computed Columns list, then target tables will be recreated during the next refresh. To confirm that you want this to occur, you see the following alert:

Field list change will cause the target table to be recreated. Proceed (Y/N).

Click Yes to proceed or No to return to the Include Computed Column list and leave the list unchanged. If you click Yes, the As of Date will be deleted to force a full refresh.

Filters/File Suite Instances

Detail on the Filters/File Suite Instances field to access the DataOrch Filter Criteria (DOFC) form in order to enter filter criteria, including file suite instances, for this source transform. Colleague uses this criteria to specify what records from the input file will be processed by the source transform when it is run.

Technical Tip: If you use a wildcard LookUp for your computed columns, you can select multiple elements from the LookUp resolution form. The LookUp list is restricted to only single-valued computed columns that have been generated for database use on the Define Computed Column (DCC) or the Bundle Generation (BNGN) forms. You cannot enter multivalued computed columns as source data elements for a transform.

Note: For a multi-part key file, each CDD element making up the key is included in this list as a separate CDD element.



Incremental Date Field

The Incremental Date Field displays the MIO-managed change date field for the input file you select for the source transform. There are two ways to control what is selected for an incremental update:

The Incremental Date Field together with the As of Date.

The Incremental Filter Criteria.

As of Date

The As of Date field affects only incremental refreshes. If the transform has an Incremental Date Field, then the records updated on or after this will be in the next incremental refresh. You can also use the As Of Date field in the expression specified for the incremental filter criteria to control what data is included during the next incremental refresh. For more information on using incremental filter criteria, see Defining Filter Criteria on page 89.

If the transform has an Incremental Date field, then all records from the source file that have a MIO-maintained change date on or after this As of Date will be processed by the transform in the next incremental refresh.

The refresh updates this date with the current date when a transform is refreshed to the target database and no errors are encountered. This As of Date will not be modified if any source file records cannot be written to the ODS target database due to data errors. This ensures that the refresh will include these records in a future refresh when the data errors are corrected in the source file.

You can manually change the As of Date to affect which records are selected during an incremental refresh. You may want to do this if you do not want the refresh to attempt to send previously identified error records in future incremental refreshes. However, this will mean that not all records from the source environment will exist on the target database.

Note: Not all files will have a MIO-managed change date field. If this is the case for this file, you see the following message in this field: NO MIO CHANGE DATE FIELD.



Truncate Strings (Y/N)

The Truncate Strings field allows you to indicate whether or not you want the refresh to truncate input fields for string-type columns in this transform if the data exceeds the length specified in Colleague for the source field.


Use the Termination Error Threshold field to enter the error threshold for the selected transform. When the threshold is reached, data load will stop for the transform to the ODS target database table.


SQL Server. The number of errors allowed per batch commit, as specified on the DataOrch Parameters (DOPA) form.

Oracle. The number of errors allowed for the entire data load of this transform.

This setting overrides the default in the Termination Error Threshold field on the DataOrch Parameters (DOPA) form.

View Generated Transforms

Each DataOrchestrator source transform will create and update one or more tables on the ODS target database. Each source transform will be implemented by a set of target transforms, each of which will create and modify one of these target tables. You can detail to the Generated Target Transforms (DOGT) form to view the set of target transforms generated from this source transform.

Note: Truncation applies only to string-type columns. Also, truncation does not apply to the primary key of a transform.



Defining Filter Criteria

Detail to the Filter Criteria (DOFC) form to enter and verify MIOSEL expressions to use against the input file of a transform to limit the set of records from the file that will be processed through the transform. When saving from the form, the MIOSEL statements (containing the entered expressions against the input file for the transform) will be run to verify their syntax.

The expressions will be used during a full refresh as:

SELECT [file name] [filter criteria to run]

The expressions will be used during an incremental refresh as:

SELECT [file name] [filter criteria to run] AND ( [ incremental date field GE “as of date” ] OR ( [incremental filter criteria to run] ))

For example:

SELECT PERSON WITH SPOUSE AND ( PERSON.CHANGE.DATE GE "12/30/08" OR ( LAST.NAME EQ “Smith” ))

The expressions will be used during an incremental refresh when a transform does not have an Incremental Date Field as:

SELECT [filename][filter criteria to run] AND [incremental filter

criteria to run]

For example:

SELECT PERSON WITH SPOUSE AND LAST.NAME EQ "Smith"

If the input file is a file suite template, you can use the DOFC form to filter the instances of that file suite to include in the refresh. You can also indicate which of these instances to update on an ongoing basis, and which contain static data that will not change and therefore do not need to be updated as part of the refresh.

Note: If you add a new file suite instance, but enter No in the Update Instance field, the file suite instance will not be included when this transform is refreshed. You must run the refresh at least once with the Update Instance field set to Yes.



Figure 12: DataOrch Filter Criteria (DOFC) Form

Note: Figure 12 shows an example of a filter criteria for a source transform. However, you can also use a filter criteria for a target transform.



Noteworthy Fields on the DOFC Form

The fields described in this section are particularly important when limiting the set of records from the file that will be processed through a transform.

File Name

The File Name field displays the name of the file to which the filter criteria will be applied.

Filter Criteria to Run

Use the Filter Criteria to Run field to add or modify a MIOSEL expression you want to use to filter the input records for the transform. The expression must start with the selection clause keyword WITH.

The following keywords are also available to use as a filter criteria to be replaced by a value when the refresh is run:

TODAY(). This returns the current date when the refresh is run. For example:

ACPG.START.DATE LE TODAY()

ASOFDATE(). This returns the As of Date specified for the transform.

If you modify this field, the next refresh of this transform will be a full refresh.

Incremental Filter Criteria to Run

Use the Incremental Filter Criteria to Run field to construct a MIOSEL expression you want to use to filter the input records for the transform for an incremental refresh. If the transform has an Incremental Date Field, then this expression will be added to the Incremental Date Field filter for an incremental refresh with the OR keyword. Otherwise, the Incremental Filter Criteria expression alone will be added to the overall filter with the AND keyword. The expression must start with the selection clause keyword WITH and cannot have any other WITH keywords in its expression (use AND or OR instead).

Note: You cannot enter an SQL statement as the filter criteria.



The following keywords are also available to use as a filter criteria to be replaced by a value when the refresh is run:

TODAY(). This returns the current date when the refresh is run. For example:

ACPG.START.DATE LE TODAY()

ASOFDATE(). This returns the As of Date specified for the transform.

File Suite Instances

Use the File Suite Instances list to enter the file suite instance IDs to include when the transform is refreshed.

Update Instance (Y/N)

Use the Update Instance (Y/N) field to indicate whether this instance needs to be updated when the transform is refreshed. Enter Yes to include the instance; otherwise, enter No. If you enter No, the instance is updated only if the columns selected from the file suite template for the transform change. In this case, the data for all file suite instances will be deleted and recreated.

Note: If you are adding a new file suite instance, but enter No in this field, the file suite instance will not be included when this transform is refreshed. You must run the refresh at least once with this field set to Yes.



Viewing Generated Target Transforms

Detail on the View Generated Transforms field from the DOST form to access the Generated Target Transforms (DOGT) form. The DOGT form allows you to view target transforms that have been generated to implement a specific source transform. Each of the target transforms represents a table on the target database.

Figure 13: Generated Target Transforms (DOGT) Form



Noteworthy Fields on the DOGT Form

The fields described in this section are particularly important when viewing target transforms that have been generated to implement a specific source transform.

Generated Target Transforms

The Generated Target Transforms column displays the name of a generated target transform that is associated with a particular source transform. This name represents a table that will be created on the target database.

Status

The Status column displays the status of the most recent refresh of the transform.

Start Date and Start TIme

These columns display the start date and time of the most recent refresh of the target transform.

Total Rows

The Total Rows column displays the count of the total number of rows written to the target database during the most recent run of the target transform.

Deleted

The Deleted column displays the count of the number of rows deleted on the target database during the most recent run of the target transform.

Error

The Error column displays the count of the number of transform output rows that could not be loaded into the target database due to data or execution errors during the most recent run of the target transform.



End Date and End Time

These columns display the end date and time of the most recent refresh of the target transform.

Procedure for Defining Source Transforms

See Procedure for Defining a Target beginning on page 157.




Setting Up Targets

Defining Target Transforms

In This ChapterThis chapter describes how to set up the target transforms for the DataOrch Target (DOTA) form.



Topic Page

Forms Used 98

Defining Target Transforms 100

Using the DataOrch Target Transform (DOTT) Form 101

Defining Target Transform Columns 107

Defining Properties for a Target Transform Column 113

Defining Operations for Transform Columns 116


Setting Up Targets: Defining Target Transforms



Form Purpose

DataOrch Target Transform (DOTT) Define a target transform that extracts data from Colleague and creates a table containing that data on the target database.


Transform Columns (DOTC) Specify the output columns for a target transform. Each target column is defined by an input field and an optional transform operation that produces the value to store in the corresponding target database column.

Column Properties (DOPR) Define properties for a transform column.

For Target Transform Columns only - Optional Operations

Field Extract (DOFE) Define a field extract operation. This returns subfields of the data in the source field.

Multivalue Operation (DOMV) Define a multivalue operation. Either the Multivalue Position or the Multivalue Aggregation operation can be performed on a source field.

Pointer Reference (DOFR) Define a pointer reference operation. This references a field in a file, which is pointed to by a value in the source field.

Null Value Replacement (DONV) Define a null value replacement operation. Either the Null Replacement or the Null Test operation can be performed on a source field.

String Concatenation (DOCA) Define specifications for a concatenation operation for a transform column.

Substring Selection (DOSS) Define a substring selection operation. Set up a starting position of a substring for the source field and the length of the substring.


Forms Used

Validation Code LookUp (DOVC) Define a validation code lookup operation. Enter a validation code table name that corresponds to the value in the source field and choose which field to return from the validation code.

Expression Entry (DOEE) Define an expression to use to return a value for a transform column

Table 21: Forms Used in This Chapter(cont’d)

Form Purpose



Defining Target TransformsTo define a target transform for a target, use the following forms:

DataOrch Target Transform (DOTT)

Filter Criteria (DOFC)

Transform Columns (DOTC)

You can also choose to detail from the DOTC form to define the following for a target transform column:

Define properties for the column on the Columns Properties (DOPR) form.

Define an operation for the column using one of the following forms:• Field Extract (DOFE)• Multivalue Operation (DOMV)• Pointer Reference (DOFR)• Null Value Replacement (DONV)• String Concatenation (DOCA)• Substring Selection (DOSS)• Validation Code LookUp (DOVC)• Expression Entry (DOEE)



Using the DataOrch Target Transform (DOTT) Form

Detail on a Target Transform on the DataOrch Target (DOTA) form to access the DOTT form. Use the DOTT form to define a target transform that extracts data from Colleague and creates a table containing that data on the target database. You can also view information about when a specific target transform was last refreshed. To view and modify the columns associated with this target transform, detail to the Transform Columns (DOTC) form. In addition, you can detail to the DataOrch Filter Criteria (DOFC) form to specify criteria to filter the records of the source file of the transform.

You also have the option to:

Manipulate the As of Date field for an incremental update to control the content of the update. The As of Date field must have a value to perform an incremental update, whether or not the transform has an Incremental Date Field.

Indicate that the transform needs to be refreshed as a full update.

Set the refresh to truncate input fields for string-type columns in the transform.

Specify an error threshold for the transform at which the data export will stop.

Enter a description of the transform.



Figure 14: DataOrch Target Transform (DOTT) Form



Noteworthy Fields on the DOTT Form

The fields described in this section are particularly important when defining a target transform that extracts data from Colleague and creates a table containing that data on the target database.

Status

The Status field displays the status of the most recent refresh of the transform.

Start Date/Time

These fields display the start date and time of the most recent refresh of the target transform.

End Date/Time

These fields display the end date and time of the most recent refresh of the target transform.

Refresh ID

The Refresh ID field displays the ID of the most recent refresh that processed the target transform.

Total Rows

The Total Rows field displays the count of the total number of rows written to the target database during the most recent run of the target transform.

Deleted

The Deleted field displays the count of the number of rows deleted from the target database during the most recent run of the target transform.



Error

The Error field displays the count of the number of transform output rows on which an error occurred during the most recent run of the target transform.

Source File

The Source File field displays the source file on which the target transform is based. The source file selected will be used to control which fields and computed columns you will be allowed to select when creating the target transform. Also, the source file’s incremental change date field will control the incremental updates of the target transform.

Column Definitions

Detail on the Column Definitions field to access the Transform Columns (DOTC) form in order to view and modify the columns associated with the target transform.

Filters/File Suite Instances

Detail on the Filters/File Suite Instances field to access the DataOrch Filter Criteria (DOFC) form in order to enter filter criteria, including file suite instances, for the target transform. Colleague uses this criteria to specify what records from the source file are processed by the target transform when it is run. For more information on the DOFC form, see Defining Filter Criteria on page 89.

Incremental Date Field

The Incremental Date Field displays the MIO-managed change date field for the source file you selected for the target transform. There are two ways to control what is selected for an incremental update:

The Incremental Date Field together with the As of Date

The Incremental Filter Criteria

Note: The file entered cannot be a logical view. The file must be a physical file in Colleague.



As of Date

The As of Date field affects only incremental refreshes. If the transform has an Incremental Date Field, then the records updated on or after this will be in the next incremental refresh. You can also use the As Of Date field in the expression specified for the incremental filter criteria to control what data is included during the next incremental refresh. For more information on using incremental filter criteria, see Defining Filter Criteria on page 89.

If the transform has an Incremental Date Field, then all records from the source file that have a MIO-maintained change date on or after this As of Date will be processed by the transform in the next incremental refresh.

The refresh updates this date with the current date when a transform is refreshed to the target database and no errors are encountered. This As of Date will not be modified if any source file records cannot be written to the ODS target database due to data errors. This ensures that the refresh will include these records in a future refresh when the data errors are corrected in the source file.

You can manually change the As of Date to affect which records are selected during an incremental refresh. You may want to do this if you do not want the refresh to attempt to send previously identified error records in future incremental refreshes. However, this will mean that not all records from the source environment will exist on the target database.

Refresh as Full Only (Y/N)

In the Refresh as Full Only (Y/N) field, indicate whether or not this transform can be updated only as full when refreshed using the DataOrch Refresh (DORE) process. For example, a target transform should not be refreshed incrementally in the following instances:

The transform contains computed columns dependent on data from files other than the transform’s source file that change on a regular basis.

The transform has pointer-reference operations and the referenced data values change on a regular basis.

Note: Not all source files will have a MIO-managed change date field. If this is the case for this source file, you see the following message in this field: NO MIO CHANGE DATE FIELD.



In both these instances, the inclusion of the source records is not triggered in an incremental refresh. For this reason, you would enter Yes for the transform to update only as full during a refresh.

Truncate Strings (Y/N)

Use the Truncate Strings (Y/N) field to indicate whether you want the refresh to truncate input fields for string type columns in the transform if the data exceeds the specified length for the column.

Truncation applies only to string type columns. Also, truncation does not apply to the primary key of a transform.


Use the Termination Error Threshold field to specify the error threshold for the transform. When the threshold is reached, data load will stop for the transform to the ODS target database table.


SQL Server. The number of errors allowed per batch commit, as specified on the DataOrch Parameters (DOPA) form.

Oracle. The number of errors allowed for the data load of the transform.

This setting overrides the default in the Termination Error Threshold field on the DataOrch Parameters (DOPA) form.

Description

Enter or view a description for this target transform.

Note: If this field is set to Yes, the target transform will be refreshed as a full update regardless of the setting on the DataOrch Refresh (DORE) form in the Update Type or Override Update Type fields.



Defining Target Transform Columns

Detail from the DOTT form to the Transform Columns (DOTC) form to specify the output columns for a target transform. Each target column is defined by an input field and an optional transform operation that will produce the output value to be stored in the corresponding ODS target database column.

The available input fields that can be used are fields and computed columns from the source file of this target transform. Other target columns of this transform can also be used as source fields, in order to nest or combine column operations. However, to be used as a source field for a column, other transform columns must precede the column being defined in the transform column list.

When you create a target transform, be aware of the following cardinality rules that apply to columns marked “Yes” in the Export (Y/N) field:

Only one unassociated multivalue column can be included in a transform unless its data type has been changed to a single value by a transform operation.

Multiple associated data elements may be included only if they are from the same association.

Unassociated multivalues and associated multivalues cannot be included in the same target transform.

When you save from this form, these rules will be checked. If the rules are not followed in the transform, you will receive an error message.

You can also detail from this form to the Column Properties (DOPR) form to specify properties for a column. If you define properties on the DOPR form, then the DOTC form will display an “X” in the Addl field.



Figure 15: Transform Columns (DOTC) Form



Noteworthy Fields on the DOTC Form

The fields described in this section are particularly important when specifying the output columns for a target transform.

Source File

The Source File field displays the source file on which the target transform is based. The source file will be used as the reference from which fields and computed columns can be chosen when creating the target transform. Also, the source file’s incremental change date field will control the incremental updates of the target transform.

Target Key Column

The Target Key Column field displays the name of the target primary key column to be created on the target database for a target transform.

Source Key Field

The Source Key Field field displays the source key field to which the target primary key column is mapped.

Key Size

The Key Size field displays the size of the target key column to be created on the target database.

Key Target Data Type

The Key Target Data Type field displays the data type of the target key column to be created on the target database.

Export Key (Y/N)

The Export Key (Y/N) field displays the export flag for the Target Key Column. The Target Key Column will always have “Yes” as its export flag. This column will always be exported to the target database.



Target Column

Use the Target Column field to enter the name of the column to be created in the target table for this transform in the ODS target database.

To add a timestamp column, enter DO_REFRESH_TIMESTAMP, or use LookUp to select that value. The timestamp column indicates when the refresh for this target was started, not when the rows were created in the target database.

Source Field

Use the Source Field field to enter the source field or computed column from the source file that is the input to the target column. In order to nest or combine operations, the source field can also be another target column, which must be defined in the transform column list preceding its use as a source field.

Size

Use the Size field to enter the size of the target column to be created on the target database.

Target Data Type

The Target Data Type field displays the Envision data type that describes the column to be created on the target database. The data type may be modified based on the transform operation that is selected for the target column.

Note: The computed columns available are restricted to only single-valued computed columns that have been generated for database use on the Define Computed Column (DCC) or Bundle Generation (BNGN) forms. You cannot enter multivalued computed columns as source data elements for a transform.



Operation

Use the Operation field to choose an operation to perform in order to generate an output value for the target column. When you choose an operation, you will detail automatically to the appropriate form to allow you to enter specifications to use with that operation. There are ten operations available:

Field Extract. Colleague opens the Field Extract (DOFE) form. This form allows you to extract a subset of the fields of a source delimited string by specifying the field delimiter and the number of fields to extract from a starting field.

Multivalue Aggregation. (For multivalue target data types only.) Colleague opens the Multivalue Operation (DOMV) form. This form allows you to select from different aggregation operations including: Sum, Count, Min, and Max. If you select this operation, the column’s data type is changed to single-value.

Multivalue Position. (For multivalue target data types only.) Colleague opens the Multivalue Operation (DOMV) form. This form allows you to choose a specific position or the last position in a multivalued field to become a column in the target transform. If you select this operation, the column’s data type is changed to single-value.

Pointer Reference. Colleague opens the Pointer Reference (DOFR) form. This form allows you to specify a pointer field as the source field for the column and use this operation to select data fields from another Envision file using the pointer field as the link. Multivalued target columns are restricted to referencing only single-valued fields from another file.

Null Replacement. Colleague opens the Null Value Replacement (DONV) form. This form allows you to specify a value in the If Null Return field that will be returned if the source field value for the column is null. The data type in this field must match the data type of the source field.

Null Test. Colleague opens the Null Value Replacement (DONV) form. This form allows you to specify a value for both the If Null Return and If Not Null Return fields to create a Null Test Operation to choose the value in the column based on whether the source field value is null or not. The data types of the two fields must match.

String Concatenation. Colleague opens the String Concatenation (DOCA) form. This form allows you to create a concatenation operation using quoted strings, single-valued string type fields from the source file, or single-valued string type target columns from the associated transform defined before this column. All arguments must be single-valued source data elements or literals.

Substring. Colleague opens the Substring Selection (DOSS) form. This form allows you to define a starting position and length of a substring for the source field.

Validation Code LookUp. Colleague opens the Validation Code LookUp (DOVC) form. This form allows you to enter a validation code table name



that corresponds to the value in the source field and choose which field to return from the validation code.

Expression Entry. Colleague opens the Expression Entry (DOEE) form. This form allows you to enter an expression to use to return a value for this column. Operations can be nested.

Addl

The Additional field displays an “X” if properties were defined for the column on the Column Properties (DOPR) form. These properties include:

Creating an index.

Entering an output format string.

Specifying foreign keys.

Export (Y/N)

Use the Export (Y/N) field to export the target column to the target database. To export the target column, enter Yes; otherwise, enter No.

You would typically enter “No” if a target column is used in another operation, but the column’s data is not needed on the target database. Columns that are marked “No” will not be created on the target database and will not affect the cardinality of the target transform.



Defining Properties for a Target Transform Column

Detail from the DOTC form to the Column Properties (DOPR) form to define properties for a transform column, including the following:

An index for the column.

The column’s output format string.

Foreign keys for the column.

Figure 16: Column Properties (DOPR) Form

Note: You cannot modify the output format string for the key field of a transform.



Noteworthy Fields on the DOPR Form

The fields described in this section are particularly important when defining properties for a target transform column.

Target ID

The Target ID field displays the ID of the target containing the column.

Target Transform

The Target Transform field displays the name of the target transform containing the column.

Target Column

The Target Column field displays the name of the target transform column.

Create Index (Y/N)

Use the Create Index (Y/N) field to indicate whether you want the refresh to create an index in the ODS target database for the transform column. Enter Yes to create an index for this column; otherwise, enter No or leave this field blank.

Output Format String

Use the Output Format String field to enter a column’s output format string, which corresponds to the Conversion String allowable in Envision. The format defined for the column determines the type of column on the ODS target database.

Note: Indexes are created in the ODS target database during a refresh.



Related Transform

In the Related Transform field, enter transform names to relate to the column. A foreign key will be created in the ODS target database from the column to the table created for the related transform.

Create Foreign Key (Y/N)

Use the Create Foreign Key (Y/N) field to indicate whether you want the refresh to create a foreign key in the ODS target database to point to the corresponding related transform in the Related Transform field.

Enter Yes to create a foreign key for the column. Enter No if you do not want a foreign key to be created.

Description

Use the Description field to enter or view a description for this target transform column.

Note: To detect foreign key relationships automatically, choose the “Calculate Foreign Keys” operation in the Operation field on the DataOrch Transform Maint (DOMA) form.



Defining Operations for Transform Columns

Detail from the DOTC form to define an operation for a target transform column using one of the following forms:

Field Extract (DOFE)

Multivalue Operation (DOMV)

Pointer Reference (DOFR)

Null Value Replacement (DONV)

String Concatenation (DOCA)

Substring Selection (DOSS)

Validation Code LookUp (DOVC)

Expression Entry (DOEE)



The Field Extract (DOFE) Form

Detail from the DOTC form to use this form to define a field extract column operation. Use the field extract operation to return one or more subfields of the data in the source field where the subfields are delimited by the passed character. This form also provides a description field to enter a description of the column.

Figure 17: Field Extract (DOFE) Form



Noteworthy Fields on the DOFE Form

The fields described in this section are particularly important when defining a field extract column operation.

Target ID

The Target ID field displays the ID of the target associated with the target transform.

Target Transform

The Target Transform field displays the name of the current target transform.

Target Column

The Target Column field displays the name of the target column that will have its value determined by the result of the operation.

Source File

The Source File field displays the source file on which the definition of the target transform is based.

Source Field

The Source Field field displays the source field to which the target column is mapped. This is the field against which the field extraction operation will be applied. The source field can also be another target column, as long as it precedes this column in the transform’s column list.

Delimiter

The Delimiter field allows you to enter the delimiter to use as a separator for the source field’s data. If you leave this field blank, Colleague uses a single space as the delimiter. Only one-character entries are accepted, with the exception of the @VM and @SM expressions, which represent value mark and sub-value mark delimiters, respectively.



For example, an asterisk is used as the delimiter to get the first element of a multi-part key.

Start Field Number

Use the Start Field Number field to enter the starting field number, from the fields as delineated by the delimiter, at which to start extracting fields for the field extract.

Number of Fields to Extract

Use the Number of Fields to Extract field to enter the number of fields to extract, beginning with the start field number.

Description

Use the Description field to enter a description for the target transform column.



The Multivalue Operation (DOMV) Form

Detail from the DOTC form to use this form to define a multivalue transform operation. Either the Multivalue Position or the Multivalue Aggregation operations can be performed on the source field. The result of either operation is a single value. This form also provides a description field to enter a description of the target column.

Figure 18: Multivalue Operation (DOMV) Form



Noteworthy Fields on the DOMV Form

The fields described in this section are particularly important when defining or viewing a multivalue transform operation.

Target ID


Target Transform


Target Column

The Target Column field displays the name of the target column from the transform that will have its value determined by the result of the operation.

Source File

The Source File field displays the source file on which the target transform is based.

Source Field

The Source Field field displays the multivalued source field to which the target column is mapped. This is the field to which the aggregation operation will be applied. The source field can also be another target column, which needs to be defined before being used as a source field.

Position in List

The Position in List field displays the position in a multivalued list contained in the source field that the column will reference. You can enter a positive integer or the keyword “last” in this field to select a specific value in the multivalued source field that the column will reference. The result of the operation will be a single value.



Aggregation Operation

Use the Aggregation Operation field to select from a list of available aggregation operations to perform on the source field. The result of the operation will be a single value.

Count. A count of the number of values in the list.

Min. The minimum or lowest value in the list.

Max. The maximum or highest value in the list.

Sum. The sum of all values in the list.

Description




The Pointer Reference (DOFR) Form

Detail from the DOTC form to use this form to define a pointer reference column operation. This allows a reference to a field of an Envision file other than the source file defined for the target transform. The pointer reference operation uses the value in the Source Field field as a pointer to a record of this second file, and returns the value of the field named in the operation. Multivalued source fields are restricted to referencing only single-valued fields from another file.

This form also provides a description field to enter a description of the column.

Figure 19: Pointer Reference (DOFR) Form

Note: Only fields from another file can be selected. Also, computed columns from another file are not available for selection.



Noteworthy Fields on the DOFR Form

The fields described in this section are particularly important when defining a pointer reference column operation.

Target ID


Target Transform


Target Column


Source File

The Source File field displays the source file for the overall target transform of which the column is a component. This file contains the source field, unless the source file is itself another column defined in the target transform.

Source Field

The Source Field field displays the source field that the target column will use as a pointer to a secondary file. The source field can also be another target column from the transform as long as it is defined before being used as a source field.



File

Use the File field to enter the file that contains the field referenced by the selected pointer.

Pointer

The Pointer field displays the key to the secondary file. The value of the pointer field selected is used to look up the referenced record in the secondary file.

Field

Use the Field field to enter the field in the secondary file that is being returned by the operation.

Description


Note: If the Source File is a file suite template file (for example, GLS.FYR, CS.ACYR, etc.), then you can specify a different file suite template file of the same type (for example, .FYR, .ACYR, etc.) in this field. When a refresh is run, the file suite instance string for the file specified in this field will be replaced with the file suite instance being processed.



The Null Value Replacement (DONV) Form

Detail from the DOTC form to use this form to define a null value replacement operation.

If you selected the Null Replacement operation for this column on the Transform Columns (DOTC) form, you can specify a value in the If Null Return field that will be returned if the source field value for the column is null. However, the data type in this field must match the data type of the source field.

If you selected the Null Test operation for this column on the DOTC form, you can specify a value for both the If Null Return and If Not Null Return fields to create a Null Test Operation to choose the value in the column based on whether the source field value is null or not. However, the data types of the two fields must match. For example, to output a column in a transform indicating whether a Faculty member has a special status, you can specify the Null Test operation with an input field of FAC.SPECIAL.STATUS and enter No in the If Null Return field and Yes in the If Not Null Return field. In addition, you can enter a description of the target column.

Figure 20: Null Value Replacement (DONV) Form



Noteworthy Fields on the DONV Form

The fields described in this section are particularly important when defining a a null value replacement operation.

Target ID


Target Transform


Target Column


Source File

The Source File field displays the name of the source file on which the definition of the target transform is based.

Source Field

The Source Field field displays the source field to which the target column is mapped. This is the field against which the operation will be applied. The source field can also be another target column, as long as it precedes the column in the transform’s column list.

If Null Return

Use the If Null Return field to enter a value to replace the source field data when that data is null. You can also enter quoted strings, numeric values, fields from the source file, or target columns from the associated transform defined before the column.



If Not Null Return

Use the If Not Null Return field to enter a value to replace the source field data when that data is not null. You can also enter quoted strings, numeric values, fields from the source file, and target columns from the associated transform defined before the column.

Description

Use the Description field to enter a description for the target column.



The String Concatenation (DOCA) Form

Detail from the DOTC form to use this form to define the specifications for a concatenation operation. All arguments must be single-valued source data elements or literals. You can also enter a description of the target column.

Figure 21: String Concatenation (DOCA) Form



Noteworthy Fields on the DOCA Form

The fields described in this section are particularly important when defining a a string concatenation operation.

Target ID


Target Transform


Target Column


Source File


Source Field


Concatenation Operand List

Use the Concatenation Operand List field to enter quoted strings, single-valued string type fields from the source file, or single-valued string type target columns from the associated transform defined before the column.



Description




The Substring Selection (DOSS) Form

Detail from the DOTC form to use this form to define the specifications for a substring operation. You can also enter a description of the target column.

Figure 22: Substring Selection (DOSS) Form



Noteworthy Fields on the DOSS Form

The fields described in this section are particularly important when defining a a substring operation.

Target ID


Target Transform


Target Column


Source File


Source Field

The Source Field field displays the source field to which the target column is mapped. This is the field against which the operation will be applied. The source field can also be another target column, as long as it precedes this column in the transform’s column list.

Starting Position

Use the Starting Position field to enter or view the starting position of the substring for the source field.



Substring Length

Use the Substring Length field to enter the length of the string to be returned from the substring of the source field, based on the Starting Position field.

Description




The Validation Code LookUp (DOVC) Form

Detail from the DOTC form to use this form to define the specifications for a validation code lookup operation. This form allows you to enter a validation code table name that corresponds to the value in the source field and choose which field to return from the validation code. You can also enter a description of the target column.

Figure 23: Validation Code LookUp (DOVC) Form



Noteworthy Fields on the DOVC Form

The fields described in this section are particularly important when defining a a validation code lookup operation.

Target ID


Target Transform


Target Column


Source File


Source Field


Validation Code Name

Use the Validation Code Name field to enter the validation code table name that corresponds to the value in the source field. You can select a validation code from any installed application validation code table. This operation will match the value in the input field with the appropriate code from the validation code table specified in the Validation Code Name field.



Return Field Name

Use the Return Field Name field to select which field you want to return from the validation code for this column. This identifies the information associated with the matching validation code value that the transform will return as the value of the operation. Typically, you will enter description to return the text description of the validation code value.

Description




The Expression Entry (DOEE) Form

Detail from the DOTC form to use this form to define the specifications for an operation manually. You can also enter a description of the target column.

When you save from the DOEE form, the expression is run against its source file to verify the syntax. If the expression fails, then the Export (Y/N) field on the Transform Columns (DOTC) form is set to “No” for the corresponding column to prevent an error from being encountered when the transform is run.

Figure 24: Expression Entry (DOEE) Form



Noteworthy Fields on the DOEE Form

The fields described in this section are particularly important when defining a an expression entry operation.

Target ID


Target Transform


Target Column


Source File


Source Field

The Source Field field displays the source field selected for the column.

Expression

Use the Expression field to enter the expression to use to return a value for the column.

Note: If you use a literal string argument, the argument must be quoted.



The following operation names and arguments are available.

Field Extract. SubFieldExtract(FieldName,“Delimiter”,StartFieldNum,NumFieldsTo Extract). For example: SubFieldExtract(INSTITUTIONS_ATTEND_ID,"*",1,1)

Multivalue Aggregation. ListAggregate(FieldName,“Function”). For example:

ListAggregate(SEC.WITHDRAWN.STUDENTS,"COUNT")

Multivalue Position. ListElementByPos(FieldName,PositionInMultiValue). For example:

ListElementByPos(APP.RECRUIT.PURPOSES,3)

Pointer Reference. PointerRef(SourceFieldName,“TargetFileName”,“TargetFieldName”). For example:

PointerRef(INV.TERM,"TERMS","TERM.DESC")

Null Replacement. ReplaceNull(SourceFieldName,AlternateSourceField/String). For example:

ReplaceNull(ARD.REVERSAL.AMT,0)

If Null. IfNull(SourceFieldName,NullReturnField/String,NotNullReturnField/String). For example:

IfNull (INV.SPONSOR.PAYMENT, "MC","SB")

String Concatenation. ConCat(SourceFieldName/String1,SourceFieldName/String2,...SourceFieldName/StringX). For example:

ConCat(STPR_STUDENT,"*",STPR_ACAD_LEVEL)

Substring. SubString(SourceFieldName,StartPosition,Length). For example:

SubString (CITY, 1, 10)

Validation Code LookUp. AssocLookupByPtr(“ValcodeName”,“TargetFile”,“LookupField”, LookupValue,“ReturnField”). For example:

AssocLookupByPtr("GRADUATION.TYPES", "CORE.VALCODES","VAL.INTERNAL.CODE", INSTA.GRAD.TYPE,"VAL.EXTERNAL.REPRESENTATION")

Note: Operations can be nested; operations can be used anywhere that a Field Name is referenced in these syntax descriptions.

Note: You can use only fields from the source file in an expression; you cannot use previously defined fields of the transform.



Description


Procedure for Defining Target Transforms

See Procedure for Defining a Target beginning on page 157.




Setting Up Targets

Defining and Creating SQL Views

In This ChapterThis chapter describes how to define, create, and maintain SQL views for a target. SQL views are an optional feature that you can utilize, and they can be particularly helpful for your reporting users. Views simplify accessing information in your operational data store and provide predefined queries that can be shared by all reporting users.

You also have the option to indicate if an SQL view should be created as a materialized view. This means that the SQL statement that defines the view is run and all the resulting data rows are stored in a database table. This may significantly improve the response time for queries using the SQL view. However, this also means that the data content of the view must be refreshed whenever the tables providing data for the view are updated. You can either directly update the data content for a materialized view, or set up a materialized view to refresh automatically whenever any of the transforms on which it depends are updated.

Setting up SQL views is a two-step process. You must first define the view, then create the view on the ODS target database. This allows you to control when views are created on the ODS target database.



Topic Page

Forms Used 144

Defining SQL Views 145

Defining the SQL Select Statement for an SQL View 147

Procedure for Defining SQL Views 152

Creating SQL Views 153

Procedure for Creating SQL Views Using the DOTV Form 154


Setting Up Targets: Defining and Creating SQL Views



Form Purpose

DataOrch Target Views (DOTV) Define, create, and maintain SQL views for a target, and view information about the statuses of these SQL views.

DataOrch View Spec (DOVS) Modify the select statement used to create an SQL view, modify the column documentation for the SQL view, or define or modify the materialized view parameters for the SQL view. Test the SQL view select statement on the ODS target database.


Defining SQL Views

Defining SQL ViewsDetail on the SQL Views field from the DataOrch (DOTA) form. This allows you to access the DataOrch Target Views (DOTV) form where you can view or define the SQL views associated to a target.

When you detail from the DOTA form, you can only define SQL views; you cannot create them, so the following fields are not accessible:

View Creation Selection

Create on Save (Y/N)

Figure 25: DataOrch Target Views (DOTV) Form

Status information is available for the SQL view, including when the SQL view was last modified and the last date it was successfully created on the ODS target database.

Note: When you are ready to create views for a target, you have two options for creating them. For more details, see Creating SQL Views on page 153.



Noteworthy Fields on the DOTV Form

The fields described in this section are particularly important for defining SQL views associated to a target.

SQL View Name

In the SQL View Name field, enter a name for the SQL view that you want to associate to the target. The name you enter here will be used as the name of the corresponding SQL view when the view is created on the ODS target database.

Detail from this field to access the DataOrch View Spec (DOVS) form if you want to define or modify the SQL select statement, column documentation, or materialized view parameters for an SQL view.

Modify Date

The Modify Date field shows the date of the last modification of the SQL view.

Success Date

The Success Date field shows the date of the last successful creation of the view on the ODS target database. If the SQL view is modified, or an error is encountered when creating it on the target database, the Success Date will be reset to a null value.

Material View

The Material View field displays “Yes” if the SQL select statement defined on the DataOrch View Specs (DOVS) form will produce a table on the target database. If this field is blank, the SQL select statement will produce a view on the target database. You can detail on the SQL View Name field to change this setting.

Note: Deleting a name from the SQL View Name list does not automatically delete the corresponding SQL view on the target database. This must be done manually by the system administrator.


Defining SQL Views

Defining the SQL Select Statement for an SQL View

To view and define the SQL statement, column documentation, or the materialized view parameters for an SQL view, detail on an SQL View Name from the DOTV form to the DataOrch View Spec (DOVS) form.

Use the DOVS form to perform the following functions:

View or modify the select statement used to create the SQL view on the ODS target database.

View or modify the description.

View or modify the column documentation for the SQL view.

View or set the option indicating that a view column or set of view columns make up the primary key of the SQL view.

Test the SQL view select statement on the ODS target database.

Set the option indicating the view is a materialized view.

View or modify the list of ODS target transforms on which a materialized view depends.

Note: This information is used to create the primary key on the database table for a materialized view. For SQL views that are not marked as materialized, this is informational only.



Figure 26: DataOrch View Spec (DOVS)

Status information is also available for the SQL view, including when the SQL view was last modified and the last date it was successfully created on the ODS target database.

Note: There can be two versions of an SQL view, one with Oracle-specific SQL syntax and one with SQL Server syntax. In the header of the DOVS form, the version is displayed in the Database Type field. This is set up in the Database Type field on the DataOrch Target (DOTA) form for the associated target.


Defining SQL Views

Noteworthy Fields on the DOVS Form

The fields described in this section are particularly important for defining SQL select statements and column documentation for an SQL view.

SQL Select Statement

Use the SQL Select Statement field to enter an SQL select statement used to create the SQL view on the ODS target database. For example:

SELECT ID, LAST_NAME, BIRTH_DATE FROM PERSON WHERE LAST_NAME EQ “Smith”

The &schema& keyword is also available for use in the SQL Select Statement. For example:

&schema&.TABLE_NAME will be replaced with the schema name (dbo for SQL Server) of the target database when the view is created.

Key

In the Key field, enter Yes to identify a column as being part of a materialized view’s key. By entering “Yes”, this column will be included in the primary key for the table created for the materialized view. If more than one column is marked “Yes”, then a multi-column key will be created on this table, and the order of the columns in the key will be in the same order in which they are listed on the DOVS form.

Description

In the Description field, enter a description for the SQL view.

Note: SELECT must be the first word of the SQL Select Statement.

Note: The set of columns identified as the primary key must contain unique values, or the creation of the key will fail when the materialized view is refreshed.



SQL View Columns

In the SQL View Columns field, enter the column names in the SQL select statement. For materialized SQL views, the SQL View Column entries that have the Key field set to “Yes” are used to create the primary key for the table on the database. These field names must exist in the SQL view or the creation of the materialized SQL view will fail. Otherwise, the column names entered here are for your institution's documentation purposes only. No column names will be added to the SQL select statement. Also, the SQL select statement will not be checked to verify the column names exist in the select statement.

Description

Enter a description for the SQL view column.

Create Materialized View (Y/N)

In the Create Materialized View (Y/N) field, enter “Yes” to create a table on the target database from the SQL select statement. Creating a table from the SQL select statement allows you to create indexes and foreign keys on the resulting target table. Enter “No” if you want to create a view on the target database from the SQL select statement.

Dependent Transforms

In the Dependent Transforms field, enter the transforms on which this materialized view is dependent.

Dependent transforms are required when defining a materialized view. If the SQL select statement references a transform, then enter the transform’s name in this list.

Note: When you run a refresh using the DataOrch Refresh (DORE) form, if the refresh processes a transform on which a materialized view is dependent, then the materialized view is automatically updated. This ensures that the most recent data from the transform is included in the materialized view.


Defining SQL Views

Test SQL View Now

Detail on the Test SQL View Now field to test the SQL view on the ODS target database. The select statement will be validated by creating a temporary view named DATAORCH_TEST_VIEW on the ODS target database, which will be deleted after the view is created.

Note: A select statement must be entered in the SQL Select Statement field in order to attempt to test the SQL view.



Procedure for Defining SQL Views

Step 1. Access the DOTV form by detailing on the SQL Views field from the DataOrch (DOTA) form.

Step 2. Define the SQL View Names on the DOTV form. For more details, see online help.

Step 3. If you want to define or modify the select statement, column documentation, and/or materialized view parameters for the SQL view, detail on the SQL View Name field to access the DataOrch View Spec (DOVS) form.

a. Enter the SQL select statement that defines the view and the column documentation on the DOVS form. For more details, see online help.

b. If you want to create a materialized view, enter Yes in the Create Materialized View (Y/N) field, and then enter the transforms on which the materialized view is dependent in the Dependent Transforms field. For more details, see online help.

c. If you want to test the SQL view on the ODS target database, detail on the Test SQL View Now field.

d. When you are finished viewing the results of the test, save from the DOVS form.

Step 4. Save from DOTV form to return to the DOTA form.


Creating SQL Views

Creating SQL ViewsAfter you have defined your SQL views for a target and saved from the DOTV form, you must save from the DOTA form before you can create your SQL views. This saves all of your SQL view definitions for the target.

When you are ready to create the SQL views on the ODS target database, you have two options for creating the views:

Use the DataOrch Refresh (DORE) form. Creating views on this form allows you an easy way to create all the views by simply running a refresh of the target. For more details, refer to Defining and Running an ODS Refresh beginning on page 165.

Access the DOTV form from the Quick Access field. If you access the DOTV form from the Quick Access field (not by detailing from the DOTA form), you can create the SQL views. This form allows you more flexibility in controlling which views you want to create on your ODS target database.

Note: In addition to these two options for creating views, any SQL view defined as a materialized view will be refreshed when any of the target transforms defined in the Dependent Transforms list on the DOVS form is refreshed.



Procedure for Creating SQL Views Using the DOTV Form

Use the following steps to create SQL views using the DOTV form.

Step 1. Access the DOTV form by entering DOTV in the Quick Access field. At the DataOrch Target LookUp prompt, enter the target for which you want to create SQL views.

The following fields are now accessible:

View Creation Selection

Create on Save (Y/N)

Figure 27: DataOrch Target Views (DOTV) Form


Creating SQL Views

Step 2. (Optional) Choose an option from the View Creation Selection drop-down list. The following options are available:

All Views. This option updates the Create on Save (Y/N) field to Yes for all SQL views.

Unsuccessful Views. This option updates the Create on Save (Y/N) field to Yes for all SQL views without a corresponding Success Date.

Deselect Views. This option updates the Create on Save (Y/N) field to No for all SQL views.

Step 3. Alternately, you can use the Create on Save (Y/N) field to select views to create. Enter Yes to create one or more SQL views on the ODS target database when saving from this form; otherwise, enter No.

Step 4. Save your entries on the DOTV form.

When you save from the DOTV form, SQL views are created if their associated Create on Save (Y/N) field is set to Yes, and if no errors are encountered.

Note: If an SQL view does not receive errors during creation, then its corresponding Success Date field will be updated to the current date. Otherwise, the Success Date field for the SQL view will be cleared.




Setting Up Targets


In This ChapterThis chapter describes the procedure for defining a target.

Table 24 lists the topics covered in this Chapter.


Topic Page

Procedure for Defining a Target 158


Setting Up Targets: Procedure for Defining a Target


Perform the steps below to define or modify a target.

Step 1. Access the DataOrch Target (DOTA) form.

Step 2. Enter the information for this target. See online help for further information.

Step 3. (Required) On the DOTA form, detail on the Temporary File Path field to access the Additional Target Parameters (DOTP) form.

a. (Target Windows server only). Enter the target drive name of the temporary file path.

b. Enter the directory path on the ODS target database server where the temporary files, created by the refresh, will be stored.

ALERT! Datatel highly recommends that you secure access to the DOTA form.

Note: Do not use an existing Colleague database as a target database for an operational data store. If you try to do this, you will receive an error message when you run the refresh.

ALERT! The target database must be refreshed by only one target. If two targets refresh the same target database, the operational data store status information will be inaccurate, the data written to that database can be corrupted, and the refresh may encounter errors.

ALERT! This directory must have sufficient space allocated for these temporary files. The space needed for temporary files can be significant depending on how you define the target. Datatel recommends that you monitor the space usage in this directory as your refresh runs. Be sure that access to this directory is in accordance with the online help for the Temporary File Path field on DOTP. Also, the temporary files in this directory may contain data that might be considered sensitive at your institution. Access to the directory should be restricted accordingly.


In This Chapter

c. Save and exit from the DOTP form.

Step 4. On the DOTA form, to add a source transform to this target, enter the file name in the Source Transforms table. Source transforms allow you to specify the data to move to an ODS target database by selecting a set of fields and/or computed columns from a single Colleague file.

File names entered here can be refreshed to an ODS target database by using the DataOrch Refresh (DORE) form. If you specify a file containing stored computed columns in this field, you must run the Update Stored Computed Column (USCC) process to ensure that the stored computed columns are refreshed before the DORE process is run. For more information on using stored computed columns with the DataOrchestrator ODS, see Support Solution 6877. (If the target also contains the STUDENT.TERMS.CC file, the Update GPA SCC Flags [UGSF] process should be run before the USCC process.)

Step 5. If you are adding a source transform, Colleague automatically opens the DataOrch Source Transform (DOST) form and can specify which fields and/or computed columns of the file to include in this source transform. To view or modify an existing transform, detail on the specific source transform to access the DOST form. See online help for further information.

a. (Optional) On the DOST form, detail on the Filters/File Suite Instances field to access the Filter Criteria (DOFC) form in order to enter, view, or modify filter criteria for this source transform. Colleague uses this criteria to specify what records from the input file will be processed by the source transform when it is run. See online help for further information. Save and exit from the DOFC form to return to the DOST form.

b. (Optional) On the DOST form, if you want to see the set of tables that will be created on the ODS target database, detail on the View Generated Transforms field to access the Generated Target Transforms (DOGT) form to view the target transforms generated from this source transform. Save and exit from the DOGT form to return to the DOST form.

c. Save and exit from the DOST form.

Step 6. On the DOTA form, to add a target transform to this target, enter a name for the transform in the Target Transforms table. The target transform name you

Note: Deleting a file from the Source Transforms table does not automatically delete the corresponding table on the target database. This must be done manually by the system administrator.



enter will be used as the name of the corresponding target table when this transform is refreshed using the DORE process.

Target transforms allow you to specify the format of the data on the target database, as well as any data transformation operation to run when the data is moved.

Step 7. If you are adding a target transform, Colleague automatically opens the DataOrch Target Transform (DOTT) form to define the transform. To view or modify an existing transform, detail on the specific target transform to access the DOTT form. See online help for further information.

a. On the DOTT form, detail on the Column Definitions field to access the Transform Columns (DOTC) form. Use the DOTC form to add, view, or modify the output columns for the transform. Each column is defined by an input field (and an optional transformation operation) that produces the output value for the corresponding target database column. See online help for further information.

If you specify a stored computed column as an input field, you must run the Update Stored Computed Column (USCC) process to ensure that the stored computed columns are refreshed before the DORE process is run. For more information on using stored computed columns with the DataOrchestrator ODS, see Support Solution 6877. (If you specify a stored computed column from the STUDENT.TERMS.CC file, the Update GPA SCC Flags [UGSF] process should be run before the USCC process.)

b. (Optional) On the DOTC form, detail on a target column name to access the Column Properties (DOPR) form in order to specify properties for this column. See online help for further information. If you detailed, save and exit from the DOPR form to return to the DOTC form.

c. (Optional) On the DOTC form, you can choose a transformation operation to perform in order to generate the output value for the target database column. To do this, select an operation from the Operation field drop-down list. The following options are available:

Field Extract. Details to the Field Extract (DOFE) form.

Multivalue Aggregation. Details to the Multivalue Operation (DOMV) form.

Multivalue Position. Details to the Multivalue Operation (DOMV) form.

Note: Deleting a name from the Target Transforms table does not automatically delete the corresponding table on the target database. This must be done manually by the system administrator.


In This Chapter

Pointer Reference. Details to the Pointer Reference (DOFR) form.

Null Replacement. Details to the Null Value Replacement (DONV) form.

Null Test. Details to the Null Value Replacement (DONV) form.

String Concatenation. Details to the String Concatenation (DOCA) form.

Substring. Details to the Substring Selection (DOSS) form.

Validation Code LookUp. Details to the Validation Code LookUp (DOVC) form.

Expression Entry. Details to the Expression Entry (DOEE) form.

See online help for further information on these forms.

d. If you detailed from the Operation field on DOTC to use any of the forms mentioned in the list above, save and exit from the form to return to the DOTC form.

e. Save and exit from the DOTC form to return to the DOTT form.

f. (Optional) On the DOTT form, detail on the Filters/File Suite Instances field to access the DOFC form in order to enter, view, or modify filter criteria for this target transform. Colleague uses this criteria to specify what records from the source file are processed by the target transform when it is run. See online help for further information. Save and exit from the DOFC form to return to the DOTT form.

Step 8. If you have finished defining target transforms, save and exit from the DOTT form to return to the DOTA form.

Step 9. (Optional) On the DOTA form, detail on the SQL View field to define SQL views.

a. Define the SQL View Names for the views on the DataOrch Target Views (DOTV) form.

b. Detail on the SQL View Name field to define the select statement and column documentation for a view on the DataOrch View Spec (DOVS) form.

c. Optionally, on the DOVS form, you can indicate that the SQL view should be created as a materialized view by entering Yes in the Create Materialized View field, and then adding all transforms referenced in the SQL view definition to the Dependent Transforms list. You should also enter Yes in the Key field for the view columns defining the primary key



for the materialized view, as this will greatly improve the performance of queries against the materialized view.

d. Optionally, you can detail on the Test SQL View Now field to test the SQL view on the ODS target database, then save to return to the DOVS form.

e. Save and exit from both the DOVS and DOTV forms

Step 10. If you have finished defining the target, save and exit from the DOTA form.

Step 11. To export data to the target database, run the DataOrch Refresh (DORE) process. If you want, you can also create all SQL views for the target when running the refresh. For more information, see Defining and Running an ODS Refresh beginning on page 165.

Alternately, you can create the SQL views without running a refresh by accessing the DOTV form from the Quick Access menu. For more information, see Procedure for Creating SQL Views Using the DOTV Form on page 154.



Refreshing ODS Data

Refreshing ODS Data

Defining and Running an ODS Refresh

In This ChapterThis chapter describes how to define the parameters that control the way the DataOrchestrator ODS refreshes a target database with information from Colleague source files.


Forms UsedTable 26 shows the forms used in this chapter.


Topic Page

Forms Used 165

Defining Settings for Refreshing Data 166

Defining Additional Parameters for the Refresh 174

Procedure for Defining a Refresh 177

Table 26: Forms Used to Set Up Refreshes for ODS Target Databases

Form Purpose

DataOrch Refresh (DORE) Specify or update settings that control refreshes of a target database. Run a refresh that updates the specified target database.

Additional Refresh Parameters (DORP)

Enter additional parameters for a refresh.


Refreshing ODS Data: Defining and Running an ODS Refresh

Defining Settings for Refreshing DataUse the DataOrch Refresh (DORE) form to define the settings to be used when doing an initial export, or when updating data from the Colleague source environment to an ODS target database. You can also use the DORE form to run an export of data to this ODS target database and to view the statistics that resulted from the previous export. Each Refresh ID that you set up is associated with one specific target.

When running a refresh, you can specify a default Update Type (Full or Incremental). A Full update processes all records of the source file for the transform. An Incremental update processes only those records of the source file that either:

Changed since the last refresh if the transform has an Incremental Date Field, or

Match the Incremental Filter Criteria.

If you choose Incremental, only the non-computed column data elements in the source file will be checked for updates since the last refresh. This means that if the transform references data elements from other Colleague files (either directly or through the use of computed columns), no dependency checking is done on these elements. Updates to their values will not automatically trigger the inclusion of these input elements in the Incremental update.

You can also choose to include specific source or target transforms in the refresh, or you can include all source and/or target transforms.

If you enter specific transforms (in the Include Source Transforms and/or Include Target Transforms tables), you can choose to override the default Update Type for a transform to either Full or Incremental.

The DataOrch Target (DOTA), DataOrch Source Transform (DOST), and Transform Columns (DOTC) forms allow you to select which fields and computed columns will be refreshed for each transform. The fewer fields (and particularly the fewer computed columns) that are referenced for each transform, the better the refresh performance will be. However, if any of these output fields change for a transform, then a full refresh will be done for the transform, and the target table will be dropped and recreated. Depending on the transform, this could significantly impact the time needed for the subsequent refresh.



You can also specify whether, when this refresh is run, you want to create all of the SQL views associated to the target.

To improve performance of the refresh, you can specify the number of separate concurrent Envision processes to use for a refresh. Using more concurrent Envision processes to run a refresh will result in better performance for the refresh, but will use resources on the application server. This may also impact performance of other Colleague processes.

In addition, you can specify whether you want a refresh to run immediately, or choose just to save the entered settings for this target.

You can view the refresh status on the DORE form. The following statuses may be displayed in the header:

InProc. A transform is currently being refreshed.

Success. The transform was successfully refreshed without any errors.

Warning. The transform or refresh received errors during processing.

Fail. The transform or refresh received fatal errors during processing.

You can also detail from the DORE form to access the Additional Refresh Parameters (DORP) form.

Note: When you run a refresh, if the refresh processes a transform on which a materialized view is dependent, then the materialized view is automatically updated. This ensures that the most recent data from the transform is included in the materialized view.

Technical Tip: When the refresh of an ODS target database takes place, all information stored in Colleague about the target, the transforms and SQL views being refreshed, and the refresh are written to the target database. This information can be queried on the target database using SQL-based tools.

Technical Tip: When you run a full refresh (after the initial refresh), each ODS target database table is cleared only after the data extract and file transfer occurs for the corresponding transform, so that users are minimally impacted.

Note: A maximum of five transforms are allowed to fail before the refresh will be stopped automatically.



Figure 28: DataOrch Refresh (DORE) Form

When you first enter the DORE form, Colleague displays the DataOrch Refresh LookUp prompt where you can enter one of the following:

An ID for a refresh that you want to add and set up.

An ID for an existing refresh for which you want to run a refresh and/or modify the settings.



Noteworthy Fields on the DORE Form

The fields described in this section are particularly important when defining the settings for an ODS refresh.

Target ID

Use the Target ID field to enter one of the following for a new Refresh ID:

The ID for a target that you want to create and set up.

The ID for an existing target that you want to associate with this new Refresh ID.

Also, use this field to view or modify the target information that is associated with an existing Refresh ID. To do this, detail to the DataOrch Target (DOTA) form.

For more information on creating or modifying a target, see Defining a Target beginning on page 67.

RDAS Listener

Use the RDAS Listener field to enter the name of the DMI Listener with the RDAS role on the ODS target database server. This Listener will be used to refresh data from the source Colleague environment to the ODS target database.

DMI Listener Host Name

The DMI Listener Host Name field displays the TCP/IP address or DNS alias name of the computer where the DMI Listener is installed. This is the DMI Listener with the RDAS role on the ODS target database server.

Port

The Port field displays the port number associated to the DMI Listener.

Note: After you specify a Target ID for this Refresh ID and then save, you cannot change the Target ID.



Update Type

When running a refresh, you can specify whether all records of the input files for the listed transforms should be processed (Full), or only those records of the input files that have changed since the last refresh (Incremental).

Transform Selection Option

Use the Transform Selection Option field to select one of the following options for this refresh:

All source transforms. All source transforms from the associated target will be refreshed, using the default in the Update Type field. In addition, you can enter specific target transforms to be refreshed in the Include Target Transforms table (and override the default Update Type).

All target transforms. All target transforms from the associated target will be refreshed, using the default in the Update Type field. You can also enter specific source transforms to be refreshed in the Include Source Transforms table (and override the default Update Type).

All transforms. All source and target transforms from the associated target will be refreshed, using the default defined in the Update Type field.

Specific transforms. You can enter specific source and/or target transforms to be refreshed in the Include Source Transforms and/or Include Target Transforms tables (and override the default Update Types).

Note: Choosing this option clears previous entries in the Include Source Transforms table and the associated Override Update Type table. The tables become inquiry only.

Note: Choosing this option clears previous entries in the Include Target Transforms table and the associated Override Update Type table. The tables become inquiry only.

Note: Choosing this option clears previous entries in the Include Source Transforms and the Include Target Transforms tables, as well as the associated Override Update Type tables. The tables become inquiry only.



Include Source Transforms/Include Target Transforms

If you want to include only specific transforms in the refresh, use the Include Source Transforms and/or the Include Target Transforms tables to list those transforms. Use LookUp to select one or more transforms to process from a list of transforms associated with this target.

Override Update Type

If you enter transforms in the Include Source Transforms and/or Include Target Transform tables, you can also choose to override the default Update Type for a specific transform and control whether all records will be updated or only those records changed since the last update.

However, if a target transform is set to Refresh as Full Only on the DataOrch Target Transform (DOTT) form, the target transform will be refreshed as a full update regardless of the setting in this field.

Check Connectivity

To test connectivity, detail on the Check Connectivity field to log in to the specified target database in order to verify the connection.

Create SQL Views (Y/N)

Use the Create SQL Views (Y/N) field to indicate whether or not you want to create all of the SQL views associated to the target, including any materialized views. Enter Yes to create the SQL views; otherwise, enter No.

Note: To test connectivity, the DataOrch Target (DOTA) form must already be set up with the DMI Listener Host Name, DMI Listener Port, and the associated target information. If this information has not been entered, the following error is displayed: Incomplete Setup.

Note: This setting allows you an easy way to create all the views for a target when you run the refresh. However, you do not need to create all the views each time you refresh a target if none of the definitions for the views have changed. You also have the option of using the DOTV form instead to create views. The DOTV form allows you more flexibility in controlling which views you want to create on your ODS target database. For more information, see Defining and Creating SQL Views beginning on page 143.



Temporary File Path

Detail on the Temporary File Path field to access the Additional Refresh Parameters (DORP) form to enter the source drive name and temporary file path for this refresh.

Run Stored Procedure

In the Run Stored Procedure field, enter Yes to run the stored procedure for this refresh ID.

After the refresh runs, indicate whether you want to run the stored procedure that was created for this refresh ID. Enter Yes to run the stored procedure; otherwise, enter No.

The stored procedure has access to the status information for the DataOrchestrator ODS, which allows you to create a stored procedure that contains more complex processing. The status information for the entire refresh is stored in the DO_REFRESH table, and the refresh status for individual transforms is stored in the DO_TARGET_SPEC table on the target database.

Run Refresh (Y/N)

Use the Run Refresh (Y/N) field to indicate whether or not you want to refresh the selected target database when you save your entries on the DORE form. Enter Yes to run a refresh, or enter No if you want only to save the refresh settings.

Number of Concurrent Threads

Specify the number of concurrent Envision processes to use when running transforms to export data to the target database. Using more threads will usually result in better performance for the refresh, but will use resources on the Colleague database server. This may impact performance of other Colleague processes.

Note: You can create a stored procedure in the target database that can be run after a refresh is complete. This stored procedure must be named for the refresh ID, with _SP appended to the end. For example, if the refresh ID is D01_REFRESH, the stored procedure must be named D01_REFRESH_SP.



ALERT! Using multiple threads for your refresh can significantly affect the system resources available to your Colleague environment. Datatel recommends that you carefully monitor the system resource usage of your Colleague database server while running your ODS refresh and adjust the scheduling of your refresh and the number of threads you assign it accordingly.

Note: When more threads are used, not only are more system resources used as well, but also more UniData licenses, and both of these can affect the performance of other processes.



Defining Additional Parameters for the Refresh

Optionally, you can detail from the DataOrch Refresh (DORE) form by using the Temporary File Path field to go to the Additional Refresh Parameters (DORP) form. The DORP form allows you to define a Source Drive Name and Source Temporary File Path that overrides the corresponding settings on the DataOrch Parameters (DOPA) form for this refresh.

Figure 29: Additional Refresh Parameters (DORP) Form

Technical Tip: The reason you may want to override the corresponding settings on DOPA is because this allows you to limit access to the source temporary file path for this refresh.



Noteworthy Fields on the DORP Form

The fields described in this section are particularly important when defining additional parameters for a refresh.

Refresh ID

The Refresh ID field displays the ID of the refresh associated with the parameters.

Source Drive Name

If you use a Windows source environment and use a source database other than SQL Server, use the Source Drive Name field to enter the source drive name of the temporary file path. This value specifies which drive on the source Colleague database server will store the temporary files created by the refresh.

Source Temporary File Path

Use the Source Temporary File Path field to enter the directory path on the source Colleague database server where the temporary files, created by the refresh, will be stored. Enter each element in the path on a separate line and omit any slashes (/).


Note: If you are using a SQL Server source database environment, leave this field blank.

Note: This value will override the default Source Drive Name on the DataOrch Parameters (DOPA) form.

Note: If you are using a Windows SQL Server source environment, leave this field blank.




UNIX UniData. For the directory where the files are being written: • The user logged in to Colleague must have read/write permissions. • The user who starts the source DMI_DAS must have read/write

permissions.

Windows UniData. For the directory where the files are being written: • The user logged in to Colleague must have read/write permissions. • The SYSTEM user must have read/write permissions.

UNIX Oracle. For the directory where the files are being written: • Oracle needs read/write permissions. • The user who starts the source DMI_DAS must have read/write

permissions.


Note: If the Listener is ever started manually (instead of through SA Valet as a Windows service), then the user who starts the Listener must also have read/write permissions.

Note: The path you enter in this field will override the default Source Temporary File Path on the DataOrch Parameters (DOPA) form.



Procedure for Defining a Refresh

Perform the steps below to enter settings for a refresh for a specific target, and to run the refresh, if desired.

Step 1. Access the DataOrch Refresh (DORE) form.

Step 2. At the DataOrch Refresh LookUp prompt, either add a new Refresh ID or enter an existing one.

Step 3. If you entered an existing Refresh ID, the Target ID is displayed and cannot be changed. If you are adding a new Refresh ID, enter the ID of a new or existing target in the Target ID field.

If you add a new Target ID, you will automatically detail to the DataOrch Target (DOTA) form so that you can define the new target. For an existing target, you can manually detail to the DOTA form to view or update the target information. For information on defining or updating a target, see Defining a Target on page 67.

For information about the fields on the DOTA form, see online help. When you are finished on the DOTA form, save to return to the DORE form.

Step 4. On the DORE form, enter or modify the settings for refreshing this target. For more information on these fields, see online help.

ALERT! You should ensure that the target database to which you are exporting Colleague data is adequately secured using the native security controls for your database platform. Some files and fields may contain data that might be considered sensitive in your institution. Access to the target database containing this data should be restricted accordingly. Datatel also highly recommends that you secure access to the DORE form. This form provides a user with the ability to export any data in Colleague to an external database.

Note: After you specify a Target ID for this Refresh ID and then save, you cannot change this association.



Step 5. (Optional) To test connectivity, detail on the Check Connectivity field to log in to the specified target database in order to verify the connection. When you are finished checking connectivity, save to return to the DORE form.

Step 6. (Optional) On the DORE form, you can detail on the Temporary File Path field to access the Additional Refresh Parameters (DORP) form to override the source drive name and temporary file path defined on the DOPA form for this refresh. When you are finished entering parameters, save and exit from the DORP form to return to the DORE form.

Step 7. (Optional) In the Create SQL Views (Y/N) field, indicate whether you want to create all of the SQL views associated to the target. Enter Yes to create the SQL views; otherwise, enter No.

Step 8. (Optional) After the refresh runs, indicate whether you want to run the stored procedure that was created for this refresh ID. Enter Yes in the Run Stored Procedure field to run the stored procedure; otherwise, enter No.

Step 9. In the Run Refresh (Y/N) field, enter Yes if you want to refresh the selected target when you save your entries on the DORE form. Enter No if you want only to save the refresh settings.

Step 10. In the Number of Concurrent Threads field, specify the number of threads you want to use for exporting the files to the target. Remember that using more concurrent Envision processes will usually result in better performance for the refresh, but will use resources on the application server. This may impact performance of other Colleague processes.

Step 11. Save from the DORE form.

ALERT! This directory must have sufficient space allocated for these temporary files. The space needed for temporary files can be significant depending on how you define the target. Datatel recommends that you monitor the space usage in this directory as your refresh runs. Be sure that access to this directory is in accordance with the online help for the Temporary File Path field on the DORP form. Also, the temporary files in this directory may contain data that might be considered sensitive at your institution. Access to the directory should be restricted accordingly.



Step 12. If you are running a refresh, complete the Peripheral Settings and Process Handler forms according to your preferences.

If you entered “Yes” in the Run Refresh (Y/N) field, the DataOrchestrator ODS refreshes the target database and reports errors on the DataOrch Error Analysis report. For more information on this report, see Viewing Errors for a Refresh on page 193.




Setting Up Targets


In This Chapter

This chapter describes how to activate and calculate stored computed columns for any ODS target database that contains stored computed columns.


Note: To use the iStrategy data warehouse solution with the DataOrchestrator ODS, you must run the processes in this chapter. If your institution does not use any targets containing stored computed columns, you can skip this chapter. Stored computed column files are indicated by having .CC as the last three characters of the file name. Stored computed columns are delivered inactive and must be activated.


Topic Page

Forms Used 182

Calculating Stored Computed Columns 183

Procedure for Activating Stored Computed Columns 184

Procedure for Calculating Stored Computed Columns 186

Procedure to Update Flags in the STUDENT.TERMS.CC File 188


Setting Up Targets: Calculating Stored Computed Columns

Forms UsedTable 28 lists and describes the forms used in this chapter.


Form Purpose

Define Stored Computed Column (DSCC)

Activate stored computed columns.

Update Stored Computed Column (USCC)

Calculate stored computed columns for an ODS target database.

Update GPA SCC Flags (UGSF) Update flags in the STUDENT.TERMS.CC file for the Cumulative GPA by Term.

Note: Use the USCC process for the initial calculation of stored computed columns in this file. For subsequent updates, Datatel recommends running the UGSF process before the USCC process.



Calculating Stored Computed ColumnsTo calculate stored computed columns referenced by a target, you first use the Define Stored Computed Column (DSCC) form to activate all stored computed columns, and then use the Update Stored Computed Column (USCC) process to populate the stored computed columns in the source files. No information is available for these stored computed columns until both processes are run against the source files.

You need to activate a given stored computed column on the DSCC form only once. However, Datatel recommends that you run the USCC process every time you run the DataOrch Refresh (DORE) process to export data to an ODS target database containing stored computed columns.

How often you run the USCC process depends on how often your stored computed columns need to be recalculated. You may also consider how frequently reports are run. Based on that you can run the USCC process before running reports to get the most up-to-date data.

Note: If you run the DORE process without running the USCC process, the ODS target database will contain stored computed column data only for the most recent time the USCC process was run.

Technical Tip: If your target contains the STUDENT.TERMS.CC file, use the USCC process for the initial calculation of stored computed columns in this file. For subsequent updates, Datatel recommends that you use the UGSF process (before using the USCC process) to update the STUDENT.TERMS.CC file.

Note: For any ODS target database that contains stored computed columns, Datatel recommends that you set up the process handler to automatically run the USCC process. (If the target also contains the STUDENT.TERMS.CC file, the UGSF process should be set up to run automatically before the USCC process.) This ensures that the stored computed columns will be updated when the ODS target database is updated. Refer to Envision Runtime Administration for further information on how to set up a batch process to run at scheduled intervals.



Procedure for Activating Stored Computed Columns

To activate stored computed columns, follow the steps below. You need to activate a given stored computed column only once.

Step 1. To activate stored computed columns, access the Define Stored Computed Column (DSCC) form in the Envision Tool Kit for the appropriate application. For example, use the Envision Tool Kit for Colleague Student if your institution uses the iStrategy data warehouse solution.

Figure 30: Define Stored Computed Column (DSCC) Form

Step 2. At the Computed Column LookUp prompt, enter the stored computed column that you want to make active.

Note: If you are using Colleague Studio, the functionality of the DSCC form is available using the following path: Entity editor > Attributes > Calculation.



For example, the iStrategy data warehouse solution contains the following stored computed columns that you must make active:

APPLCC.READMIT.FLAG (APPLICATIONS.CC file)

STTRCC.TERM.GPA (STUDENT.TERMS.CC file)

STTRCC.TRGR.CUM.GPA (STUDENT.TERMS.CC file)

Step 3. In the Recalculation Method field, select Batch to make this stored computed column active.

Step 4. Update the form to save your work.

Step 5. At the Computed Column LookUp prompt, enter any other stored computed column that you want to make active, or save your work and exit from the form.



Procedure for Calculating Stored Computed Columns

To calculate stored computed columns, follow the steps below. The following example shows how to calculate stored computed columns if your ODS target database contains the APPLICATIONS.CC and STUDENT.TERMS.CC files.

Step 1. Access the Update Stored Computed Column (USCC) form in the appropriate application. If you are in Colleague Student, for example, the following message is displayed:

Scanning ST application for stored computed columns...

The USCC form then lists the names of all ST files with active stored computed columns, as shown in Figure 31.

Figure 31: Update Stored Computed Column (USCC) Form



Step 2. In the Files to Recalculate field, remove all files from the list except those associated with the ODS target database. For example, the APPLICATIONS.CC and STUDENT.TERMS.CC files are necessary for the iStrategy data warehouse solution.

Figure 32: The USCC Form After Removing All Files Except Needed Stored Computed Columns

Step 3. For the initial update of these files, enter Yes in the Force Recalculation field to calculate all stored computed columns.

For subsequent updates of other files containing stored computed columns, enter Yes in the Force Recalculation field to recalculate all stored computed columns, or enter No to recalculate only those that have changed.

Note: For subsequent updates of the STUDENT.TERMS.CC file, Datatel recommends that you run the UGSF process before the USCC process, and do not force recalculation on this file. This is because, for many Colleague environments, the USCC process may take an extended period to run. For further details on the UGSF process, see Procedure to Update Flags in the STUDENT.TERMS.CC File on page 188.



Step 4. Save your work and exit from the form. For more details on the DSCC and USCC processes, refer to Stored Computed Columns.

Procedure to Update Flags in the STUDENT.TERMS.CC File

Step 1. Access the Update GPA SCC Flags (UGSF) form in Colleague Student.

Figure 33: Update GPA SCC Flags (UGSF) Form

Note: If your ODS target database does not use the STUDENT.TERMS.CC file or you are performing the initial calculation of the STUDENT.TERMS.CC file, you can skip this procedure.



Step 2. In the Update change flags for Cumulative GPA by Term? field, enter Yes to set the change flags in the STUDENT.TERMS.CC file when you save from UGSF.

This allows the correct Cumulative GPA by Term values to be recalculated when you run the USCC process without a forced recalculation of these stored computed columns of all records in the file.

These flags are set on records in the STUDENT.TERMS.CC file in the following case: a student’s GPA changes for a given term (thereby affecting the Cumulative GPA by Term for all subsequent terms). Flags are set on all terms following the changed term.

Step 3. Save your work and exit from the form.







Viewing Errors for a Refresh

In This ChapterThis chapter describes how to run an error analysis report for the errors encountered when exporting data from a Colleague source file to an ODS target database.




Topic Page

Form Used 193

Viewing the Error Analysis Report for a Refresh 194

Procedure for Running the Error Analysis Report 197


Form Purpose

DataOrch Error Analysis (DOEA) View the errors received for transforms after running a refresh.


Viewing Reports and Refresh History: Viewing Errors for a Refresh

Viewing the Error Analysis Report for a Refresh

Use the DataOrch Error Analysis (DOEA) form after running a refresh to view the errors encountered for transforms. The error information displayed is for those data rows of a transform that were not written to the ODS target database.

You can display errors for all transforms associated with a refresh, or you can limit the number of errors displayed by selecting only specific transforms. You can also choose to show a report that contains only summary information about the errors in each transform, or you can include detailed information about each recorded error. In addition, you can choose to sort error rows per transform by error type or by source file record ID.

The number of errors shown for a transform can be affected by the Max Errors Saved field on the DataOrch Parameters (DOPA) form. This field is used to limit the number of errors reported for transforms. Only this number of errors will be included for a specific transform in this report.

Technical Tip: The Error Analysis Report for the refresh includes warning messages for each transform that results in an empty target ODS table. These warnings are included because these transforms do not result in useful information in the operational data store and may indicate either an error in the design of the transform, or that the transform can be omitted from the refresh.

Technical Tip: To help you correct data errors in the source environment, you may want to use the Colleague Data Scanner Utility (CDSU) process. For more information, see Preparing for Release 18, available on the Datatel website.



Figure 34: DataOrch Error Analysis (DOEA) Form

Noteworthy Fields on the DOEA Form

The fields described in this section are important when setting up the error analysis report to run.

Refresh ID

The Refresh ID field displays the ID of the refresh.

Target ID

The Target ID field displays the ID of the associated target.



Status

The Status field displays the status of the refresh.

Last Start Date/Time

The Last Start Date/Time field displays the date and time when the most recent refresh started.

Last End Date/Time

The Last End Date/Time field displays the date and time when the most recent refresh ended.

Limit to Transforms

Use the Limit to Transforms field to enter the transforms associated with this refresh for which you want to view data or refresh errors. You can enter only those transforms from the target that have data or refresh errors.

Rows with Errors

The Rows with Errors field displays the number of rows with errors.

Show Detailed Errors (Y/N)

Use the Show Detailed Errors (Y/N) field to enter Yes to view error summary information and to display each error row for the transforms selected. Enter No to view only error summary information for each transform.

Sort by Error Type (Y/N)

Use the Sort by Error Type (Y/N) field to enter Yes to sort error rows per transform by error type. Enter No to sort error rows per transform by source file record ID.

Note: If you leave this field blank, then all transforms for this refresh that have data or refresh errors will be included.



Procedure for Running the Error Analysis Report

Use the following steps to run the Error Analysis Report.

Step 1. Access the Error Analysis (DOEA) form. At the DataOrch Refresh LookUp prompt, enter the refresh for which you want to view error analysis.

Step 2. On the DOEA form, enter or modify the settings for running the report. For more information on these fields, see online help.

Step 3. Save your entries on the DOEA form.

Step 4. Complete the Peripheral Settings and Process Handler forms according to your preferences.

Step 5. When you are finished viewing, exit from the report.





Viewing Target Transform Data for a Target

In This ChapterThis chapter describes how to run the Transform Summary report in order to view the information for target transforms for a specified target.




Topic Page

Form Used 199

Viewing the Transform Summary Report for a Target 200

Procedure for Running the Transform Summary Report 202


Form Purpose

DataOrch Transform Summary (DOTS) View the information for target transforms associated with a target.


Viewing Reports and Refresh History: Viewing Target Transform Data for a Target

Viewing the Transform Summary Report for a Target

Use the DataOrch Transform Summary (DOTS) form to view a report of the target transforms for a specific target. This report includes data associated with a target transform, and also information about each column associated to the transform.

You can choose whether or not to include the description of the transform and the descriptions of the transform’s columns.

Figure 35: DataOrch Transform Summary (DOTS) Form

Note: This report can be used only for target transforms. You cannot enter source transforms.


Viewing the Transform Summary Report for a Target

Noteworthy Fields on the DOTS Form

The fields described in this section are important when setting up the Transform Summary report to run.

Target ID

The Target ID field displays the ID of the selected target.

Target Transforms

Use the Target Transforms field to enter target transform names, associated to the Target ID, to include in the report.

Include Transform Description (Y/N)

Use the Include Transform Description (Y/N) field to indicate whether to include the description of the transform in the report. Enter Yes to include the description, or enter No to exclude the description.

Include Column Descriptions (Y/N)

Use the Include Column Descriptions (Y/N) field to indicate whether to include the columns’ descriptions for the transform in the report. Enter Yes to include the descriptions, or enter No to exclude the descriptions.


Viewing Reports and Refresh History: Viewing Target Transform Data for a Target

Procedure for Running the Transform Summary Report

Use the following steps to run the Transform Summary report.

Step 1. Access the DataOrch Transform Summary (DOTS) form. At the DataOrch Target LookUp prompt, enter the ID of the target for which you want to view transform information.

Step 2. On the DOTS form, enter or modify the settings for running the report. For more information on these fields, see online help.

Step 3. Save your entries on the DOTS form.

Step 4. Complete the Peripheral Settings and Process Handler forms according to your preferences.

Step 5. When you are finished viewing, exit from the report.




In This ChapterThis chapter describes how to view the historical information for a refresh, including the settings, status, and statistics of previously run refreshes for a specific target.




Topic Page

Form Used 203

Viewing the History of a Refresh 204

Procedure for Viewing the History of a Refresh 206


Form Purpose

DataOrch Refresh History (DORH) View the settings, status, and statistics of a previously run refresh for a specific target.


Viewing Reports and Refresh History: Viewing the History of a Refresh

Viewing the History of a RefreshThe DataOrch Refresh History (DORH) form is used to view historical information about previously run refreshes. This form shows the refresh status, settings, and statistics of a previously run refresh for a specific ODS target database. The settings and statistics include the following:

Settings for the refresh:

Refresh ID

Target ID

Database name/TNS

DMI Listener host name

DMI Listener port number

Number of concurrent threads

For each target transform run as part of the refresh, the following information is included:

Target transform and source file names.

The Update Type for the transform. This would indicate the Override Update Type if the default Update Type was overridden for a transform.

Status of the transform, indicating whether it was successfully refreshed or not.

Date/time the target transform started and finished processing.

Transform information for the refresh:

Total number of rows written for the transform.

Number of rows deleted on the target database.

Number of errors encountered per transform.

For each materialized SQL view that is run as part of the refresh, the following information is included:

Materialized SQL view name. Note that an entry for a materialized SQL view is indicated by the words “Materialized View” in the Source File field.

An Update Type of “Full.” (Materialized SQL views cannot be incrementally updated).

The status of the refresh of the materialized SQL view, indicating whether it was successfully refreshed or not.

Date/time the materialized SQL view started and finished processing.



Figure 36: DataOrch Refresh History (DORH) Form


Viewing Reports and Refresh History: Viewing the History of a Refresh

You can also view the refresh status on the DORH form. The following statuses may be displayed.

InProc. The transform is currently being refreshed.

Success. The transform was successfully refreshed without any errors.

Warning. The transform or refresh received errors during processing.

Fail. A transform or refresh received fatal errors during processing.

Procedure for Viewing the History of a Refresh

Use the following step to view the history of a refresh.

Step 1. Access the DataOrch Refresh History (DORH) form. At the DataOrch Refresh History LookUp prompt, enter the ID of the history record you want to view.

Step 2. When you are finished viewing, exit from the DORH form.

Technical Tip: The history record ID consists of three parts: 1. The Refresh ID. 2. The date, in internal format, when the user ran the refresh. 3. The exact time, in internal format, when the user ran the refresh. For example, DO.REFRESH.ABC_14016_52885. The order of the Refresh IDs are sorted by date and time. The most recent refresh appears first.





Maintaining Transforms

In This ChapterThis chapter describes how to maintain DataOrchestrator ODS transforms. You can copy existing transforms, delete transforms as needed, and rename target transforms. You can also regenerate source transforms, and calculate foreign keys for both source and target transforms.




Topic Page

Form Used 209

Maintaining DataOrchestrator ODS Transforms 210

Procedure for Deleting Transforms 213

Procedure for Copying Transforms 214

Procedure for Renaming Transforms 215

Procedure for Copying and Renaming Transforms 216

Procedure for Calculating Foreign Keys 217

Procedure for Regenerating Source Transforms 218


Form Purpose

DataOrch Transform Maint (DOMA) Maintains DataOrchestrator ODS transforms.


Maintaining the DataOrchestrator ODS: Maintaining Transforms

Maintaining DataOrchestrator ODS Transforms

Use the DataOrch Transform Maint (DOMA) form to perform the following functions:

Delete transforms from a target.

Copy transforms from one target to another target.

Rename target transforms.

Copy transforms from a target, and then rename them to a different target or to the same target.

Calculate foreign keys for transforms.

Regenerate source transforms.

Figure 37: Example of Deleting a Transform using the DataOrch Transform Maint (DOMA) Form



Noteworthy Fields on the DOMA Form

The fields described in this section are important when maintaining transforms using the DataOrchestrator ODS.

Operation

Use the Operation field to choose an operation to perform on a target. Select one of the following operations:

Delete. Delete transforms from a target.

Copy. Copy transforms from one target to another.

Rename. Rename target transforms associated with a target. Source transforms cannot be renamed.

Calculate Foreign Keys. Generate foreign key metadata for transforms. Foreign key metadata will be created in the following cases: • If the input field for a column is a pointer to a Colleague file, then a foreign

key is created to any transform with that file as its source file. • A foreign key is created for each multivalued transform to all single-

valued transforms with the same source file.

Regenerate Source Transforms. Regenerate source transforms associated with a target. This will regenerate the set of target transforms to create and populate data for each target database table needed to hold the specified input fields of the source file. You will need to regenerate source transforms if any Colleague metadata changes for the fields you export in a source transform.

From Target ID

Use the From Target ID field to enter the name of the target from which to select associated transforms to process.

Transform List

Use the Transform List field to enter the transforms to process that are associated with the target selected in the From Target ID field.

Rename To

Use the Rename To field to enter a new name for a target transform on the Transform List. Source transforms cannot be renamed.



To Target ID

If you are copying to another Target ID, you must enter the name of the target to which you want to copy transforms. For a DataOrchestrator ODS target, you can enter the same target name as you entered in the From Target ID field. You would do this if you want to copy transforms from a target onto the same target with a new name. This allows you to create new transforms from existing transforms and modify them as needed.



Procedure for Deleting Transforms

To delete transforms from a target, use the following steps.

Step 1. Access the DataOrch Transform Maint (DOMA) form.

Step 2. In the Operation field, select Delete.

Step 3. In the From Target ID field, enter the ID of the target from which you want to delete transforms.

Step 4. In the Transform List field, enter the transforms you want to delete.

Step 5. Save from the DOMA form to delete the transforms you selected.



Procedure for Copying Transforms

To copy transforms from one target to another, use the following steps.


Step 2. In the Operation field, select Copy from the drop-down list.

Step 3. In the From Target ID field, enter the ID of the target from which you want to copy transforms.

Step 4. In the Transform List field, enter the transforms you want to copy.

Step 5. In the To Target ID field, enter the ID of the target to which you want to copy the transforms.

Step 6. Save from the DOMA form to copy the transforms you selected.

Note: If you copy to an existing transform, you will overwrite the filter of the transform being copied to with the filter of the transform being copied from. Also, if you copy to an existing transform that has any of the same column names, and the column sizes are larger in the existing transform, the larger size is used.



Procedure for Renaming Transforms

To rename transforms, use the following steps.


Step 2. In the Operation field, select Rename.

Step 3. In the From Target ID field, enter the ID of the target for which you want to rename transforms.

Step 4. In the Transform List field, enter the transforms you want to rename.

Step 5. In the Rename To field, enter the new name for each transform you listed.

Step 6. Save from the DOMA form to rename the transforms you selected.



Procedure for Copying and Renaming Transforms

To copy and rename transforms from a target to another target, or to the same target, use the following steps.



Step 3. In the From Target ID field, enter the ID of the target from which you want to copy transforms.

Step 4. In the Transform List field, enter the transforms you want to copy.

Step 5. In the Rename To field, enter the new name for each transform you listed.

Step 6. In the To Target ID field, enter the ID of the target to which you want to copy the transforms.

Step 7. Save from the DOMA form to copy and rename the transforms you selected.

Note: If you copy to an existing transform, you will overwrite the filter of the transform being copied to with the filter of the transform being copied from. Also, if you copy to an existing transform that has any of the same column names, and the column sizes are larger in the existing transform, the larger size is used.



Procedure for Calculating Foreign Keys

To calculate foreign keys for source and target transforms, use the following steps.


Step 2. In the Operation field, select Calculate Foreign Keys from the drop-down list.

Step 3. In the From Target ID field, enter the ID of the target from which you want to calculate foreign keys for transforms.

Step 4. In the Transform List field, enter the transforms for which you want to calculate foreign keys.

Step 5. Save from the DOMA form to calculate foreign keys for the transforms you selected.

If you want to view or modify foreign key metadata for target transforms, follow these steps:

a. Access the DataOrch Target (DOTT) form.

b. Detail from the Column Definitions field to access the Transform Columns (DOTC) form.

c. Detail from a target column name to the Column Properties (DOPR) form.

For more information, see Defining Target Transforms beginning on page 97.

Note: The foreign key metadata for source transforms cannot be modified after it has been calculated.



Procedure for Regenerating Source Transforms

To regenerate source transforms, use the following steps.


Step 2. In the Operation field, select Regen Source Transforms from the drop-down list.

Step 3. In the From Target ID field, enter the ID of the target from which you want to regenerate source transforms.

Step 4. In the Transform List field, enter the source transforms you want to regenerate.

Step 5. Save from the DOMA form to regenerate the source transforms you selected.



Maintaining SQL Views

In This ChapterThis chapter describes how to maintain SQL views. You can copy existing SQL views, delete views as needed, and rename SQL views.




Topic Page

Form Used 219

Maintaining SQL Views 220

Procedure for Deleting SQL Views 223

Procedure for Copying SQL Views 224

Procedure for Renaming SQL Views 225

Procedure for Copying and Renaming SQL Views 226


Form Purpose

DataOrch View Maintenance (DOVM) Maintains SQL views.


Maintaining the DataOrchestrator ODS: Maintaining SQL Views

Maintaining SQL ViewsUse the DataOrch View Maintenance (DOVM) form to perform the following functions:

Delete SQL views from a target.

Copy SQL views from one target to another target.

Rename SQL views.

Copy SQL views from a target, and then rename them to a different target or to the same target.

Note that when you when you copy an SQL view to a target and an SQL view with the same name exists on that target, the information in the target SQL view is updated as follows:

The SQL statement associated with view is replaced with that of the SQL view being copied.

If the SQL view being copied is a materialized SQL view, the materialized SQL view parameters, including the Create Materialized View flag and the Dependent Transforms list are overwritten. However, if the SQL view being copied is not a materialized SQL view, the existing data in the target SQL view is not changed.

The SQL view’s column information for the view being copied will be integrated into the target SQL view column information. Any new column entries will be added to the SQL View Columns list (on the DataOrch View Spec [DOVS] form) and information for existing columns will be overwritten. The only exception to this is the Key field. If this field is set to “Yes” in the existing SQL view, it will not be overwritten with a value of “No” from the SQL view being copied.

Note: This process modifies only the SQL view metadata. No processing will occur on the ODS target database.



Figure 38: Example of Deleting an SQL View using the DataOrch View Maintenance (DOVM) Form

Noteworthy Fields on the DOVM Form

The fields described in this section are important when deleting, copying, or renaming SQL views using the DataOrchestrator ODS.

Operation

Use the Operation field to choose an operation to perform on a target. Select one of the following operations:

Delete. Delete SQL views from a target.

Copy. Copy SQL views from one target to another.

Rename. Rename SQL views associated with a target.

Note: Deleting an SQL view from a target does not automatically delete the corresponding SQL view on the ODS target database. This must be done manually by the system administrator.



From Target ID

Use the From Target ID field to enter the name of the target from which to select associated SQL views to process.

SQL Views

Use the SQL Views list to enter the SQL views to process that are associated with the target selected in the From Target ID field.

Rename To

Use the Rename To field to enter a new name for an SQL view on the SQL Views list.

To Target ID

If you are copying to another Target ID, you must enter the name of the target to which you want to copy SQL views. You can enter the same target name as you entered in the From Target ID field to copy and rename SQL views to the same target. You would do this if you want to copy SQL views from a target onto the same target with a new name. This allows you to create new SQL views from existing SQL views and modify them as needed.



Procedure for Deleting SQL Views

To delete SQL views from a target, use the following steps.

Step 1. Access the DataOrch View Maintenance (DOVM) form.

Step 2. In the Operation field, select Delete.

Step 3. In the From Target ID field, enter the ID of the target from which you want to delete SQL views.

Step 4. In the SQL Views list, enter the SQL views you want to delete.

Step 5. Save from the DOVM form to delete the SQL views you selected.



Procedure for Copying SQL Views

To copy SQL views from one target to another, use the following steps.



Step 3. In the From Target ID field, enter the ID of the target from which you want to copy SQL views.

Step 4. In the SQL Views list, enter the SQL views you want to copy.

Step 5. In the To Target ID field, enter the ID of the target to which you want to copy the SQL views.

Step 6. Save from the DOVM form to copy the SQL views you selected.



Procedure for Renaming SQL Views

To rename SQL views, use the following steps.


Step 2. In the Operation field, select Rename.

Step 3. In the From Target ID field, enter the ID of the target for which you want to rename SQL views.

Step 4. In the SQL Views list, enter the SQL views you want to rename.

Step 5. In the Rename To field, enter the new name for each SQL view you listed.

Step 6. Save from the DOVM form to rename the SQL views you selected.



Procedure for Copying and Renaming SQL Views

To copy and rename SQL views from a target to another target, or to the same target, use the following steps.



Step 3. In the From Target ID field, enter the ID of the target from which you want to copy SQL views.

Step 4. In the SQL Views list, enter the SQL views you want to copy.

Step 5. In the Rename To field, enter the new name for each SQL view you listed.

Step 6. In the To Target ID field, enter the ID of the target to which you want to copy the SQL view.

Step 7. Save from the DOVM form to copy and rename the SQL views you selected.



Copying Targets to Another Target

In This ChapterThis chapter describes how to copy all transforms and SQL views from one or more targets to a single destination target.




Topic Page

Form Used 227

Copying One or More Targets to Another Target 228

Procedure for Copying One or More Targets to Another Target 230


Form Purpose

DataOrch Target Copy (DOTY) Copy transforms and SQL views from one or more targets to another single target.


Maintaining the DataOrchestrator ODS: Copying Targets to Another Target

Copying One or More Targets to Another Target

Use the DataOrch Target Copy (DOTY) form to copy all transforms and SQL views from one or more targets to a single destination target. This form provides the ability to install all target templates with different versions all at once. The target IDs listed in the Copy from Target IDs field will be sorted alphabetically before processing to ensure that the target templates are installed in the correct order.

Figure 39: DataOrch Target Copy (DOTY) Form

Note: This form copies all transforms and SQL views from the targets listed in the Copy from Target IDs field. To select specific transforms and SQL views from a target, use the DataOrch Transform Maint (DOMA) form and the DataOrch View Maintenance (DOVM) form. For further details on how individual transforms and SQL views will be copied, see the following information for the DOMA and DOVM forms: – Maintaining Transforms beginning on page 209 – Maintaining SQL Views beginning on page 219


Copying One or More Targets to Another Target

Noteworthy Fields on the DOTY Form

The fields described in this section are important when copying all transforms and SQL views from one or more targets to a single destination target.

Copy from Target IDs

Use the Copy from Target IDs field to enter one or more targets whose transforms and SQL views you want to copy to the destination target.

Copy to Target ID

Use the Copy to Target ID field to enter the target ID to which you want to add the transforms and SQL views from the targets you listed in the Copy from Target IDs field.

Note: Target templates are not allowed, as they cannot be modified. Also, the destination target cannot be a target that is listed in the Copy from Target IDs field.


Maintaining the DataOrchestrator ODS: Copying Targets to Another Target

Procedure for Copying One or More Targets to Another Target

Use the following steps to copy all transforms and SQL views from one or more targets to a single destination target.

Step 1. Access the DataOrch Target Copy (DOTY) form.

Step 2. On the DOTY form, enter one or more targets from which you want to copy all transforms and SQL views. Also, enter the destination target. For more information on these fields, see online help.

Step 3. Save from the DOTY form to copy the transforms and SQL views from the targets you selected to the destination target.




In This ChapterThis chapter describes how to delete DataOrchestrator ODS objects. You can delete existing refreshes, targets, or refresh history records from the source Colleague database.




Topic Page

Form Used 231

Deleting DataOrchestrator ODS Objects 232

Procedure for Deleting DataOrchestrator ODS Objects 233


Form Purpose

DataOrch Object Delete (DOOD) Delete DataOrchestrator ODS objects.


Maintaining the DataOrchestrator ODS: Deleting DataOrchestrator ODS Objects

Deleting DataOrchestrator ODS Objects Use the DataOrch Object Delete (DOOD) form to delete DataOrchestrator ODS objects. You can delete existing refreshes, targets, or refresh history records from the source Colleague database.

Figure 40: DataOrch Object Delete (DOOD) Form

Note: If you delete a target, you will also delete all of the target’s associated transforms, transform columns, and transform error records.



Noteworthy Fields on the DOOD Form

The fields described in this section are important when deleting objects using the DataOrchestrator ODS.

Object Type

Use the Object Type field to choose the DataOrchestrator ODS object type for which you want to delete IDs.

IDs to Delete

Use the IDs to Delete field to enter the IDs of the records to delete for the object type you selected. When you save, these IDs will be deleted from the source Colleague database.

Procedure for Deleting DataOrchestrator ODS Objects

Use the following steps to delete an object.

Step 1. Access the DataOrch Object Delete (DOOD) form.

Step 2. On the DOOD form, enter the settings for deleting objects. For more information on these fields, see online help.

Step 3. Save from the DOOD form to delete the objects you selected.

Note: The object type you select determines which IDs you can enter in the IDs to Delete field.


Maintaining the DataOrchestrator ODS: Deleting DataOrchestrator ODS Objects



Appendices

Appendices

Checklist for Setting Up the DataOrchestrator ODS

In This AppendixThis appendix provides a quick checklist of the activities that the system administrator and system programmer perform to set up and maintain the DataOrchestrator ODS.


Appendices: Checklist for Setting Up the DataOrchestrator ODS


Table 1: Checklist for Setting Up the DataOrchestrator ODS

Step User Description Reference

1. System administrator

Check that your institution has the prerequisites needed for the DataOrchestrator ODS.



Decide where to build the operational data stores. Where to Build the Operational Data Stores beginning on page 36.


Prepare the target database where the operational data stores will be built.

Preparing the ODS Target Database beginning on page 38.


Note: If your institution does not use SQL Server, skip this step.

For SQL Server, you can use a script to create your ODS target database.

SQL Server Script for Creating the ODS Target Database on page 40.


Note: If your institution does not use SQL Server, skip this step.

If you are using a SQL Server target database, the filegroup “IDX” must be in your target database.

Adding the IDX Filegroup on page 42.


Add the DODS optional module for the DataOrchestrator ODS.

Adding the Optional Module on page 46.


Create a new DMI Listener on the ODS target database server and select the roles DBAS and RDAS for the Listener. Use the Colleague release system to retrieve and load the appropriate software updates.

Setting Up the ODS Target Database Server and Installing Software Updates on page 47.

See Updating Colleague Software for information on retrieving and installing software updates.


Note: If your institution does not use Oracle, skip this step.

If the target database is Oracle, install the Oracle JDBC driver on the ODS target database. If your source environment is UniData and your target database is Oracle, you also must install the Oracle JDBC driver on your source environment.

Installing the Oracle JDBC Driver on page 51.




Define parameters for the DataOrchestrator ODS. Defining DataOrchestrator ODS Parameters beginning on page 59.


Create a target, which defines the configuration of an operational data store on a target database. This includes defining source and target transforms to refresh on the target database. (Optional) Define and create SQL views as shown.

Defining a Target beginning on page 67 (all chapters in Part 4).


Create a Refresh ID with an associated target. Define parameters to control how the DataOrchestrator ODS refreshes a target database with data from Colleague source files.

Defining and Running an ODS Refresh beginning on page 165.


Note: If your institution does not use stored computed columns, skip this step.

Activate and calculate stored computed columns for any target database that contains stored computed columns. If your ODS target database uses the STUDENT.TERMS.CC file, update the flags as shown.

Note: Datatel recommends that any operational data store that contains stored computed columns be set up to automatically run the following processes:

• Update GPA SCC Flags (UGSF) if using the STUDENT.TERMS.CC file

• Update Stored Computed Column (USCC)

This ensures that the stored computed columns will be updated when the ODS target database is updated.

Calculating Stored Computed Columns beginning on page 181. Refer to Envision Runtime Administration for information on how to set a batch process to run at scheduled intervals.


Run a refresh that updates the specified ODS target database. To run the refresh, access the DataOrch Refresh (DORE) form and enter Yes in the Run Refresh (Y/N) field.

(Optional) Create all SQL views for the target when you run the refresh by entering Yes in the Create SQL Views (Y/N) field.

Save from the DORE form to run the refresh.

Defining and Running an ODS Refresh beginning on page 165.

14. System administrator and system programmer

Run an error analysis report for any errors encountered when exporting data from a Colleague source file to an ODS target database.

Viewing Errors for a Refresh beginning on page 193.

Table 1: Checklist for Setting Up the DataOrchestrator ODS (cont’d)



Appendices: Checklist for Setting Up the DataOrchestrator ODS


(Optional) Run a Transform Summary report in order to view information for target transforms for a specified target.

Viewing Target Transform Data for a Target beginning on page 199.


(Optional) View historical information for previously run refreshes for a specific target.

Viewing the History of a Refresh beginning on page 203.


(Optional) Copy existing target transforms, delete transforms, and rename transforms. Regenerate source transforms, and create foreign keys for source and target transforms.

Maintaining Transforms beginning on page 209.


(Optional) Copy existing SQL views, delete views, and rename SQL views.

Maintaining SQL Views beginning on page 219.


(Optional) Copy all transforms and SQL views from one or more targets to a single destination target.

Copying Targets to Another Target beginning on page 227.


(Optional) Delete existing refreshes, targets, or refresh history records from the source Colleague database.

Deleting DataOrchestrator ODS Objects beginning on page 231.

Table 1: Checklist for Setting Up the DataOrchestrator ODS (cont’d)



Appendices

Frequently Asked Questions

In This AppendixThis appendix describes provides answers to frequently asked questions when using the DataOrchestrator ODS and the data models and views for reporting.

Table 2 lists the topics covered in this appendix.

Note: For more information on the DataOrchestrator ODS and the data models and views, reference AnswerNet page 5983, “DataOrchestrator Support Solutions.”

Table 2: Topics in this Chapter

Topic Page

Frequently Asked Questions about the DataOrchestrator ODS 242


Appendices: Frequently Asked Questions

Frequently Asked Questions about the DataOrchestrator ODS

Listed below are questions and answers for frequently asked questions about the DataOrchestrator ODS and the data models and views associated with the DataOrchestrator ODS.

Q What port will I need to open up in my firewall on the source database server?

A The port on which the database is running on the source database server should allow incoming requests.

Q What ports will I need to open up in my firewall on the target database server?

A The following ports on the target database server should allow incoming requests:

The port where the Datatel daemon is installed (usually port 9000).

The port you have chosen for the RDAS listener.

Q When I set up the RDAS Listener on the target database server, do I set up a new local product repository on that server?

A No. You do not need to set up a new local product repository on the target database server. The only prerequisite for installing the new listener on the target database server is that you have installed a Datatel daemon on the target database server.

Q I have a SQL Server target database. When testing connectivity on the DataOrch Refresh (DORE) form, I receive the following message. What does this mean?:

Unable to log in to target database.

Failed Login to ods_test on sdw2k3qasql1 by abcdef: Fatal: 18456(14):

Login failed for user 'abcdef'.



A Check the target SQL Server database instance where ods_test is set up to be sure a user named “abcdef” is defined with SQL Server authentication. Also, be sure the security for that SQL Server instance is set up to allow for both Windows and SQL Server authentication.

Q What permissions should I give users on the target SQL Server database?

A Give server roles of public and bulkadmin to the user. Give database role membership of public, db_ddladmin, db_datareader, and db_datawriter for that particular database.

Q When checking connectivity on the DORE form, I receive “DMI_OPEN” errors. How do I resolve these?

A First, check to be sure all listeners for the environment are running. If some are not running, be sure to start them, and then look for the following:

On the DORE form, be sure the RDAS listener installed on your target database server was selected.

If your source database is UniData or Oracle, then go to the DataOrch Parameters (DOPA) form and be sure the DMI_DAS listener is the DMI_DAS listener on your Colleague database server.

Q I created a new source transform using the PERSON file. The tables subsequently created on the SQL target database were not what I expected. How can I know which tables will be created and can I choose the tables, or is this based solely on the columns/fields that I select from the PERSON file?

For example, our institution does not use the MILITARY_INFO table, but because I selected the PERSON file, it was created. How could I skip this table?

A To understand tables created on the SQL target database, refer to Mapping Envision Files for SQL Server and Oracle, available on the Datatel website. This mapping pertains to source transforms only which create the Release 18 SQL schema table structure (that is, all single-valued columns in one table, all unassociated multivalues in the filename_LS table, and all associated columns in a table named after the association).

Note: This question could be posed for any file and association name; PERSON and MILITARY_INFO are examples.



The tables created also depend on which fields you choose on the DataOrch Source Transform (DOST) form when setting up your source transform. If you choose all the fields for the PERSON file, for example, you will have all 15 tables that make up the PERSON file in your target database.

If you want to omit tables from the target database, select specific columns from the PERSON file. For example, to prevent the MILITARY_ASSOC table from being created on the target database, do not select any of the following columns on the DOST form, which are all part of the MILITARY_ASSOC association:

MILITARY.STATUSES

MILITARY.BRANCHES

MILITARY.GOVT.BENEFITS

MILITARY.START.DATES

MILITARY.END.DATES

Also, you can view which tables will be created on the target database for a specific source transform by detailing on the View Generated Transforms field on the DOST form.

Q When running a refresh, I receive the following message:

User error 10015: - unable to authenticate your login.

A This message occurs because a pre-authenticated server is used for the file transfer in the DataOrchestrator ODS refresh. This means that on the DMI Pre-Authenticated Server (DMCC) form, you must have your application server and your Colleague database server defined. If you are a UniData client who has not distributed the DAS, then your application server and Colleague database server are the same, so you will need to enter only one line.

If you have the DMCC form set up and are still experiencing issues, then follow these steps:

Step 1. Start the following listeners with logging turned on:

The listener defined on the DORE form.

The listener installed on the DataOrch Parameters (DOPA) form.



Step 2. Rerun the Check Connectivity function on the DORE form to receive the error again, then search for an entry in the dmi.log file such as the following:

Unable to authenticate a client login from (hp.datatel.com,127.0.0.1)

Use the DNS name and the IP address in this entry to define the pre-authenticated server on the DMCC form.

Step 3. Restart the listeners again with logging turned off.

Q When running a refresh, I receive the following error:

Unable to log in to target database.

Failed Login to <db_name> on <server_name> by <user>: Premature end of Stream: server won't transmit TDS header

A This is probably due to an incorrect database port specified on the DOTA form. Access the DOTA form and verify that the correct database port is specified.

Q When running a refresh, I receive the following error:

The file separator detected (“/”) is not the same separator in the file path “...”

A Look for the following:

In SA Valet, make sure the listener on the target ODS database that is specified on DOTA has both the RDAS and DBAS roles.

If your source database is UniData or Oracle, then go to the DOPA form and make sure the correct source listener is specified in the Source DMI_DAS Listener Name field.




Appendices

Troubleshooting the DataOrchestrator ODS

In This AppendixThis appendix provides suggestions for items to check or steps to take if you encounter issues in using the DataOrchestrator ODS.

Note: For the most up-to-date information about the troubleshooting issues listed in this appendix, as well as any additional issues reported since the publication of this manual, see AnswerNet page 6063. If you do not have access to AnswerNet at your institution, consult with your system administrator.


Appendices: Troubleshooting the DataOrchestrator ODS

Dat

aOrc

hes

trat

or

OD

S Is

sues

Tab

le 3

list

s th

e is

sues

, pos

sibl

e ca

uses

, and

res

olut

ions

for

trou

bles

hoot

ing.

Tab

le 3

: T

roub

lesh

ootin

g Is

sues

Issu

eP

oss

ible

Cau

seR

eso

luti

on

1.R

ecei

ve o

ne o

f the

follo

win

g er

rors

on

the

Dat

aOrc

h R

efre

sh (

DO

RE

) fo

rm w

hen

chec

king

con

nect

ivity

or

runn

ing

a re

fres

h:

Uni

Dat

a so

urce

:

Error - encountered during Data

Extract

Cannot open D:\...\DOR... file.

Ora

cle

sour

ce:


Extract

ORA-20014: EXTRACT_DATA: Error

extracting data. Message: ORA-

20002: OPEN_FILE:

INVALID_OPERATION: File could

not be opened as requested

The

sou

rce

dire

ctor

y pa

th o

n th

e D

ataO

rch

Par

amet

ers

(DO

PA

) fo

rm

coul

d be

inco

rrec

t or

the

perm

issi

on

on th

e di

rect

ory

coul

d be

inco

rrec

t.

Ver

ify th

e pa

th o

n th

e D

OP

A fo

rm is

a v

alid

pat

h on

the

sour

ce d

atab

ase

serv

er. A

lso,

ver

ify th

e pe

rmis

sion

s to

th

e di

rect

ory

corr

espo

nd w

ith th

e in

stal

latio

n in

stru

ctio

ns

on s

ettin

g up

the

sour

ce d

atab

ase

serv

er te

mpo

rary

di

rect

ory.


DataOrchestrator ODS Issues

2.R

ecei

ve o

ne o

f the

follo

win

g er

rors

on

the

DO

RE

form

whe

n ch

ecki

ng c

onne

ctiv

ity o

r ru

nnin

g a

refr

esh:

SQ

L S

erve

r so

urce

:


Extract:

SQLState = S1000, NativeError =

0Error = [Microsoft][ODBC SQL

Server Driver]Unable to open BCP

host data-file

Uni

Dat

a an

d O

racl

e so

urce

:

Error encountered during file

transfer

D:\thisIsNotADirectory\... (The

system cannot find the path

specified)

The

targ

et d

irect

ory

path

on

the

Add

ition

al T

arge

t Par

amet

ers

(DO

TP

) fo

rm c

ould

be

inco

rrec

t or

the

perm

issi

on o

n th

e di

rect

ory

coul

d be

inco

rrec

t.

Ver

ify th

e pa

th o

n th

e D

OT

P fo

rm is

a v

alid

pat

h on

the

targ

et d

atab

ase

serv

er. A

lso

verif

y th

e pe

rmis

sion

s to

th

e di

rect

ory

corr

espo

nd w

ith th

e in

stal

latio

n in

stru

ctio

ns

on s

ettin

g up

the

targ

et d

atab

ase

serv

er te

mpo

rary

di

rect

ory.

Tab

le 3

: T

roub

lesh

ootin

g Is

sues

(co

nt’d

)

Issu

eP

oss

ible

Cau

seR

eso

luti

on



3.R

ecei

ve o

ne o

f the

follo

win

g er

rors

on

the

DO

RE

form

whe

n ch

ecki

ng c

onne

ctiv

ity o

r ru

nnin

g a

refr

esh.

UniSession Exception-39207:

Error [39207] occurred on

server. Possible client-side

licensing failure.:

Unable to connect to the

registry database.

OR

Unable to login to target

database. Registry error-30008:

Unable to access the registry

database for the request.

Uni

Dat

a lic

ense

s on

the

sour

ce

data

base

hav

e ru

n ou

t.S

peci

fy fe

wer

thre

ads

to u

se o

n th

e D

OR

E fo

rm, o

r ru

n th

e re

fres

h at

a ti

me

whe

n fe

wer

use

rs a

re lo

gged

in to

C

olle

ague

.

Tab

le 3

: T

roub

lesh

ootin

g Is

sues

(co

nt’d

)

Issu

eP

oss

ible

Cau

seR

eso

luti

on



4.R

ecei

ve o

ne o

f the

follo

win

g er

rors

whe

n ru

nnin

g a

refr

esh.


Extract for table:

Fatal: 4121(16): Cannot find

either column "dbo" or the user-

defined function or aggregate

computedColumn, or the name is

ambiguous.

OR


Extract for table:

computedColumn is not a column

in table

The

com

pute

d co

lum

n do

es n

ot

exis

t in

the

sour

ce d

atab

ase.

T

he c

ompu

ted

colu

mn

mus

t be

able

to b

e ge

nera

ted

to

the

data

base

usi

ng th

e D

efin

e C

ompu

ted

Col

umns

(D

CC

) fo

rm. I

t may

be

that

the

colu

mn

need

s to

be

rege

nera

ted

in o

rder

for

the

Dat

aOrc

hest

rato

r to

use

it.

5.R

ecei

ve a

n er

ror

such

as

the

follo

win

g w

hen

runn

ing

a re

fres

h:

The file separator detected

("/") is not the same separator

in the file path "..."

The

targ

et R

DA

S li

sten

er d

oes

not

also

hav

e th

e D

BA

S r

ole

it ne

eds,

or

the

liste

ner

spec

ified

on

the

DO

PA

fo

rm is

inco

rrec

t.

Ver

ify th

at th

e ta

rget

RD

AS

list

ener

has

bot

h th

e D

BA

S

and

RD

AS

rol

es a

ccor

ding

to th

e in

stal

latio

n in

stru

ctio

ns. V

erify

that

the

liste

ner

nam

e on

the

DO

PA

fo

rm is

the

corr

ect s

ourc

e da

taba

se li

sten

er.

6.R

ecei

ve a

n er

ror

such

as

the

follo

win

g w

hen

runn

ing

a re

fres

h:

No source listener host name

specified

OR

No source extract file path

specified

The

DO

PA

form

is n

ot d

efin

ed fo

r a

Uni

Dat

a or

Ora

cle

sour

ce

envi

ronm

ent.

Ver

ify th

at th

e pa

ram

eter

s on

the

DO

PA

form

are

co

rrec

t.

Tab

le 3

: T

roub

lesh

ootin

g Is

sues

(co

nt’d

)

Issu

eP

oss

ible

Cau

seR

eso

luti

on



7.R

ecei

ve a

n er

ror

such

as

the

follo

win

g w

hen

runn

ing

a re

fres

h:

Could not RPC execute command

The

tran

sfor

m is

taki

ng o

ver

an h

our

for

a U

niD

ata

envi

ronm

ent t

o pr

oces

s.

The

Uni

Dat

a R

PC

con

nect

ion

defa

ult t

imeo

ut n

eeds

to

be in

crea

sed.

The

Uni

Dat

a un

irpcs

ervi

ces

file

need

s th

e ud

api_

serv

er ti

meo

ut in

crea

sed

from

360

0 to

144

00.

The

set

ting

requ

ires

unirp

cd to

be

rest

arte

d in

ord

er to

ta

ke e

ffect

.

For

exa

mpl

e:

/usr/ud71/unishared/unirpc/

unirpcservices:

defcs /cm/tools/ud71_10/bin/udapi_server

* TCP/IP 0 14400

8.R

ecei

ve a

n er

ror

such

as

the

follo

win

g w

hen

runn

ing

a re

fres

h w

ith a

SQ

L S

erve

r ta

rget

dat

abas

e:

Unable to log in to target

database. Failed Login to

ods_test on sdw2k3qasql1 by

abcdef: Fatal: 18456(14): Login

failed for user 'abcdef'

A u

ser

nam

ed "

abcd

ef"

is n

ot a

ble

to

conn

ect t

o th

e ta

rget

SQ

L S

erve

r da

taba

se w

ith S

QL

Ser

ver

auth

entic

atio

n.

Che

ck th

e ta

rget

SQ

L S

erve

r da

taba

se in

stan

ce w

here

od

s_te

st is

set

up

to b

e su

re th

at a

use

r na

med

"ab

cdef

" is

def

ined

with

SQ

L S

erve

r au

then

ticat

ion.

Als

o, b

e su

re

the

secu

rity

for

that

SQ

L S

erve

r in

stan

ce is

set

up

to

allo

w fo

r bo

th W

indo

ws

and

SQ

L S

erve

r au

then

ticat

ion.

9.W

hen

chec

king

con

nect

ivity

on

the

DO

RE

fo

rm, t

he fo

llow

ing

erro

r is

rec

eive

d:

DMI_OPEN

The

ref

resh

pro

cess

is n

ot a

ble

to

conn

ect t

o ei

ther

the

sour

ce

DM

I_D

AS

list

ener

or

targ

et R

DA

S

liste

ner.

Che

ck th

at a

ll lis

tene

rs fo

r th

e en

viro

nmen

t are

run

ning

. If

som

e ar

e no

t run

ning

, sta

rt th

em, a

nd th

en lo

ok fo

r the

fo

llow

ing:

•O

n th

e D

OR

E fo

rm, b

e su

re th

at th

e R

DA

S li

sten

er

inst

alle

d on

you

r ta

rget

dat

abas

e se

rver

was

se

lect

ed.

•If

your

sou

rce

data

base

is U

niD

ata

or O

racl

e, th

en

acce

ss th

e D

OP

A fo

rm a

nd b

e su

re th

e D

MI_

DA

S

liste

ner

is th

e D

MI_

DA

S li

sten

er o

n yo

ur C

olle

ague

da

taba

se s

erve

r.

Tab

le 3

: T

roub

lesh

ootin

g Is

sues

(co

nt’d

)

Issu

eP

oss

ible

Cau

seR

eso

luti

on



10.

Rec

eive

an

erro

r su

ch a

s th

e fo

llow

ing:

User error 10015: - unable to

authenticate your login.

Thi

s m

essa

ge o

ccur

s be

caus

e a

pre-

auth

entic

ated

ser

ver

is u

sed

for

the

file

tran

sfer

in th

e D

ataO

rche

stra

tor

OD

S r

efre

sh.

On

the

DM

I Pre

-Aut

hent

icat

ed S

erve

r (D

MC

C)

form

, yo

u m

ust h

ave

your

app

licat

ion

serv

er a

nd y

our

Col

leag

ue d

atab

ase

serv

er d

efin

ed. I

f you

r in

stitu

tion

uses

Uni

Dat

a an

d ha

s no

t dis

trib

uted

the

DA

S, t

hen

your

app

licat

ion

serv

er a

nd C

olle

ague

dat

abas

e se

rver

ar

e th

e sa

me,

so

you

will

nee

d to

ent

er o

nly

one

line.

If yo

u ha

ve th

e D

MC

C fo

rm s

et u

p an

d ar

e st

ill

expe

rienc

ing

issu

es, t

hen

follo

w th

ese

step

s:

Ste

p 1:

Sta

rt th

e fo

llow

ing

liste

ners

with

logg

ing

turn

ed

on: •

The

list

ener

def

ined

on

the

DO

RE

form

.

•T

he li

sten

er in

stal

led

on th

e D

OP

A fo

rm.

Ste

p 2:

Rer

un th

e C

heck

Con

nect

ivity

func

tion

on th

e D

OR

E fo

rm to

rec

eive

the

erro

r ag

ain,

then

sea

rch

for

an e

ntry

in th

e dm

i.log

file

suc

h as

the

follo

win

g:

"Unable to authenticate a client login

from (hp.datatel.com,127.0.0.1)"

Use

the

DN

S n

ame

and

the

IP a

ddre

ss in

this

ent

ry to

de

fine

the

pre-

auth

entic

ated

ser

ver

on th

e D

MC

C fo

rm.

Ste

p 3:

Res

tart

the

liste

ners

aga

in w

ith lo

ggin

g tu

rned

of

f.

11.

Rec

eive

an

erro

r su

ch a

s th

e fo

llow

ing:

Unable to log in to target

database. Failed Login to

<db

_nam

e> on <

serv

er_n

ame> by

<us

er>: Premature end of Stream:

server won't transmit TDS header

Thi

s is

pro

babl

y du

e to

an

inco

rrec

t da

taba

se p

ort s

peci

fied

on th

e D

OTA

form

.

Acc

ess

the

DO

TA fo

rm a

nd v

erify

that

the

corr

ect

data

base

por

t is

spec

ified

.

Tab

le 3

: T

roub

lesh

ootin

g Is

sues

(co

nt’d

)

Issu

eP

oss

ible

Cau

seR

eso

luti

on



12.

Rec

eive

an

erro

r su

ch a

s th

e fo

llow

ing:

No suitable driver.

Thi

s er

ror

occu

rs w

hen

usin

g th

e fo

llow

ing

form

s to

cre

ate

or te

st S

QL

view

s on

an

Ora

cle

targ

et d

atab

ase:

•D

ataO

rch

Targ

et V

iew

s (D

OT

V)

•D

ataO

rch

Vie

w S

pec

(DO

VS

)

The

re is

no

Ora

cle

JDB

C d

river

in

stal

led

in th

e U

niD

ata

sour

ce

envi

ronm

ent.

If yo

u ha

ve a

Uni

Dat

a so

urce

en

viro

nmen

t and

you

r ta

rget

da

taba

se is

Ora

cle,

then

you

als

o m

ust i

nsta

ll th

e O

racl

e JD

BC

driv

er

on y

our

sour

ce e

nviro

nmen

t.

Inst

all t

he O

racl

e JD

BC

driv

er u

sing

the

inst

ruct

ions

in

Inst

allin

g th

e O

racl

e JD

BC

Driv

er o

n pa

ge 5

1, b

ut o

n th

e U

niD

ata

sour

ce e

nviro

nmen

t. T

hen

rest

art t

he

sour

ce e

nviro

nmen

t DM

I_D

AS

for

the

chan

ge to

take

ef

fect

.

13.

The

ref

resh

will

“ha

ng”

in a

SQ

L S

erve

r so

urce

env

ironm

ent.

The

re m

ay b

e m

ultip

le v

ersi

ons

of

the

bcp.

exe

file.

Thi

s oc

curs

whe

n m

ultip

le v

ersi

ons

of S

QL

Ser

ver

have

bee

n in

stal

led

on th

e O

DS

ta

rget

dat

abas

e se

rver

, whi

ch is

us

ed fo

r th

e da

ta e

xtra

ct.

Mak

e su

re th

e pa

th e

nviro

nmen

t var

iabl

e fo

r th

e O

DS

ta

rget

dat

abas

e se

rver

is p

oint

ing

to th

e la

test

ver

sion

of

bcp.

If y

ou u

pdat

e th

e pa

th, y

ou m

ust r

esta

rt th

e ta

rget

R

DA

S li

sten

er fo

r th

e ch

ange

to ta

ke e

ffect

.

14.

Rec

eive

the

follo

win

g er

ror:

ERROR, cannot read source file

<<file path>>

The

Sou

rce

Tem

pora

ry F

ile P

ath

ente

red

on th

e D

OP

A fo

rm is

in

corr

ect.

The

Sou

rce

Tem

pora

ry F

ile P

ath

is s

et u

p on

the

DO

PA

fo

rm. C

heck

that

this

pat

h ex

ists

, and

that

the

SY

ST

EM

us

er a

nd/o

r th

e us

er r

unni

ng D

MI h

as fu

ll re

ad/w

rite

acce

ss to

the

fold

er. A

lso,

try

dele

ting

and

re-e

nter

ing

the

entir

e pa

th. S

tray

spa

ces

may

cau

se in

corr

ect p

aths

an

d ar

e no

t eas

y to

see

in U

I for

ms.

Com

men

t: T

he fu

ll er

ror

may

look

som

ethi

ng li

ke th

is:

Error encountered during file transfer

SSFRQ: ERROR, cannot read source file

<<file path>>

15.

Rec

eive

the

follo

win

g er

ror

in a

Uni

Dat

a en

viro

nmen

t:

In BP/_S_RPC_EXEC at line 2

In BP/_S_RPC_EXEC at line 2 too

many names in LIST

The

tran

sfor

m b

eing

ref

resh

ed h

as

over

150

col

umns

spe

cifie

d. F

or

mor

e in

form

atio

n, s

ee S

uppo

rt

Sol

utio

n 43

04: U

niQ

uery

LIS

T fi

eld

limit.

Spe

cify

150

col

umns

or

less

for

the

tran

sfor

m. Y

ou c

an

eith

er b

e m

ore

sele

ctiv

e ab

out t

he c

olum

ns in

clud

ed fo

r th

e tr

ansf

orm

, or

split

the

colu

mns

into

mul

tiple

tr

ansf

orm

s.

Tab

le 3

: T

roub

lesh

ootin

g Is

sues

(co

nt’d

)

Issu

eP

oss

ible

Cau

seR

eso

luti

on



16.

Rec

eive

the

follo

win

g ty

pe o

f err

or:

Violation of PRIMARY KEY

constraint

Thi

s is

usu

ally

a s

peci

fic d

ata

issu

e w

here

dup

licat

e ke

ys e

xist

for o

ne o

r m

ore

files

/tabl

es. T

he s

ourc

e of

the

prob

lem

may

not

alw

ays

be

appa

rent

, bec

ause

the

extr

act a

nd

tran

sfor

mat

ion

proc

ess

split

s so

urce

fil

es in

to m

any

diffe

rent

tabl

es o

n th

e ta

rget

.

See

Sup

port

Sol

utio

n 64

38: O

DS

: “V

iola

tion

of

PR

IMA

RY

KE

Y c

onst

rain

t” S

QL

erro

r.

17.

Rec

eive

the

follo

win

g er

rors

in C

olle

ague

S

QL

Ser

ver

envi

ronm

ent o

n th

e D

OR

E

form

whe

n ch

ecki

ng c

onne

ctiv

ity o

r ru

nnin

g a

refr

esh:

SQLState = 08001, NativeError =

53Error = [Microsoft][SQL Native

Client]Named Pipes Provider:

Could not open a connection to

SQL Server [53]. SQLState =

HYT00, NativeError = 0Error =

[Microsoft][SQL Native

Client]Login timeout

expiredSQLState = 08001,

NativeError = 53Error =

[Microsoft][SQL Native Client]An

error has occurred while

establishing a connection to the

server. When connecting to SQL

Server 2005, this failure may be

caused by the fact that under

the default settings, SQL Server

does not allow remote

connections.

Thi

s er

ror

can

occu

r du

ring

exec

utio

n of

the

bcp

stat

emen

t tha

t ex

trac

ts d

ata

from

the

Col

leag

ue

SQ

L S

erve

r da

taba

se. T

o co

nfirm

th

is, t

urn

logg

ing

on in

the

DA

S a

nd

then

rer

un th

e re

fres

h an

d vi

ew th

e lo

g. In

the

log,

find

the

SB

XT

Q

inco

min

g an

d ou

tgoi

ng tr

ansa

ctio

ns

whi

ch lo

ok s

omet

hing

like

this

:

incoming:

DMIþ2.0þDBIQþþþsql17746142

01þþþþþþþþþþþSBXTQþ9þ0þf:\

datatel\coll18\test\ODS_LI

STENER\ods_temp\þ~|~þDOR70

64.datþSYSDEFSþSELECT

SYSDEFS SAMPLE 5 RETURNING

DELIM @IDþSBXTQ.END

outgoing:

DMIþ1.4þDBISþþþsql17746142

01þþýj3þ14821þ54787þþþþþþþ

SERRSþ7þ0þþþSQLState =

08001, NativeError =

53Error = [Microsoft][SQL

Native Client]Named Pipes

Use

the

follo

win

g M

icro

soft

reso

urce

to u

pdat

e th

e se

rver

nam

e re

turn

ed b

y th

e S

QL

stat

emen

t SE

LEC

T

@@

SE

RV

ER

NA

ME

:

http

://m

sdn.

mic

roso

ft.co

m/e

n-us

/libr

ary/

ms1

8794

4.as

px

Tab

le 3

: T

roub

lesh

ootin

g Is

sues

(co

nt’d

)

Issu

eP

oss

ible

Cau

seR

eso

luti

on


http://msdn.microsoft.com/en-us/library/ms187944.aspx


(con

tinue

d)Provider: Could not open a

connection to SQL Server

[53]. SQLState = HYT00,



Client]Login timeout

expiredSQLState = 08001,



Client]An error has

occurred while establishing

a connection to the

server.When connecting to

SQL Server 2005, this

failure may be caused by

the fact that under the

default settings SQL Server

does not allow remote

connections.þSERRS.END

If yo

u se

e an

SE

RR

S r

espo

nse,

this

m

eans

the

bcp

stat

emen

t tha

t the

R

DA

S r

an fa

iled.

To

figur

e ou

t the

se

rver

nam

e to

use

in th

e bc

p st

atem

ent,

the

RD

AS

run

s th

e fo

llow

ing

SQ

L st

atem

ent t

o ge

t the

se

rver

nam

e:

SELECT @@SERVERNAME

You

can

run

this

in M

anag

emen

t S

tudi

o. If

the

serv

er n

ame

is

inco

rrec

t, us

e th

e M

icro

soft

link

give

n in

the

Res

olut

ion

on p

age

255

to u

pdat

e th

e se

rver

nam

e.

Tab

le 3

: T

roub

lesh

ootin

g Is

sues

(co

nt’d

)

Issu

eP

oss

ible

Cau

seR

eso

luti

on



(con

tinue

d)T

he @

@S

ER

VE

RN

AM

E m

ight

be

inco

rrec

t fro

m c

hang

ing

a se

rver

's

nam

e. If

you

cha

nge

the

nam

e, b

ut

do n

ot u

pdat

e th

e @

@S

ER

VE

RN

AM

E in

SQ

L S

erve

r, th

en y

ou w

ill li

kely

run

into

this

issu

e du

ring

a D

ataO

rche

stra

tor

OD

S

refr

esh.

To s

ee if

the

issu

e is

due

to a

m

ism

atch

, run

the

follo

win

g on

the

sour

ce d

atab

ase

serv

er a

nd c

heck

to

see

if it

ret

urns

the

corr

ect s

erve

r:

SELECT @@SERVERNAME

18R

ecei

ve a

mes

sage

in S

QL

Ser

ver

Col

leag

ue s

ourc

e en

viro

nmen

t sim

ilar

to

the

follo

win

g:

SQ

LSta

te =

280

00, N

ativ

eErr

or =

18

456E

rror

= [M

icro

soft]

[SQ

L N

ativ

e C

lient

][SQ

L S

erve

r]Lo

gin

faile

d fo

r us

er.

The

SQ

L S

erve

r bc

p co

mm

and

trea

ts c

erta

in c

hara

cter

s as

pa

ram

eter

/com

man

d te

rmin

ator

s.

Ver

ify y

our

sour

ce e

nviro

nmen

t us

erna

me

and

pass

wor

d do

not

in

clud

e an

y of

the

follo

win

g ch

arac

ters

: []

{}()

, ; ?

* !

@

If yo

ur S

QL

Ser

ver

Col

leag

ue s

ourc

e en

viro

nmen

t us

erna

me

or p

assw

ord

incl

udes

one

of t

hese

ch

arac

ters

, cha

nge

your

pas

swor

d so

that

non

e of

thes

e ch

arac

ters

is u

sed.

Tab

le 3

: T

roub

lesh

ootin

g Is

sues

(co

nt’d

)

Issu

eP

oss

ible

Cau

seR

eso

luti

on




Index

From this index you can click on any entry to access the information about the topic.

AActivating, stored computed columns 181, 183, 184

Additional Refresh Parameters (DORP) form 174

Additional Target Parameters (DOTP) form 78

Additional target parameters for 77

Aggregation operations 122

Alertscollation sequence for iStrategy 40directory must have sufficient space 158, 178secure DOTA and DORE forms 38secure ODS target database 38, 177target database must be refreshed by only one

target 158using multiple threads for refresh affects system

resources 173

BBNGN form 86, 110

Bundle Generation (BNGN) form 86, 110

CCalculating, stored computed columns 181

Cardinality rules 107

CDSU form 194

Colleague Data Scanner Utility (CDSU) form 194

Column documentation for SQL view 147

Column Properties (DOPR) form 113

CopyingSQL views 219targets to another target 227transforms 209

Copying and renamingSQL views 219transforms 209

Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.

CreatingSQL view 153SQL views using the DOTV form 154

DData mapping of Colleague data 25

DataOrch Error Analysis (DOEA) form 194

DataOrch Object Delete (DOOD) form 232

DataOrch Parameters (DOPA) form 60

DataOrch Refresh (DORE) form 153, 166

DataOrch Source Transform (DOST) form 83

DataOrch Target (DOTA) form 71

DataOrch Target Copy (DOTY) form 228

DataOrch Target Transform (DOTT) form 101

DataOrch Target Views (DOTV) form 145

DataOrch Transform Maint (DOMA) form 210

DataOrch Transform Summary (DOTS) form 200

DataOrch View Maintenance (DOVM) form 220

DataOrch View Spec (DOVS) form 147

DataOrchestrator ODSarchitecture 26benefits 22configuration of hardware and software 29defining parameters 59deleting objects 231installing 45, 47overview of using 55prerequisites 35suggested workflow 55understanding 22

DCC form 86, 110

Define Computed Column (DCC) form 86, 110

Define Stored Computed Column (DSCC) form 183, 184

259

Index

Definingadditional target parameters 77DataOrchestrator ODS parameters 59ODS refresh 165operations for transform columns 116source transforms 82SQL select statement 147SQL view 145target transform column properties 113target transform columns 107target transforms 100targets 67

DeletingDataOrchestrator ODS objects 231refresh history records 231refreshes 231SQL views 219targets 231transforms 209

Delimiters 118

Dependent transforms 150

DOCA form 129

DODS optional module 46

DOEA form 194

DOEE form 138

DOFC form 89

DOFE form 117

DOFR form 123

DOGT form 93

DOMA form 210

DOMV form 120

DONV form 126

DOOD form 232

DOPA form 60

DOPR form 113

DORE form 153, 166

DORP form 174

DOSS form 132

DOST form 83

DOTA form 71

DOTC form 107

DOTP form 78

DOTS form 200

260

DOTT form 101

DOTV form 145

DOTVS form 147

DOTY form 228

DOVC form 135

DOVM form 220

DSCC form 183, 184

EEnvision data mapping 25

Error analysis report 193

Errorscorrecting in source environment using CDSU 194viewing after ODS refresh 77, 193

Expression Entry (DOEE) form 138

Expression Entry operation 112, 138

FField Extract (DOFE) form 117

Field Extract operation 111, 117

File suite instances 86, 92, 104, 125

Filter Criteria (DOFC) form 89

Forms, Datatel softwareAdditional Refresh Parameters (DORP) 174Additional Target Parameters (DOTP) 78Bundle Generation (BNGN) 86, 110Colleague Data Scanner Utility (CDSU) 194Column Properties (DOPR) 113DataOrch Error Analysis (DOEA) 194DataOrch Object Delete (DOOD) 232DataOrch Parameters (DOPA) 60DataOrch Refresh (DORE) 153, 166DataOrch Source Transform (DOST) 83DataOrch Target (DOTA) 71DataOrch Target Copy (DOTY) 228DataOrch Target Transform (DOTT) 101DataOrch Target Views (DOTV) 145DataOrch Transform Maint (DOMA) 210DataOrch Transform Summary (DOTS) 200DataOrch View Maintenance (DOVM) 220DataOrch View Spec (DOVS) 147Define Computed Column (DCC) 86, 110Define Stored Computed Column (DSCC) 183,

184Expression Entry (DOEE) 138Field Extract (DOFE) 117


Index

Forms, Datatel software (cont’d)Filter Criteria (DOFC) 89Generated Target Transforms (DOGT) 93Multivalue Operation (DOMV) 120Null Value Replacement (DONV) 126Pointer Reference (DOFR) 123String Concatenation (DOCA) 129Substring Selection (DOSS) 132Transform Columns (DOTC) 107Update GPA SCC Flags (UGSF) 183, 188Update Stored Computed Column (USCC) 183,

186Validation Code LookUp (DOVC) 135

Full refresh 166definition 15

GGenerated Target Transforms (DOGT) form 93

HHistory record ID, three parts of 206

History, viewing for ODS refresh 203

IIDX filegroup 42

Improving refresh performance 166

Incremental refreshconsiderations 166definition 15

InstallingDataOrchestrator ODS 45, 47JDBC driver for Oracle 51overview for the DataOrchestrator ODS 34software updates 47

JJDBC driver, installing for Oracle 51

MMaterialized view

copying 220defining parameters 147definition 143dependent transforms 150

Multivalue Aggregation operation 111, 120

Multivalue Operation (DOMV) form 120

Multivalue Position operation 111, 120


NNull Replacement operation 111, 126

Null Test operation 111, 126

Null Value Replacement (DONV) form 126

OODS history, viewing 203

ODS refreshcreating SQL views 171defining 165viewing errors for 193viewing history for 203

ODS target databasepreparing 38SQL Server script for creating 40

ODS target database server, setting up 47

ODS targetsactivating stored computed columns 181calculating stored computed columns 181

Operational data storesdefinition 14facts about 24limitations 24where to build 36

Operationsaggregation 122available arguments 140definitions 111expression entry 138field extract 117forms used to define 116multivalue aggregation 120multivalue position 120null replacement 126null test 126pointer reference 123string concatenation 129substring 132validation code lookup 135

Optional module, adding 46

Oracleconnection considerations 37database considerations 37installing JDBC driver 51

261

Index

PPerformance of refresh 166

Permissionsdirectory for temporary files for Bulk Load

transaction 48directory for temporary files from Bulk Extract

transaction. 49for user refreshing DataOrchestrator ODS 48source temporary files directory 63, 176target SQL Server database 243target temporary files directory 79troubleshooting 248, 249

Pointer Reference (DOFR) form 123

Pointer Reference operation 111, 123

Prerequisites for DataOrchestrator ODS 35

Proceduresactivating stored computed columns 184adding the IDX filegroup 42calculating foreign keys for transforms 217calculating stored computed columns 186copying and renaming SQL views 226copying and renaming transforms 216copying one or more targets to another target 230copying SQL views 224copying transforms 214creating SQL views using the DOTV form 154defining a refresh 177defining a target 158defining DataOrchestrator ODS parameters 64defining SQL views 152deleting DataOrchestrator ODS objects 233deleting SQL views 223deleting transforms 213installing the Oracle JDBC driver 51regenerating source transforms 218renaming SQL views 225renaming transforms 215running the error analysis report 197running the transform summary report 202updating flags in STUDENT.TERMS.CC file 188viewing the history of a refresh 206

262

RRDAS Listener 169

Recommendationsbatch commit size set at 1000 62monitor space usage in temporary directory 48, 49,

63, 79, 175run USCC before run DORE 183secure access to the DORE form 177secure DOTA 38secure ODS target database 38set up automatic refresh for an ODS with stored

computed columns 24set up process handler 183use DNS alias name 73use the UGSF process before the USCC process

187workflow for DataOrchestrator ODS 55

Refresh history records, deleting 231

Refresh ID, cannot change associated target ID 177

Refresh performance 166

Refresh statuses 167

Refreshes, deleting 231

RenamingSQL views 219transforms 209

Reporting Data Access Server (RDAS) 15

SSchema name, definition 74

SecuringDORE 38, 177DOTA 38ODS target database 38, 177

Setting upforms to set up targets 68ODS target database server 47

Software updatesinstalling 47


Index

Source transformsdefining 82definition 14, 71setting up 83

SQL select statementdefining 147

SQL Serveradding IDX filegroup 42connection considerations 36database considerations 36

SQL viewcolumn documentation 147copy and renaming 226copying 224creating 153defining 145defining SQL select statement 147definition 15deleting 223renaming 225testing 151

Stored computed columnsactivating 181, 183, 184calculating 181

Stored procedure 172

String Concatenation (DOCA) form 129

String Concatenation operation 111, 129

Substring operation 111, 132

Substring Selection (DOSS) form 132

System prerequisites for DataOrchestrator ODS 35

TTarget transform columns

defining 107defining properties for 113

Target transformsdefining 100definition 14, 71viewing for a target 199

Targetsadditional parameters 77copying to another target 227defining 67definition 14deleting 231forms used to set up 68


TestingSQL view 151

Transform Columns (DOTC) form 107

Transform columns, defining operation for 116

Transform Summary report 199

Transformscalculating foreign keys for 217copying 214copying and renaming 216deleting 213regenerating source transforms 218renaming 215

UUGSF form 183, 188

Update GPA SCC Flags (UGSF) form 183, 188

Update Stored Computed Column (USCC) form 183, 186

USCC form 183, 186

VValidation Code LookUp (DOVC) form 135

Validation Code LookUp operation 111, 135

Viewingerrors for an ODS refresh 193history of an ODS refresh 203target transforms for a target 199

WWhere to build operational data stores 36

263

Index

264

Date post:	21-May-2018
Category:	Documents
Upload:	ngodiep
View:	215 times
Download:	2 times

Using the DataOrchestrator ODS™sgps.wau.edu/Datatel/UsingTheDataOrchestrator.pdf · Using the...

Documents