Post on 21-May-2018
transcript
Using the DataOrchestrator ODS™Reissued Manual as of June 18, 2010
This is a new edition of the Using the DataOrchestrator ODS™ manual. This edition replaces the previous edition dated October 20, 2009, and incorporates the changes delivered with software update SU50297.04-485.
The Primary Changes Made
Section Pages Changes Made
Using the DataOrch Source Transform (DOST) Form
Defining Filter Criteria
Using the DataOrch Target Transform (DOTT) Form
83, 87 89, 91 101, 104, 105
An Incremental Date Field is no longer required to perform an incremental refresh of a transform. The As of Date, together with the Incremental Filter Criteria, now allows a transform to be run as an incremental.
Using the DataOrch Source Transform (DOST) Form
86 Individual fields that are a part of a multi-part key are now available on the DataOrch Source Transform (DOST) form. Each field is included in the Include Computed Columns list for a multi-part key file and can be used to extract the individual piece of data from a multi-part key.
Defining Target Transform Columns
110 A timestamp column can be added to each transform on the Transform Columns (DOTC) form to record the time at which the refresh was run.
Defining SQL Views 145 A new field, Material View, was added to the DataOrch Target Views (DOTV) form. This field displays “Yes” for each SQL view that is defined as a materialized view on the DataOrch View Specs (DOVS) form.
Defining the SQL Select Statement for an SQL View
147 - 152 The ability to create a table from an SQL select statement (materialized view) has been added to the DataOrch View Spec (DOVS) form.
Defining Settings for Refreshing Data
172 A new option, Run Stored Procedure, has been added to the DataOrch Refresh (DORE) form to automatically run a stored procedure on the ODS target database at the completion of a refresh.
Viewing the History of a Refresh
204 Information for materialized views is displayed on the DataOrch Refresh History (DORH) form.
Maintaining SQL Views 220 Information was added to explain what now happens when you copy an SQL view to a target and an SQL view with the same name already exists on that target.
Using the DataOrchestrator ODS™
Release 18
June 18, 2010
Datatel Colleague®
For last-minute updates and additional information about this manual, see AnswerNet page 5696.
Using the DataOrchestrator ODS™
© 2010 Datatel, Inc. All Rights Reserved
The information in this document is confidential and proprietary to and considered a trade secret of Datatel, Inc., and shall not be reproduced in whole or in part without the written authorization of Datatel, Inc. The information in this document is subject to change without notice.
Colleague and ActiveCampus are registered trademarks of Datatel, Inc. ActiveAlumni and ActiveAdmissions are trademarks of Datatel, Inc. Other brand and product names are trademarks or registered trademarks of their respective holders.
Datatel, Inc.4375 Fair Lakes CourtFairfax, VA 22033(703) 968-9000(800) DATATELwww.datatel.com
Table of Contents
11 Introduction
13 About This Manual13 In This Chapter14 Who Should Read This Manual14 Terms Used in This Manual16 What This Manual Covers17 How This Manual is Organized19 Where to Find More Information
21 About the DataOrchestrator ODS21 In This Chapter22 Understanding the DataOrchestrator ODS24 What Operational Data Stores Cannot Do24 Facts about Operational Data Stores25 Envision Data Mapping on an ODS Target Database26 Architecture of the DataOrchestrator ODS29 Hardware and Software Configuration for the DataOrchestrator
ODS
31 Overview and Installation
33 Overview33 In This Chapter34 Installation Overview for the DataOrchestrator ODS35 Prerequisites for the DataOrchestrator ODS36 Where to Build the Operational Data Stores36 SQL Server37 Oracle38 Preparing the ODS Target Database40 SQL Server Script for Creating the ODS Target Database42 Adding the IDX Filegroup42 Procedure for Adding the IDX Filegroup
Using the DataOrchestrator ODS™, June 18, 2010 5© 2010 Datatel, Inc.
Table of Contents
45 Installing the DataOrchestrator ODS45 In This Chapter46 Before You Begin46 Adding the Optional Module47 Setting Up the ODS Target Database Server and Installing
Software Updates51 Installing the Oracle JDBC Driver51 Procedure for Installing the Oracle JDBC Driver
53 Setting Up DataOrchestrator ODS Parameters
55 Overview of Using the DataOrchestrator ODS55 In This Chapter56 Workflow for Using the DataOrchestrator ODS
59 Defining DataOrchestrator ODS Parameters59 In This Chapter59 Form Used60 Defining Parameters61 Noteworthy Fields on the DOPA Form64 Procedure for Defining DataOrchestrator ODS Parameters
65 Setting Up Targets
67 Defining a Target67 In This Part68 Forms Used70 Defining a Target71 Overview of the DataOrch Target (DOTA) Form72 Noteworthy Fields on the DOTA Form
77 Defining Additional Target Parameters77 In This Chapter77 Form Used78 Defining Additional Target Parameters79 Noteworthy Fields on the DOTP Form80 Procedure for Defining Additional Target Parameters
6 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Table of Contents
81 Defining Source Transforms81 In This Chapter81 Forms Used82 Defining Source Transforms83 Using the DataOrch Source Transform (DOST) Form85 Noteworthy Fields on the DOST Form89 Defining Filter Criteria91 Noteworthy Fields on the DOFC Form93 Viewing Generated Target Transforms94 Noteworthy Fields on the DOGT Form95 Procedure for Defining Source Transforms
97 Defining Target Transforms97 In This Chapter98 Forms Used
100 Defining Target Transforms101 Using the DataOrch Target Transform (DOTT) Form103 Noteworthy Fields on the DOTT Form107 Defining Target Transform Columns109 Noteworthy Fields on the DOTC Form113 Defining Properties for a Target Transform Column114 Noteworthy Fields on the DOPR Form116 Defining Operations for Transform Columns117 The Field Extract (DOFE) Form118 Noteworthy Fields on the DOFE Form120 The Multivalue Operation (DOMV) Form121 Noteworthy Fields on the DOMV Form123 The Pointer Reference (DOFR) Form124 Noteworthy Fields on the DOFR Form126 The Null Value Replacement (DONV) Form127 Noteworthy Fields on the DONV Form129 The String Concatenation (DOCA) Form130 Noteworthy Fields on the DOCA Form132 The Substring Selection (DOSS) Form133 Noteworthy Fields on the DOSS Form135 The Validation Code LookUp (DOVC) Form136 Noteworthy Fields on the DOVC Form138 The Expression Entry (DOEE) Form139 Noteworthy Fields on the DOEE Form141 Procedure for Defining Target Transforms
Using the DataOrchestrator ODS™, June 18, 2010 7© 2010 Datatel, Inc.
Table of Contents
143 Defining and Creating SQL Views143 In This Chapter144 Forms Used145 Defining SQL Views146 Noteworthy Fields on the DOTV Form147 Defining the SQL Select Statement for an SQL View149 Noteworthy Fields on the DOVS Form152 Procedure for Defining SQL Views153 Creating SQL Views154 Procedure for Creating SQL Views Using the DOTV Form
157 Procedure for Defining a Target157 In This Chapter158 Procedure for Defining a Target
163 Refreshing ODS Data
165 Defining and Running an ODS Refresh165 In This Chapter165 Forms Used166 Defining Settings for Refreshing Data169 Noteworthy Fields on the DORE Form174 Defining Additional Parameters for the Refresh175 Noteworthy Fields on the DORP Form177 Procedure for Defining a Refresh
181 Calculating Stored Computed Columns181 In This Chapter182 Forms Used183 Calculating Stored Computed Columns184 Procedure for Activating Stored Computed Columns186 Procedure for Calculating Stored Computed Columns188 Procedure to Update Flags in the STUDENT.TERMS.CC File
8 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Table of Contents
191 Viewing Reports and Refresh History
193 Viewing Errors for a Refresh193 In This Chapter193 Form Used194 Viewing the Error Analysis Report for a Refresh195 Noteworthy Fields on the DOEA Form197 Procedure for Running the Error Analysis Report
199 Viewing Target Transform Data for a Target199 In This Chapter199 Form Used200 Viewing the Transform Summary Report for a Target201 Noteworthy Fields on the DOTS Form202 Procedure for Running the Transform Summary Report
203 Viewing the History of a Refresh203 In This Chapter203 Form Used204 Viewing the History of a Refresh206 Procedure for Viewing the History of a Refresh
207 Maintaining the DataOrchestrator ODS
209 Maintaining Transforms209 In This Chapter209 Form Used210 Maintaining DataOrchestrator ODS Transforms211 Noteworthy Fields on the DOMA Form213 Procedure for Deleting Transforms214 Procedure for Copying Transforms215 Procedure for Renaming Transforms216 Procedure for Copying and Renaming Transforms217 Procedure for Calculating Foreign Keys218 Procedure for Regenerating Source Transforms
Using the DataOrchestrator ODS™, June 18, 2010 9© 2010 Datatel, Inc.
Table of Contents
219 Maintaining SQL Views219 In This Chapter219 Form Used220 Maintaining SQL Views221 Noteworthy Fields on the DOVM Form223 Procedure for Deleting SQL Views224 Procedure for Copying SQL Views225 Procedure for Renaming SQL Views226 Procedure for Copying and Renaming SQL Views
227 Copying Targets to Another Target227 In This Chapter227 Form Used228 Copying One or More Targets to Another Target229 Noteworthy Fields on the DOTY Form230 Procedure for Copying One or More Targets to Another Target
231 Deleting DataOrchestrator ODS Objects231 In This Chapter231 Form Used232 Deleting DataOrchestrator ODS Objects233 Noteworthy Fields on the DOOD Form233 Procedure for Deleting DataOrchestrator ODS Objects
235 Appendices
237 Checklist for Setting Up the DataOrchestrator ODS
237 In This Appendix238 Checklist for Setting Up the DataOrchestrator ODS
241 Frequently Asked Questions241 In This Appendix242 Frequently Asked Questions about the DataOrchestrator ODS
247 Troubleshooting the DataOrchestrator ODS247 In This Appendix248 DataOrchestrator ODS Issues
259 Index
10 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Introduction
About This Manual
In This ChapterThis chapter includes information about the users who should read this manual, terms used in this manual, the product that is covered in this manual, the organization of the topics in this manual, and additional resources for more information.
Table 1 lists the topics in this chapter.
Table 1: Topics in This Chapter
Topic Page
Who Should Read This Manual 14
Terms Used in This Manual 14
What This Manual Covers 16
How This Manual is Organized 17
Where to Find More Information 19
Using the DataOrchestrator ODS™, June 18, 2010 13© 2010 Datatel, Inc.
Introduction: About This Manual
Who Should Read This ManualThis manual is for system administrators and system programmers who set up and support reporting for your institution. No specific knowledge of the structure of source Envision files is assumed.
Terms Used in This ManualThe following are the terms and descriptions that are important to understanding and using the concepts presented in this manual. These terms are used throughout this manual:
Operational Data Store (ODS). An ODS is a repository of data that can deal with a specific area of knowledge and provides easy access to the data for reporting and analysis. It is a snapshot of data captured at a point in time.
Source Database. This refers to the Colleague environment from which you want to create an ODS.
Target Database. This refers to the database where you want to place an ODS.
Target. A target contains all of the necessary information about an ODS target database including:
Connection information.
The set of source and target transforms that should be run to create and update the tables in the ODS.
Current update status information of the target.
Source Transform. A source transform allows you to specify the data to move to an ODS target database by selecting a set of fields and/or computed columns from a single Colleague file. The selected data will be structured on the target database in one or more tables in the same way as the standard SQL representation for that data in a Colleague environment. Source transforms do not allow you to make any data transformations when the data is populated.
Target Transform. A target transform allows you to specify the format of the data in a table on the ODS target database, the Colleague source data elements that will populate each column, as well as any data transformation operations to be run when the data is populated. Target transforms are a powerful tool for the creation of ODS tables that are optimized for reporting purposes, and provide greater flexibility in defining how the data is structured on the ODS.
14 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Terms Used in This Manual
Refresh. The refresh is the specification for an executable process that refreshes a subset of the transforms associated with a target.
Full Refresh. A full refresh means that all records of the source file for the transform you choose will be processed through the transform and updated on the ODS target database.
Incremental Refresh. An incremental refresh means that only those records of the source file that changed since the last refresh will be processed through the transform.
Reporting Data Access Server (RDAS). A DMI Listener role defined for use with the DataOrchestrator ODS. This DMI Listener role includes a set of transactions for bulk data management.
SQL View. An SQL view combines data from various physical tables where it is stored so that the result appears to be data from a single table. As a result, a report or query is easier to build because the user creating the report does not need to know the tables where data elements are stored or their join relationships.
Using the DataOrchestrator ODS™, June 18, 2010 15© 2010 Datatel, Inc.
Introduction: About This Manual
What This Manual CoversThis manual discusses the DataOrchestrator ODS™, which provides the ability to create and maintain operational data stores, which are individual data stores comprised of a defined subset of information from Colleague.
The information in the operational data stores can be kept up-to-date through periodic refreshes of current data from the Colleague source database. The operational data stores can be used as a source for reporting against this Colleague data and can also be used as a source for data extracts for propagation into data warehouse systems.
The chapters in this manual contain the information and procedures for installing the software for the DataOrchestrator ODS, as well as implementing and using the DataOrchestrator ODS.
In this manual you will find high-level installation and setup information, and references to more detailed installation documentation. This documentation provides information on how to perform the following tasks for implementing the DataOrchestrator ODS and operational data stores.
Install the following software components:• DataOrchestrator ODS, Envision, and DMI software updates.• An additional DMI Listener for the DataOrchestrator ODS.• JDBC driver for an Oracle target database.
Define the settings on which the DataOrchestrator ODS depends, including the set of source and target transforms for an ODS.
Export and refresh data to the operational data stores.
16 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
How This Manual is Organized
How This Manual is OrganizedTable 2 shows how this manual is organized.
Table 2: Organization of This Manual
Part Chapter Title Summary
Part 1
Introduction
About This Manual Includes information about who should read this manual, terms used in the manual, what this manual covers, how it is organized, and where to find information on related topics.
About the DataOrchestrator ODS
Discusses what the DataOrchestrator ODS is and how the DataOrchestrator ODS can enhance your institution’s reporting.
Part 2
Overview and Installation
Overview Provides an overview of the installation process and the prerequisites for implementing the DataOrchestrator ODS. Also included is information on preparing the database where you want to build operational data stores.
Installing the DataOrchestrator ODS
Provides information about adding the optional module for the DataOrchestrator ODS, retrieving the Datatel software updates, performing the installation steps, and installing the Oracle JDBC driver.
Part 3
Setting Up DataOrchestrator ODS Parameters
Overview of Using the DataOrchestrator ODS
Provides a high-level overview of the suggested workflow for using the DataOrchestrator ODS.
Defining DataOrchestrator ODS Parameters
Provides information on how to set up the parameters for the DataOrchestrator ODS.
Part 4
Setting Up Targets
Defining a Target Provides an overview on setting up and modifying a target to export and transform Colleague source data to an operational data store using the DataOrchestrator ODS.
Defining Additional Target Parameters
Describes how to define the additional parameters for a target.
Defining Source Transforms
Provides an explanation of how to define source transforms for a target.
Defining Target Transforms Provides an explanation of how to define target transforms.
Defining and Creating SQL Views
Describes how to define, create, and maintain SQL views for a target.
Procedure for Defining a Target
Provides the procedure for defining a target.
Using the DataOrchestrator ODS™, June 18, 2010 17© 2010 Datatel, Inc.
Introduction: About This Manual
Part 5
Refreshing ODS Data
Defining and Running an ODS Refresh
Provides information on how to set up a refresh for exporting Colleague source data using the DataOrchestrator ODS, and how to refresh data.
Calculating Stored Computed Columns
Includes information on how to activate, update, and export stored computed columns that are used as source data elements for operational data stores.
Part 6
Viewing Reports and Refresh History
Viewing Errors for a Refresh
Discusses how to view the error analysis report for a previously run refresh.
Viewing Target Transform Data for a Target
Discusses how to view information for target transforms for a specified target.
Viewing the History of a Refresh
Discusses how to view the settings and the export statistics for a previously run refresh.
Part 7
Maintaining the DataOrchestrator ODS
Maintaining Transforms Provides information on how to copy existing target transforms, delete transforms, and rename transforms. Also details how to regenerate source transforms, and create foreign keys for transforms.
Maintaining SQL Views Provides information on how to copy existing SQL views, delete views, and rename SQL views.
Copying Targets to Another Target
Gives information on how to copy all transforms and SQL views from one or more targets to a single destination target.
Deleting DataOrchestrator ODS Objects
Includes information on how to delete existing refreshes, targets, or refresh history records from the source Colleague database.
Appendices Checklist for Setting Up the DataOrchestrator ODS
Provides a quick checklist of the activities to perform to set up the DataOrchestrator ODS.
Frequently Asked Questions
Provides answers to frequently asked questions when using the DataOrchestrator ODS and the data models and views for reporting.
Troubleshooting the DataOrchestrator ODS
Provides suggestions for items to check or steps to take if you encounter issues in using the DataOrchestrator ODS.
Table 2: Organization of This Manual (cont’d)
Part Chapter Title Summary
18 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Where to Find More Information
Where to Find More InformationTable 3 lists additional resources for finding more information.
Table 3: Additional Resources
Type of Information Resource
Information on using the DataOrchestrator ODS data models for reporting.
Reporting from the DataOrchestrator ODS Data Models
Information on the mapping of Colleague data to an ODS target database when using DataOrchestrator source transforms. (See page 24.)
Mapping Envision Files for SQL Server and Oracle
Installing a newly licensed optional module and retrieving software updates. (See page 46.)
Updating Colleague Software
Creating a new DMI_DAS. (See page 47.)
Installation Procedures for Colleague Release 18.0
Setting up batch processes to run at scheduled intervals. (See page 183.)
Envision Runtime Administration
How to activate and calculate stored computed columns. (See page 183.)
Stored Computed Columns
Assistance with implementing SQL-based reporting solutions
Datatel Consulting Services. Contact Services Scheduling:
1-800-DATATEL (328-2835)
Technical Support The Datatel Solution Center. Contact the Tools and Technology Team:
1-800-DATATEL (328-2835)
Using the DataOrchestrator ODS™, June 18, 2010 19© 2010 Datatel, Inc.
Introduction: About This Manual
20 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Introduction
About the DataOrchestrator ODS
In This ChapterThis chapter provides information that you should understand about the DataOrchestrator ODS before you use the product to create and use operational data stores for your institution. It addresses the following questions:
What is the DataOrchestrator ODS, and what value does it bring to reporting?
What are the limitations of the operational data stores?
How can you maximize your use of the operational data stores?
Also included is information about Envision data mapping on an ODS target database. Table 4 lists the topics in this chapter.
Table 4: Topics in This Chapter
Topic Page
Understanding the DataOrchestrator ODS 22
Facts about Operational Data Stores 24
Envision Data Mapping on an ODS Target Database 25
Architecture of the DataOrchestrator ODS 26
Hardware and Software Configuration for the DataOrchestrator ODS 29
Using the DataOrchestrator ODS™, June 18, 2010 21© 2010 Datatel, Inc.
Introduction: About the DataOrchestrator ODS
Understanding the DataOrchestrator ODS
The DataOrchestrator ODS was developed to help provide increased reporting capabilities to your institution’s staff, managers, and executives to support their institutional effectiveness strategies and long-range planning efforts.
The DataOrchestrator ODS is a powerful tool that allows you to export and transform data from Colleague applications to create operational data stores that encompass specific areas of knowledge.
Ease of access and usability, as well as fast processing of data, are some of the advantages of operational data stores. Also, operational data stores are easy to use and convenient because they are not restricted to the same server or location as the source database. Data can be exported to a target SQL-based (SQL Server or Oracle) database.
The following are some of the benefits derived from the DataOrchestrator ODS:
Operational data stores, as discussed in this manual, enhance your reporting capability. By utilizing an operational data store strategy, Datatel provides a better means for you to report across Colleague applications while using a variety of reporting tools appropriate for your institution.
When extracting data from Colleague, you can transform the data using flexible functions so that the data is available in a form that best suits your reporting purposes.
Specific subsets of Colleague data can be created in the ODS target database to optimize reporting. This means that you no longer have to search through thousands of elements in Colleague to select information for reporting. Instead, you are able to simply utilize the DataOrchestrator ODS to select your own specific files and fields for each operational data store for your reporting purposes.
For the decision makers at your institution, who need consistent and accurate data on a daily basis, operational data stores provide a snapshot of information, or operational picture, from a particular point in time. For example, all the reports created from an operational data store today will
Note: The DataOrchestrator ODS does not support exporting from a Colleague SQL Server platform to an Oracle database as this configuration is not used by Datatel clients.
22 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Understanding the DataOrchestrator ODS
have consistent information from one report to another, irrespective of who created the report.
An operational data store can be used to combine data from multiple sources for reporting purposes. You can incorporate data from your non-Colleague systems into an operational data store created using the DataOrchestrator ODS, producing an integrated reporting platform.
You have complete flexibility in determining how often operational data stores should be refreshed. A decision may be reached between the technical staff and the various professional staff at your institution to determine how fresh the data needs to be for reporting from each operational data store.
You have the ability to do full or incremental refreshes of data, and you can also set up the process handler to automatically refresh this data.
Operational data stores improve performance of the transactional database by reducing the load on that database because information for reporting is derived from snapshots of data on another database server.
Your institution may have specific needs that might necessitate adding or removing files or data elements from an operational data store. The DataOrchestrator ODS allows you to modify the set of files and fields that are exported to an operational data store and how this data is transformed, resulting in a customized reporting solution for your institution.
Using the DataOrchestrator ODS™, June 18, 2010 23© 2010 Datatel, Inc.
Introduction: About the DataOrchestrator ODS
What Operational Data Stores Cannot Do
In addition to understanding what an operational data store can do for your institution, it is also important to understand what it is not designed to do.
An operational data store is not a data warehouse; it is an independent database containing subsets of application data. The information in a data warehouse is highly optimized and aggregated for analytic reporting, and contains a complete historical record of institutional data over time. However, an operational data store retains the level of detail of the data structures in the transactional database and does not contain historical data beyond what is available in the transactional database. In addition, the data in an operational data store is not intended to be used for transactional purposes, such as up-to-the-minute reporting. Operational data store information is taken at a point in time, and the information may not be the most current.
Facts about Operational Data Stores
To further understand the nature of operational data stores and to maximize their utility, consider the following facts:
An operational data store includes static data, not dynamic or live data. The data is captured at the time the data is exported to the operational data store. Thus, it is a snapshot of data at a point in time. The operational data store is updated with new data each time the refresh is run. Therefore, the resulting reports will consistently reflect the data existing at the time of the operational data store’s update.
The latest transactional data will be selected from the database whenever you export data to the operational data store.
You can set up the Process Handler to export data to an operational data store automatically on a preset schedule.
Note: Datatel recommends that any operational data store that contains stored computed columns be set up to automatically run the following processes: – Update GPA SCC Flags (UGSF) if using the STUDENT.TERMS.CC file – Update Stored Computed Column (USCC) This ensures that the stored computed columns will be updated when the ODS target database is updated. Refer to Envision Runtime Administration for further information on how to set a batch process to run at scheduled intervals.
24 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Understanding the DataOrchestrator ODS
Envision Data Mapping on an ODS Target Database
Data mapping on an ODS target database is different for source transforms and target transforms. For source transforms, the mapping of Colleague data on an ODS target database is identical to the mapping of Colleague data on an R18 SQL Server or Oracle environment. For further information on this mapping, see Mapping Envision Files for SQL Server and Oracle, available on the Datatel website. However, the DataOrchestrator ODS will create on the target database only those tables and columns that are necessary to store the Envision data elements selected for inclusion in an operational data store.
In contrast, target transforms allow a much greater flexibility in the structure of the data on the ODS target database. The number of columns of each target table, as well as their names and data types, can be defined in the way that best fits your institution’s reporting requirements.
Using the DataOrchestrator ODS™, June 18, 2010 25© 2010 Datatel, Inc.
Introduction: About the DataOrchestrator ODS
Architecture of the DataOrchestrator ODS
A high-level overview of the system architecture of the DataOrchestrator ODS is shown in Figure 1 on page 27. The main elements of this architecture are:
Colleague Application Environment. The top of Figure 1 displays the Colleague application environment. This environment is the source of data for an operational data store and controls refreshes of that data to the operational data store. A key element in this environment is the Envision batch process displayed on the right, which controls the execution of target transforms.
DMI_DAS. The middle of Figure 1 displays the DMI communication layer. This is used by the Colleague application environment to communicate with both the source (environment) database and the target database where the operational data store is created. The three DMI transactions shown in this layer are specifically designed to support bulk extracting and loading of data for the DataOrchestrator ODS.
Source and Target Database Servers. The source server (on the bottom left) contains the Colleague environment database. The target server (on the bottom right) contains the ODS target database.
26 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Architecture of the DataOrchestrator ODS
Figure 1: DataOrchestrator ODS Architecture
The directional arrows in Figure 1 show how control and data flows between these elements during the refresh of a target transform. The process for executing a transform is as follows:
In the Colleague application environment, the batch process creates and sends a DAS Bulk Export transaction from the information in the target transform. Using the information specified in this transaction, the DMI_DAS creates a query statement to extract the data necessary for the transaction from the source database. This query statement is in the native query language of the source database (UniQuery for UniData, and SQL for Oracle or Microsoft SQL Server). The query statement runs, using the bulk extract function of the environment database. The results are stored to a delimited flat file.
In the Colleague application environment, the batch process then uses a DAS Streaming Data Transfer transaction to move the delimited flat file from the source server to the target server.
In the Colleague application environment, the batch process then uses a DAS Bulk Load transaction that utilizes the target database’s native bulk load function (BULK INSERT statement for SQL Server, and SQL*Loader for Oracle) to load the data in the delimited flat file to the appropriate table on the target database.
Using the DataOrchestrator ODS™, June 18, 2010 27© 2010 Datatel, Inc.
Introduction: About the DataOrchestrator ODS
This process is the core of the design of the DataOrchestrator ODS. This leverages the bulk extract and load functions of your native database platform, while maintaining the Colleague database-independent architecture where the DMI_DAS layer provides all database platform-specific processing.
28 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Hardware and Software Configuration for the DataOrchestrator ODS
Hardware and Software Configuration for the DataOrchestrator ODS
A high-level overview of the configuration of the hardware and software elements required in a DataOrchestrator ODS installation is shown in Figure 2. This hardware and software infrastructure supports the architecture described in Architecture of the DataOrchestrator ODS on page 26.
As Figure 2 shows, there must be a DMI_DAS Listener located on the source database server, and there must also be a DMI_DAS Listener with the RDAS role located on the target database server. For more information on the DMI RDAS Listener, see Setting Up the ODS Target Database Server and Installing Software Updates beginning on page 47.
Figure 2: DataOrchestrator ODS Hardware and Software Configuration
Using the DataOrchestrator ODS™, June 18, 2010 29© 2010 Datatel, Inc.
Introduction: About the DataOrchestrator ODS
30 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Overview and Installation
Overview
In This ChapterThis chapter provides you with an overview of the setup requirements and installation steps for the DataOrchestrator ODS. In addition, you will find the prerequisites for the DataOrchestrator ODS, as well as considerations for building your resulting operational data stores and for preparing the target database.
Table 5 lists the topics in this chapter.
Table 5: Topics in This Chapter
Topic Page
Installation Overview for the DataOrchestrator ODS 34
Prerequisites for the DataOrchestrator ODS 35
Where to Build the Operational Data Stores 36
Preparing the ODS Target Database 38
Using the DataOrchestrator ODS™, June 18, 2010 33© 2010 Datatel, Inc.
Overview and Installation: Overview
Installation Overview for the DataOrchestrator ODS
This section provides a high-level overview of the setup requirements and installation procedures for the DataOrchestrator ODS.
Consult Table 6 to be sure that your software environment is ready for working with the DataOrchestrator ODS.
Table 6: Setup Requirements and Installation Procedures
Setup Requirement/ ReferenceInstallation Procedure
Check that your institution has the prerequisites needed for the DataOrchestrator ODS.
Prerequisites for the DataOrchestrator ODS on page 35.
Decide where to build the operational data stores.
Where to Build the Operational Data Stores beginning on page 36.
Prepare the target database where the operational data stores will be built.
Preparing the ODS Target Database on page 38.
For SQL Server, you can use a script to create your ODS target database.
SQL Server Script for Creating the ODS Target Database on page 40
If your institution is using a SQL Server target database, you must be sure that the filegroup named “IDX” is present in your target database.
Adding the IDX Filegroup on page 42
Add the optional module for the DataOrchestrator ODS.
Adding the Optional Module beginning on page 46.
Retrieve the required software updates. Setting Up the ODS Target Database Server and Installing Software Updates beginning on page 47.
Install and configure the target database DMI_DAS, update DAS roles, and install software updates.
Setting Up the ODS Target Database Server and Installing Software Updates on page 47.
If the target database is Oracle, install the Oracle JDBC driver on the ODS target database. If the source environment is UniData and the target database is Oracle, also install the Oracle JDBC driver on the source environment.
Installing the Oracle JDBC Driver on page 51.
34 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Prerequisites for the DataOrchestrator ODS
Prerequisites for the DataOrchestrator ODS
Table 7 lists the prerequisites that your system must meet before you install the software updates for the DataOrchestrator ODS.
Table 7: Prerequisites for the DataOrchestrator ODS
Prerequisite Comment
Colleague R18 Make sure Colleague is current on all software updates. If you need help, contact the Datatel Solution Center.
UniData 7.1 For UniData, you must have the source Colleague database on UniData version 7.1 or later. For information about installing UniData 7.1, see the installation procedures provided by IBM.
SQL Server 2005 or SQL Server 2008 You must have SQL Server 2005 or SQL Server 2008. Other releases are not supported.
Oracle Database 11g or 10g Release 2 You must have Oracle Database 11g or 10g Release 2. Other releases are not supported.
Datatel Messaging Interface (DMI) with Data Access Server (DAS)
Use the DMI_DAS installed in the Colleague application environment on the source (Colleague) database server.
Ports used by the DMI Listeners must be open between the Colleague and ODS database servers.
Open ports are required so that the Colleague database server DMI_DAS can communicate with the ODS database server DMI RDAS.
The port used by the Datatel daemon on the ODS target database server must be open to the client PC using SA Valet in order to install software updates.
The port for the daemon must be open in order to manage and update the DMI RDAS through SA Valet.
Using the DataOrchestrator ODS™, June 18, 2010 35© 2010 Datatel, Inc.
Overview and Installation: Overview
Where to Build the Operational Data Stores
The most important thing to think about is how you will use the operational data stores you create. How do you intend to report from them? Will you use Crystal Reports, Safari ReportWriter, Excel, SQL Server, or a data warehouse solution? The following information is intended to help you think through these questions before beginning. Knowing which reporting tool you intend to use can save you time and effort by eliminating the need to do setup work more than once.
SQL Server
Consider the following points for the target database:
Database considerations:• You must have SQL Server 2005 or SQL Server 2008.• You will need to create the SQL Server database in which you want to
build the operational data stores. The process that exports data will create the tables, but cannot create the database itself.
Connection considerations:• Ports used by the DMI Listeners must be open between the Colleague and
ODS servers.• The port used by the Datatel daemon on the ODS server must be open to
the client PC using SA Valet in order to install software updates.
Third-party tool considerations:• None
Note: Do not use an existing Colleague database as a target database for an operational data store. If you try to do this, you will receive an error message when you run the refresh.
Note: This section assumes a high level of general technical knowledge; no detailed explanations of the technology are provided. This section is intended as a quick look at what will be required.
36 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Where to Build the Operational Data Stores
Oracle
Consider the following points for the target database:
Database considerations:• You must have Oracle Database 10g Release 2 or later. • You will need to create the Oracle instance in which you want to build the
operational data stores. The process that exports data will create the tables, but cannot create the instance itself.
Connection considerations:• Ports used by the DMI Listeners must be open between the Colleague and
ODS servers.• The port used by the Datatel daemon on the ODS server must be open to
the client PC using SA Valet in order to install software updates.
Third-party tool considerations:• None
Note: The DataOrchestrator ODS does not support exporting from a Colleague SQL Server platform to an Oracle database as this configuration is not used by Datatel clients.
Using the DataOrchestrator ODS™, June 18, 2010 37© 2010 Datatel, Inc.
Overview and Installation: Overview
Preparing the ODS Target Database
Select the database where you want to build the operational data stores. The operational data stores can be stored in any existing or new target SQL-based (SQL Server or Oracle) database.
Make sure a valid target database exists and note the name of that target database.
Make sure that a database user exists for the DataOrchestrator ODS to use for accessing the target database. This user must have full administrative privileges to create database tables and to create and edit data in the target database.
If you are using SQL Server as the target database, this user must have been created with SQL Server Authentication (not Windows Authentication). In addition, you must set the authentication of your SQL Server to “SQL Server and Windows.”
To do this, access Microsoft SQL Server Management Studio. Select the SQL Server and Windows Authentication mode option under Server authentication on the Security tab of the Server Properties window of your SQL Server instance, as shown in Figure 3 on page 39.
ALERT! You should ensure that the target database to which you are exporting Colleague data is adequately secured using the native security controls for your database platform. Some files and fields may contain data that might be considered sensitive in your institution. Access to the ODS target database containing this data should be restricted accordingly. Datatel also highly recommends that you secure access to the DataOrch Target (DOTA) and DataOrch Refresh (DORE) forms. These forms provide a user with the ability to export any data in Colleague to an external database.
Note: You do not have to create any tables. The DataOrchestrator ODS software creates the tables in the target database as part of the export.
Note: If you are using the iStrategy data warehouse solution, you must use a SQL Server target database. You must set the default collation sequence of your database to “SQL_Latin1_General_CP1_CI_AS”.
38 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Preparing the ODS Target Database
Figure 3: Server Properties Window in Microsoft SQL Server Management Studio
Using the DataOrchestrator ODS™, June 18, 2010 39© 2010 Datatel, Inc.
Overview and Installation: Overview
SQL Server Script for Creating the ODS Target Database
For SQL Server, you can use a script to create your ODS target database. Figure 4 on page 41 shows a sample script for using the CREATE DATABASE statement to create a local ODS target database in SQL Server. Note the following:
This script uses directory paths D:\sqlsrvr\data and E:\sqlsrvr\txlog, and assumes a target database name of testods. You must replace these paths with directory structures that exist on your SQL Server file system and replace testods with your preferred database name.
The script places the database files on a separate drive (D:) from the transaction log (E:). This is expected to provide performance benefits, because the SQL server will write to the transaction log at the same time it writes to the database.
The COLLATE statement defines alphanumeric sorting. Use Latin1_General_BIN, as shown in the example (unless you are using the iStrategy data warehouse solution). This sorts data in the same order as your Colleague database.
ALERT! If you are using the iStrategy data warehouse solution, you must use the “SQL_Latin1_General_CP1_CI_AS” collation sequence for the ODS database in this script.
Technical Tip: If you copy the script in Figure 4 on page 41 from the PDF of this manual, you may need to first paste it into a text editor (such as Windows Notepad) to strip out any hidden characters in the PDF text, and then copy it from the text editor to the SQL application. If you copy directly from the PDF to the SQL application, the script may not run properly. However, if you cut and paste as suggested, you may remove valid carriage returns. After pasting the text in the query window, compare your script to Figure 4 on page 41 and enter carriage returns, if necessary.
40 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Preparing the ODS Target Database
Figure 4: Sample CREATE DATABASE Script for Creating an ODS Target Database in SQL Server
CREATE DATABASE testods
ON PRIMARY
( NAME = testods_dat01,
FILENAME = 'D:\sqlsvr\data\testods_dat01.mdf',
SIZE = 500MB,
MAXSIZE = UNLIMITED,
FILEGROWTH = 10% ),
FILEGROUP IDX
( NAME = testods_idx01,
FILENAME = 'D:\sqlsvr\data\testods_idx01.ndf',
SIZE = 250MB,
MAXSIZE = UNLIMITED,
FILEGROWTH = 10% )
LOG ON
( NAME = 'testods_log01',
FILENAME = 'E:\sqlsvr\txlog\testods_log01.ldf',
SIZE = 500MB,
MAXSIZE = UNLIMITED,
FILEGROWTH = 10% )
COLLATE Latin1_General_BIN
GO
Using the DataOrchestrator ODS™, June 18, 2010 41© 2010 Datatel, Inc.
Overview and Installation: Overview
Adding the IDX Filegroup
If your institution is using a SQL Server target database, you must be sure that the filegroup named “IDX” is present in your target database (in addition to the default/primary filegroup that you use). If the IDX filegroup does not exist on your target database, create a filegroup with this name and assign a file to it. The IDX filegroup is used for the indices.
Procedure for Adding the IDX Filegroup
To add the IDX filegroup and assign a file to it, complete the following steps.
Step 1. Right-click on the name of the ODS target database in Management Studio, and select Properties to display the Properties page.
Step 2. Click the Filegroups page. Click Add and enter IDX in the Name field of the first available open line as shown in Figure 5 on page 43, and then click OK to close the Properties page.
Note: If you used the script described in SQL Server Script for Creating the ODS Target Database on page 40, you can skip the rest of this chapter. The IDX filegroup was created in your target database.
42 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Preparing the ODS Target Database
Figure 5: Filegroups Page
Step 3. Open the Properties page again for the database. Click the Files page.
Step 4. Click Add, and then enter the logical name for the new data file in the Logical Name field of the first available line.
Using the DataOrchestrator ODS™, June 18, 2010 43© 2010 Datatel, Inc.
Overview and Installation: Overview
Step 5. Click the value shown in the Filegroup field. A dropdown list of available Filegroups is displayed. Select IDX as shown in Figure 6.
Figure 6: Files Page
Step 6. Enter an initial space allocation for the new file in the Initial Size (MB) column.
Step 7. Check the Autogrowth column and adjust it as needed, as the default of 1 MB is probably not sufficient.
Step 8. Check the Path column to confirm that this is the path where you want the index file created.
Step 9. Click OK to save your updates.
44 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Overview and Installation
Installing the DataOrchestrator ODS
In This ChapterThis chapter provides information about adding the optional module, setting up the ODS target database server, and performing additional installation steps.
Table 8 lists the topics in this chapter.
Table 8: Topics in This Chapter
Topic Page
Adding the Optional Module 46
Setting Up the ODS Target Database Server and Installing Software Updates
47
Installing the Oracle JDBC Driver 51
Using the DataOrchestrator ODS™, June 18, 2010 45© 2010 Datatel, Inc.
Overview and Installation: Installing the DataOrchestrator ODS
Before You BeginTable 9 lists the tasks that must be complete before you can continue with the procedures in this chapter.
Adding the Optional ModuleTo add the DODS optional module for the DataOrchestrator ODS, see Updating Colleague Software, available from the Documentation section of the Datatel website, for information on installing newly licensed optional modules.
Table 9: Before You Begin
Task Reference
License the DODS optional module from Datatel.
Datatel client sales.
Make sure you have the prerequisites for the DataOrchestrator ODS.
Prerequisites for the DataOrchestrator ODS on page 35.
Know where to build your operational data stores.
Where to Build the Operational Data Stores on page 36.
Have the target database prepared. Preparing the ODS Target Database on page 38.
46 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up the ODS Target Database Server and Installing Software Updates
Setting Up the ODS Target Database Server and Installing Software Updates
To install the DataOrchestrator ODS, perform the following steps to set up the ODS target database and retrieve and install software updates.
Step 1. Install the standard Datatel daemon on the ODS target database server.
Step 2. Use SA Valet to create a new DMI Listener on the ODS target database server. See Installation Procedures for Colleague Release 18.0, available from the Datatel website. See “Procedure for Installing a New DMI Listener.”
When creating the new DMI Listener, in the DMI Listener Roles Selection window, click DBAS and RDAS.
Step 3. Use SA Valet to retrieve the necessary software updates to your local product repository. See Updating Colleague Software (available for downloading from the Documentation section of the Datatel website) for information on retrieving and installing software updates.
Step 4. Install the DMI software update in your environment. This will update all existing DMI Listeners in your Colleague environment and your Colleague local product repository.
Step 5. Install the Envision software updates in your Colleague environment.
Technical Tip: When you create the new Listener, it will inherit the original DAS values of the existing Listener. Be sure to review the Java memory, so that the performance parameters are acceptable for your servers.
Using the DataOrchestrator ODS™, June 18, 2010 47© 2010 Datatel, Inc.
Overview and Installation: Installing the DataOrchestrator ODS
Step 6. Create the ODS target database.
a. For SQL Server, you can use a script to create your ODS target database. For more information, see SQL Server Script for Creating the ODS Target Database beginning on page 40. If you do not use this script, you must be sure that the filegroup named “IDX” is present in your target database. For more information, see Adding the IDX Filegroup beginning on page 42.
b. For Oracle, use the Database Creation Assistant (DBCA) to create your ODS target database. Set the database character set as follows:
• Default database character set: WE8ISO8859P1 • Default national character set: AL16UTF16 - Unicode UTF-16 Universal
character set
Step 7. Create a subdirectory to be used by the DMI RDAS Listener on the target database server to store temporary files for the Bulk Load transaction. The following permissions are needed for this directory:
UNIX Oracle target database. The user who starts up the target DMI RDAS Listener must have read/write permissions.
SQL Server target database. • The Windows SYSTEM user must have read/write permissions.
• The DMI RDAS Listener logs in with SQL Server authentication, so the SQL Server process account must have read/write permissions for the BULK INSERT command.
This directory must have sufficient space allocated for these temporary files. The space needed for temporary files can be significant depending on how you define the target. Datatel recommends that you monitor the space usage in this directory as your refresh runs.
Step 8. If you are using a UniData or Oracle source environment, skip to Step 9. For SQL Server only, ensure that the user who will be running the DataOrchestrator ODS refresh has sufficient permissions on the source database to create SQL views.
If this is a SQL server source environment, you will skip Step 9 through Step 13. You are done with installation.
Note: If the Listener is ever started manually (instead of via SA Valet as a Windows service), then the user who starts the Listener must also have read/write permissions.
48 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up the ODS Target Database Server and Installing Software Updates
Step 9. Create a subdirectory on the source database server for use by the DMI_DAS Listener. This subdirectory will be used to store temporary files from the Bulk Extract transaction. The following permissions are needed for this directory:
UNIX UniData source.• The user logged in to Colleague must have read/write permissions.• The user who starts the source DMI_DAS must have read/write
permissions.
Windows UniData source. • The user logged in to Colleague must have read/write permissions.• The SYSTEM user must have read/write permissions.
UNIX Oracle source.• Oracle needs read/write permissions.• The user who starts the source DMI_DAS must have read/write
permissions.
This directory must have sufficient space allocated for these temporary files. The space needed for temporary files can be significant depending on how you define the target. Datatel recommends that you monitor the space usage in this directory as your refresh runs.
Step 10. Log in to Colleague as the DMI administrative user (for example, dmiadmin).
Step 11. In the UT application, access the DMI Pre-Authenticated Server (DMCC) form. Enter the DMI Admin password.
Note: If the Listener is ever started manually (instead of via SA Valet as a Windows service), then the user who starts the Listener must also have read/write permissions.
Using the DataOrchestrator ODS™, June 18, 2010 49© 2010 Datatel, Inc.
Overview and Installation: Installing the DataOrchestrator ODS
Step 12. On the DMCC form, enter information for the application server and the Colleague database server.
a. In the Server Name field, enter the name the server is known by in the DMI Registry environment. This is usually the DNS name.
b. In the Enabled field, enter Yes for enabled.
c. In the Domain Name field, enter the fully qualified domain name for this server.
d. In the IP Address field, enter the IP address of the server. (Do not reuse the DNS name.)
Step 13. Save and exit from the DMCC form.
Note: If your institution's Colleague database is UniData, the application server and the Colleague database server are the same server, so you need to enter only one row of information.
50 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Installing the Oracle JDBC Driver
Installing the Oracle JDBC Driver
Procedure for Installing the Oracle JDBC Driver
Step 1. Download the JDBC driver from the Oracle website. See AnswerNet document 4527 for download instructions and the current supported version of the driver.
Step 2. Save the file to your server.
Step 3. Use FTP to transfer the file to the machine where DMI is installed and place it in the lib folder under your home DMI folder. For example, if the DMI folder is /datatel/coll18/production/ods_das, then place the file under the /datatel/coll18/production/ods_das/lib folder.
Step 4. Stop and restart the DMI Listener.
Note: You must be a registered user of the Oracle website to download the Oracle JDBC driver. If your target database is Oracle, you need to install the Oracle JDBC driver on the ODS target database. In addition, if you have a UniData source environment and your target database is Oracle, then you also must install the Oracle JDBC driver on your source environment.
Using the DataOrchestrator ODS™, June 18, 2010 51© 2010 Datatel, Inc.
Overview and Installation: Installing the DataOrchestrator ODS
52 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up DataOrchestrator ODS Parameters
Overview of Using the DataOrchestrator ODS
In This ChapterThis chapter provides a high-level overview of the suggested workflow for using the DataOrchestrator ODS. The purpose of the overview is to give you a an understanding of the steps involved in setting up and maintaining the DataOrchestrator ODS.
Table 10 lists the topic covered in this chapter.
Table 10: Topic in This Chapter
Topic Page
Workflow for Using the DataOrchestrator ODS 56
Using the DataOrchestrator ODS™, June 18, 2010 55© 2010 Datatel, Inc.
Setting Up DataOrchestrator ODS Parameters: Overview of Using the DataOrchestrator ODS
Workflow for Using the DataOrchestrator ODS
Table 11 shows the steps to set up, refresh, and maintain the DataOrchestrator ODS and operational data stores.
Table 11: Steps to Set Up, Refresh, and Maintain the DataOrchestrator ODS and Operational Data Stores
Step Form Reference
1. Define parameters for the DataOrchestrator ODS.
DataOrch Parameters (DOPA)
Defining DataOrchestrator ODS Parameters on page 59.
2. Create a target, which defines the configuration of an operational data store on a target database. This includes defining source and target transforms to refresh on the target database. (Optional) Define and create SQL views as shown.
DataOrch Target (DOTA)
Use detail forms as needed.
Defining a Target beginning on page 67 (all chapters in Part 4).
3. Create a Refresh ID with an associated target. Define parameters to control how the DataOrchestrator ODS refreshes a target database with data from Colleague source files. (Optional) Create SQL views.
DataOrch Refresh (DORE)
Defining and Running an ODS Refresh on page 165.
4. Note: If your institution does not use stored computed columns, skip this step.
Activate and calculate stored computed columns for any ODS target database that contains stored computed columns. If your ODS target database uses the STUDENT.TERMS.CC file, update the flags as shown.
Define Stored Computed Column (DSCC)
Update Stored Computed Column (USCC)
Update GPA SCC Flags (UGSF)
Calculating Stored Computed Columns on page 181.
56 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Workflow for Using the DataOrchestrator ODS
5. Run an error analysis report for any errors encountered when exporting data from a Colleague source file to an ODS target database.
DataOrch Error Analysis (DOEA)
Viewing Errors for a Refresh on page 193.
6. (Optional) Run the Transform Summary report to view information for target transforms associated with a target.
DataOrch Transform Summary (DOTS)
Viewing Target Transform Data for a Target on page 199.
7. (Optional) View historical information for previously run refreshes for a specific target.
DataOrch Refresh History (DORH)
Viewing the History of a Refresh on page 203.
8. (Optional) Copy existing transforms from one target to another, delete transforms, rename target transforms, and copy and rename transforms. Also, regenerate source transforms, and calculate foreign keys for transforms.
DataOrch Transform Main (DOMA)
Maintaining Transforms on page 209.
9. (Optional) Copy existing SQL views from one target to another, delete SQL views, rename SQL views, and copy and rename SQL views.
DataOrch View Maintenance (DOVM)
Maintaining SQL Views on page 219.
10.(Optional) Copy all transforms and SQL views from one or more targets to a single destination target.
DataOrch Target Copy (DOTY)
Copying Targets to Another Target on page 227
11.(Optional) Delete existing refreshes, targets, or refresh history records from the source Colleague database.
DataOrch Object Delete (DOOD)
Deleting DataOrchestrator ODS Objects on page 231.
Table 11: Steps to Set Up, Refresh, and Maintain the DataOrchestrator ODS and Operational Data Stores (cont’d)
Step Form Reference
Using the DataOrchestrator ODS™, June 18, 2010 57© 2010 Datatel, Inc.
Setting Up DataOrchestrator ODS Parameters: Overview of Using the DataOrchestrator ODS
58 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up DataOrchestrator ODS Parameters
Defining DataOrchestrator ODS Parameters
In This ChapterThis chapter describes how to set up the parameters for the DataOrchestrator ODS.
Table 12 lists the topics covered in this chapter.
Form UsedTable 13 lists and describes the form used in this chapter.
Table 12: Topics in This Chapter
Topic Page
Form Used 59
Defining Parameters 60
Procedure for Defining DataOrchestrator ODS Parameters 64
Table 13: Form Used in This Chapter
Form Purpose
DataOrch Parameters (DOPA) Enter setup information for the DataOrchestrator ODS.
Using the DataOrchestrator ODS™, June 18, 2010 59© 2010 Datatel, Inc.
Setting Up DataOrchestrator ODS Parameters: Defining DataOrchestrator ODS Parameters
Defining ParametersUse the DataOrch Parameters (DOPA) form to define parameters that the DataOrchestrator ODS will use when performing refreshes.
Figure 7: DataOrch Parameters (DOPA) Form
60 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Parameters
Noteworthy Fields on the DOPA Form
The fields described in this section are important for setting up parameters for the DataOrchestrator ODS.
Source DMI_DAS Listener Name
Use the Source DMI_DAS Listener Name field to enter the name of the DMI Listener on the source database server.
Source DMI Listener Host Name
The Source DMI Listener Host Name field displays the TCP/IP address or the DNS alias name of the computer where the DMI Listener is installed.
Source DMI Listener Port
The Source DMI Listener Port field displays the port number associated with the DMI Listener.
Termination Error Threshold
Use the Termination Error Threshold field to specify the default termination error threshold for all transforms. When this threshold is reached, data load will stop to the ODS target database table for a specific transform.
Note: If you are using a SQL Server Colleague environment, leave this field blank.
Note: If you are using a SQL Server Colleague environment, this field will be blank.
Note: If you are using a SQL Server Colleague environment, this field will be blank.
Using the DataOrchestrator ODS™, June 18, 2010 61© 2010 Datatel, Inc.
Setting Up DataOrchestrator ODS Parameters: Defining DataOrchestrator ODS Parameters
You can use the error analysis report generated by the DataOrch Error Analysis (DOEA) form to identify the source of errors and correct the source data or modify the transform. For more information on this report, see Viewing the Error Analysis Report for a Refresh on page 194.
Depending on your target database type, this setting defines the following:
SQL Server. This value specifies the number of errors allowed per batch commit of a transform.
Oracle. This value specifies the number of errors allowed for the entire data load of a transform.
Batch Commit Size
Use the Batch Commit Size field to enter the batch commit size for the data load into the ODS target database. This is the number of rows to write and commit to the ODS target database for each batch load. Datatel recommends that you leave the default set at 1000.
Max Errors Saved
Use the Max Errors Saved field to enter the maximum number of error rows to return and store for a transform. An error row is a database row that could not be inserted into the target database, but which also doesn't stop the database from loading subsequent rows. There may be more errors than this number encountered by the refresh, but this is the maximum number stored by the DataOrchestrator ODS for later analysis.
Source Drive Name
For a Windows source environment, use the Source Drive Name field to enter the default source drive name of the temporary file path. If you are running on any other source environment, leave this field blank. This value specifies which drive on the source Colleague database server will store the temporary files created by the refresh.
Note: A refresh exports data to the ODS target database in batches. This setting allows you to control the size of these batches.
62 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Parameters
Source Temporary File Path
Use the Source Temporary File Path field to enter the default directory path on the source Colleague database server where the temporary files, created by the refresh, will be stored. Enter each element in the path on a separate line and omit any slashes (/).
This directory must have sufficient space allocated for these temporary files. The space needed for temporary files can be significant depending on how you define the target. Datatel recommends that you monitor the space usage in this directory as your refresh runs.
The following permissions are needed for this directory:
UNIX UniData. For the directory where the files are being written: • The user logged in to Colleague must have read/write permissions. • The user who starts the source DMI_DAS must have read/write
permissions.
Windows UniData. For the directory where the files are being written: • The user logged in to Colleague must have read/write permissions. • The SYSTEM user must have read/write permissions.
UNIX Oracle. For the directory where the files are being written: • Oracle needs read/write permissions. • The user who starts the source DMI_DAS must have read/write
permissions.
The temporary files in this directory may contain data that might be considered sensitive at your institution. Access to the directory should be restricted accordingly.
Note: If you are using a SQL Server source environment, leave this field blank.
Note: If the Listener is ever started manually (instead of via SA Valet as a Windows service), then the user who starts the Listener must also have read/write permissions.
Using the DataOrchestrator ODS™, June 18, 2010 63© 2010 Datatel, Inc.
Setting Up DataOrchestrator ODS Parameters: Defining DataOrchestrator ODS Parameters
Procedure for Defining DataOrchestrator ODS Parameters
Step 1. From the Envision Runtime (UT) application, access the DataOrch Parameters (DOPA) form.
Step 2. Enter the parameters for the DataOrchestrator ODS. Refer to online help for more information.
Step 3. Save and exit from the DOPA form.
64 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up Targets
Defining a Target
In This PartThis Part describes how to create and modify a target, which defines the configuration of an operational data store on a target database.
Table 14 lists the topics covered in this Part.
Table 14: Topics in This Part
Topic Page
Forms Used 68
Defining a Target 70
Defining Additional Target Parameters 77
Defining Source Transforms 81
Defining Target Transforms 97
Defining Target Transform Columns 107
Defining Operations for Transform Columns 116
Defining and Creating SQL Views 143
Procedure for Defining a Target 157
Using the DataOrchestrator ODS™, June 18, 2010 67© 2010 Datatel, Inc.
Setting Up Targets: Defining a Target
Forms UsedTable 15 shows the forms used in this Part.
Table 15: Forms Used to Define a Target
Form Purpose
DataOrch Target (DOTA) Create and maintain a target, including the source and target transforms to refresh on a target database. Also view data on the status of this target.
Additional Target Parameters (DOTP)
Enter additional parameters for a target.
DataOrch Filter Criteria (DOFC) Enter a MIOSEL expression to limit the records of a file processed through a transform. Also, enter any file suite instances to be included when the transform is refreshed.
DataOrch Target Views (DOTV) Define SQL views for a target and view information about their statuses.
DataOrch View Spec (DOVS) Define the select statement, column documentation, and materialized view parameters for an SQL view.
For Source Transforms only
DataOrch Source Transform (DOST) Define the fields and computed columns of a file to refresh on the target database.
Generated Target Transforms (DOGT)
View the target transforms that were generated to implement a source transform.
For Target Transforms only
DataOrch Target Transform (DOTT) Define a target transform that extracts data from Colleague and creates a table containing that data on the target database.
Transform Columns (DOTC) Specify the output columns for a target transform. Each target column is defined by an input field and an optional transform operation that produces the value to store in the corresponding target database column.
Column Properties (DOPR) Define properties for a transform column.
68 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Forms Used
For Target Transform Columns only - Optional Operations
Field Extract (DOFE) Define a field extract operation. This returns subfields of the data in the source field.
Multivalue Operation (DOMV) Define a multivalue operation. Either the Multivalue Position or the Multivalue Aggregation operation can be performed on a source field.
Pointer Reference (DOFR) Define a pointer reference operation. This references a field in a file, which is pointed to by a value in the source field.
Null Value Replacement (DONV) Define a null value replacement operation. Either the Null Replacement or the Null Test operation can be performed on a source field.
String Concatenation (DOCA) Define specifications for a concatenation operation for a transform column.
Substring Selection (DOSS) Define a substring selection operation. Set up a starting position of a substring for the source field and the length of the substring.
Validation Code LookUp (DOVC) Define a validation code lookup operation. Enter a validation code table name that corresponds to the value in the source field and choose which field to return from the validation code.
Expression Entry (DOEE) Define an expression to use to return a value for a transform column
Table 15: Forms Used to Define a Target (cont’d)
Form Purpose
Using the DataOrchestrator ODS™, June 18, 2010 69© 2010 Datatel, Inc.
Setting Up Targets: Defining a Target
Defining a TargetUse the DataOrch Target (DOTA) form as the starting point to create or modify a target. Depending on what you want to include in the target, you will also need to access additional forms. Figure 8 shows the additional forms that are available when defining a target.
Figure 8: Forms Available for Defining a Target
DOTA Define Source Transform
Define Target Transform
DOTT (define a transform that extracts data from Colleague)
DOST (select fields and computed columns of a file to refresh)
DOFC (limit records from the file) (optional) DOGT (view target transforms generated from the source transform)
DOFC (limit records from the file) (optional) DOTC (select output columns for a transform)
DOFE (field extract operation) DOMV (multivalue operation) DOFR (pointer reference operation) DONV (null value replacement operation) DOCA (string concatenation operation) DOSS (substring selection operation) DOVC (validation code lookup operation) DOEE (expression entry operation)
Define Operation (optional)
DOPR (define properties for a column)
Define Additional Target Parameters
DOTP (define additional parameters for target)
DOTV (define SQL views)
Define SQL Views (Optional)
DOVS (define select statement, column documentation, and materialized view parameters for an SQL view)
70 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining a Target
Overview of the DataOrch Target (DOTA) Form
The DataOrch Target (DOTA) form is used to define and maintain targets, and to view information about the status of these targets. On the DOTA form, you can enter source and target transforms that you want to refresh to a target database.
Source transforms allow you to specify the data to move to a target ODS database by selecting a set of fields and/or computed columns from a single Colleague file. The selected data will be structured on the target database in the same way as the standard SQL representation for that data in a Colleague environment. Source transforms do not allow you to make any data transformations when the data is moved. Detail to the DataOrch Source Transform (DOST) form to include fields and computed columns that will be sent for a transform.
Target transforms allow you to specify the format of the data on the target ODS database, as well as any data transformation operations to be run when the data is moved. Target transforms are a powerful tool for the creation of ODS tables that are optimized for reporting purposes, and provide greater flexibility in defining how the data is structured on the ODS. Detail to the DataOrch Target Transform (DOTT) form to specify the input fields and computed columns for a transform, and to define how these data elements will be transformed when written to the target ODS database.
You must also detail from the DOTA form to the Additional Target Parameters (DOTP) form to define the Target Drive Name and Target Temporary File Path for the temporary files created by this process.
In addition, you can detail from the DOTA form to the DataOrch Target Views (DOTV) form to define the SQL views associated to this target.
After data has been refreshed to this target database, you can view statistics for the ODS that show its overall status.
Using the DataOrchestrator ODS™, June 18, 2010 71© 2010 Datatel, Inc.
Setting Up Targets: Defining a Target
Figure 9: DataOrch Target (DOTA) Form
Noteworthy Fields on the DOTA Form
The fields described in this section are particularly important for understanding as well as defining and modifying a target.
Status
The Status field displays the status of the target after the most recent refresh.
Oldest Transform Refresh and Latest Transform Refresh
These fields tell you the refresh dates and times for both the oldest and the most recently updated transforms associated with the operational data store. This allows you to see easily the date range of the transforms and determine whether you should take any action based on these dates. For example, if the
72 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining a Target
Oldest Transform Refresh field has an older date than expected, this may alert you that one or more of your transforms is not being updated on a regular basis.
Description
Use the Description field to enter the text description to be displayed for this target.
Database Type
Use the Database Type field to enter the type of database where the operational data store will reside. Select the database type from the drop-down list. You can select SQL Server or Oracle.
Database Host Name
Use the Database Host Name field to enter the IP address of the computer on which the target database resides. You may have an existing DNS alias name created for the computer where the target database resides. Datatel recommends that you use that DNS alias name in the Database Host Name field for ease of use.
Database Port
Use the Database Port field to enter the port number of the server on which the target database resides.
Leave this field blank if your Oracle or Microsoft SQL Server database listener service is using the default port number; otherwise, you must enter a port number in this field.
The database port number is the database listener service port number.
For Oracle the default is either 1521 or 1526.
For Microsoft SQL Server the default is 1433.
Note: This is not the Datatel DMI Listener port number.
Using the DataOrchestrator ODS™, June 18, 2010 73© 2010 Datatel, Inc.
Setting Up Targets: Defining a Target
Database Name/TNS
Use the Database Name/TNS field to enter the target database name for SQL Server or the Transparent Network Substrate (TNS) name for Oracle.
This is the name of the target database to which you want to connect.
For a Microsoft SQL Server target database, enter the database name.
For an Oracle target database, enter the TNS.
Schema Name
Use the Schema Name field to enter the schema name for an Oracle target database.
The schema name is an “owner” string that is prefixed to every target table to uniquely identify the table in a database instance. By default, all tables created on the target database will be created with the schema name of the user who is logged in. To create tables in another schema on the target database, specify the name of the schema in this field.
For a Microsoft SQL Server database, the only schema name that Datatel supports is “dbo”. Therefore, if you selected “SQL Server” in the Database Type field, the Schema Name field is populated with this value and is inquiry only.
Database User ID
Use the Database User ID field to enter a valid user ID for accessing the target database. This ID must have full access to the target database.
Password
Use the Password field to enter the password for accessing the target database.
Enter a valid password for accessing the target database, corresponding to the user ID entered in the Database User ID field. When you enter the password, the value is displayed as asterisks.
74 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining a Target
Temporary File Path
Detail on the Temporary File Path field to access the Additional Target Parameters (DOTP) form to enter the name of the directory that is used for temporary files created on the ODS target database during a refresh. For more information, see Defining Additional Target Parameters beginning on page 77.
SQL Views
To view and define SQL views, detail on the SQL Views field to access the DataOrch Target Views (DOTV) form.
From the DOTV form, you can detail to the DataOrch View Spec (DOVS) form to define the select statement, column documentation and materialized view parameters for an SQL view. For more information, see Defining and Creating SQL Views beginning on page 143.
Source Transforms
Use the Source Transforms table to enter a file name to be included in the ODS target database. File names entered here can be refreshed to an ODS target database by using the DataOrch Refresh (DORE) form. You cannot enter application-hierarchy files here (for example, ST.VALCODES, CORE.VALCODES, etc.). Instead, specify the file suite template file name (for example, VALCODES, etc.).
If you want to select fields to be included, detail on the specific source transform to access the DataOrch Source Transform (DOST) form.
If you specify a file containing stored computed columns in this field, you must run the Update Stored Computed Column (USCC) process to ensure that the stored computed columns are refreshed before the DORE process is run. For more information, see Defining Source Transforms beginning on page 81.
Note: The file entered cannot be a logical view. The file must be a physical file in Colleague.
Note: Deleting a file from the Source Transforms table does not automatically delete the corresponding table on the target database. This must be done manually by the database administrator.
Using the DataOrchestrator ODS™, June 18, 2010 75© 2010 Datatel, Inc.
Setting Up Targets: Defining a Target
Target Transforms
Use the Target Transforms table to enter a name for a target transform to be included in the target database.
The target transform name entered here will be used as the name of the corresponding target table when this transform is refreshed using the DataOrch Refresh (DORE) form.
To define the input fields and column transform operations for the transform, detail on the specific target transform to access the DataOrch Target Transform (DOTT) form. For more information, see Defining Target Transforms beginning on page 97.
Refresh Date/Time
The Refresh Date/Time columns display the date and time when the most recent refresh started for this target for source and/or target transforms.
Status
The Status column displays the status of the most recent refresh for this target for source and/or target transforms.
Columns
For source transforms, the Columns column displays the number of columns included in the source transform. This number is a sum of the columns from the generated target transforms for this source transform (not including the primary key, which is inferred).
For target transforms, the Columns field displays the number of columns included in the target transform, including the primary key.
Note: Deleting a name from the Target Transforms table does not automatically delete the corresponding table on the target database. This must be done manually by the database administrator.
76 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up Targets
Defining Additional Target Parameters
In This ChapterThis chapter describes how to set up the additional target parameters for the DataOrch Target (DOTA) form.
Table 16 lists the topics covered in this chapter.
Form Used Table 17 lists and describes the form used in this chapter.
Table 16: Topics in This Chapter
Topic Page
Form Used 77
Defining Additional Target Parameters 78
Table 17: Form Used in This Chapter
Form Purpose
Additional Target Parameters (DOTP) Enter additional parameters for a target.
Using the DataOrchestrator ODS™, June 18, 2010 77© 2010 Datatel, Inc.
Setting Up Targets: Defining Additional Target Parameters
Defining Additional Target ParametersDetail on the Temporary File Path field from the DataOrch (DOTA) form. This allows you to access the Additional Target Parameters (DOTP) form to define the drive name and temporary file path for a target.
Figure 10: Additional Target Parameters (DOTP) Form
78 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Additional Target Parameters
Noteworthy Fields on the DOTP Form
The fields described in this section are particularly important for defining parameters for a target.
Target Drive Name
Use the Target Drive Name field to enter the target drive name of the temporary file path for a Windows server. If you are running on any other source environment, this field should be left blank.
This field specifies which drive on the ODS target database server will store the temporary files created by the refresh.
Target Temporary File Path
Use the Target Temporary File Path to enter the directory path on the ODS target database server where the temporary files created by the refresh will be stored. Enter each element in the path on a separate line and omit any slashes (/).
This directory must have sufficient space allocated for these temporary files. The space needed for temporary files can be significant depending on how you define the target. Datatel recommends that you monitor the space usage in this directory as your refresh runs.
The following permissions are needed for this directory:
UNIX Oracle. For the directory where the files are transferred to and will be bulk loaded from:• The user who starts up the source DMI_DAS must have read/write
permissions.
Windows SQL Server. For the directory where the files are created (via the Microsoft SQL Server bcp utility) and will be bulk loaded from: • The SYSTEM user must have read/write permissions.
• The DMI_DAS logs in with SQL Server authentication, so the SQL Server process account must have read/write permissions for the BULK INSERT command.
Note: If the Listener is ever started manually (instead of via SA Valet as a Windows service), then the user who starts the Listener must also have read/write permissions.
Using the DataOrchestrator ODS™, June 18, 2010 79© 2010 Datatel, Inc.
Setting Up Targets: Defining Additional Target Parameters
The temporary files in this directory may contain data that might be considered sensitive at your institution. Access to the directory should be restricted accordingly.
Procedure for Defining Additional Target Parameters
Step 1. See Procedure for Defining a Target beginning on page 157.
80 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up Targets
Defining Source Transforms
In This ChapterThis chapter describes how to set up the source transforms for the DataOrch Target (DOTA) form.
Table 18 lists the topics covered in this chapter.
Forms Used Table 19 lists and describes the forms used in this chapter.
Table 18: Topics in This Chapter
Topic Page
Forms Used 81
Defining Source Transforms 82
Using the DataOrch Source Transform (DOST) Form 83
Defining Filter Criteria 89
Viewing Generated Target Transforms 93
Table 19: Forms Used in This Chapter
Form Purpose
DataOrch Source Transform (DOST) Define the fields and computed columns of a file to refresh on the target database.
DataOrch Filter Criteria (DOFC) Enter a MIOSEL expression to limit the records of a file processed through a transform. Also, enter any file suite instances to be included when the transform is refreshed.
Generated Target Transforms (DOGT) View the target transforms that were generated to implement a source transform.
Using the DataOrchestrator ODS™, June 18, 2010 81© 2010 Datatel, Inc.
Setting Up Targets: Defining Source Transforms
Defining Source TransformsTo define a source transform for a target, use the following forms:
DataOrch Source Transform (DOST)
Filter Criteria (DOFC)
To view the target transforms generated from a source transform, detail to the Generated Target Transforms (DOGT) form from the DOST form.
82 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Source Transforms
Using the DataOrch Source Transform (DOST) Form
Detail on a Source Transform on the DataOrch Target (DOTA) form to access the DOST form. Use the DOST form to select the fields and computed columns of a file that will be refreshed to the target database when you use the DataOrch Refresh (DORE) form. You can also detail to the DataOrch Filter Criteria (DOFC) form to specify filters, including file suite instances, to limit the records of the input file of the transform. In addition, you can view data about when the source transform was last refreshed.
You also have the option to:
Manipulate the As of Date field for an incremental update to control the content of the update. The As of Date field must have a value to perform an incremental update, whether or not the transform has an Incremental Date Field.
Set the refresh to truncate input fields for string-type columns to the length specified in the transform.
Specify an error threshold for the transform at which the data export will stop.
The source transform will be run by generating a set of target transforms to create and populate data for each target database table needed to hold the specified input fields of the source file. To view the target transforms generated from this source transform, detail to the Generated Target Transforms (DOGT) form.
When target transforms are generated, their output fields are built with the data types and sizes of the referenced Colleague fields. However, if data types or sizes of any referenced Colleague fields change, the target transforms need to be recreated. To do this, you must use the Regenerate Source Transforms operation on the DataOrch Transform Maint (DOMA) form. For more information, see Maintaining Transforms beginning on page 209.
Using the DataOrchestrator ODS™, June 18, 2010 83© 2010 Datatel, Inc.
Setting Up Targets: Defining Source Transforms
Figure 11: DataOrch Source Transform (DOST) Form
84 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Source Transforms
Noteworthy Fields on the DOST Form
The fields described in this section are particularly important when selecting the fields and computed columns of a file that will be refreshed to the target database.
Select All Fields (Y/N)
Use the Select all Fields field to indicate whether you want to select all available fields for the source transform. Enter Yes to select all available fields (which clears any previous entries in the Include Fields table). Enter No to choose specific fields in the Include Fields table or to select no fields.
Include Fields
Use the Include Fields list to select which fields will be refreshed for the file you chose. The fewer fields you select, the better the refresh performance will be and the less chance errors will be encountered.
If you modify the Include Fields list, then target tables will be recreated during the next refresh. To confirm that you want this to occur, you see the following alert:
Field list change will cause the target table to be recreated. Proceed (Y/N).
Click Yes to proceed or No to return to the Include Fields list and leave the list unchanged. If you click Yes, the As of Date will be deleted to force a full refresh.
Technical Tip: If you use a wildcard LookUp for your fields, you can select multiple elements from the LookUp resolution form.
Using the DataOrchestrator ODS™, June 18, 2010 85© 2010 Datatel, Inc.
Setting Up Targets: Defining Source Transforms
Include Computed Columns
Use the Include Computed Columns list to select which computed columns will be refreshed for the file you chose. Computed columns are recalculated during the refresh, so the fewer computed columns you select, the better the refresh performance will be and the less chance errors will be encountered.
If you modify the Include Computed Columns list, then target tables will be recreated during the next refresh. To confirm that you want this to occur, you see the following alert:
Field list change will cause the target table to be recreated. Proceed (Y/N).
Click Yes to proceed or No to return to the Include Computed Column list and leave the list unchanged. If you click Yes, the As of Date will be deleted to force a full refresh.
Filters/File Suite Instances
Detail on the Filters/File Suite Instances field to access the DataOrch Filter Criteria (DOFC) form in order to enter filter criteria, including file suite instances, for this source transform. Colleague uses this criteria to specify what records from the input file will be processed by the source transform when it is run.
Technical Tip: If you use a wildcard LookUp for your computed columns, you can select multiple elements from the LookUp resolution form. The LookUp list is restricted to only single-valued computed columns that have been generated for database use on the Define Computed Column (DCC) or the Bundle Generation (BNGN) forms. You cannot enter multivalued computed columns as source data elements for a transform.
Note: For a multi-part key file, each CDD element making up the key is included in this list as a separate CDD element.
86 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Source Transforms
Incremental Date Field
The Incremental Date Field displays the MIO-managed change date field for the input file you select for the source transform. There are two ways to control what is selected for an incremental update:
The Incremental Date Field together with the As of Date.
The Incremental Filter Criteria.
As of Date
The As of Date field affects only incremental refreshes. If the transform has an Incremental Date Field, then the records updated on or after this will be in the next incremental refresh. You can also use the As Of Date field in the expression specified for the incremental filter criteria to control what data is included during the next incremental refresh. For more information on using incremental filter criteria, see Defining Filter Criteria on page 89.
If the transform has an Incremental Date field, then all records from the source file that have a MIO-maintained change date on or after this As of Date will be processed by the transform in the next incremental refresh.
The refresh updates this date with the current date when a transform is refreshed to the target database and no errors are encountered. This As of Date will not be modified if any source file records cannot be written to the ODS target database due to data errors. This ensures that the refresh will include these records in a future refresh when the data errors are corrected in the source file.
You can manually change the As of Date to affect which records are selected during an incremental refresh. You may want to do this if you do not want the refresh to attempt to send previously identified error records in future incremental refreshes. However, this will mean that not all records from the source environment will exist on the target database.
Note: Not all files will have a MIO-managed change date field. If this is the case for this file, you see the following message in this field: NO MIO CHANGE DATE FIELD.
Using the DataOrchestrator ODS™, June 18, 2010 87© 2010 Datatel, Inc.
Setting Up Targets: Defining Source Transforms
Truncate Strings (Y/N)
The Truncate Strings field allows you to indicate whether or not you want the refresh to truncate input fields for string-type columns in this transform if the data exceeds the length specified in Colleague for the source field.
Termination Error Threshold
Use the Termination Error Threshold field to enter the error threshold for the selected transform. When the threshold is reached, data load will stop for the transform to the ODS target database table.
Depending on your target database type, this setting defines the following:
SQL Server. The number of errors allowed per batch commit, as specified on the DataOrch Parameters (DOPA) form.
Oracle. The number of errors allowed for the entire data load of this transform.
This setting overrides the default in the Termination Error Threshold field on the DataOrch Parameters (DOPA) form.
View Generated Transforms
Each DataOrchestrator source transform will create and update one or more tables on the ODS target database. Each source transform will be implemented by a set of target transforms, each of which will create and modify one of these target tables. You can detail to the Generated Target Transforms (DOGT) form to view the set of target transforms generated from this source transform.
Note: Truncation applies only to string-type columns. Also, truncation does not apply to the primary key of a transform.
88 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Source Transforms
Defining Filter Criteria
Detail to the Filter Criteria (DOFC) form to enter and verify MIOSEL expressions to use against the input file of a transform to limit the set of records from the file that will be processed through the transform. When saving from the form, the MIOSEL statements (containing the entered expressions against the input file for the transform) will be run to verify their syntax.
The expressions will be used during a full refresh as:
SELECT [file name] [filter criteria to run]
The expressions will be used during an incremental refresh as:
SELECT [file name] [filter criteria to run] AND ( [ incremental date field GE “as of date” ] OR ( [incremental filter criteria to run] ))
For example:
SELECT PERSON WITH SPOUSE AND ( PERSON.CHANGE.DATE GE "12/30/08" OR ( LAST.NAME EQ “Smith” ))
The expressions will be used during an incremental refresh when a transform does not have an Incremental Date Field as:
SELECT [filename][filter criteria to run] AND [incremental filter
criteria to run]
For example:
SELECT PERSON WITH SPOUSE AND LAST.NAME EQ "Smith"
If the input file is a file suite template, you can use the DOFC form to filter the instances of that file suite to include in the refresh. You can also indicate which of these instances to update on an ongoing basis, and which contain static data that will not change and therefore do not need to be updated as part of the refresh.
Note: If you add a new file suite instance, but enter No in the Update Instance field, the file suite instance will not be included when this transform is refreshed. You must run the refresh at least once with the Update Instance field set to Yes.
Using the DataOrchestrator ODS™, June 18, 2010 89© 2010 Datatel, Inc.
Setting Up Targets: Defining Source Transforms
Figure 12: DataOrch Filter Criteria (DOFC) Form
Note: Figure 12 shows an example of a filter criteria for a source transform. However, you can also use a filter criteria for a target transform.
90 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Source Transforms
Noteworthy Fields on the DOFC Form
The fields described in this section are particularly important when limiting the set of records from the file that will be processed through a transform.
File Name
The File Name field displays the name of the file to which the filter criteria will be applied.
Filter Criteria to Run
Use the Filter Criteria to Run field to add or modify a MIOSEL expression you want to use to filter the input records for the transform. The expression must start with the selection clause keyword WITH.
The following keywords are also available to use as a filter criteria to be replaced by a value when the refresh is run:
TODAY(). This returns the current date when the refresh is run. For example:
ACPG.START.DATE LE TODAY()
ASOFDATE(). This returns the As of Date specified for the transform.
If you modify this field, the next refresh of this transform will be a full refresh.
Incremental Filter Criteria to Run
Use the Incremental Filter Criteria to Run field to construct a MIOSEL expression you want to use to filter the input records for the transform for an incremental refresh. If the transform has an Incremental Date Field, then this expression will be added to the Incremental Date Field filter for an incremental refresh with the OR keyword. Otherwise, the Incremental Filter Criteria expression alone will be added to the overall filter with the AND keyword. The expression must start with the selection clause keyword WITH and cannot have any other WITH keywords in its expression (use AND or OR instead).
Note: You cannot enter an SQL statement as the filter criteria.
Using the DataOrchestrator ODS™, June 18, 2010 91© 2010 Datatel, Inc.
Setting Up Targets: Defining Source Transforms
The following keywords are also available to use as a filter criteria to be replaced by a value when the refresh is run:
TODAY(). This returns the current date when the refresh is run. For example:
ACPG.START.DATE LE TODAY()
ASOFDATE(). This returns the As of Date specified for the transform.
File Suite Instances
Use the File Suite Instances list to enter the file suite instance IDs to include when the transform is refreshed.
Update Instance (Y/N)
Use the Update Instance (Y/N) field to indicate whether this instance needs to be updated when the transform is refreshed. Enter Yes to include the instance; otherwise, enter No. If you enter No, the instance is updated only if the columns selected from the file suite template for the transform change. In this case, the data for all file suite instances will be deleted and recreated.
Note: If you are adding a new file suite instance, but enter No in this field, the file suite instance will not be included when this transform is refreshed. You must run the refresh at least once with this field set to Yes.
92 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Source Transforms
Viewing Generated Target Transforms
Detail on the View Generated Transforms field from the DOST form to access the Generated Target Transforms (DOGT) form. The DOGT form allows you to view target transforms that have been generated to implement a specific source transform. Each of the target transforms represents a table on the target database.
Figure 13: Generated Target Transforms (DOGT) Form
Using the DataOrchestrator ODS™, June 18, 2010 93© 2010 Datatel, Inc.
Setting Up Targets: Defining Source Transforms
Noteworthy Fields on the DOGT Form
The fields described in this section are particularly important when viewing target transforms that have been generated to implement a specific source transform.
Generated Target Transforms
The Generated Target Transforms column displays the name of a generated target transform that is associated with a particular source transform. This name represents a table that will be created on the target database.
Status
The Status column displays the status of the most recent refresh of the transform.
Start Date and Start TIme
These columns display the start date and time of the most recent refresh of the target transform.
Total Rows
The Total Rows column displays the count of the total number of rows written to the target database during the most recent run of the target transform.
Deleted
The Deleted column displays the count of the number of rows deleted on the target database during the most recent run of the target transform.
Error
The Error column displays the count of the number of transform output rows that could not be loaded into the target database due to data or execution errors during the most recent run of the target transform.
94 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Source Transforms
End Date and End Time
These columns display the end date and time of the most recent refresh of the target transform.
Procedure for Defining Source Transforms
See Procedure for Defining a Target beginning on page 157.
Using the DataOrchestrator ODS™, June 18, 2010 95© 2010 Datatel, Inc.
Setting Up Targets: Defining Source Transforms
96 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up Targets
Defining Target Transforms
In This ChapterThis chapter describes how to set up the target transforms for the DataOrch Target (DOTA) form.
Table 20 lists the topics covered in this chapter.
Table 20: Topics in This Chapter
Topic Page
Forms Used 98
Defining Target Transforms 100
Using the DataOrch Target Transform (DOTT) Form 101
Defining Target Transform Columns 107
Defining Properties for a Target Transform Column 113
Defining Operations for Transform Columns 116
Using the DataOrchestrator ODS™, June 18, 2010 97© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Forms Used Table 21 lists and describes the forms used in this chapter.
Table 21: Forms Used in This Chapter
Form Purpose
DataOrch Target Transform (DOTT) Define a target transform that extracts data from Colleague and creates a table containing that data on the target database.
DataOrch Filter Criteria (DOFC) Enter a MIOSEL expression to limit the records of a file processed through a transform. Also, enter any file suite instances to be included when the transform is refreshed.
Transform Columns (DOTC) Specify the output columns for a target transform. Each target column is defined by an input field and an optional transform operation that produces the value to store in the corresponding target database column.
Column Properties (DOPR) Define properties for a transform column.
For Target Transform Columns only - Optional Operations
Field Extract (DOFE) Define a field extract operation. This returns subfields of the data in the source field.
Multivalue Operation (DOMV) Define a multivalue operation. Either the Multivalue Position or the Multivalue Aggregation operation can be performed on a source field.
Pointer Reference (DOFR) Define a pointer reference operation. This references a field in a file, which is pointed to by a value in the source field.
Null Value Replacement (DONV) Define a null value replacement operation. Either the Null Replacement or the Null Test operation can be performed on a source field.
String Concatenation (DOCA) Define specifications for a concatenation operation for a transform column.
Substring Selection (DOSS) Define a substring selection operation. Set up a starting position of a substring for the source field and the length of the substring.
98 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Forms Used
Validation Code LookUp (DOVC) Define a validation code lookup operation. Enter a validation code table name that corresponds to the value in the source field and choose which field to return from the validation code.
Expression Entry (DOEE) Define an expression to use to return a value for a transform column
Table 21: Forms Used in This Chapter(cont’d)
Form Purpose
Using the DataOrchestrator ODS™, June 18, 2010 99© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Defining Target TransformsTo define a target transform for a target, use the following forms:
DataOrch Target Transform (DOTT)
Filter Criteria (DOFC)
Transform Columns (DOTC)
You can also choose to detail from the DOTC form to define the following for a target transform column:
Define properties for the column on the Columns Properties (DOPR) form.
Define an operation for the column using one of the following forms:• Field Extract (DOFE)• Multivalue Operation (DOMV)• Pointer Reference (DOFR)• Null Value Replacement (DONV)• String Concatenation (DOCA)• Substring Selection (DOSS)• Validation Code LookUp (DOVC)• Expression Entry (DOEE)
100 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Using the DataOrch Target Transform (DOTT) Form
Detail on a Target Transform on the DataOrch Target (DOTA) form to access the DOTT form. Use the DOTT form to define a target transform that extracts data from Colleague and creates a table containing that data on the target database. You can also view information about when a specific target transform was last refreshed. To view and modify the columns associated with this target transform, detail to the Transform Columns (DOTC) form. In addition, you can detail to the DataOrch Filter Criteria (DOFC) form to specify criteria to filter the records of the source file of the transform.
You also have the option to:
Manipulate the As of Date field for an incremental update to control the content of the update. The As of Date field must have a value to perform an incremental update, whether or not the transform has an Incremental Date Field.
Indicate that the transform needs to be refreshed as a full update.
Set the refresh to truncate input fields for string-type columns in the transform.
Specify an error threshold for the transform at which the data export will stop.
Enter a description of the transform.
Using the DataOrchestrator ODS™, June 18, 2010 101© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Figure 14: DataOrch Target Transform (DOTT) Form
102 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Noteworthy Fields on the DOTT Form
The fields described in this section are particularly important when defining a target transform that extracts data from Colleague and creates a table containing that data on the target database.
Status
The Status field displays the status of the most recent refresh of the transform.
Start Date/Time
These fields display the start date and time of the most recent refresh of the target transform.
End Date/Time
These fields display the end date and time of the most recent refresh of the target transform.
Refresh ID
The Refresh ID field displays the ID of the most recent refresh that processed the target transform.
Total Rows
The Total Rows field displays the count of the total number of rows written to the target database during the most recent run of the target transform.
Deleted
The Deleted field displays the count of the number of rows deleted from the target database during the most recent run of the target transform.
Using the DataOrchestrator ODS™, June 18, 2010 103© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Error
The Error field displays the count of the number of transform output rows on which an error occurred during the most recent run of the target transform.
Source File
The Source File field displays the source file on which the target transform is based. The source file selected will be used to control which fields and computed columns you will be allowed to select when creating the target transform. Also, the source file’s incremental change date field will control the incremental updates of the target transform.
Column Definitions
Detail on the Column Definitions field to access the Transform Columns (DOTC) form in order to view and modify the columns associated with the target transform.
Filters/File Suite Instances
Detail on the Filters/File Suite Instances field to access the DataOrch Filter Criteria (DOFC) form in order to enter filter criteria, including file suite instances, for the target transform. Colleague uses this criteria to specify what records from the source file are processed by the target transform when it is run. For more information on the DOFC form, see Defining Filter Criteria on page 89.
Incremental Date Field
The Incremental Date Field displays the MIO-managed change date field for the source file you selected for the target transform. There are two ways to control what is selected for an incremental update:
The Incremental Date Field together with the As of Date
The Incremental Filter Criteria
Note: The file entered cannot be a logical view. The file must be a physical file in Colleague.
104 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
As of Date
The As of Date field affects only incremental refreshes. If the transform has an Incremental Date Field, then the records updated on or after this will be in the next incremental refresh. You can also use the As Of Date field in the expression specified for the incremental filter criteria to control what data is included during the next incremental refresh. For more information on using incremental filter criteria, see Defining Filter Criteria on page 89.
If the transform has an Incremental Date Field, then all records from the source file that have a MIO-maintained change date on or after this As of Date will be processed by the transform in the next incremental refresh.
The refresh updates this date with the current date when a transform is refreshed to the target database and no errors are encountered. This As of Date will not be modified if any source file records cannot be written to the ODS target database due to data errors. This ensures that the refresh will include these records in a future refresh when the data errors are corrected in the source file.
You can manually change the As of Date to affect which records are selected during an incremental refresh. You may want to do this if you do not want the refresh to attempt to send previously identified error records in future incremental refreshes. However, this will mean that not all records from the source environment will exist on the target database.
Refresh as Full Only (Y/N)
In the Refresh as Full Only (Y/N) field, indicate whether or not this transform can be updated only as full when refreshed using the DataOrch Refresh (DORE) process. For example, a target transform should not be refreshed incrementally in the following instances:
The transform contains computed columns dependent on data from files other than the transform’s source file that change on a regular basis.
The transform has pointer-reference operations and the referenced data values change on a regular basis.
Note: Not all source files will have a MIO-managed change date field. If this is the case for this source file, you see the following message in this field: NO MIO CHANGE DATE FIELD.
Using the DataOrchestrator ODS™, June 18, 2010 105© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
In both these instances, the inclusion of the source records is not triggered in an incremental refresh. For this reason, you would enter Yes for the transform to update only as full during a refresh.
Truncate Strings (Y/N)
Use the Truncate Strings (Y/N) field to indicate whether you want the refresh to truncate input fields for string type columns in the transform if the data exceeds the specified length for the column.
Truncation applies only to string type columns. Also, truncation does not apply to the primary key of a transform.
Termination Error Threshold
Use the Termination Error Threshold field to specify the error threshold for the transform. When the threshold is reached, data load will stop for the transform to the ODS target database table.
Depending on your target database type, this setting defines the following:
SQL Server. The number of errors allowed per batch commit, as specified on the DataOrch Parameters (DOPA) form.
Oracle. The number of errors allowed for the data load of the transform.
This setting overrides the default in the Termination Error Threshold field on the DataOrch Parameters (DOPA) form.
Description
Enter or view a description for this target transform.
Note: If this field is set to Yes, the target transform will be refreshed as a full update regardless of the setting on the DataOrch Refresh (DORE) form in the Update Type or Override Update Type fields.
106 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Defining Target Transform Columns
Detail from the DOTT form to the Transform Columns (DOTC) form to specify the output columns for a target transform. Each target column is defined by an input field and an optional transform operation that will produce the output value to be stored in the corresponding ODS target database column.
The available input fields that can be used are fields and computed columns from the source file of this target transform. Other target columns of this transform can also be used as source fields, in order to nest or combine column operations. However, to be used as a source field for a column, other transform columns must precede the column being defined in the transform column list.
When you create a target transform, be aware of the following cardinality rules that apply to columns marked “Yes” in the Export (Y/N) field:
Only one unassociated multivalue column can be included in a transform unless its data type has been changed to a single value by a transform operation.
Multiple associated data elements may be included only if they are from the same association.
Unassociated multivalues and associated multivalues cannot be included in the same target transform.
When you save from this form, these rules will be checked. If the rules are not followed in the transform, you will receive an error message.
You can also detail from this form to the Column Properties (DOPR) form to specify properties for a column. If you define properties on the DOPR form, then the DOTC form will display an “X” in the Addl field.
Using the DataOrchestrator ODS™, June 18, 2010 107© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Figure 15: Transform Columns (DOTC) Form
108 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Noteworthy Fields on the DOTC Form
The fields described in this section are particularly important when specifying the output columns for a target transform.
Source File
The Source File field displays the source file on which the target transform is based. The source file will be used as the reference from which fields and computed columns can be chosen when creating the target transform. Also, the source file’s incremental change date field will control the incremental updates of the target transform.
Target Key Column
The Target Key Column field displays the name of the target primary key column to be created on the target database for a target transform.
Source Key Field
The Source Key Field field displays the source key field to which the target primary key column is mapped.
Key Size
The Key Size field displays the size of the target key column to be created on the target database.
Key Target Data Type
The Key Target Data Type field displays the data type of the target key column to be created on the target database.
Export Key (Y/N)
The Export Key (Y/N) field displays the export flag for the Target Key Column. The Target Key Column will always have “Yes” as its export flag. This column will always be exported to the target database.
Using the DataOrchestrator ODS™, June 18, 2010 109© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Target Column
Use the Target Column field to enter the name of the column to be created in the target table for this transform in the ODS target database.
To add a timestamp column, enter DO_REFRESH_TIMESTAMP, or use LookUp to select that value. The timestamp column indicates when the refresh for this target was started, not when the rows were created in the target database.
Source Field
Use the Source Field field to enter the source field or computed column from the source file that is the input to the target column. In order to nest or combine operations, the source field can also be another target column, which must be defined in the transform column list preceding its use as a source field.
Size
Use the Size field to enter the size of the target column to be created on the target database.
Target Data Type
The Target Data Type field displays the Envision data type that describes the column to be created on the target database. The data type may be modified based on the transform operation that is selected for the target column.
Note: The computed columns available are restricted to only single-valued computed columns that have been generated for database use on the Define Computed Column (DCC) or Bundle Generation (BNGN) forms. You cannot enter multivalued computed columns as source data elements for a transform.
110 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Operation
Use the Operation field to choose an operation to perform in order to generate an output value for the target column. When you choose an operation, you will detail automatically to the appropriate form to allow you to enter specifications to use with that operation. There are ten operations available:
Field Extract. Colleague opens the Field Extract (DOFE) form. This form allows you to extract a subset of the fields of a source delimited string by specifying the field delimiter and the number of fields to extract from a starting field.
Multivalue Aggregation. (For multivalue target data types only.) Colleague opens the Multivalue Operation (DOMV) form. This form allows you to select from different aggregation operations including: Sum, Count, Min, and Max. If you select this operation, the column’s data type is changed to single-value.
Multivalue Position. (For multivalue target data types only.) Colleague opens the Multivalue Operation (DOMV) form. This form allows you to choose a specific position or the last position in a multivalued field to become a column in the target transform. If you select this operation, the column’s data type is changed to single-value.
Pointer Reference. Colleague opens the Pointer Reference (DOFR) form. This form allows you to specify a pointer field as the source field for the column and use this operation to select data fields from another Envision file using the pointer field as the link. Multivalued target columns are restricted to referencing only single-valued fields from another file.
Null Replacement. Colleague opens the Null Value Replacement (DONV) form. This form allows you to specify a value in the If Null Return field that will be returned if the source field value for the column is null. The data type in this field must match the data type of the source field.
Null Test. Colleague opens the Null Value Replacement (DONV) form. This form allows you to specify a value for both the If Null Return and If Not Null Return fields to create a Null Test Operation to choose the value in the column based on whether the source field value is null or not. The data types of the two fields must match.
String Concatenation. Colleague opens the String Concatenation (DOCA) form. This form allows you to create a concatenation operation using quoted strings, single-valued string type fields from the source file, or single-valued string type target columns from the associated transform defined before this column. All arguments must be single-valued source data elements or literals.
Substring. Colleague opens the Substring Selection (DOSS) form. This form allows you to define a starting position and length of a substring for the source field.
Validation Code LookUp. Colleague opens the Validation Code LookUp (DOVC) form. This form allows you to enter a validation code table name
Using the DataOrchestrator ODS™, June 18, 2010 111© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
that corresponds to the value in the source field and choose which field to return from the validation code.
Expression Entry. Colleague opens the Expression Entry (DOEE) form. This form allows you to enter an expression to use to return a value for this column. Operations can be nested.
Addl
The Additional field displays an “X” if properties were defined for the column on the Column Properties (DOPR) form. These properties include:
Creating an index.
Entering an output format string.
Specifying foreign keys.
Export (Y/N)
Use the Export (Y/N) field to export the target column to the target database. To export the target column, enter Yes; otherwise, enter No.
You would typically enter “No” if a target column is used in another operation, but the column’s data is not needed on the target database. Columns that are marked “No” will not be created on the target database and will not affect the cardinality of the target transform.
112 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Defining Properties for a Target Transform Column
Detail from the DOTC form to the Column Properties (DOPR) form to define properties for a transform column, including the following:
An index for the column.
The column’s output format string.
Foreign keys for the column.
Figure 16: Column Properties (DOPR) Form
Note: You cannot modify the output format string for the key field of a transform.
Using the DataOrchestrator ODS™, June 18, 2010 113© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Noteworthy Fields on the DOPR Form
The fields described in this section are particularly important when defining properties for a target transform column.
Target ID
The Target ID field displays the ID of the target containing the column.
Target Transform
The Target Transform field displays the name of the target transform containing the column.
Target Column
The Target Column field displays the name of the target transform column.
Create Index (Y/N)
Use the Create Index (Y/N) field to indicate whether you want the refresh to create an index in the ODS target database for the transform column. Enter Yes to create an index for this column; otherwise, enter No or leave this field blank.
Output Format String
Use the Output Format String field to enter a column’s output format string, which corresponds to the Conversion String allowable in Envision. The format defined for the column determines the type of column on the ODS target database.
Note: Indexes are created in the ODS target database during a refresh.
114 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Related Transform
In the Related Transform field, enter transform names to relate to the column. A foreign key will be created in the ODS target database from the column to the table created for the related transform.
Create Foreign Key (Y/N)
Use the Create Foreign Key (Y/N) field to indicate whether you want the refresh to create a foreign key in the ODS target database to point to the corresponding related transform in the Related Transform field.
Enter Yes to create a foreign key for the column. Enter No if you do not want a foreign key to be created.
Description
Use the Description field to enter or view a description for this target transform column.
Note: To detect foreign key relationships automatically, choose the “Calculate Foreign Keys” operation in the Operation field on the DataOrch Transform Maint (DOMA) form.
Using the DataOrchestrator ODS™, June 18, 2010 115© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Defining Operations for Transform Columns
Detail from the DOTC form to define an operation for a target transform column using one of the following forms:
Field Extract (DOFE)
Multivalue Operation (DOMV)
Pointer Reference (DOFR)
Null Value Replacement (DONV)
String Concatenation (DOCA)
Substring Selection (DOSS)
Validation Code LookUp (DOVC)
Expression Entry (DOEE)
116 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
The Field Extract (DOFE) Form
Detail from the DOTC form to use this form to define a field extract column operation. Use the field extract operation to return one or more subfields of the data in the source field where the subfields are delimited by the passed character. This form also provides a description field to enter a description of the column.
Figure 17: Field Extract (DOFE) Form
Using the DataOrchestrator ODS™, June 18, 2010 117© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Noteworthy Fields on the DOFE Form
The fields described in this section are particularly important when defining a field extract column operation.
Target ID
The Target ID field displays the ID of the target associated with the target transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column that will have its value determined by the result of the operation.
Source File
The Source File field displays the source file on which the definition of the target transform is based.
Source Field
The Source Field field displays the source field to which the target column is mapped. This is the field against which the field extraction operation will be applied. The source field can also be another target column, as long as it precedes this column in the transform’s column list.
Delimiter
The Delimiter field allows you to enter the delimiter to use as a separator for the source field’s data. If you leave this field blank, Colleague uses a single space as the delimiter. Only one-character entries are accepted, with the exception of the @VM and @SM expressions, which represent value mark and sub-value mark delimiters, respectively.
118 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
For example, an asterisk is used as the delimiter to get the first element of a multi-part key.
Start Field Number
Use the Start Field Number field to enter the starting field number, from the fields as delineated by the delimiter, at which to start extracting fields for the field extract.
Number of Fields to Extract
Use the Number of Fields to Extract field to enter the number of fields to extract, beginning with the start field number.
Description
Use the Description field to enter a description for the target transform column.
Using the DataOrchestrator ODS™, June 18, 2010 119© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
The Multivalue Operation (DOMV) Form
Detail from the DOTC form to use this form to define a multivalue transform operation. Either the Multivalue Position or the Multivalue Aggregation operations can be performed on the source field. The result of either operation is a single value. This form also provides a description field to enter a description of the target column.
Figure 18: Multivalue Operation (DOMV) Form
120 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Noteworthy Fields on the DOMV Form
The fields described in this section are particularly important when defining or viewing a multivalue transform operation.
Target ID
The Target ID field displays the ID of the target associated with the target transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the source file on which the target transform is based.
Source Field
The Source Field field displays the multivalued source field to which the target column is mapped. This is the field to which the aggregation operation will be applied. The source field can also be another target column, which needs to be defined before being used as a source field.
Position in List
The Position in List field displays the position in a multivalued list contained in the source field that the column will reference. You can enter a positive integer or the keyword “last” in this field to select a specific value in the multivalued source field that the column will reference. The result of the operation will be a single value.
Using the DataOrchestrator ODS™, June 18, 2010 121© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Aggregation Operation
Use the Aggregation Operation field to select from a list of available aggregation operations to perform on the source field. The result of the operation will be a single value.
Count. A count of the number of values in the list.
Min. The minimum or lowest value in the list.
Max. The maximum or highest value in the list.
Sum. The sum of all values in the list.
Description
Use the Description field to enter a description for the target transform column.
122 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
The Pointer Reference (DOFR) Form
Detail from the DOTC form to use this form to define a pointer reference column operation. This allows a reference to a field of an Envision file other than the source file defined for the target transform. The pointer reference operation uses the value in the Source Field field as a pointer to a record of this second file, and returns the value of the field named in the operation. Multivalued source fields are restricted to referencing only single-valued fields from another file.
This form also provides a description field to enter a description of the column.
Figure 19: Pointer Reference (DOFR) Form
Note: Only fields from another file can be selected. Also, computed columns from another file are not available for selection.
Using the DataOrchestrator ODS™, June 18, 2010 123© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Noteworthy Fields on the DOFR Form
The fields described in this section are particularly important when defining a pointer reference column operation.
Target ID
The Target ID field displays the ID of the target associated with the target transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the source file for the overall target transform of which the column is a component. This file contains the source field, unless the source file is itself another column defined in the target transform.
Source Field
The Source Field field displays the source field that the target column will use as a pointer to a secondary file. The source field can also be another target column from the transform as long as it is defined before being used as a source field.
124 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
File
Use the File field to enter the file that contains the field referenced by the selected pointer.
Pointer
The Pointer field displays the key to the secondary file. The value of the pointer field selected is used to look up the referenced record in the secondary file.
Field
Use the Field field to enter the field in the secondary file that is being returned by the operation.
Description
Use the Description field to enter a description for the target transform column.
Note: If the Source File is a file suite template file (for example, GLS.FYR, CS.ACYR, etc.), then you can specify a different file suite template file of the same type (for example, .FYR, .ACYR, etc.) in this field. When a refresh is run, the file suite instance string for the file specified in this field will be replaced with the file suite instance being processed.
Using the DataOrchestrator ODS™, June 18, 2010 125© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
The Null Value Replacement (DONV) Form
Detail from the DOTC form to use this form to define a null value replacement operation.
If you selected the Null Replacement operation for this column on the Transform Columns (DOTC) form, you can specify a value in the If Null Return field that will be returned if the source field value for the column is null. However, the data type in this field must match the data type of the source field.
If you selected the Null Test operation for this column on the DOTC form, you can specify a value for both the If Null Return and If Not Null Return fields to create a Null Test Operation to choose the value in the column based on whether the source field value is null or not. However, the data types of the two fields must match. For example, to output a column in a transform indicating whether a Faculty member has a special status, you can specify the Null Test operation with an input field of FAC.SPECIAL.STATUS and enter No in the If Null Return field and Yes in the If Not Null Return field. In addition, you can enter a description of the target column.
Figure 20: Null Value Replacement (DONV) Form
126 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Noteworthy Fields on the DONV Form
The fields described in this section are particularly important when defining a a null value replacement operation.
Target ID
The Target ID field displays the ID of the target associated with the target transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the name of the source file on which the definition of the target transform is based.
Source Field
The Source Field field displays the source field to which the target column is mapped. This is the field against which the operation will be applied. The source field can also be another target column, as long as it precedes the column in the transform’s column list.
If Null Return
Use the If Null Return field to enter a value to replace the source field data when that data is null. You can also enter quoted strings, numeric values, fields from the source file, or target columns from the associated transform defined before the column.
Using the DataOrchestrator ODS™, June 18, 2010 127© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
If Not Null Return
Use the If Not Null Return field to enter a value to replace the source field data when that data is not null. You can also enter quoted strings, numeric values, fields from the source file, and target columns from the associated transform defined before the column.
Description
Use the Description field to enter a description for the target column.
128 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
The String Concatenation (DOCA) Form
Detail from the DOTC form to use this form to define the specifications for a concatenation operation. All arguments must be single-valued source data elements or literals. You can also enter a description of the target column.
Figure 21: String Concatenation (DOCA) Form
Using the DataOrchestrator ODS™, June 18, 2010 129© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Noteworthy Fields on the DOCA Form
The fields described in this section are particularly important when defining a a string concatenation operation.
Target ID
The Target ID field displays the ID of the target associated with the target transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the name of the source file on which the definition of the target transform is based.
Source Field
The Source Field field displays the source field to which the target column is mapped. This is the field against which the operation will be applied. The source field can also be another target column, as long as it precedes the column in the transform’s column list.
Concatenation Operand List
Use the Concatenation Operand List field to enter quoted strings, single-valued string type fields from the source file, or single-valued string type target columns from the associated transform defined before the column.
130 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Description
Use the Description field to enter a description for the target column.
Using the DataOrchestrator ODS™, June 18, 2010 131© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
The Substring Selection (DOSS) Form
Detail from the DOTC form to use this form to define the specifications for a substring operation. You can also enter a description of the target column.
Figure 22: Substring Selection (DOSS) Form
132 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Noteworthy Fields on the DOSS Form
The fields described in this section are particularly important when defining a a substring operation.
Target ID
The Target ID field displays the ID of the target associated with the target transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the name of the source file on which the definition of the target transform is based.
Source Field
The Source Field field displays the source field to which the target column is mapped. This is the field against which the operation will be applied. The source field can also be another target column, as long as it precedes this column in the transform’s column list.
Starting Position
Use the Starting Position field to enter or view the starting position of the substring for the source field.
Using the DataOrchestrator ODS™, June 18, 2010 133© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Substring Length
Use the Substring Length field to enter the length of the string to be returned from the substring of the source field, based on the Starting Position field.
Description
Use the Description field to enter a description for the target column.
134 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
The Validation Code LookUp (DOVC) Form
Detail from the DOTC form to use this form to define the specifications for a validation code lookup operation. This form allows you to enter a validation code table name that corresponds to the value in the source field and choose which field to return from the validation code. You can also enter a description of the target column.
Figure 23: Validation Code LookUp (DOVC) Form
Using the DataOrchestrator ODS™, June 18, 2010 135© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
Noteworthy Fields on the DOVC Form
The fields described in this section are particularly important when defining a a validation code lookup operation.
Target ID
The Target ID field displays the ID of the target associated with the target transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the name of the source file on which the definition of the target transform is based.
Source Field
The Source Field field displays the source field to which the target column is mapped. This is the field against which the operation will be applied. The source field can also be another target column, as long as it precedes the column in the transform’s column list.
Validation Code Name
Use the Validation Code Name field to enter the validation code table name that corresponds to the value in the source field. You can select a validation code from any installed application validation code table. This operation will match the value in the input field with the appropriate code from the validation code table specified in the Validation Code Name field.
136 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Return Field Name
Use the Return Field Name field to select which field you want to return from the validation code for this column. This identifies the information associated with the matching validation code value that the transform will return as the value of the operation. Typically, you will enter description to return the text description of the validation code value.
Description
Use the Description field to enter a description for the target column.
Using the DataOrchestrator ODS™, June 18, 2010 137© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
The Expression Entry (DOEE) Form
Detail from the DOTC form to use this form to define the specifications for an operation manually. You can also enter a description of the target column.
When you save from the DOEE form, the expression is run against its source file to verify the syntax. If the expression fails, then the Export (Y/N) field on the Transform Columns (DOTC) form is set to “No” for the corresponding column to prevent an error from being encountered when the transform is run.
Figure 24: Expression Entry (DOEE) Form
138 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Noteworthy Fields on the DOEE Form
The fields described in this section are particularly important when defining a an expression entry operation.
Target ID
The Target ID field displays the ID of the target associated with the target transform.
Target Transform
The Target Transform field displays the name of the current target transform.
Target Column
The Target Column field displays the name of the target column from the transform that will have its value determined by the result of the operation.
Source File
The Source File field displays the name of the source file on which the definition of the target transform is based.
Source Field
The Source Field field displays the source field selected for the column.
Expression
Use the Expression field to enter the expression to use to return a value for the column.
Note: If you use a literal string argument, the argument must be quoted.
Using the DataOrchestrator ODS™, June 18, 2010 139© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
The following operation names and arguments are available.
Field Extract. SubFieldExtract(FieldName,“Delimiter”,StartFieldNum,NumFieldsTo Extract). For example: SubFieldExtract(INSTITUTIONS_ATTEND_ID,"*",1,1)
Multivalue Aggregation. ListAggregate(FieldName,“Function”). For example:
ListAggregate(SEC.WITHDRAWN.STUDENTS,"COUNT")
Multivalue Position. ListElementByPos(FieldName,PositionInMultiValue). For example:
ListElementByPos(APP.RECRUIT.PURPOSES,3)
Pointer Reference. PointerRef(SourceFieldName,“TargetFileName”,“TargetFieldName”). For example:
PointerRef(INV.TERM,"TERMS","TERM.DESC")
Null Replacement. ReplaceNull(SourceFieldName,AlternateSourceField/String). For example:
ReplaceNull(ARD.REVERSAL.AMT,0)
If Null. IfNull(SourceFieldName,NullReturnField/String,NotNullReturnField/String). For example:
IfNull (INV.SPONSOR.PAYMENT, "MC","SB")
String Concatenation. ConCat(SourceFieldName/String1,SourceFieldName/String2,...SourceFieldName/StringX). For example:
ConCat(STPR_STUDENT,"*",STPR_ACAD_LEVEL)
Substring. SubString(SourceFieldName,StartPosition,Length). For example:
SubString (CITY, 1, 10)
Validation Code LookUp. AssocLookupByPtr(“ValcodeName”,“TargetFile”,“LookupField”, LookupValue,“ReturnField”). For example:
AssocLookupByPtr("GRADUATION.TYPES", "CORE.VALCODES","VAL.INTERNAL.CODE", INSTA.GRAD.TYPE,"VAL.EXTERNAL.REPRESENTATION")
Note: Operations can be nested; operations can be used anywhere that a Field Name is referenced in these syntax descriptions.
Note: You can use only fields from the source file in an expression; you cannot use previously defined fields of the transform.
140 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Target Transforms
Description
Use the Description field to enter a description for the target column.
Procedure for Defining Target Transforms
See Procedure for Defining a Target beginning on page 157.
Using the DataOrchestrator ODS™, June 18, 2010 141© 2010 Datatel, Inc.
Setting Up Targets: Defining Target Transforms
142 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up Targets
Defining and Creating SQL Views
In This ChapterThis chapter describes how to define, create, and maintain SQL views for a target. SQL views are an optional feature that you can utilize, and they can be particularly helpful for your reporting users. Views simplify accessing information in your operational data store and provide predefined queries that can be shared by all reporting users.
You also have the option to indicate if an SQL view should be created as a materialized view. This means that the SQL statement that defines the view is run and all the resulting data rows are stored in a database table. This may significantly improve the response time for queries using the SQL view. However, this also means that the data content of the view must be refreshed whenever the tables providing data for the view are updated. You can either directly update the data content for a materialized view, or set up a materialized view to refresh automatically whenever any of the transforms on which it depends are updated.
Setting up SQL views is a two-step process. You must first define the view, then create the view on the ODS target database. This allows you to control when views are created on the ODS target database.
Table 22 lists the topics covered in this chapter.
Table 22: Topics in This Chapter
Topic Page
Forms Used 144
Defining SQL Views 145
Defining the SQL Select Statement for an SQL View 147
Procedure for Defining SQL Views 152
Creating SQL Views 153
Procedure for Creating SQL Views Using the DOTV Form 154
Using the DataOrchestrator ODS™, June 18, 2010 143© 2010 Datatel, Inc.
Setting Up Targets: Defining and Creating SQL Views
Forms Used Table 23 lists and describes the forms used in this chapter.
Table 23: Forms Used in This Chapter
Form Purpose
DataOrch Target Views (DOTV) Define, create, and maintain SQL views for a target, and view information about the statuses of these SQL views.
DataOrch View Spec (DOVS) Modify the select statement used to create an SQL view, modify the column documentation for the SQL view, or define or modify the materialized view parameters for the SQL view. Test the SQL view select statement on the ODS target database.
144 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining SQL Views
Defining SQL ViewsDetail on the SQL Views field from the DataOrch (DOTA) form. This allows you to access the DataOrch Target Views (DOTV) form where you can view or define the SQL views associated to a target.
When you detail from the DOTA form, you can only define SQL views; you cannot create them, so the following fields are not accessible:
View Creation Selection
Create on Save (Y/N)
Figure 25: DataOrch Target Views (DOTV) Form
Status information is available for the SQL view, including when the SQL view was last modified and the last date it was successfully created on the ODS target database.
Note: When you are ready to create views for a target, you have two options for creating them. For more details, see Creating SQL Views on page 153.
Using the DataOrchestrator ODS™, June 18, 2010 145© 2010 Datatel, Inc.
Setting Up Targets: Defining and Creating SQL Views
Noteworthy Fields on the DOTV Form
The fields described in this section are particularly important for defining SQL views associated to a target.
SQL View Name
In the SQL View Name field, enter a name for the SQL view that you want to associate to the target. The name you enter here will be used as the name of the corresponding SQL view when the view is created on the ODS target database.
Detail from this field to access the DataOrch View Spec (DOVS) form if you want to define or modify the SQL select statement, column documentation, or materialized view parameters for an SQL view.
Modify Date
The Modify Date field shows the date of the last modification of the SQL view.
Success Date
The Success Date field shows the date of the last successful creation of the view on the ODS target database. If the SQL view is modified, or an error is encountered when creating it on the target database, the Success Date will be reset to a null value.
Material View
The Material View field displays “Yes” if the SQL select statement defined on the DataOrch View Specs (DOVS) form will produce a table on the target database. If this field is blank, the SQL select statement will produce a view on the target database. You can detail on the SQL View Name field to change this setting.
Note: Deleting a name from the SQL View Name list does not automatically delete the corresponding SQL view on the target database. This must be done manually by the system administrator.
146 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining SQL Views
Defining the SQL Select Statement for an SQL View
To view and define the SQL statement, column documentation, or the materialized view parameters for an SQL view, detail on an SQL View Name from the DOTV form to the DataOrch View Spec (DOVS) form.
Use the DOVS form to perform the following functions:
View or modify the select statement used to create the SQL view on the ODS target database.
View or modify the description.
View or modify the column documentation for the SQL view.
View or set the option indicating that a view column or set of view columns make up the primary key of the SQL view.
Test the SQL view select statement on the ODS target database.
Set the option indicating the view is a materialized view.
View or modify the list of ODS target transforms on which a materialized view depends.
Note: This information is used to create the primary key on the database table for a materialized view. For SQL views that are not marked as materialized, this is informational only.
Using the DataOrchestrator ODS™, June 18, 2010 147© 2010 Datatel, Inc.
Setting Up Targets: Defining and Creating SQL Views
Figure 26: DataOrch View Spec (DOVS)
Status information is also available for the SQL view, including when the SQL view was last modified and the last date it was successfully created on the ODS target database.
Note: There can be two versions of an SQL view, one with Oracle-specific SQL syntax and one with SQL Server syntax. In the header of the DOVS form, the version is displayed in the Database Type field. This is set up in the Database Type field on the DataOrch Target (DOTA) form for the associated target.
148 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining SQL Views
Noteworthy Fields on the DOVS Form
The fields described in this section are particularly important for defining SQL select statements and column documentation for an SQL view.
SQL Select Statement
Use the SQL Select Statement field to enter an SQL select statement used to create the SQL view on the ODS target database. For example:
SELECT ID, LAST_NAME, BIRTH_DATE FROM PERSON WHERE LAST_NAME EQ “Smith”
The &schema& keyword is also available for use in the SQL Select Statement. For example:
&schema&.TABLE_NAME will be replaced with the schema name (dbo for SQL Server) of the target database when the view is created.
Key
In the Key field, enter Yes to identify a column as being part of a materialized view’s key. By entering “Yes”, this column will be included in the primary key for the table created for the materialized view. If more than one column is marked “Yes”, then a multi-column key will be created on this table, and the order of the columns in the key will be in the same order in which they are listed on the DOVS form.
Description
In the Description field, enter a description for the SQL view.
Note: SELECT must be the first word of the SQL Select Statement.
Note: The set of columns identified as the primary key must contain unique values, or the creation of the key will fail when the materialized view is refreshed.
Using the DataOrchestrator ODS™, June 18, 2010 149© 2010 Datatel, Inc.
Setting Up Targets: Defining and Creating SQL Views
SQL View Columns
In the SQL View Columns field, enter the column names in the SQL select statement. For materialized SQL views, the SQL View Column entries that have the Key field set to “Yes” are used to create the primary key for the table on the database. These field names must exist in the SQL view or the creation of the materialized SQL view will fail. Otherwise, the column names entered here are for your institution's documentation purposes only. No column names will be added to the SQL select statement. Also, the SQL select statement will not be checked to verify the column names exist in the select statement.
Description
Enter a description for the SQL view column.
Create Materialized View (Y/N)
In the Create Materialized View (Y/N) field, enter “Yes” to create a table on the target database from the SQL select statement. Creating a table from the SQL select statement allows you to create indexes and foreign keys on the resulting target table. Enter “No” if you want to create a view on the target database from the SQL select statement.
Dependent Transforms
In the Dependent Transforms field, enter the transforms on which this materialized view is dependent.
Dependent transforms are required when defining a materialized view. If the SQL select statement references a transform, then enter the transform’s name in this list.
Note: When you run a refresh using the DataOrch Refresh (DORE) form, if the refresh processes a transform on which a materialized view is dependent, then the materialized view is automatically updated. This ensures that the most recent data from the transform is included in the materialized view.
150 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining SQL Views
Test SQL View Now
Detail on the Test SQL View Now field to test the SQL view on the ODS target database. The select statement will be validated by creating a temporary view named DATAORCH_TEST_VIEW on the ODS target database, which will be deleted after the view is created.
Note: A select statement must be entered in the SQL Select Statement field in order to attempt to test the SQL view.
Using the DataOrchestrator ODS™, June 18, 2010 151© 2010 Datatel, Inc.
Setting Up Targets: Defining and Creating SQL Views
Procedure for Defining SQL Views
Step 1. Access the DOTV form by detailing on the SQL Views field from the DataOrch (DOTA) form.
Step 2. Define the SQL View Names on the DOTV form. For more details, see online help.
Step 3. If you want to define or modify the select statement, column documentation, and/or materialized view parameters for the SQL view, detail on the SQL View Name field to access the DataOrch View Spec (DOVS) form.
a. Enter the SQL select statement that defines the view and the column documentation on the DOVS form. For more details, see online help.
b. If you want to create a materialized view, enter Yes in the Create Materialized View (Y/N) field, and then enter the transforms on which the materialized view is dependent in the Dependent Transforms field. For more details, see online help.
c. If you want to test the SQL view on the ODS target database, detail on the Test SQL View Now field.
d. When you are finished viewing the results of the test, save from the DOVS form.
Step 4. Save from DOTV form to return to the DOTA form.
152 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Creating SQL Views
Creating SQL ViewsAfter you have defined your SQL views for a target and saved from the DOTV form, you must save from the DOTA form before you can create your SQL views. This saves all of your SQL view definitions for the target.
When you are ready to create the SQL views on the ODS target database, you have two options for creating the views:
Use the DataOrch Refresh (DORE) form. Creating views on this form allows you an easy way to create all the views by simply running a refresh of the target. For more details, refer to Defining and Running an ODS Refresh beginning on page 165.
Access the DOTV form from the Quick Access field. If you access the DOTV form from the Quick Access field (not by detailing from the DOTA form), you can create the SQL views. This form allows you more flexibility in controlling which views you want to create on your ODS target database.
Note: In addition to these two options for creating views, any SQL view defined as a materialized view will be refreshed when any of the target transforms defined in the Dependent Transforms list on the DOVS form is refreshed.
Using the DataOrchestrator ODS™, June 18, 2010 153© 2010 Datatel, Inc.
Setting Up Targets: Defining and Creating SQL Views
Procedure for Creating SQL Views Using the DOTV Form
Use the following steps to create SQL views using the DOTV form.
Step 1. Access the DOTV form by entering DOTV in the Quick Access field. At the DataOrch Target LookUp prompt, enter the target for which you want to create SQL views.
The following fields are now accessible:
View Creation Selection
Create on Save (Y/N)
Figure 27: DataOrch Target Views (DOTV) Form
154 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Creating SQL Views
Step 2. (Optional) Choose an option from the View Creation Selection drop-down list. The following options are available:
All Views. This option updates the Create on Save (Y/N) field to Yes for all SQL views.
Unsuccessful Views. This option updates the Create on Save (Y/N) field to Yes for all SQL views without a corresponding Success Date.
Deselect Views. This option updates the Create on Save (Y/N) field to No for all SQL views.
Step 3. Alternately, you can use the Create on Save (Y/N) field to select views to create. Enter Yes to create one or more SQL views on the ODS target database when saving from this form; otherwise, enter No.
Step 4. Save your entries on the DOTV form.
When you save from the DOTV form, SQL views are created if their associated Create on Save (Y/N) field is set to Yes, and if no errors are encountered.
Note: If an SQL view does not receive errors during creation, then its corresponding Success Date field will be updated to the current date. Otherwise, the Success Date field for the SQL view will be cleared.
Using the DataOrchestrator ODS™, June 18, 2010 155© 2010 Datatel, Inc.
Setting Up Targets: Defining and Creating SQL Views
156 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up Targets
Procedure for Defining a Target
In This ChapterThis chapter describes the procedure for defining a target.
Table 24 lists the topics covered in this Chapter.
Table 24: Topics in This Chapter
Topic Page
Procedure for Defining a Target 158
Using the DataOrchestrator ODS™, June 18, 2010 157© 2010 Datatel, Inc.
Setting Up Targets: Procedure for Defining a Target
Procedure for Defining a Target
Perform the steps below to define or modify a target.
Step 1. Access the DataOrch Target (DOTA) form.
Step 2. Enter the information for this target. See online help for further information.
Step 3. (Required) On the DOTA form, detail on the Temporary File Path field to access the Additional Target Parameters (DOTP) form.
a. (Target Windows server only). Enter the target drive name of the temporary file path.
b. Enter the directory path on the ODS target database server where the temporary files, created by the refresh, will be stored.
ALERT! Datatel highly recommends that you secure access to the DOTA form.
Note: Do not use an existing Colleague database as a target database for an operational data store. If you try to do this, you will receive an error message when you run the refresh.
ALERT! The target database must be refreshed by only one target. If two targets refresh the same target database, the operational data store status information will be inaccurate, the data written to that database can be corrupted, and the refresh may encounter errors.
ALERT! This directory must have sufficient space allocated for these temporary files. The space needed for temporary files can be significant depending on how you define the target. Datatel recommends that you monitor the space usage in this directory as your refresh runs. Be sure that access to this directory is in accordance with the online help for the Temporary File Path field on DOTP. Also, the temporary files in this directory may contain data that might be considered sensitive at your institution. Access to the directory should be restricted accordingly.
158 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
In This Chapter
c. Save and exit from the DOTP form.
Step 4. On the DOTA form, to add a source transform to this target, enter the file name in the Source Transforms table. Source transforms allow you to specify the data to move to an ODS target database by selecting a set of fields and/or computed columns from a single Colleague file.
File names entered here can be refreshed to an ODS target database by using the DataOrch Refresh (DORE) form. If you specify a file containing stored computed columns in this field, you must run the Update Stored Computed Column (USCC) process to ensure that the stored computed columns are refreshed before the DORE process is run. For more information on using stored computed columns with the DataOrchestrator ODS, see Support Solution 6877. (If the target also contains the STUDENT.TERMS.CC file, the Update GPA SCC Flags [UGSF] process should be run before the USCC process.)
Step 5. If you are adding a source transform, Colleague automatically opens the DataOrch Source Transform (DOST) form and can specify which fields and/or computed columns of the file to include in this source transform. To view or modify an existing transform, detail on the specific source transform to access the DOST form. See online help for further information.
a. (Optional) On the DOST form, detail on the Filters/File Suite Instances field to access the Filter Criteria (DOFC) form in order to enter, view, or modify filter criteria for this source transform. Colleague uses this criteria to specify what records from the input file will be processed by the source transform when it is run. See online help for further information. Save and exit from the DOFC form to return to the DOST form.
b. (Optional) On the DOST form, if you want to see the set of tables that will be created on the ODS target database, detail on the View Generated Transforms field to access the Generated Target Transforms (DOGT) form to view the target transforms generated from this source transform. Save and exit from the DOGT form to return to the DOST form.
c. Save and exit from the DOST form.
Step 6. On the DOTA form, to add a target transform to this target, enter a name for the transform in the Target Transforms table. The target transform name you
Note: Deleting a file from the Source Transforms table does not automatically delete the corresponding table on the target database. This must be done manually by the system administrator.
Using the DataOrchestrator ODS™, June 18, 2010 159© 2010 Datatel, Inc.
Setting Up Targets: Procedure for Defining a Target
enter will be used as the name of the corresponding target table when this transform is refreshed using the DORE process.
Target transforms allow you to specify the format of the data on the target database, as well as any data transformation operation to run when the data is moved.
Step 7. If you are adding a target transform, Colleague automatically opens the DataOrch Target Transform (DOTT) form to define the transform. To view or modify an existing transform, detail on the specific target transform to access the DOTT form. See online help for further information.
a. On the DOTT form, detail on the Column Definitions field to access the Transform Columns (DOTC) form. Use the DOTC form to add, view, or modify the output columns for the transform. Each column is defined by an input field (and an optional transformation operation) that produces the output value for the corresponding target database column. See online help for further information.
If you specify a stored computed column as an input field, you must run the Update Stored Computed Column (USCC) process to ensure that the stored computed columns are refreshed before the DORE process is run. For more information on using stored computed columns with the DataOrchestrator ODS, see Support Solution 6877. (If you specify a stored computed column from the STUDENT.TERMS.CC file, the Update GPA SCC Flags [UGSF] process should be run before the USCC process.)
b. (Optional) On the DOTC form, detail on a target column name to access the Column Properties (DOPR) form in order to specify properties for this column. See online help for further information. If you detailed, save and exit from the DOPR form to return to the DOTC form.
c. (Optional) On the DOTC form, you can choose a transformation operation to perform in order to generate the output value for the target database column. To do this, select an operation from the Operation field drop-down list. The following options are available:
Field Extract. Details to the Field Extract (DOFE) form.
Multivalue Aggregation. Details to the Multivalue Operation (DOMV) form.
Multivalue Position. Details to the Multivalue Operation (DOMV) form.
Note: Deleting a name from the Target Transforms table does not automatically delete the corresponding table on the target database. This must be done manually by the system administrator.
160 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
In This Chapter
Pointer Reference. Details to the Pointer Reference (DOFR) form.
Null Replacement. Details to the Null Value Replacement (DONV) form.
Null Test. Details to the Null Value Replacement (DONV) form.
String Concatenation. Details to the String Concatenation (DOCA) form.
Substring. Details to the Substring Selection (DOSS) form.
Validation Code LookUp. Details to the Validation Code LookUp (DOVC) form.
Expression Entry. Details to the Expression Entry (DOEE) form.
See online help for further information on these forms.
d. If you detailed from the Operation field on DOTC to use any of the forms mentioned in the list above, save and exit from the form to return to the DOTC form.
e. Save and exit from the DOTC form to return to the DOTT form.
f. (Optional) On the DOTT form, detail on the Filters/File Suite Instances field to access the DOFC form in order to enter, view, or modify filter criteria for this target transform. Colleague uses this criteria to specify what records from the source file are processed by the target transform when it is run. See online help for further information. Save and exit from the DOFC form to return to the DOTT form.
Step 8. If you have finished defining target transforms, save and exit from the DOTT form to return to the DOTA form.
Step 9. (Optional) On the DOTA form, detail on the SQL View field to define SQL views.
a. Define the SQL View Names for the views on the DataOrch Target Views (DOTV) form.
b. Detail on the SQL View Name field to define the select statement and column documentation for a view on the DataOrch View Spec (DOVS) form.
c. Optionally, on the DOVS form, you can indicate that the SQL view should be created as a materialized view by entering Yes in the Create Materialized View field, and then adding all transforms referenced in the SQL view definition to the Dependent Transforms list. You should also enter Yes in the Key field for the view columns defining the primary key
Using the DataOrchestrator ODS™, June 18, 2010 161© 2010 Datatel, Inc.
Setting Up Targets: Procedure for Defining a Target
for the materialized view, as this will greatly improve the performance of queries against the materialized view.
d. Optionally, you can detail on the Test SQL View Now field to test the SQL view on the ODS target database, then save to return to the DOVS form.
e. Save and exit from both the DOVS and DOTV forms
Step 10. If you have finished defining the target, save and exit from the DOTA form.
Step 11. To export data to the target database, run the DataOrch Refresh (DORE) process. If you want, you can also create all SQL views for the target when running the refresh. For more information, see Defining and Running an ODS Refresh beginning on page 165.
Alternately, you can create the SQL views without running a refresh by accessing the DOTV form from the Quick Access menu. For more information, see Procedure for Creating SQL Views Using the DOTV Form on page 154.
162 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Refreshing ODS Data
Defining and Running an ODS Refresh
In This ChapterThis chapter describes how to define the parameters that control the way the DataOrchestrator ODS refreshes a target database with information from Colleague source files.
Table 25 lists the topics covered in this chapter.
Forms UsedTable 26 shows the forms used in this chapter.
Table 25: Topics in This Chapter
Topic Page
Forms Used 165
Defining Settings for Refreshing Data 166
Defining Additional Parameters for the Refresh 174
Procedure for Defining a Refresh 177
Table 26: Forms Used to Set Up Refreshes for ODS Target Databases
Form Purpose
DataOrch Refresh (DORE) Specify or update settings that control refreshes of a target database. Run a refresh that updates the specified target database.
Additional Refresh Parameters (DORP)
Enter additional parameters for a refresh.
Using the DataOrchestrator ODS™, June 18, 2010 165© 2010 Datatel, Inc.
Refreshing ODS Data: Defining and Running an ODS Refresh
Defining Settings for Refreshing DataUse the DataOrch Refresh (DORE) form to define the settings to be used when doing an initial export, or when updating data from the Colleague source environment to an ODS target database. You can also use the DORE form to run an export of data to this ODS target database and to view the statistics that resulted from the previous export. Each Refresh ID that you set up is associated with one specific target.
When running a refresh, you can specify a default Update Type (Full or Incremental). A Full update processes all records of the source file for the transform. An Incremental update processes only those records of the source file that either:
Changed since the last refresh if the transform has an Incremental Date Field, or
Match the Incremental Filter Criteria.
If you choose Incremental, only the non-computed column data elements in the source file will be checked for updates since the last refresh. This means that if the transform references data elements from other Colleague files (either directly or through the use of computed columns), no dependency checking is done on these elements. Updates to their values will not automatically trigger the inclusion of these input elements in the Incremental update.
You can also choose to include specific source or target transforms in the refresh, or you can include all source and/or target transforms.
If you enter specific transforms (in the Include Source Transforms and/or Include Target Transforms tables), you can choose to override the default Update Type for a transform to either Full or Incremental.
The DataOrch Target (DOTA), DataOrch Source Transform (DOST), and Transform Columns (DOTC) forms allow you to select which fields and computed columns will be refreshed for each transform. The fewer fields (and particularly the fewer computed columns) that are referenced for each transform, the better the refresh performance will be. However, if any of these output fields change for a transform, then a full refresh will be done for the transform, and the target table will be dropped and recreated. Depending on the transform, this could significantly impact the time needed for the subsequent refresh.
166 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Settings for Refreshing Data
You can also specify whether, when this refresh is run, you want to create all of the SQL views associated to the target.
To improve performance of the refresh, you can specify the number of separate concurrent Envision processes to use for a refresh. Using more concurrent Envision processes to run a refresh will result in better performance for the refresh, but will use resources on the application server. This may also impact performance of other Colleague processes.
In addition, you can specify whether you want a refresh to run immediately, or choose just to save the entered settings for this target.
You can view the refresh status on the DORE form. The following statuses may be displayed in the header:
InProc. A transform is currently being refreshed.
Success. The transform was successfully refreshed without any errors.
Warning. The transform or refresh received errors during processing.
Fail. The transform or refresh received fatal errors during processing.
You can also detail from the DORE form to access the Additional Refresh Parameters (DORP) form.
Note: When you run a refresh, if the refresh processes a transform on which a materialized view is dependent, then the materialized view is automatically updated. This ensures that the most recent data from the transform is included in the materialized view.
Technical Tip: When the refresh of an ODS target database takes place, all information stored in Colleague about the target, the transforms and SQL views being refreshed, and the refresh are written to the target database. This information can be queried on the target database using SQL-based tools.
Technical Tip: When you run a full refresh (after the initial refresh), each ODS target database table is cleared only after the data extract and file transfer occurs for the corresponding transform, so that users are minimally impacted.
Note: A maximum of five transforms are allowed to fail before the refresh will be stopped automatically.
Using the DataOrchestrator ODS™, June 18, 2010 167© 2010 Datatel, Inc.
Refreshing ODS Data: Defining and Running an ODS Refresh
Figure 28: DataOrch Refresh (DORE) Form
When you first enter the DORE form, Colleague displays the DataOrch Refresh LookUp prompt where you can enter one of the following:
An ID for a refresh that you want to add and set up.
An ID for an existing refresh for which you want to run a refresh and/or modify the settings.
168 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Settings for Refreshing Data
Noteworthy Fields on the DORE Form
The fields described in this section are particularly important when defining the settings for an ODS refresh.
Target ID
Use the Target ID field to enter one of the following for a new Refresh ID:
The ID for a target that you want to create and set up.
The ID for an existing target that you want to associate with this new Refresh ID.
Also, use this field to view or modify the target information that is associated with an existing Refresh ID. To do this, detail to the DataOrch Target (DOTA) form.
For more information on creating or modifying a target, see Defining a Target beginning on page 67.
RDAS Listener
Use the RDAS Listener field to enter the name of the DMI Listener with the RDAS role on the ODS target database server. This Listener will be used to refresh data from the source Colleague environment to the ODS target database.
DMI Listener Host Name
The DMI Listener Host Name field displays the TCP/IP address or DNS alias name of the computer where the DMI Listener is installed. This is the DMI Listener with the RDAS role on the ODS target database server.
Port
The Port field displays the port number associated to the DMI Listener.
Note: After you specify a Target ID for this Refresh ID and then save, you cannot change the Target ID.
Using the DataOrchestrator ODS™, June 18, 2010 169© 2010 Datatel, Inc.
Refreshing ODS Data: Defining and Running an ODS Refresh
Update Type
When running a refresh, you can specify whether all records of the input files for the listed transforms should be processed (Full), or only those records of the input files that have changed since the last refresh (Incremental).
Transform Selection Option
Use the Transform Selection Option field to select one of the following options for this refresh:
All source transforms. All source transforms from the associated target will be refreshed, using the default in the Update Type field. In addition, you can enter specific target transforms to be refreshed in the Include Target Transforms table (and override the default Update Type).
All target transforms. All target transforms from the associated target will be refreshed, using the default in the Update Type field. You can also enter specific source transforms to be refreshed in the Include Source Transforms table (and override the default Update Type).
All transforms. All source and target transforms from the associated target will be refreshed, using the default defined in the Update Type field.
Specific transforms. You can enter specific source and/or target transforms to be refreshed in the Include Source Transforms and/or Include Target Transforms tables (and override the default Update Types).
Note: Choosing this option clears previous entries in the Include Source Transforms table and the associated Override Update Type table. The tables become inquiry only.
Note: Choosing this option clears previous entries in the Include Target Transforms table and the associated Override Update Type table. The tables become inquiry only.
Note: Choosing this option clears previous entries in the Include Source Transforms and the Include Target Transforms tables, as well as the associated Override Update Type tables. The tables become inquiry only.
170 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Settings for Refreshing Data
Include Source Transforms/Include Target Transforms
If you want to include only specific transforms in the refresh, use the Include Source Transforms and/or the Include Target Transforms tables to list those transforms. Use LookUp to select one or more transforms to process from a list of transforms associated with this target.
Override Update Type
If you enter transforms in the Include Source Transforms and/or Include Target Transform tables, you can also choose to override the default Update Type for a specific transform and control whether all records will be updated or only those records changed since the last update.
However, if a target transform is set to Refresh as Full Only on the DataOrch Target Transform (DOTT) form, the target transform will be refreshed as a full update regardless of the setting in this field.
Check Connectivity
To test connectivity, detail on the Check Connectivity field to log in to the specified target database in order to verify the connection.
Create SQL Views (Y/N)
Use the Create SQL Views (Y/N) field to indicate whether or not you want to create all of the SQL views associated to the target, including any materialized views. Enter Yes to create the SQL views; otherwise, enter No.
Note: To test connectivity, the DataOrch Target (DOTA) form must already be set up with the DMI Listener Host Name, DMI Listener Port, and the associated target information. If this information has not been entered, the following error is displayed: Incomplete Setup.
Note: This setting allows you an easy way to create all the views for a target when you run the refresh. However, you do not need to create all the views each time you refresh a target if none of the definitions for the views have changed. You also have the option of using the DOTV form instead to create views. The DOTV form allows you more flexibility in controlling which views you want to create on your ODS target database. For more information, see Defining and Creating SQL Views beginning on page 143.
Using the DataOrchestrator ODS™, June 18, 2010 171© 2010 Datatel, Inc.
Refreshing ODS Data: Defining and Running an ODS Refresh
Temporary File Path
Detail on the Temporary File Path field to access the Additional Refresh Parameters (DORP) form to enter the source drive name and temporary file path for this refresh.
Run Stored Procedure
In the Run Stored Procedure field, enter Yes to run the stored procedure for this refresh ID.
After the refresh runs, indicate whether you want to run the stored procedure that was created for this refresh ID. Enter Yes to run the stored procedure; otherwise, enter No.
The stored procedure has access to the status information for the DataOrchestrator ODS, which allows you to create a stored procedure that contains more complex processing. The status information for the entire refresh is stored in the DO_REFRESH table, and the refresh status for individual transforms is stored in the DO_TARGET_SPEC table on the target database.
Run Refresh (Y/N)
Use the Run Refresh (Y/N) field to indicate whether or not you want to refresh the selected target database when you save your entries on the DORE form. Enter Yes to run a refresh, or enter No if you want only to save the refresh settings.
Number of Concurrent Threads
Specify the number of concurrent Envision processes to use when running transforms to export data to the target database. Using more threads will usually result in better performance for the refresh, but will use resources on the Colleague database server. This may impact performance of other Colleague processes.
Note: You can create a stored procedure in the target database that can be run after a refresh is complete. This stored procedure must be named for the refresh ID, with _SP appended to the end. For example, if the refresh ID is D01_REFRESH, the stored procedure must be named D01_REFRESH_SP.
172 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Settings for Refreshing Data
ALERT! Using multiple threads for your refresh can significantly affect the system resources available to your Colleague environment. Datatel recommends that you carefully monitor the system resource usage of your Colleague database server while running your ODS refresh and adjust the scheduling of your refresh and the number of threads you assign it accordingly.
Note: When more threads are used, not only are more system resources used as well, but also more UniData licenses, and both of these can affect the performance of other processes.
Using the DataOrchestrator ODS™, June 18, 2010 173© 2010 Datatel, Inc.
Refreshing ODS Data: Defining and Running an ODS Refresh
Defining Additional Parameters for the Refresh
Optionally, you can detail from the DataOrch Refresh (DORE) form by using the Temporary File Path field to go to the Additional Refresh Parameters (DORP) form. The DORP form allows you to define a Source Drive Name and Source Temporary File Path that overrides the corresponding settings on the DataOrch Parameters (DOPA) form for this refresh.
Figure 29: Additional Refresh Parameters (DORP) Form
Technical Tip: The reason you may want to override the corresponding settings on DOPA is because this allows you to limit access to the source temporary file path for this refresh.
174 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Additional Parameters for the Refresh
Noteworthy Fields on the DORP Form
The fields described in this section are particularly important when defining additional parameters for a refresh.
Refresh ID
The Refresh ID field displays the ID of the refresh associated with the parameters.
Source Drive Name
If you use a Windows source environment and use a source database other than SQL Server, use the Source Drive Name field to enter the source drive name of the temporary file path. This value specifies which drive on the source Colleague database server will store the temporary files created by the refresh.
Source Temporary File Path
Use the Source Temporary File Path field to enter the directory path on the source Colleague database server where the temporary files, created by the refresh, will be stored. Enter each element in the path on a separate line and omit any slashes (/).
This directory must have sufficient space allocated for these temporary files. The space needed for temporary files can be significant depending on how you define the target. Datatel recommends that you monitor the space usage in this directory as your refresh runs.
Note: If you are using a SQL Server source database environment, leave this field blank.
Note: This value will override the default Source Drive Name on the DataOrch Parameters (DOPA) form.
Note: If you are using a Windows SQL Server source environment, leave this field blank.
Using the DataOrchestrator ODS™, June 18, 2010 175© 2010 Datatel, Inc.
Refreshing ODS Data: Defining and Running an ODS Refresh
The following permissions are needed for this directory:
UNIX UniData. For the directory where the files are being written: • The user logged in to Colleague must have read/write permissions. • The user who starts the source DMI_DAS must have read/write
permissions.
Windows UniData. For the directory where the files are being written: • The user logged in to Colleague must have read/write permissions. • The SYSTEM user must have read/write permissions.
UNIX Oracle. For the directory where the files are being written: • Oracle needs read/write permissions. • The user who starts the source DMI_DAS must have read/write
permissions.
The temporary files in this directory may contain data that might be considered sensitive at your institution. Access to the directory should be restricted accordingly.
Note: If the Listener is ever started manually (instead of through SA Valet as a Windows service), then the user who starts the Listener must also have read/write permissions.
Note: The path you enter in this field will override the default Source Temporary File Path on the DataOrch Parameters (DOPA) form.
176 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Additional Parameters for the Refresh
Procedure for Defining a Refresh
Perform the steps below to enter settings for a refresh for a specific target, and to run the refresh, if desired.
Step 1. Access the DataOrch Refresh (DORE) form.
Step 2. At the DataOrch Refresh LookUp prompt, either add a new Refresh ID or enter an existing one.
Step 3. If you entered an existing Refresh ID, the Target ID is displayed and cannot be changed. If you are adding a new Refresh ID, enter the ID of a new or existing target in the Target ID field.
If you add a new Target ID, you will automatically detail to the DataOrch Target (DOTA) form so that you can define the new target. For an existing target, you can manually detail to the DOTA form to view or update the target information. For information on defining or updating a target, see Defining a Target on page 67.
For information about the fields on the DOTA form, see online help. When you are finished on the DOTA form, save to return to the DORE form.
Step 4. On the DORE form, enter or modify the settings for refreshing this target. For more information on these fields, see online help.
ALERT! You should ensure that the target database to which you are exporting Colleague data is adequately secured using the native security controls for your database platform. Some files and fields may contain data that might be considered sensitive in your institution. Access to the target database containing this data should be restricted accordingly. Datatel also highly recommends that you secure access to the DORE form. This form provides a user with the ability to export any data in Colleague to an external database.
Note: After you specify a Target ID for this Refresh ID and then save, you cannot change this association.
Using the DataOrchestrator ODS™, June 18, 2010 177© 2010 Datatel, Inc.
Refreshing ODS Data: Defining and Running an ODS Refresh
Step 5. (Optional) To test connectivity, detail on the Check Connectivity field to log in to the specified target database in order to verify the connection. When you are finished checking connectivity, save to return to the DORE form.
Step 6. (Optional) On the DORE form, you can detail on the Temporary File Path field to access the Additional Refresh Parameters (DORP) form to override the source drive name and temporary file path defined on the DOPA form for this refresh. When you are finished entering parameters, save and exit from the DORP form to return to the DORE form.
Step 7. (Optional) In the Create SQL Views (Y/N) field, indicate whether you want to create all of the SQL views associated to the target. Enter Yes to create the SQL views; otherwise, enter No.
Step 8. (Optional) After the refresh runs, indicate whether you want to run the stored procedure that was created for this refresh ID. Enter Yes in the Run Stored Procedure field to run the stored procedure; otherwise, enter No.
Step 9. In the Run Refresh (Y/N) field, enter Yes if you want to refresh the selected target when you save your entries on the DORE form. Enter No if you want only to save the refresh settings.
Step 10. In the Number of Concurrent Threads field, specify the number of threads you want to use for exporting the files to the target. Remember that using more concurrent Envision processes will usually result in better performance for the refresh, but will use resources on the application server. This may impact performance of other Colleague processes.
Step 11. Save from the DORE form.
ALERT! This directory must have sufficient space allocated for these temporary files. The space needed for temporary files can be significant depending on how you define the target. Datatel recommends that you monitor the space usage in this directory as your refresh runs. Be sure that access to this directory is in accordance with the online help for the Temporary File Path field on the DORP form. Also, the temporary files in this directory may contain data that might be considered sensitive at your institution. Access to the directory should be restricted accordingly.
178 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Defining Additional Parameters for the Refresh
Step 12. If you are running a refresh, complete the Peripheral Settings and Process Handler forms according to your preferences.
If you entered “Yes” in the Run Refresh (Y/N) field, the DataOrchestrator ODS refreshes the target database and reports errors on the DataOrch Error Analysis report. For more information on this report, see Viewing Errors for a Refresh on page 193.
Using the DataOrchestrator ODS™, June 18, 2010 179© 2010 Datatel, Inc.
Refreshing ODS Data: Defining and Running an ODS Refresh
180 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Setting Up Targets
Calculating Stored Computed Columns
In This Chapter
This chapter describes how to activate and calculate stored computed columns for any ODS target database that contains stored computed columns.
Table 27 lists the topics covered in this chapter.
Note: To use the iStrategy data warehouse solution with the DataOrchestrator ODS, you must run the processes in this chapter. If your institution does not use any targets containing stored computed columns, you can skip this chapter. Stored computed column files are indicated by having .CC as the last three characters of the file name. Stored computed columns are delivered inactive and must be activated.
Table 27: Topics in This Chapter
Topic Page
Forms Used 182
Calculating Stored Computed Columns 183
Procedure for Activating Stored Computed Columns 184
Procedure for Calculating Stored Computed Columns 186
Procedure to Update Flags in the STUDENT.TERMS.CC File 188
Using the DataOrchestrator ODS™, June 18, 2010 181© 2010 Datatel, Inc.
Setting Up Targets: Calculating Stored Computed Columns
Forms UsedTable 28 lists and describes the forms used in this chapter.
Table 28: Forms Used in This Chapter
Form Purpose
Define Stored Computed Column (DSCC)
Activate stored computed columns.
Update Stored Computed Column (USCC)
Calculate stored computed columns for an ODS target database.
Update GPA SCC Flags (UGSF) Update flags in the STUDENT.TERMS.CC file for the Cumulative GPA by Term.
Note: Use the USCC process for the initial calculation of stored computed columns in this file. For subsequent updates, Datatel recommends running the UGSF process before the USCC process.
182 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Calculating Stored Computed Columns
Calculating Stored Computed ColumnsTo calculate stored computed columns referenced by a target, you first use the Define Stored Computed Column (DSCC) form to activate all stored computed columns, and then use the Update Stored Computed Column (USCC) process to populate the stored computed columns in the source files. No information is available for these stored computed columns until both processes are run against the source files.
You need to activate a given stored computed column on the DSCC form only once. However, Datatel recommends that you run the USCC process every time you run the DataOrch Refresh (DORE) process to export data to an ODS target database containing stored computed columns.
How often you run the USCC process depends on how often your stored computed columns need to be recalculated. You may also consider how frequently reports are run. Based on that you can run the USCC process before running reports to get the most up-to-date data.
Note: If you run the DORE process without running the USCC process, the ODS target database will contain stored computed column data only for the most recent time the USCC process was run.
Technical Tip: If your target contains the STUDENT.TERMS.CC file, use the USCC process for the initial calculation of stored computed columns in this file. For subsequent updates, Datatel recommends that you use the UGSF process (before using the USCC process) to update the STUDENT.TERMS.CC file.
Note: For any ODS target database that contains stored computed columns, Datatel recommends that you set up the process handler to automatically run the USCC process. (If the target also contains the STUDENT.TERMS.CC file, the UGSF process should be set up to run automatically before the USCC process.) This ensures that the stored computed columns will be updated when the ODS target database is updated. Refer to Envision Runtime Administration for further information on how to set up a batch process to run at scheduled intervals.
Using the DataOrchestrator ODS™, June 18, 2010 183© 2010 Datatel, Inc.
Setting Up Targets: Calculating Stored Computed Columns
Procedure for Activating Stored Computed Columns
To activate stored computed columns, follow the steps below. You need to activate a given stored computed column only once.
Step 1. To activate stored computed columns, access the Define Stored Computed Column (DSCC) form in the Envision Tool Kit for the appropriate application. For example, use the Envision Tool Kit for Colleague Student if your institution uses the iStrategy data warehouse solution.
Figure 30: Define Stored Computed Column (DSCC) Form
Step 2. At the Computed Column LookUp prompt, enter the stored computed column that you want to make active.
Note: If you are using Colleague Studio, the functionality of the DSCC form is available using the following path: Entity editor > Attributes > Calculation.
184 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Calculating Stored Computed Columns
For example, the iStrategy data warehouse solution contains the following stored computed columns that you must make active:
APPLCC.READMIT.FLAG (APPLICATIONS.CC file)
STTRCC.TERM.GPA (STUDENT.TERMS.CC file)
STTRCC.TRGR.CUM.GPA (STUDENT.TERMS.CC file)
Step 3. In the Recalculation Method field, select Batch to make this stored computed column active.
Step 4. Update the form to save your work.
Step 5. At the Computed Column LookUp prompt, enter any other stored computed column that you want to make active, or save your work and exit from the form.
Using the DataOrchestrator ODS™, June 18, 2010 185© 2010 Datatel, Inc.
Setting Up Targets: Calculating Stored Computed Columns
Procedure for Calculating Stored Computed Columns
To calculate stored computed columns, follow the steps below. The following example shows how to calculate stored computed columns if your ODS target database contains the APPLICATIONS.CC and STUDENT.TERMS.CC files.
Step 1. Access the Update Stored Computed Column (USCC) form in the appropriate application. If you are in Colleague Student, for example, the following message is displayed:
Scanning ST application for stored computed columns...
The USCC form then lists the names of all ST files with active stored computed columns, as shown in Figure 31.
Figure 31: Update Stored Computed Column (USCC) Form
186 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Calculating Stored Computed Columns
Step 2. In the Files to Recalculate field, remove all files from the list except those associated with the ODS target database. For example, the APPLICATIONS.CC and STUDENT.TERMS.CC files are necessary for the iStrategy data warehouse solution.
Figure 32: The USCC Form After Removing All Files Except Needed Stored Computed Columns
Step 3. For the initial update of these files, enter Yes in the Force Recalculation field to calculate all stored computed columns.
For subsequent updates of other files containing stored computed columns, enter Yes in the Force Recalculation field to recalculate all stored computed columns, or enter No to recalculate only those that have changed.
Note: For subsequent updates of the STUDENT.TERMS.CC file, Datatel recommends that you run the UGSF process before the USCC process, and do not force recalculation on this file. This is because, for many Colleague environments, the USCC process may take an extended period to run. For further details on the UGSF process, see Procedure to Update Flags in the STUDENT.TERMS.CC File on page 188.
Using the DataOrchestrator ODS™, June 18, 2010 187© 2010 Datatel, Inc.
Setting Up Targets: Calculating Stored Computed Columns
Step 4. Save your work and exit from the form. For more details on the DSCC and USCC processes, refer to Stored Computed Columns.
Procedure to Update Flags in the STUDENT.TERMS.CC File
Step 1. Access the Update GPA SCC Flags (UGSF) form in Colleague Student.
Figure 33: Update GPA SCC Flags (UGSF) Form
Note: If your ODS target database does not use the STUDENT.TERMS.CC file or you are performing the initial calculation of the STUDENT.TERMS.CC file, you can skip this procedure.
188 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Calculating Stored Computed Columns
Step 2. In the Update change flags for Cumulative GPA by Term? field, enter Yes to set the change flags in the STUDENT.TERMS.CC file when you save from UGSF.
This allows the correct Cumulative GPA by Term values to be recalculated when you run the USCC process without a forced recalculation of these stored computed columns of all records in the file.
These flags are set on records in the STUDENT.TERMS.CC file in the following case: a student’s GPA changes for a given term (thereby affecting the Cumulative GPA by Term for all subsequent terms). Flags are set on all terms following the changed term.
Step 3. Save your work and exit from the form.
Using the DataOrchestrator ODS™, June 18, 2010 189© 2010 Datatel, Inc.
Setting Up Targets: Calculating Stored Computed Columns
190 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Viewing Reports and Refresh History
Viewing Errors for a Refresh
In This ChapterThis chapter describes how to run an error analysis report for the errors encountered when exporting data from a Colleague source file to an ODS target database.
Table 29 lists the topics covered in this chapter.
Form UsedTable 30 lists and describes the form used in this chapter.
Table 29: Topics in This Chapter
Topic Page
Form Used 193
Viewing the Error Analysis Report for a Refresh 194
Procedure for Running the Error Analysis Report 197
Table 30: Form Used in This Chapter
Form Purpose
DataOrch Error Analysis (DOEA) View the errors received for transforms after running a refresh.
Using the DataOrchestrator ODS™, June 18, 2010 193© 2010 Datatel, Inc.
Viewing Reports and Refresh History: Viewing Errors for a Refresh
Viewing the Error Analysis Report for a Refresh
Use the DataOrch Error Analysis (DOEA) form after running a refresh to view the errors encountered for transforms. The error information displayed is for those data rows of a transform that were not written to the ODS target database.
You can display errors for all transforms associated with a refresh, or you can limit the number of errors displayed by selecting only specific transforms. You can also choose to show a report that contains only summary information about the errors in each transform, or you can include detailed information about each recorded error. In addition, you can choose to sort error rows per transform by error type or by source file record ID.
The number of errors shown for a transform can be affected by the Max Errors Saved field on the DataOrch Parameters (DOPA) form. This field is used to limit the number of errors reported for transforms. Only this number of errors will be included for a specific transform in this report.
Technical Tip: The Error Analysis Report for the refresh includes warning messages for each transform that results in an empty target ODS table. These warnings are included because these transforms do not result in useful information in the operational data store and may indicate either an error in the design of the transform, or that the transform can be omitted from the refresh.
Technical Tip: To help you correct data errors in the source environment, you may want to use the Colleague Data Scanner Utility (CDSU) process. For more information, see Preparing for Release 18, available on the Datatel website.
194 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Viewing the Error Analysis Report for a Refresh
Figure 34: DataOrch Error Analysis (DOEA) Form
Noteworthy Fields on the DOEA Form
The fields described in this section are important when setting up the error analysis report to run.
Refresh ID
The Refresh ID field displays the ID of the refresh.
Target ID
The Target ID field displays the ID of the associated target.
Using the DataOrchestrator ODS™, June 18, 2010 195© 2010 Datatel, Inc.
Viewing Reports and Refresh History: Viewing Errors for a Refresh
Status
The Status field displays the status of the refresh.
Last Start Date/Time
The Last Start Date/Time field displays the date and time when the most recent refresh started.
Last End Date/Time
The Last End Date/Time field displays the date and time when the most recent refresh ended.
Limit to Transforms
Use the Limit to Transforms field to enter the transforms associated with this refresh for which you want to view data or refresh errors. You can enter only those transforms from the target that have data or refresh errors.
Rows with Errors
The Rows with Errors field displays the number of rows with errors.
Show Detailed Errors (Y/N)
Use the Show Detailed Errors (Y/N) field to enter Yes to view error summary information and to display each error row for the transforms selected. Enter No to view only error summary information for each transform.
Sort by Error Type (Y/N)
Use the Sort by Error Type (Y/N) field to enter Yes to sort error rows per transform by error type. Enter No to sort error rows per transform by source file record ID.
Note: If you leave this field blank, then all transforms for this refresh that have data or refresh errors will be included.
196 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Viewing the Error Analysis Report for a Refresh
Procedure for Running the Error Analysis Report
Use the following steps to run the Error Analysis Report.
Step 1. Access the Error Analysis (DOEA) form. At the DataOrch Refresh LookUp prompt, enter the refresh for which you want to view error analysis.
Step 2. On the DOEA form, enter or modify the settings for running the report. For more information on these fields, see online help.
Step 3. Save your entries on the DOEA form.
Step 4. Complete the Peripheral Settings and Process Handler forms according to your preferences.
Step 5. When you are finished viewing, exit from the report.
Using the DataOrchestrator ODS™, June 18, 2010 197© 2010 Datatel, Inc.
Viewing Reports and Refresh History: Viewing Errors for a Refresh
198 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Viewing Reports and Refresh History
Viewing Target Transform Data for a Target
In This ChapterThis chapter describes how to run the Transform Summary report in order to view the information for target transforms for a specified target.
Table 31 lists the topics covered in this chapter.
Form UsedTable 32 lists and describes the form used in this chapter.
Table 31: Topics in This Chapter
Topic Page
Form Used 199
Viewing the Transform Summary Report for a Target 200
Procedure for Running the Transform Summary Report 202
Table 32: Form Used in This Chapter
Form Purpose
DataOrch Transform Summary (DOTS) View the information for target transforms associated with a target.
Using the DataOrchestrator ODS™, June 18, 2010 199© 2010 Datatel, Inc.
Viewing Reports and Refresh History: Viewing Target Transform Data for a Target
Viewing the Transform Summary Report for a Target
Use the DataOrch Transform Summary (DOTS) form to view a report of the target transforms for a specific target. This report includes data associated with a target transform, and also information about each column associated to the transform.
You can choose whether or not to include the description of the transform and the descriptions of the transform’s columns.
Figure 35: DataOrch Transform Summary (DOTS) Form
Note: This report can be used only for target transforms. You cannot enter source transforms.
200 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Viewing the Transform Summary Report for a Target
Noteworthy Fields on the DOTS Form
The fields described in this section are important when setting up the Transform Summary report to run.
Target ID
The Target ID field displays the ID of the selected target.
Target Transforms
Use the Target Transforms field to enter target transform names, associated to the Target ID, to include in the report.
Include Transform Description (Y/N)
Use the Include Transform Description (Y/N) field to indicate whether to include the description of the transform in the report. Enter Yes to include the description, or enter No to exclude the description.
Include Column Descriptions (Y/N)
Use the Include Column Descriptions (Y/N) field to indicate whether to include the columns’ descriptions for the transform in the report. Enter Yes to include the descriptions, or enter No to exclude the descriptions.
Using the DataOrchestrator ODS™, June 18, 2010 201© 2010 Datatel, Inc.
Viewing Reports and Refresh History: Viewing Target Transform Data for a Target
Procedure for Running the Transform Summary Report
Use the following steps to run the Transform Summary report.
Step 1. Access the DataOrch Transform Summary (DOTS) form. At the DataOrch Target LookUp prompt, enter the ID of the target for which you want to view transform information.
Step 2. On the DOTS form, enter or modify the settings for running the report. For more information on these fields, see online help.
Step 3. Save your entries on the DOTS form.
Step 4. Complete the Peripheral Settings and Process Handler forms according to your preferences.
Step 5. When you are finished viewing, exit from the report.
202 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Viewing Reports and Refresh History
Viewing the History of a Refresh
In This ChapterThis chapter describes how to view the historical information for a refresh, including the settings, status, and statistics of previously run refreshes for a specific target.
Table 33 lists the topics covered in this chapter.
Form UsedTable 34 lists and describes the form used in this chapter.
Table 33: Topics in This Chapter
Topic Page
Form Used 203
Viewing the History of a Refresh 204
Procedure for Viewing the History of a Refresh 206
Table 34: Form Used in This Chapter
Form Purpose
DataOrch Refresh History (DORH) View the settings, status, and statistics of a previously run refresh for a specific target.
Using the DataOrchestrator ODS™, June 18, 2010 203© 2010 Datatel, Inc.
Viewing Reports and Refresh History: Viewing the History of a Refresh
Viewing the History of a RefreshThe DataOrch Refresh History (DORH) form is used to view historical information about previously run refreshes. This form shows the refresh status, settings, and statistics of a previously run refresh for a specific ODS target database. The settings and statistics include the following:
Settings for the refresh:
Refresh ID
Target ID
Database name/TNS
DMI Listener host name
DMI Listener port number
Number of concurrent threads
For each target transform run as part of the refresh, the following information is included:
Target transform and source file names.
The Update Type for the transform. This would indicate the Override Update Type if the default Update Type was overridden for a transform.
Status of the transform, indicating whether it was successfully refreshed or not.
Date/time the target transform started and finished processing.
Transform information for the refresh:
Total number of rows written for the transform.
Number of rows deleted on the target database.
Number of errors encountered per transform.
For each materialized SQL view that is run as part of the refresh, the following information is included:
Materialized SQL view name. Note that an entry for a materialized SQL view is indicated by the words “Materialized View” in the Source File field.
An Update Type of “Full.” (Materialized SQL views cannot be incrementally updated).
The status of the refresh of the materialized SQL view, indicating whether it was successfully refreshed or not.
Date/time the materialized SQL view started and finished processing.
204 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Viewing the History of a Refresh
Figure 36: DataOrch Refresh History (DORH) Form
Using the DataOrchestrator ODS™, June 18, 2010 205© 2010 Datatel, Inc.
Viewing Reports and Refresh History: Viewing the History of a Refresh
You can also view the refresh status on the DORH form. The following statuses may be displayed.
InProc. The transform is currently being refreshed.
Success. The transform was successfully refreshed without any errors.
Warning. The transform or refresh received errors during processing.
Fail. A transform or refresh received fatal errors during processing.
Procedure for Viewing the History of a Refresh
Use the following step to view the history of a refresh.
Step 1. Access the DataOrch Refresh History (DORH) form. At the DataOrch Refresh History LookUp prompt, enter the ID of the history record you want to view.
Step 2. When you are finished viewing, exit from the DORH form.
Technical Tip: The history record ID consists of three parts: 1. The Refresh ID. 2. The date, in internal format, when the user ran the refresh. 3. The exact time, in internal format, when the user ran the refresh. For example, DO.REFRESH.ABC_14016_52885. The order of the Refresh IDs are sorted by date and time. The most recent refresh appears first.
206 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS
Maintaining Transforms
In This ChapterThis chapter describes how to maintain DataOrchestrator ODS transforms. You can copy existing transforms, delete transforms as needed, and rename target transforms. You can also regenerate source transforms, and calculate foreign keys for both source and target transforms.
Table 35 lists the topics covered in this chapter.
Form UsedTable 36 lists and describes the form used in this chapter.
Table 35: Topics in This Chapter
Topic Page
Form Used 209
Maintaining DataOrchestrator ODS Transforms 210
Procedure for Deleting Transforms 213
Procedure for Copying Transforms 214
Procedure for Renaming Transforms 215
Procedure for Copying and Renaming Transforms 216
Procedure for Calculating Foreign Keys 217
Procedure for Regenerating Source Transforms 218
Table 36: Form Used in This Chapter
Form Purpose
DataOrch Transform Maint (DOMA) Maintains DataOrchestrator ODS transforms.
Using the DataOrchestrator ODS™, June 18, 2010 209© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Maintaining Transforms
Maintaining DataOrchestrator ODS Transforms
Use the DataOrch Transform Maint (DOMA) form to perform the following functions:
Delete transforms from a target.
Copy transforms from one target to another target.
Rename target transforms.
Copy transforms from a target, and then rename them to a different target or to the same target.
Calculate foreign keys for transforms.
Regenerate source transforms.
Figure 37: Example of Deleting a Transform using the DataOrch Transform Maint (DOMA) Form
210 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining DataOrchestrator ODS Transforms
Noteworthy Fields on the DOMA Form
The fields described in this section are important when maintaining transforms using the DataOrchestrator ODS.
Operation
Use the Operation field to choose an operation to perform on a target. Select one of the following operations:
Delete. Delete transforms from a target.
Copy. Copy transforms from one target to another.
Rename. Rename target transforms associated with a target. Source transforms cannot be renamed.
Calculate Foreign Keys. Generate foreign key metadata for transforms. Foreign key metadata will be created in the following cases: • If the input field for a column is a pointer to a Colleague file, then a foreign
key is created to any transform with that file as its source file. • A foreign key is created for each multivalued transform to all single-
valued transforms with the same source file.
Regenerate Source Transforms. Regenerate source transforms associated with a target. This will regenerate the set of target transforms to create and populate data for each target database table needed to hold the specified input fields of the source file. You will need to regenerate source transforms if any Colleague metadata changes for the fields you export in a source transform.
From Target ID
Use the From Target ID field to enter the name of the target from which to select associated transforms to process.
Transform List
Use the Transform List field to enter the transforms to process that are associated with the target selected in the From Target ID field.
Rename To
Use the Rename To field to enter a new name for a target transform on the Transform List. Source transforms cannot be renamed.
Using the DataOrchestrator ODS™, June 18, 2010 211© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Maintaining Transforms
To Target ID
If you are copying to another Target ID, you must enter the name of the target to which you want to copy transforms. For a DataOrchestrator ODS target, you can enter the same target name as you entered in the From Target ID field. You would do this if you want to copy transforms from a target onto the same target with a new name. This allows you to create new transforms from existing transforms and modify them as needed.
212 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining DataOrchestrator ODS Transforms
Procedure for Deleting Transforms
To delete transforms from a target, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Delete.
Step 3. In the From Target ID field, enter the ID of the target from which you want to delete transforms.
Step 4. In the Transform List field, enter the transforms you want to delete.
Step 5. Save from the DOMA form to delete the transforms you selected.
Using the DataOrchestrator ODS™, June 18, 2010 213© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Maintaining Transforms
Procedure for Copying Transforms
To copy transforms from one target to another, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Copy from the drop-down list.
Step 3. In the From Target ID field, enter the ID of the target from which you want to copy transforms.
Step 4. In the Transform List field, enter the transforms you want to copy.
Step 5. In the To Target ID field, enter the ID of the target to which you want to copy the transforms.
Step 6. Save from the DOMA form to copy the transforms you selected.
Note: If you copy to an existing transform, you will overwrite the filter of the transform being copied to with the filter of the transform being copied from. Also, if you copy to an existing transform that has any of the same column names, and the column sizes are larger in the existing transform, the larger size is used.
214 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining DataOrchestrator ODS Transforms
Procedure for Renaming Transforms
To rename transforms, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Rename.
Step 3. In the From Target ID field, enter the ID of the target for which you want to rename transforms.
Step 4. In the Transform List field, enter the transforms you want to rename.
Step 5. In the Rename To field, enter the new name for each transform you listed.
Step 6. Save from the DOMA form to rename the transforms you selected.
Using the DataOrchestrator ODS™, June 18, 2010 215© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Maintaining Transforms
Procedure for Copying and Renaming Transforms
To copy and rename transforms from a target to another target, or to the same target, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Copy from the drop-down list.
Step 3. In the From Target ID field, enter the ID of the target from which you want to copy transforms.
Step 4. In the Transform List field, enter the transforms you want to copy.
Step 5. In the Rename To field, enter the new name for each transform you listed.
Step 6. In the To Target ID field, enter the ID of the target to which you want to copy the transforms.
Step 7. Save from the DOMA form to copy and rename the transforms you selected.
Note: If you copy to an existing transform, you will overwrite the filter of the transform being copied to with the filter of the transform being copied from. Also, if you copy to an existing transform that has any of the same column names, and the column sizes are larger in the existing transform, the larger size is used.
216 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining DataOrchestrator ODS Transforms
Procedure for Calculating Foreign Keys
To calculate foreign keys for source and target transforms, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Calculate Foreign Keys from the drop-down list.
Step 3. In the From Target ID field, enter the ID of the target from which you want to calculate foreign keys for transforms.
Step 4. In the Transform List field, enter the transforms for which you want to calculate foreign keys.
Step 5. Save from the DOMA form to calculate foreign keys for the transforms you selected.
If you want to view or modify foreign key metadata for target transforms, follow these steps:
a. Access the DataOrch Target (DOTT) form.
b. Detail from the Column Definitions field to access the Transform Columns (DOTC) form.
c. Detail from a target column name to the Column Properties (DOPR) form.
For more information, see Defining Target Transforms beginning on page 97.
Note: The foreign key metadata for source transforms cannot be modified after it has been calculated.
Using the DataOrchestrator ODS™, June 18, 2010 217© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Maintaining Transforms
Procedure for Regenerating Source Transforms
To regenerate source transforms, use the following steps.
Step 1. Access the DataOrch Transform Maint (DOMA) form.
Step 2. In the Operation field, select Regen Source Transforms from the drop-down list.
Step 3. In the From Target ID field, enter the ID of the target from which you want to regenerate source transforms.
Step 4. In the Transform List field, enter the source transforms you want to regenerate.
Step 5. Save from the DOMA form to regenerate the source transforms you selected.
218 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS
Maintaining SQL Views
In This ChapterThis chapter describes how to maintain SQL views. You can copy existing SQL views, delete views as needed, and rename SQL views.
Table 37 lists the topics covered in this chapter.
Form UsedTable 38 lists and describes the form used in this chapter.
Table 37: Topics in This Chapter
Topic Page
Form Used 219
Maintaining SQL Views 220
Procedure for Deleting SQL Views 223
Procedure for Copying SQL Views 224
Procedure for Renaming SQL Views 225
Procedure for Copying and Renaming SQL Views 226
Table 38: Form Used in This Chapter
Form Purpose
DataOrch View Maintenance (DOVM) Maintains SQL views.
Using the DataOrchestrator ODS™, June 18, 2010 219© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Maintaining SQL Views
Maintaining SQL ViewsUse the DataOrch View Maintenance (DOVM) form to perform the following functions:
Delete SQL views from a target.
Copy SQL views from one target to another target.
Rename SQL views.
Copy SQL views from a target, and then rename them to a different target or to the same target.
Note that when you when you copy an SQL view to a target and an SQL view with the same name exists on that target, the information in the target SQL view is updated as follows:
The SQL statement associated with view is replaced with that of the SQL view being copied.
If the SQL view being copied is a materialized SQL view, the materialized SQL view parameters, including the Create Materialized View flag and the Dependent Transforms list are overwritten. However, if the SQL view being copied is not a materialized SQL view, the existing data in the target SQL view is not changed.
The SQL view’s column information for the view being copied will be integrated into the target SQL view column information. Any new column entries will be added to the SQL View Columns list (on the DataOrch View Spec [DOVS] form) and information for existing columns will be overwritten. The only exception to this is the Key field. If this field is set to “Yes” in the existing SQL view, it will not be overwritten with a value of “No” from the SQL view being copied.
Note: This process modifies only the SQL view metadata. No processing will occur on the ODS target database.
220 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining SQL Views
Figure 38: Example of Deleting an SQL View using the DataOrch View Maintenance (DOVM) Form
Noteworthy Fields on the DOVM Form
The fields described in this section are important when deleting, copying, or renaming SQL views using the DataOrchestrator ODS.
Operation
Use the Operation field to choose an operation to perform on a target. Select one of the following operations:
Delete. Delete SQL views from a target.
Copy. Copy SQL views from one target to another.
Rename. Rename SQL views associated with a target.
Note: Deleting an SQL view from a target does not automatically delete the corresponding SQL view on the ODS target database. This must be done manually by the system administrator.
Using the DataOrchestrator ODS™, June 18, 2010 221© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Maintaining SQL Views
From Target ID
Use the From Target ID field to enter the name of the target from which to select associated SQL views to process.
SQL Views
Use the SQL Views list to enter the SQL views to process that are associated with the target selected in the From Target ID field.
Rename To
Use the Rename To field to enter a new name for an SQL view on the SQL Views list.
To Target ID
If you are copying to another Target ID, you must enter the name of the target to which you want to copy SQL views. You can enter the same target name as you entered in the From Target ID field to copy and rename SQL views to the same target. You would do this if you want to copy SQL views from a target onto the same target with a new name. This allows you to create new SQL views from existing SQL views and modify them as needed.
222 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining SQL Views
Procedure for Deleting SQL Views
To delete SQL views from a target, use the following steps.
Step 1. Access the DataOrch View Maintenance (DOVM) form.
Step 2. In the Operation field, select Delete.
Step 3. In the From Target ID field, enter the ID of the target from which you want to delete SQL views.
Step 4. In the SQL Views list, enter the SQL views you want to delete.
Step 5. Save from the DOVM form to delete the SQL views you selected.
Using the DataOrchestrator ODS™, June 18, 2010 223© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Maintaining SQL Views
Procedure for Copying SQL Views
To copy SQL views from one target to another, use the following steps.
Step 1. Access the DataOrch View Maintenance (DOVM) form.
Step 2. In the Operation field, select Copy from the drop-down list.
Step 3. In the From Target ID field, enter the ID of the target from which you want to copy SQL views.
Step 4. In the SQL Views list, enter the SQL views you want to copy.
Step 5. In the To Target ID field, enter the ID of the target to which you want to copy the SQL views.
Step 6. Save from the DOVM form to copy the SQL views you selected.
224 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining SQL Views
Procedure for Renaming SQL Views
To rename SQL views, use the following steps.
Step 1. Access the DataOrch View Maintenance (DOVM) form.
Step 2. In the Operation field, select Rename.
Step 3. In the From Target ID field, enter the ID of the target for which you want to rename SQL views.
Step 4. In the SQL Views list, enter the SQL views you want to rename.
Step 5. In the Rename To field, enter the new name for each SQL view you listed.
Step 6. Save from the DOVM form to rename the SQL views you selected.
Using the DataOrchestrator ODS™, June 18, 2010 225© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Maintaining SQL Views
Procedure for Copying and Renaming SQL Views
To copy and rename SQL views from a target to another target, or to the same target, use the following steps.
Step 1. Access the DataOrch View Maintenance (DOVM) form.
Step 2. In the Operation field, select Copy from the drop-down list.
Step 3. In the From Target ID field, enter the ID of the target from which you want to copy SQL views.
Step 4. In the SQL Views list, enter the SQL views you want to copy.
Step 5. In the Rename To field, enter the new name for each SQL view you listed.
Step 6. In the To Target ID field, enter the ID of the target to which you want to copy the SQL view.
Step 7. Save from the DOVM form to copy and rename the SQL views you selected.
226 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS
Copying Targets to Another Target
In This ChapterThis chapter describes how to copy all transforms and SQL views from one or more targets to a single destination target.
Table 39 lists the topics covered in this chapter.
Form UsedTable 40 lists and describes the form used in this chapter.
Table 39: Topics in This Chapter
Topic Page
Form Used 227
Copying One or More Targets to Another Target 228
Procedure for Copying One or More Targets to Another Target 230
Table 40: Form Used in This Chapter
Form Purpose
DataOrch Target Copy (DOTY) Copy transforms and SQL views from one or more targets to another single target.
Using the DataOrchestrator ODS™, June 18, 2010 227© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Copying Targets to Another Target
Copying One or More Targets to Another Target
Use the DataOrch Target Copy (DOTY) form to copy all transforms and SQL views from one or more targets to a single destination target. This form provides the ability to install all target templates with different versions all at once. The target IDs listed in the Copy from Target IDs field will be sorted alphabetically before processing to ensure that the target templates are installed in the correct order.
Figure 39: DataOrch Target Copy (DOTY) Form
Note: This form copies all transforms and SQL views from the targets listed in the Copy from Target IDs field. To select specific transforms and SQL views from a target, use the DataOrch Transform Maint (DOMA) form and the DataOrch View Maintenance (DOVM) form. For further details on how individual transforms and SQL views will be copied, see the following information for the DOMA and DOVM forms: – Maintaining Transforms beginning on page 209 – Maintaining SQL Views beginning on page 219
228 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Copying One or More Targets to Another Target
Noteworthy Fields on the DOTY Form
The fields described in this section are important when copying all transforms and SQL views from one or more targets to a single destination target.
Copy from Target IDs
Use the Copy from Target IDs field to enter one or more targets whose transforms and SQL views you want to copy to the destination target.
Copy to Target ID
Use the Copy to Target ID field to enter the target ID to which you want to add the transforms and SQL views from the targets you listed in the Copy from Target IDs field.
Note: Target templates are not allowed, as they cannot be modified. Also, the destination target cannot be a target that is listed in the Copy from Target IDs field.
Using the DataOrchestrator ODS™, June 18, 2010 229© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Copying Targets to Another Target
Procedure for Copying One or More Targets to Another Target
Use the following steps to copy all transforms and SQL views from one or more targets to a single destination target.
Step 1. Access the DataOrch Target Copy (DOTY) form.
Step 2. On the DOTY form, enter one or more targets from which you want to copy all transforms and SQL views. Also, enter the destination target. For more information on these fields, see online help.
Step 3. Save from the DOTY form to copy the transforms and SQL views from the targets you selected to the destination target.
230 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS
Deleting DataOrchestrator ODS Objects
In This ChapterThis chapter describes how to delete DataOrchestrator ODS objects. You can delete existing refreshes, targets, or refresh history records from the source Colleague database.
Table 41 lists the topics covered in this chapter.
Form UsedTable 42 lists and describes the form used in this chapter.
Table 41: Topics in This Chapter
Topic Page
Form Used 231
Deleting DataOrchestrator ODS Objects 232
Procedure for Deleting DataOrchestrator ODS Objects 233
Table 42: Form Used in This Chapter
Form Purpose
DataOrch Object Delete (DOOD) Delete DataOrchestrator ODS objects.
Using the DataOrchestrator ODS™, June 18, 2010 231© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Deleting DataOrchestrator ODS Objects
Deleting DataOrchestrator ODS Objects Use the DataOrch Object Delete (DOOD) form to delete DataOrchestrator ODS objects. You can delete existing refreshes, targets, or refresh history records from the source Colleague database.
Figure 40: DataOrch Object Delete (DOOD) Form
Note: If you delete a target, you will also delete all of the target’s associated transforms, transform columns, and transform error records.
232 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Deleting DataOrchestrator ODS Objects
Noteworthy Fields on the DOOD Form
The fields described in this section are important when deleting objects using the DataOrchestrator ODS.
Object Type
Use the Object Type field to choose the DataOrchestrator ODS object type for which you want to delete IDs.
IDs to Delete
Use the IDs to Delete field to enter the IDs of the records to delete for the object type you selected. When you save, these IDs will be deleted from the source Colleague database.
Procedure for Deleting DataOrchestrator ODS Objects
Use the following steps to delete an object.
Step 1. Access the DataOrch Object Delete (DOOD) form.
Step 2. On the DOOD form, enter the settings for deleting objects. For more information on these fields, see online help.
Step 3. Save from the DOOD form to delete the objects you selected.
Note: The object type you select determines which IDs you can enter in the IDs to Delete field.
Using the DataOrchestrator ODS™, June 18, 2010 233© 2010 Datatel, Inc.
Maintaining the DataOrchestrator ODS: Deleting DataOrchestrator ODS Objects
234 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Appendices
Checklist for Setting Up the DataOrchestrator ODS
In This AppendixThis appendix provides a quick checklist of the activities that the system administrator and system programmer perform to set up and maintain the DataOrchestrator ODS.
Using the DataOrchestrator ODS™, June 18, 2010 237© 2010 Datatel, Inc.
Appendices: Checklist for Setting Up the DataOrchestrator ODS
Checklist for Setting Up the DataOrchestrator ODS
Table 1: Checklist for Setting Up the DataOrchestrator ODS
Step User Description Reference
1. System administrator
Check that your institution has the prerequisites needed for the DataOrchestrator ODS.
Prerequisites for the DataOrchestrator ODS on page 35.
2. System administrator
Decide where to build the operational data stores. Where to Build the Operational Data Stores beginning on page 36.
3. System administrator
Prepare the target database where the operational data stores will be built.
Preparing the ODS Target Database beginning on page 38.
4. System administrator
Note: If your institution does not use SQL Server, skip this step.
For SQL Server, you can use a script to create your ODS target database.
SQL Server Script for Creating the ODS Target Database on page 40.
5. System administrator
Note: If your institution does not use SQL Server, skip this step.
If you are using a SQL Server target database, the filegroup “IDX” must be in your target database.
Adding the IDX Filegroup on page 42.
6. System administrator
Add the DODS optional module for the DataOrchestrator ODS.
Adding the Optional Module on page 46.
7. System administrator
Create a new DMI Listener on the ODS target database server and select the roles DBAS and RDAS for the Listener. Use the Colleague release system to retrieve and load the appropriate software updates.
Setting Up the ODS Target Database Server and Installing Software Updates on page 47.
See Updating Colleague Software for information on retrieving and installing software updates.
8. System administrator
Note: If your institution does not use Oracle, skip this step.
If the target database is Oracle, install the Oracle JDBC driver on the ODS target database. If your source environment is UniData and your target database is Oracle, you also must install the Oracle JDBC driver on your source environment.
Installing the Oracle JDBC Driver on page 51.
238 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Checklist for Setting Up the DataOrchestrator ODS
9. System administrator
Define parameters for the DataOrchestrator ODS. Defining DataOrchestrator ODS Parameters beginning on page 59.
10. System administrator
Create a target, which defines the configuration of an operational data store on a target database. This includes defining source and target transforms to refresh on the target database. (Optional) Define and create SQL views as shown.
Defining a Target beginning on page 67 (all chapters in Part 4).
11. System administrator
Create a Refresh ID with an associated target. Define parameters to control how the DataOrchestrator ODS refreshes a target database with data from Colleague source files.
Defining and Running an ODS Refresh beginning on page 165.
12. System administrator
Note: If your institution does not use stored computed columns, skip this step.
Activate and calculate stored computed columns for any target database that contains stored computed columns. If your ODS target database uses the STUDENT.TERMS.CC file, update the flags as shown.
Note: Datatel recommends that any operational data store that contains stored computed columns be set up to automatically run the following processes:
• Update GPA SCC Flags (UGSF) if using the STUDENT.TERMS.CC file
• Update Stored Computed Column (USCC)
This ensures that the stored computed columns will be updated when the ODS target database is updated.
Calculating Stored Computed Columns beginning on page 181. Refer to Envision Runtime Administration for information on how to set a batch process to run at scheduled intervals.
13. System administrator
Run a refresh that updates the specified ODS target database. To run the refresh, access the DataOrch Refresh (DORE) form and enter Yes in the Run Refresh (Y/N) field.
(Optional) Create all SQL views for the target when you run the refresh by entering Yes in the Create SQL Views (Y/N) field.
Save from the DORE form to run the refresh.
Defining and Running an ODS Refresh beginning on page 165.
14. System administrator and system programmer
Run an error analysis report for any errors encountered when exporting data from a Colleague source file to an ODS target database.
Viewing Errors for a Refresh beginning on page 193.
Table 1: Checklist for Setting Up the DataOrchestrator ODS (cont’d)
Step User Description Reference
Using the DataOrchestrator ODS™, June 18, 2010 239© 2010 Datatel, Inc.
Appendices: Checklist for Setting Up the DataOrchestrator ODS
15. System administrator and system programmer
(Optional) Run a Transform Summary report in order to view information for target transforms for a specified target.
Viewing Target Transform Data for a Target beginning on page 199.
16. System administrator and system programmer
(Optional) View historical information for previously run refreshes for a specific target.
Viewing the History of a Refresh beginning on page 203.
17. System administrator and system programmer
(Optional) Copy existing target transforms, delete transforms, and rename transforms. Regenerate source transforms, and create foreign keys for source and target transforms.
Maintaining Transforms beginning on page 209.
18. System administrator and system programmer
(Optional) Copy existing SQL views, delete views, and rename SQL views.
Maintaining SQL Views beginning on page 219.
19. System administrator and system programmer
(Optional) Copy all transforms and SQL views from one or more targets to a single destination target.
Copying Targets to Another Target beginning on page 227.
20. System administrator and system programmer
(Optional) Delete existing refreshes, targets, or refresh history records from the source Colleague database.
Deleting DataOrchestrator ODS Objects beginning on page 231.
Table 1: Checklist for Setting Up the DataOrchestrator ODS (cont’d)
Step User Description Reference
240 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Appendices
Frequently Asked Questions
In This AppendixThis appendix describes provides answers to frequently asked questions when using the DataOrchestrator ODS and the data models and views for reporting.
Table 2 lists the topics covered in this appendix.
Note: For more information on the DataOrchestrator ODS and the data models and views, reference AnswerNet page 5983, “DataOrchestrator Support Solutions.”
Table 2: Topics in this Chapter
Topic Page
Frequently Asked Questions about the DataOrchestrator ODS 242
Using the DataOrchestrator ODS™, June 18, 2010 241© 2010 Datatel, Inc.
Appendices: Frequently Asked Questions
Frequently Asked Questions about the DataOrchestrator ODS
Listed below are questions and answers for frequently asked questions about the DataOrchestrator ODS and the data models and views associated with the DataOrchestrator ODS.
Q What port will I need to open up in my firewall on the source database server?
A The port on which the database is running on the source database server should allow incoming requests.
Q What ports will I need to open up in my firewall on the target database server?
A The following ports on the target database server should allow incoming requests:
The port where the Datatel daemon is installed (usually port 9000).
The port you have chosen for the RDAS listener.
Q When I set up the RDAS Listener on the target database server, do I set up a new local product repository on that server?
A No. You do not need to set up a new local product repository on the target database server. The only prerequisite for installing the new listener on the target database server is that you have installed a Datatel daemon on the target database server.
Q I have a SQL Server target database. When testing connectivity on the DataOrch Refresh (DORE) form, I receive the following message. What does this mean?:
Unable to log in to target database.
Failed Login to ods_test on sdw2k3qasql1 by abcdef: Fatal: 18456(14):
Login failed for user 'abcdef'.
242 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Frequently Asked Questions about the DataOrchestrator ODS
A Check the target SQL Server database instance where ods_test is set up to be sure a user named “abcdef” is defined with SQL Server authentication. Also, be sure the security for that SQL Server instance is set up to allow for both Windows and SQL Server authentication.
Q What permissions should I give users on the target SQL Server database?
A Give server roles of public and bulkadmin to the user. Give database role membership of public, db_ddladmin, db_datareader, and db_datawriter for that particular database.
Q When checking connectivity on the DORE form, I receive “DMI_OPEN” errors. How do I resolve these?
A First, check to be sure all listeners for the environment are running. If some are not running, be sure to start them, and then look for the following:
On the DORE form, be sure the RDAS listener installed on your target database server was selected.
If your source database is UniData or Oracle, then go to the DataOrch Parameters (DOPA) form and be sure the DMI_DAS listener is the DMI_DAS listener on your Colleague database server.
Q I created a new source transform using the PERSON file. The tables subsequently created on the SQL target database were not what I expected. How can I know which tables will be created and can I choose the tables, or is this based solely on the columns/fields that I select from the PERSON file?
For example, our institution does not use the MILITARY_INFO table, but because I selected the PERSON file, it was created. How could I skip this table?
A To understand tables created on the SQL target database, refer to Mapping Envision Files for SQL Server and Oracle, available on the Datatel website. This mapping pertains to source transforms only which create the Release 18 SQL schema table structure (that is, all single-valued columns in one table, all unassociated multivalues in the filename_LS table, and all associated columns in a table named after the association).
Note: This question could be posed for any file and association name; PERSON and MILITARY_INFO are examples.
Using the DataOrchestrator ODS™, June 18, 2010 243© 2010 Datatel, Inc.
Appendices: Frequently Asked Questions
The tables created also depend on which fields you choose on the DataOrch Source Transform (DOST) form when setting up your source transform. If you choose all the fields for the PERSON file, for example, you will have all 15 tables that make up the PERSON file in your target database.
If you want to omit tables from the target database, select specific columns from the PERSON file. For example, to prevent the MILITARY_ASSOC table from being created on the target database, do not select any of the following columns on the DOST form, which are all part of the MILITARY_ASSOC association:
MILITARY.STATUSES
MILITARY.BRANCHES
MILITARY.GOVT.BENEFITS
MILITARY.START.DATES
MILITARY.END.DATES
Also, you can view which tables will be created on the target database for a specific source transform by detailing on the View Generated Transforms field on the DOST form.
Q When running a refresh, I receive the following message:
User error 10015: - unable to authenticate your login.
A This message occurs because a pre-authenticated server is used for the file transfer in the DataOrchestrator ODS refresh. This means that on the DMI Pre-Authenticated Server (DMCC) form, you must have your application server and your Colleague database server defined. If you are a UniData client who has not distributed the DAS, then your application server and Colleague database server are the same, so you will need to enter only one line.
If you have the DMCC form set up and are still experiencing issues, then follow these steps:
Step 1. Start the following listeners with logging turned on:
The listener defined on the DORE form.
The listener installed on the DataOrch Parameters (DOPA) form.
244 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Frequently Asked Questions about the DataOrchestrator ODS
Step 2. Rerun the Check Connectivity function on the DORE form to receive the error again, then search for an entry in the dmi.log file such as the following:
Unable to authenticate a client login from (hp.datatel.com,127.0.0.1)
Use the DNS name and the IP address in this entry to define the pre-authenticated server on the DMCC form.
Step 3. Restart the listeners again with logging turned off.
Q When running a refresh, I receive the following error:
Unable to log in to target database.
Failed Login to <db_name> on <server_name> by <user>: Premature end of Stream: server won't transmit TDS header
A This is probably due to an incorrect database port specified on the DOTA form. Access the DOTA form and verify that the correct database port is specified.
Q When running a refresh, I receive the following error:
The file separator detected (“/”) is not the same separator in the file path “...”
A Look for the following:
In SA Valet, make sure the listener on the target ODS database that is specified on DOTA has both the RDAS and DBAS roles.
If your source database is UniData or Oracle, then go to the DOPA form and make sure the correct source listener is specified in the Source DMI_DAS Listener Name field.
Using the DataOrchestrator ODS™, June 18, 2010 245© 2010 Datatel, Inc.
Appendices: Frequently Asked Questions
246 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Appendices
Troubleshooting the DataOrchestrator ODS
In This AppendixThis appendix provides suggestions for items to check or steps to take if you encounter issues in using the DataOrchestrator ODS.
Note: For the most up-to-date information about the troubleshooting issues listed in this appendix, as well as any additional issues reported since the publication of this manual, see AnswerNet page 6063. If you do not have access to AnswerNet at your institution, consult with your system administrator.
Using the DataOrchestrator ODS™, June 18, 2010 247© 2010 Datatel, Inc.
Appendices: Troubleshooting the DataOrchestrator ODS
Dat
aOrc
hes
trat
or
OD
S Is
sues
Tab
le 3
list
s th
e is
sues
, pos
sibl
e ca
uses
, and
res
olut
ions
for
trou
bles
hoot
ing.
Tab
le 3
: T
roub
lesh
ootin
g Is
sues
Issu
eP
oss
ible
Cau
seR
eso
luti
on
1.R
ecei
ve o
ne o
f the
follo
win
g er
rors
on
the
Dat
aOrc
h R
efre
sh (
DO
RE
) fo
rm w
hen
chec
king
con
nect
ivity
or
runn
ing
a re
fres
h:
Uni
Dat
a so
urce
:
Error - encountered during Data
Extract
Cannot open D:\...\DOR... file.
Ora
cle
sour
ce:
Error - encountered during Data
Extract
ORA-20014: EXTRACT_DATA: Error
extracting data. Message: ORA-
20002: OPEN_FILE:
INVALID_OPERATION: File could
not be opened as requested
The
sou
rce
dire
ctor
y pa
th o
n th
e D
ataO
rch
Par
amet
ers
(DO
PA
) fo
rm
coul
d be
inco
rrec
t or
the
perm
issi
on
on th
e di
rect
ory
coul
d be
inco
rrec
t.
Ver
ify th
e pa
th o
n th
e D
OP
A fo
rm is
a v
alid
pat
h on
the
sour
ce d
atab
ase
serv
er. A
lso,
ver
ify th
e pe
rmis
sion
s to
th
e di
rect
ory
corr
espo
nd w
ith th
e in
stal
latio
n in
stru
ctio
ns
on s
ettin
g up
the
sour
ce d
atab
ase
serv
er te
mpo
rary
di
rect
ory.
248 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
DataOrchestrator ODS Issues
2.R
ecei
ve o
ne o
f the
follo
win
g er
rors
on
the
DO
RE
form
whe
n ch
ecki
ng c
onne
ctiv
ity o
r ru
nnin
g a
refr
esh:
SQ
L S
erve
r so
urce
:
Error - encountered during Data
Extract:
SQLState = S1000, NativeError =
0Error = [Microsoft][ODBC SQL
Server Driver]Unable to open BCP
host data-file
Uni
Dat
a an
d O
racl
e so
urce
:
Error encountered during file
transfer
D:\thisIsNotADirectory\... (The
system cannot find the path
specified)
The
targ
et d
irect
ory
path
on
the
Add
ition
al T
arge
t Par
amet
ers
(DO
TP
) fo
rm c
ould
be
inco
rrec
t or
the
perm
issi
on o
n th
e di
rect
ory
coul
d be
inco
rrec
t.
Ver
ify th
e pa
th o
n th
e D
OT
P fo
rm is
a v
alid
pat
h on
the
targ
et d
atab
ase
serv
er. A
lso
verif
y th
e pe
rmis
sion
s to
th
e di
rect
ory
corr
espo
nd w
ith th
e in
stal
latio
n in
stru
ctio
ns
on s
ettin
g up
the
targ
et d
atab
ase
serv
er te
mpo
rary
di
rect
ory.
Tab
le 3
: T
roub
lesh
ootin
g Is
sues
(co
nt’d
)
Issu
eP
oss
ible
Cau
seR
eso
luti
on
Using the DataOrchestrator ODS™, June 18, 2010 249© 2010 Datatel, Inc.
Appendices: Troubleshooting the DataOrchestrator ODS
3.R
ecei
ve o
ne o
f the
follo
win
g er
rors
on
the
DO
RE
form
whe
n ch
ecki
ng c
onne
ctiv
ity o
r ru
nnin
g a
refr
esh.
UniSession Exception-39207:
Error [39207] occurred on
server. Possible client-side
licensing failure.:
Unable to connect to the
registry database.
OR
Unable to login to target
database. Registry error-30008:
Unable to access the registry
database for the request.
Uni
Dat
a lic
ense
s on
the
sour
ce
data
base
hav
e ru
n ou
t.S
peci
fy fe
wer
thre
ads
to u
se o
n th
e D
OR
E fo
rm, o
r ru
n th
e re
fres
h at
a ti
me
whe
n fe
wer
use
rs a
re lo
gged
in to
C
olle
ague
.
Tab
le 3
: T
roub
lesh
ootin
g Is
sues
(co
nt’d
)
Issu
eP
oss
ible
Cau
seR
eso
luti
on
250 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
DataOrchestrator ODS Issues
4.R
ecei
ve o
ne o
f the
follo
win
g er
rors
whe
n ru
nnin
g a
refr
esh.
Error - encountered during Data
Extract for table:
Fatal: 4121(16): Cannot find
either column "dbo" or the user-
defined function or aggregate
computedColumn, or the name is
ambiguous.
OR
Error - encountered during Data
Extract for table:
computedColumn is not a column
in table
The
com
pute
d co
lum
n do
es n
ot
exis
t in
the
sour
ce d
atab
ase.
T
he c
ompu
ted
colu
mn
mus
t be
able
to b
e ge
nera
ted
to
the
data
base
usi
ng th
e D
efin
e C
ompu
ted
Col
umns
(D
CC
) fo
rm. I
t may
be
that
the
colu
mn
need
s to
be
rege
nera
ted
in o
rder
for
the
Dat
aOrc
hest
rato
r to
use
it.
5.R
ecei
ve a
n er
ror
such
as
the
follo
win
g w
hen
runn
ing
a re
fres
h:
The file separator detected
("/") is not the same separator
in the file path "..."
The
targ
et R
DA
S li
sten
er d
oes
not
also
hav
e th
e D
BA
S r
ole
it ne
eds,
or
the
liste
ner
spec
ified
on
the
DO
PA
fo
rm is
inco
rrec
t.
Ver
ify th
at th
e ta
rget
RD
AS
list
ener
has
bot
h th
e D
BA
S
and
RD
AS
rol
es a
ccor
ding
to th
e in
stal
latio
n in
stru
ctio
ns. V
erify
that
the
liste
ner
nam
e on
the
DO
PA
fo
rm is
the
corr
ect s
ourc
e da
taba
se li
sten
er.
6.R
ecei
ve a
n er
ror
such
as
the
follo
win
g w
hen
runn
ing
a re
fres
h:
No source listener host name
specified
OR
No source extract file path
specified
The
DO
PA
form
is n
ot d
efin
ed fo
r a
Uni
Dat
a or
Ora
cle
sour
ce
envi
ronm
ent.
Ver
ify th
at th
e pa
ram
eter
s on
the
DO
PA
form
are
co
rrec
t.
Tab
le 3
: T
roub
lesh
ootin
g Is
sues
(co
nt’d
)
Issu
eP
oss
ible
Cau
seR
eso
luti
on
Using the DataOrchestrator ODS™, June 18, 2010 251© 2010 Datatel, Inc.
Appendices: Troubleshooting the DataOrchestrator ODS
7.R
ecei
ve a
n er
ror
such
as
the
follo
win
g w
hen
runn
ing
a re
fres
h:
Could not RPC execute command
The
tran
sfor
m is
taki
ng o
ver
an h
our
for
a U
niD
ata
envi
ronm
ent t
o pr
oces
s.
The
Uni
Dat
a R
PC
con
nect
ion
defa
ult t
imeo
ut n
eeds
to
be in
crea
sed.
The
Uni
Dat
a un
irpcs
ervi
ces
file
need
s th
e ud
api_
serv
er ti
meo
ut in
crea
sed
from
360
0 to
144
00.
The
set
ting
requ
ires
unirp
cd to
be
rest
arte
d in
ord
er to
ta
ke e
ffect
.
For
exa
mpl
e:
/usr/ud71/unishared/unirpc/
unirpcservices:
defcs /cm/tools/ud71_10/bin/udapi_server
* TCP/IP 0 14400
8.R
ecei
ve a
n er
ror
such
as
the
follo
win
g w
hen
runn
ing
a re
fres
h w
ith a
SQ
L S
erve
r ta
rget
dat
abas
e:
Unable to log in to target
database. Failed Login to
ods_test on sdw2k3qasql1 by
abcdef: Fatal: 18456(14): Login
failed for user 'abcdef'
A u
ser
nam
ed "
abcd
ef"
is n
ot a
ble
to
conn
ect t
o th
e ta
rget
SQ
L S
erve
r da
taba
se w
ith S
QL
Ser
ver
auth
entic
atio
n.
Che
ck th
e ta
rget
SQ
L S
erve
r da
taba
se in
stan
ce w
here
od
s_te
st is
set
up
to b
e su
re th
at a
use
r na
med
"ab
cdef
" is
def
ined
with
SQ
L S
erve
r au
then
ticat
ion.
Als
o, b
e su
re
the
secu
rity
for
that
SQ
L S
erve
r in
stan
ce is
set
up
to
allo
w fo
r bo
th W
indo
ws
and
SQ
L S
erve
r au
then
ticat
ion.
9.W
hen
chec
king
con
nect
ivity
on
the
DO
RE
fo
rm, t
he fo
llow
ing
erro
r is
rec
eive
d:
DMI_OPEN
The
ref
resh
pro
cess
is n
ot a
ble
to
conn
ect t
o ei
ther
the
sour
ce
DM
I_D
AS
list
ener
or
targ
et R
DA
S
liste
ner.
Che
ck th
at a
ll lis
tene
rs fo
r th
e en
viro
nmen
t are
run
ning
. If
som
e ar
e no
t run
ning
, sta
rt th
em, a
nd th
en lo
ok fo
r the
fo
llow
ing:
•O
n th
e D
OR
E fo
rm, b
e su
re th
at th
e R
DA
S li
sten
er
inst
alle
d on
you
r ta
rget
dat
abas
e se
rver
was
se
lect
ed.
•If
your
sou
rce
data
base
is U
niD
ata
or O
racl
e, th
en
acce
ss th
e D
OP
A fo
rm a
nd b
e su
re th
e D
MI_
DA
S
liste
ner
is th
e D
MI_
DA
S li
sten
er o
n yo
ur C
olle
ague
da
taba
se s
erve
r.
Tab
le 3
: T
roub
lesh
ootin
g Is
sues
(co
nt’d
)
Issu
eP
oss
ible
Cau
seR
eso
luti
on
252 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
DataOrchestrator ODS Issues
10.
Rec
eive
an
erro
r su
ch a
s th
e fo
llow
ing:
User error 10015: - unable to
authenticate your login.
Thi
s m
essa
ge o
ccur
s be
caus
e a
pre-
auth
entic
ated
ser
ver
is u
sed
for
the
file
tran
sfer
in th
e D
ataO
rche
stra
tor
OD
S r
efre
sh.
On
the
DM
I Pre
-Aut
hent
icat
ed S
erve
r (D
MC
C)
form
, yo
u m
ust h
ave
your
app
licat
ion
serv
er a
nd y
our
Col
leag
ue d
atab
ase
serv
er d
efin
ed. I
f you
r in
stitu
tion
uses
Uni
Dat
a an
d ha
s no
t dis
trib
uted
the
DA
S, t
hen
your
app
licat
ion
serv
er a
nd C
olle
ague
dat
abas
e se
rver
ar
e th
e sa
me,
so
you
will
nee
d to
ent
er o
nly
one
line.
If yo
u ha
ve th
e D
MC
C fo
rm s
et u
p an
d ar
e st
ill
expe
rienc
ing
issu
es, t
hen
follo
w th
ese
step
s:
Ste
p 1:
Sta
rt th
e fo
llow
ing
liste
ners
with
logg
ing
turn
ed
on: •
The
list
ener
def
ined
on
the
DO
RE
form
.
•T
he li
sten
er in
stal
led
on th
e D
OP
A fo
rm.
Ste
p 2:
Rer
un th
e C
heck
Con
nect
ivity
func
tion
on th
e D
OR
E fo
rm to
rec
eive
the
erro
r ag
ain,
then
sea
rch
for
an e
ntry
in th
e dm
i.log
file
suc
h as
the
follo
win
g:
"Unable to authenticate a client login
from (hp.datatel.com,127.0.0.1)"
Use
the
DN
S n
ame
and
the
IP a
ddre
ss in
this
ent
ry to
de
fine
the
pre-
auth
entic
ated
ser
ver
on th
e D
MC
C fo
rm.
Ste
p 3:
Res
tart
the
liste
ners
aga
in w
ith lo
ggin
g tu
rned
of
f.
11.
Rec
eive
an
erro
r su
ch a
s th
e fo
llow
ing:
Unable to log in to target
database. Failed Login to
<db
_nam
e> on <
serv
er_n
ame> by
<us
er>: Premature end of Stream:
server won't transmit TDS header
Thi
s is
pro
babl
y du
e to
an
inco
rrec
t da
taba
se p
ort s
peci
fied
on th
e D
OTA
form
.
Acc
ess
the
DO
TA fo
rm a
nd v
erify
that
the
corr
ect
data
base
por
t is
spec
ified
.
Tab
le 3
: T
roub
lesh
ootin
g Is
sues
(co
nt’d
)
Issu
eP
oss
ible
Cau
seR
eso
luti
on
Using the DataOrchestrator ODS™, June 18, 2010 253© 2010 Datatel, Inc.
Appendices: Troubleshooting the DataOrchestrator ODS
12.
Rec
eive
an
erro
r su
ch a
s th
e fo
llow
ing:
No suitable driver.
Thi
s er
ror
occu
rs w
hen
usin
g th
e fo
llow
ing
form
s to
cre
ate
or te
st S
QL
view
s on
an
Ora
cle
targ
et d
atab
ase:
•D
ataO
rch
Targ
et V
iew
s (D
OT
V)
•D
ataO
rch
Vie
w S
pec
(DO
VS
)
The
re is
no
Ora
cle
JDB
C d
river
in
stal
led
in th
e U
niD
ata
sour
ce
envi
ronm
ent.
If yo
u ha
ve a
Uni
Dat
a so
urce
en
viro
nmen
t and
you
r ta
rget
da
taba
se is
Ora
cle,
then
you
als
o m
ust i
nsta
ll th
e O
racl
e JD
BC
driv
er
on y
our
sour
ce e
nviro
nmen
t.
Inst
all t
he O
racl
e JD
BC
driv
er u
sing
the
inst
ruct
ions
in
Inst
allin
g th
e O
racl
e JD
BC
Driv
er o
n pa
ge 5
1, b
ut o
n th
e U
niD
ata
sour
ce e
nviro
nmen
t. T
hen
rest
art t
he
sour
ce e
nviro
nmen
t DM
I_D
AS
for
the
chan
ge to
take
ef
fect
.
13.
The
ref
resh
will
“ha
ng”
in a
SQ
L S
erve
r so
urce
env
ironm
ent.
The
re m
ay b
e m
ultip
le v
ersi
ons
of
the
bcp.
exe
file.
Thi
s oc
curs
whe
n m
ultip
le v
ersi
ons
of S
QL
Ser
ver
have
bee
n in
stal
led
on th
e O
DS
ta
rget
dat
abas
e se
rver
, whi
ch is
us
ed fo
r th
e da
ta e
xtra
ct.
Mak
e su
re th
e pa
th e
nviro
nmen
t var
iabl
e fo
r th
e O
DS
ta
rget
dat
abas
e se
rver
is p
oint
ing
to th
e la
test
ver
sion
of
bcp.
If y
ou u
pdat
e th
e pa
th, y
ou m
ust r
esta
rt th
e ta
rget
R
DA
S li
sten
er fo
r th
e ch
ange
to ta
ke e
ffect
.
14.
Rec
eive
the
follo
win
g er
ror:
ERROR, cannot read source file
<<file path>>
The
Sou
rce
Tem
pora
ry F
ile P
ath
ente
red
on th
e D
OP
A fo
rm is
in
corr
ect.
The
Sou
rce
Tem
pora
ry F
ile P
ath
is s
et u
p on
the
DO
PA
fo
rm. C
heck
that
this
pat
h ex
ists
, and
that
the
SY
ST
EM
us
er a
nd/o
r th
e us
er r
unni
ng D
MI h
as fu
ll re
ad/w
rite
acce
ss to
the
fold
er. A
lso,
try
dele
ting
and
re-e
nter
ing
the
entir
e pa
th. S
tray
spa
ces
may
cau
se in
corr
ect p
aths
an
d ar
e no
t eas
y to
see
in U
I for
ms.
Com
men
t: T
he fu
ll er
ror
may
look
som
ethi
ng li
ke th
is:
Error encountered during file transfer
SSFRQ: ERROR, cannot read source file
<<file path>>
15.
Rec
eive
the
follo
win
g er
ror
in a
Uni
Dat
a en
viro
nmen
t:
In BP/_S_RPC_EXEC at line 2
In BP/_S_RPC_EXEC at line 2 too
many names in LIST
The
tran
sfor
m b
eing
ref
resh
ed h
as
over
150
col
umns
spe
cifie
d. F
or
mor
e in
form
atio
n, s
ee S
uppo
rt
Sol
utio
n 43
04: U
niQ
uery
LIS
T fi
eld
limit.
Spe
cify
150
col
umns
or
less
for
the
tran
sfor
m. Y
ou c
an
eith
er b
e m
ore
sele
ctiv
e ab
out t
he c
olum
ns in
clud
ed fo
r th
e tr
ansf
orm
, or
split
the
colu
mns
into
mul
tiple
tr
ansf
orm
s.
Tab
le 3
: T
roub
lesh
ootin
g Is
sues
(co
nt’d
)
Issu
eP
oss
ible
Cau
seR
eso
luti
on
254 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
DataOrchestrator ODS Issues
16.
Rec
eive
the
follo
win
g ty
pe o
f err
or:
Violation of PRIMARY KEY
constraint
Thi
s is
usu
ally
a s
peci
fic d
ata
issu
e w
here
dup
licat
e ke
ys e
xist
for o
ne o
r m
ore
files
/tabl
es. T
he s
ourc
e of
the
prob
lem
may
not
alw
ays
be
appa
rent
, bec
ause
the
extr
act a
nd
tran
sfor
mat
ion
proc
ess
split
s so
urce
fil
es in
to m
any
diffe
rent
tabl
es o
n th
e ta
rget
.
See
Sup
port
Sol
utio
n 64
38: O
DS
: “V
iola
tion
of
PR
IMA
RY
KE
Y c
onst
rain
t” S
QL
erro
r.
17.
Rec
eive
the
follo
win
g er
rors
in C
olle
ague
S
QL
Ser
ver
envi
ronm
ent o
n th
e D
OR
E
form
whe
n ch
ecki
ng c
onne
ctiv
ity o
r ru
nnin
g a
refr
esh:
SQLState = 08001, NativeError =
53Error = [Microsoft][SQL Native
Client]Named Pipes Provider:
Could not open a connection to
SQL Server [53]. SQLState =
HYT00, NativeError = 0Error =
[Microsoft][SQL Native
Client]Login timeout
expiredSQLState = 08001,
NativeError = 53Error =
[Microsoft][SQL Native Client]An
error has occurred while
establishing a connection to the
server. When connecting to SQL
Server 2005, this failure may be
caused by the fact that under
the default settings, SQL Server
does not allow remote
connections.
Thi
s er
ror
can
occu
r du
ring
exec
utio
n of
the
bcp
stat
emen
t tha
t ex
trac
ts d
ata
from
the
Col
leag
ue
SQ
L S
erve
r da
taba
se. T
o co
nfirm
th
is, t
urn
logg
ing
on in
the
DA
S a
nd
then
rer
un th
e re
fres
h an
d vi
ew th
e lo
g. In
the
log,
find
the
SB
XT
Q
inco
min
g an
d ou
tgoi
ng tr
ansa
ctio
ns
whi
ch lo
ok s
omet
hing
like
this
:
incoming:
DMIþ2.0þDBIQþþþsql17746142
01þþþþþþþþþþþSBXTQþ9þ0þf:\
datatel\coll18\test\ODS_LI
STENER\ods_temp\þ~|~þDOR70
64.datþSYSDEFSþSELECT
SYSDEFS SAMPLE 5 RETURNING
DELIM @IDþSBXTQ.END
outgoing:
DMIþ1.4þDBISþþþsql17746142
01þþýj3þ14821þ54787þþþþþþþ
SERRSþ7þ0þþþSQLState =
08001, NativeError =
53Error = [Microsoft][SQL
Native Client]Named Pipes
Use
the
follo
win
g M
icro
soft
reso
urce
to u
pdat
e th
e se
rver
nam
e re
turn
ed b
y th
e S
QL
stat
emen
t SE
LEC
T
@@
SE
RV
ER
NA
ME
:
http
://m
sdn.
mic
roso
ft.co
m/e
n-us
/libr
ary/
ms1
8794
4.as
px
Tab
le 3
: T
roub
lesh
ootin
g Is
sues
(co
nt’d
)
Issu
eP
oss
ible
Cau
seR
eso
luti
on
Using the DataOrchestrator ODS™, June 18, 2010 255© 2010 Datatel, Inc.
Appendices: Troubleshooting the DataOrchestrator ODS
(con
tinue
d)Provider: Could not open a
connection to SQL Server
[53]. SQLState = HYT00,
NativeError = 0Error =
[Microsoft][SQL Native
Client]Login timeout
expiredSQLState = 08001,
NativeError = 53Error =
[Microsoft][SQL Native
Client]An error has
occurred while establishing
a connection to the
server.When connecting to
SQL Server 2005, this
failure may be caused by
the fact that under the
default settings SQL Server
does not allow remote
connections.þSERRS.END
If yo
u se
e an
SE
RR
S r
espo
nse,
this
m
eans
the
bcp
stat
emen
t tha
t the
R
DA
S r
an fa
iled.
To
figur
e ou
t the
se
rver
nam
e to
use
in th
e bc
p st
atem
ent,
the
RD
AS
run
s th
e fo
llow
ing
SQ
L st
atem
ent t
o ge
t the
se
rver
nam
e:
SELECT @@SERVERNAME
You
can
run
this
in M
anag
emen
t S
tudi
o. If
the
serv
er n
ame
is
inco
rrec
t, us
e th
e M
icro
soft
link
give
n in
the
Res
olut
ion
on p
age
255
to u
pdat
e th
e se
rver
nam
e.
Tab
le 3
: T
roub
lesh
ootin
g Is
sues
(co
nt’d
)
Issu
eP
oss
ible
Cau
seR
eso
luti
on
256 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
DataOrchestrator ODS Issues
(con
tinue
d)T
he @
@S
ER
VE
RN
AM
E m
ight
be
inco
rrec
t fro
m c
hang
ing
a se
rver
's
nam
e. If
you
cha
nge
the
nam
e, b
ut
do n
ot u
pdat
e th
e @
@S
ER
VE
RN
AM
E in
SQ
L S
erve
r, th
en y
ou w
ill li
kely
run
into
this
issu
e du
ring
a D
ataO
rche
stra
tor
OD
S
refr
esh.
To s
ee if
the
issu
e is
due
to a
m
ism
atch
, run
the
follo
win
g on
the
sour
ce d
atab
ase
serv
er a
nd c
heck
to
see
if it
ret
urns
the
corr
ect s
erve
r:
SELECT @@SERVERNAME
18R
ecei
ve a
mes
sage
in S
QL
Ser
ver
Col
leag
ue s
ourc
e en
viro
nmen
t sim
ilar
to
the
follo
win
g:
SQ
LSta
te =
280
00, N
ativ
eErr
or =
18
456E
rror
= [M
icro
soft]
[SQ
L N
ativ
e C
lient
][SQ
L S
erve
r]Lo
gin
faile
d fo
r us
er.
The
SQ
L S
erve
r bc
p co
mm
and
trea
ts c
erta
in c
hara
cter
s as
pa
ram
eter
/com
man
d te
rmin
ator
s.
Ver
ify y
our
sour
ce e
nviro
nmen
t us
erna
me
and
pass
wor
d do
not
in
clud
e an
y of
the
follo
win
g ch
arac
ters
: []
{}()
, ; ?
* !
@
If yo
ur S
QL
Ser
ver
Col
leag
ue s
ourc
e en
viro
nmen
t us
erna
me
or p
assw
ord
incl
udes
one
of t
hese
ch
arac
ters
, cha
nge
your
pas
swor
d so
that
non
e of
thes
e ch
arac
ters
is u
sed.
Tab
le 3
: T
roub
lesh
ootin
g Is
sues
(co
nt’d
)
Issu
eP
oss
ible
Cau
seR
eso
luti
on
Using the DataOrchestrator ODS™, June 18, 2010 257© 2010 Datatel, Inc.
Appendices: Troubleshooting the DataOrchestrator ODS
258 Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Index
From this index you can click on any entry to access the information about the topic.
AActivating, stored computed columns 181, 183, 184
Additional Refresh Parameters (DORP) form 174
Additional Target Parameters (DOTP) form 78
Additional target parameters for 77
Aggregation operations 122
Alertscollation sequence for iStrategy 40directory must have sufficient space 158, 178secure DOTA and DORE forms 38secure ODS target database 38, 177target database must be refreshed by only one
target 158using multiple threads for refresh affects system
resources 173
BBNGN form 86, 110
Bundle Generation (BNGN) form 86, 110
CCalculating, stored computed columns 181
Cardinality rules 107
CDSU form 194
Colleague Data Scanner Utility (CDSU) form 194
Column documentation for SQL view 147
Column Properties (DOPR) form 113
CopyingSQL views 219targets to another target 227transforms 209
Copying and renamingSQL views 219transforms 209
Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
CreatingSQL view 153SQL views using the DOTV form 154
DData mapping of Colleague data 25
DataOrch Error Analysis (DOEA) form 194
DataOrch Object Delete (DOOD) form 232
DataOrch Parameters (DOPA) form 60
DataOrch Refresh (DORE) form 153, 166
DataOrch Source Transform (DOST) form 83
DataOrch Target (DOTA) form 71
DataOrch Target Copy (DOTY) form 228
DataOrch Target Transform (DOTT) form 101
DataOrch Target Views (DOTV) form 145
DataOrch Transform Maint (DOMA) form 210
DataOrch Transform Summary (DOTS) form 200
DataOrch View Maintenance (DOVM) form 220
DataOrch View Spec (DOVS) form 147
DataOrchestrator ODSarchitecture 26benefits 22configuration of hardware and software 29defining parameters 59deleting objects 231installing 45, 47overview of using 55prerequisites 35suggested workflow 55understanding 22
DCC form 86, 110
Define Computed Column (DCC) form 86, 110
Define Stored Computed Column (DSCC) form 183, 184
259
Index
Definingadditional target parameters 77DataOrchestrator ODS parameters 59ODS refresh 165operations for transform columns 116source transforms 82SQL select statement 147SQL view 145target transform column properties 113target transform columns 107target transforms 100targets 67
DeletingDataOrchestrator ODS objects 231refresh history records 231refreshes 231SQL views 219targets 231transforms 209
Delimiters 118
Dependent transforms 150
DOCA form 129
DODS optional module 46
DOEA form 194
DOEE form 138
DOFC form 89
DOFE form 117
DOFR form 123
DOGT form 93
DOMA form 210
DOMV form 120
DONV form 126
DOOD form 232
DOPA form 60
DOPR form 113
DORE form 153, 166
DORP form 174
DOSS form 132
DOST form 83
DOTA form 71
DOTC form 107
DOTP form 78
DOTS form 200
260
DOTT form 101
DOTV form 145
DOTVS form 147
DOTY form 228
DOVC form 135
DOVM form 220
DSCC form 183, 184
EEnvision data mapping 25
Error analysis report 193
Errorscorrecting in source environment using CDSU 194viewing after ODS refresh 77, 193
Expression Entry (DOEE) form 138
Expression Entry operation 112, 138
FField Extract (DOFE) form 117
Field Extract operation 111, 117
File suite instances 86, 92, 104, 125
Filter Criteria (DOFC) form 89
Forms, Datatel softwareAdditional Refresh Parameters (DORP) 174Additional Target Parameters (DOTP) 78Bundle Generation (BNGN) 86, 110Colleague Data Scanner Utility (CDSU) 194Column Properties (DOPR) 113DataOrch Error Analysis (DOEA) 194DataOrch Object Delete (DOOD) 232DataOrch Parameters (DOPA) 60DataOrch Refresh (DORE) 153, 166DataOrch Source Transform (DOST) 83DataOrch Target (DOTA) 71DataOrch Target Copy (DOTY) 228DataOrch Target Transform (DOTT) 101DataOrch Target Views (DOTV) 145DataOrch Transform Maint (DOMA) 210DataOrch Transform Summary (DOTS) 200DataOrch View Maintenance (DOVM) 220DataOrch View Spec (DOVS) 147Define Computed Column (DCC) 86, 110Define Stored Computed Column (DSCC) 183,
184Expression Entry (DOEE) 138Field Extract (DOFE) 117
Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Index
Forms, Datatel software (cont’d)Filter Criteria (DOFC) 89Generated Target Transforms (DOGT) 93Multivalue Operation (DOMV) 120Null Value Replacement (DONV) 126Pointer Reference (DOFR) 123String Concatenation (DOCA) 129Substring Selection (DOSS) 132Transform Columns (DOTC) 107Update GPA SCC Flags (UGSF) 183, 188Update Stored Computed Column (USCC) 183,
186Validation Code LookUp (DOVC) 135
Full refresh 166definition 15
GGenerated Target Transforms (DOGT) form 93
HHistory record ID, three parts of 206
History, viewing for ODS refresh 203
IIDX filegroup 42
Improving refresh performance 166
Incremental refreshconsiderations 166definition 15
InstallingDataOrchestrator ODS 45, 47JDBC driver for Oracle 51overview for the DataOrchestrator ODS 34software updates 47
JJDBC driver, installing for Oracle 51
MMaterialized view
copying 220defining parameters 147definition 143dependent transforms 150
Multivalue Aggregation operation 111, 120
Multivalue Operation (DOMV) form 120
Multivalue Position operation 111, 120
Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
NNull Replacement operation 111, 126
Null Test operation 111, 126
Null Value Replacement (DONV) form 126
OODS history, viewing 203
ODS refreshcreating SQL views 171defining 165viewing errors for 193viewing history for 203
ODS target databasepreparing 38SQL Server script for creating 40
ODS target database server, setting up 47
ODS targetsactivating stored computed columns 181calculating stored computed columns 181
Operational data storesdefinition 14facts about 24limitations 24where to build 36
Operationsaggregation 122available arguments 140definitions 111expression entry 138field extract 117forms used to define 116multivalue aggregation 120multivalue position 120null replacement 126null test 126pointer reference 123string concatenation 129substring 132validation code lookup 135
Optional module, adding 46
Oracleconnection considerations 37database considerations 37installing JDBC driver 51
261
Index
PPerformance of refresh 166
Permissionsdirectory for temporary files for Bulk Load
transaction 48directory for temporary files from Bulk Extract
transaction. 49for user refreshing DataOrchestrator ODS 48source temporary files directory 63, 176target SQL Server database 243target temporary files directory 79troubleshooting 248, 249
Pointer Reference (DOFR) form 123
Pointer Reference operation 111, 123
Prerequisites for DataOrchestrator ODS 35
Proceduresactivating stored computed columns 184adding the IDX filegroup 42calculating foreign keys for transforms 217calculating stored computed columns 186copying and renaming SQL views 226copying and renaming transforms 216copying one or more targets to another target 230copying SQL views 224copying transforms 214creating SQL views using the DOTV form 154defining a refresh 177defining a target 158defining DataOrchestrator ODS parameters 64defining SQL views 152deleting DataOrchestrator ODS objects 233deleting SQL views 223deleting transforms 213installing the Oracle JDBC driver 51regenerating source transforms 218renaming SQL views 225renaming transforms 215running the error analysis report 197running the transform summary report 202updating flags in STUDENT.TERMS.CC file 188viewing the history of a refresh 206
262
RRDAS Listener 169
Recommendationsbatch commit size set at 1000 62monitor space usage in temporary directory 48, 49,
63, 79, 175run USCC before run DORE 183secure access to the DORE form 177secure DOTA 38secure ODS target database 38set up automatic refresh for an ODS with stored
computed columns 24set up process handler 183use DNS alias name 73use the UGSF process before the USCC process
187workflow for DataOrchestrator ODS 55
Refresh history records, deleting 231
Refresh ID, cannot change associated target ID 177
Refresh performance 166
Refresh statuses 167
Refreshes, deleting 231
RenamingSQL views 219transforms 209
Reporting Data Access Server (RDAS) 15
SSchema name, definition 74
SecuringDORE 38, 177DOTA 38ODS target database 38, 177
Setting upforms to set up targets 68ODS target database server 47
Software updatesinstalling 47
Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
Index
Source transformsdefining 82definition 14, 71setting up 83
SQL select statementdefining 147
SQL Serveradding IDX filegroup 42connection considerations 36database considerations 36
SQL viewcolumn documentation 147copy and renaming 226copying 224creating 153defining 145defining SQL select statement 147definition 15deleting 223renaming 225testing 151
Stored computed columnsactivating 181, 183, 184calculating 181
Stored procedure 172
String Concatenation (DOCA) form 129
String Concatenation operation 111, 129
Substring operation 111, 132
Substring Selection (DOSS) form 132
System prerequisites for DataOrchestrator ODS 35
TTarget transform columns
defining 107defining properties for 113
Target transformsdefining 100definition 14, 71viewing for a target 199
Targetsadditional parameters 77copying to another target 227defining 67definition 14deleting 231forms used to set up 68
Using the DataOrchestrator ODS™, June 18, 2010© 2010 Datatel, Inc.
TestingSQL view 151
Transform Columns (DOTC) form 107
Transform columns, defining operation for 116
Transform Summary report 199
Transformscalculating foreign keys for 217copying 214copying and renaming 216deleting 213regenerating source transforms 218renaming 215
UUGSF form 183, 188
Update GPA SCC Flags (UGSF) form 183, 188
Update Stored Computed Column (USCC) form 183, 186
USCC form 183, 186
VValidation Code LookUp (DOVC) form 135
Validation Code LookUp operation 111, 135
Viewingerrors for an ODS refresh 193history of an ODS refresh 203target transforms for a target 199
WWhere to build operational data stores 36
263