DATRAN Database Import Utility

DATRAN Database Import Utility

User Guide

DATRAN Database Import Utility Revision 02

DATRAN Database Import Utility– User Guide © 2019 QTech Data System Ltd.

2

Contents Introduction ..................................................................................................................... 5

About This User Guide .............................................................................................................. 5 Intended audience ................................................................................................................... 5

Product Overview ............................................................................................................. 5 Importing Data from Third party Data Loggers .......................................................................... 5 Importing Data from QTech XL4 Plus RTU logs ........................................................................... 6

Getting started ................................................................................................................. 7 Preparing for Installation .......................................................................................................... 7 System Requirements ............................................................................................................... 7 Installation Process .................................................................................................................. 7 Main Window .......................................................................................................................... 7 Configure Database Connections .............................................................................................. 8

Server .............................................................................................................................................. 8 Database ......................................................................................................................................... 8 User name and Password ............................................................................................................... 8

Importing Data from Third Party Data Loggers .................................................................. 9 Key Features of CSV import workflow ....................................................................................... 9 CSV Files and Formats .............................................................................................................. 9 Supported Common Extensions: ............................................................................................. 10

A header column ........................................................................................................................... 10 Date and Time ............................................................................................................................... 10 Site Name ...................................................................................................................................... 10 Multiple Value columns ................................................................................................................ 10 Records starting a few rows into the file ...................................................................................... 10 A different delimiter has been used ............................................................................................. 10

Unsupported Common Extensions .......................................................................................... 11 Filename Information ................................................................................................................... 11 Fixed Width Fields ......................................................................................................................... 11

DATRAN Database Fields ........................................................................................................ 11 Site name ...................................................................................................................................... 11 Equipment name ........................................................................................................................... 11 Point .............................................................................................................................................. 11 Timestamp .................................................................................................................................... 11 Value ............................................................................................................................................. 11 Units (Analogue points only) ......................................................................................................... 11 Min (Analogue points only) ........................................................................................................... 11 Max (Analogue points only) .......................................................................................................... 11

Creating a CSV Map File .......................................................................................................... 12 CSV Import .................................................................................................................................... 12 First row contains site name ......................................................................................................... 12 Column headers on row above first data row .............................................................................. 12 Data starts on row ......................................................................................................................... 12 Field Delimiter ............................................................................................................................... 12

Database Mapping ................................................................................................................. 13 Mapping Type ............................................................................................................................... 13 Creating a link ............................................................................................................................... 13 Deleting a link ............................................................................................................................... 13


3

Manually editing a link .................................................................................................................. 13 Example Data ................................................................................................................................ 13 Multiple Data Columns ................................................................................................................. 14 Save the Mapping ......................................................................................................................... 14

Importing Process .................................................................................................................. 14

Importing an RTU Log File ............................................................................................... 15 Retrieve the log file ................................................................................................................ 15 Configure the Database Connections. ..................................................................................... 15 Create an RTU Map File .......................................................................................................... 15 Import the log file .................................................................................................................. 17 Validate the Data ................................................................................................................... 17

Validating Import using QTech Trending ......................................................................... 18

Warranty ........................................................................................................................ 19

Additional Information and Support ............................................................................... 19


4

Copyright 2019 QTech Data Systems Limited

Christchurch, New Zealand.

All rights reserved.

The circuit details, Software algorithms and know how disclosed in this document are proprietary to QTech Data Systems Limited and shall remain the intellectual property of QTech Data Systems

Limited.

DISCLAIMER

The information in this document is subject to change without notice and does not represent a commitment on any part of QTech Data Systems Limited. While the information contained herein is

assumed to be accurate, QTech Data Systems Limited assumes no responsibility for any errors or omissions.

Contact QTech QTech Data Systems Limited 12 Midas Place, Middleton Christchurch 8024 New Zealand Phone: +64-3-366-3713 Fax: +64-3-365-2815 Email: [email protected] Web: www.qtech.co.nz

Revision Details 01 April 2019 Initial Issue (imported from Help file)

02 November 2019 add feature to import Historical data log from RTU

mailto:[email protected]

http://www.qtech.co.nz/


5

Introduction

About This User Guide

This User guide provides information on how to install and use the QTech DATRAN™ Database Import Utility. The utility works in conjunction with the DATRAN™ SCADA Database system.

Additional Information and reading:

• QTech DATRAN Charting/Trending User Guide

• QTech Workbench User Guide (for XL4 Plus RTU)

• QTech DATRAN Browser Application Help

Intended audience

This guide is for systems administrators of DATRAN™ SCADA systems installations. It assumes that you have been trained on and are familiar with QTech Data Systems DATRAN™ products, their configuration and their terminology. Users are also expected to have a working knowledge of spreadsheet applications such as Microsoft Excel™.

Product Overview The Database Import Utility enables users of DATRAN systems to capture telemetry log information from third party data logging devices and import the data into the DATRAN system for analysis. Information is initially stored in spreadsheet form then mapped to equivalent equipment datapoints in the DATRAN Database. The information is then imported and added as records into the DATRAN SQL database to enable tools like QTech Trending to graph the data.

QTech telemetry (RTU) products such as the XL4 Plus RTU also have a data logging capability. Under normal circumstances log information is directly transferred to DATRAN via a communications link and automatically added to the SQL Database for historical data logging. If the communications link is not available then the log can also be retrieved from the RTU and stored as a log file by using the QTech Workbench utility. This file can then be imported into DATRAN using the Database Import Utility. This provides a secondary method of capturing valuable log information in the event of extended communications outages between DATRAN and the RTU.

Importing Data from Third party Data Loggers The Database Import application provides a customisable method for taking your comma separated value (CSV) data and placing it in the DATRAN database. Once there it can be analysed using existing DATRAN tools such as QTech Trending or DATRAN Reporting. Data can be retrieved and input from datalogging devices if the data can be assembled into a CSV format.

Although many software applications can export to CSV Format, this format has no rigid structure which means that a CSV file exported from one application will, most likely, not be able to be imported in another. It is because of this an intermediate file is required. This provides the mappings between the various fields within the CSV file and the required fields in the DATRAN Database.

The typical workflow process is shown below:

Prepare Log File

• Convert device data to CSV format spreadsheet

Import Log File

• Launch Import utility

• Create map file from CSV to DATRAN fields

• Connect to DATRAN Database

• Select CSV file or folder multiple CSV files

• Import to Database

Verify and Chart Data

• Launch DATRAN Trending

• Add data points and plot data


6

Import CSV

Map CSV field to DATRAN™

Chart and Report from DATRAN™

DATRAN™ Database

Importing Data from QTech XL4 Plus RTU logs The import utility provides a workflow for capturing historical data from a QTech RTU (XL4 Plus) and importing it into the database. This is particularly useful for filling in database records if communications between DATRAN and the RTU are suspended for a period but the RTU continues to log data, for later retrieval but for some reason the log can’t be retrieved by DATRAN.

In this case the log can be retrieved using the QTech Workbench utility, stored as a binary formatted file then imported into the DATRAN SQL database using the import utility.

Retrieve Log File

•Connect to RTU

•Run QTech Workbench

•Download Historical datalog file and Save

Import Log File

• Export DATRAN Site Data (TBD map file)

•Run Import utility

• create map file to auto generate mapping to Datran fields

•Select historical datalog file

•Import to Database

Verify and Chart Data

•Launch DATRAN Trending

•Add data points and plot data


7

Getting started

Preparing for Installation

The QTech Workbench software is free to download from the QTech Website. Contact QTech for the link. The product requires a license key to be entered and this can be supplied by QTech on request.

System Requirements

Installation Requirements:

• A PC running Microsoft Windows 7 or later

• Recommended minimum 4 GB ram, 5 GB hard Drive

• .NET Framework V2.0

Installation Process

Before installing it is recommended that you close all applications running on the target machine.

Run the installer and review and accept the software licencing agreement to install the product. Follow any on screen instructions.

The licensing of the Database Import application is linked to the machine that it is installed on. The initial installation will create a temporary 14 day license that can be used for evaluation purposes. Thereafter a full license needs to be obtained from QTech. The user must enter a valid license key provided by QTech by clicking the help menu and selecting Enter License Key from the menu.

To obtain a license you must supply the hard drive disk volume identifier of the workstation. (open a command windows (cmd) and type vol at the prompt to diplay the volume ID.

Main Window

The application is designed so that the user can create a map file to map fields in the imported spreadsheet to DATRAN then by clicking the appropriate button import either a single or multiple datum input (CSV) files. Alternatively log files from an RTU (QTech XL4 Plus) can be imported one at a time.

The workflow requires you to:

1. Configure the database connections to the DATRAN SQL database and the DATRAN Service (TBD) Database (for RTU log imports only)

2. Select the type of import to undertake 3. Create and use a mapping file to match data from the log to database fields 4. Load the log file or files to import 5. Run the importer 6. Check the results using an application such as DATRAN Trending.

Progress and errors conditions are reported in the Status report area.

Figure 1: Main User Interface


8

Configure Database Connections Before using the application, a connection to a DATRAN SQL database must be made. Immediately after installation the user is prompted to configure a connection but a connection can also be configured at any time by selecting Database connection from the configuration menu.

Figure 2

This allows you to specify where your DATRAN database is and a valid user account to successfully connect to it.

Once you have entered this information you need to press the Test button. This will determine if the information you have entered is correct and allows a successful connection to the database. If the connection is not successful then you will be prompted with the reason for the failure and are then required to make appropriate changes and try again.

When a successful connection has been made the Save button will become enabled. Pressing this will store the settings for future sessions.

Server The computer name or IP Address of where the SQL Database resides. E.g.

QTech22 or 192.168.0.123 or localhost

If you are using SQL Express then the above may need to be postfixed with \SQLExpress. E.g.

QTech22\SQLExpress or 192.168.0.123\SQLExpress or localhost\SQLExpress

If the database is exposed on a specific IP Port then this needs to be specified after a comma. E.g.

QTech22,4664

Database This is the name of the DATRAN database. By default DATRAN installs this as DATRANdatabase

User name and Password A valid user account configured within your SQL database. The default administrator user is normally sa

Note: Currently Windows Authentication is not supported.


9

Importing Data from Third Party Data Loggers

Key Features of CSV import workflow

• Graphical Mapping Interface Flexible CSV column configuration via drawing mapping lines between CSB columns and database fields means that it this product work with most CSV formats, not just the ones brands of loggers listed above

• CSV Embedded Site Name Some Loggers output the name of the site as the first row in the CSV file.

• CSV Column Headings It is quite common for CSV files to have a row of column headers. These are then used by Database Import to make the mapping more user friendly by displaying these names as opposed to column numbers.

• Record Start Row This is user definable for CSV files where the records do not start on the first row

• User Specified Column Separators Database Import can also be used for Tab separated files.

• Digital and Analogue Points Digital point information in the form of 0s and 1s or Analogue information with associated scaling and units can be imported

• Multiple Data Columns It is normal for loggers to store multiple values per timestamp E.g. a single logger could be recording air temperature and soil moisture. Database Import can be configured to handle this information in a single pass of the CSV file.

• Combining CSV Columns Columns can easily be joined using the graphical mapping interface. E.g. some CSV files may have the Date and Time in separate columns. These then need to be joined in order to be imported into the DATRAN database.

• Filing Columns with Arbitrary Text If any of the required fields cannot be mapped from the CSV columns then that can be assigned a fixed text value.

• Importing Multiple Files at a Time The Import Folder feature allows all files in a directory to be imported using the same map file.

• Status Reports A status report is generated for each CSV file that has been imported

• SQL Transaction Rollbacks In the event of CSV import failure the problem is reported in the status report and all records imported from that CSV file are removed from the database leaving it in a known state

CSV Files and Formats The Comma-Separated Values (CSV) file format provides a means of storing tabulated data. There is no standard file format specification for this other than a few basic rules:

• Each row contains information about a single record

• Each column contains information about a field within that record.

• Columns are normally separated by a comma

• Rows are separated by a carriage return. Data Loggers normally store records in the form of Timestamp, Logged Value For example:

10:21:05,23 10:22:09,14


10

Supported Common Extensions: A header column Given the above example there is no way of knowing what the data refers to some file formats contain an additional row stating what each column refers to. E.g.

Time,Level 10:21:05,23 10:22:09,14

This can be handled in the mapping screen by ticking the Column headers on row above first data row check box.

Date and Time Sometime these fields are combined together and sometimes they are comma separated E.g.

19/05/2008 10:21:06,23 19/05/2008 10:22:09,14 or 19/05/2008,10:21:06,23 19/05/2008,10:22:09,14

This can be handled in the mapping screen by mapping both the Date and the Time fields to the Timestamp database field.

Site Name Some data loggers use the first row of the CSV file to contain the site name (the location from where the data came). E.g. Main Road Sewer 19/05/2008 10:21:06,23 19/05/2008 10:22:09,14 By ticking the First row contains site name check box this information can then be mapped to the Site database field.

Multiple Value columns Some loggers have multiple inputs and can therefore have multiple points logged for a single timestamp. E.g. a logger could measure soil temperature, air temperature and soil water content. The output CSV file could look something like: 19/05/2008 10:21:06,23,28,1.5 19/05/2008 10:22:09,14,18,1.2 In this example each column will represent a different Point in the DATRAN database and will therefore require a slightly different mapping. This can be achieved by selecting the Number of data columns required. Each mapping is then representing in a new tab.

Records starting a few rows into the file If there are additional header rows used for other information it may be that the actual data does start until, say, row 7. This can be handled by changing the value with the Data starts on row edit box.

A different delimiter has been used Occasionally the field delimited is not a comma. Common delimiters include Space, Tab and Semicolon. The Field Delimiter radio buttons allow the alternative to be specified.


11

Unsupported Common Extensions Filename Information Some data loggers store additional information in the actual filename of the CSV file. For example the date of the log retrieval or the name of the site where the data came from etc. This information is not available to the Database Import application.

Fixed Width Fields In this case there are no delimited characters. The fields are all of known width and padded with spaces. This is not currently supported by the Database Import application.

DATRAN Database Fields The DATRAN Database has the following mandatory database fields representing a single record entry in the database:

Site name This represents the site name or location where the data came from. Sometimes it can be extracted from the CSV file if it is present as either the first row or one of the CSV columns. If this information is not present in the CSV file then it will need to be entered manually.

Equipment name Represents the type of equipment that provided the data e.g. Flow Meter. If this information is not present in the CSV file then it will need to be entered manually.

Point The name of the record. E.g. Pump Hours. If this information is not present in the CSV file then it will need to be entered manually.

Timestamp The date and time of the record. Note that the DATRAN database operates at a resolution of 1 second and timestamps must unique. If the CSV file contains 2 or more identical timestamps then the import will fail due to a duplicate entry in the database.

Value Represents the instantaneous value of the Point at the given Timestamp. For digital points the only allowed values are 0 or 1.

Units (Analogue points only) This field is only displayed when creating an analogue mapping. It allows the units of the Value to be specified e.g. l/s, % etc.

Min (Analogue points only) Used by Historical Trending to define the lowest value for the vertical axis for this Point.

Max (Analogue points only) Used by Historical Trending to define the uppermost value for the vertical axis for this Point.


12

Creating a CSV Map File The map file editor can be invoked by either clicking on the Create/Edit button from the main window or selecting New Mapping from the File menu. The screen is divided into two sections: CSV Import and Database Mapping. Select the CSV file as source file type.

Figure 3

CSV Import This section allows you to load your CSV file using the Open Sample CSV button. The grid then shows the first few rows and columns of CSV file using the current settings.

Figure 4

First row contains site name If the first row of the CSV file contains the name of the site where the data came from then by selecting this check box will create a fictitious data column that can be used for the Site name.

Column headers on row above first data row If column headers are supplied then by selecting this check box the column headers will be used both in the grid and the Database Mapping section to make designing the map file easier.

Data starts on row Specifies the row number where the first record can be found. This field updates automatically when either of the above fields are modified.

Field Delimiter The columns in some CSV files are separated by a delimiter other than a comma. If this is the case use these radio buttons to specify what character is being used as the delimiter.


13

After changing any of these settings press the Refresh button to update the contents of the grid.

Database Mapping When the CSV Import Refresh button is pressed the CSV Columns shown at the left hand side of the Database Mapping area update to show the available columns.

Figure 5

In the above illustrations it can be seen that there are four CSV Columns in the Database Mapping but only three in the CSV Import grid. This is due to the fictitious column generated by the First row contains site name check box.

Mapping Type The mapping type of either Digital or Analogue needs to be selected first as the database fields are slightly different for each of these types. The above illustration shows the digital mapping. The analogue mapping has addition fields for Units, Min and Max. The other difference is that for a digital mapping the Value field can only have a 0 or 1 value.

Creating a link links are created by clicking on the CSV Columns selector (triangle) on the left hand side and then clicking on the Database Fields selector on the right hand side. The selectors turns orange to show that they are selected. Alternatively clicking on a column in the CSV Import grid will also highlight the appropriate left hand side selector.

Deleting a link Follows the same procedure as for creating a link. When a link is selected a subsequent time it is toggled. Alternatively the link can be edited directly in the provided edit boxes.

Manually editing a link When a link has been created the edit box shows its internal representation. Fields in square brackets represent references to columns within the CSV file. multiple CSV columns can be combined into a single database field (see the Timestamp in the above illustration). Manually edited text can also be added (see the Equipment Name field above). A combination of both references and manually edited text is also possible.

Example Data Rightmost (underneath the refresh button) shows what the data would look like for a single record in the DATRAN database. If any of the edit boxes were manually changed you will need to press the Refresh button to update these examples. In the case of any errors, for example the time format cannot be parsed or the digital value is not either 0 or 1 the example data column will show the error in red. All errors need to be fixed before the mapping file can be saved.


14

Multiple Data Columns If the CSV file contains multiple columns each containing data that you wish to add to the DATRAN database then you will need to create multiple mappings as follows:

1. Enter the number of data columns in the field provided. You will then noticed that you are presented with one tab per data field. The tab's title comes from the Point field.

2. Complete the database fields on each of the tabs. When this means is that when a row of CSV data is imported it will be passed through each of the mappings created thus creating multiple points in the DATRAN database.

Save the Mapping The mapping cannot be saved if any errors are detected. In this case fix the problems and press the save button again.

Importing Process Once you have a map file importing your CSV file is achieved by following the steps: 1) Click the Browse button or click open mapping from the file menu to load the map file created in the

pervious step. 2) Press the Import File button. Navigate to the folder containing the CSV file to import from. If multiple CSV

files are to be imported use the CTRL or SHIFT keys to select one or more files. Click open to commence the import process.

Figure 6

The Status Report area shows the progress of import along with any errors (displayed in red). This includes the names of the files being imported, how many rows have been imported and any errors that occurred during the import. If there are any errors then the import is aborted and any database changes are rolled back. This puts the database into a known state and allows you to correct the problem and then re-import with the knowledge that duplicate database entries will not occur. The progress bar provides an indication of how far through the current import the application is. When an import is in progress the Cancel button is enabled. If this is pressed during an import the database is rolled back in the same way as if an error occurred. The Status Report indicates that cancel was pressed.


15

Importing an RTU Log File Retrieve the log file To import the Historical Data Log from an XL4 Plus RTU first connect the RTU to a laptop or device on which QTech Workbench has been installed (V2.6 or later). Connect the RTU to the device via a USB cable (Type A-B) and if necessary, wait until the USB device drivers are installed automatically. The click the connect button in Workbench to commence communications with the RTU.

Figure 7

Click Read Data log to retrieve the log file from the RTU. The user is prompted to save the file on the local hard drive of the PC. The file saved as a .DB file, this file cannot be edited with a text editor.

Configure the Database Connections. Follow the steps in section Getting Started to create a connection to the DATRAN SQL Database.

Create an RTU Map File The RTU references data points by their Data type and index. This datapoint relates to a site name (RTU), equipment name and data point in DATRAN (see figure below). Typically, this information is displayed using DATRAN Browser which visualises the data in a tree structure (the TBD).

For example, The RTU log contains the raw data for a data point (e.g. RDI5 for Real Digital Input 5). In DATRAN this may map to say the “Amberley Domain” site, “Pump 1” equipment and “Overheat” measurement point. In addition, the RTU data point will need to be scaled according to the formula for before it can be logged into the SQL database. This is typically the case for analogue inputs.

Figure 8


16

To create a nodes file for an RTU log the user should first select the site name in DATRAN Browser then right click and select export from the pop-up menu. Provide a name for the file (or simply leave it as export.txt). Ensure that relative pathnames are used and that the export starts at the current node.

Figure 9

The exported TBD file becomes the “nodes” source file to load when creating the map. The map file will be parsed automatically to extract the DATRAN field names for mapping the log file data into the database.

From the file menu click new mapping. or if the main menu has no map file loaded, click the create button. Select DB Binary File as source file type. Browse to select the exported nodes file (e.g. export.txt). click the load nodes file button to load the TBD nodes into the application.

Figure 10

Browse the site (or sites) that are in the TBD nodes list and select the site that you want to import into DATRAN. Click the save mapping button to save the select site (and all the nodes associated) into a “map” file.

Note that when you select a site it lists the telemetry datapoints that are available for importing data. If the data point is not listed, then the RTU does not have the data point mapped in DATRAN.

Close the dialogue box when done to return to the main window.


17

Import the log file From the main window Select the map file by clicking the browse button and navigating to the file created in the previous step. Click open to load the file into the application.

Click the Browse button to navigate to the log file saved from Workbench and open it. Only one log file can be imported at a time. Log files are typically named database.db (or something similar)

Click the import button to commence the import process. Progress will be shown in the status report field. The importer will map the raw data in the log file to the database point identifier in the SQL database and will also scale the data according to the scaling formula, then the data will be written to the SQL database.

Errors are also logged to a text log file for inspection. Error conditions may occur if there is no valid point data in the DATRAN TBD structure associated with the Raw datapoint logged by the RTU. Errors can also occur if the raw data cannot be correctly scaled.

Validate the Data As per CSV file imports the Trending application can be used to graph the data and validate that the log was successfully imported. Refer to the DATRAN Tending User Guide for more information.


18

Validating Import using QTech Trending The illustration below shows where the various DATRAN database fields appear when using Historical Trending.

Figure 11

Historical Trending can display up to 8 pens on a single plot. Each pen represents the data for a single Point. If there are multiple analogue pens on display then the Min and Max properties can be used to shift the plot either higher or lower on the vertical axis. This allows the data to be easily viewed. Refer to the QTech Trending User Guide for more information on creating graphical plots of historical data.


19

Warranty The software is covered by QTech Limited Warranty Agreement and software End User License Agreement, respectively. Please refer to the QTech Limited Product Warranty Agreement, which may be downloaded from the QTech website: www.qtech.co.nz QTech Data Systems Limited does not warrant the suitability of this product for any particular application as the conditions in which it is used are beyond our control. This is not withstanding warranty of merchantability.

Additional Information and Support If you have problems try the following:

• Visit the QTech web site for application notes and guides

• Refer to the troubleshooting section if one is present in this document

• Contact the support desk at [email protected]

• Phone the support desk, contact details at beginning of this document

Copyright 2019 QTech Data Systems Limited, all rights reserved.

http://www.qtech.co.nz/

mailto:[email protected]

Date post:	23-Oct-2021
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

DATRAN Database Import Utility

Documents