11. Export File: CSV Column Format ........................................................................................ 16
By
Taek M. Kwon, Ph.D.
Transportation Data Research Laboratory (TDRL)
University of Minnesota Duluth
Last Updated: Dec 12, 2016
1
BullConverter User Manual BullConverter is a utility program that converts raw vendor-dependent WIM data files to a standard csv format called Bull-CSV, supported by the Office of Transportation System Management (OTSM) at the Minnesota Department of Transportation (MnDOT). BullReporter which is a reporting tool used by MnDOT OTSM uses these CSV formatted files as its inputs and produces a wide range of analysis reports. The combination of BullConverter/BullReporter software stack enable a seamless integration of the multi-vendor WIM system environment at MnDOT and produces uniform analysis reports for all sites, regardless of differences in vendors. The BullConverter graphical user interface (GUI) consists of eight tabs: Start, Site Name Edit, IRD Binary, Kistler DL, Intercomp, PVR Conv, Log File, and Export, and this manual describes how to use them.
1. Start Tab
For vehicle classification, BullConverter provides its own classification, and the user must first select a classification scheme from the Start tab. Figure 1 shows a selection of the min5.tye classification scheme. This selection is used by all conversion utilities available in remaining BullConverter tabs. In addition, Bullconveter provides ESAL computation and its parameters are set in the Start tab.
Figure 1: Start tab is used for selecting a Classification Scheme and providing ESAL parameters.
2
Class definition files contain detailed rules on how each class is determined, and
they have a filename extension, “.tye” (English units) or “.tym” (Metric units). These files are text files and located in the Windows AppData folder. This folder is defined in Windows environment variables called, Environment.SpecialFolder.ApplicationData, and the folder location in Windows-7 and typical files contained in this folder are shown in Figure 2. Notice that this AppData folder contains the basic classification definition files developed by MnDOT, which are min5.tye and min54.tye, English unit versions, and min5.tym and min54.tym, metric versions with the same classification scheme. How to write a class definition file is described in another documentation.
The AppData folder also includes a Log folder that saves information related to conversion events and processes. The AppData folder is hidden by default in Windows, and users can make it visible by setting Folder Options to “Show hidden files and folders".
BullConverter uses the “Settings.xml” file in the AppData folder to save user settings, which includes settings for directories, site names, classification scheme, etc. This is a text file and can be edited using Notepad, but users should avoid directly editing this file.
For an advanced user, the “Settings.xml” file is used in the following way. When a new version of BullConverter program is installed on a computer, the installer overwrites the existing Settings.xml file in your computer using the default file in the installation file; consequently, you will lose many of your own settings. One way of dealing with this problem is that you copy the “Settings.xml” file to a temporary location before you install BullConverter. And then you simply copy back the saved “Settings.xml” file to your AppData folder.
3
2. Site Name Edit Tab
The user must create a database of site information records, each record consisting of a site name and a three-digit site ID. This is done by using the second tab “Site Name Edit” shown in Figure 3. Clicking an item from the Site Folder Names listbox shows the site name and ID on the right panel, stored in the database. The New button simply deletes the texts in the Site Name and Site ID textboxes and creates empty textboxes. Clicking the Add button actually adds the site name and ID texts to the database. Once the site information is added in the database, it will remain in the computer until it is deleted using the Remove button in the left panel. Therefore, the user only needs to set a site name and ID once. BullConverter uses this site name and ID to create CSV site folders. Bull-CSV site folder names follow the format: “sitename_###” where ### is a three digit site ID, which are shown on the “Site Folder Names” listbox in the left panel of Figure 3. These site folder names become the destination for the converted CSV files. BullReporter recognizes and utilizes these CSV site folder names created by BullConverter.
Figure 3: Site name and ID editing
4
3. IRD Binary Tab When multiple IRD raw binary files must be converted to Bull-CSV files, the “IRD Binary” tab is used (see Figure 4). To use this utility, two folder locations must be specified. The textbox for “IRD Binary Folder” should be set to the folder where IRD raw binary files are located, and the “CSV Root Folder” should be set to the parent folder of the CSV site folders, typically named “Rawcsv” or "Bull-csv". The folder locations should be entered using the folder Browse buttons on the right side of the textbox to eliminate potential spelling errors. The CSV site folders are automatically created if the site name and ID pairs are saved in the database. If the site name is not specified for a site, it uses a default site name, "Site", and the site ID, i.e., “Site_###” where ### is the three-digit site ID. There is a check mark for selecting year and month in the IRD Binary tab as below:
If this check mark is checked, only the raw binary files in the specified year and the month are converted. In this example, only the binary files in 2013, January are converted. If this check mark is unchecked, all of the binary files starting from the first day of the specified year to present are converted. In this example, all binary files starting from January 1st, 2013 to present are converted. This option is convenient, if it is used along with the “Convert New Only” button since it will only convert new files from the specified year to present. The two buttons are described next. "Convert All (Overwrite)" button: For the selected year and month, all binary files are converted to Bull-CSV files, and then they are saved in the corresponding CSV site folder. If a same-name CSV file already exists in the CSV site folder, it overwrites the existing one without notification. "Convert New Only" button: It first checks, if the Bull-CSV file of the converting raw binary file already exists or not. It then converts only if the corresponding Bull-CSV file does not exist. This routine was designed to be used when new binary files are regularly downloaded and only to convert the new ones downloaded. Note: If all IRD binary files are in one folder, BullConverter will automatically place them in a proper CSV site folder after conversion. Other conversion tabs include the same two action buttons as the IRD binary tab,
providing the same conversion functionality.
5
Figure 4: Conversion of IRD binary files to Bull-CSV files
4. Kistler DL Data Conversion
Kistler Data Logger (DL) computes and saves WIM vehicle records from axle load and vehicle presence sensors installed on a WIM site. There is a program called KisDA (Kistler DL archiver) which downloads vehicle records from DL and then archives them as daily CSV files. The KisDA program creates files with name formatted, yyyymmdd.###.kis where "###" is a three digit site number. The Kistler DL tab shown in Figure 5 is used to convert *.kis files to Bull-CSV formatted files. The Kistler tab GUI and its usages are identical to that of the IRD binary tab. The Kistler source folder must be set to the folder that contains *.kis files and the CSV Root Folder to the destination Bull-CSV root folder. All conversions are then performed using the two buttons provided in the Kistler DL tab. The *.kis Error/Warning codes are mapped to Bull-CSV format according to Table 1.
6
Table 1: Error/Warning Code Mapping to Bull-CSV
*.kis
ErrWarning
Name Description Bull-CSV
ErrWarning
1 Out of Spec Vehicle driving out of
specifications
15
2 Vehicle Processing
Error
Cannot process vehicle 14
70 Velocity above max Driving above specified velocity 46
71 Velocity below min Driving below specified velocity 0
72 Strong acceleration
max
Strong acceleration above
specified max
34
73 Strong deceleration
min
Strong deceleration below
specified min
34
74 High imbalance High left/right weight imbalance 35
75 Sensor missing Force or presence sensor signal
missing
70
76 ADC overload ADC overload 66
77 High vehicle
dynamics
Highly dynamic driving behavior 67
78 Acceleration change Strong changes in acceleration 34
79 Driving between two
lanes
Driving between two adjacent
lanes
68
80 Single-track vehicle Single-track vehicle or vehicle
driving on only one side of the
layout
69
81 Force record missing Force record missing 7
82 Single axle vehicle Single axle vehicle 19
83 Stop and go Vehicle stopped while driving
through the WIM site
16
84 GVW above max Gross vehicle weight above
specified value
44
85 GVW below min Gross vehicle weight below
specified value
65
86 Axle load above max Axle load above specified value 43
87 Axle load below min Axle load below specified value 65
90 Undefined Kistler DL
error/warning
Undefined or unknown DL error
or waning messages, i.e., error or
warning messages that are not
included in 1, 2, 70-87.
65
7
Figure 5: Kistler DL tab for converting *.kis files
5. Intercomp Data Conversion Intercomp WIMLOGIX data files are converted to Bull-CSV files using the Intercomp tab. WIMLOGIX files are text WIM data files in which each field is separated by “;” and produced by Intercomp WIM systems. To use this utility, place WIMLOGIX data files in one folder and select that folder using the Browse button (top right in Figure 6) next to the textbox labeled “Intercomp WIMLOGIX Data Folder.” The Intercomp WIMLOGIX data folder can include data from multiple sites. The two buttons, “Convert All (Overwrite)” and Convert Only New”, work similarly to those in the IRD Binary tab. Conversion from Intercomp error codes to Bull-CSV codes is summarized in Table 2.
8
Figure 6: Conversion of Intercomp WIMLOGIX files
Table 2: Intercomp Error Codes to Bull-CSV Mapping
Intercomp
Error
Description Bull-CSV Error/Warning
Code
AM Axle count mismatch 8
WB Weight balance 31
OS Over speed 17
US Under speed 13
AC Acceleration (positive) violation 33
DE Deceleration violation 33
NL No loop 3
LV Loop violation 3
*OS, US, AC, and DE may contain valid axle weight data, depending on the user threshold setup.
9
6. PVR Conv Tab
This tab is used to convert Peek WIM binary files formatted in PVR (Peek Vehicle Record) to Bull-CSV files. The user interface is similar to that of other conversion tabs. The destination folder is the textbox labeled “CSV Root Folder” and the source folder is the folder that contains the PVR binary files. To overwrite, use the “Convert All (Overwrite)” button and use the “Convert Only New” button to exclude conversion of files already converted. This utility does not include the option of selecting Year and Month.
It should be mentioned that the site name and ID of a PVR binary file are specified inside the binary data (part of binary data). The BullConverter uses this internal name and ID as the site name and ID, not the binary file name available in the file system. This is because the external file name can be different from the internal filename, and such a case the internal name is given a higher priority.
7. Bull-CSV Filename Convention and Column Definitions
BullConverter produces csv files that conform the filename format given by:
yyyymmdd.###.csv where yyyymmdd is (year, month, day) and ### is a three digit site ID. The columns of Bull-CSV files are summarized in Table 3. Each column is separated by comma “,”. If a column data is empty, it means that no data is available or no error in the case of Err# column. The first line of a Bull-CSV file is headings of each column, and the second line is a separator line that consists of additional information on the data. The second line presently consists of three items separated by ";", which are a dash separator “----------“(Ten "-" characters), BullConverter Version # (Release date), and TypFile. For example, -------;Version 4.18 (Released June-29-2016);min5.tye Actual CSV data in Bull-CSV file starts from the third line. The Bull-CSV Error/Warning codes are summarized in Tables 4 and 5. It should be noted that Bull-CSV format allocates only a single column for errors and warnings. They are distinguishable through differences in numeric codes.
10
Table 3: Bull-CSV Column Format
Column
Number
Column name Description
1 Veh# Vehicle number
2 Lane# Lane number of this vehicle passed through
3 Time Time in “hh:mm:ss” format where hh is military
hour, mm is minute, and ss is second.
4 AxleC Number of axles on this vehicle
5 Speed Speed of this vehicle in mph
6-16 Axle Spacing (AS):
AS1,AS2,…,AS11
Axle spacing in feet. It contains a total of 11 fields
separated by comma.
17-28 Axle weights(AW):
AW1,AW2,…,AW12
Axle weights in Kips. It contains a total of 12 fields
separated by comma.
29 GVW Gross Vehicle Weight (GVW) in Kips. It is simply a
summation of all axle weights.
30 Class Vehicle class determined by the classification scheme
defined in the TypFile column (33).
31 Err# It is a collection of numeric codes that represent
errors and warnings (See Tables 4 and 5 for detail).
Multiple error/warning numbers are separated by
colon “:”. For example, “2:12” means detection of
two errors, 2 and 12.
32 100thSec 100th seconds of the vehicle time in Column 3.
33 ESAL Equivalent Single Axle Load (ESAL) computed
based on user selection of ESAL parameters in Start
tab
34 ImageF List of image file IDs of this vehicle separated by “/”.
Example, Image000479_4 (4=lane num)
35 License# License plate reading of this vehicle
36 Temp Temperature in Fahrenheit. It should be a
temperature reading of pavement.
37 calFac Calibration (multiplication) factor applied to the raw
axle weight values. It is normally an adjustment
factor computed based on pavement temperature but
other adjustment factors may be combined.
11
8. Bull-CSV Error/Warning Codes Bull-CSV format allocates a single column for both error and warning. BullConverter converts error and warning codes from raw WIM data files to the Bull-CSV Error/Warning codes summarized in Tables 4 and 5. Error numbers are allocated to 1-20, and warning numbers are allocated to 31 or above. Bull-CSV Error/Warning numbers were generated to be inclusive to error/warning codes of multiple vendors of WIM systems.
68 Driving between two adjacent lanes Kistler DL warning
69 Vehicle driving only on one side of layout Kistler DL warning
70 Force or presence sensor signal missing Kistler DL warning
13
*Vehicle records with warning numbers, 43, 44, 51, 52, 53, 54, and 57, may contain good
weight data.
9. Log File Tab The Log File tab includes the View Log File button in the left plane (Figure 7). Clicking this button displays the log information produced in the process of conversion to Bull-CSV format. Pressing the Delete Log Data button removes the current log and saves the old log to another file, i.e., it backs up the log information and then clears the clutter of the current log. Please ignore the right side plane since it is a utility tool made for BullConverter developers.
Figure 7: Log File tab
This tab is also used to test conversion of a single IRD or PVR binary file to Bull-
CSV. It is convenient to test single file conversions before conversion of a whole folder.
Clicking a conversion button brings up an “Open File” dialog, from which user navigates
and selects the source binary file. It will then convert and saved to the destination folder.
For single PVR file conversions, the output files are saved in the BullConverter AppData
folder and then loaded to Notepad.
14
10. CSV Export Utility The CSV files converted by BullConverter do not include date and site ID columns because the file name itself shows that information. Also, each file contains only a single day data. However, for database or spreadsheet applications, it is often more convenient to have dates and site IDs in separate columns, and many days of data is stored in a single file. This utility (shown in Figure 8) converts Bull-CSV formatted files to export CSV files that include date and site ID columns per vehicle record. It also computes and produces ESAL column based on user selected parameters. The csv column definitions of export files are summarized in Table 6. It is important to recognize that the exported files have “exp” prefix in their filename and are no longer compatible with BullReporter. Export file conversion period can be selected for a whole year or only a specified month by selecting one of the options in the Time group (See Figure 8). Also the output file can be produced as a single large file or as multiple daily files. For example, a single date conversion on 1/20/2013 at site 039 will produce a file with the following name. exp-20130120.039.csv If an export file was created for one month from 1/1/2013 to 1/31/2013 for the same site in a single large file, an export file with the following name will be produced. exp-20130120-20130120.039.csv
These export files are all created in the folder defined in the textbox under “Export-CSV Root Folder.” It creates site folder names under the Export-CSV Root Folder before saving export files. Therefore, “Export-CSV Root Folder” should be the root folder for all export site folders, just like the CSV Root Folder. It should be noted that export function re-computes vehicle classification based on the Start tab selection, ESAL and overweight violations based on the parameters set in the Export Parameter Setup window. Pressing the “Set Parameters” button opens up the Export Parameter Setup window shown in Figure 9. For flexible pavements, the parameters needed are structural number (SN), which is a function of thickness and modulus of each layer and the drainage conditions of base and subbase, and terminal serviceability (Pt), which indicates the pavement conditions to be considered as failures. MnDOT 2007 Pavement Design Manual recommends SN = 5 and Pt=2.5. For rigid pavements, slab thickness in inches (D) must be supplied along with Pt. MnDOT’s recommends D =8 and Pt=2.5.
Overweight vehicles are checked based on the parameters supplied again by the Export Parameter Setup. After axle-group overweight checks, the warning number "43" is added if an overweight axle group is found. For GWV, the warning number "44" is added if GVW is over the limit. If the original data contains a “44” warning but it is no longer overweight GVW by the new parameter setup, the “44” warning from the
15
original error string is removed to conform to the new parameter setup. The same rule is applied to the overweight axle groups.
Figure 8: Export csv file conversion tab.
Figure 9: Parameter setup for export tab
16
11. Export File: CSV Column Format
Table 6: Export CSV Column Format (Not compatible with BullReporter)
Column
Number
Column name Description
1 Date Date in “MM/dd/yyyy” format.
2 Site Three digit site ID
3 Veh# Vehicle number
4 Lane# Lane number of this vehicle passed through
5 Time Time in hh:mm:ss where hh is military hour, mm is
minute, and ss is second.
6 AxleC Number of axles on this vehicle
7 Speed Speed of this vehicle in mph
8-18 Axle Spacing (AS):
AS1,…,AS11
Axle spacing in feet. It contains a total of 11 fields
separated by comma.
19-30 Axle weights(AW):
AW1,…,AW12
Axle weights in Kips. It contains a total of 12 fields
separated by comma.
31 GVW Gross Vehicle Weight (GVW) in Kips. It is simply a
summation of all axle weights.
32 Class Vehicle class determined by the classification scheme
specified in Column 35.
33 Err# It is a collection of numeric codes that represent error
and warnings (See Tables 4 and 5 for detail).
Multiple error/warning numbers are separated by
colon “:”. For example, “2:12” means detection of
two errors, 2 and 12.
34 100th Sec Number of 100th seconds of the time in Column 2.
35 TypFile A “.tye” or “.tym” file that was used for
determination of vehicle classification in Column 30.
36 ImageF List of image file names of this vehicle
37 License# License plate reading of this vehicle
38 Temp Temperature in Fahrenheit. It should be a pavement
temperature.
39 calFac Calibration (multiplication) factor applied to the raw
axle weight values. It is normally an adjustment
factor computed based on pavement temperature but
other adjustment factors may be combined.
40 ESAL ESAL computed based on user selection of flexible
or rigid pavement and the corresponding parameters
