Post on 02-Jun-2018
transcript
8/11/2019 Pipe-It Manaul 20130806
1/264
PIPE-ITMANUAL
Copyright 2013 Petrostreamz AS.
All rights reserved.
2013-08-06
Questions may be addressed to:
support@petrostreamz.com
Website:
http://petrostreamz.com
mailto:support@petrostreamz.commailto:support@petrostreamz.comhttp://petrostreamz.com/http://petrostreamz.com/http://petrostreamz.com/mailto:support@petrostreamz.com8/11/2019 Pipe-It Manaul 20130806
2/264
8/11/2019 Pipe-It Manaul 20130806
3/264
Table of Contents
Pipe-It Advantages ..................................................................6
Getting Started .........................................................................8
Register for an account and a license ............................................................ 8Download Pipe-It Software ......................................................................... 10Pipe-It Installation ..................................................................................... 11Installed Components ................................................................................. 15Pipe-It Preferences..................................................................................... 15
How to use Pipe-It: A small working example ..................... 20
Pipe-It Example: Launch a GAWK program .................................................. 20
Help and Documentation ...................................................... 31
Tooltip ...................................................................................................... 31Pipe-It Help System ................................................................................... 31Support by email ........................................................................................ 32
The main software components ........................................... 33
Introduction............................................................................................... 33Terminology .............................................................................................. 35
Pipe-It Graphical Elements ................................................... 36
Resources .................................................................................................. 36Program (Script) ........................................................................................ 39Scripter Dialog Box ..................................................................................... 40Example .................................................................................................... 40Summary of Scripter Dialog Options ............................................................ 47
Pipe-It Projects & Sub-Projects ............................................ 51
Projects & Composites ................................................................................ 52Composites ................................................................................................ 53Export of Composite ................................................................................... 54Importing .................................................................................................. 55Sockets ..................................................................................................... 56Socket Types ............................................................................................. 57
Runner ...................................................................................................... 59
8/11/2019 Pipe-It Manaul 20130806
4/264
Using Projects by multiple users .................................................................. 59Example Pipe-It Projects ............................................................................. 60Designing Pipe-It projects ........................................................................... 62
Pipe-It Graphical User Interface .......................................... 65
Tool Bar .................................................................................................... 66
Multiple Views ....................................................................... 71
Operating Systems ..................................................................................... 72Interactions with Canvas ............................................................................ 73Save As Dialog ........................................................................................... 90Displaying the model status and results ....................................................... 91Exporting and Importing data ..................................................................... 92
Optimizer ............................................................................... 95
Introduction and example ........................................................................... 95Optimization .............................................................................................. 99The Solvers ............................................................................................... 99Solver plugin API ..................................................................................... 100Example .................................................................................................. 101Reference ................................................................................................ 102Solver Parameters .................................................................................... 114
Optimization Variables .............................................................................. 117
Random Sampler Solver ........................................................................... 118Case Matrix Solver ................................................................................... 118Nelder-Mead Simplex-Reflection Solver ...................................................... 118IPOPT Solver ........................................................................................... 119NOMAD ................................................................................................... 121The DotNumericsPlugin solvers ................................................................. 121
Linkz .................................................................................... 122
Introduction............................................................................................. 122Linkz Window .......................................................................................... 122
Creating a new link .................................................................................. 123Using Delimiters and Regular Expression ................................................... 126The Optimizer Connection ........................................................................ 128More than a single value .......................................................................... 128MapLinkz ................................................................................................. 130Links in Annotations ................................................................................. 131
Streamz ............................................................................... 132
Introduction............................................................................................. 132
8/11/2019 Pipe-It Manaul 20130806
5/264
Streamz Language Commands .................................................................. 134Streamz GUI (Streamz Assistants) ............................................................. 136Streamz Copier ........................................................................................ 145Streamz Tabulator.................................................................................... 146Streamz Separator ................................................................................... 149
Streamz Generic ...................................................................................... 150
Streamz Documentation ........................................................................... 150
Strexzel ............................................................................... 151
Strexzel Macro Commands ........................................................................ 151Comments ............................................................................................... 152
Accumulate.............................................................................................. 152Add Note ................................................................................................. 153Calculate ................................................................................................. 154
Change Stream Units................................................................................ 157
Edit Component ....................................................................................... 158Edit Variable ............................................................................................ 159Expose Components ................................................................................. 161Insert Component .................................................................................... 162Insert Streams ......................................................................................... 164Insert Variable ......................................................................................... 165Load Characterization ............................................................................... 168Move Component ..................................................................................... 170Move Streams .......................................................................................... 171
Move Variable .......................................................................................... 173Name Characterization ............................................................................. 174
Remove Columns ..................................................................................... 175Remove Component ................................................................................. 176Remove Streams ...................................................................................... 177Remove Variable ...................................................................................... 178Save Characterization ............................................................................... 179Set Constant ............................................................................................ 180Set Formula ............................................................................................. 181Set Precision ............................................................................................ 185
Seed ....................................................................................................... 187
Strexzel GUI ............................................................................................ 187Strexzel interaction with Pipe-It. ............................................................... 193Console mode / Command line arguments ................................................. 193
Convertors ........................................................................... 195
Ecl2Str .................................................................................................... 195Pre Ecl..................................................................................................... 197Sen2Str ................................................................................................... 202
Script Assistant ................................................................... 206
8/11/2019 Pipe-It Manaul 20130806
6/264
Overview ................................................................................................. 206Using Scripts............................................................................................ 206Creating Scripts ....................................................................................... 207Modifying Scripts...................................................................................... 209Workflow ................................................................................................. 210
Detailed Description ................................................................................. 211
Plotz..................................................................................... 219
Plotz GUI............................................................................................... 223Help ....................................................................................................... 224Graphical Interface and Functionality ........................................................ 228
Appendix ............................................................................. 232
Streamz Tabulation Reference .................................................................. 234
Terminology ............................................................................................ 237Advanced Linkz trick................................................................................. 242Interaction with 3rd party software ........................................................... 243Getting Started with Streamz (Application) ................................................ 245Plots on Canvas in Pipe-It projects and using "R" language ........................ 259Pipe-It File types ...................................................................................... 262Copyright Notice ...................................................................................... 263Disclaimer ............................................................................................... 263
8/11/2019 Pipe-It Manaul 20130806
7/264
Pipe-It Manual Page 6
Pipe-It AdvantagesPetrostreamzs Pipe-It is a new methodology for asset optimizationand exploitation where petroleum streams are optimized over the
whole value chain of, and within each, petroleum sector for:
maximum product value
better decisions for investments and technical solutions
cost reductions
Petroleum resources originate as static bodies of oil and gas inunderground reservoirs. As these resources are exploited, they flow aspetroleum streams into a production system. Streams are mixed andprocessed, splitting into new streams. Mixing and splitting of streams
occurs throughout a labyrinth of wells, surface equipment, pipelines,storage facilities, transportation with new mixing and products beinggenerated at the final refinery destination. With Pipe-It users canmanage their petroleum streams and optimize the production ofhydrocarbons.
Pipe-It has the ability to keep a detailed, quantitative upstream-to-downstream accounting of the individual components making uppetroleum resources. This bridges the work of reservoir engineers, whotalks in terms of volumetric rates, to the processing engineers, who
talk in terms of component molar rates, and ultimately to managerswho talk in terms of currency, revenue, net present value and profit.
With Pipe-It software, a petroleum resource stream can be describedsimply as crude oil and gas, or it can be quantified with the hundreds ofmolecular constituents (methane, propane, , benzene, carbon dioxide,). It also allows for every conceivable description between simple andcomplex LPG,NGL,GTL products, crude oil refinery distillation cuts,wax, asphaltenes, and mercaptons. Even monetary value in any
currency, instead of barrels.
Pipe-It allows a project to be visualized with an intuitive graphicallayout design. One visualization can be created to give a clear vision ofthe project from a top-level management point-of-view, showing howfields are connected and interact at major process facilities and exportterminals. Other visualizations detail sub-projects like a subsea welltemplate or a satellite process facility, always maintaining the actualconnectivity to other elements in the project. Custom visualizations ofthe same Project can be made by each individual or team working on a
Project.
http://en.wikipedia.org/wiki/Liquefied_petroleum_gashttp://en.wikipedia.org/wiki/Natural_gas_processinghttp://en.wikipedia.org/wiki/Gas_to_liquidshttp://en.wikipedia.org/wiki/Gas_to_liquidshttp://en.wikipedia.org/wiki/Natural_gas_processinghttp://en.wikipedia.org/wiki/Liquefied_petroleum_gas8/11/2019 Pipe-It Manaul 20130806
8/264
Pipe-It Manual Page 7
Pipe-It allows an entire project or elements of a project to be executed(run). This might involve executing external computer programsdescribing performance of reservoirs, wells, pipelines, process facilities,storage and transportation vessels, and final-destination refining. Pipe-It allows these 3rd-party programs to communicate using Pipe-Its
engine.
Not only does Pipe-It rely on computational programs for resourcestreams, it allows access to measured data provided by online meteringand spot testing stored in databases. Sophisticated data managementfeatures of the Streamz engine allows measured and calculated data tobe compared on a consistent basis in terms of quantity (mass, volume,etc.) and time period (daily, monthly, annual). Once on a commonbasis comparative stream data can be used as the basis foroptimization.
A unique technology called Linkz allows Pipe-It to "connect" to tokens(numbers and strings) in text files irrespective of formats. These tokensare then available to Pipe-Its execution and optimization engines andbecome part of the overall optimization. This ability together with thepossibility of executing any program that can be launched from the OSgives Pipe-It an unprecedented capability and flexibility making it anideal framework for integrated computational integration. No programsare imposed on the end user (they choose which programs are bestsuited for their area of expertise), but rather Pipe-It embraces all and
facilitates their integration (connected execution) and data-exchange.
Pipe-Its Optimization Interface allows (through Linkz) any number inany file to be variables, constraints and/or objectives of anoptimization. Multiple levels of optimization can be defined and beactive in a single top level global optimization. Different solvers anddirections of optimization (MAX, MIN) can be specified for each instanceof the global or local optimization. A set of robust solvers are provided,
which can be extended by means of documented API.
Pipe-It includes a command-line version that can be called by not onlyother programs (even batch files) but also by itself! This allows Pipe-Itto be embedded within any other software that can launch an OScommand-line. This also means Pipe-It can recursively call manyinstances of itself from within a top-level graphical instance. This is thebasis of multi-level local and global optimization made possible by Pipe-
It.
8/11/2019 Pipe-It Manaul 20130806
9/264
Pipe-It Manual Page 8
Getting StartedPetrostreamz Pipe-It(referred to as simply Pipe-Ithenceforth) is asoftware solution for scalable integration and optimization of petroleum
assets. With a unique graphical platform, Pipe-It intuitively integratescomplex data and model configurations typically found in our industry.Once data and models are integrated, Pipe-It provides sensitivityassessment and optimization capability to define business opportunitiesdependent on complex cause-and-affect dependencies of revenueopportunities and cost management. Our fundamental philosophy isthat current work flows should not be disturbed, with each applicationcustodian continuing to use the same software, including all utilities for
pre- and post-processing.
This document provides an introduction to the use of Pipe-It and ismeant to be a guide for understanding and using a project built byPetrostreamz AS for clients. Other sections in this document contain
example to give the users all necessary details.
Please refer to Pipe-ItExample Projectsfor some examples.
Petrostreamz Support department can also arrange necessary trainingcourses, short video tutorials or provide on-site support / consultancy
services. Please contactsupport@petrostreamz.com.
Register for an account and a license
To start using Pipe-It, you will need to install the Pipe-It software inyour machine. The software and a trial Pipe-It license is available upon
request.
The first step is to register to create an account. This registration isfree and only requires you to provide some information to be
considered for a three-month trial Pipe-It license. Once you get youraccount, you can sign in to download the Pipe-It software.
The registration procedure:
Go to the websitehttp://petrostreamz.com
Choose "Download" tab on the top menu.
Enter your e-mail address, for first time registration you will need to
choose "No, I'd like to register".
mailto:support@petrostreamz.commailto:support@petrostreamz.commailto:support@petrostreamz.comhttp://petrostreamz.com/http://petrostreamz.com/http://petrostreamz.com/http://petrostreamz.com/mailto:support@petrostreamz.com8/11/2019 Pipe-It Manaul 20130806
10/264
Pipe-It Manual Page 9
Click "Go to registration".
Fill-in all required information.
If you are registering to get a trial license, choose "Accept Trial License
terms".
If you or your company has already bought a commercial license,choose "I am a Pipe-It customer and/or my company is a Pipe-It
customer already".
Click "Create new account".
Petrostreamz will send an e-mail confirming that your registration is
successful.
The approval process could take a few days. Once your account hasbeen approved, you will get an e-mail containing information abouthow to log-in, set your password, download Pipe-It and other details.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Register/registration.png8/11/2019 Pipe-It Manaul 20130806
11/264
Pipe-It Manual Page 10
Download Pipe-It Software
Once your account at Petrostreamz has been activated, you will receivean email containing all the information you need to log in, set your
password, and other details.
After you log in and set your password by following the procedure
mentioned in the email, you can start to download the Pipe-It software.
How to download Pipe-It:
Log in at http://petrostreamz.com/user using your e-mail and
password.
Navigate to the download page by clicking Download at the top of thepage.
Click on the link provided to access the download area.
http://petrostreamz.com/userhttp://petrostreamz.com/userhttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/DownloadSoftware/download-link.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/DownloadSoftware/download-page.pnghttp://petrostreamz.com/user8/11/2019 Pipe-It Manaul 20130806
12/264
Pipe-It Manual Page 11
Once you have navigated to the download area, you can download thePipe-It installation file by choosing the newest version of Pipe-It Setup(.exe). The installation packages for Mac OS X systems and Linuxsystems are available on request.
Note:On the same page you can also download Pipe-It examples andbrochures, these could help you to get familiar with Pipe-It.
Enjoy!
Pipe-It Installation
Windows Operating System
Pipe-It is distributed as a packed installation executable that is anapplication installation and configuration service. This binary is anengine for the installation, maintenance, and removal of Petrostreamzsoftware on modern Microsoft Windows systems.
On other systems the installation procedures might be different.Installation packages are available for Mac OS X systems and Linux
systems (on request).
Quick Installation Guide
Here is an explanation of the installation procedure that allows you to
quickly install Pipe-It on your desktop system.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/DownloadSoftware/download-file.png8/11/2019 Pipe-It Manaul 20130806
13/264
Pipe-It Manual Page 12
File Naming and Versioning
Petrostreamz AS provides installation binaries with the following
naming convention, for example:
Pi peI t Set up- 1. 3. 0. exe
This allows you to keep track of build number and have up-to-dateversions. You can easily compare this build ID with that of the versioncurrently installed, which is shown in the About dialog (About Us on the
Help menu):
Executing Pipe-It Installer
Welcome Message
The first screen that you see indicates that Pipe-It installation issuccessfully started and that it will guide you through all installationsteps and choices. It also shows the build number to keep you aware of
which version you are installing:
8/11/2019 Pipe-It Manaul 20130806
14/264
Pipe-It Manual Page 13
Choose Install Location
This page allows you to check the installation path of Pipe-It and if
required change it to another one that is more convenient for yourinstallation. Note that this installation path is stored in the registry, sowhen you run a Pipe-It installer with a new version to update Pipe-It, itwill load the previous installation path automatically. Then you only
need to check that it is correct and press the "Install" button:
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/PipeItInstallation/install-wizard.png8/11/2019 Pipe-It Manaul 20130806
15/264
Pipe-It Manual Page 14
Installation Progress
The page indicates installation progress, and shows you the files beingcopied to your hard drive if "Show details" is clicked. Just wait fewseconds to get the software installed:
Installation Completed
Finally you get a screen that indicates that the installation iscompleted. Also by default it allows you to run Pipe-It directly when the
installation is closed:
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/PipeItInstallation/install-completed.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/PipeItInstallation/install-progress.png8/11/2019 Pipe-It Manaul 20130806
16/264
Pipe-It Manual Page 15
Installed Components
The Pipe-It software product includes the following components
distributed within the installer binary:
Petrostreamz components
Pipe-It.exe- Main GUI Application.
Pipe-Itc.exe - Non-GUI version of Pipe-It that can be invoked from
other programs or by Pipe-It itself!
Streamz.exe - The "Engine" of Pipe-It. Sophisticated streammanagement program with track record of industry-strength projects
under its belt. See Streamz documentation for more info.
ProjectBuilder.exe - Wizard-based builder of Pipe-It project. Uses atemplate (see below) designed for specific application (e.g. EclBOz) andcustomizes it based on user supplied data. The Wizard displays pagesfor user interaction and replaces/duplicates elements in the template to
create a ready to run Pipe-It project.
Growing set of standard conversion utilities to facilitate integration of
popular engineering software (e.g. Eclipse, Sensor)
A set of Solver for use by the Optimizer
A set of Project Builder Templates
A set of Scripter Assistant Program short-cuts.
3rd party components
Qt Libraries- DLLs to run GUI software.
UnxUtils - Set of Unix utilities built for Win32 to use in projects forcross-platform issues. See also http://unxutils.sourceforge.net/ Theinstaller program appends the path in each launched instance of Pipe-itso all these commands are available in command line.
Pipe-It Preferences
Petrostreamz Pipe-It, like most of the applications on your computer, hasa set of preferences you can customize. You use preferences to specify
http://unxutils.sourceforge.net/http://unxutils.sourceforge.net/http://unxutils.sourceforge.net/8/11/2019 Pipe-It Manaul 20130806
17/264
Pipe-It Manual Page 16
things such as auto-save settings and a default folder for storing projects.At this point it is recommended to change at least the default editor fromNotepad (on Windows) to the one you are used to. You can worry aboutthe others as you get more experience with using Pipe-It.
To view or change preferences:
Win: Choose Edit > Preferences
Mac: Choose Pipe-It > Preferences
Preferences are tabbed into 3 pages:
General
Text Editor
Canvas
General Preferences
The page of preferences defines settings:
Beep on project run finished- the checkbox makes Pipe-It beeping whenexecution of your project is finished. By default the setting is enabled, butyou can disable it when running Optimizations and it produces severalbeeps too often. Anyway this is good indication for long term executions
to attract attention.
8/11/2019 Pipe-It Manaul 20130806
18/264
Pipe-It Manual Page 17
Auto-save opened projects - the option with defined interval enablestimer to save your project after defined time from last save operation. Itsaves project into model+view files, ppm and ppv with same file name asopened project but prepended with "~". In case of power failure you havepossibility to extract lost changes to project by getting information from
auto-save files.
Text Editor Preferences
Text Editor page allows you to define default text editor for Edit Itemaction for Resources. Note that it is different than double-click onResource which invokes Application from default system association withthis file type. Instead Right click on Resource and choosing Edit Item, or
just pressing F4invokes association defined by this preferences page.
It can be very useful if you would like to have fast way of starting Fileedition process but without changing the default association in OS to starteditor with double-click. Using F4 invokes Editor specified in Pipe-It andallows to keep double-clicking for processing the file by some utility.
You can see that the page allows you to extend associations for severalfile types. By default it provides additional association for image files to
show you example of defining new bindings to your editing applications.
Canvas Preferences
You can customize next canvas settings:
8/11/2019 Pipe-It Manaul 20130806
19/264
Pipe-It Manual Page 18
Use mouse wheel for zooming By default the mouse while (or thetrackpad found on current portable computers) is used to scroll up anddown in the canvas. This setting will allow the user to use it for zoomingin and out instead.
Reverse mouse wheel direction when use for zoom Zoom-in and Zoom-
out are swapped from the default action.
Animate connectors on Project Run - when you run your project, thenactive connections (which are connected to elements that are beingprocessed this time) are animated by moving dots to show you which
Resources are currently used or produced by running Scripters.
Make new item selected when adding to canvas Fairly obvious.
Make green checkmarks disappear on first mouse move- When executionof project is completed then all processed elements got checkmarks (inright top corner) to indicate completeness of executed elements.
Checkmarks that indicates problems attracts attention to resolve situationand start execution again. But in case of successful execution all elements
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/ApplicationPreferences/AppHelpPreferencesDialog-AnimateConnectors-1.png8/11/2019 Pipe-It Manaul 20130806
20/264
Pipe-It Manual Page 19
are marked with green checkmarks and not required to be kept on canvasto continue your operations. For such case you can use the option to keepgreen checkmarks until first mouse move - you definitely notice theproject state and can continue work.
It is convenient if project run time is huge and you need indication whenyou return to computer to look at completeness of the execution.
Animate slow shading during Time used to fade the checkmarks.
8/11/2019 Pipe-It Manaul 20130806
21/264
Pipe-It Manual Page 20
How to use Pipe-It: A small workingexampleYou can use Pipe-It to "pipe" together your project from bottom-up ortop-down. Each software application being launched is called a Process.Any file connected with a Process is called a Resource. You can chaintogether as many Processes as needed using Connectors, in any order(parallel and/or series). A collection of Processes can be grouped into aComposite. The total collection of all Processes, Resources, andComposites is a Pipe-It Project. An entire Pipe-It Project can be runtime-and-again with changes to any input data using the Optimizer.The Runner orchestrates, without user intervention, intelligent andefficient launching of all Processes in a Pipe-It Project. And by usingLinkz,any data in Resource files can be accessed and used for further
operation.
In this page, you will find a step-by-step procedure to build a verysimple Pipe-It project. This example will help you to start theinteraction with Pipe-It. A detailed description of a Pipe-It element can
be examined by clicking the referenced item (a hyperlink).
Pipe-It Example: Launch a GAWK program
In this example, we will utilize Pipe-It to launch a gawk program (http://www.gnu.org/software/gawk/manual/gawk.html) that willextract data from a text file. Windows version of GAWK is installed withPipe-It as part of UnixUtils, to facilitate execution of Pipe-It projects
across operating systems.
Open the Pipe-It software. Go to the Start menu and choose the Pipe-It
program.
Save the Pipe-It project in your preferred directory.Go to the File menu and choose "Save" or "Save As". You could also
use the shortcut provided (Ctrl+S or Ctrl+Shift+S) to do this operation.
Note:A shortcut for an operation (if any) can be seen on the menu.
http://.http/www.gnu.org/software/gawk/manual/gawk.htmlhttp://.http/www.gnu.org/software/gawk/manual/gawk.htmlhttp://.http/www.gnu.org/software/gawk/manual/gawk.htmlhttp://.http/www.gnu.org/software/gawk/manual/gawk.htmlhttp://.http/www.gnu.org/software/gawk/manual/gawk.html8/11/2019 Pipe-It Manaul 20130806
22/264
Pipe-It Manual Page 21
You can start building the project by adding a title or a note in theCanvas to identify and/or give a general description of the Pipe-It
project.
Choose the text-box (Annotation) symbol in the Toolbar and place it at
the preferred location on the Canvas. Double-click to edit theannotation text and press "OK" when finished.You could also edit the display (Color/Line/Font etc.) by placing thecursor on the text-box, right-click, and choose "Open Item
Properties...".
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/PipeItExample1/example-textbox.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/PipeItExample1/example-save.png8/11/2019 Pipe-It Manaul 20130806
23/264
Pipe-It Manual Page 22
Adding a Resource to the Pipe-It project. Choose the Resource symbolon the Toolbar, or just simply press "R", and place a Resource element
on the Canvas.
Associate a data-file (employee.txt) with the Resource.Right click on the Resource box and choose "Set Resource File...". Findthe data-file and click "Save". Un this case you will create this file, so
just give it the suggested, or another, name.
Right-click this resource and choose to Open in Editor (or just press
F4 after selecting the resource:
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/PipeItExample1/example-resource-file.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/PipeItExample1/example-resource-toolbar.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/PipeItExample1/example-textbox-itemprop.png8/11/2019 Pipe-It Manaul 20130806
24/264
Pipe-It Manual Page 23
Depending on your Pipe-It Preferences the empty file will open inNotepad or your preferred text Editor. Populate it with the following
data and Save it.
00 Thomas Manager Sal es $5, 000200 J ason Devel oper Technol ogy $5, 500300 Sanj ay Sysadmi n Technol ogy $7, 000400 Ni sha Manager Market i ng $9, 500500 Randy DBA Technol ogy $6, 000
This data can be copied from the website which is the source of thisexample:
(http://www.thegeekstuff.com/2010/01/awk-introduction-tutorial-7-
awk-print-examples/)
Add one more Resources and associate it with the (actions.awk) using
the same procedure as above and populate it with the following:
{pri nt ; }
Adding a Process to the Pipe-It project. Choose Process symbol on theToolbar and choose "Program(Script)" or just simply press "P". Placethe Process element on the Canvas.
http://www.thegeekstuff.com/2010/01/awk-introduction-tutorial-7-awk-print-examples/http://www.thegeekstuff.com/2010/01/awk-introduction-tutorial-7-awk-print-examples/http://www.thegeekstuff.com/2010/01/awk-introduction-tutorial-7-awk-print-examples/http://www.thegeekstuff.com/2010/01/awk-introduction-tutorial-7-awk-print-examples/http://www.thegeekstuff.com/2010/01/awk-introduction-tutorial-7-awk-print-examples/8/11/2019 Pipe-It Manaul 20130806
25/264
Pipe-It Manual Page 24
You can rename the Process element by right-clicking on it andselecting "Rename Element...", or just simply press F2 with the Processelement on the Canvas selected.
Connect all input resources to the process by using Connectors.Click the Connectors symbol on the Toolbar to activate it and thenconnect the preferred resource with the process element. First movethe mouse to the (actions.awk) Resource, connection points (smallwhite marks) should show up on the edge of the element. Left-clickonce at one of the points, then move to the Process element and left-
click again at one of the connection points on that element (you canalso just click and hold the left mouse button at the connection point onthe Resource and then go to a connection point on the Process andrelease the button). Now the Resource and the Process should beconnected by an arrow, going from the Resource to the Process. Do the
same for the (employee.txt) Resource.
The Pipe-It project should now look similar to this. A second annotation
(text box) has been added to provide additional information.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/PipeItExample1/example-process.png8/11/2019 Pipe-It Manaul 20130806
26/264
Pipe-It Manual Page 25
To be able to launch the Gawk program, we need to write a scriptinside the Process element based on the Pipe-It usage of the AWK
syntax.
Double-click the process element and in the "Script" box type thecommand line to invoke Gawk application.
gawk
The syntax of the awk command is actually:
gawk - f act i ons- f i l e dat a- f i l e [ > resul t - f i l e]
Thus, inside the "Script" box we need to use the syntax above and insert
actions-file (.awk) first, then the input file and the output file.
The command line inside the "Script" window should look like:
8/11/2019 Pipe-It Manaul 20130806
27/264
Pipe-It Manual Page 26
Run the Pipe-It project by clicking the "Run Project" symbol on the
Toolbar or by using the shortcut "Ctrl+R".
A green check-mark symbol will appear on the elements if the projectruns successfully. If an error occurs, you will see a red-flag symbol on thebroken element.
Since the optional output file was not specified in this example, gawksends the output of the program to the standard output. Pipe-Itprovides a console to capture this output from the program. To view
this console click on the View menu item and select Show Console:
A Console Window will open to show the result of the execution. In thisexample since the actions file contained only the action to print, each
line will be printed on the console:
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/PipeItExample1/example-run-project.png8/11/2019 Pipe-It Manaul 20130806
28/264
Pipe-It Manual Page 27
The next step in this mini-tutorial is to save the result instead to a file.Create another Resource on the canvas and connect it as an Output tothe Process (start the connector from the Process and drive it to the
new Resource). Associate this resource to a file named result.txt:
Canvas
Console
Output of gawk
8/11/2019 Pipe-It Manaul 20130806
29/264
Pipe-It Manual Page 28
Since the syntax of gawkrequires redirection operator (>) to passthe output to a file instead of to the console, we need to modify theScript in the Program. To enable the > to be interpreted correctly bythe operating system (Windows), the command line needs to beprocessed via the command shell. The final command needed looks like
this:
Run the project again. This time the Console only logs what was donebut the result is sent to the file. In the picture below the result.txt file
is opened in Notepad to show the content.
8/11/2019 Pipe-It Manaul 20130806
30/264
Pipe-It Manual Page 29
Pipe-It offers many features to make your projects flexible, robust andsustainable. One such feature is the use of Resource Referencesinstead of directly coded file names (Covered in detail later in thismanual). You might want to come back to this project when you havestudied it. The screenshot below shows the content of the Process. Note
that the Script uses Resource References syntax while the parsedcommand line is identical to what we saw earlier.
Before we close this project, let us try to alter the action, which is veryboring right now. Change the content of the action.awk file to (fourthcol umn cont ai ns Technol ogy) :
$4 ~/Technology/ {print;}
Resource reference syntax
Corresponding
Command line
8/11/2019 Pipe-It Manaul 20130806
31/264
Pipe-It Manual Page 30
Pipe-It uses gawk to select only lines fulfilling certain criteria and write
the results to a file. Save and close the Pipe-It project.
Congratulations! You have just created your first Pipe-It project and
run it.
This Pipe-It project example is very simple and far from what Pipe-Itcan do in real application. This exercise comes with the intention to
help new users to start the basic interaction with Pipe-It.
For some real Pipe-It project examples, please refer to Pipe-It Example
Projectsand on the website.
8/11/2019 Pipe-It Manaul 20130806
32/264
Pipe-It Manual Page 31
Help and DocumentationThere are several ways to get help in Pipe-It, depending on the level of
help required.
Tooltip
Hypertext - short description of object in the user interface. Thedescription usually appears within a few seconds when the cursor is
placed over an object.
Pipe-It Help System
Pipe-It comes with a built-in help system where you can find Pipe-Itand Streamz documentation (in English). You can invoke this systemby pressing F1 or going to the Help menu in the main Pipe-It window.
Currently this PDF Pipe-It Manual is linked to the help system.
Future versions will be provided in a format similar to a web browserthat shows hypertext content with links and images. By using thissystem, you can access the Pipe-It and Streamz documentation, apanel with a list of keywords, a bookmarks panel and a search system
that allows you to easily find the content that you are interested in.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/HelpAndDocumentation/help-tab.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/HelpAndDocumentation/tooltips.png8/11/2019 Pipe-It Manaul 20130806
33/264
Pipe-It Manual Page 32
Support by email
Sending an e-mail - to the Support Department at
support@petrostreamz.comis the highest level of help and can be usedto resolve intricate problems when you are otherwise stuck.
mailto:support@petrostreamz.commailto:support@petrostreamz.comhttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/HelpAndDocumentation/help-system.pngmailto:support@petrostreamz.commailto:support@petrostreamz.commailto:support@petrostreamz.com8/11/2019 Pipe-It Manaul 20130806
34/264
Pipe-It Manual Page 33
The main software components
Introduction
This section will give a brief introduction to the software components in
the Pipe-It package.
The figure shows the main software components of the Pipe-It package.In the figure, solid lines indicate that the program has a graphical userinterface (GUI). Dashed line indicates that the program is started fromthe command line, without GUI. File type extensions of the input filesare also shown. Most of the programs write out to the same file or
same type of file.
Pipe-It
The main program with GUI.
Pipe-Itc
The command line version of Pipe-It.
Linkz
It is used to define links to individual data elements or matrices insidefiles. Linkz can work with any text file and with MS Excel (binary orxml) files. Through a GUI the user can "teach" Linkz how to find the
desired data elements. The link definitions are saved in a .ppl file.
Optimizer
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/MainSoftwareComponents/MainSoftwareComponents.png8/11/2019 Pipe-It Manaul 20130806
35/264
Pipe-It Manual Page 34
A control panel to set up and start automatic iterations of executions,using a solver. The optimizer can use links into data files defined with
Linkz.
(Solvers)
A solver contains the optimization algorithm. It is started by theOptimizer and takes over control during optimization. A collection ofsolvers is delivered with Pipe-It, most of them are in dynamic libraries
(.dll). Choice of solver is available from a menu in the Optimizer.
Runner
A part of Pipe-It that takes care of execution of the project. It is notvisible to the user. The Runner runs only the parts of a large project
that need updating according to changed data and dependencies.
Streamz
Uses .str (Stream files). It calculates combination and splitting ofcollections of streams. It is both a command line program and tightlyintegrated within Pipe-It.
Strexzel
A spreadsheet-like program that is specially made to do operations on.str files, which are beyond the scope of Streamz.
Plotz
Makes graphs / plots.
MapLinkz
Copies data from one file to another by using links defined with Linkz.It is integrated with Pipe-It and easy to use in projects in the Pipe-It
GUI.
(Converters)
A collection of utilities that can read various other file formats,including binary files, and produce .str files.
ProjectBuilder
A utility program that can read definitions from "templates" (made by
expert users), and based on these it shows Wizard-like question pagesto the user, collects data about the new project from the user's
8/11/2019 Pipe-It Manaul 20130806
36/264
Pipe-It Manual Page 35
answers, and generates (possibly very large) Pipe-It project files as a
result.
ScriptAssistant
A utility that helps the user to set up how to use (a new) externalprogram (what input it needs, what output it generates etc.) fromwithin Pipe-It. The result is that the external program becomes
apparently integrated in Pipe-It and easier to use in a Pipe-It project.
Terminology
Unsure about what we mean? Need a one line definition of a term used onthe website? What is a Resource? A Process?
Refer to the Appendix for a collection of all technical terms in the contextof Pipe-It. Even Pipe-It is defined here! Use the search function to search
for the term.
8/11/2019 Pipe-It Manaul 20130806
37/264
Pipe-It Manual Page 36
Pipe-It Graphical ElementsPipe-It is a unique software application that allows you to graphicallyand computationally integrate models and optimize petroleum assets.
Pipe-It can launch any software application you are currently running,on any operating system. Pipe-It chains together your applications, inseries and parallel. Pipe-It's versatile Streamz technology manages thecomplex oil and gas streams processed by models making up anintegrated asset. Our fundamental philosophy is that current work flowsshould not be disturbed, with each application custodian continuing touse the same software, including all utilities for pre- and post-processing. The main function of Pipe-It is to provide added value
through
modular & scalable integration solutions
local & global optimization capability
visual organizational functionality
Resources
Resources are representations of files on disk. The default shape is arounded rectangle with a pale blue color. They should usually berenamed to something appropriate in the context of the project. Theyneed to be linked to the appropriate file on disk via "Set Resource file"context-menu option. An easy way to create a new (or multiple)resource is to drag the file from file explorer on to canvas. This createsthe resource, gives it a default name and also set the file linking in on
quick step.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Resource/resource.jpg8/11/2019 Pipe-It Manaul 20130806
38/264
8/11/2019 Pipe-It Manaul 20130806
39/264
Pipe-It Manual Page 38
The element can be renamed to a more convenient name by pressingF2 and typing in a new name. It will still be associated with theselected file.
If the resource points to a stream file,a characterization name can be
associated with the stream file by pressing F3 and typing in the name.
Pressing + will open the item properties dialog, where thevarious properties can be inspected and modified. If characterizationshave been added to the project via Streamz Library(), these will be available in the Characterizationdropdown. Stream file input and output options can also be defined in
this property page.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Resource/itemproperties1.jpghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Resource/setResourceChar.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Resource/resourcenamechanged.jpg8/11/2019 Pipe-It Manaul 20130806
40/264
Pipe-It Manual Page 39
Program (Script)
Programs are operations that modifies an upstream resource to createa new downstream resource . The default shape is an ellipse (oval) with
a pale green color. They should usually be renamed to somethingappropriate in the context of the project. They are usually connected onthe upstream and the downstream sides to resources. They can neverbe connected directly to another program. The most often usedprogram is a generic Scripter where the user can specify any commandline for execution. A program canvas element can be created byclicking on the "P" icon in the toolbar and dragging the ellipse on to the
canvas.
The application executable and its command line arguments isdescribed in the scripter properties dialog. Select the process and press
to open this dialog, or open it from the context menu.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Program/processcontextmenu.jpghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Program/process.jpg8/11/2019 Pipe-It Manaul 20130806
41/264
Pipe-It Manual Page 40
Scripter Dialog Box
Example
The scripter command line can be explicitly entered by the user. Forexample if the scripter launches the reservoir simulator SENSOR andfirst argument is the input data file, the commands written into the
scripter would be:
SENSOR. exe i nput . dat
Pipe-It allows a flexible syntax to make projects and programs portableby using reference to connected resources. The files actually connectedto those resource will then be used to make up the command line on-
the-fly at time of execution.
To illustrate this, let's set up a process for converting a stream file withthe Strexzel application, we will need an input stream file, a macro file.
When the program executes, it will produce an output file.
Let's select the program and open the scripter properties dialog. Youcan do this by selecting the program and by pressing F4, or by
selecting "Open Scripter" from the context program context menu.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Program/OpenScripter.jpghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Program/process_example.jpg8/11/2019 Pipe-It Manaul 20130806
42/264
Pipe-It Manual Page 41
You get a window divided into 3 parts. The first part with the labelScript is a space for you to write the command required to launch
your program.
In the example below, we have used the Strexzel utility. A normalcommand line invocation for Strexzel for opening a stream file, runninga macro file and output the result to a second stream file is:
St r exzel - m - i - o
First we need to type in the name of the application that we want Pipe-It to execute when the project runs. In this case, the application nameis "Strexzel". The simplest way to populate this script area is directlyfile names. However the user needs to know exactly the names of thefiles (including the path relative to the current folder). Suppose the filesare located in the current folder. The following picture shows what a
Script may look like.
8/11/2019 Pipe-It Manaul 20130806
43/264
Pipe-It Manual Page 42
In this case the Script and the Command Line are identical.
However Pipe-It is designed to provide the names of the files needed asarguments on the command line automatically, if a Resource (set to afile) is connected to the Process using a Connector. In fact that is thepurpose of the connectors to provide Processes with files (via
Resources).
To use this flexible, and more robust, way of populating the Script youneed to use the Insert New Argument button. This results in a box
with many choices. Many of the defaults can usually be used.
8/11/2019 Pipe-It Manaul 20130806
44/264
8/11/2019 Pipe-It Manaul 20130806
45/264
Pipe-It Manual Page 44
Where clicking the button of your choice will select it and take you back
to the original window.
If the program had been created using a utility, or a user has added
more sockets, a list of recognized sockets would have been displayed.
Since we are working with a simple example with two input files andone output files, we proceed with the default sockets and stick with I1
(First Input Socket).
We need to set the resource that will be used as the first argument. Weclick on the Choose button in front of First Resource to get the
following dialog:
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Program/resourceReferenceDialog.jpghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Program/socketReferenceDialogOpen.jpg8/11/2019 Pipe-It Manaul 20130806
46/264
Pipe-It Manual Page 45
This dialog lists the resources that are connected to the chosen Socket(I1 in our case) of the program. We can select the resource usingforward referencing or backward referencing indexes. In the forwardreferencing scheme, an integer index is assigned to each resource. Inthe backward referencing scheme an integer reference relative to the
number of resources is used. If we have three resources and want toreference the second one using the forward referencing scheme, we willuse the index "2". If we want to use backward referencing, we will use
the index "n-1".
Backward referencing is for geeks. Normal people will use forwardreferences. In this example we will use forward referencing. Since wealready have the resources (with files) connected to the Program, theyshow up under respective columns. We select the macro file by clickingon its index (1) in the forward referencing column. The dialog will close
automatically after we have selected a reference.
Since this argument also needs a switch (-m) we can specify it as part
of the Bounding Text entry as shown in the figure:
This ensures that the switch is written only if the argument is written tothe command line (useful for optional arguments). When you click
Insert you get a new entry in the script.
8/11/2019 Pipe-It Manaul 20130806
47/264
8/11/2019 Pipe-It Manaul 20130806
48/264
Pipe-It Manual Page 47
Summary of Scripter Dialog Options
Field Mandatory Description Actions / dialogs
Script yes Describes an applicationcommand line using
Pipe-It referencesyntax.
edit manually orcompose by using the
"Insert newreference" button.
Socket yes socket name to beinserted in reference.This can be an input oroutput socket
Edit manually or openthe "socket reference"dialog by pressing the"choose" button nextto the field. Thisopens the socketreference dialog(described below).
Selecting a value inthis dialog will
8/11/2019 Pipe-It Manaul 20130806
49/264
Pipe-It Manual Page 48
populate the field.
Firstresource
yes The first resource nameto be inserted in thereference
Edit manually or openthe "resourcereference" dialog by
pressing the "choose"button next to thefield. This opens theresource referencedialog (describedbelow). Selecting avalue in this dialogwill populate the field.
Lastresource
no This lets you select aresource as above, but
it is only used if youwant to specify a rangeof resources in thereference. The range isgiven by two indexesusing forward orbackward referencing.
same as above.
Minimum no The field "minimum"(matches required forinsertion) indicates the
minimum number ofconnected resourcesthat has to be active forthe program to execute.
Spin button forincreasing ordecreasing the value.
Relative /Fixed filepath
yes Defines the reference asa fixed path or arelative reference
Select using radiobuttons. Defaults torelative.
ExcludeFile
Extension
no If this is selected, thereference expansion will
exlude the file extension
Checkbox.
ParentDirectoryOnly
no If this is selected, thereference expansion willonly refer to the parentdirectory for theresource
Checkbox.
Insert yes Press this button toinsert a new referencein the script field. The
reference will becomposed using
Button.
8/11/2019 Pipe-It Manaul 20130806
50/264
Pipe-It Manual Page 49
selected values from thefields described above.
Error - Area to show whenScript has un references
resources/filesWorkingdir
no Defines the workingdirectory for theprogram.
Pressing the buttondisplays a field wherethe working directorycan be defined.
Utility no Expands a list withpreconfigured utilities.Selecting a utility fromthe list will preconfigurethe contents.
The button opens thescript translationdialog (describedbelow).
Notice that Pipe-It has preconfigured Program templates (utilities) for aset of bundled applications. By using these to define a program insteadof the generic Program icon, Pipe-It will be able to provide a full set of
possible command line parameters.
Each argument in the Script is abstracted as follows
The first character in the argument specifies if the resource is
referenced as a relative or a fixed path.
"R1@I1" signifies that the resource path is relative "F1@I1" signifies thatthe resource path is fixed
The second character refers to the resource index. This is used toidentify the correct resource in the case that multiple resources are
connected to a program via the same socket
"R1@I1" signifies that the resource path is relative and that we reference
the first resource
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Program/StrexzelUtility.jpg8/11/2019 Pipe-It Manaul 20130806
51/264
Pipe-It Manual Page 50
Notice that Pipe-It uses its own abstractions for representing commandline arguments. The command line switches and options are identical toa normal command line, but the argument file references are replacedwith an alternate syntax. This syntax makes it possible for Pipe-It tocreate valid program command lines by referring to its own resource
abstractions instead of referring to specific files directly. By creating aprogram representation with the Strexzel utility, Pipe-It creates thefollowing command line for us:
St r exzel { - m "R1@I 1" } { - i "R1@I 2" } { - c "R1@I 3" } { - o "R1@O1" } {- s "R1@O2" } { - l "R1@O3" }
This predefined command line contains all known command linearguments for the applications.
To illustrate this, let's set up a process for converting a stream file with
the Strexzel application, we will need an input stream file, a macro file.When the program executes, it will produce an output file.
Since Pipe-It is aware of the command line interface for Strexzel, The
command line
St r exzel { - m "R1@I 1" }{ - i "R1@I 2" }{ - c " R1@I 3" } { - o "R1@O1"}{- s "R1@O2" } { - l "R1@O3" }
is generated automatically.
Connectors pointing to the program are interpreted as input, andconnectors pointing away from the program are interpreted as output.Resources are referenced via input or output sockets. Sockets are
identified by type and number
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Pipe-It_GUI_Program/process_example.jpg8/11/2019 Pipe-It Manaul 20130806
52/264
Pipe-It Manual Page 51
Pipe-It Projects & Sub-ProjectsExcel files are called spreadsheets. Powerpoint files are calledpresentations. Similarly Pipe-It "files" are called projects. Pipe-It
projects consists of many component files. Not just single files likemost other applications. Pipe-It Project View (*.ppv) file store thevisual content of the project. Pipe-It Project Model (*.ppm) file is mostimportant and stores the logical model describing the all the elementsin the project their connections, their grouping into composites, thefiles associated with each resource, the command lines of each scripteretc. Pipe-It Project Linkz (*.ppl) file stores all the variables collectedfrom all text files that are part of the project, their current values andalgorithms to locate them in later runs. Pipe-It Project Streamz Library(*.pps) files store Streamz characterizations and conversions used in
the project.
The location of these 4 files is called the root folder of the project. Mostother files mentioned under can be located anywhere on the disk.However, to make the project portable it is recommended that all such
files are located at or under the root folder.
If the project contains optimization, one or a series of Pipe-It ProjectOptimization (*.ppo) files are also present. They can be located
anywhere.
Since Pipe-It is a multiple application integration framework, allowingthe launch of many internal and external applications, a large numberof application specific files become part of the project. Many of thesefiles may be represented as Resources in the Pipe-It project andbecome "known" to the project. Many other files just exist and arecreated, read and written by applications. All files connected toinitiating resources (those that are not downstream of any process) arealso mandatory for the proper execution of the project. These files aretypically located at the root or in folders under the root.
Designing a Pipe-It project requires some planning. Simple projectsmight contain only a handful of data files linked to resources and thesecan live in the project root folder. But real life projects containhundreds if not thousands of files. It makes sense to organize thesefiles in a sensible folder hierarchy. The project within Pipe-It shouldalso follow the folder structure using composites and sub-composites asfar as possible. This helps in cloning of composites in case a newinstance of the functionality encapsulated in the composite is to becreated. Pipe-It estimates the % of files that resides within a particularsub-folder of the project and offers to the user the possibility of
duplicating files. This is really a powerful option as all files, their
8/11/2019 Pipe-It Manaul 20130806
53/264
Pipe-It Manual Page 52
connections to resources, their Linkz variables, and all process
command lines are cloned.
Making order of the hundreds and thousand files is another reason for
collecting related files together.
Projects & Composites
Project is container for all elements of modeling processes. Of coursekeeping all the elements on same canvas makes the scheme verycomplex for understanding and tracking processe details. To help withthis some or all elements in a project can be organized in Sub-
Processes or Composites.
We recommend to use the term Projectfor meaning whole project. Alsoit means a set of 4 files that defines the project - logical model + visual
model + variable links + Streamz Library (ppm + ppv + ppl + pps).
The term Composite is used for defining some part of whole projectthat is organized separately from other elements. Actually compositesare small sub-projects that work in same way as other elements of
canvas.
Because Composite is actually sub-project that defines someoperations, it can also have nested composites of second level and so
on.
From this point of view, the project is tree of composites:
Or similar but top to bottom structure:
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Projects/TreeOfComposites-1.png8/11/2019 Pipe-It Manaul 20130806
54/264
Pipe-It Manual Page 53
Composites
As mentioned in the section on Projects, composites are a way to makea project easier to understand. They will compose parts of a projecttogether into one box for a less cluttered appearance. For example onemay make one composite for the reservoir part of the field, and one forthe processing part of the field. Each composite may also be run
individually. Project can have any amount of nested composites:
Connections from elements within a composite to elements outside (orinside other composites) are made viaSocketsthat look visually unique
for each different type.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Projects/TreeOfComposites-2.png8/11/2019 Pipe-It Manaul 20130806
55/264
Pipe-It Manual Page 54
Export of Composite
It is possible to export complete composites to standalone projects by
using:
Selecting the Composite and right-clicking give this dialog. There are
two possibilities:
Save Composite as a Pipe-It Project: The data files connected toresources are referenced at original locations. All nested elements are
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Composites/CompositesConnections-1.png8/11/2019 Pipe-It Manaul 20130806
56/264
Pipe-It Manual Page 55
carried over. But associated Linkz variables and Streamz Library are
not.
Save Composite as a Pipe-It Archive creates a complete standalonecopy of all datafiles, folders, files, Linkz variables, Streamz Library that
are specific to this composite, and nested composites. This is apreferred way to send a complete portion of a project to someone else.
Importing
Importing composites is done Selecting the File/Import menu optionafter which the typical file browser window allows you to find the ppv
file to import.
An alternate way is to drag the ppv file, like any other resource, on to
the canvas. In this case a dialog to choose insertion of resource orimport of composite is offered:
8/11/2019 Pipe-It Manaul 20130806
57/264
Pipe-It Manual Page 56
See Also
Sockets
Socket Types
Sockets
Except for connectors, every element has a number of sockets. Whenyou connect two elements, you actually connect an output socket fromone to an input socket on the other. Each socket has a differentpurpose. Think of connecting a stereo receiver to a set of speakers. You
can't just attach your wires at random. Each speaker will have exactlytwo terminals (sockets), but the receiver might have 20 or 30, all witha different purpose. You have to connect the receiver's "Right SpeakerPositive" terminal to the positive terminal on the right-hand speaker,
and so forth.
All currently defined resources, manifolds, and distributors have exactlytwo sockets -- one for input and one for output. By default, they'renamed Input and Output, respectively. You're free to rename them(except Resources), but you can't delete these sockets or add newones. Currently defined processes (i.e., Scripters), also start with twosockets, Input and Output, but you can add any number of either type,as well as rename them. Since you can define your own sockets foreach process, it's up to you to define how they're to be used and towhat they should be connected. Generally, though, if you plan to attachresources of different types to a process, it would make sense to definea different socket for each type of input or output.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/Sockets/SocketsExample1.png8/11/2019 Pipe-It Manaul 20130806
58/264
Pipe-It Manual Page 57
From the contextual pop-up menu, bring up the Edit Socketsdialog forthe Scripter. There you'll see all of your sockets and be able to editthem. You'll also see which connectors are attached to each socket andin which order (which is also editable).
TIP: If different "types" of resources are connected to a scripter andare to be used for different types of arguments on the command linethen it is recommended to create new sockets for each type ofconnected resource, otherwise the wrong file may end up at theposition of the wrong argument.
Socket Types
All elements (except connectors) have sockets. Let's pass through
types to show what kinds of sockets can be used in appropriate items.
Resources
Resources have exactly two sockets -- one for input and one foroutput:
Resource Input Socket
Resource Output Socket
ProcessesScripters also start with two sockets, Input and Output, but you candelete either one and/or add any number of either type, as well asrename them.
Process Input Socket
Process Output Socket
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/SocketTypes/SocketTypes-1.png8/11/2019 Pipe-It Manaul 20130806
59/264
Pipe-It Manual Page 58
Composites
Composites can have all 4 types of sockets due to all available internal
elements:
Resource Input Socket
Resource Output Socket
Process Input Socket
Process Output Socket
Let's illustrate all possible types of sockets inside composites:
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/SocketTypes/SocketTypes-3.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/SocketTypes/SocketTypes-2.png8/11/2019 Pipe-It Manaul 20130806
60/264
Pipe-It Manual Page 59
Runner
The Runner is a behind-the-scene component of Pipe-It thatorchestrates the execution of the project. It checks the status of all the
elements, if all the resources have associated files connected, anddecides the order of execution. It manages the launching of scriptercommand lines to the OS and waits for their completion. It determinesthe complete dependencies of the elements and keeps the elements onhold till all the upstream elements are up to date. The runner is visually
invisible to the end user, who will only interact through the GUI.
Pipe-Itcallows running a model from a command line without invokingthe GUI at all. In such case the Pipe-It Runner is only being invoked
directly.
Runner's Project (Composite) execution logic
The rules for running composites are the following:
All active processes within the running composite, and within itsnested, active sub-composites, will be checked and run in the proper
order.
All active resourceswithin the running composite, and within its nested,active sub-composites, will be checked and updated in the proper
order.
All processes outside of the running composite will be consideredinactive (as will any connection to or from such a process).
All active resources outside of the running composite will be checkedand updated if attached to the running composite by a chain of active
connections that is either strictly upstream or strictly downstream.
If one wants to completely isolate a running composite from all outside
elements, the easiest way to do that is to deactivate the compositeitself (non-recursively). The composite can still be opened and run inthat state, but all outside connections will be considered inactive. Thiscould prove to be a useful technique for testing individual composites.Be advised that the results might differ from those obtained by running
the complete project, however.
Using Projects by multiple users
Pipe-It allows projects to be opened by single user at same time. Toprevent inconsistency due to simultaneous modifications to project
8/11/2019 Pipe-It Manaul 20130806
61/264
Pipe-It Manual Page 60
files, there is lock file is created when user opens a project. The lockfile contains information about user and host name, time stamp of last
access (load or save) to project and PID of process.
When you try to open project being in use by another user, you will get
a warning which indicates a user name and computer, so you cancontact him directly to unlock the project and have possibility to open
and modify it.
Example Pipe-It Projects
This section gives glimpses of the showcase projects built with Pipe-It.Both the scale (number of elements) and technical complexity and
variety of the projects are impressive.
Multi-Field Complex Fluid Integration (Statoil)
Over 11,000 Resources in this project and nearly 3500 Processes
(application launches).
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/MultiUsersProjects/AppHelpLockFileWarning-1.png8/11/2019 Pipe-It Manaul 20130806
62/264
Pipe-It Manual Page 61
Production Optimization with Assisted HM (Petrobras)
Master project that performs the production optimization.
Compositional Well Test Allocation (Equion/Ecopetrol)
Almost 1500 Resources and 750 Processes organized in 600
Composites.
8/11/2019 Pipe-It Manaul 20130806
63/264
8/11/2019 Pipe-It Manaul 20130806
64/264
Pipe-It Manual Page 63
If the project contains optimization, one or a series of Pipe-It ProjectOptimization (*.ppo) files are also present. They can be located
anywhere.
A complete project will also contain data files that are connected to
each resource. All files connected to initiating resources (those that arenot downstream of any process) are also mandatory for the properexecution of the project. These files are typically located at the root or
in folders under the root.
Designing a Pipe-It project requires some planning. Simple projectsmight contain only a handful of data files linked to resources and thesecan live in the project root folder. But real life projects containhundreds if not thousands of files. It makes sense to organize thesefiles in a sensible folder hierarchy. The project within Pipe-It should
also follow the folder structure using composites and sub-composites asfar as possible. This helps in cloning of composites in case a newinstance of the functionality encapsulated in the composite is to becreated. Pipe-It estimates the % of files that resides within a particularsub-folder of the project and offers to the user the possibility ofduplicating files. This is really a powerful option as all files, theirconnections to resources, their Linkz variables, and all process
command lines are cloned.
Making order of the hundreds and thousand files is another reason for
collecting related files together.
Use Recycle Loops
Pipe-It runs projects in an efficient manner. Only outdated parts areexecuted. All resources in the project are checked for modificationdates of files to decide which part of project should be executed. Thisavoids unnecessary execution of the same Process in subsequent runs
to generate identical data.
But there may be certain instances when some identified portion of the
project should be executed each time you press Run button. Forexample, imagine a project which fetches input data from some
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/UseRecycleLoops/AppHelpUseRecycleLoops-1.png8/11/2019 Pipe-It Manaul 20130806
65/264
Pipe-It Manual Page 64
network services (e.g. Web Services). Before processing the rest of theproject there should be a mechanism to get updated data from theexternal component (Web Services) since the external component maynot be automatically updating a resource within the Pipe-It project.
A simple two-manifold recycle loop is the simplest way to achieve this.Everything within and downstream of such a recycle loop is guaranteedto run every time the project is run (because the loop always outdates
itself).
Caution : Using recycle loops sparingly. Using them without properconsideration may have unexpected side-effects such as downstreamprocesses being executed before upstream processes.
Utility Method Dialog
In process of designing projects you may find that you use similarscript commands repeatedly. Usually they only differ in filenames andfew other arguments. To make the script writing easier and some-whatautomated, rather than writing it from scratch each time, you can useUtilities available to a default installation of Pipe-It to use templatesapplicable to your project. Or create new ones.
See Script Assistant for more details.
8/11/2019 Pipe-It Manaul 20130806
66/264
8/11/2019 Pipe-It Manaul 20130806
67/264
Pipe-It Manual Page 66
Tool Bar
Application Tool Bar consists of 3 main sections:
Now describe all available actions in the tool bar:
File Operations
This section has 3 main standard file actions as:
Create New Project
Open Project from disk
Save Project
Note that Open Icon has small arrow in right bottom corner. If the
Open icon is pressed and held for one second then a drop-down menuappears with list of recently opened projects.
Navigation, Selection and Zoom
First 3 buttons (greyed out in picture above) allow navigation throughcurrently opened composites. Backand Forwardwork in same way
as navigation buttons in a web browser. Up button changes view toupper-level parent composite. All these buttons can be pressed and
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/ToolBar/AppHelpToolBar-View-1.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/ToolBar/AppHelpToolBar-FileOpen-1.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/ToolBar/AppHelpToolBar-File-1.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/ToolBar/AppHelpToolBar-ToolBar-1.png8/11/2019 Pipe-It Manaul 20130806
68/264
Pipe-It Manual Page 67
held for one second to get list of history entries or, in case of Up
button, for list of all parents of current composite.
Next 3 buttons Select, Zoom In and Zoom Out manage the current
mouse cursor mode.
Selectallows selection, resizing, moving etc. canvas actions on mouseclick. This is the most common mode of operation.
Zoom Inon click to canvas changes the zoom level for +10%
Zoom Outon click to canvas changes the zoom level for -10%
When Zoom In or Zoom Out is selected, then holding Alt (on Mac:
Option) button temporally switch to opposite zoom action.
Also in Zoom Inor Zoom Out, area selection on canvas switches viewto Zoom-To-Fitthe selection area into view:
Next to it is the, 1:1button. It is not a modebutton, it just switchescurrent view to un-zoomed (1:1) scale or in other words to 100%
scale.
Finally, Last button in the tool bar is Zoom-To-Fit action with drop-down menu. Just pressing the button switch zoom level to fit all yourpresented canvas elements in current view. It has drop-down menu to
apply Zoom-To-Fitto:
Current Window - Ctrl+9
8/11/2019 Pipe-It Manaul 20130806
69/264
Pipe-It Manual Page 68
Zoom-to-Fit Canvas - Ctrl+6
Elements to insert and Run
The toolbar contains 6 buttons which to add elements to the Canvas.The behavior of these buttons can be summarized as click-and-click.This means the button should be click once on the tool bar and thenthe location on the Canvas should be clicked where that element should
be placed.
Double clicking for continuous insertion mode
You may also double click on buttons on the tool bar. This will turn onthe tool until you choose another one. This will prove practical if youwant to add more than one element of the same type to the canvas.For example if you have lot of connectors to make, just double clickand make all your connectors as you want them to be, without having
to turn the connector tool on for each one.
Insert Resource
This button means adding a Resource to canvas. The resource
object represents a file on disk. Though you can give a several name tothe resource without a connection to actual used file name of the
associated file on disk.
On canvas the object looks like:
First double-click on the object opens Select File Dialogto associate afile with the object. A double-click on the object, that already hasassociation, runs OS defined default Application associated with the file.For example, clicking on a resource with associated "text.txt" file, will
open Notepad.exe
Also you can just drag-and-dropfrom Window Explorer to Canvas a fileobject. It will create a Resource automatically, associate the file toresource object and renames the resource it to File Name of droppedfile.
8/11/2019 Pipe-It Manaul 20130806
70/264
Pipe-It Manual Page 69
Insert Scripter
This button means adding a Program (Script) to canvas. Thescripter object represents an application to execute.
On canvas the object looks in such way:
Double-click on the object opens Edit Script Dialog.
Sub-menu associated with the button allows to change default action to
insert other elements that are bound to Programcategory.
Insert Manifold, Distributor and Sockets
This button means adding a Manifold to canvas. Manifold gather
information from multiple resources into one visual connector.
Sub-menu associated with the button allows adding other type ofsimilar elements Distributors, Input/Output Resourse/Process
Sockets.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/ToolBar/AppHelpToolBar-Manifold-1.png8/11/2019 Pipe-It Manaul 20130806
71/264
Pipe-It Manual Page 70
Insert Composite
This button means adding a Composite (Sub-Project) to canvas.
On canvas the object looks in such way:
Look at Composites? and Socket types? pages for more details.
Insert Annotation
The next button allows placement of new connectors between
resources and processes.
Insert Annotation
This button means adding a object with text or image that does notplay role in logic but can represent some illustration, information orinstructions for the project.
Undo Button
provides fast way to revert your changes one step at a time.
http://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/ToolBar/AppHelpToolBar-Undo-1.pnghttp://vmware-host/Shared%20Folders/DATA/Petrostreamz/Development/Distros/BuildManagement/Release_v1.3.0/Docs/contents/ToolB