The SNAME OpenCalc System Stephen M. Hollister (M) New Wave Systems, Inc., Jamestown, Rhode Island The SNAME OpenCalc System is built upon an open source framework that allows for the creation of long- lasting and reusable calculations that can be combined in many ways to create completely new solutions by naval architects, not programmers. Flexible calculation tools are created by splitting traditional interactive program applications (“apps”) into three separately developed and tested objects: calculation engine (CE) batch programs, open source and industry standard (XML) data files, and open source user interface frameworks (UIF) that work with any CE without programming. A working prototype system has gone through two revisions and a third is now presented as a formal release. The SNAME OpenCalc System offers a new programming structure that returns creative control and flexibility to subject matter experts (SMEs) and users. It can create solutions not possible without access to all source code and expensive custom programming. This is critically important in fields like naval architecture, where small market size and demand have not created lasting, flexible, and cost-effective computer solutions and design tools. KEY WORDS: Programming; XML; CAD; design; calculations; optimization.
Transcript
The SNAME OpenCalc System
Stephen M. Hollister (M)New Wave Systems, Inc., Jamestown, Rhode Island
The SNAME OpenCalc System is built upon an open source framework that allows for the creation of long-lasting and reusable calculations that can be combined in many ways to create completely new solutions by naval architects, not programmers. Flexible calculation tools are created by splitting traditional interactive program applications (“apps”) into three separately developed and tested objects: calculation engine (CE) batch programs, open source and industry standard (XML) data files, and open source user interface frameworks (UIF) that work with any CE without programming. A working prototype system has gone through two revisions and a third is now presented as a formal release. The SNAME OpenCalc System offers a new programming structure that returns creative control and flexibility to subject matter experts (SMEs) and users. It can create solutions not possible without access to all source code and expensive custom programming. This is critically important in fields like naval architecture, where small market size and demand have not created lasting, flexible, and cost-effective computer solutions and design tools.
Batch Programs - Computer programs that do not have any user interface. They can be run automatically.CAD - Computer Aided Design - General purpose interactive programs for geometry design of any object or structure.CE - Calc Engine - TSPA/OpenCalc component that isolates calculations into separate batch programs.CFD - Computational Fluid Dynamics - A class of programs that use meshes to perform detailed fluid flow calculations.Command File - A text batch or command file with commands that an operating system understands and can perform. They are often called BAT or COM files.CSV - Comma Separated Value File - A data file storage format used by spreadsheets to transfer worksheet data to other applications.DLL - Dynamic Link Library - A method for combining separate library routines with programs at run time.DOS - Disk Operating System - The operating system for IBM PC compatible computers in the 1980s and 1990s.EXE - Executable File - A file type that defines a program that can be launched by an operating system.FEM - Finite Element Method - A class of programs that use meshes to perform detailed structural stress, strain, and deformation analyses.GUI - Graphical User Interface - A front end to a program that offers users interactive and graphical control, as in a CAD program.HTML - Hypertext Markup Language - Tag-based text file commands and data for display of information on web pages.ISO - International Standards Organization - World-wide organization that defines and promotes common standards for all.SME - Subject Matter Expert - A person who has expertise in a subject like naval architecture, but limited expertise in computer science.SVG - Scalable Vector Graphics - An XML standard for storing vector graphics geometry and display information.TSPA - Tri-Split Programming Architecture - Divides a program into three separate parts: User Interface Frameworks, Calc Engines, and XML text file data for any variable or data structure. OpenCalc is built on this open architecture.UI - User Interface - Any interactive front end to a program that offers direct user control.UIF - User Interface Framework - TSPA/OpenCalc component that refers to a user interface that can work with any calc engine. Spreadsheets are the main vehicle used in this work for UIFs.Unix - A multitasking, multiuser operating system developed in the 1970s at Bell Labs.Unix Pipes and Filters - A technique for combining text files and batch text processing programs in an automated process command file.VBA - Visual Basic for Applications - Programming language for the Excel spreadsheet.W3C - World Wide Web Consortium - The main international standards organization for the World Wide Web.XML - Extensible Markup Language - An ISO standard for storing tagged data in text files. It also refers to the XML data portion of TSPA/OpenCalc.
INTRODUCTIONTime, changing technology, and a small market have resulted in a lack of marine design and analysis computer software for naval architects, yacht designers, students, and hobbyists. The SNAME OpenCalc System establishes a website, open technology, and a process within SNAME that will provide members and others with a computer program design and
development framework where everyone can offer pieces that support compatible and flexible solutions for all areas of the design process. This system is more than just a way to collect and offer "lost" or basic design programs. It provides both technical tools and organizational guidance that fix the issues of time, technology, cost, and market, and is a scalable and agnostic framework that will help everyone from students to naval architects to large shipbuilders.
OpenCalc is not about completely free software, although it comes seeded with a few reliable and validated calculations such as hydrostatics and stability. Instead, it is a free and open framework of tools for developing low cost, lasting, and compatible calculations that can be combined by users to create new solutions without programming. It can also be a valuable tool for independent software developers by lowering costs and increasing market size. It is a simple process to isolate calculations and insert them into one of the provided open source programming templates for calculations (calc engines). OpenCalc provides all the file I/O routines.
The key organizational glue to this system is the definition of a simple and flexible variable and geometry data file definition, especially for the conceptual and preliminary design stages. Development will be an ongoing process, but the integration of common and lasting computer design and analysis programming tools from a community of independent participants rests on this task. After more than 50 years of marine computing development, there is not even one common data definition for hull station offsets with open source I/O code that allows everyone to easily work together to create compatible tools.
The technological solution offered by this system is one that separates naval architecture programming knowledge from the complex and fast-changing world of writing modern graphical user interfaces (GUIs) - even spreadsheet programming. Too many programs are lost, as in the change from DOS to Windows, or not even created because of time, cost, and expertise. The framework provided by this SNAME system offers an easier way for naval architects to create their own calculations in a general form that can be used for many purposes. This is done using spreadsheets as the common, but not only, User Interface Framework (UIF) interface for launching independent calc engines.
Technology has changed over the decades to where PCs are now more than 10,000 times faster than the VAX 11/780 of 1978, and even the cheapest PC’s have 4 GB of memory and a terabyte of storage - and some PCs have 16 GB of fast memory and a terabyte of SSD storage. Reading/writing from/to a file in storage has little delay. When a UIF front end transfers data to a text file and then launches an external calc engine, the delay is unnoticeable. It became apparent that there is no reason to bind calculations and data processing routines to any user interface.
There is also no reason for independent software developers to define proprietary and binary data file definitions. They need to
Hollister The SNAME OpenCalc System 2
be open and coordinated by industry groups to allow compatible tools to be developed by many users for all types and sizes of vessels. The solution is not to separate and isolate all users into competing suites of computer programs, especially in the preliminary design stage which requires much more coordination and cooperation between independent naval architects, businesses, and approval organizations.
OpenCalc defines and separates the role of subject experts to better match their importance in many smaller market segments. When subject experts become the customer in the development of large computer solutions, they end up losing a lot of control and flexibility. They become simple “users” of fixed solutions, not creators of new ones. TSPA/OpenCalc changes that.
The OpenCalc initiative and process is a new and unique solution to difficulties in the programming world that has enormous and far-reaching benefits for many industries. By separating the user interface from the calculations, one not only eliminates the cost and complexity of GUI programming, but opens up the possibility of creating and sharing complex user-created combinations of calculations and optimizations. The SNAME OpenCalc System provides open source spreadsheet code that can launch any independent calc engine (or sequence of calc engines) without writing custom or macro code. The calculations are completely separated from the spreadsheet and can then be used for many other interactive or automated tasks.
HISTORY OF OPENCALCThe SNAME OpenCalc System started in 2014 as "Project 114" to try to save or restructure old code for boat and ship design to create more compatible, flexible and lasting tools. This has been an on-going problem since the beginning of the computer industry. Mainframe computers gave way to minicomputers, workstations, and PCs, and operating systems changed from DOS to Windows (and Apple OS to MacOS) to web-based systems. Each time, applications had to change and be rebuilt at high cost. The maritime industry can’t support those changes. Calculations might stay the same, but the cost of rebuilding a new application with a new user interface was and is too expensive, thereby losing much good software. The author, for instance, has old DOS-based code, such as a Wageningen propeller optimization program and a Velocity Prediction Program for sailboats, that never justified conversion to Windows-based user interfaces.
Excel and other spreadsheet programs are not a full solution even if one imposes format and data standards. Calculations are mixed in directly with the user interface, difficult to change and test, poorly supported, and cannot be easily used for other purposes. There is now a desire to be able to combine and sequence different calculations automatically to search thousands or millions of iterations for an optimum solution. While it may be possible to write a custom program to do this, it’s extraordinarily expensive and one needs all the source code.
For example, in a SNAME New England Section paper (Hollister 1996), this author combined hull variation, hydrostatics, and Holtrop resistance calculations into one DOS program to search for an optimum shape starting from a parent hull. This required all the source code and a custom user interface front end that needed a lot of development and testing. That this could be done was of no real benefit since the market would not support the cost. Also, that was when DOS was replaced by Windows and the recoding cost was no longer viable for the new technology and size of market.
Twenty years later, the author completed another attempt at the task using the separate component framework of OpenCalc. That paper was presented at the 2016 SNAME Maritime Convention in Seattle (Hollister 2016). Each calculation: hull variation, hydrostatics, and Holtrop resistance became a separately developed and tested stand-alone EXE batch (no user interface) calc engine program that read and wrote a standard text file of data using open source I/O code. These separate programs were then “wrapped” in an automatic command file and launched by the open source spreadsheet (UIF) that can execute any separate calculation program or sequence of programs. The spreadsheet macro code launches these calculations automatically and produces displayed, printed, and plotted output with no extra programming. The spreadsheet launches the calc engines hidden in the background and it appears to the user that those calculations are done inside of the spreadsheet. There is no delay.
With OpenCalc, one can write and test a separate program (calc engine) for a set of calculations in any programming language that can have a long life and be used for many different and new solutions by users, not programmers. In most cases, there is no need to write a standard user interface because the open source Excel spreadsheet Visual Basic for Applications (VBA) code will do the job. The calc engines can be located anywhere on the web and be copy protected and fee-based if desired. Spreadsheet VBA code and formulas are difficult or impossible to hide or protect, but calc engines offer a way to separate and protect proprietary methods.
Several years ago, the author received a call from a naval architect who wrote hull generation code for the Pro/E CAD program and wanted to automate the connection to a meshing program and then to a computational fluid dynamics (CFD) program in the hopes of searching for an optimal hull shape. In theory, this is technically possible, but the custom programming cost could not be justified even if one could get access to all the source code or automated batch calculations. TSPA/OpenCalc now makes that solution possible technically and financially. The designer would create a stand-alone calc engine of his hull shape generator and combine it with a meshing calc engine followed by a CFD calc engine. They could be launched by an open source front end that searched for an optimal solution.
OpenCalc is based on the idea that many interactive user interfaces are standardized and can be written to work with any
Hollister The SNAME OpenCalc System 3
number of independent and automated batch calculations without writing any custom code. The user interface for the sequence of separate hull variation, hydrostatics, and Holtrop resistance calculation engines was provided by an open source Excel front end that launched the three independent calculation programs without having the source code or writing new code. This cannot be done in any way using traditional interactive apps.
TRI-SPLIT PROGRAMMING ARCHITECURE The latest revision of OpenCalc generalizes the definition of calc engines as external stand-alone executable subroutines that are more stable, adaptable, and flexible, and it formalizes the definition of XML data and structures into more powerful and flexible hierarchical industry standards.
OpenCalc has evolved into a solution that goes far beyond standard marine industry calculations. It can form the basis for complex and flexible Multi-Disciplinary Engineering and Optimization (MDE/MDO) solutions that can combine and automate geometry manipulations, calculations, meshing, and CFD/FEM calculations.
The splitting of interactive applications into three components or programming objects causes a fundamental change in all areas of programming. That is why OpenCalc is now divided into two levels - a base open source structure called Tri-Split Programming Architecture (TSPA) (www.trisplit.com ) that is industry independent, and on top of that, industry specific levels to define their own XML data definitions and calc engines. SNAME’s OpenCalc system for the marine industry is built using the base tools and open source structure of TSPA. There are also cross-industry calculations and data definitions for geometry and basic engineering data. TSPA allows all programming objects to work together and individual industries can build and control their own product definitions on top of cross-industry and common definitions for basic geometric shapes. Naval architecture programming tools can and should be compatible with tools from associated engineering fields.
PROGRAMMING ARCHITECTUREThe traditional programming model in computer science is the interactive app where one software developer or team creates the user interface, calculations, and the data definition and file format. Separate calculations from multiple apps cannot be automated and the data is often incompatible and must go through a two-stage filter process to and from a neutral file format (Fig. 1)
Fig. 1 Traditional Programming Architecture
TSPA splits this structure into three separate “external” programming objects: open source User Interface Frameworks (UIF), Calculation “engines” (CE), and open source hierarchical levels of cross-industry and specific-industry defined variables and data structures (XML) that can be developed and tested separately by subject matter experts. (Fig. 2)
Fig. 2 Tri-Split Programming Architecture
These new external programming objects can be mixed and matched in many new and creative ways by users. This will be explained in more detail later in the paper.
As illustrated in Fig. 3, not only can one calc engine have more than one user interface, but one user interface can launch many different calc engines. The glue holding these pieces together is industry-defined data and structures built using open source XML files.
The fundamental change defined by this work involves splitting traditional interactive “apps” into three separately developed and tested parts. Calculations and data processing tools can be written separately by subject matter experts (SMEs), tested once, have long lives, and be used and reused for many interactive and automatic applications.
OpenCalc has shown an example where a general Excel spreadsheet using open source VBA code can launch any external calc engine. The spreadsheet will prompt for all input, launch the external CE or sequenced CEs, read the results back in, and then display, print, and graph the results. Tests have shown this to work with a variety of CEs without any changes to the spreadsheet UIF or the calc engines.
The spreadsheet launches the CEs in the background and the user never knows that the calculations are done externally from the spreadsheet. There is no delay. Results are displayed immediately even for a CE as large as a full damaged stability calculation. The benefit is that the user can now add in additional custom calculations and save the changes to a new spreadsheet. This also means that the separate calculation can be used in combination with other CEs to solve completely new problems without writing additional code. A key element in this process is the use of common data and data structure formats or schemas using XML. Industries have to take control over “their” data to enable compatible and low cost tools.
The next sections discuss each of these three external TSPA objects in more detail.
CALC ENGINES (CE)The first TSPA object type is a "Calc Engine" (CE) executable batch program that has no user interface. It's a stand-alone executable program that reads a text file of input, processes it,
and writes the results out to the same or a different XML text file. (Fig. 4)
Fig 4 Separate calc engine model that reads and writes files
Calc engines can be written in any computer language, validated separately by subject experts, used for many purposes, located anywhere on the web, and have a long life not affected by computer technology changes. It's just a simple EXE (or other type of executable program) batch file that reads and writes a text file and acts like a subroutine - one that is tested and validated and can be used automatically in many different ways. The mental shift required is to see that computers are so fast that automatic launching of external calculations can be hidden with no delay in processing time.
As described in the next section, separate and open user interface programs can be constructed to launch any calc engine automatically. This calc engine "object" is like well-known Unix filters such as "grep" (A batch program to search for a string in a text file.) except that the calculations can be as complex as computational fluid dynamics (CFD) analyses and support any program variables and data structures - not just strings in text files. Also, calc engines do not just filter data out from one text file to another. They treat the text files as random-access databases of any program variable or data structure. To turn separate calculations into reusable stand-alone tools for any data, TSPA requires that all CEs include an XML text file that defines all input and output data (like a subroutine argument list) using the TSPA XML schema layer for any variable and data structure. Figure 5 shows a TSPA/XML subroutine argument list definition for a simple CE that adds variable ‘A’ to variable ‘B’ to produce variable ‘C’. The UIF spreadsheet reads this definition to prompt for all input and show all output without knowing any more about the external calc engine.
Fig 5 XML CE/subroutine definition file
Batch CE programs can be launched from inside other programs and passed “arguments” just like a subroutine. For the case of
Hollister The SNAME OpenCalc System 5
this Add2.exe calc engine, the string that would start the batch program looks like this:
Add2 (A,B,C) [io=MyDataFile.xml]
The open source I/O code of TSPA/OpenCalc reads and parses the Add2 arguments to know what variables to access in the xml file. Also, like standard subroutines, arguments are passed by position, so someone could launch Add2 with “(X,Y,Z)” and the code would read the input variables (X,Y) from the file (MyDataFile.xml) and assign them to “A” and “B” in the calc engine. Writing out the “C” value would go into the “Z” variable in the file.
If the argument list becomes long, which is not uncommon, it can be put into a text file and referred to like this:
Add2 [args=add2args.txt io=MyDataFile.xml]
If no argument list is given, the CE will use its own variable names to access data in the XML data file. This might be useful for carefully designed and coordinated CEs.
Eventually, one will be allowed to use expressions in an argument list as in most programming languages. In addition, if an output argument is left blank, then the CE will not write that variable to the output XML file. This allows one to pick out only those results needed without cluttering up the data XML file. A traditional Unix filter is a calc engine that reads a simple text file for input and writes a simple text file for output. A Unix filter can be turned into a TSPA calc engine by providing an XML definition file that defines the two I/O variables as strings. Unix text filters are just one small case of TSPA and Unix filters can be used with any TSPA object.
TSPA allows calc engines that define any complex sequence of input and output variables and structures equivalent to any programming subroutine. A TSPA calc engine is an external executable batch program that works just like an internal subroutine or function with the XML data file acting as memory. However, calc engines will typically perform more complex tasks than simple calculations. With modern computer speeds and fast static data file storage, the overhead is minimal and more flexible compared with linking to a dynamic link library (DLL). The benefits of unlinking, open and flexible definitions, separate development and validation, and generality offer many more possibilities.
TSPA provides open source tools, templates, and data schemas to create, read, and write any program variable and data structure. To create a new CE, a subject expert opens a TSPA calc engine program template that includes all the necessary I/O code, adds in their calculation or processing routines, compiles and links the code, and then defines the input/output XML text file subroutine argument list. It can then be immediately used by
anyone on the web without writing user interface or connecting code.
There is no reason to combine and bind one set of calculations or data processing to one custom user interface. That increases complexity, validation problems, computer and content expert interaction issues, and cost. It also makes those calculations unavailable for other uses. Even if the calculations were isolated in software libraries, their attachment to and validation with new custom Windows user interfaces is not justified in many cases, especially in smaller markets.
Many subject experts build "apps" using spreadsheets, but the calculations are tied to the cell locations of the user interface and cannot be combined with other calculations automatically. Comma Separated Value (CSV) files can be used to transfer data to other applications, but users still need to manually transfer those files and agree upon common data definitions with other users. Use of CSV prevents combination and automatic execution of a series of separate calculations. Also, one study (Panko 1998, 2008) has shown that 88% of all spreadsheets have errors. Many find that it's not worth the time and effort to understand spreadsheet formulas or macro code to be able to use the spreadsheet, let alone improve or use the calculations for other purposes. TSPA provides subject matter experts with a better model for spreadsheet applications.
A Finite Element Modeling (FEM) calculation could be located on a very fast computer and offered on a subscription or cost per use basis. Users could create their own automatic geometry generator calc engine, send the output to a mesh generator calc engine, and then pass the mesh to the external FEM calculation program which returns the results back to the original user. This wrapped collection of external calc engines could be launched from an open front-end UIF program that automatically searches for an optimum shape based on a user-defined structural criteria merit function. This open spreadsheet UIF could expand to include multi-objective optimization problems that calculate a Pareto Front. With TSPA, this could be done without writing new code or having the programs locally, let alone having the source for each application. This type of automation and user control is not possible with the traditional interactive app and private data approach to program development.
TSPA will provide program templates for different languages that include all the necessary source code for data file I/O. All one needs to do to create a new calc engine is to add in their own calculation routines, compile, link, and test. Once testing has been done, it can be immediately used for many different purposes by users anywhere on the web.
USER INTERFACE FRAMEWORKS (UIF)The second TSPA file object type is a "User Interface Framework" (UIF) program (preferably a free and open source program) that can launch one or more calc engines without writing any code. Calc engines can be grouped or classed by the
Hollister The SNAME OpenCalc System 6
type of processing they do and that defines a basis for providing open source UIF front end programs that work with any calc engine of that class. Figure 6 shows the separation and connection of a graphical user interface (GUI) object with a calc engine object.
Fig 6 TSPA Separate GUI/CALC engine model
The most common form of data manipulation is the creation, editing, and saving of data. In the future, TSPA will provide an open source user interface framework that can do those tasks for any data file defined with an XML, HTML, or TSPA data schema. It will also allow users to apply any external calc engine that might generate or alter the data automatically. One calc engine might translate string data from one language to another, and another might convert numeric data from English to Metric units. Yet another might transpose music from one key to another. The calc engine possibilities are unlimited, but the basic UIF framework is the same for all - read a data file, offer interactive or calc engine data manipulation, and save the results back to the file. The calc engines are all external, and if a user finds (or creates) a better or different one, it can be swapped in with no other changes. There is no reason to write a separate and custom user interface for each calc engine manipulation of data.
A very large class of calc engines takes a set of input variables (X), processes them, and then writes out the resultant numeric variables (Y) back to the same data file. This is a classic black box calculation of Y = f(X). The calculations could be as simple as C = A + B [C=f(A,B)], or they could be as complex as a finite element analysis with forces and a very large mesh as input and stresses as output. Note that a calc engine (large or small) does just one set of calculations for one set of input. This is a key point. The standard UIF front end in OpenCalc provides all the data input, looping, display, printing, graphing, searching, and visualization. The UIF does this for any calc engine of its class without coding by reading the XML list of inputs and outputs of each CE. The UIF can launch one calc engine or it can start a batch (BAT) or command (COM) file that launches many different calc engines, automatically feeding the results of one calculation into the next using the common XML data file. Users can define command files or scripts that process any sequence of calc engines either separately or under direct user control from a UIF.
TSPA will also provide open source tools that make it easy for calc engine writers to output 2D views into standard XML/SVG (Scalable Vector Graphics) files - one for each launching of a calc engine. This would allow the standard UIF to display either one view or animate a series of SVG views generated from looping input over one or many wrapped calc engines. An XML schema for 3D visualization and rendering is another goal, but it will be built on top of open cross-industry XML geometry tags, not their own internal geometry tags. Instead of defining a rectangle or a polygon with internal SVG tags, it will link to a general cross-industry polygon tag definition in another schema namespace.
The Y = f(X) form of UIF can also take any one or more wrapped calc engines, prompt for the input variables (X), launch the external calc engine or engines one or more times, and then read back in the result variables (Y). The UIF can also calculate, print, plot, or contour any set of input and output values. Searching for optimum solutions based on user-defined merit functions will be added next. This will work for any calc engine of this class. Again, there is no reason for each set of calculations to have its own custom and attached interactive user interface. The CEs are launched automatically from the UIF and the user never sees that happen. CEs require less effort to develop and they can be used for many purposes, including the recreation of a traditional interactive “app” form.
A variation of this UIF class could take two different data sets and run them through the same sequence of calc engines. Printed and plotted results could be compared side-by-side or on the same graph. Yet another variation could have a UIF framework where one data set is run through two different sequences of calculations and compared side-by-side. A calc engine writer could compare the results of an old version to a new version without custom programming or piecing together output from two different apps.
Another class of calc engines might read raw sensor data anywhere on the web and save the results into a TSPA compatible data file. A general sensor data UIF located anywhere else on the web might read and display the raw data from one or more sensors in real time and apply any number of processing or time averaging calc engines. Much of this is possible now due to the interconnectivity and bandwidth of the internet along with fast computers and massive amounts of memory and data storage.
Many engineering calculations are based on a static or dynamic balance of forces and moments - a Free Body Diagram (FBD). One could have an open source UIF front end that processes any set of forces and moments, and each one could come from an external calc engine located anywhere on the web. Groups could work on the same FBD model, where each member develops a different force or moment calc engine. The FBD UIF could be located anywhere on the web and point to force and moment
Hollister The SNAME OpenCalc System 7
calc engines located anywhere else, such as on Google Drive or Microsoft's OneDrive.
Yet another engineering analysis model looks to find an optimal geometric shape based on a complex set of separate calculations and a potentially infinite geometry shape input domain. One can start with a geometry shape generator calc engine followed by one or more geometry evaluation calc engines in an automated UIF loop that searches for an optimum shape based on one or more user-defined merit functions, like cost and weight. This UIF would automate the sequential processing of any number of geometry variations and calculations. This is possible only if the calculations are created as separate batch calc engines that use compatible data. This process has been shown to work using a standard UIF Excel front end, a complex curved ship hull geometry, and three separate calc engines. No coding was required to combine and launch the separate calc engines, and it can be done by a subject expert, not a programmer.
There is also the possibility of having UIFs located on web sites so that they can launch a calc engine from anyone on the web. Instead of custom programming web sites, one would just link standard web UIF code to one or more CEs.
The possibilities for calc engine UIF front ends are endless, and the TSPA goal is to make them open and free. Note that spreadsheets provide a perfect platform for many of these UIF frameworks. It might seem odd to use a spreadsheet to launch external calculations, but users can still add in their own customizations or calculations in the spreadsheet to alter the standard UIF for each use case. Instead of tying one user interface with one set of calculations, users can customize the standard UIF spreadsheet that launches one calc engine or they can use the standard UIF to launch many calc engines wrapped in a batch or command file script.
The big justification for splitting off user interfaces from calculations is that large classes of calculations share the same user interface needs. Most general UIF frameworks can be provided by spreadsheets using open VBA code or script. Separate calc engines can be launched alone or wrapped with any other calc engine, and they can be reused in many different and creative ways by end users.
OPEN AND LAYERED DATA SCHEMAS (XML)The third and key TSPA object type is open and industry-defined text data files built in layers of XML schemas. This is the glue that ties everything together. The days are over when independent software developers define and control industry data and structures in proprietary file formats that work with only one program. The Web is built on separate and compatible programs that work off a common HTML data file definition, and HTML has now been defined with XML and is governed by the World-Wide Web Consortium (W3C) and the International Standards Organization (ISO). Web programmers do not define
their own data file formats. Common data drives the Web, but that’s missing in the rest of the programming world.
HTML defines a file definition for interactive data (text, graphics, audio, video) with display and format information, while TSPA defines a layered industry standard XML data file framework for all program variables and structures. It’s no longer necessary to have a separate custom data file for each application.
The TSPA/XML text file data definition acts like a general database for program variables and structures for all programs. There is no need for two-step neutral file format conversion. Programmers can use this organization to define all their own data and structures, but the structures won’t be hidden away in binary files. Industries need to assert some level of influence to ensure better compatibility and data flow between calculations from different sources. In naval architecture, that might involve some naming and organization recommendations for things like different hull definitions, weights, and operating conditions.
TSPA will offer open source I/O code in many languages so that software developers can access any program variable or data structure from a common file without knowing anything about the file format, and one text data file can contain data from all programs. When separate calc engines are launched in sequence, they can pass data via this common data file automatically.
XML is not a file format in a traditional sense. It’s a way to surround data with opening and closing tags in a text file. TSPA/XML wraps each program variable and data structure in tags so that they can be put into the file in any order or combination. Access is made by name and not position. One might think of it as program memory in a file. Programmers don’t know where a variable is located in memory. They just get and put the data by name. It works the same way for TSPA/XML, which reads in the entire file as a string, builds a list of tags and variables, and then offers routines to get/put any variable by name. When done, the programmer can overwrite that entire string to the file.
The lowest layers of TSPA/XML, including a geometry layer, will be submitted to W3C/ISO as a world standard. They are built upon their base XML definition. Industries can then build their data and structures on top of a common and cross-industry compatible base. Industries can control their data, but they can be cross-industry compatible using common lower level definitions for data like geometry.
TSPA data schemas are built in layers to define everything from simple variables to complex geometric data structures. Since XML is a text markup language that surrounds data with tags that are accessed by name, anyone can add in their own custom data tags without affecting any program or the open TSPA I/O code. This is a very important concept that allows for both common and custom data content at the same time in a single file using the standard I/O code. Custom tag data just gets
Hollister The SNAME OpenCalc System 8
passed along using the common tag format. It’s up to each program to use or ignore the data. This is a key element that should satisfy both industries and independent software developers.
Again, XML data definitions are defined by schemas, not file organizations or formats. Text files are just streams of characters and in many cases, the tag data can be in any order and broken into any number of separate files to fit the needs of a solution process. This is quite different than the traditional method where independent programmers define specific data files and line formats for each program. Data in TSPA form is more like a random-access database definition than a sequential collection of data useful for only one program. Figure 7 shows how data definitions form layers that are built on top of each other.
Fig 7 Layered XML schema example
The first layer of TSPA data definition builds on top of a base XML tag layer and uses that to define any programming variable and data structure. TSPA constructs the next layer to define any program variable, array, matrix, or data structure - linked lists, tables, tree structures, network organizations - to match any that can be created in a computer program. This makes it a simple matter to save and retrieve any variable and data structure organization to/from a file. This variable tag layer also includes pointer and link variables so that it can directly match any data type and structure that can be defined inside of a program.
Figure 8 shows a small TSPA XML file that contains 4 variables; two as single variables and two as arrays. Notice that all variables can have units defined and the I/O routines can translate units on the fly going into and out of a calc engine.
Fig 8 Variable and array definitions
TSPA also allows for common program “Structure” definitions. Structures collect a group of variables together as a single entity. A structure can be used as a single variable or as an array. Figure 9 shows a variable (HULL2) that has a structure data type (DT) called STATION2 in the NAME (Naval Architecture and Marine Engineering) namespace consisting of variables that define one station of a station definition of a hull. In this example, HULL2 consists of two stations with offset points and breakpoint/curve indicators. Each station is surrounded by <I> … </I> tags. This definition contains enough information to allow a program to use the open I/O code to get or put any number or array using a name like “HULL2[1].Z[3]”. This is the 4th Z-value of the second station of HULL2 (zero-based arrays). Note that HULL2 defines a 100 ft long box barge. Programmers no longer need to do any file I/O programming.
Fig 9 Structure definition for a hull using stations
The next TSPA schema layer uses the lower variable and structure tag level to define common cross-industry definitions, like geometry. The goal with geometry data is not only to eliminate neutral file formats, like DXF, IGES and ISO-10303 (especially because of the cost, complexity, and no open source I/O code) for product design, but to define geometry from the bottom up rather than the top down. A TSPA XML definition of a polyline geometry tag variable can be used in many different product definitions.
Programmers modeling different types of geometry should not be defining their own polyline definitions as incompatible sub-tags of custom and complex XML schema. A polyline tag should be part of a common cross-industry XML schema (namespace) that can stand by itself. A user who needs to work with polyline data should only have to learn that one simple tag definition. Software developers no longer need to redevelop the wheel for data file formats for things like polylines or curves using formats that are readable only by their program.
Figure 10 shows a layered variation of the structure shown in Figure 9, where the 2D stations are defined using a lower level “GEOM” namespace structure called “PATH2D.” This is a general 2D geometric object that defines a combination
Hollister The SNAME OpenCalc System 9
polyline/curve path using indicators (zero indicates a knuckle or polyline point rather than a curve-fit point).
Fig 10 Layered structure definition for a hull
Note that the offsets (OFF) for each station points (P) to a separate named variable called “Box2010”, which is a cross-industry definition of any 2D path shape. Since both stations are the same, they can point to the same 2D shape. A calc engine could get the ‘Y’ array of the second station by passing the string “HULL1[1].OFF.Y” to the open source I/O routines. The routine would return a set of 4 doubles: 0.0;0.0;10.0;10.0.
What this allows for is access by other non-naval architecture programs to that data because they know what a PATH2D tag is and not what NAME:STATION2 Y and Z arrays are. One could have a CAD program read, edit and store PATH2D tag data without knowing that it has anything to do with a hull. Common cross-industry data definitions in this form are much more powerful than the use of neutral files. The TSPA project will define cross-industry definitions for many types of geometry, from points through NURB curves and surfaces. These definitions will be offered for use in an open source format, but they will not preclude others from creating their own definitions.
Higher TSPA schema layers can be defined by industries that want to take control over their variables, data structures, and geometries so that subject experts can create compatible calc engines and combinations. Since many geometries can be built on top of the lower level geometry tags, calc engines developed for a lower level tag will work for that tag when it’s used in any higher-level schema definition in any industry. The low level calc engines will be able to pull out and operate on the lower level data tags in any higher level XML data structure files.
For example, a general calc engine that creates a 2D airfoil section curve could be used as part of a larger airplane XML definition. The open TSPA I/O code allows any calc engine to find the tag variables they need from any XML file, process the tag information, and put it back into that same file without ever knowing about or affecting any other tag data. A 2D airfoil variation calc engine could be combined with a 3D airplane analysis calc engine to automatically search for the best foil sections to use. Up until now, users have been happy enough to have ways to manually connect data between separate interactive apps, but that will fundamentally not work when the goal is to automatically search through thousands or millions of separate calculations. Interactive and manual apps are not the only programming need anymore. Separate calc engines are needed.
In another example, if some complex object is made up of thousands of polygons defined with the standard lower level polygon tag definition, one could apply a generic stand-alone polygon area calc engine to determine the entire surface area of the object. A TSPA UIF front end could take any calc engine and apply its processing to any data file that matches the TSPA layered data structure. The UIF will look in any XML file for specific tag data to pass to any calc engine.
One could write a calc engine that curve-fits a polyline data tag and converts it into a spline curve PATH2D tag definition. This calc engine could be applied to any polyline tag located in any complex XML TSPA data definition for any geometry in any industry. Common calculation tools do not have to be recreated and custom built into each industry application. TSPA greatly reduces the need to recreate user interface and calculation code and gives those tools to subject experts to create new solutions without custom programming.
USERS AS CREATORS Now that computers are tens of thousands of times faster than the original PCs and have massive amounts of memory and storage, and now that computers can transfer data to and from any other computer on the web, a new, more flexible programming model is needed than the traditional all-in-one interactive and manual program “app” created by programming experts who tie everything together into fixed manual solutions for “users.” TSPA defines a new open framework that breaks apart the app and allows subject matter experts to develop their own pieces to combine and use in new creative ways.
Calc engines can be launched by automatic script, command files, or by other calc engines, and they can be launched from interactive UIF frameworks that perform the input/analysis job without any programming. Users can also alter a general UIF to provide multiple custom user interfaces for one calc engine. This could be a way to provide simple versus complex front ends to the same set of calculations. As programs get more features (inputs) and complexity, one can set default values for input values and the UIF will adjust automatically to hide them.
Hollister The SNAME OpenCalc System 10
Users can also make copies of a spreadsheet UIF and add in extra formulas for specific jobs. It’s difficult for traditional programmers to add in the custom programming capabilities of a spreadsheet to each stand-alone app. One can think of this combination of UIF to a single calc engine as an N-to-1 relationship, and users can create their own spreadsheet front end.
Users can also have one general UIF launch several independent calc engines in sequence without writing any new code. The UIF that handles any “black box” external calc engine can also start any batch file that launches several separate calc engines located anywhere on the web. The combination of separate calc engines can be viewed as a “wrapped” black box calculation with its own larger set of inputs and outputs that can be created from the individual I/O calc engine definition files. The standard UIF front end adjusts accordingly. One can think of this relationship as 1-to-N. Therefore, TSPA offers an N-to-N relationship between user interfaces and calc engines that can be created and controlled by subject experts and users.
To get these new combinations to work, one does not need the source code for the calculations and the executable programs can be located anywhere on the web. CEs can be created separately, validated by subject experts, and used for many different purposes, either alone or in combination with other calc engines. TSPA defines a standard process for layered data and structure definitions that adapts to both cross-industry and industry-specific needs. Most importantly, these separate programming objects can be mixed and matched to solve completely new problems without additional programming. With TSPA, the calculations can stay as separately-validated EXE files even if technology changes. One might create a new UIF front end to adapt, but that would not change the calculations at all. In the world of computers, EXE batch programs and text files have not changed over the years. Calc engines and text data files can have long lives and many uses.
TSPA defines and separates the role of subject experts to better match their importance in many smaller market segments. When subject experts become the “customer” in the development of large computer solutions, they end up losing a lot of control and flexibility. They become “users” of fixed solutions, not creators of new ones. TSPA/OpenCalc changes that.
COMPUTER-AIDED DESIGNThe current trend in CAD is the move away from stand-alone programs for ship design to those built inside of or on top of general purpose programs. This divides the market into smaller communities and isolates those who cannot support the extra CAD costs and learning curves. It also increases the complexity and development costs for independent software developers. The tools created for one CAD environment are not available to other CAD environments without expensive custom programming. The goal is not to hope that one CAD program wins the market battle.
In addition, these CAD-specific tools are interactive in nature and not available for combined or automated use, as with searching for optimum solutions. OpenCalc, however, defines a way to create stand-alone calculations that could be combined and launched by any CAD program using the open source I/O tools. In this way, a naval architect/programmer might create and test one standard calc engine program that would work with any compatible (CAD or other) program from anyone.
In a CAD program, a user could take the selected hull geometry and send it off to an external calc engine to perform a calculation or geometry variation task. This could be done automatically in the background with no custom programming. One could also have an external calc engine generate geometry to pass into any CAD program as a starting shape. This calc engine could be written without any knowledge of the CAD program. It would be the CAD program's responsibility to use the open source code to provide a standard and compatible connection that would work with any external calculation program. This would only have to be done once using the open source tools to have it work with any external calc engine. Each independent programmer would not have to learn and build custom apps for each CAD program. This is the Holy Grail of programming - defining separately developed and tested objects that work together automatically. OpenCalc/TSPA will provide open source programming templates for calc engines that include all the necessary XML I/O code. Programmers only need to write their calculation routines. They do not need to do any I/O programming or know anything about how the tag format works.
The problem in the general CAD world is the need for customization of its base program to each industry and application. This creates costly tools that are incompatible with tools and calculations in competing CAD programs. Even though the geometry might be based on the same NURB surface definition, the tools will not work together in any common or stand-alone environment. The tools are all tied to a proprietary interactive format and process, not an open one that allows combined and automatic solutions.
In a paper by the author (Hollister 2016), three separate and standard calc engines: hull variation, hydrostatics, and Holtrop resistance, were combined to search for an optimum hull shape without writing custom code and without having the source code for any of the calculations. With a compatible CAD program, one could launch this search sequence using the current “parent” hull shape being designed in CAD, ultimately replacing it with the optimum shape determined by the external calculations.
The keys to this process are the separate and open definition of external calc engines and a common XML data file definition of industry geometry and variables. This cannot be achieved with the current proprietary independent "app" model of programming.
Hollister The SNAME OpenCalc System 11
FUTUREOpenCalc has split into two parts: the base TSPA open source system that can be applied to all areas of programming, and OpenCalc, a layer built on top of TSPA that provides for the specific needs of the maritime industry. The job for SNAME includes identifying and converting legacy calculations into modern and reusable calc engines and defining data variable and structure XML standards for conceptual and preliminary design and analysis for data such as hulls, compartments, tanks, and operating conditions. Each industry needs to take control over their data. Programming and analysis cannot rely on separate and incompatible program file definitions that require two-step neutral file conversions.
Separating calculations from inside of a traditional application to a separate calc engine batch format is not difficult, but industry organizations need to push to achieve a critical mass of support. Independent software developers should no longer completely define their own data and proprietary file formats. Industries can use the common data definition and I/O tools of TSPA to create their own data definitions and schemas. These can be added to by software developers using custom tags, but industries won’t be fully successful without some level of influence or guidance. This doesn’t mean the elimination of successful legacy formats because they can be used as alternate definitions with conversion routines that work on-the-fly. The hydrostatics calc engine included with OpenCalc can read several different legacy hull station formats, like GHS and SHCP. However, all other variables and structures are defined with XML tags.
The goal is to have SNAME play a lead role in introducing and pioneering this needed and valuable approach to ship design software development. With contributions from all and buy in from developers the software resources developed will be highly useful for hobbyists, students, professionals, and software developers alike. There is no other path ahead for programming. The world of separate interactive apps tied to calculations and proprietary data formats has severe limits, and engineers and naval architects need creative tools that can solve new problems without becoming expert programmers or programmers at all.
All programming templates are currently written in C/C++, but the next implementations will be in Java and Java Script. The goal is to provide open source XML I/O code in many languages and UIF connection code written for many different spreadsheets.
With SNAME’s help, work will begin on education, converting legacy code, and building libraries of historical hulls using new XML definitions for hulls and compartments.
MARKET CONSIDERATIONSOpen source software does not mean that everything is free. The UIF components may be based on spreadsheets like Excel,
which is not free, but the goal is to make the macro code in Visual Basic for Applications (VBA) or some other spreadsheet macro language free. The XML data definition components and I/O code will be free and open source. Unlike ISO, the TSPA definitions will be free and include free source code in many different languages.
TSPA is also not free in the sense that some things built on top may not be free. This open source opportunity encourages others to use the programming structure, including those who create their own proprietary calc engines to use or sell. The OpenCalc resource will make it easier to create and maintain software developments by all and will be less expensive and offer a much larger market of users at all levels.
This external calc engine structure is different from custom VBA programming in Excel, where it’s difficult to hide proprietary VBA source code. With separate CEs in EXE form, developers can hide and protect their code. However, the hope is that many designers will offer their tools to others for free. For the case of an optimization combination of CEs, designers might create their own proprietary hull generator CE, add to it the free SNAME hydrostatics/stability CE, add in a hull meshing CE, and finish off with a complex CFD CE that is offered at another web site for a subscription service. It is also a goal to get an OpenFoam version that works as a stand-alone CE in batch form.
CONCLUSION"There continues to exist competition between the proponents of self-contained, relatively closed turnkey systems of increasing functional scope and the advocates of open systems and a modular, in part heterogeneous approach to system growth. This argument will probably go through many more cycles, but by and large, the long-range trend has been in favor of more and more open systems."..."Such openness tends to lie in the interest of not only the user communities, but in the long run also of the system vendors."
(Nowacki 2010)
As a student of Professor Horst Nowacki in the 1970’s, and after 45 years of frustrations from dealing with changing technology in a small, technically complex market, the author developed TSPA and OpenCalc as the solution. After losing programs when the world changed from mainframe computing to PCs (programs were put on 9-track tape and forgotten), and then after losing a huge amount of programming development when DOS evolved to Windows (many still keep old XP computers for those programs), and now viewing more code being lost in the change to cloud computing, a new programming model was created to isolate and provide separate tools for computer scientists, subject experts, users, and industries. This change is made more viable with the open source movement and Web standards like HTML and XML.
Hollister The SNAME OpenCalc System 12
There is an historical mindset inertia that needs to be moved now that technology offers massive amounts of speed, memory, storage and web connections. In the program development world, the traditional interactive app developed by separate and competing software development companies that concentrate computer expertise and data control has not evolved. This is most noticeable in smaller and more complex markets that do not justify expensive app development and where calculations/data processing require more subject expert creation and control.
Excitement over popular web apps hides historical and smaller market issues. The fundamental flaw is that the traditional programming model has not evolved into what it could be. The interactive and manual user-controlled app model of programming where one user interface is tied to one set of calculations or processing might have been a great solution for the beginnings of the PC era, but that's no longer the case. There is a limit to what independent software company development and interactive apps can commercially justify and offer. Some of this has to do with the limitations and complexities of small markets, and some of it has to do with the needs of users to combine and automate complex tasks without user interaction. Apps require too much code that must be created by programming specialists working with subject experts and tied into interactive and fixed use tools with limited customization. Development costs limit the creation of tools for smaller markets and changing technology causes many valuable tools to be lost.
New program organizations are possible with the web connecting fast computers, their programs, and their data files. However, what really must change is the idea that subject experts need to be simple users of other people's software even if the software does offer some level of customization. Computer use will never reach its potential until complex tools for creating completely new solutions are given to subject experts without having to become expert programmers. That appears to be an unfortunate new meme - to have everyone become coders (see Hour of Code), not better subject experts who can define new solutions given separate and flexible computer object tools.
TSPA working examples have shown how this is possible and how it offers ways to build new automated solutions that can’t be built any other way unless one has all the source code and a very large budget. Calculations are separated from interactive user interfaces and common data is defined with open definitions and code. Everyone can add their own compatible piece to large web-based solutions. Computing will no longer be controlled by all-in-one apps, proprietary data formats, and suites defined and dominated by independent software developers. Industries will assert control over their data and subject experts will have flexible tools to create completely new solutions without programming.
TSPA is being developed as an open framework with many tested and open source tools. Some might create calc engines that are not open or free, but the XML layered data schemas and I/O code will be free along with many UIF frameworks. After more than 50 years of rapid computer developments, changes, lost programs, and ignoring the needs of small markets, it’s time to define an open, flexible, independent, and external object-oriented approach that is web-based, cross-platform, and cross-industry. Subject matter experts are not just users or customers of an app development process defined and controlled by computer scientists and independent software developers. They should not have to wait for the next software revision in the hope that what they want will be included. For small markets, that may never happen. Subject matter experts can and should be creators of completely new computer solutions without becoming programming experts. TSPA/OpenCalc gives them that direct, flexible, and creative control over their subject domain.
REFERENCES
Hollister, Stephen, “Automatic Hull Variation and Optimization”, Presented at the February 28, 1996 meeting of the New England Section of The Society of Naval Architects and Marine Engineers
Hollister, Stephen, “A New Paradigm for Ship Design Calculations”, Presented at the Nov. 2015 SNAME Maritime Convention (WMTC) conference http://www.sname.org/project114/home
Hollister, Stephen, “Automatic Hull Variation and Optimization using SNAME’s OpenCalc System”, Presented at the Nov. 2016 SNAME Maritime Convention, Seattle, WA
Panko, Raymond R., "What We Know About Spreadsheet Errors." Journal of End User Computing's Special issue on Scaling Up End User Development, Volume 10, No 2. Spring 1998, pp. 15-21 Revised May 2008.
Stephen M. Hollister, an engineer and software developer living in Jamestown, Rhode Island, has BSE and MSE degrees in Naval Architecture and Marine Engineering along with an MSE degree in Computer, Information and Control Engineering, all from the University of Michigan. He has written over a million lines of code over 45 years as a combination naval architecture subject expert and a computer science subject expert. Software developed includes a full relational trimmed NURB surface CAD program and a suite of programs for intact and damaged ship hydrostatics and stability. He has also developed many other calculation programs for hull resistance, propeller analysis, sailboat velocity prediction, and stress-based plate development, along with many vessels and products that have been CNC cut and built. Technology used has covered the historical range from mainframes, cards, Calcomp plotters and Tektronix graphics terminals, to writing PC graphics drivers in assembler and full applications for DOS, to writing a full 3D graphics package, and then converting everything over to Windows and MFC. Mr. Hollister has written programs in FORTRAN, Snowball, assembly languages of all sorts, Turbo Pascal, Algol, VBA, C and C++.
In the 1980s, Mr. Hollister taught college Math and Computer Science full time for six years, and since 1985, when he started New Wave Systems, Inc. (www.newavesys.com and www.pilot3d.com), has specialized in NURB surfaces, ship design, and general CAD/CAM/CAE software development. See also http://www.pilot3d.com/NurbSecrets.htm. Mr. Hollister obtained a PE license and is a 40 plus year member of SNAME, IEEE, and ACM.
