BASIS IntroductionBuild system And Software Implementation Standard

by Andreas Schuh

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 2

Outline

Introduction

Project Template

CMake Modules

Utilities

Milestones

Tutorials9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 3

IntroductionWhy BASIS? What is it?

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 4

We want…

the lifetime of your software to be longer than the research time spent on the development of the algorithm it implements.

Therefore, we want to establish a unified software structure. to ease integration of packages. to reduce maintenance burden. to make our software more robust against

failure. to increase the impact factor of your work!

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 5

Objective

Specify standards/conventions for the software development process.

Standardize project organization and software build including testing and packaging.

Provide basic utility functions in order to promote compliance with lab conventions and to reduce common development burden.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 6

Vision

9/30/2011

BASIS

…

Toolkit

ODVBA

COMPAREPREDICT

GLISTR

MICO

Integrated Distribution Package

Projects•COMPARE•PREDICT•…

Toolkit

BASIS

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 7

Ah, another Toolkit…

BASIS is not a toolkit!

It focuses on more basic aspects.

It consists of specifications in plain text or on Wiki.

It accompanies this standard with a preset project template and basic utility functions for supported programming languages.

However, distinction may be fuzzy at times…9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 8

BASIS Overview

9/30/2011

Standard Specifications

Template

File

syst

em

Hie

rarc

hy

Tem

pla

te F

iles

CMake Modules

Basi

sPro

ject

.cm

ake

(CM

ake

)

Basi

sTest

.cm

ake

(CTe

st)

Basi

sPack

.cm

ake

(CPa

ck)

ToolsPro

ject

Cre

ati

on

Auto

mate

d T

est

ing

Docu

men

tati

on

Utilities

Com

mand E

xecu

tion

Com

mand-l

ine

Pars

ing

Test

ing F

ram

ew

ork

s

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 9

Project TemplateHow does it look like?

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 10

<prefix> is the installation prefix, e.g., /usr/local/ on Unix.

<sinfix> is a project- and possibly version-specific string, e.g., <project>.

<prefix>/

bin/ <sinfix>/

include/ sbia/ <project>/

lib/

cmake/ <project>/

<sinfix>/

perl5/ 5.x.x/ SBIA/

pythonx.x/ site-packages/ sbia/

share/ <sinfix>/

doc/

example/

…

<source> is the root directory of the source tree.

<project> is the project name in lowercase only.

Filesystem Hierarchy

<source>/ config/

data/

doc/example

/include/ sbia/ <project

>/src/

test/

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 11

Filesystem Hierarchy

Note: We distinguish between main executables in <prefix>/bin/<sinfix>/ and auxiliary executables located in <prefix>/lib/<sinfix>/.

By default, symbolic links are created upon installation in the <prefix>/bin/ directory which link to (selected) executables in <prefix>/bin/<sinfix>/.

More details in the specification.9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 12

Template Files

Common project files which are part of the project template and copied upon project creation.

For an example, see root CMakeLists.txt template.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 13

CMake ModulesWhat are they good for?

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 14

Implement functions and macros which extend and replace most of the standard CMake commands.

Know not only C++, but also other languages:Java (not yet), Python, Perl, BASH, and MATLAB.

Implement filesystem hierarchy standard,and adhere to guidelines regarding software documentation, packaging, and testing.

basis_add_executable()

CMake Modules

9/30/2011

add_executable()

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 15

CMake Modules

The main modules are: BasisProject.cmake▪ To be included by every BASIS project. Initializes

BASIS and includes definition of basis_* functions and macros.

BasisTest.cmake – Extends CTest.cmake.▪ Used for software tests.▪ Included by basis_project_initialize().

BasisPack.cmake – Extends CPack.cmake.▪ Used for packaging.▪ Included by basis_project_finalize().

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 16

CMake Modules

The most important functions to know are:

basis_find_package() basis_add_executable() basis_add_library() basis_add_test() basis_add_doc() basis_include_directories() basis_target_link_libraries() basis_set_target_properties() basis_get_target_property() basis_install()

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 17

CMake Example

<source>/config/Depends.cmake# find NIfTI-1 tools for MATLABbasis_find_package (MatlabNiftiTools REQUIRED)

basis_add_include_directories (“${MatlabNiftiTools_INCLUDE_DIRS}”)

<source>/src/CMakeLists.txt# MEX-file build from C++basis_add_library (look_for_gold_instead MEX look_for_gold.cxx)

# executable build from MATLAB scriptsbasis_add_executable (miner neurominer.m)basis_target_link_libraries (miner look_for_gold_instead)

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 18

CMake Modules

See the API documentation of BASIS for details on the CMake variables, functions, and macros.

The documentation is generated automatically every night (less frequently in later stages) from thein-source code comments using Doxygen.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 19

ToolsWhat tools do we get?

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 20

basisproject

Used to create a new BASIS project or to modify a previously created project.

Can upgrade a previously created BASIS project to a newer version of BASIS.

Uses 3-way file merge to merge your changes with the changes in the newer BASIS template.

Therefore, stores meta-data in .basis/ directories in project source tree.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 21

basistest

Wrapper for command-line tools implemented for the automated execution of software tests:

The basistest_cron is executed, e.g., every 5 minutes. It in turn just calls basistest_master with the

arguments specific to the given system. A configuration file tells the basistest_master what

branches of which projects to test at which time interval,…

The actual tests are executed by instances of basistest_slave which are submitted to SGE in our case and am using CTest.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 22

basistest

See CDash dashboard of BASIS for examples of test result submissions of both Continuous▪ Executed after a change was committed to the

repository.

and Nightly▪ Run once every certain time interval, e.g., every

night.

tests.9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 23

Documentation

Natively, Doxygen knows how to parse C/C++, Java, and Python (not satisfactory, though)

Filters can be used to translate other languages into something Doxygen understands best: C++.

BASIS includes such filters for the missing languages, i.e.(, Python), Perl, BASH, and MATLAB.

Application of filters hidden to the developer. A simple basis_add_doc(api GENERATOR Doxygen) in the build configuration is sufficient.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 24

Documentation

See the API documentation of the BASIS utilities for examples of documentation generated fromin-source comments only.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 25

UtilitiesWhat implementations does it provide?

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 26

Command Execution

Executing several utility commands from within one or more main commands is common practice on Linux and in particular in research environment.

Several factors affect which commands are actually executed and which shared libraries are loaded.

How can we make sure that the right commands are executed with the correct libraries loaded?

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 27

Command Execution

BASIS provides utility functions for each supported language to do just that in a safe way.

Further, these functions satisfy the Calling Conventions, where executables are referenced by build target name rather then filename.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 28

Command-line Parsing

Any command-line tool needs to parse the given command-line arguments.

Each main executable is required by BASIS to output a help screen on --help (-h), a shorter usage information on --helpshort, and its version and copyright on --version.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 29

Command-line Parsing

Therefore, BASIS provides command-line parsing libraries for each supported language.

These libraries create a help screen from brief descriptions provided for each option. No need to format the help screen manually. Possibility to output a man page or XML

description. Such help is more likely to be up-to-date rather

then being forgotten when an option was renamed, removed, or added…

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 30

Command-line Parsing

C++ TCLAP gflags

Java args4j

Python argh (wrapper for argparse)

Perl Getopt::Tabular Getopt::Declare

BASH shflags (extended shFlags)

MATLAB none available yet

9/30/2011

See Command-line Parsing.txt document in BASIS repository for more details.

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 31

Testing Frameworks

C++ gtest gmock libtap++

Java JUnit JTap

Python unittest PyTap

Perl Test::Simple Test::More Test::MockObject

BASH shtap (slightly modified tap-functions library)

MATLAB MATLAB xUnit Test Framework

9/30/2011

Already part of Perl installation.

Test Anything Protocol (TAP) ProducersCommon use and standardized output which can be processed, for example,by Test::Harness written in Perl.Already part of Python

installation.

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 32

MilestonesWhat’s the plan?

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 33

Beta Release 0.1

Release date: Monday, 10/3/2011

Stable features: Languages: C++, Python, Perl, BASH, MATLAB Project Template. CMake Modules. Tools basisproject and basistest. Build of executables from MATLAB scripts. Build of MEX-files from C++. Command Execution using build target name.

See Wiki for further release details.9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 34

Beta Release 0.1

Yes, it is beta. Thus,

for ongoing projects, it is up to you whether to migrate to BASIS or postpone this for until prior to publication.

▪ Plan ahead! This might take some efforts.

9/30/2011

for new projects, it is strongly recommended to use the beta release of BASIS already.

▪ This makes you familiar with it and reduces the burden of adopting it once version 1.0 is released.

▪ Your feedback and experience in using BASIS is important to shape the stable release version.

▪ Before the stable release, changes can be easily made and decisions can be reconsidered.

▪ After such release, it’s not!

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 35

Release 1.0

Release date: Monday, 11/7/2011

Stable features: Command-line parsing. Build of shared libraries from MATLAB

scripts. Tutorial-like How-To Guides.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 36

Release 1.1

Release date: Monday, 12/12/2011

New stable features: Testing frameworks. Improved Doxygen filters for Python and

Perl.

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 37

Release 1.2

Release date: Monday, 2/27/2012

New stable features: Packaging, including creation of source

packages that bundle also external dependencies and considerations regarding decision that were be made by then on how to distribute our software and make it noticed.▪ licensing permitting

Coding Guidelines More How-To Guides

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 38

TutorialsPlease help me!

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 39

Tutorials

Short tutorial-like meetings to go more into details regarding certain aspects with hands-on examples over the next few months.

Invitations will be send in advance via SBIA mailing list including links to tutorial materials.

Remember, your participation and feedback is required to form the first release version of BASIS!

What do you like? What concerns you? What hinders you? What is missing?

9/30/2011

BASIS IntroductionCopyright (c) 2011 University of Pennsylvania. All rights reserved. 40

Thank you

9/30/2011

Let’s talk about it!