Www.nitrc.org nitrc.nih.gov F-T4: Best Practices in Software Development MICCAI 2010 - September 24,...

transcript

www.nitrc.orgnitrc.nih.gov

F-T4: Best Practices in Software DevelopmentMICCAI 2010 - September 24, 2010

Best Practices in Software Development:Users and Developers Unite!

Bennett Landman, Ph.D.

David N. Kennedy, Ph.D.

Xiaodong Tao, Ph.D.

Paul A. Yushkevich, Ph.D.

Gary Hui Zhang, Ph.D.

Schedule13:30 - 15:00 Session 1: Presentations

13:30 - 13:40 Introduction to the Tutorial13:40 - 14:00 Neuroimaging Informatics Tools and Resources Clearinghouse: NITRC

David N. Kennedy14:00 - 14:20 3DSlicer3 Extension Modules: Concept and Implementation

Xiaodong Tao14:20 - 14:40 Challenges of Collaborative Open-Source Development: Lessons learned from the

early evolution of the Java Image Science ToolkitBennett Landman

14:40 - 15:00 Making others use your software: Lessons learned from ITK-SNAPPaul A. Yushkevich

15:00 - 15:30 Tea/Coffee Break

15:30 - 17:00 Session 2: Panel Discussion & Demonstrations15:30 - 16:15 Panel Discussion16:15 - 17:00 Demos

Academic Objectives

All participants should gain knowledge regarding:• Methods for open source software development• Methods for software distribution, including licensing

issues• Techniques for software validation• Techniques for user interactions (documentation,

tutorials, feedback/communication, etc.)• Resources for finding existing relevant software

projects.• See http://www.nitrc.org/projects/best_practices

FacultyBennett Landman, Ph.D.Assistant ProfessorDepartment of Electrical EngineeringVanderbilt University Institute of Image Sciencebennett.landman@vanderbilt.edu

David N. Kennedy, Ph.D.Professor, Department of PsychiatryUniversity of Massachusetts Medical SchoolWorcester, MA 01605David.Kennedy@umassmed.edu

Xiaodong Tao, Ph.D.Computer ScientistGE Global Research CenterImaging Technologiestaox@research.ge.com

Gary Hui Zhang, Ph.D.PICSL, Department of RadiologyUniversity of Pennsylvania School of Medicine3600 Market Street, Suite 320Philadelphia, PA 19104huiz@seas.upenn.edu

Paul A. Yushkevich, Ph.D.Assistant ProfessorPenn Image Computing and Science LaboratoryDepartment of RadiologyUniversity of Pennsylvaniapauly2@mail.med.upenn.edu

Neuroimaging Informatics Tools and Resources Clearinghouse - NITRC

Software Tools - A Users Perspective

David N. Kennedy, Ph.D.

Director, Division of Neuroinformatics

Department of Psychiatry, UMass Medical School

Child and Adolescent NeuroDevelopment Initiative

With apologies and credit to Gary Larson - The Far Side

users developers

How do I change the contrast? How do I add layers? How do I invert the transform?

blah blah blah blah blah blah RTFM blah blah

usersdevelopers

Well, this is really a very complex piece of code. Have you read my paper from MICCAI? It’s probably too complex for you…

• Data Acquisition & Analysis - Normal Development and Studies in Child Psychiatry (Autism, Bipolar Disorder,

– Structure

– Diffusion

– Resting State

User Scope:

Finding the Right Tool

“MRI segmentation software brain”

• Google?– About 157,000 results– Top Hit: Rescue

• Source Forge?

• NITRC?

Google Top Hit

Finding the Right Tool“MRI segmentation software brain”• Google?

– About 157,000 results– Top Hit: Rescue– Results include: papers, pdfs, ppts, programs,

pictures, videos, etc.

• Source Forge?• NITRC?

– About 157,000 results

• Source Forge?– 15,539 results– Top Hit: PortableApps.com: Portable Software/USB – Results include: you name it…

• NITRC?

Source Forge Results

– About 157,000 results

• Source Forge?– 15,539 results

• NITRC?– 39 Software Packages– Top Hit: ITK-SNAP– Results include: All the ‘usual suspect’ tools, and

NITRC Result

Refining the Search

• Operating System: MacOS

• Supported Image Type: DICOM

• Results in: 16 Resources

NITRC ResultRefined

Best Practices Take-Home Points:

1. Findability Description/Promotion

User Requirements Developer Requirements

Installation

• Findability is just first step. Can I install it, easily?

• Examples:– FSL– Mango

FSL Installation Steps1. Click download link

2. Agree to license

3. Provide some personal information

4. Download appropriate release

5. Run FSL installer script (from command line)

6. Test the installation

Mango Installation Steps

1. Click download link

2. Open the download w/ ‘Archive Utility’

3. Run Installer (Double click from UI) - Answer usual questions

4. Run Program (Double click from ‘Applications’ directory)

2. Easy to Install Easy to Install/Verify

Ease of Use• Now what? You’ve found and installed

your program, how do you use it?

• From the UI, is it clear what to do?

User Interface

Documentation• Of course the software is very powerful, and

it’s impossible for everything to be intuitively obvious from the UI, who do you think we are, Microsoft?

• Users are obligated to RTM (read the manual) prior to bugging the developers with questions.

• But, where is the manual? Is it easy to use? Are there examples? Tutorials? FAQ?

FSL Documentation

3. Easy to Use Intuitive UI/Good Documentation

• OK, I REALLY did read the manual, and I just can’t figure out how to [insert problem here]…

• I checked the FAQ, and searched on the mail list archive…

• Oh, wait a minute, is there a FAQ? Is there a mail list? Is there a users ‘forum’? Where are they?

• OK, I checked those too, how do I get more help?– User Support Forums, Mailing Lists– Email an administrator?– What’s the preferred method for this project?

3. Easy to Use Intuitive/Good Documentation

4. Get Help Bug Tracker/Forums/Mailing List

Latest Version?• Once you succeed at contacting the

developers, how often does the following happen to you?– Oh, you can’t do [blah]? ‘That’s been added in

the ‘most recent’ version’ Or worse yet, ‘that’s only available in the development version’. Or, ‘that’s planned for release 3.X’.

• Users: Know your versions!• Developers: Keep your public releases up

to date! (yes, ALL of them…)

5. Up-to-Date Versioning/Synchronization/Dashboard

I want the best!• Of course, I only want to use the best software know to

human kind for my research.• Unfortunately, ‘best’ is exceedingly domain dependent, and

many problems (and software packages) cross many domains in a specific problem (skull striping, registration, pre-filtering, statistical test, etc.).

• Also, ‘best’ is data dependent. The ‘best’ surface extraction in post mortem tissue may not be the best in patients w/ AD.

• Users: Know what best means in your context!• Developers: Provide quality metrics (on standard cases)

when possible!

6. High Quality Validation/Comparison Metrics

Say I’m not the only one…• Even if using the ‘best’ tool, the user will run into

publications problems if no one else has ever used that tool before.

• Have you gotten reviewers who say: “You should have used the Talairach coordinate system”, or “You should have used SPM…”?

• Even if it’s not fair or optimal, users need to know how popular a resource is. # of downloads # of users # of publications, etc.

7. Use Metrics Downloads/Users/Publication Counts

I can make it better…• I’ve found a great tool, it works really well for me,

and does everything I need… almost.• I submitted a ‘Feature Request’, and I completely

understand that your funding and time is limited, and you will not get to my request until 2021.

• But hey, I’m a wicked good [insert programming language here] programmer myself. Let me at your source code…

• Open Source is not the solution for everything, but with proper management and policy, it’s an important tool for extensibility, transparency, and public relations. Besides, it ought to be the law…

8. Contribute/Extend Source Code Availability/Policy

Summary• Top 10 reasons users use a specific

software package:

10. It’s what my lab uses…

9. Has the most publications…

8. Is easiest to use…

7. Is part of the LONI pipeline…

6. I wrote it… when I was a graduate student… 20 years ago…

Summary• Top 10 reasons users use a specific

software package:

5. My graduate student wrote it… who is a botany major… with no formal programming training…

4. It works with 3D Slicer…

3. It’s written in ITK…

2. Everyone uses SPM…

1. At least it’s not SPM…

Summary, continued• Top 1 reason NOT listed as a reason to use

a specific software package:

1. It is the best…

This is where developers and users have to continue to work together to:

• Insist on the best

• Define the parameters of best

• Develop standards for performance

• Document the performance

• And use the guidelines of Best Practices for software development!

Acknowledgements

• All NITRC Users and Developers• Today’s Presenters & Audience• MICCAI Organizers• NITRC Development Team

(Neuromorphometrics Inc, TCG Inc, UCSD)• NITRC Sponsors (NIH Blueprint, NIBIB)• The ‘tutorial’ lives on at

http://www.nitrc.org/projects/best_practices

8. Contribute/Extend Source Code Availability/Policy

Www.nitrc.org nitrc.nih.gov F-T4: Best Practices in Software Development MICCAI 2010 - September 24,...

Documents