Acknowledgements ª Portions of this material are supported by or based upon work supported by the
Defense Advanced Research Projects Agency (DARPA) under its Agreement No. HR0011-07-9-0002, the United States Department of Energy under Contract No. DE-FG02-06ER25752, the Blue Waters sustained petascale computing project, which is supported by the National Science Foundation under award number OCI 07-25070, and the SI2-SSI Productive and Accessible Development Workbench for HPC Applications, which is supported by the National Science Foundation under award number OCI 1047956
ª The SI2-SSI team is lead by Jay Alameda (NCSA), Greg Watson (IBM), Steven Brandt (LSU), and Allen Malony (U Oregon). Team members and senior personnel include Beth Tibbitts (IBM), Ralph Johnson (U Illinois), Chris Navarro (NCSA), Sameer Shende (U Oregon), Wyatt Spear (U Oregon), Brian Jewett (U Illinois), Galen Arnold (NCSA), and Rui Liu (NCSA)
1
2
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
3
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
4
What is Eclipse?
ª A vendor-neutral open-source workbench for multi-language development
ª A extensible platform for tool integration ª Plug-in based framework to create, integrate
and utilize software tools
5
Launching & Monitoring
Eclipse Parallel Tools Platform (PTP)
Debugging
Coding & Analysis
Performance Tuning
6
Parallel Tools Platform (PTP)
ª The Parallel Tools Platform aims to provide a highly integrated environment specifically designed for parallel application development
ª Features include: ª An integrated development environment (IDE) that
supports a wide range of parallel architectures and runtime systems
ª A scalable parallel debugger ª Parallel programming tools
(MPI, OpenMP, UPC, etc.) ª Support for the integration
of parallel tools ª An environment that simplifies the
end-user interaction with parallel systems ª http://www.eclipse.org/ptp
7
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
Why WHPC?
ª Stable, portable platform for tool development ª Focus on tool functionality, manage rapid evolution
of HPC platforms ª Encourage consistent tool look and feel ª Support for HPC application development practices
ª Edit, build, test, debug, maintain, for maximum developer productivity
ª Remote development, batch execution mandatory ª Track, store, search, browse code artifact provenance ª Share tool functionality through an integration framework ª Maintain tool identity
ª Provides for independent tool development pathways and funding
8
Why Parallel Tools Platform?
ª High potential to meet needs of a WHPC. ª Target next generation of HPC developers
growing up with IDEs (Eclipse, Visual Studio, ...)
ª For PTP to become a WHPC need to: ª Cultivate community of users ª Make substantial improvements to PTP around two
themes: ª Improving usability ª Improving productivity
9
10
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
Requirements and Impact
ª Application-centric approach ª Use real application codes, with PTP, on production
computational resources ª Identify specific goals to accomplish with each
application ª Use Eclipse PTP to accomplish the goals ª Identify shortcomings in Eclipse PTP that need to
be rectified for Eclipse PTP to be effective with that application workplan
ª This is part of our project team’s responsibility ª Work with application community and learn from their
experience with Eclipse PTP
11
Requirements and Impact (2)
ª Application-centric approach ª Work with application community and learn from their
experience with Eclipse PTP ª Bridge to TeraGrid and (now) XSEDE Advanced
User Support ª Work with targeted organizations to assist with
adoption of PTP ª Monthly user calls ª Annual user group meeting ª Hands on tutorials ª Conference Birds of a Feather
12
13
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
Improvements
ª Work within Eclipse release cycle ª Major (API-breaking) improvements with coordinated
June release ª Last major release Eclipse 4.2 “Juno” released June 27,
2012 ª Minor enhancements and bug-fixes with two
coordinated service releases in September and February ª Eclipse 4.2 SR1 released Sept 28, 2012
ª Foci of improvements ª Improve usability ª Improve productivity
14
Improve Usability
ª Remote support and scalability enhancements ª Broaden support of remote capabilities to full PTP ª Provide for easy platform configuration management ª Provide additional remote features
ª Automatic remote service deployment ª Multiple authentication mechanism ª Support wide range of resource managers ª Full remote debug support
15
Improve Usability
ª Integration with other tools ª Improve External Tools Framework (ETFw)
ª Full remote support ª Integration of tool output with Eclipse views
ª Improve and broaden parallel paradigm support ª Driven by user needs and feedback
16
Improve Productivity
ª Provide support for performance driven refactoring
ª Track source and executable code provenance
17
Significant Recent Improvements
ª User-configurable machine configuration ª Wide variety of configurations now available: ª Documentation, tutorial at
ª http://wiki.eclipse.org/PTP/designs/Resource_Manager_Configuration
18
Scalable System Monitoring ª System view
ª Jobs running on system
ª Active jobs
ª Inactive jobs
ª Messages
ª Console
Run-19 Running an Application
Sync-20
Synchronized Projects ª Projects types can be:
-20
File Service Index Service
Launch Service
Build Service
Debug Service
Local source code
Source code copy
Local Remote
Run
Debug
Compute
Edit Search/Index NavigaAon
Synchronize
Executable
Synchronized Projects
Blue Waters Enhancements
• Blue Waters: Cray XE6/XK7 at NCSA • PTP did not work with Crays “out of the box”
• Could not submit jobs with appropriate aprun options • Could not monitor status of compute nodes • Could not set environment modules for build • Did not recognize Cray, PGI compilers’ errors messages • Did not support OpenACC • …
• Less than 6 months to fix these for PTP 6.0 (!)
21 The Eclipse Parallel Tools Platform
22 The Eclipse Parallel Tools Platform
Integrated OpenACC documentation and PLDT support (added for BW)
Documentation also available for MPI, OpenMP
23 The Eclipse Parallel Tools Platform 23 The Eclipse Parallel Tools Platform Code completion for OpenACC directives (added for BW)
24 The Eclipse Parallel Tools Platform
After the build, compiler errors, warnings, and loopmark information are shown in the Problems view and source code editor
(Cray, PGI support added for BW)
25 The Eclipse Parallel Tools Platform Graphical interface for launching a job (customized for BW)
Additional Plug-ins from NCSA
ª NCSA publishes additional plug-ins can be added onto an existing PTP installation ª http://forecaster.ncsa.uiuc.edu/help/index.jsp
ª Contribute a System menu to the menu bar with XSEDE- and NCSA-specific commands
NCSA-26 Advanced Features: NCSA/XSEDE
System Menu
ª Open Web content in Eclipse: ª Open XSEDE User Portal
ª Open User Guide for a machine
ª Open an SSH terminal (as an Eclipse view)
NCSA-27 Advanced Features: NCSA/XSEDE
Eclipse-integrated SSH terminals are provided by the Remote System Explorer (RSE), one of the features that is included in the Eclipse for Parallel Application Developers package.
System Menu
ª Shortcuts for common PTP tasks: ª Add Remote Environment adds a
Remote Tools connection for a particular machine
ª Add System Monitor opens the System Monitoring perspective and begins monitoring a particular machine
NCSA-28 Advanced Features: NCSA/XSEDE
System Menu
ª The plug-in is preconfigured with information about XSEDE and NCSA resources
ª The bottom four commands generally prompt for a system
ª Select System can be used to eliminate this prompt, so these commands always act on a particular system
NCSA-29 Advanced Features: NCSA/XSEDE
MyProxy Logon
ª MyProxy Logon allows you to authenticate with a MyProxy server ª Often myproxy.teragrid.org
ª It stores a “credential,” which is usually valid for 12 hours
ª During these 12 hours, SSH connections to XSEDE resources will not require a password; they can use the stored credential ª However, you must enter the
correct username for that machine!
NCSA-30 Advanced Features: NCSA/XSEDE
Improvements in the works…
ª External Tools Framework (ETFW) being transitioned to use the resource manager “JAXB” XML descriptions. ª Enhanced flexibility for tool integration
ª Environment Management enhancements ª Ordering of modules (important for some machines) ª Evaluate proper role and relationship between
environment management, scanner-discovery and tool chains ª Would like to properly set remote include paths
(semi)-automatically! ª Separate Build system from CDT build
ª Cleaner multi-system build management ª Builds in any language
Improvements in the works…
ª External Tools Framework (ETFW) being transitioned to use the resource manager “JAXB” XML descriptions. ª Enhanced flexibility for tool integration
ª Environment Management enhancements ª Ordering of modules (important for some machines) ª Evaluate proper role and relationship between
environment management, scanner-discovery and tool chains ª Would like to properly set remote include paths
(semi)-automatically! ª Separate Build system from CDT build
ª Cleaner multi-system build management ª Builds in any language
Increasing ambitiousness!
33
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
Software Engineering
ª Code Visibility
34
Software Engineering
ª Code Visibility
35
Code navigation
Software Engineering
ª Code Visibility
36
Code navigation
Syntax-aware editing (navigate to program units and declarations)
Software Engineering
ª Code Visibility
37
Code navigation
Syntax-aware editing (navigate to program units and declarations)
Code Outline
Software Engineering
ª Code visibility: deducing call hierarchy
38
Would like to understand call hierachy of this code in relation to “main()” in startup.c
Software Engineering: Call Hierarchy (C/C++)
ª After selecting main, right click and select <Open Call Hierarchy>
39
40
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
Multi-machine build management
ª Local ª Source is located on local machine, builds happen
locally ª Synchronized
ª Source is local, then synchronized with remote machine(s)
ª Building and launching happens remotely (can also happen locally)
ª Remote ª Source is located on remote machine(s), build and
launch takes place on remote machine(s)
41
Synchronized Projects ª Projects types can be:
42
File Service Index Service
Launch Service
Build Service
Debug Service
Local source code
Source code copy
Local Remote
Run
Debug
Compute
Edit Search/Index NavigaAon
Synchronize
Executable
43
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
Performance Tuning: PTP TAU plug-ins http://www.cs.uoregon.edu/research/tau
ª TAU (Tuning and Analysis Utilities) ª First implementation of External Tools Framework (ETFw) ª Eclipse plug-ins wrap TAU functions, make them available
from Eclipse ª Full GUI support for the TAU command line interface ª Performance analysis integrated with development
environment
44
45
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
Source Code Control: “Team” Features
ª Eclipse supports integration with multiple version control systems (VCS) ª CVS, SVN, Git, and others ª Collectively known as “Team” services
ª Many features are common across VCS ª Compare/merge ª History ª Check-in/check-out
ª Some differences ª Version numbers ª Branching
46
CVS Features
ª Shows version numbers next to each resource
ª Marks resources that have changed ª Can also change color (preference
option) ª Context menu for Team
operations ª Compare to latest, another
branch, or history ª Synchronize whole project (or any
selected resources) 47
48
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
Issue Tracking
ª Mylyn Bridge ª Tracks tasks, links to source and bug repositories
49
Connections to Jira, bugzilla, …
50
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
Eclipse Documentation
ª Eclipse Help System – built in and standalone (http://help.eclipse.org)
51
Adapting Eclipse Documentation to Other Projects: QMCPack
ª See http://code.google.com/p/qmcpack-doc/
52
53
Outline ª Overview of Eclipse and Eclipse Parallel Tools Platform
(PTP) ª Overview of WHPC: NSF-funded SI2-SSI project to
produce a productive and accessible development workbench using Eclipse PTP ª Determining Requirements, Ensuring Impact ª Improvements to Eclipse PTP
ª Software Engineering Practices Enabled by Eclipse PTP ª Code visibility ª Multi-system build management ª Performance tuning ª Source code control ª Issue Tracking ª Documentation
ª Eclipse PTP Resources
Online Information
ª Information about PTP ª Main web site for downloads, documentation, etc.
ª http://eclipse.org/ptp ª Wiki for designs, planning, meetings, etc.
ª http://wiki.eclipse.org/PTP ª Articles and other documents
ª http://wiki.eclipse.org/PTP/articles
ª Information about Photran ª Main web site for downloads, documentation, etc.
ª http://eclipse.org/photran ª User’s manuals
ª http://wiki.eclipse.org/PTP/photran/documentation
54
Mailing Lists
ª PTP Mailing lists ª Major announcements (new releases, etc.) - low volume
ª http://dev.eclipse.org/mailman/listinfo/ptp-announce ª User discussion and queries - medium volume
ª http://dev.eclipse.org/mailman/listinfo/ptp-user ª Developer discussions - high volume
ª http://dev.eclipse.org/mailman/listinfo/ptp-dev
ª Photran Mailing lists ª User discussion and queries
ª http://dev.eclipse.org/mailman/listinfo/photran ª Developer discussions –
ª http://dev.eclipse.org/mailman/listinfo/photran-dev
55
Getting Involved
ª See http://eclipse.org/ptp ª Read the developer documentation on the wiki ª Join the mailing lists ª Attend the monthly developer meetings
ª Conf Call Monthly: Second Tuesday, 1:00 pm ET ª Details on the PTP wiki
ª Attend the monthly user meetings ª Teleconference Monthly ª Each 4th Wednesday, 2:00 pm ET ª Details on the PTP wiki
PTP will only succeed with your participation!
56