Interactive Supercomputing Update

IDC HPC User’s Forum, September 2008

Agenda

Why am I here?Some trends…What does Interactive Supercomputing do?What’s new? (and app examples if there is time)

2

Why I’m here (at least partly)

At the April User’s Forum meeting, somebody on a panel said something like;

‘I don’t want to learn MPI, I wish computer scientists would build tools to make my life easier.’

At that very moment, I was interviewing with Interactive Supercomputing…

3

HPC Conventional Wisdom Includes;

• Computing cost continues to decline while reality cost continues to rise – creating pull for “in silico” techniques

• More compute power is needed for multiple reasons;• More fidelity; multi-physics; data explosion…

• Increasing complexity in the compute engine• More cores, not faster cores; Potentially less capability / core;

Multi-threading HW; The usual pain points are only getting worse. E.g. memory and i/o BW/FLOP, latencies…

• Creating a more difficult strategy choice for development; multicore, manycore, gpu, thin, thick or fat nodes…

4

There is a strong need for new development tools -- even for experienced parallel programmers. But in the meantime…

The Domain Expert View

• Swamped by the velocity of their own domain• Long ago moved from 3GL’s to VHLL’s

• E.g. from FORTRAN to some variant of the M language (most likely Matlab®)

• … and don’t want to move back• Now have enough data and math to need more

than one desktop worth of compute• Our surveys show as many as 40% of users are

performance limited for some applications

5

What we do:

• Make high performance computing accessible to the widest possible range of users; • enable domain experts to develop and deploy

high performance parallel applications easily6

Note: “server” includes “cluster”

Star-P Value PropositionHigher Productivity, Quicker Results, No complex programming

Star-P Open Software Platform

What’s New? (courtesy PNNL)

9

Data-Intensive Computing

Manage the Explosion

of Data(high throughput

data capture)

ExtractKnowledge

from Massive Datasets

(fusion, activeanalysis, predictive

modeling)

Reduce Data(facilitate human understanding)

Modeling & Simulations

Instruments

Sensors

time

We call this stepKnowledge Discovery

Why Star-P for Knowledge Discovery?

• Need to match algorithm to data means users need to experiment with multiple algorithms• VHLL makes code changes easy• Note, we see this requirement often – e.g. in finance and

intelligence where codes must be continually adapted• Size of data means HPC is required for experiments

• With Star-P, good enough speed-up is achieved quickly• Star-P includes KD functions which run in parallel• and Parallel I/O to remove that potential bottleneck

10

Factoring network flow behavior [Karpinski, Almeroth, Belding]

Algorithmic exploration

Many NMF variants exist in the literature– Not clear how useful on large data– Not clear how to calibrate (i.e., number of iterations to converge)

NMF algorithms combine linear algebra and optimization methods

Basic and “improved” NMF factorization algorithms implemented:– euclidean (Lee & Seung 2000)– K-L divergence (Lee & Seung 2000)– semi-nonnegative (Ding et al. 2006)– left/right-orthogonal (Ding et al. 2006)– bi-orthogonal tri-factorization (Ding et al. 2006)– sparse euclidean (Hoyer et al. 2002)– sparse divergence (Liu et al. 2003)– non-smooth (Pascual-Montano et al. 2006)

NMF traffic analysis results• NMF identifies essential components of the traffic• Analyst labels different types of external behavior

Computational Ecology

• Modeling dispersal of species within a habitat (to maximize range)

• Large geographic areas, linked with GIS data

• Blend of numerical and combinatorial algorithms

Brad McRae and Paul Beier, “Circuit theory predicts gene flow in plant and animal populations”, PNAS, Vol. 104, no. 50, December 11, 2007

Results

Solution time reduced from 3 days (desktop) to 5 minutes (14p) for typical problems

Aiming for much larger problems: Yellowstone-to-Yukon (Y2Y)

16

Thank You!

David RichVP Marketing

+1 781 419 5095drich@interactivesupercomputing.com