"Parallel MATLAB in production supercomputing with applications in signal and image processing"

Ashok KrishnamurthyDavid HudakJohn NehrbassSiddharth SamsiVijay Gadepally

2

Functions - Scope of ActivitySupercomputing. Computation, software, storage, and support services empower Ohio’s scientists, engineers, faculty, students, businesses and other clients.

Networking. Ohio’s universities, colleges, K-12 and state government connect to the network. OSC also provides engineering services, video conferencing, and support through a 24x7 service desk.

Research. Lead science and engineering projects, assist researchers with custom needs, partner with regional, national, and international researchers in groundbreaking initiatives, and develop new tools.

Education. The Ralph Regula School of Computational Science delivers computational science training to students and companies across Ohio.

3

OSC provides a stable computational infrastructure to support scientific research computing

IBM 1350 • Opteron dual-core w/IBM Cell • 4,000+ cores• 8 TBytes memory• 22+ teraflops• Blend of 4 core and 8 core nodes

– Large processor count– Large memory SMP jobs

Intel P4 Cluster•2.46 TF•512 processors•1 TBytes memory

Itanium2 Cluster•2.7 TF•596 processors 3x16 Alt•1 TBytes memory

PRODUCTION COMPUTING

Mass Storage470 TBytes disk80 TBytes tapeNFS, PVFS, iSCSI

Infiniband or Myrinet Interconnect

Gateway to User Science

4

Visualization Cluster•AMD Opteron•72 processors•144 GBytes memory•nVIDIA Quadro 5600 graphics card(330 GF)

Providing agile computational infrastructure to support the research and innovation process

Apple G5 Cluster•PowerPC G5•64 processors•128 GBytes mem

MATLAB/GRI Cluster•AMD Opteron•164 processors•328 GBytes memory

RESEARCH COMPUTING

Mass Storage470 TBytes disk80 TBytes tapeNFS, PVFS, iSCSI

Gateway to User Science

BALE Cluster•AMD Athlon 64•110 processors•220 GBytes memory•nVIDIA GeForce 6150 GPU

5

OSC Instrumentation and Analytics Services

• Remote instrumentation uses OSC’s state-wide resources– Networking, Storage, HPC, Analytics (web service)

ParallelMATLAB

Parallel MATLAB Choices

• MathWorks Distributed Computing Engine + Toolbox

• Interactive Supercomputing’s Star-P

• MIT Lincoln Labs pMATLAB + OSC bcMPI

EMPOWER. PARTNER. LEAD.

EMPOWER. PARTNER. LEAD.

MATLAB® Distributed Computing Toolbox Architecture

Image source: http://www.mathworks.com

EMPOWER. PARTNER. LEAD.

Star-P Architecture

Image source: http://www.interactivesupercomputing.com

Dedicated Cluster:

Desktop client can directly address cluster compute nodes

Typical Shared Production Cluster:

Desktop client cannot directly address cluster compute nodes

Parallel MATLAB Computing Configurations• Have to map client+engines to a number of computer

configurations

• Local or Remote– Local - single administrative domain

• Uniform set of user accounts• Implicit trust relationship

– Remote - multiple administrative domains• Authentication required

• Dedicated or shared– Dedicated - preallocated resources available on demand– Shared - allocation request must complete prior to interactive

session

EMPOWER. PARTNER. LEAD.

MATLAB Distributed Computing Engine & Toolbox

• Mathworks implementation of parallel MATLAB

• Consists of two products :– MATLAB® Distributed Computing Engine (MDCE)– Distributed Computing Toolbox (DCT)

• MDCE enables users to run MATLAB applications on a cluster

• DCT provides toolbox migration : Client’s toolbox licenses are available when the parallel job runs on the cluster

• Can be used in interactive mode as well as in non-interactive batch jobs

EMPOWER. PARTNER. LEAD.

MATLAB® Distributed Computing Engine

• Customization of scripts to run under shared, remote resources which includes

– System specific batch scripts must be generated at run time

– Authentication and remote connection setup when a job is submitted to the cluster

EMPOWER. PARTNER. LEAD.

Star-P

• Star-P is a client-server parallel computing platform available from Interactive Supercomputing

• Designed to work with high level languages such as MATLAB and Python

• Designed for interactive usage

EMPOWER. PARTNER. LEAD.

Star-P

• Customization required to run under the Torque workload manager

– Custom parameters to be used include command-line options that control job submission to the cluster

– Custom MPI launch mechanism developed by Interactive Supercomputing for use on OSC clusters

• Does not support toolbox migration

• Monitoring and debugging server backend processes is still a challenge

EMPOWER. PARTNER. LEAD.

pMATLAB + bcMPI• Runs on UNIX: tested on Linux, NetBSD, MacOS X

• API with MatlabMPI– If you can use MatlabMPI, you can probably use bcMPI– bcMPI tags are numeric, MatlabMPI alphanumeric

• Broadcast, barrier, reduce operations

• bcMPI supports synchronous or asynchronous sends– MPI_Buffer_attach, MPI_Buffer_detatch, MPI_Probe

• MPI communicator support (new in v1.1)– Supports many MATLAB data types, but no sparse support

EMPOWER. PARTNER. LEAD.

bcMPI

• Advantages :– Extensible - core library makes it easy to add

additional MPI functions and interpreter data types – Portable - no dependencies on any machine, or

specific MPI library implementations – Scalable - use efficient algorithms; take advantage of

native MPI library and communications hardware – Open source software developed at OSC

• Can be used only in non-interactive batch jobs

20

Example of remote instrumentation application: access to electron microscope

Parallel MATLAB Example: Image analysis of Scanning Electron Microscope Images

• Scanning Electron Microscope at OSU Center for Accelerated Maturation of Materials

• Demonstrates real-time user control

• Adding analytics and collaboration services for image analysis and computational modeling

CAMM Portal Interface

21

Upload Search

Home

Images

Filter Setup

Help

BatchAnalysis

Results

Logout Services About Contact

Welcome John Doe CAMM Image Storage & Analysis Portal

Browse

Browse for Filesamsi\siron\0784A1k.tiffilename

Lab notes about this imageÉ

Notes:

0784Sample id

SEMInstrument

1000xMagnification

ALocation

EMPOWER. PARTNER. LEAD.

MATLAB GUI to set image processing parameters

SEGMENTS ADDED TOGETHER

Parallel Processing : MATLAB GUI

Parallel MATLAB : Results

Run Time (minutes) vs. Number Of CPUs

0.00

4.00

8.00

12.00

16.00

20.00

24.00

28.00

32.00

36.00

40.00

44.00

48.00

52.00

56.00

60.00

0 2 4 6 8 10 12 14 16 18

Number of CPUs

Tim

e (M

inu

tes)

• 396 images processed on 2, 4, 8, 12 and 16 processors

EMPOWER. PARTNER. LEAD.

Parallel MATLAB Application : Acoustics Signal Processing

• MATLAB used to analyze acoustic signatures used for self-localization of sensors

• Comparative analysis using multiple algorithms on multiple data sets – embarrassingly parallel

EMPOWER. PARTNER. LEAD.

Parallel MATLAB application : Synthetic Aperture Radar Model[1]

• Develop synthetic aperture radar model for forest clusters

• MATLAB used to study invertibility of forest model and to fit model parameters to SAR data from several forests

• Parallel MATLAB used to perform above studies

[1] A Model for Generating Synthetic VHF SAR Forest Clutter Images – Julie Ann Jackson, Randolph L. Moses. Paper submitted to IEEE Aerospace and Electronic Systems Journal

Parallel MATLAB Application: Pathology slide image processing

Summary

• Parallel MATLAB is of interest to many in OSC’s user community

• Parallel MATLAB is bringing in new users to High Performance Computing

• Users would like the same experience as desktop MATLAB, but are willing to give up some of it for quicker turn-around time

• Providing a complete solution is a lot of work – even for embarrassingly parallel cases

EMPOWER. PARTNER. LEAD.