Some Visualization Models applied tothe Analysis of Parallel Applications

Lucas Mello Schnorr

Advisors:Philippe O. A. Navaux & Denis Trystram & Guillaume Huard

Federal University of Rio Grande do Sul, BrazilGrenoble Institute of Technology, France

– Porto Alegre, Brazil – October 26, 2009 –

1 / 1

Introduction - Context

Distributed Systems→ GridsGrid Interconnection and Scalability

Topology and ConnectivityPerformance: bandwidth and latencyNew resources can be added very easily

100 100

1

Bandwidth

Limitations

Hierarchical 

Topology

Influence in the application executionVisualization – Performance Analysis

2 / 1

Introduction - Existing Tools/Techniques

Statistical TechniquesParaGraph (1990) – bar charts, utilization CountPablo (1993) – bar charts + 3D scatter plot

Behavioral TechniquesVampir (1996) – time-line system viewJumpshot (1999), Paje (2000) – space-time

Structural TechniquesParaGraph (1990) – network display / hypercube

3 / 1

Introduction - Problem Identification

Lack of a network-aware analysisDifficult to analyze using space-time viewsStructural techniques undeveloped

Problems of visualization scalabilityVisualization techniques limitations reachedAnalysis are limited to hundreds of entities

Desirable Characteristics for Application Analysis→ The Objectives

Consider network propertiesVisualization scalability in the analysis

4 / 1

Introduction - The Thesis Approach

Explore techniques from Information VisualizationContext of parallel application analysis

Grid resourcesThread/Process parallel applications

Proposed Visualization ModelsBehavioral and Structural/Statistical (3D)

Communication PatternNetwork topology + Communication PatternLogical representation

Visual AggregationLarge-scale tracesLocal and Global summaries

5 / 1

Outline

6 / 1

3D Model - Visual Conception

Resources represented in 2DStructural (e.g. a graph)Statistical

Vertical dimension is timeObjects’ Behavior EvolutionStates and Links

Interaction TechniquesNotion of a CameraRotationTranslationObjects AnimationReplay step-by-step

7 / 1

3D Model - Differences from existing tools

3D Statistical RepresentationPablo→ 3D Scatter PlotParadyn→ 3D TerrainParaProf→ Triang Mesh, 3D Bar and 3D Scatter Plot

3D Behavioral RepresentationParaProf→ 2 metrics and timeVirtue→ the time-tunnel view

Our ApproachPresence of a timeline to show objects’ evolutionMultiple Configurations in the visualization base

8 / 1

3D Model - Abstract Component Model

Input DataApplication Traces

Timestamp-based eventsBehavior registered

Resources DescriptionNetwork topology: graphLogical resource organization: tree

9 / 1

3D Model - The Trace Reader

Deals directly with application traces and eventsOnly trace-dependent part of the modelTransform events into high-level visual objects

Container→ EntitiesState/Variable/Event→ EvolutionLink→ Communications

No semantics→ Visualization is generic

10 / 1

3D Model - The Extractor

Supply entity matcher needs: links and entitiesAttribute entities with location data

where a process is executedwhich process a thread belongs to

Input is also redirected to the Visualization module

11 / 1

3D Model - The Entity Matcher

Responsible for the Visualization Base layoutThree possibilities of configuration are proposed

Communication Pattern (deadlocks, ...)Network Topology (network utilization, routes, ..)Logical Organization (load balancing, ...)

12 / 1

3D Model - Visualization

How the visual objects are represented in 3D

Rendering the visualization base

Application Communication PatternNetwork Topology + App. Communication PatternLogical Organization of Resources

13 / 1

3D Model - Visualization

How the visual objects are represented in 3DRendering the visualization base

Application Communication Pattern

Network Topology + App. Communication PatternLogical Organization of Resources

Visualization

P0P1

P2

P3

P4

P5

P0 P1

P2

P3

P4

P5

P0

P3

P1

P2

P5 P4

P4 P3

P4 P2

P5 P2

P5 P1

P0 P4

P0 P5

Communication Patterngenerated by the Entity Matcher

Flow of Visual objectsfrom the Extractor

14 / 1

3D Model - Visualization

How the visual objects are represented in 3DRendering the visualization base

Application Communication PatternNetwork Topology + App. Communication Pattern

Logical Organization of Resources

Visualization

P0

P1

P2

P3P4

P5

R1R0

R5

R3

R2R4

P0P1

P2

P3

P4

P5

P0

P3

P1

P2

P5 P4

P4 P3

P4 P2

P5 P2

P5 P1

P0 P4

P0 P5

Flow of Visual objectsfrom the Extractor

Network Topology + Communication Patterngenerated by the Entity Matcher

15 / 1

3D Model - Visualization

How the visual objects are represented in 3DRendering the visualization base

Application Communication PatternNetwork Topology + App. Communication PatternLogical Organization of Resources

16 / 1

Outline

17 / 1

Aggregation Model - Overview

Enable large-scale trace analysisVisualy compare entities behaviorDetect global and local characteristics

Steps of the Model

1 Hierarchical Monitoring Data

2 Time-Slice algorithm (temporal integration)

3 Aggregation model (spatial integration)

4 Treemap representation

Visualization differences from existing toolsPlanetLab’s CoVisualize→ resourcesTreemap for Workload Visualization [Stephen 2003]

Lack of configurable time intervals, aggregated data18 / 1

Hierarchical Monitoring Data

Monitoring systems register entities behaviorEntities can be processes and threadsThey can be organized as a hierarchy

Logical hierarchyGeographical Location hierarchyOther possibilities: libraries, components

Grid’5000 example

19 / 1

Time-Slice Algorithm - Basics

Objective: annotate leaf nodes of the hierarchyTime-slice definitionSummary of trace events on the interval

States, Variables, Links, Events, ...

Output of the AlgorithmHierarchy of input + computed values on leaves

20 / 1

Time-Slice Algorithm - Example

21 / 1

Time-Slice Algorithm - Example

22 / 1

Aggregation Model

Objective: aggregated values at intermediary levels

Aggregation Functionsadd, subtract, multiply, divide, max, min, median, ...Depends on

what type of value the leaves havethe desired statistical result

23 / 1

Aggregation Model

Objective: aggregated values at intermediary levels

Aggregation Functionsadd, subtract, multiply, divide, max, min, median, ...Depends on

what type of value the leaves havethe desired statistical result

24 / 1

Aggregation Model

Objective: aggregated values at intermediary levels

Aggregation Functionsadd, subtract, multiply, divide, max, min, median, ...Depends on

what type of value the leaves havethe desired statistical result

25 / 1

Aggregation Model

Objective: aggregated values at intermediary levels

Aggregation Functionsadd, subtract, multiply, divide, max, min, median, ...Depends on

what type of value the leaves havethe desired statistical result

26 / 1

Visualization of the Approach - Treemaps

Technique created in 1991Scalable hierarchical representationAlgorithm

Top-down drawingFor a given node, split screen space among children

Original algorithm has several evolutionsSquarified treemap is used here→ Keeps rectangles as close to squares as possible

27 / 1

Treemap to view the Aggregated Hierarchy

28 / 1

Treemap to view the Aggregated Hierarchy

29 / 1

Treemap to view the Aggregated Hierarchy

30 / 1

Treemap to view the Aggregated Hierarchy

31 / 1

Outline

32 / 1

Triva Prototype Implementation

Developed in Objective-C and C++Combine several existing tools

DIMVisual libraryPaje Components (the Simulator)Graphviz, Ogre3D, wxWidgets

Performance evaluation of PajeAble to handle large-scale tracesSmall response-timeMemory limitations

33 / 1

DIMVisualReader - Trace Reader

Built-in instrumentation of KAAPI library

MPIRastro wrapper for MPI applications

34 / 1

TrivaView - The 3D Approach

Model: Extractor, Entity Matcher & VisualizationInteraction Techniques (Ambient, CameraManager)Base configuration

Application Comm. Pattern created with GraphVizNetwork Topology description (dot format)Logical Organization (plist format)

Placement on the Visualization BaseRendering the 3D Timestamped Paje Objects

35 / 1

TimeSliceView - The Aggregation Model

Only two componentsTimeSlice FilterTriva2DFrame

Time-Slice Algorithm and Aggregation ModelImplementation of the Squarified Treemap AlgorithmDrawing the rectangles with the wxWidgets

36 / 1

Outline

37 / 1

Results

Different application traces are used as inputResults are composed of screenshots of theprototype

ObjectiveCheck if 3D visualizations enable a betterunderstanding of traces with the network topologyCheck if large-scale analysis are possible with theaggregation model

Traces Description3D VisualizationTreemap Visualization

38 / 1

Results - Trace Description

Synthetic tracesLarge-scale hierarchies (up to 100 thousand)Typical Communication Patterns

Real tracesKAAPI Traces

Executing Steal

MPI Traces

Grid’5000 platform in FranceXiru Cluster at Porto Alegre

39 / 1

3D Visualization - Communication Patterns

Differences from the space-time diagram→ 3D enables Graph-like representations→ with time evolution

Process A (Master)

Process B (Slave)

Process C (Slave)

Process D (Slave)

Process E (Slave)

Ring Communication Pattern Fully-Connected Star

Process A

Process B

Process C

Process D

Process EProcess A

Process B

Process C

Process D

Process E

0 5 10 15 20 0 5 10 15 20 0 5 10 15 20

40 / 1

3D Visualization - KAAPI Trace

Fibonacci Application26 processes, two sites, two clustersLines represent steal requestsDifferent number of communication between clusters

beggining→ big tasks, less communicationend→ smaller tasks, more communication

Grelon XiruNancy Router Porto Alegre Router

Run

Steal

41 / 1

3D Visualization - KAAPI Trace

60 processes, two sites, three clustersTotal execution time of a KAAPI fibonacci applicationObserve number of requests in time

Rennes Nancy

Paraquad (25)

Paramount (5)

Grelon (30)

Rennes

Router

Nancy

Router

Paramount (5)

Grelon (30)

Paraquad (25)

Rennes

Router

Nancy

Router

More WS

Requests

Less WS

Requests

42 / 1

3D Visualization - KAAPI Trace

200 processes, 200 machines, two sites, five clustersAnnotated manually with bandwidth limitations

Nancy

Rennes

Grelon (87)Grillon (13)

Paramount (6)

Paraquad (33)

Paravent (61)

100

1

100

Initial Execution of Applicationwith Link Properties 

Interconnection becomes bottleneck,possible hints to better allocation 

Too many WSRequests on lowbandwidth Link

100

43 / 1

3D Visualization - KAAPI Trace

2900 processes, four sites, thirteen clusters

Lille

BordeauxSophiaRennes

(12)

(40) (69)

(840)

(72)(366) (370)

(120)

(504)

(288)

(40)

(88)

(92)

End of Execution

44 / 1

Treemap Visualization - DescriptionTime-Slice and Aggregated Hierarchies

Interaction Techniques: mouse wheel, mouse overDetailed information is available in the status bar

Hierarchy: Site (2) - Cluster(3) - Machine (5)

Executing

Blocked

Hierarchy: Site (2) - Cluster(3) - Machine (5) Hierarchy: maximum aggregation possible

Executing Blocked

Hierarchy: maximum aggregation possible

Hierarchy: Site (2) - Cluster (3) - Machine (5)

Executing

Blocked

Hierarchy: Site (2) - Cluster (3) - Machine (5)Hierarchy: Site (2) - Cluster (3) - Machine (5) Hierarchy: Site (2) - Cluster (3) - Machine (5) 

45 / 1

Treemap Visualization - KAAPI Trace

Paraquad

ParaventParamount

Chinqchint

Chti ChuqueChicon Bordemer Bordeplage Bordereau

Sol

Azur

Helios

Run and RSteal states, 2900 processes, 310 processors

46 / 1

Treemap Visualization - Large-Scale

Synthetic trace with 100 thousand processesTwo states, four-level hierarchy

A Hierarchy: Site (10) - Cluster(10) - Machine (10) -  Processor (100)  B Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) 

C Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100)  D Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) 

E Maximum Aggregation

47 / 1

Treemap Visualization - KAAPI Trace

400 processes, 50 machines, one site8 processes per machine

Overload of some machines with 2 CPUsUnusual amount of time in Steal state

Machines with 4 CPUs show normal behavior

Bordemer Bordeplage

Bordereau

Bordemer Bordeplage

Bordereau

~42 s

~39 s

~39 s ~39 s

~39 s

~39 s

~39 s

~39 s

~39 s

~31 s~34 s

A Larger RSteal states, for each K-Processor B Showing only RSteal state, for each K-Processor

48 / 1

Treemap Visualization - KAAPI Trace

188 processes, 188 machines, five sitesDifferent behavior at Porto AlegreProbably due to the interconnection

Latency for Grid’5000 in France: ~10 msLatency between Porto Alegre and France: ~300 ms

More time spent in work stealing functions

Rennes Toulouse

Porto AlegreBordeauxNancy Rennes

Toulouse

Porto Alegre Bordeaux

Nancy

A Run and RSteal states B Showing only RSteal state

~43 s

~110 s~148 s

~78 s ~65 s

~67 s

~17 s

49 / 1

Treemap Visualization - MPI Trace

Traces from the EP application – NAS Benchmark32 processes – time spent in each MPI operationInit/Barrier: might indicate a linear implementation

B Only MPI_INIT state

~4.5 s

~0.9 s

C Only  MPI_BARRIER state

~5.7 s

~0.3 s

A With States Running, MPI_Init, MPI_Barrier and MPI_AllReduce

Running

MPI_Init

MPI_Barrier

MPI_AllReduce

Running

MPI_Init

MPI_Barrier

MPI_AllReduce

Maximum Aggregation

Only Process Rank 21

50 / 1

Conclusions

The problem identified in the ThesisLack of structural visualization analysisVisualization scalability

Main AchievementsBehavioral with Structural/Statistical Model (3D)

Analysis considering network structureExperiments using Grid’5000 platformIdentification of behavior in KAAPI work stealing

Time-Slice Technique & Aggregation ModelValidated with real-scenario with 2900 processesTested with synthetic traces up to 100K processesLoad-balance efficiency / global and local summaries

51 / 1

Perspectives and Implications

PerspectivesShow aggregated objects in the 3D visualizationOther types of information for the time-slice techniqueUse of other aggregation functionsAggregation model to merge communication patterns...

ImplicationsBetter understanding of parallel applications→ consider execution environment details→ large-scale visual analysis

Re-thinking behavioral visualization→ Do we need a timeline in representations?→ Aggregated data

Use of information visualization techniques

52 / 1

Publications

5 TRIVA PROTOTYPE IMPLEMENTATION . . . . . . . . . . . . . . . . . 75

2 PARALLEL APPLICATION VISUALIZATION . . . . . . . . . . . . . . . 232.1 Historical Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.2 Examplesof PerformanceVisualizationTools . . . . . . . . . . . . . . . 262.2.1 ParaGraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.2.2 TraceView . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.2.3 Pablo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.2.4 Paradyn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.2.5 Vampir . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.2.6 Virtue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.2.7 Jumpshot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.2.8 ParaProf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332.2.9 Pajé . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352.3 Summaryof VisualizationTechniques . . . . . . . . . . . . . . . . . . . 352.3.1 Behavioral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.3.2 Structural . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382.3.3 Statistical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

3 THE THREE-DIMENSIONAL MODEL . . . . . . . . . . . . . . . . . . . 433.1 Visual Conception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443.2 Model Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463.3 TheTraceReader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473.4 TheExtractor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483.5 TheEntityMatcher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493.5.1 Case1: Parallel Application’sCommunicationPattern . . . . . . . . . . . 503.5.2 Case2: Network TopologycombinedwithCommunicationPattern . . . . 513.5.3 Case3: Logical OrganizationandtheCommunicationPattern . . . . . . . 523.6 TheVisualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533.6.1 RenderingtheVisualizationBase . . . . . . . . . . . . . . . . . . . . . . 543.6.2 InteractionMechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . 563.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4 VISUAL AGGREGATION MODEL . . . . . . . . . . . . . . . . . . . . . 594.1 Hierarchical Organizationof MonitoringData . . . . . . . . . . . . . . 604.2 TheTime-SliceAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 624.2.1 States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634.2.2 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644.2.3 Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654.2.4 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664.2.5 Morestatistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664.2.6 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.3 TheAggregationModel . . . . . . . . . . . . . . . . . . . . . . . . . . . 684.3.1 AggregationFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . 694.4 Visualizationof theApproach . . . . . . . . . . . . . . . . . . . . . . . . 704.4.1 TreemapsBasicConcepts . . . . . . . . . . . . . . . . . . . . . . . . . . 704.4.2 TheScalability Issue . . . . . . . . . . . . . . . . . . . . . . . . . . . . 714.4.3 UsingTreemapintheExample . . . . . . . . . . . . . . . . . . . . . . . 724.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

2 PARALLEL APPLICATION VISUALIZATION . . . . . . . . . . . . . . . 232.1 Historical Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.2 Examplesof PerformanceVisualizationTools . . . . . . . . . . . . . . . 262.2.1 ParaGraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.2.2 TraceView . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.2.3 Pablo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292.2.4 Paradyn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.2.5 Vampir . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.2.6 Virtue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.2.7 Jumpshot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.2.8 ParaProf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332.2.9 Pajé . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352.3 Summaryof VisualizationTechniques . . . . . . . . . . . . . . . . . . . 352.3.1 Behavioral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.3.2 Structural . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382.3.3 Statistical . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Grid 2008

Sbac 2009

CCGrid 2009

Future GenerationComputer Systems

Journal

Submit toJournal of Grid

Computing6 RESULTS AND EVALUATION . . . . . . . . . . . . . . . . . . . . . . . 956.1 TracesDescription . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 956.1.1 SyntheticTraces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 966.1.2 KAAPI Traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 996.1.3 MPI Traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1006.2 3DVisualizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1016.2.1 Descriptionof theVisualization . . . . . . . . . . . . . . . . . . . . . . . 1016.2.2 CommunicationPatternsAnalysis . . . . . . . . . . . . . . . . . . . . . 1026.2.3 KAAPI andtheGrid’5000Topology . . . . . . . . . . . . . . . . . . . . 1056.3 TreemapVisualizations . . . . . . . . . . . . . . . . . . . . . . . . . . . 1106.3.1 Descriptionof theVisualization . . . . . . . . . . . . . . . . . . . . . . . 1116.3.2 Large-ScaleVisualizations . . . . . . . . . . . . . . . . . . . . . . . . . 1126.3.3 KAAPI WorkStealingAnalysis . . . . . . . . . . . . . . . . . . . . . . . 1146.3.4 MPI OperationsAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . 119

53 / 1

Acknowledgements

Thesis financed with scholarships byCAPES and CNPqCAPES/Cofecub – Project 4602/06-4

Thanks toAdvisors: Navaux, Guillaume and DenisFamily and friends

54 / 1