Visualizing More Performance Data Than What Fits on Your...

transcript

Visualizing More Performance DataThan What Fits on Your Screen

Lucas Mello Schnorr(LIG – CNRS)

6th International Parallel Tools WorkshopStuttgart, Germany

September 25th, 2012

Challenges and MotivationSupercomputers today

– with very large applications

Sequoia (IBM BlueGene/Q) with 1,572,864 cores (#1 - Top500 - June/2012)

→ Sequoia (4 threads per core): 6 millions threads

Space/Time trace size explosion

Many entities in space + Very detailed behavior in time

Performance Visualization

→ How to keep the representation useful on scale?→ Considering the limited screen space available

Challenges and MotivationSupercomputers today – with very large applications

Performance Visualization→ How to keep the representation useful on scale?→ Considering the limited screen space available

Trace visualizationReal BOINC availability trace file→ Availability is either true or falseOne volunteer client

GNUPlot to a vector file: 8-month and 12-day zoom

Trace visualizationReal BOINC availability trace file→ Availability is either true or falseOne volunteer clientGNUPlot to a vector file: 8-month and 12-day zoom

Trace visualization – trust the rendering?Acroread

Evince

Same vector file, two different views→ Should we trust the rendering?

Trace visualization – trust the rendering?Evince

Space/Time views

Widespread, useful, intuitive, fast adoptionAll trace events represented, causal order

Pajehttp://paje.sf.net

Vitehttp://vite.gforge.inria.fr

Vampirhttp://vampir.eu

However...

Also impacted by ever larger trace sizesLimited visualization scalability

Space/Time views

Widespread, useful, intuitive, fast adoptionAll trace events represented, causal order

Pajehttp://paje.sf.net

Vitehttp://vite.gforge.inria.fr

Vampirhttp://vampir.eu

However...

Also impacted by ever larger trace sizesLimited visualization scalability

Space/Time views – limitations

MPI Sweep3D – 16 processesSimulated with SMPI, traced with SimGrid→ pj_dump’ed to a csv file, loaded into RGantt-charts by R, dumped to a vector file

Space/Time views – limitationsEvince

Space/Time views – limitationsGhostscript

Space/Time views – limitations

MPI Sweep3D – 16 processesReal execution on the Griffon cluster of Grid’5000→ traced with TAU, converted with tau2paje

Once again, Gantt-charts by R→ vector file

Space/Time views – limitationsEvince

Space/Time views – limitationsAcroread

Space/Time views – closer look (ViTe tool)

Trust the OpenGL rendering, no data aggregation

Source: http://vite.gforge.inria.fr8/ 30

Space/Time views – closer look (new Pajé)Trust the rendering → without or with OpenGL

Space/Time views – closer look (old Pajé)

Opaque aggregating filter (no user interaction)→ Slashed rectangles represent time-integrated statesSelf-configure depending on temporal zoom

Source: http://paje.sourceforge.net10/ 30

Space/Time views – closer look (old Pajé)

Space dimension: one process per vertical pixel→ at best, 1000 process represented at the same time

11/ 30

Data aggregation for trace visualizationAlways present (in different forms)→ with large traces

Three groups

Implicit Data Aggregation→ uncontrolled, no visual feedbackExplicit Data Aggregation→ analyst control + visual feedbackForbidden Data Aggregation→ detect the amount of data, forbids overview

Tools might be classified in more than one group→ multiple approaches depending on data dimensionMain objective→ Explicit Data Aggregation

12/ 30

Three groups

Implicit Data Aggregation→ uncontrolled, no visual feedback

Explicit Data Aggregation→ analyst control + visual feedbackForbidden Data Aggregation→ detect the amount of data, forbids overview

12/ 30

Three groups

Implicit Data Aggregation→ uncontrolled, no visual feedbackExplicit Data Aggregation→ analyst control + visual feedback

Forbidden Data Aggregation→ detect the amount of data, forbids overview

12/ 30

Three groups

12/ 30

Three groups

Tools might be classified in more than one group→ multiple approaches depending on data dimension

Main objective→ Explicit Data Aggregation

12/ 30

Three groups

12/ 30

So, why controlled data aggregation?

Technical need: too much data to fit on small screens

Data aggregation→ key for large-scale visualizationSemantic: aggregated data→ more meaningful

13/ 30

Data aggregation→ key for large-scale visualization

Semantic: aggregated data→ more meaningful

13/ 30

Goal: Visualization techniques for aggregated traces

13/ 30

Outline

1 Multi-Scale Data Aggregation

2 Visualization techniquesSquarified Treemap ViewHierarchical Graph View

3 Tools and framework

4 Conclusion

14/ 30

Multi-Scale Data Aggregation

15/ 30

Multi-Scale Data AggregationIn shortFΓ,∆ :R× T → R

(r , t) 7→∫∫

NΓ,∆(r ,t)ρ(r ′, t ′).dr ′.dt ′

Method requires dimension ordering→ Time scale is trivial – what about space scale?Additional scales could be added

Some Examples

16/ 30

(r , t) 7→∫∫

NΓ,∆(r ,t)ρ(r ′, t ′).dr ′.dt ′

Method requires dimension ordering→ Time scale is trivial – what about space scale?

Additional scales could be added

Some Examples

16/ 30

(r , t) 7→∫∫

NΓ,∆(r ,t)ρ(r ′, t ′).dr ′.dt ′

Some Examples

16/ 30

(r , t) 7→∫∫

NΓ,∆(r ,t)ρ(r ′, t ′).dr ′.dt ′

Some Examples

16/ 30

Multi-Scale Data Aggregation – Time

1 Time interval defined during the analysis2 Summary of events for each monitored entity

BlockedExecution

9 seconds

Time-integrated summary for processes

Numbers are in seconds (Execution, Blocked)B=(7,2)A=(4,5) C=(3,6) E=(4,5)D=(9,0)

17/ 30

Multi-Scale Data Aggregation – Time

1 Time interval defined during the analysis2 Summary of events for each monitored entity

BlockedExecution

9 seconds

Time-integrated summary for processes

Numbers are in seconds (Execution, Blocked)B=(7,2)A=(4,5) C=(3,6) E=(4,5)D=(9,0)

17/ 30

Multi-Scale Data Aggregation – Space1 Define a neighborhood for each monitored entity2 Apply an aggregating operator on the neighborhood

Neighborhood as a hierarchyResource-basedApplication groups

Deeper the hierarchy→ higher the quality

Space-integrated summary

Aggregating operator: addition (Execution, Blocked)

B=(7,2)

A=(4,5)

C=(3,6)

E=(4,5)

D=(9,0)

M1=(11,7)

M2=(12,6)

M3=(4,5)

C1=(11,7)

C2=(16,11)

G=(27,18)

18/ 30

Multi-Scale Data Aggregation – Space1 Define a neighborhood for each monitored entity2 Apply an aggregating operator on the neighborhood

Neighborhood as a hierarchyResource-basedApplication groups

Deeper the hierarchy→ higher the quality

Space-integrated summary

Aggregating operator: addition (Execution, Blocked)

B=(7,2)

A=(4,5)

C=(3,6)

E=(4,5)

D=(9,0)

M1=(11,7)

M2=(12,6)

M3=(4,5)

C1=(11,7)

C2=(16,11)

G=(27,18)

18/ 30

Visualization Techniques

19/ 30

Visualization techniques

Squarified Treemap ViewObserve outliers, differences of behaviorHierarchical aggregation

B Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) C Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) D Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) E Maximum Aggregation

Hierarchical Graph ViewCorrelate application behavior to network topologyPin-point resource contention

All nodes Clusters Sites Fullaggregation

20/ 30

Visualization techniques

Squarified Treemap ViewObserve outliers, differences of behaviorHierarchical aggregation

B Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) C Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) D Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100) E Maximum Aggregation

Hierarchical Graph ViewCorrelate application behavior to network topologyPin-point resource contention

All nodes Clusters Sites Fullaggregation

20/ 30

Squarified Treemap View

Scalable representation for hierarchies→ better visualization scalability for large trees

Space-filling top-down recursive layout algorithmNode value→ space occupied in the screenSquarified version→ keeps rectangles ratio close to 1

M=3 N=2 O=1

A=1 B=1 C=1 D=1 E=1 F=1

T=6 M=3N=2

21/ 30

Scalable representation for hierarchies→ better visualization scalability for large treesSpace-filling top-down recursive layout algorithm

Node value→ space occupied in the screenSquarified version→ keeps rectangles ratio close to 1

M=3 N=2 O=1

A=1 B=1 C=1 D=1 E=1 F=1

T=6 M=3N=2

21/ 30

M=3 N=2 O=1

A=1 B=1 C=1 D=1 E=1 F=1

M=3N=2

21/ 30

M=3 N=2 O=1

A=1 B=1 C=1 D=1 E=1 F=1

T=6 M=3N=2

21/ 30

M=3 N=2 O=1

A=1 B=1 C=1 D=1 E=1 F=1

T=6 M=3N=2

21/ 30

Squarified Treemap View – an exampleSynthetic trace with 100 thousand processesTwo states, four-level hierarchyVisualization artifacts without spatial aggregation

A Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100)

22/ 30

Squarified Treemap View – an exampleSynthetic trace with 100 thousand processesTwo states, four-level hierarchy

Visualization artifacts without spatial aggregation

B Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100)

22/ 30

C Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100)

22/ 30

D Hierarchy: Site (10) - Cluster(10) - Machine (10) - Processor (100)

22/ 30

E Maximum Aggregation

22/ 30

Squarified Treemap View – KAAPI exampleKAAPI (run DAGs, work stealing for load balacing)188 processes running on five clusters

Rennes

Toulouse

Porto Alegre Bordeaux

~110 s~148 s

~78 s ~65 s

Analysis: stealing requests depends on latencyPorto Alegre – France: ~300 ms In France: ~10 ms

23/ 30

Squarified Treemap View – KAAPI exampleKAAPI (run DAGs, work stealing for load balacing)188 processes running on five clusters

Rennes

Toulouse

Porto Alegre Bordeaux

~110 s~148 s

~78 s ~65 s

Analysis: stealing requests depends on latencyPorto Alegre – France: ~300 ms In France: ~10 ms

23/ 30

Hierarchical Graph View

Scalable representation for graphsTopology, with application-level metricsIdentify resource bottleneck in space and time

Use spatial-temporal aggregated tracesInteractive force-directed layout (Barnes-Hut algorithm)

Trace metrics → geometrical properties→ Size, shape, filling, colors→ Nodes: monitored entities→ Edges: relationship among entities

hostA hostB

link utilization

time slice

24/ 30

hostA hostB

link utilization

time slice

24/ 30

hostA hostB

link utilization

time slice

24/ 30

Hierarchical Graph View - an example

Squares are hosts, diamonds are network linksColors represent different applicationsor parts of it (task type, phase)

Two clusters interconnected by four network links

time slice

25/ 30

Hierarchical Graph View - a larger exampleSquares are hosts, computer power defines sizeFrench Grid5000 platform: 2170 nodes

All nodes

26/ 30

Clusters

26/ 30

Fullaggregation

26/ 30

Tools and Framework

27/ 30

Open-source toolsPajeNG – Next Generation (Space/Time view), GPL3

http://github.com/schnorr/pajeng/

Only pj_dump and pj_validate (partial pajeng)Link against libpaje.so – then use Paje API to get data

Data Liberation Front®

$ ./pj_dump input.paje > output.csv

Then use your preferred visualization tool (gnuplot, R, GNU Octave, ...)

Viva (Aggregated Treemaps, Hierarchical graph), GPL3

http://github.com/schnorr/viva/

Partial graph visualization for now (soon)Deprecates Triva, serves as research framework

Some tracers and converters: SimGrid, Akypuera(with Score-P support, such as otf22paje), Poti,XKaapi, EZTrace, rastro2paje, libRastro, GTG,JRastro and counting...

$ ./otf22paje ./scorep-20120827/traces.otf2 | ./viva ...$ ./otf22paje ./scorep-20120827/traces.otf2 | pj_dump > output.csv

28/ 30

Only pj_dump and pj_validate (partial pajeng)Link against libpaje.so – then use Paje API to get dataData Liberation Front®

28/ 30

Conclusion and Future WorkLarge scale traces – limited screen space

Explicit Data aggregation→ Controlled + visual feedback

Concerns with behavior attenuationAggregation may remove important detailsFlexible aggregation: operators & neighborhood

Visualization techniques for aggregated dataContinuous evaluation of visualization scalability

With larger data-sets, does it remain useful?

Future Work

Revisit Space/Time representationsAggregation operators to deal with time uncertaintyTo quantify loss of information when aggregating

29/ 30

Future Work

29/ 30

Future Work

29/ 30

Future Work

29/ 30

Thank you for your attentionSome references

Visualizing More Performance Data Than What Fitson Your Screen. Lucas Mello Schnorr, ArnaudLegrand. The 6th International Parallel ToolsWorkshop. Springer. 2012. (Invited paper)Detection and Analysis of Resource Usage Anomalies in LargeDistributed Systems Through Multi-scale Visualization. Lucas MelloSchnorr, Arnaud Legrand, Jean-Marc Vincent. Concurrency andComputation: Practice and Experience. Wiley. 2012.A Hierarchical Aggregation Model to achieve Visualization Scalability inthe analysis of Parallel Applications. Lucas Mello Schnorr, GuillaumeHuard, Philippe Olivier Alexandre Navaux. Parallel Computing. Volume38, Issue 3, March 2012, Pages 91-110.

More information→ http://mescal.imag.fr/membres/lucas.schnorr/

INFRA-SONGS Project (WP-7)http://infra-songs.gforge.inria.fr/

Simulation of Next Generation SystemsWP-7: Visualization and Analysis

30/ 30

Visualizing More Performance Data Than What Fits on Your...

Documents