PerformanceEngineeringusingMVAPICHandTAU
SameerShendeUniversityofOregon
MVAPICHUser’sGroup(MUG)Meeting
11:15am–11:45am,Wednesday,August8,2018OhioSupercomputingCenter,Columbus,Ohio
Acknowledgments
• The MVAPICH2 team The Ohio State University • http://mvapich.cse.ohio-state.edu
• TAU team at the University of Oregon • http://tau.uoregon.edu
2http://tau.uoregon.edu/mug18.pdf
Outline
• Introduction• TheMPIToolsInterfacesandBenefits• IntegratingTAUandMVAPICH2withMPI_T• UseCases• TAUPerformanceSystem®
3http://tau.uoregon.edu/mug18.pdf
Canwedesigndynamicandadaptivepoint-to-pointcommunicationmechanismsthatcandeliverthebest
1. Communicationperformance2. Overlapofcomputationandcommunication3. Memoryfootprint
BroadChallenge
4http://tau.uoregon.edu/mug18.pdf
TAU Performance System®
• Tuning and Analysis Utilities (22+ year project) • Comprehensive performance profiling and tracing
• Integrated, scalable, flexible, portable • Targets all parallel programming/execution paradigms
• Integrated performance toolkit • Instrumentation, measurement, analysis, visualization • Widely-ported performance profiling / tracing system • Performance data management and data mining • Open source (BSD-style license) • Uses performance and control variables to interface with MVAPICH2
• Integrates with application frameworks • http://tau.uoregon.edu
5http://tau.uoregon.edu/mug18.pdf
Understanding Application Performance using TAU
• How much time is spent in each application routine and outer loops? Within loops, what is the contribution of each statement?
• How many instructions are executed in these code regions? Floating point, Level 1 and 2 data cache misses, hits, branches taken?
• What is the memory usage of the code? When and where is memory allocated/de-allocated? Are there any memory leaks?
• What are the I/O characteristics of the code? What is the peak read and write bandwidth of individual calls, total volume?
• What is the contribution of each phase of the program? What is the time wasted/spent waiting for collectives, and I/O operations in Initialization, Computation, I/O phases?
• How does the application scale? What is the efficiency, runtime breakdown of performance across different core counts?
• How can I tune MPI for better performance? What performance and control does MVAPICH2 export to observe and control its performance?
6http://tau.uoregon.edu/mug18.pdf
7
Limited Profile
Flat Profile
Loop Profile
Phase Profile
Callpath Profile
Trace
All levels support multiple metrics/counters
O(KB) O(TB)
Howmuchdatadoyouwant?
http://tau.uoregon.edu/mug18.pdf
8
DirectviaProbes IndirectviaSampling
• Exact measurement • Fine-grain control • Calls inserted into code
• No code modification • Minimal effort • Relies on debug symbols (-g
option) • TAU_SAMPLING=1
call TAU_START(‘potential’) // code call TAU_STOP(‘potential’)
PerformanceDataMeasurement
http://tau.uoregon.edu/mug18.pdf
Outline
• Introduction• TheMPIToolsInterfacesandBenefits• IntegratingTAUandMVAPICH2withMPI_T• UseCases• TAUPerformanceSystem®
9http://tau.uoregon.edu/mug18.pdf
MVAPICH2 and TAU
● TAUandMVAPICH2areenhancedwiththeabilitytogeneraterecommendationsandengineeringperformancereport
● MPIlibrarieslikeMVAPICH2arenow“reconfigurable”atruntime● TAUandMVAPICH2communicateusingtheMPI-Tinterface
10http://tau.uoregon.edu/mug18.pdf
MPI_TsupportwithMVAPICH2
MemoryUsage:- currentlevel
- maximumwatermark
Registrationcache:- hits
- misses
Pt-to-ptmessages:- unexpectedqueuelength- unexp.matchattempts
- recvq.length
Shared-memory:- limic/CMA
- bufferpoolsize&usage
Collectiveops:- comm.creation
- #algorithminvocations[Bcast–8;Gather–10]
…
InfiniBandN/W:- #controlpackets
- #out-of-orderpackets
• Supportperformancevariables(PVAR)
• VariablestotrackdifferentcomponentswithintheMPIlibrary
• InitialsupportforControlVariables(CVAR)• VariablestomodifythebehaviorofMPILibrary
11http://tau.uoregon.edu/mug18.pdf
Outline
• Introduction• TheMPIToolsInterfacesandBenefits• IntegratingTAUandMVAPICH2withMPI_T• UseCases• TAUPerformanceSystem®
12http://tau.uoregon.edu/mug18.pdf
Interacting TAU with MVAPICH2 through MPI_T Interface
● EnhanceexistingsupportforMPI_TinMVAPICH2toexposearichersetofperformanceandcontrolvariables
● GetanddisplayMPIPerformanceVariables(PVARs)madeavailablebytheruntimeinTAU
● Controltheruntime’sbehaviorviaMPIControlVariables(CVARs)
● AddsupporttoMVAPICH2andTAUforinteractiveperformanceengineeringsessions
13http://tau.uoregon.edu/mug18.pdf
Plugin-based Infrastructure for Non-Interactive Tuning
● PerformancedatacollectedbyTAU● SupportforPVARsandCVARs● SettingCVARstocontrolMVAPICH2● StudyingperformancedatainTAU’s
ParaProfprofilebrowser● Multiplepluginsavailablefor
● Tuningapplicationatruntimeand● Generatepost-runrecommendations
14http://tau.uoregon.edu/mug18.pdf
Enhancing MPI_T Support ● IntroducedsupportfornewMPI_TbasedCVARstoMVAPICH2
○ MPIR_CVAR_MAX_INLINE_MSG_SZ■ Controlsthemessagesizeuptowhich“inline”transmissionofdatais
supportedbyMVAPICH2○ MPIR_CVAR_VBUF_POOL_SIZE
■ Controlsthenumberofinternalcommunicationbuffers(VBUFs)MVAPICH2allocatesinitially.Also,MPIR_CVAR_VBUF_POOL_REDUCED_VALUE[1]([2…n])
○ MPIR_CVAR_VBUF_SECONDARY_POOL_SIZE■ ControlsthenumberofVBUFsMVAPICH2allocateswhenthereareno
morefreeVBUFsavailable○ MPIR_CVAR_IBA_EAGER_THRESHOLD
■ ControlsthemessagesizewhereMVAPICH2switchesfromeagertorendezvousprotocolforlargemessages
● TAUenhancedwithsupportforsettingMPI_TCVARsinanon-interactivemodeforuninstrumentedapplications
15http://tau.uoregon.edu/mug18.pdf
MVAPICH2
● SeveralnewMPI_TbasedPVARsaddedtoMVAPICH2○ mv2_vbuf_max_use,mv2_total_vbuf_memoryetc
● EnhancedTAUwithsupportfortrackingofMPI_TPVARsandCVARsforuninstrumentedapplications○ ParaProf,TAU’svisualizationfrontend,enhancedwithsupportfor
displayingPVARsandCVARs○ TAUprovidestau_exec,atooltotransparentlyinstrumentMPIroutines
○ Uninstrumented:%mpirun–np1024./a.out
○ Instrumented:%mpirun–np1024tau_exec[options]./a.out%paraprof
16http://tau.uoregon.edu/mug18.pdf
PVARs Exposed by MVAPICH2
17http://tau.uoregon.edu/mug18.pdf
CVARs Exposed by MVAPICH2
18http://tau.uoregon.edu/mug18.pdf
Using MVAPICH2 and TAU
● TosetCVARsorreadPVARsusingTAUforanuninstrumentedbinary:%exportTAU_TRACK_MPI_T_PVARS=1%exportTAU_MPI_T_CVAR_METRICS=
MPIR_CVAR_VBUF_POOL_REDUCED_VALUE[1],MPIR_CVAR_IBA_EAGER_THRESHOLD
%exportTAU_MPI_T_CVAR_VALUES=32,64000%exportPATH=/path/to/tau/x86_64/bin:$PATH%mpirun-np1024tau_exec-Tmvapich2,mpit./a.out%paraprof
19http://tau.uoregon.edu/mug18.pdf
VBUF usage without CVARs
20http://tau.uoregon.edu/mug18.pdf
VBUF usage with CVARs
Total memory used by VBUFs is reduced from 3,313,056 to 1,815,056
21http://tau.uoregon.edu/mug18.pdf
VBUF Memory Usage Without CVAR
22http://tau.uoregon.edu/mug18.pdf
VBUF Memory Usage With CVAR
% export TAU_TRACK_MPI_T_PVARS=1 % export TAU_MPI_T_CVAR_METRICS=MPIR_CVAR_VBUF_POOL_SIZE % export TAU_MPI_T_CVAR_VALUES=16 % mpirun -np 1024 tau_exec -T mvapich2 ./a.out
23http://tau.uoregon.edu/mug18.pdf
TAUPerformanceSystem
24
Instrumentation • Fortran, C++, C, UPC, Java, Python, Chapel, Spark • Automatic instrumentation
Measurement and analysis support • MPI, OpenSHMEM, ARMCI, PGAS, DMAPP • pthreads, OpenMP, OMPT interface, hybrid, other thread models • GPU, CUDA, OpenCL, OpenACC • Parallel profiling and tracing • Use of Score-P for native OTF2 and CUBEX generation
Analysis • Parallel profile analysis (ParaProf), data mining (PerfExplorer) • Performance database technology (TAUdb) • 3D profile browser
http://tau.uoregon.edu/mug18.pdf
Instrumentation
Sourceinstrumentationusingapreprocessor• Addtimerstart/stopcallsinacopyofthesourcecode.• UseProgramDatabaseToolkit(PDT)forparsingsourcecode.• RequiresrecompilingthecodeusingTAUshellscripts(tau_cc.sh,tau_f90.sh)• Selectiveinstrumentation(filterfile)canreduceruntimeoverheadandnarrowinstrumentationfocus.
Compiler-basedinstrumentation• Usesystemcompilertoaddaspecialflagtoinserthooksatroutineentry/exit.• RequiresrecompilingusingTAUcompilerscripts(tau_cc.sh,tau_f90.sh…)
RuntimepreloadingofTAU’sDynamicSharedObject(DSO)• Noneedtorecompilecode!Usempiruntau_exec./appwithoptions.• Requiresdynamicexecutable(linkusing–dynamiconTheta).
Addhooksinthecodetoperformmeasurements
25
TAU Execution Command (tau_exec) Uninstrumented execution
• % mpirun -np 256 ./a.out Track GPU operations
• % mpirun –np 256 tau_exec –cupti ./a.out • % mpirun –np 256 tau_exec –cupti -um ./a.out (for Unified Memory) • % mpirun –np 256 tau_exec –opencl ./a.out • % mpirun –np 256 tau_exec –openacc ./a.out
Track MPI performance using MPI_T interface • % mpirun -np 256 tau_exec –T mpit ./a.out
Track OpenMP, and MPI performance (MPI enabled by default) • % export TAU_OMPT_SUPPORT_LEVEL=full;
% export TAU_OMPT_RESOLVE_ADDRESS_EAGERLY=1 • % mpirun -np 256 tau_exec –T ompt,tr6,mpi –ompt ./a.out
Track memory operations • % export TAU_TRACK_MEMORY_LEAKS=1 • % mpirun –np 256 tau_exec –memory_debug ./a.out (bounds check)
Use event based sampling (compile with –g) • % mpirun –np 256 tau_exec –ebs ./a.out • Also –ebs_source=<PAPI_COUNTER> -ebs_period=<overflow_count>
26 http://tau.uoregon.edu/mug18.pdf
Using TAU TAU supports several measurement and thread options
Phase profiling, profiling with hardware counters (papi), MPI library, CUDA, Beacon (backplane for event notification – online monitoring), PDT (automatic source instrumentation) … Each measurement configuration of TAU corresponds to a unique stub makefile and library that is generated when you configure it
To instrument source code automatically using PDT Choose an appropriate TAU stub makefile in <arch>/lib: % source ~tg457572/tau.bashrc % export TAU_MAKEFILE=$TAU/Makefile.tau-mvapich2-icpc-mpi-pdt % export TAU_OPTIONS=‘-optVerbose …’ (see tau_compiler.sh )
Use tau_f90.sh, tau_cxx.sh, tau_upc.sh, or tau_cc.sh as F90, C++, UPC, or C compilers respectively: % mpif90 foo.f90 changes to % tau_f90.sh foo.f90
Set runtime environment variables, execute application and analyze performance data:
% pprof (for text based profile display) % paraprof (for GUI)
27http://tau.uoregon.edu/mug18.pdf
Choosing TAU_MAKEFILE % source ~tg457572/tau.bashrc % ls $TAU/Makefile.* /home1/00494/tg457572/pkgs/tau_latest/x86_64/lib/Makefile.tau-mvapich2-icpc-mpi-pdt/home1/00494/tg457572/pkgs/tau_latest/x86_64/lib/Makefile.tau-mvapich2-icpc-mpi-pthread-pdt/home1/00494/tg457572/pkgs/tau_latest/x86_64/lib/Makefile.tau-mvapich2-icpc-ompt-mpi-pdt-openmp/home1/00494/tg457572/pkgs/tau_latest/x86_64/lib/Makefile.tau-mvapich2-icpc-papi-mpi-pdt-mpit/home1/00494/tg457572/pkgs/tau_latest/x86_64/lib/Makefile.tau-mvapich2-icpc-papi-ompt-mpi-pdt-openmp/home1/00494/tg457572/pkgs/tau_latest/x86_64/lib/Makefile.tau-pdt
For an MPI+F90 application with MPI, you may choose Makefile.tau-mvapich2-icpc-mpi-pdt
• Supports MPI instrumentation, papi, and PDT for automatic source instrumentation % export TAU_MAKEFILE=$TAU/Makefile.tau-mvapich2-icpc-mpi-pdt % tau_f90.sh matrix.f90 -o matrix OR with build systems: % make CC=tau_cc.sh CXX=tau_cxx.sh F90=tau_f90.sh % cmake –DCMAKE_Fortran_COMPILER=tau_f90.sh
–DCMAKE_C_COMPILER=tau_cc.sh –DCMAKE_CXX_COMPILER=tau_cxx.sh % mpirun -np 1024 ./matrix % paraprof
28http://tau.uoregon.edu/mug18.pdf
Configuration tags for tau_exec % ./configure –pdt=<dir> -mpi –papi=<dir>; make install Creates in $TAU: Makefile.tau-papi-mpi-pdt(Configuration parameters in stub makefile) shared-papi-mpi-pdt/libTAU.so % ./configure –pdt=<dir> -mpi; make install creates Makefile.tau-mpi-pdt shared-mpi-pdt/libTAU.so To explicitly choose preloading of shared-<options>/libTAU.so change: % mpirun -np 256 ./a.out to % mpirun -np 256 tau_exec –T <comma_separated_options> ./a.out % mpirun -np 256 tau_exec –T papi,mpi,pdt ./a.out Preloads $TAU/shared-papi-mpi-pdt/libTAU.so % mpirun -np 256 tau_exec –T papi ./a.out Preloads $TAU/shared-papi-mpi-pdt/libTAU.so by matching. % mpirun –np 256 tau_exec –T papi,mpi,pdt –s ./a.out Does not execute the program. Just displays the library that it will preload if executed without the –s option. NOTE: -mpi configuration is selected by default. Use –T serial for Sequential programs.
29 29http://tau.uoregon.edu/mug18.pdf
IntrospectingImpactofEagerThresholdon3DStencilBenchmark
30
Default Optimized
LessOverlap,LessUsefulworkdonebyApplication!
MoreOverlap,MoreUsefulworkdoneby
Application!
http://tau.uoregon.edu/mug18.pdf
Other Runtime Environment Variables EnvironmentVariable Default Description
TAU_TRACE 0 Settingto1turnsontracing
TAU_CALLPATH 0 Settingto1turnsoncallpathprofiling
TAU_TRACK_MEMORY_FOOTPRINT 0 Settingto1turnsontrackingmemoryusagebysamplingperiodicallytheresidentsetsizeandhighwatermarkofmemoryusage
TAU_SELECT_FILE Specifythepathtoruntimeselectiveinstrumentationfileforfilteringeventsusingexcludeandincludelistsofroutinesand/orfiles.
TAU_CALLPATH_DEPTH 2 Specifiesdepthofcallpath.Settingto0generatesnocallpathorroutineinformation,settingto1generatesflatprofileandcontexteventshavejustparentinformation(e.g.,HeapEntry:foo)
TAU_SAMPLING 0 Settingto1enablesevent-basedsampling.
TAU_TRACK_SIGNALS 0 Settingto1generatedebuggingcallstackinfowhenaprogramcrashes
TAU_COMM_MATRIX 0 Settingto1generatescommunicationmatrixdisplayusingcontextevents
TAU_THROTTLE 1 Settingto0turnsoffthrottling.Enabledbydefaulttoremoveinstrumentationinlightweightroutinesthatarecalledfrequently
TAU_THROTTLE_NUMCALLS 100000 Specifiesthenumberofcallsbeforetestingforthrottling
TAU_THROTTLE_PERCALL 10 Specifiesvalueinmicroseconds.Throttlearoutineifitiscalledover100000timesandtakeslessthan10usecofinclusivetimepercall
TAU_COMPENSATE 0 Settingto1enablesruntimecompensationofinstrumentationoverhead
TAU_PROFILE_FORMAT Profile Settingto“merged”generatesasinglefile.“snapshot”generatesxmlformat
TAU_METRICS TIME Settingtoacommaseparatedlistgeneratesothermetrics.(e.g.,TIME,ENERGY,PAPI_FP_INS,PAPI_NATIVE_<event>:<subevent>)
31http://tau.uoregon.edu/mug18.pdf
Runtime Environment Variables (contd.) EnvironmentVariable Default Description
TAU_TRACK_MEMORY_LEAKS 0 Tracksallocatesthatwerenotde-allocated(needs–optMemDbgortau_exec–memory)
TAU_EBS_SOURCE TIME AllowsusingPAPIhardwarecountersforperiodicinterruptsforEBS(e.g.,TAU_EBS_SOURCE=PAPI_TOT_INSwhenTAU_SAMPLING=1)
TAU_EBS_PERIOD 100000 Specifiestheoverflowcountforinterrupts
TAU_MEMDBG_ALLOC_MIN/MAX 0 Bytesizeminimumandmaximumsubjecttoboundschecking(usedwithTAU_MEMDBG_PROTECT_*)
TAU_MEMDBG_OVERHEAD 0 SpecifiesthenumberofbytesforTAU’smemoryoverheadformemorydebugging.
TAU_MEMDBG_PROTECT_BELOW/ABOVE
0 Settingto1enablestrackingruntimeboundscheckingbeloworabovethearraybounds(requires–optMemDbgwhilebuildingortau_exec–memory)
TAU_MEMDBG_ZERO_MALLOC 0 Settingto1enablestrackingzerobyteallocationsasinvalidmemoryallocations.
TAU_MEMDBG_PROTECT_FREE 0 Settingto1detectsinvalidaccessestodeallocatedmemorythatshouldnotbereferenceduntilitisreallocated(requires–optMemDbgortau_exec–memory)
TAU_MEMDBG_ATTEMPT_CONTINUE 0 Settingto1allowsTAUtorecordandcontinueexecutionwhenamemoryerroroccursatruntime.
TAU_MEMDBG_FILL_GAP Undefined Initialvalueforgapbytes
TAU_MEMDBG_ALINGMENT sizeof(int) Bytealignmentformemoryallocations
TAU_EVENT_THRESHOLD 0.5 Defineathresholdvalue(e.g.,.25is25%)totriggermarkereventsformin/max
32http://tau.uoregon.edu/mug18.pdf
DownloadTAUfromU.Oregon
http://www.hpclinux.com[OVAfile]http://tau.uoregon.edu/tau.tgz
formoreinformation
Freedownload,opensource,BSDlicense 33
http://tau.uoregon.edu/mug18.pdf
PRL, University of Oregon, Eugene
www.uoregon.edu
34http://tau.uoregon.edu/mug18.pdf
Support Acknowledgments US Department of Energy (DOE)
• ANL • Office of Science contracts, ECP • SciDAC, LBL contracts • LLNL-LANL-SNL ASC/NNSA contract • Battelle, PNNL and ORNL contract
CEA, France Department of Defense (DoD)
• PETTT, HPCMP National Science Foundation (NSF)
• SI2-SSI, Glassbox Intel Corporation NASA Partners:
• University of Oregon • The Ohio State University • ParaTools, Inc. • University of Tennessee, Knoxville • T.U. Dresden, GWT • Jülich Supercomputing Center
http://tau.uoregon.edu/mug18.pdf35