Overview of Software Environment
on Fugaku Hitoshi MuraiRIKEN R-CCS
June 30, 2021 16th meeting for application code tuning on
A64FX computer systems.
Introduction• Programming Env. WG in FS2020 project
• responsible for designing the programming env. of Fugaku.• languages & compilers• tools (performance profiler, debugger, etc.)
• based on the knowledge learned from our experience of the K computer and requests from the pilot users.
• Goals:• higher performance, efficiency, and usability of Fugaku.• contribution to the software ecosytem of Arm processors.
June 30, 2021 26th meeting for application code tuning on
A64FX computer systems.
Basic Concept of the Design• The vendor (Fujitsu) provides essential software.
• compilers• libraries (e.g. MPI, BLAS)• tools
• We provide Open Source Software (OSS).• as supplements or alternatives to vendor-provided ones.• through Spack
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 3
Languages & Compilers (for Fugaku)
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 4
Version Languages OpenMP TypeFujitsu 4.5.0 F2008, C11, C++14 4.0+ native & crossGCC 8 & 10† F2008-, C17, C++17+ 4.5+ native & WIP for crossLLVM 10‡ & 11+† F2008-, C11, C++17+ 4.0+ native
† provided through Spack‡ available on the Arm compile node
MPI
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 5
Spec. Thread Support Level Optimized for Fugaku Compiler Note
Fujitsu MPI 3.1+ MPI_THREAD_SERIALIZED Yes Fujitsu or GCC† based on Open MPI
MPICH on Tofu 3.0 MPI_THREAD_MULTIPLE Not fully Fujitsu (WIP) or GCCprovided as a Spackpackage "mpich-tofu"
† See Section 4 of Supercomputer Fugaku Users Guide - Language and development environment - Version 1.05.
Math Library• Fujitsu provides:
• SSL II / C-SSL II (Fujitsu Scientific Subroutine Library II),• BLAS / LAPACK / ScaLAPACK,• High-speed quadruple precision basic arithmetic library.• They are fully optimized for Fugaku.
• We provide many others through Spack, including:• FFTW (fujitsu-fftw), optimized for Fugaku,• PETSC,• EigenExa,• etc.
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 6
Vendor-provided Tools• Integrated Development Environment (based on Eclipse)• Performance profilers
• Instant Performance Profiler (FIPP) ― profiles overall performance based on sampling.
• Advanced Performance Profiler (FAPP) ― profiles regional performance based on manual instrumentation of programs.
• CPU Performance Analysis Report ― visualizes FAPP outputs as tables and graphs in MS Excel.
• Debugging features• abnormal termination inspection• deadlock inspection• scripted debugging
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 7
Other Tools• OSS (ported and maintained in collaborative works)
• Score-P/Scalasca @ JSC ― performance instrumentation, measurement, trace analyzer (WIP)
• TAU @ U. Oregon ― profiling and tracing toolkit for performance analysis of parallel programs (WIP)
• VeloC @ ANL ― very low overhead transparent multilevel checkpoint/restart
• Commercial Software• Vampir• Arm Forge (WIP)
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 8
Visualization Tools• The following usage scenarios (tools) for visualization are
available on Fugaku.• CLI (Command Line Interface)
ImageMagick, POV-Ray, GrADS• GUI (Graphical User Interface)
ImageMagick, Ncview, ParaView• Distributed (Client / Server)
ParaView• VNC (GNOME Desktop)
• Refer to Fugaku Visualization User Guide for more detail.
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 9
Basic Policy about OSS on Fugaku• We select and provide ones from the most widely-used
OSS packages;• Users are able to install packages for themselves if needed.• The above two are achieved with Spack.• Best effort support:
• No specific supports for packages are provided in principle.• Packages are provided as-is (not optimized).• not fully tested.
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 10
Spack• A package management tool designed for large
supercomputing centers.• Developed by LLNL in Exascale Computing Project (ECP).• Features:
• supporting multiple versions and configurations of software on a wide variety of platforms and environments including Fugaku (A64FX),
• >5000 recipes of packages available,• handling dependencies between packages.
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 11
spack-mirror.r-ccs.riken.jp/oss/public/
For the compatibility of OSS with Fugaku (A64FX), visit:
Spack on Fugaku (1)• Basic usage
• System Spack (public instance) + User Spack (private instance)
• The operators provides important or widely-used packages in the system Spack.
• Users can add their necessary packages in their own Spack.• These two instances can be "chained."
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 12
$ . /vol0004/apps/oss/spack/share/spack/setup-env.sh$ spack load xxx
// setup Spack// load a package XXX
Spack on Fugaku (2)• Some very important packages (e.g. FFTW) have been
optimized for Fugaku.• Fore more detail, refer to the documents available on the
Fugaku portal.• Resource – Use of OSS• Resource – Manuals – User's Guide – Fugaku Spack User Guide
• Important Notice:• Spack does not support cross compilation yet!• You have to install packages for Fugaku on the compute nodes.
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 13
Available Packages on Compute Nodes
June 30, 2021 14
Name Versionadios2 2.6.0
alps 2.3.0bcftools 1.10.2
bedtools2 2.27.1blitz 1.0.1boost 1.74.0
bwa 0.7.17cblas 2015-06-06cmake 3.18.4dssp 3.1.4
eigenexa 2.6fujitsu-fftw masterfujitsu-mpi 4.3.1
fujitsu-ssl2 4.3.1gcc 10.2.0
genesis 1.5.1grads 2.2.1
gromacs 2021.1hdf5 1.10.7
htslib 1.10.2julia 1.5.2
kokkos 3.1.01lammps 20201029
Name Versionlis 2.0.27
llvm 11.0.0lz4 1.9.2
mapslice2 2.2.1metis 5.1.0
mpich-tofu mastermptensor 0.3.0
n2p2 2.1.1netcdf-c 4.7.4
netcdf-cxx 4.2netcdf-cxx4 4.3.1
netcdf-fortran 4.5.3netlib-
scalapack 2.1.0
openblas 0.3.12openfoam 2006
openfoam-org 8openjdk 11.0.0-2020-01-01
parallel-netcdf 1.12.1paraview 5.8.1parmetis 4.0.3
petsc 3.14.1pfapack 41899picard 2.20.8povray 3.7.0.8
Name Versionprocess-in-
process 2 & 3
py-dask 2.16.0py-h5py 2.10.0
py-jupyterhub 1.0.0py-keras 2.2.4
py-mpi4py 3.0.3py-netcdf4 1.5.3
py-numpy 1.19.4py-pydmd 0.3py-pygps 1.3.5py-pysam 0.15.2
py-scikit-learn 0.23.2py-scipy 1.5.4
py-seaborn 0.9.0py-xarray 0.14.0
quantum-espresso 6.6r 4.0.3
raja 0.12.1samtools 1.1
screen 4.8.0star 2.7.6atmux 3.1bxios 2.5
Note: many other ones on which these packages are depend are also available.
R-CCS Software & Supported Software• R-CCS Software
• Process-in-Process (PiP)• MPICH on Tofu• Omni XcalableMP• EigenExa• GENESIS• NTChem• SCALE (TBA)• McKernel (TBA)
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 15
• Supported Software• ABINIT-MP
Fore more detail of each software, visit the page on the Fugaku portal.
AI Frameworks• PyTorch 1.7.0
• to be updated to the latest around September.• TensorFlow 2.2.0
• to be updated to the latest around September.• Chainer 4.5.0
• These are fully optimized for Fugaku in the DL4Fugaku project and currently provided w/o Spack.• Please contact to the Fugaku support desk if interested.
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 16
• Vendor-provided and open-source software complement each other to achieve higher efficiency and usability of Fugaku.• OSS packages are available through Spack, which allows
both the operators and users to easily manage them.• For some specific applications (e.g. visualization & deep
learning), we provide and/or optimize a set of software.
June 30, 20216th meeting for application code tuning on
A64FX computer systems. 17
Summary