MANUFACTURINGWEATHER FORECASTING
SIMULATIONSON HPC INFRASTRUCTURES
LADISLAV HLUCHÝV. ŠIPKOVÁ, M. DOBRUCKÝ, J. BARTOK, B.M. NGUYEN
INSTITUTE OF INFORMATICS, SLOVAK ACADEMY OF SCIENCES
ECW 2016 - ENVIRONMENTAL COMPUTING WORKSHOP - ESCIENCE 2016
PARTNERS
• IISAS: INSTITUTE OF INFORMATICS, SLOVAK ACADEMY OF SCIENCES (ACADEMIC SECTOR)
• MICROSTEP-MIS: MONITORING AND INFORMATION SYSTEMS (COMMERCIAL SECTOR)
• IMS MODEL SUITE: COMPLEX SOFTWARE SYSTEM FOR METEOROLOGY AND CRISIS MANAGEMENT
• THIS PAPER PRESENTS A PART OF MANUFACTURING WRFON HPC INFRASTRUCTURE FOR IMS MODEL SUITE
WRF - WEATHER RESEARCH AND FORECASTING
• DESIGNED FOR RESEARCH AND OPERATIONAL PURPOSES
• NUMERICAL WEATHER PREDICTION
• ATMOSPHERIC SIMULATION
• TWO DYNAMIC SOLVERS
• ARW: ADVANCE RESEARCH WRF
• NMM: NON-HYDROSTATIC MESOSCALE MODEL
• FLEXIBLE AND PORTABLE CODE
• SEQUENTIAL
• PARALLEL (MPI) WITH OR WITHOUT MULTI-THREADING
• SUPPORTS A TWO-LEVEL DOMAIN DECOMPOSITION
• AT FIRST INTO PATCHES FOR DISTRIBUTED MEMORY,
• THEN WITHIN EACH PATCH MULTI-THREADING IS APPLIED FOR SHARED MEMORY
OBJECTIVES
• DEVELOPMENT OF MANAGEMENT TOOLS TO FACILITATE THE EXECUTION OF THE WRF SIMULATION PROCESS ON HPC INFRASTRUCTURES• LOCAL HPC CLUSTER
• GRID INFRASTRUCTURE (EGI)
• PERFORMANCE INVESTIGATION OF PARALLEL WRF MODELS TO FIND OUT THE MOST SUITABLE CONFIGURATION WITH THE GIVEN INPUT SCENARIO FOR 3D METEOROLOGICAL MODELLING• MPI
• MPI + OPENMP
• THE NUMBER OF COMPUTE NODES, CORES, MPI PROCESSES, OPENMP THREADS
• THE MANAGEMENT TOOLS ARE ALSO USED FOR PARAMETER TUNING OF THE MODELS (FOR IMS BY MICROSTEP-MIS) THAT REQUIRES • TENS OF EVALUATIONS OF THE PARAMETERIZED MODEL ACCURACY
• EACH EVALUATION OF THE MODEL PARAMETERS REQUIRES RE-RUNNING OF THE HUNDREDS OF METEOROLOGICAL SITUATIONS COLLECTED OVER THE YEARS AND COMPARISON OF THE MODEL OUTPUT WITH THE OBSERVED DATA
3D METEOROLOGICAL MODELLING• DOMAINS - WEATHER MODELLING
• HORIZONTAL, VERTICAL AND TIME RESOLUTION,
• SO THE MODEL CAN CATCH LOCAL CONDITIONS
• METEOROLOGICAL INITIAL AND BOUNDARY CONDITIONS
• FROM THE GLOBAL MODEL GFS (GLOBAL FORECASTING SYSTEM) OF US NATIONAL WEATHER SERVICE
• THE SETTING ENABLED TO MODEL THE ARABIAN PENINSULA WEATHER
• THE UPPERMOST DOMAIN WITH THE RESOLUTION 50X50 KM
• THE FINAL DOMAIN WITH THE RESOLUTION 1.8 KM, AROUND DUBAI AND ABU DHABI
WRF SIMULATION
• WRF SIMULATION CONSISTS OF MANY EXECUTABLE PROGRAMS
• VARIOUS TYPE AND COMPLEXITY, SEQUENTIAL AND PARALLEL
• TAKING A DIFFERENT NUMBER OF PROCESSOR CORES FOR EXECUTION
• WRF WORKFLOW - DAG GRAPH• (JOB 1) WPS PREPROCESSING,
• (JOB 2) WRF MODELING,
• (JOB 3) UPP POST-PROCESSING
Pi – MPI process Tj – OpenMP
thread
WRF WORKFLOW – MORE DETAILS• JOB 1 - WPS PREPROCESSING: CONVERSION OF INPUTS FROM GRIB TO NETCDF
FORMAT USING
• GEOGRID.EXE (SERIAL/MPI)
• UNGRIB.EXE (SERIAL)
• METGRID.EXE (SERIAL/MPI)
• JOB 2 - WRF MODELING - NUMERICAL MODELING USING
• REAL.EXE – INITIALIZATION - REAL DATA PREPROCESSOR (MPI/MPI+OPENMP)
• WRF.EXE – NUMERICAL INTEGRATION - ARW SOLVER (MPI/MPI+OPENMP)
• JOB 3 - UPP POST-PROCESSING
• CONVERSION OF OUTPUTS FROM NETCDF TO GRIB FORMAT USING UNIPOST.EXE(SERIAL/MPI) IN A NESTED CYCLE FOR ALL HOURS OF THE PREDICTED TIME PERIOD
• THERE IS NO DEPENDENCY BETWEEN PROCESSING DATA OF INDIVIDUAL HOURS, SO, THE JOB CAN BE STRUCTURED AS A PARAMETRIC STUDY (PS), WHERE EACH SUB-JOB HANDLES A SECTION OF THE TIME PERIOD
WRF WORKFLOW EXECUTION
• STARTS ON THE UI MACHINE THROUGH THE INVOCATION OF THE WRF WORKFLOW-MANAGER ENCOMPASSED WITH NEEDED INPUT PARAMETERS
• IS REALIZED WITHIN THE “RUNNING-ENVIRONMENT” LOCATED IN THE SHARED ADDRESS SPACE WHICH HAS THE DIRECTORY STRUCTURE• GEOG – GEOGRAPHICAL DATA, SEVERAL GEO-TABLES
• CFG – CONFIGURATION FILES FOR INPUT SCENARIO AND SIMULATION OPTIONS
• PARM – UPP POST-PROCESSING PARAMETERS
• BIN – RUN-SCRIPTS AND EXECUTABLES
• INPUT_ARCH – INPUT DATA FILES
• OUTPUT_ARCH – OUTPUT DATA FILES
• WPS_RUN – WPS PREPROCESSING
• MODEL_RUN – WRF MODELING
• POSTPR_RUN – UPP POST-PROCESSING
IISAS HPC CLUSTER
v HARDWARE CONFIGURATION
52X IBM DX360 M3 (2X INTEL E5645 @2.4GHZ, 48 GB RAM, 2X 500 GB SCRATCH DISK), 2X IBM DX360 M3 (2X INTEL E5645 @2.4GHZ, 48 GB RAM, 2X 500 GB SCRATCH DISK, NVIDIA TESLA M2070: 6 GB RAM + 448 CUDA CORES), 2X X3650 M3 MANAGING SERVERS (2X INTEL E5645 @2.4GHZ, 48 GB RAM, 6X 500 GB DISKS), 4X X3650 M3 DATA-MANAGING SERVERS (2X INTEL E5645 @2.4GHZ, 48 GB RAM, 2X 500 GB DISKS, 2X 8 GBPS FC), 1X X3550 M4 SERVER (1X INTEL E5-2640 @2.5GHZ, 8 GB RAM, 2X 500 GB DISKS), INFINIBAND 2X 40 GBPS (IN 52+2+2+4 NODES), 2X DS3512 WITH 72TB DISKS
v SOFTWARE INSTALATION
• WRF PACKAGE VERSION 3.7.1 (WRF, WPS, TERRESTRIAL DATASETS),
• UPP VERSION 3.0,
• LIBRARIES NETCDF 4, JASPER 1.7,
• GNU COMPILERS VERSION 4.4.7 (GFORTRAN, GCC, OPENMP LIBRARY),
• OPEN MPI VERSION 1.10.0
PERFORMANCE RESULTS WRF MODEL: SEQUENTIAL ON THE LOCAL CLUSTER
• PREDICTION TIME PERIOD
• 3 HOURS IN THIS PAPER FOR SCALING WRF SIMULATIONS FOR TESTING PURPOSE WITH GIVEN HW/SW CONFIGURATIONS
• 48 HOURS IN REAL SIMULATIONS (MICROSTEP-MIS) TO MODEL THE ARABIAN PENINSULA WEATHER
• THE NEED OF HPC TO ACCELERATE SIMULATIONS
Number ofnodes
Number ofcores per
node
Execution time
hh:mm:ssWPS 1 1 00:39:54WRF 1 1 15:57:53UPP (2 jobs) 1 1 00:03:48Complete simulation process 16:41:35
PERFORMANCE RESULTSWRF MODEL: MPI ON LOCAL CLUSTERFIXED NUMBER OF CORES PER NODE
Number of
nodes
Number of
cores per node
Number of
MPI processes
Execution time
hh:mm:ss
WPS 1 10 10 00:04:22WRF 1 8 8 02:36:33WRF 2 8 16 01:27:01WRF 4 8 32 00:49:03WRF 8 8 64 00:30:13WRF 16 8 128 00:20:47WRF 32 8 256 00:13:57UPP (2 jobs) 1 3 3 00:01:44Complete simulation process (best) 00:20:03
PERFORMANCE RESULTSWRF MODEL: MPI + OPENMP ON LOCAL CLUSTER
FIXED MPI PROCESSES
Number ofnodes x cores
Number ofMPI
processes( per node)
Number ofOpenMPthreads
Execution time
hh:mm:ss
WRF 8x12 32 (4) 2 00:31:31
WRF 16x12 32 (2) 4 00:20:52
WRF 16x12 32 (2) 6 00:17:21
WRF 32x12 32 (1) 8 00:15:47
WRF 32x12 32 (1) 10 00:15:15
WRF 32x12 32 (1) 12 00:15:20
PERFORMANCE RESULTSWRF MODEL: MPI + OPENMP ON LOCAL CLUSTER
FIXED NUMBER OF OPENMP THREADS
Number ofnodes x cores
Number ofMPI
processes(per node)
Number ofOpenMP threads
Execution time
hh:mm:ss
WRF 8x12 32 (4) 3 00:24:44
WRF 12x12 48 (4) 3 00:19:28
WRF 16x12 64 (4) 3 00:17:27
WRF 24x12 96 (4) 3 00:13:49
WRF 32x12 128 (4) 3 00:12:24
WRF 16x12 32 (2) 6 00:17:21
WRF 24x12 48 (2) 6 00:14:31
WRF 32x12 64 (2) 6 00:12:09
WRF 40x12 80 (2) 6 00:12:01
WRF MODELMPI ON GRID INFRASTRUCTURE EGI
• WRF RUNNING-ENVIRONMENT IN ITS INITIAL STATE, ALL EXECUTABLES AND INPUT FILES ARE STORED IN GRID STORAGE ELEMENT (SE), FROM WHICH THEY ARE DOWNLOADED
• GEOGRAPHICAL DATASETS (174 GB) ARE LOCATED IN CLUSTER SHARED ADDRESS SPACE, THEY DO NOT PARTICIPATE ON THE DATA TRANSFER
• GRID WRF WORKFLOW IS DESIGNED AS ONE GRID JOB ENCAPSULATING ALL TASKS: WPS+WRF+UPP
• MPI PROGRAMS ARE EXECUTED USING MPI-START
• OUTPUT OF SIMULATION IS UPLOADED TO STORAGE ELEMENT (SE)
• TIME OVERHEAD BY DATA TRANSFERS BETWEEN CE AND SE: 2 MINUTES
Grid UI – Grid User InterfaceWMS – Workload Management System VO – Virtual OrganizationCE – Computing ElementsGG – Grid GateLRMS – Local Resource Management SystemWN – Working NodeSE – Storage Element
PBS – Portable Batch System
CONCLUSION
• MANAGEMENT TOOLS ARE BUILT AND FULFILL DESIGNED PURPOSES
• TO LOCATE THE OPTIMAL CONFIGURATION WITH GIVEN SCENARIO
• FOR IMS MODEL PARAMETER TUNING (MICROSTEP-MIS)
• HYBRID PROGRAMMING MODEL (MPI + OPENMP) SEEMS A NATURAL FIT FOR THE WAY MOST CLUSTERS ARE BUILT TODAY
• THE GRID OVERHEAD IS CAUSED MAINLY BY THE TRANSFER OF BIG FILES BETWEEN THE SE AND CE
FUTURE DIRECTIONS
ØGRID
• AT THE MOMENT, IN EUROPEAN GRID INFRASTRUCTURE (EGI), ONLY A FEW GRID SITES AND VIRTUAL ORGANIZATIONS (VO) ARE SUPPORTING MPI ANDOPENMP APPLICATIONS
ØCLOUD
• PERFORMANCE OVERHEAD ASSOCIATED WITH VIRTUALIZATION OF INTERCONNECTION NETWORK
• WRF IS REPORTED TO RUN ON VIRTUALIZED INFINIBAND INTERCONNECT WITH ONLY 15% OVERHEAD WHICH MAKES FULLY VIRTUALIZED HPC CLUSTERS VIABLE SOLUTION
ØACCELERATORS
• PARTS OF WRF WERE PORTED TO NVIDIA GPU AND INTEL XEON PHI WITH PROMISING RESULTS
THANK YOU FOR YOUR ATTENTION
MANUFACTURING WEATHER FORECASTING SIMULATIONS ON HPC INFRASTRUCTURES
INSTITUTE OF INFORMATICS, SLOVAK ACADEMY OF SCIENCES
ECW 2016 - ENVIRONMENTAL COMPUTING WORKSHOP - ESCIENCE 2016