211
CASTRO An adaptive, parallel, radiation hydrodynamics code for self-gravitating astrophysical flows User’s Guide November 14, 2018
CASTRO
An adaptive, parallel, radiation hydrodynamics codefor self-gravitating astrophysical flows
User’s Guide
November 14, 2018
Chapter Listing
list of figures xii
list of tables xiv
preface xv
1 Introduction 1
2 Getting Started 3
3 Inputs Files 9
4 Software Framework 21
5 Single-Level Flow Chart 45
6 Runtime Parameters 59
7 Compilers, Managing Jobs, and Scaling 77
8 Frequently Asked Questions 79
9 Hydrodynamics 85
10 Gravity 103
11 Diffusion 113
iii
iv CHAPTER LISTING
12 Rotation 117
13 Radiation 123
14 Particles 135
15 Equation of State and Burning Network 139
16 AMR 143
17 ConvertCheckpoint 151
18 Initializing Castro with Maestro Data 157
19 Visualization 161
20 Verification Test Problems 175
References 187
Index 191
Contents
list of figures xii
list of tables xiv
preface xv
1 Introduction 1
1.1 Introduction to Castro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Units and Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 Getting Started 3
2.1 Downloading the Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Building the Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Running the Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.4 Visualization of the Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.5 Other Distributed Problem Setups . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3 Inputs Files 9
3.1 Working with probin Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Common inputs Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.1 Problem Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.2 Domain Boundary Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2.3 Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2.4 Regridding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2.5 Simulation Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.2.6 Time Step . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.2.7 Subcycling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.2.8 Restart Capability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.9 Controlling Plotfile Generation . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2.10 Screen Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
v
vi CONTENTS
3.2.11 Other parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4 Software Framework 21
4.1 Code structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2 Major data structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2.1 Amr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.2 AmrLevel and Castro classes . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.3 Floating point data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.4 Box and FArrayBox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.5 MultiFab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.2.6 StateData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.2.7 Various source MultiFabs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.3 MFIter and interacting with Fortran . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.3.1 Non-Tiling MFIter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.3.2 AMReX’s Current Tiling Approach In C++ . . . . . . . . . . . . . . . . . . . 30
4.4 Boundaries: FillPatch and FillPatchIterator . . . . . . . . . . . . . . . . . . . . 34
4.4.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.4.2 Physical Boundaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.4.3 FluxRegister . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.5 Other AMReX Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.5.1 Geometry class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.5.2 ParmParse class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.5.3 Error Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.6 Gravity class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.7 Fortran Helper Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.8 Setting Up Your Own Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.8.1 Optional Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.8.2 Dimension Agnostic Problem Initialization . . . . . . . . . . . . . . . . . . . 43
4.9 Parallel I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5 Single-Level Flow Chart 45
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.2 Overview of a single step (no SDC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.2.1 Main Hydro, Reaction, and Gravity Advancement (CTU w/ Strang-splitting) 48
5.2.2 Main Hydro, Reaction, and Gravity Advancement (MOL w/ Strang-splitting) 52
5.3 Overview of a single step (with SDC) . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.3.1 Main Hydro and Gravity Advancement (SDC) . . . . . . . . . . . . . . . . . 56
6 Runtime Parameters 59
6.1 Introduction to Runtime Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
6.1.1 C++ parameter format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
6.1.2 Microphysics/extern parameter format . . . . . . . . . . . . . . . . . . . . . . 60
6.2 Removed Runtime Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6.3 castro Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6.4 diffusion Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6.5 gravity Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
6.6 particles Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
CONTENTS vii
7 Compilers, Managing Jobs, and Scaling 77
7.1 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
8 Frequently Asked Questions 79
8.1 Compiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
8.2 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
8.3 Profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
8.4 Managing Runs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
8.5 Runtime Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
8.6 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
9 Hydrodynamics 85
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
9.2 Hydrodynamics Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
9.3 Conservation Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
9.4 Source Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
9.5 Primitive Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
9.5.1 Internal energy and temperature . . . . . . . . . . . . . . . . . . . . . . . . . 90
9.5.2 Primitive Variable System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
9.6 Hydrodynamics Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
9.6.1 Compute Primitive Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
9.6.2 Edge State Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
9.6.3 Riemann Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
9.6.4 Compute Fluxes and Update . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
9.7 Temperature Fixes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
9.8 Resets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
9.8.1 Density Resets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
9.8.2 Energy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
9.8.3 Flux Limiting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
10 Gravity 103
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
10.1.1 Integration Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
10.1.2 Controls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
10.2 Types of Approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
10.2.1 ConstantGrav . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
10.2.2 MonopoleGrav . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
10.2.3 PoissonGrav . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
10.2.4 PrescribedGrav . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
10.2.5 Point Mass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
10.3 GR correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
10.4 Hydrodynamics Source Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
11 Diffusion 113
11.1 Thermal Diffusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
11.2 Enthalpy Diffusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
11.3 Species Diffusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
viii CONTENTS
11.4 Viscosity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
12 Rotation 117
12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
12.2 Coordinate transformation to rotating frame . . . . . . . . . . . . . . . . . . . . . . 118
12.3 Momentum equation in rotating frame . . . . . . . . . . . . . . . . . . . . . . . . . . 120
12.4 Energy equations in rotating frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
12.5 Switching to the rotating reference frame . . . . . . . . . . . . . . . . . . . . . . . . 121
12.6 Adding the forcing to the hydrodynamics . . . . . . . . . . . . . . . . . . . . . . . . 121
13 Radiation 123
13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
13.2 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
13.2.1 Getting the Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
13.3 Microphysics: EOS, Network, and Opacity . . . . . . . . . . . . . . . . . . . . . . . . 124
13.3.1 EOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
13.3.2 Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
13.3.3 Opacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
13.4 Radiation Solver Physics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
13.4.1 Verbosity and I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
13.4.2 Flux Limiter and Closure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
13.4.3 Boundary Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
13.4.4 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
13.4.5 Parameters for Gray Solver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
13.4.6 Grouping in the MG Solver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
13.4.7 Parameters for MG Solver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
13.4.8 Linear System Solver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
13.5 Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
13.5.1 Gray Solver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
14 Particles 135
14.1 Tracer particles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
14.2 Initializing the Particles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
14.3 Output file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
14.3.1 Run-time Screen Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
15 Equation of State and Burning Network 139
15.1 Equation of State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
15.1.1 Standard Castro EOSes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
15.1.2 EOS Interfaces and Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 139
15.2 Nuclear Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
15.2.1 Required Thermodynamics Quantities . . . . . . . . . . . . . . . . . . . . . . 141
16 AMR 143
16.1 Tagging for Refinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
16.2 Synchronization Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
16.2.1 Synchronization Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
CONTENTS ix
16.2.2 Source Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
16.2.3 Synchronization Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
17 ConvertCheckpoint 151
17.1 Star in Corner (star at center = 0) . . . . . . . . . . . . . . . . . . . . . . . . . . 151
17.1.1 Converting the Checkpoint File . . . . . . . . . . . . . . . . . . . . . . . . . . 151
17.1.2 Restarting from a Grown Checkpoint File . . . . . . . . . . . . . . . . . . . . 152
17.2 Star at Center of Domain (star at center = 1) . . . . . . . . . . . . . . . . . . . . 153
17.2.1 Converting the Checkpoint File . . . . . . . . . . . . . . . . . . . . . . . . . . 153
17.2.2 Restarting from a Grown Checkpoint File . . . . . . . . . . . . . . . . . . . . 154
17.2.3 Cylindrical Coordinates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
18 Initializing Castro with Maestro Data 157
18.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
18.2 MAESTRO Plotfile Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
18.3 List of Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
18.3.1 Examples of Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
18.4 New Subroutines in Prob Xd.f90 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
18.5 Additional Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
18.5.1 Multilevel Restart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
19 Visualization 161
19.1 Controlling What’s in the PlotFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
19.2 amrvis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
19.3 VisIt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
19.4 yt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
19.4.1 Castro-Specific Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
19.4.2 Interacting with yt: Command Line and Scripting . . . . . . . . . . . . . . . 163
19.4.3 yt Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
19.4.4 Data Containers and Selection . . . . . . . . . . . . . . . . . . . . . . . . . . 165
19.4.5 Grid Inspection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
19.4.6 Example Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
20 Verification Test Problems 175
20.1 Hydrodynamics Test Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
20.1.1 Sod’s Problem (and Other Shock Tube Problems) . . . . . . . . . . . . . . . 175
20.1.2 Sedov Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
20.1.3 Rayleigh-Taylor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
20.2 Gravity Test Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
20.3 Radiation Test Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
20.3.1 Light Front . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
20.3.2 Diffusion of a Gaussian Pulse . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
20.3.3 Radiation Source Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
20.3.4 Radiating Sphere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
20.4 Regression Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
References 187
x CONTENTS
Index 191
List of Figures
4.1 Single-level grid structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.2 A simple domain showing 4 Boxes labeled 0–3, and their tiling regions (dotted lines) 30
12.1 Inertial frame C and non-inertial frame C. We consider a fluid element P , whosedistance in the two frames is related by r = r + l . . . . . . . . . . . . . . . . . . . . 119
14.1 A model atmosphere (left panel) and the trajectories of 500 particles (right panel)following the fluid motion on the atmosphere. The particles are initially positionedat five different heights, y = 13000 km, 11000 km, 8000 km, 6000 km, 38000 km(100 particles at each height). In the left panel, the arrows roughly show the fluidmotion. In the right panel, the solid lines represent the trajectories of the particles. 137
17.1 Data from checkpoint file before and after the domain has been coarsened and grown.This case uses star at center = 0 and ref ratio=2. The first grown example hasgrown factor=2, the second has grown factor=3. In all figures the level 0 gridsare shown in white, the level 1 grids in red, the level 2 grids in yellow, and in thegrown figures, the level 3 grids are in pink. . . . . . . . . . . . . . . . . . . . . . . . . 155
17.2 Data from checkpoint file before and after the domain has been coarsened and grown.This case uses star at center = 0 and ref ratio=2. The first grown example hasgrown factor=2, the second has grown factor=3. In all figures the level 0 gridsare shown in white, the level 1 grids in red, the level 2 grids in yellow, and in thegrown figure, the level 3 grids are in pink. . . . . . . . . . . . . . . . . . . . . . . . 156
19.1 Pressure slice with annotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
19.2 Volume rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
19.3 Pressure isocontour rendering colored with velocity magnitude . . . . . . . . . . . . 171
19.4 1-D velocity magnitude profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
19.5 Density/velocity magnitude/kinetic energy phase plot . . . . . . . . . . . . . . . . . 173
19.6 Off-axis density projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
xi
xii LIST OF FIGURES
20.1 Castro solution for Sod’s problem run in 3-d, with the newest ppm limiters, alongthe x, y, and z axes. A coarse grid of 32 zones in the direction of propagation, with2 levels of refinement was used. The analytic solution appears as the red line. . . . . 176
20.2 Castro solution for Sod’s problem run in 3-d, with the piecewise-linear Godunovmethod with limiters, along the x, y, and z axes. A coarse grid of 32 zones in thedirection of propagation, with 2 levels of refinement was used. The analytic solutionappears as the red line. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
20.3 Castro solution for the double rarefaction problem run in 3-d, along the x, y, andz axes. A coarse grid of 32 zones in the direction of propagation, with 2 levels ofrefinement was used. The analytic solution appears as the red line. . . . . . . . . . . 178
20.4 Castro solution for the strong shock problem run in 3-d, along the x, y, and z axes.A coarse grid of 32 zones in the direction of propagation, with 2 levels of refinementwas used. The analytic solution appears as the red line. . . . . . . . . . . . . . . . . 179
20.5 Castro solution for the Sedov blast wave problem run in 1-d spherical, 2-d axisym-metric, and 3-d Cartesian coordinates. Each of these geometries produces a sphericalSedov explosion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
20.6 Castro solution for the Sedov blast wave problem run in 2-d Cartesian coordinates.This corresponds to a cylindrical Sedov explosion. . . . . . . . . . . . . . . . . . . . 182
20.7 Rayleigh-Taylor with different PPM types. . . . . . . . . . . . . . . . . . . . . . . . . 18320.8 Castro solution for radiating source test problem. Heating and cooling solutions are
shown as a function of time, compared to the analytic solution. The gray photonsolver was used. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
20.9 Castro solution for radiating sphere problem, showing the radiation energy densityas a function of energy group. This test was run with 64 photon energy groups. . . . 185
List of Tables
1.1 Common quantities and units. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
4.1 Physics boundary conditions in Castro . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6.1 castro : AMR parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616.2 castro : diagnostics, I/O parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . 616.3 castro : diffusion parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626.4 castro : embiggening parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 636.5 castro : gravity and rotation parameters . . . . . . . . . . . . . . . . . . . . . . . . . 636.6 castro : hydrodynamics parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646.7 castro : parallelization parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676.8 castro : particles parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676.9 castro : reactions parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686.10 castro : refinement parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 696.11 castro : timestep control parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . 696.12 diffusion parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 726.13 gravity parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746.14 particles parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
9.1 The integer variables to index the conservative state array . . . . . . . . . . . . . . . 869.2 The integer variable keys for accessing the primitive state vector. Note, unless oth-
erwise specified the quantities without a subscript are “gas” only and those with the“tot” subscript are “gas + radiation”. . . . . . . . . . . . . . . . . . . . . . . . . . . 86
9.3 The integer variable keys for accessing the auxiliary primitive state vector, quax.Note, unless otherwise specified the quantities without a subscript are “gas” onlyand those with the “tot” subscript are “gas + radiation”. . . . . . . . . . . . . . . . 87
9.4 The integer variable keys for accessing the Godunov interface state vectors. Note,unless otherwise specified the quantities without a subscript are “gas” only and thosewith the “tot” subscript are “gas + radiation”. . . . . . . . . . . . . . . . . . . . . . 87
20.1 Sedov inputs files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
xiii
xiv LIST OF TABLES
20.2 Analysis routines for Sedov . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
Preface
Welcome to the Castro User’s Guide!
In this User’s Guide we describe how to download and run Castro, a massively parallel code thatsolves the multicomponent compressible hydrodynamic equations for astrophysical flows includingself-gravity, nuclear reactions and radiation. Castro uses an Eulerian grid and incorporates adaptivemesh refinement (AMR). Our approach to AMR uses a nested hierarchy of logically-rectangulargrids with simultaneous refinement in both space and time, utilizing the AMReX library1.
The core algorithms in Castro are described in a series of papers:
• CASTRO: A New Compressible Astrophysical Solver. I. Hydrodynamics and Self-gravity,A. S. Almgren, V. E. Beckner, J. B. Bell, M. S. Day, L. H. Howell, C. C. Joggerst, M. J. Li-jewski, A. Nonaka, M. Singer, & M. Zingale, 2010, ApJ, 715, 1221http://dx.doi.org/10.1088/0004-637X/715/2/1221
• CASTRO: A New Compressible Astrophysical Solver. II. Gray Radiation Hydrodynamics,W. Zhang, L. Howell, A. Almgren, A. Burrows, & J. Bell, 2011, ApJS, 196, 20http://dx.doi.org/10.1088/0067-0049/196/2/20
• CASTRO: A New Compressible Astrophysical Solver. III. Multigroup Radiation Hydrody-namics, W. Zhang, L. Howell, A. Almgren, A. Burrows, J. Dolence, & J. Bell, 2013, ApJS,204, 7http://dx.doi.org/10.1088/0067-0049/204/1/7
Improvements to the gravity solver and rotation were described in:
• Double White Dwarf Mergers on Adaptive Meshes I. Methodology and Code Verification,M. P. Katz, M. Zingale, A. C. Calder, F. D. Swesty, A. S. Almgren, W. Zhang 2016, ApJ,819, 94.http://dx.doi.org/10.3847/0004-637X/819/2/94
1earlier versions of Castro used the BoxLib library
xv
xvi Chapter 0. Preface
The development of AMReX library is led by the Center for Computational Sciences and Engineering/ Lawrence Berkeley National Laboratory. Castro development is done collaboratively, includingthe CCSE and Stony Brook University.
Castro core developers are those who have made substantial contributions to the code. The processfor becoming a core developer is described in the README.md in the Castro root directory. CurrentCastro core developers are:
Ann AlmgrenMaria G. Barrios SazoJohn BellVince BecknerMarc DayMax KatzMike LijewskiChris MaloneAndy NonakaDon WillcoxWeiqun ZhangMichael Zingale
All Castro development takes place on the project’s github page
https://github.com/AMReX-Astro/Castro
External contributions are welcomed. Fork the Castro repo, modify your local copy, and issue apull-request to the AMReX-Astro/Castro project. Further guidelines are given in the README.md
file.
To get help, subscribe to the castro-help google group mailing list: https://groups.google.com/forum/#!forum/castro-help
Acknowledging and Citing Castro
If you use Castro in your research, we would appreciate it if you cited the relevant code papersdescribing its design, features, and testing. A list of these can be found in the CITATION file in theroot Castro/ directory.
The development Castro is supported by the science application interests of the contributors. Thereis a lot of effort behind the scenes: testing, optimization, development of new features, bug fixing,. . ., that is often done under the radar. Nevertheless, we are happy to volunteer our time to helpnew users come up to speed with Castro. When significant new development / debugging for youapplication is provided by a member of the Castro development community, we would appreciateconsideration of inviting the developer(s) for co-authorship on any science paper that results.
CHAPTER 1
Introduction
1.1 Introduction to Castro
Castro is a adaptive mesh, radiation hydrodynamics code that is designed to model astrophysicalreacting flows on massively parallel computers.
The major capabilities:
• 1-, 2-, and 3-dimensional unsplit, 2nd-order hydrodynamics
• multigroup flux-limited diffusion radiation hydrodynamics
• adaptive mesh refinement with subcycling; jumps of 2x and 4x between levels
• arbitrary equation of state (gamma-law and stellar EOSes are bundled)
• general nuclear reaction networks
• explicit thermal diffusion
• full Poisson gravity (with isolated boundary conditions)
• rotation (in the co-rotating frame) in 2-d axisymmetric and 3-d.
• parallelization via MPI + OpenMP
1.2 Units and Conventions
Castro works in CGS units unless otherwise specified. Table 1.1 shows some of the common symbols/ names used throughout the code documentation and papers.
Physical constants, again using the CGS system are available in Castro/constants/constants cgs.f90
1
2 Chapter 1. Introduction
name units description
t s time
ρ g cm−3 mass density
u cm s−1 velocity vector
p dyn cm−2 pressure
g cm s−2 gravitational acceleration
S varies source term
E erg g−1 specific total energy
e erg g−1 specific internal energy
T K temperature
kth erg cm−1 s−1 K −1 thermal conductivity
Xk – mass fraction of species k
ωk s−1 species creation rate (from reactions)
Table 1.1: Common quantities and units.
CHAPTER 2
Getting Started
2.1 Downloading the Code
Castro is built on top of the AMReX framework. In order to run Castro you must download twoseparate git modules.
First, make sure that git is installed on your machine—we recommend version 1.7.x or higher.
1. Clone/fork the AMReX repository from the AMReX-Codes github page (https://github.com/AMReX-Codes/amrex/). To clone via the command line, simply type:
git clone https://github.com/AMReX-Codes/amrex.git
Alternately, if you have a github account with your machine’s SSH-keys registered, you cando:
git clone ssh://[email protected]/AMReX-Codes/amrex.git
This will create a directory called amrex/ on your machine.
You will want to periodically update AMReX by typing
git pull
in the amrex/ directory.
Note: actively development is done on the development branch in each repo, and mergedinto the master branch periodically. If you wish to use the Castro development branch, thenyou should also switch to the development branch for AMReX.
2. Set the environment variable, AMREX HOME, on your machine to point to the path name whereyou have put AMReX. You can add this to your .bashrc as:
3
4 Chapter 2. Getting Started
export AMREX_HOME= /path/to/amrex/
where you replace /path/to/amrex/ will the full path to the amrex/ directory.
3. Clone/fork the Castro repository from the same github organization as above, using eitherHTTP access:
git clone https://github.com/AMReX-Astro/Castro.git
or SSH access if you have it enabled:
git clone ssh://[email protected]:/AMReX-Astro/Castro.git
Or, as above, you can download a ZIP file of the code from our main github page, by clickingon the Castro link.
As with AMReX, development on Castro is done in the development branch, so you shouldwork there if you want the latest source.
4. We recommend setting the CASTRO HOME environment variable to point to the path namewhere you have put Castro. Add the following to your .bashrc:
export CASTRO_HOME="/path/to/Castro/"
5. (optional) An additional repository, Microphysics.git is available at the starkiller-astrogithub page. This add additional reaction networks and EOSes and can be cloned followingthe same procedure as above1:
git clone https://github.com/starkiller-astro/Microphysics.git
or via SSH as
git clone ssh://[email protected]:/starkiller-astro/Microphysics.git
To access the Microphysics routines, set the MICROPHYSICS HOME environment variable to pointto the Microphysics/ directory.
2.2 Building the Code
In Castro each different problem setup is stored in its own sub-directory under Castro/Exec/. Youbuild the Castro executable in the problem sub-directory. Here we’ll build the Sedov problem:
1. From the directory in which you checked out the Castro git repo, type
cd Castro/Exec/hydro_tests/Sedov
This will put you into a directory in which you can run the Sedov problem in 1-d, 2-d or 3-d.
2. In Sedov/, edit the GNUmakefile, and set
• DIM = 2
This is the dimensionality—here we pick 2-d.
1Note: previously the radiation solver was distributed separately as CastroRadiation.git, but this has beenmerged into the main Castro respository
2.3—Running the Code 5
• COMP = gnu
This is the set of compilers. gnu are a good default choice (this will use g++ andgfortran. You can also choose pgi and intel for example.
If you want to try other compilers than the GNU suite and they don’t work, please letus know.
• DEBUG = FALSE
This disabled debugging checks and results in a more optimized executable.
• USE MPI = FALSE
This turns off parallelization via MPI. Set it to TRUE to build with MPI—this requiresthat you have the MPI library installed on your machine. In this case, the build systemwill need to know about your MPI installation. This can be done by editing the makefilesin the AMReX tree, but the default fallback is to look for the standard MPI wrappers(e.g. mpic++ and mpif90) to do the build.
3. Now type make.
The resulting executable will look something like Castro2d.Linux.gnu.ex, which means thisis a 2-d version of the code, made on a Linux machine, with COMP = gnu.
2.3 Running the Code
1. Castro takes an input file that overrides the runtime parameter defaults. The code is run as:
Castro2d.Linux.gcc.gfortran.ex inputs.2d.cyl_in_cartcoords
This will run the 2-d cylindrical Sedov problem in Cartesian (x-y coordinates). You can seeother possible options, which should be clear by the names of the inputs files.
2. You will notice that running the code generates directories that look like plt00000/,plt00020/, etc, and chk00000/, chk00020/, etc. These are “plotfiles” and “checkpoint”files. The plotfiles are used for visualization, the checkpoint files are used for restarting thecode.
2.4 Visualization of the Results
There are several options for visualizing the data. The popular VisIt package supports the AMReXfile format natively, as does the yt python package2. The standard tool used within the AMReX-community is Amrvis, which we demonstrate here. Amrvis is available on github.
1. Get Amrvis:
git clone https://github.com/AMReX-Codes/Amrvis
2Each of these will recognize it as the BoxLib format.
6 Chapter 2. Getting Started
Then cd into Amrvis/, edit the GNUmakefile there to set DIM = 2, and again set COMP tocompilers that you have. Leave DEBUG = FALSE.
Type make to build, resulting in an executable that looks like amrvis2d...ex.
If you want to build amrvis with DIM = 3, you must first download and build volpack:
git clone https://ccse.lbl.gov/pub/Downloads/volpack.git
Then cd into volpack/ and type make.
Note: Amrvis requires the OSF/Motif libraries and headers. If you don’t have these youwill need to install the development version of motif through your package manager. Onmost Linux distributions, the motif library is provided by the openmotif package, and itsheader files (like Xm.h) are provided by openmotif-devel. If those packages are not installed,then use the package management tool to install them, which varies from distribution todistribution, but is straightforward. lesstif gives some functionality and will allow you tobuild the amrvis executable, but Amrvis may not run properly.
You may then want to create an alias to amrvis2d, for example
alias amrvis2d /tmp/Amrvis/amrvis2d...ex
where /tmp/Amrvis/amrvis2d...ex is the full path and name of the Amrvis executable.
2. Configure Amrvis:
Copy the amrvis.defaults file to your home directory (you can rename it to .amrvis.defaultsif you wish). Then edit the file, and change the palette line to point to the full path/filenameof the Palette file that comes with Amrvis.
3. Visualize:
Return to the Castro/Exec/hydro tests/Sedov directory. You should have a number ofoutput files, including some in the form *pltXXXXX, where XXXXX is a number correspondingto the timestep the file was output. amrvis2d filename to see a single plotfile, or amrvis2d-a *plt*, which will animate the sequence of plotfiles.
Try playing around with this—you can change which variable you are looking at, select aregion and click “Dataset” (under View) in order to look at the actual numbers, etc. You canalso export the pictures in several different formats under ”File/Export”.
Some users have found that Amrvis does not work properly under X with the proprietaryNvidia graphics driver. A fix for this is provided in the FAQ (§ 8.6)—this is due to thedefault behavior of the DAC in mappuing colors.
Note: yt is a great alternative to using Amrvis for visualization, and understands Castroplotfiles well.
Please know that we do have a number of conversion routines to other formats (such asmatlab), but it is hard to describe them all. If you would like to display the data in anotherformat, please let us know (again, [email protected]) and we will point you to whateverwe have that can help.
You have now completed a brief introduction to Castro.
2.5—Other Distributed Problem Setups 7
2.5 Other Distributed Problem Setups
There are a number of standard problem setups that come with Castro. These can be used as astarting point toward writing your own setup. We organize these into subdirectories by broad type(radiation, hydro, gravity, etc.): The standard categories and some of the included problems are:
• gravity tests:
– DustCollapse:
A pressureless cloud collapse that is a standard test problem for gravity. An analyticsolution that describes the radius of the sphere as a function of time is found in Colgateand White [10]. This problem is also found in the FLASH User’s Guide.
– hydrostatic adjust:
Model a 1-d stellar atmosphere (plane-parallel or spherical/self-gravitating) and dumpenergy in via an analytic heat source and watch the atmosphere’s hydrostatic state adjustin response. This is the counterpart to the Maestro test basestate unit test.
• hydro tests:
– double bubble:
Initialize 1 or 2 bubbles in a stratified atmosphere (isothermal or isentropic) and allowfor the bubbles to have the same or a different γ from one another / the backgroundatmosphere. This uses the multigamma EOS.
An analogous problem is implemented in Maestro.
– HCBubble:
– KH:
A Kelvin-Helmholtz shear instability problem.
– oddeven:
A grid-aligned shock hitting a very small density perturbation. This demonstratesthe odd-even decoupling problem discussed in [22]. This setup serves to test thecastro.hybrid riemann option to hydrodynamics.
– reacting bubble:
A reacting bubble in a stratified white dwarf atmosphere. This problem was featured inthe Maestro reaction paper [2].
– RT:
A single-model Rayleigh-Taylor instability problem.
– RT particles:
– Sedov:
The standard Sedov-Taylor blast wave problem. This setup was used in the first Castropaper [1].
8 Chapter 2. Getting Started
– Sod:
A one-dimensional shock tube setup, including the classic Sod problem. This setup wasused in the original Castro paper.
– Sod stellar:
A version of the Sod shock tube for the general stellar equation of state. This setup andthe included inputs files was used in [28].
– toy convect:
A simple nova-like convection problem with an external heating source. This problemshows how to use the model parser to initialize a 1-d atmosphere on the Castro grid,incorporate a custom tagging routine, sponge the fluid above the atmosphere, and writea custom diagnostics routine.
A Maestro version of this problem setup also exists.
• radiation tests:
• science:
• unit tests:
CHAPTER 3
Inputs Files
The Castro executable uses two inputs files at runtime to set and alter the behavior of the algorithmand initial conditions.
The main inputs file, typically named inputs is used to set BoxLib parameters and the control flowin the C++ portions of the Castro code. Each parameter here has a namespace (like amr.optionnameor castro.optionname). Parameters set here are read using the BoxLib ParmParse class infras-tructure.
The second inputs file, typically named probin is used by the Fortran code that initializes theproblem setup. It is read at problem initialization (via a Fortran namelist) and the problem-specificquantities are stored in a Fortran module probdata module defined in the problem’s probdata.f90file.
Only the inputs file is specified on the commandline. The associated probin file is specified in theinputs file using the amr.probin file parameter, e.g.,
amr.probin_file = my_special_probin
for example, has the Fortran code read a file called my special probin.
3.1 Working with probin Files
There are three different Fortran namelists that can be defined in the probin file:
• &fortin is the main namelist read by the problem’s probinit subroutine in the Prob ?d.f90
file.
• &extern is used to set different microphysics options
9
10 Chapter 3. Inputs Files
• &tagging is used to get the parameters (defined in tagging module) that affect how we tagfor refinement.
3.2 Common inputs Options
Important: because the inputs file is handled by the C++ portion of the code, any quantitiesyou specify in scientific notation, must take the form 1.e5 and not 1.d5—the ‘d’ specifier is notrecognized.
Additionally, note that in Castro, all quantities are in CGS units.
3.2.1 Problem Geometry
The geometry namespace is used by BoxLib to define the computational domain. The main pa-rameters here are:
• geometry.prob lo: physical location of low corner of the domain (type: Real; must be set)
Note: a number is needed for each dimension in the problem
• geometry.prob hi: physical location of high corner of the domain (type: Real; must be set)
Note: a number is needed for each dimension in the problem
• geometry.coord sys: coordinate system, 0 = Cartesian, 1 = r-z (2-d only), 2 = spherical(1-d only) (must be set)
• geometry.is periodic: is the domain periodic in this direction? 0 if false, 1 if true (default:0 0 0)
Note: an integer is needed for each dimension in the problem
• castro.center: physical location of problem center on the domain (type: Real; default: 0.00.0 0.0). The problem center is used for gravity, rotation, and some other quantities. Thisis not necessarily the geometric center of the domain—often you should choose it to coincidewith the center of mass of your system. See § 4.7 for more details.
Note: a number is needed for each dimension in the problem
As an example, the following:
geometry.prob_lo = 0 0 0
geometry.prob_hi = 1.e8 2.e8 2.e8
geometry.coord_sys = 0
geometry.is_periodic = 0 1 0
castro.center = 5.e7 1.e8 1.e8
This defines the domain to run from (0, 0, 0) at the lower left to (108, 2 × 108, 2 × 108) at theupper right in physical space, specifies a Cartesian geometry, and makes the domain periodic inthe y-direction only. The problem center is set to be halfway in between the lower left and upperright corners.
3.2—Common inputs Options 11
3.2.2 Domain Boundary Conditions
Boundary conditions are specified using integer keys that are interpreted by BoxLib. The runtimeparameters that we use are:
• castro.lo bc: boundary type of each low face (must be set)
• castro.hi bc: boundary type of each high face (must be set)
The valid boundary types are:
0 – Interior / Periodic 3 – Symmetry1 – Inflow 4 – Slip Wall2 – Outflow 5 – No Slip Wall
Note: castro.lo bc and castro.hi bc must be consistent with geometry.is periodic—if thedomain is periodic in a particular direction then the low and high bc’s must be set to 0 for thatdirection.
As an example, the following:
castro.lo_bc = 1 4 0
castro.hi_bc = 2 4 0
geometry.is_periodic = 0 0 1
This defines a problem with inflow (1) in the low-x direction, outflow (2) in the high-x direction,slip wall (4) on the low and high y-faces, and periodic in the z-direction. See § 4.4.2 for moreinformation.
3.2.3 Resolution
The grid resolution is specified by defining the resolution at the coarsest level (level 0) and thenumber of refinement levels and factor of refinement between levels. The relevant parameters are:
• amr.n cell: number of cells in each direction at the coarsest level (Integer > 0; must be set)
• amr.max level: number of levels of refinement above the coarsest level (Integer ≥ 0; mustbe set)
• amr.ref ratio: ratio of coarse to fine grid spacing between subsequent levels (2 or 4; mustbe set)
• amr.regrid int: how often (in terms of number of steps) to regrid (Integer; must be set)
• amr.regrid on restart: should we regrid immediately after restarting? (0 or 1; default: 0)
Note: if amr.max level = 0 then you do not need to set amr.ref ratio or amr.regrid int.
Some examples:
amr.n_cell = 32 64 64
12 Chapter 3. Inputs Files
would define the domain to have 32 cells in the x-direction, 64 cells in the y-direction, and 64 cellsin the z-direction at the coarsest level. (If this line appears in a 2D inputs file then the final numberwill be ignored.)
amr.max_level = 2
would allow a maximum of 2 refined levels in addition to the coarse level. Note that these ad-ditional levels will only be created only if the tagging criteria are such that cells are flagged asneeding refinement. The number of refined levels in a calculation must be ≤ amr.max level, butcan change in time and need not always be equal to amr.max level.
amr.ref_ratio = 2 4
would set factor of 2 refinement between levels 0 and 1, and factor of 4 refinement between levels1 and 2. Note that you must have at least amr.max level values of amr.ref ratio (Additionalvalues may appear in that line and they will be ignored).
amr.regrid_int = 2 2
tells the code to regrid every 2 steps. Thus in this example, new level 1 grids will be createdevery 2 level-0 time steps, and new level 2 grids will be created every 2 level-1 time steps. Ifamr.regrid int < 0 for any level, then regridding starting at that level will be disabled. Ifamr.regrid int = -1 only, then we never regrid for any level. Note that this is not compatiblewith amr.regrid on restart = 1.
3.2.4 Regridding
The details of the regridding strategy are described in § 16.1; here we cover how the input parameterscan control the gridding.
As described later, the user defines Fortran subroutines which tag individual cells at a given levelif they need refinement. This list of tagged cells is sent to a grid generation routine, which uses theBerger-Rigoutsos algorithm [3] to create rectangular grids that contain the tagged cells.
The relevant runtime parameters are:
• amr.regrid file: name of file from which to read the grids (text; default: no file)
If set to a filename, e.g. fixed girds, then list of grids at each fine level are read in from thisfile during the gridding procedure. These grids must not violate the amr.max grid size cri-terion. The rest of the gridding procedure described below will not occur if amr.regrid file
is set.
• amr.n error buf: radius of additional tagging around already tagged cells (Integer ≥ 0;default: 1)
• amr.max grid size: maximum size of a grid in any direction (Integer > 0; default: 128 (2-d),32 (3-d))
Note: amr.max grid size must be even, and a multiple of amr.blocking factor at everylevel.
3.2—Common inputs Options 13
• amr.blocking factor: grid size must be a multiple of this (Integer > 0; default: 2)
Note: amr.blocking factor at every level must be a power of 2 and the domain size mustbe a multiple of amr.blocking factor at level 0.
This can be very important for elliptic problems with multigrid. A higher blocking factorallows the multigrid algorithm to coarsen more at the lowest level, reducing the amount ofwork required by the bottom solver.
• amr.grid eff: grid efficiency (Real > 0 and < 1; default: 0.7)
When creating a refined grid, do we make boxes that only include the coarse cells that wereexplicitly tagged for refinement? or do we allow ourselves to encompass nearby, untaggedcells in order to make larger and more regular boxes? This is the grid efficiency.
When blocking factor = 1, grid efficiency is exactly the fraction of refined cells in the fineBoxArray which correspond to coarse cells which were tagged. For other blocking factors, weactually apply grid eff at the level which has been coarsened by blocking factor, so it isno longer strictly this fraction, but the idea is still the same.
• amr.refine grid layout: refine grids more if # of processors > # of grids (0 if false, 1 iftrue; default: 1)
Note also that amr.n error buf, amr.max grid size and amr.blocking factor can be read in asa single value which is assigned to every level, or as multiple values, one for each level.
As an example, consider:
amr.grid_eff = 0.9
amr.max_grid_size = 64
amr.blocking_factor = 32
The grid efficiency, amr.grid eff, means that during the grid creation process, at least 90% of thecells in each grid at the level at which the grid creation occurs must be tagged cells. A higher gridefficiency means fewer cells at higher levels, but may result in the production of lots of small grids,which have inefficient cache and OpenMP performance and higher communication costs.
The amr.max grid size parameter means that the final grids will be no longer than 64 cellson a side at every level. Alternately, we could specify a value for each level of refinement as:amr.max grid size = 64 32 16, in which case our final grids will be no longer than 64 cellson a side at level 0, 32 cells on a side at level 1, and 16 cells on a side at level 2. Theamr.blocking factor means that all of the final grids will be multiples of 32 at all levels. Again,this can be specified on a level-by-level basis, like amr.blocking factor = 32 16 8, in which casethe dimensions of all the final grids will be multiples of 32 at level 0, multiples of 16 at level 1, andmultiples of 8 at level 2.
3.2.4.1 Getting good performance
These parameters can have a large impact on the performance of Castro, so taking the time toexperiment with is worth the effort. Having grids that are large enough to coarsen multiple levelsin a V-cycle is essential for good multigrid performance in simulations that use self-gravity.
14 Chapter 3. Inputs Files
Need more experience here
3.2.4.2 How grids are created
The gridding algorithm proceeds in this order:
1. Grids are created using the Berger-Rigoutsos clustering algorithm modified to ensure that allnew fine grids are divisible by amr.blocking factor.
2. Next, the grid list is chopped up if any grids are larger than max grid size. Note thatbecause amr.max grid size is a multiple of amr.blocking factor the amr.blocking factor
criterion is still satisfied.
3. Next, if amr.refine grid layout = 1 and there are more processors than grids, and ifamr.max grid size / 2 is a multiple of amr.blocking factor, then the grids will be re-defined, at each level independently, so that the maximum length of a grid at level `, in anydimension, is amr.max grid size[`] / 2.
4. Finally, if amr.refine grid layout = 1, and there are still more processors than grids, andif amr.max grid size / 4 is a multiple of amr.blocking factor, then the grids will beredefined, at each level independently, so that the maximum length of a grid at level `, in anydimension, is amr.max grid size[`] / 4.
3.2.5 Simulation Time
There are two paramters that can define when a simulation ends:
• max step: maximum number of level 0 time steps (Integer ≥ 0; default: -1)
• stop time: final simulation time (Real ≥ 0; default: -1.0)
To control the number of time steps, you can limit by the maximum number of level 0 time steps(max step) or by the final simulation time (stop time), or both. The code will stop at whichevercriterion comes first.
Note that if the code reaches stop time then the final time step will be shortened so as to endexactly at stop time, not past it.
As an example:
max_step = 1000
stop_time = 1.0
will end the calculation when either the simulation time reaches 1.0 or the number of level 0 stepstaken equals 1000, whichever comes first.
3.2.6 Time Step
If castro.do hydro = 1, then typically the code chooses a time step based on the CFL number:
∆t = CFL · mini,j,k
[min
∆x
|u|i,j,k + ci,j,k,
∆y
|v|i,j,k + ci,j,k,
∆z
|w|i,j,k + ci,j,k
](3.1)
3.2—Common inputs Options 15
If method-of-lines integration is used instead, then we have
∆t = CFL · mini,j,k
[(∆x
|u|i,j,k + ci,j,k
)−1
+
(∆y
|v|i,j,k + ci,j,k
)−1
+
(∆z
|w|i,j,k + ci,j,k
)−1]−1
(3.2)
(If we are simulating in 1D or 2D, the extraneous parts related to v and/or w are removed.)
The following parameters affect the timestep choice:
• castro.cfl: CFL number (Real > 0 and ≤ 1; default: 0.8)
• castro.init shrink: factor by which to shrink the initial time step (Real > 0 and ≤ 1;default: 1.0)
• castro.change max: factor by which the time step can grow in subsequent steps (Real ≥ 1;default: 1.1)
• castro.fixed dt: level 0 time step regardless of cfl or other settings (Real > 0; unused ifnot set)
• castro.initial dt: initial level 0 time step regardless of other settings (Real > 0; unusedif not set)
• castro.dt cutoff: time step below which calculation will abort (Real > 0; default: 0.0)
• castro.hard cfl limit: whether or not to abort the simulation if the hydrodynamics updatecreates velocities that violate the CFL criterion (Integer; default: 1)
As an example, consider:
castro.cfl = 0.9
castro.init_shrink = 0.01
castro.change_max = 1.1
castro.dt_cutoff = 1.e-20
This defines the cfl parameter in Eq. 3.1 to be 0.9, but sets (via init shrink) the first timestepwe take to be 1% of what it would be otherwise. This allows us to ramp up to the hydrodynamictimestep at the start of a simulation. The change max parameter restricts the timestep fromincreasing by more than 10% over a coarse timestep. Note that the time step can shrink by anyfactor; this only controls the extent to which it can grow. The dt cutoff parameter will force thecode to abort if the timestep ever drops below 10−20. This is a safety feature—if the code hits sucha small value, then something likely went wrong in the simulation, and by aborting, you won’t burnthrough your entire allocation before noticing that there is an issue.
If we know what we are doing, then we can force a particular timestep:
castro.fixed_dt = 1.e-4
This sets the level 0 time step to be 1.e-4 for the entire simulation, ignoring the other timestepcontrols. Note that if castro.init shrink 6= 1 then the first time step will in fact becastro.init shrink · castro.fixed dt.
castro.initial_dt = 1.e-4
16 Chapter 3. Inputs Files
sets the initial level 0 time step to be 10−4 regardless of castro.cfl or castro.fixed dt. Thetime step can grow in subsequent steps by a factor of castro.change max each step.
[DIFFUSION] If diffusion is enabled, the timestep will also be limited by:
∆t =1
2mini,j,k
[min
∆x2
Di,j,k,
∆y2
Di,j,k,
∆z2
Di,j,k
](3.3)
where D ≡ k/(ρcV ) if we are diffusing temperature, and D ≡ k/(ρcP ) if we are diffusing enthalpy.No input parameter is necessary to enable this constraint. See Chapter 11 for more details.
[REACTIONS] If reactions are enabled, the timestep will also be limited by two constraints:
∆t = dtnuc e mini,j,k
ei,j,kei,j,k
(3.4)
∆t = dtnuc X mini,j,k
minn
Xni,j,k
Xni,j,k
(3.5)
where e is the internal energy, and Xn is the mass fraction of the nth species. The safety factorscorrespond to the runtime parameters castro.dtnuc e and castro.dtnuc X. These limiters saythat the timestep must be small enough so that no zone can change its internal energy by more thanthe fraction dtnuc e in one step, and so that no zone can change the abundance of any isotope bymore than the fraction dtnuc X in one step. The time derivatives e and Xn are estimated by callingthe right-hand-side of the nuclear network given the state at the time the timestep limiter is beingcalculated. (We use a small number floor to prevent division by zero.) To prevent the timestepfrom being dominated by trace species, there is an additional option castro.dtnuc X threshold
which is the mass fraction threshold below which a species will not be considered in the timestepconstraint. dtnuc e and dtnuc X are set to a large number by default, effectively disabling them.Typical choices for these values in the literature are ∼ 0.1.
3.2.7 Subcycling
Castro supports a number of different modes for subcycling in time, set via amr.subcycling mode.
• amr.subcycling mode = Auto (default): the code will run with equal refinement in spaceand time. In other words, if level n + 1 is a factor of 2 refinement above level n, then n + 1will take 2 steps of half the duration for every level n step.
• If amr.subcycling mode = None: the code will not refine in time. All levels will advancetogether with a timestep dictated by the level with the strictest dt. Note that this is identicalto the deprecated command amr.nosub = 1.
• If amr.subcycling mode = Manual: the code will subcycle according to the values suppliedby amr.subcycling iterations.
In the case of amr.subcycling mode = Manual, we subcycle in manual mode with largest allowabletimestep. The number of iterations at each level is then specified as:
amr.subcycling_iterations = 1 2 1 2
3.2—Common inputs Options 17
Here, we take 1 level-0 timestep at a time (required). Take 2 level-1 timesteps for each level-0 step,1 timestep at level-2 for each level-1 step, and take 2 timesteps at level-3 for each level-2 step.
Alternately, we could do:
amr.subcycling_iterations = 2
which will subcycle twice at every level (except level 0).
3.2.8 Restart Capability
Castro has a standard sort of checkpointing and restarting capability. In the inputs file, the followingoptions control the generation of checkpoint files (which are really directories):
• amr.check file: prefix for restart files (text; default: chk)
• amr.check int: how often (by level 0 time steps) to write restart files (Integer > 0; default:-1)
• amr.check per: how often (by simulation time) to write restart files (Real > 0; default: -1.0)
Note that amr.check per will write a checkpoint at the first timestep whose ending time ispast an integer multiple of this interval. In particular, the timestep is not modified to matchthis interval, so you won’t get a checkpoint at exactly the time you requested.
• amr.restart: name of the file (directory) from which to restart (Text; not used if not set)
• amr.checkpoint files output: should we write checkpoint files? (0 or 1; default: 1)
If you are doing a scaling study then set amr.checkpoint files output = 0 so you can testscaling of the algorithm without I/O.
• amr.check nfiles: how parallel is the writing of the checkpoint files? (Integer ≥ 1; default:64)
See the § 4.9 for more details on parallel I/O and the amr.check nfiles parameter.
• amr.checkpoint on restart: should we write a checkpoint immediately after restarting? (0or 1; default: 0)
• castro.grown factor: factor by which domain has been grown (Integer ≥ 1; default: 1)
Note:
• You can specify both amr.check int or amr.check per, if you so desire; the code will printa warning in case you did this unintentionally. It will work as you would expect – you willget checkpoints at integer multiples of amr.check int timesteps and at integer multiples ofamr.check per simulation time intervals.
• amr.plotfile on restart and amr.checkpoint on restart require amr.regrid on restart
to be in effect.
As an example,
amr.check_file = chk_run
amr.check_int = 10
18 Chapter 3. Inputs Files
means that restart files (really directories) starting with the prefix “chk run” will be generated every10 level-0 time steps. The directory names will be chk run00000, chk run00010, chk run00020,etc.
If instead you specify
amr.check_file = chk_run
amr.check_per = 0.5
then restart files (really directories) starting with the prefix “chk run” will be generated every 0.1
units of simulation time. The directory names will be chk run00000, chk run00043, chk run00061,etc, where t = 0.1 after 43 level-0 steps, t = 0.2 after 61 level-0 steps, etc.
To restart from chk run00061, for example, then set
amr.restart = chk_run00061
3.2.9 Controlling Plotfile Generation
The main output from Castro is in the form of plotfiles (which are really directories). The followingoptions in the inputs file control the generation of plotfiles:
• amr.plot file: prefix for plotfiles (text; default: “plt”)
• amr.plot int: how often (by level-0 time steps) to write plot files (Integer > 0; default: -1)
• amr.plot per: how often (by simulation time) to write plot files (Real > 0; default: -1.0)
Note that amr.plot per will write a plotfile at the first timestep whose ending time is pastan integer multiple of this interval. In particular, the timestep is not modified to match thisinterval, so you won’t get a checkpoint at exactly the time you requested.
• amr.plot vars: name of state variables to include in plotfiles (valid options: ALL, NONE or alist; default: ALL)
• amr.derive plot vars: name of derived variables to include in plotfiles (valid options: ALL,NONE or a list; default: NONE
• amr.plot files output: should we write plot files? (0 or 1; default: 1)
If you are doing a scaling study then set amr.plot files output = 0 so you can test scalingof the algorithm without I/O.
• amr.plotfile on restart: should we write a plotfile immediately after restarting? (0 or 1;default: 0)
• amr.plot nfiles: how parallel is the writing of the plotfiles? (Integer ≥ 1; default: 64)
See the Software Section for more details on parallel I/O and the amr.plot nfiles parameter.
• castro.plot X: include all the species mass fractions in the plotfile (0 or 1; default: 0)
All the options for amr.derive plot vars are kept in derive lst in Castro setup.cpp. Feel freeto look at it and see what’s there.
Some notes:
3.2—Common inputs Options 19
• You can specify both amr.plot int or amr.plot per, if you so desire; the code will printa warning in case you did this unintentionally. It will work as you would expect – youwill get plotfiles at integer multiples of amr.plot int timesteps and at integer multiples ofamr.plot per simulation time intervals.
As an example:
amr.plot_file = plt_run
amr.plot_int = 10
means that plot files (really directories) starting with the prefix “plt run” will be generated every10 level-0 time steps. The directory names will be plt run00000, plt run00010, plt run00020,etc.
If instead you specify
amr.plot_file = plt_run
amr.plot_per = 0.5
then restart files (really directories) starting with the prefix “plt run” will be generated every 0.1units of simulation time. The directory names will be plt run00000, plt run00043, plt run00061,etc, where t = 0.1 after 43 level-0 steps, t = 0.2 after 61 level-0 steps, etc.
3.2.10 Screen Output
There are several options that set how much output is written to the screen as Castro runs:
• amr.v: verbosity of Amr.cpp (0 or 1; default: 0)
• castro.v: verbosity of Castro.cpp (0 or 1; default: 0)
• gravity.v: verbosity of Gravity.cpp (0 or 1; default: 0)
• diffusion.v: verbosity of Diffusion.cpp (0 or 1; default: 0)
• mg.v: verbosity of multigrid solver (for gravity) (allow values: 0,1,2,3,4; default: 0)
• amr.grid log: name of the file to which the grids are written (text; not used if not set)
• amr.run log: name of the file to which certain output is written (text; not used if not set)
• amr.run log terse: name of the file to which certain (terser) output is written (text; notused if not set)
• amr.sum interval: if > 0, how often (in level-0 time steps) to compute and print integralquantities (Integer; default: -1)
The integral quantities include total mass, momentum and energy in the domain everycastro.sum interval level-0 steps. The print statements have the form
TIME= 1.91717746 MASS= 1.792410279e+34
for example. If this line is commented out then it will not compute and print these quanitities.
20 Chapter 3. Inputs Files
• castro.do special tagging: allows the user to set a special flag based on user-specifiedcriteria (0 or 1; default: 1)
castro.do special tagging = 1 can be used, for example, to calculate the bounce time ina core collapse simulation; the bounce time is defined as the first time at which the maximumdensity in the domain exceeds a user-specified value. This time can then be printed into aspecial file as a useful diagnostic.
As an example:
amr.grid_log = grdlog
amr.run_log = runlog
Every time the code regrids it prints a list of grids at all relevant levels. Here the code will writethese grids lists into the file grdlog. Additionally, every time step the code prints certain statementsto the screen (if amr.v = 1), such as:
STEP = 1 TIME = 1.91717746 DT = 1.91717746
PLOTFILE: file = plt00001
The run log option will output these statements into runlog as well.
Terser output can be obtained via:
amr.run_log_terse = runlogterse
This file, runlogterse differs from runlog, in that it only contains lines of the form
10 0.2 0.005
in which “10” is the number of steps taken, “0.2” is the simulation time, and “0.005” is the level-0time step. This file can be plotted very easily to monitor the time step.
3.2.11 Other parameters
There are a large number of solver-specific runtime parameters. We describe these together withthe discussion of the physics solvers in later chapters.
CHAPTER 4
Software Framework
4.1 Code structure
Castro is built upon the AMReX C++ framework. This provides high-level classes for managing anadaptive mesh refinement simulation, including the core data structures we will deal with. A keydesign pattern in AMReX is that the overall memory management and parallelization is done inthe C++ layer, while the heavy computational work is done in Fortran kernels. AMReX providesconvenient data structures that allow for this workflow—high level objects in C++ that communicatewith Fortran through pointers to data regions that appear as multidimensional arrays.
Castro uses a structured-grid approach to hydrodynamics. We work with square/cubic zones thathold state variables (density, momentum, etc.) and compute the fluxes of these quantities throughthe interfaces of the zones (this is a finite-volume approach). Parallelization is achieved by domaindecomposition. We divide our domain into many smaller boxes, and distributed these across pro-cessors. When information is needed at the boundaries of the boxes, messages are exchanged andthis data is held in a perimeter of ghost cells. AMReX manages this decompostion and communica-tion for us. Additionally, AMReX implements adaptive mesh refinement. In addition to the coarsedecomposition of our domain into zones and boxes, we can refine rectangular regions by addingfiner-gridded boxes on top of the coarser grid. We call the collection of boxes at the same resolutiona level.
Castro uses a hybrid MPI + OpenMP approach to parallelism. MPI is at used to communicateacross nodes on a computer and OpenMP is used within a node, to loop over subregions of a boxwith different threads.
The code structure in the Castro/ directory reflects the division between C++ and Fortran.
• constants/: contains a file of useful constants in CGS units
• Docs/: you’re reading this now!
21
22 Chapter 4. Software Framework
• Exec/: various problem implementations, sorted by category:
– gravity tests/: test problems that primarily exercise the gravity solver
– hydro tests/: test problems of the hydrodynamics (with or without reactions)
– radiation tests/: test problems that primarily exercise the radiation hydrodynamicssolver
– science/: problem setups that were used for scientific investigations
– unit tests/: test problems that exercise primarily a single module
• Microphysics/: contains directories for different default microphysics (these are all imple-mented in Fortran)
– conductivity/: the thermal conductivity
– EOS/: the equation of state
– networks/: the nuclear reaction networks
– opacity/: the radiative opacity (used with radiation)
– viscosity/: the viscous transport coefficient
• Source/: source code. In this main directory is all of the code. Sources are mixed C++ andFortran and are organized by topic as:
– diffusion/ : thermal diffusion code
– driver/ : the main driver, I/O, runtime parameter support
– gravity/ : self-gravity code
– hydro/ : the compressible hydrodynamics code
– particles/ : support for particles
– problems/ : template code for implementing a problem
– radiation/ : the implicit radiation solve code
– reactions/ : nuclear reaction code
– rotation/ : rotating code
– sources/ : hydrodynamics source terms support
• Util/: a catch-all for additional things you may need
– ConvertCheckpoint/: a tool to convert a checkpoint file to a larger domain
– . . .
4.2 Major data structures
The following data structures are the most commonly encountered when working in the C++ portionsof Castro. This are all AMReX data-structures / classes.
4.2—Major data structures 23
4.2.1 Amr
This is the main class that drives the whole simulation. This is the highest level in Castro.
4.2.2 AmrLevel and Castro classes
An AmrLevel is a virtual base class provided by AMReX that stores all the state data on a singlelevel in the AMR hierarchy and understands how to advance that data in time.
The most important data managed by the AmrLevel is an array of StateData, which holds thefluid quantities, etc., in the boxes that together make up the level.
The Castro class is derived from the AmrLevel. It provides the Castro-specific routines to evolveour system of equations. Like the AmrLevel, there is one Castro object for each level in the AMRhierarchry.
A lot of the member data in the Castro class are static member variables—this means that they areshared across all instances of the class. So, in this case, every level will have the same data. This isdone, in particular, for the values of the runtime parameters, but also for the Gravity, Diffusion,and Radiation objects. This means that those objects cover all levels and are the same object ineach instantiation of the Castro class.
4.2.3 Floating point data
Floating point data in the C++ AMReX frame work is declared as Real. This is typedef to eitherfloat or double depending on the make variable PRECISION.
The corresponding type for Fortran is provided by the bl fort module as c real. We typicallyrename this to rt when using it. An example of a declaration of a parameter is:
use amrex_fort_module , only : rt => amrex_real
real(rt) :: tol = 1.0e-10 _rt
The bl constants module provides common constants that can be used in the code, like ZERO,THIRD, ONE, etc.
Note: single precision support in Castro is not yet complete. In particular, a lot of the supportingmicrophysics has not been updated.
4.2.4 Box and FArrayBox
A Box is simply a rectangular region in space. It does not hold data. In AMReX, an AMR levelhas a global index space, with (0, 0, 0) being the lower left corner of the domain at that level, and(Nx− 1, Ny − 1, Nz − 1) being the upper right corner of the domain (for a domain of Nx×Ny ×Nz
zones). The location of any Box at a level can be uniquely specified with respect to this global indexspace by giving the index of its lower-left and upper-right corners. Figure 4.1 shows an example ofthree boxes at the same level of refinement.
24 Chapter 4. Software Framework
Figure 4.1: Three boxes that comprise a single level. At this resolution, the domain is 20×18zones. Note that the indexing in AMReX starts with 0.
AMReX provides other data structures that collect Boxes together, most importantly the BoxArray.We generally do not use these directly, with the exception of the BoxArray grids, which is definedas part of the AmrLevel class that Castro inherits. grids is used when building new MultiFabsto give the layout of the boxes at the current level.
A FArrayBox or FAB, for Fortran array box is a data structure that contains a Box locating itin space, as well as a pointer to a data buffer. The real floating point data are stored as one-dimensional arrays in FArrayBoxes. The associated Boxcan be used to reshape the 1D array intomulti-dimensional arrays to be used by Fortran subroutines. The key part of the C++ AMReXdata structures is that this data buffer can be sent to Fortran, where it will appear as a DIM+1dimensional array (DIM space + 1 component).
Note: Castro is complied for a specific dimensionality.
4.2.5 MultiFab
At the highest abstraction level, we have the MultiFab (mulitple FArrayBoxes). A MultiFab
contains an array of Boxes, including Boxes owned by other processors for the purpose of commu-nication, an array of MPI ranks specifying which MPI processor owns each Box, and an array ofpointers to FArrayBoxes owned by this MPI processor. Note: a MultiFab is a collection of the
is this still anaccuratedescription?
boxes that together make up a single level of data in the AMR hierarchy.
A MultiFab can have multiple components (like density, temperature, ...) as well as a perimeter ofghost cells to exchange data with neighbors or implement boundary conditions (this is all reflectedin the underlying FArrayBox).
4.2—Major data structures 25
Parallelization in AMReX is done by distributing the FABs across processors. Each processor knowswhich FABs are local to it. To loop over all the boxes local to a processor, an MFIter is used (moreon this below).
High-level operations exist on MultiFabs to add, subtract, multiply, etc., them together or withscalars, so you don’t need to write out loops over the data directly.
In Castro, MultiFabs are one of the main data structures you will interact with in the C++ portionsof the code.
4.2.6 StateData
StateData is a class that essentially holds a pair of MultiFabs: one at the old time and oneat the new time. AMReX knows how to interpolate in time between these states to get data atany intermediate point in time. The main data that we care about in Castro (the fluid state,gravitational potential, etc.) will be stored as StateData. Essentially, data is made StateData inCastro if we need it to be stored in checkpoints / plotfiles, and/or we want it to be automaticallyinterpolated when we refine.
An AmrLevel stores an array of StateData (in a C++ array called state). We index this arrayusing integer keys (defined via an enum in Castro.H). The state data is registered with AMReX inCastro setup.cpp.
Note that each of the different StateData carried in the state array can have different numbersof components, ghost cells, boundary conditions, etc. This is the main reason we separate all thisdata into separate StateData objects collected together in an indexable array.
The current StateData names Castro carries are:
• State Type : this is the NUM STATE hydrodynamics components that make up the conservedhydrodynamics state (usually referred to as U in these notes. But note that this does notinclude the radiation energy density.
In Fortran, the components of a FAB derived from State Type is indexed using the integerkeys defined in Castro nd.F90 and stored in meth params module, e.g., URHO, UMX, UMY, ...
Note: regardless of dimensionality, we always carry around all three velocity components. The“out-of-plane” components will simply be advected, but we will allow rotation (in particular,the Coriolis force) to affect them.
State Type MultiFabs have no ghost cells by default for pure hydro and a single ghost cellby default when RADIATION is enabled. There is an option to force them to have ghost cellsby setting the parameter castro.state nghost at runtime.
Note that the prediction of the hydrodynamic state to the interface will require 4 ghost cells.This accomodated by creating a separate MultiFab, Sborder that lives at the old-time leveland has the necessary ghost cells. We will describe this more later.
• Rad Type : this stores the radiation energy density, commonly denoted Er in these notes.It has nGroups components—the number of energy groups used in the multigroup radiationhydrodynamics approximation.
not sure howneutrinos factor inhere
• PhiGrav Type : this is simply the gravitational potential, usually denoted Φ in these notes.
26 Chapter 4. Software Framework
• Gravity Type : this is the gravitational acceleration. There are always 3 components, re-gardless of the dimensionality (consistent with our choice of always carrying all 3 velocitycomponents).
• PhiRot Type : this is the rotational potential. When rotation is enabled, this will store theeffective potential corresponding to the centrifugal force.
• Rotation Type : this is the rotational acceleration. There are always 3 components, re-gardless of the dimensionality (consistent with our choice of always carrying all 3 velocitycomponents). This includes the terms corresponding to the Coriolis force, the centrifugalforce, as well as optional terms due to the change in rotation rate, Ω.
• Source Type : this holds the time-rate of change of the source terms, dS/dt, for each of theNUM STATE State Type variables.
SDC doesdifferently
Note: we do not make use of the old-time quantity here. In fact, we never allocate theFArrayBoxs for the old-time in the Source Type StateData, so there is not wasted memory.
• Reactions Type : this holds the data for the nuclear reactions. It has NumSpec+2 components:the species creation rates (usually denoted ωk in these notes), the specific energy generationrate (enuc), and its density (ρenuc).
These are stored as StateData so we have access to the reaction terms outside of advance,both for diagnostics (like flame speed estimation) and for reaction timestep limiting (this inparticular needs the data stored in checkpoints for continuity of timestepping upon restart).
why do we need rhoedot and edotseparately?
• SDC React Type : this is used with the SDC time-advancement algorithm. This stores theQVAR terms that describe how the primitive variables change over the timestep due only toreactions. These are used when predicting the interface states of the primitive variables forthe hydrodynamics portion of the algorithm.
We access the multifabs that carry the data of interest by interacting with the StateData usingone of these keys. For instance:
MultiFab& S_new = get_new_data(State_Type );
gets a pointer to the multifab containing the hydrodynamics state data at the new time.
4.2.7 Various source MultiFabs
There are a number of different MultiFabs (and arrays of MultiFabs) that hold source term infor-mation.
• hydro source : this is a MultiFab that holds the update to the hydrodynamics (basicallythe divergence of the fluxes). This is filled in the conservative update routine of the hydro-dynamics.
As this is expressed as a source term, what is actually stored is
Sflux = −∇ · F (4.1)
So the update of the conserved state appears as:
∂U
∂t= Sflux (4.2)
4.3—MFIter and interacting with Fortran 27
• sources for hydro : a single MultiFab that stores the sum of sources over each physicalprocess.
4.3 MFIter and interacting with Fortran
The process of looping over boxes at a given level of refinement and operating on their datain Fortran is linked to how Castro achieves thread-level parallelism. The OpenMP approach inCastro has evolved considerably since the original paper was written, with the modern approach,called tiling, gearing up to meet the demands of many-core processors in the next-generation ofsupercomputers. We discuss the original and new approach together here.
In both cases, the key construct is the MFIter—this is a C++ iterator that knows how to loop overthe FArrayBoxes in the MultiFab that are local to the processor (in this way, a lot of the parallelismis hidden from view).
4.3.1 Non-Tiling MFIter
The non-tiling way to iterate over the FArrayBoxs is 1:
for (MFIter mfi(mf); mfi.isValid (); ++mfi) // Loop over boxes
// Get the index space of this iteration
const Box& box = mfi.validbox ();
// Get a reference to the FAB , which contains data and box
FArrayBox& fab = mf[mfi];
// Get the index space for the data region in th FAB.
// Note "abox" may have ghost cells , and is thus larger than
// or equal to "box" obtained using mfi.validbox ().
const Box& abox = fab.box();
// We can now pass the information to a Fortran routine ,
// fab.dataPtr () gives a double*, which is reshaped into
// a multi -dimensional array with dimensions specified by
// the information in "abox". We will also pass "box",
// which specifies our "work" region.
do_work(ARLIM_3D(box.loVect ()), ARLIM_3D(box.hiVect ()),
fab.dataPtr(), fab.nComp(),
ARLIM_3D(abox.loVect ()), ARLIM_3D(abox.hiVect ())
A few comments about this code
1Note: some older code will use a special AMReX preprocessor macro, BL TO FORTRAN, defined in ArrayLim.H, thatconverts the C++ multifab into a Fortran array and its lo and hi indices. Additionally, some older code will wrapthe Fortran subroutine name in an additional preprocessor macro, BL FORT PROC CALL to handle the name manglingbetween Fortran and C. This later macro is generally not needed any more because of Fortran 2003 interoperabilitywith C (through the Fortran bind keyword).
28 Chapter 4. Software Framework
• In this example, we are working off of a MultiFab named mf. This could, for example, comefrom state data as:
MultiFab& mf = get_old_data(State_Type );
• We are passing the data in mf one box at a time to the Fortran function do work.
• Here the MFIter iterator, mfi, will perform the loop only over the boxes that are local to theMPI task. If there are 3 boxes on the processor, then this loop has 3 iterations.
++mfi iterates to the next FArrayBox owned by the MultiFab mf, and mfi.isValid() returnsfalse after we’ve reached the last box contained in the MultiFab, terminating the loop.
• box as returned from mfi.validbox() does not include ghost cells. This is the valid dataregion only. We can get the indices of the valid zones as box.loVect() and box.hiVect().
In passing to the Fortran function, we use the macro ARLIM 3D, defined in ArrayLim.H topass the lo and hi vectors as pointers to an int array. This array is defined to always be3D, with 0s substituted for the higher dimension values if we are running in 1- or 2D.
Passing the data in this 3D fashion is a newer approach in Castro. This enables writingdimension agnostic code. There are many other approaches that will pass only the DIM valuesof lo and hi using alternate macros in ArrayLim.H.
• fab.dataPtr() returns a double *—a pointer to the data region. This is what is passed toFortran.
• fab.nComp() gives an int—the number of components in the MultiFab. This will be usedfor dimensioning in Fortran.
• To properly dimension the array in Fortran, we need the actual bounds of the data region,including any ghost cells. This is the Box abox, obtained as fab.box(). We pass the lo andhi of the full data region as well.
To properly compile, we need a prototype for the Fortran function. These are placed in the * F.H
files in the Castro Source/ directory. Here’s the prototype for our function:
void do_work
(const int* lo, const int* hi,
Real* state , const Real& ncomp
const int* s_lo , const int* s_hi)
A few comments on the prototype:
• we use the const qualifier on the many of the arguments. This indicates that the data thatis pointed to cannot be modified2 means that the pointers themselves are to be unmodified.But the contents of the memory space that they point to can be modified.
• For ncomp, we in the calling sequence, we just did fab.nComp(). This returns a int. ButFortran is a pass-by-reference language, so we make the argument in the prototype a reference.This ensures that it is passed by reference.
In our Fortran example, we want to loop over all of the data, including 1 ghost cell all around. Thecorresponding Fortran function will look like:
2the way to read these complicated C++ declarations is right-to-left. So ‘const int* lo‘ means ‘lo‘ is a integerpointer to a memory space that is constant. See https://isocpp.org/wiki/faq/const-correctness#ptr-to-const
4.3—MFIter and interacting with Fortran 29
subroutine do_work(lo, hi, &
state , ncomp , &
s_lo , s_hi) bind(C, name="do_work")
use prob_params_module , only : dg
integer , intent(in) :: lo(3), hi(3)
integer , intent(in) :: s_lo(3), s_hi(3), ncomp
real (kind=dp_t), intent(inout) :: state(s_lo (1): s_hi(1), &
s_lo (2): s_hi(2), &
s_lo (3): s_hi(3), ncomp)
! loop over the data
do k = lo(3) -1*dg(3), hi (3)+1* dg(3)
do j = lo(2) -1*dg(2), hi (2)+1* dg(2)
do i = lo(1) -1*dg(1), hi (1)+1* dg(1)
! work on state(i,j,k,:), where the last index
! is the component of the multifab
enddo
enddo
enddo
end subroutine do_work
Finally, comments on the Fortran routine;
• We use the Fortran 2003 bind keyword to specify that we want this to be interoperable withC. Ordinarily we would not need to specify the optional argument name in the binding, butthe PGI compiler requires this if our Fortran subroutine is part of a module.
• We dimension state using s lo and s hi—these are the bounds we got from the FArrayBox,and are for the entire data region, including ghost cells.
Note, in Fortran, the spatial indices of state don’t necessarily start at 1—they reflect theglobal index space for the entire domain at this level of refinement. This means that we knowwhere the box is located.
Later we’ll see how to compute the spatial coordinates using this information.
• Our loop uses lo and hi—these are the indices of the valid data region (no ghost cells). Sincewe want a single ghost cell all around, we subtract 1 from lo and add 1 to hi.
Finally, since this is dimension-agnostic code (it should work correctly in 1-, 2-, and 3D),we need to ensure the loops over the higher dimensions do nothing when we compile for alower dimensionality. This is the role of dg—dg is 1 if our simulation includes that spatialdimension and 0 otherwise.
If we were not looping over ghost cells too, then we would not need to invoke dg, since lo
and hi are both set to 0 for any dimensions not represented in our simulation.
Up to this point, we have not said anything about threading. In this style of using the MFIter, we
30 Chapter 4. Software Framework
0
1
2
3
Figure 4.2: A simple domain showing 4 Boxes labeled 0–3, and their tiling regions (dotted lines)
implement the OpenMP in Fortran, for instance by putting a pragma around the outer loop in thisexample.
4.3.2 AMReX’s Current Tiling Approach In C++
There are two types of tiling that people discuss. In logical tiling, the data storage in memoryis unchanged from how we do things now in pure MPI. In a given box, the data region is storedcontiguously). But when we loop in OpenMP over a box, the tiling changes how we loop over thedata. The alternative is called separate tiling—here the data storage in memory itself is changedto reflect how the tiling will be performed. This is not considered in AMReX.
We have recently introduced logical tiling into parts of AMReXIt is off by default, to make thetransition smooth and because not everything should be tiled. It can be enabled on a loop-by-loopbasis by setting an optional argument to MFIter. We demonstrate this below. Further examplescan be found at Tutorials/Tiling C, and Src/LinearSolvers/C CellMG/.
In our logical tiling approach, a box is logically split into tiles, and a MFIter loops over each tile ineach box. Note that the non-tiling iteration approach can be considered as a special case of tilingwith the tile size equal to the box size.
Let us consider an example. Suppose there are four boxes—see Figure 4.2. The first box is dividedinto 4 logical tiles, the second and third are divided into 2 tiles each (because they are small), andthe fourth into 4 tiles. So there are 12 tiles in total. The difference between the tiling and non-tilingversion are then:
4.3—MFIter and interacting with Fortran 31
• In the tiling version, the loop body will be run 12 times. Note that tilebox is different foreach tile, whereas fab might be referencing the same object if the tiles belong to the samebox.
• In the non-tiling version (by constructing MFIter without the optional second argument orsetting to false), the loop body will be run 4 times because there are four boxes, and a callto mfi.tilebox() will return the traditional validbox. The non-tiling case is essentiallyhaving one tile per box.
The tiling implementation of the same call to our the Fortran do work routine is show below:
bool tiling = true;
for (MFIter mfi(mf , tiling ); mfi.isValid (); ++mfi) // Loop over tiles
// Get the index space of this iteration.
const Box& box = mfi.growntilebox (1);
// Get a reference to the FAB , which contains data and box
FArrayBox& fab = mf[mfi];
// Get the index space for the data pointed by the double *.
const Box& abox = fab.box();
// We can now pass the information to a Fortran routine.
do_work(ARLIM_3D(box.loVect ()), ARLIM_3D(box.hiVect ()),
fab.dataPtr(), fab.nComp(),
ARLIM_3D(abox.loVect ()), ARLIM_3D(abox.hiVect ())
Note that the code is almost identical to the one in § ??. Some comments:
• The iterator now takes an extra argument to turn on tiling (set to true).
There is another interface fo MFIter that can take an IntVect that explicitly gives the tilesize in each coordinate direction. If we don’t explictly specify the tile size at the loop, thenthe runtime parameter fabarray.mfiter tile size can be used to set it globally.
• .validBox() has the same meaning as in the non-tile approach, so we don’t use it. Sincein this example, we want to include a single ghost cell in our loop over the data, we use.growntilebox(1) (where the 1 here indicates a single ghost cells) to get the Box (andcorresponding lo and hi) for the current tile, not the entire data region. If instead, we justwanted the valid region in Fortran, without any ghost cells, we would use .tilebox().
• When passing into the Fortran routine, we still use the index space of the entire FArrayBox
(including ghost cells), as seen in the abox construction. This is needed to properly dimensionthe array in Fortran.
The Fortran routine will declare a multidimensional array that is of the same size as the entirebox, but only work on the index space identified by the tile-box (box).
The Fortran code is almost the same as before, but now our loop simply uses lo and hi, since, byconstruction with .growntilebox(1), this already includes the single ghost cell all around:
subroutine do_work(lo, hi, &
32 Chapter 4. Software Framework
state , ncomp , &
s_lo , s_hi) bind(C, name="do_work")
integer , intent(in) :: lo(3), hi(3)
integer , intent(in) :: s_lo(3), s_hi(3), ncomp
real (kind=dp_t), intent(inout) :: state(s_lo (1): s_hi(1), &
s_lo (2): s_hi(2), &
s_lo (3): s_hi(3), ncomp)
! loop over the data
do k = lo(3), hi(3)
do j = lo(2), hi(2)
do i = lo(1), hi(1)
! work on state(i,j,k,:), where the last index
! is the component of the multifab
enddo
enddo
enddo
end subroutine do_work
The function prototype is unchanged.
Tiling provides us the opportunity of a coarse-grained approach for OpenMP. Threading can beturned on by inserting the following line above the for (MFIter...) line.
#pragma omp parallel
Note that the OpenMP pragma does not have a for—this is not used when working with aniterator.
Assuming four threads are used in the above example, thread 0 will work on 3 tiles from the firstbox, thread 1 on 1 tile from the first box and 2 tiles from the second box, and so forth. Note thatOpenMP can be used even when tiling is turned off. In that case, the OpenMP granularity is atthe box level (and good performance would need many boxes per MPI task).
The tile size for the three spatial dimensions can be set by a parameter, e.g., fabarray.mfiter tile size
= 1024000 8 8. A huge number like 1024000 will turn off tiling in that direction. As noted above,the MFIter constructor can also take an explicit tile size: MFIter(mfi(mf,IntVect(128,16,32))).
Note that tiling can naturally transition from all threads working on a single box to each threadworking on a separate box as the boxes coarsen (e.g., in multigrid).
The MFIter class provides some other useful functions:
• mfi.validbox() : The same meaning as before independent of tiling.
• mfi.tilebox() : The standard way of getting the bounds of the current tile box. This willtile over the valid data region only.
• mfi.growntilebox(int) : A grown tile box that includes ghost cells at box boundaries only.Thus the returned boxes for a FArrayBox are non-overlapping.
4.3—MFIter and interacting with Fortran 33
• mfi.nodaltilebox(int) : Returns non-overlapping edge-type boxes for tiles. The argumentis for direction.
• mfi.fabbox() : Same as mf[mfi].box().
Finally we note that tiling is not always desired or better. The traditional fine-grained approachcoupled with dynamic scheduling is more appropriate for work with unbalanced loads, such aschemistry burning in cells by an implicit solver. Tiling can also create extra work in the ghost cellsof tiles.
4.3.2.1 Practical Details in Working with Tiling
With tiling, the OpenMP is now all in C++, and not in Fortran for all modules except reactionsand initdata.
It is the responsibility of the coder to make sure that the routines within a tiled region are safe touse with OpenMP. In particular, note that:
• tile boxes are non-overlapping
• the union of tile boxes completely cover the valid region of the fab
• Consider working with a node-centered MultiFab, ugdnv, and a cell-centered MultiFab, s:
– with mfi(s), the tiles are based on the cell-centered index space. If you have an 8 × 8box, then and 4 tiles, then your tiling boxes will range from 0→ 3, 4→ 7.
– with mfiugdnv, the tiles are based on nodal indices, so your tiling boxes will range from0→ 3, 4→ 8.
• When updating routines to work with tiling, we need to understand the distinction betweenthe index-space of the entire box (which corresponds to the memory layout) and the index-space of the tile.
– In the C++ end, we pass (sometimes via the BL TO FORTRAN() macro) the loVect andhiVect of the entire box (including ghost cells). These are then used to allocate thearray in Fortran as:
double precision :: a(a_l1:a_h1 , a_l2:a_h2 , ...)
When tiling is used, we do not want to loop as do a l1, a h1, but instead we need toloop over the tiling region. The indices of the tiling region need to be passed into the For-tran routine separately, and they come from the mfi.tilebox() or mfi.growntilebox()statement.
– In Fortran, when initializing an array to 0, do so only over the tile region, not for theentire box. For a Fortran array a, this means we cannot do:
a = 0.0
a(:,:,:,:) = 0.0
but instead must do:
a(lo(1):hi(1),lo(2):hi(2),lo(3):hi(3),:) = 0.0
34 Chapter 4. Software Framework
where lo() and hi() are the index-space for the tile box returned from mfi.tilebox()
in C++ and passed into the Fortran routine.
– Look at r old s in Exec/gravity tests/DustCollapse/probdata.f90 as an exampleof how to declare a threadprivate variable—this is then used in sponge nd.f90.
4.4 Boundaries: FillPatch and FillPatchIterator
AMReX calls the act of filling ghost cells a fillpatch operation. Boundaries between grids are oftwo types. The first we call “fine-fine”, which is two grids at the same level. The second type is”coarse-fine”, which needs interpolation from the coarse grid to fill the fine grid ghost cells. Both ofthese are part of the fillpatch operation. Fine-fine fills are just a straight copy from “valid regions”to ghost cells. Coarse-fine fills are enabled because the StateData is not just arrays, they’re “StateData”, which means that the data knows how to interpolate itself (in an anthropomorphical sense).The type of interpolation to use is defined in Castro setup.cpp—search for cell cons interp,for example—that’s “cell conservative interpolation”, i.e., the data is cell-based (as opposed tonode-based or edge-based) and the interpolation is such that the average of the fine values createdis equal to the coarse value from which they came. (This wouldn’t be the case with straight linearinterpolation, for example.)
Additionally, since StateData has an old and new timelevel, the fill patch operation can interpolateto an intermediate time.
4.4.1 Examples
To illustrate the various ways we fill ghost cells and use the data, let’s consider the followingscenarios:
• You have state data that was defined with no ghost cells. You want to create a new MultiFab
containing a copy of that data with NGROW ghost cells.
This is the case with Sborder—the MultiFab of the hydrodynamic state that we use tokick-off the hydrodynamics advance.
Sborder is declared in Castro.H simply as:
Multifab Sborder;
It is then allocated in Castro::initialize do advance()
Sborder.define(grids , NUM_STATE , NUM_GROW , Fab_allocate );
const Real prev_time = state[State_Type ]. prevTime ();
expand_state(Sborder , prev_time , NUM_GROW );
Note in the call to .define(), we tell AMReX to already allocate the data regions for theFArrayBoxs that are part of Sborder.
The actually filling of the ghost cells is done by Castro::expand state():
AmrLevel :: FillPatch (*this , Sborder , NUM_GROW ,
prev_time , State_Type , 0, NUM_STATE );
4.4—Boundaries: FillPatch and FillPatchIterator 35
Here, we are filling the ng ghost cells of MultiFabSborder at time prev time. We are usingthe StateData that is part of the current Castro object that we are part of. Note: FillPatchtakes an object reference as its first argument, which is the object that contains the relevantStateData—that is what the *this pointer indicates. Finally, we are copying the State Type
data components 0 to NUM STATE3.
The result of this operation is that Sborder will now have NUM GROW ghost cells consistentwith the State Type data at the old time-level.
• You have state data that was defined with NGROW ghost cells. You want to ensure that theghost cells are filled (including any physical boundaries) with valid data.
This is very similar to the procedure shown above. The main difference is that for theMultiFab that will be the target of the ghost cell filling, we pass in a reference to theStateData itself.
The main thing you need to be careful of here, is that you need to ensure that the the timeyou are at is consistent with the StateData’s time. Here’s an example from the radiationportion of the code MGFLDRadSolver.cpp:
Real time = castro ->get_state_data(Rad_Type ). curTime ();
MultiFab& S_new = castro ->get_new_data(State_Type );
AmrLevel :: FillPatch (*castro , S_new , ngrow , time , State_Type ,
0, S_new.nComp(), 0);
In this example, S new is a pointer to the new-time-level State Type MultiFab. So thisoperation will use the State Type data to fill its own ghost cells. we fill the ngrow ghost cellsof the new-time-level State Type data, for all the components.
Note that in this example, because the StateData lives in the Castro object and we areworking from the Radiation object, we need to make reference to the current castro objectpointer. If this were all done within the Castro object, then the pointer will simply be *this,as we saw above.
• You have a MultiFab with some derived quantity. You want to fill its ghost cells.
MultiFabs have a FillBoundary() method that will fill all the ghost cells between boxes atthe same level. It will not fill ghost cells at coarse-fine boundaries or at physical boundaries.
what is the use casefor this?
• You want to loop over the FABs in state data, filling ghost cells along the way
This is the job of the FillPatchIterator—this iterator is used to loop over the grids and fillghostcells. A key thing to keep in mind about the FillPatchIterator is that you operate ona copy of the data—the data is disconnected from the original source. If you want to updatethe data in the source, you need to explicitly copy it back. Also note: FillPatchIterator
takes a multifab, but this is not filled—this is only used to get the grid layout. Finally, theway the FillPatchIterator is implemented is that all the communication is done first, andthen the iterating over boxes commences.
For example, the loop that calls CA UMDRV (all the hydrodynamics integration stuff) startswith
3for clarity and continuity in this documentation, some of the variable names have been changed compared to theactual code
36 Chapter 4. Software Framework
for (FillPatchIterator fpi(*this , S_new , NUM_GROW ,
time , State_Type , strtComp , NUM_STATE );
fpi.isValid (); ++fpi)
FArrayBox &state = fpi();
Box bx(fpi.validbox ());
// work on the state FAB. The interior (valid) cells will
// live between bx.loVect () and bx.hiVect ()
Here the FillPatchIterator is the thing that distributes the grids over processors and makesparallel “just work”. This fills the single patch “fpi” , which has NUM GROW ghost cells, withdata of type “State Type” at time “time”, starting with component strtComp and includinga total of NUM STATE components.
how do tiling andFillPatchIterator
work together?
In general, one should never assume that ghostcells are valid, and instead do a fill patch operationwhen in doubt. Sometimes we will use a FillPatchIterator to fill the ghost cells into a multifabwithout an explict look. This is done as:
FillPatchIterator fpi(*this ,S_old ,1,time ,State_Type ,0,NUM_STATE );
MultiFab& state_old = fpi.get_mf ();
In this operation, state old points to the internal MultiFab in the FillPatchIterator, by gettinga reference to it as fpi.get mf(). This avoids a local copy.
Note that in the examples above, we see that only StateData can fill physical boundaries (becausethese register how to fill the boundaries when they are defined). There are some advanced operationsin AMReX that can get around this, but we do not use them in Castro.
ok?
4.4.2 Physical Boundaries
Physical boundary conditions are specified by an integer index4 in the inputs file, using thecastro.lo bc and castro.hi bc runtime parameters. The generally supported boundary con-ditions are, their corresponding integer key, and the action they take for the normal velocity,transverse velocity, and generic scalar are shown in Table 4.1
The definition of the specific actions are:
• INT DIR: data taken from other grids or interpolated
• EXT DIR: data specified on EDGE (FACE) of bndry
• HOEXTRAP: higher order extrapolation to EDGE of bndry
• FOEXTRAP: first order extrapolation from last cell in interior
• REFLECT EVEN: F (−n) = F (n) true reflection from interior cells
• REFLECT ODD: F (−n) = −F (n) true reflection from interior cells
4the integer values are defined in BC TYPES.H
4.5—Other AMReX Concepts 37
name integer normal velocity transverse velocity scalars
interior 0 INT DIR INT DIR INT DIR
inflow 1 EXT DIR EXT DIR EXT DIR
outflow 2 FOEXTRAP FOEXTRAP FOEXTRAP
symmetry 3 REFLECT ODD REFLECT EVEN REFLECT EVEN
slipwall 4 REFLECT ODD REFLECT EVEN REFLECT EVEN
noslipwall 5 REFLECT ODD REFLECT EVEN REFLECT EVEN
Table 4.1: Physical boundary conditions supported in Castro. why does slipwall and noslipwall dothe same thing?
The actual registration of a boundary condition action to a particular variable is done inCastro setup.cpp. At the top we define arrays such as “scalar bc”, “norm vel bc”, etc, whichsay which kind of bc to use on which kind of physical boundary. Boundary conditions are set infunctions like “set scalar bc”, which uses the scalar bc pre-defined arrays. We also specify thename of the Fortran routine that is responsible for filling the data there (e.g., hypfill). Theseroutines are discussed more below.
If you want to specify a value at a function (like at an inflow boundary), then you choose an inflowboundary at that face of the domain. You then need to write the implementation code for this. Anexample is the problem toy convect which implements a hydrostatic lower boundary (through itscustom bc fill ?d.F90 routines.
4.4.3 FluxRegister
A FluxRegister holds face-centered data at the boundaries of a box. It is composed of a set ofMultiFabs (one for each face, so 6 for 3D). A FluxRegister stores fluxes at coarse-fine interfaces,and isused for the flux-correction step.
4.5 Other AMReX Concepts
There are a large number of classes that help define the structure of the grids, metadata associatewith the variables, etc. A good way to get a sense of these is to look at the .H files in the amrex/Src/directory.
4.5.1 Geometry class
There is a Geometry object, geom for each level as part of the Castro object (this is inheretedthrough AmrLevel).
correct?
38 Chapter 4. Software Framework
4.5.2 ParmParse class
4.5.3 Error Estimators
4.6 Gravity class
There is a single Gravity object, gravity, that is a static class member of the Castro object. Thismeans that all levels refer to the same Gravity object.
Within the Gravity object, there are pointers to the Amr object (as parent), and all of theAmrLevels (as a PArray, LevelData). The gravity object gets the geometry information at eachlevel through the parent Amr class.
The main job of the gravity object is to provide the potential and gravitation acceleration for usein the hydrodynamic sources. Depending on the approximation used for gravity, this could meancalling the AMReX multigrid solvers to solve the Poisson equation.
4.7 Fortran Helper Modules
There are a number of modules that make data available to the Fortran side of Castro or performother useful tasks.
• bl constants module:
This provides double precision constants as Fortran parameters, like ZERO, HALF, and ONE.
• bl types:
This provides a double precision type, dp t for use in Fortran. This should be identical todouble precision on most architectures.
• extern probin module:
This module provides access to the runtime parameters for the microphysics routines (EOS,reaction network, etc.). The source for this module is generated at compile type via a make
rule that invokes a python script. This will search for all of the parameters files in theexternal sources, parse them for runtime parameters, and build the module.
• fundamental constants module:
This provides the CGS values of many physical constants.
• math module:
This provides simple mathematical functions. At the moment, a cross product routine.
• meth params module:
This module provides the integer keys used to access the state arrays for both the conservedvariables (URHO, UMX, . . .) and primitive variables (QRHO, QU, . . .), as well as the number ofscalar variables.
It also provides the values of most of the castro.xxxx runtime parameters.
4.8—Setting Up Your Own Problem 39
• model parser module:
This module is built if USE MODELPARSER = TRUE is set in the problem’s GNUmakefile. Itthen provides storage for the an initial model and routines to read it in and interpolate ontothe Castro grid.
• prob params module:
This module stores information about the domain and current level, and is periodically syncedup with the C++ driver. The information available here is:
– physbc lo, physbc hi: these are the boundary condition types at the low and high endsof the domain, for each coordinate direction. Integer keys, Interior, Inflow, Outflow,Symmetry, SlipWall, and NoSlipWall allow you to interpret the values.
– center is the center of the problem. Note—this is up to the problem setup to define (inthe probinit subroutine). Alternately, it can be set at runtime via castro.center.
which wins if bothtry to set it?
Usually center will be the physical center of the domain, but not always. For instance,for axisymmetric problems, center may be on the symmetry axis.
center is used in the multipole gravity, hybrid advection algorithm, rotation sources,for the point mass gravity, in defining the center of the sponge, and in deriving the radialvelocity.
other importantplaces?
– coord type
– dim
– dg
– refining information
4.8 Setting Up Your Own Problem
To define a new problem, we create a new directory in one of the subdirectories of Exec/, andplace in it a Prob 2d.f90 file (or 1d/3d, depending on the dimensionality of the problem), aprobdata.f90 file, the inputs and probin files, and a Make.package file that tells the buildsystem what problem-specific routines exist. Finally, if you need custom boundary conditions, abc fill 2d.F90 (or 1d/3d) file is needed. The simplest way to get started is to copy these filesfrom an existing problem. Here we describe how to customize your problem.
The purpose of these files is:
• probdata.f90: this holds the probdata module Fortran module that allocates storage for allthe problem-specific runtime parameters that are used by the problem (including those thatare read from the probin file.
• Prob ?d.f90: this holds the main routines to initialize the problem and grid and performproblem-specific boundary conditions:
– probinit():
40 Chapter 4. Software Framework
This routine is primarily responsible for reading in the probin file (by defining the&fortin namelist and reading in an initial model (usually through the model parser module—see the toy convect problem setup for an example). The parameters that are initializedhere are those stored in the probdata module.
– ca initdata():
This routine will initialize the state data for a single grid. The inputs to this routineare:
∗ level: the level of refinement of the grid we are filling
∗ time: the simulation time
∗ lo(), hi(): the integer indices of the box’s valid data region lower left and upperright corners. These integers refer to a global index space for the level and identifywhere in the computational domain the box lives.
∗ nscal: the number of scalar quantities—this is not typically used in Castro.
∗ state l1, state l2, (state l3): the integer indices of the lower left corner of thebox in each coordinate direction. These are for the box as allocated in memory, sothey include any ghost cells as well as the valid data regions.
∗ state h1, state h2, (state h3): the integer indices of the upper right corner of thebox in each coordinate direction. These are for the box as allocated in memory, sothey include any ghost cells as well as the valid data regions.
∗ state(): the main state array. This is dimensioned as:
double precision state(state_l1:state_h1,state_l2:state_h2,NVAR)
(in 2-d), where NVAR comes from the meth params module.
When accessing this array, we use the index keys provided by meth params module
(e.g., URHO) to refer to specific quantities
∗ delta(): this is an array containing the zone width (∆x) in each coordinate direc-tion: delta(1) = ∆x, delta(2) = ∆y, . . ..
∗ xlo(), xhi(): these are the physical coordinates of the lower left and upper rightcorners of the valid region of the box. These can be used to compute the coordinatesof the cell-centers of a zone as:
do j = lo(2), hi(2)
y = xlo(2) + delta (2)*( dble(j-lo(2)) + 0.5d0)
...
(Note: this method works fine for the problem initialization stuff, but for rou-tines that implement tiling, as discussed below, lo and xlo may not refer to thesame corner, and instead coordinates should be computed using problo() from theprob params module.)
• bc fill ?d.F90:
4.8—Setting Up Your Own Problem 41
These routines handle how Castro fills ghostcells at physical boundaries for specific data. Mostproblem setups won’t need to do anything special here, and inclusion of this file is optional –only use it if you need to set specific boundary conditions.
These routines are registered in Castro setup.cpp, and called as needed. By default, theyjust pass the arguments through to filcc, which handles all of the generic boundary con-ditions (like reflecting, extrapolation, etc.). The specific ‘fill’ routines can then supplythe problem-specific boundary conditions, which are typically just Dirichlet boundary con-ditions (usually this means looking to see if the bc() flag at a boundary is EXT DIR. Theproblem-specific code implementing these specific conditions should follow the filcc call.
– ca hypfill: This handles the boundary filling for the hyperbolic system.
– ca denfill: At times, we need to fill just the density (always assumed to be the firstelement in the hyperbolic state) instead of the entire state. When the fill patch routine iscalled with first comp = Density and num comp = 1, then we use ca denfill insteadof ca hypfill.
(Note: it seems that this may be used for more than just density, but it is only used fortagging and the plotfile)
– ca grav?fill: These routines fill will the ghostcells of the gravitational accelerationgrids with the gravitational acceleration.
Note: for constant gravity, these routines will never be called. For one of the Poisson-type gravities, you only need to do something special here if you are implementing anInterior boundary type (which you can test for by comparing bc(:,:,:) to EXT DIR.
For the other standard physical boundary types, the ghost cell filling will be handledautomatically by the default filcc call in these routines.
The gravitational acceleration in the ghost cells is used during the hydrodynamics portionof the code in predicting the interface states.
– ca reactfill: This handles boundary filling for any Reactions Type MultiFABs, whichare sometimes used to interface with the nuclear burning module. It stores the normalstate data in addition to components for the energy release and species change.
These routines take the following arguments:
– adv l1, adv l2, (adv l3): the indicies of the lower left corner of the box holding thedata we are working on. These indices refer to the entire box, including ghost cells.
– adv h1, adv h2, (adv h3): the indicies of the upper right corner of the box holding thedata we are working on. These indices refer to the entire box, including ghost cells.
– adv(): the array of data whose ghost cells we are filling. Depending on the routine, thismay have an additional index refering to the variable.
This is dimensioned as:
double precision adv(adv_l1:adv_h1,adv_l2:adv_h2)
– domlo(), domhi(): the integer indices of the lower left and upper right corners of thevalid region of the entire domain. These are used to test against to see if we are fillingphysical boundary ghost cells.
42 Chapter 4. Software Framework
This changes according to refinement level: level-0 will range from 0 to castro.max grid size,and level-n will range from 0 to castro.max grid size ·
∏n castro.ref ratio(n).
– delta(): is the zone width in each coordinate direction, as in initdata() above.
– xlo(): this is the physical coordinate of the lower left corner of the box we are filling—including the ghost cells.
Note: this is different than how xlo() was defined in initdata() above.
– time: the simulation time
– bc(): an array that holds the type of boundary conditions to enforce at the physicalboundaries for adv.
Sometimes it appears of the form bc(:,:) and sometimes bc(:,:,:)—the last indexof the latter holds the variable index, i.e., density, pressure, species, etc.
The first index is the coordinate direction and the second index is the domain face (1 islow, 2 is hi), so bc(1,1) is the lower x boundary type, bc(1,2) is the upper x boundarytype, bc(2,1) is the lower y boundary type, etc.
To interpret the array values, we test against the quantities defined in bc types.fi
included in each subroutine, for example, EXT DIR, FOEXTRAP, . . .. The meaning of theseare explained below.
4.8.1 Optional Files
The follow problem-specific files are optional. There are stubs for each of these in the main sourcetree.
• Problem.f90 :
This provides two routines, problem checkpoint and problem restart that can be used toadd information to the checkpoint files and read it in upon restart. This is useful for someglobal problem-specific quantities. For instance, the wdmerger5 problem uses this to storecenter of mass position and velocity information in the checkpoint files that are used forruntime diagnostics.
The name of the checkpoint directory is passed in as an argument. Problem F.H provides theC++ interfaces for these routines.
• problem tagging ?d.F90, problem tagging nd.F90
This implements problem-specific tagging for refinement, through a subroutine set problem tags.The full hydrodynamic state (State Type) is passed in, and the problem can mark zones forrefinement by setting the tag variable for a zone to set. An example is provided by thetoy convect problem which refines a rectangular region (fuel layer) based on a density pa-rameter and the H mass fraction.
• Problem Derive F.H, Problem Derives.H, problem derive nd.f90
5available separately at https://github.com/BoxLib-Codes/wdmerger
4.9—Parallel I/O 43
Together, these provide a mechanism to create derived quantities that can be stored in theplotfile. Problem Derives.H provides the C++ code that defines these new plot variables. Itdoes this by adding them to the derive lst—a list of derived variables that Castro knowsabout. When adding new variables, a descriptive name, Fortran routine that does the deriv-ing, and component of StateData are specified.
The Fortran routine that does the deriving is put in the problem-specific problem derive nd.f90
(and a prototype for C++ is put in Problem Derives.H). A example is provided by thereacting bubble problem, which derives several new quantities (perturbations against abackground one-dimensional model, in this case).
• Prob.cpp, Problem.H, Problem F.H
These files provide problem-specific routines for computing global diagnostic informationthrough the sum integrated quantities functionality that is part of the Castro class.
An example is provided by toy flame, where an estimate of the flame speed is computed byintegrating the mass of fuel on the grid.
4.8.2 Dimension Agnostic Problem Initialization
Most of the problem setups have separate implementations for 1-, 2-, and 3D. A new method existsthat allows you to write just a single set of files for any dimensionality (this is called the dimensionagnostic format). To use this mode, set DIMENSION AGNOSTIC= TRUE in your GNUmakefile. Thenwrite you problem initialization in Prob nd.F90. Analogous routines exist for tagging and boundaryconditions. See the rotating torus and Noh problem setups for an example.
4.9 Parallel I/O
Both checkpoint files and plotfiles are really directories containing subdirectories: one subdirectoryfor each level of the AMR hierarchy. The fundamental data structure we read/write to disk is aMultiFab, which is made up of multiple FAB’s, one FAB per grid. Multiple MultiFabs may bewritten to each directory in a checkpoint file. MultiFabs of course are shared across CPUs; a singleMultiFab may be shared across thousands of CPUs. Each CPU writes the part of the MultiFabthat it owns to disk, but they don’t each write to their own distinct file. Instead each MultiFab iswritten to a runtime configurable number of files N (N can be set in the inputs file as the parameteramr.checkpoint nfiles and amr.plot nfiles; the default is 64). That is to say, each MultiFabis written to disk across at most N files, plus a small amount of data that gets written to a headerfile describing how the file is laid out in those N files.
What happens is N CPUs each opens a unique one of the N files into which the MultiFab is beingwritten, seeks to the end, and writes their data. The other CPUs are waiting at a barrier for thoseN writing CPUs to finish. This repeats for another N CPUs until all the data in the MultiFab iswritten to disk. All CPUs then pass some data to CPU 0 which writes a header file describing howthe MultiFab is laid out on disk.
We also read MultiFabs from disk in a “chunky” manner, opening only N files for reading at atime. The number N , when the MultiFabs were written, does not have to match the number Nwhen the MultiFabs are being read from disk. Nor does the number of CPUs running while reading
44 Chapter 4. Software Framework
in the MultiFab need to match the number of CPUs running when the MultiFab was written todisk.
Think of the number N as the number of independent I/O pathways in your underlying parallelfilesystem. Of course a “real” parallel filesytem should be able to handle any reasonable value ofN . The value -1 forces N to the number of CPUs on which you’re running, which means that eachCPU writes to a unique file, which can create a very large number of files, which can lead to inodeissues.
CHAPTER 5
Single-Level Flow Chart
5.1 Introduction
There are several different time-evolution methods currently implemented in Castro. As best aspossible, they share the same driver routines and use preprocessor or runtime variables to separatethe different code paths.
• Strang-splitting: the Strang evolution does the burning on the state for ∆t/2, then updatesthe hydrodynamics using the burned state, and then does the final ∆t/2 burning. No explicitcoupling of the burning and hydro is done. Within the Strang code path, there are twomethods for doing the hydrodynamics, controlled by castro.do ctu.
– Corner-transport upwind (CTU): this implements the unsplit, characteristic tracingmethod of [6].
– Method of lines (MOL): this discretizes the space part of our system without any char-acteristic tracing and uses an ODE integrator to advance the state. Multiple stages canbe done, each requiring reconstruction, Riemann solve, etc., and the final solution ispieced together from the intermediate stages.
• SDC: the SDC path is enabled by the SDC preprocessor variable. This iteratively couples thereactions and hydrodynamics together.
Several helper functions are used throughout:
• clean state: There are many ways that the hydrodynamics state may become unphysical inthe evolution. The clean state() routine enforces some checks on the state. In particular,it
double check
1. enforces that the density is above castro.small dens
45
46 Chapter 5. Single-Level Flow Chart
2. normalizes the species so that the mass fractions sum to 1
3. resets the internal energy if necessary (too small or negative) and computes the temper-ature for all zones to be thermodynamically consistent with the state.
5.2 Overview of a single step (no SDC)
The main evolution for a single step is contained in Castro advance.cpp, as Castro::advance().This does the following advancement. Note, some parts of this are only done depending on whichpreprocessor directives are defined at compile-time—the relevant directive is noted in the [ ] at thestart of each step.
1. Initialization (initialize advance())
This sets up the current level for advancement. The following actions are performend (note,we omit the actions taken for a retry, which we will describe later):
• Sync up the level information to the Fortran-side of Castro
• Do any radiation initialization
• Initialize all of the intermediate storage arrays (like those that hold source terms, etc.).
• Swap the StateData from the new to old (e.g., ensures that the next evolution startswith the result from the previous step).
• Do a clean state
• Create the MultiFabs that hold the primitive variable information for the hydro solve.
• For method of lines integration: allocate the storage for the intermediate stage updates,k mol, and the Sburn MultiFab that holds the post burn state.
• Zero out all of the fluxes
2. Advancement
The update strategy differs for CTU vs MOL:
• CTU: Calls do advance to take a single step, incorporating hydrodynamics, reactions,and source terms.
• MOL: Call do advance mol MOL STAGES times (i.e., once for each of the intermediatestages in the ODE integration). Within do advance we will use the stage number,mol iteration, to do an pre- or post-hydro sources (e.g., burning).
In either case, for radiation-hydrodynamics, this step does the advective (hyperbolic) portionof the radiation update only. Source terms, including gravity, rotation, and diffusion areincluded in this step, and are time-centered to achieve second-order accuracy.
If castro.use retry is set, then we subcycle the current step if we violated any stabilitycriteria to reach the desired ∆t. The idea is the following: if the timestep that you tookhad a timestep that was not sufficient to enforce the stability criteria that you would like toachieve, such as the CFL criterion for hydrodynamics or the burning stability criterion forreactions, you can retry the timestep by setting castro.use retry = 1 in your inputs file.
5.2—Overview of a single step (no SDC) 47
This will save the current state data at the beginning of the level advance, and then if thecriteria are not satisfied, will reject that advance and start over from the old data, with aseries of subcycled timesteps that should be small enough to satisfy the criteria. Note thatthis will effectively double the memory footprint on each level if you choose to use it.
3. [AUX UPDATE] Auxiliary quantitiy evolution
Auxiliary variables in Castro are those that obey a continuity equation (with optional sources)that are passed into the EOS, but not subjected to the constraint on mass fractions (summingto one).
The advection and source terms are already dealt with in the main hydrodynamics advance(above step). A user-supplied routine ca auxupdate can be provided here to further updatethese quantities.
4. Radial data and [POINTMASS] point mass
If castro.spherical star is set, then we average the state data over angles here to createa radial profile. This is then used in the boundary filling routines to properly set DirichletBCs when our domain is smaller than the star, so the profile on the boundaries will not beuniform.
If castro.point mass fix solution is set, then we change the mass of the point mass thatoptionally contributes to the gravitational potential by taking mass from the surroundingzones (keeping the density in those zones constant).
5. [RADIATION] Radiation implicit update
The do advance() routine only handled the hyperbolic portion of the radiation update. Thisstep does the implicit solve (either gray or multigroup) to advance the radiation energies tothe new time level. Note that at the moment, this is backward-difference implicit (first-orderin time) for stability.
This is handled by final radiation call().
6. [PARTICLES] Particles
If we are including passively-advected particles, they are advanced in this step.
7. Finalize
This cleans up at the end of a step:
• Update the flux registers to account for mismatches at coarse-fine interfaces. This cleansup the memory used during the step.
• If castro.track grid losses is set, then we also add up the mass that left through theboundary over this step.1
• Free any memory allocated for the level advance.
1Note: this functionality assumes that only the coarse grid touches the physical boundary. It does not use anyuse masks to prevent double counting if multiple levels touch the boundary.
48 Chapter 5. Single-Level Flow Chart
5.2.1 Main Hydro, Reaction, and Gravity Advancement (CTU w/ Strang-splitting)
The explicit portion of the system advancement (reactions, hydrodynamics, and gravity) is doneby do advance(). Consider our system of equations as:
∂U
∂t= −A(U) + R(U) + S, (5.1)
where A(U) = ∇ · F(U), with F the flux vector, R are the reaction source terms, and S are thenon-reaction source terms, which includes any user-defined external sources, Sext. We use Strangsplitting to discretize the advection-reaction equations. In summary, for each time step, we updatethe conservative variables, U, by reacting for half a time step, advecting for a full time step (ignoringthe reaction terms), and reacting for half a time step. The treatment of source terms complicatesthis a little. The actual update, in sequence, looks like:
U? = Un +∆t
2R(Un) (5.2)
Un+1,(a) = U? + ∆tS(U?) (5.3)
Un+1,(b) = Un+1,(a) −∆tA(U?) (5.4)
Un+1,(c) = Un+1,(b) +∆t
2[S(Un+1,(b))− S(U?)] (5.5)
Un+1 = Un+1,(c) +∆t
2R(Un+1,(c)) (5.6)
Note that in the first step, we add a full ∆t of the old-time source to the state. This predictionensures consistency when it comes time to predicting the new-time source at the end of the update.The construction of the advective terms, A(U) is purely explicit, and based on an unsplit second-order Godunov method. We predict the standard primitive variables, as well as ρe, at time-centerededges and use an approximate Riemann solver construct fluxes.
At the beginning of the time step, we assume that U and φ are defined consistently, i.e., ρn and φn
satisfy equation (10.3). Note that in Eq. 5.5, we actually can actually do some sources implicitlyby updating density first, and then momentum, and then energy. This is done for rotating andgravity, and can make the update more akin to:
Un+1,(c) = Un+1,(b) +∆t
2[S(Un+1,(c))− S(Un)] (5.7)
Castro also supports radiation. This part of the update algorithm only deals with the advective /hyperbolic terms in the radiation update.
Here is the single-level algorithm. The goal here is to update the State Type StateData from theold to new time (see § 4.2.6). We will use the following notation here, consistent with the namesused in the code:
• S old is a multifab reference to the old-time-level State Type data.
• Sborder is a multifab that has ghost cells and is initialized from S old. This is what thehydrodynamic reconstruction will work from.
• S new is a multifab reference to the new-time-level State Type data.
5.2—Overview of a single step (no SDC) 49
In the code, the objective is to evolve the state from the old time, S old, to the new time, S new.
1. Initialize
(a) In initialize do advance() :
i. Create Sborder, initialized from S old
(b) Check for NaNs in the initial state, S old.
2. React ∆t/2. [strang react first half()]
Update the solution due to the effect of reactions over half a time step. The integrationmethod and system of equations used here is determined by a host of runtime parametersthat are part of the Microphysics package. But the basic idea is to evolve the energy releasefrom the reactions, the species mass fractions, and temperature through ∆t/2.
Using the notation above, we begin with the time-level n state, Un, and produce a state thathas evolved only due to reactions, U?.
(ρe)? = (ρe)? − ∆t
2ρHnuc (5.8)
(ρE)? = (ρE)? − ∆t
2ρHnuc (5.9)
(ρXk)? = (ρXk)
? +∆t
2(ρωk)
n. (5.10)
Here, Hnuc is the energy release (erg/g/s) over the burn, and ωk is the creation rate for speciesk.
After exiting the burner, we call the EOS with ρ?, e?, and X?k to get the new temperature,
T ?.
Note that the density, ρ, does not change via reactions in the Strang-split formulation.
The reaction data needs to be valid in the ghost cells. The logic in this routine (accomplishedthrouh the use of a mask) will burn only in the valid interior cells or in any ghost cellsthat are on a coarse-fine interface or physical boundary. This allows us to just use a levelFillBoundary() call to fill all of the ghost cells on the same level with valid data.
An experimental option (enabled via use custom knapsack weights) will create a customdistribution map based on the work needed in burning a zone and redistribute the boxesacross processors before burning, to better load balance..
need to explain thelogic here, especiallywhat the parallelcopies are doing
After reactions, clean state is called.
At the end of this step, Sborder sees the effects of the reactions.
3. Construct time-level n sources and apply [construct old gravity(), do old sources()]
The time level n sources are computed, and added to the StateData Source Type. Thesources are then applied to the state after the burn, U? with a full ∆t weighting (this will becorrected later). This produces the intermediate state, Un+1,(a).
The sources that we deal with here are:
50 Chapter 5. Single-Level Flow Chart
(a) sponge : the sponge is a damping term added to the momentum equation that is designedto drive the velocities to zero over some timescale. Our implementation of the spongefollows that of Maestro [2]
(b) external sources : users can define problem-specific sources in the ext src ?d.f90 file.Sources for the different equations in the conservative state vector, U, are indexed usingthe integer keys defined in meth params module (e.g., URHO).
This is most commonly used for external heat sources (see the toy convect problemsetup) for an example. But most problems will not use this.
(c) [DIFFUSION] diffusion : thermal diffusion can be added in an explicit formulation.Second-order accuracy is achieved by averaging the time-level n and n+ 1 terms, usingthe same predictor-corrector strategy described here.
Note: thermal diffusion is distinct from radiation hydrodynamics.
Also note that incorporating diffusion brings in an additional timestep constraint, sincethe treatment is explicit. See Chapter 11 for more details.
(d) [HYBRID MOMENTUM] angular momentumneed to write thisup
(e) [GRAVITY] gravity:
For full Poisson gravity, we solve for for gravity using:
gn = −∇φn, ∆φn = 4πGρn, (5.11)
The construction of the form of the gravity source for the momentum and energy equationis dependent on the parameter castro.grav source type. Full details of the gravitysolver are given in Chapter 10.
we should add adescription ofwhether we do alevel solve or acomposite solvewhat do we store?phi and g? source?
(f) [ROTATION] rotation
We compute the rotational potential (for use in the energy update) and the rotationalacceleration (for use in the momentum equation). This includes the Coriolis and centrifu-gal terms in a constant-angular-velocity co-rotating frame. The form of the rotationalsource that is constructed then depends on the parameter castro.rot source type.More details are given in Chapter 12.
The source terms here are evaluated using the post-burn state, U? (Sborder), and latercorrected by using the new state just before the burn, Un+1,(b). This is compatible withStrang-splitting, since the hydro and sources takes place completely inside of the surroundingburn operations.
Note that the source terms are already applied to S new in this step, with a full ∆t—this willbe corrected later.
4. Construct the hydro update [construct hydro source()]
The goal is to advance our system considering only the advective terms (which in Cartesiancoordinates can be written as the divergence of a flux).
We do the hydro update in two parts—first we construct the advective update and store itin the hydro source MultiFab, then we do the conservative update in a separate step. This
5.2—Overview of a single step (no SDC) 51
separation allows us to use the advective update separately in more complex time-integrationschemes.
In the Strang-split formulation, we start the reconstruction using the state after burning, U?
(Sborder). There are two approaches we use, the corner transport upwind (CTU) methodthat uses characteristic tracing as described in [6], and a method-of-lines approach. Thechoice is determined by the parameter castro.do ctu.
(a) CTU method:
For the CTU method, we predict to the half-time (n+1/2) to get a second-order accuratemethod. Note: Sborder does not know of any sources except for reactions. The advec-tion step is complicated, and more detail is given in Section 9.6. Here is the summarizedversion:
i. Compute primitive variables.
ii. Convert the source terms to those acting on primitive variables
iii. Predict primitive variables to time-centered edges.
iv. Solve the Riemann problem.
v. Compute fluxes and update.
To start the hydrodynamics, we need to know the hydrodynamics source terms at time-level n, since this enters into the prediction to the interface states. This is essentiallythe same vector that was computed in the previous step, with a few modifications. Themost important is that if we set castro.source term predictor, then we extrapolatethe source terms from n to n+ 1/2, using the change from the previous step.
Note: we neglect the reaction source terms, since those are already accounted for in thestate directly, due to the Strang-splitting nature of this method.
The update computed here is then immediately applied to S new.
(b) method of linesthis needs to befilled
5. Clean State [clean state()]we only seem to dothis for the MOLintegration
This is done on S new.
After these checks, we check the state for NaNs.
6. Update radial data and center of mass for monopole gravityis that right?
These quantities are computed using S new.
7. Correct the source terms with the n+1 contribution [construct new gravity(), do new sources]
Previously we added ∆tS(U?) to the state, when we really want a time-centered approach,(∆t/2)[S(U? + S(Un+1,(b))]. We fix that here.
We start by computing the source term vector S(Un+1,(b)) using the updated state, Un+1,(b).We then compute the correction, (∆t/2)[S(Un+1,(b)) − S(U?)] to add to Un+1,(b) to give us
52 Chapter 5. Single-Level Flow Chart
the properly time-centered source, and the fully updated state, Un+1,(c). This correction isstored in the new sources MultiFab2.
In the process of updating the sources, we update the temperature to make it consistent withthe new state.
8. React ∆t/2. [strang react second half()]
We do the final ∆t/2 reacting on the state, begining with Un+1,(c) to give us the final stateon this level, Un+1.
This is largely the same as strang react first half(), but it does not currently fill thereactions in the ghost cells.
confirm this
9. Finalize [finalize do advance()]
Finalize does the following:
(a) for the momentum sources, we compute dS/dt, to use in the source term prediction/ex-trapolation for the hydrodynamic interface states during the next step.
(b) If we are doing the hybrid momentum algorithm, then we sync up the hybrid and linearmomenta
A summary of which state is the input and which is updated for each of these processes is presentedbelow:
step S old Sborder S new
1. init input updated
2. react input / updated
3. old sources input updated
4. hydro input updated
5. clean input / updated
6. radial / center input
7. correct sources input / updated
8. react input / updated
5.2.2 Main Hydro, Reaction, and Gravity Advancement (MOL w/ Strang-splitting)
The handling of sources differs in the MOL integration, as compared to CTU. Again, consider oursystem as:
∂U
∂t= −A(U) + R(U) + S . (5.12)
2The correction for gravity is slightly different since we directly compute the time-centered gravitational sourceterm using the hydrodynamic fluxes.
5.2—Overview of a single step (no SDC) 53
We will again use Strang splitting to discretize the advection-reaction equations, but the hydroupdate will consist of s stages. The update first does the reactions, as with CTU:
U? = Un +∆t
2R(Un) (5.13)
We then consider the hydro update discretized in space, but not time, written as:
∂U
∂t= −A(U) + S(U) (5.14)
Using a Runge-Kutta (or similar) integrator, we write the update as:
Un+1,? = U? + ∆ts∑l=1
bikl (5.15)
where bi is the weight for stage i and ki is the stage update:
kl = −A(Ul) + S(Ul) (5.16)
with
Ul = U? + ∆t
l−1∑m=1
almkm (5.17)
Finally, there is the last part of the reactions:
Un+1 = Un+1,? +∆t
2R(Un+1,?) (5.18)
In contrast to the CTU method, the sources are treated together with the advection here.
The time at the intermediate stages is evaluated as:
tl = cl∆t (5.19)
The integration coefficients are stored in the vectors a mol, b mol, and c mol, and the stage updatesare stored in the MultiFab k mol.
Here is the single-level algorithm. We use the same notation as in the CTU flowchart.
In the code, the objective is to evolve the state from the old time, S old, to the new time, S new.
1. Initialize
In initialize do advance(), set the starting point for the stage’s integration:
(a) if mol iteration = 0: initialize Sborder from S old
(b) if mol iteration > 0: we need to create the starting point for the current stage. Westore this, temporarily in the new-time slot (what we normally refer to as S new):
S newiter = Sburn + ∆titer−1∑l=0
aiter,lk moll (5.20)
Then initialize Sborder from S new.
54 Chapter 5. Single-Level Flow Chart
Check for NaNs in the initial state, S old.
2. React ∆t/2. [strang react first half()]
This step is unchanged from the CTU version. At the end of this step, Sborder sees theeffects of the reactions.
Each stage needs to build its starting point from this point, so we store the effect of the burnin a new MultiFab, Sburn, for use in the stage initialization.
3. Construct sources from the current stage’s state [construct old gravity(), do old sources()]fix: gravity is stillusing S old
The time level n sources are computed, and added to the StateData Source Type. Thesources are then applied to the state after the burn, U? with a full ∆t weighting (this will becorrected later). This produces the intermediate state, Un+1,(a).
For full Poisson gravity, we solve for for gravity using:
gn = −∇φn, ∆φn = 4πGρn, (5.21)
4. Construct the hydro update [construct hydro source()]
The hydro update in the MOL branch will include both the advective and source terms. Ineach stage, store in k mol[istage] the righthand side for the current stage.
In constructing the stage update, we use the source evaluated earlier, and compute:
k moll = −A(Ul) + S(Ul) (5.22)
Each call to do advance mol only computes this update for a single stage. On the last stage,we compute the final update as:
S new = Sburn + ∆t
n stages−1∑l=0
bl k moll (5.23)
5. Clean State [clean state()]we only seem to dothis for the MOLintegration
This is done on S new.
After these checks, we check the state for NaNs.
6. React ∆t/2. [strang react second half()]
We do the final ∆t/2 reacting on the state, begining with Un+1,(c) to give us the final stateon this level, Un+1.
This is largely the same as strang react first half(), but it does not currently fill thereactions in the ghost cells.
confirm this
7. Finalize [finalize do advance()]
Finalize does the following:
(a) for the momentum sources, we compute dS/dt, to use in the source term prediction/ex-trapolation for the hydrodynamic interface states during the next step.
5.3—Overview of a single step (with SDC) 55
(b) If we are doing the hybrid momentum algorithm, then we sync up the hybrid and linearmomenta
A summary of which state is the input and which is updated for each of these processes is presentedbelow:
step S old Sborder S new
1. init input updated
2. react input / updated
3. old sources input updated
4. hydro input updated
5. clean input / updated
6. radial / center input
7. correct sources input / updated
8. react input / updated
5.3 Overview of a single step (with SDC)
We express our system as:
Ut = A(U) + R(U) (5.24)
here A is the advective source, which includes both the flux divergence and the hydrodynamicsource terms (e.g. gravity):
A(U) = −∇ · F(U) + S (5.25)
The SDC version of the main advance loop looks similar to the no-SDC version, but includes aniteration loop over the hydro, gravity, and reaction update. So the only difference happens in step2 of the flowchart outlined in § 5.2. In particular this step now proceeds as:
2. Advancement
Loop k from 0 to sdc iters, doing:
(a) Hydrodynamics advance: This is done through do advance—in SDC mode, this onlyupdates the hydrodynamics, including the non-reacting sources. However, in predictingthe interface states, we use an iteratively-lagged approximation to the reaction sourceon the primitive variables, Ik−1
q .first time through isthis source 0?
The result of this is an approximation to A(U), stored in hydro sources (the fluxdivergence) and old sources and new sources.
(b) React: Reactions are integrated with the advective update as a source—this way thereactions see the time-evolution due to advection as we integrate:
dU
dt= [A(U)]n+1/2 + R(U) (5.26)
56 Chapter 5. Single-Level Flow Chart
The advective source includes both the divergence of the fluxes as well as the time-centered source terms. This is computed by sum of sources() by summing over allsource components hydro source, old sources, and new sources.
(c) Clean state: This ensures that the thermodynamic state is valid and consistent.
(d) Construct reaction source terms: Construct the change in the primitive variables dueonly to reactions over the timestep, Ikq . This will be used in the next iteration.
Note that is it likely that some of the other updates (like any non-advective auxiliary quantityupdates) should be inside the SDC loop, but presently they are only done at the end. Also notethat the radiation implicit update is not done as part of the SDC iterations.
5.3.1 Main Hydro and Gravity Advancement (SDC)
The evolution in do advance is substantially different than the Strang case. In particular, reactionsare not evolved. Here we summarize those differences.
1. Initialize [initialize do advance()]
This is unchanged from step 1 in the Strang algorithm.
2. Construct time-level n sources and apply [construct old gravity(), do old sources()]
This corresponds to step 3 in the Strang algorithm. There are not differences compared tothe Strang algorithm, although we note, this only needs to be done for the first SDC iterationin the advancement, since the old state does not change.
we need to fix thecode to only do thisonce
3. Construct the hydro update [construct hydro source()]
This corresponds to step 4 in the Strang algorithm. There are a few major differences withthe Strang case:
• There is no need to extrapolate source terms to the half-time for the prediction (thecastro.source term predictor parameter), since SDC provides a natural way to ap-proximate the time-centered source—we simply use the iteratively-lagged new-timesource.
• The primitive variable source terms that are used for the prediction include the con-tribution due to reactions (from the last SDC iteration). This addition is done inconstruct hydro source() after the source terms are converted to primitive variables.
4. Update radial data and center of mass for monopole gravity
This is the same as the Strang step 6
5. Clean State [clean state()]
This is the same as the Strang step 5
6. Correct the source terms with the n+1 contribution [construct new gravity(), do new sources]
This is the same as the Strang step 6
7. Finalize [finalize do advance()]
5.3—Overview of a single step (with SDC) 57
This differs from Strang step 7 in that we do not construct dS/dt, but instead store the totalhydrodynamical source term at the new time. As discussed above, this will be used in thenext iteration to approximate the time-centered source term.
CHAPTER 6
Runtime Parameters
6.1 Introduction to Runtime Parameters
Castro has 2 sets of runtime parameters—those controlled by C++ and those controlled by Fortran.The C++ parameters are set in the inputs file and managed by the AMReX ParmParse class. ForCastro-specific parameters, we list the runtime parameters in a file cpp parameters and generatethe C++ code and headers using the mk params.sh script—note this script needs to be run everytime the cpp parameters file is updated.
The behavior of the network, EOS, and other microphysics routines are controlled by a differentset of runtime parameters. These parameters are defined in plain-text files parameters located inthe different directories that hold the microphysics code. At compile time, a script in the AMReXbulid system, findparams.py, locates all of the parameters files that are needed for the givenchoice of network, integrator, and EOS, and assembles all of the runtime parameters into a modulenamed extern probin module (using the write probin.py script). The parameters are set in yourprobin file in the &extern namelist.
6.1.1 C++ parameter format
The C++ parameters take the form of:
# comment describing the parameter
name type default need in Fortran? ifdef fortran name fortran type
Here, name is the name of the parameter that will be looked for in the inputs file, type is oneof int, Real, or string, and default is the default value of the parameter. The next columnsare optional, but you need to fill in all of the information up to and including any of the optionalcolumns you need (e.g., if you are going to provide the fortran name, you also need to provide need
59
60 Chapter 6. Runtime Parameters
in Fortran? and ifdef. The need in Fortran? column is y if the runtime parameter should bemade available in Fortran (through meth params module). The ifdef field provides the name of apreprocessor name that should wrap this parameter definition—it will only be compiled in if thatname is defined to the preprocessor. The fortran name is the name that the parameter shoulduse in Fortran—by default it will be the same as name. The fortran type is the data type of theparameter in Fortran—by default it will be the Fortran-equivalent to type. Finally, any commentimmediately before the parameter definition will be used to generate the LATEX documentationdescribing the parameters.
6.1.2 Microphysics/extern parameter format
The microphysics/extern parameter definitions take the form of:
# comment describing the parameter
name data-type default-value priority
Here, the priority is simply an integer. When two directories define the same parameter, but withdifferent defaults, the version of the parameter with the highest priority takes precedence. Thisallows specific implementations to override the general parameter defaults.
The documentation below for the Castro C++ parameters is automatically generated, using thecomments in the cpp parameters file.
6.2 Removed Runtime Parameters
The following runtime parameters have been removed for Castro.
• castro.ppm flatten before integrals : this parameter controlled whether we applied theflattening of the parabolic profiles before we integrated under their profiles or afterwards.The default was switched to flattening before the integration, which is more consistent withthe original PPM methodology. This parameter was removed since the variation enabled bythis parameter was not that great.
(removed in commit: 9cab697268997714919de16db1ca7e77a95c4f98)
• castro.ppm reference and castro.ppm reference edge limit : these parameters con-trolled whether we use the integral under the parabola for the fastest wave moving towardthe interface for the reference state and whether in the case that the wave is moving awayfrom the interface we use the cell-center value or the limit of the parabola on the interface.These were removed because there is little reason to not use the reference state.
(removed in commit: 213f4ffc53463141084551c7be4b37a2720229aa)
6.3 castro Namespace
6.3—castro
Nam
espace
61
Table 6.1: castro : AMR parameters
parameter description default value
do reflux do we do the hyperbolic reflux at coarse-fine interfaces? 1
lin limit state interp how to do limiting of the state data when interpolating 0: only prevent new extrema 1:preserve linear combinations of state variables
0
state interp order highest order used in interpolation 1
state nghost Number of ghost zones for state data to have. Note that if you are using radiation,choosing this to be zero will be overridden since radiation needs at least one ghost zone.
0
update sources after reflux whether to re-compute new-time source terms after a reflux 1
use custom knapsack weights should we have state data for custom load-balancing weighting? 0
Table 6.2: castro : diagnostics, I/O parameters
parameter description default value
hard cfl limit abort if we exceed CFL = 1 over the cource of a timestep 1
job name a string describing the simulation that will be copied into the plotfile’s job info file ””
output at completion write a final plotfile and checkpoint upon completion 1
print fortran warnings display warnings in Fortran90 routines (0, 1)
print update diagnostics display information about updates to the state (how much mass, momentum, energyadded)
(0, 1)
reset checkpoint step Do we want to reset the number of steps in the checkpoint? This ONLY takes effectif amr.regrid on restart = 1 and amr.checkpoint on restart = 1, (which require thatmax step and stop time be less than the value in the checkpoint) and you set it to valuegreater than this default value.
-1
continued on next page
62C
hap
ter6.
Ru
ntim
eP
arameters
Table 6.2—continued
parameter description default value
reset checkpoint time Do we want to reset the time in the checkpoint? This ONLY takes effect ifamr.regrid on restart = 1 and amr.checkpoint on restart = 1, (which require thatmax step and stop time be less than the value in the checkpoint) and you set it tovalue greater than this default value.
-1.e200
show center of mass display center of mass diagnostics 0
sum interval how often (number of coarse timesteps) to compute integral sums (for runtime diagnos-tics)
-1
sum per how often (simulation time) to compute integral sums (for runtime diagnostics) -1.0e0
track grid losses calculate losses of material through physical grid boundaries 0
Table 6.3: castro : diffusion parameters
parameter description default value
diffuse cond scale fac scaling factor for conductivity 1.0
diffuse cutoff density set a cutoff density for diffusion – we zero the term out below this density -1.e200
diffuse enth enable enthalpy diffusion 0
diffuse spec enable species diffusion 0
diffuse temp enable thermal diffusion 0
diffuse vel enable velocity diffusion 0
6.3—castro
Nam
espace
63
Table 6.4: castro : embiggening parameters
parameter description default value
grown factor the factor by which to extend the domain upon restart for embiggening 1
star at center used with the embiggening routines to determine how to extend the domain -1
Table 6.5: castro : gravity and rotation parameters
parameter description default value
do grav permits gravity calculation to be turned on and off -1
do rotation permits rotation calculation to be turned on and off -1
grav source type determines how the gravitational source term is added to the momentum and energystate variables.
4
implicit rotation update we can do a implicit solution of the rotation update to allow for better coupling of theCoriolis terms
1
moving center to we recompute the center used for the multipole gravity solve each step? 0
point mass mass of the point mass 0.0
point mass fix solution if we have a central point mass, we can prevent mass from building up in the zonesadjacent to it by keeping their density constant and adding their mass to the pointmass object
0
rot axis the coordinate axis (x = 1, y = 2, z = 3) for the rotation vector 3
rot source type determines how the rotation source terms are added to the momentum and energyequations
4
rotation include centrifugal permits the centrifugal terms in the rotation to be turned on and off 1
rotation include coriolis permits the Coriolis terms in the rotation to be turned on and off 1
rotation include domegadt permits the d(omega)/dt terms in the rotation to be turned on and off 1
continued on next page
64C
hap
ter6.
Ru
ntim
eP
arameters
Table 6.5—continued
parameter description default value
rotational dPdt the rotation periods time evolution—this allows the rotation rate to change durning thesimulation time
0.0
rotational period the rotation period for the corotating frame -1.e200
state in rotating frame Which reference frame to measure the state variables with respect to. The standard inthe literature when using a rotating reference frame is to measure the state variableswith respect to an observer fixed in that rotating frame. If this option is disabled bysetting it to 0, the state variables will be measured with respect to an observer fixed inthe inertial frame (but the frame will still rotate).
1
use point mass include a central point mass 1
Table 6.6: castro : hydrodynamics parameters
parameter description default value
add ext src if true, define an additional source term 0
allow small energy Whether or not to allow the internal energy to be less than the internal energy corre-sponding to small temp
1
cg blend for the Colella & Glaz Riemann solver, what to do if we do not converge to a solutionfor the star state. 0 = do nothing; print iterations and exit 1 = revert to the originalguess for p-star 2 = do a bisection search for another 2 * cg maxiter iterations.
2
cg maxiter for the Colella & Glaz Riemann solver, the maximum number of iterations to take whensolving for the star state
12
cg tol for the Colella & Glaz Riemann solver, the tolerance to demand in finding the star state 1.0e-5
density reset method Which method to use when resetting a negative/small density 1 = Reset to character-istics of adjacent zone with largest density 2 = Use average of all adjacent zones for allstate variables 3 = Reset to the original zone state before the hydro update
1
difmag the coefficient of the artificial viscosity 0.1
continued on next page
6.3—castro
Nam
espace
65
Table 6.6—continued
parameter description default value
do ctu do we do the CTU unsplit method or a method-of-lines approach? 1
do hydro permits hydro to be turned on and off for running pure rad problems -1
do sponge permits sponge to be turned on and off 0
dual energy eta1 Threshold value of (E - K) / E such that above eta1, the hydrodynamic pressure isderived from E - K; otherwise, we use the internal energy variable UEINT.
1.0e0
dual energy eta2 Threshold value of (E - K) / E such that above eta2, we update the internal energyvariable UEINT to match E - K. Below this, UEINT remains unchanged.
1.0e-4
first order hydro set the flattening parameter to zero to force the reconstructed profiles to be flat, resultingin a first-order method
0
fix mass flux 0
fourth order do we do fourth-order accurate MOL hydro? 0
hse interp temp if we are doing HSE boundary conditions, should we get the temperature via interpola-tion (using model parser) or hold it constant?
0
hse reflect vels if we are doing HSE boundary conditions, how do we treat the velocity? reflect? oroutflow?
0
hse zero vels if we are doing HSE boundary conditions, do we zero the velocity? 0
hybrid hydro whether to use the hybrid advection scheme that updates z-angular momentum, cylin-drical momentum, and azimuthal momentum (3D only)
0
hybrid riemann do we drop from our regular Riemann solver to HLL when we are in shocks to avoidthe odd-even decoupling instability?
0
limit fluxes on small dens Should we limit the density fluxes so that we do not create small densities? 0
mol order integration order for MOL integration 1 = first order, 2 = second order TVD, 3 = 3rdorder TVD, 4 = 4th order RK
2
plm iorder for piecewise linear, reconstruction order to use 2
continued on next page
66C
hap
ter6.
Ru
ntim
eP
arameters
Table 6.6—continued
parameter description default value
ppm predict gammae do we construct γe = p/(ρe) + 1 and bring it to the interfaces for additional thermody-namic information (this is the Colella & Glaz technique) or do we use (ρe) (the classicCastro behavior). Note this also uses τ = 1/ρ instead of ρ.
0
ppm reference eigenvectors do we use the reference state in evaluating the eigenvectors? 0
ppm temp fix various methods of giving temperature a larger role in the reconstruction—see Zingale& Katz 2015
0
ppm type reconstruction type: 0: piecewise linear; 1: classic Colella & Woodward ppm; 2:extrema-preserving ppm
1
riemann solver which Riemann solver do we use: 0: Colella, Glaz, & Ferguson (a two-shock solver); 1:Colella & Glaz (a two-shock solver) 2: HLLC
0
small dens the small density cutoff. Densities below this value will be reset -1.e200
small ener the small specific internal energy cutoff. Internal energies below this value will be reset -1.e200
small pres the small pressure cutoff. Pressures below this value will be reset -1.e200
small temp the small temperature cutoff. Temperatures below this value will be reset -1.e200
source term predictor extrapolate the source terms (gravity and rotation) to n + 1/2 timelevel for use in theinterface state prediction
0
sponge implicit if we are using the sponge, whether to use the implicit solve for it 1
transverse reset density if the transverse interface state correction, if the new density is negative, then replaceall of the interface quantities with their values without the transverse correction.
1
transverse reset rhoe if the interface state for (ρe) is negative after we add the transverse terms, then replacethe interface value of (ρe) with a value constructed from the (ρe) evolution equation
0
transverse use eos after we add the transverse correction to the interface states, replace the predictedpressure with an EOS call (using e and ρ).
0
use eos in riemann should we use the EOS in the Riemann solver to ensure thermodynamic consistency? 0
use flattening flatten the reconstructed profiles around shocks to prevent them from becoming toothin
1
continued on next page
6.3—castro
Nam
espace
67
Table 6.6—continued
parameter description default value
use pslope for the piecewise linear reconstruction, do we subtract off (ρg) from the pressure beforelimiting?
1
xl ext bc type if we are doing an external -x boundary condition, who do we interpret it? ””
xr ext bc type if we are doing an external +x boundary condition, who do we interpret it? ””
yl ext bc type if we are doing an external -y boundary condition, who do we interpret it? ””
yr ext bc type if we are doing an external +y boundary condition, who do we interpret it? ””
zl ext bc type if we are doing an external -z boundary condition, who do we interpret it? ””
zr ext bc type if we are doing an external +z boundary condition, who do we interpret it? ””
Table 6.7: castro : parallelization parameters
parameter description default value
bndry func thread safe 1
do acc determines whether we use accelerators for specific loops -1
Table 6.8: castro : particles parameters
parameter description default value
do tracer particles permits tracer particle calculation to be turned on and off 0
68C
hap
ter6.
Ru
ntim
eP
arameters
Table 6.9: castro : reactions parameters
parameter description default value
disable shock burning disable burning inside hydrodynamic shock regions 0
do react permits reactions to be turned on and off – mostly for efficiency’s sake -1
dtnuc X Limit the timestep based on how much the burning can change the species mass fractionsof a zone. The timestep is equal to dtnuc · (X/X).
1.e200
dtnuc X threshold If we are using the timestep limiter based on changes in X, set a threshold on the speciesabundance below which the limiter is not applied. This helps prevent the timestep frombecoming very small due to changes in trace species.
1.e-3
dtnuc e Limit the timestep based on how much the burning can change the internal energy of azone. The timestep is equal to dtnuc · (e/e).
1.e200
dxnuc limit the zone size based on how much the burning can change the internal energy ofa zone. The zone size on the finest level must be smaller than dxnuc · cs · (e/e), wherecs is the sound speed. This ensures that the sound-crossing time is smaller than thenuclear energy injection timescale.
1.e200
dxnuc max Disable limiting based on dxnuc above this threshold. This allows zones that havealready ignited or are about to ignite to be de-refined.
1.e200
max dxnuc lev Disable limiting based on dxnuc above this AMR level. -1
react T max maximum temperature for allowing reactions to occur in a zone 1.e200
react T min minimum temperature for allowing reactions to occur in a zone 0.0
react rho max maximum density for allowing reactions to occur in a zone 1.e200
react rho min minimum density for allowing reactions to occur in a zone 0.0
6.3—castro
Nam
espace
69
Table 6.10: castro : refinement parameters
parameter description default value
do special tagging 0
spherical star 0
Table 6.11: castro : timestep control parameters
parameter description default value
cfl the effective Courant number to use—we will not allow the hydrodynamic waves to crossmore than this fraction of a zone over a single timestep
0.8
change max the maximum factor by which the timestep can increase from one step to the next. 1.1
clamp subcycles If we do request more than the maximum number of subcycles, should we fail, or shouldwe clamp to that maximum number and perform that many?
1
dt cutoff the smallest valid timestep—if we go below this, we abort 0.0
fixed dt a fixed timestep to use for all steps (negative turns it off) -1.0
init shrink a factor by which to reduce the first timestep from that requested by the timestepestimators
1.0
initial dt the initial timestep (negative uses the step returned from the timestep constraints) -1.0
max dt the largest valid timestep—limit all timesteps to be no larger than this 1.e200
max subcycles Do not permit more subcycled timesteps than this parameter. Set to a negative valueto disable this criterion.
10
plot per is exact enforce that the AMR plot interval must be hit exactly 0
retry neg dens factor If we’re doing retries, set the target threshold for changes in density if a retry is triggeredby a negative density. If this is set to a negative number then it will disable retries usingthis criterion.
1.e-1
continued on next page
70C
hap
ter6.
Ru
ntim
eP
arameters
Table 6.11—continued
parameter description default value
retry subcycle factor When performing a retry, the factor to multiply the current timestep by when tryingagain.
0.5
retry tolerance Tolerance to use when evaluating whether to do a retry. The timestep suggested by theretry will be multiplied by (1 + this factor) before comparing the actual timestep toit. If set to some number slightly larger than zero, then this prevents retries that arecaused by small numerical differences.
0.02
sdc iters Number of iterations for the SDC advance. 2
small plot per is exact enforce that the AMR small plot interval must be hit exactly 0
use post step regrid Check for a possible post-timestep regrid if certain stability criteria were violated. 0
use retry Retry a timestep if it violated the timestep-limiting criteria over the course of an ad-vance. The criteria will suggest a new timestep that satisfies the criteria, and we willdo subcycled timesteps on the same level until we reach the original target time.
0
6.4— diffusion Namespace 71
6.4 diffusion Namespace
72C
hap
ter6.
Ru
ntim
eP
arameters
Table 6.12: diffusion parameters
parameter description default value
mlmg maxorder Use MLMG as the operator 4
v the level of verbosity for the diffusion solve (higher number means more output) 0
6.5— gravity Namespace 73
6.5 gravity Namespace
74C
hap
ter6.
Ru
ntim
eP
arameters
Table 6.13: gravity parameters
parameter description default value
const grav if doing constant gravity, what is the acceleration 0.0
direct sum bcs Check if the user wants to compute the boundary conditions using the brute forcemethod. Default is false, since this method is slow.
0
do composite phi correction should we apply a lagged correction to the potential that gets us closer to the compositesolution? This makes the resulting fine grid calculation slightly more accurate, at thecost of an additional Poisson solve per timestep.
1
drdxfac ratio of dr for monopole gravity binning to grid resolution 1
get g from phi For non-Poisson gravity, do we want to construct the gravitational acceleration by takingthe gradient of the potential, rather than constructing it directly?
0
gravity type what type ”fillme”
max multipole order the maximum mulitpole order to use for multipole BCs when doing Poisson gravity 0
max solve level For all gravity types, we can choose a maximum level for explicitly calculating thegravity and associated potential. Above that level, we interpolate from coarser levels.
MAX LEV-1
mlmg agglomeration Do agglomeration? 1
mlmg consolidation 1
mlmg max fmg iter how many FMG cycles? 0
mlmg nsolve Do N-Solve? 0
no composite do we do a composite solve? 0
no sync do we perform the synchronization at coarse-fine interfaces? 0
v the level of verbosity for the gravity solve (higher number means more output on thestatus of the solve / multigrid
0
6.6— particles Namespace 75
6.6 particles Namespace
76C
hap
ter6.
Ru
ntim
eP
arameters
Table 6.14: particles parameters
parameter description default value
particle init file the name of an input file containing the total particle number and the initial positionof each particle.
””
particle output file the name of timestamp files. ””
particle restart file the name of a file with new particles at restart ””
restart from nonparticle chkfile to restart from a checkpoint that was written with USE PARTICLES=FALSE 0
timestamp density whether the local densities at given positions of particles are stored in output files 1
timestamp dir the name of a directory in which timestamp files are stored. ””
timestamp temperature whether the local temperatures at given positions of particles are stored in output files 0
v the level of verbosity for the tracer particle (0 or 1) 0
CHAPTER 7
Compilers, Managing Jobs, and Scaling
Castro requires both a C++ compiler (with support for C++ 11) and a Fortran compiler with 2003+support.
The information on running at various HPC centers has been moved to https://amrex-astro.
github.io/workflow/.
7.1 Scaling
Data from scaling studies is archived in Castro/Docs/ManagingJobs/scaling/
Needs to be updated
77
CHAPTER 8
Frequently Asked Questions
8.1 Compiling
1. Compiling fails giving me a cryptic message about a module not being found, usually bl types
or bl error module, like:
mpif90 -fno-range-check -fno-second-underscore
-Jo/3d.Linux.gcc.gfortran.MPI.EXE -I o/3d.Linux.gcc.gfortran.MPI.EXE
-ffixed-line-length-0 -g -O3
-I. -I/home/zingale/development/Microphysics/util
-I../../Microphysics/EOS -I../../Microphysics/EOS/gamma_law
-I../../Microphysics/networks
-I../../Microphysics/networks/general_null
-I. -I/home/zingale/development/BoxLib//Src/C_BaseLib
-I/home/zingale/development/BoxLib//Src/C_AMRLib
-I/home/zingale/development/BoxLib//Src/C_BoundaryLib -I../../Source
-I../../Source/Src_3d -I../../Source/Src_nd -I../../constants
-I/home/zingale/development/BoxLib//Src/F_BaseLib
-I/home/zingale/development/BoxLib//Tools/C_scripts -c
../../Microphysics/EOS/eos.f90 -o
o/3d.Linux.gcc.gfortran.MPI.EXE/eos.o
../../Microphysics/EOS/eos.f90:3:6:
use bl_types
1
Fatal Error: Can’t open module file bl_types.mod for reading at (1):
No such file or directory compilation terminated.
79
80 Chapter 8. Frequently Asked Questions
/home/zingale/development/BoxLib//Tools/C_mk/Make.rules:122: recipe
for target ’o/3d.Linux.gcc.gfortran.MPI.EXE/eos.o’ failed
This usually indicates that the build system cannot find a source file (note: the problem aboveis not bl types, that just seems to be the way the error manifests itself). The source files arespecified in the various Make.package files throughout the Castro directory hierarchy. make
will look through the directories in the VPATH LOCATIONS to find the files.
There are 2 things you can do to check what’s happening. First, inspect the directories inVPATH LOCATIONS. This can be done via:
make print-VPATH_LOCATIONS
Next, ask make to tell you where it is finding each of the source files. This is done through ascript find files vpath.py that is hooked into Castro’s build system. You can run this as:
make file_locations
At the end of the report, it will list any files it cannot find in the vpath. Some of these are tobe expected (like extern.f90 and buildInfo.cpp—these are written at compiletime. Butany other missing files need to be investigated.
2. I’m still having trouble compiling. How can I find out what all of the make variables are setto?
Use:
make help
This will tell you the value of all the compilers and their options.
3. How do I use a system’s BLAS library instead of compiling and linking the one that comeswith the StarKiller microphysics?
To use a system’s BLAS library, set the Make variable USE SYSTEM BLAS to TRUE. This willthen look at the Make variable BLAS LIBRARY for the library to link (defaults to -lopenblas).
4. How can I check to make sure the function signatures defined in C++ are consistent with theirimplementations in Fortran?
Use:
make typecheck
This will compile the code and report on any mismatched function signatures.
8.2 Debugging
1. Castro crashes with a floating point exception—how can I get more information?
The best thing to do is to recompile the code with TEST=TRUE set in the GNUmakefile. Thiswill have AMReX catch the signals raised in both C++ and Fortran functions. Behind thescenes, this defines the BL TESTING preprocessor flag, which will initialize memory allocated infabs or multifabs to signaling NaNs (sNaN), and use the BLBackTrace::handler() functionto handle various signals raised in both C++ and Fortran functions. This is a Linux/UNIX
8.2—Debugging 81
capability. This gives us a chance to print out backtrace information. The signals include segfault, floating point exceptions (NaNs, divided by zero and overflow), and interruption by theuser and system. What signals are handed to AMReX are controlled by AMReX(e.g., usinginterruption by the user, this was once used to find an MPI deadlock.) It also includes theBL ASSERTION statements if USE ASSERTION=TRUE or DEBUG=TRUE.
The AMReX parameters that affect the behavior are:
• amrex.fpe trap invalid
• amrex.fpe trap zero
• amrex.fpe trap overflow
For further capabilities, defining BACKTRACE=TRUE enables you to get more information thanthe backtrace of the call stack info by instrumenting the code. (This is in C++ code only).Here is an example. You know the line “Real rho = state(cell,0);” is causing a segfault.You could add a print statement before that. But it might print out thousands (or evenmillions) of line before it hits the segfault. With BACKTRACE, you could do
#ifdef BL_BACKTRACING
std::ostringstream ss;
ss << ‘‘state.box() = ‘‘ << state.box() << ‘‘ cell = ‘‘ << cell;
BL_BACKTRACE_PUSH(ss.str()); // PUSH takes std::string
#endif
Real rho = state(cell,0); // state is a Fab, and cell is an IntVect.
The “print” prints to a stack of string, not stdout. When it hits the segfault, you will onlysee the last print out.
2. How can I monitor the state in a zone from the C++ side at various points in the evolution?
Given a MultiFab mf, you can dump out the state as:
print_state(mf, IntVect(D_DECL(10, 20, 30)));
Here, the IntVect has the dimension that we were compiled with (and this is handled throughthe preprocessor D DECL). In this case, we are inspecting zone (10, 20, 30), in the global indexspace. Note that since a multifab exists only on a single level, the integer indices here referto the global index space on that level.
3. What if I want to see all the data in a FArrayBox?
You can simply output a FAB to std::cout. Imagine that you are in an MFIter loop, witha MultiFab mf:
S = FArrayBox& mf[mfi];
std::cout << S << std::endl;
This will output the contents on the FAB, one zone per line.
82 Chapter 8. Frequently Asked Questions
8.3 Profiling
1. How can I get line-by-line profiling information?
With the GNU compliers, you can enabling profiling with gprof by compiling with
USE_GPROF=TRUE
in your GNUmakefile.
When you run, a file named gmon.out will be produced. This can be processed with gprof
by running:
gprof exec-name
where exec-name is the name of the executable. More detailed line-by-line information canbe obtained by passing the -l argument to gprof.
8.4 Managing Runs
1. How can I force the running code to output, even it the plot or checkpoint interval parametersdon’t require it?
Create a file called dump and continue, e.g., as:
touch dump_and_continue
This will force the code to output a checkpoint file that can be used to restart. Otheroptions are plot and continue to output a plotfile, dump and stop to output a checkpointfile and halt the code, and stop run to simply stop the code. Note that the parameteramr.message int controls how often the existence of these files is checked; by default it is10, so the check will be done at the end of every timestep that is a multiple of 10. Set thatto 1 in your inputs file if you’d like it to check every timestep.
2. How can I output plotfiles in single precision?
The AMReX runtime parameter:
fab.format = NATIVE_32
controls this (put this in your inputs file). Note: checkpoint files are unaffected by this andwill always be written out in the native precision (the ‘fab.format‘ parameter is overridden inthe checkpoint code in AMReX).
8.5 Runtime Errors
1. When running with retries, Castro requests too many substeps and crashes
This can occur due to CFL violations or negative densities. If there are density resets,try running with castro.limit fluxes on small dens = 1. This will use a flux limiter toprevent the density from going negative.
8.6—Visualization 83
8.6 Visualization
1. When I try to use Amrvis with the Nvidia driver, all I see is black—no data. How do I fixthis?
You need to edit your xorg.conf file (usually found in /etc/X11/ to enable the Dac8Bit
option. The section will look like:
Section "Device"
Identifier "Device0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
Option "Dac8bit" "True"
EndSection
If you don’t already have an xorg.conf then you can create one by running nvidia-xconfig
first.
CHAPTER 9
Hydrodynamics
9.1 Introduction
The hydrodynamics scheme in Castro implements an unsplit second-order Godunov method. Char-acteristic tracing is used to time-center the input states to the Riemann solver. The same hydro-dynamics routines are used for pure hydro and radiation hydrodynamics.
Some general notes:
• Regardless of the dimensionality, we always carry around all 3 components of velocity/momentum—this allows for rotation sources easily.
• When radiation is enabled (via RADIATION), we discuss the gas and radiation quantities sepa-rately. This generally applies to the temperature, pressure, internal energy, various adiabaticindices, and sound speed. When we refer to the “total” value of one of these, it means thatboth gas and radiation contributions are included. When we refer to the “gas” quantity, thisis what the equation of state would return.
For continuity, we continue to use the “gas” qualifier even if we are not solving the radiationhydrodynamics equations. In this case, it still means that it comes through the equation ofstate, but note some of our equations of state (like the helmeos) include a radiation pressurecontribution when we are running without radiation hydrodynamics enabled. In this case, westill refer to this as the “gas”.
9.2 Hydrodynamics Data Structures
Within the Fortran routines that implement the hydrodynamics, there are several main data struc-tures that hold the state.
85
86 Chapter 9. Hydrodynamics
variable quantity note
URHO ρUMX ρuUMY ρvUMZ ρwUEDEN ρEUEINT ρe this is computed from the other quantities using ρe =
ρE − ρu · u/2UTEMP T this is computed from the other quantities using the
EOSUFA ρA1 the first advected quantityUFS ρX1 the first speciesUFX ρY1 the first auxiliary variableUSHK a shock flag (used for shock detection)UMR radial momentum (if HYBRID MOMENTUM is defined)UML angular momentum (if HYBRID MOMENTUM is defined)UMP vertical momentum (if HYBRID MOMENTUM is defined)
Table 9.1: The integer variables to index the conservative state array
variable quantity note
QRHO ρQU uQV vQW wQPRES pQREINT ρeQTEMP TQGAME p/(ρe) + 1QFA A1 the first advected quantityQFS X1 the first speciesQFX Y1 the first auxiliary variableQPTOT ptot the total pressure, gas + radiationQREITOT etot the total specific internal energy, gas + radiationQRAD Er the radiation energy (there are ngroups of these)
Table 9.2: The integer variable keys for accessing the primitive state vector. Note, unless otherwisespecified the quantities without a subscript are “gas” only and those with the “tot” subscript are“gas + radiation”.
• conserved state: these arrays generally begin with u, e.g., uin, uout. The NVAR componentsfor the state data in the array are accessed using integer keys defined in 9.1.
• primitive variable state: these arrays generally simply called q, and has NQ components. Note:if RADIATION is defined, then there are QVAR components that are pure hydro out of the totalNQ components, and the pure hydro components always come first in the state array.
Table 9.2 gives the names of the primitive variable integer keys for accessing these arrays.
9.2—Hydrodynamics Data Structures 87
variable quantity note
QGAMC γ1 the first adiabatic exponent, as returned from theEOS
QC cs the sound speed, as returned from the EOSQCSML a small sound speed used for cutoffsQDPDR ∂p/∂ρ|e computed via the EOSQDPDE ∂p/∂e|ρ computed via the EOSQGAMCG Γ1tot includes radiation components (defined only if
RADIATION is defined)QCG cstot total sound speed including radiation (defined only if
RADIATION is defined)QLAMS λf the ngroups flux limiters (defined only if RADIATION
is defined)
Table 9.3: The integer variable keys for accessing the auxiliary primitive state vector, quax. Note,unless otherwise specified the quantities without a subscript are “gas” only and those with the“tot” subscript are “gas + radiation”.
variable quantity note
QGDRHO ρQDU uQDV vQDW wQDPRES p regardless of whether RADIATION is defined, this is
always just the gas pressureQDGAME γe = p/(ρe) + 1 regardless of whether RADIATION is defined, this is
always just the gas contributionQDLAMS λf the starting index for the flux limiter—there are
ngroups components (defined only if RADIATION isdefined)
QDERADS Er the starting index for the radiation energy—there arengroups components (defined only if RADIATION isdefined)
Table 9.4: The integer variable keys for accessing the Godunov interface state vectors. Note,unless otherwise specified the quantities without a subscript are “gas” only and those with the“tot” subscript are “gas + radiation”.
• auxiliary primitive variables: these arrays are generally called qaux. The main differencebetween these and the regular primitive variables is that we do not attempt to do any recon-struction on their profiles. There are NQAUX quantities, indexed by the integer keys listed intable 9.3.
• interface variable: these are the time-centered interface states returned by the Riemann solver.They are used to discretize some non-conservative terms in the equations. These arrays aregenerally called qx, qy, and qz for the x, y, and z interfaces respectively (in some places thenumbers 1, 2, and 3 are used instead). There are NGDNV components accessed with the integerkeys defined in table 9.4
88 Chapter 9. Hydrodynamics
9.3 Conservation Forms
We begin with the fully compressible equations for the conserved state vector, U = (ρ, ρu, ρE, ρAk, ρXk, ρYk) :
∂ρ
∂t= −∇ · (ρu) + Sext,ρ, (9.1)
∂(ρu)
∂t= −∇ · (ρuu)−∇p+ ρg + Sext,ρu, (9.2)
∂(ρE)
∂t= −∇ · (ρuE + pu) + ρu · g −
∑k
ρqkωk +∇ · kth∇T + Sext,ρE , (9.3)
∂(ρAk)
∂t= −∇ · (ρuAk) + Sext,ρAk
, (9.4)
∂(ρXk)
∂t= −∇ · (ρuXk) + ρωk + Sext,ρXk
, (9.5)
∂(ρYk)
∂t= −∇ · (ρuYk) + Sext,ρYk . (9.6)
Here ρ,u, T, p, and kth are the density, velocity, temperature, pressure, and thermal conductivity,respectively, and E = e + u · u/2 is the total energy with e representing the internal energy. Inaddition, Xk is the abundance of the kth isotope, with associated production rate, ωk, and energyrelease, qk. Here g is the gravitational vector, and Sext,ρ,Sextρu, etc., are user-specified sourceterms. Ak is an advected quantity, i.e., a tracer. We also carry around auxiliary variables, Yk,which have a user-defined evolution equation, but by default are treated as advected quantities.
In the code we also carry around T and ρe in the conservative state vector even though they arederived from the other conserved quantities. The ordering of the elements within U is defined byinteger variables into the array—see Table 9.1
Some notes:
• Regardless of the dimensionality of the problem, we always carry all 3 components of thevelocity. This allows for, e.g., 2.5-d rotation (advecting the component of velocity out of theplane in axisymmetric coordinates).
You should always initialize all velocity components to zero, and always construct the kineticenergy with all three velocity components.
• There are NADV advected quantities, which range from UFA: UFA+nadv-1. The advectedquantities have no effect at all on the rest of the solution but can be useful as tracer quantities.
• There are NSPEC species (defined in the network directory), which range from UFS: UFS+nspec-1.
• There are NAUX auxiliary variables, from UFX:UFX+naux-1 The auxiliary variables are passedinto the equation of state routines along with the species; An example of an auxiliary variableis the electron fraction, Ye, in core collapse simulations.note about
qpass map here
9.4 Source Terms
We now compute explicit source terms for each variable in Q and U. The primitive variable sourceterms will be used to construct time-centered fluxes. The conserved variable source will be used
9.5—Primitive Forms 89
to advance the solution. We neglect reaction source terms since they are accounted for in Steps 1and 6. The source terms are:
SnQ =
SρSu
SpSρeSAk
SXk
SYk
n
=
Sext,ρ
g + 1ρSext,ρu
1ρ∂p∂eSext,ρE + ∂p
∂ρSextρ
∇ · kth∇T + Sext,ρE1ρSext,ρAk1ρSext,ρXk1ρSext,ρYk
n
, (9.7)
SnU =
SρuSρESρAk
SρXk
SρYk
n
=
ρg + Sext,ρu
ρu · g +∇ · kth∇T + Sext,ρE
Sext,ρAk
Sext,ρXk
Sext,ρYk
n
. (9.8)
9.5 Primitive Forms
Castro uses the primitive form of the fluid equations, defined in terms of the state Q =(ρ,u, p, ρe, Ak, Xk, Yk), to construct the interface states that are input to the Riemann problem.
The primitive variable equations for density, velocity, and pressure are:
∂ρ
∂t= −u · ∇ρ− ρ∇ · u + Sext,ρ (9.9)
∂u
∂t= −u · ∇u− 1
ρ∇p+ g +
1
ρ(Sext,ρu − u Sext,ρ) (9.10)
∂p
∂t= −u · ∇p− ρc2∇ · u +
(∂p
∂ρ
)e,X
Sext,ρ
+1
ρ
∑k
(∂p
∂Xk
)ρ,e,Xj ,j 6=k
(ρωk + Sext,ρXk−XkSext,ρ)
+1
ρ
(∂p
∂e
)ρ,X
[−eSext,ρ −
∑k
ρqkωk +∇ · kth∇T
+ Sext,ρE − u ·(Sext,ρu −
u
2Sext,ρ
)](9.11)
The advected quantities appear as:
∂Ak∂t
= −u · ∇Ak +1
ρ(Sext,ρAk
−AkSext,ρ), (9.12)
∂Xk
∂t= −u · ∇Xk + ωk +
1
ρ(Sext,ρXk
−XkSext,ρ), (9.13)
∂Yk∂t
= −u · ∇Yk +1
ρ(Sext,ρYk − YkSext,ρ). (9.14)
90 Chapter 9. Hydrodynamics
All of the primitive variables are derived from the conservative state vector, as described in Section9.6.1. When accessing the primitive variable state vector, the integer variable keys for the differentquantities are listed in Table 9.2.
9.5.1 Internal energy and temperature
We augment the above system with an internal energy equation:
∂(ρe)
∂t= −u · ∇(ρe)− (ρe+ p)∇ · u−
∑k
ρqkωk +∇ · kth∇T + Sext,ρE
− u ·(
Sext,ρu −1
2Sext,ρu
), (9.15)
This has two benefits. First, for a general equation of state, carrying around an additional thermo-Since (rhoe) is inthe conserved state,I don’t think wederive the internalenergy source fromtotal andmomentum sourcesanymore
dynamic quantity allows us to avoid equation of state calls (in particular, in the Riemann solver,see e.g. [7]). Second, it is sometimes the case that the internal energy calculated as
eT ≡ E −1
2v2 (9.16)
is unreliable. This has two usual causes: one, for high Mach number flows, the kinetic energycan dominate the total gas energy, making the subtraction numerically unreliable; two, if you usegravity or other source terms, these can indirectly alter the value of the internal energy if obtainedfrom the total energy.
To provide a more reasonable internal energy for defining the thermodynamic state, we have im-plemented the dual energy formalism from ENZO [5, 4], where we switch between (ρe) and (ρeT )depending on the local state of the fluid. To do so, we define parameters η1, η2, and η3, cor-responding to the code parameters castro.dual energy eta1, castro.dual energy eta2, andcastro.dual energy eta3. We then consider the ratio eT /E, the ratio of the internal energy(derived from the total energy) to the total energy. These parameters are used as follows:
• η1: If eT > η1E, then we use eT for the purpose of calculating the pressure in the hydrody-namics update. Otherwise, we use the e from the internal energy equation in our EOS callto get the pressure.
• η2: At the end of each hydro advance, we examine whether eT > η2E. If so, we reset e to beequal to eT , discarding the results of the internal energy equation. Otherwise, we keep e asit is.
Optionally we can also update E so that it gains the difference of the old and and new e, bysetting castro.dual energy update E from e to 1.
• η3: Similar to η1, if eT > η3E, we use eT for the purposes of our nuclear reactions, otherwise,we use e.
Note that our version of the internal energy equation does not require an artificial viscosity, as usedin some other hydrodynamics codes. The update for (ρe) uses information from the Riemann solveto calculate the fluxes, which contains the information intrinsic to the shock-capturing part of thescheme.
In the code we also carry around T in the primitive state vector.
9.6—Hydrodynamics Update 91
9.5.2 Primitive Variable System
The full primitive variable form (without the advected or auxiliary quantities) is
∂Q
∂t+∑d
Ad∂Q
∂xd= SQ. (9.17)
For example, in 2D:
ρuvpρeXk
t
+
u ρ 0 0 0 00 u 0 1
ρ 0 0
0 0 u 0 0 00 ρc2 0 u 0 00 ρe+ p 0 0 u 00 0 0 0 0 u
ρuvpρeXk
x
+
v 0 ρ 0 0 00 v 0 0 0 00 0 v 1
ρ 0 0
0 0 ρc2 v 0 00 0 ρe+ p 0 v 00 0 0 0 0 v
ρuvpρeXk
y
= SQ
(9.18)The eigenvalues are:
Λ(Ax) = u− c, u, u, u, u, u+ c, Λ(Ay) = v − c, v, v, v, v, v + c. (9.19)
The right column eigenvectors are:
R(Ax) =
1 1 0 0 0 1− cρ 0 0 0 0 c
ρ
0 0 1 0 0 0c2 0 0 0 0 c2
h 0 0 1 0 h0 0 0 0 1 0
, R(Ay) =
1 1 0 0 0 10 0 1 0 0 0− cρ 0 0 0 0 c
ρ
c2 0 0 0 0 c2
h 0 0 1 0 h0 0 0 0 1 0
. (9.20)
The left row eigenvectors, normalized so that Rd · Ld = I are:
Lx =
0 − ρ2c 0 1
2c20 0
1 0 0 − 1c2
0 00 0 1 0 0 0
0 0 0 − hc2
1 00 0 0 0 0 10 ρ
2c 0 12c2
0 0
, Ly =
0 0 − ρ2c
12c2
0 01 0 0 − 1
c20 0
0 1 0 0 0 0
0 0 0 − hc2
1 00 0 0 0 0 10 0 ρ
2c1
2c20 0
. (9.21)
9.6 Hydrodynamics Update
There are four major steps in the hydrodynamics update:
1. Converting to primitive variables
2. Construction the edge states
3. Solving the Riemann problem
4. Doing the conservative update
92 Chapter 9. Hydrodynamics
Each of these steps has a variety of runtime parameters that affect their behavior. Additionally,there are some general runtime parameters for hydrodynamics:
• castro.do hydro: time-advance the fluid dynamical equations (0 or 1; must be set)
• castro.add ext src: include additional user-specified source term (0 or 1; default 0)
• castro.do sponge: call the sponge routine after the solution update (0 or 1; default: 0)
The purpose of the sponge is to damp velocities outside of a star, to prevent themfrom dominating the timestep constraint. The sponge parameters are set in your probinfile, in the &sponge namelist. You can sponge either on radius from the center (usingsponge lower radius and sponge upper radius) or on density (using sponge lower density
and sponge upper density). The timescale of the damping is set through sponge timescale.
• castro.normalize species: enforce that∑
iXi = 1 (0 or 1; default: 0)
• castro.fix mass flux: enforce constant mass flux at domain boundary (0 or 1; default: 1)
• castro.allow negative energy: is internal energy allowed to be negative? (0 or 1; default:1)
• castro.spherical star: this is used to set the boundary conditions by assuming the star isspherically symmetric in the outer regions (0 or 1; default: 0)
When used, Castro averages the values at a given radius over the cells that are inside thedomain to define a radial function. This function is then used to set the values outside thedomain in implementing the boundary conditions.
• castro.show center of mass: (0 or 1; default: 0)
Several floors are imposed on the thermodynamic quantities to prevet unphysical behavior:
• castro.small dens: (Real; default: -1.e20)
• castro.small temp: (Real; default: -1.e20)
• castro.small pres: (Real; default: -1.e20)
9.6.1 Compute Primitive Variables
We compute the primtive variables from the conserved variables.
• ρ, ρe: directly copy these from the conserved state vector
• u, Ak, Xk, Yk: copy these from the conserved state vector, dividing by ρ
• p, T : use the EOS.
First, if castro.allow negative energy is 0 (it defaults to 1) and e < 0, we do the following:
1. Use the EOS to set e = e(ρ, Tsmall, Xk).
2. If e < 0, abort the program with an error message.
Now, use the EOS to compute p, T = p, T (ρ, e,Xk).
9.6—Hydrodynamics Update 93
We also compute the flattening coefficient, χ ∈ [0, 1], used in the edge state prediction to furtherlimit slopes near strong shocks. We use the same flattening procedure described in the the theoriginal PPM paper [9] and the Flash paper [11]. A flattening coefficient of 1 indicates that noadditional limiting takes place; a flattening coefficient of 0 means we effectively drop order to afirst-order Godunov scheme (this convention is opposite of that used in the Flash paper). For eachcell, we compute the flattening coefficient for each spatial direction, and choose the minimum valueover all directions. As an example, to compute the flattening for the x-direction, here are the steps:
1. Define ζ
ζi =pi+1 − pi−1
max (psmall, |pi+2 − pi−2|). (9.22)
2. Define χχi = min 1,max[0, a(ζi − b)] , (9.23)
where a = 10 and b = 0.75 are tunable parameters. We are essentially setting χi = a(ζi − b),and then constraining χi to lie in the range [0, 1]. Then, if either ui+1 − ui−1 < 0 or
pi+1 − pi−1
min(pi+1, pi−1)≤ c, (9.24)
where c = 1/3 is a tunable parameter, then set χi = 0.
3. Define χ
χi =
1−max(χi, χi−1) pi+1 − pi−1 > 0
1−max(χi, χi+1) otherwise. (9.25)
The following runtime parameters affect the behavior here:
• castro.use flattening turns on/off the flattening of parabola near shocks (0 or 1; default1)
9.6.2 Edge State Prediction
We wish to compute a left and right state of primitive variables at each edge to be used as inputs tothe Riemann problem. There are several reconstruction techniques, a piecewise linear method thatfollows the description in Colella (1990) [6], the classic PPM limiters [9], and the new PPM limitersintroduced in Colella & Sekora (2008) [8]. The choice of limiters is determined by castro.ppm type.
For the new PPM limiters, we have further modified the method of [8] to eliminate sensitivity dueto roundoff error (modifications via personal communication with Colella).
We also use characteristic tracing with corner coupling in 3D, as described in Miller & Colella(2002) [19]. We give full details of the new PPM algorithm, as it has not appeared before in theliterature, and summarize the developments from Miller & Colella.
The PPM algorithm is used to compute time-centered edge states by extrapolating the base-timedata in space and time. The edge states are dual-valued, i.e., at each face, there is a left state anda right state estimate. The spatial extrapolation is one-dimensional, i.e., transverse derivatives areignored. We also use a flattening procedure to further limit the edge state values. The Miller &Colella algorithm, which we describe later, incorporates the transverse terms, and also describesthe modifications required for equations with additional characteristics besides the fluid velocity.
94 Chapter 9. Hydrodynamics
There are four steps to compute these dual-valued edge states (here, we use s to denote an arbitraryscalar from Q, and we write the equations in 1D, for simplicity):
• Step 1: Compute si,+ and si,−, which are spatial interpolations of s to the hi and lo side ofthe face with special limiters, respectively. Begin by interpolating s to edges using a 4th-orderinterpolation in space:
si+1/2 =7
12(si+1 + si)−
1
12(si+2 + si−1) . (9.26)
Then, if (si+1/2−si)(si+1−si+1/2) < 0, we limit si+1/2 a nonlinear combination of approximationsto the second derivative. The steps are as follows:
1. Define:
(D2s)i+1/2 = 3(si − 2si+1/2 + si+1
)(9.27)
(D2s)i+1/2,L = si−1 − 2si + si+1 (9.28)
(D2s)i+1/2,R = si − 2si+1 + si+2 (9.29)
2. Defines = sign
[(D2s)i+1/2
], (9.30)
(D2s)i+1/2,lim = smax
min[Cs∣∣(D2s)i+1/2,L
∣∣ , Cs ∣∣(D2s)i+1/2,R
∣∣ , s ∣∣(D2s)i+1/2
∣∣] , 0 ,(9.31)
where C = 1.25 as used in Colella and Sekora 2009. The limited value of si+1/2 is
si+1/2 =1
2(si + si+1)− 1
6(D2s)i+1/2,lim. (9.32)
Now we implement an updated implementation of the Colella & Sekora algorithm whicheliminates sensitivity to roundoff. First we need to detect whether a particular cell correspondsto an “extremum”. There are two tests.
– For the first test, defineαi,± = si±1/2 − si. (9.33)
If αi,+αi,− ≥ 0, then we are at an extremum.
– We only apply the second test if either |αi,±| > 2|αi,∓|. If so, we define:
(Ds)i,face,− = si−1/2 − si−3/2 (9.34)
(Ds)i,face,+ = si+3/2 − si−1/2 (9.35)
(Ds)i,face,min = min [|(Ds)i,face,−| , |(Ds)i,face,+|] . (9.36)
(Ds)i,cc,− = si − si−1 (9.37)
(Ds)i,cc,+ = si+1 − si (9.38)
(Ds)i,cc,min = min [|(Ds)i,cc,−| , |(Ds)i,cc,+|] . (9.39)
If (Ds)i,face,min ≥ (Ds)i,cc,min, set (Ds)i,± = (Ds)i,face,±. Otherwise, set (Ds)i,± =(Ds)i,cc,±. Finally, we are at an extreumum if (Ds)i,+(Ds)i,− ≤ 0.
9.6—Hydrodynamics Update 95
Thus concludes the extremum tests. The remaining limiters depend on whether we are at anextremum.
– If we are at an extremum, we modify αi,±. First, we define
(D2s)i = 6(αi,+ + αi,−) (9.40)
(D2s)i,L = si−2 − 2si−1 + si (9.41)
(D2s)i,R = si − 2si+1 + si+2 (9.42)
(D2s)i,C = si−1 − 2si + si+1 (9.43)
Then, define
s = sign[(D2s)i
], (9.44)
(D2s)i,lim = max
min[s(D2s)i, Cs
∣∣(D2s)i,L∣∣ , Cs ∣∣(D2s)i,R
∣∣ , Cs ∣∣(D2s)i,C∣∣] , 0 .
(9.45)Then,
αi,± =αi,±(D2s)i,lim
max [(D2s)i, 1× 10−10](9.46)
– If we are not at an extremum and |αi,±| > 2|αi,∓|, then define
s = sign(αi,∓) (9.47)
δIext =−α2
i,±4 (αj,+ + αj,−)
, (9.48)
badnotation–shouldn’treuse I here
δs = si∓1 − si, (9.49)
If sδIext ≥ sδs, then we perform the following test. If sδs− αi,∓ ≥ 1× 10−10, then
αi,± = −2δs− 2s[(δs)2 − δsαi,∓
]1/2 (9.50)
otherwise,
αi,± = −2αi,∓ (9.51)
Finally, si,± = si + αi,±.
• Step 2: Construct a quadratic profile using si,−, si, and si,+.
sIi (x) = si,− + ξ [si,+ − si,− + s6,i(1− ξ)] , (9.52)
s6 = 6si − 3 (si,− + si,+) , (9.53)
ξ =x− ihh
, 0 ≤ ξ ≤ 1. (9.54)
• Step 3: Integrate quadratic profiles. We are essentially computing the average value sweptout by the quadratic profile across the face assuming the profile is moving at a speed λk.
96 Chapter 9. Hydrodynamics
Define the following integrals, where σk = |λk|∆t/h:
I(k)+ (si) =
1
σkh
∫ (i+1/2)h
(i+1/2)h−σkhsIi (x)dx (9.55)
I(k)− (si) =
1
σkh
∫ (i−1/2)h+σkh
(i−1/2)hsIi (x)dx (9.56)
Plugging in (9.52) gives:
I(k)+ (si) = si,+ −
σk2
[si,+ − si,− −
(1− 2
3σk
)s6,i
], (9.57)
I(k)− (si) = si,− +
σk2
[si,+ − si,− +
(1− 2
3σk
)s6,i
]. (9.58)
• Step 4: Obtain 1D edge states by performing a 1D extrapolation to get left and right edgestates. Note that we include an explicit source term contribution.
sL,i+1/2 = si − χi∑
k:λk≥0
lk ·[si − I(k)
+ (si)]
rk +∆t
2Sni , (9.59)
sR,i−1/2 = si − χi∑
k:λk<0
lk ·[si − I(k)
− (si)]
rk +∆t
2Sni . (9.60)
Here, rk is the kth right column eigenvector of R(Ad) and lk is the kth left row eigenvectorthis is old—wedon’t do flatteningthis way, nor is thisthe reference statewe use
lf L(Ad). The flattening coefficient is χi.
In order to add the transverse terms in an spatial operator unsplit framework, the details followexactly as given in Section 4.2.1 in Miller & Colella, except for the details of the Riemann solver,which are given below.
For the reconstruction of the interface states, the following apply:
• castro.ppm type: use piecewise linear vs PPM algorithm (0, 1, 2; default: 1)
Values of 1 and 2 are both piecewise parabolic reconstruction, with 2 using updated limitersthat better preserve extrema.
• castro.ppm temp fix does various attempts to use the temperature in the reconstruction ofthe interface states. This is experimental.
• castro.ppm predict gammae reconstructs γe = p/(ρe) + 1 to the interfaces and does thenecessary transverse terms to aid in the conversion between the conserved and primitiveinterface states in the transverse flux routines (0 or 1; default 0)
• castro.ppm reference eigenvectors uses the reference states in the evaluation of the eigen-vectors for the characteristic projection (0 or 1; default 0)
The interface states are corrected with information from the transverse directions to make this asecond-order update. These transverse directions involve separate Riemann solves. Sometimes, theupdate to the interface state from the transverse directions can make the state ill-posed. There areseveral parameters that help fix this:
9.6—Hydrodynamics Update 97
• castro.transverse use eos: If this is 1, then we call the equation of state on the interface,using ρ, e, and Xk, to get the interface pressure. This should result in a thermodynamicallyconsistent interface state.
• castro.transverse reset density: If the transverse corrections result in a negative densityon the interface, then we reset all of the interface states to their values before the transversecorrections.
• castro.transverse reset rhoe: The transverse updates operate on the conserved state.Usually, we construct the interface (ρe) in the transverse update from total energy and thekinetic energy, however, if the interface (rhoe) is negative, and transverse reset rhoe =
1, then we explicitly discretize an equation for the evolution of (ρe), including its transverseupdate.
9.6.3 Riemann Problem
Castro has three main options for the Riemann solver—the Colella & Glaz solver [7] (the samesolver used by Flash), a simpler solver described in an unpublished manuscript by Colella, Glaz, &Ferguson, and an HLLC solver. The first two are both two-shock approximate solvers, but differin how they approximate the thermodynamics in the “star” region.
Inputs from the edge state prediction are ρL/R, uL/R, vL/R, pL/R, and (ρe)L/R (v represents all ofthe transverse velocity components). We also compute Γ ≡ d log p/d log ρ|s at cell centers and copythese to edges directly to get the left and right states, ΓL/R. We also define cavg as a face-centeredvalue that is the average of the neighboring cell-centered values of c. We have also computedρsmall, psmall, and csmall using cell-centered data.
Here are the steps. First, define (ρc)small = ρsmallcsmall. Then, define:
(ρc)L/R = max[(ρc)small,
∣∣ΓL/R, pL/R, ρL/R∣∣] . (9.61)
Define star states:
p∗ = max
[psmall,
[(ρc)LpR + (ρc)RpL] + (ρc)L(ρc)R(uL − uR)
(ρc)L + (ρc)R
], (9.62)
u∗ =[(ρc)LuL + (ρc)RuR] + (pL − pR)
(ρc)L + (ρc)R. (9.63)
If u∗ ≥ 0 then define ρ0, u0, p0, (ρe)0 and Γ0 to be the left state. Otherwise, define them to be theright state. Then, set
ρ0 = max(ρsmall, ρ0), (9.64)
and define
c0 = max
(csmall,
√Γ0p0
ρ0
), (9.65)
ρ∗ = ρ0 +p∗ − p0
c20
, (9.66)
(ρe)∗ = (ρe)0 + (p∗ − p0)(ρe)0 + p0
ρ0c20
, (9.67)
98 Chapter 9. Hydrodynamics
c∗ = max
(csmall,
√∣∣∣∣Γ0p∗
ρ∗
∣∣∣∣)
(9.68)
Then,
cout = c0 − sign(u∗)u0, (9.69)
cin = c∗ − sign(u∗)u∗, (9.70)
cshock =cin + cout
2. (9.71)
If p∗ − p0 ≥ 0, then cin = cout = cshock. Then, if cout = cin, we define ctemp = εcavg. Otherwise,ctemp = cout − cin. We define the fraction
f =1
2
[1 +
cout + cin
ctemp
], (9.72)
and constrain f to lie in the range f ∈ [0, 1].
To get the final “Godunov” state, for the transverse velocity, we upwind based on u∗.
vgdnv =
vL, u∗ ≥ 0
vR, otherwise. (9.73)
Then, define
ρgdnv = fρ∗ + (1− f)ρ0, (9.74)
ugdnv = fu∗ + (1− f)u0, (9.75)
pgdnv = fp∗ + (1− f)p0, (9.76)
(ρe)gdnv = f(ρe)∗ + (1− f)(ρe)0. (9.77)
Finally, if cout < 0, set ρgdnv = ρ0, ugdnv = u0, pgdnv = p0, and (ρe)gdnv = (ρe)0. If cin ≥ 0, setρgdnv = ρ∗, ugdnv = u∗, pgdnv = p∗, and (ρe)gdnv = (ρe)∗.
If instead the Colella & Glaz solver is used, then we define
γ ≡ p
ρe+ 1 (9.78)
on each side of the interface and follow the rest of the algorithm as described in the original paper.
For the construction of the fluxes in the Riemann solver, the following parameters apply:
• castro.riemann solver: this can be one of the following values:
– 0: the Colella, Glaz, & Ferguson solver.
– 1: the Colella & Glaz solver
– 2: the HLLC solver. Note: this should only be used with Cartesian geometries becauseit relies on the pressure term being part of the flux in the momentum equation.
The default is to use the solver based on an unpublished Colella, Glaz, & Ferguson manuscript(it also appears in [21]), as described in the original Castro paper [1].
The Colella & Glaz solver is iterative, and two runtime parameters are used to control itsbehavior:
9.7—Temperature Fixes 99
– castro.cg maxiter: number of iterations for CG algorithm (Integer; default: 12)
– castro.cg tol: tolerance for CG solver when solving for the “star” state (Real; default:1.0e-5)
– castro.cg blend: this controls what happens if the root finding in the CG solver fails.There is a nonlinear equation to find the pressure in the star region from the jumpconditions for a shock (this is the two-shock approximation—the left and right statesare linked to the star region each by a shock). The default root finding algorithm is asecant method, but this can sometimes fail.
The options here are:
∗ 0 : do nothing. The pressure from each iteration is printed and the code abortswith a failure
∗ 1 : revert to the original guess for p-star and carry through on the remainder of theRiemann solve. This is almost like dropping down to the CGF solver. The p-starused is very approximate.
∗ 2 : switch to bisection and do an additional cg maxiter iterations to find the root.Sometimes this can work where the secant method fails.
• castro.hybrid riemann: switch to an HLL Riemann solver when we are in a zone with ashock (0 or 1; default 0)
This eliminates an odd-even decoupling issue (see the oddeven problem). Note, this cannotbe used with the HLLC solver.
9.6.4 Compute Fluxes and Update
Compute the fluxes as a function of the primitive variables, and then advance the solution:
Un+1 = Un −∆t∇ · Fn+1/2 + ∆tSn. (9.79)
Again, note that since the source term is not time centered, this is not a second-order method.After the advective update, we correct the solution, effectively time-centering the source term.
9.7 Temperature Fixes
There are a number of experimental options for improving the behavior of the temperature in thereconstruction and interface state prediction. The options are controlled by castro.ppm temp fix,which takes values:
• 0: the default method—temperature is not considered
• 1: do parabolic reconstruction on T , giving I(k)+ (Ti). We then derive the pressure and internal
energy (gas portion) via the equation of state as:
I(k)+ (pi) = p(I(k)
+ (ρi), I(k)+ (Ti)) (9.80)
I(k)+ ((ρe)i) = (ρe)(I(k)
+ (ρi), I(k)+ (Ti)) (9.81)
100 Chapter 9. Hydrodynamics
The remainder of the hydrodynamics algorithm then proceeds unchanged.
• 2: on entering the Riemann solver, we recompute the thermodynamics on the interfaces toensure that they are all consistent. This is done by taking the interface values of ρ, e, Xk,and computing the corresponding pressure, p from this.
• 3: This does the characteristic tracing using the (τ, u, T ) eigensystem. Note: this is notwidely implemented—see the Sod stellar for an implementation.
9.8 Resets
9.8.1 Density Resets
Need to document density reset method
9.8.2 Energy
Need to document allow negative energy and allow small energy
9.8.3 Flux Limiting
Multi-dimensional hydrodynamic simulations often have numerical artifacts that result from thesharp density gradients. A somewhat common issue, especially at low resolution, is negative densi-ties that occur as a result of a hydro update. Castro contains a prescription for dealing with negativedensities, that resets the negative density to be similar to nearby zones. Various choices exist forhow to do this, such as resetting it to the original zone density before the update or resetting itto some linear combination of the density of nearby zones. The reset is problematic because thestrategy is not unique and no choice is clearly better than the rest in all cases. Additionally, it isnot specified at all how to reset momenta in such a case. Consequently, we desired to improve thesituation by limiting fluxes such that negative densities could not occur, so that such a reset wouldin practice always be avoided. Our solution implements the positivity-preserving method of [14].This behavior is controlled by castro.limit fluxes on small dens.
A hydrodynamical update to a zone can be broken down into an update over every face of the zonewhere a flux crosses the face over the timestep. The central insight of the positivity-preservingmethod is that if the update over every face is positivity-preserving, then the total update mustbe positivity-preserving as well. To guarantee positivity preservation at the zone edge i + 1/2, the
flux Fn+1/2i+1/2 at that face is modified to become:
Fn+1/2i+1/2 → θi+1/2F
n+1/2i+1/2 + (1− θi+1/2)FLF
i+1/2, (9.82)
where 0 ≤ θi+1/2 ≤ 1 is a scalar, and FLFi+1/2 is the Lax-Friedrichs flux,
FLFi+1/2 =
1
2
[Fn
i + Fni+1 + CFL
∆x
∆t
1
α
(Un
i −Uni+1
)], (9.83)
9.8—Resets 101
where 0 < CFL < 1 is the CFL safety factor (the method is guaranteed to preserve positivity aslong as CFL < 1/2), and α is a scalar that ensures multi-dimensional correctness (α = 1 in 1D, 1/2in 2D, 1/3 in 3D). Fi is the flux of material evaluated at the zone center i using the cell-centeredquantities U. The scalar θi+1/2 is chosen at every interface by calculating the update that wouldbe obtained from Equation 9.82, setting the density component equal to a value just larger thanthe density floor, castro.small dens, and solving for the value of θ at the interface that makesthe equality hold. In regions where the density is not at risk of going negative, θ ≈ 1 and theoriginal hydrodynamic update is recovered. Further discussion, including a proof of the method, adescription of multi-dimensional effects, and test verification problems, can be found in [14].
CHAPTER 10
Gravity
10.1 Introduction
10.1.1 Integration Strategy
Castro uses subcycling to integrate levels at different timesteps. The gravity algorithm needs torespect this. Self-gravity is computed via multigrid. At coarse-fine interfaces, the stencil used inthe Laplacian understands the coarse-fine interface and is different than the stencil used in theinterior.
There are two types of solves that we discuss with AMR:
• composite solve : This is a multilevel solve, starting at a coarse level (usually level 0) andsolving for the potential on all levels up to the finest level.
• level solve : This solves for the potential only on a particular level. Finer levels are ignored.At coarse-fine interfaces, the data from the coarse levels acts as Dirichlet boundary conditionsfor the current-level-solve.
The overall integration strategy is unchanged from the discussion in [1]. Briefly:
• At the beginning of a simulation, we do a multilevel composite solve (if gravity.no composite=
0).
We also do a multilevel composite solve after each regrid.
• The old-time gravity on the coarse level is defined based on this composite solve, but wealso do a level solve on the coarse level, and use it to determine the difference between thecomposite solve and the level solve, and store that in a MultiFab.
103
104 Chapter 10. Gravity
• After the hydro advance on the coarse level, we do another level solve, and use the (level solve− compositive solve) as a lagged predictor of how much we need to add back to that levelsolve to get an effective new-time value for phi on the coarse level, and that’s what definesthe phi used for the new-time gravity
• Then we do the fine grid timestep(s), each using the same strategy
• At an AMR synchronization step across levels (see Section 16.2 for a description of whenthese synchronizations occur), if we’re choosing to synchronize the gravitational field acrosslevels (gravity.no sync= 0) we then do a solve starting from the coarse grid that adjustsfor the mismatch between the fine-grid phi and the coarse-grid phi, as well as the mismatchbetween the fine-grid density fluxes and the coarse-grid density fluxes, and add the resultingsync solve phi to both the coarse and the fine level
Thus, to within the gravity error tolerance, you get the same final result as if you had donea full composite solve at the end of the timestep (assuming gravity.no sync= 0).
If you do gravity.no composite= 1, then you never do a full multilevel solve, and the gravity onany level is defined only by the solve on that level. The only time this would be appropriate is ifthe fine level(s) cover essentially all of the mass on the grid for all time.
10.1.2 Controls
Castro can incorporate gravity as a constant, monopole approximation, or a full Poisson solve. Toenable gravity in the code, set:
USE_GRAV = TRUE
in the GNUmakefile, and then turn it on in the inputs file via castro.do grav = 1. If you wantto incorporate a point mass (through castro.point mass), you must have
USE_POINTMASS = TRUE
in the GNUmakefile.
There are currently four options for how gravity is calculated, controlled by setting gravity.gravity type.The options are ConstantGrav, PoissonGrav, Monopole Grav or PrescribedGrav. Again, theseare only relevant if USE GRAV = TRUE in the GNUmakefile and castro.do grav = 1 in the inputs
file. If both of these are set then the user is required to specify the gravity type in the inputs fileor the program will abort.
Some additional notes:
• For the full Poisson solver (gravity.gravity type= PoissonGrav), the behavior of the fullPoisson solve / multigrid solver is controlled by gravity.no sync and gravity.no composite.
• For isolated boundary conditions, and when gravity.gravity type= PoissonGrav, the pa-rameters gravity.max multipole order and gravity.direct sum bcs control the accuracyof the Dirichlet boundary conditions. These are described in Section 10.2.3.2.
• For MonopoleGrav, in 1D we must have coord sys = 2, and in 2D we must have coord sys
= 1.
The following parameters apply to gravity solves:
10.2—Types of Approximations 105
• gravity.gravity type : how should we calculate gravity? Can be ConstantGrav, PoissonGrav,MonopoleGrav, or PrescribedGrav
• gravity.const grav : if gravity.gravity type = ConstantGrav, set the value of constantgravity (default: 0.0)
• gravity.no sync : gravity.gravity type = PoissonGrav, do we perform the “sync solve”?(0 or 1; default: 0)
• gravity.no composite : if gravity.gravity type = PoissonGrav, whether to perform acomposite solve (0 or 1; default: 0)
• gravity.max solve level : maximum level to solve for φ and g; above this level, interpolatefrom below (default: MAX LEV− 1)
• gravity.abs tol : if gravity.gravity type = PoissonGrav, this is the absolute tolerancefor the Poisson solve. You can specify a single value for this tolerane (or do nothing, andget a reasonable default value), and then the absolute tolerance used by the multigrid solveis abs tol × 4πGρmax where ρmax is the maximum value of the density on the domain. Onfine levels, this absolute tolerance is multiplied by ref ratio2 to account for the change in theabsolute scale of the Laplacian operator. You can also specify an array of values for abs tol,one for each possible level in the simulation, and then the scaling by ref ratio2 is not applied.
• gravity.rel tol : if gravity.gravity type = PoissonGrav, this is the relative tolerancefor the Poisson solve. By default it is zero. You can specify a single value for this toleranceand it will apply on every level, or you can specify an array of values for rel tol, one for eachpossible level in the simulation. This replaces the old parameter gravity.ml tol.
• gravity.max multipole order : if gravity.gravity type = PoissonGrav, this is the max` value to use for multipole BCs (must be ≥ 0; default: 0)
• gravity.direct sum bcs : if gravity.gravity type = PoissonGrav, evaluate BCs usingexact sum (0 or 1; default: 0)
• gravity.drdxfac : ratio of dr for monopole gravity binning to grid resolution
The follow parameters affect the coupling of hydro and gravity:
• castro.do grav : turn on/off gravity
• castro.moving center : do we recompute the center used for the multipole gravity solvereach step?
• castro.point mass : point mass at the center of the star (must be ≥ 0; default: 0.0)
Note that in the following, MAX LEV is a hard-coded parameter in Source/Gravity.cpp which iscurrently set to 15. It determines how many levels can be tracked by the Gravity object.
10.2 Types of Approximations
10.2.1 ConstantGrav
Gravity can be defined as constant in direction and magnitude, defined in the inputs file by
106 Chapter 10. Gravity
gravity.const grav = -9.8
for example, to set the gravity to have magnitude 9.8 in the negative y-direction if in 2D, negativez-direction if in 3-D. The actual setting is done in Gravity.cpp as:
grav.setVal(const_grav , BL_SPACEDIM -1, 1, ng);
Note that at present we do not fill the gravitational potential φ in this mode; it will be set to zero.
Note: ConstantGrav can only be used along a Cartesian direction (vertical for 2D axisymmetric).
10.2.2 MonopoleGrav
MonopoleGrav integrates the mass distribution on the grid in spherical shells, defining an enclosed
mass and uses this to compute the gravitational potential and acceleration in a spherically-symmetric fashion.
• In 1D spherical coordinates we compute
g(r) = −GMenclosed
r2(10.1)
where Menclosed is calculated from the density at the time of the call.
For levels above the coarsest level we define the extent of that level’s radial arrays as rangingfrom the center of the star (r = 0) to the cell at that level farthest away from the origin. Ifthere are gaps between fine grids in that range then we interpolate the density from a coarserlevel in order to construct a continuous density profile. We note that the location of valuesin the density profile and in the gravitational field exactly match the location of data at thatlevel so there is no need to interpolate between points when mapping the 1D radial profile ofg back onto the original grid.
• In 2D or 3D we compute a 1D radial average of density and use this to compute gravity asa one-dimensional integral, then interpolate the gravity vector back onto the Cartesian gridcells. At the coarsest level we define the extent of the 1D arrays as ranging from the centerof the star to the farthest possible point in the grid (plus a few extra cells so that we can fillghost cell values of gravity). At finer levels we first define a single box that contains all boxeson that fine level, then we interpolate density from coarser levels as needed to fill the valueof density at every fine cell in that box. The extent of the radial array is from the center ofthe star to the nearest cell on one of the faces of the single box. This ensures that all cells atthat maximum radius of the array are contained in this box.
We then average the density onto a 1D radial array. We note that there is a mapping fromthe Cartesian cells to the radial array and back; unlike the 1D case this requires interpolation.We use quadratic interpolation with limiting so that the interpolation does not create newmaxima or minima.
The default resolution of the radial arrays at a level is the grid cell spacing at that level,i.e., ∆r = ∆x. O For increased accuracy, one can define gravity.drdxfac as a numbergreater than 1 (2 or 4 are recommended) and the spacing of the radial array will then satisfy∆x/∆r = drdxfac. Individual Cartesian grid cells are subdivided by drdxfac in each coor-dinate direction for the purposing of averaging the density, and the integration that createsg is done at the finer resolution of the new ∆r.
10.2—Types of Approximations 107
Note that the center of the star is defined in the subroutine PROBINIT and the radius iscomputed as the distance from that center.
there is anadditional correctionat the corners inmake radial grav
that accounts forthe volume in ashell that is not partof the grid
What about the potential in this case? when does make radial phi come into play?
10.2.3 PoissonGrav
The most general case is a self-induced gravitational field,
g(x, t) = ∇φ (10.2)
where φ is defined by solving
∆φ = 4πGρ. (10.3)
We only allow PoissonGrav in 2D or 3D because in 1D, computing the monopole approximationin spherical coordinates is faster and more accurate than solving the Poisson equation.
10.2.3.1 Poisson Boundary Conditions: 2D
In 2D, if boundary conditions are not periodic in both directions, we use a monopole approximationat the coarsest level. This involves computing an effective 1D radial density profile (on level =
0 only), integrating it outwards from the center to get the gravitational acceleration g, and thenintegrating g outwards from the center to get φ (using φ(0) = 0 as a boundary condition, since nomass is enclosed at r = 0). For more details, see Section 10.2.2.
10.2.3.2 Poisson Boundary Conditions: 3D
The following describes methods for doing isolated boundary conditions. The best reference forCastro’s implementation of this is [17].
• Multipole Expansion
In 3D, by default, we use a multipole expansion to estimate the value of the boundary condi-tions. According to, for example, Jackson’s Classical Electrodynamics (with the correspondingchange to Poisson’s equation for electric charges and gravitational ”charges”), an expansionin spherical harmonics for φ is
φ(x) = −G∞∑l=0
l∑m=−l
4π
2l + 1qlm
Ylm(θ, φ)
rl+1, (10.4)
The origin of the coordinate system is taken to be the center variable, that must be declaredand stored in the probdata module in your project directory. The validity of the expansionused here is based on the assumption that a sphere centered on center, of radius approxi-mately equal to the size of half the domain, would enclose all of the mass. Furthermore, thelowest order terms in the expansion capture further and further departures from sphericalsymmetry. Therefore, it is crucial that center be near the center of mass of the system, forthis approach to achieve good results.
108 Chapter 10. Gravity
The multipole moments qlm can be calculated by expanding the Green’s function for thePoisson equation as a series of spherical harmonics, which yields
qlm =
∫Y ∗lm(θ′, φ′) r′
lρ(x′) d3x′. (10.5)
Some simplification of Equation 10.4 can be achieved by using the addition theorem forspherical harmonics:
4π
2l + 1
l∑m=−l
Y ∗lm(θ′, φ′)Ylm(θ, φ) = Pl(cos θ)Pl(cos θ′)
+ 2l∑
m=1
(l −m)!
(l +m)!Pml (cos θ)Pml (cos θ′)
[cos(mφ) cos(mφ′) + sin(mφ) sin(mφ′)
]. (10.6)
Here the Pml are the associated Legendre polynomials and the Pl are the Legendre polyno-mials. After some algebraic simplification, the potential outside of the mass distribution canbe written in the following way:
φ(x) ≈ −Glmax∑l=0
[Q
(0)l
Pl(cos θ)
rl+1+
l∑m=1
[Q
(C)lm cos(mφ) +Q
(S)lm sin(mφ)
] Pml (cos θ)
rl+1
]. (10.7)
The modified multipole moments are:
Q(0)l =
∫Pl(cos θ′) r′
lρ(x′) d3x′ (10.8)
Q(C)lm = 2
(l −m)!
(l +m)!
∫Pml (cos θ′) cos(mφ′) r′
lρ(x′) d3x′ (10.9)
Q(S)lm = 2
(l −m)!
(l +m)!
∫Pml (cos θ′) sin(mφ′) r′
lρ(x′) d3x′. (10.10)
Our strategy for the multipole boundary conditions, then, is to pick some value lmax that is ofsufficiently high order to capture the distribution of mass on the grid, evaluate the discretizedanalog of the modified multipole moments for 0 ≤ l ≤ lmax and 1 ≤ m ≤ l, and then directlycompute the value of the potential on all of the boundary zones. This is ultimately an O(N3)operation, the same order as the monopole approximation, and the wall time required tocalculate the boundary conditions will depend on the chosen value of lmax.
The number of l values calculated is controlled by gravity.max multipole order in yourinputs file. By default, it is set to 0, which means that a monopole approximation is used.There is currently a hard-coded limit of lmax = 50. This is because the method used togenerate the Legendre polynomials is not numerically stable for arbitrary l (because thepolynomials get very large, for large enough l).
• Direct Sum
Up to truncation error caused by the discretization itself, the boundary values for the potentialcan be computed exactly by a direct sum over all cells in the grid. Suppose I considersome ghost cell outside of the grid, at location r′ ≡ (x′, y′, z′). By the principle of linearsuperposition as applied to the gravitational potential,
φ(r′) =∑ijk
−Gρijk ∆Vijk
[(x− x′)2 + (y − y′)2 + (z − z′)2]1/2, (10.11)
10.2—Types of Approximations 109
where x = x(i), y = y(j) and z = z(k) are constructed in the usual sense from the zoneindices. The sum here runs over every cell in the physical domain (that is, the calculation isO(N3) for each boundary cell). There are 6N2 ghost cells needed for the Poisson solve (sincethere are six physical faces of the domain), so the total cost of this operation is O(N5) (thisonly operates on the coarse grid, at present). In practice, we use the domain decompositioninherent in the code to implement this solve: for the grids living on any MPI task, we createsix N2 arrays representing each of those faces, and then iterate over every cell on each of thosegrids, and compute their respective contributions to all of the faces. Then, we do a globalreduce to add up the contributions from all cells together. Finally, we place the boundarycondition terms appropriate for each grid onto its respective cells.
This is quite expensive even for reasonable sized domains, so this option is recommended onlyfor analysis purposes, to check if the other methods are producing accurate results. It can beenabled by setting gravity.direct sum bcs= 1 in your inputs file.
10.2.4 PrescribedGrav
With PrescribedGrav1, gravity can be defined as a function that is specified by the user. Theoption is allowed in 2D and 3D. To define the gravity vector, copy prescribe grav nd.f90 fromSource/gravity/ to your run directory. The makefile system will always choose this local copy ofthe file over the one in another directory. Then define the components of gravity inside a loop overthe grid inside the file. If your problem uses a radial gravity in the form g(r), you can simply adaptca prescribe grav gravityprofile, otherwise you will have to adapt ca prescribe grav, bothare located in prescribed grav nd.90.
10.2.5 Point Mass
Pointmass gravity works with all other forms of gravity, it is not a separate option. Since thePoisson equation is linear in potential (and its derivative, the acceleration, is also linear), the pointmass option works by adding the gravitational acceleration of the point mass onto the accelerationfrom whatever other gravity type is under in the simulation.
Note that point mass can be < 0.
A useful option is point mass fix solution. If set to 1, then it takes all zones that are adjacentto the location of the center variable and keeps their density constant. Any changes in densitythat occur after a hydro update in those zones are reset, and the mass deleted is added to thepointmass. (If there is expansion, and the density lowers, then the point mass is reduced andthe mass is added back to the grid). This calculation is done in pm compute delta mass() inSource/gravity/pointmass nd.f90.
1Note: The PrescribedGrav option and text here were contributed by Jan Frederik Engels of University ofGottingen.
110 Chapter 10. Gravity
10.3 GR correction
In the cases of compact objects or very massive stars, the general relativity (GR) effect startsto play a role2. First, we consider the hydrostatic equilibrium due to effects of GR then deriveGR-correction term for Newtonian gravity. The correction term is applied to the monopole ap-proximation only when USE GR = TRUE is set in the GNUmakefile.
The formulae of GR-correction here are based on [18]. For detailed physics, please refer to [27].For describing very strong gravitational field, we need to use Einstein field equations
Rik −1
2gikR =
κ
c2Tik , κ =
8πG
c2, (10.12)
where Rik is the Ricci tensor, gik is the metric tensor, R is the Riemann curvature, c is the speedof light and G is gravitational constant. Tik is the energy momentum tensor, which for ideal gashas only the non-vanishing components T00 = %c2 , T11 = T22 = T33 = P ( contains rest mass andenergy density, P is pressure). We are interested in spherically symmetric mass distribution. Thenthe line element ds for given spherical coordinate (r, ϑ, ϕ) has the general form
ds2 = eνc2dt2 − eλdr2 − r2(dϑ2 + sin2 ϑdϕ) , (10.13)
with ν = ν(r), λ = λ(r). Now we can put the expression of Tik and ds into (10.12), then fieldequations can be reduced to 3 ordinary differential equations:
κP
c2= e−λ
(ν ′
r+
1
r2
)− 1
r2, (10.14)
κP
c2=
1
2e−λ
(ν ′′ +
1
2ν ′
2+ν ′ − λ′
r− ν ′λ′
2
), (10.15)
κ% = e−λ(λ′
r− 1
r2
)+
1
r2, (10.16)
where primes means the derivatives with respect to r. After multiplying with 4πr2, (10.16) can beintegrated and yields
κm = 4πr(1− e−λ) , (10.17)
the m is called “gravitational mass” inside r defined as
m =
∫ r
04πr2%dr . (10.18)
For the r = R, m becomes the mass M of the star. M contains not only the rest mass butthe whole energy (divided by c2), that includes the internal and gravitational energy. So the% = %0 + U/c2 contains the whole energy density U and rest-mass density %0. Differentiation of(10.14) with respect to r gives P = P ′(λ, λ′, ν, ν ′, r), where λ, λ′, ν, ν ′ can be eliminated by (10.14),(10.15), (10.16). Finally we reach Tolman-Oppenheinmer-Volkoff(TOV) equation for hydrostaticequilibrium in general relativity:
dP
dr= −Gm
r2%
(1 +
P
%c2
)(1 +
4πr3P
mc2
)(1− 2Gm
rc2
)−1
. (10.19)
2Note: The GR code and text here were contributed by Ken Chen of Univ. of Minnesota.
10.4—Hydrodynamics Source Terms 111
For Newtonian case c2 →∞, it reverts to usual form
dP
dr= −Gm
r2% . (10.20)
Now we take effective monopole gravity as
g = −Gmr2
(1 +P
%c2)(1 +
4πr3P
mc2)(1− 2Gm
rc2)−1 . (10.21)
For general situations, we neglect the U/c2 and potential energy in m because they are usuallymuch smaller than %0. Only when T reaches 1013K (KT ≈ mpc
2, mp is proton mass) before itreally makes a difference. So (10.21) can be expressed as
g = −GMenclosed
r2
(1 +
P
%c2
)(1 +
4πr3P
Menclosedc2
)(1− 2GMenclosed
rc2
)−1
, (10.22)
where Menclosed has the same meaning as with the MonopoleGrav approximation.
10.4 Hydrodynamics Source Terms
There are several options to incorporate the effects of gravity into the hydrodynamics system. Themain parameter here is castro.grav source type.
• castro.grav source type = 1 : we use a standard predictor-corrector formalism for updat-ing the momentum and energy. Specifically, our first update is equal to ∆t× Sn , where Sn
is the value of the source terms at the old-time (which is usually called time-level n). Atthe end of the timestep, we do a corrector step where we subtract off ∆t/2× Sn and add on∆t/2× Sn+1, so that at the end of the timestep the source term is properly time centered.
• castro.grav source type = 2 : we do something very similar to 1. The major differenceis that when evaluating the energy source term at the new time (which is equal to u · Sn+1
ρu ,where the latter is the momentum source term evaluated at the new time), we first updatethe momentum, rather than using the value of u before entering the gravity source terms.This permits a tighter coupling between the momentum and energy update and we have seenthat it usually results in a more accurate evolution.
• castro.grav source type = 3 : we do the same momentum update as the previous two,but for the energy update, we put all of the work into updating the kinetic energy alone. Inparticular, we explicitly ensure that (rhoe) maintains the same, and update (rhoK) with thework due to gravity, adding the new kinetic energy to the old internal energy to determine thefinal total gas energy. The physical motivation is that work should be done on the velocity,and should not directly update the temperature—only indirectly through things like shocks.
• castro.grav source type = 4 : the energy update is done in a “conservative” fashion. Theprevious methods all evaluate the value of the source term at the cell center, but this methodevaluates the change in energy at cell edges, using the hydrodynamical mass fluxes, permittingtotal energy to be conserved (excluding possible losses at open domain boundaries). See [16]for some more details.
CHAPTER 11
Diffusion
11.1 Thermal Diffusion
Castro incorporates explicit thermal diffusion into the energy equation. In terms of the specificinternal energy, e, this appears as:
ρDe
Dt+ p∇ · u = ∇ · kth∇T (11.1)
where kth is the thermal conductivity, with units erg cm−1 s−1 K−1.
To see the similarity to the thermal diffusion equation, consider the special case of constant con-ductivity, kth, and density, and assume an ideal gas, so e = cvT , where cv is the specific heat atconstant volume. Finally, ignore hydrodynamics, so u = 0. This gives:
∂T
∂t= D∇2T (11.2)
where D ≡ kth/(ρcv). Solving this equation explicitly requires a timestep limiter of
∆tdiff ≤1
2
∆x2
D(11.3)
(this is implemented in ca estdt temp diffusion in Castro/Source/driver/timestep.F90).
Support for diffusion must be compiled into the code by setting USE DIFFUSION = TRUE in yourGNUmakefile. It is treated explicitly, by constructing the contribution to the evolution as a sourceterm. This is time-centered to achieve second-order accuracy in time.
The following parameter affects diffusion:
• castro.diffuse temp: enable thermal diffusion (0 or 1; default 0)
113
114 Chapter 11. Diffusion
A pure diffusion problem (with no hydrodynamics) can be run by setting
castro.diffuse_temp = 1
castro.do_hydro = 0
To complete the setup, a thermal conductivity must be specified. The interface for the conductivityis:
subroutine thermal_conductivity(eos_state , therm_cond)
use extern_probin_module , only: const_conductivity
type (eos_t), intent(in) :: eos_state
real (kind=dp_t), intent(inout) :: therm_cond
The density, temperature, and mass fractions come in through the eos state type. An EOS callis done in Castro just before the call to thermal conductivity, so you can assume that the entirestate is consistent.
There are two conductivity routines provided with Castro by default:
• constant : A simple constant thermal conductivity. This can be selected by setting
Conductivity_dir := constant
in your GNUmakefile. To set the value of the conductivity (e.g., to 100), you add to yourprobin file’s &extern namelist:
const_conductivity = 100.0
• constant opacity : A simple constant opacity. This is converted to an opacity as:
kth =16σBT
3
3κconstρ(11.4)
where κconst is the opacity, with units cm2 g−1. This is selected by setting
Conductivity_dir := constant_opacity
in your GNUmakefile. To set the value of the opacity, e.g., to 0.2 (e.g., for electron scattering),set:
const_opacity = 0.2
in the &extern namelist of your probin.
The diffusion approximation breaks down at the surface of stars, where the density rapidly dropsand the mean free path becomes large. In those instances, you should use the flux limited diffusionmodule in Castro to evolve a radiation field. However, if your interest is only on the diffusion in theinterior, you can use the castro.diffuse cutoff density parameter to specify a density, belowwhich, diffusion is not modeled. This is implemented in the code by zeroing out the conductivityand skipping the estimation of the timestep limit in these zones.
A simple test problem that sets up a Gaussian temperature profile and does pure diffusion isprovided as diffusion test.
11.2—Enthalpy Diffusion 115
11.2 Enthalpy Diffusion
Castro can also diffuse enthalpythese need to bedocumented
Note this uses the same interface for the transport coefficients as thermal diffusion, so the twocannot be used at the same time.
11.3 Species Diffusion
Castro can also diffuse species.
Note this uses the same interface for the transport coefficients as thermal diffusion, so the twocannot be used at the same time.
11.4 Viscosity
CHAPTER 12
Rotation
12.1 Introduction
Currently, Castro supports contant, solid-body rotation about a fixed (in space and time) axis in2D and 3D by transforming the evolution equations to the rotating frame of reference.
To include rotation you must set
USE_ROTATION = TRUE
in the GNUMakefile. Rotation can then be enabled via
castro.do_rotation = 1
in the inputs file. The rotational period must then be set via castro.rotational period. Therotational period is internally converted to an angular frequency for use in the source term equations.
The axis of rotation currently depends on the dimensionality of the problem and the value ofcoord sys; in all cases, however, the default axis of rotation points from center, which is typicallydefined in a Prob $(DIM)d.f90 routine, to the typical “vertical direction.” The vertical directionis defined as follows:
• 2D
– coord sys = 0, (x,y): out of the (x,y)-plane along the “z”-axis
– coord sys = 1, (r,z): along the z-axis
• 3D
– coord sys = 0, (x,y,z): along the z-axis
117
118 Chapter 12. Rotation
To change these defaults, modify the omega vector in the ca rotate routine found in theRotate $(DIM)d.f90 file.
The main parameters that affect rotation are:
• castro.do rotation : include rotation as a forcing term (0 or 1; default: 0)
• castro.rotational period : period (s) of rotation (default: 0.0)
• castro.rotational dPdt : d(period) / dt for rotation (default: 0.0)
• castro.rotation include centrifugal : whether to include the centrifugal forcing (de-fault: 1)
• castro.rotation include coriolis : whether to include the Coriolis forcing (default: 1)
• castro.rotation include domegadt : whether to include the forcing from the time deriva-tive of the rotation frequency (default: 1)
• castro.state in rotating frame : whether state variables are measured in the rotatingframe (default: 1)
• castro.rot source type : method of updating the energy during a rotation update (default:4)
• castro.implicit rotation update : for the Coriolis term, which mixes momenta in thesource term, whether we should solve for the update implicitly (default: 1)
• castro.rot axis : rotation axis (default: 3 (Cartesian); 2 (cylindrical))
For completeness, we show below a derivation of the source terms that appear in the momentumand total energy evolution equations upon switching to a rotating reference frame.
12.2 Coordinate transformation to rotating frame
Consider an inertial reference frame C and a non-inertial reference frame C whose origins areseparated by the vector l (see Figure 12.1). The non-inertial frame is rotating about the axis ωwith a constant angular velocity ω; furthermore, we assume the direction of the rotational axis isfixed. Consider a fluid element at the point P whose location is given by r in C and by r in C:
r = r + l, (12.1)
or in component notation
riei = riei + liei, (12.2)
where ei and ei are the ith unit vectors in the C and C coordinate systems, respectively. The totaltime rate of change of 12.2 is given by
DriDt
ei =DriDt
ei + riDeiDt
+DliDt
ei, (12.3)
where we have used the fact that the unit vectors of the inertial frame C are not moving (or at leastcan be considered stationary, and the change in l gives the relative motion of the two coordinatesystems). By definition, a unit vector can not change its length, and therefore the only change of
12.2—Coordinate transformation to rotating frame 119
Figure 12.1: Inertial frame C and non-inertial frame C. We consider a fluid element P , whosedistance in the two frames is related by r = r + l
ei with time can come from changing direction. This change is carried out by a rotation about theω axis, and the tip of the unit vector moves circumferentially, that is
DeiDt
= ω × ei. (12.4)
Plugging 12.4 into 12.3 and switching back to vector notation, we have
Dr
Dt=Dr
Dt+ ω × r +
Dl
Dt. (12.5)
The left hand side of 12.5 is interpretted as the velocity of the fluid element as seen in the inertialframe; the first term on the right hand side is the velocity of the fluid element as seen by a stationaryobserver in the rotating frame C. The second and third terms on the right hand side of 12.5 describethe additional velocity due to rotation and translation of the frame C as seen in C. In other words,
v = v + ω × r + vl, (12.6)
where we use vl to represent the translational velocity.
Similarly, by taking a second time derivative of 12.6 we have
Dv
Dt=Dv
Dt+ 2ω × v + ω × [ω × r] +
DvlDt
. (12.7)
Henceforth we will assume the two coordinate systems are not translating relative to one another,vl = 0. It is also worth mentioning that derivatives with respect to spatial coordinates do notinvolve additional terms due to rotation, i.e. ∇ ·v = ∇ · v. Because of this, the continuity equationremains unchanged in the rotating frame:
∂ρ
∂t= −∇ · (ρv) , (12.8)
orDρ
Dt= −ρ∇ · v. (12.9)
120 Chapter 12. Rotation
12.3 Momentum equation in rotating frame
The usual momentum equation applies in an inertial frame:
D (ρv)
Dt= −ρv ·∇v −∇p+ ρg. (12.10)
Using the continuity equation, 12.9, and substituting for the terms in the rotating frame from 12.7,we have from 12.10:
ρ
(Dv
Dt+ 2ω × v + ω × [ω × r]
)− ρv∇ · v = −ρv ·∇v −∇p+ ρg
ρ
(∂v
∂t+ v ·∇v
)= −∇p+ ρg − 2ρω × v − ρω × [ω × r]
∂ (ρv)
∂t= −∇ · (ρvv)−∇p+ ρg − 2ρω × v
− ρω × [ω × r] (12.11)
orD (ρv)
Dt= −ρv ·∇v −∇p+ ρg − 2ρω × v − ρω × [ω × r] . (12.12)
12.4 Energy equations in rotating frame
From 12.12, we have the velocity evolution equation in a rotating frame
Dv
Dt= −1
ρ∇p+ g − 2ω × v − ω × [ω × r] . (12.13)
The kinetic energy equation can be obtained from 12.13 by mulitplying by ρv:
ρv · Dv
Dt= −v ·∇p+ ρv · g − 2ρv · [ω × v]− ρv · ω × [ω × r]
1
2
D (ρv · v)
Dt− 1
2v · vDρ
Dt= −v ·∇p+ ρv · g − ρv ·
[(ω · r)ω − ρω2r
]1
2
D (ρv · v)
Dt= −1
2ρv · v∇ · v − v ·∇p+ ρv · g − ρv ·
[(ω · r)ω − ρω2r
].(12.14)
The internal energy is simply advected, and, from the first law of thermodynamics, can change dueto pdV work:
D (ρe)
Dt= − (ρe+ p)∇ · v. (12.15)
Combining 12.14 and 12.15 we can get the evolution of the total specific energy in the rotatingframe, ρE = ρe+ 1
2ρv · v:
D (ρe)
Dt+
1
2
D (ρv · v)
Dt= −
(ρe+ p+
1
2ρv · v
)∇ · v − v ·∇p+ ρv · g − ρv ·
[(ω · r)ω − ρω2r
]D(ρE)
Dt= −ρE∇ · v −∇ · (pv) + ρv · g − ρv ·
[(ω · r)ω − ρω2r
](12.16)
or∂(ρE)
∂t= −∇ ·
(ρEv + pv
)+ ρv · g − ρv ·
[(ω · r)ω − ρω2r
]. (12.17)
12.5—Switching to the rotating reference frame 121
12.5 Switching to the rotating reference frame
If we choose to be a stationary observer in the rotating reference frame, we can drop all of the tildes,which indicated terms in the non-inertial frame C. Doing so, and making sure we account for theoffset, l, between the two coordinate systems, we obtain the following equations for hydrodynamicsin a rotating frame of reference:
∂ρ
∂t= −∇ · (ρv) (12.18)
∂ (ρv)
∂t= −∇ · (ρvv)−∇p+ ρg − 2ρω × v − ρ (ω · r)ω + ρω2r (12.19)
∂ (ρE)
∂t= −∇ · (ρEv + pv) + ρv · g − ρ (ω · r) (ω · v) + ρω2 (v · r) . (12.20)
12.6 Adding the forcing to the hydrodynamics
There are several ways to incorporate the effect of the rotation forcing on the hydrodynamicalevolution. We control this through the use of the runtime parameter castro.rot source type.This is an integer with values currently ranging from 1 through 4, and these values are all analogousto the way that gravity is used to update the momentum and energy. For the most part, thedifferences are in how the energy update is done:
• castro.rot source type = 1 : we use a standard predictor-corrector formalism for updatingthe momentum and energy. Specifically, our first update is equal to ∆t × Sn , where Sn isthe value of the source terms at the old-time (which is usually called time-level n). At theend of the timestep, we do a corrector step where we subtract off ∆t/2 × Sn and add on∆t/2× Sn+1, so that at the end of the timestep the source term is properly time centered.
• castro.rot source type = 2 : we do something very similar to 1. The major difference isthat when evaluating the energy source term at the new time (which is equal to u · Sn+1
ρu ,where the latter is the momentum source term evaluated at the new time), we first updatethe momentum, rather than using the value of u before entering the rotation source terms.This permits a tighter coupling between the momentum and energy update and we have seenthat it usually results in a more accurate evolution.
• castro.rot source type = 3 : we do the same momentum update as the previous two, butfor the energy update, we put all of the work into updating the kinetic energy alone. Inparticular, we explicitly ensure that (rhoe) maintains the same, and update (rhoK) with thework due to rotation, adding the new kinetic energy to the old internal energy to determinethe final total gas energy. The physical motivation is that work should be done on the velocity,and should not directly update the temperature – only indirectly through things like shocks.
• castro.rot source type = 4 : the energy update is done in a “conservative” fashion. Theprevious methods all evaluate the value of the source term at the cell center, but this methodevaluates the change in energy at cell edges, using the hydrodynamical mass fluxes, permittingtotal energy to be conserved (excluding possible losses at open domain boundaries). Addi-tionally, the velocity update is slightly different—for the corrector step, we note that thereis an implicit coupling between the velocity components, and we directly solve this coupledequation, which results in a slightly better coupling and a more accurate evolution.
122 Chapter 12. Rotation
The other major option is castro.implicit rotation update. This does the update of the Cori-olis term in the momentum equation implicitly (e.g., the velocity in the Coriolis force for the zonedepends on the updated momentum). The energy update is unchanged.
A detailed discussion of these options and some verification tests is presented in [17].
CHAPTER 13
Radiation
13.1 Introduction
Castro has three radiation solvers:
• SingleGroupSolver: this solver does not have radiation pressre. It is pure hydro plus radi-ation diffusion. This is only applicable when the medium is optically thick and the pressureis small.
• SGFLDSolver: this is the gray flux-limited diffusion radiation hydrodymamics solver. Herethe radiation pressure is separate from the gas pressure, and both pressures participate in theRiemann solver.
• MGFLDSolver: this is the multigroup flux-limited diffusion radiation hydrodynamics solver.As with the gray solver, radiation pressure contributes to the pressure in the Riemann solver.Here a number of energy groups are used to represent the radiation field, and the opacity canbe frequency-dependent.
The gray solver has a comoving frame mode and a mixed frame mode, whereas the MG solver usesthe comoving frame approach. More details about the formulation and algorithm can be found inthe series of Castro papers.
13.2 Getting Started
13.2.1 Getting the Code
The Castro radiation solver is part of the main Castro git repo, so you already have all the Castrocode and problem setups to exercise radiation. The only other requirement is a copy of the Hypre
123
124 Chapter 13. Radiation
library. Hypre provides the algebraic multigrid solvers used by the implicit radiation update. Youcan get a copy at https://computation.llnl.gov/casc/linear_solvers/sls_hypre.html. Youwill need to follow their installation instructions.
In addition to the environment variables you set for the main Castro hydrodynamics problems, youalso need to tell the code where to find Hypre. This is done via one of two variables:
• the environment variable HYPRE DIR should point to the location of your Hypre installation(e.g., /home/J.Doe/hypre) or can be set directly in CastroRadiation/Exec/Make.Castro.This applies is you build with USE OMP=FALSE
• the variable HYPRE OMP DIR should be set (either as an environment variable or in Castro/E
xec/Make.Castro) to the directory for openmp enabled Hypre (e.g., /home/J.Doe/hypre-omp) if you build with USE OMP=TRUE
Now go to a “run” directory, say Castro/Exec/radiation tests/RadThermalWave, edit the fileGNUmakefile, and set
• COMP = your favorite compiler suite (e.g., gnu, pgi, intel)
• DIM = 1 or 2 or 3
• USE RAD = TRUE—this is important. This tells the build system to compile in, and link thethe radiation code.
Then type make to generate an executable file.
13.3 Microphysics: EOS, Network, and Opacity
13.3.1 EOS
Castro provides several types of equation of state (EOS), including gamma-law and Helmholtz. Touse the gamma-law EOS, set
EOS_DIR := gamma_law_general
in the GNUmakefile.
The original Helmholtz EOS for stellar interiors includes a radiation contribution. However, for ra-diation hydrodynamics calculations, the radition contribution should be taken out of EOS becauseradiation has been treated in other places. To use Helmholtz EOS, we will use the version in Micro-physics, as with the pure hydrodynamics code, but this will interpret the RADIATION preprocessorvariable and disable the radiation portion of the EOS1 If you have your own EOS, you can put itin Microphysics.
13.3.1.1 EOS Parameters
The following parameters affect how the radiation solver used the EOS:
1at the moment, we don’t have a way to allow for the EOS to provide radiation pressure if the Castro radiation isused solely for neutrinos, but this is something that could be added easily.
13.3—Microphysics: EOS, Network, and Opacity 125
• radiation.do real eos = 1
Usually you do not want to change this from the default. Setting this to 0 is only for contrivedtests that assume the specific heat is in the form of a power-law,
cv = const ρmT−n (13.1)
13.3.2 Network
The radiation solver uses the same networks as we saw for pure hydro, so nothing needs to changehere. Again, if you are not modeling reactions, then the general null network can be used todefine the appropriate composition for your problem.
13.3.3 Opacity
By default, we assume that
κ = const ρmT−nνp, (13.2)
where κ is either Planck or Rosseland mean absorption coefficients, ρ is density, T is temperature,ν is frequency, and m, n and p are constants. For the gray solver, p = 0. If Equation (13.2) issufficient, set
Opacity_dir := null
in GNUmakefile. Otherwise, put your own opacity in Castro/Microphysics/opacity and set theinput parameter, radiation.use opacity table module = 1 (see § 13.3.3.1).
Some notes:
• Here, κ has units of cm−1. Some papers or texts may instead have an implicit density factorin κ, yielding units cm2 g−1.
• Castro allows for two temperatures (different radiation and gas temperature, so Er 6= aT 4gas).
Correspondingly, Castro cares about both the Planck mean, κP , and Rosseland mean, κR,opacities—these have different weightings.
If we set κP∆x 1 (κP is really large), then the two temperatures become the same.
If we set κP = κR, then we can see how different the two temperature are.
In an optically thick medium, we would not expect the two temperatures to be very different.
13.3.3.1 Opacity Parameters
The parameters describing the opacity include:
• radiation.use opacity table module = 0
For neutrino problems, this parameter is not ignored. For photon problems, this determineswhether the opacity module at Opacity dir (which is set in GNUmakefile) will be used tocompute opacities. If this is set to 1, the following parameters for opacities will be ignored.
126 Chapter 13. Radiation
• For the Planck mean opacity of the form in Eq. (13.2), the following parameters set thecoefficient and exponents:
– radiation.const kappa p = -1.0
– radiation.kappa p exp m = 0.0
– radiation.kappa p exp n = 0.0
– radiation.kappa p exp p = 0.0
• For the Rosseland mean opacity of the form in Eq. (13.2), the following parameters set thecoefficient and exponents:
– radiation.const kappa r = -1.0
– radiation.kappa r exp m = 0.0
– radiation.kappa r exp n = 0.0
– radiation.kappa r exp p = 0.0
• For the scattering coefficient of the form in Eq. (13.2), the following parameters set thecoefficient and exponents:
– radiation.const scattering = 0.0
– radiation.scattering exp m = 0.0
– radiation.scattering exp n = 0.0
– radiation.scattering exp p = 0.0
• radiation.kappa r floor = 0.0
Floor for Rosseland mean.
• radiation.do kappa stm emission = 0
If it is 1, correction for stimulated emission is applied to Planck mean as follows
κ = const ρmT−nνp[1− exp
(− hνkT
)]. (13.3)
• radiation.surface average = 2
How the averaging of opacity is done from faces to center for the radiation solver. 0 isarithmetic averaging, 1 is harmonic averaging, and 2 is a combination of the two. This isimplemented in RAD ?D.F in kavg.
Note that the unit for opacities is cm−1. For the gray solver, the total opacity in the diffusion coeffi-cient is the sum of kappa r and scattering, whereas for the MG solver, there are two possibilities.If const kappa r is greater than 0, then the total opacity is set by kappa r alone, otherwise thetotal opacity is the sum of kappa p and scattering.
13.4—Radiation Solver Physics 127
13.4 Radiation Solver Physics
In this section, we list some radiation related parameters that you can set in an inputs file. Hereare some important parameters:
• radiation.SolverType:
Set it to 5 for the gray solver, and 6 for the MG solver.
• castro.do hydro
Usually you want to set it to 1. If it is set to 0, hydro will be turned off, and the calculationwill only solve radiation diffusion equation.
• castro.do radiation
If it is 0, the calculation will be pure hydro.
Below are more parameters. For each parameter, the default value is on the right-hand side of theequal sign.
13.4.1 Verbosity and I/O
• radiation.v = 0
Verbosity
• radiation.verbose = 0
Verbosity
• radiation.plot lambda = 0
If 1, save flux limiter in plotfiles.
• radiation.plot kappa p = 0
If 1, save Planck mean opacity in plotfiles.
• radiation.plot kappa r = 0
If 1, save Rosseland mean opacity in plotfiles.
• radiation.plot lab Er = 0
If 1, save lab frame radiation energy density in plotfiles. This flag is ignored when the mixed-frame gray solver is used.
• radiation.plot com flux = 0
If 1, save comoving frame radiation flux in plotfiles.
• radiation.plot lab flux = 0
If 1, save lab frame radiation flux in plotfiles.
128 Chapter 13. Radiation
13.4.2 Flux Limiter and Closure
• radiation.limiter = 2
Possible values are:
– 0: No flux limiter
– 2: Approximate limiter of Levermore & Pomraning
– 12: Bruenn’s limiter
– 22: Larsen’s square root limiter
– 32: Minerbo’s limiter
• radiation.closure = 3
Possible values are:
– 0: f = λ, where f is the scalar Eddington factor and λ is the flux limiter.
– 1: f = 13
– 2: f = 1− 2λ
– 3: f = λ+ (λR)2, where R is the radiation Knudsen number.
– 4: f = 13 + 2
3( FcE )2, where F is the radiation flux, E is the radiation energy density, andc is the speed of light.
Note the behavior of the radiative flux in the optically thin and optically thick limits. The fluxlimiter, λ = λ(R), where
R =|∇E(0)
r |χRE
(0)r
(13.4)
Regardless of the limiter chosen, when we are optically thick, χR →∞, R→ 0, and λ→ 1/3. Theradiative flux then becomes
F (0)r = − cλ
χR∇E(0)
r →1
3
c
χR∇E(0)
r (13.5)
And when we are optically thin, χR → 0, R→∞, and λ→ 1/R = χRE(0)r /|∇E0
r |, and the radiativeflux then becomes
F (0)r = − cλ
χR∇E(0)
r → −c
χR
χRE(0)r
|∇E0r |∇E(0)
r = −cE0r (13.6)
See Krumholz et al. 2007 for some discussion on this.
13.4.3 Boundary Conditions
Castro needs to know about the boundary conditions for both the hydrodynamics and radiationportions of the evolution.
13.4—Radiation Solver Physics 129
13.4.3.1 Hydrodynamics Evolution
For the hydrodynamics portion of the solve, the boundary conditions for the normal hydrodynamicstate values will be set by the problem’s hypfill routine (which typically just calls filcc to handlethe usual hydrodynamics boundary types: outflow, symmetry, etc.).
A corresponding radfill routine needs to be written to fill the ghost cells for the radiation energydensity during the hydrodynamics evolution. Again, this usually will just default to calling filcc.
Note: if any of the hydrodynamic boundary conditions types are set to Inflow, then you will needto ensure that the radfill routine explicitly handles the boundary condition implementation forthe radiation energy density in that case—the filcc routine will not do a hydrodynamic Inflow
boundary.
13.4.3.2 Radiation Evolution
The following parameters are for radiation boundary in the diffusion equation. They do not affecthydrodynamic boundaries.
• radiation.lo bc
This sets the action to take at the lower edge of the domain in each coordinate direction.Possible values are:
– 101 Dirchlet:
Specify the radiation energy density on the boundary. For gray radiation, this could beEr = aT 4.
For multigroup radiation, Castro stores the energy density as erg cm−3, so the totalradiation energy can be found by simply summing over the groups. So if you wantto set the radiation BCs using the Planck function, you simply multiply by the groupwidth—see Exec/radiation tests/RadSphere/Tools/radbc.f90 for an example.
– 102 Neumann:
Here, you specify the radiation flux on the boundary. For gray radiation, this is theexpression given in the gray Castro paper (Eq. 7, 8),
Fr = − cλκR∇Er (13.7)
where λ is the flux limiter.
Note that if your boundary represents an incoming flux through a vacuum (like stellarirradiation), then κ→ 0, leaving
Fr = −cEr (13.8)
(see § 13.4.2) in that case.
– 104 Marshak (vacuum):
Here, you specify the incident flux and the outside is a vacuum. This differs from theNeumann condition because there is also a flux coming from inside, for the net fluxacross the boundary is different than the incident flux.
130 Chapter 13. Radiation
– 105 Sanchez-Pomraning:
This is a modified form of the Marshak boundary condition that works with FLD. Thisis like the Marshak condition, but λ = 1/3 is not assumed inside the boundary (opticalthickness).
• radiation.hi bc
See radiation.lo bc.
• radiation.lo bcflag = 0 0 0
If it is 0, bcval is used for that dimension, otherwise subroutine rbndry in RadBndry 1d.f90
is called to set boundary conditions.
• radiation.hi bcflag = 0 0 0
See radiation.lo bcflag
• radiation.lo bcval = 0.0 0.0 0.0
The actual value to impose for the boundary condition type set by radiation.lo bc. Thisparameter is interpreted differently depending on the boundary condition:
– Dirchlet: Dirichlet value of rad energy density
– Neumann: inward flux of rad energy
– Marshak: incident flux
– Sanchez-Pomraning: incident flux
• radiation.hi bcval = 0.0 0.0 0.0
See radiation.lo bcval
13.4.4 Convergence
For the gray solver, there is only one iteration in the scheme, whereas for the MG solver, thereare two iterations with an inner iteration embedded inside an outer iteration. In the following,the iteration in the gray solver will also be referred as the outer iteration for convenience. Theparameters for the inner iteration are irrelevant to the gray solver.
radiation.maxiter = 50Maximal number of outer iteration steps.
radiation.miniter = 1Minimal number of outer iteration steps.
radiation.reltol = 1.e-6Relative tolerance for the outer iteration.
radiation.abstol = 0.0Absolute tolerance for the outer iteration.
radiation.maxInIter = 30Maximal number of inner iteration steps.
13.4—Radiation Solver Physics 131
radiation.minInIter = 1Minimal number of inner iteration steps.
radiation.relInTol = 1.e-4Relative tolerance for the inner iteration.
radiation.absInTol = 0.0Absolute tolerance for the inner iteration.
radiation.convergence check type = 0For the MG solver only. This specifiy the way of checking the convergence of an outeriteration. Possible values are
• 0: Check T , Ye, and the residues of the equations for ρe and ρYe
• 1: Check ρe
• 2: Check the residues of the equations for ρe and ρYe
• 3: Check T and Ye
13.4.5 Parameters for Gray Solver
radiation.comoving = 1Do we use the comoving frame approach?
radiation.Er Lorentz term = 1If the mixed-frame approach is taken, this parameter decides whether Lorentz transformationterms are retained.
radiation.delta temp = 1.0This is used in computing numerical derivativas with respect to T . So it should be a smallnumber compared with T , but not too small.
radiation.update limiter = 1000Stop updating flux limiter after update limiter iteration steps.
radiation.update planck = 1000Stop updating Planck mean opacity after update planck iteration steps.
radiation.update rosseland = 1000Stop updating Rosseland mean opacity after update rosseland iteration steps.
13.4.6 Grouping in the MG Solver
We provide two methods of setting up groups based upon logarithmic spacing. In both methods,you must provide:
radiation.nGroupsNumber of groups.
radiation.lowestGroupHzFrequency of the lower bound for the first group.
132 Chapter 13. Radiation
In addition, if the parameter groupGrowFactor is provided, then the first method will be used, oth-erwise the second method will be used. In the first way, you must also provide firstGroupWidthHz(the width of the first group). The width of other groups is set to be groupGrowFactor times thewidth of its immediately preceding group. In the second way, you must provide highestGroupHz
as the upper bound of the last group. It should be noted that lowestGroupHz can be 0 in thefirst method, but not the second method. However, when we compute the group-integrated Planckfunction, the lower bound for the first group and the upper bound for the last group are assumedto be 0 and ∞, respectively.
13.4.7 Parameters for MG Solver
radiation.delta e rat dt tol = 100.0Maximally allowed relative change in e during one time step.
radiation.delta T rat dt tol = 100.0Maximally allowed relative change in T during one time step.
radiation.delta Ye dt tol = 100.0Maximally allowed absolute change in Ye during one tim estep.
radiation.fspace advection type = 2Possible value is 1 or 2. The latter is better.
radiation.integrate Planck = 1If 1, integrate Planck function for each group. For the first group, the lower bound in theintegration is assumed to be 0 no matter what the grouping is. For the last group, the upperbound in the integration is assumed to be ∞.
radiation.matter update type = 0How to update matter. 0 is proabaly the best.
radiation.accelerate = 2The inner iteration of the MG solver usually requires an acceleration scheme. Choices are
• 0: No acceleration
• 1: Local acceleration
• 2: Gray acceleration
radiation.skipAccelAllowed = 0If it is set to 1, skip acceleration if it does not help.
radiation.n bisect = 1000Do bisection for the outer iteration after n bisec iteration steps.
radiation.use dkdT = 1If it is 1, ∂κ
∂T is retained in the Jacobi matrix for the outer (Newton) iteration.
radiation.update opacity = 1000Stop updating opacities after update opacity outer iteration steps.
radiation.inner update limiter = 0Stop updating flux limiter after inner update limiter inner iteration steps. If it is 0, the
13.4—Radiation Solver Physics 133
limiter is lagged by one outer iteration. If it is -1, the limiter is lagged by one time step. Ifthe inner iteration has difficulty in converging, setting this parameter it to -1 can help. Sincethe flux limiter is only a kludge, it is justified to lag it.
13.4.8 Linear System Solver
There are a number of choices for the linear system solver. The performance of the solvers usuallydepends on problems and the computer. So it is worth trying a few solvers to find out which oneis best for your problem and computer.
radsolve.level solver flag: the linear solver in Hypre to use. The available choices are:
• 0: SMG
• 1: PFMG (≥ 2-d only)
• 100: AMG using ParCSR ObjectType
• 102: GMRES using ParCSR ObjectType
• 103: GMRES using SStruct ObjectType
• 104: GMRES using AMG as preconditioner
• 109: GMRES using Struct SMG/PFMG as preconditioner
• 150: AMG using ParCSR ObjectType
• 1002: PCG using ParCSR ObjectType
• 1003: PCG using SStruct ObjectType
As a general rule, the SMG is the most stable solver, but is usually the slowest. The asym-metry in the linear system comes from the adaptive mesh, so the PFMG should be your firstchoice. Note: in you cannot use PFMG.
Setting this to 109 (GMRES using Struct SMG/PFMG as preconditioner) should work rea-sonably well for most problems.
radsolve.maxiter (default: 40): Maximal number of iteration in Hypre.
radsolve.reltol (default: 1.e-10): Relative tolerance in Hypre
radsolve.abstol (default: 0): Absolute tolerance in Hypre
radsolve.v (default: 0): Verbosity
radsolve.verbos (default: 0): Verbosity
habec.verbose (default: 0): Verbosity for level solver flag < 100
hmabec.verbose (default: 0): Verbosity for level solver flag >= 100
134 Chapter 13. Radiation
13.5 Output
13.5.1 Gray Solver
For the gray radiation solver, the radiation energy density is stored in plotfiles as rad. Note thatthis quantity has units of erg cm−3, which is different that the specify internal energy of the gaserg g−1.
CHAPTER 14
Particles
14.1 Tracer particles
Tracer particles are to track the Lagrangian evolution of a model fluid using discrete particles.In hydrodynamical simulations based on an Eulerian grid (including CASTRO), thermodynamicvariables at a given time are derived by solving the equations of motion of a fluid between cells.Therefore, in this scheme, the physical quantities that we can access to are not discretized quan-tities at any given position, but rather average values over each cell. However, employing discreteparticles, passively advected with the fluid flow, allows us to obtain local instantaneous thermody-namic variables, such as the temperature and the density, at well-defined positions, independent ofthe spatial resolution, i.e., the spatial cell size. This means that we can follow the evolution of thefluid at any given position and time.
CASTRO provides a tracer particle scheme with useful options. In this scheme, particles areadvanced using the midpoint method either with the cell-centered velocities or the face-centeredvelocities (Marker-And-Cell method)1. The number and the initial positions of particles are flexiblydetermined according to the purpose of a given model.
14.2 Initializing the Particles
One must include the tracer particles in the GNUmakefile by setting
USE PARTICLES = TRUE.
1One can simplify interpolation with the cell-centered velocity. However, this can lead to decoupling of the pressureand the velocity components, possibly resulting in instability. This can be avoided with the face-centered velocity
135
136 Chapter 14. Particles
And the particles can be initialized via
castro.do tracer particles = 1
in the inputs file.
If one wants to investigate the evolution of fluid motions starting from specific positions (or a certainrange of area or volume), one should manually specify the positions of particles by providing aninput file containing the total number and the initial positions of the particles. The input fileshould be in the same directory where your inputs file is located. The name of the input file isdetermined via :
particles.particle init file =particle file
Here particle file is the user-specified name of the file. The first line in this file is assumed to containthe number of particles. Each line after that contains the positions in a coordinate system adoptedfor your model. For 3-D cartesian coordinates,
x y z
For example, an input file for a model fluid with 6 particles in 2-D Cartesian coordinates may looklike,
6
3.28125e+08 9.9198e+08
5.46875e+08 9.9198e+08
7.65625e+08 9.9198e+08
9.84375e+08 9.9198e+08
1.20312e+09 9.9198e+08
1.42188e+09 9.9198e+08
According to this input file, the 6 particles will be positioned at the same height (same y coordinatein the second column), equally spaced in x direction (the first column except for the particle numberon the first line) from 3.28× 108 cm to 1.42× 109 cm.
14.3 Output file
The output files are stored in a directory whose name is determined by a variable particles.timestamp dir.For example, if the variable is set as follows,
particles.timestamp dir= particle dir,
A directory particle dir is automatically made with the directories for the main CASTRO outputfile (plt****) once a simulation starts and the particle output files are stored inside that directory.
The name of the output file consists of Timestamp along with a number at the end. The numberincreases (typically from 00) as more processors are involved in following the trajectories of particles.
14.3—Output file 137
In parallel computing, a computational domain is divided according to the number of processorsrequested. Then each processor only follows the particles on the domain assigned to that processorand records their positions and velocities at any given time in a different output file. Since it ispossible for particles to move from one domain to another during the evolution, its history can bestored in different files. More output files (with larger numbers at the end of the file name) can beproduced as more processors track the particles.
By default, the output file contains the positions and velocities of all particles at a given time,meaning [3 + 2×dimensionality] columns. For example, for particles in a 3-D domain, the columnsin the output file are,
index1 index2 x y z t vx vy vz [ρ T ]
The first two integers correspond to the particle index and the processor number. One should usethe two numbers in order to identify a particle and extract its history (i.e., the trajectory in Figure14.1).
Figure 14.1: A model atmosphere (left panel) and the trajectories of 500 particles (right panel)following the fluid motion on the atmosphere. The particles are initially positioned at five differentheights, y = 13000 km, 11000 km, 8000 km, 6000 km, 38000 km (100 particles at each height).In the left panel, the arrows roughly show the fluid motion. In the right panel, the solid linesrepresent the trajectories of the particles.
One can also add the last two columns [ρ T ], i.e., the local density and local temperature of fluidat the position of each particle by setting the following,
particles.timestamp temperature = 1,particles.timestamp density = 1.
For example, let’s consider 10 particles on a domain. If 4 out 10 particles are initially on a processorand the rest are on another processor, this means two processors are tracking the particles and twooutput files are produced. In the output file written by the processor with 4 particles, one can findthat four lines are stored at the same time and each line corresponds to each particle info. whilein the other output file for the other 6 particles, 6 lines are stored at the same time.
If particles.write in plotfile=1, the particle data are stored in a binary file along with the mainCASTRO output plotfile in directories plt*****/Tracer/.
138 Chapter 14. Particles
14.3.1 Run-time Screen Output
The verbosity written to the screen at run-time is constrolled by setting:
particles.v = 0 or 1 (default: 0)
CHAPTER 15
Equation of State and Burning Network
15.1 Equation of State
15.1.1 Standard Castro EOSes
Castro is written in a modular fashion so that the EOS and network burning routines can besupplied by the user. However, for the examples presented later we use several EOS and networkroutines that come with the Microphysics distribution.
Castro relies on routines to calculate the equation of state (EOS) of a fluid, as well as a speciesnetwork to define the components of the fluid. The network optionally has the ability to do nuclearburning, but for this section its main purpose is in defining the species so that the EOS can calculatefluid properties that depend on composition, such as electron fraction.
By default, Castro comes with the gamma law EOS. This represents a gamma law gas, with equationof state:
P = (γ − 1)ρe. (15.1)
The gas is currently assumed to be monatomic and ideal. (Only a restricted set of thermodynamicvariables are actually calculated, the minimum necessary for the hydrodynamics. A fuller set ofthermodynamic variables, for example the entropy from the Sackur-Tetrode equation, are calculatedin the gamma law general EOS inside the Microphysics repository.)
15.1.2 EOS Interfaces and Parameters
Each EOS should have two main routines by which it interfaces to the rest of Castro. At thebeginning of the simulation, eos init will perform any initialization steps and save EOS variables
139
140 Chapter 15. Equation of State and Burning Network
(mainly smallt, the temperature floor, and smalld, the density floor). Then, whenever you wantto call the EOS, use
call eos(eos input, eos state).
The first argument specifies the inputs to the EOS. The options that are currently available arestored in EOS/eos data.F90, and are always a combination of two thermodynamic quantities. Forexample, eos input rt means that we call the EOS with rho (density) and T (temperature) andwe expect the EOS to return the associated thermodynamic quantities such as internal energy e
and entropy s.
We note that for real (non-analytic) equations of state in which ρ, T and species are the independentvariables, such as the Helmholtz EOS, eos input rt directly calls the EOS and obtains the otherthermodynamic variables. But for other inputs, e.g. eos input re, a Newton-Raphson iteration isperformed to find the density or temperature that corresponds to the given input.
The eos state variable is a Fortran derived type (similar to a C++ struct). It stores a completeset of thermodynamic variables. When calling the EOS, you should first fill the variables that arethe inputs, for example with
use eos_type_module
...
type (eos_t) :: eos_state
...
eos_state % rho = state(i,j,k,URHO)
eos_state % T = state(i,j,k,UTEMP)
eos_state % e = state(i,j,k,UEINT) / state(i,j,k,URHO)
eos_state % xn = state(i,j,k,UFS:UFS+nspec-1) / state(i,j,k,URHO)
eos_state % aux = state(i,j,k,UFX:UFX+naux-1) / state(i,j,k,URHO)
Whenever the eos state type is initialized, the thermodynamic state variables are filled withunphysical numbers. If you do not input the correct arguments to match your input quantities, theEOS will call an error. This means that it is good practice to fill the quantities that will be iteratedover with an initial guess. Indeed, this initial guess is typically required for equations of state thatiterate over this variable, as the values they are initialized with will likely not converge. Usually aprior value of the temperature or density suffices if it’s available, but if not then use small temp
or small dens.
If you are interested in using more realistic and sophisticated equations of state, you should down-load the Microphysics repository. This is a collection of microphysics routines that are compatiblewith the BoxLib codes. We refer you to the documentation in that repository for how to set it upand for information on the equations of state provided. That documentation also goes into moredetail about the details of the EOS code, in case you are interested in how it works (and in caseyou want to develop your own EOS).
15.2 Nuclear Network
The nuclear network serves two purposes: it defines the fluid components used in both theequation of state and the hydrodynamics, and it evolves those components through a nuclear
15.2—Nuclear Network 141
burning step. Castro comes with a general null network (which lives in the Networks/ di-rectory). This is a bare interface for a nuclear reaction network. No reactions are enabled,and no auxiliary variables are accepted. It contains several sets of isotopes; for example,Networks/general null/triple alpha plus o.net would describe the isotopes needed to rep-resent the triple-α reaction converting helium into carbon, as well as oxygen and iron.
The main interface file, network.f90, is a wrapper function. The actual network details are definedin actual network.f90, a file which is automatically generated in your work directory when youcompile. It supplies the number and names of species and auxiliary variables, as well as otherinitializing data, such as their mass numbers, proton numbers, and the binding energies.
The burning front-end interface, Networks/burner.f90, accepts a different derived type called theburn t type. Like the eos t, it has entries for the basic thermodynamic quantities:
use burn_type_module
...
type (burn_t) :: burn_state
...
burn_state % rho = state(i,j,k,URHO)
burn_state % T = state(i,j,k,UTEMP)
burn_state % e = state(i,j,k,UEINT) / state(i,j,k,URHO)
burn_state % xn = state(i,j,k,UFS:UFS+nspec-1) / state(i,j,k,URHO)
It takes in an input burn t and returns an output burn t after the burning has completed. Thenuclear energy release can be computed by taking the difference of burn state out % e andburn state in % e. The species change can be computed analogously. In normal operation inCastro the integration occurs over a time interval of ∆t/2, where ∆t is the hydrodynamics timestep.
If you are interested in using actual nuclear burning networks, you should download the Microphysicsrepository. This is a collection of microphysics routines that are compatible with the BoxLib codes.We refer you to the documentation in that repository for how to set it up and for information onthe networks provided. That documentation also goes into more detail about the details of thenetwork code, in case you are interested in how it works (and in case you want to develop your ownnetwork).
15.2.1 Required Thermodynamics Quantities
Three input modes are required of any EOS:
• eos input re: ρ, e, and Xk are input
• eos input rt: ρ, T , and Xk are input
• eos input rp: ρ, P , and Xk are input
The eos t derived type holds a large number of thermodynamics quantities, but not all of theseare needed for basic Castro operation. The main quantities that any EOS in any mode needs tosupply, if they are not input, are:
• eos state % T: the temperature
• eos state % P: total pressure
142 Chapter 15. Equation of State and Burning Network
• eos state % e: the specific energy
• eos state % gam1: the first adiabatic index, Γ1 = d logP/d log ρ|s
Additionally the eos input re mode also needs to supply:
• eos state % cs: the adiabatic sound speed
• eos state % dpdr e: the derivative, ∂p/∂ρ|e—note that the specific internal energy, e is heldconstant here.
• eos state % dpde: the derivative, ∂p/∂e|ρ
For radiation hydro, the eos input rt model needs to supply:
• eos state % cv: the specific heat capacity.
Other quantities (e.g., entropy) might be needed for the derived variables that are optional outputinto the plotfiles.
For burning, when the temperature equation is evolved, the EOS needs to supply:
• eos state % dhdX(nspec): the derivative of the specific enthalpy with respect to mass frac-tion at constant T and p. This is commonly computed as:
ξk = eXk+
1
pρ
(p
ρ2− eρ
)pXk
. (15.2)
with
pXk=
∂p
∂A
∣∣∣∣ρ,T,Z
∂A
∂Xk+
∂p
∂Z
∣∣∣∣ρ,T,A
∂Z
∂Xk
= − A2
Ak
∂p
∂A
∣∣∣∣ρ,T,Z
+A
Ak
(Zk − Z
) ∂p∂Z
∣∣∣∣ρ,T,A
, (15.3)
eXk=
∂e
∂A
∣∣∣∣ρ,T,Z
∂A
∂Xk+
∂e
∂Z
∣∣∣∣ρ,T,A
∂Z
∂Xk
= − A2
Ak
∂e
∂A
∣∣∣∣ρ,T,Z
+A
Ak
(Zk − Z
) ∂e∂Z
∣∣∣∣ρ,T,A
. (15.4)
(see [2], Appendix A).
CHAPTER 16
AMR
Our approach to adaptive refinement in Castro uses a nested hierarchy of logically-rectangular gridswith simultaneous refinement of the grids in both space and time. The integration algorithm onthe grid hierarchy is a recursive procedure in which coarse grids are advanced in time, fine gridsare advanced multiple steps to reach the same time as the coarse grids and the data at differentlevels are then synchronized.
During the regridding step, increasingly finer grids are recursively embedded in coarse grids untilthe solution is sufficiently resolved. An error estimation procedure based on user-specified criteria(described in Section 16.1) evaluates where additional refinement is needed and grid generationprocedures dynamically create or remove rectangular fine grid patches as resolution requirementschange.
A good introduction to the style of AMR used here is in Lecture 1 of the Adaptive Mesh RefinementShort Course at https://ccse.lbl.gov/people/jbb/shortcourse/lecture1.pdf.
16.1 Tagging for Refinement
Castro determines what zones should be tagged for refinement at the next regridding step by usinga set of built-in routines that test on quantities such as the density and pressure and determiningwhether the quantities themselves or their gradients pass a user-specified threshold. This may thenbe extended if amr.n error buf > 0 to a certain number of zones beyond these tagged zones.This section describes the process by which zones are tagged, and describes how to add customizedtagging criteria.
The routines for tagging cells are located in the Tagging nd.f90 file in the Source/driver/ di-rectory. (These are dimension-agnostic routines that loop over all three dimensional indices evenfor 1D or 2D problems.) The main routines are ca denerror, ca temperror, ca presserror,
143
144 Chapter 16. AMR
ca velerror, and ca raderror. They refine based on density, temperature, pressure, velocity, andradiation energy density (if enabled), respectively. The same approach is used for all of them. As anexample, we consider the density tagging routine. There are four parameters that control tagging.If the density in a zone is greater than the user-specified parameter denerr, then that zone will betagged for refinement, but only if the current AMR level is less than the user-specified parametermax denerr lev. Similarly, if the absolute density gradient between a zone and any adjacent zoneis greater than the user-specified parameter dengrad, that zone will be tagged for refinement, butonly if we are currently on a level below max dengrad lev. Note that setting denerr alone will notdo anything; you’ll need to set max dengrad lev >= 1 for this to have any effect.
All four of these parameters are set in the &tagging namelist in your probin file. If left unmodified,they default to a value that means we will never tag. The complete set of parameters that can becontrolled this way is the following:
• density:
– value: denerr, max denerr lev
– gradient: dengrad, max dengrad lev
• temperature:
– value: temperr, max temperr lev
– gradient: tempgrad, max tempgrad lev
• velocity (magnitude):
– value: velerr, max velerr lev
– gradient: velgrad, max velgrad lev
• pressure:
– value: presserr, max presserr lev
– gradient: pressgrad, max pressgrad lev
• radiation energy density:
– value: raderr, max raderr lev
– gradient: radgrad, max radgrad lev
Since there are multiple algorithms for determining whether a zone is tagged or not, it is worthwhileto specify in detail what is happening to a zone in the code during this step. We show this in thefollowing pseudocode section. A zone is tagged if the variable itag = SET, and is not tagged ifitag = CLEAR (these are mapped to 1 and 0, respectively).
itag = CLEAR
for errfunc[k] from k = 1 ... N
// Three possibilities for itag: SET or CLEAR or remaining unchanged
call errfunc[k](itag)
end for
16.1—Tagging for Refinement 145
In particular, notice that there is an order dependence of this operation; if errfunc[2] CLEARs azone and then errfunc[3] SETs that zone, the final operation will be to tag that zone (and viceversa). In practice by default this does not matter, because the built-in tagging routines neverexplicitly perform a CLEAR. However, it is possible to overwrite the Tagging nd.f90 file if you wantto change how ca denerror, ca temperror, etc. operate. This is not recommended, and if youdo so be aware that CLEARing a zone this way may not have the desired effect.
We provide also the ability for the user to define their own tagging criteria. This is done throughthe Fortran function set problem tags in the problem tagging *d.f90 files. This function isprovided the entire state (including density, temperature, velocity, etc.) and the array of taggingstatus for every zone. As an example of how to use this, suppose we have a 3D Cartesian simulationwhere we want to tag any zone that has a density gradient greater than 10, but we don’t care aboutany regions outside a radius r > 75 from the problem origin; we leave them always unrefined. Wealso want to ensure that the region r ≤ 10 is always refined. In our probin file we would setdenerr = 10 and max denerr lev = 1 in the &tagging namelist. We would also make a copy ofproblem tagging 3d.f90 to our work directory and set it up as follows:
subroutine set_problem_tags(tag ,tagl1 ,tagl2 ,tagl3 ,tagh1 ,tagh2 ,tagh3 , &
state ,state_l1 ,state_l2 ,state_l3 , &
state_h1 ,state_h2 ,state_h3 ,&
set ,clear ,&
lo ,hi ,&
dx ,problo ,time ,level)
use bl_constants_module , only: ZERO , HALF
use prob_params_module , only: center
use meth_params_module , only: URHO , UMX , UMY , UMZ , UEDEN , NVAR
implicit none
integer ,intent(in ) :: lo(3),hi(3)
integer ,intent(in ) :: state_l1 ,state_l2 ,state_l3 , &
state_h1 ,state_h2 ,state_h3
integer ,intent(in ) :: tagl1 ,tagl2 ,tagl3 ,tagh1 ,tagh2 ,tagh3
double precision ,intent(in ) :: state(state_l1:state_h1 , &
state_l2:state_h2 , &
state_l3:state_h3 ,NVAR)
integer ,intent(inout) :: tag(tagl1:tagh1 ,tagl2:tagh2 ,tagl3:tagh3)
double precision ,intent(in ) :: problo (3),dx(3),time
integer ,intent(in ) :: level ,set ,clear
double precision :: x, y, z, r
do k = lo(3), hi(3)
z = problo (3) + (dble(k) + HALF) * dx(3) - center (3)
do j = lo(2), hi(2)
y = problo (2) + (dble(j) + HALF) * dx(2) - center (2)
do i = lo(1), hi(1)
x = problo (1) + (dble(i) + HALF) * dx(1) - center (2)
r = (x**2 + y**2 + z**2)**( HALF)
if (r > 75.0) then
146 Chapter 16. AMR
tag(i,j,k) = clear
elseif (r <= 10.0) then
tag(i,j,k) = set
endif
enddo
enddo
enddo
end subroutine set_problem_tags
16.2 Synchronization Algorithm
Here we present the AMR algorithm for the compressible equations with self-gravity. The gravitycomponent of the algorithm is closely related to (but not identical to) that in Miniati and Colella,JCP, 2007. The content here is largely based on the content in the original Castro paper ([1]).The most significant difference is the addition of a different strategy for when to employ thesynchronization; but regardless of whether the original or new strategy is used, the fundamentalsynchronization step is identical.
16.2.1 Synchronization Methodology
Over a coarse grid time step we collect flux register information for the hyperbolic part of thesynchronization:
δF = −∆tcAcF c +
∑∆tfA
fF f (16.1)
Analogously, at the end of a coarse grid time step we store the mismatch in normal gradients of φat the coarse-fine interface:
δFφ = −Ac∂φc
∂n+∑
Af∂φf
∂n(16.2)
We want the composite φc−f to satisfy the multilevel version of (10.3) at the synchronization time,just as we want the coarse and fine fluxes at that time to match. So the goal is to synchronize φacross levels at that time and then zero out this mismatch register.
At the end of a coarse grid time step we can define Uc−f
and φc−f
as the composite of the datafrom coarse and fine grids as a provisional solution at time n + 1. (Assume U has been averageddown so that the data on coarse cells underlying fine cells is the average of the fine cell data aboveit.)
The synchronization consists of two parts:
• Step 1: Hyperbolic reflux
In the hyperbolic reflux step, we update the conserved variables with the flux synchronizationand adjust the gravitational terms to reflect the changes in ρ and u.
Uc,? = Uc
+δF
V, (16.3)
where V is the volume of the cell and the correction from δF is supported only on coarse cellsadjacent to fine grids.
16.2—Synchronization Algorithm 147
Note: this can be enabled/disabled via castro.do reflux. Generally, it should be enabled(1).
Also note that for axisymmetric or 1D spherical coordinates, the reflux of the pressure gradientis different, since it cannot be expressed as a divergence in those geometries. We use a separateflux register in the hydro code to store the pressure term in these cases.
• Step 2: Gravitational synchronization
In this step we correct for the mismatch in normal derivative in φc−f at the coarse-fineinterface, as well as accounting for the changes in source terms for (ρu) and (ρE) due to thechange in ρ.
On the coarse grid only, we define
(δρ)c = ρc,? − ρc. (16.4)
We then form the composite residual, which is composed of two contributions. The first is
the degree to which the current φc−f
does not satisfy the original equation on a composite
grid (since we have solved for φc−f
separately on the coarse and fine levels). The second isthe response of φ to the change in ρ. We define
R ≡ 4πGρ?,c−f −∆c−f φc−f
= −4πG(δρ)c − (∇ · δFφ)|c. (16.5)
Then we solve
∆c−f δφc−f = R (16.6)
as a two level solve at the coarse and fine levels. We define the update to gravity,
δgc−f = ∇(δφc−f ). (16.7)
Finally, we need to
– add δφc−f directly to to φc and φf and interpolate δφc−f to any finer levels and add tothe current φ at those levels.
– if level c is not the coarsest level in the calculation, then we must transmit the effect ofthis change in φ to the coarser levels by updating the flux register between level c andthe next coarser level, cc. In particular, we set
δFφcc−c = δF cc−cφ +
∑Ac∂(δφ)c−f
∂n. (16.8)
The gravity synchronization algorithm can be disabled with gravity.no sync = 1. Thisshould be done with care. Generally, it is okay only if he refluxing happens in regions of lowdensity that don’t affect the gravity substantially.
148 Chapter 16. AMR
16.2.2 Source Terms
After a synchronization has been applied, the state on the coarse grid has changed, due to thechange in fluxes at the coarse-fine boundary as well as the change in the gravitational field. Thisposes a problem regarding the source terms, all of which generally rely either on the state itself, oron the global variables affected by the synchronization such as the gravitational field. The new-timesources constructed on the coarse grid all depended on what the state was after the coarse-gridhydrodynamic update, but the synchronization and associated flux correction step retroactivelychanged that hydrodynamic update. So one can imagine that in a perfect world, we would havecalculated the hydrodynamic update first, including the coarse-fine mismatch corrections, and onlythen computed the source terms at the new time. Indeed, an algorithm that did not subcycle, butmarched every zone along at the same timestep, could do so – and some codes, like FLASH, actuallydo this, where no new-time source terms are computed on any level until the hydrodynamic updatehas been fully completed and the coarse-fine mismatches corrected. But in Castro we cannot do this;in general we assume the ability to subcycle, so the architecture is set up to always calculate thenew-time source terms on a given level immediately after the hydrodynamic update on that level.Hence on the coarse level we calculate the new-time source terms before any fine grid timestepsoccur.
One way to fix this, as suggested by Miniati and Colella for the case of gravity, is to explicitlycompute what the difference in the source term is as a result of any flux corrections across coarse-fine boundaries. They work out the form of this update for the case of a cell-centered gravitationalforce, which has contributions from both the density advected across the coarse-fine interfaces (i.e.δρg, where δρ is the density change due to the coarse-fine synchronization on the coarse rid), as wellas the global change in the gravitational field due to the collective mass motion (see Miniati andColella for the explicit form of the source term). This has a couple of severe limitations. First, itmeans that when the form of the source term is changed, the form of the corrector term is changedtoo. For example, it is less easy to write down the form of this corrector term for the flux-basedgravitational energy source term that is now standard in Castro. Second, gravity is a relativelyeasy case due to its linearity in the density and the gravitational acceleration; other source termsrepresenting more complicated physics might not have an easily expressible representation in termsof the reflux contribution. For example, for a general nuclear reaction network (that does nothave an analytic solution), it is not possible to write down an analytic expression for the nuclearreactions that occur because of δρ.
Instead we choose a more general approach. On the coarse level, we save the new-time source termsthat were applied until the end of the fine timesteps. We also save the fine level new-time sourceterms. Then, when we do the AMR synchronization after a fine timestep, we first subtract thepreviously applied new-time source terms to both the coarse and the fine level, then do the flux cor-rection and associated gravitational sync solve, and then re-compute the new-time source terms onboth the coarse and the fine level1. In this way, we get almost the ideal behavior – if we aren’t sub-cycling, then we get essentially the same state at the end of the fine timestep as we would in a codethat explicitly had no subcycling. The cost is re-computing the new-time source terms that second
1In the absence of a global field like the gravitational potential, this would only need to be done on the coarselevel, as we always assume that the solution on the fine grid is correct and average it down to the coarse grid. InCastro we do it by default on the fine level too in anticipation of the fact that gravity is a common component ofmany of our production science simulations. This could be generalized so that if you aren’t using any global forcefields, you don’t bother updating the fine level. If this is important to the science you want to do, please let theCastro developers know and we can look into it.
16.2—Synchronization Algorithm 149
time on each level. For most common source terms such as gravity, this is not a serious problem –the cost of re-computing ρg (for example, once you already know g) is negligible compared to thecost of actually computing g itself (say, for self-gravity). If you believe that the error in not recom-puting the source terms is sufficiently low, or the computational cost of computing them too high,you can disable this behavior2 using the code parameter castro.update sources after reflux.
Note that at present nuclear reactions are not enabled as part of this scheme, and at present arenot automatically updated after an AMR synchronization. This will be amended in a future releaseof Castro.
16.2.3 Synchronization Timing
The goal of the synchronization step is for the coarse and fine grid to match at the end of a coarsetimesteps, after all subcycled fine grid timesteps have been completed and the two levels havereached the same simulation time. If subcycling is disabled, so that the coarse and fine grid takethe same timestep, then this is sufficient. However, in the general subcycling case, the situationis more complicated. Consider the discussion about source terms in 16.2.2. If we have a coarselevel and one fine level with a refinement ratio of two, then for normal subcycling the fine gridtakes two timesteps for every one timestep taken by the coarse level. The strategy advocated bythe original Castro paper (and Miniati and Colella) is to only do the AMR synchronization at theactual synchronization time between coarse and fine levels, that is, at the end of the second finetimestep. Consequently, we actually only update the source terms after that second fine timestep.Thus note that on the fine grid, only the new-time source terms in the second fine timestep areupdated. But a moment’s thought should reveal a limitation of this. The first fine grid timestepwas also responsible for modifying the fluxes on the coarse grid, but the algorithm as presentedabove didn’t take full account of this information. So, the gravitational field at the old time inthe second fine timestep is actually missing information that would have been present if we hadupdated the coarse grid already. Is there a way to use this information? For the assumptions wemake in Castro, the answer is actually yes. If we apply the effect of the synchronization not at thesynchronization time but at the end of every fine timestep, then every fine timestep always hasthe most up-to-date information possible about the state of the gravitational field. Now, of course,in fine timesteps before the last one, we have not actually reached the synchronization time. Butwe already know at the end of the first fine timestep what the synchronization correction will befrom that fine timestep: it will be equal to 1/2 of the coarse contribution to the flux register andthe normal contribution to the flux register for just that timestep. This is true because in Castro,we assume that the fluxes provided by the hydrodynamic solver are piecewise-constant over thetimestep, which is all that is needed to be second-order accurate in time if the fluxes are timecentered3. So it is fair to say that halfway through the coarse timestep, half of the coarse fluxhas been advected, and we can mathematically split the flux register into two contributions thathave equal weighting from the coarse flux. (In general, of course, the coarse flux contribution ateach fine timestep is weighted by 1/R where R is the refinement ratio between the coarse and finelevels.) So, there is nothing preventing us from updating the coarse solution at the synchronization
2in general it may be desirable for this to be a source-term specific setting, so that some source terms that are cheapor physically important are re-computed after a synchronization can be set to update, while others can be disabled.If this is important for your science application, please let the developers know, as this would be a straightforwardextension of the current architecture.
3If this scheme is generalized to higher-order methods, in principle all one would need to do is integrate the fluxesuntil ∆t/2, which is what we are doing here for the constant-in-time flux case.
150 Chapter 16. AMR
time tn+1c after this first fine timestep; we already know at that point how the coarse solution will
change, so why not use that information? We can then update the gravitational potential at tn+1/2c
that is used to construct the boundary conditions for the gravitational potential solve on the finegrid at the beginning of the second fine timestep.
In practice, this just means calling the synchronization routine described in 16.2.1, with the onlymodification being that the flux register contribution from the coarse grid is appropriately weightedby the fine grid timestep instead of the coarse grid timestep, and we only include the current finestep:
δF = −∆tfAcF c + ∆tfA
fF f (16.9)
The form of the φ flux register remains unchanged, because the intent of the gravity sync solve is tosimply instantaneously correct the mismatch between the fine and coarse grid. The only difference,then, between the old strategy and this new method is that we call the synchronization at theend of every fine timestep instead of only the last subcycled one, and we change the weightingappropriately. This new method is more expensive as currently implemented because we haveto do R gravitational sync solves, refluxes, and source term recalculations instead of only one.However, it results in maximal possible accuracy, especially in cases where significant amounts ofmaterial are crossing refinement boundaries. The reflux strategy is controlled by the parametercastro.reflux strategy. At present the old method is still the default.
Note that one does not need to be using self-gravity for this to be beneficial. Even in pure hy-drodynamics this can matter. If a regrid occurs on the fine level, new zones on the boundariesof the current fine level are filled by interpolation from the coarse level. In the old method, thatinterpolation is not using the most up-to-date data that accounts for the synchronization.
For multiple levels of refinement, the scheme extends naturally. In the old method, we alwayscall the synchronization at the synchronization time between any two levels. So for example withtwo jumps in refinement by a factor of two, there is a synchronization at the end of the first twotimesteps on level 2 (between level 1 and level 2), a synchronization after the next two timestepson level 2 (again between level 1 and level 2), and then a synchronization between level 0 and level1. In the new method, we always call the synchronization at the end of every timestep on the finestlevel only, and we simultaneously do the synchronization on every level. The timestep ∆tf in theflux register is just the timestep on the finest level. (If this is unclear, give it a sanity check: whenthe sum of all flux register totals is added up, the level 0 contribution will have a factor of ∆t equalto the coarse grid timestep since the sum of the timesteps on the finest level over the entire advancemust equal the level 0 timestep. So, the final contribution from the flux register is the same as ifwe had saved up the flux mismatch until the end of the level 0 timestep.) The synchronization nolonger needs to be called at the end of any coarser level’s timestep because it will already be up todate as a result of the synchronizations applied at the end of the fine level timesteps.
CHAPTER 17
ConvertCheckpoint
Within the CASTRO distribution, there is the capability to “grow” a checkpoint file so that a calcu-lation can be restarted in a larger domain covered by grid cells a factor of two or four coarser than theexisting coarsest level. Instructions for how to do so are in the Castro/Util/ConvertCheckpoint/READMEfile and are included here. Upon restart the existing data in the checkpoint file will be used to fillthe region of the previous computational domain, and the new regions will be filled by some othermeans, typically interpolation from a 1D model file.
17.1 Star in Corner (star at center = 0)
In this section we consider the case where the star (or feature of interest) is centered at the lowerleft corner of the domain, e.g. you are modeling only one quarter of the star in 2D, or an octant ofthe star in 3D. Then you only want to grow the domain in the “high side” directions (e.g., to theupper right).
17.1.1 Converting the Checkpoint File
Let’s say you have a checkpoint file, chk00100, say, with 5 levels of refinement and a (real) problemdomain size P and (integer) domain size D at level 0.
The inputs file that created this might have contained:
• max step = 100
• amr.max level = 5
• amr.n cell = D D
• geometry.prob lo = 0 0
151
152 Chapter 17. ConvertCheckpoint
• geometry.prob hi = P P
• amr.ref ratio = 4 4 4 4
Now let’s suppose that you want to grow the domain by a factor of 8 and cover that new largerdomain with a level that is a factor of 2 coarser than the existing level 0 grids.
1. First, set DIM = in the GNUmakefile, and type “make” in the Util/ConvertCheckpoint/
directory. This will make an executable from the Embiggen.cpp code.
2. Run the embiggening code as follows:
Embiggen2d.Linux.Intel.Intel.ex checkin=chk00100 checkout=newchk00050 ref_ratio=2 grown_factor=8 star_at_center=0
(Your executable may have a slightly different name depending on the compilers you built itwith.)
This will create a new checkpoint directory, called newchk00050, that represents a simulationwith one additional level of refinement coarser than the previous level 0 grids by a factor ofref ratio (in this case, 2). The new domain will be a factor of grown factor (in this case,8) larger than the previous domain.
Note that ref ratio must be 2 or 4, because those are the only acceptable values of ref ratio
in Castro.
grown factor can be any reasonable integer; but it’s only been tested with 2, 3, 4 and 8. Itdoes not need to be a multiple of 2.
17.1.2 Restarting from a Grown Checkpoint File
You should now be able to restart your calculation using newchk00050.
Your inputs file should now contain lines like:
• max step = 51
• amr.restart = newchk00050
• amr.max level = 6
• amr.n cell = 4D 4D
• geometry.prob lo = 0 0
• geometry.prob hi = 8P 8P
• castro.grown factor = 8
• castro.star at center = 0
• amr.ref ratio = 2 4 4 4 4
IMPORTANT:
1. Unlike earlier, you may now set amr.max level to be at most one greater than before, butyou need not set it that high. For example, you could set amr.max level the same as beforeand you would lose data at the finest refinement level. You may not set amr.max level =
17.2—Star at Center of Domain (star at center = 1) 153
0, however, because we have no data at the new level 0 until we average down from the newlevel 1 after the restart.
2. You must set amr.n cell = (grown factor / ref ratio) times the previous value ofamr.n cell. In this case amr.n cell = (8/2)*D = 4D.
3. You must set amr.prob hi to be a factor of grown factor greater than the previous valueof amr.prob hi.
4. You must insert the value of ref ratio used in the Embiggen call as the first value in the listof amr.ref ratio, since that will now be the refinement ratio between the new level 0 andthe new level 1.
5. You must set castro.grown factor in your inputs file equal to the value of grown factoryou used when you called Embiggen*ex so that the CASTRO code knows how big the originaldomain was.
6. Note that if you have run 100 steps at the original level 0, that would be equivalent to 50steps at the new level 0 because you coarsened by a factor of 2. Thus once you re-start fromthe new checkpoint directory, the next step will be 51, not 101. Make sure to keep track ofyour plotfiles accordingly.
7. Don’t forget to adjust max denerr lev and comparable variables to control the number of finelevels you now want. If you want to have 6 levels of refinement after restart, then make suremax denerr lev, etc, are set high enough. If you only want to have 5 levels of refinement(where the new level 5 would now be a factor of ref ratio coarser than the previous level 5),make sure to adjust max denerr lev accordingly as well.
17.2 Star at Center of Domain (star at center = 1)
Now let’s assume that the star (or feature of interest) is centered at the center of the domain in 2Dor 3D Cartesian coordinates. We will later consider the case of 2D cylindrical (r-z) coordinates inwhich the star is centered at the left midpoint.
17.2.1 Converting the Checkpoint File
Suppose that you want to grow the domain by a factor of 2 and cover that new larger domain witha level that is a factor of 2 coarser than the existing level 0 grids.
After you build the Embiggen executable, you type:
• Embiggen2d.Linux.Intel.Intel.ex checkin=chk00100 checkout=newchk00050 ref ratio=2grown factor=2 star at center=1
Note that
• ref ratio must still be 2 or 4
• grown factor can only be 2 or 3 in this case.
154 Chapter 17. ConvertCheckpoint
17.2.2 Restarting from a Grown Checkpoint File
Your inputs file for restarting would now look like
• max step = 51
• amr.restart = newchk00050
• amr.max level = 6
• amr.n cell = D D
• geometry.prob lo = -P/2 -P/2
• geometry.prob hi = 3P/2 3P/2
• castro.grown factor = 2
• castro.star at center = 1
• amr.ref ratio = 2 4 4 4 4
17.2.3 Cylindrical Coordinates
In the case of 2D cylindrical (r-z) coordinates in which the star is centered at the left edge but verti-cal midpoint of the domain, the embiggening procedure is the same as above (with star at center= 1) but the inputs file for restart is slightly different in that geometry.prob lo is modified inthe z- but not the r-direction. If we consider the original inputs file to look like:
• max step = 100
• amr.max level = 6
• amr.n cell = D 2D
• geometry.prob lo = 0 0
• geometry.prob hi = P 2P
• amr.ref ratio = 4 4 4 4
then an inputs file for restart would look like:
• amr.restart = newchk00050
• amr.max level = 6
• amr.n cell = D 2D
• geometry.prob lo = 0 -P
• geometry.prob hi = 2P 3P
• castro.grown factor = 2
• castro.star at center = 1
• amr.ref ratio = 2 4 4 4 4
17.2—Star at Center of Domain (star at center = 1) 155
Figure 17.1: Data from checkpoint file before and after the domain has been coarsened andgrown. This case uses star at center = 0 and ref ratio=2. The first grown example hasgrown factor=2, the second has grown factor=3. In all figures the level 0 grids are shownin white, the level 1 grids in red, the level 2 grids in yellow, and in the grown figures, the level 3grids are in pink.
156 Chapter 17. ConvertCheckpoint
Figure 17.2: Data from checkpoint file before and after the domain has been coarsened andgrown. This case uses star at center = 0 and ref ratio=2. The first grown example hasgrown factor=2, the second has grown factor=3. In all figures the level 0 grids are shownin white, the level 1 grids in red, the level 2 grids in yellow, and in the grown figure, the level 3grids are in pink.
CHAPTER 18
Initializing Castro with Maestro Data
18.1 Overview
We can now initialize a Castro simulation using data from a Maestro plotfile. This should not bethought of as a restart mode, but rather a new simulation with a special initialization. In orderto use this feature, you must make sure the Maestro plotfile has the proper variables, add somenew parameters to your inputs file, and add a few subroutines to Prob Xd.f90. You need to builda special executable with “USE MAESTRO INIT=TRUE”, which will add “.MAESTRO” to theexecutable string. For multilevel problems, there are a few extra steps relating to the fact that youhave to supply a grids file consistent with the Maestro grid structure.
18.2 MAESTRO Plotfile Requirements
The Maestro plotfile needs to have the following variables:
• “x vel”, “y vel”, (and “z vel”, depending on dimensionality of the problem)
• “density” (castro.MAESTRO init type = 1 and 2 only)
• Optional species (such as “X(C12)”) - there is an option to not read any species from theMaestro plotfile. In this case, you must make sure your code manually defines the speciescell-by-cell in the initial Castro data
• “tfromp”
• “pi” (castro.MAESTRO init type = 2, 3, and 4 only)
• “entropy” (castro.MAESTRO init type = 4 only)
157
158 Chapter 18. Initializing Castro with Maestro Data
Also, model cc XXXXX needs to list variables in the following order, which is the default orderfound in MAESTRO/Source/base io.f90: r, base r, rho0, p0, gamma1bar, rhoh0, div coeff, psi,tempbar, etarho cc, tempbar init.
18.3 List of Parameters
Here are the additional parameters you must add to your inputs file.
Parameter Definition Type Defaultcastro.MAESTRO plotfile name of the Maestro plotfile std::string must be setcastro.MAESTRO modelfile name of the Maestro “model cc” file std::string must be setcastro.MAESTRO npts model number of points in the Maestro model cc file int must be setcastro.MAESTRO first species name of the first species std::string must be set or else nothing will be read incastro.MAESTRO nspec number of species in the Maestro plotfile std::string NumSpec in Castrocastro.MAESTRO cutoff density controls how we overwrite data at the edge of the star Real must be setcastro.MAESTRO init type determines how we initialize the Castro state int must be setcastro.MAESTRO spherical specifies planar or spherical problem int must be set
18.3.1 Examples of Usage
• castro.MAESTRO plotfile = ”wd 384 6.25e8K norotate plt120578”
• castro.MAESTRO modelfile = ”./wd 384 6.25e8K norotate plt120578/model cc 120578”
• castro.MAESTRO npts model = 1663This is the number of points in castro.MAESTRO modelfile. Note that this is not thesame thing as “npts model”, which is the number of points in the initial model file used forstandard simulations where we do not initialize from a Maestro plotfile.
• castro.MAESTRO first species = “X(C12)” If you do not specify this, no species will beread in. You can always manually specify or overwrite the species cell-by-cell later.
• castro.MAESTRO nspec = 3If you do not specify this, it will default to the number of species in the Castro network,“NumSpec”. We have this here because sometimes Maestro and Castro will use differentnetworks with different number of species.
• castro.MAESTRO cutoff density = 1.e6The code will use this density to figure out the radial coordinate, r model start, which is thelast radial coordinate before rho0 falls below castro.MAESTRO cutoff density. It is pos-sible to set castro.MAESTRO cutoff density to a tiny value, such that rho0 never fallsbelow this value, in which case we set r model start to ∞. In INITDATA MAKEMODEL,we create a new 1D model integrating outward starting from r model start. Then, in INIT-DATA OVERWRITE, we overwrite newly initialized Castro data in any cell that maps intoa radial coordinate greater than r model start by interpolating from the new 1D model.
• castro.MAESTRO init type = 2Castro will read in data from the Maestro plotfile, and then call the EOS to make surethat ρ, e, T , and Xk are consistent. The inputs to the EOS are based on the value ofcastro.MAESTRO init type:
1. e = e(ρ, T,Xk)
18.4—New Subroutines in Prob Xd.f90 159
2. e, T = e, T (ρ, p0 + π,Xk)
3. ρ, e = ρ, e(p0 + π, T,Xk)
4. ρ, T, e = ρ, T, e(p0 + π, s,Xk)
• castro.MAESTRO spherical = 10 = planar; 1 = spherical.
18.4 New Subroutines in Prob Xd.f90
There are three routines that need to be added to your local copy of Prob Xd.f90. SeeCastro/Exec/wdconvect/Prob 3d.f90 for a standard spherical Maestro initialization.
1. INITDATA MAESTROThis fills in the Castro state by taking the Maestro data, calling the EOS, and making theproper variables conserved quantities. Specifically, we need a thermodynamically consistentρ, T , e, and Xk, and then algebraically compute ρu, ρe, ρE, and ρXk,
2. INITDATA MAKEMODELThis creates a user-defined 1D initial model starting from r model start.
3. INITDATA OVERWRITEThis overwrites the initialized Castro data using the new 1D initial model for all cells thatmap into radial coordinates greater than r model start.
18.5 Additional Notes
Note that for both single-level and multilevel Maestro to Castro initialization, the Castro base gridstructure does not have to match the Maestro base grid structure, as long as the problem domainis the same. For example, if the coarsest level in a Maestro plotfile contains 643 cells divided into8-323 grids, it is ok to use a Castro base grid structure with 1-643 grid, 64-163 grids, or anythingelse you can imagine - the grids don’t even have to be the same size. As is normally the case,the Castro base grid structure is created based on the parameters in the Castro inputs file, such asamr.max grid size, amr.blocking factor, etc.
18.5.1 Multilevel Restart
When initialing from a multilevel Maestro plotfile, there are some extra steps. First, you needto create a Castro-compatible grids file from the Maestro plotfile. This can be done with theBoxLib/Tools/Postprocessing/F Src/fboxinfo.f90 utility. Compile and run this using the“--castro” option, e.g., “fboxinfo.Linux.gfortran.exe --castro pltxxxxx | tee gr0.maestro”, to gen-erate the Castro-compatible grids file. Note that the base grid structure is still controlled byamr.max grid size, amr.blocking factor, etc., since in C++ BoxLib, the grids file only indi-cates the refined grid structure, whereas in Fortran BoxLib the grids file contains the base grid andrefined grid structures.
160 Chapter 18. Initializing Castro with Maestro Data
Now, when you initialize the Castro simulation, you need to specify the grid file using amr.regrid file= ”gr0 3d.128 2levels”, for example. You can happily run this now, but note that the regriddingalgorithm will never be called (since Castro thinks it’s started a new simulation from scratch with agrids file, thus disabling the regridding). If you wish for the grid structure to be changed, you mustdo a traditional Castro restart from the Castro-generated checkpoint file (you can still use the same“.MAESTRO” executable or an executable built with USE MAESTRO INIT=FALSE), makingsure that you do not specity amr.regrid file (or else the grids will stay fixed). You are free to spec-ify amr.regrid on restart, amr.compute new dt on regrid, and amr.plotfile on restart.
Sometimes a Maestro plotfile will only have 1 or 2 total levels, but you ultimately want to run aCastro simulation with many more levels of refinement. My recommended strategy is the following:
1. Initialize a Castro simulation from the Maestro plotfile while preserving the exact same gridstructure and run for 10 time steps.
2. Do a traditional Castro restart from chk00010, but do not increase amr.max level, and runfor 10 more time steps. This allows a new grid structure with the same effective resolutionas before settle in using the C++ BoxLib regridding algorithm.
3. Do a traditional Castro restart from chk00020, but increase amr.max level by 1, and runfor 10 time steps.
4. Repeat the procedure from the previous step (using the most updated checkpoint of course)as many times as desired.
CHAPTER 19
Visualization
There are a large number of tools that can be used to read in Castro or BoxLib data and make plots.Here we give a brief overview of some of the tools as well as some examples.
19.1 Controlling What’s in the PlotFile
There are a few options that can be set at runtime to control what variables appear in the plotfile.
• amr.plot vars : this controls which of the main state variables appear in the plotfile. Thedefault is for all of them to be stored. But you can specify a subset by name, e.g.
amr.plot_vars = density
to only store that subset.
• amr.derive plot vars : this controls which of the derived variables to be stored in theplotfile. Derived variables are created only when the plotfile is being created, using theinfrastructure provided by BoxLib to register variables and the associated Fortran routine todo the deriving (Derive nd.F90).
By default, no derived variables are stored. You can store all derived variables that Castroknows about by doing:
amr.derive_plot_vars = ALL
or a subset by explicitly listing them, e.g.,
amr.derive_plot_vars = entropy pressure
161
162 Chapter 19. Visualization
19.2 amrvis
Our favorite visualization tool is amrvis. We heartily encourage you to build the amrvis2d andamrvis3d executables, and to try using them to visualize your data. A very useful feature isView/Dataset, which allows you to actually view the numbers – this can be handy for debugging.You can modify how many levels of data you want to see, whether you want to see the grid boxesor not, what palette you use, etc.
If you like to have amrvis display a certain variable, at a certain scale, when you first bring upeach plotfile (you can always change it once the amrvis window is open), you can modify theamrvis.defaults file in your directory to have amrvis default to these settings every time you run it.The directories CoreCollapse, HSE test, Sod and Sedov have amrvis.defaults files in them. If youare working in a new run directory, simply copy one of these and modify it.
19.3 VisIt
VisIt is also a great visualization tool, and it directly handles our plotfile format (which it callsBoxlib). For more information check out visit.llnl.gov.
[Useful tip:] To use the Boxlib3D plugin, select it from File→ Open file→ Open file as type Boxlib,and then the key is to read the Header file, plt00000/Header, for example, rather than telling to toread plt00000.
19.4 yt
yt is a free and open-source software that provides data analysis and publication-level visualizationtools for astrophysical simulation results such as those CASTRO produces. As yt is script-based,it’s not as easy to use as VisIt, and certainly not as easy as amrvis, but the images can be worthit! Here we do not flesh out yt, but give an overview intended to get a person started. Fulldocumentation and explanations from which this section was adapted can be found at http://yt-project.org/doc/index.html.
yt can be installed by the following commands:
$ wget https://raw.githubusercontent.com/yt-project/yt/master/doc/install script.sh
$ bash install script.sh
This installs yt in your current directory. To update ytin the future, simply do
$ conda update yt
assuming you have conda installed.
19.4.1 Castro-Specific Data
yt was originally created for simple analysis and visualization of data from the Enzo code. Since,it has grown to include support for a variety of codes, including Castro. However, ytwill still
19.4—yt 163
sometimes make assumptions, especially about data field names, that favor Enzo and cause errorswith Castro data. These problems can usually be avoided by taking care to specify the data fieldsdesired in visualization. For example, Enzo’s density field is called “Density,” and is the defaultfor many plotting mechanisms when the user does not specify the field. However, Castro does nothave a field called “Density”; instead, the density field is called “density.” If a user does not specifya field while plotting with Castro data, chances are that yt will try (and fail) to find “Density” andreturn an error. As you will see in the examples, however, there is a way to create your own fieldsfrom existing ones. You can use these derived fields as you would use any other field.
There are also a few imperatives when it comes to reading in your Castro simulation data andassociated information. First and foremost is that the inputs file for the simulation must existin the same directory as where the plotfile directory is located, and it must be named “inputs.”yt reads information from the inputs file such as the number of levels in the simulation run, thenumber of cells, the domain dimensions, and the simulation time. yt will also optionally parse theprobin file for pertinent information if it is similarly included with the name “probin” in the samedirectory as the plotfile of interest. When specifying a plotfile as the data source for plots, you maysimply call it by its directory name, rather than using the Header file as in VisIt. As a final caveat,the existence of the job info file within the plotfile directory is what currently distinguishes Castrodata from MAESTRO data in yt; unless you like surprises, we suggest you ensure your plotfile hasone.
19.4.2 Interacting with yt: Command Line and Scripting
ytis written completely in python (if you don’t have python, yt will install it for you) and thereare a number of different ways to interact with it, including a web-based gui. Here we will covercommand-line yt and scripts/the python interactive prompt, but other methods are outlined on theyt webpage at http://yt-project.org/doc/interacting/index.html.
The first step in starting up yt is to activate the yt environment:
$ source $YT DEST/bin/activate
From the command line you can create simple plots, perform simple volume renderings, print thestatistics of a field for your data set, and do a few other things. Try $ yt to see a list of commands,and $ yt <command> --help to see the details of a command. The command line is the easiestway to get quick, preliminary plots – but the simplicity comes at a price, as yt will make certainassumptions for you. We could plot a projection of density along the x-axis for the plotfile (yt callsit a parameter file) plt def 00020 by doing the following:
$ yt plot -p -a 0 -f density plt def 00020
Or a temperature-based volume rendering with 14 contours:
$ yt render -f Temp --contours 14 plt def 00020
Any plots created from the command line will be saved into a subdirectory called “frames.” Thecommand line is nice for fast visualization without immersing yourself too much in the data, butusually you’ll want to specify and control more details about your plots. This can be done eitherthrough scripts or the python interactive prompt. You can combine the two by running scriptswithin the interactive prompt by the command
164 Chapter 19. Visualization
>>> execfile(‘script.py’)
which will leave you in the interactive prompt, allowing you to explore the data objects you’vecreated in your script and debug errors you may encounter. While in the yt environment, you canaccess the interactive prompt by $ python or the shortcut
$ pyyt
Once you’re in the yt environment and in a .py script or the interactive prompt, there are a coupleof points to know about the general layout of yt scripting. Usually there are five sections to a ytscript:
1. Import modules
2. Load parameter files and saved objects
3. Define variables
4. Create and modify data objects, image arrays, plots, etc. → this is the meat of the script
5. Save images and objects
Note that neither saving nor loading objects is necessary, but can be useful when the creation ofthese objects is time-consuming, which is often the case during identification of clumps or contours.
19.4.3 yt Basics
The first thing you will always want to do is to import yt:
>>> from yt.mods import *
Under certain circumstances you will be required to import more, as we will see in some of theexamples, but this covers most of it, including all of the primary functions and data objects providedby yt. Next, you’ll need yt to access the plotfile you’re interested in analyzing. Remember, youmust have the “inputs” file in the same directory:
>>> ds = load(‘plt def 00020’)
When this line is executed, it will print out some key parameters from the simulation. However,in order to access information about all of the fluid quantities in the simulation, we must usethe “index” object. (Note that for yt versions earlier than 3.0, this information was contained inthe “hierarchy” object; for these versions, replace pf.index with pf. h in the following examples.The “hierarchy” object was removed in yt-3.0 and its associated functionality for accessing datawas moved directly to the datasets themselves.) It contains the geometry of the grid zones, theirparentage relationships, and the fluid states within each one. It is easily created:
>>> ds.index
Upon execution, yt may print out a number of lines saying it’s adding unknown fields to the list offields. This is because Castro has different names for fields than what yt expects. We can see whatfields exist through the commands
>>> print ds.index.field list
>>> print ds.index.derived field list
19.4—yt 165
There may not be any derived fields for Castro data. We can find out the number of grids and cellsat each level, the simulation time, and information about the finest resolution cells:
>>> ds.index.print stats()
The dataset itself also stores a number of associated methods; for example, you can find the valueand location of the maximum of a field in the domain:
>>> value, location = ds.find max(‘density’)
(Note that in yt versions before 3.0, this type of method was primarily associated with the hierarchyobject and was accessed with ds.h.find max.)
The list goes on. A full list of methods and attributes associated with the index object (and mostany yt object or function) can be accessed by the help function:
>>> help(pf.index)
You can also use >>> dir() on an object or function to find out which names it defines. Don’t beshy about searching the yt documentation for help. Note that creating the index object in its ownline is not always needed before calling functions like find max; yt will construct it automaticallyif it does not already exist.
19.4.4 Data Containers and Selection
Sometimes, you’ll want to select, analyze, or plot only portions of your simulation data. To thatend, yt includes a way to create data “containers” that select data based on geometric bounds orfluid quantity values. There are many, including rays, cylinders, and clumps (some in the examples,all described in the documentation), but the easiest to create is a sphere, centered on the locationof the maximum density cell we found above:
>>> my data container = ds.sphere(location, (5.0e4, ‘km’))
Here, specify that the radius is in units of kilometers using a dimensionful quantity. When specifyingdistances in yt, the default is to use the simulation-native unit named “code length”, which forCastro is “cm”, and if you just put in 5.0e4 instead of (5.0e4, ‘km’), you will get a 50,000 cmradius. The pf.index.print stats() command lists available units. We can access the data withinthe container:
>>> print my data container[‘density’]
>>> print my data container.quantities[‘Extrema’]([‘density’, ‘pressure’])
When the creation of objects is time-consuming, it can be convenient to save objects so they canbe used in another session. To save an object as part of the .yt file affiliated with the index:
>>> pf.index.save object(my data container, ‘sphere to analyze later’)
Once it has been saved, it can be easily loaded later:
>>> sphere to analyze = pf.index.load object(‘sphere to analyze later’)
166 Chapter 19. Visualization
19.4.5 Grid Inspection
yt also allows for detailed grid inspection. The index object possesses an array of grids, from whichwe can select and examine specific ones:
>>> print pf.index.grids
>>> my grid = pf.index.grids[4]
Each grid is a data object that carries information about its location, parentage relationships (gridswithin which it resides, and grids that reside within it, at least in part), fluid quantities, and more.Here are some of the commands:
>>> print my grid.Level
>>> print my grid ActiveDimensions
>>> print my grid.LeftEdge
>>> print my grid.RightEdge
>>> print my grid.dds
(dds is the size of each cell within the grid).
>>> print my grid.Parent
>>> print my grid.Children[2].LeftEdge
>>> print my grid[‘Density’]
You can examine which cells within the grid have been refined with the child mask attribute, arepresentative array set to zero everywhere there is finer resolution.To find the fraction of your gridthat isn’t further refined:
>>>print my grid.child mask.sum()/float(my grid.ActiveDimensions.prod())
Rather than go into detail about the many possibilities for plotting in yt, we’ll provide someexamples.
19.4.6 Example Scripts
In these examples, we investigate 3-D simulation data of two stars orbiting in the center of thedomain, which is a box of sides 1010 cm.
# Pressure Contoursfrom yt.mods import *
pf = load(‘plt00020’)
field = ‘pressure’
pf.index
# Most Castro fields have no inherent units, so we add them in, in the form of a raw string# with some LaTeX-style formatting.pf.field info[field]. units = r‘\rmBa’
19.4—yt 167
# SlicePlot parameters include: parameter file, axis, field, window width (effectively the# x and y zoom), and fontsize. We can also create projections with ProjectionPlot().p = SlicePlot(pf, ‘z’, field, width=((5.0e9, ‘cm’), (3.0e9, ‘cm’)),
fontsize=13)
# Zlim is the range of the colorbar. In other words, the range of the data we want to display.# Names for many colormaps can be found at wiki.scipy.org/Cookbook/Matplotlib/Show colormaps.p.set zlim(field, 2.85e13, 2.95e13)
p.set cmap(field, ‘jet’)
# Here we add 5 density contour lines within certain limits on top of the image. We overlay# our finest grids with a transparency of 0.2 (lower is more transparent). We add a quiver# plot with arrows every 16 pixels with x velocity in the x-direction and y velocity in# the y-direction. We also mark the center with an ‘x’ and label one of our stars.p.annotate contour(‘density’, clim=(1.05e-4, 1.16e-4), ncont=5, label=False)
p.annotate grids(alpha=0.2, min level=2)
p.annotate quiver(‘x velocity’, ‘y velocity’, factor=16)
p.annotate marker([5.0e9, 5.0e9], marker=‘x’)
p.annotate point([5.95e9, 5.1e9], ‘Star!’)
# This saves the plot to a file with the given prefix. We can alternatively specify# the entire filename.p.save(‘contours.press den ’)
Figure 19.1: Pressure slice with annotations
#————————
168 Chapter 19. Visualization
# Volume Renderingfrom yt.mods import *
pf = load(‘plt00020’)
field = ‘pressure’ dd = pf.all data()
# We take the log of the extrema of the pressure field, as well as a couple other interesting# value ranges we’d like to visualize.h mi, h ma = dd.quantities[‘Extrema’](field)[0]
h mi, h ma = np.log10(h mi), np.log10(h ma)
s mi, s ma = np.log10(2.90e13), np.log10(3.10e13)
pf.index
# We deal in terms of logarithms here because we have such a large range of values.# It can make things easier, but is not necessary.pf.field info[field].take log=True
# This is what we use to visualize volumes. There are a couple of other, more complex# ways. We set the range of values we’re interested in and the number of bins in the# function. Make sure to have a lot of bins if your data spans many orders of magnitude!# Our raw data ranges from about 1013 to 1022.tf = ColorTransferFunction((h mi-1, h ma+1), nbins=1.0e6)
# Here we add several layers to our function, either one at a time or in groups. We# specify the value-center and width of the layer. We can manipulate the color by# individually setting the colormaps and ranges to spread them over. We can also# change the transparency, which will usually take some time to get perfect.tf.sample colormap(np.log10(2.0e21), 0.006, col bounds=[h mi,h ma],
alpha=[27.0], colormap=‘RdBu r’)
tf.sample colormap(np.log10(2.0e19), 0.001, col bounds=[h mi,h ma],
alpha=[5.5], colormap=‘RdBu r’)
tf.add layers(6, mi=np.log10(2.95e13), ma=s ma,
col bounds=[s mi,s ma],
alpha=19*na.ones(6,dtype=‘float64’), colormap=‘RdBu r’)
tf.sample colormap(np.log10(2.95e13), 0.000005, col bounds=[s mi,s ma],
alpha=[13.0], colormap=‘RdBu r’)
tf.sample colormap(np.log10(2.90e13), 0.000007, col bounds=[s mi,s ma],
alpha=[11.5], colormap=‘RdBu r’)
tf.sample colormap(np.log10(2.85e13), 0.000008, col bounds=[s mi,s ma],
alpha=[9.5], colormap=‘RdBu r’)
# By default each color channel is only opaque to itself. If we set grey opacity=True,# this is no longer the case. This is good to use if we want to obscure the inner
19.4—yt 169
# portions of our rendering. Here it only makes a minor change, as we must set our# alpha values for the outer layers higher to see a strong effect.tf.grey opacity=True
# Volume rendering uses a camera object which centers the view at the coordinates we’ve# called ‘c.’ ‘L’ is the normal vector (automatically normalized) between the camera# position and ‘c,’ and ‘W’ determines the width of the image—again, like a zoom.# ‘Nvec’ is the number of pixels in the x and y directions, so it determines the actual# size of the image.c = [5.0e9, 5.0e9, 5.0e9]
L = [0.15, 1.0, 0.40]
W = (pf.domain right edge - pf.domain left edge)*0.5
Nvec = 768
# ‘no ghost’ is an optimization option that can speed up calculations greatly, but can# also create artifacts at grid edges and affect smoothness. For our data, there is no# speed difference, so we opt for a better-looking image.cam = pf.camera(c, L, W, (Nvec,Nvec), transfer function = tf,
fields=[field], pf=pf, no ghost=False)
# Obtain an image! However, we’ll want to annotate it with some other things before# saving it.im = cam.snapshot()
# Here we draw a box around our stars, and visualize the gridding of the top two levels.# Note that draw grids returns a new image while draw box does not. Also, add# background color in front of draw box is necessary to make the box appear over# blank space (draw grids calls this internally). For draw box we specify the left# (lower) and right(upper) bounds as well its color and transparency.im.add background color(‘black’, inline=True)
cam.draw box(im, np.array([3.0e9, 4.0e9, 4.0e9]),
np.array([7.0e9, 6.0e9, 6.0e9]), np.array([1.0, 1.0, 1.0, 0.14]))
im = cam.draw grids(im, alpha=0.12, min level=2)
im = cam.draw grids(im, alpha=0.03, min level=1, max level=1)
# ‘im’ is an image array rather than a plot object, so we save it using a different# function. There are others, such as ‘write bitmap.’im.write png(‘pressure shell volume.png’)
#————————
# Isocontour Rendering# Here we extract isocontours using some extra modules and plot them using matplotlib.from mpl toolkits.mplot3d import Axes3D
from mpl toolkits.mplot3d.art3d import Poly3DCollection
import matplotlib.pyplot as plt
170 Chapter 19. Visualization
Figure 19.2: Volume rendering
from yt.mods import *
pf = load(‘plt00020’)
field = ‘pressure’
field weight = ‘magvel’
contour value = 2.83e13
domain = pf.all data()
# This object identifies isocontours at a given value for a given field. It returns# the vertices of the triangles in that isocontour. It requires a data source, which# can be an object—but here we just give it all of our data. Here we find a pressure# isocontour and color it the magnitude of velocity over the same contour.surface = pf.surface(domain, field, contour value)
colors = apply colormap(np.log10(surface[field weight]), cmap name=‘RdBu’)
fig = plt.figure()
ax = fig.gca(projection=‘3d’)
p3dc = Poly3DCollection(surface.triangles, linewidth=0.0)
p3dc.set facecolors(colors[0,:,:]/255.)
ax.add collection(p3dc)
# By setting the scaling on the plot to be the same in all directions (using the x scale),# we ensure that no warping or stretching of the data occurs.
19.4—yt 171
ax.auto scale xyz(surface.vertices[0,:], surface.vertices[0,:],
surface.vertices[0,:])
ax.set aspect(1.0)
plt.savefig(‘pres magvel isocontours.png’)
Figure 19.3: Pressure isocontour rendering colored with velocity magnitude
#————————
#1-D and 2-D Profiles# Line plots and phase plots can be useful for analyzing data in detail.from yt.mods import *
pf = load(‘plt00020’)
pf.index
# Just like with the pressure contours script, we can set the units for fields that# have none.pf.field info[‘magvel’]. units = r‘\rmcm/\rms’
pf.field info[‘kineng’]. units = r‘\rmergs’
# We can create new fields from existing ones. ytassumes all units are in cgs, and# does not do any unit conversions on its own (but we can make it). Creating new fields# requires us to define a function that acts on our data and returns the new data,# then call add field while supplying the field name, the function the data comes from,# and the units. Here, we create new fields simply to rename our data to make the plot# look prettier.def newT(field, data):
return data[‘t’]
172 Chapter 19. Visualization
add field(‘X’, function= newT, units=r‘\rmdomain \rmfraction’)
def newDen(field, data):
return data[‘density’]
add field(‘Density’, function= newDen, units=r‘\rmg/\rmcmˆ3’)
# PlotCollections are one of the most commonly used tools in yt, alongside SlicePlots and# ProjectionPlots. They are useful when we want to create multiple plots from the same# parameter file, linked by common characteristics such as the colormap, its bounds, and# the image width. It is easy to create 1-D line plots and 2-D phase plots through a# PlotCollection, but we can also create thin projections and so on. When we create a# PlotCollection, it is empty, and only requires the parameter file and the ’center’ that# will be supplied to plots like slices and sphere plots.pc = PlotCollection(pf, ‘c’)
# Now we add a ray—a sample of our data field along a line between two points we define# in the function call.ray = pc.add ray([0.0, 5.0e9, 5.0e9],[1.e10, 5.0e9, 5.0e9], ‘magvel’)
# This is where our derived fields come in handy. Our ray is drawn along the x-axis# through the center of the domain, but by default the fraction of the ray we have gone# along is called ‘t.’ We now have the same data in another field we called ‘X,’ whose# name makes more sense, so we’ll reassign the ray’s first field to be that. If we wanted,(# we could also reassign names to ‘magvel’ and ‘kineng.’ray.fields = [‘X’, ‘magvel’]
# Next, we’ll create a phase plot. The function requires a data source, and we can’t# just hand it our parameter file, but as a substitute we can quickly create an object# that spans our entire domain (or use the method in the isocontour example). The# specifications of the region (a box) are the center, left bound, and right bound.region = pf.region([5.0e9, 5.0e9, 5.0e9], [0.0, 0.0, 0.0],
[1.0e10, 1.0e10, 1.0e10])
# The phase object accepts a data source, fields, a weight, a number of bins along both# axes, and several other things, including its own colormap, logarithm options,# normalization options, and an accumulation option. The first field is binned onto# the x-axis, the second field is binned onto the y-axis, and the third field is# binned with the colormap onto the other two. Subsequent fields go into an underlying# profile and do not appear on the image.phase = pc.add phase object(region, [‘Density’, ‘magvel’,‘kineng’], weight=None,
x bins=288, y bins=288)
pc.save(‘profile’)
#————————
#Off-Axis Projection# If we don’t want to take a projection (this can be done for a slice as well) along# one of the coordinate axes, we can take one from any direction using an# OffAxisProjectionPlot. To accomplish the task of setting the view up, the plot
19.4—yt 173
Figure 19.4: 1-D velocity magnitude profile
Figure 19.5: Density/velocity magnitude/kinetic energy phase plot
# requires some of the same parameters as the camera object: a normal vector, center,# width, and field, and optionally we can set no ghost (default is False). The normal# vector is automatically normalized as in the case of the camera. The plot also# requires a depth—that is, how much data we want to sample along the line of sight,# centered around the center. In this case ‘c’ is a shortcut for the domain center.pf = load(‘plt00020’)
field = ‘density’
174 Chapter 19. Visualization
L = [0.25, 0.9, 0.40]
plot = OffAxisProjectionPlot(pf, L, field, center=‘c’,
width=(5.0e9, 4.0e9), depth=3.0e9)
# Here we customize our newly created plot, dictating the font, colormap, and title.# Logarithmic data is used by default for this plot, so we turn it off.plot.set font(‘family’:‘Bitstream Vera Sans’, ‘style’:‘italic’,
‘weight’:‘normal’, ‘size’:14, ‘color’:‘red’)
plot.set log(field, False)
plot.set cmap(field, ‘jet’)
plot.annotate title(‘Off-Axis Density Projection’)
# The actual size of the image can also be set. Note that the units are in inches.plot.set window size(8.0)
plot.save(‘off axis density’)
Figure 19.6: Off-axis density projection
CHAPTER 20
Verification Test Problems
20.1 Hydrodynamics Test Problems
20.1.1 Sod’s Problem (and Other Shock Tube Problems)
The Exec/hydro tests/Sod problem directory sets up a general one-dimensional shock tube. Theleft and right primitive-variable states are specified and the solution evolves until a user-specifiedend time. For a simple discontinuity, the exact solution can be found from an exact Riemann solver.For this problem, the exact solutions were computed with the exact Riemann solver from Toro [26],Chapter 4.
20.1.1.1 Sod’s Problem
The Sod problem [24] is a simple shock tube problem that exhibits a shock, contact discontinuity,and a rarefaction wave. The initial conditions are:
ρL = 1uL = 0pL = 1
ρR = 0.125uR = 0pR = 0.1
(20.1)
The gamma law equation of state is used with γ = 1.4. The system is evolved until t = 0.2 s. Setupsfor 1-, 2-, and 3-d are provided. The following inputs files and probin files setup the Sod’s problem:
For multi-dimensional runs, the directions transverse to the jump are kept constant. We use a CFLnumber of 0.9, an initial timestep shrink (castro.init shrink) of 0.1, and the maximum factorby which the timestep can increase (castro.change max) of 1.05.
175
176 Chapter 20. Verification Test Problems
inputs file probin file description
inputs-sod-x probin-sod-x Sod’s problem along x-directioninputs-sod-y probin-sod-y Sod’s problem along y-directioninputs-sod-z probin-sod-z Sod’s problem along z-direction
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
de
nsity
x/y/z
xyz
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
ve
locity
x/y/z
xyz
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
pre
ssu
re
x/y/z
xyz
1.6
1.8
2
2.2
2.4
2.6
2.8
3
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
inte
rna
l e
ne
rgy
x/y/z
xyz
Figure 20.1: Castro solution for Sod’s problem run in 3-d, with the newest ppm limiters, alongthe x, y, and z axes. A coarse grid of 32 zones in the direction of propagation, with 2 levels ofrefinement was used. The analytic solution appears as the red line.
Figure 20.1 shows the Castro solution using the newest PPM limiters compared to the analyticsolution, showing the density, velocity, pressure, and internal energy. Figure 20.2 is the same asFigure 20.1, but with the piecewise-linear Godunov method with limiters, shown for comparison.
The Verification subdirectory includes the analytic solution for the Sod problem sod-exact.out,with γ = 1.4. 1-d slices can be extracted from the Castro plotfile using the fextract tool fromBoxLib/Tools/Postprocessing/F Src/. The steps to generate this verification plot with Castroare:
1. in Exec/hydro tests/Sod, build the Castro executable in 3-d
2. run the Sod problem with Castro in the x, y, and z directions:./Castro3d.Linux.Intel.Intel.ex inputs-sod-x
./Castro3d.Linux.Intel.Intel.ex inputs-sod-y
./Castro3d.Linux.Intel.Intel.ex inputs-sod-z
3. build the fextract tool in BoxLib/Tools/Postprocessing/F Src/.
4. run fextract on the Castro output to generate 1-d slices through the output:fextract3d.Linux.Intel.exe -d 1 -s sodx.out -p sod x plt00034
fextract3d.Linux.Intel.exe -d 2 -s sody.out -p sod y plt00034
fextract3d.Linux.Intel.exe -d 3 -s sodz.out -p sod z plt00034
20.1—Hydrodynamics Test Problems 177
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
de
nsity
x/y/z
xyz
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
ve
locity
x/y/z
xyz
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
pre
ssu
re
x/y/z
xyz
1.6
1.8
2
2.2
2.4
2.6
2.8
3
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
inte
rna
l e
ne
rgy
x/y/z
xyz
Figure 20.2: Castro solution for Sod’s problem run in 3-d, with the piecewise-linear Godunovmethod with limiters, along the x, y, and z axes. A coarse grid of 32 zones in the direction ofpropagation, with 2 levels of refinement was used. The analytic solution appears as the red line.
5. copy the sodx/y/z.out files into the Verification directory.
6. in Verification run the gnuplot script sod 3d.gp as:gnuplot sod 3d.gp
This will produce the figure sod 3d.eps.
20.1.1.2 Double Rarefaction
The double rarefaction is the “Test 2” problem described by Toro [26], Chapter 6. In this test, thecenter of the domain is evacuated as two rarefaction waves propagate in each direction, outwardfrom the center. It is difficult to get the internal energy to behave at the center of the domainbecause we are creating a vacuum. The initial conditions are:
ρL = 1uL = −2pL = 0.4
ρR = 1uR = 2pR = 0.4
(20.2)
The gamma law equation of state is used with γ = 1.4. The system is evolved until t = 0.15 s.Setups for 1-, 2-, and 3-d are provided. The following inputs files and probin files setup the Sod’sproblem:
inputs file probin file description
inputs-test2-x probin-test2-x Double rarefaction problem along x-directioninputs-test2-y probin-test2-y Double rarefaction problem along y-directioninputs-test2-z probin-test2-z Double rarefaction problem along z-direction
178 Chapter 20. Verification Test Problems
We use a CFL number of 0.8, an initial timestep shrink (castro.init shrink) of 0.1, and themaximum factor by which the timestep can increase (castro.change max) of 1.05. The PPMsolver with the new limiters are used.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
de
nsity
x/y/z
xyz
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
ve
locity
x/y/z
xyz
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
pre
ssu
re
x/y/z
xyz
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
inte
rna
l e
ne
rgy
x/y/z
xyz
Figure 20.3: Castro solution for the double rarefaction problem run in 3-d, along the x, y, andz axes. A coarse grid of 32 zones in the direction of propagation, with 2 levels of refinement wasused. The analytic solution appears as the red line.
Figure 20.3 shows the Castro output, run along all 3 coordinate axes in 3-d, compared to theanalytic solution.
The comparison to the analytic solution follows the same procedure as described for the Sod’sproblem above. The gnuplot script test2 3d.gp will generate the figure, from the 1-d slices createdby fextract named test2x.out, test2y.out, and test2z.out.
20.1.1.3 Strong Shock
The strong shock test is the “Test 3” problem described by Toro [26], Chapter 6. In this test, alarge pressure jump at the initial interface creates a very strong rightward moving shock, followedvery closely by a contact discontinuity. The initial conditions are:
ρL = 1uL = 0pL = 1000
ρR = 1uR = 0pR = 0.01
(20.3)
The gamma law equation of state is used with γ = 1.4. The system is evolved until t = 0.012 s.Setups for 1-, 2-, and 3-d are provided. The following inputs files and probin files setup the Sod’sproblem:
20.1—Hydrodynamics Test Problems 179
inputs file probin file description
inputs-test3-x probin-test3-x Strong shock problem along x-directioninputs-test3-y probin-test3-y Strong shock problem along y-directioninputs-test3-z probin-test3-z Strong shock problem along z-direction
0
1
2
3
4
5
6
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
de
nsity
x/y/z
xyz
0
5
10
15
20
25
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
ve
locity
x/y/z
xyz
0
100
200
300
400
500
600
700
800
900
1000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
pre
ssu
re
x/y/z
xyz
0
500
1000
1500
2000
2500
3000
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
inte
rna
l e
ne
rgy
x/y/z
xyz
Figure 20.4: Castro solution for the strong shock problem run in 3-d, along the x, y, and z axes.A coarse grid of 32 zones in the direction of propagation, with 2 levels of refinement was used. Theanalytic solution appears as the red line.
We use a CFL number of 0.9, an initial timestep shrink (castro.init shrink) of 0.1, and themaximum factor by which the timestep can increase (castro.change max) of 1.05. The PPMsolver with the new limiters are used.
Figure 20.4 shows the Castro output, run along all 3 coordinate axes in 3-d, compared to theanalytic solution.
The comparison to the analytic solution follows the same procedure as described for the Sod’sproblem above. The gnuplot script test3 3d.gp will generate the figure, from the 1-d slices createdby fextract named test3x.out, test3y.out, and test3z.out.
20.1.2 Sedov Problem
The Sedov (or Sedov-Taylor) blast wave is a standard hydrodynamics test problem. A large amountof energy is placed into a very small volume, driving a spherical (or cylindrical in 2-d Cartesiancoordinates) blast wave. Analytic solutions were found by Sedov [23].
A cylindrical blast wave (e.g. a point explosion in a 2-d plane) can be modeled in 2-d Cartesiancoordinates. A spherical blast wave can be modeled in 1-d spherical, 2-d axisymmetric (cylindrical
180 Chapter 20. Verification Test Problems
r-z), or 3-d Cartesian coordinates. This provides a good test on the geometric factors in thehydrodynamics solver. We use a publically available code, sedov3.f [15], to generate the analyticsolutions.
The Castro implementation of the Sedov problem is in Exec/hydro tests/Sedov. A number ofdifferent inputs/probin files are provided, corresponding to different Sedov/Castro geometries. Themain ones are:
inputs file probin file descriptioninputs.1d.sph probin.1d.sph Spherical Sedov explosion modeled in
1-d spherical coordinates
inputs.2d.sph in cylcoords probin.2d.sph in cylcoords Spherical Sedov explosion modeledin 2-d cylindrical (axisymmetric)coordinates
inputs.2d.cyl in cartcoords probin.2d.cyl in cartcoords Cylindrical Sedov explosion modeledin 2-d Cartesian coordinates
inputs.3d.sph probin.3d.sph Spherical Sedov explosion modeled in3-d Cartesian coordinates
Table 20.1: Sedov inputs files
In the Sedov problem, the explosion energy, Eexp (in units of energy, not energy/mass or ener-gy/volume) is to be deposited into a single point, in a medium of uniform ambient density, ρambient,and pressure, pambient. Initializing the problem can be difficult because the small volume is typicallyonly a cell in extent. This can lead to grid imprinting in the solution. A standard solution (see forexample [20] and the references therein) is to convert the explosion energy into a pressure containedwithin a certain volume, Vinit, of radius rinit as
pinit =(γ − 1)Eexp
Vinit. (20.4)
This pressure is then deposited in all of the cells where r < rinit.
To further minimize any grid effects, we do subsampling in each zone: each zone is divided it intoNsub subzones in each coordinate direction, each subzone is initialized independently, and then thesubzones are averaged together (using a volume weighting for spherical or cylindrical/axisymmetricCastro grids) to determine the initial state of the full zone.
For these runs, we use ρambient = 1, pambient = 10−5, Eexp = 1, rinit = 0.01, and Nsub = 10. A basegrid with 32 zones in each coordinate direction plus 3 levels of refinement is used (the finest meshwould coorespond to 256 zones in a coordinate direction). The domain runs from 0 to 1 in eachcoordinate direction.
Analysis routines for the Sedov problem are provided in Castro/Diagnostics/Sedov/. Theseroutines will average the Castro solution over angles, using the proper geometric weighting, toproduce an average profile as a function of radius. The following routines correspond to the inputsfiles described above:
20.1—Hydrodynamics Test Problems 181
inputs file analysis routineinputs.1d.sph fsedov1d.f90
inputs.2d.sph in cylcoords fsedov2d sph in cylcoords.f90
inputs.2d.cyl in cartcoords fsedov2d cyl in cartcoords.f90
inputs.3d.sph fsedov3d sph.f90
Table 20.2: Analysis routines for Sedov
20.1.2.1 Spherical Blast Wave
A spherical Sedov explosion can be modeled in 1-d spherical, 2-d cylindrical (axisymmetric), or 3-dCartesian coordinates, using the inputs files described in Table 20.1. A 1-d radial profile can beextracted using the appropriate fsedov routine, as listed in Table 20.2. For example, to run andprocess the 2-d cylindrical Sedov explosion, one would do:
1. in Exec/hydro tests/Sedov, build the Castro executable in 2-d
2. run the spherical Sedov problem with Castro in 2-d cylindrical coordinates:./Castro2d.Linux.Intel.Intel.ex inputs.2d.sph in cylcoords
3. build the fsedov2d sph in cylcoords tool in Castro/Diagnostics/Sedov.
4. run fsedov2d sph in cylcoords on the Castro output to generate 1-d radial profiles:fsedov2d sph in cylcoords.Linux.Intel.exe -s sedov 2d sph in cyl.out \
-p sedov 2d sph in cyl plt00246
A similar procedure can be used for the 1-d and 3-d spherical Sedov explosions (with theoutput named sedov 1d sph.out and sedov 3d sph.out respectively). Once this is done, thesedov sph.gp gnuplot script can be used to make a plot comparing the 3 solutions to the analyticsolution, spherical sedov.dat.
Figure 20.5 shows the comparison of the 3 Castro spherical Sedov explosion simulations to theanalytic solution.
20.1.2.2 Cylindrical Blast Wave
20.1.3 Rayleigh-Taylor
2D. Domain size 0.5 by 1.0. 256 by 512 cells, single level calculation. Periodic in x, solid walls ontop and bottom in y. Gamma law gas with γ = 1.4, no reactions. Zero initial velocity. Constant|g| = 1. The density profile is essentially ρ = 1 on bottom, ρ = 2 on top, but with a perturbation.A single-mode perturbation is constructed as:
y(x) = 0.5 + 0.01cos(4πx) + cos(4π(Lx − x))
2(20.5)
We note that the symmetric form of the cosine is done to ensure that roundoff error does notintroduce a left-right asymmetry in the problem. Without this construction, the R-T instabilitywill lose its symmetry as it evolves. This then applied to the interface with a tanh profile to smooththe transition between the high and low density material:
ρ(x, y) = 1 + 0.5
[1 + tanh
(y − y(x)
0.005
)](20.6)
182 Chapter 20. Verification Test Problems
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0 0.05 0.1 0.15 0.2 0.25 0.3
de
nsity
r
1-d2-d3-d
0
1
2
3
4
5
6
0 0.05 0.1 0.15 0.2 0.25 0.3
ve
locity
r
1-d2-d3-d
0
5
10
15
20
25
30
0 0.05 0.1 0.15 0.2 0.25 0.3
pre
ssu
re
r
1-d2-d3-d
10-6
10-4
10-2
100
102
104
106
108
1010
1012
1014
0 0.05 0.1 0.15 0.2 0.25 0.3in
tern
al e
ne
rgy
r
1-d2-d3-d
Figure 20.5: Castro solution for the Sedov blast wave problem run in 1-d spherical, 2-d axisymmet-ric, and 3-d Cartesian coordinates. Each of these geometries produces a spherical Sedov explosion.
0
1
2
3
4
5
6
0 0.1 0.2 0.3 0.4 0.5
de
nsity
r
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0 0.1 0.2 0.3 0.4 0.5
ve
locity
r
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
0 0.1 0.2 0.3 0.4 0.5
pre
ssu
re
r
10-6
10-4
10-2
100
102
104
106
108
1010
0 0.1 0.2 0.3 0.4 0.5
inte
rna
l e
ne
rgy
r
Figure 20.6: Castro solution for the Sedov blast wave problem run in 2-d Cartesian coordinates.This corresponds to a cylindrical Sedov explosion.
Hydrostatic pressure with p = 5.0 at bottom of domain, assuming ρ = 1 on the lower half of thedomain, and ρ = 2 on the upper half and no density perturbation. We run to t = 2.5 with piecewise
20.2—Gravity Test Problems 183
linear, old PPM, and new PPM. CFL=0.9. See Figure 20.7.
Figure 20.7: Rayleigh-Taylor with different PPM types.
20.2 Gravity Test Problems
20.3 Radiation Test Problems
There are two photon radiation solvers in Castro—a gray solver and a multigroup solver. The graysolver follows the algorithm outlined in [13]. We use the notation described in that paper. Inparticular, the radiation energy equation takes the form of:
∂ER∂t
= ∇ ·(cλ(ER)
κR∇ER
)+ κP (4σT 4 − cER) (20.7)
Here, ER is the radiation energy density, κR is the Roseland-mean opacity, κP is the Planck-meanopaciy, and λ is a quantity ≤ 1/3 that is subjected to limiting to keep the radiation field causal.Castro allows for κR and κP to be set independently as power-laws.
184 Chapter 20. Verification Test Problems
20.3.1 Light Front
The light front problem tests the ability of the radiation solver to operate in the free-streaminglimit. A radiation front is estabilished by initializing one end of the computational domain witha finite radiation field, and zero radiation field everywhere else. The speed of propagation of theradiation front is keep in check by the flux-limiters, to prevent it from exceeding c.
20.3.2 Diffusion of a Gaussian Pulse
The diffusion of a Gaussian pulse problem tests the diffusion term in the radiation energy equation.The radiation energy density is initialized at time t = t0 to a Gaussian distribution:
ER = (ER)0 exp
− 1
4Dt0|r − r0|2
. (20.8)
As the radiation diffuses, the overall distribution will remain Gaussian, with the time-dependentsolution of:
ER = (ER)0t0
t0 + texp
− 1
4D(t0 + t)|r − r0|2
(20.9)
20.3.3 Radiation Source Problem
The radiation source problem tests the coupling between the radiation field and the gas energythrough the radiation source term. The problem begins with the radiation field and gas temperatureout of equilibrium. If the gas is too cool, then the radiation field will heat it. If the gas is toohot, then it will radiate and cool. In each case, the gas energy and radiation field will evolve untilthermal equilibrium is achieved.
Our implementation of this problem follows that of [25].
20.3.4 Radiating Sphere
The radiating sphere (RadSphere) is a multigroup radiation test problem. A hot sphere is centeredat the origin in a spherical geometry. The spectrum from this sphere follows a Planck distribution.The ambient medium is at a much lower temperature. A frequency-dependent opacity makes thedomain optically thin for high frequecies and optically thick for low frequency. At long times,the solution will be a combination of the blackbody radiation from the ambient medium plus theradiation that propagated from the hot sphere. An analytic solution exists [12] which gives theradiation energy as a function of energy group at a specified time and distance from the radiatingsphere.
Our implementation of this problem is in Exec/radiation tests/RadSphere and follows that of[25]. The routine that computes the analytic solution is provided as analytic.f90.
20.3—Radiation Test Problems 185
102
103
104
105
106
107
108
109
10-16
10-15
10-14
10-13
10-12
10-11
10-10
10-9
10-8
10-7
10-6
10-5
ρe
(e
rg/c
m3)
time (s)
analytic heating solutionCastro heating solution
analytic cooling solutionCastro cooling solution
Figure 20.8: Castro solution for radiating source test problem. Heating and cooling solutions areshown as a function of time, compared to the analytic solution. The gray photon solver was used.
1.0 x 10-90
1.0 x 10-80
1.0 x 10-70
1.0 x 10-60
1.0 x 10-50
1.0 x 10-40
1.0 x 10-30
1.0 x 10-20
1.0 x 10-10
1.0 x 100
1.0 x 1014
1.0 x 1015
1.0 x 1016
1.0 x 1017
1.0 x 1018
1.0 x 1019
1.0 x 1020
Era
d (
erg
/cm
3)
nu (Hz)
analytic solutionCastro solution
Figure 20.9: Castro solution for radiating sphere problem, showing the radiation energy densityas a function of energy group. This test was run with 64 photon energy groups.
186 Chapter 20. Verification Test Problems
20.4 Regression Testing
An automated regression test suite for Castro (or any BoxLib-based code) written in Python existsin BoxLib/Tools/RegressionTesting. Details of its use are provided in the BoxLib User’s Guide.
References
[1] A. S. Almgren, V. E. Beckner, J. B. Bell, M. S. Day, L. H. Howell, C. C. Joggerst, M. J.Lijewski, A. Nonaka, M. Singer, and M. Zingale. CASTRO: A New Compressible AstrophysicalSolver. I. Hydrodynamics and Self-gravity. ApJ, 715:1221–1238, June 2010.
[2] A. S. Almgren, J. B. Bell, A. Nonaka, and M. Zingale. Low Mach Number Modeling of TypeIa Supernovae. III. Reactions. ApJ, 684:449–470, September 2008.
[3] Marsha Berger and Isidore Rigoutsos. An algorithm for point clustering and grid generation.Systems, Man and Cybernetics, IEEE Transactions on, 21(5):1278–1286, 1991.
[4] G. L. Bryan, M. L. Norman, B. W. O’Shea, T. Abel, J. H. Wise, M. J. Turk, D. R. Reynolds,D. C. Collins, P. Wang, S. W. Skillman, B. Smith, R. P. Harkness, J. Bordner, J.-h. Kim,M. Kuhlen, H. Xu, N. Goldbaum, C. Hummels, A. G. Kritsuk, E. Tasker, S. Skory, C. M.Simpson, O. Hahn, J. S. Oishi, G. C. So, F. Zhao, R. Cen, Y. Li, and Enzo Collaboration.ENZO: An Adaptive Mesh Refinement Code for Astrophysics. ApJS, 211:19, April 2014.
[5] G. L. Bryan, M. L. Norman, J. M. Stone, R. Cen, and J. P. Ostriker. A piecewise parabolicmethod for cosmological hydrodynamics. Computer Physics Communications, 89:149–168,August 1995.
[6] P. Colella. Multidimensional upwind methods for hyperbolic conservation laws. Journal ofComputational Physics, 87:171–200, March 1990.
[7] P. Colella and H. M. Glaz. Efficient solution algorithms for the Riemann problem for realgases. Journal of Computational Physics, 59:264–289, June 1985.
[8] P. Colella and M. D. Sekora. A limiter for PPM that preserves accuracy at smooth extrema.Journal of Computational Physics, 227:7069–7076, July 2008.
[9] P. Colella and P. R. Woodward. The Piecewise Parabolic Method (PPM) for Gas-DynamicalSimulations. Journal of Computational Physics, 54:174–201, September 1984.
[10] S. A. Colgate and R. H. White. The Hydrodynamic Behavior of Supernovae Explosions. ApJ,143:626, March 1966.
187
188 REFERENCES
[11] B. Fryxell, K. Olson, P. Ricker, F. X. Timmes, M. Zingale, D. Q. Lamb, P. MacNeice, R. Ros-ner, J. W. Truran, and H. Tufo. FLASH: An Adaptive Mesh Hydrodynamics Code for ModelingAstrophysical Thermonuclear Flashes. ApJS, 131:273–334, November 2000.
[12] F. Graziani. The prompt spectrum of a radiating sphere: Benchmark solutions for diffusionand transport. In F. Graziani, editor, Computational Methods in Transport: Verification andValidation, number 62 in Lecture Notes in Computational Science and Engineering, pages151–167. Springer, 2008.
[13] L. H. Howell and J. A. Greenough. Radiation diffusion for multi-fluid Eulerian hydrodynamicswith adaptive mesh refinement. Journal of Computational Physics, 184:53–78, January 2003.
[14] X. Y. Hu, N. A. Adams, and C.-W. Shu. Positivity-preserving method for high-order con-servative schemes solving compressible Euler equa tions. Journal of Computational Physics,242:169–180, June 2013.
[15] J. R. Kamm and F. X. Timmes. submitted to ApJ supplement, May 2007, seehttp://cococubed.asu.edu/code pages/sedov.shtml.
[16] M. P. Katz. White Dwarf Mergers on Adaptive Meshes. PhD thesis, State University of NewYork at Stony Brook, 2016.
[17] M. P. Katz, M. Zingale, A. C. Calder, F. D. Swesty, A. S. Almgren, and W. Zhang. WhiteDwarf Mergers on Adaptive Meshes. I. Methodology and Code Verification. APJ, 819:94,March 2016.
[18] R. Kippenhahn and A. Weigert. Stellar Structure and Evolution. Springer, 1990.
[19] G. H. Miller and P. Colella. A Conservative Three-Dimensional Eulerian Method for CoupledSolid-Fluid Shock Capturing. Journal of Computational Physics, 183:26–82, November 2002.
[20] M. Omang, S. Børve, and J. Trulsen. SPH in spherical and cylindrical coordinates. Journalof Computational Physics, 213:391–412, March 2006.
[21] Richard B. Pember, Jeffrey A. Greenough, and Phillip Colella. An adaptive, higher-ordergodunov method for gas dynamics in three-dimensional orthogonal curvilinear coordinates.Technical report, 1996.
[22] James J Quirk. A contribution to the great Riemann solver debate. Springer, 1997.
[23] L. I. Sedov. Similarity and Dimensional Methods in Mechanics. Academic Press, 1959. trans-lated from the 4th Russian Ed.
[24] G. A. Sod. A survey of several finite difference methods for systems of nonlinear hyperbolicconservation laws. Journal of Computational Physics, 27:1–31, April 1978.
[25] F. D. Swesty and E. S. Myra. A Numerical Algorithm for Modeling Multigroup Neutrino-Radiation Hydrodynamics in Two Spatial Dimensions. Astrophysical Journal Supplement,181:1–52, March 2009.
[26] E. F. Toro. Riemann Solvers and Numerical Methods for Fluid Dynamics. Springer, 2ndedition, 1997.
[27] Y. B. Zeldovich and D. Novikov. Relativistic astrophysics. Vol.1: Stars and relativity. Univer-sity of Chicago Press, 1971.
REFERENCES 189
[28] M. Zingale and M. P. Katz. On the Piecewise Parabolic Method for Compressible Flow WithStellar Equations of State. ApJS, 216:31, February 2015.
Index
probin, 9
inputs, 9
probin, 144
BLAS, 80
Code reference
&tagging, 144
cpp parameters, 59
parameters, 38, 59
AmrLevel, 23
BoxArray, 24
Box, 23
FArrayBox, 24
FillPatchIterator, 35
MFIter, 27
MultiFab, 24
StateData, 25
ARLIM 3D, 28
ArrayLim.H, 27, 28
bc fill ?d.F90, 37, 40
BC TYPES.H, 36
bl constants module, 38
bl fort module, 23
BL FORT PROC CALL, 27
BL TO FORTRAN, 27
bl types, 38
ca auxupdate, 47
ca denerror, 143
ca estdt temp diffusion, 113
ca initdata(), 40
ca presserror, 143
ca raderror, 144
ca temperror, 143
ca velerror, 144
Castro::advance(), 46
Castro::expand state(), 34
Castro::initialize do advance(), 34
Castro advance.cpp, 46
Castro nd.F90, 25
Castro setup.cpp, 25
clean state, 45, 46, 49
clean state(), 45, 51, 54, 56
construct hydro source(), 50, 54, 56
construct new gravity(), 51, 56
construct old gravity(), 49, 54, 56
Derive nd.F90, 161
do advance, 46
do advance(), 48
do advance mol, 46
do new sources, 51, 56
do old sources(), 49, 54, 56
dump and continue, 82
dump and stop, 82
ext src ?d.f90, 50
extern probin module, 38, 59
final radiation call(), 47
finalize do advance(), 52, 54, 56
findparams.py, 59
hypfill, 37
initialize advance(), 46
initialize do advance(), 49, 53, 56
k mol, 46
meth params module, 25, 60
mk params.sh, 59
plot and continue, 82
Prob.cpp, 43
Prob ?d.f90, 39
Prob nd.F90, 43
191
192 INDEX
probdata.f90, 39
probdata module, 9
Problem.f90, 42
Problem.H, 43
problem checkpoint, 42
Problem Derive F.H, 42
problem derive nd.f90, 42
Problem Derives.H, 42
Problem F.H, 42, 43
problem restart, 42
problem tagging *d.f90, 145
problem tagging ?d.F90, 42
problem tagging nd.F90, 42
Sburn, 46
set problem tags, 42, 145
stop run, 82
strang react first half(), 49, 52, 54
strang react second half(), 52, 54
sum integrated quantities, 43
sum of sources(), 56
tagging module, 10
Tagging nd.f90, 143
thermal conductivity, 114
write probin.py, 59
Makefiles
BLAS LIBRARY, 80
COMP, 5
DEBUG, 5
DIMENSION AGNOSTIC, 43
DIM, 4
PRECISION, 23
USE MPI, 5
USE RAD, 124
USE SYSTEM BLAS, 80
multigrid, 13
plotfiles
derived variables, 43
Preprocessor
AUX UPDATE, 47
DIFFUSION, 16, 50
GRAVITY, 50
HYBRID MOMENTUM, 50, 86
PARTICLES, 47
POINTMASS, 47
RADIATION, 47, 85--87, 124
REACTIONS, 16
ROTATION, 50
SDC, 45
Problem setups
Noh, 43
RadSphere, 184
Sedov, 179
Sod stellar, 100
Sod, 175
reacting bubble, 43
rotating torus, 43
toy convect, 37, 42, 50
toy flame, 43
wdmerger, 42
Runtime parameters
gravity.rel tol, 105
amr.blocking factor, 13
amr.check file, 17
amr.check int, 17
amr.check nfiles, 17
amr.check per, 17
amr.checkpoint files output, 17
amr.checkpoint nfiles, 43
amr.checkpoint on restart, 17
amr.derive plot vars, 18, 161
amr.grid eff, 13
amr.grid log, 19
amr.max grid size, 12
amr.max level, 11
amr.message int, 82
amr.n cell, 11
amr.n error buf, 12
amr.plot files output, 18
amr.plot file, 18
amr.plot int, 18
amr.plot nfiles, 18, 43
amr.plot per, 18
amr.plot vars, 18, 161
amr.plotfile on restart, 18
amr.ref ratio, 11
amr.refine grid layout, 13
amr.regrid file, 12
amr.regrid int, 11
amr.regrid on restart, 11
amr.restart, 17
amr.run log terse, 19
amr.run log, 19
amr.subcycling iterations, 16
INDEX 193
amr.subcycling mode, 16
amr.sum interval, 19
amr.v, 19
castro.add ext src, 64, 92
castro.allow negative energy, 92
castro.allow small energy, 64
castro.bndry func thread safe, 67
castro.center, 39
castro.cfl, 15, 69
castro.cg blend, 64, 99
castro.cg maxiter, 64, 99
castro.cg tol, 64, 99
castro.change max, 15, 69
castro.clamp subcycles, 69
castro.density reset method, 64
castro.diffuse cond scale fac, 62
castro.diffuse cutoff density, 62, 114
castro.diffuse enth, 62
castro.diffuse spec, 62
castro.diffuse temp, 62, 113
castro.diffuse vel, 62
castro.difmag, 64
castro.disable shock burning, 68
castro.do acc, 67
castro.do ctu, 45, 51, 65
castro.do grav, 63, 105
castro.do hydro, 65, 92
castro.do react, 68
castro.do reflux, 61
castro.do rotation, 63, 118
castro.do special tagging, 20, 69
castro.do sponge, 65, 92
castro.do tracer particles, 67
castro.dt cutoff, 15, 69
castro.dtnuc X threshold, 16, 68
castro.dtnuc X, 16, 68
castro.dtnuc e, 16, 68
castro.dual energy eta1, 65, 90
castro.dual energy eta2, 65, 90
castro.dual energy eta3, 90
castro.dual energy update E from e, 90
castro.dxnuc max, 68
castro.dxnuc, 68
castro.first order hydro, 65
castro.fix mass flux, 65, 92
castro.fixed dt, 15, 69
castro.fourth order, 65
castro.grav source type, 50, 63
castro.grown factor, 17, 63
castro.hard cfl limit, 15, 61
castro.hi bc, 36
castro.hse interp temp, 65
castro.hse reflect vels, 65
castro.hse zero vels, 65
castro.hybrid hydro, 65
castro.hybrid riemann, 65
castro.implicit rotation update, 63, 118
castro.init shrink, 15, 69
castro.initial dt, 15, 69
castro.job name, 61
castro.limit fluxes on small dens, 65,
100
castro.lin limit state interp, 61
castro.lo bc, 36
castro.max dt, 69
castro.max dxnuc lev, 68
castro.max subcycles, 69
castro.mol order, 65
castro.moving center, 63, 105
castro.normalize species, 92
castro.output at completion, 61
castro.plm iorder, 65
castro.plot X, 18
castro.plot per is exact, 69
castro.point mass fix solution, 47, 63
castro.point mass, 63, 105
castro.ppm predict gammae, 66, 96
castro.ppm reference eigenvectors, 66,
96
castro.ppm temp fix, 66, 96, 99
castro.ppm type, 66, 93, 96
castro.print fortran warnings, 61
castro.print update diagnostics, 61
castro.react T max, 68
castro.react T min, 68
castro.react rho max, 68
castro.react rho min, 68
castro.reset checkpoint step, 61
castro.reset checkpoint time, 62
castro.retry neg dens factor, 69
castro.retry subcycle factor, 70
castro.retry tolerance, 70
castro.riemann solver, 66, 98
castro.rot axis, 63, 118
castro.rot source type, 50, 63, 118, 121
194 INDEX
castro.rotation include centrifugal, 63,
118
castro.rotation include coriolis, 63, 118
castro.rotation include domegadt, 63, 118
castro.rotational dPdt, 64, 118
castro.rotational period, 64, 117, 118
castro.sdc iters, 70
castro.show center of mass, 62, 92
castro.small dens, 45, 66, 92, 101
castro.small ener, 66
castro.small plot per is exact, 70
castro.small pres, 66, 92
castro.small temp, 66, 92
castro.source term predictor, 51, 56, 66
castro.spherical star, 47, 69, 92
castro.sponge implicit, 66
castro.star at center, 63
castro.state in rotating frame, 64, 118
castro.state interp order, 61
castro.state nghost, 25, 61
castro.sum interval, 62
castro.sum per, 62
castro.track grid losses, 47, 62
castro.transverse reset density, 66, 97
castro.transverse reset rhoe, 66, 97
castro.transverse use eos, 66, 97
castro.update sources after reflux, 61
castro.use custom knapsack weights, 61
castro.use eos in riemann, 66
castro.use flattening, 66, 93
castro.use point mass, 64
castro.use post step regrid, 70
castro.use pslope, 67
castro.use retry, 46, 70
castro.v, 19
castro.xl ext bc type, 67
castro.xr ext bc type, 67
castro.yl ext bc type, 67
castro.yr ext bc type, 67
castro.zl ext bc type, 67
castro.zr ext bc type, 67
denerr, 144
dengrad, 144
diffusion.mlmg maxorder, 72
diffusion.v, 19, 72
fabarray.mfiter tile size, 31
geometry.coord sys, 10
geometry.is periodic, 10
geometry.prob hi, 10
geometry.prob lo, 10
gravity.abs tol, 105
gravity.const grav, 74, 105
gravity.direct sum bcs, 74, 104, 105, 109
gravity.do composite phi correction, 74
gravity.drdxfac, 74, 105, 106
gravity.get g from phi, 74
gravity.gravity type, 74, 104, 105
gravity.max multipole order, 74, 105, 108
gravity.max solve level, 74, 105
gravity.ml tol, 105
gravity.mlmg agglomeration, 74
gravity.mlmg consolidation, 74
gravity.mlmg max fmg iter, 74
gravity.mlmg nsolve, 74
gravity.no composite, 74, 103--105
gravity.no sync, 74, 104, 105
gravity.v, 19, 74
habec.verbose, 133
hmabec.verbose, 133
max denerr lev, 144
max dengrad lev, 144
max step, 14
mg.v, 19
particles.particle init file, 76
particles.particle output file, 76
particles.particle restart file, 76
particles.restart from nonparticle chkfile,
76
particles.timestamp density, 76
particles.timestamp dir, 76
particles.timestamp temperature, 76
particles.v, 76
radsolve.abstol, 133
radsolve.level solver flag, 133
radsolve.maxiter, 133
radsolve.reltol, 133
radsolve.verbos, 133
radsolve.v, 133
sponge lower density, 92
sponge lower radius, 92
sponge timescale, 92
sponge upper density, 92
sponge upper radius, 92
stop time, 14
use custom knapsack weights, 49
INDEX 195
sponge, 50
Variables
Gravity Type, 26
MOL STAGES, 46
NADV, 88
NAUX, 88
NGDNV, 87
NQAUX, 87
NQ, 86
NSPEC, 88
NVAR, 86
PhiGrav Type, 25
PhiRot Type, 26
QVAR, 86
Rad Type, 25
Reactions Type, 26
Rotation Type, 26
SDC React Type, 26
S new, 48, 50, 51, 53, 54
S old, 48, 53
Sborder, 25, 34, 48--51, 53, 54
Sburn, 54
Source Type, 26, 49, 54
State Type, 25, 48
a mol, 53
b mol, 53
c mol, 53
c real, 23
center, 39
coord type, 39
derive lst, 43
dg, 39
dim, 39
geom, 37
grids, 24
hydro sources, 55
hydro source, 26, 50, 56
k mol, 53, 54
mol iteration, 46, 53
nGroups, 25
new sources, 52, 55, 56
ngroups, 86, 87
old sources, 55, 56
physbc hi, 39
physbc lo, 39
qaux, 87
rt, 23
set, 42
sources for hydro, 27
tag, 42
uin, 86
uout, 86
visualization, 5
