Current Status ofAthena++

Kengo TOMIDA (Osaka)

James Stone (Princeton)

and

The Athena++ Development Team

11/30/2016

CfCA Users’ Meeting

Athena++• Completely redesign Athena (Stone et al. 2008) from scratch

• Flexible coordinates: non-uniform spacing, Spherical…

• Static / Adaptive Mesh Refinement (AMR)

• Support various physical processes (not limited to star formation)

- MHD(CT), self-gravity, radiation, general relativity…

• Highly parallelized： Z-ordering & dynamic scheduling

• Hybrid parallelization: MPI + OpenMP

• High performance： vectorization, memory hierarchy, etc.

• Parallel IO with MPI/HDF5, support standard software (VisIt)

• Easy-to-use, easy-to-learn, easy-to-maintain: documents & tests

• Carefully tested; we even found >3 bugs in Intel Compiler(!)

• Support various architectures; Intel, IBM BG/Q, Xeon Phi etc.

RED: New (or improved) features in Athena++

http://princetonuniversity.github.io/athena/

http://princetonuniversity.github.io/athena/

What you can do with Athena++

With the current public BETA version:

• (M)HD in Cartesian, Cylindrical, and Spherical coordinates.

• Special/General Relativity (fixed metric, e.g. Kerrschild)

• Approximate Riemann solver + Constrained Transport

• Second-order in time and space (higher order in progress)

• Massively parallel simulations on most systems in Japan

(Currently Intel and IBM computers, K/Fujitsu in progress)

• Static Mesh Refinement in any coordinate system

• Adaptive Mesh Refinement in non-relativistic systems

• User-defined source terms

• (Support by one of the core developers in Japanese)

Performance on NAOJ Cray XC30

No significant loss

with SMR (or faster!)

97% scalability btw.

24 → 6144 cores

>x2 faster than Athena

MHD SMR costs

more due to level

boundaries

New Additions / Improvements

Since the last Users’ Meeting…

• Redesign file formats and improve IO performance

(use of collective IO calls is crucial)

• Support polar singularities in spherical coordinates

• Add user-defined source/coordinate/output functions

• Implement Ohmic dissipation and ambipolar diffusion

(with X. Bai, explicit only, not public yet)

• Add tests especially for curvilinear coord. and SMR/AMR

• Improve compatibility

• Change license to BSD 3 clause from GPLv3

• Lots of debugging

• Prepare the documentation and website

Star and Disk InteractionCollaboration with Takasao-san

• 3D spherical polar coordinates with

SMR to reduce the cost

• High resolution to resolve MRI

• Stable initial condition

• Dumping zone near the star

→ improved long-term stability

X-point at R=3, corotation at R=5

65 orbits at the stellar surface ⇒MRI develops up to R ~ 10 R*

The polar region still costs too much

→need additional care.

Rossby Wave Instability

Collaboration with Ono-san

Non-linear development of

shear at pressure bumps

2D cylindrical coordinates

with static mesh refinement

Fukagawa+13

HD142527But note that the simulation is gas,

while the observation is dust.

ISM with Ambipolar Diffusion

A master thesis project at Osaka U. (Fukuda)

(Decaying) turbulent box simulations with heating/cooling

AD is crucial in the thermal structure of the ISM: J → C-shocks

Ideal MHD with AD

ISM with Ambipolar Diffusion

The thermal structure of

the ISM is not uniform

even within one phase.

• H2 rotation transition

• CH+ abundance

Stochastic heating by

turbulence?

Ideal MHD produces too

strong shocks?

→ non-ideal MHD(c.f. Myers et al. 2015)

temperature PDF at 1 crossing time

n=30 cm-3, Teq = 76K, B=5μG, Mach 5

note: this is a low-resolution test run

Future Plan• Self Gravity on AMR

• Higher-order integrator (being implemented)

• Chemistry (almost done)

• Radiation MHD (direct solver, almost done)

• Viscosity (explicit, almost done)

• Shearing box (almost done)

• Non-ideal MHD (operator split / Hall)

• Optimization for new architectures (Xeon Phi KNL)

• ポスト京萌芽的課題「惑星科学」Sub A→円盤の乱流計算

宣伝１： CfCA数値流体力学講習会 2月18日～21日予定初学者含む学部生～PDを対象。ドキュメント等を日本語化予定。Athena++で実際に自分でXC30上で科学計算ができるところまで・・・近日中にアナウンス予定。是非御参加・周知下さい。

宣伝2: ALMA共同科学事業「ALMAのための観測的可視化フレームワークの構築」研究員１名＠大阪大学公募中（12/15 3pm締切、tennet:14482）これまでの分野は問いません。コード開発・輻射輸送の経験を考慮。

↓ nested-grid MHD + RADMC-3D (+CASA) による観測的可視化これをAthena++内で完結し、より高速かつ誰でも扱えるようにしたい

Elias 2-27, ALMA Cycle-2

1.3mm, Perez+ 2016

Synthetic observation

Tomida+ submitted to ApJL

on arXiv today

Thanks!

Planet-Disk Interaction

(Figures provided by Zhaohuan Zhu)

Planet-disk interaction simulation in 3D spherical coord. with SMR

SMR allows us to resolve the small scale circumplanetary disk

while covering the whole disk with an affordable cost.

(A related paper & NASA PR: http://hubblesite.org/newscenter/archive/releases/2015/40/)

GRMHD accretion Torus

GRMHD (fixed-metric) simulations (by C. White, arXiv:1511.00943)

any Riemann solver can be used → more accurate (and faster)

than other codes using LLF. Costs ～x2.7 (HD) - x4.6 (MHD).

(Preliminary) AMR

• Implemented in a week based on the SMR

- The Black Friday is the official birthday

• “AMR part” is only less than 1000 lines

- “SMR part” is much longer, sharing most codes

• Being tested - does not work with MPI yet

• Low overhead, almost as fast as uniform grids,

at least on a single core.