The yt Project:Growing & Engaging a Community of Practice

Matthew TurkColumbia University

What is yt?

astro-ph/1011.3514astro-ph/1112.4482

yt-project.org

yt is a platform for analysis and visualization of astrophysical* simulations

install script:

Full dependency stackSource code

Development environmentGUI

Sample data

There are many simulation codes.

There are many simulation codes, but there’s only one sky.

data structures,methods,

assumptions,IO formats,

units,variable names,

...

Fully-Supported Semi-Supported In-Progress

EnzoFLASHOrionNyx

Uniform Data

ChomboAthena

ARTRAMSES

GDFPiernik

CactusHydra

Gadget

yt is designed to address physical,not computational,

entities and questions.

Orthogonal RaysNon-orthogonal Rays 1D

SlicesOblique SlicesProjections

2D

SpheresRectangular PrismsDisks/CylindersInclined BoxesClumpsExtracted RegionsEllipsoids

3D

Scripting interface, HTML5 GUI, in-situ processing, transparent multi-level parallelism...

yt is supposed to get out of the way.

from yt.mods import *pf = load(’galaxy0030/galaxy0030’)p = SlicePlot(pf, 2, ’Density’, ’c’, (200,’kpc’))p.save(’Galaxy’)

10-12 g/cc

6 C.B. Hummels et al.

Figure 1. Sample projections from the medium feedback and cooling suppression run at z = 0.5. Each image represent a column-densitymap of a region with 500 comoving kpc on a side. The white circle denotes r200 = 160 kpc.

tributions of a wide variety of atomic species, observable inabsorption against background quasars. We initially restrictourselves to low redshift, as there exists a substantial bodyof observations which link absorption strength to projecteddistance from the galaxy. In the following sections, we firstexamine the column density distributions, then explore thephysical properties of the gas giving rise to this absorption,and finally investigate the redshift evolution of the predictedabsorption strength.

3.1 Radial Profiles for Atomic Species

In this study, we examine the radial column density distri-bution for a number of absorption-line-generating species,well-sampling the range of ionization energies (and thus tem-perature/density regimes) of the CGM as shown in Table 2.At the two ends of the spectrum, the H I traces the cold,dense clouds, whereas the O VII probes the hot coronal gas.Between these regimes is a continuum of layers of mate-rial acting as the interface between these two phases. Thislayering e!ect takes place on many di!erent spatial scalesfrom small star-forming clumps of cold gas up to the galacticscale. Figure 1 demonstrates this at the latter scale, wherethe cold H I of the inner halo is enshrouded in warmer CIV-bearing gas, which is further encased and extended bygas exhibiting O VI. This provides us with some intuition

for understanding the following galactic profiles for thesespecies.

We produce radial profiles of the median gas columndensity in nine di!erent ionic species as a function of pro-jected radius extending from 2 kpc to 1 Mpc for our simu-lated galaxies. We probe this gas from the disk-dominatedregion (r < rinner ! 20 kpc) through the CGM (rinner <r < r200) into the intergalactic medium (r > r200). Figures2, 3, and 4 present these results (Figure 5 shows total gascolumn densities for comparison). The left and right sidesof each figure show the radial profiles of the same quantitybut for di!erent simulations. The left sides display a com-parison of simulations with and without di!erent forms offeedback: V, MF and MFCS, whereas the right sides exhibita progression of thermal feedback intensities: LF, MF andHF (the MF simulation is repeated on both sides to easecomparison). Each profile gives the median value expectedin an observation of a galaxy at a given radius over the red-shift range z = 0.1 " 0.5 (selected to approximately matchthe redshift ranges of the observations). To give some idea ofthe variation around the median absorber strength, in eachpanel we overplot a color band representing the quartiles ofthe column density distribution (i.e. 25% to 75%) for onemodel: MFCS (left) and HF (right). The quartile distribu-tion associated with the HF model is most indicative of thespread of all the other models, whereas MFCS produces a

c! 2012 RAS, MNRAS 000, 1–19

Canned Analysis Tasks

Absorption SpectrumCoordinate Transformations

Halo FindingMass FunctionsMerger TreesHalo Profiling

Level SetsLight ConesLight RaysTime Series

Star AnalysisTwo-Point Analysis

“Community”?

“Users”

“Developers”

Traditional View

“Users”

“Developers”

Most Scientific Software

“Devusers”

Community of Practice

“Developers”

“Developers”

Inspection and verification

“Developers”

Inspection and verificationTracking modifications

“Developers”

Inspection and verificationTracking modificationsSharing information

“Developers”

Inspection and verificationTracking modificationsSharing informationAdding functionality

“Developers”

Inspection and verificationTracking modificationsSharing informationAdding functionality

All are necessary characteristics of the scientific process as a whole.

“Users”

“Users”

Uncritical acceptance of code...?

“Users”

Uncritical acceptance of code...?

“These are people we give the code to that don’t care how it works.”

Developing as a Team

Challenges

Reward Structure

de facto and de jure

de facto and de jure

•Utilization of developed tools

•Respect from community

•Project involvements

•Invitations to speak

de facto and de jure

•Funding

•Publications

•Citation count

•Influence

Traditional astrophysics does not favor tool builders.

Chores

Chores

Documentation,testing,

outreach,infrastructure.

Tasks not fully-aligned with reward structure goals present great

motivational challenges.

Co-opetition

Co-opetition

•Funding

•Publications

•Citation count

•Influence( )

How developer community engagement -- as seen in cohesion, excitement, energy -- is

affected by funded improvements is unclear.

Strategies

The Art of Community by Bacon

Producing Open Source Software by Fogel

Team Geek by Fitzpatrick & Collins-Sussman

Organizing Simulation Code Collectives by Sundberg

Scientific Software Production by Howison & Herbsleb

You must design the community you want.

Technical& Social

Technical& Social

SVN hg

Repository

Users

Users & Repos

Users & Repos

Users & Repos

Users & Repos

Users & Repos

Users & Repos

Forky development:

very low barrier to entry; everything comes in the box.

Testing:

answer as well as integration tests get run every 30 minutes.

Code review:

eyes on (nearly) every changesetmentoring new developers

The path upstream must be obvious.

•Happy application

•Itch-scratching

•Submission

•Code review & mentoring

•Participation

Non-core contributed code:

a place for sharing scripts,data and images

Communication

Communication

All business is conducted openly.

Communication

Immediate

2 Communication

Low-Latency

Communication

High-Latency

Technical& Social

HRT

HumilityRT

HumilityRespectT

HumilityRespectTrust

HumilityRespectTrust

(Fitzpatrick & Collins-Sussman)

The most important thing I try to remember is that I want to foster a community of peers, not of elites.

I’ve noticed something is acting strangely with ...

““

I’ve noticed something is acting strangely with ...

You’re probably doing it wrong.““

““

Can you tell us how you would expect it to act?

I’ve noticed something is acting strangely with ...

““

““

I think that there might be a bug in the way ...

““

It’s like that for very good reasons. Don’t touch it.

I think that there might be a bug in the way ...

““

““

It behaves the way it does because ...

I think that there might be a bug in the way ...

““

““

Thoughtfulness,inquiry,

confidence,letting go.

This culture must be seeded directly.

This culture must be seeded directly,because culture will self-propagate.

letting go

Pride rather than ownership

Projects passing between people

Too much control can smother.

Too much control can smother.will

Successes

Developed by working astrophysicists.

Tom AbelDavid CollinsBrian CrosbyAndrew CunninghamNathan GoldbaumCameron HummelsJi-hoon KimSteffen KlemerKacper KowalikMichael KuhlenEve LeeChris MaloneChris MoodyAndrew MyersJeff Oishi

Jean-Claude PassyThomas RobitailleAnna RosenAnthony ScopatzDevin SilviaSam SkillmanStephen SkoryBritton SmithGeoffrey SoCasey StarkElizabeth TaskerMatthew TurkRick WagnerJohn WiseJohn ZuHone

Tom AbelDavid CollinsBrian CrosbyAndrew CunninghamNathan GoldbaumCameron HummelsJi-hoon KimSteffen KlemerKacper KowalikMichael KuhlenEve LeeChris MaloneChris MoodyAndrew MyersJeff Oishi

Jean-Claude PassyThomas RobitailleAnna RosenAnthony ScopatzDevin SilviaSam SkillmanStephen SkoryBritton SmithGeoffrey SoCasey StarkElizabeth TaskerMatthew TurkRick WagnerJohn WiseJohn ZuHone

“Usage of a HPC Data Analysis and Visualization System,” Szczepanski et al 2012

Usage on NSF NICS Nautlius

(play movie)

“... it seems likely that significant software contributions to existing scientific software projects are not likely to be rewarded through the traditional reputation economy of science. Together these factors provide a reason to expect the over-production of independent scientific software packages, and the under-production of collaborative projects in which later academics build on the work of earlier ones.”

Howison & Herbsleb (2011)

Thank you.

Tom AbelDavid CollinsBrian CrosbyAndrew CunninghamNathan GoldbaumCameron HummelsJi-hoon KimSteffen KlemerKacper KowalikMichael KuhlenEve LeeChris MaloneChris MoodyAndrew MyersJeff Oishi

Jean-Claude PassyThomas RobitailleAnna RosenAnthony ScopatzDevin SilviaSam SkillmanStephen SkoryBritton SmithGeoffrey SoCasey StarkElizabeth TaskerMatthew TurkRick WagnerJohn WiseJohn ZuHone