21
Millennium Data Dissemination 2012-11-26 MPA institute seminar 1 possible extensions how ambitious can/should we be?
MPA institute seminar 1
Millennium Data Dissemination
2012-11-26
possible extensionshow ambitious can/should we
be?
Raw data:Particles
FOF groups and Subhalos
Density fields
Subhalo merger trees
Synthetic galaxies (SAM)
Mock catalogues
Mock images
2012-12-17 Millennium Workshop 2
SUBFIND Tree Builder
Halo Merger Trees
SAMSSP
FOF GroupFinder
(Sub-)HalosFOF GroupsSimulation Particles
Gadget
MoMaF’
SAM Galaxies
Light-cone Galaxies
SkyMaker’SExtractor
Synthetic Images
SExtractor Catalogues
PSF, Noise,Background
Scaling
IGM
MRObs
2012-12-17 Millennium Workshop 3
MPA institute seminar 4
Raw data• Serve particles from Millennium
simulations– 10 billion/snapshot, ~25TB
• from database?– with spatial queries– links to halos/FOF groups
• through JobRunner services– cutout– gridding + FFTs etc
2012-11-26
MPA institute seminar 5
Other simulations?• Hydro Example: – http://gavo.mpa-garching.mpg.de/HydroClusters– online visualisation:
http://galformod.mpa-garching.mpg.de/phoxybrowser • Others– Millennium+gas?– Eagle– …
• How?– In DB ok for objects– Visualisation support for “fluffy things”
2012-11-26
6
SUBFIND Tree Builder
Halo Merger Trees
SAMSSP
FOF GroupFinder
(Sub-)HalosFOF GroupsSimulation Particles
Gadget
MoMaF’
SAM Galaxies
Light-cone Galaxies
SkyMaker’SExtractor
Synthetic Images
SExtractor Catalogues
PSF, Noise,Background
Scaling
IGM
MRObs
2012-12-17 Millennium Workshop 7
MPA institute seminar 8
Group catalogs• Have
– FOF– SUBFIND subhalos– Mhalo merger trees– Dhalo merger trees
• More properties– shapes, density profiles (ala AIP, or ala MR-XXL): good for lensing– others?
• More?– BDM, NEXUS, ORIGAMI, …– All of Knebe etal
• LSS objects?– Voids, filaments, walls in the DB– “more precise” environmental queries possible– per grid cell or as object (“stacking voids”)
2012-11-26
SUBFIND Tree Builder
Halo Merger Trees
SAMSSP
FOF GroupFinder
(Sub-)HalosFOF GroupsSimulation Particles
Gadget
MoMaF’
SAM Galaxies
Light-cone Galaxies
SkyMaker’SExtractor
Synthetic Images
SExtractor Catalogues
PSF, Noise,Background
Scaling
IGM
MRObs
2012-12-17 Millennium Workshop 9
MPA institute seminar 10
SAM catalogues• More models– Durham, MPA
• Complete MR-XXL?–~200x Millennium, 80TB
• Extensions
2012-11-26
MPA institute seminar 11
SFH-s• Allows SEDs from post-processing• how to store? – rows vs arrays (in SQLServer)– functions (easy in SQLServer):
• create spectra– “exact” observer frame bands
• PCA amplitudes in the database– find nearby galaxies for given observed
spectrum2012-11-26
MPA institute seminar 12
SAM online• GALFORM online !!!
– http://galformod.mpa-garching.mpg.de/dev/jobrunner/
– on millimil• Do we want others?– Galacticus?
• Need constraints, validated models.– How afraid should we be of “bad science”?
• SAM online on Millennium?– input to MRObs pipeline
2012-11-26
SUBFIND Tree Builder
Halo Merger Trees
SAMSSP
FOF GroupFinder
(Sub-)HalosFOF GroupsSimulation Particles
Gadget
MoMaF’
SAM Galaxies
Light-cone Galaxies
SkyMaker’SExtractor
Synthetic Images
SExtractor Catalogues
PSF, Noise,Background
Scaling
IGM
MRObs
2012-12-17 Millennium Workshop 13
MPA institute seminar 14
Light cones• In Database: not just from MPA!
– common model– linked to SAMs
• More– deep: new code for Millennium-II– all sky: MR-XXL– with scaling!
• What properties needed?– who wants them? – what for (lensing?)?
• Light cone building online– Will soon make our MoMaF available on JobRunner– new code in preparation (for MR-II etc)
2012-11-26
SUBFIND Tree Builder
Halo Merger Trees
SAMSSP
FOF GroupFinder
(Sub-)HalosFOF GroupsSimulation Particles
Gadget
MoMaF’
SAM Galaxies
Light-cone Galaxies
SkyMaker’SExtractor
Synthetic Images
SExtractor Catalogues
PSF, Noise,Background
Scaling
IGM
MRObs
2012-12-17 Millennium Workshop 15
MPA institute seminar 16
MRObs• Store of images will keep growing
– Overzier!• Create images on demand
– telescope models custom?• Post processing as well?
– Sextractor– other?
• Improvement galaxy images– realistic structure (weak lensing systematics, strong
lensing structure)– from simulations ?– form observations?
2012-11-26
SUBFIND Tree Builder
Halo Merger Trees
SAMSSP
FOF GroupFinder
(Sub-)HalosFOF GroupsSimulation Particles
Gadget
MoMaF’
SAM Galaxies
Light-cone Galaxies
SkyMaker’
SExtractor
Synthetic Images
SExtractor Catalogues
PSF, Noise,Background
Scaling
IGM
MRObs
2012-12-17 Millennium Workshop 17
Lensing
MPA institute seminar 18
Lensing• Under construction – with Ben Metcalf
• Online?–Why not, JobRunner will handle it.
• How much?– what is required for weak lensing?
2012-11-26
MPA institute seminar 19
Even more• L-Galaxies online++– updating physics modules?–MCMC mode?
• Dark matter light cones?– for lensing– what do we need? particles, fields?
• Interface– dashboard (M.Egger)– improve SQL interface (Roys talk)
2012-11-26
MPA institute seminar 20
Galformod infrastructure• Currently– 12+1 12-core nodes– 180 TB raw storage– 20TB DB storage
• What do others have?– AIP ~500TB– JHU ~5PB (?)
• What do we want?
2012-11-26
100 billion …
MPA institute seminar 21
What do we want need?• Depends on answers to all previous questions• What can we afford?– Where can we get the money?
• Where do we put “it”?– RZG?
• Pool our resources together?– beyond mirror– distributed over Virgo nodes?
• Larger scale collaborations (EUDat)?– see Hermann’s talk
2012-11-26
