Date post: | 04-Jan-2016 |
Category: |
Documents |
Upload: | patricia-dinah-little |
View: | 216 times |
Download: | 3 times |
Page 1 LWS Teams Day JSOC Overview
HMI-AIA Joint Science Operations Center Science Data Processing a.k.a. JSOC-SDP
Overview
Page 2 LWS Teams Day JSOC Overview
HMI & AIA JSOC Concept
Science TeamForecast Centers
EPOPublic
Catalog
Primary Archive
HMI & AIAOperations
House-keeping
Database
MOCSDOGSDDS
Redundant Data
Capture System
10-DayArchive
OffsiteArchiv
eLMSA
L
OfflineArchiv
e
HMI JSOC Pipeline Processing System
DataExport& WebService
Stanford
LMSAL
High-LevelData Import
AIA AnalysisSystem
Local Archive
QuicklookViewing
GSFCWhite Sands
World
Page 3 LWS Teams Day JSOC Overview
JSOC Dataflow Rates
Dataflow (GB/day)
Joint Ops
ScienceArchive440TB/yr(Offiste)
Data Capture
2 processors each
1230
1610
HMI &AIA Science
Hk
0.04
30d cache40TB each
Quick Look
LMSAL secure host
Level 0(HMI & AIA)
2 processors
75
Level 1(HMI)
16 processors
Online Data
325TB+50TB/yr
HMI High LevelProcessingc. 200 processors
HMI Science Analysis Archive 650TB/yr
Redundant data capture system
1210
1210
Data Exports
1200
LMSAL Link(AIA Level 0, HMI Magnetograms)
240
1610
1820
1230
rarelyneeded
1230
2 processorsSDO Scientist &User Interface
Dataflow (GB/day)
Joint Ops
ScienceArchive440TB/yr(Offiste)
Data Capture
2 processors each
1230
1610
HMI &AIA Science
Hk
0.04
30d cache40TB each
Quick Look
LMSAL secure host
Level 0(HMI & AIA)
2 processors
75
Level 1(HMI)
16 processors
Online Data
325TB+50TB/yr
HMI High LevelProcessingc. 200 processors
HMI Science Analysis Archive 650TB/yr
Redundant data capture system
1210
1210
Data Exports
1200
LMSAL Link(AIA Level 0, HMI Magnetograms)
240
1610
1820
1230
rarelyneeded
1230
2 processorsSDO Scientist &User Interface
Page 4 LWS Teams Day JSOC Overview
JSOC-SDP Major Components
Page 5 LWS Teams Day JSOC Overview
JSOC DRMS/SUMS Basic Concepts
• Each “image” is stored as a record in a data “series”.
• There will be many series: e.g. hmi_ground.lev0 is ground test data
• The image metadata is stored in a relational database – our Data Record Management System (DRMS)
• The image data is stored in SUMS (Storage Unit Management System) which itself has database tables to manage its millions of files.
• SUMS owns the disk and tape resources.
• Users interact with DRMS via a programming lauguage, e.g. c, FORTRAN, IDL.
• The “name” of a dataset is actually a query in a simplified DRMS naming language that also allows general SQL clauses.
• Efficient use of the system relies on direct use of DRMS
• Data may be exported from DRMS as FITS or other (TBD) protocols for remote users.
• Several Remote DRMS (RDRMS?) sites will be established which will “subscribe” to series of their choice. They will maintain RSUMS containing their local series and cached JSOC series.
• The JSOC may act as an RDRMS to access products made at remote sites.
Page 6 LWS Teams Day JSOC Overview
Pipeline Program, “module”
Record Manage
ment
Keyword Access
Data Access
DRMS Library
Link Manage
ment
Utility LibrariesJSOC Science
Libraries
Record Cache
JSOC Pipeline Processing System Components
Database Server
SUMSStorage Unit
Management System
DRMSData Record
Management SystemSUMS Tape Farm
SUMS Disks
PUIPipeline User
Interface
Pipeline processing
plan
Processing script, “mapfile”
List of pipeline modules with needed datasets for input, output
Pipeline Operato
r
Processing History Log
Page 7 LWS Teams Day JSOC Overview
Simple example – find and look at an image
P% P% show_keys "ds=hmi_ground.lev0[][2007.09.09_23:50/1m]" key=FSN,T_OBSFSN T_OBS0566684 2007.09.09_23:50:01_UTC0566685 2007.09.09_23:50:06_UTC0566686 2007.09.09_23:50:11_UTC0566687 2007.09.09_23:50:16_UTC0566688 2007.09.09_23:50:21_UTC0566689 2007.09.09_23:50:26_UTC0566690 2007.09.09_23:50:31_UTC0566691 2007.09.09_23:50:36_UTC0566692 2007.09.09_23:50:41_UTC0566693 2007.09.09_23:50:46_UTC0566694 2007.09.09_23:50:51_UTC0566695 2007.09.09_23:50:56_UTCP% P% ds9 `show_keys "ds=hmi_ground.lev0[566686]" seg=file -p -q`P%
Example of simple utility “module” called “show_”keys”
First find images in the minute starting 9 Sept at 23:50 then look atone with ds9
Page 8 LWS Teams Day JSOC Overview
JSOC Export
• JSOC will support VSO access
• JSOC will also have a direct web access
• There will be remote DRMS/SUMS systems at key Co-I institutions
• ALL HMI and AIA data will be available for export at level-0 through standard products (level-1 for both and level-2 for HMI)
• It would be unwise to expect to export all of the data. It is simply not a reasonable thing to expect and would be a waste or resources.
• Our goal is to make all useful data easily accessible.
• This means “we” must develop browse and search tools to help generate efficient data export requests.
Page 9 LWS Teams Day JSOC Overview
DRMS/SUMS Configuration
• DRMS and SUMS use the open-source PostgreSQL database engine.
• DRMS will run on a pair of dedicated servers likely 4-quad-core processors with up to 10TB fast disk.
• SUMS will consist of file servers with attached tape systems.
• SUMS will manage 200TB cache disk with 150TB/year of permanent archive for level-1 and higher level products.
• SUMS archive will use LTO-4 tapes in a robotic system with at least 10 drives and 2000 tapes near-line.
• The pipeline processing system will have about 50 processor cores dedicated to level-0 to level-1 processing and about 450 cores for higher level processing in the pipeline. HMI standard products will need about half of these.
Page 10 LWS Teams Day JSOC Overview
Extra Info
Page 11 LWS Teams Day JSOC Overview
Data Record Management Service
(DRMS)
Data Record Management Service
(DRMS)
Pipeline client-server architecture
JSOC Disks
Analysis codeC/Fortran/IDL/Matlab
JSOC Library
Record Cache (Keywords+Links+Data paths)
OpenRecordsCloseRecords
GetKeyword, SetKeywordGetLink, SetLink
OpenDataSegmentCloseDataSegment
Pipeline client process
JSOC DisksJSOC DisksJSOC Disks
Data Record Management Service
(DRMS)
Storage Unit Management Service
(SUMS)
Tape ArchiveService
SeriesTables
Record Catalogs
Storage UnitTables
Record CatalogsRecord
Tables
PostgreSQL DatabaseServer
AllocUnitGetUnitPutUnit
SQL queries
SQL queries
SQL queries
Storage unit transfer
Data Segment I/O
Storage unit transfer
Generic file I/O
DRMS socket protocol
Page 12 LWS Teams Day JSOC Overview
Pipeline batch processing
• A pipeline batch is encapsulated in a single database transaction, “DRMS session”:
– If no module fails all data records are commited and become visible to other clients of the JSOC catalog at the end of the session
– If failure occurs all data records are deleted and the database rolled back
– It is possible to commit data produced up to intermediate checkpoints during sessions
DRMS Service = Session Master
Input data records
Output data records
DRMS API
Registersession
DRMS API
Module 1
DRMS API
Module 2.2
…
DRMS API
Module N
DRMS API
Commit Data&
Deregister
Pipeline batch = atomic transaction
Record & SeriesDatabase SUMS
DRMS API
Module 2.1
Page 13 LWS Teams Day JSOC Overview
HMI module status and MDI heritage
DopplerVelocity
HeliographicDoppler velocity
maps
Tracked TilesOf Dopplergrams
StokesI,V
ContinuumBrightness
Tracked full-disk1-hour averagedContinuum maps
Brightness featuremaps
Solar limb parameters
StokesI,Q,U,V
Full-disk 10-minAveraged maps
Tracked Tiles
Line-of-sightMagnetograms
Vector MagnetogramsFast algorithm
Vector MagnetogramsInversion algorithm
Egression andIngression maps
Time-distanceCross-covariance
function
Ring diagrams
Wave phase shift maps
Wave travel times
Local wave frequency shifts
SphericalHarmonic
Time series
Mode frequenciesAnd splitting
Brightness Images
Line-of-SightMagnetic Field Maps
Coronal magneticField Extrapolations
Coronal andSolar wind models
Far-side activity index
Deep-focus v and cs
maps (0-200Mm)
High-resolution v and cs
maps (0-30Mm)
Carrington synoptic v and cs maps (0-30Mm)
Full-disk velocity, sound speed, Maps (0-30Mm)
Internal sound speed
Internal rotation
Vector MagneticField Maps
MDI pipeline modules exist
Standalone production codes in use at Stanford
Research codes in use by team
Codes to be developed at HAO
Codes being developed in the community
Codes to be developed at Stanford
Primary observables
Intermediate and high level data products
Page 14 LWS Teams Day JSOC Overview
AIA Level-2
Page 15 LWS Teams Day JSOC Overview
JSOC Data Volumes from Proposal
this version modified to show the links to the hardware plan
img size channels
cadence
compress
HMI: 55,000,000 bps ** SU 553 30 16 200% 395 90 49
AIA: 67,000,000 bps ** SU 674 30 20 200% 482 90 59
HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189 180 93
AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386 1,900 2,004
HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0 0
AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39 0
HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25 0 0
AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77 0 0
HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0 0
AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0 0
HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0 0
AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0 0
HMI: tlm SU 553 100% 198 24
AIA: tlm SU 674 100% 241 30
HMI: Lev0, Lev-1, All Higher SU 730 412 93
AIA: Lev0, Lev1a SU 1,296 743 2,004
HMI Totals 68 71 610 118
AIA Totals 146 77 984 2,034
Combined (TB) 214 148 1,594 2,151
Tape shelf size (TB) 7,968
Tape shelf number of tapes - mixed density 11,257
Export
Near-line retain days
Near-line Cache (TB)
Tape Archive Fraction
Totals
2,026Local tape
LMSAL Link 1,193
1,227Offsite tape
Higher level 286
Level-1
Data Path Assumptions Combined (GB/day)
Fixed Disk cache (TB)
Online disk cache days
Perm disk per year (TB)
Level-01,610
Volume (GB/day)
Processed at
In from DDS
Tape per year (TB)
1,227
1,210
797
Page 16 LWS Teams Day JSOC Overview
Sample of DRMS/SUMS Resource Assessment
Page 17 LWS Teams Day JSOC Overview
JSOC Level-0 processing flow
Page 18 LWS Teams Day JSOC Overview
JSOC-SDP Status
• Capture system complete, waiting testing with DDS at White Sands
• DRMS and SUMS virtually done
• Level-0 work progressing, to be done by instrument deliveries
• Level-1 work to start after delivery
• Getting serious about work on basic pipeline modules
• Review of status and computer hardware plans in early November
Page 19 LWS Teams Day JSOC Overview
Stanford JSOC effort plan
HMI-SU Effort Distribution
0
100
200
300
400
500
600
700
800
900
1000
FY2006Q1
FY2006Q2
FY2006Q3
FY2006Q4
FY2007Q1
FY2007Q2
FY2007Q3
FY2007Q4
FY2008Q1
FY2008Q2
FY2008Q3
FY2008Q4
FY2009Q1
Pe
rce
nt
FT
E
On-Demand Support
Irradiance
Forecast - farside…
Coronal Inferences
Mag Field - Vector Field
Mag. Field - Line-of-Sight
Level 2 - Local HS Holography
Level 2 - Local HS Ring Diagram
Level 2 - Local HS Time Distance
Level 2 - Global HS
Level 2 - Quick Look
HK & FDS
Level-0
JSOC Verification & Test
Data Quality and Proc. Metadata
General Env.: cvs,os,oracle,compile
Archive h/w arch.
Processing Hardware Arch.
User Tools: API, Data Export, & Browsing
pui
Data Capture
DRMS
SUMS
Support for AIA