Date post: | 02-Jan-2016 |
Category: |
Documents |
Upload: | eileen-mitchell |
View: | 219 times |
Download: | 0 times |
AIA Core-Team Meeting20-22 April 2009
JSOC Stuff
Phil Scherrer
5. Data export and centers, US and beyond; data importa. Data distribution: system and volume requirements
i. JSOC to world Scherrer 20min 14:40ii. JSOC to SAO Scherrer/Davey 10min 15:00iii. Elsewhere (ROB, Lancashire, …) Fleck/Boyes/Dalla 15min 15:10iv. Internal flow: Stanford to LM , and back Hurlburt/Serafin 10min 15:25
Break 15:35 – 15:55
b. JSOC (direct) data interfacesi. DRMS system Scherrer 20min 15:55ii. Web interface Summers [TBD] 10min 16:15iii. IDL+… interfaces Freeland 20min 16:25iv. VSO Gurman 15min 16:45
c. Data from other instruments: i. SDO: HMI and EVE, discussion Hurlburt/Hock 15min 17:00ii. Other – Discussion Schrijver 10min 17:15
d. Documentation review and action items Scherrer/Green 15min 17:25
6. Summary: HMI data for AIA science investigation Hoeksema/Scherrer 10min 17:40
Data export and centers, US and beyond; data import a. Data distribution: system and volume requirements
i. JSOC to world
http://jsoc.stanford.edu/jsocwiki/TeamMeetings Link to SDO Pre-Ship Review(ppt) http://hmi.stanford.edu/Presentations/SDO-PSR/25-AIA_HMIInst&ScienceOps-PhilScherrer.ppt
And
Link to JSOC Status shown at 2008 SDO Teams Meeting http://hmi.stanford.edu/TeamMeetings/Mar_2008/Proceedings/JSOC_Status_March_2008.ppt
First page
HMI and AIA JSOC Architecture
Science TeamForecast Centers
EPOPublic
Catalog
Primary Archive
MOCDDS
Redundant Data
Capture System
12-DayArchive
OffsiteArchiv
e
OfflineArchiv
e
HMI JSOC Pipeline Processing System
DataExport& WebService
JSOC-SDP
LMSAL
High-LevelData Import
AIA AnalysisSystem
Local Archive
HMI & AIAOperations
House-keeping
Database
QuicklookViewing
housekeeping GSFCWhite Sands
World
JSOC-IOC
Stanford
JSOC-AVC
JSOC – SDP Locations at Stanford
Cedar South
Cypress North
P&A 1st floor
P&A Basement
JSOC Interfaces with SDO Ground System
JSOC-SDP Stanford JSOC-IOC LMSAL
DDS Handshakefiles
S-bandKa-band
RT HK Telemetry (S-band)
Science data files (Ka-band)
DDS at WSC
MOC at GSFC
InstrumentCommands
RT HK Telemetry
RT HK telemetry L-0 HK files FDS products Planning data
JSOC-SDP Primary responsibilities: Capture, archive and process science data
Additional: Instrument H&S monitoring
JSOC-IOC Primary responsibilities:Monitor instruments health and safety in real-time, 24/7Control instrument operations and generate commandsSupport science planning functions
Instrument Commands
Spare Science Data
Capture SystemAIA Science Data Capture
SystemHMI Science Data Capture
System
AIA MON monitoringPlanning
HMI MON monitoringPlanning
AIA OPS Real-time
Inst monitor and Control
HMI OPS Real-time
Inst monitor and Control
AIAQLQuicklook Planning Analysis
HMI QLQuicklookPlanningAnalysis
L-0 HK files FDS products Mission support data
SDP segmentT&C-Segment
JSOC-SDP Major Components
4 Quad Core X86-64
Processors
10 TB Disk
4 Quad Core X86-64
Processors
10 TB Disk
4 Quad Core X86-64
Processors
10 TB Disk
Primary Secondary Export
Database – DRMS & SUMS
2 Quad Core X86-64
Processors
Support W/S, FDS, L0 HK, Pipeline User Interface, etc.
Web Server
& Export Cache
2 Dual Core X86-64
Processors
10 TB Disk
2 Dual Core X86-64
Processors
8 TB Disk
LTO-4 Tape Library
HMI SPARE AIA
SPARE@MOCOffsite
Data Capture System
DDS
2 Dual Core X86-64
Processors
8 TB Disk
LTO-4 Tape Library
2 Dual Core X86-64
Processors
8 TB Disk
LTO-4 Tape Library
2 Dual Core X86-64
Processors
1 TB Disk
LTO-4 Tape Library
2 Dual Core X86-64
Processors
2 TB Disk
LTO-4 Tape Library
Pipeline Processor Cluster
512 cores in 64 nodes
2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors
2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors2 Quad Core X86-64
Processors
400 TB Disk
150 TB Disk
Per year
2200 Cartridge
Library
Tapes
12 LTO-4 Drives
10-Gig
LMSAL
Link
1-Gig
Web
Link
ethernet
Fast interconnect
SUMS Server
Local Science
Workstations
Firew
all
JSOC Dataflow RatesDataflow (GB/day)
Joint Ops
ScienceArchive440TB/yr(Offiste)
Data Capture
2 processors each
1230
1610
HMI &AIA Science
Hk
0.04
30d cache40TB each
Quick Look
LMSAL secure host
Level 0(HMI & AIA)
2 processors
75
Level 1(HMI)
16 processors
Online Data
325TB+50TB/yr
HMI High LevelProcessingc. 200 processors
HMI Science Analysis Archive 650TB/yr
Redundant data capture system
1210
1210
Data Exports
1200
LMSAL Link(AIA Level 0, HMI Magnetograms)
240
1610
1820
1230
rarelyneeded
1230
2 processorsSDO Scientist &User Interface
Dataflow (GB/day)
Joint Ops
ScienceArchive440TB/yr(Offiste)
Data Capture
2 processors each
1230
1610
HMI &AIA Science
Hk
0.04
30d cache40TB each
Quick Look
LMSAL secure host
Level 0(HMI & AIA)
2 processors
75
Level 1(HMI)
16 processors
Online Data
325TB+50TB/yr
HMI High LevelProcessingc. 200 processors
HMI Science Analysis Archive 650TB/yr
Redundant data capture system
1210
1210
Data Exports
1200
LMSAL Link(AIA Level 0, HMI Magnetograms)
240
1610
1820
1230
rarelyneeded
1230
2 processorsSDO Scientist &User Interface
JSOC Data Volumes from Proposalthis version modified to show the links to the hardware plan
img size channels
cadence
compress
HMI: 55,000,000 bps ** SU 553 30 16 200% 395 90 49
AIA: 67,000,000 bps ** SU 674 30 20 200% 482 90 59
HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189 180 93
AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386 1,900 2,004
HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0 0
AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39 0
HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25 0 0
AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77 0 0
HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0 0
AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0 0
HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0 0
AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0 0
HMI: tlm SU 553 100% 198 24
AIA: tlm SU 674 100% 241 30
HMI: Lev0, Lev-1, All Higher SU 730 412 93
AIA: Lev0, Lev1a SU 1,296 743 2,004
HMI Totals 68 71 610 118
AIA Totals 146 77 984 2,034
Combined (TB) 214 148 1,594 2,151
Tape shelf size (TB) 7,968
Tape shelf number of tapes - mixed density 11,257
Export
Near-line retain days
Near-line Cache (TB)
Tape Archive Fraction
Totals
2,026Local tape
LMSAL Link 1,193
1,227Offsite tape
Higher level 286
Level-1
Data Path Assumptions Combined (GB/day)
Fixed Disk cache (TB)
Online disk cache days
Perm disk per year (TB)
Level-01,610
Volume (GB/day)
Processed at
In from DDS
Tape per year (TB)
1,227
1,210
797
this version modified to show the links to the hardware plan
img size channels
cadence
compress
HMI: 55,000,000 bps ** SU 553 30 16 200% 395 90 49
AIA: 67,000,000 bps ** SU 674 30 20 200% 482 90 59
HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189 180 93
AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386 1,900 2,004
HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0 0
AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39 0
HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25 0 0
AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77 0 0
HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0 0
AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0 0
HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0 0
AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0 0
HMI: tlm SU 553 100% 198 24
AIA: tlm SU 674 100% 241 30
HMI: Lev0, Lev-1, All Higher SU 730 412 93
AIA: Lev0, Lev1a SU 1,296 743 2,004
HMI Totals 68 71 610 118
AIA Totals 146 77 984 2,034
Combined (TB) 214 148 1,594 2,151
Tape shelf size (TB) 7,968
Tape shelf number of tapes - mixed density 11,257
Export
Near-line retain days
Near-line Cache (TB)
Tape Archive Fraction
Totals
2,026Local tape
LMSAL Link 1,193
1,227Offsite tape
Higher level 286
Level-1
Data Path Assumptions Combined (GB/day)
Fixed Disk cache (TB)
Online disk cache days
Perm disk per year (TB)
Level-01,610
Volume (GB/day)
Processed at
In from DDS
Tape per year (TB)
1,227
1,210
797
this version modified to show the links to the hardware plan
img size channels cadence
compress
HMI: 55,000,000 bps ** SU 553 30 16 200% 395
AIA: 67,000,000 bps ** SU 674 30 20 200% 482
HMI: 4k*4k*2 bytes/2-seconds*(pi/4) 3.4E+07 2 4 0.39 SU 530 100 52 100% 189
AIA: 4k*4k*2 bytes * 8 imgs per 10 seconds 3.4E+07 8 10 0.50 SU 1,080 30 32 100% 386
HMI: V,M,Ic @ 45s & B, ld, ff @ 90s*(pi/4) 3.4E+07 5.5 45 0.39 SU 130 0 0 46 0
AIA: Level 1.0 same as level-0 3.4E+07 8 10 0.50 tbd 1,080 90 95 10% 39
HMI: See below 7.5E+10 1 86400 1.00 SU 70 0 0 25 100% 25
AIA (lev1a): movies & extracted regions. @ 20% 6.7E+06 8 10 0.50 LM 216 0 0 77 100% 77
HMI: Magnetograms (M, B) 3.4E+07 5 90 0.39 na 59 100 6 0
AIA: Full Level-0 data+lev1_extract 3.5E+07 8 10 0.50 na 1,134 100 111 0
HMI: 2 * Higher Level products + 5*10 min B SU 149 60 1 0
AIA: 3* higher Level products (TRACE < 1) SU 648 60 6 0
HMI: tlm SU 553 100% 198
AIA: tlm SU 674 100% 241
HMI: Lev0, Lev-1, All Higher SU 730 412
AIA: Lev0, Lev1a SU 1,296 743
HMI Totals 68 71 610
AIA Totals 146 77 984
Combined (TB) 214 148 1,594
Tape shelf size (TB) 7,968
Tape shelf number of tapes - mixed density 11,257
Tape per year (TB)
1,227
1,210
797
Level-01,610
Volume (GB/day)
Processed at
In from DDS
Higher level 286
Level-1
Data Path Assumptions Combined (GB/day)
Fixed Disk cache (TB)
Online disk cache days
Perm disk per year (TB)
Tape Archive Fraction
Totals
2,026Local tape
LMSAL Link 1,193
1,227Offsite tape
Export
JSOC Data Volumes from Proposal
JSOC Processing “Levels”
• Tlm is raw telemetry files as received from SDOGS
• Level-0 is images extracted from tlm with added meta-data, no change to pixels
• Level-1 is cleaned up and calibrated into physical units in standardized form
• Level-2 is science data products
• Level-3 is higher level products or user produced products and are not JSOC products but may be archived and distributed as desired by owner
JSOC DCS Science Telemetry Data Archive
• Telemetry data is archived twice
• The Data Capture System (DCS) archives tlm files for offsite storage
• Archive tapes are shipped to the offsite location and verified for reading
• The Data Capture System copies tlm files to the Pipeline Processing System
• The Pipeline Processing System generates Level-0 images and archives both tlm and Level-0 data to tape
• Only when the DCS has received positive acks on both tlm archive copies does it inform the DDS, which is now free to remove the file from its tracking logic
HMI and AIA Level-0
• Level-0 Processing is the same for HMI and AIA
• Level 0.1 – Immediate – Used for Ops quicklook– Reformat images– Extract Image Header meta-data– Add “Image Status Packet” high-rate HK Packet (per image)– Export for JSOC IOC Quicklook
• Level 0.3 – Few minute lag - Used for quicklook science data products– Add other RT HK meta-data– Add FDS S/C info
• Level 0.5 – Day or more lag – Used for final science data products– Update FDS data– Add SDO roll info– Includes final images
Level 1 HMI and AIA Basic Observable Quantities
• HMI and AIA level-1 “levels” are similar but the details differ (a lot).• HMI – from filtergrams to physical quantities
– 1.0 Flat field applied to enable limb fit and registration– 1.5 Final product types
• 1.5q – Quicklook available in ~10 minutes, saved ~10 days• 1.5p – Provisional mix of 1.5q and 1.5 final• 1.5 – Final best possible product
– Products• Continuum Intensity• Doppler Velocity• Line of Sight Field• Vector Field
• AIA – Filtergrams are basic product– Quicklook and Final both produced– Planning movies from quicklook– Full details in development
Newer Processing Flow Diagrams
http://jsoc.stanford.edu/jsocwiki/Lev1Doc
Link to Cmap at diagram to
http://jsoc.stanford.edu/Cmaps/web/JSOC_SDP_Data_Flow.html
On following pages ---
HK 1553Other APID
Level-0
Level-0.3
FDS predict datadayfiles
Command logs if needed
FDS series,temp Few minutes lag
Select nearest or average
Level-0HSB image
Immediate orRetransmitted,
permanent
Level-0 HKISP
Level-0.1Ground Tables
HK 1553 APID dayfiles Level-0.5
Level-0 HK,temp
FDS final datadayfiles
FDS series,temp
SDO HK dayfilesFrom MOC
SDO HK lev0 temp
JSOC-IOCquicklook,Temp, 5d
Level 1.0qFlat fielded and
bad pixel list included,Temp 1d
Immediate
Day lag
Level 1.0Flat fielded and
bad pixel list included,Temp 60d
Level 1.5Final Observables,
permanent
Level 1.5qQuicklook Observables,
Temp, 5d
DDS
DDS
HK via MOC
JSOCIOC
JSOCIOC
MOC
MOC
MOC
MOC
Level 1.5pProvisional Observables,
Links to best avail
JSOC LEVEL-0 Processing
Configuration Management & Control• Capture System
– Managed by JSOC-SDP CCB after August freeze
– Controlled in CVS
• SUMS, DRMS, PUI, etc. Infrastructure– Managed by JSOC-SDP CCB after launch
– Controlled in CVS
• PUI Processing Tables– Managed by HMI and/or AIA Instrument Scientist
– Controlled in CVS
• Level 0,1 Pipeline Modules– Managed by HMI and/or AIA Instrument Scientist
– Controlled in CVS
• Science Analysis Pipeline Modules– Managed by program author
– Controlled in CVS
Data export and centers, US and beyond; data import a. Data distribution: system and volume
requirements
And b. JSOC (direct) data interfaces
i. DRMS system ii. Web interface
http://jsoc.stanford.edu/jsocwiki/TeamMeetings Link to JSOC Status shown at 2008 SDO Teams Meeting http://hmi.stanford.edu/TeamMeetings/Mar_2008/Proceedings/JSOC_Status_March_2008.ppt
First page
JSOC Export• ALL HMI and AIA data will be available for export at level-1 through
standard products (level-1 for both and level-2 for HMI)• It would be unwise to expect to export all of the data. It is simply not a
reasonable thing to expect and would be a waste of resources.• Our goal is to make all useful data easily accessible.• This means “we” must develop browse and search tools to help generate
efficient data export requests.• Quicklook Products
– Quicklook raw images to JSOC IOC– Quicklook Basic Products to Space Weather Users
• Prime Science Users– JSOC will support Virtual Solar Observatory (VSO) access– JSOC will also have a direct web access– There will be remote DRMS/SUMS systems at key Co-I institutions – JSOC In Situ Delivery and processing– Special Processing at JSOC-SDP as needed and practical
• Public Access– Web access for all data; Special products for E/PO and certain solar events
JSOC DRMS/SUMS Basic Concepts
• Each “image” is stored as a record in a data “series”.• There will be many series: e.g. hmi_ground.lev0 is ground test data• The image metadata is stored in a relational database – our Data Record
Management System (DRMS)• The image data is stored in SUMS (Storage Unit Management System)
which itself has database tables to manage its millions of files.• SUMS owns the disk and tape resources.• Users interact with DRMS via a programming language, e.g. C, FORTRAN,
IDL.• The “name” of a dataset is actually a query in a simplified DRMS naming
language that also allows general SQL clauses.• Users are encouraged to use DRMS for efficient use of system resources• Data may be exported from DRMS as FITS or other protocols for remote
users.• Several Remote DRMS (RDRMS) sites will be established which will
“subscribe” to series of their choice. They will maintain RSUMS containing their local series and cached JSOC series.
• The JSOC may act as an RDRMS to access products made at remote sites.
JSOC data organization• Evolved from FITS-based MDI dataset concept to
– Fix known limitations/problems– Accommodate more complex data models required by higher-level processing
• Main design features– Lesson learned from MDI: Separate meta-data (keywords) and image data
• No need to re-write large image files when only keywords change (lev1.8 problem)
• No (fewer) out-of-date keyword values in FITS headers
• Can bind to most recent values on export
– Easy data access through query-like dataset names• All access in terms of sets of data records, which are the “atomic units” of a data series
• A dataset name is a query specifying a set of data records (possibly from multiple data series):
– Storage and tape management must be transparent to user• Chunking of data records into “storage units” and tape files done internally
• Completely separate storage and catalog databases: more modular design
• Legacy MDI modules should run on top of new storage service
– Store meta-data (keywords) in relational database (PostgreSQL)• Can use power of relational database to rapidly find data records
• Easy and fast to create time series of any keyword value (for trending etc.)
• Consequence: Data records for a given series must be well defined (i.e. have a fixed set of keywords)
DRMS DataSeries
• A Dataseries consists of: – A SeriesName which consists of
• <projectname>.<productname>
– a sequence of Records which consist of a set of: • Keywords and • Segments which consist of:
– structure information and
– storage unit identifier
• Links which provide pointers to associated records in other series.
– A list of 0 or more PrimeKeys which are keywords sufficient to identify each record (default to “recnum”)
DRMS DataSeries - cont
• Data is accessed in RecordSets which are collections of records identified by seriesname and primekeys
• RecordSets are identified by a “name” which is really a query.
• Records may have versions which have the same set of primekey values, most recent is current record.
• See: http://jsoc.stanford.edu/jsocwiki/DataSeries
hmi.lev0_cam1_fg
Logical Data Organization
JSOC Data Series Data records for series hmi.fd_V
Single hmi.fd_V data record
aia.lev0_cont1700
hmi.lev1_fd_M
hmi.lev1_fd_Vaia.lev0_FE171
hmi.lev1_fd_V#12345
hmi.lev1_fd_V#12346
hmi.lev1_fd_V#12347
hmi.lev1_fd_V#12348
hmi.lev1_fd_V#12349
hmi.lev1_fd_V#12350
hmi.lev1_fd_V#12351
…
…
Keywords:RECORDNUM = 12345 # Unique serial numberT_OBS = ‘2009.01.05_23:22:40_TAI’DATAMIN = -2.537730543544E+03DATAMAX = 1.935749511719E+03...P_ANGLE = LINK:ORBIT,KEYWORD:SOLAR_P…
Storage Unit= Directory
Links:ORBIT = hmi.lev0_orbit, SERIESNUM = 221268160CALTABLE = hmi.lev0_dopcal, RECORDNUM = 7L1 = hmi.lev0_cam1_fg, RECORDNUM = 42345232R1 = hmi.lev0_cam1_fg, RECORDNUM = 42345233…
Data Segments:
Velocity =
hmi.lev1_fd_V#12352
hmi.lev1_fd_V#12353
JSOC Pipeline Processing System Components
Database Server
SUMSStorage Unit
Management System
DRMSData Record
Management SystemSUMS Tape Farm
SUMS Disks
Pipeline Program, “module”
Record Manage
ment
Keyword Access
Data Access
DRMS Library
Link Manage
ment
Utility LibrariesJSOC Science
Libraries
Record Cache
PUIPipeline User
Interface
Pipeline processing
plan
Processing script, “mapfile”
List of pipeline modules with needed datasets for input, output
Pipeline Operato
r
Processing History Log
Pipeline batch processing• A pipeline “session” is encapsulated in a single database transaction:
– If no module fails all data records are commited and become visible to other clients of the JSOC catalog at the end of the session
– If failure occurs all data records are deleted and the database rolled back– It is possible to commit data produced up to intermediate checkpoints during sessions
DRMS Service = Session Master
Input data records
Output data records
DRMS ServerInitiate session
…
DRMS API
Module N
DRMS ServerCommit Data
&Deregister
Pipeline session = atomic transaction
Record & SeriesDatabase
SUMS
DRMS API
Module 1
DRMS API
Module 2.2
DRMS API
Module 2.1
Analysis pipeline
Data export and centers, US and beyond; data import
ii. JSOC to SAO
http://jsoc.stanford.edu/netdrms/
First page
Remote DRMS/SUMS
• Cooperating sites run NetDRMS code which is the JSOC DRMS/SUMS code base.– They maintain their own PostgreSQL database– Remote systems can “subscribe” to series created at
other DRMS sites– Subscribed series DRMS records are synchronized
automatically with a short lag– SUMS Storage Units (SUs) which contain the file data
are imported on demand to the remote SUMS when a non-local sunum is requested.
– JSOC will serve all and will receive data from some remote sites.
Remote DRMS Sites
• Site Location Contact SUMS ID
• CFA Cambridge, MA, USA Alisdair Davey 0x0004• CORA Boulder, CO, USA Aaron Birch 0x0005• GSFC Greenbelt, MD, USA Joe Hourclé 0x0002• IAS Toulouse, France Frederic Auchere 0x0018• IIAP Bangalore, India Paul Rajaguru 0x000c• JSOC Stanford, CA, USA Art Amezcua 0x0000• JILA Boulder, CO, USA Deborah Haber 0x0008• LMSAL Palo Alto, CA, USA John Serafin 0x0023• MPI Katlenburg-Lindau, Germany Raymond Burston 0x0001• MSSL Dorking, UK Elizabeth Auden 0x0020• NSO Tucson, AZ, USA Igor Suarez-Sola 0x0003• ROB Brussels, Belgium Benjamin Mampaey 0x001d• Yale New Haven, CT, USA Charles Baldner 0x0010
Web Access
• JSOC page at http://jsoc.stanford.edu– Semantics see: Jsocwiki at
http://jsoc.stanford.edu/jsocwiki– Syntax for code see: Man Pages– Access for data see: e.g.
http://jsoc.stanford.edu/ajax/lookdata.html– Also links for CVS repository and trouble reports
First page
Work Remaining
• Oh, gee…• Web browsable catalog• Better “user experience”• Links to HKB• VSO provided SU availability catalog• …• Testing• Not to mention HMI analysis code…• Testing• Data from the Sky
6. Summary: HMI data for AIA science investigation
http://hmi.stanford.edu/Presentations/LWS-2007-TeamsDay/HMI_Dataproducts_Sept_2007.ppt
First page
Page 40 LWS Teams Day JSOC Overview
Primary goal: origin of solar variability
• The primary goal of the Helioseismic and Magnetic Imager (HMI) investigation is to study the origin of solar variability and to characterize and understand the Sun’s interior and the various components of magnetic activity.
• HMI produces data to determine the interior sources and mechanisms of solar variability and how the physical processes inside the Sun are related to surface and coronal magnetic fields and activity.
Page 41 LWS Teams Day JSOC Overview
Key Features of HMI Science Plan
• Data analysis pipeline: standard helioseismology and magnetic field analyses
• Development of new approaches to data analysis
• Targeted theoretical and numerical modeling
• Focused data analysis and science working groups
• Joint investigations with AIA and EVE
• Cooperation with other space- and ground-based projects (SOHO, Hinode, PICARD, STEREO, RHESSI, GONG+, SOLIS, HELAS)
Page 42 LWS Teams Day JSOC Overview
HMI Major Science Objectives1.B – Solar Dynamo
1.C – Global Circulation
1.D – Irradiance Sources
1.H – Far-side Imaging
1.F – Solar Subsurface Weather
1.E – Coronal Magnetic Field
1.I – Magnetic Connectivity
1.J – Sunspot Dynamics
1.G – Magnetic Stresses
1.A – Interior Structure
NOAA 9393
Far-side
Page 43 LWS Teams Day JSOC Overview
1. Convection-zone dynamics and solar dynamo– Structure and dynamics of the tachocline– Variations in differential rotation.– Evolution of meridional circulation.– Dynamics in the near-surface shear layer.
2. Origin and evolution of sunspots, active regions and complexes of activity– Formation and deep structure of magnetic complexes.– Active region source and evolution.– Magnetic flux concentration in sunspots.– Sources and mechanisms of solar irradiance variations.
3. Sources and drivers of solar activity and disturbances– Origin and dynamics of magnetic sheared structures and delta-type sunspots.– Magnetic configuration and mechanisms of solar flares and CME.– Emergence of magnetic flux and solar transient events.– Evolution of small-scale structures and magnetic carpet.
4. Links between the internal processes and dynamics of the corona and heliosphere– Complexity and energetics of solar corona.– Large-scale coronal field estimates.– Coronal magnetic structure and solar wind
5. Precursors of solar disturbances for space-weather forecasts– Far-side imaging and activity index.– Predicting emergence of active regions by helioseismic imaging.– Determination of magnetic cloud Bs events.
Primary Science Objectives
Page 44 LWS Teams Day JSOC Overview
HMI Science Analysis Plan
Magnetic Shear
Tachocline
Differential Rotation
Meridional Circulation
Near-Surface Shear Layer
Activity Complexes
Active Regions
Sunspots
Irradiance Variations
Flare Magnetic Configuration
Flux Emergence
Magnetic Carpet
Coronal energetics
Large-scale Coronal Fields
Solar Wind
Far-side Activity Evolution
Predicting A-R Emergence
IMF Bs Events
Brightness Images
Global Helioseismology
Processing
Local Helioseismology
Processing
Version 1.0w
Filtergrams
Line-of-sightMagnetograms
Vector Magnetograms
DopplerVelocity
ContinuumBrightness
Line-of-SightMagnetic Field Maps
Coronal magneticField Extrapolations
Coronal andSolar wind models
Far-side activity index
Deep-focus v and cs
maps (0-200Mm)
High-resolution v and cs
maps (0-30Mm)
Carrington synoptic v and cs
maps (0-30Mm)
Full-disk velocity, v(r,Θ,Φ),And sound speed, cs(r,Θ,Φ),
Maps (0-30Mm)
Internal sound speed,cs(r,Θ) (0<r<R)
Internal rotation Ω(r,Θ)(0<r<R)
Vector MagneticField Maps
Science ObjectiveData ProductProcessing
Observables
HMI Data
Page 45 LWS Teams Day JSOC Overview
HMI module status and MDI heritage
DopplerVelocity
HeliographicDoppler velocity
maps
Tracked TilesOf Dopplergrams
StokesI,V
ContinuumBrightness
Tracked full-disk1-hour averagedContinuum maps
Brightness featuremaps
Solar limb parameters
StokesI,Q,U,V
Full-disk 10-minAveraged maps
Tracked Tiles
Line-of-sightMagnetograms
Vector MagnetogramsFast algorithm
Vector MagnetogramsInversion algorithm
Egression andIngression maps
Time-distanceCross-covariance
function
Ring diagrams
Wave phase shift maps
Wave travel times
Local wave frequency shifts
SphericalHarmonic
Time series
Mode frequenciesAnd splitting
Brightness Images
Line-of-SightMagnetic Field Maps
Coronal magneticField Extrapolations
Coronal andSolar wind models
Far-side activity index
Deep-focus v and cs
maps (0-200Mm)
High-resolution v and cs
maps (0-30Mm)
Carrington synoptic v and cs maps (0-30Mm)
Full-disk velocity, sound speed, Maps (0-30Mm)
Internal sound speed
Internal rotation
Vector MagneticField Maps
MDI pipeline modules exist
Standalone production codes in use at Stanford
Research codes in use by team
Codes to be developed at HAO
Codes being developed in the community
Codes to be developed at Stanford
Primary observables
Intermediate and high level data products
Page 46 LWS Teams Day JSOC Overview
JSOC - HMI Pipeline
HMI Data Analysis Pipeline
DopplerVelocity
HeliographicDoppler velocity
maps
Tracked TilesOf Dopplergrams
StokesI,V
Filtergrams
ContinuumBrightness
Tracked full-disk1-hour averagedContinuum maps
Brightness featuremaps
Solar limb parameters
StokesI,Q,U,V
Full-disk 10-minAveraged maps
Tracked Tiles
Line-of-sightMagnetograms
Vector MagnetogramsFast algorithm
Vector MagnetogramsInversion algorithm
Egression andIngression maps
Time-distanceCross-covariance
function
Ring diagrams
Wave phase shift maps
Wave travel times
Local wave frequency shifts
SphericalHarmonic
Time seriesTo l=1000
Mode frequenciesAnd splitting
Brightness Images
Line-of-SightMagnetic Field Maps
Coronal magneticField Extrapolations
Coronal andSolar wind models
Far-side activity index
Deep-focus v and cs
maps (0-200Mm)
High-resolution v and cs
maps (0-30Mm)
Carrington synoptic v and cs
maps (0-30Mm)
Full-disk velocity, v(r,Θ,Φ),And sound speed, cs(r,Θ,Φ),
Maps (0-30Mm)
Internal sound speed,cs(r,Θ) (0<r<R)
Internal rotation Ω(r,Θ)(0<r<R)
Vector MagneticField Maps
HMI DataData ProductProcessing
Level-0
Level-1
Page 47 LWS Teams Day JSOC Overview
Magnetic Fields
StokesI,VFiltergrams
StokesI,Q,U,V
Full-disk 10-minAveraged maps
Tracked Tiles
Line-of-sightMagnetograms
Vector MagnetogramsFast algorithm
Vector MagnetogramsInversion algorithm
Line-of-SightMagnetic Field Maps
Coronal magneticField Extrapolations
Coronal andSolar wind models
Vector MagneticField Maps
Code: Stokes I,V,Lev0.5 V & LOS fieldJ. SchouS. Tomzcyk
Status: in development
Code: Stokes I,Q,U,VJ. SchouS. Tomzcyk
Status: in development
Page 48 LWS Teams Day JSOC Overview
Line-of Sight Magnetic Field
StokesI,VFiltergrams
Line-of-sightMagnetograms Line-of-Sight
Magnetic Field Maps
SynopticMagnetic Field Maps
Magnetic Footpoint Velocity Maps
Code: LOS magnetogramsJ. SchouS. TomzcykR. Ulrich (cross calib)
Status: in development
Code: LOS magnetic maps (project?)T. HoeksemaR. Bogart
Status: in development
Code: Synoptic Magnetic Field Maps T. HoeksemaX. ZhaoR. Ulrich
Status: in development
Code: Velocity Maps of Magnetic FootpointsY. LiuG. Fisher
Status: in development
Page 49 LWS Teams Day JSOC Overview
Vector Magnetic Field
Filtergrams StokesI,Q,U,V
Full-disk 10-minAveraged maps
Tracked Tiles
Vector MagnetogramsFast algorithm
Coronal magneticField Extrapolations
Vector MagneticField Maps
Code: fastrackR. Bogart
Status: needs modifications for fields
Code: Vector Field Fast and Inversion AlgorithmsJ. SchouS. Tomzcyk
Status: in development
Code: Vector Field MapsT. HoeksemaY.Liu
Status: in development
Code: Coronal Field Extrapolations &Ambiguity issue T.HoeksemaY.Liu, X.ZhaoC. SchrijverP.Goode T.MetcalfK.D.LekaStatus: in development
Code: Coronal Magnetic Field Topological PropertiesJ.LinkerV. Titov
Status: needs implementation
Code: Solar Wind ModelsX.ZhaoK.HayshiJ.LinkerP.GoodeV.Yurchishin
Status: in development
Coronal andSolar wind models
Vector MagnetogramsInversion algorithm
Need $$
Page 50 LWS Teams Day JSOC Overview
Intensity
Filtergrams ContinuumBrightness
Brightness Images
Code: Continuum MapsSchou
Status: in development
Code: Solar Limb Parameters, Lev0.5 used to make other Lev1 products, Lev2 for science goals
R. BushJ. Kuhn
Status: in development
Code: Brightness Feature Maps(European contribution)
Status: in development
Code: Averaged Continuum MapsBush
Status: in development
Solar limb parametersper image for Lev0.5
Brightness Synoptic Maps
Tracked full-disk1-hour averagedContinuum maps
Code: Brightness Synoptic MapsScherrer
Status: in development
Solar limb parameters
Brightness featuremaps