Date post: | 30-Dec-2015 |
Category: |
Documents |
Upload: | terence-bruno-barber |
View: | 220 times |
Download: | 1 times |
2
Agenda
TESS Ground System Architecture MAST Operations Overview Interfaces MAST design
Archive System Archive User Interface
Schedule Testing plan Deliverables
IT Security Document Status Forward Work
4
MAST Overview
300 TB – Current size archive holdings. 3 TB – Average Ingest Rate per month. 19 TB – average data distribution rate per month. 1,5000,000 – Average number of searches per month.
5
TESS Raw Data
Data types Kepler:
• long cadence target – 150,000 targets, 32 pixels/target, 30 minutes cadence
• short cadence target – 300 targets, 85 pixels/target, 1 minute cadence• full frame image – one per month
TESS: • Target – 25000 targets/month, 100 pixels/target, 1 minute cadence• Full Frame Image – 30 minute cadence
TESS raw data volume: 6600 GB/ year (uncompressed) Target data: 3500 GB/year Full Frame Image data: 3100 GB/year
Kepler raw data volume: 380 GB/year (uncompressed) Long cadence: 293 GB/year Short cadence: 84 GB/year FFI: 2.3 GB/year
6
Archive Operations
TESS monthly processing (Kepler quarterly) Transferred from SPOC via hard drive
TESS archive data volume – 50 TB/year (to be refined) Total Kepler archive data volume – 19 TB (Q0-Q17)
• Does not consider reprocessed data
No proprietary period on data Availability depends primarily on SPOC processing
time (<1 month) Primary TESS web site hosted by MIT
Links to MAST web site
7
MAST Interfaces
SPOC
TSO
MAST
Target Pixel Data
Full Frame Images
Collateral Data
Light Curves
Centroid Data
Cotrending Basis Vectors
Threshold Crossing Events
Data Validation Reports
TESS Objects of Interest Catalog
S/c Data (from POC)
Threshold Crossing Events
Data Validation Reports
8
TESS Archive Data Products from SPOC
Full Frame Image Target Pixel Files
All pixels for a single target Raw and calibrated values
Light Curves Output of photometric analysis and cotrending
Collateral Pixel Files Calibration data collected on-board
Ground calibration data Cotrending Basis Vectors
• Systematic trends present in the ensemble of flux data for each CCD
Pixel Response Function• Point spread function and pointing and electronic systematics
Flat Field• Pixel response non-uniformity map
9
TESS Data Products from TSO
TESS Input Catalog (TIC) Threshold Crossing Events (TCE) summary report
Identified from light curves Data Validation (DV) results
Characterize and validate TCEs TESS Objects of Interest (TOI) Catalog Follow-up observations
10
Archive System Overview
query results
queries
Ingest
Distribution
Archive User Interface
data files
data files
catalog updates
queriesquery results
data request
quer
ies
quer
y re
sults
Archive user
catalog updates
data
base
repl
icati
on
data files
Storage Broker
Data Receiptmanifest
Primary data store
Safe store
On-line cache
data files
data files
data request
science metadata
file informationdata files
Archive Catalog
Archive Catalog
11
Archive Functional Overview (1/3)
Data Receipt Receives data from the SPOC Receives data from the TESS Science Office Accepts data and generate an archive manifest
Ingest Stores TESS data in the Storage Broker Catalogs TESS metadata the Archive Database
Storage Broker Provides a storage service manager for the file I/O on the primary, secondary safestore, and public data stores
12
Archive Functional Overview (2/3)
Archive Catalog Provides an on-line science catalog of TESS
metadata for searches, data mining, and data retrieval
• Implemented on relational DBMS
• Contains metadata extracted from the science data keyword values
Stages TESS catalogs for archival use Data Distribution
Retrieves requested data from storage broker Distributes data to archive users via electronic
mechanisms or removable media Archive Operator Interface
Maintains an operator interface for monitoring and control of the archive system
13
Archive Functional Overview (3/3)
Archive User Interface Makes TESS data available to the astronomical community
• Web-based query/preview/retrieval interface
Hosts software tools required to search the TESS archive catalog
• This includes user access to target lists.
Hosts software tools required to access data
Provides data mining, visualization, and correlation with other major mission datasets
• International Virtual Observatory Alliance (IVOA) standards compliant
14
Archive Catalog
Three main types of database tables File tracking tables – What datasets and files
exist, the file/dataset status, and the relationship between them.
Instrument and data specific tables Common Archive Observation Model (CAOM)
Database
15
Common Archive Observation Model (CAOM)
Unified model for metadata about observation (position, time, wavelength) implemented in a database enabling easy uniform cross-mission searches
Extensible for mission-specific data May require additional static data not in FITS file
Enables easy creation of Virtual Observatory (VO) services VO allows s/w access to data holdings
Current version enabled as part of the MAST Data Discovery or Portal Interface
16
Data Distribution Options
Electronic mechanisms Streamed through browser Data delivered to MAST staging area for user down load via browser or FTP
Data delivered directly to user disk via SFTP Wget and cURL scripts created and delivered to user
Removable media: Hard drive, DVD Archive Operator has the ability to override specified
user preference.
17
Archive Distribution Performance
MAST has a requirement to be capable of distributing 2.5 TBytes of archived science data products externally and 9.4 TBytes internally per month.
MAST is implementing scalable hardware and networking technologies, with high-bandwidth ISP to the internet to handle increased loads.
Historic data distribution metric includes internal transfers and uncompressed volumes. Internal data transfers do not use ISP bandwidth. JWST and TESS estimates include internal transfers.
18
Archive User Interface: Overview
Archive User Interface (AUI) provides ability to: Search for data products by:
• Data file name• Target name, position or list of positions• Other observation parameters (instrument, waveband, program ID,
etc.)
Explore and discover data through the Data Portal Request retrieval of those products
26
Archive Development Schedule
Phase B (April 1, 2014 – October 1, 2014 ) Continue developing TESS Archive Operations Concept Continue developing TESS-MAST ICD
• Work with SPOC to develop SPOC-MAST ICD section• Work with TSO to develop TSO-MAST ICD section
Continue to participate in Ground Segment Monthly Technical Reviews
Participate in TESS Mission PDR (September 2014) Phase C (October 2, 2014 – November 29, 2016)
Develop TESS specific archive components based on Archive Operations Concept and TESS-MAST ICD
Participate in TESS Mission CDR Participate in Ground Segment testing
Phase D (November 30, 2016 – July 17, 2017) Develop TESS Archive User’s Guide
27
query results
queries
Ingest
Distribution
Archive User Interface
data files
data files
catalog updates
queriesquery results
data request
quer
ies
quer
y re
sults
Archive user
catalog updates
data
base
repl
icati
on
data files
Storage Broker
Data Receiptmanifest
Primary data store
Safe store
On-line cache
data files
data files
data request
science metadata
file information
data files
Archive Catalog
Archive Catalog
Archive System Enhancements for TESS
TESS mission specific s/w
28
Archive Development Effort
Staff Phase(s) Total FTE Avg. FTEProgram Manager B-D 2.0 0.050Project Manager B-D 1.4 0.035Systems Engineer B-D 5.4 0.14Archive Database Engineer C-D 3.6 0.12Archive Software Engineer C-D 3.6 0.12Archive Scientist D 1.2 0.11Archive System Admin. C 0.01 0.01
Phase B: April 1, 2014 – October 1, 2014 Phase C: October 2, 2014 – November 29, 2016Phase D: November 30, 2016 – July 17, 2017
29
Testing Plan
MAST will work with the SPOC and TSO to develop test data.
MAST will participate in Ground Segment end-to-end tests.
30
MAST Deliverables
Deliverable Responsibility Phase Date of DeliveryTESS Archive Operations Concept Systems Engineer B PDRTESS-Archive ICD
SPOC-ArchiveTSO-Archive
Systems Engineer B/C PDR/CDR
Archive Catalog Design Specification Systems Engineer B/C PDR/CDRArchive Architecture Systems Engineer B/C PDR/CDRArchive Software Design Systems Engineer C CDRTESS Archive User Interface Software Engineers C/D GS TestArchive Test Procedures Systems Engineer D GS TestArchive Operations Procedures Archive Operator D LaunchTESS Archive User’s Guide Archive Scientist E Launch
31
IT Security
MAST will protect all TESS data against unauthorized access per requirements established by NPR 2810.1, Security of Information Technology.
STScI is moving towards becoming a NASA external system STScI will have a single Moderate system Security
Plan Compliance with FISMA and NPR 2810.1A still
required
32
Document Status
TESS Archive Operations Concept Initial draft available dated June 12, 2014
TESS-MAST ICD SPOC-MAST ICD
• Initial draft available
TSO-MAST ICD• Initial draft available by PDR
33
Forward Work
Systems Engineering Complete TESS Archive Operations Concept Work with SPOC to complete SPOC-MAST section
of TESS-MAST ICD Work with TSO to complete TSO-MAST section of
TESS-MAST ICD Prepare for PDR
Archive Software Development Data Receipt and Ingest Archive Catalog Archive User Interface