U.S. Department of the InteriorU.S. Geological Survey
ASPRS Hurricane Katrina ASPRS Hurricane Katrina Lessons LearnedLessons Learned
Brenda Jones and Ron Brenda Jones and Ron RistyRistySAIC*SAIC*
USGS Center for Earth Resources Observation and ScienceUSGS Center for Earth Resources Observation and Science*Work performed under U.S. Geological Survey contract 03CRCN0001*Work performed under U.S. Geological Survey contract 03CRCN0001
EROSCenter for Earth Resources Observations and Science
Our MissionOur MissionScience:Science:To promote and conduct applications, users, and knowledge of land information to better understand our planet
Data Access:Data Access:To ensure that scientists, researchers, businesses, decision makers, and the public have ready access to land information
Data Archives:Data Archives:To safeguard and expand the national archive of remotely sensed land data
Current Data Ingest FlowsData Supported - Restricted
IKONOSQuickBirdOrbimageSPOTRadarsatIRSHomeland Security Infrastructure Program dataEnvisatGIS DataGraphicsDisaster Monitoring Constellation
Data Supported - PublicLandsatAsterHyperionAliAerial data
DOQNAIPNOAA AerialUSACE/FEMA
NEDNHDDeclass DataLidarGIS DataGraphics
Quickbird/Ikonos Data Flow
Aerial Data Flow
Aster Data Flow
SPOT Data Flow
Current statusEROS is investigating the re-engineering of its Emergency Response data ingest routines
AutomationShould this task be distributedHow much can vendors provideNeed users to define requirements
Convert data to standard and useable format Create browse
Provided in some cases, but inconsistentControlled and reliable format if automated
Create compressed data
Current status (continued)
Shapefiles/KML files EventData TypeExport files by area of interest or extent
Metadata Requirements for CSV, XML, or EXCEL spreadsheet Define minimal FGDC compliant metadata
Package data for deliveryProvide tiered web access capabilities
Data format requirementsDelivery data format
Datatype - 8 bit, 11 bit, 16 bitFormat – GeoTiff, NITF, flat binaryProcessing level – raw, geo-referenced, orthoBands – color, NIR, pan, merged, everythingOthers?
Browse requirements
Browse examples and analysisBrowse generated from the 8-bit image with consistent resolution down-sampling for each datasetSome Imagery is not consistent size (X by Y pixels; real estate)
Therefore, browse will also be various sizesGoals:
Fit on users screenFile sizes reduced for quick viewsRetain reasonable resolution for minimal detail reduction while earning benefits of a browse
Browse requirementsAnalyze browse for usability
What browse are needed?Multi-spectral true color Pan bandNIR
Previously made available any browse packaged with the productNow recommending we create the browse in most casesWhat is appropriate data size reduction?Determine common acceptable format for each dataset
Quickbird Browse ExamplesRGB Multi-spectral bands true color browse:
Notice the slight size differenceBoth have same resolution down-sampling applied
This is about 61m resolution
IKONOS Browse Examples
RGB Multi-spectralbands true colorbrowse:
Notice the sizedifferenceBoth have sameresolutiondown-samplingapplied
This is about 80mresolution
Landsat Browse Examples
720m true colorbrowse
Simply reduced size(and resolution) ofexisting browse createdby the EROS LAMsystem
Aerial PhotographyBrowse Examples
500x572 pixels
Compression requirements
Type of compressionMrSIDJpegJpeg2000ECW
Analyze statistics for acceptable compressionDetermine compression that is acceptable for all
Can automate if a base compression that will work for all is foundIf other options for compression are needed, may be able to provide real-time, but this introduces wait time for users that must be balanced
Compression overview
MrSID compression analysisLossy vs. lossless examples; quality loss analysis8-bit vs. native pixel depthStatistics regarding:
File size reductionTime to compress (real-time generation viability)Are World files desired with the Sid images?
Mr Sid Statistics
Both resolution reductions creates browse generally <1% of the original GeoTIFF full resolution browse file size
Quick download time to a user interface viewerExample scenes file sizes:
Image Type
Original product size
Mr Sid Lossless size
Encoding Ratio
EncodingTime
Aerial 323 M 187M16M4MB5.8M
2M 0SAerial 323 M
Lossless (1.71:1)Lossy (20:1)Lossy (20:1)
6M 4SQuickbird 81 M 1M 29S
Lossy (20:1)Ikonos 239M 1M 23S
Aerial PhotographyMr Sid Examples
Original Image Mr Sid Lossy
Aerial photography – lossy Mr Sid
Original Image – 4x Lossy – 4x
Lossy – 8xOriginal Image – 8x
Aerial Photography – Mr Sid
Original Image Mr Sid LossLess
Aerial Photography – lossless Mr Sid
Original Image – 4x Lossless – 4x
Original Image – 8x Lossless – 8x
Mr Sid results
Lossy vs LosslessImagery looked very similarSize of data
Lossless size approximately ½ size of original imageLossy – 20 times smallerSmaller size may be more advantageous for first responders
Encoding timeVarious between 1-5 minutes per imageShould be able to include this into the normal production flow as an additional product
Requirements for Managing DataEach data type will require a unique index to reference the image
2 Character data type 6 Character K/J or Path/RowYYYY - YearMM - MonthDD – DayHH -- HourMM – MinuteSS – SecondP/M – Pan or Multispectral+/-ddd_ddd+/-ddd_ddd - centerpoint00 – Unique number for duplicates or frames covering the same area
Requirements for Managing Data
Additional processing may be requiredMosaicsPan mergesOrthorectificationGeo-referencingOthers
The data will need to be grouped for downloading
TarTar-gzZip
Requirements for Managing DataMake data available for FTP and HTTP downloadGenerate status graphics
Jpgspdfshapefileskml
Pull or copy all data by event for customersCopy data to firewire or other media for delivery to customersCapture metadata and generate minimal FGDC metadataDump metadata in a comma delimited, csv, or XML format
Requirements for Managing DataStore the metadata in a database for managing the dataStop light web page to understand the current data flow, predicted taskingsCustomer notification of new data (email, web page, RSS service)Viewer integration…GIS files tied to the event
Data Management requirementsArchiving
Data will move from the disk cache to 9840 tape on siloAfter 60 days of no use, the data will be moved to the long term archiveData received on firewire will also be added to the long term archive strategy
Event specification requirementsName of eventLat/Long extentDate of the eventAny data loaded to the archive will be related to this event based on the date loaded and the extent
Data download requirements
Users unable to download data with current flows – need user requirements
Need to determine bottleneckSilo?Users network?Size of image?User interface?Others?
Data download requirements
Tiered web accessSimple web page with different access capabilities
ShapefileKML fileNon-ims indexmapIMS lite (vector based) with a tile toolFull IMS
Google EarthOthers?
Contact info
Brenda JonesWork – 605.594.6503Cell – 605.321.3995Email – [email protected]
Ron RistyWork – 605.594.6969Cell – 605.321.3698Email – [email protected]
Questions?Questions?