Accessing Raster Data in Rasdaman Data Cubes via R using a Shiny App and the CubeR Package
Mattia Rossi1,2, Daniel Frisinghelli1, and Alexander Jacob1
1 Eurac Research, Institute for Earth Observation, Bolzano, Italy ([email protected])2 Free University of Bolzano, Faculty of Science and Technology, Italy
R- Package CubeR
Click below to explore!
Shiny Web Interface
Background
Integration of in-situ data
Outlook
Two Minute Madness
PICO Presentation at European Geosciences Union General Assembly 2018, Vienna 08-13 April 2018
https://gitlab.inf.unibz.it/earth_observation_public/CubeR
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Oliver, Woodcock (2015). „Australian Geoscience Data Cube - CEOS WGISS 40“
Two Minute Madness IBack to Main Slide
Index
Baumann 2018: “Rasdaman – raster data manager (Version 9.5.0)”; rasdaman.org
During a research with extensive spatial data diverse problems have to be faced:
• Where to Store• Enough Storage• Data management and
standardization• Intermediate Results• Processing Power• Time Consumption
?
Multi-dimensional Arrays (often referred to as Data Cubes)
Principles:• Store Once• Process Once• Use many times• You only get what you need• Direct Processing
Standards:• Coverage Implementation Schema• Web Coverage Service• Web Coverage Processing Service
+ Efficient Data Management+ Scalable+ Processing Power
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Background
Two Minute Madness II
1. CubeR Package 2. Shiny Web App
• Create WC(P)S queries • Transform Server Response• Use Case Rasdaman Data Cubes
Back to Main Slide
Main Functionalities• Perform Access • Request Metainformation• Get Products • Perform Calculation
http://saocompute.eurac.edu/rasdaman/owshttps://gitlab.inf.unibz.it/earth_observation_public/CubeR
Combination with an in-situ Database PICO Screen 4.3
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Back to Main Slide
Background
Data Cubes?Advantages ofData Cubes ?
Interaction withData Cubes ?
EURACs implementation?
Click the red boxes to explore!
2 Minute Madness
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Spatial Data Cubes
Basic Facts about Data Cubes
• A multi-dimensional array significantly bigger than the main memory
• Used to store huge amounts of datasets from one or multiple sensors based on the extent and acquisition time
• Data values of same type sit at grid points
• Defined by d axes of the d-dimensional data cube
• Each grid point has two neighbors in each dimension
Back to Background
Oliver, Woodcock (2015). „Australian Geoscience Data Cube - CEOS WGISS 40“
• Spatio-temporal Information (e.g. Time Series of Raster Images) from one sensor stored in one cube.
• On an easiest scale the cube is three-dimensional (x, y, Time)
• Can increase dimensions (e.g. Bands)
From the Acquisition to a multi-dimensional array (Data Cube):
During a research project intermediate spatial results or final products could be stored together based on time and extent in one array of images
Giuliani et al. (2017) “Building an Earth Observations Data Cube: lessons learned from the Swiss Data Cube (SDC) on generating Analysis Ready Data (ARD)”
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Advantages of using Spatial Data Cubes
Lewis et al., 2017: “The Australian Data Cube – Foundations and Lessons Learned”
Schematic Hyperspectral Data Cube
Started mostly for the huge amount of data in hyperspectral images. The storage of several hundred spectral bands and multiple acquisitions over time in data cubes saves time and resources
Behman et al. 2014: “Detection of early plant stress responses in hyperspectral images”
Zhang et al. 2015: “Compression of hyperspectral remote sensing images by tensor approach “
Also the data acquired by multispectral satellites and sensors is augmenting and will continue to rise in the next years.
Storing time series of multispectral data in multi-dimensional arrays provides a powerful opportunity to centralize the storage, distribution and processing of data and to save resources and processing time as well as to increase the reproducibility of a research.Recently many solutions and standards emerged focusing on:
• Efficient Data Management• Scalability• Processing power
Back to Background
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Interaction with Data CubesData Cubes are accessed with predefinedstandards as the ones defined by theOGC:
Web Coverage Service (WCS)This service provides standardized access to gridded data represented in the coverage standard (CIS).
Web Coverage Processing Service (WCPS)This is an extension of the WCS standard that adds on the fly processing functionality to coverage retrieval request.
Coverage Implementation Schema (CIS)This standard describes a common schema of representation of collections of values distributed over space and time.
Many Data Cubes offer the possibility to directly interactwith and process data on a Data Cube. An example is theRasdaman DataCube exposing data via WC(P)S:
Get an Image at a given Time
Get a subset of an image
Get a SubsettedImage in a time
interval
Get value of a Pixel over Time
PerformCalculations
Back to Background
Trim Slice
Formula, e.g.
Baumann 2018: “Rasdaman – raster data manager (Version 9.5.0)”; rasdaman.org
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
EURAC Implementation
R - Package
Back to Background
CEPH_FS CEPH_RBD
Hardware
File Systems
Cluster Orchestration &
Virtual Environments
Applications
WCS
WCPS
SOS
Data Models &
Data Bases
1.4 PB
2 x 40 Gb/s336 Cores 3 TB RAM
~50 Cores ~200 MB RAM
e.g. pre -processingof S1 / S2
The following chart shows the implementation of the Rasdaman Data Cubes within the computing environment at Eurac Research.
The grey box suggests where the CubeR Package is usable.
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Back to Index
R-Package CubeR
The Accessibility of the Rasdaman implementation at EURAC, as provided by using WCPS queries, can be done throughout all programming languages. We focused on developing an R-Package to facilitate the access and to enhance the understanding of EO based Data Cubes for our research center. We named this Package “CubeR”
CubeR consists of 4 main pillars:
Discover Describe
Request/Return
Imports: httr
lubridate
magrittr
ncdf4
raster
sp
stringr
urltools
xml2
Process
Click the red boxes to explore!
Background
https://gitlab.inf.unibz.it/earth_observation_public/CubeR
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
DiscoverBack to R-Package
The access to the Rasdaman Database can contain multiple Data Cubes, referred to as „coverages“. The package automatically adds the necessary suffixes to the main Rasdaman URL (e.g. http://saocompute.eurac.edu/rasdaman/ows)
GetCapabilitiesReturns an XML File
getCapabilites()
Coverage1Coverage2...Coverage n
XML Parsing Get the Meta-
information
Return the Data
Capability of coverages stored at the moment*Sensor*_*Level*_*Tile*_*Resolution*
Depending on the type of coverage
Perform mathematical operations
Next Slides
Hosts various Coverages (often referred to as Data Cubes)
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Describe
After the access we added core functionalities to receive the metainformation of the coverages
The “Describe Coverage” command requires an XML File for each coverage of interest. We wrapped functions around this XML to return the relevant coverage information such as:
The Coverage is the input for eachMetainformation query(Here a Level-2A Sentinel 2 MSI Coverage)
All Function for findingmetainformation havethe prefix „coverage_get“
Back to R-Package
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Request/Return
Central Functions of the Package
• Create WCS queries
• Handle the Server Response
• Return the Tables, Images or Raster
Processing time of the three functions for 170 images (all Sentinel 2 Images of 1 Tile) takes roughly 30s (Depending on the connection to the Server)
Return Image Return Pixel over Time Return Area over Time
Back to R-Package
Baumann 2018: “Rasdaman – raster data manager (Version 9.5.0); rasdaman.org
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Process
Shiny App
Direct processing in the Rasdaman Data Cube environment.
For our own scientific goals we implemented a function allowing the computation of Normalized Difference between two bands so far.
NDVI one Image NDVI one Pixel
Products resulting could be gathered into a new Data Cube depending on processing complexity and - time
Back to R-Package
Baumann 2018: “Rasdaman – raster data manager (Version 9.5.0)”; rasdaman.org
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Back to Index
Shiny Web Application
Analogous to the Development of the Package we set up a web application based on the Shiny and Leaflet Package for near-real time visualization of requests. We used the functions described in the R-Package beforehand and added responsive panels for the single operations (Discovery, Description, Request, Response and Process) on the Data Cubes as well as a visualization with Leaflet in the Background.
The Shiny App has two main Windows:
Main Interface Pixel Plots
R Package
Click the red boxes to explore!
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
The Shiny Main Interface
Back to Shiny Web Application
The Main interface is divided in three main panels:
• Visualization PanelSelecting the Band combination based on the Coverage and a rescale factor (faster computation)
• Calculation PanelBasic Calculation on the Coverage(e.g. NDVI)
• Data Set SelectorSelect Dataset, Spatial Extent and Timestamp of interest
• Based on the Package Functions
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
The Shiny Pixel Plots
In situ Integration
Back to Shiny Web Application
The Pixel History tab has also three different Functionalities:
Pixel CoordinatesPanel to insert the exact location of one Pixel
BandsSelect the Bands of interest in base of the coverage selected in the previous window
Normalized DifferenceOn the Cube calculation of the Normalized Difference Indexes
For now the plots are with the base utility since it seems to be the fastest among packages
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Outlook
Back to Index
Integration of in-situ data
Shiny App
At Eurac Research we also implemented an SOS based Database for Point observations at fixed stations within the MONALISA Project.
For handling the Data from this Database we created a smaller second package MonalisR aimed at accessing the SOS API directly.
With both these packages it is possible to combine Data Cubes and Station Data information
https://gitlab.inf.unibz.it/earth_observation_public/MonalisR
In Situ NDVI + Sentinel 2 Pixel NDVI Values (MONALISA Station)
Processing time for all images from 2015 – 2017 and 2016 In-situ NDVI in 15min resolution, combination and plot ~5sec (Depending on the connection)
Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]
Back to Index
Outlook
In situ integration
Future Developments of the Package• The package development is in first stable version (v.1.0)• Further WCPS queries based on the needs of the community/researchers. The base computation of a few spectral indices should be
extended to more difficult mathematical operations• Further use cases within the Sentinel Alpine Observatory research (http://sao.eurac.edu/)
• Vegetation Stress (Drought) detection• Mosaicking
• Highly dependent on the use cases and your interest!
Future Developments in Data Cube Technology at Eurac Research• Organize also Climate Data to easily combine with EO-Data• Closer integration of data and metadata• Discovery looking into “OpenSearch”• Federate and integrate with other Platform
• OpenEO (http://openeo.org/)• Common interface to various other Backends
Thank you for your interest in this PICO Presentation at EGU2018!