Date post: | 22-Jan-2018 |
Category: |
Technology |
Upload: | bigdataeurope |
View: | 395 times |
Download: | 0 times |
1ST BDE SC5 PILOT:RATIONALE, COMPONENTS AND REUSABILITYNCSR–Demokritos11/10/2016
Overview
¥ Downscaling¥ NetCDF, big data and abstraction¥ BDE SC5 pilot #1 functionality¥ Next steps
18-oct.-16
Downscaling¥ Downscaling of climatic and / or meteorological data:
o Essential first step for any further analysis, assessment or processing in climate and related domains
NetCDF Format, Model, Abstraction¥ Numerical array data format¥ Embedded metadata / variables, attributes¥ Dimensions¥ De facto standard for climate, weather and other Earth
observation datao ESGFo Australia’s National Environmental Research Data Interoperability
Platform (NERDIP)¥ Transparent big data connectors to move from and to
NetCDF format and file abstractions18-oct.-16
WRF: Weather Research and Forecasting Model
¥ Widely used and available¥ Operational forecasting and atmospheric – weather
and climate – research ¥ Open source / public domain
18-oct.-16
Specification¥ Supplement climate research community with big
data technologyo Discrepancy between big-data and data-intensive
advances and research practiceo Rigid policies at research sites – need a more flexible
approach to technology¥ To be used in conjunction with institutional
infrastructure already in use18-oct.-16
BDE SC5 Pilot I Components
CassandraMetadata & data lineage
Hive/HadoopRaw data &
analytics
WRF ModelInstitutional resource
connectors
NetCDFInterfacing and transformation, Semagrow tools
SC5 1st Pilot
Operations Implemented¥ Operations
o Data ingestion (NetCDF files)v Both manually, for bootstrapping, as well as after downscaling
o Data export (NetCDF files)v Selection of variables / time slices
o Start and monitor WRF-based downscaling on institutional resourcesv If requested results already exist, they are retrievedv If not, WRF is started
o Maintain data lineage records on BDE platformv Monitoring and further analysis v Subset of W3C PROV, http://www.w3.org/TR/prov-overview18-oct.-16
Sample Analytics ¥ Climate-change indices / analytics (indicative)
o Number of summer days, frost days o Tropical nights o Monthly minimum value of daily maximum temperatureo Precipitation-based statistics
¥ Analytics for other applicationso Comfort indices (temperature, humidity)o Risk for forest fires (wind speed, temperature, humidity)o Atmospheric pollution (wind speed, vertical gradient of temperature, heat
fluxes)18-oct.-16
Hangout and Evaluation
¥ Carried out an online hands-on and evaluation session (12 July 2016)
¥ Python UI for components¥ Most promising components warranting further
future development:o Tools to enable analyticso Data lineage
18-oct.-16
Conclusions and Next Steps¥ Big data technologies can aid climate and weather
research o Advances on climate research feeds into a number of
societal challenges and areas of interest¥ Data abstraction and data lineage are generic
components which will enable further progresso We may contextualise and investigate further during
the 2nd pilot18-oct.-16