A procedure to manage open access data for post-processing in GIS environment DataBases (DB) are a widespread source of data, useful for many applications in different scientific fields. The present contribution describes an automatic procedure to access, download and store open access data from different sources, to be processed in a GIS environment. In particular, it refers to the specific need of the authors to manage both meteorological data (pressure and temperature) and GNSS (Global Navigation Satellite System) Zenith Total Delay (ZTD) estimates. Such data allow to produce Precipitable Water Vapor (PWV) maps, thanks to the so called GNSS for Meteorology(G4M) procedure, developed through GRASS GIS software ver. 7.4, for monitoring in time and interpreting severe meteorological events. Actually, the present version of the procedure includes the meteorological pressure and temperature data coming from NOAA’s Integrated Surface Database (ISD), whereas the ZTD data derive from the RENAG DB, that collects ZTD estimates for 181 GNSS Permanent Stations (PSs) from 1998 to 2015 in the French-Italian boundary region. Several Python scripts have been implemented to manage the download of data from NOAA and RENAG DBs, their import on a PostgreSQL/PostGIS geoDB, besides the data elaboration with GRASS GIS to produce PWV maps. The key features of the data management procedure are its scalability and versatility for different sources of data and different contexts. As a future development, a web-interface for the procedure will allow an easier interaction for the users both for post-processing and real-time data. The data management procedure repository is available at https://github.com/gtergeomatica/G4M- data PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.27227v1 | CC BY 4.0 Open Access | rec: 19 Sep 2018, publ: 19 Sep 2018
Transcript
A procedure to manage open access data for post-processingin GIS environment
DataBases (DB) are a widespread source of data useful for many applications in different
scientific fields The present contribution describes an automatic procedure to access
download and store open access data from different sources to be processed in a GIS
environment In particular it refers to the specific need of the authors to manage both
meteorological data (pressure and temperature) and GNSS (Global Navigation Satellite
System) Zenith Total Delay (ZTD) estimates Such data allow to produce Precipitable
Water Vapor (PWV) maps thanks to the so called GNSS for Meteorology(G4M) procedure
developed through GRASS GIS software ver 74 for monitoring in time and interpreting
severe meteorological events Actually the present version of the procedure includes the
meteorological pressure and temperature data coming from NOAArsquos Integrated Surface
Database (ISD) whereas the ZTD data derive from the RENAG DB that collects ZTD
estimates for 181 GNSS Permanent Stations (PSs) from 1998 to 2015 in the French-Italian
boundary region Several Python scripts have been implemented to manage the download
of data from NOAA and RENAG DBs their import on a PostgreSQLPostGIS geoDB besides
the data elaboration with GRASS GIS to produce PWV maps The key features of the data
management procedure are its scalability and versatility for different sources of data and
different contexts As a future development a web-interface for the procedure will allow
an easier interaction for the users both for post-processing and real-time data The data
management procedure repository is available at httpsgithubcomgtergeomaticaG4M-
data
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
A procedure to manage open access data1
for post-processing in GIS environment2
Lorenzo Benvenuto12 Ilaria Ferrando2 Roberto Marzocchi12 Bianca3
Federici2 and Domenico Sguerso24
1Gter srl Innovazione in Geomatica Gnss e Gis (University of Genoa Spin-Off company)5
ndash Piazza De Marini 361 16123 Genoa (Italy)6
2Department of Civil and Environmental Engineering (DICCA) University of Genoa Via7
Montallegro 1 16145 Genoa (Italy)8
Corresponding author9
Lorenzo Benvenuto1210
Email address lorenzobenvenutogterit11
ABSTRACT12
DataBases (DB) are a widespread source of data useful for many applications in different scientificfields The present contribution describes an automatic procedure to access download and store openaccess data from different sources to be processed in a GIS environment In particular it refers tothe specific need of the authors to manage both meteorological data (pressure and temperature) andGNSS (Global Navigation Satellite System) Zenith Total Delay (ZTD) estimates Such data allow toproduce Precipitable Water Vapor (PWV) maps thanks to the so called GNSS for Meteorology (G4M)procedure developed through GRASS GIS software ver 74 for monitoring in time and interpreting severemeteorological events Actually the present version of the procedure includes the meteorological pressureand temperature data coming from NOAArsquos Integrated Surface Database (ISD) whereas the ZTD dataderive from the RENAG DB that collects ZTD estimates for 181 GNSS Permanent Stations (PSs) from1998 to 2015 in the French-Italian boundary region Several Python scripts have been implementedto manage the download of data from NOAA and RENAG DBs their import on a PostgreSQLPostGISgeoDB besides the data elaboration with GRASS GIS to produce PWV maps The key features of thedata management procedure are its scalability and versatility for different sources of data and differentcontexts As a future development a web-interface for the procedure will allow an easier interaction forthe users both for post-processing and real-time data The data management procedure repository isavailable at httpsgithubcomgtergeomaticaG4M-data
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
INTRODUCTION30
This work is part of the research activities conceived by the Geomatics Laboratory of the University31
of Genoa to detect severe meteorological events using the remote sensing of the water vapour content32
in the atmosphere through GNSS (Global Navigation Satellite System) signals (Bevis et al 1992)33
A GIS (Geographic Information System) procedure called G4M (GNSS for Meteorology) has been34
conceived in order to produce 2D Precipitable Water Vapour (PWV) maps with high spatial and temporal35
resolution (Ferrando et al 2018) The developed procedure is able to use observations coming from36
existing monitoring infrastructures of pressure P temperature T and GNSS Zenith Total Delay (ZTD)37
not necessarily co-located and distributed over an orographically complex area In G4M procedure the38
orographic effect influencing PWV maps is reduced by means of a time differentiation of PWV maps with39
respect to a calm period meaning a period of quiet before a storm thus with the realization of ∆PWV40
maps Furthermore an index called Heterogeneity Index (HI) accounting the ∆PWV spatial variability41
has been conceived as a promising indicator to detect severe meteorological events in space and time42
A scheme of the G4M procedure is depicted in figure 143
44
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 1 Scheme of the G4M procedure
In order to automate the procedure to create PWV ∆PWV and HI 2D maps G4M has been recently45
updated using Python as script language GRASS 74 (Neteler et al 2012 GRASS Development Team46
2018) to perform raster interpolation while 1D P T and ZTD data are stored in a PostgreSQLPostGIS47
geodatabase48
This work is intended to describe data collection organization and storage which are the essential49
preliminary phases to launch the G4M procedure First of all the locations of GNSS Permanent Stations50
(PSs) and meteorological stations are uploaded on the PostgreSQLPostGIS geoDB Three boolean (true-51
false) fields have been added to allow users to decide whether use or not the measuring stations depending52
on data availability The monitoring data (ZTD T and P) are uploaded on the PostgreSQLPostGIS53
geoDB using the script described in the following section Hence the main data management workflow54
illustrated in figure 2 can be summarized as follows55
- the computational region is set according to the Digital Terrain Model (DTM) boundaries while the56
resolution is set as defined by the user57
- starting and ending time of elaboration and the time step between two consecutive maps are set by58
the user59
- for each time the geoDB is queried to verify the presence of ZTD T and P data (if the corresponding60
boolean field is set to true)61
- the existing data are elaborated to produce 2D high resolution PWV ∆PWV and HI maps62
Thanks to the GRASS GIS time series modules and all the available Python libraries (time psycopg263
grass-script etc) the management of time series data is made easier if compared with other GIS software64
or programming languages and it allows a straightforward realization of time series plots and analysis65
27
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 2 Workflow of G4M procedure
The following sections describe the techniques used to download data from existing monitoring66
infrastructures and upload them into a spatial database specifically designed particular attention is given67
to the Python scripts conceived to automate these processes68
DATA69
The studied area covers the French-Italian neighbouring area In this area GNSS (ZTD data) and70
meteorological (P and T data) stations are differently distributed71
Available GNSS PSs come from different networks such as the International Gnss Service (IGS)72
Tracking network the EUropean REference Frame (EUREF) Permanent Network and finally also73
national transnational and regional ones In the present work GNSS data come from 181 GNSS PSs with74
an average spacing of 30-40 km acquisition rate is 30rdquo in accordance to an international standard75
ZTD were estimated splitting the whole network in 3 sub-net according to the decreasing station age76
(net1 net2 and net3 respectively) to obtain stable solutions77
A database 1 (RENAG DB) containing two-hourly ZTD estimates from 1998 to 2012 was created78
and made available to researchers (Sguerso et al 2013) Recently it was updated to include data up to79
December 2015 (Sguerso et al 2015) P and T data come from existing network of NOAA (National80
Oceanic and Atmospheric Administration) with an average spacing of sim100 km Meteorological data are81
available through CDO2 service which provides free access to NCEI3 of global historical weather and82
climate data P and T stations are not always co-located because not all the stations measure both P and T83
or because of occasional lack of data Figure 3 shows the GNSS PSs and meteorological stations falling84
in the area of interest85
1ftprenagunicefrproductsGPS climatology Sguerso Labbouz Walpersdorf2Climate Data Online httpswwwncdcnoaagovcdo-web3National Centers for Environmental Information httpswwwnceinoaagovarchive
37
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 3 GNSS PSs in the working area For the GNSS PSs the division in sub-networks ishighlighted net1 net2 and net3 are represented with red blue and green dots respectively whereas the 15common PSs are represented with yellow dots P and T stations are reported with cyan and purple squaresrespectively
To describe the terrain elevation the Global Digital Elevation Model (GDEM) of ASTER space86
project (joint product of METI Ministry of Economy Trade and Industry of Japan and NASA National87
Aeronautics and Space Administration httpasterwebjplnasagovgdemasp) is used88
METHOD89
The G4M procedure computes 2D PWV maps starting from 1D ZTD P and T observations exploiting90
open-sources software GRASS GIS 74 To support a near future web application of G4M the 1D input91
data are stored in a geodatabase specifically designed using PostgreSQL+PostGIS geodatabase which is92
highly compatible with GRASS 74 Moreover the procedure originally in Fortran has been re-written93
in Python a more modern programming language that has many libraries to interact with PostgreSQL94
GRASS and temporal series of data allowing to obtain a more automate procedure In this section the95
Python scripts conceived to automatically upload ZTD P and T data to the geodatabase are exposed All96
the scripts are available at httpsgithubcomgtergeomaticaG4M-data97
Python script for ZTD98
ZTD data have a temporal step of 2 hours and initially are saved into 3 different folders named net1 net299
and net3 according to the already mentioned division into subnets Every folder contains a file for each100
stations with ZTD estimates and the relative RMS (Root Mean Square) The files are named in a proper101
way which describes the station name the year of observation and the network in which the station is102
inserted (eg AJAC2012net1az)103
The Python script conceived for uploading data into the geodatabase is named ztd2postgresqlpy It104
reads every line of a specified file and puts the data (ie year day hour ZTD and RMS) in the proper105
column of a table in the database A scheme of the ztd2postgresqlpy script is shown in figure 4 To106
upload all the data contained into the RENAG DB to the geodatabase this procedure has to be executed107
47
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
for every station and for every year of observation To speed up the upload process a bash script has been108
conceived to automatically execute the ztd2postgresqlpy script for every file in a folder109
Figure 4 A scheme of the script to upload ZTD data into the geodatabase
Python script for P and T110
P and T data can be downloaded from NOAA in two way 1) in the ISD-lite format from the website4 2)111
manually from NOAA website5 specifying the time period and a list of desired stations This second112
downloading way is not easy to automate and highly time consuming for this reason in this work the113
meteorological data have been downloaded in ISD-lite format114
Files in this format contain a fixed-width formatted subset of the complete Integrated Surface Data115
(ISD) for a selected number of observations The data are typically stored in a single file corresponding to116
the ISD data ie one file per station per year117
To download data from NOAA and upload them into the geoDB in an automatic and fast way a118
Python script called noaa2postresql ftppy has been implemented This script needs two inputs a year119
and the station id While the year must be specified by the user the stations ids are automatically read120
from the database using a query Having this two inputs it is possible to download the files containing P121
and T data for all the stations in the database and for the specified year The downloaded files are saved122
in compressed folders so the following step is the extraction and read of those files Finally data are123
uploaded to the geoDB and all the downloaded file are removed from the local computer A scheme of the124
noaa2postgresql ftppy script is shown in figure 5125
Sguerso D Labbouz L and Walpersdorf A (2013) 14 years of GPS tropospheric delays in the French-153
Italian border region a data base for meteorological and climatological analyses The International154
Archives of the Photogrammetry Remote Sensing and Spatial Information Sciences XL-5W37ndash14155
67
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Sguerso D Labbouz L and Walpersdorf A (2015) 14 years of GPS tropospheric delays in the156
FrenchndashItalian border region comparisons and first application in a case study Appl Geomat 8(1)1ndash157
13158
77
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
A procedure to manage open access data1
for post-processing in GIS environment2
Lorenzo Benvenuto12 Ilaria Ferrando2 Roberto Marzocchi12 Bianca3
Federici2 and Domenico Sguerso24
1Gter srl Innovazione in Geomatica Gnss e Gis (University of Genoa Spin-Off company)5
ndash Piazza De Marini 361 16123 Genoa (Italy)6
2Department of Civil and Environmental Engineering (DICCA) University of Genoa Via7
Montallegro 1 16145 Genoa (Italy)8
Corresponding author9
Lorenzo Benvenuto1210
Email address lorenzobenvenutogterit11
ABSTRACT12
DataBases (DB) are a widespread source of data useful for many applications in different scientificfields The present contribution describes an automatic procedure to access download and store openaccess data from different sources to be processed in a GIS environment In particular it refers tothe specific need of the authors to manage both meteorological data (pressure and temperature) andGNSS (Global Navigation Satellite System) Zenith Total Delay (ZTD) estimates Such data allow toproduce Precipitable Water Vapor (PWV) maps thanks to the so called GNSS for Meteorology (G4M)procedure developed through GRASS GIS software ver 74 for monitoring in time and interpreting severemeteorological events Actually the present version of the procedure includes the meteorological pressureand temperature data coming from NOAArsquos Integrated Surface Database (ISD) whereas the ZTD dataderive from the RENAG DB that collects ZTD estimates for 181 GNSS Permanent Stations (PSs) from1998 to 2015 in the French-Italian boundary region Several Python scripts have been implementedto manage the download of data from NOAA and RENAG DBs their import on a PostgreSQLPostGISgeoDB besides the data elaboration with GRASS GIS to produce PWV maps The key features of thedata management procedure are its scalability and versatility for different sources of data and differentcontexts As a future development a web-interface for the procedure will allow an easier interaction forthe users both for post-processing and real-time data The data management procedure repository isavailable at httpsgithubcomgtergeomaticaG4M-data
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
INTRODUCTION30
This work is part of the research activities conceived by the Geomatics Laboratory of the University31
of Genoa to detect severe meteorological events using the remote sensing of the water vapour content32
in the atmosphere through GNSS (Global Navigation Satellite System) signals (Bevis et al 1992)33
A GIS (Geographic Information System) procedure called G4M (GNSS for Meteorology) has been34
conceived in order to produce 2D Precipitable Water Vapour (PWV) maps with high spatial and temporal35
resolution (Ferrando et al 2018) The developed procedure is able to use observations coming from36
existing monitoring infrastructures of pressure P temperature T and GNSS Zenith Total Delay (ZTD)37
not necessarily co-located and distributed over an orographically complex area In G4M procedure the38
orographic effect influencing PWV maps is reduced by means of a time differentiation of PWV maps with39
respect to a calm period meaning a period of quiet before a storm thus with the realization of ∆PWV40
maps Furthermore an index called Heterogeneity Index (HI) accounting the ∆PWV spatial variability41
has been conceived as a promising indicator to detect severe meteorological events in space and time42
A scheme of the G4M procedure is depicted in figure 143
44
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 1 Scheme of the G4M procedure
In order to automate the procedure to create PWV ∆PWV and HI 2D maps G4M has been recently45
updated using Python as script language GRASS 74 (Neteler et al 2012 GRASS Development Team46
2018) to perform raster interpolation while 1D P T and ZTD data are stored in a PostgreSQLPostGIS47
geodatabase48
This work is intended to describe data collection organization and storage which are the essential49
preliminary phases to launch the G4M procedure First of all the locations of GNSS Permanent Stations50
(PSs) and meteorological stations are uploaded on the PostgreSQLPostGIS geoDB Three boolean (true-51
false) fields have been added to allow users to decide whether use or not the measuring stations depending52
on data availability The monitoring data (ZTD T and P) are uploaded on the PostgreSQLPostGIS53
geoDB using the script described in the following section Hence the main data management workflow54
illustrated in figure 2 can be summarized as follows55
- the computational region is set according to the Digital Terrain Model (DTM) boundaries while the56
resolution is set as defined by the user57
- starting and ending time of elaboration and the time step between two consecutive maps are set by58
the user59
- for each time the geoDB is queried to verify the presence of ZTD T and P data (if the corresponding60
boolean field is set to true)61
- the existing data are elaborated to produce 2D high resolution PWV ∆PWV and HI maps62
Thanks to the GRASS GIS time series modules and all the available Python libraries (time psycopg263
grass-script etc) the management of time series data is made easier if compared with other GIS software64
or programming languages and it allows a straightforward realization of time series plots and analysis65
27
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 2 Workflow of G4M procedure
The following sections describe the techniques used to download data from existing monitoring66
infrastructures and upload them into a spatial database specifically designed particular attention is given67
to the Python scripts conceived to automate these processes68
DATA69
The studied area covers the French-Italian neighbouring area In this area GNSS (ZTD data) and70
meteorological (P and T data) stations are differently distributed71
Available GNSS PSs come from different networks such as the International Gnss Service (IGS)72
Tracking network the EUropean REference Frame (EUREF) Permanent Network and finally also73
national transnational and regional ones In the present work GNSS data come from 181 GNSS PSs with74
an average spacing of 30-40 km acquisition rate is 30rdquo in accordance to an international standard75
ZTD were estimated splitting the whole network in 3 sub-net according to the decreasing station age76
(net1 net2 and net3 respectively) to obtain stable solutions77
A database 1 (RENAG DB) containing two-hourly ZTD estimates from 1998 to 2012 was created78
and made available to researchers (Sguerso et al 2013) Recently it was updated to include data up to79
December 2015 (Sguerso et al 2015) P and T data come from existing network of NOAA (National80
Oceanic and Atmospheric Administration) with an average spacing of sim100 km Meteorological data are81
available through CDO2 service which provides free access to NCEI3 of global historical weather and82
climate data P and T stations are not always co-located because not all the stations measure both P and T83
or because of occasional lack of data Figure 3 shows the GNSS PSs and meteorological stations falling84
in the area of interest85
1ftprenagunicefrproductsGPS climatology Sguerso Labbouz Walpersdorf2Climate Data Online httpswwwncdcnoaagovcdo-web3National Centers for Environmental Information httpswwwnceinoaagovarchive
37
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 3 GNSS PSs in the working area For the GNSS PSs the division in sub-networks ishighlighted net1 net2 and net3 are represented with red blue and green dots respectively whereas the 15common PSs are represented with yellow dots P and T stations are reported with cyan and purple squaresrespectively
To describe the terrain elevation the Global Digital Elevation Model (GDEM) of ASTER space86
project (joint product of METI Ministry of Economy Trade and Industry of Japan and NASA National87
Aeronautics and Space Administration httpasterwebjplnasagovgdemasp) is used88
METHOD89
The G4M procedure computes 2D PWV maps starting from 1D ZTD P and T observations exploiting90
open-sources software GRASS GIS 74 To support a near future web application of G4M the 1D input91
data are stored in a geodatabase specifically designed using PostgreSQL+PostGIS geodatabase which is92
highly compatible with GRASS 74 Moreover the procedure originally in Fortran has been re-written93
in Python a more modern programming language that has many libraries to interact with PostgreSQL94
GRASS and temporal series of data allowing to obtain a more automate procedure In this section the95
Python scripts conceived to automatically upload ZTD P and T data to the geodatabase are exposed All96
the scripts are available at httpsgithubcomgtergeomaticaG4M-data97
Python script for ZTD98
ZTD data have a temporal step of 2 hours and initially are saved into 3 different folders named net1 net299
and net3 according to the already mentioned division into subnets Every folder contains a file for each100
stations with ZTD estimates and the relative RMS (Root Mean Square) The files are named in a proper101
way which describes the station name the year of observation and the network in which the station is102
inserted (eg AJAC2012net1az)103
The Python script conceived for uploading data into the geodatabase is named ztd2postgresqlpy It104
reads every line of a specified file and puts the data (ie year day hour ZTD and RMS) in the proper105
column of a table in the database A scheme of the ztd2postgresqlpy script is shown in figure 4 To106
upload all the data contained into the RENAG DB to the geodatabase this procedure has to be executed107
47
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
for every station and for every year of observation To speed up the upload process a bash script has been108
conceived to automatically execute the ztd2postgresqlpy script for every file in a folder109
Figure 4 A scheme of the script to upload ZTD data into the geodatabase
Python script for P and T110
P and T data can be downloaded from NOAA in two way 1) in the ISD-lite format from the website4 2)111
manually from NOAA website5 specifying the time period and a list of desired stations This second112
downloading way is not easy to automate and highly time consuming for this reason in this work the113
meteorological data have been downloaded in ISD-lite format114
Files in this format contain a fixed-width formatted subset of the complete Integrated Surface Data115
(ISD) for a selected number of observations The data are typically stored in a single file corresponding to116
the ISD data ie one file per station per year117
To download data from NOAA and upload them into the geoDB in an automatic and fast way a118
Python script called noaa2postresql ftppy has been implemented This script needs two inputs a year119
and the station id While the year must be specified by the user the stations ids are automatically read120
from the database using a query Having this two inputs it is possible to download the files containing P121
and T data for all the stations in the database and for the specified year The downloaded files are saved122
in compressed folders so the following step is the extraction and read of those files Finally data are123
uploaded to the geoDB and all the downloaded file are removed from the local computer A scheme of the124
noaa2postgresql ftppy script is shown in figure 5125
Sguerso D Labbouz L and Walpersdorf A (2013) 14 years of GPS tropospheric delays in the French-153
Italian border region a data base for meteorological and climatological analyses The International154
Archives of the Photogrammetry Remote Sensing and Spatial Information Sciences XL-5W37ndash14155
67
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Sguerso D Labbouz L and Walpersdorf A (2015) 14 years of GPS tropospheric delays in the156
FrenchndashItalian border region comparisons and first application in a case study Appl Geomat 8(1)1ndash157
13158
77
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 1 Scheme of the G4M procedure
In order to automate the procedure to create PWV ∆PWV and HI 2D maps G4M has been recently45
updated using Python as script language GRASS 74 (Neteler et al 2012 GRASS Development Team46
2018) to perform raster interpolation while 1D P T and ZTD data are stored in a PostgreSQLPostGIS47
geodatabase48
This work is intended to describe data collection organization and storage which are the essential49
preliminary phases to launch the G4M procedure First of all the locations of GNSS Permanent Stations50
(PSs) and meteorological stations are uploaded on the PostgreSQLPostGIS geoDB Three boolean (true-51
false) fields have been added to allow users to decide whether use or not the measuring stations depending52
on data availability The monitoring data (ZTD T and P) are uploaded on the PostgreSQLPostGIS53
geoDB using the script described in the following section Hence the main data management workflow54
illustrated in figure 2 can be summarized as follows55
- the computational region is set according to the Digital Terrain Model (DTM) boundaries while the56
resolution is set as defined by the user57
- starting and ending time of elaboration and the time step between two consecutive maps are set by58
the user59
- for each time the geoDB is queried to verify the presence of ZTD T and P data (if the corresponding60
boolean field is set to true)61
- the existing data are elaborated to produce 2D high resolution PWV ∆PWV and HI maps62
Thanks to the GRASS GIS time series modules and all the available Python libraries (time psycopg263
grass-script etc) the management of time series data is made easier if compared with other GIS software64
or programming languages and it allows a straightforward realization of time series plots and analysis65
27
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 2 Workflow of G4M procedure
The following sections describe the techniques used to download data from existing monitoring66
infrastructures and upload them into a spatial database specifically designed particular attention is given67
to the Python scripts conceived to automate these processes68
DATA69
The studied area covers the French-Italian neighbouring area In this area GNSS (ZTD data) and70
meteorological (P and T data) stations are differently distributed71
Available GNSS PSs come from different networks such as the International Gnss Service (IGS)72
Tracking network the EUropean REference Frame (EUREF) Permanent Network and finally also73
national transnational and regional ones In the present work GNSS data come from 181 GNSS PSs with74
an average spacing of 30-40 km acquisition rate is 30rdquo in accordance to an international standard75
ZTD were estimated splitting the whole network in 3 sub-net according to the decreasing station age76
(net1 net2 and net3 respectively) to obtain stable solutions77
A database 1 (RENAG DB) containing two-hourly ZTD estimates from 1998 to 2012 was created78
and made available to researchers (Sguerso et al 2013) Recently it was updated to include data up to79
December 2015 (Sguerso et al 2015) P and T data come from existing network of NOAA (National80
Oceanic and Atmospheric Administration) with an average spacing of sim100 km Meteorological data are81
available through CDO2 service which provides free access to NCEI3 of global historical weather and82
climate data P and T stations are not always co-located because not all the stations measure both P and T83
or because of occasional lack of data Figure 3 shows the GNSS PSs and meteorological stations falling84
in the area of interest85
1ftprenagunicefrproductsGPS climatology Sguerso Labbouz Walpersdorf2Climate Data Online httpswwwncdcnoaagovcdo-web3National Centers for Environmental Information httpswwwnceinoaagovarchive
37
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 3 GNSS PSs in the working area For the GNSS PSs the division in sub-networks ishighlighted net1 net2 and net3 are represented with red blue and green dots respectively whereas the 15common PSs are represented with yellow dots P and T stations are reported with cyan and purple squaresrespectively
To describe the terrain elevation the Global Digital Elevation Model (GDEM) of ASTER space86
project (joint product of METI Ministry of Economy Trade and Industry of Japan and NASA National87
Aeronautics and Space Administration httpasterwebjplnasagovgdemasp) is used88
METHOD89
The G4M procedure computes 2D PWV maps starting from 1D ZTD P and T observations exploiting90
open-sources software GRASS GIS 74 To support a near future web application of G4M the 1D input91
data are stored in a geodatabase specifically designed using PostgreSQL+PostGIS geodatabase which is92
highly compatible with GRASS 74 Moreover the procedure originally in Fortran has been re-written93
in Python a more modern programming language that has many libraries to interact with PostgreSQL94
GRASS and temporal series of data allowing to obtain a more automate procedure In this section the95
Python scripts conceived to automatically upload ZTD P and T data to the geodatabase are exposed All96
the scripts are available at httpsgithubcomgtergeomaticaG4M-data97
Python script for ZTD98
ZTD data have a temporal step of 2 hours and initially are saved into 3 different folders named net1 net299
and net3 according to the already mentioned division into subnets Every folder contains a file for each100
stations with ZTD estimates and the relative RMS (Root Mean Square) The files are named in a proper101
way which describes the station name the year of observation and the network in which the station is102
inserted (eg AJAC2012net1az)103
The Python script conceived for uploading data into the geodatabase is named ztd2postgresqlpy It104
reads every line of a specified file and puts the data (ie year day hour ZTD and RMS) in the proper105
column of a table in the database A scheme of the ztd2postgresqlpy script is shown in figure 4 To106
upload all the data contained into the RENAG DB to the geodatabase this procedure has to be executed107
47
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
for every station and for every year of observation To speed up the upload process a bash script has been108
conceived to automatically execute the ztd2postgresqlpy script for every file in a folder109
Figure 4 A scheme of the script to upload ZTD data into the geodatabase
Python script for P and T110
P and T data can be downloaded from NOAA in two way 1) in the ISD-lite format from the website4 2)111
manually from NOAA website5 specifying the time period and a list of desired stations This second112
downloading way is not easy to automate and highly time consuming for this reason in this work the113
meteorological data have been downloaded in ISD-lite format114
Files in this format contain a fixed-width formatted subset of the complete Integrated Surface Data115
(ISD) for a selected number of observations The data are typically stored in a single file corresponding to116
the ISD data ie one file per station per year117
To download data from NOAA and upload them into the geoDB in an automatic and fast way a118
Python script called noaa2postresql ftppy has been implemented This script needs two inputs a year119
and the station id While the year must be specified by the user the stations ids are automatically read120
from the database using a query Having this two inputs it is possible to download the files containing P121
and T data for all the stations in the database and for the specified year The downloaded files are saved122
in compressed folders so the following step is the extraction and read of those files Finally data are123
uploaded to the geoDB and all the downloaded file are removed from the local computer A scheme of the124
noaa2postgresql ftppy script is shown in figure 5125
Sguerso D Labbouz L and Walpersdorf A (2013) 14 years of GPS tropospheric delays in the French-153
Italian border region a data base for meteorological and climatological analyses The International154
Archives of the Photogrammetry Remote Sensing and Spatial Information Sciences XL-5W37ndash14155
67
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Sguerso D Labbouz L and Walpersdorf A (2015) 14 years of GPS tropospheric delays in the156
FrenchndashItalian border region comparisons and first application in a case study Appl Geomat 8(1)1ndash157
13158
77
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 2 Workflow of G4M procedure
The following sections describe the techniques used to download data from existing monitoring66
infrastructures and upload them into a spatial database specifically designed particular attention is given67
to the Python scripts conceived to automate these processes68
DATA69
The studied area covers the French-Italian neighbouring area In this area GNSS (ZTD data) and70
meteorological (P and T data) stations are differently distributed71
Available GNSS PSs come from different networks such as the International Gnss Service (IGS)72
Tracking network the EUropean REference Frame (EUREF) Permanent Network and finally also73
national transnational and regional ones In the present work GNSS data come from 181 GNSS PSs with74
an average spacing of 30-40 km acquisition rate is 30rdquo in accordance to an international standard75
ZTD were estimated splitting the whole network in 3 sub-net according to the decreasing station age76
(net1 net2 and net3 respectively) to obtain stable solutions77
A database 1 (RENAG DB) containing two-hourly ZTD estimates from 1998 to 2012 was created78
and made available to researchers (Sguerso et al 2013) Recently it was updated to include data up to79
December 2015 (Sguerso et al 2015) P and T data come from existing network of NOAA (National80
Oceanic and Atmospheric Administration) with an average spacing of sim100 km Meteorological data are81
available through CDO2 service which provides free access to NCEI3 of global historical weather and82
climate data P and T stations are not always co-located because not all the stations measure both P and T83
or because of occasional lack of data Figure 3 shows the GNSS PSs and meteorological stations falling84
in the area of interest85
1ftprenagunicefrproductsGPS climatology Sguerso Labbouz Walpersdorf2Climate Data Online httpswwwncdcnoaagovcdo-web3National Centers for Environmental Information httpswwwnceinoaagovarchive
37
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 3 GNSS PSs in the working area For the GNSS PSs the division in sub-networks ishighlighted net1 net2 and net3 are represented with red blue and green dots respectively whereas the 15common PSs are represented with yellow dots P and T stations are reported with cyan and purple squaresrespectively
To describe the terrain elevation the Global Digital Elevation Model (GDEM) of ASTER space86
project (joint product of METI Ministry of Economy Trade and Industry of Japan and NASA National87
Aeronautics and Space Administration httpasterwebjplnasagovgdemasp) is used88
METHOD89
The G4M procedure computes 2D PWV maps starting from 1D ZTD P and T observations exploiting90
open-sources software GRASS GIS 74 To support a near future web application of G4M the 1D input91
data are stored in a geodatabase specifically designed using PostgreSQL+PostGIS geodatabase which is92
highly compatible with GRASS 74 Moreover the procedure originally in Fortran has been re-written93
in Python a more modern programming language that has many libraries to interact with PostgreSQL94
GRASS and temporal series of data allowing to obtain a more automate procedure In this section the95
Python scripts conceived to automatically upload ZTD P and T data to the geodatabase are exposed All96
the scripts are available at httpsgithubcomgtergeomaticaG4M-data97
Python script for ZTD98
ZTD data have a temporal step of 2 hours and initially are saved into 3 different folders named net1 net299
and net3 according to the already mentioned division into subnets Every folder contains a file for each100
stations with ZTD estimates and the relative RMS (Root Mean Square) The files are named in a proper101
way which describes the station name the year of observation and the network in which the station is102
inserted (eg AJAC2012net1az)103
The Python script conceived for uploading data into the geodatabase is named ztd2postgresqlpy It104
reads every line of a specified file and puts the data (ie year day hour ZTD and RMS) in the proper105
column of a table in the database A scheme of the ztd2postgresqlpy script is shown in figure 4 To106
upload all the data contained into the RENAG DB to the geodatabase this procedure has to be executed107
47
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
for every station and for every year of observation To speed up the upload process a bash script has been108
conceived to automatically execute the ztd2postgresqlpy script for every file in a folder109
Figure 4 A scheme of the script to upload ZTD data into the geodatabase
Python script for P and T110
P and T data can be downloaded from NOAA in two way 1) in the ISD-lite format from the website4 2)111
manually from NOAA website5 specifying the time period and a list of desired stations This second112
downloading way is not easy to automate and highly time consuming for this reason in this work the113
meteorological data have been downloaded in ISD-lite format114
Files in this format contain a fixed-width formatted subset of the complete Integrated Surface Data115
(ISD) for a selected number of observations The data are typically stored in a single file corresponding to116
the ISD data ie one file per station per year117
To download data from NOAA and upload them into the geoDB in an automatic and fast way a118
Python script called noaa2postresql ftppy has been implemented This script needs two inputs a year119
and the station id While the year must be specified by the user the stations ids are automatically read120
from the database using a query Having this two inputs it is possible to download the files containing P121
and T data for all the stations in the database and for the specified year The downloaded files are saved122
in compressed folders so the following step is the extraction and read of those files Finally data are123
uploaded to the geoDB and all the downloaded file are removed from the local computer A scheme of the124
noaa2postgresql ftppy script is shown in figure 5125
Sguerso D Labbouz L and Walpersdorf A (2013) 14 years of GPS tropospheric delays in the French-153
Italian border region a data base for meteorological and climatological analyses The International154
Archives of the Photogrammetry Remote Sensing and Spatial Information Sciences XL-5W37ndash14155
67
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Sguerso D Labbouz L and Walpersdorf A (2015) 14 years of GPS tropospheric delays in the156
FrenchndashItalian border region comparisons and first application in a case study Appl Geomat 8(1)1ndash157
13158
77
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
Figure 3 GNSS PSs in the working area For the GNSS PSs the division in sub-networks ishighlighted net1 net2 and net3 are represented with red blue and green dots respectively whereas the 15common PSs are represented with yellow dots P and T stations are reported with cyan and purple squaresrespectively
To describe the terrain elevation the Global Digital Elevation Model (GDEM) of ASTER space86
project (joint product of METI Ministry of Economy Trade and Industry of Japan and NASA National87
Aeronautics and Space Administration httpasterwebjplnasagovgdemasp) is used88
METHOD89
The G4M procedure computes 2D PWV maps starting from 1D ZTD P and T observations exploiting90
open-sources software GRASS GIS 74 To support a near future web application of G4M the 1D input91
data are stored in a geodatabase specifically designed using PostgreSQL+PostGIS geodatabase which is92
highly compatible with GRASS 74 Moreover the procedure originally in Fortran has been re-written93
in Python a more modern programming language that has many libraries to interact with PostgreSQL94
GRASS and temporal series of data allowing to obtain a more automate procedure In this section the95
Python scripts conceived to automatically upload ZTD P and T data to the geodatabase are exposed All96
the scripts are available at httpsgithubcomgtergeomaticaG4M-data97
Python script for ZTD98
ZTD data have a temporal step of 2 hours and initially are saved into 3 different folders named net1 net299
and net3 according to the already mentioned division into subnets Every folder contains a file for each100
stations with ZTD estimates and the relative RMS (Root Mean Square) The files are named in a proper101
way which describes the station name the year of observation and the network in which the station is102
inserted (eg AJAC2012net1az)103
The Python script conceived for uploading data into the geodatabase is named ztd2postgresqlpy It104
reads every line of a specified file and puts the data (ie year day hour ZTD and RMS) in the proper105
column of a table in the database A scheme of the ztd2postgresqlpy script is shown in figure 4 To106
upload all the data contained into the RENAG DB to the geodatabase this procedure has to be executed107
47
PeerJ Preprints | httpsdoiorg107287peerjpreprints27227v1 | CC BY 40 Open Access | rec 19 Sep 2018 publ 19 Sep 2018
for every station and for every year of observation To speed up the upload process a bash script has been108
conceived to automatically execute the ztd2postgresqlpy script for every file in a folder109
Figure 4 A scheme of the script to upload ZTD data into the geodatabase
Python script for P and T110
P and T data can be downloaded from NOAA in two way 1) in the ISD-lite format from the website4 2)111
manually from NOAA website5 specifying the time period and a list of desired stations This second112
downloading way is not easy to automate and highly time consuming for this reason in this work the113
meteorological data have been downloaded in ISD-lite format114
Files in this format contain a fixed-width formatted subset of the complete Integrated Surface Data115
(ISD) for a selected number of observations The data are typically stored in a single file corresponding to116
the ISD data ie one file per station per year117
To download data from NOAA and upload them into the geoDB in an automatic and fast way a118
Python script called noaa2postresql ftppy has been implemented This script needs two inputs a year119
and the station id While the year must be specified by the user the stations ids are automatically read120
from the database using a query Having this two inputs it is possible to download the files containing P121
and T data for all the stations in the database and for the specified year The downloaded files are saved122
in compressed folders so the following step is the extraction and read of those files Finally data are123
uploaded to the geoDB and all the downloaded file are removed from the local computer A scheme of the124
noaa2postgresql ftppy script is shown in figure 5125
