PostgreSQL for Buoy DataSteve Ruberg, Steve Constant, Ron Muzzi, Russ

Miller, Joe Smith

Utilization of PostgreSQL Database for Real-Time Western Lake Erie Data

Storage and DisseminationRUBERG, S.A., CONSTANT, S.A., MUZZI, R.W., MILLER, R.J., and SMITH, J.P.

Many thanks!

What are we talking about today?

1. Western Lake Erie (WLE) Real-time (RT) data collection2. Database3. Interface

Map made by Kaye LaFond

Map made by Kaye LaFond

1. WLE RT-Data Collection

● Meteorological● Water Quality

○ Phosphorus○ Chlorophyll○ Blue-green Algae○ Nitrogen

Format

● Flat text● ~50 columns● Split by day● Several MBytes per season● High temporal resolution (every 15 min)

1. WLE RT-Data Collection

Problem

● Client-side computer variability● Data require intense processing to visualize● Tried complete client-side interface to this

○ Not great...

1. WLE RT-Data Collection

2. Database

● SQL database○ Relational Database Management System (RDBMS)○ Tables connected by keys○ Through PHP, server power can be leveraged

■ Data delivered in smaller, more digestible chunks

● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP

2. Database - choices

● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,

ACID compliance, etc, but you do care about ease and speed, you tend

to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848

2. Database - choices

● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,

ACID compliance, etc, but you do care about ease and speed, you tend

to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848

● Deployment environment = Federal Laboratory

2. Database - choices

● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,

ACID compliance, etc, but you do care about ease and speed, you tend

to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848

● Deployment environment = Federal Laboratory● National Buoy Data Center uses Oracle

○ According to: Handbook of Automated Data Quality Control Checks and Procedures (2009)○ http://www.ndbc.noaa.gov/NDBCHandbookofAutomatedDataQualityControl2009.pdf

2. Database - choices

● Adhere to the LAMP stack?○ Linux, Apache, MySQL, PHP○ “If you don't really know or care much about databases, data validity,

ACID compliance, etc, but you do care about ease and speed, you tend

to go with MySQL.” - User Orthogonality on AskMetafilter■ http://ask.metafilter.com/92162/MySQL-vs-PostgreSQL#1349848

● Deployment environment = Federal Laboratory● National Buoy Data Center uses Oracle

○ According to: Handbook of Automated Data Quality Control Checks and Procedures (2009)○ http://www.ndbc.noaa.gov/NDBCHandbookofAutomatedDataQualityControl2009.pdf

● Go for something a little more free

2. Database - choices

https://www.postgresql.org/about/

○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant

2. Database - PostgreSQL

https://www.postgresql.org/about/

○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant

■ Atomicity - all or nothing

2. Database - PostgreSQL

https://www.postgresql.org/about/

○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant

■ Atomicity - all or nothing■ Consistency - transactions done right

2. Database - PostgreSQL

https://www.postgresql.org/about/

○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant

■ Atomicity - all or nothing■ Consistency - transactions done right■ Isolation - result of concurrent transactions same

as if they were run separately

2. Database - PostgreSQL

https://www.postgresql.org/about/

○ 15+ years development○ Open-source○ Nearly unbounded scalability○ ACID Compliant

■ Atomicity - all or nothing■ Consistency - transactions done right■ Isolation - result of concurrent transactions same

as if they were run separately■ Durability - transaction permanence

2. Database - PostgreSQL

● 2 parallel tables○ Configurations○ Data and Quality Flags

● R used to process text data, Unix for automation

2. Database - Structure

● Configuration ID (primary key)● Start Date● End Date● Notes● GMT Offset● Matrix with the configurations, 7 columns

2. Database - Configuration table

● Matrix with the configurations, 7 columns○ Variable name universal across tables○ Variable name unique to the data under config○ Data units (as applicable)○ Statistic of data (min, max, mean, etc.)○ Data type○ QARTOD inspired gross/climatological range limits

■ Min■ Max

2. Database - Configuration table

● Data and Quality Flags○ Data Label (primary key)○ Corresponding configuration ID○ Measure Date○ Data matrix○ Quality flag matrix

2. Database - Data table

● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix

● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix

2. Database

● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix

● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix

2. Database

● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix

● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix

2. Database

● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix

● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix

2. Database

● Configuration Table○ Configuration ID ○ Start Date○ End Date○ Notes○ GMT Offset○ Config matrix

● Data and Quality Flags○ Data Label○ Configuration ID○ Measure Date○ Data matrix○ Quality flag matrix

Rows here match with... columns here.

2. Database

https://www.glerl.noaa.gov/res/HABs_and_Hypoxia/rtMonSQL.php

https://www.glerl.noaa.gov/res/HABs_and_Hypoxia/rtMonSQL.php

https://www.glerl.noaa.gov/res/HABs_and_Hypoxia/rtMonSQL.php

Features

● FAST

Features

● FAST● Click location on map to

view by location○ Use the drop down too

Features

● FAST● Click location on map to

view by location○ Use the drop down too

● OR Select handful of variables to view across all sampling locations

Features

● FAST● Click location on map to

view by location○ Use the drop down too

● OR Select handful of variables to view across all sampling locations

● Pick central date and day range of interest, then click ‘Go!’ to refresh the page and view that time’s data

jpthesmithe.com/iaglr.html