AidstothePresentationandAnalysisofWSPRSpots:
TimescaleDBdatabaseandGrafanaGwynGriffiths,G3ZILandRobRobinett,AI6VNFormoreinformationvisitwsprdaemon.org
WithenormousthankstoGary,Corrieandtheteamprovidingwsprnet.org
LX1D
Q
Synopsis
! Motivation–Original,leadingtoagrowingambition
! StepsfromDCC2019toreachtoday’sWsprDaemon
! Today’sTimescaledatabaseandHardware
! ExploitingtheTimescaleDB" Real-timedataservicetoThirdParties
" GrafanadatavisualisationofWSPRdata
" AccesstoadditionaldatafromWsprDaemonusers
• Noiseestimates
• Completewsprdparameterset
! NextSteps
Motivation
! Initial" RobRobinett,AI6VN’sconcept:ArobustandreliableWSPRreportingtoolforusersoftheKiwiSDRmulti-channelrx.
" WithGlennElmore,N6GN,seespotentialforKiwiSDRstoestimatenoiseatthesametimeasWSJT-XestimatesSNR.
" Benchinvestigations(Griffiths,DCC2019)leadtosimpleInfluxdatabase&GrafanagraphsbyTommyNourse,KI6NKO.
! Earlyevolution" Recognisebenefitsofdatabasewithnoiseandspotdata,andderivedvariables,e.g.azimuthatthereceiver;latsandlons.
! Developingambition" Addspotsfromwsprnet.orgtoourdatabase,helpreduceloadonwsprnet.orgbyservingdatatopopularthird-partyAppsassecondarysource.
" Servedataon-lineforlongerperiodthanwsprnet.orgtointerestedresearchersandcitizenscientists,e.g.HamScicommunity.
WsprDaemon:FromDCC2019…Global~4000
reporters
UDPdatatowsprnet
MySQLDB
InnoDBstorageengine
Webappusers
WsprDaemonUsers
InputspotsHTTPPutstandarddatatowsprnet.org
Noiseestimatedatato3rdPartydatabase
WsprDaemonclient
InfluxdatabasefromThirdParty
• WsprDaemonuseslatestwsprddecoderfromWSJT-X
• Robustreportingtowsprnet.org• InSeptember2019alsoreportednoise
estimatestoanInfluxdatabasesetupandmaintainedbyTommyNourse,KI6NKO,withsimpleGrafanatimeseriesgraphs.
Grafanausers
GrafanaNoisedata
Panel
Global~4000
reporters
UDPdatatowsprnet
MySQLDB
InnoDBstorageengine
API
Webappusers
WsprDaemonUsers
BashandPythonwhitelistingandderivedvariables
WsprDaemonBashandPythonpreprocessing
postgreSQLDBwithTimescaleDB
extensions
InputspotscachedHTTPPutstandarddatatowsprnet.orgFTPextendeddatatologs.wsprdaemon.org
3rdpartydatae.g.NASASpace
Weather
3rdpartydataBashandPythonpreprocessing
Webappusers
Node.js
3rdPartyPartner
SwiftandPostgresClientKit
3rdPartyPartner
Mobileusers
GrafanaDash-boards
Webappusers
psql
Commandlineusers
WsprDaemonclient
WsprDaemonServerlogs.wsprdaemon.orgOwned,8-core192GB,550GBSSD,7.3TBRAID5disk,colocatedatHurricaneElectric.
FTPWsprDaaemondatamirror
FTPdatamirrorin
WsprDaemonBackupServerlogs1.wsprdaemon.org
postgreSQLDBwithTimescaleDB
extensions
WsprDaemonDataArchitectureToday
FromwsprnetAPI
TimescaleDatabase&DataRetention
PackagedasanextensiontoPostgreSQL–
benefitingfromits20+yearsasawidelyusedOpenSourcedatabase.
Bringsconceptoftime‘chunks’toPostgreSQLto
handletimeseriesefficiently.Fastdatainsertion.Fastretrievalofrecentdata.
B-treeindexingbettersuitedtohighcardinality
WSPRdatathanInflux–whichwetriedfirst.
Current‘Chunk’inRAM
48GBof192GB30dayscurrently
Older,stillqueried‘Chunks’inSSD
550GB~1year
Historical,rarelyqueried‘Chunks’rotatingdisks
7.3TBRAIDSunspotCycle
Datamovementovertimetohighercapacity,lessexpensivestorage
Real-timedataservicetoThirdParties
wsprd.vk7jj.comPhilBarnardVK7JJ WsprWatchiOSappPeterMarksVK2TPM
Dashboardname DashboardSettings
DashboardSave
Settimewindow
Zoomin/out
Reload
Databasetouse
QueryBuilder
VisualisationDisplay
Firstquery‘A’
ToggleQueryEditorandTextEntry
PanelTitle
Addquery
EnterPanelTitle
Axes
GrafanadatavisualisationofWSPRdata
Aguideisavailableatwsprdaemon.org/grafana.html
Derivesignallevels:Dataakintothatinresearchpublications
! TimescaleDB:JoinSNRandnoisetables
! Grafana:Timeseriesplots,withSignalLevelasHeatmap.
Above:Signalpowerreceivedatnr.BostonfromCHUtimesignalstationnr.Ottawa,1Sept.2003on7.335MHz.FromMcNamaraetal.,2008.RadioScience,43(2).
N1ZPY,MainereceivedatKD2OM,nrRochester,NYon7.04MHz
SporadicE
Peakdaytimeabsorption
Greaterdaytimevariation(fading)
Heatmaps:NumberofWSPRspotsin200kmrangeringsevery20min! TimescaleDB:SelectallreceiversingridFN13%
whereazimuthbetween250–290˚! Grafana:StandardHeatmapin200kmand20min
Antarctica
Cote d'Ivoire
Afghanistan
Albania
Algeria
American Samoa
Andorra
Angola
Antigua and Barbuda
Argentina
Armenia
Aruba
Australia
Austria
Azerbaijan
Bahamas
Bahrain
Bangladesh
Barbados
Belarus
Belgium
Belize
Bhutan
Bolivia
Bosnia and Herzegovina
Botswana
Brazil
British Virgin Islands
Brunei Darussalam
Bulgaria
Burkina Faso
Burundi
Cambodia
Cameroon
Canada
Cape Verde
Cayman Islands
Central African Republic
Chad
Chile
China
Colombia
Comros
Congo
D. R. Congo
Costa Rica
Croatia
Cuba
Cyprus
Czech RepublicDenmark
Djibouti
Dominica Republic
Dominica
East Timor
Ecuador
Egypt
El Salvador
Equatorial Guinea
EritreaEstonia
Ethiopia
Falkland Islands
Faroe Islands
Fiji
Finland
France
French Guiana
French Polynesia
Gabon
Gambia
Georgia
Germany
Ghana
Greece
Greenland
Guadeloupe
Guatemala
Guernsey
GuineaGuinea-Bissau
Guyana
Haiti
Honduras
Hungary
Iceland
India
Indonesia
Iran
Iraq
Ireland
Israel
Italy
Jamaica
Japan
Jordan
Kazakhstan
Kenya
Kiribati
Korea
Kuwait
Kyrgyzstan
Laos
Latvia
Lebanon
Lesotho
Libya
Liberia
Liechtenstein
Lithuania
Luxembourg
Macao, China
Madagascar
Malawi
Malaysia
Maldives
Mali
Malta
Martinique
Mauritania
Mayotte
Mexico
Micronesia
Moldova
Mozambique
Myanmar
Namibia
Nepal
Netherlands Antilles
Netherlands
New Caledonia
New Zealand
Nicaragua
Niger
Nigeria
Norfolk Island
Northern Mariana Islands
Norway
Oman
Pakistan
Palau
Panama
Papua New Guinea
Paraguay
Peru
Philippines
Poland
Portugal
Puerto Rico
Qatar
Rawanda
Romania
Russian Federation
Saint Kitts and Nevis
Saint Lucia
Saint Pierre and Miquelon
Samoa
San Marino
Sao Tome and Principe
Saudi Arabia
Senegal
Sierra Leone
Slovakia
Slovenia
Solomon Islands
Somalia
South Africa
Spain
Sudan
Suriname
Swaziland
Sweden
Switzerland
Syrian Arab Republic
Tajikistan
Tanzania
Thailand
Togo
Tonga
Tunisia
Turkey
Turkmenistan
Tuvalu
Uganda
Ukraine
United Arab Emirates
United Kingdom
United States of America
Uruguay
Uzbekistan
Vanuatu
Venezuela
Viet Nam
Yugoslavia
Zambia
Zimbabwe
Alabama
Alaska
Arizona
Arkansas
California
ColoradoConnecticut
Delaware
Florida
Georgia
Hawaii
Idaho
IllinoisIndiana
Iowa
Kansas
Kentucky
Louisiana
Maine
MassachusettsMichigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
PennsylvaniaRhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah Vermont
Virginia
Washington
West Virginia
Wisconsin
Wyoming
Alberta
British Columbia
Manitoba
New Brunswick
Newfoundland & Labrador
Newfoundland
Northwest Territories
Nova Scotia
Nunavut
OntarioQuébec
Saskatchewan
Yukon Territory
90°80°
70°
60°
50°
40°
30°
20°10°0°350°
340°
330°
320°
310°
300°
290°
280°
270°
260°250°
240°230°
220°210°
200° 190° 180° 170° 160°150°
140°
130°
120°
110°
100°
Azimuthal MapCenter: 43°0'0"N 77°30'0"W
Courtesy of Tom (NS6T)
Map from http://ns6t.net/
250-290˚ Grid
FN13
14MHz7MHz0000UTC28Aug–0000UTC29AugYaxis:Range0–4000km
MapcourtesyTomNS6T
SNRdifferences:2antennas1location
! TimescaleDB:Calculatemedianandquartilesin20minuteintervals.
! Grafana:Plotlyplug-inforX-axisotherthantime–hereAzimuthatrx.
G3ZIL:40minvertedVdipolepairwithadaptivenoisereductionG3ZIL/P:Activeshortverticaldipole(2mtiptotip)N6GNdesign.
Longview:Heatmapsbyhourofday
DatafromNorthernUtahSDRKA7OEI-1
On14MHz11July–29Aug
! TimescaleDB:SimpleSQL
! Grafana:HourlyHeat-mapplug-in
Noiselevel(dBmin1Hz)
Numberofspotsin30minuteintervals
Meandistanceofspotsin30minuteintervals(km)
Yaxisishouroftheday(UTC)0000attop,2359atbottom,eachblockonXaxisisoneday
WsprDaemonlogsallwsprdparametersNeedsnon-parametricdensitycontours–notinGrafana
! Blocksize:numberofsymbolsoverwhichdecodeattempted
! TimeOffset:resolutionof10ms! DecodeCycles:Iterationsofthe
Fanodecoderperbit! ipass,nhardmin,decodetype
! Jitter:finetimeoffset! Metric(left):“closenessofapathto
thereceivedsequence”! SyncQuality(right):howwell
receivedsyncsymbolsequencematcheslocal
NextSteps
! WsprDaemon:Extendrobustreporting(andnoise)toreceiverswithaSoapyinterface.
! TimescaleDB:Appdeveloperswelcometoevaluateourdatabase;looktoeaseloadonwsprnet.org.
! Grafana:Improvefrontpagelayoutandhelp,findplug-ins(orlearntocodejson)forunavailablegraphtypes
Welcomemostheartedlycontactwithanyoneinterestedinthetypesofanalysisanddatapresentationinthistalktostudypropagationortheirownstationperformanceor…
WithenormousthankstoGary,Corrieandtheteamprovidingwsprnet.organdtoTommyNourse,SteveSykes,ClintTurner,PhilBarnard,PeterMarks,GlennElmore
