QA/QC of Solar spectral QA/QC of Solar spectral UV-measurements:UV-measurements:

Checking for wavelength shifts and spectral shape deviations with

focus on error flagging

Harry Slaper, PN den Outer, HAJM Reinen

Flagging outputFlagging outputShift in two wavelength regions:< 325 nm green 0.1 nm yellow 0.2 nm red> 325 nm green 0.2 nm yellow 0.4 nm red

starting wavelength (SZA dependent) < 2-3% error => GREEN

spike error flag: shape error > 10% =>2-fold error at single wavelength=> YELLOWat 305 nm corresponds to 2-3 % CIE-error

atmospheric variability during the scan:green is subdivided in INSTABLE (shape>3%)HIGHLY INSTABLE (shape>5%)

Shift and deconvolution Shift and deconvolution analysis: why?analysis: why?

Spectral instruments show variation in slit functions (FWHM 0.3 - 2.5 nm)=>comparing spectra

High accuracy wavelength calibration required (wavelength shift of 0.03-0.05nm => 1% change in Effective UV)

Wavelength Shift AnalysisWavelength Shift AnalysisS T E P 1R a t i o o f m e a s u r e m e n t M a t w a v e l e n g t h ,w i t h t h e s u m o f t w o s u r r o u n d i n gm e a s u r e m e n t s :

MR ( ) = 2 M ( )

M ( - s ) + M ( + s )

S T E P 2S i m i l a r r a t i o f o r s i m u l a t e d m e a s u r e m e n tR M o d ( s l i t c o n v o l u t e d h i g h r e s o l u t i o n ) .

Wavelength Shift AnalysisWavelength Shift AnalysisS T E P 3C a l c u l a t e E r r o r R a t i o t a k i n g t h e r a t i o o f t h e m e a s u r e d a n dm o d e l l e d r a t i o ’ s f r o m s t e p s 1 a n d 2 .M i n i m a l i z e t h e E R b y c h o o s i n g o p t i m a l s h i f t :

E R =

( MR ( )

M o dR ( + )2)

( n - 1 )

l n ( )

w h e r e s u m m a t i o n o v e r w a v e l e n g t h i n t e r v a l ( 1 6 n m ) n - 1 n u m b e r o f r a t i o s i n t h e i n t e r v a l t h e w a v e l e n g t h s h i f t

Correction for slit function Correction for slit function effectseffects

Measured spectrum: M() Iterative ‘Deconvolution’ technique -

start spectrum S0() (high resol.) - simulate measurement Ms0(); - correct start spectrum: S1() = (M() / Ms0()) S0()

Repeat steps until: S9()

Wavelength shift Wavelength shift uncertaintiesuncertainties

Reproducability error < 0.01 nm (FWHM <= 1 nm)

Alignment error 0.01-0.02 nm Absolute acccuracy < 0.01 nm

(depends on Extra Terrestrial)

-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

30 40 50 60 70 80 90

Solar Zenith Angle (degrees)

Waveeln

gth

sh

ift

(nm

)

shift < 325 nm

shift >325 nm

Solar zenith angle (in)depence of wavelength shiftsusing SHICrivm (spectra from RIVM UV monitoring)

Shift and deconvolution Shift and deconvolution analysisanalysis

Enables high accuracy wavelength calibration

Facilitates direct comparison of spectral readings from different instruments/sites

Enables calculation of Standardized spectral UV-data

Shift and deconvolution Shift and deconvolution software: SHICrivmsoftware: SHICrivm

wavelength dependent shift standardized spectrum

- shift and slit corrected- shift corrected

effective UV-determination for several Action Spectra

Check on spectral anomalies

Further application of Further application of SHICrivmSHICrivm

Determine high resolution quality of ET-spectra (Kitt Peak, Atlas 3)

Slit function; FWHM estimates Analysis of requirements

regarding over/under sampling Facilitate model/measurement

intercomparison

Analysis of spectral shape errors Analysis of spectral shape errors (spikes, cloud variability)(spikes, cloud variability)

Use ratio of readings at two adjacent wavelengths and compare with expected ratio (SHIC)

identify shortest wavelength with reliable readings (SHIC and ratio method)

identify spikes and cloud variability

relative CIE weighted UV

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 20 40 60 80 100

SZA (degrees)

Fra

ctio

n b

elo

w in

dic

ated

wav

elen

gth

290

295

300

303

305

Data from Blumthaler SUSPEN 1997

Ratio of spectra for DEG

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

1.2

290 300 310 320 330 340 350 360 370 380 390 400

Wavelength

Rat

io

DEG 185 0400/0430

Check spectral structure

-30

-20

-10

0

10

20

30

290 300 310 320 330 340 350 360 370 380 390 400

wavelength (nm)

de

via

tio

n (

%)

SHIC

Spectral ratio

Check spectral structure

-30

-20

-10

0

10

20

30

290 300 310 320 330 340 350 360 370 380 390 400

wavelength (nm)

dev

iati

on

(%

)

SHIC

Spectral ratio

Start wavelength

286

288

290

292

294

296

298

300

302

285 290 295 300 305

SHIC analysed (nm)

Ra

tio

an

aly

sed

ATI 185

Linear (ATI 185)

Start wavelength

288

290

292

294

296

298

300

302

285 290 295 300 305

SHIC analysed (nm)

Ra

tio

ba

sed

(n

m)

DEG 185

Linear (DEG 185)

Start wavelength

290

295

300

305

310

315

290 295 300 305 310

SHIC analysed start (nm)

Ra

tio

ba

sed

sta

rt (

nm

)

NLR 185

Linear (NLR 185)

starting wavelength CAT

288

290

292

294

296

298

300

302

304

288 290 292 294 296 298 300 302 304

SHIC based start (nm)

Ra

tio

ba

sed

sta

rt (

nm

)

185 CAT

Linear (185 CAT)

Starting wavelength NZY

288

290

292

294

296

298

300

302

304

306

290 295 300 305

SHIC based start (nm)

Ra

tio

ba

sed

sta

rt (

nm

)

NZY

Linear (NZY)

Identification of spikes and Identification of spikes and spectral shape errorsspectral shape errors

Spectral shape errors by comparing readings at subsequent wavelengths with ‘expected’ratio

problem how to distinguish between errors and variability

proposal shape error >0.03 instable conditions; >0.05 highly instable; >0.10 YELLOW (spikes flag)

Shape errors versus pyrano variabilitySZA 35-45 RIVM data march-october 2000

0

5

10

15

20

25

30

35

40

0 3 6 9 12 15

shape (10%) error (%)

pyr

ano

var

iab

ility

(%

)

meansd%

R = 0.812 mean SD of 1 min readings

Correlation of Correlation ofshape 10% with mean SD

SZA nr mean SD SD <min> SD <min>R R R

0 - 35 692 0.851 0.58 0.68435 - 45 1817 0.812 0.49 0.70745 - 55 1805 0.764 0.48 0.6955 - 65 1857 0.722 0.41 0.64665 - 75 1640 0.634 0.37 excl 1% 0.50575 - 85 1648 0.591 0.48 0.64885 - 90 844 0.611 0.59 0.613

shape derived from spectral shape errorsSD derived from pyranometer variabilitywithin minute or between minutes

Correlation of shape errors (10%) and pyranometervariability (mean of SD within minutes; SD between 1 minute readings)

Flagging outputFlagging outputShift in two wavelength regions:< 325 nm green 0.1 nm yellow 0.2 nm red> 325 nm green 0.2 nm yellow 0.4 nm red

starting wavelength (SZA dependent) < 2-3% error => GREEN

spike error flag: shape error > 10% =>2-fold error at single wavelength=> YELLOWat 305 nm corresponds to 2-3 % CIE-error

atmospheric variability during the scan:green is subdivided in INSTABLE (shape>3%)HIGHLY INSTABLE (shape>5%)

!SHICrivm version 3.012 10-06-2001; Harry Slaper LSO/RIVM, harry.slaper@RIVM.NL! Quality check shift and structure! sh1 refers to average shift over wavelength interval: 300.00 325.00! sh2 refers to average shift over wavelength interval: 325.00 390.00! start_wave indicates effective UV below lowest wavelength where 5 successive readings! meet criterium for structural checks: 0.250! spike and instability checking above the starting wavelength1061024 sza: 44.842 shift1 flagging : GREEN 0.016 (nm) shift2 flagging : GREEN 0.030 (nm) spike and instability flag: GREEN INSTABLE 3.04 % start wavelength flag : YELLOW 299.04 (nm) efuv below: 0.49 % last wavelength flag : GREEN 379.50 (nm) efuv above: 1.92 %

SHIC statusSHIC status

Version 2.79 released october 2000; 3.012 now tested; UNIX version tested

flagging included; faster (1-2 sec per spectrum)

spectral shape checks improved shifts below 300 nm possible

further improvement: output control

Next year activitiesNext year activities New release of SHIC december 2001 Application of flagging on subsets of data Use in climatological analysis (calculation of

effective UV in standardized way) improvement of algorithm to identify

atmospheric variability during the scans (OE) implementation at the database (in

collaboration with FMI, Ola Engelsen) uncertainty analysis in relation to FWHM