Post on 13-Dec-2015
transcript
ORGANISATION OF THE MSDP DATA PROCESSING
Thierry ROUDIERNadège MEUNIERPierre MEIN
MSDP Workshop, Tarbes, 18-20th January 2006
PLAN
Codes : choice and availability Organisation of the directories, files : input The parameter files (short) The different steps through the processing (short) The output files : interpretation More details about the parameter files associated to each step
Code: choice and availibility
Only existing code : developped by Pierre Mein
Public : available on our web site http://bass2000.bagn.obs-mip.fr/
Acknowledgements in publications
SOFTWARE AND DOCUMENTATION
SOFTWARE: http://bass2000.bagn.obs-mip.fr/New2003/Pages/DPSM/dpsm_acceuil.html
DOCUMENTATION:
GENERAL: - readme.txt general user guide - auto.txt user guide for msdpauto - sequence.txt example of data for msdpauto - param.txt parameter list of ms.par
OTHERS : - correction.txt parameter list to modify in various cases - captions.txt plot control -filenames.txt filename description at the different processing steps - remarks.txt a few examples and difficulties - journal.txt list of successive improvements of the code - signs.txt give the sign of the result - widget.txt widgets information (not updated) - vtt.txt information for the VTT observations
Organisation of the files and directories
Parameter files Data files (scan, flat-fields, dark
current, field-stop) IDL routine msdpauto
create the directory for output files cut raw 3D files into 2D files (1 im / file) create the parameter file ms.par start the fortran code ms1
Fortran code ms1 process the data
Individual 2D files 1 b3 file (scan), with n images : 1 starting
time Creation of n 2D image file : filename
including an artificial time (increment of 1) example :
Usefull to limit the number of files to process after this step (tob1, tob2)
Actual time of each image : .log file obtained at THEMIS
c031031_13182784_00111c031031_13182785_00111c031031_13182786_00111c031031_13182787_00111c031031_13182788_00111
t*fts sequence.par
/data/ /data2/auto/
msdpauto
tyyyy.par
N, L, S
/data2/auto/dir_date_Nseq_L/key1
key.par
b*.fts
ms.par
Parameters
Conversion
ms1Computation
Option /no_fort
MSDP DATA PROCESSING
N=sb=seq.
L=cm=line
S=qv=Stokes
ms1
x*L*z*L*
y*L*Sc*L*Sd*L*Sq*L*Sr*L*Sp*L*S
geo.ps g*Lflat.ps f*L*Sgrid.pscmd*.ps quick.ps j*L*Scmr*.psprof.pssq*L*S.pssp*L*S.ps
ms.lisscan.lis
readmsdp
Quick-look
Spectroheliog
Averages
Channels
Bisect.
Profiles
Calib.
steps results .ps files Ascii files
The parameter files Telescope related file : tyyyy.par
include instrumentalk set-up informations
can change over the years Sequence related file : sequence.par
1 line per sequence liste of files to process association obs / calib info steps, polarimetry … see header keywords
Processing info : ms.par (BIG FILE !) characterizes sequence + line
Steps Corrections Files Output results
geom calib
Power fctsScattered lightNormalizationSmoothing Profile curvatureFourier filteringCospatiality
2D - correlAverage departures
geoflatbmc
cmd
b
c
d
q
r
p
Aligned and calibrated channelsPossible direct inversionavoiding interpolation corrections
Individual maps I, v, B//Possible destretching
Large maps I, v, B//
Individual maps Profiles I, Q, U, V with calibrated central wavelength
Large spectrohéliog. I,Q,U,V Inversions with constant
quick
cmr
prof
Like cmd except cospatiality
Like quick except 2D - correl
The different steps through the processing
The ouput files: interpretations
One postscript file per step Binary files with results Ascii files with messages
GEO.PS
Intensity gradients
A D
BE
The extrema define the channel edges in i
The program computes the regression line for projected vectors (AD,BE,…) on i and j
Channels location
Intensities
Flat fields
Line curvature correction Mean profile determination Elimination of the mean profile Check that the result is « flat » :
flat.ps
FLAT.PSMinimum signal (line core ) + parabolic adjustement
Shift at same between 2 successive channels (ltrj) Mean profile of successive channels
Mean profile after transmission correction for the 1st window
Control of the even and odd interlacing channels (box 16 channels)
Mean profile kept
Start of 1st channel
Idem 2nd window
Channel cut along i
Start of the last channel
Channel cut along j
Cuts mean along j for all the channels
Idem 2nd box
Results
For each step, one file containing everything (I, B// … )
Order and number of the images in the file depends on : observing condition (polarization or not) number of steps chosen for the output Stokes
profiles As many q* or p* files as the number of
Stokes parameters A file per scan To read the files : IDL routine readmsdp
Standard quicklook output filewith no polarization
images #1, 2, 3, 4 : intensities (close to line center, aver. I at ±6Δλ , diff. between I at ±6Δλ ~ V// and aver. I at ±6Δλ bissector)
image #5 : V// at ±6Δλ (bissector)
Δλ=dlambda/2 if 9 channels, dlambda/4 if 16 channels
dlambda = distance between channels
images #1, 3, 5, 7 : intensities (close to line center, aver I at ±6Δλ , diff. between I at ±6Δλ ~V// and aver. I at ±6Δλ bissector) ; repeated n times (n=number of Stokes meas.)
image #8 : co-spatiality map : diff. between I at ±6Δλ
images #2, 4, 6 : Stokes Q (or U, V) close to line center and at ±6Δλ, + difference when Stokes V (~ B//)
image #9 : V// at ±6Δλ (bissector) image #10 : B// at ±6Δλ (bissector)
Standard quicklook output filewith polarization
Final output p* filewith no polarization
images #1 to17 : Stokes I profile around line center, ±nΔλ, and n from –8 to +8
images #19, 21 : V// at ±4Δλ and ±8Δλ (bissector)
images #18, 20 : aver. I at ±4Δλ and ±8Δλ (bissector)
images #1 to 17 : Stokes I profile around line center, ±nΔλ, and n from –8 to +8
images #18 to 34 : Stokes profile around line center, ±nΔλ, and n from –8 to +8
images #37, 41 : V// at ±4Δλ and ±8Δλ (bissector)
images #38, 42 : B// at ±4Δλ and ±8Δλ (bissector) if Stokes V
images #35, 39 : aver. I at ±4Δλ and ±8Δλ (bissector)
images #36, 40 : diff. between I at ±4Δλ and ±8Δλ for cospatiality tests (bissector)
Final output p* filewith polarization
ASCII files
scan.lis : small text file ms.lis : very long file, prints and
warning for all steps of the computation
tyyyy.PAR
tyyyy.par (THEMIS), pyyyy.par (Pic du Midi), vyyyy.par (VTT), myyyy.par (Meudon)
yyyy : year (may be constant or change)
Contents instrumental configuration processing and output options :
WARNING ; example number of points in the
profiles lmpr1*2+1 ; Δλ = lbd1r1
(nl) lbd ncha grorder nbox jt1000 ja1000 jb1000 1 4861 9 47 1 2 4861 16 46 2 3 5173 16 44 2 4 5876 16 38 2 2903 83 5 5890 16 38 2 6 5896 16 38 2 7 6103 16 37 3 8 6563 9 34 1 9 6563 16 34 2 10 8542 16 26 2
(nbox) inveri inverj invi invj invern inverl invers nlisd nlisr 1 1 1 1 0 0 1 0 0 0 2 0 0 1 0 1 0 1 2 2 3 0 0 1 0 1 0 1 2 2 4 1 1 1 0 0 1 0 0 0
tl sb sx sy sz cm bs yy mm dd lbd go stx dt sty ny ng nq qv nb bt qp sd
1 3 3 3 3 2 16 03 10 17 0 0 0 60 0 0 4 3 0 1 0 0 2
1 5 5 5 5 0 16 00 08 24 8542 0 5000 60 8500 4 3 1 1 1 0 0 1
1 6 6 6 6 0 16 00 08 24 5890 0 5000 60 11000 3 3 3 3 1 0 0 1
1 8 8 8 8 0 16 00 08 24 4861 0 5000 60 11000 3 3 1 1 1 0 0 1
1 9 9 9 9 0 16 00 08 24 4861 0 5000 60 11000 3 3 1 1 1 0 0 1
end
Télescop
séquence number
obs.
d.c.
f.f.
f.s.
caméra
channelnumber
date
grating order
X step
t between scans in 1/10 de s.
polarisation
burst
up to the stage « q » ou « p »
0 = sun1 = dec2=linux
SEQUENCE.PAR
Manual or =0 for file header
MS.PAR
Parameters : fixed (derived from tyyyy.par,
sequence.par, headers, …) variables depending on the
options, problems
Choose the data level : ixy, igeo, iflat, ibmc, icmd, iquick, icmr, iprof, igrayq, igrayp
Modify the thresholds (geometry determination, rejection, …) : milgeo, si, sj, sgi, sgj, etc.
Remove pieces of images (borders) : nob, nob2, ix1, ix2, etc…
Choose the output spatial step : milsec Normalize intensities (in case of clouds) : norma Symetrize the image (scanning, Stokes sign,
direction) : inveri, inverj, invi, invj, invers, etc … Filter and smooth : crecd, w1d, w2d, w3d, lcrecq,
etc. Choose the chords : lmpd, lbd1d, lbpasd Choose to print the results
Main options
tel dob nseq nline ncam1 ncam2 1 20031017 3 2
MSDPBMS WAVELNTH GRORDER FSLTH FSWTH STEP_X NBSTEP_X 16 5896 0 60000 300000 5000 20 STEPGRID NBSTGRID GRID_MAX GRID_PER GRID_WID SEQ_STOK BURST 8500 4 0 0 0 3 0
Télescop Date obsSequence number
Camera number
Parameters non used in ms.par
MS.PAR
FILE obs.par
nm lbda dlbd mupris mustep minpro xfirst
8 5896 80 3300 800 500
Number of channel c/ (window)
Distance between 2 channels (mAngs.).
Lambda (Angs.)
multi-slit stepbox (micron)
Translation between channels(prisms box) (micron)
Normalisation of the profile, value ajusted at the line center
nwinp mgrim nquv ipos burst select polord
2 4 3 4 1
ntmax priscan jypas interc uint
0 0 5000 15
Number of (window) / image
Nombre d’état de polarisation
Number of positions Y-scan (in polarisation)
Number of imagesby burst
Maximun number , step of the grid (in polarisation)
Number of’imagesby scan
Prisms order For the field
Step in X of the sweep (here 5’’.0) (arcsec/1000) Approximative distance
Between the end and the beginning of the channels fUnity=pixel CCD
win kdecal
2 0
1 50
nbcln nblgn li lj invern
1035 921 133000 9000 1
1035 921 133000 9000 1
Number of the window
Channels interlacing
Number of pixels in the window in i
Nombre de pixelsDe la fenêtre en j
Field size arcsec in i (*1000)
Field size in arcsec en j (*1000)
To modify the channel order
cqp
inveri inverj inverl norma scatter etal
1 0 0 0 0 0
ix1 ix2 jy1 jy2 jyq1 jyq2
0 133000 500 8500 500 8500
0 133000 500 8500 500 8500
Symmetrize the maps / i
Symmetrize maps / j
Reverse the orientation (lambda)
Normalize intensity(example: clouds)
Diffusion rate(scatter/1000)
not used
Take off the edgein x ,in arcsec
Take off the edgein y , in arcsec
Same for the out files« p » et « q »
invi invj istep invers (istep et invers echange)
1 0 8500 0
Reverse out mapsReverse the signs of Stokes parame ters
Step in Y (STEP_Y)(arcsec/1000) en polarisation
FILE exe.par
dir
/home/lafon/dpsm/data/dir3_2/
filter b000000_000_000_000000_m0000_00000000.fts
ixy igeo iflat ibmc
1 1 1 1
icmd iquick icmr iprof igrayq igrayp
1 1 0 0 1 0
Directory of files b
Filter of files b
Différentes step : 1 for use 0 else
tob1 tob2
0000000024000000
tdc1 tdc2
0000000024000000
tfs1 tfs2
0000000024000000
tff1 tff2 nff
0000000024000000 1
24000000
24000000
Start and end of the observation to be traited
Hours min et max of dark current
Hours min et max of field stop
Hours min et max of flat field Numbre offlat fields used divided by nqff
tcl1 tcl2
0000000024000000
sundec iswap intert ipermu nqseul milsec
0 1 600 1 0 250
bmg
si sj sgi sgj milang milgeo nleft nright
0 15 15 15 0 3500 0 0
0 15 15 15 0 3500 0 0
Hours for geometric calibrations
Ordinateur type
No used
Swap or non
Minimun time-stepBetween 2 scans (1/100) seconde
Echange X et Y
Number of couples (if polarisation)
out put pixel size, here 0.25 arcsec
IntensitY threshold in i et j to detect the channels
Geometry thresholdRegression differencein 1/1000 de pixel
Channels anglegradients intensity threshold inn i et j to detect the channels
To determine the channel left edge du (right) from neighbourg channel.
cmf
inclin milrec calfs caldeb
1 500 0 1
cqp
ideb igri itgri itana jtana calana milalp milzero ijlis
0 12000 33500 16298 0 0 0 0 0
Intery
ilisdr jlisdr mincmd maxcmd ilisqp jlisqp
Type of computation for the relative Channels transmission
Threshold for alignementbetween FF and FS
Beam translations of the separator for polarisation in i and j.
Computation by the program of grid position (polarisation)
Util size in arcsec/1000 of grid plages (polarisation)
Grid period arcsec/1000
1st point of the first util plage of the grid arcsec/1000 (position)
Shift adjustement xy of of the analysor (polar. circ.)
Intensity change for the signalbefore interpol. (I **a )
Spatial smoothing ,noise
Type of the detection of the line shape
cmd
cented sumd nlisd curvd crecd w1d w2d w3d ratiod
1 0 2 0 2000 0 1 0
Intensity line core computation
Direct output from channels
Profil smoothing Curvature correction by using neighbourg points .
Fourier Filtering to correct « cannelures «
lmpd lbd1d lbpasd
0 0 0
0 0 0
2 1500 1500
quick
crecq milsigq lcrecq
0 2000 0
cmr
center sumr nlisr curvr crecr w1r w2r w3r ratior
1 0 2 0 2000 0 1 0
lmpr lbd1r lbpasr
7 500 500
0 0 0
0 1500 1500
Réjection by computing the mean of values with gap graeter than sigma *milsigq/1000.
Mean gap correction
Smooth in y
Parameter définitions identical to those of « cmd »
Spectrohéliogrammes (no used at cmd step because car non calibratec).
bissectors
Sum and différence (blue and red wings)
1s rope : 1.5 * dlbd=1.5 * 80= 120 mA2 2nd rope : 3.0 * dlbd=3.0 * 80= 240 mA
prof
crecp milsigp lcrecp
0 2000 0
FILE fix.par
reg lin linref iplotg iplotf nqff
0 0 0 2 4 3
npol
1
bmg
(win) i1 i2m j1 j2m lip jeps intvi intvj
1 1 0 1 0 40 20 30 20
2 1 0 1 0 40 20 30 20
Plot géo.ps
Plot of flat.ps
FIX PARAMETERS
Define the Stokes parameters succession for flat field
No used
No used
Window number
1st pixel used and gap to the last pixel in i et j
Interval between in i used to measure the channels curvature, here 40%
Interval to search in j the edges i n i (gretaer length ) at + or - jeps pixels
Intervals in i et j to compute the means to detect the edges in j and i
(win) leps n1 distor normsq dlxy
1 40 1 1
2 40 1 1
bmc
idc dxr100 dyr100 dxrmms dyrmms
1 0 0
cmf
smoothi smoothj il1p il2p isym iextra iff
0 0 10 90 0
l
1st util channel Take into account the channel curvature
For the dark current
Window number
The corrections in each Falt field channel are replaced or not by means
Restric the mean profile computations of the spectral line
Symmetrized profile
Search interval of points with gradients maximun to +/- leps
Small shift between flat field and scan images
dxdust dydust x1dust x2dust y1dust y2dustWcs ncs acs1 zcs1 bcs1 acs2 zcs2 bcs2
(win) curv iliss jparli lispro deconv
1 1 10 5 10
2 1 10 5 10
(win) jt100 ja100 jb100 jz100 jtcor
1 0 0 0 0
2 0 0 0 0
cmd/cmr
longw lat absord absorr mps cstok
0 0 1 1 1
Smoothing in i before the detection of the line core
Parabolic smoothing in j before the detection of the line core
Mean profil smoothing used to compute the corrections
Window number
No used Profil in absorption or emission for files d
To take care of the curvature of the line
Window number
Translation in j, in pixel/100, corresponding to the difference in between 2 channels
Define the tilt and curvature of the line in each channel
Same for files r
Specify the velocity unit in m/s
If 0 parameters are computed by the program
quick
lcorq jlap2q icormq copasq milcoq decmq
0 0 0 0 0 0
prof
lcorp jlap2p icormp copasp milcop decmp
0 0 0 0 0 0
gray
igrq jgrq igrp jgrp imax
3 2 4 2 0
Indice of the used board for the 2D spatial correlation
½ interval of overposition between 2 frames of the scan
size for the correlation computation Step for the computation of the first derivative over x
The result is not taking into accountif the maximun pf the 2D correlation is less than milcoq /1000
No usedParameters identical to quick
Number of plots in horizontal et vertical files q
Idem files p Maximum number of pixels in y direction y for all the sweep. Allows to adjust the graphic scale p and q