LTPDA A data analysis framework for LISA PathÞnderhewitson/presentations/presentations...A data...

transcript

LTPDAA data analysis framework for LISA PathfinderM Hewitson for the LTP team

LISA BB Meeting, Hannover, January 2007

On the critical path to LISATechnology demonstrator for LISA

drag-free controlfree-falling test-mass acceleration noise

LPF Mission

Laser Interferometer Space Antenna

5x106 km

LPF Mission

30cm5x106 km

LPF Mission

5x106 km

LPF Mission

5x106 km

LPF Mission

5x106 km

LPF Mission

5x106 km

Launch: mid-2010

Measurements

IFO signalsmeasure longitudinal for TM1/SC and TM1/TM2measure TM angles

Capitative sensorsx,y,z for each TM3 angles for each TM

Many environmental monitors, e.g.temperaturetest-mass chargeparticle countersmagnetometers

DA challenges

DA challengesVery limited data download rate (15kbps)

DA challengesVery limited data download rate (15kbps)Interesting features are on time-scales similar to experiment lengths

each experiment ~24Hlooking at <mHz in 24 hours of data

DA challengesVery limited data download rate (15kbps)Interesting features are on time-scales similar to experiment lengths

each experiment ~24Hlooking at <mHz in 24 hours of data

Signals sampled with different clocks at different rates

1/600Hz - a few Hz sample ratesneed to resample and synchronise signals before doing coherence analyses

Data analysisNeed the ‘usual’ commissioner’s tool-set

Spectral estimationsDigital filtersModel fittingSimulated signals...

Data analysis requirementsUnusual mission

data analysis is part of mission operationsdata analysis software is operational software

Use commercial software environment

Use commercial software environmentMATLAB

All data analysis results should be accountable and reproducible

Concept of Analysis Objects

All data analysis to be done ‘graphically’

All data analysis to be done ‘graphically’allow for non-experts in MATLAB

Multi-user data access

Multi-user data accessclient/server system

Analysis Objects

Analysis ObjectsWhat a result isn’t:

an image

an imagea plot

an imagea plota text file full of numbers

Analysis Object

Numerical

Provenance

Processing

history

Additional

meta-data

- creator

- date

- IP address

- Hostname

- Operating System

- software versions

- Name

- ID number

- Comment

- pipeline file(s)

- Name

- Algorithm version

- Parameter list

- Creation date/time

- Input histories

- Name

- Numerical data

vectors

- Creation date/time

- Additional flags

Taking care of history

Output

Algorithmic step

history

Algorithm

history

Intelligent algorithms

Taking care of history

Output

Algorithmic step

history

Algorithm

history

Intelligent algorithms

Implementation

Environment is:object-orientedimplemented as MATLAB toolboxGraphical programming achieved using SIMULINK as a ‘drawing pad’

Toolbox contents

Toolbox contents•toolbox comprises

• new classes

• helper functions

• signal processing functions (algorithms)

• documentation

• new classes

• documentation

Core functionality is ~65000 lines of MATLAB code

• new classes

• documentation

GUIs ~6000 lines of MATLAB code

• new classes

• documentation

GUIs ~6000 lines of MATLAB code

Documentation is ~62000 lines of html (includes src, some automatically

generated)

Class hierarchy

timespan

pzmodel

miir mfir specwin

timeformat

pole zero

tsdatafsdataxydata cdata

provenance

history

ao User Classes

Class hierarchy

timespan

pzmodel

miir mfir specwin

timeformat

pole zero

tsdatafsdataxydata cdata

provenance

history

ao User Classes

Benefits of classes

Benefits of classesMany and various constructors

Benefits of classesMany and various constructorsDefine object properties in central location

'name' 'data' 'hist'

'provenance' 'description'

'mfile' 'mfilename'

'mdlfile' 'mdlfilename'

'version' 'created'

Benefits of classesMany and various constructorsDefine object properties in central locationOverload operators

can directly operate with AOs

'mfile' 'mfilename'

'version' 'created'

% Create cdata AOsa1 = ao(10);a2 = ao('foo.xml'); % Add thema3 = a1+a2; % Subtract constanta4 = a3-10; % Testsif a1 < 100 disp('small ao');end

Overload functionsexamples:

'mfile' 'mfilename'

'version' 'created'

Overload functionsexamples:

exp mean sqrt abs export median stairs acos fft minus std ao filter mpower string ao2m filtfilt mrdivide submit aosplit find mtimes submit_fastinsert asin ge mux subsasgn atan get ne subsref attachm getAOdata norm sum attachmdl gt phase svd cat hist plot tag char imag plus tan complex index polyfit testAO conj interp power times cos inv rdivide transpose ctranspose iplot real var decimate join resample xml demux le save xmladd det len select xmlparse diag ln set display log simple_plot eig log10 sin eq lt split

'mfile' 'mfilename'

'version' 'created'

Visualising data

>> figure, iplot(a4)

Visualising data

>> figure, iplot(a4)

Inside the functions

b = foo(a,pl)

output AOs

b = foo(a,pl)

output AOs input AOs

b = foo(a,pl)

output AOs input AOs input parameters

b = foo(a,pl)Description

in standard MATLAB format so that ‘help fcn’ works as it should

Unpack input AO(s) and PLIST(s)

Unpack input AO(s) and PLIST(s)do something to data

Unpack input AO(s) and PLIST(s)do something to datacreate new output data object (tsdata or fsdata)

Unpack input AO(s) and PLIST(s)do something to datacreate new output data object (tsdata or fsdata)create new output history

h = history(ALGONAME, VERSION, pl, [a.hist b.hist])

History of input AOs

Create new output AO(s)

Create new output AO(s)same for wrappers and new functions

Example wrapper

function varargout = ltpda_pwelch(varargin)% LTPDA_PWELCH makes power spectral density estimates of the time-series objects%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% DESCRIPTION: LTPDA_PWELCH makes power spectral density estimates of the% time-series objects in the input analysis objects.%% CALL: b = ltpda_pwelch(a1,a2,a3,...,pl)%% INPUTS: b - output analysis objects% aN - input analysis objects% pl - input parameter list%% Makes PSD estimates using ltpda_psd() of each input AO.%% If the last input argument is a parameter list (plist) it is used.% The following parameters are recognised.%% PARAMETERS: 'Win' - a specwin window object [default: Kaiser -200dB psll]% 'Nolap' - segment overlap (number of samples) [default: taken from window function]% 'Nfft' - number of samples in each fft [default: fs of input data]% 'Debug' - debug level for terminal output (0-1)%% VERSION: $Id: ltpda_pwelch.m,v 1.12 2007/07/16 12:52:21 ingo Exp $%% HISTORY: 07-02-2007 M Hewitson% Creation%% The following call returns a parameter list object that contains the% default parameter values:%% >> pl = ltpda_pwelch('Params')%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Example wrapper

Win = find(pl, 'Win'); % Window object if isempty(Win) Win = specwin('Kaiser', Nfft, 200) disp(sprintf('! Using default Window of %s', strrep(Win.name, '_', '\_'))) end plo = append(plo, param('Win', Win)); Nolap = find(pl, 'Nolap'); % Amount to overlap each fft if isempty(Nolap) Nolap = round(Win.rov*Nfft/100); disp(sprintf('! Using default overlap of %d samples', Nolap)) end plo = append(plo, param('Nolap', Nolap));

Example wrapper

% Compute PSD using pwelch [pxx, f] = pwelch(d.x, Win.win, Nolap, Nfft, d.fs);

Example wrapper

% Compute PSD using pwelch [pxx, f] = pwelch(d.x, Win.win, Nolap, Nfft, d.fs);

% create new output fsdata fs = fsdata(f, pxx, d.fs); fs = set(fs, 'name', sprintf('PSD(%s)', d.name)); fs = set(fs, 'yunits', [d.yunits '/\surdHz']); fs = set(fs, 'enbw', Win.nenbw); % create new output history h = history(ALGONAME, VERSION, plo, a.hist); % make output analysis object b = ao(fs, h); % set name b = set(b, 'name', sprintf('PSD(%s)', a.name)); % add to outputs bs = [bs b];

History structure

history

version

History structure

history

version

input histories

version

params

History structure

history

version

input histories

version

params

input histories

version

params

History structure

history

version

input histories

version

params

input histories

version

params

input histories

version

params

History structure

history

version

input histories

version

params

input histories

version

params

input histories

version

params

History structure

history

version

input histories

version

params

input histories

version

params

input histories

version

params

Entire processing history of every AO is recorded

Reliving history

%% Reproduce from history% Write an m-file from AOao2m(a4, 'test.m'); edit 'test.m'

Reliving history

function a_out = test % TEST.M % % % written by ao2m / $Id: ao2m.m,v 1.11 2007/11/14 16:30:18 ingo Exp $% % based on analysis object:% name: a3 - 10 / ((Data) + (Data)) - (Data)% provenance: created by hewitson@localhost[192.168.2.104] on MACI/7.5 (R2007b)/0.7 (R2007b) at 2008-01-06% description: % original m-file: % % a7331931 = ao(plist([param('VALS', [10]) ]));a7331814 = ao(plist([param('VALS', [[1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 a7331773 = ao(plist([param('VALS', [10]) ]));a7331865 = plus(a7331773, a7331814);a7331960 = minus(a7331865, a7331931);a_out = a7331960; % END

Reliving history

function a_out = test % TEST.M % % % written by ao2m / $Id: ao2m.m,v 1.11 2007/11/14 16:30:18 ingo Exp $% % based on analysis object:% name: a3 - 10 / ((Data) + (Data)) - (Data)% provenance: created by hewitson@localhost[192.168.2.104] on MACI/7.5 (R2007b)/0.7 (R2007b) at 2008-01-06% description: % original m-file: % % a7331931 = ao(plist([param('VALS', [10]) ]));a7331814 = ao(plist([param('VALS', [[1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 a7331773 = ao(plist([param('VALS', [10]) ]));a7331865 = plus(a7331773, a7331814);a7331960 = minus(a7331865, a7331931);a_out = a7331960; % END

Visualise history

>> figure, plot(a4.hist)

Visualise history

>> figure, plot(a4.hist)

Saving your objectsAll LTPDA (user) objects can be saved to file (and loaded from)File format is XML

.mat works as well

>> p = param('a', 2)---- param 1 ----key: aval: 2-----------------

.mat works as well

>> p = param('a', 2)---- param 1 ----key: aval: 2----------------->> save(p, 'param.xml')

.mat works as well

>> p = param('a', 2)---- param 1 ----key: aval: 2-----------------

<?xml version="1.0" encoding="utf-8"?><param> <Object> <param> <version>$Id: param.m,v 1.9 2007/08/31 17:40:08 hewitson Exp $</version> <key>a</key> <value>2</value> </param> </Object></param>

>> save(p, 'param.xml')

Signal processing

% Load raw datao1 = ao('o1_1.xml');o12 = ao('o12_1.xml'); % Compute ASDs pl = plist('Win', specwin('Kaiser', 10, 150), 'Nfft', o1.data.fs*20000); o1_xx = ltpda_pwelch(o1, pl);o12_xx = ltpda_pwelch(o12, pl); % Save objectssave(o1_xx, 'o1_xx.xml');save(o12_xx, 'o12_xx.xml');

Signal processing

Load Data

Signal processing

Process Data

Load Data

Signal processing

Process Data

Load Data

pl = plist('Win', specwin('Kaiser', 10, 150), 'Nfft', o1.data.fs*20000);

Signal processing

Process Data

Load Data

Signal processing

Process Data

Load Data

Save Data

More complicated e.g.Frequency-domain calibration of IFO outputs from MDC1

Cdf = df_controller(Sdf.data.x); %ao('Cdf.xml');Ssus = ao('Ssus.xml');Sw1 = ao('Sw1.xml');Sw3 = ao('Sw3.xml'); % spectrao12xx = ao('o12_xx.xml');o1xx = ao('o1_xx.xml'); % Other valuesw1 = ao('w1.xml');w3 = ao('w3.xml');delta = ao('delta.xml');wd = ao('wd.xml'); % Load reference tracesa11r = ao('../../../ltp_noise_models/A11xx.xml');a12r = ao('../../../ltp_noise_models/A1212xx.xml'); %% Calibratet1 = abs(Sw1 - Cdf); a1xx = o1xx .* t1;a1xx = set(a1xx, 'name', 'ASD(a1)');beta = wd.^2 - delta.*Sw3; t3 = Sw3./Ssus;a12xx = abs(o12xx.*t3 + beta.*o1xx);a12xx = set(a12xx, 'name', 'ASD(a12)'); %% Plot

figureiplot(a1xx , a11r) figureiplot(a1xx ./ a11r)allyscale('lin') % END

Load Objects

Manipulate Objects

Looking at the history

Data access

LTPDA Repository

MySQL Server

converter

Datafrom S/C

MATLAB Client

Centralised data distributionMultiple clients can access the data at the same timeData can be accessed from anywhere

firewall permitting

idhashxml

objsidobj_idobj_typenamecreatedversioniphostnameossubmittedcomment1comment2comment3comment4comment5comment6validatedvdate

objmeta

idobj_iduser_idtransdatedirection

transactionsidnobjsobj_ids

collectionsidfirstnamefamilynameusernameemailtelephoneinstitution

idobj_iddata_typedata_idmfilenamemdlfilename

aoidobj_idin_filefs

miir/mfir

idxunitsyunitsfsnsecst0

tsdataidxunitsyunitsfs

fsdataidxunitsyunits

cdataidxunitsyunits

xydata

The LTPDA Repository

Share your LTPDA objects with your friendsclient/server systemserver:

mysql database

client:MATLAB LTPDA functions built around Database Toolbox

Can store:filters, parameter lists, AOs, etcjust pass object ref ID to your friends

Interaction in MATLABSince we have a database at the core, we can:

submit objectsquery for objectsretrieve objects

all from within MATLAB

Repository GUI - submit

Describe object(s)

Retrieve

Retrieve:object 12

collection 1

Retrieve

Retrieve:object 12

collection 1

Graphical programmingusing SIMULINK

Analysis diagrams

Diagram is executed from main control panelCalls functions in the underlying toolbox

Get your copy now!

http://www.lisa.uni-hannover.de/ltpda/index.html

LTPDA A data analysis framework for LISA PathÞnderhewitson/presentations/presentations...A data...

Documents