A Test Framework for Movement of MC Data

using SAM & SRM Interface

2nd March, 2008CDF Offline Meeting

2

Overview

Test FrameworkComposition of filesMonitoring Parameters

Time PlotTransfer RateCumulative Data PlotQuery Map

Conclusion & Next StepWish List !

Data Movement: Test Framework

Data from SAM, dccp

fcdflnx1cdfsam15

FNAL SRM V2srm://cmssrm.fnal.gov:8443

UCSD SRM V2srm://t2data2.t2.ucsd.edu:8443

scpsrmcp

srmcp

Composition of Files

4A SAM data file is assosciated with its meta data file.

Composition of Files: File Size

5

No. of Files No. of Files

Meta Data File Size in Bytes Data File Size in Mega Bytes

File S

ize

Meta data files are of the order of few Kilo bytes.Data files are the order of several Mega Bytes.

Monitoring Parameters: Transfer Time

6

No. of Files No. of Files

Meta Data Transfer Time in Sec Data Transfer Time in Sec

Tim

e T

aken

Meta data files (few Kilo bytes) take around 14 seconds.Data files takes (several Mega bytes) around 34 seconds.

Monitoring Parameters: Evolution of files with time

7

Time Time

Meta Data File Size in Bytes Data File Size in Mega Bytes

File S

ize

A dip in the right hand side plot (18:32 – 19:02) is being balanced by copying meta data files during this period.

Monitoring Parameters: Transfer Rate

8

No. of Files No. of Files

Meta Data Tr. Rate in Bytes/sec Data File Tr. Rate in MBs/sec

Tra

nsfe

r R

ate

Transfer rate for larger size files is much higher than smaller size files.

Monitoring Parameters: Cumulative Flow of Data

9

File S

ize

Time

More than ~30 Giga Bytes/hr of data has been transferred. More test is needed to confirm a robust transfer rate.

Monitoring Parameters: Query Map

10

Not easy to draw any conclusions from the above plot. But, it will be helpful to display quality data for multiple sites versus time. Each box represents a fixed duration worth of transfers.

Conclusion & Next StepConclusion:

Larger the file size, higher the efficiency of the framework.

Next Steps Different sizes of files will be used

For files of other than above different sizes, joining files with “tar” is one option. But, it is too much cpu consuming.

In the next offline meeting, we will show plots on monitoring parameters for trans Atlantic movement of data. 11

DataSet Each File Size (in MB)

1. atop48 500

2. bhel0d 737

3. b0dsdka 850

4. ftop1b 1050

5. bhmu0i 1350

Wish List !

Addition of more SRM managed storage element sites for studying the performance of the framework.

Compatibility of SAM with SRM version 2 Present version of SRM in SAM is 1 Latest version of SRM is 2. SRM version 2 is not backward compatible

with SRM version 1. Present version of SAM needs some development such that it would be

compatible with SRM version 2 A rough idea of the time frame ( under which the above issues

will be resolved) will be helpful in speedy development of this work !

12