Upgrade D0 farm. Reasons for upgrade RedHat 7 needed for D0 software New versions of –ups/upd v4_6...

transcript

Upgrade D0 farm

Reasons for upgrade

• RedHat 7 needed for D0 software

• New versions of – ups/upd v4_6– fbsng v1_3f+p2_1– sam

• Use of farm for MC and analysis

• Integration in farm network

MC production on farm

• Input: requests

• Request translated in mc_runjob macro

• Stages:1. mc_runjob on batch server (hoeve)

2. MC job on node

3. SAM store on file server (schuur)

farm server file server

SAM DB

datastore

fbs(rcp,sam)

fbs(mcc)

mcc request

mcc input

mcc output

1.2 TB

FNALSARA

control

metadata

fbs job:1 mcc2 rcp3 sam

100 cpu’s

SAM DB

datastore

fbs(rcp[,sam])

fbs(mcc)

mcc request

mcc input

mcc output

1.2 TB

FNALSARA

control

metadata

fbs job:1 mcc2 rcp

100 cpu’s

cron:sam

fbsuser:cpfbsuser:mcc

fbsuser: rcp

willem:sam

hoeve node schuur

fbsuser:mc_runjob

fbs submit

control

SECTION mcc EXEC=/d0gstar/curr/minbias-02073214824/batch NUMPROC=1 QUEUE=FastQ STDOUT=/d0gstar/curr/minbias-02073214824/stdout STDERR=/d0gstar/curr/minbias-02073214824/stdoutSECTION rcp EXEC=/d0gstar/curr/minbias-02073214824/batch_rcp NUMPROC=1 QUEUE=IOQ DEPEND=done(mcc) STDOUT=/d0gstar/curr/minbias-02073214824/stdout_rcp STDERR=/d0gstar/curr/minbias-02073214824/stdout_rcp

#!/bin/sh

. /usr/products/etc/setups.shcd /d0gstar/mcc/mcc-dist. mcc_dist_setup.sh

mkdir -p /data/curr/minbias-02073214824cd /data/curr/minbias-02073214824cp -r /d0gstar/curr/minbias-02073214824/* .touch /d0gstar/curr/minbias-02073214824/.`uname -n`sh minbias-02073214824.sh `pwd` > logtouch /d0gstar/curr/minbias-02073214824/`uname -n`/d0gstar/bin/check minbias-02073214824

#!/bin/shi=minbias-02073214824if [ -f /d0gstar/curr/$i/OK ];thenmkdir -p /data/disk2/sam_cache/$icd /data/disk2/sam_cache/$inode=`ls /d0gstar/curr/$i/node*`node=`basename $node`job=`echo $i | awk '{print substr($0,length-8,9)}'`rcp -pr $node:/data/dest/d0reco/reco*${job}* .rcp -pr $node:/data/dest/reco_analyze/rAtpl*${job}* .rcp -pr $node:/data/curr/$i/Metadata/*.params .rcp -pr $node:/data/curr/$i/Metadata/*.py .rsh -n $node rm -rf /data/curr/$irsh -n $node rm -rf /data/dest/*/*${job}*touch /d0gstar/curr/$i/RCPfi

batchruns on node

batch_rcpruns on schuur

#!/bin/shlocate(){file=`grep "import =" import_${1}_${job}.py | awk -F \" '{print $2}'`sam locate $file | fgrep -q [return $?}. /usr/products/etc/setups.shsetup samSAM_STATION=hoeveexport SAM_STATION

tosam=$1LIST=`cat $tosam`

for job in $LISTdo cd /data/disk2/sam_cache/${job} list='gen d0g sim' for i in $list do until locate $i || (sam declare import_${i}_${job}.py && locate ${i}) do sleep 60; done done

list='reco recoanalyze' for i in $list do sam store --descrip=import_${i}_${job}.py --source=`pwd` return=$? echo Return code sam store $returndonedoneecho Job finished ...

declare gen, d0g, sim

store reco, recoanalyze

runs on schuurcalled by fbs or cron

Filestream

• Fetch input from sam

• Read input file from schuur

• Process data on node

• Copy output to schuur

hoeve node schuur

mc_runjob

fbs submit

control

attach filestream

Analysis on farm

• Stages:– Read files from sam– Copy files to node(s)– Perform analysis on node– Copy files to file server– Store files in sam

SAM DB

datastore

1.2 TB

FNALSARA

control (fbs)

metadata

100 cpu’s

1. sam + rcp2. analyze3. rcp + sam

fbs(1), fbs(3)

fbs(2)

triviaal node-2

fbsuser:rcp

fbsuser:

analysisprogram

willem:sam

output

SECTION sam EXEC=/home/willem/batch_sam NUMPROC=1 QUEUE=IOQ STDOUT=/home/willem/stdout STDERR=/home/willem/stdout

#!/bin/sh

. /usr/products/etc/setups.shsetup samSAM_STATION=triviaalexport SAM_STATION

sam run project get_file.py --interactive > log

/usr/bin/rsh -n -l fbsuser triviaal rcp -r /stage/triviaal/sam_cache/boo node-2:/data/test >> log

batch.jdf

batch_sam

SAM DB

datastore

1.2 TB

FNALSARA

control (fbs)

metadata

100 cpu’s

1. sam2. rcp + analyze + rcp3. rcp + sam

fbs(1), fbs(3)

fbs(2)

triviaal node-2

fbsuser:rcpanalysisprogram

willem:sam

output

fbsuser:fbs submit

SECTION sam EXEC=/d0gstar/batch_node NUMPROC=1 QUEUE=FastQ STDOUT=/d0gstar/stdout STDERR=/d0gstar/stdout

#!/bin/shuname -adate

rsh -l fbsuser triviaal fbs submit ~willem/batch_node.jdf

#!/bin/sh. /usr/products/etc/setups.shsetup fbsngsetup samSAM_STATION=triviaalexport SAM_STATIONsam run project get_file.py --interactive > log/usr/bin/rsh -n -l fbsuser triviaal fbs submit /home/willem/batch_node.jdf

SECTION sam EXEC=/home/willem/batch NUMPROC=1 QUEUE=IOQ STDOUT=/home/willem/stdout STDERR=/home/willem/stdout

SECTION ana EXEC=/d0gstar/batch_node NUMPROC=1 QUEUE=FastQ STDOUT=/d0gstar/stdout STDERR=/d0gstar/stdout

#!/bin/shrcp -pr server:/stage/triviaal/sam_cache/boo /data/test. /d0/fnal/ups/etc/setups.shsetup root -q KCC_4_0:exception:opt:threadsetup kailibroot -b -q /d0gstar/test.C

{gSystem->cd("/data/test/boo");gSystem->Exec("pwd");gSystem->Exec("ls -l");}

## This file sets up and runs a SAM project.#import os, sys, string, time, signalfrom re import *from globals import *import run_projectfrom commands import *########################################### Set the following variables to appropriate values

# Consult database for valid choicessam_station = "triviaal"

# Consult Database for valid choicesproject_definition = "op_moriond_p1014"

# A particular snapshot version, last or newsnapshot_version = 'new'

# Consult database for valid choicesappname = "test"version = "1"group = "test"

# The maximum number of files to get from sammax_file_amt = 5

# for additional debug info use "--verbose"#verbosity = "--verbose"verbosity = ""

# Give up on all exceptionsgive_up = 1

def file_ready(filename): # Replace this python subroutine with whatever # you want to do # to process the file that was retrieved. # This function will only be called in the event of # a successful delivery. print "File ",filename," has been delivered!"# os.system('cp '+filename+' /stage/triviaal/sam') return

get_file.py

Disk partitioning hoeve

/d0dist /d0usr

/mcc-dist /mc_runjob /curr/ups

/db /etc /prd

/fnal -> /d0/fnal/d0usr -> /fnal/d0usr/d0dist -> /fnal/d0dist/usr/products -> /fnal/ups

/fbsng

ana_runjob

• Is analogous to mc_runjob

• Creates and submits analysis jobs

• Input– get_file.py with SAM project name

• Project defines files to be processed

– analysis script

Integration with grid (1)

• At present separate clusters:– D0, LHCb, Alice, DAS cluster

• hoeve and schuur in farm network

Present network layout

hoeve schuur

switch

node node node

router

hefnet

surfnet

New network layout

farmrouter

switch switch switch

D0LHCb

hefnet

lambda

hoeve schuur

booder

New network layout

farmrouter

switch switch switch

D0LHCb

hefnet

lambda

hoeve schuur

booder

Server tasks

• hoeve– software server– farm server

• schuur– fileserver– sam node

• booder– home directory server– in backup scheme

• Replace fbs with pbs or condor– pbs on Alice and LHCb nodes– condor on das cluster

• Use EDG installation tool LCGF– Install d0 software with rpm

• Problem with sam (uses ups/upd)

• Package mcc in rpm

• Separate programs from working space

• Use cfg commands to steer mc_runjob

• Find better place for card files

• Input structure now created on node

Grid job

#!/bin/sh

macro=$1

pwd=`pwd`

cd /opt/fnal/d0/mcc/mcc-dist. mcc_dist_setup.sh

cd $pwddir=/opt/fnal/d0/mcc/mc_runjob/py_scriptpython $dir/Linker.py script=$macro

[willem@tbn09 willem]$ cat test.pbs# PBS batch job script

#PBS -o /home/willem/out#PBS -e /home/willem/err#PBS -l nodes=1

# Changing to directory as requested by user

cd /home/willem

# Executing job as requested by user

./submit minbias.macro

PBS job submit

RunJob class for gridclass RunJob_farm(RunJob_batch) : def __init__(self,name=None) : RunJob_batch.__init__(self,name) self.myType="runjob_farm"

def Run(self) : self.jobname = self.linker.CurrentJob() self.jobnaam = string.splitfields(self.jobname,'/')[-1] comm = 'chmod +x ' + self.jobname commands.getoutput(comm) if self.tdconf['RunOption'] == 'RunInBackground' : RunJob_batch.Run(self) else : bq = self.tdconf['BatchQueue'] dirn = os.path.dirname(self.jobname) print dirn comm = 'cd ' + dirn + '; sh ' + self.jobnaam + ' `pwd` >& stdout' print comm runcommand(comm)

To be decided

• Location of minimum bias files

• Location of MC output

Job status

• Job status is recorded in– fbs– /d0/mcc/curr/<job_name>– /data/mcc/curr/<job_name>

SAM servers

• On master node:– station– fss

• On master and worker nodes:– stager– bbftp

Upgrade D0 farm. Reasons for upgrade RedHat 7 needed for D0 software New versions of –ups/upd v4_6...

Documents