27/10/98 R. Moore, Trigger Workshop 1

L2 Object I/O Format

• L2 needs to read and writeobjects online and offlineä pre-processors transfer their

outputs to L3 and L2 globalä simulation will need to read

and write L2 objects in anoffline environmentä offline analysis will need to

read L2 trigger objects• Code must be flexible

enough for offline butefficient enough for online

27/10/98 R. Moore, Trigger Workshop 2

Requirements

• Changing the format shouldbe simpleä the format will evolve over

time and so should not befixed in stone

• Must support multipleformats offline (but notnecessarily online)

• Must be portableä needs to run on all offline

platforms• Must read from a block of

alpha byte-ordered memory

27/10/98 R. Moore, Trigger Workshop 3

Solutions

• Hand code routines4very flexible8not easy to change format8probably need two versions of

reader for off and online8read and write code independent =

BUGS!• XDR using rpcgen4easy to change format4machine independent4read and write code automatically

generated8not easy to add different formats

(but can be done)8performance and compatibility

issues with Alpha board

27/10/98 R. Moore, Trigger Workshop 4

Solutions II

• CORBA8too unwieldy for use online!

• Python/Perl script to parseheader file and generate I/Oroutines to compile4fully flexible4read and write code linked4changing format simple4can generate online and offline

version of reading code8possibly more work to write

initially? May pay off in reducedmaintenance work?

27/10/98 R. Moore, Trigger Workshop 5

Python/Perl Script

class Electron { // Version=3.5 L2OBJECT(Stream,Electron); private: float emFrac; // Min=0. Max=1. Size=1 float ET; // Min=0. Max=1000. Size=2 Eta eta; // Version<=2.0 … .}

Stream &operator>>(Stream &str,Electron &ele) { int8 tmp8bit; // temporary 8bit variable str >> ele.version; // read electron version info str >> tmp8bit; ele.emFrac=(float)((tmp8bit*(1.-0.)/256)+0.);// Similarly for ET… ...#ifdef OFFLINE if(ele.version <= 2.0) str >> eta;#endif … .}

Python/Perl

Script

27/10/98 R. Moore, Trigger Workshop 6

ReleaseEnvironment

• Attempt to follow standardDØ environmentä SoftRelToolsä CVSä CTEST

• Statusä currently trying to set up ups

and upd at MSUä L2 code already in CVSä hope to update to

SoftRelTools once ups online(and understood!)ä ready within a month

27/10/98 R. Moore, Trigger Workshop 7

Software Libraries

Utilities

Emulator Hardware

Test Code

L2 Global

L2util

L2GblWorker

L2common

L2hwL2emu

L2GblAdmin

L2utilTest

L2commonTest

{OR}Common Code

27/10/98 R. Moore, Trigger Workshop 8

L2 Code Structure

• Choice of release procedure:ä link releases of data transport

and user code together to give asingle L2 version numberä separate data transport and

crate code so that each has aseparate version number

MBus

VMEAdm

inistr

ator

Worker Crate

Code

Data Transport Code

27/10/98 R. Moore, Trigger Workshop 9

Version Numbering

• Single version number4easy to keep track of exactly what

code was running when8updating ANY crate’s code means

ALL crates have to have a newversion number

• Multiple version numbers4each crate can update its code

independently4only data movement code change

will update all crate code versions8harder to keep track of multiple

versions8compatibility issues now that data

movement and crate code canchange independently

27/10/98 R. Moore, Trigger Workshop 10

Release Grouping

L2util

L2GblWorker

L2common

L2hwL2emu

L2GblAdmin

L2utilTest

L2commonTest

{OR}

Data Transport Version

L2 Global Version

27/10/98 R. Moore, Trigger Workshop 11

Coding Guidelines

• Follow DØ draft standardä obtainable from WWWä explains how to write clear,

consistent C++ code• Use DOC++ comments to

document the codeä only change from DØ draft

standardä every function, variable, typedef

etc. should have a DOC++ entryexplaining it purpose AND itsparameters/return valueä documentation on WWW “DØ

Computing Tools” page

27/10/98 R. Moore, Trigger Workshop 12

//// File: EventMessage.hpp// Purpose: EventMessage class for L2 Trigger code// Created: 24-SEP-1998 by Roger Moore//// Comments:// Header file for EventMessage class. See Doc++ comments for// full documentation.//// Revisions://#ifndef _L2UTIL_EVENTMESSAGE_HPP#define _L2UTIL_EVENTMESSAGE_HPP#include…//================================================// CLASS : EventMessage/** Class which reads and writes an admin event message to a worker. The format of the message as send is: \begin{center} \begin{tabular}{|r|l|} \hline Byte & Function \\ \hline 0 & message ID (Message::EVENT_MSG) \\ 1 & buffer number of event processed \\ 2 & event processed=0/failed=1/passed=2 \\ 8-15 & 128 L2 trigger bits \\ \hline \end{tabular} \end{center} @author Roger Moore (moore@pa.msu.edu) @version 0.1 24-SEP-1998 *///================================================

Example L2 Class

27/10/98 R. Moore, Trigger Workshop 13

HTML Output

27/10/98 R. Moore, Trigger Workshop 14

Additional Rules

• Restrictions on online code*no dynamic memory

allocation*no C++ style exceptions*no RTTI*no Standard Template Library

• Additional guidelines– only use virtual functions

where ABSOULTELY needed– can be pretty free with use of

“inline” since entire programwill sit in cache and branchingslows down Alpha processorsignificantly

27/10/98 R. Moore, Trigger Workshop 15

Prototype Boards

• Two environments¶Alpha Linux·bare system (no OS)

• Three setups to test with– 1, 2 and 3-4 nodes

Serial

Ethernet

Hard drive

Alpha Card (128Mb)

Floppy drive

27/10/98 R. Moore, Trigger Workshop 16

Single Node I

• Setup both softwareenvironments (Linux andDBM)

• Test hardware devicesî VME interfaceîMBus interface

• Test existing low level codeî interrupt handlerî Real Time Clock handler

• Test hardware monitor portî needs new FPGA program

Get the boards to boot!

27/10/98 R. Moore, Trigger Workshop 17

Single Node II

• Run “dumb” Administratorî no MBT, Workers, VBD etcî only respond to Bit3(TCC)

• Run “dumb” Workerî no MBT, Admin, VBD etc.î responds to Bit3(TCC)

• Stabilize “dumb” Workerî perform simple timing tests on

Pre-processor/Global workercode

• Add and test interfaces toVBD and MBT

27/10/98 R. Moore, Trigger Workshop 18

Twin Nodes I

• Test inter-alpha MBuscommunicationîMBus speed tests(?)îmore comprehensive test of

broadcast engine(?)• Test inter-alpha VME

communicationî setup of the VME mapping by a

remote node• Test simple Admin-Worker

interactionsî no MBT, VBDî responds to Bit3(TCC)

27/10/98 R. Moore, Trigger Workshop 19

Twin Nodes II

• Add MBT and VBD to crateî perform full test of single

worker systemî no interrupts on MBus (can we

poll MBT?)î send simulation events through

entire chain and examineoutputî perform preliminary speed tests

27/10/98 R. Moore, Trigger Workshop 20

Multiple Nodes

• Setup crate with 2+ workersî use Bit3 to start withî go to MBT and VBD later

• Test different Administratorsetupsî lockstep OR non-lockstepî pre-processor OR globalîî shadow nodes?shadow nodes?

• Test Calorimeter PPî three workers running in one

crate

27/10/98 R. Moore, Trigger Workshop 21

How far can we gowith Linux?

• Linux useful for debuggingcode “offline”â runs on Alpha boardâ same source code (or even

executable!) behind a UNIXinterfaceâ nice, cosy development

environment• Can we conceive of using

Linux at run-time?â what about performance?â interrupt latency?â preemptive multi-tasking?

27/10/98 R. Moore, Trigger Workshop 22

The Linux Kernel

Virtual Memoryspace for processes

Kernel Memory Space

Physical Memory

Hardware

Interrupt Handler

Timer interrupts usedby kernel to perform“book keeping” tasks

Device interruptspassed to drivers

which request them

27/10/98 R. Moore, Trigger Workshop 23

L2 Run-time Kernel

Hardware

Kernel Memory Space

PhysicalMemory

L2 ExecutableCode

Reserved Physical Memory

Interrupt Handler

L2 InterruptHandlerSwitch turns off all

or some interruptsto the kernel

Communicate withprocess using reserved

physical memory

(maybe PAL code)

27/10/98 R. Moore, Trigger Workshop 24

Run-time Linux

• Reserved Physical Memoryâ used to store event Buffersâ used to communicate between

interrupt handler & executable• Interrupt Handlerâ switch can turn off interrupts

to Linux kernelâ executable disables timer

interrupts to kernel and socannot be preemptedâ updates FIFOs in reserved

memory when new eventarrives

27/10/98 R. Moore, Trigger Workshop 25

Linux Performance

• Linux environment willincur some overheadâ interrupt handler has an extra

‘if’ statement to select L2 orkernelâ executable runs in virtual

memory so it’s very hard toensure no L3 cache overwritesâ initialization may be a little

slower since code has to domore (load executable intovirtual memory, turn offkernel etc.)

27/10/98 R. Moore, Trigger Workshop 26

So what does Linuxgive you?

•Core Dumps!â easy stack unwinding since

executable runs as a Linuxprocessâ able to store multiple core

dumps on local disk so severalcrashes can be examined

• Full UNIX environmentâ run debugger on same

machine as executableâ remote access simple

• Same development and run-time environment