5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

e-VLBI Developments at JIVE

Arpad Szomoru

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Outline

• Current status, recent results

• Network upgrades

• EXPReS: first results

• Connectivity improvements

• The future..

EVN Symposium 2004, A. Szomoru, JIVE

155 Mbps10 Gbps

2.5 Gbps

1 Gbps

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Recent developments

• Regular science/test sessions throughout the year

• First open calls for e-VLBI science proposals

• First science run completely lost, but, first ever real-time fringes to Mc (128 Mbps)

• Second and third science runs: many hours of smooth sailing at 128 Mbps

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Current status

• Technical tests:

• 6-station fringes at 256 Mbps

• first European 512 Mbps fringes (Jb and Wb, May 18)

• 3-station 512 Mbps fringes (Cm, Wb, On, August 21)

• first fringes using new 5 GHz receiver at Mc

• Current connectivity:• Ar: 64 Mbps in the past, but <32 Mbps this year

• European telescopes: 128 Mbps always, 256 Mbps often, 512 Mbps to Wb, Jb and On

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Tr connectivity bottleneck - solved

Black Diamond 6808 switches:

New interfaces (10GE) system in old architecture (1GE)• Originally 8x1GE interface per card• 10GE NIC served by 8 x 1GE queues• Queuing regime – RR (packet based) and flow-based

Flow based:Max. flow capacity – 1Gbit/s – backround traffic.

There is no known reordering workaround to solve this problem.

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

e-VLBI to South America? SMART-1SMART-1 factsheet

 Testing solar-electric propulsion and other deep-space technologies Name SMART stands for Small Missions for Advanced Research in Technology.   Description SMART-1 is the first of ESA’s Small Missions for Advanced Research in Technology. It travelled to the Moon using solar-electric propulsion and carrying a battery of miniaturised instruments. As well as testing new technology, SMART-1 is making the first comprehensive inventory of key chemical elements in the lunar surface. It is also investigating the theory that the Moon was formed following the violent collision of a smaller planet with Earth, four and a half thousand million years ago.  Launched 27 September 2003   Status Arrived in lunar orbit, 15 November 2004. Conducting lunar orbit science operations.   Notes SMART-1 is the first European spacecraft to travel to and orbit around the Moon. This is only the second time that ion propulsion has been used as a mission's primary propulsion system (the first was NASA's Deep Space 1 probe launched in October 1998). SMART-1 is looking for water (in the form of ice) on the Moon. To save precious xenon fuel, SMART-1 uses 'celestial mechanics', that is, techniques such as making use of 'lunar resonances' and fly-bys.

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

e-VLBI to Sourth America (2)

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Australia:

• Telescopes connected

• PCEVN-Mk5 interface needed

China:

• Shanghai Observatory connected at 2.5Gbps

• Connection via TEIN (622Mbps), ORIENT? Issues with CERNET, CSTNet

• Direct lightpath Hong Kong-Netherlight?

And other continents..

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Hybrid networks in the Netherlands..

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Switch from Cisco to Nortel/Avici equipment has been completed: for now, 7 * 1 Gbps, ultimately 16 * 1 Gbps.

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Outline..and across Europe: GÉANT2 network upgrade

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

EXPReS: now what?SA1: new hires at JIVE (two software engineers, one network

engineer, one e-VLBI postdoc)

Inclusion of e-MERLIN telescopes in e-EVN

Operational improvements (deliverable driven):

• Robustness

• Reliability

• Speed

• Ease of operation

• Station feedback

And still, pushing data rates, protocols, UDP, Circuit TCP? Get rid of fairness… Better usage of available bandwidth.

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

e-VLBI control interface

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Interface to station Mk5s

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Runtime control

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Integrating fringe display

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Data status monitor

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Streamlining of post processing

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Web-based Post-processing

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

OngoingNew control computers (Solaris AMD servers)

• Cut down on (re-)start time

• Powerful code development platform

• Tightening up of existing code

• Operational within next few weeks

Other hardware upgrades:

• SX optics (fibres + NICs), managed switches at JIVE and SARA

• Mark5A→B: motherboards, memory, power supplies, serial links, CIBs

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

FABRIC:

• Distributed software correlation

• High bandwidth data transport (On part of e-MERLIN @ 4Gbps)

• Two new hires at JIVE

And coming..SCARIe:

• Collaboration with SARA and UvA

• Distributed software correlation using Dutch grid

• Lambda switching, dynamical allocation of lightpaths

• JIVE postdoc hired, still looking for UvA postdoc

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

FABRIC=

The GRID

FABRIC componentsobserving schedule

in VEX format

user correlatorparametersGRID

resources data

correlator controlincluding model

calculation

field systemcontrols antennaand acquisition

DBBCVSI

VSIe??on??

outputdata

earth orientationparameters

PC-EVN#2

resource allocationand routing

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Martin Swany

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

• 2 Heavy duty gamer PCs• Tyan Thunder K8WE Motherboards• Dual AMD Opteron 2.4GHz processors• 4GB RAM• 2 1Gb PCI-Expres Nics

• First one at Torun, back-to-back to Mark5• Second one located at Poznan Supercomputing Centre

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Protocols (1) Mix of High Speed and Westwood TCP (Sansa)

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Protocols (2)

Circuit TCP (Mudambi, Zheng and Veeraraghavan)

Meant for Dedicated End-to-End Circuits, fixed congestion window

No slow start, no backoff: finally, a TCP rude enough for e-VLBI?

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

Protocols (3)

Home-grown version of CTCP using pluggable TCP congestion avoidance algorithms in newer Linux kernels (Mark Kettenis)

Rock-steady 560 Mbps transfer using iperf from Mc to JIVE

Serious problem with new version of Mk5A software under newer kernels

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

e-EVN: the future• Aim: 16 * 1 Gbps production e-EVN network • IP: not possible/affordable • 10 Gbps lightpath across Europe: currently ~20k€/year

• Lightpaths across GÉANT terminating at JIVE• If possible, all the way from telescopes. If not,

overprovisioned IP connections from telescopes to GÉANT, lightpaths from there on.

• Guaranteed bandwidth, possibility to use ethernet frames, no more need to worry about congestion..

• A true connected-element interferometer

5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE

External LPs

OMEnetwork

Switch Mk5

Mk5

Mk5

10 G IP

OMEnetwork

Switch

Dynamic capabilities through DRAC

N GE

N GE

GEs, STM-64 direct

Proposed connection Surfnet-JIVE