Lambda Station: On-demand flow based

routing for data intensive GRID applications over

multitopology networks

GridNets 2006, San Jose,CA, October 1 – 2, 2006

Fermi National Accelerator Laboratory Don Petravick (PI), Phil Demar, Matt Crawford

California Institute of Technology. Harvey Newman (co-PI)

by Andrey Bobyshev, Fermilab.

Outline of the talk:

● goals and major directions of the project● software architecture, API, SOAP/XML, some details● results of using Lambda Station service, SC05 Demo ● problems and challenges, plans

Basic terms

● Lambda Station – a host with special software to control traffic path across LAN and WAN on-demand of applications● PBR – policy based routing● PBR Client – a system or cluster and applications running on it sourcing traffic flows that can be subject for policy based routing● Flow - a stream of packets with some attributes in common such as endpoint IP addresses (or range of addresses), protocols, protocol's ports if applicable and differentiated services code point (DSCP).

The main goal of Lambda Station project is to design, develop and deploy a network path selection services to interface production storage and computing facilities with advanced research networks.

– selective forwarding on a per flow basis

– alternate network paths for high impact data movement

– access control in site edge routers for those selected flows

– on-demand from applications (authentication & authorization)

– current implementation based on policy-based routing & including the

support of DSCP marking

The goal of the project... deals with the last-mile problem in local area networks

Lambda Station TestBed

● two HEP sites, Fermilab and

Caltech

● 10G & 1G End-To-End paths

via DOE UltraScienceNet and

NFS UltraLight

● components of test and

production

computing/storage/network

infrastructures

● 10Gb and 1Gb servers

Lambda Station Building Blocks

LSInterfaceLS-Management &Reporting Interface

mySQL:requests, history,security

LSDIScovery Service

LSRESource Scheduler LSController

SiSi

local definitions

online updates

SiSi

SOAP/Clarens

Vendor specificmodules

NETWORK CONFIGURATOR

CISCO Force10 WAN

Storage & application spaceManagement

SOAP/Clarens

LSInterface

Remote lambdastation

Data Exchange

Control & Management

Service-based Architecture:● CLARENS is framework for service based architecture, mutual authentication of requests●LSController – synchronizes work of all services, has control functions ●LSI - unified interface for intercommunication between LS and applications, and LS-to-LS● LSDIScovery service – detects new lambdastations, and PBR clients at remote and local sites● LSRESource Scheduler – estimates bandwidth allocation, monitor real-time usage of resources● NETWORK CONFIGURATOR – dynamic reconfiguring of LAN and WAN

SOAP

Lambda Station Software Multiple API-calls or primitives (~ 20) divided into several groups:● informative (whoiam,sayHello,getClientInfo, getKnownLambdastations....)● service or operational (openSvcTicket,cancelTicket,getTicket,getFlowSpec...)● internal (getNextDSCP, assignDSCP,updateMyLambdaDefs)● single interface for client-to-LS and LS-to-LS communication

Scenarios to configure LANs:● dynamically configurable local networks with dynamically or statically assigned DSCP● without DSCP tagging● statically configured networks with predefined matching criterion for flows

LS Software v1.0 – a fully functional prototype but no interoperability across different platforms

Java implementation of LS Software● Service Oriented Architecture● build by using JClarens framework as a grid-aware tool● messages are defined by XML schemas● core authentication is based on gLite library to support standard Grid proxies and KCA-issued certificates● secure document/literal wrapped SOAP messages, Web Services Interoperability Profile (WS-I Basic Profile Version 1.1)

Multi threaded LS implementation (LS Controller, NetConfig

and LS/PBR Config as JClarens background processors)

Flows and DSCP tagging Any combination of flow's attributes can be used by Lambda Station software to identify flows on per ticket's basis.

Typical steps of alternate paths:● generate request for service (application, application's proxy, admin)● LS negotiates service and parameters with remote site● configure local and wide area network (not yet available)● marking traffic (if specified).

Current LS software is capable to complete all these steps within 3 – 5 mins. That is why it is desirable to know flow selection parameters before transferring is started:● endpoint IP addresses ● DSCP

DSCP Tagging

Complexity of using DSCP tagging:● preservation of DSCP is not guaranteed in WAN● DSCP tagging needs to be synchronize between sites for dynamically configurable networks ( asymmetry is bad for high-performance transferring )

LS software does support two different modes of DSCP tagging :● fixed DSCP values to identify site's traffic. ● DSCP value is assigned dynamically on per ticket base.

Interaction between applications and LSMultiple scenarios how applications and Lambda Station may communicate:● modified applications with LS awareness capability to place and manipulate tickets● w/ and w/o DSCP traffic marking ● unmodified applications, a proxy controls paths for their traffic

Operational modes of openSvcTicket request (subject to authentication, authorization and quotas):

● new ticket - creating a new ticket● join - provides ID and required parameters(DSCP) of already opened ticket● extend ticket – allows to extend already created ticket.

Lambda Station Aware Applications lsiperf, lsTraceRoute:● Full support of Lambda Station awareness● Places a ticket for an alternative path, negotiates and synchronizes with remote server's application the parameters (DSCP tagging with IPTables ) of request● watches ticket's status SRM/dCache:● limited Lambda Station awareness● places ticket, join and extend● recognizes absence of Lambda Station● tries to use it only for specific destinations

1. Data transfer started:– 10GE host; 5 tcp streams

– Network path is via ESnet

– OC12 bottleneck…

– Path MTU is 1500B

– LambdaStation service ticket is

opened

2. LambdaStation changes

network path to USN

3. Host path MTUD check

detects a larger path MTU

4. LambdaStation service ticket

expires:– Network path changed back to

ESnet

LSiperf End-to-End Test

1

2 3

4

Flows based switching demo at SC05

Project accomplishments

● Software version 1 (a fully functional prototype supporting whole cycle of LS functionality)● positive results of testing between Fermilab and Caltech● positive results during SC05 via general Internet and SCinet (10G path)● lsiperf, lsTraceroute – wrappers around well known applications to add Lambda Station awareness● SRM/dCache integration – testing, added in production 1.7.0 release

Problems and challenges● Traffic Asymmetry is bad for high performance applications● Network (Lambda Station) awareness is too complex● Definition of PBR Clients is a complex issue, auto definition is not yet available

References: http://www.lambdastation.org/

Plans● release jLS, interoperability across platforms● SRM/dCache with production quality LS support● add real-time monitoring of resources● add WAN control plane module

END

Miscellaneous Slides

● Simulation of multiple Lss● screen shoots of ticket's queue● SC05 Demo

● Single web services ServiceTicket request interface for client2LS and LS2LS message flows

● Multi threaded LS implementation:

– LS Controller, NetConfig and LS/PBR Config as JClarens background

processors

● Apache XmlBeans API for XML data binding

● JClarens utilized to provide User Session management, authorization,

authentication and transparent DB connections pooling

● Interoperability tests with perl, python clients

LS Java API

Netconfig Module● dynamically modifies the configurations of local network devices. ● a vendor dependent components. ● Cisco routers with IOS version supporting sequencing type of named ACLs.

Tasks to configure PBR in Cisco devices:● interface on which PBR is applied needs to be configured with “ip policy route-map” statement● route map needs to be configured as ordered list of match/action statements● match criteria need to be associated with ACLs

openSvcTicket request

● new ticket - creating a new ticket● join - provides ID and required parameters(DSCP) of already opened ticket● extend ticket – allows to extend already created ticket.

Accepts many input parameters, most of them could also be determined automatically:● Dst site's identifiers● Src/Dst PBRClient identifier or/and list of Src/Dst IP in (CIDR)● IP protocols, protocol's ports● localPath, remotePath identifiers● BWout,BWin● boardTime, startTime, endTime

Returns ID of locally assigned ticket

Operational modes (subject authentication, authorization and quoting):

Directions of the project.

● building a wide-are testbed infrastructure● designing, developing Lambda Station software, Lambda Station (network) aware applications● researching effects of flow based switching on applications

Application's behavior in flow based

switching environment

Tuning of end systems for maximum rates is not a subject of the Lambda Station project, however, we need to see an increase of data movement performance when selectively switch flows

● DSCP tagging with IPtables ● switching between two paths with different MTUs

Effect of DSCP tagging with IPTables

Tomcat + Axis container

JClarens

LS Service Request

Interface

LS PBR clients

Interface

LS Controller

Client API

LS NetConfig

LS/PBR Config Manager

LS JAVA API Components

Motivation of the project

● unprecedented demands for data movement in physics experiments such as CDF, D0, BarBar and coming LHC experiments ● massive, globally distributed datasets growing to the 100 petabytes by 2010● collaborative data analysis by global communities of thousands of scientists ● available data communication technology will not be able to satisfy these demands simply by plain increasing bandwidth in LANs and WANs due to technology limitations and high deployment and operation costs.● Advance Research optical networks – greater capacity, no production quality of service, are not universally available for all pairs of communicating endpoints

LS multitopology network model

PBR-client B

Multiple Network Toplogies

Red

Green

BlueAdmission Group of network devices

RT1 RT2 RT3

RT1 RT2 RT3

PBR-client A

H1 H2 H3

NG-A

NG-C

NG-B

NG-ADM

H1 H2 H3

PBR-client

PBR-clients or regular clientsat the remote sites

ClientA rules for Red& GREEN topologies:

RT1 RT2 RT3

BLUE is ProductionPath

RED-ClientA-INGREEN-ClientB-INRED-ClientB-INGREEN-ClientB-IN

RED-B-IN

GREEN-OUT RED-OUT

Client A, RED & GREEN rules for NGC

Cisco IOS, dynamic configuring of PBR, extended sequencing ACLs + access policy ACLs

lambdastation@FNALlambdastation@SC05

Fermilab SC05/Seattle

nws-lab.fnal.gov

charley.fnal.gov

NAA-2-NAA

protocol

netconfig

LS-2-LS

protocol

A122.302.sc05.org

CommodityInternet/SCinet

A126.302.sc05.org

lsiperf

srmcp

ls-request

reply

default path

alternative path forspecific flows only

SC05/HighSpeedLinks

LambdaStation SC05 Demo

Note A

Note A: We believe it is a HW/ASIC problem with SNMP monitoring, a time to time SNMP -get returns the same counters as in previous cycle.

PMTUD