Lambda Station: On-demand flow based routing for data intensive GRID applications over multitopology networks

Fermi National Accelerator Laboratory Don Petravick (PI), Phil Demar, Matt Crawford, Andrey Bobyshev, Maxim Grigoriev

California Institute of Technology. Harvey Newman (co-PI)

ESCC/Internet2 Joint Techs Workshop

Fermilab, July 15-19, 2007

Outline of the talk

● goals and major directions of the project

● software architecture, API, middleware

● results of using Lambda Station service

● problems and challenges, plans

Basic terms

● Lambda Station – a host with special software to control traffic path across LAN and WAN on-demand of applications

● PBR – policy based routing

● PBR Client – a system or cluster and applications running on it sourcing traffic flows that can be subject for policy based routing

● Flow - a stream of packets with some attributes in common such as endpoint IP addresses (or range of addresses), protocols, protocol's ports if applicable and differentiated services code point (DSCP).

The main goal of Lambda Station project is to design, develop and deploy a network path selection services to interface production storage and computing facilities with advanced research networks.

– selective forwarding on a per flow basis– alternate network paths for high impact data movement – access control in site edge routers for those selected flows– on-demand from applications (authentication & authorization)– current implementation based on policy-based routing & including the

support of DSCP marking

The goal of the project... deals with the last-mile problem in local area networks

Flows and DSCP tagging

Any combination of flow's attributes can be used by Lambda Station software to identify flows on per ticket's basis.

Typical steps of alternative path reservation:

● generate request for service (application, application's proxy, admin)● LS negotiates service and parameters with remote site● configure local and wide area network (not yet available)● marking traffic (if specified).

Current LS software is capable to complete all these steps within 3 – 5 mins. That is why it is desirable to know flow selection parameters before transferring is started:● endpoint IP addresses ● DSCP

Lambda Station Building Blocks

LS request interfaceLS-Management &Reporting Interface

mySQL:requests, Authorization

LS-to-LS service

LS Controller

SiSi

local definitions

online updates

SiSi

SOAP/JClarens

NETWORK CONFIGURATOR

CISCO Force10 WAN

Storage & application spaceManagement

SOAP/JClarens

LSInterface

RemoteLambdaStation

Data Exchange

Control & Management

Service-based Architecture:●LS Controller – manages LS persistence, controls other services●LS persistence is stores current state of the system●LS request Iinterface - webservice for placing all kinds of “ticket” related requests ●LS-to-LS – webservice for LS definitions propagation and LS discovery● NETWORK CONFIGURATOR – dynamic reconfiguring of LAN and WAN

SOAP over HTTPS

LS persistence

LS-2-LS service

Vendor specific modules

Tomcat + Axis container

JClarens

LS Service RequestInterface

LS PBR clientsInterface

LS Controller

Client API

LS NetConfig

LS/PBR Config Manager

LS JAVA API Components

Java implementation of LS Software

● Service Oriented Architecture● utilized JClarens and Axis framework as a web-services toolkit● messages are defined and strongly validated by XML schema●LS service is multi-threaded, one thread for LSController, one thread for LS2LS service and threads pool for openSvcTicket requests● LS2LS and client-2-LS authentication is based on gLite library and supports standard Grid proxies and KCA-issued certificates●Authorization is based on rules set●General framework persistence is accomplished by MySQL DB backend● secure document/literal wrapped SOAP messages, Web Services Interoperability Profile (WS-I Basic Profile Version 1.1)

Java implementation of LS Software(continued)

● Automated LS and PBR client configuration management●Automated deployment ( one can install on any Linux box)●LS Controller, LS2LS , LS AAA, LS client interface are ready for deployment. Supported java and perl clients.●Some interest from ANL to support C client for Globus toolkit●Network Config calls implemented in interface and may relay requests to perl service ( SOA at work ) ●Currently deployed and work ( exchanging PBR and LS configurations) at FNAL ( 2 stations) and CalTech ( 1 station)

DSCP Tagging

Complexity of using DSCP tagging:

● preservation of DSCP is not guaranteed in WAN

● DSCP tagging needs to be synchronize between sites for dynamically configurable networks ( asymmetry is bad for high-performance transferring ) LS software does support two different modes of DSCP tagging :

● fixed DSCP values to identify site's traffic.

● DSCP value is assigned dynamically on per ticket base.

Netconfig Module● dynamically modifies the configurations of local network devices. ● a vendor dependent components. ● Cisco routers with IOS version supporting sequencing type of named ACLs.

Tasks to configure PBR in Cisco devices:● interface on which PBR is applied needs to be configured with “ip policy route-map” statement● route map needs to be configured as ordered list of match/action statements● match criteria need to be associated with ACLs

openSvcTicket request

● new ticket - creating a new ticket● join - provides ID and required parameters(DSCP) of already opened ticket● extend ticket – allows to extend already created ticket.

Accepts many input parameters, most of them could also be determined automatically:● Remote LambdaStation site's identifiers● Source/Destination PBRClient identifier or/and list of IP netblocks●IP protocols, protocol's ports or port range● localPath, remotePath identifiers ( subject to available resources)● Bandwidth out, Bandwidth in● boardTime, startTime, endTime, travelTime

Returns ID of locally assigned ticket

Operational modes (subject to authentication, authorization):

1. Data transfer started:– 10GE host; 5 tcp streams– Network path is via ESnet

– OC12 bottleneck…

– Path MTU is 1500B– LambdaStation service ticket is

opened

2. LambdaStation changes network path to USN

3. Host path MTUD check detects a larger path MTU

4. LambdaStation service ticket expires:– Network path changed back to

ESnet

LSiperf End-to-End Test

1

2 3

4

Lambda Station TestBed

SRM/DCache 1.7 LS-awareness

Wide Area Network

normal traffic flow

High Impact traffic

USCMS Tier1

Advanced Networks

CMS core router

FNAL LambdaStation

Production USCMS SRM server sends requests to Lambda Station to stir a high-impact traffic into Advanced Network infrastructure

Site Network

StarLight

How to demonstrate increasing transfer rate ?? In tests we could generate a data stream and monitor its rates when switching traffic between alternative paths. Production traffic is not deterministic.

CMS SRM

CalTech

CalTech LambdaStation

SRM

Project accomplishments

● Software version 1.0 (a fully functional prototype supporting whole cycle of LS functionality)● positive results of testing between Fermilab and Caltech●lsiperf, lsTraceroute – wrappers around well known applications to add Lambda Station awareness ( based on prototype version 1.0)● SRM/dCache integration – testing, added in production 1.7.0 release● run LS-aware SRM/dCache on production cluster at Fermilab●Interoperable Java implementation of the major components of LambdaStation ( perl, java clients available)

Problems and challenges● Traffic Asymmetry is bad for high performance applications● Network (Lambda Station) awareness is too complex● Definition of PBR Clients is a complex issue, auto definition is not yet available

References: http://www.lambdastation.org/

Plans● release fully functional Java LS● SRM/dCache with production quality LS support● add real-time monitoring of resources ( perfSONAR )● add WAN control plane module● integration with OSCARS ( unified Network Path Reservation Model ?)

END

Miscellaneous Slides

● Simulation of multiple Lss● screen shoots of ticket's queue● SC05 Demo

Netconfig Module● dynamically modifies the configurations of local network devices. ● a vendor dependent components. ● Cisco routers with IOS version supporting sequencing type of named ACLs.

Tasks to configure PBR in Cisco devices:● interface on which PBR is applied needs to be configured with “ip policy route-map” statement● route map needs to be configured as ordered list of match/action statements● match criteria need to be associated with ACLs

openSvcTicket request

● new ticket - creating a new ticket● join - provides ID and required parameters(DSCP) of already opened ticket● extend ticket – allows to extend already created ticket.

Accepts many input parameters, most of them could also be determined automatically:● Dst site's identifiers● Src/Dst PBRClient identifier or/and list of Src/Dst IP in (CIDR)● IP protocols, protocol's ports● localPath, remotePath identifiers● BWout,BWin● boardTime, startTime, endTime

Returns ID of locally assigned ticket

Operational modes (subject authentication, authorization and quoting):

Directions of the project.

● building a wide-are testbed infrastructure● designing, developing Lambda Station software, Lambda Station (network) aware applications● researching effects of flow based switching on applications

Application's behavior in flow based switching environment

Tuning of end systems for maximum rates is not a subject of the Lambda Station project, however, we need to see an increase of data movement performance when selectively switch flows

● DSCP tagging with IPtables ● switching between two paths with different MTUs

Effect of DSCP tagging with IPTables

Motivation of the project

● unprecedented demands for data movement in physics experiments such as CDF, D0, BarBar and coming LHC experiments ● massive, globally distributed datasets growing to the 100 petabytes by 2010● collaborative data analysis by global communities of thousands of scientists ● available data communication technology will not be able to satisfy these demands simply by plain increasing bandwidth in LANs and WANs due to technology limitations and high deployment and operation costs.● Advance Research optical networks – greater capacity, no production quality of service, are not universally available for all pairs of communicating endpoints

LS multitopology network model

PBR-client B

Multiple Network Toplogies

Red

Green

BlueAdmission Group of network devices

RT1 RT2 RT3

RT1 RT2 RT3

PBR-client A

H1 H2 H3

NG-A

NG-C

NG-B

NG-ADM

H1 H2 H3

PBR-client

PBR-clients or regular clientsat the remote sites

ClientA rules for Red& GREEN topologies:

RT1 RT2 RT3

BLUE is ProductionPath

RED-ClientA-INGREEN-ClientB-INRED-ClientB-INGREEN-ClientB-IN

RED-B-IN

GREEN-OUT RED-OUT

Client A, RED & GREEN rules for NGC

Cisco IOS, dynamic configuring of PBR, extended sequencing ACLs + access policy ACLs

lambdastation@FNALlambdastation@SC05

Fermilab SC05/Seattle

nws-lab.fnal.gov

charley.fnal.gov

NAA-2-NAAprotocol

netconfig

LS-2-LSprotocol

A122.302.sc05.org

CommodityInternet/SCinet

A126.302.sc05.orglsiperf

srmcp

ls-request

reply

default path

alternative path forspecific flows only

SC05/HighSpeedLinks

LambdaStation SC05 Demo

Note A

Note A: We believe it is a HW/ASIC problem with SNMP monitoring, a time to time SNMP -get returns the same counters as in previous cycle.

PMTUD