Using Prioritized Network Traffic to Achieve End-to-End Predictability

BBN Technologies OOMWorks LLC Cambridge, MA Metuchen, NJ

Craig Rodrigues Yamuna Krishnamurthy

Irfan Pyarali Pradeep Gore

Real-Time and Embedded Distributed Object Computing

June 15-18, 2002

Arlington, Virginia, USA

Managed Behavior for Computational Resources

• In order to preserve end-to-end Quality of Service (QoS) requirements, different technologies must be used to manage the behavior of computational resources

• Processor Resources– RT-CORBA

• Network Resources– RSVP– DiffServ

Talk Outline

• Real-Time CORBA (RTCORBA) Overview

• RSVP and Diffserv Overviews

• Diffserv enhancements to RTCORBA middleware

• Performance results

Real-time CORBA Overview

• RT-CORBA adds QoS control to regular CORBA to improve the application predictability

– Bounding priority inversions– Managing resources end-to-end

• Policies & mechanisms for resource configuration/control in RT-CORBA include:

Processor Resources– Thread pools– Priority models– Portable prioritiesCommunication Resources– Protocol policies– Explicit bindingMemory Resources– Request buffering

• These capabilities address some (but by no means all) important real-time application development challenges

ClientOBJREF

Object(Servant)

in argsoperation()

out args + return

IDLSTUBS

IDLSKEL

Object Adapter

ORB CORE GIOP

Protocol Properties

End-to-End PriorityPropagation

ThreadPools

StandardSynchronizersExplicit

Binding

Portable Priorities

SchedulingService

Preserving Priorities End-to-End

• RT-CORBA preserves end-to-end priorities by:– Mapping importance of activities to corresponding OS priorities

– Propagating priorities across the network as activity spans multiple hosts

• However , RT-CORBA specification is less explicit about:— Communication transport and underlying network

• Unless this behavior is carefully considered and modeled:— In-transit activities will be affected by network

idiosyncrasies — End-to-end predictability in the system difficult to achieve

What is RSVP?

• Resource Reservation Protocol, specified in IETF RFC 2205

• QoS properties are requested by an “out-of-band” signalling protocol

• Can be used to request a reserved bandwidth between a sender and receiver host

• Information about the reservation is stored in each intermediate router: “soft-state”

Problems with RSVP

• Protocol is very complex to implement– much complexity due to anticipated use-cases of

large scale multicast trees; multicast not widely used• Storing reservation per-flow reservation state is costly

– some work being done into looking reservations for aggregate flows

• Microsoft implemented RSVP in Win2K, disabled it in WinXP– equipment vendors have less incentive to implement

it

Problems with RSVP (continued)

• IETF Next Steps in Signaling (NSIS) Working group– realized that RSVP has it shortcomings, but a

signaling protocol for QoS is useful– developing new signaling mechanism (“RSVP

Lite”?)– specification due end of 2002

• Maybe worth looking at in future, but right nowindustry seems to be moving away RSVP

Solution: Differentiated Services

• Use Differentiated Services to prioritize the RT-CORBA network traffic

• Differentiated Services (Diffserv) architecture provides different types or levels of service for network traffic

• Diffserv Code Points (DSCP) are added to data packet headers to specify the expected type of service

• Diffserv enabled routers and network elements use DSCP codepoints to differentiate the network traffic

• We are enhancing the existing TAO ORB's real-time implementation by making it Diffserv aware

What is Diffserv?

Diffserv Field(8-bits)

HeaderLength(4-bits)

Total Length(16-bits)

Source Address (32-bits)

Destination Address (32-bits)

TTL (8-bits) Protocol (8-bits) Checksum (16-bits)

Flags(3-bits)

Version(4-bits)

Identification (16-bits) Fragment offset (13-bits)

• Diffserv field: 6 bits of Diffserv Codepoint, 2 bits ECN• Each DSCP (0-63) specifies a Per-Hop-Behavior (PHB), which is a type of router-level QoS (RFC 2475)

IP Datagram Header

Diffserv QoS Enhancements to the TAO ORB

• Provided an efficient and flexible way of setting the DSCP codepoints in the ORB data packets– Done by extending the RTCORBA protocol properties

– Setting the DSCP codepoint in the message sent and the reply received

• Provided a mechanism to map the RT-CORBA priorities to the Diff-Serv network priorities

Middle-tierServer

Client QNXpriority

= 16

LynxOSpriority = 128

Server Solarispriority = 136

ServiceContextpriority = 100

ServiceContextpriority = 100

Router

DiffservCodePoint

= EF

DiffservCodePoint

= EF

Router

RTCORBA IDL Modification

local interface TCPProtocolProperties : ProtocolProperties

{ attribute long send_buffer_size; attribute long recv_buffer_size; attribute boolean keep_alive; attribute boolean dont_route; attribute boolean no_delay; attribute boolean enable_network_priority; };

Initialize Protocol Properties

//Set the tcp protocol properties

RTCORBA::TCPProtocolProperties_var tcp_properties = rt_orb->create_tcp_protocol_properties

(send_buffer_size, recv_buffer_size, 1, // keep_alive 0, // dont_route 1, // no_delay enable_network_priority);

Client Side – Initializing the Policy List

RTCORBA::ProtocolList protocols; protocols.length (1); protocols[0].protocol_type = 0; protocols[0].transport_protocol_properties = RTCORBA::ProtocolProperties::_duplicate (tcp_properties.in ()); protocols[0].orb_protocol_properties = RTCORBA::ProtocolProperties::_nil ();

poa_policy_list[1] = rt_orb->create_client_protocol_policy (protocols );

Client Side - Change Transport Protocol Policy

• ORB Level

object = orb->resolve_initial_references ("ORBPolicyManager");

CORBA::PolicyManager_var policy_manager =

CORBA::PolicyManager::_narrow (object.in () );

policy_manager->set_policy_overrides (policy_list, CORBA::SET_OVERRIDE );

Client Side - Change Transport Protocol Policy

• Thread Levelorb->orb_core () >policy_current().set_policy_overrides

(policy_list, CORBA::SET_OVERRIDE);

• Object LevelCORBA::Object_var object =

server->_set_policy_overrides (policy_list, CORBA::SET_OVERRIDE); server = Test::_narrow (object.in () );

Server Side - Initializing the Policy List

RTCORBA::ProtocolList protocols;

protocols.length (1);

protocols[0].protocol_type = 0;

protocols[0].transport_protocol_properties =

RTCORBA::ProtocolProperties::_duplicate (tcp_properties.in ());

protocols[0].orb_protocol_properties =

RTCORBA::ProtocolProperties::_nil ();

poa_policy_list[1] =

rt_orb->create_server_protocol_policy (protocols );

Server Side – Set Policy on the POA

//Create POA with Diffserv enabled

PortableServer::POA_var poa_with_ds =

root_poa->create_POA ("POA_WITH_DS",

poa_manager.in (),

poa_policy_list);

RTCORBA to Network Priority Mapping

• Added a pluggable RTCORBA to Network Priority Mapping framework

• Similar to the RTCORBA to Thread Priority Mapping framework

• RTCORBA IDL modification:

typedef short NetworkPriority;

native NetworkPriorityMapping;• The framework allows the user to implement their

own mapping algorithms and plug it in using the ACE Service Configurator

RTCORBA/Diffserv Testbed

Client Server

TG

Router

192.168.1.* 192.168.10.*

Diffserv DiffservT1

T2

Linux Linux

Linux

Linux

Priority of T1 = Priority of T2

IIOP IIOP

Congestion

Traffic

Performance Results

1. No network traffic congestion

2. With network traffic congestion

3. With network traffic congestion and DSCP set on Thread 1

0

20000

40000

60000

80000

100000

1 2 3

La

ten

cy

(m

icro

se

cs

)

Thread 2

Thread 1

Performance Results

With congestion traffic and DSCP set on T1

Potential Use Case for RT-CORBA/Diffserv

“Predator” Squadron Operations CenterADSI with JCPI & PowerScene Equipped

Air Operations CenterAir Defense System Integrator

With JCPI STU III Back-Up

Nellis – ISAFAF Fiber Optical LinkPrimary

TADIL & Intel Feeds

UAV Precise Position Location Information

PPLIPrioritized Data Feeds

GCS #1 GCS #2 GCS #3 GCS #4

JTIDS Network

ISR Constellation

UAV PPLI

Source: UAV Battlelab, Eglin Air Force BaseU.S. Air Force

Summary

• End-to-end QoS requires resource management of CPU and network resources

• RT-CORBA today primarily manages CPU resources

• Our enhancement allows RT-CORBA middleware to manage network resourcesusing Diffserv, which is priority based

Obtaining Software

• http://deuce.doc.wustl.edu/Download.html

• Contacts:– Craig Rodrigues, crodrigu@bbn.com,

1-617-873-4725– Yamuna Krishnamurthy,

yamuna@oomworks.com, 1-732-205-0346

Further References

• “Protocol Selection and Explicit Binding”,Schmidt & Vinoski http://www.cuj.com/experts/2005/vinoski.htm

• “An Architectural for Differentiated Services”, RFC 2475 http://www.ietf.org/rfc/rfc2475.txt