Date post: | 21-Dec-2015 |
Category: |
Documents |
View: | 214 times |
Download: | 0 times |
1
Quality of Service vs.Any Service at All
IWQoS 2005 Passau Germany
Randy H. KatzComputer Science Division
Electrical Engineering and Computer Science DepartmentUniversity of California, Berkeley
Berkeley, CA 94720-1776
2
Presentation Outline
• The Problem• System and Network Trends• Checking-Observing-Protecting Services• Inspection-and-Action Boxes• Annotation Layer• Scenario• Call to Action
3
Presentation Outline
• The Problem• System and Network Trends• Checking-Observing-Protecting Services• Inspection-and-Action Boxes• Annotation Layer• Scenario• Call to Action
4
Some Observations
• Internet reasonably robust to point problems like link and router failures (“fail stop”)
• Successfully operates under a wide range of loading conditions and over diverse technologies
• During 9/11/01, Internet worked reasonable well, under heavy traffic conditions and with some major facilities failures in Lower Manhattan
5
The Problem
• Networks awash in illegitimate traffic: port scans, propagating worms, p2p file swapping
– Legitimate traffic starved for bandwidth– Essential network services (e.g., DNS, NFS)
compromised
• Needed: better network management of services/applications to achieve good performance and resilience even in the face of network stress
– Self-aware network environment– Observing and responding to traffic changes– While sustaining the ability to control the network
6
From the Frontlines
• Berkeley Campus Network– Unanticipated traffic surges render the network
unmanageable (and may cause routers to fail)– Denial of service attacks, latest worm, or the newest file
sharing protocol largely indistinguishable– In-band control channel is starved, making it difficult to
manage and recover the network
• Berkeley EECS Department Network (12/04)– Suspected denial-of-service attack against DNS– Poorly implemented/configured spam appliance adds to
DNS overload– Traffic surges render it impossible to access Web or
mount file systems
• Network problems contribute to brittleness of distributed systems
7
Why and How Networks Fail
• Complex phenomenology of failure• Recent Berkeley experience suggests that traffic
surges render enterprise networks unusable• Indirect effects of DoS traffic on network
infrastructure: role of unexpected traffic patterns– Cisco Express Forwarding: random IP addresses flood route
cache forcing all traffic to go through router slow path—high CPU utilization yields inability to manage router table updates
– Route Summarization: powerful misconfigured peer overwhelms weaker peer with too many router table entries
– SNMP DoS attack: overwhelm SNMP ports on routers– DNS attack: response-response loops in DNS queries generate
traffic overload
8
Possible Approach• New technology: packet flow manipulations at
L4-L7 made possible by new PNEs and stateful routers
• Enables identification/segregation of traffic– Good: protect it– Bad: block it– Suspicious: slow it
• Check/Observe/Protect Services (COPS)– Inspection-and-Action Boxes (iBoxes)– Annotation layer between routing and transport
• Yielding new service building blocks– Beyond packet marking and annotation …– To flow extraction and path-oriented statistics collection …– Based on traffic analysis, model extraction, statistical
correlation & causality testing
9
Presentation Outline
• The Problem• System and Network Trends• Checking-Observing-Protecting Services• Inspection-and-Action Boxes• Annotation Layer• Scenario• Call to Action
10
Edge Network
WideArea
Network
ServerServer
ServerServer
Managing Edge Network Services and Applications
• Not shrink wrap software—but cascaded “appliances”• Data Center in-a-box blade servers, network storage• Brittle to traffic surges and shifts, yielding network
disruption
FirewallIDSTrafficShaper
EgressChecker
LoadBalancer
Blades
Edge Network Middleboxes
11
Appliances Proliferate:Management Nightmare!
F5 Networks BIG-IP LoadBalancerWeb server load balancer
Packeteer PacketShaperTraffic monitor and shaper
Ingrian i225SSL offload appliance
Network Appliance NetCacheLocalized content delivery platform
Nortel Alteon Switched FirewallCheckPoint firewall and L7 switch
Cisco IDS 4250-XLIntrusion detection system
Cisco SN 5420IP-SAN storage gateway
Extreme Networks SummitPx1L2-L7 application switch
NetScreen 500Firewall and VPN
12
Network Support for Tiered Applications
LAN LAN LAN WideArea
Network
ServerAppTier
EgressChecker
ServerServerWebTier
ServerServer
ServerDatabaseTier
Firewall
LoadBalancer
Datacenter Network(s)
Unified LAN
WideArea
Network
ServerServer
ServerServerson
Demand
LoadBalancer
+Firewall
+Egress
CheckerBlades
ServerServer
ServerServerson
DemandConfigure servers, storage, connectivitynet functionality as needed
13
“The Computer is the Network”
• Emergence of Programmable Network Elements– Network components where net services/applications execute– Virtualization (hosts, storage, nets) and flow filtering (blocking,
delaying)
• Computation-in-the-Network is NOT Unlimited– Packet handling complexity limited by latency/processing
overhead– NOT arbitrary per packet programming (aka active networking)– Allocate general computation like proxies to network blades
• Beyond Per Packet Processing: Network Flows– Managing/configuring network for performance and resilience– Emergence of stateful routers for flow-based management– Adaptation based on Observe (Monitor), Analyze (Detect,
Diagnose), Act (Redirect, Reallocate, Balance, Throttle)
14
Presentation Outline
• The Problem• System and Network Trends• Checking-Observing-Protecting Services• Inspection-and-Action Boxes• Annotation Layer• Scenario• Call to Action
15
Check
• Checkable Protocols: Maintain invariants and techniques for checking and enforcing protocols
– Listen & Whisper: well-formed BGP behavior– Traffic Rate Control: Self-Verifiable Core Stateless Fair
Queuing (SV-CSFQ)
• Existing work requires changes to protocol end points or routers on the path
– Difficult to retrofit checkability to existing protocols without embedded processing in PNEs
– Develop building blocks for new protocols » Observable protocol behavior» Cryptographic techniques» Statistical methods
16
Observe
• Observation and Action Points– Network points where control is exercised,
traffic classified, resources allocated– In the datapath statistical collection +
annotating, prioritizing, shaping, blocking, …– Inspection-and-Action Boxes (iBoxes)
» Prototyped on commercial PNEs» Placed at Internet and Server edges of enterprise
net» Cascaded with existing routers to extend their
functionality» Migration into (some current and) future router
architectures
17
Protect
• Protect Crucial Services– Minimize and mitigate effects of attacks and traffic surges– Classify traffic into good, bad, and ugly (suspicious)
» Good: standing patterns and operator-tunable policies» Bad: evolves faster, harder to characterize» Ugly: cannot immediately be determined as good or bad
– Filter the bad, slow the suspicious, maintain resources for the good (e.g., control traffic)
» Sufficient to reduce false positives» Some suspicious-looking good traffic may be slowed
down, but won’t be blocked
18
Presentation Outline
• The Problem• System and Network Trends• Checking-Observing-Protecting Services• Inspection-and-Action Boxes• Annotation Layer• Scenario• Call to Action
19
iBoxes: Observe, Analyze, Act
AccessTier
AccessTier
AccessTier
R R
R R
I nternet orWAN Edge
RDistribution
Tier
I
EE
EE
EE
EE
EE
EE
UserEnd Hosts
ServerEnd Hosts
Network ServicesEnd Hosts
I
I
I R I
R
Enterprise Network Architecture
Inspection-and-Action Boxes:Deep multiprotocol packet inspectionNo routing; observation & markingPolicing points: drop, fence, block
20
Generic Network Element Architecture
InterconnectionFabric
Inp
ut
Port
s
Outp
ut
Port
s
Buffers
Buffers
Buffers
“Tag”Mem
CPCPCPAP
ActionProcessor
CPCPCPCP
ClassificationProcessor
Rules &Programs
21
RouterVM
• High-level specification environment for describing packet processing
• Virtualized: abstracted view of underlying hardware resources of target PNEs
– Portable across diverse architectures– Simulate before deployment
• Services, policies, and standard routing functions managed thru composed packet filters
– Generalized packet filter: trigger + action bundle, cascaded, allows new services and policies to be implemented / configured thru GUI
– New services can be implemented without new code through library building blocks
Mel Tsai
22
QoS ModuleQoS ModuleL2 Switching
Engine w/ ARP
L2 SwitchingEngine w/ ARP
IP Router Engine
IP Router Engine
ControlProcessor
ControlProcessor
Packet filter 1
Packet filter 2
Packet filter n
Default filter
EthernetPort
QoS ModuleQoS Module
Backp
laneBackp
lane
L2 SwitchingEngine w/ ARP
L2 SwitchingEngine w/ ARP
IP Router Engine
IP Router Engine
Packet filter 1
Packet filter 2
Packet filter n
Default filter
EthernetPort
ComputeEngine
ComputeEngine
ComputeEngine
Extended Router Architecture• Virtualized components representative of a
“common” router implementation• Structure independent of particular hardware
Virtual line card instantiated for every
port required by application
Virtual backplane shuttles packets between line cards
CPU handles routing protocols & mgmt tasks
Compute engines perform complex, high-latency processing on flows
Blue “standard” components
Yellow components added & configured per-application
Filters are key to flexibility
Mel Tsai
23
GPF “Fill-in” Specification
“Packet filter” as high-level, programmable building-block
for network appliance apps
FILTER 19 SETUP
NAME - SIP -
SMASK - DIP -
DMASK -PROTO -
SRC PORT -DST PORT -
VLAN - ACTION -
exampleany255.255.255.25510.0.0.0255.255.255.0tcp,udpany80defaultdrop
ClassificationParameters
Action
Traditional Filter
RouterVM Generalized Packet Filter (type L7)
24
GPF Action Language
• Basic set of assignments, evaluations, expressions, control-flow ops, “physical” actions on packets/flows– Control-flow: If-then-else, if-not– Evaluation: ==, <=, >=, !=– Packet flow control: Allow,
unallow, drop, skip filter, jump filter
– Packet manipulation: Field rewriting (ip_src == blah, tcp_src = blah), truncation, header additions
– Actions: NAT, loadbalance, ratelimit, (perhaps others)
– Meta actions: packet generation, logging, statistics gathering
• Basic Filter– Simple L2-L4 header classifications– Any RouterVM actions
• L7 Filter– REs, TCP termination, ADU recon
• NAT Filter– Capabilities beyond simple NAT action
available to all GPFs
• Content Caching– Builds on L7 filter functionality
• WAN Link Compression– Simple to specify, but requires lots of
computation
• IP-to-FC Gateway– Requires own table format & processing
• XML Preprocessing– Not very well documented, and
difficulty is unknown…
25
Presentation Outline
• The Problem• System and Network Trends• Checking-Observing-Protecting Services• Inspection-and-Action Boxes• Annotation Layer• Scenario• Call to Action
26
Network-LevelObserve-Analyze-Act
• Observe – Packet, path, protocol, service invocation statistical
collection and sampling: frequencies, latencies, completion rates
– Construct the collection infrastructure
• Analyze– Determine correlations among observations– “Normal” model discovery + anomaly detection– Exploit SLT
• Act – Experiment to test correlations– Prioritize and throttle– Mark and annotate– Control theory? Distributed analyses and actions
27
Network Layer Mechanism: Annotations
• Enhance network visibility: disseminate observations, communicate actions, provide in-band network management actions, iBox-to-iBox communications
• iBoxes label packets at annotation layer but do not rewrite packet contents
• Annotations stack, must be removed from packets before delivery to A-layer unaware end nodes
• Expose annotations to application layer?
Phy
Link
Network
Annotation
Transport
Session
Presentation
Application
28
Annotation Layer:Simple Marking Example
• Marking vs. rewriting approach– E.g., mark packets as internally vs. externally sourced using
IP header options
• Prioritize internal vs. external access to services solves some but not all traffic surge problems
iBoxBoundaryRouter
NetworkServices
iBoxI nternalRouter
Enterprise Network
External Traffi c
I nternal Traffi c
Packet
PacketLabel PacketLabel
PacketLabel
Action: Markpackets
Detect load and trigger action: Slow traffi c with “external” labels
29
Annotation Layer:iBox Piggybacked Control
Plane• Problem: Control plane starvation• Use A-layer for iBox-to-iBox communication
– Passively piggyback on existing flows– “Busy” parts of network have lots of control plane b/w– Actively inject control packets to less active parts– Embedded control info authenticated and sent
redundantly– Priority given to packets w/control when net under stress
• Network monitoring and statistics collection dissemination subsystem
30
Presentation Outline
• The Problem• System and Network Trends• Checking-Observing-Protecting Services• Inspection-and-Action Boxes• Annotation Layer• Scenario• Call to Action
31
R
R
DistributionTier
E
E
E
S
S
II
R IA
E
InternetEdge
AccessEdge
ServerEdge
SpamAppliance
Primary &Secondary
DNSServers
ISS
MailServer
S
Scenario: Traffic Surge Inhibiting Network Services
• DNS Server swamped by excessive request traffic– Observe: DNS time outs, Web access traffic slowed, but also
higher than normal mail delivery latency implying busy server edge (correlation between Mail Server and DNS Server utilization?)
– Root Cause: High DNS request rates generated by Spam Appliance triggered by mail surge
32
Scenario Continued
• How Diagnosed?– I-S detects high link utilization but abnormally high DNS traffic– Stats from I-I: high mail traffic, low outgoing web traffic, in
traffic high but link utilization not high– Stats from I-A: lower web traffic, no unusual mail origination– Problem localized to Server edge, but visibility limited
R
R
DistributionTier
E
E
E
S
S
II
R IA
E
InternetEdge
AccessEdge
ServerEdge
SpamAppliance
Primary &Secondary
DNSServers
ISS
MailServer
S
33
Scenario Continued
• Possible Action Responses– Experiment: Redirect local DNS requests to Secondary DNS
server: if these complete, can infer the server is the problem, not the network
– Throttle: Due to MS-DNS correlation, block/slow email traffic at Server Edge: should expect reduced DNS server utilization
R
R
DistributionTier
E
E
E
S
S
II
R IA
E
InternetEdge
AccessEdge
ServerEdge
SpamAppliance
Primary &Secondary
DNSServers
ISS
MailServer
S
34
Presentation Outline
• Problem and Approach• System and Network Trends• Checking-Observing-Protecting Services• Inspection-and-Action Boxes• Annotation Layer• Scenario• Call to Action
35
EdgeNetwork
System Perspective Needed!
DistributedMiddleware
Client
SLT Services DistributedMiddleware
Server
InternetIP Network
Router Router
EdgeNetwork
iBox iBox
Prototype ApplicationsProgrammingAbstractions
For Roll-back andwide-area distributed
computations
Crash-only services+ Observation
Infrastructure forSystem SLT
Checkable ProtocolsFast Detection &Route Recovery
ObservationInfrastructure for
network SLT
CommodityInternet
OperatorUser
Application-Specific
Overlay Network
36
iBoxes implemented on commercial PNEs
– Don’t: route or implement (full) protocol stacks
– Do: protect routers and shield network services
» Classify packets» Extract flows» Redirect traffic» Log, count, collect stats» Filter/shape traffic
Hope for EmergingPlatforms
37
Summary and Conclusions
• Processing-in-the-Network is real– Networking plus processing in switched and routed
infrastructures– Configuration and management of packet processing
cast onto PNEs (network appliances, blades, stateful routers)
• Needed: Unifying Framework– Methods to specify functionality and processing
» RouterVM: Filtering, Redirecting, Transformation» Map from policy intentions to network actions?» Local observations/correct global behavior?
• Application-specific network processing based on session extraction
38
Summary and Conclusions
• PNEs: foundation of a pervasive infrastructure for observation and action at the network level
– iBoxes Observation and Action points– Annotation Layer for marking and control
• Check-Observe-Protect paradigm for protecting critical resources when network is under stress
• Functionality eventually migrates into future generations of routers
– E.g., Blades embedded in routers
39
Quality of Servicevs.
Any Service at All
Randy H. Katz
Thank You!
40