Traffic Instrumentation and Management

CSG, January 2002

Traffic Instrumentation

✦ What are you looking for?✧ How’s the bandwidth being spent?✧ Locate anomalies

✧ Intrusions✧ Outgoing Denial of Service (DoS) attacks

✦ Where should you look?✧ Gateway routers get us most of what we want

Solution: Network Logs

✦ Network logs let you analyze past events✧ Log specific information: source, dest, time,

amount of traffic, etc. ✧ Packet contents are overkill

✧ Privacy issues✧ Disk space

✦ Do you need to log all connections?✧ Doing so allows forensics✧ Our disk space usage: 100GB gives 3 months

Network Forensics

✦ What happened on the network?✦ Three Examples

✧ Who’s launching a DoS?✧ Where’s the network bandwidth gone?✧ A compromised machine

✧ How was it compromised? When? Where?

Outgoing DoS

✦ DoSes are generally spoofed✧ Network logs aren’t too useful✧ Egress filtering helps, but the DoS tools figure

out how much spoofing they can safely do (spoof from the same class C)

✧ Blocked spoof attacks can flood network logs

Graphs for identifying DoS

✦ Graphs are useful as DoS attacks stand out✦ How much can you graph?

✧ Graphing each network port may be impractical✧ Other traffic may interfere

DoS Identification

✦ If you catch the DoS while it’s occurring, you can check the current bandwidth usage on the switches✧ “show top pkts”

✦ If it’s spoofed, and you don’t catch it while it’s happening… now what?

Where’s the Bandwidth Gone?

✦ The “Napster” question✦ Use statistical analysis

✧ Which udp/tcp ports and/or ip addresses are using a lot of bandwidth at times of high bandwidth?

Is it an abuser?

✦ Is one machine using more than their fair share of bandwidth?✧ Look at the top ten bandwidth users✧ Maybe… most of the IPs are of known high bandwidth

services (usenet, ftp, backup)

% flow-stat -f11 < ft-v06.2002-01-06.140000 | sort -nr +2 -3 | head -10# IPaddr flows octets packets128.135.137.92 9543 2477490152 2897200224.2.177.155 160 2337752653 4446397128.135.136.147 1258 1947979335 2123565128.135.108.92 4775 1676599523 2105520128.135.12.170 1391 1510492347 2570530128.135.147.43 6765 1079172157 1396155198.49.215.223 16 868834755 979761128.135.221.135 1610 848575034 866508128.135.112.72 3855 829361150 94089166.27.181.42 43 807246316 876126

Is it a specific program✦ File sharing is high

✧ KaZaA (port 1214) and eDonkey (4662)✦ http is high (no surprise)✦ Port 55524 only has a few flows.

✧ Probably a few large file transfers✧ Flow-extract shows us that it is multicast traffic

% flow-stat -f7 < ft-v06.2002-01-06.140000 | sort -nr +2 -3 | head -10# port flows octets packets1214 232837 16971643884 2070535480 1696461 10397156269 219713074662 14292 2652388190 352664155524 83 2313245819 4410164119 1503 1571612208 25108336346 86042 1067187821 30342931737 787 809319373 8821346348 1695 799340259 14035761156 1592 715081006 75491147087 10 678618965 691006

The Compromise

✦ willard.uchicago.edu compromsed✦ We know the approximate time of the

compromise: the morning of December 18th.

✦ We want to know what else they got into and how they got in.

Logs, Part 1

✦ Look for connections to machine at right time✦ Compromise was via ssh✦ ftp’d to a home.com address✦ Weird connections to port 40911

% flow-extract -d willard.uchicago.edu.ft-v06 -e ' since 2001-12-18 00:00 { print }'

12/18/2001 04:59:42 -> 12/18/2001 04:59:42 6 02 willard.uchicago.edu 22 <-> 101 host230.avlogic.com 4658 2 100 00 -SR-A-12/18/2001 04:59:42 -> 12/18/2001 04:59:42 6 11 willard.uchicago.edu 22 <-> 01 host230.avlogic.com 4658 2 100 00 -SR-A-12/18/2001 04:59:42 -> 12/18/2001 04:59:43 6 11 willard.uchicago.edu 22 <-> 01 host230.avlogic.com 4658 6 543 10 F-RPA-12/18/2001 04:59:42 -> 12/18/2001 04:59:43 6 02 willard.uchicago.edu 22 <-> 101 host230.avlogic.com 4658 6 543 10 F-RPA-12/18/2001 04:59:42 -> 12/18/2001 04:59:42 6 103 host230.avlogic.com 4658 <-> 02 willard.uchicago.edu 22 5 296 00 FS-PA-12/18/2001 04:59:42 -> 12/18/2001 04:59:43 6 103 host230.avlogic.com 4658 <-> 02 willard.uchicago.edu 22 3 120 10 --R---12/18/2001 04:59:42 -> 12/18/2001 04:59:42 6 11 willard.uchicago.edu 22 <-> 01 host230.avlogic.com 4157 1 60 00 -S--A- [clip]12/18/2001 05:03:57 -> 12/18/2001 05:03:58 6 11 willard.uchicago.edu 2170 <-> 01 cc17926-a.wlgrv1.pa.home.com ftp 2 112 00 -S--A-12/18/2001 05:03:58 -> 12/18/2001 05:03:58 6 103 cc17926-a.wlgrv1.pa.home.com ftp <-> 02 willard.uchicago.edu 2170 1 60 00 -S--A-12/18/2001 05:03:57 -> 12/18/2001 05:03:58 6 02 willard.uchicago.edu 2170 <-> 101 cc17926-a.wlgrv1.pa.home.com ftp 2 112 00 -S--A-12/18/2001 05:05:46 -> 12/18/2001 05:05:46 6 02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 1 48 00 -S--A-12/18/2001 05:05:49 -> 12/18/2001 05:06:11 6 02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 48 3107 10 ---PA-12/18/2001 05:05:46 -> 12/18/2001 05:06:12 6 103 55.icafe.euroweb.ro 1705 <-> 02 willard.uchicago.edu 40911 43 2602 00 -S-PA-

More forensics

✦ What’s on port 40911✦ Looks like a back door

% telnet willard.uchicago.edu 40911Trying 128.135.149.73...Connected to willard.uchicago.edu (128.135.149.73).Escape character is '^]'.SSH-1.5-1.2.27

Investigating 40911

✦ Did they connect to other machines on port 40911? (Yes, ultraviolet)

✦ Could also scan the whole network for port 40911

% flow-cat * | flow-extract -e 'port = 40911 { print }'12/18/2001 04:52:01 -> 12/18/2001 04:52:01 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 1 48 00 -S--A-12/18/2001 04:52:01 -> 12/18/2001 04:52:29 6 103 55.icafe.euroweb.ro 1662 <-> 02 ultraviolet.uchicago.edu 40911 36 2294 00 -S-PA-12/18/2001 04:52:01 -> 12/18/2001 04:52:29 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 37 3131 10 ---PA-12/18/2001 05:01:50 -> 12/18/2001 05:02:10 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 39 2512 10 ---PA-12/18/2001 05:01:50 -> 12/18/2001 05:02:11 6 103 55.icafe.euroweb.ro 1662 <-> 02 ultraviolet.uchicago.edu 40911 48 2800 00 ---PA-12/18/2001 05:03:03 -> 12/18/2001 05:03:05 6 103 55.icafe.euroweb.ro 1662 <-> 02 ultraviolet.uchicago.edu 40911 6 340 00 ---PA-12/18/2001 05:03:03 -> 12/18/2001 05:03:04 6 02 ultraviolet.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1662 4 296 10 ---PA-12/18/2001 05:05:46 -> 12/18/2001 05:05:46 6 02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 1 48 00 -S--A-12/18/2001 05:05:49 -> 12/18/2001 05:06:11 6 02 willard.uchicago.edu 40911 <-> 103 55.icafe.euroweb.ro 1705 48 3107 10 ---PA-12/18/2001 05:05:46 -> 12/18/2001 05:06:12 6 103 55.icafe.euroweb.ro 1705 <-> 02 willard.uchicago.edu 40911 43 2602 00 -S-PA-

Reading the logs through

✦ What happened later on in the logs?✧ This can give us more information on what else was

compromised

✦ Connections in from avanti0.hab.de

12/17/2001 21:50:40 -> 12/17/2001 21:50:41 6 90 avanti0.hab.de 4222 <-> 02 willard.uchicago.edu 22 5 296 00 FS-PA-12/17/2001 21:50:40 -> 12/17/2001 21:50:41 6 02 willard.uchicago.edu 22 <-> 90 avanti0.hab.de 4222 2 100 00 -SR-A-12/17/2001 21:50:41 -> 12/17/2001 21:50:42 6 02 willard.uchicago.edu 22 <-> 90 avanti0.hab.de 4222 6 543 10 F-RPA-12/17/2001 21:50:41 -> 12/17/2001 21:50:42 6 90 avanti0.hab.de 4222 <-> 02 willard.uchicago.edu 22 3 120 10 --R---12/18/2001 06:12:05 -> 12/18/2001 06:12:05 6 02 willard.uchicago.edu 22 <-> 90 avanti0.hab.de 1023 1 60 00 -S--A-12/18/2001 06:12:05 -> 12/18/2001 06:12:05 6 90 avanti0.hab.de 1023 <-> 02 willard.uchicago.edu 22 2 112 00 -S--A-12/18/2001 06:12:05 -> 12/18/2001 06:12:10 6 90 avanti0.hab.de 1023 <-> 02 willard.uchicago.edu 22 20 1647 10 ---PA-

Repeat the process

✦ Looking at who avanti0.hab.de connected to can reveal more compromised machines

✦ We find one more… aupc1.uchicago.edu

% flow-cat * |flow-extract -e 'host = avanti0.hab.de && host != willard { print }'12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 90 avanti0.hab.de 4284 <-> 02 aupc1.uchicago.edu 22 5 248 00 FS-PA-12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 90 avanti0.hab.de 4284 <-> 02 aupc1.uchicago.edu 22 2 80 10 --R---12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 02 aupc1.uchicago.edu 22 <-> 90 avanti0.hab.de 4284 4 283 10 F--PA-12/18/2001 00:40:14 -> 12/18/2001 00:40:14 6 02 aupc1.uchicago.edu 22 <-> 90 avanti0.hab.de 4284 1 44 00 -S--A-12/18/2001 08:00:52 -> 12/18/2001 08:00:52 6 02 aupc1.uchicago.edu 22 <-> 90 avanti0.hab.de 1023 1 44 00 -S--A-

Logging Methods

✦ We use flow logs from routers (and some switched). ✧ Mark Fulmer’s flow-tools

✧ <http://www.splintered.net/sw/flow-tools>✧ Flow-extract

✧ <http://security.uchicago.edu/tools/net-forensics

Flow Logs

✦ Advantages✧ Straight from router✧ No sense of state✧ Authoritative

✦ Disadvantages✧ Need to have a router

that supports flows where you want to log

✧ Missing useful information (e.g. sequence number)

✧ No sense of state

Logging Methods

✦ Argus – QoSient, LLC – Carter Bullard✧ <http://www.qosient.com/argus>✧ OpenSource effort and proprietary version

✧ Same flow model, performance and scaling✧ Origin/History:

✧ Early 1990’s Work at CERT✧ Guerilla work until startup in 1999✧ Continued analysis/experimentation at CMU

✧ Validation, IDS, web logging (FlowScan-style)

Argus

✦ Applications – audit✧ Edge Traffic Characterization✧ Security✧ Anonymized research data (use analysis)✧ Traffic accounting✧ Service/Policy Discovery

✧ who/how/how much✧ Unexpected service delivery?

✧ QoS validation✧ Internet Call records

✧ Who talks to whom – not what’s said✧ Contrast to Carnivore

Argus Flow Logs✦ Advantages

✧ Authoritative✧ Transaction flow aggregation✧ Strong flow model/semantic

✧ TCP window delta/retrans✧ ICMP aggregation✧ Accurate timestamps

✧ TCPdump selection syntax✧ Scalable – multiple probes✧ Flexible – put probe anywhere

✧ Subnet/switch/host✧ Limited access to user data✧ Higher level tools for

analysis/indexing

✦ Disadvantages✧ Technology, no sexy apps✧ Limited documentation✧ Probe Architecture

✧ Vs switches, IPSEC, etc✧ Scaling factors

✧ DoS vulnerability

Argus

✦ Quick Demo

Interesting Questions

✦ Aggregate transaction analysis✧ Web trans frames smtp spam✧ Probes followed by specific connections

✦ Application fingerprinting✧ Regardless of port

✦ Network service Provision✧ End2End or Edge2Ether✧ Ask for a service, not a connection

Problems in identifying traffic

✦ What if the port number jumps around✧ Many file sharing programs are beginning to

do this to evade firewalls✧ If it’s used by a lot of people it will look like

random traffic from a statistical view point and will just appear as noise

✧ Application layer analysis can help✧ What if the traffic is encrypted?✧ Need lots of storage and a fast machine to keep up

Network Graphs

✦ Allows quick visualization of network use✦ MRTG

✧ <http://people.ee.ethz.ch/~oetiker/webtools/mrtg.html>

✦ Cricket✧ <http://cricket.sourceforge.net>

✦ FlowScan✧ <http://net.dois.wisc.edu/~plonka/FlowScan>

Traffic Management

✦ Traditional Rate Limiting✧ Who to rate limit?

✧ Just the dorms?✧ Everyone?✧ Known abusers?

✧ How much to Rate Limit?✧ Can’t do application layer limiting, so it may

be ineffective to programs that jump ports

Traffic Management

✦ Packeteer, etc.✧ Can do application level

✧ What if the traffic is encrypted?✧ Can’t do high bandwidth

✧ ~100Mb/sec okay, ~1Gb/sec not

✦ Other options…?