NetFlows & Network NetFlows & Network ForensicsForensicsE. Larry Lidzellidz@pobox.com

NetFlowsNetFlowsLogs right from routers and switchesSupported by most network vendorsTakes some CPU time on the routers

◦Might be turned on for network performance reasons

Quick to search throughMultiple ways to save on a system;

multitude of available commercial tools

General Intro to FlowsGeneral Intro to FlowsLogs flows, not connections!

◦This is good… and bad.Flows are uni-directional

◦You’ll see two flows for each connection: one in each direction.

◦Tools can reconnect them using heuristics

Introduction to Network Introduction to Network AuditingAuditingObligatory "audit" definition:

◦1. trans. To make an official systematic examination of (accounts), so as to ascertain their accuracy.

So, what is "Network" auditing?◦Making a systematic examination of

network traffic to understand the traffic and verify its legitimacy.

Introduction to Network Introduction to Network ForensicsForensicsObligatory "forensic" definition:

◦A. adj. Pertaining to, connected with, or used in courts of law; suitable or analogous to pleadings in court. forensic medicine: medicine in its relations to law; medical jurisprudence.

So, what is "Network" forensics?

◦Looking at traffic on a computer network in such a way that it could be used for evidence in a court of law.

So, what am I talking So, what am I talking about?about?Basically, the ability to log

information about your network traffic in order to help with an investigation. ◦Generally, a compromise.

Also useful for trying to explain unexplained network problems.

A Quick StoryA Quick StoryMachine on network compromisedLaunched an evil DoS attack

◦Took out an ISP for a weekendISP thought they knew who did itWe were contacted by Alaska State

Police and Secret Service◦Suspect is in Anchorage, AK◦Suspect is suspected of Credit Card

FraudCompromise is a big, big problem

◦Lots of damage caused to ISP◦Credit Card fraud via our network

A Quick Story, continuedA Quick Story, continuedThe network audit logs show that the

compromise occurred from *.ak.us.ibm.net◦ (IBM was a large ISP at the time)

We turn logs over to Alaska State Police◦ Note: none of our sensitive data was turned

over; much, much easier to convince the powers that be to turn over

Kid denies it, confesses when shown the logs; turns on friends◦ (You can tell this is an old story – these days

it’d be a professional crook)Parents offer to pay restitution

◦ (Do you think that ever happened?)Network logs save the day!

Some TerminologySome Terminology Network Flow

◦ A group of packets related by having attributes in common.

◦ Distinct from a “network connection.” AS

◦ Autonomous System Number◦ Every ISP has one, used in routing.

VLAN◦ Virtual LAN◦ A single broadcast segment of a network (a

subnet).

These definitions are slightly simplified as the complexities aren’t relevant for this conversation. I’m sure you’ll hear more details on these later on… if you haven’t already.

TCP/IPTCP/IP When looking at netflows, the more you know about TCP/IP,

the better. 3 main protocols: TCP, UDP, ICMP UDP/ICMP are conectionless.

◦ No way to easily tell if traffic is the start of a "connection" or part way through.

ICMP: Internet Control Message Protocol.◦ Ping, traceroute, etc.◦ Type: destination unreachable (3), echo req (8), echo rep (0).◦ Code: 3.0: network unreachable, 3.3: port unreachable.

TCP has connections.◦ Controlled by TCP Flags: FIN, SYN, RST, PSH, ACK, URG◦ FIN: Sender is finished sending data.◦ SYN: Synchronize sequence numbers to initiate connection.◦ RST: Reset the connection (closing it).◦ PSH: Push data to the application as soon as possible.◦ ACK: Acknowledge◦ URG: Urgent flag set.

A TCP ConnectionA TCP Connection

Client sends a SYN packet to server. If the server is listening, it responds with a SYN-

ACK packet. Client then sends back an ACK packet. Communication takes place with ACK (and often

PSH) set.◦ Push flag is generally set when the application is

waiting on input from a user or otherwise expects to be idle.

Side closing the connection sends a FIN. Other side sends an FIN-ACK. Other side sends a FIN. Closing side sends an FIN-ACK.

If the server isn't listening, it sends a RST.

IP Addresses (v4)IP Addresses (v4)Netmasking128.135.0.0/255.255.0.0

◦128.135.0.255 is broadcast for 128.135.0 subnet.

127.0.0.1/255.0.0.0◦local net. 127.0.0.1 is local host

10.0.0.0/255.0.0.0, 192.168.0.0/255.255.0.0, 172.16.0.0/255.240.0.0◦private IP space

TCP/UDP PortsTCP/UDP PortsUseful to know a variety of ports. In particular the ones that are used a lot on

your networks.General

◦ 21: ftp◦ 22: ssh◦ 25: smtp◦ 53: dns◦ 80: http◦ 135, 137, 139, 445: windows ports

Details beyond the scope of this conversation◦ 443: https◦ 6667: irc

NetFlows vs. other NetFlows vs. other securitysecurityvs. Intrusion Detection System

◦ “We have an IDS, what else do we need?“vs. System Forensics

◦ "We can always look at the system to figure out what happened.“

vs. Sniffers◦ "We can always install a sniffer when we

expect something is wrong.“vs. Firewalls

◦ "We have firewall logs, who needs anything more?"

"When I've already got X, why do I need Network Audit Logs?"

NetFlow vs. …NetFlow vs. …IDS

◦Generally do not log traffic unless it is “suspicious”

System Forensics◦With systems, you’re always looking at

a system that has been tampered withSniffers

◦Disk consumption; privacy concernsFirewalls

◦Often only at the network border; not optimized for searching logs

Defense in DepthDefense in DepthUse it all.

(Based on available budget, resources, etc.)

Network Design Network Design ConsiderationsConsiderationsDecide what you want to log.

◦ Which machines? Which networks?Choose places that handle interesting traffic.

◦ Some examples: Gateways Peering points DMZs Important machines

Keep logging server close to router/switchTheoretically possible to inject fake packets

into stream◦ Difficult

Network traffic requirements for logs fairly small.◦ About a 1.5Mb/sec per 100 Mb/sec of traffic.

PreparePreparePrepare ahead of time!

◦ That way, when you you need it, you'll have logs to consult.

◦ Too late if the incident has already happened.Allocate hardware

◦ Start small if necessary, build a case for the usefulness of the tools.

Install softwareTest! See if you can avoid the logs.

◦ If you can, it's probably misconfigured.◦ We noticed that we were missing logs from one

segment to external networks.◦ Turns out we were missing an interface on our

switch.

What can you see in a What can you see in a NetFlow?NetFlow? Think of it like phone call records: who called whom, when. Without the conversation.

Source/Destination IP; Input/Output Router Interface Protcol Type of Service NextHop Packet Count Octet Count Start/End Time TCP Flags Source/Destination AS Source/Dest Network Mask Input Output Interface encapsulation Size IP Address of next hop within the peer Router IP of cache shortcut in supervisor

Reading NetFlowsReading NetFlowsDepends a bit on the tool… but:

04/5/2011 11:59:46 -> 04/5/2011 11:59:46 6 01 65.219.38.62 3844 <-> 11 128.135.12.7 http 8 480 00 FS-PA-

04/5/2011 11:59:46 -> 04/5/2011 11:59:46 6 11 128.135.12.7 http <-> 01 65.219.38.62 3844 11 11636 00 FS-PA-

Flow start time, flow stop time Flows sorted by stop time Protocol number Source interface, source hostname, source port <-> Source and destination are the source and destination of the flow, not of the

connection. Should be two flows for each connection (assuming traffic is sent in both

directions). Destination interface, destination hostname, destination port Packets in flow, Bytes in flow, Type of Service TCP Flags: All TCP flags set in that flow.

An IncidentAn IncidentReceived a report of a

compromised system from a sysadmin◦Well… of ten systems.◦Systems had a file which listed the

other machines in the same BotNet – 35 on our network

◦Each system we looked at had a different list of systems in that file.

The Challenge: find all compromised machines.

Incident, continuedIncident, continued We chose a system that wasn’t doing much

(network-wise). ◦ Why?

Pulled all the NetFlows for that system. By reading the Flows, we were able to find the

signs of a compromise (attack against an RPC service).

Also saw another host (38.31.107.236) connecting to a backdoor on a port (1524)

Then connections out from the machine to download a rootkit (to 152.3.127.99).

Then the BotNet traffic starts up.

Note: I’m skipping some of the analysis of how we figure out that it was downloading a rootkit, etc.

Incident, continuedIncident, continuedLooked for all connections from our

network to 152.3.127.99Sorted the output and got the list of our IP

addresses in it.

We were able to easily pull together the list of compromised machines by looking at the network trail of the one machine.

In practice, we hand verified a lot of this.◦ It’s always good to double check.◦ Compared the machines to those listed in the

file on the infected machines.◦ Did other analysis of the systems to ensure

that they were compromised. (Forensics in depth?)

NetForensics: “when?”NetForensics: “when?”When was the first traffic to a

back door?When did traffic first seem to

successfully connect to ports that previous had no traffic?

When did BotNet traffic start up?Did the machine start

transferring lots more traffic at a certain point in time?

NetForensics: “how?”NetForensics: “how?”What type of machine is it?When you look at it, what

vulnerable services, if any, do you see?

Did unexpected traffic start up shortly after a connection to a service running on the machine?

Connections to a port's service that isn't used.◦ftp/http out to a third site◦Particularly if it is grabbing a rootkit.

NetForensics: “who?”NetForensics: “who?”Who is connecting to it that you

don't expect to connect?Which IP caused the compromise?

◦If you know when/how it was compromised.

Who connected to the backdoors?Look for weird connections.

◦May be a backdoor or a covert channel.

NetForensics: “what?”NetForensics: “what?”How much data was transferred?

◦Useful if you fear that information was stolen.

◦If they only transferred 500k of data, they didn't take the 20 Gig master plans for the new virtual-hyper-cyber-widget of doom.

What was on the machine in the first place?

NetForensics: “why?”NetForensics: “why?”In my experience, NetFlows

aren’t so useful for determining motive.

Sorry.

When in Doubt:When in Doubt: Read through the traffic logs for the

compromised machine. Extract logs for the specific host. Very Time Consuming

◦ You won't miss anything...◦ Unless there's too much data to look at.◦ Start ignoring stuff that you suspect to be

legitimate (ntp, dns, etc.) Don't worry about stuff before the compromise. Build a timeline.

◦ Keep notes as to the connections that you see.◦ Who connected out to where at what time?◦ Which are legitimate?◦ Interview system administrators and users.

Things to look for:Things to look for: Intruders often script attacks

◦ Machines will have similar traffic patterns◦ Same source IPs.◦ Same backdoor ports.◦ Connect to same external site.

Known bad traffic◦ IPs/hosts list on “bad traffic” reports◦ Known malware ports

Example: finding SQL Slammer◦ flow-extract -nd ft-v06.2003-01-25.030000 -e 'proto = 17 && port = 1434 && pkts = 1 && octets = 404 && ((srcnet = 10.0.0.0/255.255.0.0 && dstport = 1434 ) || (dstnet = 10.0.0.0/255.255.0.0 && srcport = 1434)) { print }

ArgusArgusLogs off of a spanning port

◦No need to have router supportFrom QoSient.

◦Free for most purposes. Read the license agreement.

Can log bytes of application data, too. packet contents, smtp exchange, ftp username/password

A bit slower than NetFlows

I use it at home

SummarySummaryNetFlows are a powerful tool.

Extremely useful for Security, but also for network management.◦ If I can find machines compromised by a

malefactor trying to remain anonymous, I can easily find the bandwidth hogs.

Heavily adopted behind the scenes in Security and Network tools.

Very useful at making network traffic tangible as you’re learning more about it. ◦ They’re a little bit higher level than a Sniffer

would show you, and therefore easier to consume.

Questions?Questions?Questions?Comments?Thoughts?

Anything else you’d like to learn about in the time remaining?