Stuff

Steve Romig

romig@net.ohio-state.edu

Introduction

• Summary: things we’ve learned about incident response, computer crime.– Things we’ve done right– Things we’ve done wrong

• Vehicle: an investigation that started 4 years ago.

Pre-incident

• OSU didn't have much of an incident response team– Incident response was ad-hoc– Response depended on who responded– Had recently hired me part time

• I started some minimal initiatives:– Tracking incidents– Logging (authentication, network traffic)– Education/awareness meetings

19:00 August 27, 1996

• California ISP calls me at home: they'd been compromised– Attack came via our modem pool.– They named a suspect: someone using the

nickname XXX on IRC.

• I confirmed the activity– Intruder had been logged in through modem pool

since 2:00 that morning.– We had several previous incidents for this intruder

Lessons

• Publish your contact info

• Log lots, log often, retain your logs

• Early action can prevent later nastiness

00:30 August 28, 1996

• Intruder is *still* logged in

• Phone traces through Ameritech: – A promising start, sort of– Phone traces work “just like in the movies”

10:00 August 28, 1996

• Intruder is *still* logged in• Phone traces through Ameritech:

– I've definitely seen too many movies– It doesn't work the way I thought!

• Lessons:– Publish contact info

• Everyone you talk to• Carry at all times

August 28, 1996

• Phone traces through Ameritech:– They keep records– We can request traces after the fact

• Lessons:– Work out procedures, info required with

your local police, phone company.

August 29, 1996

• Set up tcpdump logging of intruder sessions.• We had to identify sessions through our

authentication logs, start/stop tcpdump by hand. Ick.

• Also raised legal issues – ECPA?– Talked to our lawyer – “no”.– This indemnifies me (to some degree) - now its the

University's problem

Lessons:

• Talk to your lawyers.

• Create an incident response “team”– Not necessarily full time– Key players: legal, IT, communications,

student affairs, help desk, etc.– Make a plan – who decides how/whether

incidents will be handled.

August 30, 1996

• We enter the next level of phone trace hell:– Confusion over what sorts of court

order/subpoena/search warrant we needed to request the trace.

– I don’t recall how this was resolved.

September 3, 1996

• Got tired of starting tcpdump by hand• tacacs-action

– Config file lists accounts and actions to take on login/logout.

– Actions include "log" and "page"– "page" does what you'd expect– "log" invokes tcpdump on a sniffer on the correct

subnet to capture their traffic on login (filtering for just their IP address), or stops tcpdump for that session on logout.

Lessons

• Automation is a wonderful thing.

• We discovered that there were several people using several accounts.

September 5, 1996

• I got insanely sick of getting paged all the time. Turned off the paging in the control file for tacacs-action.

• We discovered that one of the local groups hangs out in #614 on IRC.

• Started lurking in #614…

Meanwhile…

• Tcpdump logs are piling up– We read through the logs with tcpdump

and strings and a program called cleanup that Mark Fullmer wrote.

– This is tedious, icky, and prone to errors. Its hard to read terminal escape sequences and other obfuscated traffic.

Review

• GUI to browse list of logs, view contents of logs (by "sessions") and contents of sessions.

Log Listing Window

• List of logs, sizes

• Double click log to see summary

Session Summary Window

• List of sessions from one log

• Double click to see contents

Session Contents

Session Replay

• Escape sequences are hard to read

• Replay takes the server to client traffic and writes it at a controlled rate to a terminal emulator

September 13, 1996

• Morning ritual - check mail, download tcpdump logs, run the pre-processing stuff, get a cup of coffee, and settle down to read.

• They were doing lots of IRC, email, some probing, some exploits.

• They used SSH and PGP– Through telnet sessions– Sent passphrases for private keys via telnet– Sent private keys via FTP and IRC

Lessons:

• Weakest link

• When you send encrypted email, encrypt it to your public key also :-(

September 21, 1996

• We have been trying to ID the suspects– Maintained “players” list– Original theory: ID them and jump directly

to search warrants. – Nope, it doesn’t work that way: phone

trace, pen register, search warrant. Builds body of proof.

• Phone traces are still up in the air...

Ah, breaking news…

• YYY notes that XXX gets accounts by sniffing passwords in an OSU public lab and shares them with friends.– Yes, the labs were sniffable– Despite recommendations to fix this the

year before

Lesson

• Fix known security problems

• Learn from past mistakes– Our labs are mostly fixed now– Now we’re deploying wireless networking…

October 1, 1996

• We tried to find the local 2600 meeting– 2600 magazine claimed they met at a local

mall– Not as far as we could tell

• XXX says that the local 2600 meeting isn't where its advertised. Aha! – Took some time before we learned true

location

October 15, 1996

• The first of the military/government intrusions. – The issue of notification arises again. – We call the FBI and the various military

CERTs.

The Issue of Notification

• These guys ran a domain/host– They’d run probes, exploits from there– Guess who answers postmaster email?

• They’d receive complaints about their activity– Rarely– They’d respond with a polite note “so sorry,

we’ve been hacked…”

Notification

• At one point they broke into host Q.com

• We were all set to send q.com a warning about it

• Saw email between “our” crackers and them joking about the breakin – they were friends!

Lesson

• To notify or not, that is the question…

• Don’t know who you are talking to

• Don’t know whether they will follow your instructions (if you have any)

• Sticky question

Phf Exploits

• They were using the canonical "execute xterm on the remote box as root with DISPLAY set to my X server" version of the phf exploit.

• Tom’s nasty xterm…

Review Revisited

• X traffic is obscure – requests, results, events are sent in binary form.

• I mangled an X debugger called xmond to replay X sessions from the tcpdump logs

• Later, Justin Dolske rewrote this in Perl.

Browsing an X session

• Server side (next 2 slides)

• Key press and other events, replies, errors

• What the user typed

Browsing an X session

• Client side traffic now (next 2 slides)

• Requests sent to the server

• ``What the user sees''

• What is the user seeing now?

Replay of an X session

• More obvious now: the user was running vi

• Works for simple cases

October 23, 1996

• Many of “our” intruders made various confessions to other crimes: drugs, credit card fraud, cell phone fraud…

• XXX passes out OSU accounts

• Practice sessions, training, playing with new exploits. particularly XXX, WWW.

November 1, 1996

• They use our stolen modem pool accounts to get legit accounts from ISPs where they gives their real names and addresses.– That’s helpful…– Still missing identities for many of these

guys

November 7, 1996

• We learn the real 2600 location – a coffee shop in the 'burbs.

• So we start attending 2600…

• We also started to learn lots about the local groups– Lotek, Dark Data Lordz (ddl)– Sprang from some older group

November-ish, 1996

• We discover that one of the intruders is parking in front of Detective Rick’s house every day after school

• A picture’s worth a thousand words…

February 28, 1997

• The new phone traces are here!

• “we can’t send it electronically”

• “its never on a computer, sir”– 4 boxes of green bar paper– With email headers

Spring, 1997

• A dozen officers from different federal, military investigation groups arrive– Importance of carefully documenting

everything– Patience in dealing with law enforcement– You have as much to learn about what

they do and why, as they have to learn about your domain

Review Revisited, Again

• Lots of questions, but tedious to search N gigabytes of tcpdump logs for answers.

• Created a report generator for review:– IRC nicks used– email sent from/to– files transferred by ftp, irc dcc send– urls visited on web– detects some probes, some exploits

• The report is clickable – takes you to the session

Report Generator

• Generates reports for logs

• IRC nicks used, some simple intrusion detection, web & ftp URLs visited…

Intrusion Detection and Review

• Would have been nice to use a real IDS– Most couldn’t read pcap recordings– Wanted to double check my analysis of the logs

• Testing with George Jones– Standalone network, packet blaster, IDS agent,

IDS management station– Blasted 9 months worth of logs out– Lit it up like a Christmas tree!

• I didn’t miss much. IDS system did

Summer 1997

• Tom and I play far too much quake– Tom wrote a kick-ass proxy.– We both learned lots about the quake

protocol…

• “our” hackers play far too much quake… – ¼ of the tcpdump logs is quake traffic...

More than you wanted to know about Quake…

• Client tells server where it is moving, what weapon is firing, in what direction

• Server tells the client where it is, what’s happening around it

• Client does its rendering based on what direction its looking, location, surrounding events

• Common map information used by both

Yet More About Quake

• You can record “demos” in quake and replay them

• A demo file is essentially a recording of the server to client traffic, with some timing and camera angles thrown in.

“Honestly Boss, We ARE Working…”

• Quake-replay– Reads server to client traffic from a

tcpdump log– Massages it with view direction assumed

from the client to server traffic– Constructs a demo recording that you can

play

• Now we can see how well our intruders play

Summer 1997

• Traces all done– Confirmed that the intruders are who we

thought they were, sigh– Get permission to set up pen registers

• Pen registers– Record numbers called, caller-id– Left running for a month or so

September, 1997

• Search warrants are obtained– The night before…XXX says ” i don't worry about

breaking in through my accounts at the university because they'll never catch me..."

– They arranged to serve all 9? simultaneously.

• Served at 7 AM– Coffee and donuts with a few dozen officers…

• “Oh no, Bob's here!?"

1997 and Beyond

• We don’t know how this story will end– Law enforcement community is busy– Larger cases elevated to higher levels of

federal investigation take longer– Long lines at the forensics lab…

Some Final Remarks

• Scared the local cracker community - greatly reduced amount of nasty stuff coming through our modem pool

• The black hats work together better than the white hats do– fewer reasons not to– especially not worried about violating the

law :-)

Some Final Remarks

• How to assess the cost of computer crime?– most of what these guys did was pretty trivial– real cost is in the investigation and cleanup– ounce of prevention worth a pound of cure :-)– doesn't negate their responsibility– 1% of the modem pool capacity - what's that worth

in opportunity cost to our students?

Some Final Remarks

• Document everything that you do, learn, meticulously– daily journal a good idea– what did you learn, how did you learn it,

when did you first learn it?– what did you do, why? be detailed - how?

• Reactions to not being caught– Reaffirms view of self – I am elite!– Reaffirms belief that they won’t be caught

• Reactions to being caught– Reaffirms view of self – I am really elite!– Doesn’t appear to phase some of them– Badge of honor

Some Final Remarks

References

• You can (probably) find updated versions of this talk at http://www.net.ohio-state.edu/security. Look under “talks”.