Analysts International

Performing a Computer Security Investigation

2

Introductions• Mark Lachniet from Analysts International,

Sequoia Services Group• Member of the HTCIA• Not in law enforcement or a lawyer• Senior Security Engineer and Security

Services technical lead• Frequent presenter and trainer• Certified Information Systems Security

Professional (CISSP)• Microsoft MCSE, Novell Master CNE, Linux

LPI Certified LPIC-1, Check Point Certified CCSE, TruSecure TICSA, etc.

3

Agenda• Where a technical security engineer fits into

an investigation• Frequent types of incidents• Anonymous hacks vs. targeted• How hacking happens• Types of investigation• Expanding the scope of investigation• Documentation and procedures• Real life examples• Security services – detection and prevention

4

The Security Engineer• My perspective is no doubt very different from the

other presenters you will hear from today

• My job is to understand the technical details of computer security, and to know enough about forensics and the legal system not to mess things up

• A security engineer is (usually) from the private sector, or internal Information Security staff for larger organizations

• There are a variety of security professionals who work in the industry with different emphasis– Policies and procedures

– Networking

– Server / workstation

– “White Hat” ethical hacking

5

The Security Engineer• Recently, there has been a massive influx of people

with questionable credentials and skills• Look for engineers that have industry-accepted

certifications from respected organizations:– #1 The Certified Information Systems Security

Professional (CISSP) from isc2.org– Other low-level technical certs exist (TICSA, Security+)

but are not appropriate for sensitive work

• Certifications also exist for forensic specialists, but this is somewhat different from what I do

• Also look for specific product certifications on the products used (Windows, Linux, etc.)

• Using an engineer with certifications may make for an easier day in court because they have been accredited by a recognized body

6

Where the Security Engineer Fits• Dedicated security consultants can help in both

prevention and response• In prevention – designing and maintaining secure

technological and organizational systems (not just technology!)

• In response – the topic at hand– For specific tools and technical expertise for a variety

of systems (servers, workstations, network devices)– To investigate an incident before deciding whether or

not to prosecute– To help weigh costs and benefits of various courses of

action – how to investigate, how to secure– To assist in prosecution by thoroughly researching

and documenting findings without the constrictions that law enforcement would have

7

Frequent Security Incidents• The vast majority of calls I get are in regard

to a “hacking incident” • Almost of these incidents are on Internet-

connected machines• Most incidents are precipitated by:

– An external complaint (your mail server is sending me a lot of spam e-mail)

– A change in the system (the hard drive is full, strange new programs are running, tape backups are taking a lot longer)

– The Internet is “slow” or we see strange activity– A threat from an insider – usually a network

administrator making casual statements about how they could “take them out” if they ever got fired

8

Frequent Security Incidents• Many complaints focus on inappropriate use

of company technology:– Employees looking at pornography at work– A user is suspected of having “hacking” tools– Suspected theft of trade secrets / proprietary info

• Another frequent event is an “employee termination” scenario:– Employee is usually a computer administrator– Employee has extensive access to many systems– Employee is a “troublemaker” – Employer wishes help in terminating the employee, and

wants to remove their access FIRST before firing him– Typically involves a lot of brainstorming to identify all

possible points of ingress to the computing environment

9

An Impersonal World• There are really two different types of computer

security incidents – personal and impersonal• In my work, they are almost always impersonal

hacking attacks, not someone who intentionally targeted the victim

• Most hackers could care less who you are, or what sensitive information you have, they simply want to control an Internet-connected server

• Usually this access is used in a few ways:– To commit crimes, using you as the staging point– To share questionable material, using your Internet

connection and server space (the “warez”server)– To access questionable material, using you as a relay to

hide their origin (frequently porn)– To use you as a SPAM relay to send junk e-mail to

thousands of people

10

How Hacking Happens• Hacking is generally possible due to a vulnerability or

a mis-configuration in some server or device• Vulnerabilities exist, and are constantly discovered,

in all types of systems by hackers and “white hats”• Patches are released, but rarely applied due to lack of

resources, awareness, or just plain apathy• Case in point – the latest major Internet worm called

“slammer” took advantage of a hole that has had a software fix for over a year!

• Hacking also occurs due to a variety of mis-configuration issues such as:– Not using a firewall to restrict access from the Internet– Running programs that are not necessary– Poor passwords, default passwords– Default configurations

11

Understanding Networks

Internal Network(Protected Machines)

DMZ Network(Internet Accessible Machines)

The Internet

Bad Person

Good Person

Company Firewall

Exchange e-Mail

ACME Corp Network

Internet Router

Web Server

User Laptop Printer

File Server User Workstation

12

Understanding Networks• The example given previously is an example of “best

practices” in network design, and provides some defense against Internet attacks

• Many (most?) organizations do not have an adequate network design, and have significant risk from the Net

• Even the BEST network design can’t protect a machine that is insecure!

• Each machine that can talk to the Internet has a unique identifier called an “IP Address”

• IP addresses are sometimes static, and sometimes change frequently (especially for dial-up users)

• Regardless, tracking IP addresses is frequently our only recourse to track network attacks

• For example, if the IP address of a hacker can be tracked to AOL, it is then possible to obtain further info from AOL through legal action

13

Types of Investigation• Once a call comes in requesting help in investigation,

the engineer is dispatched on-site• The first (and perhaps most important) step is discuss

the situation with the victim before doing any work• There are basically three ways to approach an

investigation:– “Pull the Plug” – don’t touch the machine– “Limited Investigation” – tread lightly– “Extensive Investigation” – heavy footprint

• Each of these approaches have advantages and disadvantages, depending on your goals

• The most important question to ask is how strongly the customer feels about trying to prosecute

• The second most important question to ask is how much $$ they have to spend

14

“Pull the Plug”• Used when a company is VERY intent on prosecution and

does not want to risk any tampering w/ evidence• As the title implies, the only investigation physically

performed on the target system would be to pull the power and network cords

• This is highly disruptive and expensive, as the server is no longer available

• There are also potential immediate results (you might miss evidence that would lead you to investigate other systems, for example)

• There is also no opportunity to examine the “state” of the machine that will be lost when turned off:– Which programs are running– Current network connections

• Investigation of other data sources should still be performed (for all types)

15

“Limited Investigation”• Used when the company hasn’t decided if they want to

prosecute, and are willing to obtain more information at the risk of having evidence modified

• Is less disruptive and less expensive – the server doesn’t need to be taken down to do the work

• Must analyze the system with tools that leave a very light “footprint” and will not modify much system information:– File (M)odify, (A)ccess, (C)reate date flags on files– System registry settings (for Windows machines)

• The goal is to determine what happened without modifying the system in a way that we lose evidence that a forensic investigator could use in court

• Doing this is technically difficult• Some information cannot be easily found without

leaving a footprint

16

“Extensive Investigation”• Most extensive data-gathering, thus slightly more

expensive due to labor• Still non-disruptive, the server is up and running,

although it may need to be restarted occasionally• Includes all of the work of the previous• After all “light footprint” methods have been tried, a

decision should be made whether to continue with more invasive techniques

• More invasive tools can be used – these will leave a trail, but will provide the maximum of information

• For example, it may be possible to do things such as:– Monitor all file accesses on the system in real-time– Monitor and record network traffic– Improve the logging data collected (usually none by default) – Read logs, files, view disk contents– Plant honeypots (password.xls, etc.)

17

Analyze Other Log Sources• In the networked world, no machine is an island• If systems have been appropriately designed and

implemented, which isn’t that often, there will be useful information in a variety of places

• The investigator must expand the scope from the “victim system” and look elsewhere

• Additional evidence can be found in many places:– Network and security devices on location– Internet Service Providers (AOL, DSL providers, etc)– Other servers on the network– Client workstations (especially if an insider is

suspected)– Authentication systems– The attacker’s workstation

18

Expanding the Scope of Investigation

Internal Network(Protected Machines)

DMZ Network(Internet Accessible Machines)

The Internet

Bad Person

Good Person

Company Firewall

Exchange e-Mail

ACME Corp Network

Internet Router

Web Server

User Laptop Printer

File Server User Workstation

!!

!!

!

! !

!

Best Source for Logging

19

Analyzing Router/Firewall logs• Some of the best information for figuring out how an

attack occurred and subsequent activity is by examining the logs of network devices such as routers and firewalls

• Unfortunately, many people don’t collect this data and store it, or even know that its possible

• Network device logs can provide a detail of what type of information traveled between network systems:– Determine how the system was profiled (reconnaissance)

– Determine how the system was attacked (vulnerability)

– Determine what happened after the attack – did the hacker use your system to store files? Attack other systems?

– Determine if multiple parties were involved (hackers tend to run in packs in different parts of the world)

20

Analyzing User Workstations• In the event that some internal involvement is

suspected, or even just to be thorough, other servers and workstations should be examined

• Computers that are in regular use store a lot of interesting information such as:– Internet history (Internet Explorer, Netscape)– E-mail (settings that lead to servers, old mail)– Content (naughty pictures, confidential info)– Hacking tools and software

• Once an attack has been tracked to a particular computer (perhaps through IP address) a forensic analyst can pick apart the workstation to find evidence

• Organizations with strong security policies will enforce mandatory vacations and analyze the user’s workstation as a part of standard practice

21

Record Keeping and Static Procedures

• When doing this work, the security engineer should take detailed written (physical) notes Actions taken should be detailed along with the time it was done

• Note: Time is a big issue! The time of each device is probably a little bit different – what is the time of the victim system vs. local time? Other devices?

• It is good if more than one person is involved, with the second person signing off on it

• Static procedures should be used to eliminate the risk of error and to have a standardized methodology

• Electronic record keeping must also be secured to minimize the risk of modification – one way is through digital signatures (cryptographic hashes that prove the integrity of data)

22

Create a Deliverable Document• Once you have as much information as possible, you

need to document all of the data you have collected and provide an analysis of the raw data

• This document should attempt to summarize:– What happened (chronological sequence of events)– How it happened (what vulnerability was used)– Problem areas (what couldn’t be done / analyzed)– Next steps (both short term recovery and long term security

steps that should be taken)– Full appendix of collected data

• All of this information needs to be thoroughly explained so that non-technical people can understand the scope and impact of the incident and make decisions

• This document can be given to law enforcement to save time – a nice tidy package

23

Next Steps• The decision to prosecute is not an easy one to make

because there are many implications:– What will be the cost of prosecuting, in terms of legal

expenses, time spent, interruption to operations, etc.– What is the likelihood of success?– What is to be gained by prosecuting?– What are the implications to public image? Nobody

wants to be in the newspaper, nobody wants to be exposed as having poor security

– There is no guarantee that you will even be able to prosecute if you want to. What if the perpetrator lives in a developing country with now computer laws?

• Unless it was an insider job, or a specifically targeted attack, most people consider it a “learning experience” and hopefully secure their systems

24

Examples: The Warez Server• For this presentation, I did a little experiment, and

set up a “honeypot” server on the Internet• This server was a standard Windows 2000 server,

and was fully up to date (no vulnerabilities)• The only change made from the default

configuration was a single (confusing) checkbox that said to allow write access on the File Transfer Protocol (FTP) server – an easy mistake to make

• I put the machine on the Internet to see how long it would take for hackers to find it and abuse it

• The answer is: 3 days. Within 3 days, hackers had found the server, and discovered that it was possible to store files there anonymously

25

Examples: The Warez Server• Within a week, a “tag” had been placed

(hacker lingo for claiming the server – there is honor among thieves)

• A few days later, a huge number of “hidden” directories were created on the server, and software was uploaded to it.

• A few days after that, people from the Internet were downloading the illicit content, and I pulled the plug

• I’m still not sure what they uploaded, but most of the time its porn

• The lesson here is that they WILL find you, and quickly at that

26

Examples: Manufacturing• A manufacturing company was getting

complaints from people claiming that spam was coming from their mail server

• Their ISP shut them down due to abuse calls• They had investigated internally and couldn’t

figure out what was happening• Analysis of the server found that they were

directly connected to the Internet without a firewall or other protection

• Further analysis found several problems:– An open mail relay (allows spam)– An open proxy server (allows anonymous web access)– An open socks server (allows full Internet access)

27

Examples: Manufacturing• Analysis of log files showed that people from all

over the world had been relaying connections through their server

• Abuse included people looking at pornographic web sites, sending spam

• A search of the Internet found that the company server had been listed on multiple hacker sites as being an “open” relay

• Thus, not only are the hackers who find you going to abuse you, but they are going to share their good fortune with others

• What are the legal liabilities of being a third party to this type of activity?

28

Examples: Marketing• A marketing firm calls with concerns because

the network administrator found a remote-control program on the server (very bad)

• The server was connected to the Internet without a firewall

• Additional user ID’s had been created and granted administrative access

• Client suspected internal involvement• Logging on the server was turned off, so no

good data was collected• Logging on the network devices was also

turned off, so there was no data there either

29

Examples: Marketing• Examination of the server turned up some

evidence, such as the time and date that the remote control software was installed, and evidence that there was a hack but not much!

• However, because there was no logging, there was no sure way to know if the attack was internal or external

• Also because there was no logging, there was no way to track to an offending workstation by IP address

• The only real option was to clean up the damage, and start recommending some security services to stop it from happening again

30

Examples: K12 District• School district in Michigan with a fast

connection to the Internet• No problems were known• The district contracted with us to have a

managed firewall installed• As soon as we turned it on and started

analyzing traffic, it was obvious that they were currently being abused

• Investigation showed that they were unknowingly hosting child pornography – not a good thing for a school

• Many other people have found existing problems just by logging

31

Prevention and Response• None of the previous incidents made it to the

legal system, it just wasn’t worth it for them• None the less, it was an expensive, emotional and

painful experience for them• Much of that pain could have been minimized

through prevention instead of response• Unfortunately, computer security is a somewhat

like the wild west – its somewhat lawless, although serious crimes can be pursued its usually not worth it

• We use the metaphor of the neighborhood when describing computer security – the best approach is to make your own home hard enough to break into that they go to your neighbor instead

32

Security Services to Know• There are some security services that are simply

mandatory for anyone who has important data• Failure in security due diligence can, in itself,

lead to prosecution of corporate officers• Privacy laws, especially the Health Insurance

Portability and Accountability act of 1996 (HIPAA) mandate security best practices

• In my opinion, this will be a huge area of emphasis in the next two decades, both for criminal and civil action

• Security breaches are becoming commonplace in the media, 6 million credit card numbers compromised, etc.

• Thus, people need prevention!

33

Security Services to Know• The following list doesn’t do justice to the field, but

here are a few things that every company needs to do:

– Design secure solutions - networks, systems and software with security in mind. At least a firewall

– Have vulnerability assessments performed (ethical hacking, or security needs analysis)

– Ensure that all servers that are Internet connected or store important data are properly “hardened”

– Use some kind of auditing and logging system to maintain an audit trail

– Maintain appropriate computer use policies

– Retain security staff to regularly evaluate log data, perform analysis, etc.

34

Thank You!

Mark Lachniet, Sr. Security EngineerCISSP, MCNE, MCSE, CCSE, LPIC-1, TICSAAnalysts International - Sequoia Services

3101 Technology Blvd. Suite ALansing, MI 48910

phone: 517.336.1004fax: 517.336.1004