Post on 19-Jul-2020
transcript
© 2019 CrySyS Lab, BME
Incident Response
Levente Buttyán
CrySyS Lab, BME HIT
buttyan@crysys.hu
w w w . c r y s y s . h u
|
Terminology
Attack
A deliberate action or activity carried out with malicious intent
aiming at defeating the security policy of a system
System compromise
Result of a successful attack
Security incident
A detected system compromise
Incident Response 2
|
Security incidents
Examples for security incidents
– Admin password is leaked
– Private customer records are stolen and published
– Data storage is encrypted by ransomware
– Service is disabled by a flood of requests
Security incidents can have a grave impact on an organization’s
business operations and reputation
Effective and efficient response to security incidents is
extremely important
Incident Response 3
|
Incident response goals
Minimize disruption to business operations
– Containment of compromise
– System recovery
Prevent future incidents of the same kind
– Incident analysis
– Recommendations for eliminating identified vulnerabilities
Support law enforcement
– Forensically sound retrieval and handling of evidence
– Accurate analysis reports
Incident Response 4
|
Incident response process
Incident Response
Pre-incident preparation
Strategy formulation
System recovery
Data collection and analysis
Reporting and recommendations
EducationEliminating vulnerabilities
Detection of the incident
pre-incident
post-incident
incident
5
|
Pre-incident preparation
Risk assessment
Preparation of hosts/devices (e.g., hardening, golden images,
backups, logging, host based IDS)
Preparation of networks (e.g., segmentation, logging, network
IDS, free monitoring ports)
Establishment of incident response policies and procedures
Establishment and training of a Computer Security Incident
Response Team (CSIRT)
Incident Response 6
|
Need for deep investigation
usually, primary goal is to recover from an incident (business continuity), but ...
you should also make efforts to better understand– the scope of the attack
» what parts of your system are affected? (space)
» for how long has the attack been going on? (time)
» what is the amount and nature of information that has been stolen? (impact)
– how the attacker actually compromised your system» what was the initial entry point?
» how did the attacker move laterally in your system?
» what are the vulnerabilities that made the attack possible?
– the goals of the attacker» why have you been attacked? (~ who is the attacker?)
» based on the collected evidence, did the attacker reach their goals?
results of deep investigation enable more complete response and better protection in the future
Incident Response 7
|
Some good advise
Be prepared!
– Security incidents did, do, and will happen
– Prepare your IR plan, have policies and procedures in place!
– Make backups, switch on logging!
Be capable of taking the first steps!
– Timely response is usually crucial
– You must have a CSIRT!
Don’t spare deep investigation!
– Better understanding of what happened enables better future protection
– Outsource deep analysis work to specialized companies (contracted well before the incident)!
Learn from the experience!
– What doesn’t kill you makes you stronger
– Improve your protection and educate your employees!
Incident Response 8
PRE-INCIDENT PREPARATION
|
Introduction
incident response is reactive in nature, with pre-incident
preparation being the only phase that comprises of proactive
measures
good preparation can
– save time, effort, and money later when an incident happens
– greatly increase the chances of successful incident response
common pre-incident preparation steps:
– identify corporate risks
– prepare hosts and networks for incident response and recovery
– establish policies and procedures that support incident response
– create and sustain a CSIRT that can assemble to handle incidents
– create and maintain a response toolkit (used by the CSIRT)
Incident Response 10
|
Preparing individual hosts
common steps that help any investigator respond effectively:
– record cryptographic checksums of critical files
– increase or enable secure audit logging
– build up your host’s defenses
– back up critical data and store media securely
– educate users about host-based security
as hosts change over time with new users, software, and
network configuration, the above steps need to be repeated
regularly
– a good approach is to incorporate them into organizational policies and
procedures
Incident Response 11
|
Recording cryptographic checksums
fingerprint a “known-good” system state by computing the cryptographic hashes of all important files (e.g., executables, DLLs, and drivers used by the OS) and storing those hashes in a secure location– checksums of “known-good” system state should be taken before an
intruder has the opportunity to compromise the system
then, regularly fingerprint the current system state and compare itagainst the stored “known-good” state– any changes to the system state will be identified and investigated
tool support– simple digest computing programs (e.g., md5sum)
» available on most Linux systems
» on Windows, use CygWin or download a Windows equivalent (e.g., md5sums)
– automate checksum computation for multiple files with scripts
– free and commercial products (e.g., TripWire, FCIV utility for Windows)» http://www.tripwire.com/it-security-software/scm/file-integrity-monitoring/
» https://support.microsoft.com/en-us/kb/841290/en-us
Incident Response 12
|
Enabling secure audit logging
when logging is enabled, answering the question of “What
happened?” after an incident becomes much easier
almost every operating system and many applications provide
significant logging capabilities
some hints to consider:
– default logging capabilities are often less than ideal some tweaking is
necessary
– log as much useful information as possible, but not too much…
– log messages to a file that only the administrator can access, or to a
secure, remote log host
Incident Response 13
|
Building up host defenses
actions taken to secure hosts
– reduce the exposure to security incidents
– make it easier to resolve incidents when they happen
cornerstones of host security:
– make sure that all operating system and application software are the most recent
» all patches, hot fixes, and updates are installed
– disable unnecessary services
» unnecessary services introduce unnecessary risk
– when faced with configuration choices, choose wisely
» many security exposures are introduced through sloppy system administration
for a complete discussion of secure host configuration choices
– refer to a book devoted to the subject
– or check the web for security configuration of a particular platform
» e.g., http://www.microsoft.com/security/pc-security/default.aspx
» e.g., http://www.debian.org/security/
» e.g., https://help.ubuntu.com/community/Security
» ...
Incident Response 14
|
Backing up critical data
regular, complete system backups may be
– a useful reference during incident response
» help to discover what was deleted and what was added
» help checking the times files and directories were last accessed, modified, or created
– indispensable for recovery from an incident
Unix/Linux backup tools
– common utilities include dump/restore, rsync, tar, dd, ...
» caution: some backup utilities reset the time of last access
» choose a backup solution that preserves all three time/date stamps (e.g., dump)
– there are many other backup tools available on the Internet...
Windows backup tools
– Windows has built-in backup solutions (Backup, File History, ...)
– there are many other backup tools available on the Internet...
backups are only as good as the original from which they were created
Incident Response 15
|
Educating users about host security
users play a critical role in an organization’s overall security
educating them improves security and incident response
users should know what types of actions they should and should not take on their systems– computer security perspective:
» users should follow the organization’s password policy
» users should not be allowed to install software on their computer– in particular, a web server or FTP server accessible from the Internet
» users should not be allowed to change configuration of their computer– e.g., to disable virus scanning or automatic updates
» users should be trained to identify phishing attempts
» users should not open attachements or URLs originating from unknown sources
» ...
– incident response perspective:» users should be trained to identify potential security incidents
» users should not take investigative actions
» rather, they should immediately notify a designated contact
Incident Response 16
|
Common network security actions
installing firewalls and intrusion detection systems
– rather than configuring firewalls and IDS devices to simply protect thenetwork, one should also configure them to log activities
using access control lists on routers
– allows certain types of traffic while prohibiting potentially dangerous traffic
encrypting network traffic
– encrypting network traffic enhances the security of any network
– one can use IPsec or other VPN solutions at the network layer, and TLS and SSH for applications
– caution: encrypting network traffic can also hinder the detection and the investigation of an incident
requiring authentication
– authenticating users can increase network security and provide an effective audit trail for incident response teams
Incident Response 17
|
Creating a network topology map
without information about the network topology, it is difficult to figure out which other systems may be affected by an incident
example:
– assume you identify a rough network sniffer on a compromised host
– the attacker might have obtained passwords by sniffing
– you need to know which other systems may have been compromised by the stolen passwords
network topology maps include details on – how the hosts are connected
– which networks use switches versus routers
– locations of external connectivity
even if all details are not available in a large network, one should have a detailed view of mission-critical networks such as the DMZ or Internet-facing e-commerce applications
topology maps can be created manually (e.g., in Visio) or automatically (as the output of some network scanning program)
information about the physical location of network devices and cables can alsobe very useful during the handling of an incident
Incident Response 18
|
Supporting network monitoring
network monitoring is often one of the first steps that one may
take when responding to an incident
make sure that the network architecture supports monitoring
– it should be possible to attach a network-monitoring platform to a
network device that has access to all network traffic on a given network
segment
– e.g., leave open ports on hubs and a spanning port on switches
network monitoring should be supported by appropriate
policies and procedures
Incident Response 19
|
Incident response policies and procedures
an incident response policy defines
– how potential incidents should be reported?
– who has the authority to approve response actions?
– when monitoring or investigation will be started?
– which type of incident warrants which type of monitoring or
investigation?
– when and how employees will be notified about investigative actions?
incident response procedures
– the policy states what you intend to do
– procedures are the implementation of the policy
– procedures entail much of technical details about how monitoring and
investigations will be carried out
Incident Response 20
|
Need for policies
policies save money and headaches– with proper policies, corporate investigators can advance the pace of an
investigation, and perhaps take investigative steps that usually require court orders when performed by law enforcement personnel
» performing a trap-and-trace of traffic (only headres) on corporate networks
» performing full-content monitoring on corporate networks
» searching and reviewing an employee’s machine
» coordinating with upstream sites (e.g., network service providers) involved in anincident
policies should be defined before an incident occurs, and employees should be informed about them– policies need to be advertised throughout the corporation and incorporated
into new employee orientation
– all current employees need to positively acknowledge their existence with a written signature
– a refresher overview course should be given when major changes are made to policies
Incident Response 21
|
Computer Security Incident Response Team
a team of designated individuals (employees of the
organization) with the appropriate technical, legal, and other
expertise necessary to resolve a computer security incident
may involve technical experts, security professionals, corporate
security officers, business managers, legal counsel, HR
personnel, helpdesk workers, ...
as incident response is not required at all times, the CSIRT can
be a dynamic team assembled when an organization requires
its capabilities
Incident Response 22
|
Creating a CSIRT
it is too late to establish the CSIRT after a computer security
incident has occured
– one cannot expect untrained and unprepared personnel to succeed
steps of CSIRT establishment
– define the team’s mission
– obtain top-level management support
» helps to involve team members from different departments
» helps to define and enforce policies
» important to get resources for the operation of the team
– select the team members
– train the team
» the team needs hands-on hacking and incident response training
» team building exercises may also be a good idea
Incident Response 23
|
Creating a response toolkit
includes the hardware, software, and documentation used during response
forensic hardware platform– should have high performance
» lot of disk space or external disk drives (for storing evidence duplicates)
» high-end processor and lot of memory (for analysis tasks)
– should be robust and configurable
– should have attachments for various external devices
– portable hardware duplicating tools may be useful
– cables, converters, adapters, plugs, labels and markers, digital camera, ...
response software– multiple operating systems on the forensics computer
– a forensic software package (e.g., EnCase, X-Ways, ...)
– other special tools (e.g., Volatility)
– all drivers for all the hardware of the forensics platform (computer and external devices)
– a selection of bootable USB pen drives and disks with different OSs
– disk-write blocking utilities
– a backup of the complete setup
Incident Response 24
|
Creating a response toolkit
network monitoring platform
– should be able to handle the amount of traffic the observed network has
» performant CPU, lot of memory and disk space
– network card should support promiscous mode
– network card should not respond to Address Resolution Protocol (ARP) packets and maintain network silence
– special network tap devices may be useful
documentation
– documentation is necessary for furtherdisciplinary, civil, or criminal action, as well as for a thorough response
– the CSIRT should document
» how the evidence is obtained
» all actions taken
» where and how the evidence is stored
– standardized report templates and forms are useful
– evidence tags and evidence labels
Incident Response 25
|
Summary on pre-incident preparation
proper prior preparation prevents poor performance during incident response
– preparation for system administrators involves configuring hosts and networks in a manner that reduces the risk of incidents and eases the task of resolving incidents
– preparation for the organization as a whole involves the establishment of a CSIRT and creation of a response toolkit from appropriate hardware and software tools, as well as templates that help documenting investigations
in the real world, pre-incident preparation is extremely difficult, both technically and ideologically
– few controls are in place to monitor user activities (due to basic human rights)
– logging is not enabled for performance reasons
– hosts do not have security software
– updates are not installed regularly
– no regular backup
– networks have too many entry points and configuration nightmares
– ...
Incident Response 26
INCIDENT INVESTIGATION
|
Introduction
the goal of this phase is to determine the who, what, when,
where, how, and why surrounding an incident
– however, determining the identity of the source of an incident may
sometimes be very difficult
– thus, many organizations choose to focus solely on what was damaged,
how it was damaged, and how to fix it
incident investigation can be divided into two phases:
– data collection
» gather all the relevant information needed to resolve the incident in a
manner that meets the response strategy
– forensic analysis
» examine all the data collected to determine the who, what, when, where,
and how
Incident Response 28
|
Incident investigation – data collection
generic requirements on data collection:
– the collection process itself may change relevant data and destroy evidence electronic data must be collected in a forensically sound manner
– collected data must be handled in a manner that protects its integrity (proper evidence handling)
types of data collected:
– host-based information
» live data collection
– volatile information obtained before the system is shut down
» forensic duplication
– provides a “mirror image” of the hard drive(s) of the investigated system, which can be used as a working copy for analysis purposes
– network-based information
» IDS, router, firewall logs
» full network traffic capture (may be initiated as part of the incident response)
– other information
» testimony and other information obtained from people (e.g., via interviews)
Incident Response 29
|
Incident investigation – forensics analysis
preparation of data
– create a working copy of all evidence data collected
– create a file list
– perform file signature analysis
– identify known good files (e.g., known OS components)
– recover deleted data and unallocated space
analysis of data
– extract e-mails and attachments
– review browser history
– review files in the file system
– identify and decrypt encrypted files
– search for relevant strings (in mails, attachments, browser history, files, deleted and unallocated parts of the hard disk, ...)
– review installed applications
– perform analysis of unkown programs
– review data collected during live response (e.g., recover OS structures from memory)
– review network-based evidence
Incident Response 30
|
Live data collection
objectives of live data collection:
– to confirm that there is an incident
– to retrieve the system’s volatile data that will no longer be available after the system is powered off
it is important that during the initial live response, one performsas few operations as possible
– in order to minimize alteration of the investigated system
but gather enough information for decision making in later steps
live data collection steps in general:
– creating a response toolkit (preparation)
– obtaining volatile data
– storing acquired information
– in-depth live analysis (if needed)
Incident Response 31
|
Creating a response toolkit
during live data collection, it is critical to use trusted commands, and make particular attention to not destroying or altering the evidence
collect and write a carefully selected set of tools to a USB stick
– collect tools that are compatible with the target system
– command line tools are preferred, because they do less ”behind the scene” interactions with the system
– check for dependencies and ensure that all DLLs and any other files theresponse tools depend on are written to the stick
– create a checksum for all the files on the stick and store them separately
– preferably, use a USB stick that has a physical write protect switch
» e.g., Kenguru FlashTrust or Kenguru Defender
label the response stick
– put case number, time and date, name of the investigator
– indicate whether or not the response stick contains output files or evidence from the victim system
Incident Response 32
|
Creating a response toolkit
at a minimum, you may want to collect the following volatile
data prior to forensic duplication:
– system date and time
– a list of the users who are currently logged on
– a list of currently running processes
– a list of past and current network connections
– the applications using those network connections
– a list of the systems that have or recently had connections to the system
– time/date stamps for the entire file system
Incident Response 33
|
Forensic duplication
forensic duplication is the process of creating copies of hard disks and other storage media for the purpose of analysis
– also called acquisition or (forensic) imaging
important general requirements:
– the duplication process must not alter the original evidence
– the duplicate should be an accurate representation of the original evidence
types of duplicates:
– forensic duplicate
– qualified forensic duplicate
– restored image
– mirror image
– logical copy
Incident Response 34
|
Forensically imaging a drive
any time when the original medium is accessed, take precautions to avoid writing to it
be aware that the following actions may modify the suspect drive:
– booting up the forensic acqusition system in Windows with the suspect drive attached to it
– plugging in the suspect drive with a USB/FireWire hard drive enclosure in Windows
– mounting the suspect drive in read/write mode in Linux
– booting up the suspect system in Windows with the drive still in it
– choosing the wrong drive to write to when performing duplication
if you create a mirror image, then wipe the image drive before using it
– wiping means overwriting every sector on the drive with some data
– this will remove old data from previous cases
– you can use dd or any forensics software package for wiping
» e.g., dd if=/dev/random of=/dev/<image drive>
Incident Response 35
|
Analysis of forensic duplicates
analysis of forensic duplicates of storage media can be
performed at two levels:
– disk level (dealing with partitions, clusters, and meta-data)
» usually requires special tools (e.g., DFF) which can open raw images and
qualified forensic duplicates (e.g., EWF or AFF files)
» allows for recovering deleted files
» allows for recovering unallocated and slack spaces
– file system level
» requires restoring the file system from the forensic duplicate
» in Linux, it is easy to mount a raw image
» qualified forensic duplicates need special mount programs (e.g., affuse) or
they need to be converted to a raw image
» once the file system is mounted, analysis can use standard Linux commands
and special tools for file system level analysis
Incident Response 36
|
Generating file lists
browsing through the entire file system manually is not feasible
creating informative file listings can be of great value, where each item on the list contains:– full path of the file
– MAC timestamps of the file
– logical size of the file
– an MD5 hash of the file
the file list should be in a format that can be imported into other tools for further analysis and correlation– spread sheet or database applications
– other analysis tools
MD5 hashes can be used – to filter out known good system files (and thus reduce the number of
remaining files that may need to be reviewed)
– to identify known bad files (e.g., based on some indicator of compromise information obtained by threat intelligence)
Incident Response 37
|
Reviewing relevant files
one can identify potentially interesting files in the file system
– by file type
» determine file type by extension, as well as by true file headers (signature)
» typically interesting file types: .doc(x), .xls(x), ..., .txt, .wpd, ..., .gif, .jpg, .png, ..., .pdf, .tmp, .log, .exe, .dll, ...
» you can limit the possible file types to those relevant to the case you are investigating
– by determining the timeframe in which the incident occurred, and scrutinizing those files created, modified, or accessed during this timeframe
» most forensics software packages offer possibility to filter files by their access times
» use standard Linux find command
– by performing a keyword search on the entire file system...
find /usr/tom –ctime -24
Incident Response 38
|
Performing keyword searches
string searches can be conducted on the logical file system or at the physical level to examine the contents of an entire drive
– a potential difficulty is that files may be compressed or encrypted
– some applications store their data in binary format
– so these files must be first uncompressed / decrypted / converted
useful keywords are case specific
examples:
– marijuana, dope, ...
– girls, teens, sex, ...
– ...
tools:
– use special programs such as Findwild
– use standard Linux grep command
Incident Response 39
|
Recovering deleted files
one of the most common tasks requested in any investigation is to
find and recover the files that have been deleted from the system
fortunately, most forensic analysis tools allow us to find, recover, and
examine many deleted items on a system
– when file systems are designed, different requirements are considered
– for end-user systems, the top two priorities are usually speed and
throughput
– so when a deletion takes place, data is not really wiped (that would take
time), but only meta-data on disk is modified to mark the clusters used by
the deleted file as unused (this is much faster)
– analysis tools rely on this and look for remains of files in unallocated disk
space
Incident Response 40
|
Summary on incident investigation
the goal of incident investigation is to determine the who, what, when, where, how, and why surrounding an incident
it has two phases: data collection and forensic analasys
types of data collected:– host-based information
» live data collection
» forensic duplication
– network-based information
– other information
data collection requires appropriate tools and careful procedures– live data collector toolkit
– forensic duplication
analysis of forensic duplicates can be carried out at two levels:disk level and file system level
forensic analysis can be based on standard (e.g., grep comand) and special tools (e.g., DFF) for searching disk or file system content,recovering deleted files, review and analyze logs, ...
Incident Response 41
INCIDENT RESPONSE IN ICS/SCADA
|
General challenges in ICS/SCADA
Impact of an incident may be cross-domain
Availability and safety concerns
Legacy equipment
Proprietary systems and protocols
Lack of ICS specific security incident response tools
Incident Response 43
|
Containment challenges
Containment of the compromise may require fast system
reconfiguration
– Network segmentation, firewall reconfiguration
– Disconnection of infected hosts/devices, disabling services
System reconfigurations in ICS/SCADA
– may not be allowed at all
– should preserve safety
– should have no adverse effect on the controlled processes
Incident Response 44
|
Recovery challenges
System recovery should be possible from backups, golden
images, and known good configurations (if available)
However, system recovery
– may require stopping and re-starting physical processes
– may take a considerable amount of time
– should be carefully scheduled
– should take into account interdependence between
different sub-systems
» requirements on the order in which sub-systems can be re-started
» requirements on consistent global state of the entire system
Incident Response 45
|
Data collection challenges
In ICS/SCADA systems, logging focuses mostly on process monitoring and troubleshooting, not on incident response
– Legacy equipment may not create any useful logs
– Field devices (e.g., sensors) typically don’t create any logs
– Network traffic is typically not logged
– PLCs may have proprietary or non-standard log format
– IT equipment may be modified in order to serve the system in the most efficient way --» logging may be switched off
– Logging may have adverse effects on system performance (availability, real-time guarantees)
– Logging would need storage capacity in PLCs and field devices
– Network based log data gathering increases network load
– Gathering log information manually may be slow and difficult
Incident Response 46
|
Ingredients of a solution
Up-to-date asset inventory, network topology, dependency
graph --» makes reconfiguration easier
Backups, golden images, proper configuration management --»
makes system recovery easier
Passive network monitoring --» makes intrusion detection and
data collection possible even in legacy systems
Physically separated management network for log gathering
and orchestration of security controls --» to avoid interference
with the system and increased network load
Incident Response 47