EFFECTIVE AND EFFICIENT MALWARE DETECTION

AT THE END HOST

Presentation by Clark Wachsmuth

C. Kolbitsch, P. M. Comparetti, C. Kreugel, E. Kirda, X. Zhou and X. Wang

or: How I Learned to Stop Worrying and Love

the Malware-infested Internet

2

THE PROBLEM

Malware! Ineffective (and/or inefficient) detection

models Can be evaded by fairly simple means by

malware authors, such as using polymorphism, obfuscation or system call reordering

Resource-heavy detectors may be effective, but not efficient enough for average consumer computer

3

PAST IMPLEMENTATIONS

Network-based detection Pros:

Useful for detecting some network malware Modern malware is heavily network-bound

Cons: For network-based malware only Content sniffers thwarted by encrypted data Blending attacks make malicious data match

normal data signatures

4

PAST IMPLEMENTATIONS

Host-based detection Pros:

Has the resources to see complete work of malware programs – not limited to specific host resource

Some pre-emptive strategies Cons:

Code obfuscation and polymorphism can easily bypass methods such as byte signatures while keeping the same functionality

System call sequence based detection, again, easily bypassed by reordering calls or making unused calls

5

PAST IMPLEMENTATIONS

Static analysis-based detection Pros:

More effective due to focus on malware behavior, thus less stymied by obfuscation and polymorphism

Cons: Method itself is difficult to employ Has its own vulnerabilities such as detecting

metamorphic code and runtime packaging Takes a heavy toll on system resources making

it unreasonable for home computer systems

6

PAST IMPLEMENTATIONS

Dynamic analysis-based detection Pros:

Less rigid focus on malware behavior allowing for a more general and broad way of detecting malware

Cons: Can require special hardware for detection

(data tainting) Large associated overhead making it unusable

in the home computer realm

7

THE SOLUTION

Effective and Efficient malware detection, duh! But how?

Effective: Can’t be duped by simple order-changing,

rearranging schemes Doesn’t rely only on known quantities; can detect

unknown running programs No false positives

Efficient: Not incurring a very significant chunk of system

resource overhead

8

THE PLAN

In a sandboxed environment, observe different malware and develop fine-grained models

Efficiently match these models up with the run-time behavior of an unknown program

If a match is found, terminate and eliminate

9

BUT HOW?!

By creating a behavior graph where each node is an “interesting” system call

The nodes store a symbolic expression (simple node) or a program “slice” (complex node) that can calculate the output of the system call

These expressions/slices used to detect if output is the argument of another interesting system call during runtime If found, an edge is created between the

two nodes

10

THE CONTROLLED ENVIRONMENT Uses Anubis (Analyzing Unknown Binaries)

Disassembles instructions (including system calls) and keeps an instruction log

Keeps memory log for instructions that read from memory, where (in memory) the instruction reads and writes

Each bite tainted to detect data dependencies between system calls

Any labels within a branch operation are labeled with the taint of the controlling instruction for control dependencies

11

THE INITIAL BEHAVIOR GRAPH

With all the instructions labeled, an initial graph is creating placing on it all system calls (as nodes)

Edges are created when a dependency is found Using the logs, a recursive backwards trace of

system call arguments is made to determine how the argument’s bits were created

These instructions are gathered into a program slice until either an instruction that can’t be traced further (from the outside) or a value produced by an immediate operand from an instruction or coming from the initialized data segment

12

PROGRAM SLICES FUNCTIONS With the slice, we know how and who

created the argument of the sys call It’s not necessarily the direct program

code, though (unrolled loops won’t match with different sizes)

Each line in binary that appears at least once in slice is marked and appropriate code copied to function. Non-marked lines become nops.

Stack needs fixing because stack creating code often not part of slice (uses instruction log)

13

SIMPLIFYING FUNCTIONS

Yay! We have a function that gives an expected output for a given input

Some functions can be quite long and fairly basic We can optimize it to a smaller symbolic

expression This optimization can have huge overhead

reduction at the end host Other functions aren’t so basic, so we

retain the program code of the function rather than have a symbolic reduction

14

SCANNING END HOST

Scanner monitors running program for sys calls Has admin privileges running is user-mode Assume programs can’t get to kernel

All nodes inactive in initial behavior graph When a system call is made, the scanner checks

graph for inactive nodes of the same type and sees if parent nodes are active If found, checks all arguments from sys calls for simple

functions; defers complex functions for later but allows complex function to hold

If all simple function arguments hold, node becomes active

15

SCANNING END HOST

When do we check the complex functions? When we reach an interesting node

Interesting if it is a security-relevant system call (writing to file system, network or registry, starting new processes)

Also interesting if node has no outgoing edges If complex function holds, the interesting

node is confirmed Otherwise, the node with the complex function

becomes inactive and any subgraph rooted under it becomes inactive as well as the edge being formed

16

MATCHING MALWARE

If an interesting node is confirmed, then the program is matched as malware However, if there is no complex function

dependency, then the graph created is not used to help detect future malware programs

The subgraph created with the interesting node is also a behavior graph that denotes a trait of the particular malware running

17

DETECTION EFFECTIVENESS

Generated behavior graphs for six popular malware families (Table 1)

100 samples of each family wereSelected from the database and the non-

interesting samples were tossed out

50 random samples chosen from remaining bunch to create behavior graphs and train dataset Not all samples could be detected due to

non-interesting behavior and complex function crashes

18

TESTING DATA

19

IS IT EFFECTIVE?

Some were effective and some weren’t so much AV software notoriously bad at classifying malware Confirmed by manual inspection, especially for Agent Restricting samples to 155 known variants yielded

92% effectiveness Also restricted data samples to 108 unknown variants

and still achieved 23% effectiveness, indicating that this method can even detect some unknown variants

This behavior-based method is more general than an AV scanner, therefore requires less graphs than signatures

20

WHAT ABOUT FALSE POSITIVES? Tested on WinXP using IE, Firefox,

Thunderbird, putty and Notepad Yielded no false positives When complex functions were unchecked

and allowed to hold, all of the above yielded false positives

Therefore, system call dependencies are at the root of this method’s success

21

OK, BUT IS IT EFFICIENT?

System setup for testing: WinXP, single-core 1.8Ghz P4 with 1GB RAM Tested using 7-Zip, IE, Visual Studio

22

UMM, DID THAT SAY 40%?

CPU / I/O-Bound tests showed low overhead

Compiling seems quite high at 40% System calls in compiling 5000/sec

compared to 7-zip’s 700/sec Compilation is worst-case scenario Improved symbolic execution engine could

possibly reduce high complex function evaluation of 16.7%

Still performed well for common tasks

23

LIMITATIONS

Authors could use time-triggered behavior or command and control mechanisms to prevent malware behavior during test

A reactive method that only works on running malware But, new graphs can be employed quickly and it can

detect some unknown variants Authors could change algorithms rendering

program slices unusable Changing algorithms is a lot of work and this

method still raises the bar considerably higher for malware authors

24

TECHNICAL CONTRIBUTIONS Developed effective models with

detailed semantic information about the malware family

Created a scanner that efficiently matches the behavior of an unknown, running program against the models by tracking system call dependencies

Experimental evidence that approach is feasible and usable in practice

25

CONCLUSION

Effective? Check With correctly labeled, known variants, a

92% effectiveness was obtained with no false positives

Efficient? Check While compiling was a worst-case scenario,

tasks common to the average end user incurred only a low overhead