Efficient Detection of Split Efficient Detection of Split Personalities in MalwarePersonalities in Malware

Davide Balzarotti, Marco Cova, Christoph Karlberger, Christopher Kruegel, Engin Kirda

and Giovanni Vigna NDSS 2011 Feb.

1

OUTLINEOUTLINEIntroduction and Related WorkOur ApproachImplementationEvaluationConclusion

2

IntroductionIntroductionMalware detection

◦ Static◦ Dynamic

Sandboxes(Anubis, CWSandbox, Joebox, Norman Sandbox)

Counterattack◦ Attacks on Virtual Machine Emulators◦ CPU semantics, timing attacks◦ Environment attacks

Processes, drivers, or registry values

3

SoluSolu11:Transparent malware :Transparent malware analysisanalysisCobra

◦ Code blocks◦ Replace instruction with a safe version

Ether◦ Hardware virtualization

More difficult to detect by malicious code.Great, but slow.

4

Solu2:Detect different Solu2:Detect different behavesbehaves“Emulating Emulation-Resistant

Malware”, 2009◦ Reference system vs. emulated

environment◦ Compare execution path◦ Use Ether to produce the reference trace

But executing the same program twice can lead to different execution runs.

5

OUTLINEOUTLINEIntroduction and Related WorkOur ApproachImplementationEvaluationConclusion

6

Our approachOur approachRecording and Replaying

◦ Reference system vs. emulated environment

◦ system call trace: types and arguments

If there is a different behavior◦ Rerun it in a transparent framework(Ether)

Detect malware reliably and efficiently

7

Reliability Two systems are execution-

equivalence if all program that◦ Start from the same initial state◦ Same inputs on both systems => Same runtime behavior => Same sequence of the system calls?

Assume no race condition

8

Reliability(cont.)If our reference system and the

analysis system are execution-equivalence, any difference in the observed behavior => split-personality

Also, this discrepancy is the result of CPU semantics or timing attacks

9

Making Systems Making Systems Execution-Execution-EquivalenceEquivalenceSame OS environment

Same address space layout of a process at load time

Same inputs to a program◦ Run program on the reference system in

log mode◦ Run program on the analysis system in

replay mode◦ System call matching

10

ReplayReplay ProblemProblemA number of system calls are not safe

to replay◦ Allocating memory, spawning threads

Only replay for those system calls that read data from the environment◦ other system calls are passed directly to the

underlying OS

Delay cause additional system calls◦ WaitForSingleObject()

11

System Call MatchingSystem Call Matching

12

OUTLINEOUTLINEIntroduction and Related WorkOur ApproachImplementationEvaluationConclusion

13

ImplementationImplementationA kernel driver

◦ Trap all the system calls Hook “System Service Descriptor Table”

◦ Each system call, two handler, log and replay

A user-space application◦ Start and control the driver◦ Start the process that has to be analyzed◦ Store the data generated during the

logging phase

14

Practical aspectsPractical aspectsHandles consistency

◦ Live handles and replayed handles◦ Check a list of all replayed handles

Networking◦ NtDeviceIOControlFile()◦ Device-dependent parameters

15

Practical aspects(cont.)Practical aspects(cont.)Deferred results

◦ STATUS_PENDING◦ NtWaitForSingleObject()

Thread Management◦ NtCreateThread()◦ Each thread has a new log

16

LimitationsLimitationsMemory Mapped Files

◦ DLLs◦ Create file with memory-mapped

Remove the system calls

Multiple processesRandom numbers

◦ KsecDDInter-process communication and

asynchronous callsPostponing check

17

OUTLINEOUTLINEIntroduction and Related WorkOur ApproachImplementation EvaluationConclusion

18

EvaluationEvaluationMicrosoft Windows XP Service Pack 3VMware virtual machineAnubis system(Qemu)

1. Log and Replay six programs(success)2. SDBot(fail)

◦ spawning new process, like NtCreateProcess◦ six different versions to detect

VMware(success) Red Pill, Scoopy, VMDetect, and SourPill

19

Evaluation(cont.)Evaluation(cont.)3. Real Malware with no VM-checks

20

Evaluation(cont.)Evaluation(cont.)4. Real malware with VM-checks

21

PerformancePerformanceDepends on the type of operation

◦ Average 1% overhead

Compresses a 1KB-long random fileCMD: 7za.exe a test.zip 1KB_rand_file

Anubis: 4.267 secEther: 77.325 secOur Vmware reference system: 1.640

sec22

OUTLINEOUTLINEIntroduction and Related WorkOur ApproachImplementation EvaluationConclusion

23

ConclusionConclusionA prototypeRecording system calls and replay

themNeed a fully transparent, analysis

system for further examination

24