Malware Detection based on Application Behavior Modeling

Mrs P.R.Lakshmi EswariC-DAC, Hyderabad

NWMTD’11Jun 20–21, 2011

Evolution of Malware Attacks

Era Who is Leading

Initially Hobbyists

Late 90s Criminals

Early of this decade

Terrorists (more dangerous criminals)

Now Spies

Malware Definition (Wikipedia)

• A software which is designed to infiltrate a computer system without the owner’s informed consent

• Refers to a variety of forms of hostile, intrusive, annoying software code

• MALicious softWARE

Threat from the Malware

• A code – which collects the credit card number or any

other personal info– Which makes an application do the buffer

overflow and crash– Loosing the private and sensitive information– which shows annoying advertisements without

your consent– Which encrypts the data and asks for money to

decrypt it

Malware Categories

Category

Description

Virus Attaches itself to a file (preferably binary)

Trojan Look as if useful program but invites attacker

Worm Same as virus but got the capability of spreading at its own

Exploit Malware code which exploits a vulnerability in the app

Root-kit To hide the actual malware from system information

Spyware Spies on the user habits and data and sends it

Phishing A website made to mimic an existing website

Spam Sending unwanted emails

Bots Code in command and control network to launch DDOS Attacks and other malicious operations

A Typical Malware

Exploit Logic

Motivational Logic• Spam• Data theft• Ransom• Disrupt the routine

Protection Logic• Packing• Anti Debugging• Anti Virtualization

Propagation Logic

Mails

USBs

Attacks - Classified

• Untargeted attacks– Attacking websites– Infecting portable storage devices– Attacking social networking websites– Wild malware (worms etc)

• Botnets• Targeted Attacks

Targeted Attacks

A Typical Attack

Doc file

Originally a executable

Opens the file,and executes the malware

Malware

Changes the windows update program

Whenever updates windows, also downloads the malware, sends the data out etc.

Botnet

IRC Server

Victim

1. Exploit / Attack2. Download malware (bot)

3. Join a channel on IRC

4. Attacker will also join this channel (preferably through a program) and issue commands (for e.g. update)

Receives the command (update)

Botnet

• DDoS (distributed denial of service attacks)

• Collecting lot of bank related data

• Spidering attacks (on websites)

• Spams

• Using victim for other sensitive attack

• Shutdown the computer etc

Motivation and Business

Motivation and Business

14

Vulnerability, Exploit and Race

Vulnerability, Exploit and Race

Malware Detection Techniques

• Black listing– Anti Virus– Intrusion Detection System– Behavior Based Malware Detection

• White listing– Specification Based Detection– Anomaly Detection

Commercial Solutions

Desktop security software (major anti malware products)

Behavior based anti malware solutions

AVGAvirMcAfeeNortonF-SecureESETBit DefenderZone AlarmTrend MicroSunbelt

Sana Security Primary ResponseMalware DefenderMamutuMalware Resist[C-DAC Hyderabad]Nova ShieldPC Tools - Threat Fire

End System Security Suites

• Centralized configuration on all clients• Centrally controlled

– Firewall– Encryption– Device Control– Anti Malware– Security policies

White listing Solutions

• Core Trace Bouncer• Bit9 Parity• Robot Genius• Microsoft App Locker• McAfee Application Control

Don’t want to pay ? !

• Free Anti Virus [AVG, AVIRA, AVAST]• Free Firewall [Zone Alarm]• URL Scanner [AVG, WOT, RG Guard]• Trend Micro Web Protection Add on• Disable Auto runs• Returnil Virtual System / Windows Steady State• Wehn-Trust HIPS [MUST for Windows XP – ASLR

Tool]• Win-pooch HIPS [Windows XP]• OSSEC HIDS• WinPatrol [BillP Studios]

How anti malware works?

Basic Activity Scanning *

Behavior Based Engine(On Process Activities)

Anti Virus Scanning(On file content)

White listing(On process

creation)

Behaviors database

Malware Signature database

Known Applications

database

( * Process activity, file read or write )

Malware Prevention System (MPS)

MPS - Approach

• Each application makes sequence of system calls for accessing various OS resources through multiple control paths (normal behaviour)

• When the application is infected with malware, its behaviour changes

User Process

1

User Process

2

User Process

n

……………..

User Space

Kernel SpaceSystem CallsOperating System

Detects malicious activity before it causes damage to end system i.e. before the system calls are executed by the operating system

MPS - Architecture

Flowchart

Protection against overall threats - Process Execution Control Model

Enforcement Module

4. Client

1. Application Profiling and Model Generation Process in a Sandbox

2. Server Manages the models and admin can set the policies here

3. Based on the policies the model gets pushed to clients

Server communication module

Malware Prevention System

27

Optimization of the representation of the profiled dataConsiders the system calls that are made on to a resource

Resource specific clustering ModelEach cluster can be defined as a 2-tuple<R,S>

–R : Resource–S : System call

Example:–Cluster 1- <A, {1,2,4}> –Cluster 2- <B, {1,3,4,2}> –Cluster 3- <C, {1,2,4}>

It is platform independent implementation

Resource - ASystem calls :

{1,2,4}

Resource - B System calls:

{1,3,4,2}

Resource - CSystem calls:

{1,2,4}

Model Generation

Operations Hooked in MPS

File System Calls Process hooks Network Calls Registry Calls

Deployment Scenario

System Architecture

Database Structure @ Server

Database Structure @ Client

Index File @ Server

Update Request

UPDATE_REQUESTUPDATE_REQUEST

UPDATE_RESPONSEUPDATE_RESPONSE

MPS ClientMPS Server

Major No,Minor No,OS type,

ModelUpdate,Db Major No,Db Minor No No.of Model Files,

Model File names,ModelFile Path

File Transfer Request

TRANSFER_REQUESTTRANSFER_REQUEST

TRANSFER_RESPONSETRANSFER_RESPONSE

MPS ClientMPS Server

Model File Name with

path

Contents of the Model File

Log Message Request

Application name,OS type,

Date,IP,

Operation,Path

Success or Fail

Client and Server – Technologies used

Server on Linux– Apache Server 2.2– Virtual Machine– Windows XP, Vista and 7 images– Linux 2.6.23 kernel image– Java runtime environment– PHP– HTTP message format– XML, OpenSSL

Windows Client– Mini Filter Driver– Call out Drivers– Win32 programming– C, C++ programming– PE Executable format Open SSL

Linux Client– Linux Security Modules– C, C++ programming– Qt Programming– OpenSSL

Server GUI

Client GUI

Malicious Pdf

• Creation of Axsle.dll• Creation of Icucnv34.dll• Write file on cvs.exe• The malware repeatedly tries to write

cvs.exe file and it gets blocked. The document doesn’t open until the write file operation on cvs.exe is completed.

Malicious Pdf

• Behaviors Detected– Hides view of system files– Hidden image file– File has system attribute– Creates logon entry– Unsigned binary– Drops executable– Modifies internet settings– Spawns process

Stuxnet

Stuxnet

Stuxnet

ATT27390 doc file

• Activities blocked– Dropping of zipfldr.dll in system32 folder– Dropping of wuaueng.dll in system32 folder

• MPS is compared with similar best commercial tools available in the market like NovaShield, Mamutu, Malware Defender, Sana Security Primary Response, Safe Connect, Threat fire etc.

Field Testing Report

properties claimed for Malware Prevention System assurance level (mark to 5) Remarks

protects from the malware before they do any harm to your system.

3.75

is a very effective and low cost anti malware solution 4

has the capability to detect unknown malware. 4

is able to detect malware using its unique heuristic technology to detect malicious behaviors.

3.5

Database can be expanded and we can update you with new malicious behaviors.

-Not checked-

is easy to use. Even if your antivirus hasn’t detected a malware, you can quarantine a process

4

Enforcement model applied 3

False positive generation 5

It doesn’t use any sort of malware signature database. 5

• MPS is found sensitive against blended MS office and PDF documents wherein the MPS solution alone identified the malicious activity as the other industry product remain silent

• Application has a tendency to raise false alarm against benign documents as it might match the enforcement policies defined

• Overall it is felt that the solution is detecting high level targeted malware behaviours, but there is a need to improve the capabilities by suppressing the false alarms.

Field Testing Report

Detection Based on Runtime Behaviour. All running programs are monitored for a set of critical behaviors that could affect the normal functioning

Malware ResistSimplifying and Strengthening Security

Salient Features

Detection Based on Runtime Behavior

Small memory footprint and high detection rate

Co-exists with Anti Virus Solutions

Low False Positive Rate

Easy to Deploy and Use

Behavior modeling of application

Verification of application against critical resource access

Process Execution Control

Enforcing the model at run time

Guard from application exploits and implicit malicious activity

Fine grained monitoring of file, process, network and registry access

Co-existence with other antivirus solutions

Malware Prevention System (MPS)

Ongoing Research @ C-DAC Hyderabad

Design and Development of Anti Malware Solution for Web Applications and Mobiles

Malware Analysis

The approach to analyze the Malware

• Run the malware in isolated lab

• Monitor network and system connections

• Understand the program’s code

• Repeat until satisfied with gathered info

How to?

• Manual– Dedicated system (ready to be

compromised)– Virtualized System

• Automated Analysis

Anubis [analyzing unknown binaries]• http://anubis.iseclab.org/

Virus total [analyze suspicious file]• http://www.virustotal.com/

Bit-Blaze [Malware Analysis Service]• https://aerie.cs.berkeley.edu/

Norman Sandbox

Joe Box Sandbox

Sunbelt CWSandBox

Comodo [Comodo Instant Malware Analysis]• http://camas.comodo.com/

Automated Analysis

Two Steps / Phases

• Behavioral (Dynamic) Analysis

• Code (Static) Analysis

• Gather as much as from behavioral analysis

• Fill the gaps from the code analysis

Analysis

Malware Analysis

• To analyze malware, we requires basic and advanced knowledge in Windows and Linux concepts (depends)

• For example: while doing behavioral analysis of the malware, we find malware modifies file A. – To get more out of it, we must know what is the significance of file A

Prepare the System

• Use VMWare and use the snapshot feature to restore state after malware execution

• Use Virtual PC – execute the malware – Close and Delete changes

• Physical System State Restore– Returnil Virtual System– Windows Steady State

Behavioral Analysis

• Activate various monitoring tools

• Execute the malware

• Terminate / suspend the malware process– Sometimes malware process comes again and

again

• Observe the results of monitoring tools

Process Explorer

• Free from Microsoft TechNet• Super Task Manager• Shows process tree

– We can know if malware created the new processes

• Also shows files which a process is using• Can see the strings also

Process Monitor

• Free from Microsoft TechNet• Monitors the following activities

– Process creation – File related– Registry– Network related

• Captures for all the process– Best is to do it for all and then apply the filters

Regshot

Using IDAPro

• Can reveal a lot of information

• Great tool if user can reverse the C/C++ code

Use OllyDbg

• OllyDbg is a great debugger

• Open the sample using OllyDbg

Snort

• Either use snort in a separate virtual machine to monitor its network activity

• Or use tools like wire shark• Find

– IRC server to whom this sample connects– Web servers?

• May notice DNS queries

Packed Malicious Executables

• Packers compress / encrypt the executable• This is used

– Difficult to analyze– Smaller size on hard disk

• However runs unpacked and original in memory

How it executes?

Executable Decryptor

Packed program

stored as data

Unpacked program in

memory

Small Decryptor extracts the packed code and executes the code

PE Format

IMAGE_DOS_HEADER

MS-DOS Stub Program

IMAGE_NT_HEADERS

Signature

IMAGE_FILE_HEADER

IMAGE_OPTIONAL_HEADER

IMAGE_SECTION_HEADER

IMAGE_SECTION_HEADER

SECTION

SECTION

If it is packed

IMAGE_DOS_HEADER

MS-DOS Stub Program

IMAGE_NT_HEADERS

IMAGE_SECTION_HEADER

IMAGE_SECTION_HEADER

SECTION

SECTION

IMAGE_DOS_HEADER

MS-DOS Stub Program

IMAGE_NT_HEADERS

IMAGE_SECTION_HEADER

IMAGE_SECTION_HEADER

SECTION

SECTION

This is Decryptor

code

Original PE

Packers Availiable

• UPX• ASPack• Themida• Petite• VMProtect

PEiD

Process dumping with LordPE

• LordPE shows all the processes and can dump there images from memory

• We can run the process from packed executable – Anyways it has to unpack itself in the memory

• We can dump from memory using LordPE

Thank You