From Prey To Hunter∗

Transforming Legacy Embedded Devices Into Exploitation Sensor Grids

Ang CuiDepartment of Computer

ScienceColumbia University

New York NY, 10027, USAang@columbia.edu

Jatin KatariaDepartment of Computer

ScienceColumbia University

New York NY, 10027, USAjk3319@columbia.edu

Salvatore J. StofoDepartment of Computer

ScienceColumbia University

New York NY, 10027, USAsal@columbia.edu

ABSTRACTOur global communication infrastructures are powered bylarge numbers of legacy embedded devices. Recent advancesin offensive technologies targeting embedded systems haveshown that the stealthy exploitation of high-value embeddeddevices such as router and firewalls is indeed feasible. How-ever, little to no host-based defensive technology is availableto monitor and protect these devices, leaving large numbersof critical devices defenseless against exploitation. We de-vised a method of augmenting legacy embedded devices, likeCisco routers, with host-based defenses in order to create astealthy, embedded sensor-grid capable of monitoring andcapturing real-world attacks against the devices which con-stitute the bulk of the Internet substrate. Using a softwaremechanism which we call the Symbiote, a white-list basedcode modification detector is automatically injected in situinto Cisco IOS, producing a fully functional router firmwarecapable of detecting and capturing successful attacks againstitself for analysis. Using the Symbiote-protected router asthe main component, we designed a sensor system which re-quires no modification to existing hardware, fully preservesthe functionality of the original firmware, and detects unau-thorized modification of memory within 450 ms. We believethat it is feasible to use the techniques described in thispaper to inject monitoring and defensive capability into ex-isting routers to create an early attack warning system toprotect the Internet substrate.

1. INTRODUCTIONThe Internet is a dynamically changing network of many

different kinds of devices, predominantly general purposehosts and servers connected by a large collection of special-ized embedded devices. Embedded devices such as routers,

∗Please note that Figures 1 and 2, along with portions ofSection 5 is taken from a companion paper [9], and arepresent here so that our exposition has the appropriate back-ground and completeness.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ACSAC ’10 Dec. 6-10, 2010, Austin, Texas USACopyright 2010 ACM 978-1-4503-0133-6/10/12 ...$10.00.

switches and firewalls constitutes the Internet’s communi-cation substrate. Devices such as VoIP, IPTV, power man-agement and physical access control units provide a myriadof other specialized services. Most host-based security tech-nologies deployed today are designed primarily to protectgeneral purpose servers and hosts, leaving vast numbers ofembedded devices, the Internet substrate itself, undefendedagainst exploitation.

We present a new embedded device defense system de-signed make the internet substrate a safer environment. Webelieve it is technically feasible to inject security functional-ity in situ into legacy embedded systems to:

1. Provide security features to protect these devices againstexploitation and rootkitting.

2. Create a large scale sensor grid providing new detec-tion capability to identify attacks against embeddeddevices that are currently unmonitored.

Recent studies suggest that large populations of vulnera-ble embedded devices on the Internet are ripe for exploita-tion [8]. However, examples of successful exploits againstsuch devices are rarely observed in the wild, despite theavailability of proof-of-concept malware, known vulnerabili-ties and high monetization potential. We posit that our in-ability to monitor embedded devices for malware installationis a factor in this phenomenon. When deployed through-out the Internet substrate, the sensor system discussed inthis paper will provide visibility into black-box embeddeddevices, allowing us to capture and analyze exploitation ofembedded devices in real-time.

As a first step to show feasibility, we demonstrate a generalmethod of transforming existing legacy embedded devicesinto exploitation detection sensors. We use Cisco firmwareand hardware as the main demonstrative platform in thispaper. However, the techniques described are not specificto any particular operating system or vendor, and can bedirectly applied to many other types of embedded devices.

In order to detect and capture successful attacks againstCisco routers for analysis, we engineered a system which au-tomatically injects generic whitelist-based anti-rootkit func-tionality into standard IOS firmwares. Once injected, theaugmented router firmware can be loaded onto physical Ciscorouters, essentially transforming such devices into highly in-teractive router honeypots. As Section 8 shows, the result-ing devices are fully functional, and can be deployed intoproduction environments.

The main challenge of creating an embedded device hon-eypot rests with the difficulties of injecting arbitrary detec-tion code into proprietary, close-source, embedded deviceswith complex and undocumented operating systems. In or-der to overcome this challenge, we’ve created a software con-structed called the Symbiote [9]. As Section 5 illustrates, theSymbiote, along with its payload, is injected in situ into anarbitrary host binary, in this case, Cisco IOS. The injectionis achieved through a generic process which is agnostic tothe operating environment of the host program. Figure 1shows how a Symbiote is typically injected into a host pro-gram. In general, Symbiotes can inject arbitrary host-baseddefenses into black-box embedded device firmwares. For afull discussion of Symbiotes, please see [9]

The unique capabilities of the Symbiote construct allowsus to overcome the complexities of injecting generic exploita-tion detection code into what is essentially an unknownblack-box device. The original functionality of resultingSymbiote-injected embedded device firmware remains un-changed. A portion of the router’s computational resourcesis diverted to a proof of concept Symbiote payload, whichcontinuously monitors for unauthorized modifications to anystatic areas within the router’s memory address space, a keyside-effect of rootkit installation. As we demonstrate in Sec-tion 9, the portion of the CPU diverted to the Symbiote’spayload is a configurable parameter, and directly effects theperformance of the Symbiote payload; in this case, the de-tection latency of any unauthorized modification.

A monitoring system is constructed around the main com-ponent of our system, the Symbiote-injected IOS image. TheSymbiote within the IOS firmware simultaneously performschecksums on all protected regions of the router’s memorywhile periodically communicating with an external monitorvia a covert channel. In the event of an unauthorized mem-ory modification within the router, the Symbiote will raisean alarm to the external monitor, which then triggers thecapture and analysis component of our system.

As Section 8 discusses, our monitoring system can be de-ployed in one of three ways; native deployment, emulateddeployment, and shadow deployment. Due to the uniquelimitations of each deployment scenario, the capture andanalysis mechanisms differ slightly. For example, when theSymbiote-injected firmware image is loaded into a physi-cal Cisco router (native deployment), IOS’s own core dumpmechanism is used to capture the router’s runtime state foranalysis. This is less than ideal because, due to the hard-ware constraint of the Cisco device, we can not guaranteethat the memory capture is performed atomically. Further-more, since the core dump is generated by IOS’s own (poten-tially compromised) code, the integrity of the output can notbe fully trusted. In contrast, when the Symbiote-injectedfirmware is executed within Dynamips, a Cisco router em-ulator, on a general purpose computer (emulated deploy-ment), the external monitor triggers a response which haltsemulation of the compromised IOS image before initiatinga full memory dump using the general purpose host com-puter. Thus, emulated deployment of our sensor can guar-antee that the capture and analysis process can be doneatomically without relying on potentially compromised code.Section 8 discusses the tradeoffs and advantages of all threedeployments in detail.

Symbiote Manager

Host Program

Symbiote Payload

= intercept point

Figure 1: Logical overview of SEM injected into em-bedded device firmware. SEM maintains control ofCPU by using large-scale randomized control-flowinterception. SEM payload executes alongside orig-inal OS.

2. MOTIVATIONSeveral recent studies demonstrate that there are vast

numbers of unsecured, vulnerable embedded devices on theinternet [8], such devices are vulnerable to the same typesof attacks as general purpose computers [3, 12], and canbe systematically exploited in much the same way [1, 3, 5].For example, various exploitable vulnerabilities [15, 13] androotkits [16] have been found and disclosed for Cisco’s routeroperating system, IOS. Cisco devices running IOS consti-tutes a significant portion of our global communication in-frastructure, and are deployed within critical areas of ourresidential, commercial, financial, government, military andbackbone networks.

Typical of the embedded security landscape, IOS is anaging system which does not employ standard protectionschemes found within modern operating systems [16], anddoes not have any host-based anti-virus to speak of. In fact,not only is the installation of third-party anti-virus (whichdoes not yet exist for IOS) not possible via any published OSinterface, any attempt to do so will also violate the vendor’sEULA and thus void existing support contracts.

Consider the availability of proof-of-concept exploits androotkits, the wide gamut of high-value targets which can becompromised by the exploitation of devices like routers andfirewalls, and the lack of host-based defenses within close-source embedded device firmwares. Such conditions shouldmake the vast numbers of vulnerable embedded devices onthe Internet highly attractive targets. Indeed, we have ob-served successful attempts to create botnets using Linux-based home routers [4]. As Section 4 shows, the necessarytechniques of exploiting Cisco IOS and installing root-kits onrunning Cisco routers are well understood. The works pre-sented within academic and blackhat circles, combined withanecdotal evidence of the systematic exploitation of embed-ded network devices within the last decade suggests thatreal-world exploitation of Cisco routers is not only possible,but likely an undetected reality.

Documented cases of embedded device exploitation arestill relatively rare. High-value embedded targets like en-terprise networking equipment have seemingly eluded ex-

ploitation. It is possible that the exploitation of deviceslike Cisco routers is still beyond the technical capabilities ofthe blackhat community. However, it is far more plausiblethat stealthy, targeted attacks against high-value embeddeddevices have eluded detection due to our inability to gainvisibility into the internals of such devices. It is quite pos-sible routers have been successfully attacked and are com-promised without anyone’s knowledge except the UE sellerswho offer them for sale.

Whether or not stealthy exploitation of embedded de-vices is a reality today, we can confidently anticipate thatattacks against such defenseless, high-value targets is in-evitable. Therefore, analysis and mitigation of embeddeddevice exploitation is crucial to the integrity of the Internetsubstrate. We believe that accurate, real-time detection ofsuch attacks is an important first step towards understand-ing the realities of the embedded security threat. Further-more, we believe the ability to inject host-based security intoexisting legacy devices will be instrumental in mounting arealistic defense of existing embedded devices.

The Symbiote structure presented in this paper is designedspecifically to abstract away the technical challenges of in-jecting third-party security into a diverse range of embeddeddevices. This device agnostic foundation allows us to lookbeyond specific hardware and firmware in order to create ageneral body of embedded defense methodologies which canbe feasibly applied to all existing devices.

3. THREAT MODELWe are interested in detecting, capturing and analyzing

successful injection of rootkits into IOS at runtime. We as-sume that the attacker is technically sophisticated and hasaccess to both zero-day vulnerabilities as well as a reliablerootkit which persistently alters the behavior of the victimdevice’s OS, yielding covert root access to the attacker. Weassume that the injected rootkit will patch specific portionsof the router’s code in order to create a hidden backdoor forthe attacker. In other words, we assume that the rootkit willalter regions of memory within the router that is meant to bestatic during normal execution. Static sections within IOSfirmware image typically include the .txt, .rodata, .firmware,.sdata and large portions of the .data sections. Furthermore,the boot-loader (rommon) section of the router, as well as allassociated configuration files (startup configuration, runningconfiguration, etc) can also be monitored for unauthorizedmodification.

While our current threat model encompasses all publishedIOS rootkitting techniques to date, it is probable that acovert backdoor can be created within an IOS router with-out modifying static regions of the router’s memory. Theproposed detection payload will not detect exploits whichleave no persistent change within the victim device. How-ever, the Symbiote-based injection scheme described in thispaper can be extended to monitor for anomalies within dy-namic sections of the target device, extending our whitelist-based detector into a full-blown host-based anomaly detector(See Section 10).

Furthermore, it is possible for sophisticated attacks to at-tempt to disable the Symbiote prior to the actual exploita-tion of the victim device. Since the Symbiote structure de-scribed in this paper is a software-based defense, absoluteintegrity of the Symbiote cannot be guaranteed. In the gen-eral case, Symbiotes can be fortified with the introduction of

specialized hardware. However, such a solution is not feasi-ble when considering the realm of legacy embedded devices.Instead, Section 6 illustrates a general method of increasingthe computational complexity of a successful bypass of theSymbiote defense without relying on additional hardware.

4. RELATED WORKRelatively little work has been done to detect and capture

sophisticated attacks against embedded devices. However,such problems have been well studied for general purposecomputers and operating systems. Numerous rootkit andmalware detection and mitigation mechanisms have beenproposed in the past but largely target general purpose com-puters. Commercial products from vendors like Symantec,Mcafee/Intel, Kapersky and Microsoft [2] all advertise someform of protection against kernel level rootkits. Kernel in-tegrity validation and security posture assessment capabilityhas been integrated into several Network Admission Control(NAC) systems. These commercial products largely dependon signature-based detection methods and can be subvertedby well known methods [18, 19, 20]. Sophisticated detec-tion and prevention strategies have been proposed by theresearch community. Virtualization-based strategies usinghypervisors, VMM’s and memory shadowing [17] have beenapplied to kernel-level rootkit detection. Others have pro-posed detection strategies using binary analysis [11], func-tion hook monitoring [23] and hardware-assisted solutionsto kernel integrity validation [22].

The above strategies may perform well within general pur-pose computers and well known operating systems but havenot been adapted to operate within the unique characteris-tics and constraints of embedded device firmware. Effectiveprevention of binary exploitation of embedded devices re-quires a rethinking of detection strategies and deploymentvehicles.

Our methodology transforms standard legacy embeddeddevices into exploitation detectors. This is similar to exist-ing honeypot-based IDS strategies, which generally involvesthe use of intentionally vulnerable systems to log, captureand analyze attacks levied against it. Many honeypot-basedsystems have been proposed. Few focus on the use or protec-tion of embedded devices. For example, Ghourabi et al. re-cently proposed the use of simulated honey routers to studyprotocol attacks against BGP [10].

In general, honeypots can be native, emulated or simu-lated, and can involve a single machine or a vast network ofsimulated nodes. Many off-the-shelf honeypot systems existfor general purpose computers. However, such systems arenot without flaws. For example, simulated honeypots dis-guises themselves as vulnerable systems but does not exposeany actual vulnerabilities to the attacker. Therefore, min-imizing false-positives in such systems is a challenge. Fur-thermore, simulated honeypots may catch indiscriminate ex-ploitation attempts, but will rarely fool sophisticated attack-ers in highly targeted attacks. Thus, native and emulatedhoneypots which exposes real vulnerabilities to the attackerare much better suited for detecting sophisticated, targetedattacks.

Guards, originally proposed by Chang and Atallah [6], isanother technology which uses mechanisms of action similarto Symbiotes. A Guard is a simple piece of security codewhich is injected into the protected software using binaryrewriting techniques similar to our Symbiote system. Once

injected, a guard will perform tamper-resistance function-ality like self-checksumming and software repair. However,Guards have no mechanism to pause and resume its com-putation, the entire Guard routine must complete executioneach time it is invoked. This limits the sophistication ofwhat each Guard can realistically perform, especially whenGuards are used in time sensitive software and real-time em-bedded devices.

Devices like Cisco routers are black-box systems utiliz-ing large numbers of undocumented proprietary hardwarecomponents. The injection of new code into proprietaryfirmware and the emulation of specialized and undocumentedhardware makes the creation native and emulated honeypotsfor embedded devices challenging. As Section 5 describes,the unique capabilities of the Symbiote construct allows usto overcome the above challenges in order to transform stan-dard Cisco IOS firmware and hardware into highly believablenative router honeypots.

5. MEET SYMBIOTEFor a full discussion of Symbiotic Embedded Machines,

please see [9]. The Symbiote is a software construct that isinjected in situ into a host program to provide the followingfour fundamental security properties.

1. The Symbiote has full visibility into the code and ex-ecution state of its host program, and can either pas-sively monitor or actively react to the observed eventsat runtime.

2. The Symbiote executes along side the host software.In order for the host to function as before, it’s injectedSymbiote must execute, and vice versa.

3. The Symbiote is an autonomous entity which is hard-ened to defend against unauthorized modification orremoval once it is injected into the host program.

4. No two instantiations of the same Symbiote are thesame. Each time a Symbiote is created, its code israndomized and mutated, rendering signature baseddetection methods and attacks requiring predictablememory and code structures within the Symbiote in-effective.

Host Program

= Live Code = Intercept Point

Host Program

Host Program

= Symbiote Binary

Original Unmodified Host Program Binary

Live Code Found Through Static Analysis or Profiling

Symbiote Binary Injected into Host Program. Live Code is Randomly Intercepted

1

2

3

Figure 2: Symbiote Injection Process.

Figure 1 shows the three logical components of Symbiotes:Control-Flow Interceptors, Symbiotic Embedded MachineManager (SEMM) and the Symbiote Payload. Together,

all three components are injected in situ into the target em-bedded device firmware. Since the Symbiote is injected insitu, the size of the resulting firmware image is unchanged.For example, the current implementation of the SymbioteManager, along with the rootkit detection payload requiresonly approximately 1600 bytes to be injected into IOS.

Figure 2 illustrates the three step Symbiote injection pro-cess. First, analysis is performed on the original host pro-gram in order to determine areas of live code, or code thatwill be run with high probability at runtime. Second, in-tercept points are chosen randomly from the host program.Lastly, the Symbiote Manager, Symbiote payload and a largenumber of control-flow intercepts are injected into the hostprogram binary, yielding a Symbiote protected host pro-gram.

The Symbiote randomly intercepts a large number of func-tions as a means to divert periodically and consistently asmall portion of the device’s CPU cycles to execute its pay-load. This approach allows the Symbiote to remain agnos-tic to operating system specifics while executing its payloadalongside the original OS. The Symbiote payload has full ac-cess to the internals of the original OS but is not constrainedby it. This allows the payload to carry out functionalitywhich might not be possible under the original OS. In thecase of Cisco IOS for example, a process watchdog timer willforcibly terminate any process which executes for more thanseveral seconds. However, since the Symbiote payload exe-cutes in time-slices randomly distributed throughout manyunrelated processes, the Symbiote payload can execute in-definitely, circumventing the watchdog timer entirely.

Stealth is a byproduct of the SEM structure. In the caseof IOS, no diagnostic tool available within the OS (short ofa full memory dump) can detect the presence of the SEMpayload because it manipulates no OS specific structure andis effectively invisible to the OS. The impact of the SEMpayload is further hidden by the fact that CPU utilization ofthe payload is not reported within any single process underIOS and is distributed randomly across a large number ofunrelated processes.

Once the Symbiote Manager gains control of the CPU, itallocates a certain number of cycles for the execution of itsSymbiote payload (in this case, a checksuming mechanism).After the payload completes its execution burst, control ofthe CPU is returned to the Symbiote Manager, which inturn resumes the execution of the original host program.

The Symbiote Manager acts as a job scheduler, treatingthe entire host program as one process, and its Symbiotepayload as the other. Traditional scheduling strategies canbe used to determine the proper CPU resource distributionbetween the Symbiote and its host program. In general, thisinvolves the optimization of both the frequency of contextswitches as well as the duration of the Symbiote payload’sexecution bursts.

The proposed Symbiote payload detects unauthorized codemodification through the computation of checksums overstatic regions of memory. Therefore, a delay exists betweenthe time of the code modification and its detection. In gen-eral we refer to the time between the occurrence of an unau-thorized event and its detection as the detection latency.Intuitively, the amount of CPU resources diverted to theSymbiote payload should be inversely proportional to thedetection latency, and thus directly proportional to the per-formance of our detector. In the case of Cisco IOS, and

many other embedded systems, an over allocation of CPUresources to the Symbiote can adversely affect the perfor-mance of the protected host device. In practice, we havefound that it is beneficial to frequently interleave the hostprogram’s execution with short Symbiote payload executionbursts. This allows the Symbiote payload to compute at ac-ceptable rates while minimizing the impact on the real-timenature of Cisco routers.

The Symbiote scheduling problem is arguably simple asit involves only two ”tasks”. However, performing such atask safely in an OS agnostic manner on embedded systemspresents several interesting complexities. A full discussion ofpotential Symbiote scheduling algorithms is out of the scopeof this paper. However, in the case of our IOS exploitationdetection Symbiote, the performance and overhead charac-teristics of several scheduling strategies are discussed in Sec-tion 9.

6. SELF-MONITORING SYMBIOTESWe must consider ways to protect the Symbiote itself

against attack and removal. The polymorphic nature ofthe Symbiote and its payload makes signature-based attacksagainst it ineffective. To further raise the bar, multiple Sym-biotes within a protected host program can be configuredin a self-monitoring monitor arrangement. As proposed byStolfo, Greenbaum and Sethumadhavan [21], a network ofmonitors can be constructed, such that an alarm will beraised if any subset of monitors are compromised or deacti-vated, or if any critical condition monitored by the systemis violated. Consider Figure 3, which shows three indepen-dent Symbiotes arranged in a full-mesh monitoring network.In this arrangement, each Symbiote monitors a specific crit-ical condition, i.e., the output of their Symbiote Payload,while simultaneously monitoring the operational status ofthe other two Symbiotes within the network. If one or moreof the Symbiotes are corrupted or disabled, the remainingSymbiotes within the network will raise an alarm. Similarly,if all three Symbiotes are simultaneously deactivated, an ex-ternal sensor can also detect this event and raise an alarm.Note that the three Symbiotes shown in Figure 3 need not belocated within the same host router. Large networks of em-bedded device sensors can be collectively protected in thismutually defensive arrangement. Using Symbiotes in thisfashion is a topic of ongoing research.

7. EXPLOITATION DETECTORIOS rootkit and malware code is generally not publicly

available. However, a survey of published persistent rootkittechniques reveals a commonality in their modus operandi.Specifically, rootkits such as [16, 13, 7] all modify some re-gion of static IOS memory in order to inject their rootkitpayload into the victim router. Thus, we implemented awhite-list strategy to detect IOS malcode and rootkits de-scribed previously.

Known rootkits operate by hooking into and altering keyfunctions within IOS. To do this, specific binary patchesmust be made to executable code. Therefore, a continuousintegrity check on all static areas of Cisco IOS will detectall function hooking and patching attempts made by rootk-its and malware. The rootkit detection payload describedbelow is not specific to IOS, and can be used on other em-bedded operating systems as well. As Section 9 shows, our

Symbiote payload accurately detects unauthorized modifica-tion of any monitored region of memory within milliseconds,and will accurately detect [16, 13, 7] immediately after suc-cessful exploitation of the victim device.

In the case of Cisco IOS, several large contiguous segmentsof the router’s memory address space can be monitored usingthe checksumming mechanism described above. Figure 4 il-lustrates the memory layout of a typical IOS firmware imageon a Cisco router. The darkened regions represent areas ofthe router’s firmware which can be safely monitored by ourchecksumming mechanism. For example, regions containingexecutable code (text and firmware), and static data (ro-data, ctors, sdata sections) should clearly not be modified atruntime. In practice, the typical IOS firmware contains largecontiguous sections of memory which should semantically re-main static during the normal operation of the router.

8. DESIGN AND OPERATIONOur sensor system has three components; a Symbiote-

protected router, a monitoring station, and a capture andanalysis system which automatically collects and analyzesforensics data once an alarm is triggered. The Symbiotewithin the IOS firmware simultaneously performs checksumson all protected regions of the router’s memory while period-ically communicating with an external monitor via a covertchannel. In the event of an unauthorized memory modifica-tion within the router, the Symbiote will raise an alarm tothe monitor, which then triggers the capture and analysiscomponent of our system.

The proposed exploitation detection sensor can be de-ployed in one of at least three ways; natively, emulatedwithin a general purpose computer, or as a shadow replicafor a production device. The implementation of the moni-toring station and capture and analysis engine changes de-pending on how the Symbiote-injected router firmware isexecuted; natively on embedded hardware or emulated on ageneral purpose computer.

When deployed natively, the monitor and capture compo-nents are integrated into the Symbiote payload and injecteddirectly into Cisco hardware, producing a standalone sen-sor. When the detection payload raises an alarm, the Sym-biote immediately triggers the core dump functionality fromwithin IOS. This causes the bulk of the router’s executionstate to be captured and transferred via FTP or TFTP.

When deployed as an emulated sensor, using Dynamipsfor example, the monitoring and capture components of thesensor are implemented within the emulator. This reducesthe footprint of the Symbiote and allows us to perform moresophisticated capture and analysis on the server running theemulation. For example, Dynamips was modified to contin-uously monitor a region of the router’s memory for an en-coded marker, which is set by the Symbiote payload onlywhen an alarm is raised.

For testing purposes, we chose to modify a portion of thetext that is printed when the ”show version” command isinvoked. In practice, many better covert channels can beused to communicate between the Symbiote and the routeremulator.

In order to transform large populations of embedded de-vices into massive embedded exploitation sensor-grids, thenative deployment is the most efficient and practical. For thepurposes of testing and validation of our approach, the emu-lated deployment scenario is most appropriate. The shadow

Figure 3: Full Mesh Self-Monitoring SymbioteNetwork

.text

.rodata

firmware.eh_frame

.data

.ctors.sdata.sbss

.bss

= Static Regions

0x80008000

0x84669060

Figure 4: Memory layout of a typical Cisco IOSrouter

deployment is best for capturing and analyzing IOS exploitsin mission critical production environments.

8.1 Native Sensor DeploymentIn the native deployment scenarios, the Symbiote-injected

firmware is loaded directly onto the target embedded device,i.e. a Cisco router. The Symbiote payload executes nativelyon the embedded hardware, alongside the original firmware.Native deployment allows the Symbiote to operate in em-bedded systems for which emulation is not feasible. For ex-ample, a large portion of Cisco devices can not be emulatedby existing software due to the use of undocumented, propri-etary hardware. In practice, most modern high-performancenetworking equipment falls within this category. Therefore,native deployment is most practical for injecting Symbioticdefenses into the diverse range of embedded devices foundon the Internet substrate.

8.2 Emulated Sensor Deployment

Symbiote ProtectedRouter (Emulated)

Internal MonitoringProcess

Network

Capture and AnalysisEngine

General Purpose Server

Figure 5: Emulated Deployment of Symbiote-basedCisco IOS Detector

Figure 5 illustrates a typical emulated deployment of oursensor. Instead of running the Symbiote-injected firmware

natively on embedded hardware, the firmware is emulatedon a general purpose computer. In the case of Cisco IOS,Dynamips is used to emulate specific devices such as 7200 se-ries routers used in our testing and evaluation environment.This differs from simulated honeypots in a significant way:the use of actual IOS firmware. The emulation of real CiscoIOS allows us to create highly interactive honeypots whichexposes real IOS vulnerabilities to potential attackers.

The emulated deployment has several advantages whichmake it the ideal approach for developing and testing ex-perimental prototype Symbiotes. First, debugging proof ofconcept Symbiotes in an emulated environment is slightlymore convenient than doing so on native embedded hard-ware. Second, the general purpose computer which hoststhe emulation usually has far greater computational capacitythan the embedded hardware which it is emulating. There-fore, computation can be offloaded from the Symbiote pay-load onto the general purpose host computer. This canpotentially allow the Symbiote payload to perform com-plex computations not feasible on actual embedded hard-ware. The Symbiote payload presented in this paper is sim-ple and requires relatively little CPU power. However, thiscan be replaced with payloads more akin to behavior basedanomaly detectors which can require significantly more re-sources. The development of Symbiote-based anomaly de-tection mechanisms is an area of active research (See Section10).

Lastly, the emulated sensor deployment can usually sim-plify the capture and analysis component of the sensor. Inthe case of the sensor presented in this paper, we modi-fied the Dynamips emulator to atomically capture the entirememory state of the IOS router once the Symbiote payloademits an alarm. The Dynamips emulator conveniently al-lows us to halt the router’s CPU briefly while the memorycapture takes place on the host computer. Once this opera-tion completes, the memory snapshot, along with all network

traffic received by the router is automatically processed andarchived for analysis. Once the Symbiote payload emits analarm, our modified Dynamips emulator continuously dumpsthe memory state of the router at a configurable frequency.

8.3 Shadow Sensor Deployment

Symbiote ProtectedRouter (Emulated Shadow)

Internal MonitoringProcess

Network

Capture and AnalysisEngine

General Purpose Server

Production Router SPAN Port

Figure 6: Shadow Deployment of Symbiote-basedCisco IOS Detector

In order to detect exploitation against high-performanceembedded devices within production environments, we mustbe able to deploy Symbiote-based sensors in a way which willnot cause unintentional service outages on the monitored de-vices. In such cases, the use of a second, identical embeddeddevice as a shadow sensor is most appropriate. Figure 6illustrates a typical shadow sensor deployment.

As the name suggests, incoming network traffic is mir-rored from the production embedded device to a Symbiote-injected shadow device, which runs the same firmware asthe production device. The Symbiote sensor injected intothe shadow device continuously monitors the shadow device,quietly emitting alerts when malicious activity is detected.

The performance of the shadow sensor is critical, as itmust be able to keep up with the production router. Thus,minimizing the control-plane latency and computational over-head of the Symbiote is critical to the effectiveness of thedetection system. We discuss preliminary performance datain the next section. The development of Symbiote-basedshadow sensors is an area of active research.

9. PERFORMANCE AND OVERHEADWe measure the performance and overhead of our Symbiote-

based exploit detector using two quantitative metrics: com-putational overhead and detection latency. The Symbiote-protected router is an emulated Cisco 7200 series routerrunning IOS 12.3. Two neighbor routers are used to ver-ify that the Symbiote-protected router’s original functional-ity is unchanged. One neighbor router is an emulated 7200series router running standard IOS 12.3. The other neigh-bor router is a physical Cisco 2921 router running IOS 12.5.Each router is configured to expose a cross-section of func-tionality typically seen on production routers. Specifically,a large number of local loopback interfaces are configured oneach router. OSPF routing is enabled on all three routers,along with a suite of standard services like IP-SLA, SNMP,HTTP{S} and SSH.

A stress-test script automatically generates network trafficthroughout the test environment, and periodically accesses

services on all the test routers. All routers are continu-ously monitored to ensure that all services operate correctlythroughout testing. The workload script also periodicallyforces route-table re-calculations by perturbing the variousOSPF routers on the network. In effect, the stress-test scriptsimulates a typical use profile for the IOS routers in the testenvironment. The same stress-test script is run against sev-eral variants of the Symbiote-injected IOS firmware in orderto illustrate key performance features of our system.

The computational overhead and performance of our sys-tem is a configurable parameter. As the figures in this sec-tion shows, the scheduling algorithm used within the Sym-biote Manager directly impacts the resource consumption ofthe Symbiote payload, and thus the overall utilization of thehost device as well as the detection latency. Two schedulingalgorithms are discussed in this section: fixed burst-rate andinverse-adaptive.

As the name suggests, the fixed burst-rate scheduling al-gorithm instructs the Symbiote payload to execute for afixed burst-rate each time the Symbiote Manager is invokedthrough a randomly placed execution intercept. On theother hand, the inverse-adaptive scheduling algorithm cal-culates the payload burst-rate based on the elapsed timesince the Symbiote Manager was last invoked; the longerthe elapsed time, the longer the burst-rate.

Intuitively, we can expect the fixed burst-rate schedulingalgorithm to execute the Symbiote payload more frequentlyas the host system becomes more utilized. This simple algo-rithm executes the Symbiote payload more frequently whenthe Cisco router is heavily utilized, and less frequently whenthe router is idle. In contrast, the inverse-adaptive schedul-ing algorithm increases Symbiote payload burst-rate whenthe system is under-utilized, and throttles back the Sym-biote payload when the router is under high load.

We analyze the performance of 15 Symbiote-injected IOSimages under the same stress-test; 7 variants using the fixedburst-rate Symbiote scheduler and 8 variants using the inverse-adaptive Symbiote scheduler. As the next three subsectionsshow, the fixed burst-rate Symbiote scheduler aggressivelyexecutes the Symbiote payload, and achieves the least detec-tion latency (approximately 400 ms). However, this aggres-sive scheduler tends to amplify CPU utilization of the pro-tected router, causing very high control-plane latency whenthe router is under load. Although the higher fixed burst-rate values like 0x7FF and 0xFFF detected IOS modificationvery quickly, it also caused the router’s control-plane to beless responsive.

In contrast, the inverse-adaptive Symbiote scheduler pro-duced slightly longer detection latencies (approximately 450ms), but was able to significantly reduce the control-planelatency of the host router, even under high load.

9.1 Computational OverheadThe same stress-test script is run against various ver-

sions of the Symbiote-injected IOS image in order to showhow the Symbiote Manager’s scheduling algorithm affectsthe CPU utilization of the router. Two major schedulingalgorithms are measured: fixed burst-rate (Figure 7) andinverse-adaptive (Figure 8). Burst-rate values presentedin the next two sections represent the number of iterationsof the main Symbiote payload executed each time the Sym-biote Manager is invoked.

Figure 7 shows the CPU utilization of 7 variants of the

fixed burst-rate Symbiote scheduler, which unconditionallyexecutes the Symbiote payload for a constant number ofCPU cycles each time the Symbiote is invoked via its manycontrol-flow intercepts. The units used, burst-rate, is thenumber of iterations of the checksum Symbiote payload thatis executed each time the Symbiote Manager is invoked.

This Symbiote scheduler disregards the current CPU uti-lization of the host device. At higher burst-rate values like0x7FF and 0xFFF, the router’s CPU utilization tends to re-main above 95% under heavy load, causing large spikes incontrol-plane latency. (See Figure 11)

Figure 8 shows the CPU utilization of 8 variants of theinverse-adaptive Symbiote scheduler, compared with the base-line CPU utilization of the unmodified IOS image under thesame stress-test. The inverse-adaptive scheduler is config-ured with maximum burst-rates from 0x1FFFF to 0xFFFFFF.Unlike the fixed burst-rate Symbiote scheduler, the inverse-adaptive scheduler throttles how much the CPU is divertedto the Symbiote based on current host device utilization.As a result, Symbiotes with inverse-adaptive schedulers canachieve comparable detection latencies while significantly re-ducing its impact on the host router’s control-plane latency.(Compare Figure 11 and Figure 12).

Figure 7: CPU Utilization: Fixed Burst-Rate SEMManager

Figure 8: CPU Utilization: Inverse-Adaptive SEMManager

9.2 Detection PerformanceIn order to measure the detection latency of our exploita-

tion detection Symbiote, a simple vulnerability which al-lows arbitrary memory modification is artificially introducedinto the Symbiote-injected IOS image. Using an automatedscript, this vulnerability is triggered, modifying a randombyte within monitored memory regions. A timer is simul-taneously started in order to measure the time it takes theSymbiote payload to detect the event.

Figure 9 shows a roughly linear relationship between theSymbiote’s fixed burst-rate value and the Symbiote’s detec-tion latency. As expected, the Symbiote detection latencydecreases as the Symbiote payload’s execution burst-rateincreases. However, as Figure 11 shows, the fixed burst-rate Symbiote scheduler causes significant increases in therouter’s control-plane latency.

Figure 10 shows the detection latency of Symbiotes usingthe inverse-adaptive scheduler. As the figure shows, theseSymbiotes can achieve comparable detection latency valuesas the fixed burst-rate versions, but as Figure 12 shows, theSymbiote’s impact on the router’s control-plane is signifi-cantly reduced.

0

0.5

1

1.5

2

2.5

3

3.5

4

0xF

0x3F

0x6F

0x7F

0x9F

0xC

F

0xF

F

0x3F

F

0x6F

F

0x9F

F

0xC

FF

0xF

FF

0x3F

FF

0x6F

FF

0x9F

FF

0xC

FF

F

0xF

FF

F

Det

ecti

on L

aten

cy (

Sec

onds)

Payload Execution Rate

Detection Latency: Fixed Burst−Rate SEM ManagerDetection Latency: Fixed Burst−Rate SEM ManagerDetection Latency: Fixed Burst-Rate SEM Manager

Figure 9: Detection Latency: Fixed Burst-RateSEM Manager

0

0.5

1

1.5

2

2.5

3

3.5

4

0x1F

FF

F

0x3F

FF

F

0x7F

FF

F

0xF

FF

FF

0x1F

FF

FF

0x3F

FF

FF

0x7F

FF

FF

0xF

FF

FF

F

0x1F

FF

FF

0x3F

FF

FF

0x7F

FF

FF

0xF

FF

FF

F

Det

ecti

on L

aten

cy (

Sec

onds)

Maximum Payload Execution Rate

Detection Latency: Inversely Adaptive SEM ManagerDetection Latency: Inversely Adaptive SEM Manager

Figure 10: Detection Latency: Inverse-AdaptiveSEM Manager

9.3 Control-Plane LatencyControl-plane latency is an indicator of how responsive

the router is. High control-plane latency can cause a routerto drop routing adjacencies and break various time-sensitivenetwork protocols. Note, however, that this measurementwill not significantly affect the latency of traffic passingthrough the router, as most modern routers have hardware-accelerated forwarding engines which are decoupled from thecontrol-plane.

Control-plane latency is measured by sending ICMP-echomessages from the test PC to the router’s local loopback in-terface. The round-trip-time is collected and shown in Fig-ure 11 for Symbiotes using fixed burst-rate scheduler vari-ants, and in Figure 12 for Symbiotes using inverse-adaptivescheduler variants. Clearly, the inverse-adaptive Symbiotescheduler significantly reduces the Symbiote’s impact on thehost router’s control-plane latency while achieving compa-rable detection latency values as fixed burst-rate Symbiotes.

Figure 11: Ping Latency: Fixed Burst-Rate SEMManager

Figure 12: Ping Latency: Inverse-Adaptive SEMManager

9.4 DiscussionPreliminary performance results shown in this section sug-

gests that high performance exploitation detection is pos-sible in Cisco IOS. Furthermore, an optimized Symbiote

scheduling algorithm can greatly improve performance ofthe overall sensor system by reducing both detection latencyand the Symbiote’s impact on the router’s control-plane la-tency. Optimization of the detection latency and the in-duced control-plane latency is an area of active research.

10. FUTURE WORKThe Symbiote-based sensor presented in this paper is a

first step towards demonstrating the feasibility and novel ca-pability of Symbiotic defense systems. The Symbiote struc-ture allows complex payloads to be injected into legacy em-bedded devices, allowing the payload to safely execute along-side the original firmware without altering the embedded de-vice’s functionality. The checksumming payload we injectedinto Cisco IOS can be replaced with a wide range of de-fensive payloads. Below are several new Symbiote payloadscurrently under development.

10.1 Embedded Self-HealingThe checksumming Symbiote payload discussed in this pa-

per can be extended to reverse unauthorized modificationof memory after it is detected. A self-healing Symbiote pay-load can be used to identify and restore regions of memorywhich have been maliciously modified.

10.2 Embedded Anomaly DetectorSymbiote payloads can implement existing anomaly detec-

tion algorithms. For example, behavior modeling strategieswhich monitor resource utilization, control and data flowpatterns can be injected into embedded devices via Sym-biote payloads.

10.3 Large-Scale Embedded Sensor GridThe exploitation detection sensor described in this paper

can be injected into large numbers of embedded devices likeCisco routers in order to monitor and analyze 0-day exploita-tion of embedded devices. We believe the use of Symbiote-based exploitation sensors in the wild is a feasible and effec-tive way of monitoring and analyzing exploits levied againstthe internet substrate. A large-scale Symbiote-based sensorgrid can potentially give us real-time visibility into embed-ded device exploitation on a global scale.

Furthermore, Symbiotes can be used to transform embed-ded devices into other kinds of sensor grids as well. Sym-biotes can allow us to use hardware components of embed-ded devices in novel ways not intended by its original de-sign. For example, many power-consuming, EM emittingcomponents can be transformed into covert communicationchannels. Existing sensors on embedded devices, combinedwith such covert channels can transform a wide gamut of in-nocuous embedded devices into a web of remotely controlledmobile sensors.

11. CONCLUSIONThe Symbiote mechanism can be used to augment legacy

embedded devices with novel functionality in an OS agnosticmanner. The applications of this capability are numerous,and will help make the introduction of modern host-baseddefenses on existing embedded devices a feasible reality. Thechecksumming Symbiote payload described in this paper isa starting point in demonstrating the unique advantages ofSymbiotic defense systems. We have demonstrated that the

Symbiote can automatically augment Cisco IOS with effec-tive anti-rootkitting capabilities. This accomplishment haslaid the foundation for the construction of a large sensor-gridof legacy embedded devices in order to accurately detect andanalyze the exploitation of the devices which make up thefabrics of our global communication infrastructures.

12. ACKNOWLEDGEMENTSThis material is based on research sponsored by Air Force

Research labs under agreement number FA8750-09-1-0075.The U.S. Government is authorized to reproduce and dis-tribute reprints for Governmental purposes notwithstand-ing any copyright notation thereon. This material is alsobased on research sponsored by DARPA contract: CRASHprogram, SPARCHS, FA8750-10-2-0253.

13. REFERENCES[1] kaiten.c IRC DDOS Bot.

http://packetstormsecurity.nl/irc/kaiten.c.

[2] Microsoft Corporation, Kernel Patch Protection:Frequently Asked Questions.http://tinyurl.com/y7pss5y, 2006.

[3] The End of Your Internet: Malware for HomeRouters, 2008. http://tinyurl.com/3d9yv9l.

[4] Network Bluepill. Dronebl.org, 2008.http://www.dronebl.org/blog/8.

[5] New worm can infect home modem/routers.APCMAG.com, 2009.http://apcmag.com/Content.aspx?id=3687.

[6] Hoi Chang and Mikhail J. Atallah. Protectingsoftware code by guards. In Tomas Sander, editor,Digital Rights Management Workshop, volume 2320 ofLecture Notes in Computer Science, pages 160–175.Springer, 2001.

[7] Ang Cui, Jatin Kataria, and Salvatore J. Stolfo.Killing the myth of cisco ios diversity: Towardsreliable, large-scale exploitation of cisco ios, 2011. 5thUSENIX Workshop on Offensive Technologies.

[8] Ang Cui and Salvatore J. Stolfo. A quantitativeanalysis of the insecurity of embedded networkdevices: results of a wide-area scan. In Carrie Gates,Michael Franz, and John P. McDermott, editors,ACSAC, pages 97–106. ACM, 2010.

[9] Ang Cui and Savaltore J. Stolfo. Defending legacyembedded devices with software symbiotes. In RobinSommer, Davide Balzarotti, and Gregor Maier,editors, RAID, volume 6961 of Lecture Notes inComputer Science. Springer, 2011.

[10] Abdallah Ghourabi, Tarek Abbes, and Adel Bouhoula.Honeypot router for routing protocols protection. InAnas Abou El Kalam, Yves Deswarte, and MahmoudMostafa, editors, CRiSIS, pages 127–130. IEEE, 2009.

[11] Christopher Krugel, William K. Robertson, andGiovanni Vigna. Detecting kernel-level rootkitsthrough binary analysis. In ACSAC, pages 91–100.IEEE Computer Society, 2004.

[12] Felix ”FX” Linder. Cisco Vulnerabilities. In InBlackHat USA, 2003.

[13] Felix ”FX” Linder. Cisco IOS Router Exploitation. InIn BlackHat USA, 2009.

[14] Richard Lippmann, Engin Kirda, and AriTrachtenberg, editors. Recent Advances in IntrusionDetection, 11th International Symposium, RAID 2008,Cambridge, MA, USA, September 15-17, 2008.Proceedings, volume 5230 of Lecture Notes inComputer Science. Springer, 2008.

[15] Michael Lynn. Cisco IOS Shellcode, 2005. In BlackHatUSA.

[16] Sebastian Muniz. Killing the myth of Cisco IOSrootkits: DIK, 2008. In EUSecWest.

[17] Ryan Riley, Xuxian Jiang, and Dongyan Xu.Guest-transparent prevention of kernel rootkits withvmm-based memory shadowing. In Lippmann et al.[14], pages 1–20.

[18] Dror-John Roecher and Michael Thumann. NACAttack. In In BlackHat USA, 2007.

[19] Skywing. Subverting PatchGuard Version 2, 2008.Uninformed,Volume 6.

[20] Yingbo Song, Pratap V. Prahbu, and Salvatore J.Stolfo. Smashing the stack with hydra: The manyheads of advanced shellcode polymorphism. In Defcon17, 2009.

[21] Salvatore J. Stolfo, Issac Greenbaum, and SimhaSethumadhavan. Self-monitoring monitors. TechnicalReport cucs-026-09, Columbia University ComputerScience Department, April 2009.

[22] Vikas R. Vasisht and Hsien-Hsin S. Lee. Shark:Architectural support for autonomic protectionagainst stealth by rootkit exploits. In MICRO, pages106–116. IEEE Computer Society, 2008.

[23] Zhi Wang, Xuxian Jiang, Weidong Cui, and XinyuanWang. Countering persistent kernel rootkits throughsystematic hook discovery. In Lippmann et al. [14],pages 21–38.