Towards Co-existing of Linux and Real-Time OSes

Hitoshi Mitake, Tsung-Han Lin, Hiromasa Shimada, Yuki Kinebuchi, Ning Li, Tatsuo NakajimaDepartment of Computer Science and Engineering, Waseda University

{mitake, johnny, yukikine, lining, tatsuo}@dcl.info.waseda.ac.jp

Abstract

The capability of real-time resource management inthe Linux kernel is dramatically improving due to theeffective contribution of the real-time Linux commu-nity. However, to develop commercial products cost-effectively, it must be possible to re-use existing real-time applications from other real-time OSes whose OSAPI differs significantly from the POSIX interface. Avirtual machine monitor that executes multiple oper-ating systems simultaneously is a promising solution,but existing virtual machine monitors such as Xen andKVM are hard to used for embedded systems due totheir complexities and throughput oriented designs. Inthis paper, we introduce a lightweight processor abstrac-tion layer named SPUMONE. SPUMONE provides vir-tual CPUs (vCPUs) for respective guest OSes, andschedules them according to their priorities. In a typicalcase, SPUMONE schedules Linux with a low priorityand an RTOS with a high priority. The important fea-tures of SPUMONE are the exploitation of an interruptprioritizing mechanism and a vCPU migration mech-anism that improves real-time capabilities in order tomake the virtualization layer more suitable for embed-ded systems. We also discuss why the traditional virtualmachine monitor design is not appropriate for embed-ded systems, and how the features of SPUMONE allowus to design modern complex embedded systems withless efforts.

1 Introduction

Modern real-time embedded systems like smart phonesbecome highly functional along with the enhancementsof CPUs targeting their market. But their functional fea-tures introduced significant engineering cost. The maindifficulty in the development of such devices comesfrom the conflicting requirement of them: low latencyand high throughput must be established in one system.This requirement is hard to satisfy with existing OSes,

because all of them are categorized as either Real-TimeOperating System (RTOS) or General Purpose Oper-ating System (GPOS). RTOSes, like eCos [2] or TOP-PERS1 [3], are designed and developed for executingreal-time tasks such as processing wireless communica-tion protocols. In the typical case, such a task runs pe-riodically for short time. The feature of executing suchdeadline-sensitive tasks imposes limitation on RTOSes.For example, most RTOSes cannot change the num-ber of tasks dynamically. On the other hand, GPOSessuch as Linux, are designed and developed for execut-ing tasks which consist of significant amount of compu-tation. Of course some of them in desktop computersare latency sensitive, to offer a comfortable user experi-ence, but missing a deadline is not fatal for them. Thecontribution from the real-time Linux community hassignificantly improved the real-time resource manage-ment capability of Linux [7]. However, there is alwaysa tradeoff between satisfying real-time constraints andachieving maximum throughput [8].

In order to develop a modern real-time embedded sys-tem which satisfies conflicting requirements, combin-ing multiple OSes on virtual machine monitors can bean effective approach. A virtual machine monitor, e.g.KVM [6], Xen [4] and VMware [5], is traditionally usedin the area of data center or desktop computing for ex-ecuting multiple OS instances in one physical machine.Their capability of executing multiple OSes is also at-tractive for embedded systems because they make it pos-sible to implement a system which has multiple OS per-sonalities. If there is a virtualization layer which has acapability of executing GPOS and RTOS in one phys-ical machine, developing real-time embedded systemscan be simpler.

Armand and Gien [9] presented several requirements fora virtualization layer to be suitable for embedded sys-tems:

1TOPPERS is an open source RTOS that offers µITRON inter-face, and it is used in many Japanese commercial products.

• 55 •

56 • Towards Co-existing of Linux and Real-Time OSes

1. It should execute an existing operating system andits supported applications in a virtualized environ-ment, such that modifications required to the op-erating system are minimized (ideally none), andperformance overhead is as low as possible.

2. It should be straightforward to move from one ver-sion of an operating system to another one; thisis especially important to keep up with frequentLinux evolutions.

3. It should reuse native device drivers from their ex-isting execution environments with no modifica-tion.

4. It should support existing legacy often real-timeoperating systems and their applications whileguaranteeing their deterministic real-time behavior.

Unfortunately, there is no open source virtualizationlayer that satisfies all the above requirements. Virtual-Logix2 VLX [9] is a virtualization layer designed forcombining RTOS and GPOS, but it is proprietary soft-ware. OKL4 microvisor [12] is a microkernel based vir-tualization technology for embedded systems, but per-forms poorly as the nature of microkernels [9]. In addi-tion, we found that there are fatal performance degra-dation of guest OSes when RTOS and SMP GPOSshare a same physical CPU. This performance prob-lem comes from the phenomenon called Lock HolderPreemption(LHP) [11]. It is a general phenomenon ofvirtualization layers, hence a solution for this problemwas already proposed. However these existing solu-tions only focus on the throughput of guest OSes, there-fore the virtualization layers that execute RTOSes can-not adopt these solutions. To the best of our knowl-edge, there is no virtualization layer that can executeRTOS and GPOS on a multicore processor without per-formance degradation caused by LHP, and is distributedas open source software.

Our laboratory is developing an open source virtual-ization layer for combining RTOS and Linux on em-bedded systems that adopt multicore processors, namedSPUMONE (Software Processing Unit, MultiplexingONE into two or more). During the development of thisvirtualization layer, we faced many difficulties specificto embedded systems. They come from the limitation

2VirtualLogix, Inc. was acquired by Red Bend Software in Sep.2010

of hardware resources, the requirement of engineeringcost, or scheduling RTOS and SMP GPOS on the sameCPU. Because of these difficulties, we believe that virtu-alization layers for real-time embedded systems shouldbe developed as open source software for incorporatingvarious insights from a wide range of community.

This paper is structured as follows: in Section 2, thedetailed motivation of our project is described. Sec-tion 3 describes the basic architecture of SPUMONE.Section 4 and Section 5 are catalogs of the problemswe encountered. Section 4 describes the difficulties ofdealing with real-time virtualization layers which adoptmulticore processors. Section 5 describes the methodfor isolating OSes spatially in embedded systems. Sec-tion 6 shows related work and the differences betweenSPUMONE and them. Finally Section 7 concludesthis paper and mentions about future directions of thisproject.

2 Why Virtualization

This section presents three advantages of using virtu-alization layers in embedded systems. The first ad-vantage is that control processing can be implementedas application software on RTOS. Embedded systemsusually include control processing like mechanical mo-tor control, wireless communication control or chem-ical control. Using software-based control techniquesenables us to adopt a more flexible control strategy, sorecent advanced embedded systems contain micropro-cessors instead of hardware implemented controllers forimplementing flexible control strategies. On the otherhand, recent embedded systems need to process vari-ous information. For example, applications which re-quire significant computation, such as multimedia play-ers and full featured web browsers, are crucial elementsof modern smart phones. Therefore, recent embeddedsystems have to contain both control and informationprocessing functionalities. In traditional embedded sys-tems, dedicated processors are assigned for respectiveprocessing. A general purpose processor with sufficientcomputational capability offers a possibility to combinethese multiple processing on a single processor. A vir-tualization layer can host RTOS and GPOS on one sys-tem, therefore this approach requires less hardware con-trollers and reduces the cost of embedded systems hard-ware.

The second advantage is that a virtualization layermakes it possible to reuse existing software. Even if the

2011 Linux Symposium • 57

virtualization layer is based on the para-virtualizationtechnique (which requires the modification of guestOSes), application programs running on the guest OSesdo not need to be modified. In a typical case of devel-oping embedded systems, vendors have their own OSesand applications running on them. The virtualizationlayer can execute such in-house software with standardOS platforms like Symbian or Android. If the in-housesoftware is developed against such a standard platform,it should be modified when a standard platform is re-placed. Actually, the standard platform is frequently re-placed according to various business reasons. On theother hand, if the in-house software is developed as ap-plication programs that run on the vendor specific OSes,porting application programs is not required even if astandard platform is replaced.

The third advantage is the isolation of source code. Forexample, proprietary device drivers can be mixed withGPL code without license violation. This may solve var-ious business issues when adopting Linux in embeddedsystems.

3 Basic Architecture

3.1 User-Level Guest OS vs. Kernel Level GuestOS

There are several traditional approaches to execute mul-tiple operating systems on a single processor in orderto compose multiple functionalities. Microkernels ex-ecute guest OS kernels at the user level. When usingmicrokernels, various privileged instructions, traps andinterrupts in the OS kernel need to be virtualized by re-placing their code. In addition, since OS kernels exe-cuted as user level tasks, application tasks need to com-municate with the OS kernel via inter-process commu-nication. Therefore, many parts of the OS need to bemodified.

VMMs are another approach to execute multiple OSes.If a processor offers a hardware virtualization support,all instructions that need to be virtualized trigger trapsto VMM. This makes it possible to use any OSes with-out any modification. But if the hardware virtualizationsupport is incomplete, some instructions still need tobe complemented by replacing some code to virtualizethem.

Most of the processors used for the embedded systemsonly have two protection levels. So when kernels are lo-cated in the privileged level, they are hard to isolate. Onthe other hand, if the kernels are located in the user level,the kernels need to be modified significantly. Most ofembedded system industries prefer not to modify a largeamount of the source code of their OSes, so it is desir-able to put them in the privileged level. Also, the virtu-alization of MMU introduces significant overhead if thevirtualization is implemented by software. Therefore,we need reorder mechanisms to reduce the engineeringcost, to ensure the reliability of the kernels and to exploitsome advanced characteristics of multicore processors.

The following three issues are most serious problems,when a guest OS is implemented in the user level.

1. The user level OS implementation requires heavymodification of the kernel.

2. Emulating an interrupt disabling instruction is veryexpensive if the instruction cannot be replaced.

3. Emulating a device access instruction is very ex-pensive if the instruction cannot be replaced.

In a typical RTOS, both the kernel and application codeare executed in the same address space. Embedded sys-tems have dramatically increased their functionalities inevery new product. To reduce the development cost,the old version of application code should be reusedand extended. The limitation of hardware resources isalways the most important issue to reduce the productcost. Therefore, the application code sometimes usesvery ad-hoc programming styles. For example, appli-cation code running on RTOS usually contains manyprivileged instructions like interrupt disable/enable in-structions to minimize the hardware resources. Also,device drivers may be highly integrated into the appli-cation code. Thus, it is very hard to modify these ap-plications to execute at the user level without changinga significant amount of application code, even if theirsource code is available. Therefore, it is hard to executethe application code and RTOS in the user level with-out violating the requirements described in Section 1.Therefore, executing RTOS is very hard if the proces-sor does not implement the hardware virtualization sup-port. Even if there is a proper hardware virtualizationsupport, we expect that the performance of RTOS andits application code may be significantly degraded. Our

58 • Towards Co-existing of Linux and Real-Time OSes

approach chooses to execute both guest OS kernels anda virtualization layer at the same privileged level. Thisdecision makes the modification of OS kernels minimal,and there is no performance degradation by introducinga virtualization layer. However, the following two issuesare very serious in the approach.

1. Instructions which disable interrupts have seriousimpact on the task dispatching latency of RTOS.

2. There is no spatial protection mechanism amongOS kernels.

The first issue is serious because replacing interrupt dis-able instructions is very hard for RTOS and its applica-tion code as described above. The second issue is alsoa big problem because executing guest OS kernels invirtual address spaces requires significant modificationon them. SPUMONE proposes a technique presented inSection 4 and Section 5 to overcome the problems.

3.2 SPUMONE: A Multicore Processor Based Vir-tualization Layer for Embedded Systems

SPUMONE is a thin software layer for multiplexing asingle physical CPU (pCPU) core into multiple virtualCPU (vCPU) cores [19, 20]. The current target pro-cessor of SPUMONE is the SH4a architecture, whichis very similar to the MIPS architecture, and is adoptedin various Japanese embedded system products. Also,standard Linux and various RTOSes support this proces-sor. The latest version of SPUMONE runs on a singleand multicore SH4a chip. Currently, SMP Linux, TOP-PERS [3], and the L4 [12] are running on SPUMONEas a guest OS.

The basic abstraction of SPUMONE is vCPU as de-picted in Figure 1. In this example, SPUMONE hoststwo guest OSes, Linux and RTOS. Linux has two vC-PUs, vCPU0 and vCPU1. vCPU0 is executed by pCPU0and vCPU1 is executed by pCPU1. RTOS has onevCPU, vCPU2. This is executed by pCPU1. So bothof vCPU1 and vCPU2 are executed on pCPU1. Un-like typical microkernels or VMMs, SPUMONE itselfand guest OS kernels are executed in the privilegedlevel as mentioned in Section 3.1. Since SPUMONEprovides an interface slightly different from the one ofthe underlying processor, we simply modify the source

pCPU0

User space

pCPU1

Kernel space

RTOS

Time multiplexed vCPUs

Kernel mode thread

User mode thread

with its address space

Figure 1: An Overview of SPUMONE

code of guest OS kernels, a method known as para-virtualization. This means that some privileged instruc-tions should be replaced to hypervisor calls, functioncalls to invoke SPUMONE API, but the number of re-placements is very small. Thus, it is very easy to port anew guest OS or to upgrade the version of a guest OSon SPUMONE.

To spatially protect multiple OSes, if it is necessary,SPUMONE may assume that underlying processorssupport the mechanisms to protect physical memoriesused by respective OS like VIRTUS [24]. The approachmay be suitable for enhancing the reliability of guestOSes on SPUMONE without significantly increasingoverhead. In section 5, we propose an alternative novelapproach to use a functionality of multicore processorto realize the spatial protection among guest OSes. Theapproach does not assume that the processor providesan additional hardware support to spatially isolate guestOSes.

SPUMONE does not virtualize peripheral devices be-cause traditional approaches incur significant overheadthat most of embedded systems could not tolerate. InSPUMONE, since device drivers are implemented in thekernel level, they do not need to be modified when thedevice is not shared by multiple OSes.

Multicore processor version of SPUMONE is designedon the distributed model similar to the Multikernel ap-proach [18]. A dedicated instance of SPUMONE is as-signed to each physical core. Therefore, data structuresused in SPUMONE need not to be protected by usingsynchronization mechanisms. This design is chosen inorder to eliminate the unpredictable overhead of syn-chronization among multiple physical cores. This maysimplify the design of SPUMONE.

2011 Linux Symposium • 59

They communicate with each other via the specially al-located shared memory area and the inter-core interrupt(ICI) mechanism. First, a sender stores data on a specificmemory area, then it sends an interrupt to a receiver,and the receiver copies or simply reads the data fromthe shared memory area.

3.2.1 Interrupt/Trap Delivery

Interrupt virtualization is a key feature of SPUMONE.Interrupts are intercepted by SPUMONE before they aredelivered to each guest OS. When SPUMONE receivesan interrupt, it looks up the interrupt destination tableto make a decision to which OS it should be delivered.Traps are also delivered to SPUMONE first, then aredirectly forwarded to the currently executing guest OS.

To allow interrupts to be intercepted by SPUMONE,the interrupt entry point of the guest OSes should notbe registered to hardware directly. The entry point ofeach guest OS must notify SPUMONE via a hypervisorcall to registering their real vector table. An interruptis first examined by the interrupt handler of SPUMONEin which the destination vCPU is determined, and thecorresponding scheduler is invoked. When the interrupttriggers OS switching, all the registers including MMUstate of the current OS are saved into the stack, then theregisters in the stack of the previous OS are restored.Finally, the execution is switched to the entry point ofthe destination OS. The processor initializes the inter-rupt just as if the real interrupt occurred, so the sourcecode of the OS entry points does not need to be changed.

3.2.2 vCPU Scheduling

Multiple guest OSes run by multiplexing a physicalCPU. The execution states of the guest OSes are man-aged by data structures that we call vCPUs. Whenswitching the execution of vCPUs, all the hardware reg-isters are stored into the corresponding register tableof vCPU, and then restored from the table of the nextexecuting vCPU. The mechanism is similar to the pro-cess implementation of a typical OS, however the vCPUsaves the entire processor state, including the privilegedcontrol registers.

The scheduling algorithm of vCPUs is the fixed prioritypreemptive scheduling. When RTOS and Linux share

the same pCPU, the vCPU owned by RTOS would gaina higher priority than the vCPU owned by Linux in or-der to maintain the real-time responsiveness of RTOS.This means that Linux is executed only when the vCPUof RTOS is in an idle state and has no real-time taskto be executed. The process scheduling is left up toOSes so the scheduling model for each OS need not tobe changed. Idle RTOS resumes its execution when itreceives an interrupt. The interrupt to RTOS should pre-empt Linux immediately, even if Linux has disabled theexecution of its interrupt handlers. The details of thisrequirement and the solution for it is described in Sec-tion 4.1.1.

3.2.3 Modifying Guest OS Kernels

Each guest OS is modified to be aware of the exis-tence of the other guest OSes, because hardware re-sources other than the processor are not multiplexed bySPUMONE as described below. Thus those are exclu-sively assigned to each OS by reconfiguring or by mod-ifying their kernels. The following describes how theguest OS kernels are modified in order to run on the topof SPUMONE.

• Interrupt Vector Table Register Instruction: The in-struction registering the address of a vector table isreplaced to notify the address to the interrupt man-ager of SPUMONE. Typically this instruction is in-voked once during the OS initialization.

• Bootstrap: In addition to the features supported bythe single-core SPUMONE, the multicore versionprovides the virtual reset vector device, which isresponsible for resetting the program counter of thevCPU that resides on a different pCPU.

• Physical Memory: A fixed size of physical mem-ory area is assigned to each guest OS. The phys-ical address for the OSes can be simply changedby modifying the configuration files or their sourcecode. Virtualizing the physical memory would in-crease the size of the virtualization layer and thesubstantial performance overhead. In addition, un-like the virtualization layer for enterprise systems,embedded systems need to support a fixed numberof guest OSes. For these reasons we simply assigna fixed amount of physical memory to each guestOS.

60 • Towards Co-existing of Linux and Real-Time OSes

• Idle Instruction: On a real processor, the idle in-struction suspends a processor until it receives aninterrupt. On a virtualized environment, this isused to yield the use of real physical core to anotherOS. We prevent the execution of this instruction byreplacing it with the hypervisor call of SPUMONE.Typically this instruction is located in a specificpart of the kernel, which is fairly easy found andmodified.

• Peripheral Devices: Peripheral devices are as-signed by SPUMONE to each OS exclusively. Thisis done by modifying the configuration of eachOS not to share the same peripherals. We as-sume that most of the devices can be assigned ex-clusively to each OS. This assumption is reason-able because, in embedded systems, multiple guestOSes are usually assigned different functionalitiesand use different physical devices. It usually con-sists of RTOS and GPOS, where RTOS is used forcontrolling special purpose peripherals such as aradio transmitter and some digital signal proces-sors, and GPOS is used for controlling generic de-vices such as various human interaction devicesand storage devices. However some devices cannotbe assigned exclusively to each OS because bothsystems need to share them. For instance, the pro-cessor we used offers only one interrupt controller.Usually a guest OS needs to clear some of its regis-ters during its initialization. In the case of runningon SPUMONE, a guest OS booting after the firstone should be careful not to clear or overwrite thesettings of the guest OS executed first. For exam-ple, we modified the Linux initialization code topreserve the settings done by TOPPERS.

3.2.4 Dynamic Multicore Processor Management

As described in the previous section, SPUMONE en-ables multiplexing of virtual CPUs on physical CPUs.The mapping between pCPUs and vCPUs is dynami-cally changed to balance the tradeoffs among real-timeconstraints, performance and energy consumption. InSPUMONE, a vCPU can be migrated to another coreaccording to the current situation. The mechanism iscalled the vCPU migration mechanism. In SPUMONE,all kernel images are located in the shared memory.Therefore, the vCPU migration mechanism just movesthe register states to manage vCPUs, and the cost of the

Save power when the processor

utilization of all the OS is low

Linux

Let Linux dominate every core

Linux

Balance processor utilization between OSes

Linux

Exclude OS execution with

dedicated processors

Linux

RTOSRTOS

Figure 2: Dynamically Changing the Mapping BetweenVirtual CPUs and Physical CPUs

migration can be reduced significantly. Actually, theround trip time of the vCPU migration in the currentversion of SPUMONE on the RP1 platform3 is about50 µs when a vCPU is moved to anther pCPU and backto the original pCPU.

There are several advantages of our approach. The firstadvantage is to change the mapping between vCPUsand pCPUs to reduce energy consumption. As shownin Figure 2, we assume that a processor offers two p-CPUs. Linux uses two vCPUs and the real-time OS usesone vCPU. When the utilization of RTOS is high, twovCPUs of Linux are mapped on one pCPU (Left Top).When RTOS is stopped, each vCPU of Linux uses a dif-ferent pCPU (Right Top). Also, one pCPU is used bya vCPU of Linux and another pCPU is shared by Linuxand RTOS when the utilization of RTOS is low (RightBelow). Finally, when it is necessary to reduce energyconsumption, all vCPUs run on one pCPU (Left Below).This approach enables us to use very aggressive policiesto balance real-time constraints, performance, and en-ergy consumption.

3The RP1 platform is our current hardware platform that containsa multicore processor. The processor has four SH4 CPUs and theyare communicated with a shared memory. The platform is developedby Hitach and Renesas.

2011 Linux Symposium • 61

Configuration Time OverheadLinux Only 68m 5.9s -

Linux and TOPPERS 69m 3.1s 1.4%

Figure 3: Linux kernel build time

OS(Linux version) Added LoC Removed LoCLinux/SPUMONE(2.6.24.3) 161 8

RTLinux 3.2(2.6.9) 2798 1131RTAI 3.6.2(2.6.19) 5920 163OK Linux (2.6.24) 28149 -

Figure 4: The total number of modified LoC in *.c, *.h,*.S and Makefile

3.2.5 Performance and Engineering Cost

Figure 3 shows the time required to build the Linux ker-nel on native Linux and modified Linux executed on thetop of SPUMONE together with TOPPERS. TOPPERSonly receives the timer interrupts every 1ms, and exe-cutes no other task. The result shows that SPUMONEand TOPPERS impose the overhead of 1.4% to theLinux performance. Note that the overhead includes thecycles consumed by TOPPERS. The result shows thatthe overhead of the existence of SPUMONE to the sys-tem throughput is sufficiently small.

We evaluated the engineering cost of reusing RTOS andGPOS by comparing the number of modified lines ofcode (LoC) in each OS kernel. Figure 4 shows the LoCadded and removed from the original Linux kernels. Wedid not count the lines of device drivers for inter-kernelcommunication because the number of lines will differdepending on how many protocols they support and howcomplex they are. We did not include the LoC of util-ity device drivers provided for communication betweenLinux and RTOS or Linux and servers processes be-cause it depends on how many protocols and how com-plex those are implemented.

The table also shows the modified LoC for RTLinux,RTAI and OK Linux, all of which are previous ap-proaches to support multiple OS environments. Sincewe could not find RTLinux, RTAI, OK Linux for theSH4a processor architecture, we evaluated them devel-oped for the Intel architecture. OK Linux is a Linuxkernel virtualized to run on the L4 microkernel. For OKLinux, we only counted the code added to the archi-tecture dependent directory arch/l4 and include/

asm-l4. The results show that our approach requires

0xf

0xe

0xd

0xc

0xb

0xa

0x9

0x8

0x7

0x6

0x5

0x4

0x3

0x2

0x1

0x0

0xf

0xe

0xd

0xc

0xb

0xa

0x9

0x8

0x7

0x6

0x5

0x4

0x3

0x2

0x1

0x0

(a) Without IPL separation (b) With IPL separation

Interrupt disable Interrupt disable

Interrup enable Interrup enable

Timer IPL

Serial IPL

Timer & Serial IPL

Linux RTOS

Linux

RTOS

Interrupt disable

Interrupt disable

Interrup enable

Interrup enable

Timer & Serial IPL

Serial IPL

Timer IPL

Figure 5: Separating Interrupt Priorities Between GuestOSes

only small modifications to the Linux kernel. The resultshows that the strategy of SPUMONE, virtualizing pro-cessors only, succeeds in reducing the number of mod-ification of guest OSes and to satisfy the requirementsdescribed in Section 1.

4 Real-Time Resource Management inSPUMONE

4.1 Reducing RTOS Dispatch Latency

In order to minimize the dispatch latency of RTOStasks during concurrent activities of Linux on a sin-gle device, we propose the following two techniques inSPUMONE.

4.1.1 Interrupt Priority Level Separation

The first technique is to replace the interrupt enablingand disabling instructions with the hypervisor calls.A typical OS disables all interrupt sources when dis-abling interrupts for the atomic execution. For example,local_irq_enable() of Linux enables all interruptand local_irq_disable() disables all interrupt. Onthe other hand, our approach leverages the interrupt pri-ority mechanism of the processor. The SH4a processorarchitecture provides 16 interrupt priority levels (IPLs).We assign the higher half of the IPLs to RTOS andthe lower half to Linux as shown in Figure 5. WhenLinux tries to block the interrupts, it modifies its inter-rupt mask to the middle priority. RTOS may thereforepreempt Linux even if it is disabling the interrupts. Onthe other hand, when RTOS is running, the interrupts for

62 • Towards Co-existing of Linux and Real-Time OSes

1

10

100

1000

10000

100000

0 20 40 60 80 100 120

Sam

ple

[num

]

Delay [us]

delay

Figure 6: Interrupt dispatch latency of TOPPERS with-out IPL separation

Linux are blocked by the processor. These blocked in-terrupts could be delivered immediately when Linux isdispatched.

The instructions for enabling and disabling inter-rupts are typically provided by the kernel internalAPI like local_irq_enable() and local_irq_

disable(). They are typically coded as inline func-tions or macros in the kernel source code. For Linux,we replace local_irq_enable() with the hypervi-sor call which enables entire level of interrupts andlocal_irq_disable() with the other hypervisor callwhich disables the lower priority interrupts. For RTOS,we replace the API for interrupt enabling with the hy-pervisor call enabling only high priority interrupts, andthe API for interrupt disabling with the other hypervisorcall disabling the entire level of interrupts. Therefore,interrupts assigned to RTOS are immediately deliveredto RTOS, while interrupts assigned to Linux are blockedduring execution of the RTOS. Figure 5 shows the inter-rupt priority levels assignment for each OS, which weused in the evaluation environment.

Figures 6 and 7 show the task dispatch latency of TOP-PERS under two configurations of SPUMONE. In Fig-ure 6, the evaluation result of SPUMONE without theIPL separation executing Linux and TOPPERS is de-picted. Linux executes write() on the file stored on aCompact Flash card repeatedly and TOPPERS measuresthe task dispatch latency of the interrupts from timemanagement unit. In Figure 7, the result of SPUMONEwith the IPL separation is shown. The guest OSes andtheir workloads are the same as the condition used in acase when the IPL separation is not used.

1

10

100

1000

10000

100000

0 20 40 60 80 100 120

Sam

ple

[num

]

Delay [us]

delay

Figure 7: Interrupt dispatch latency of TOPPERS withIPL separation

As these results show, the workload of Linux heavily in-terference with the task dispatch latency of RTOS if theIPL separation is not configured. Therefore we can saythat separating IPL is an effective method to guaranteethe low interrupt dispatch latency of RTOS.

However, the approach assumes that all activities inTOPPERS are processed at the higher priority than theactivities of Linux. The current version of Linux is im-proving real-time capabilities. So, in the near future,some applications that requires to satisfy real-time con-straints will be developed on Linux. In this case, theapproach described here cannot be used. Also, the ap-proach increases the number of modifications of Linux.It is desirable not to replace interrupt enable/disable in-structions in terms of the engineering cost. Therefore,we have developed an alternative method described inthe next section.

4.1.2 Reducing Task Dispatching Latency withvCPU migration

The second technique is based on the vCPU migrationmechanism introduced in Section 3.2.4. The first tech-nique, replacing API for interrupt enabling/disabling re-quires slight but non-trivial modification of Linux. Inaddition, the technique may not work correctly whenthe device drivers or kernel modules are programmedin a bad manner, which enable or disable interruptswith a non-standard way. The second technique ex-ploits the vCPU migration mechanism. Under this tech-nique, SPUMONE migrates a vCPU, which is assigned

2011 Linux Symposium • 63

to Linux and shares the same pCPU with the vCPU ofRTOS, to another pCPU when it traps into kernel mode,or when interrupts are received. In this way, only theuser level code of Linux is executed concurrently on theshared pCPU, which will never change the priority lev-els. Therefore, RTOS may preempt Linux immediatelywithout separating IPL used in the first technique.

4.2 Increasing the Throughput of SMP Linux

Generally speaking, porting OSes to virtualization lay-ers produces semantic gap because the assumptionswhich guest OSes rely on may not be preserved. For ex-ample, OSes assume that they dominate CPU, memory,and storage. In the ordinary environment where OSesrun directly on the real hardware, this assumption istrue. But when virtualization layers execute guest OSes,this assumption is no longer held. CPU and memory areshared by multiple OSes.

The semantic gap produced by virtualization layers cancause some new problems. One of the typical prob-lems is called the Lock Holder Preemption (LHP) prob-lem [11].

The LHP problem occurs when the vCPU of the guestOS is preempted by the virtualization layer duringthe execution of critical sections protected by mutexbased on busy waiting (e.g. spinlock_t, rwlock_tin Linux). Figure 8 depicts the typical scenario ofLHP in SPUMONE. On SPUMONE, the executionof vCPU2 belonging to RTOS is started immediatelyeven if vCPU1 executing Linux is currently runningon the same processor, because the activities of RTOSare scheduled at a higher priority than the activities inLinux. Let us assume that the execution of the Linuxkernel is preempted while the kernel keeps a lock. Inthis case, other vCPUs owned by Linux and running onother pCPUs may wait to acquire the lock via busy wait-ing.

4.2.1 Existing Solutions of LHP

This performance degradation problem caused by LHPis a general one of every virtualization layer. So, thereare existing solutions for solving the problem. Uh-lig, et al. pointed this problem [11]. They also intro-duced the methods to avoid the problem. The methodis named as Delayed Preemption Mechanism (DPM).

DPM is suitable for a virtualization layer based on thepara-virtualization technology because it does not wasteCPU time and can be implemented with a less effort.However, this solution increases the dispatch latency ofguest OSes, making it unsuitable for embedded systemsthat need to satisfy real-time constraints.

VMware ESX employs a scheduling algorithm calledco-scheduling [15] in its vCPU scheduler [16]. Thissolution wastes lots of CPU time. VMware ESX em-ploys the technique because it is the full-virtualizationtechnique. Also, it does not assume to execute mul-tiple vCPUs for a guest OS on one pCPU. Sukwongand Kim introduced the improved co-scheduling algo-rithm named balance scheduling and implemented it onKVM [17].

Wells, et al. introduced a hardware based solution calledspin detection buffer (SDB) for detecting meaninglessspin of vCPUs produced by LHP [13]. They found thatthe execution pattern can be distinguished when a CPUis spinning before acquiring a lock. SDB inspects thenumber of store instructions and counts the number ofupdated memory address if a thread is executed in ker-nel mode. If the number of counted addresses does notexceed the threshold (they set it as 1024), SDB judgesthe thread is spinning in vain. This hardware informa-tion can be used by a virtualization layer to avoid theLHP problem.

Friebel and Biemueller introduced a method for avoid-ing LHP on Xen [14]. Respective threads in the guestOSes count the number of spinning on a busy wait mu-tex. When the count exceeds the threshold, the spinningthread invokes the hypervisor call in order to switch toanother vCPU.

4.2.2 Solving LHP in SPUMONE

The methods described in Section 4.2.1 improve thethroughput of SMP Linux on traditional VMMs. Butall of them assume that there are no real-time activities.

In this section, we propose a new method for avoid-ing LHP. In our approach, the vCPU of Linux, whichshares pCPU with the vCPU of RTOS, is migrated toanother pCPU when an interrupt for RTOS is received.Then, it returns to the original pCPU when RTOS yieldspCPU and becomes idle. When two vCPUs for Linuxare executed on the same pCPU, they also cause the

64 • Towards Co-existing of Linux and Real-Time OSes

pCPU0

User space

pCPU1

Kernel space

RTOS

vCPU2 activated

pCPU0 pCPU1

RTOS

Interrupt from the device

allocated by RTOS

Figure 8: Typical example of Lock Holder Preemption in SPUMONE

1.5

2

2.5

3

3.5

4

4.5

5

5.5

0 10 20 30 40 50 60 70 80 90Sco

re o

f hac

kben

ch [s

econ

d] (

low

er is

bet

ter)

Time consumption by RTOS [%] (500ms periodic)

raw scheduler of SPUMONE2 cores3 cores4 cores

ideal scorevCPU Migration

Figure 9: Result of hackbench on Various Configuration

LHP problem. But, in this case, we assume that the de-layed preemption mechanism can be used since Linuxdoes not have real-time activities. The vCPU migrationmechanism is similar to the thread migration in normalOSes, but in the case of SPUMONE, interrupt assign-ments have to be reconfigured because peripherals de-vices are not virtualized. In our evaluation environment,timer interrupts and ICI should be taken into account.Let us assume that vCPU0 is migrated from pCPU0to pCPU1 while executing an activity on vCPU1. Thetimer device raising interrupts periodically for vCPU0on pCPU1 should be stopped before the vCPU migra-tion. Then, the timer device on pCPU1 should be mul-tiplexed for both vCPU0 and vCPU1. Also, ICI forvCPU0 on pCPU0 should be forwarded to vCPU0.

Figure 9 shows the hackbench score on various config-urations of SPUMONE. In this evaluation environment,four pCPUs execute five vCPUs. Therefore two vCPUshare one pCPU. One vCPU belongs to TOPPERS andfour vCPUs belong to Linux. TOPPERS executes a task

which consumes CPU time in the 500ms period. Linuxexecutes hackbench for measuring its throughput. TheX axis means the CPU consumption rate of the task onTOPPERS, and the Y axis means the score of hack-bench. Three horizontal lines describe the score of hack-bench under the case that Linux dominates pCPUs.

The line indicated as “ideal score” describes the scorewhich we expected at first. When RTOS consumes thetime of f (0 ≤ f < 1) on one pCPU, Linux should ex-ploit the rest of CPU resources: 4− f (When f = 1,which means RTOS never yields pCPU, the rest of CPUresources is not equal to 3. Because this is the same asthe situation when 1 CPU stops execution suddenly fromthe perspective of Linux). The line of the ideal scoreis calculated as: I( f ) = S1

4− f where f means the CPUconsumption rate of RTOS and S1 means the score ofLinux dominating one core. hackbench has enough par-allelism, therefore we predicted that the score might belinear according to the CPU consumption rate of RTOS.

The score actually measured is presented as the lineindicated as “raw scheduler of SPUMONE”. We no-ticed that the rapid degradation of the performance iscaused by LHP. So we designed and implemented thenew method described above. The score measured whenusing the new method is described as the line indicatedas “vCPU migration”. This score is still worse than theideal score, but it sufficiently utilizes the CPU resourcebecause it is better than or nearly equal to the case whenLinux dominates three cores.

Current score when using our new method is still worsethan the ideal score, so more optimization or bettervCPU scheduling policy is required. We are planning toapply the method described in Section 4.1.2. In mod-ern systems, mutexes based on busy wait mechanismare only used in kernel space. Therefore if the vCPU

2011 Linux Symposium • 65

of Linux, shareing pCPU with the vCPU of RTOS, ismigrated to another pCPU when the thread invokes sys-tem calls or the interrupts for Linux rises, LHP can beavoided.

4.3 Real-Time Task Aware Scheduler

One of the ongoing projects of SPUMONE, we planto use these additional resources to further improve thereal-time capability of guest OSes, especially Linux, bydynamically scheduling the vCPU of the guest OSes ontop of the SPUMONE. In the original design strategyof SPUMONE, we gave a high priority to the vCPU ofRTOS which is higher than the priority of the vCPU ofLinux. But this is not always the case; there might ex-ist some real-time processes that have quicker responsetime requirements than the RTOS processes. In this sit-uation, we can mark one of the vCPUs of Linux as rt-vCPU and schedule it against vCPU of RTOS. Whenthe priority of this rt-vCPU is higher than that of thevCPU of RTOS, it can gain the control of the pCPU, butsimultaneously, because we have some other pCPU inmulticore system, we can migrate the vCPU of RTOSto another core and compete with other vCPUs, so theoverall performance will not be harmed too much. Butthe overhead of this migration operation has to be care-fully taken care of.

5 Offering Spatial Protection in SPUMONE

As described in Section 3, SPUMONE locates guest OSkernels and SPUMONE in the same privileged level.However, the Linux kernel might contain security holesbecause of its huge source code, and there are possibili-ties to infect the Linux kernel. For increasing the relia-bility of the entire system, a virtualization layer shouldoffer mechanisms to protect a virtualization later and co-existing RTOSes. In traditional approaches, strict mem-ory isolation is used for the protection, but our approachcannot rely on the traditional solution because it is tooexpensive for typical embedded systems. SPUMONEoffers two mechanism to increase the reliability of theentire system. In the following sections, we will explainthe mechanisms in detail.

5.1 Protecting SPUMONE and RTOS

In the virtualization environment of SPUMONE, guestOS kernels are running side by side with SPUMONE

Processor

Figure 10: Separated Core-Local Memory and Its Ap-plication for Security

in the most privileged level. This means that these ker-nels and SPUMONE could be affected by one and an-other. In order to further improve the security of thesystem, we try to better protect these kernels withoutimplementing too much functionality in SPUMONE.We did so by taking advantage of the distributed designof SPUMONE in the multicore platform [21]. Multi-core design of SPUMONE is different from traditionalVMMs in that each physical core has its own dedi-cated virtualization layer instance, while the traditionalones have only one instance across all available physi-cal cores. We then simply install each SPUMONE in-stance into the local memory area of each physical core.Because the content of local memory is only accessi-ble from its own physical core, the attacks or intrusionsfrom the other cores are therefore prohibited as shownin Figure 10. This also means that the attacks will notpropagate. When one part of the system is broken andtries to affect others, it will not make it. So the remain-ing part of the systems can operate normally.

5.1.1 Core-Local Memory

Let us assume that two OS kernels running on top ofa dual-core processor where each core has an indepen-dent core-local memory. If the following assumptionsare satisfied, an OS kernel is protected from others.

1. The size of an OS kernel is small enough to fit incore-local memory.

2. Each core should be restricted to reset other corewhere the reset cleans up the content of the core-local memory.

66 • Towards Co-existing of Linux and Real-Time OSes

Processor

INVALID

Storage

INVALID

INVALID

0

1

2

3

4

5

6

7

Vulnable

OS Pages

execute

Compromise

Protected

OS Pages

INVALID

Boot

Loader1.

remap

Virtual Address Space

2.

3.

3. 4.

Figure 11: An Overview of Hash-Based Integrity Man-agement

3. The boot image of an OS kernel should not be in-fected, and a secure boot loader can load the kernelimage in the shared memory correctly.

4. Each core should be restricted to access I/O de-vices. I/O devices that are managed by a coreshould not be accessed from other cores.

5.1.2 Hash-Based Integrity Management

The problem of the solution presented in the previoussection is the size of core-local memory. Currently theyare a few hundred KBs. It is too small to load a mod-ern RTOS. In order to virtually extend the size of a lo-cal memory, we propose a hash-based integrity manage-ment mechanism assisted by the core-local memory pro-tection. The original kernel image is stored in the sharedmain memory, and a subset of the kernel image is copiedto the core-local memory before execution by the core.When a part of the kernel image is loaded in the core-local memory, this part is verified every time to makesure that it is not corrupted or infected.

We present how the hash-based integrity managementworks in Figure 11. The page allocation in a core-localmemory and the calculation of cryptographic hash val-ues are managed by the local memory (LMEM) man-ager that resides permanently in the core-local memory.

An OS kernel image that can be protected from otherOS kernels is called a protected OS (pOS). An OS ker-nel that may be infected by malicious activities is calleda vulnerable OS (vOS). pOS and vOS must run on dif-ferent cores.

1. The boot loader loads the LMEM manager into thecore-local memory. The OS kernel images of pOSand vOS are loaded at the same time into the mainmemory. LMEM calculates the hash value of eachpage of pOS, and stores it in a hash table also lo-cated in the core-local memory. The manager loadsa memory page that contains the entry point of pOSinto the core-local memory. Then the other coremay start to execute vOS.

2. The pages of pOS are mapped in a virtual addressspace, and a page table for managing the virtualaddress space should be in the core-local mem-ory. When the size of the page table is biggerthan the size of the core-local memory, LMEM canswap out unused page tables to the shared mem-ory. LMEM also manages the hash table for main-taining the integrity of the swapped page tables.LMEM manages page faults when the page tabledoes not contain a corresponding page table entry.

3. When LMEM handles a page fault, a correspond-ing page is copied from the shared memory to thecore-local memory. LMEM calculates the hashvalue of the page and compares it with the pre-calculated value stored in the core-local memory.A mismatch of the hash value means that the im-age of pOS in the shared memory is corrupted. Ifthere is no mismatch, the page fault is correctlycompleted and the execution of pOS is resumed.

4. When there is no space available in the core-localmemory, LMEM swaps out some pages to theshared memory. LMEM checks whether the pageis updated or not, and if it is updated, LMEM re-calculates the hash value of the page and updatesthe hash table entries. The pages will be used forloading other pages.

In this approach, the image of pOS in the shared mem-ory may be corrupted by vOS. Our current policy is torestart pOS by reloading a new undamaged kernel imageby a secure loader. We are also considering a techniqueto protect a kernel image by using a memory error cor-rection technique and an encryption technique.

2011 Linux Symposium • 67

6 Related Work

Virtualization technologies are already used in the areaof the desktop and data center computing today [4, 5, 6].It is also becoming a strong technique for constructingreal-time embedded systems because of an enhancementof processors targeting embedded systems market.

RTLinux [22] is a well known hard real-time solutionfor Linux, but it is also known with its patent problems.RTAI [23] is another real-time extension for Linux andis distributed as free software, but it requires significantmodification of the source code of Linux.

KVM for ARM [10] is a KVM based lightweight para-virtualization technology for ARM processors. Thismight be a strong candidate of virtualization technologyfor real-time embedded systems because it only requiresautomated modification of the guest Linux.

OKL4 [12] is a hypervisor based on the micro-kerneldesign. Armand and Gien introduced the poorness of itsperformance come from the design of micro-kernel [9].VirtualLogix VLX [9] is a practical designed VMM forreal-time embedded systems.

And in our best knowledge, none of them can handleLHP on multicore processors while guaranteeing real-time responsiveness when the guest OSes have asym-metrical priorities and roles.

7 Conclusion and Future Directions

Before concluding our paper, we would like to share ourexperiences with the difficulties to promote open sourcesoftware in Japanese embedded system industries.

We have been discussing various aspects of open sourcesoftware with embedded system industries for a longtime. We found that a lot of people in industries whoare working on open source software are aware of thethe merits. Especially, asking questions to communitiesis very useful to find good solutions to problems. How-ever, their bosses who were hardware engineers beforedo not understand the merits because in their cases, thesolutions should be solved by themselves inside of theirindustries. The cultural gaps between the generationsinside industries becomes one of the biggest obstructionto work with open source communities.

Now, we already know that social networks have signifi-cantly strong power on sharing knowledge. Open source

communities are kind of social networks to share knowl-edge about open source software, and engineers who askquestions to communities also need to answer questionof other people in the communities, but it sometimes toodifficult to make time for discussing open source com-munities while they are working.

Also, it is not easy that embedded system industries con-tribute their software on open source communities be-cause they sometimes use the old version of software.Because the modification of the old version is not easy tobe integrated into the current version of the open sourcesoftware.

In this paper, we introduced SPUMONE that is a virtu-alization layer for multicore processor based embeddedsystems. We described an overview of SPUMONE andshowed how SPUMONE reduces the RTOS dispatch la-tency and protects SPUMONE and RTOS from mali-cious attacks on the Linux kernel. Currently, we arepreparing to distribute SPUMONE as open source soft-ware.

References

[1] Tatsuo Nakajima, Yuki Kinebuchi, Hiromasa Shi-mada, Alexandre Courbot, Tsung-Han Lin. Tempo-ral and Spatial Isolation in a Virtualization Layerfor Multi-core Processor based Information Appli-ances. In Proceedings of the 16th Asia and SouthPacific Design Automation Conference, 2011.

[2] eCos. http://ecos.sourceware.org

[3] TOPPERS Project. http://www.toppers.jp/en/index.html

[4] Paul Barham, Boris Dragovic, Keir Fraser, StevenHand, Tim Harris, Alex Ho, Rolf Neugebauer, IanPratt, Andrew Warfield. Xen and the art of virtual-ization. In Proceedings of the nineteenth ACM sym-posium on Operating systems principles, 2003.

[5] VMware. http://www.vmware.com

[6] Avi Kivity, Yaniv Kamay, Dor Laor, Uri LublinQumranet, Anthony Liguori. kvm: the Kernel-based Virtual Machine. In Proceedings of OttawaLinux Symposium, 2007.

[7] Ingo Molnar. RT-patch. http://www.kernel.org/pub/linux/kernel/projects/rt/

68 • Towards Co-existing of Linux and Real-Time OSes

[8] Paul E McKenney. ’Real Time’ vs. ’Real Fast’:How to Choose? In Proceedings of the OttawaLinux Symposium, 2008.

[9] François Armand and Michel Gien. A PracticalLook at Micro-Kernels and Virtual Machine Mon-itors. In Proceedings of the 6th IEEE Conference onConsumer Communications and Networking Con-ference, 2009.

[10] Christoffer Dall and Jason Nieh. KVM for ARM.In Proceedings of Ottawa Linux Symposium, 2010.

[11] Volkmar Uhlig, Joshua LeVasseur, EspenSkoglund, and Uwe Dannowski. Towards ScalableMultiprocessor Virtual Machines. In VM’04: Pro-ceedings of the 3rd conference on Virtual MachineResearch And Technology Symposium

[12] Open Kernel Labs. OKL4 Microvisor.http://www.ok-labs.com/products/okl4-microvisor

[13] Philip M. Wells, Koushik Chakraborty, andGurindar S. Sohi. Hardware Support for Spin Man-agement in Overcommitted Virtual Machines. InProc. of the 15th International Conference onParallel Architectures and Compilation Techniques(PACT-2006), Sept. 2006, Seattle, WA

[14] Thomas Friebel and Sebastian Biemueller. Howto Deal with Lock Holder Preemption. http://www.amd64.org/fileadmin/user_upload/pub/2008-Friebel-LHP-GI_OS.pdf

[15] J. K. Ousterhout. Scheduling Techniques for Con-current Systems. Proceedings of Third Interna-tional Conference on Distributed Computing Sys-tems, 1982

[16] VMware, Inc. VMware vSphere(TM) 4: TheCPU Scheduler. in VMware(R) ESX(TM) 4http://www.vmware.com/files/pdf/perf-vsphere-cpu_scheduler.pdf

[17] Orathai Sukwong and Hyong S. Kim. Is Co-scheduling Too Expensive for SMP VMs? In Pro-ceedings of the ACM European conference on Com-puter systems, 2011.

[18] Andrew Baumann, Paul Barham, Pierre-EvaristeDagand, Tim Harris Rebecca Isaacs, Simon Pe-ter Timothy Roscoe, Adiran Schüpbach, Akhilesh

Singhania. The multikernel: a new OS architecturefor scalable multicore systems. In SOSP ’09: Pro-ceedings of the ACM SIGOPS 22nd Symposium onOperating Systems Principles, 2009.

[19] Yuki Kinebuchi, Takushi Morita, Kazuo Maki-jima, Midori Sugaya, Tatsuo Nakajima. Construct-ing a Multi-OS Platform with Minimal EngineeringCost. In proceedings of Analysis, Architectures andModelling of Embedded Systems, 2009.

[20] Tatsuo Nakajima, Yuki Kinebuchi, AlexandreCourbot, Hiromasa Shimada, Tsung-Han Lin, Hi-toshi Mitake. Composition kernel: a multi-core pro-cessor virtualization layer for rich functional smartproducts. In Proceedings of the 8th IFIP WG 10.2international conference on Software technologiesfor embedded and ubiquitous systems, 2010.

[21] Tsung-Han Lin, Yuki Kinebuchi, Alexandre Cour-bot, Hiromasa Shimada, Takushi Morita, Hi-toshi Mitake, Chen-Yi Lee and Tatsuo Naka-jima. Hardware-assisted Reliability Enhancementfor Embedded Multicore Virtualization Design. Inthe Proceedings of 14th IEEE International Sympo-sium on Object/Component/Service-oriented Real-time Distributed Computing, 2011.

[22] Victor Yodaiken. The RTLinux Manifesto. In theProceedings of the 5th Linux Expo, March 1999, inRaleigh North Carolina

[23] P. Mantegazza, E. L. Dozio, S. Papacharalambous.RTAI: Real Time Application Interface. In LinuxJournal, volume 2000. Specialized Systems Consul-tants, Inc. Seattle, WA, USA, 2000.

[24] Hiroaki Inoue, Junji Sakai, Masato Edahiro. Pro-cessor virtualization for secure mobile terminals. InACM Transactions on Design Automation of Elec-tronic Systems, Volume 13 Issue 3, July 2008.