Design and Implementation of a Safe, Reflective

Middleware Framework

Nalini Venkatasubramanian, Sebastian Gutierrez-Nolasco,

Shivjit Mohapatra & Jehan Wickramasuriya

Dept. of Information & Computer Science

University of California, Irvine

Irvine, CA 92697-3425, USA

{nalini,seguti,mopy,jwickram}@ics.uci.edu

Abstract

With the rapid evolution of the global information infrastructure and ubiquitous com-puting environments, service providers will need to provide effective and adaptive resourcemanagement mechanisms that can serve concurrent applications in the presence of chang-ing system conditions. Flexible, scalable and customizable middleware can be used asenabling technology for next generation systems that can satisfy application requirements(real-time, security, mobility, fault tolerance etc.) executing in highly dynamic distributedenvironments. To enable application aware resource management, we are developing acustomizable and safe middleware framework called CompOSE|Q based on a reflectivemetamodel. In this paper, we motivate the need for safe composability of concurrentlyexecuting middleware that can interact and interfere with each other. We describe thearchitecture and runtime environment for CompOSE|Q and assess the performance over-head of the additional flexibility. We also illustrate how application requirements such assecurity and timeliness can be supported efficiently in the CompOSE|Q framework.

Keywords: Actors, Distributed systems, Meta-Architecture, Middleware, Reflection,Composability.

1 Introduction

In the near future, large scale distributed systems must handle applications that are open and

interactive. Applications over the digital wireless Internet such as smart houses, intelligent

transportation systems, mobile information appliances such as palmtops running Web-based

multimedia applications require dynamic reconfiguration of systems, services and protocols.

1

Open Distributed Systems (ODS) allow services, end-systems and protocols to dynamically

attach to and detach from the distributed environment; such systems evolve dynamically and

their components interact with an environment that is not under their control. Increasing levels

of distribution in information access, openness and dynamism results in increasing complex-

ity. Middleware is the fundamental infrastructure that enables distributed computing. Flexible

middleware platforms incorporate the notion of reflection in order to provide the desired level

of configurability and openness in a controlled manner [35]. Yet, there exist several challenges

in delivering a safe plug-and-play middleware environment where protocols and services are

installed and uninstalled on the fly to accommodate dynamicity. Firstly, distributed appli-

cations have varying requirements, often stated as QoS (Quality of Service) parameters that

define the extent to which performance specifications such as responsiveness, reliability, avail-

ability, cost-effective utilization and security may be violated. These application requirements

can be satisfied using appropriate resource management policies, e.g. replication, migration,

checkpointing. Satisfying application QoS in a dynamic environment implies dynamically cus-

tomizing the underlying management mechanisms to meet QoS requirements.

A prime issue in the execution of middleware entities is that of safe composability. During

the concurrent execution of multiple distributed resource management services and protocols,

complications arise in the dynamic customization of these services and protocols. Sometimes

the interactions can be complex and rapidly changing, causing violations in correctness and

alteration of the semantics of applications built above the generic distributed management in-

frastructures. Much of this has been ignored in existing commercial frameworks and distributed

middleware infrastructures. Frameworks such as CORBA and DCOM represent a step toward

compositional software architectures but do not deal with interactions of multiple object ser-

vices executing at the same time, and the implication of composing object services. Building a

QoS-enabled customizable middleware framework requires characterizing and reasoning about

the interactions between multiple resource management activities in ODSs, their dynamic in-

stallation and customization in the presence of QoS requirements.

In this paper, we describe the design and implementation a QoS-enabled reflective middle-

ware framework called CompOSE|Q currently being developed at the University of California,

Irvine. CompOSE|Q is based on a meta-architectural model that facilitates specifying and

2

reasoning about the composability of multiple resource management services in ODSs. We

illustrate how effective mechanisms can be incorporated into a composable middleware frame-

work to ensure safety and QoS enforcement in distributed environments. The rest of this paper

is organized as follows. We begin by briefly describing the TLAM, a formal semantic model

for reasoning about middleware in Section 2. Section 3 discusses the basic CompOSE|Q ar-

chitecture and metalevel services. Section 4 addresses the implementation of the CompOSE|Qsystem and presents initial performance results. Related work is discussed in Section 5. We

conclude in Section 6 with future research directions.

2 The Two Level Meta Architectural Model (TLAM):

A Formal Model for Reflective Middleware

To simplify development of applications, we use Actors [1], a model of concurrent active objects

that has a built-in notion of encapsulation and interaction among the concurrent components

of an ODS. In the actor paradigm, the universe contains computational agents called actors,

distributed over a network. Traditional passive objects encapsulate state and a set of procedures

that manipulate the state; actors extend this by encapsulating a thread of control as well. Each

actor potentially executes in parallel with other actors and may communicate with other actors

via asynchronous message passing. Using Actors, we define a meta-architecture framework that

permits customization of resource management mechanisms such as placement, scheduling and

synchronization.

Ensuring correctness in a purely reflective model involves reasoning about system level inter-

actions by characterizing the semantics of shared distributed resources and understanding what

correctness of the overall system means. The TLAM (Two Level Actor Machine) model [46] is

a first step towards providing a formal semantics for specifying and reasoning about properties

of and interactions between components of ODSs. In the TLAM, a system is composed of two

kinds of actors, base actors and meta actors, distributed over a network of processing nodes.

Base level actors carry out application level computation, while meta-actors are part of the

runtime system, which manages system resources and controls the runtime behavior of the base

level. Meta-actors communicate with each other via message passing as do base level actors,

3

but they may also examine and modify the state of the base actors located on the same node.

The TLAM uses reification (base object state as data at the meta object level) and reflec-

tion (modification of base object state by meta objects) with support for implicit invocation of

meta objects in response to changes of base level state. The TLAM provides for full actor-style

interaction of meta level objects; we are currently looking into providing restricted support

for the explicit invocation of meta objects by base objects. Our earlier work has focused on

defining a rigorous mathematical semantics and on using these semantics to develop concepts

and methods for expressing and reasoning about properties of middleware components and

their interactions. In the TLAM model, meta-level controllers define protocols and mecha-

nisms that customize various aspects of distributed systems management. In practice, multiple

system and application activities occur concurrently in a distributed system, e.g. scheduling,

protocol processing, stream synchronization etc., and can therefore interfere with each other.

Composing multiple resource management mechanisms leads to complex interactions. Consider

the following example of a system where distributed garbage collection and process migration

can proceed concurrently. If processes in migration (transit) are not accounted for, the garbage

collection process can potentially destroy accessible information. Similarly, if the process is con-

tinuously migrating, the garbage collection process runs the risk of potential non-termination.

In general, risks that arise due to mechanism composition include loss of information, possible

non-terminations that cause deadlocks and livelocks, dangling resources, inconsistencies, and

incorrect execution semantics.

One approach to deal with interference during mechanism composition in an open system

is to serialize or delay activities to ensure overall safety of the system. However, this can

result in over-serialization of resource management activities causing performance degradation.

Furthermore, global delays and halts may cause violations of timing based QoS constraints.

We also argue that composability of resource management activities is not just desirable, but

essential to ensure cost-effective QoS in distributed systems. For instance, system protocols

and activities must not enforce arbitrary delays in the presence of timing based QoS constraints.

4

3 The CompOSE|Q Architecture

Based on the two level meta-architecture, we are developing a customizable and safe distributed

systems middleware infrastructure, called CompOSE|Q (Composable Open Software Environ-

ment with QoS) that has the ability to provide cost-effective QoS-based distributed resource

management. CompOSE|Q provides composable distributed resource management, i.e., it allows

the concurrent execution of multiple resource management policies in a distributed system in a

safe and correct manner. This will allow safe integration of resource management mechanisms

for services such as mobility, load balancing, fault tolerance and end-to-end QoS management.

The CompOSE|Q architecture, as illustrated in Fig. 1 contains:

• Modules that implement the four basic core services - remote creation, dissemination services,

distributed snapshot and directory services with interaction constraints that ensure their con-

current execution with each other and other metalevel services.

• Common object services built using core services - actor migration, replication of services and

data, actor scheduling, distributed garbage collection, name services, customized communica-

tion etc. Each of these services has its own interface definitions and interaction constraints.

• Components that provide middleware services - real time, security, fault tolerance and mo-

bility. These components leverage from common object services and allow to customize and

adapt their machanisms and policies.

• A QoS Brokerage Service enforces (application) QoS requirements by selecting a subset of

middleware services, such as security and real time, that provide the required QoS in a safe

and correct manner .

3.1 Core Services

To ensure non-interference and manage the complexity of reasoning about components of ODSs

in general, our strategy is to identify key system services where non-trivial interactions between

the application and system occur, i.e. base-meta interactions. We refer to these key services

as core services. Core services are used in specifying and implementing more complex ac-

tivities within the framework as purely meta-level interactions. The development of suitable

non-interference requirements allows us to reason about the composition of multiple system

services; these services have constraints that must be obeyed to maintain composability (i.e.

5

Fig. 1: CompOSE|Q high-level architecture.

safe concurrent execution). Examples of such non-interference requirements have been spelled

out in [46, 38]. We use commonly observed patterns in distributed systems services to extract

implementable abstractions and identify four metalevel core activities:

• Remote Creation: Recreation of services/data at a remote site. Remote creation can be

used as the basis for designing algorithms for activities such as migration, replication and load

balancing.

• Distributed Snapshot: Capturing information at multiple nodes/sites used as a basis for

checkpointing, distributed garbage collection [41], etc.

• Distributed Dissemination: Disseminating information at multiple nodes/sites used as a basis

for common services that require different degrees of consistency or synchronization such as

replication consistency, clock synchronization, key management, etc.

• Directory Services: Interactions with a global repository. Directory services can be used to

provide access control, resource discovery and implement group communication protocols.

Remote Creation: Remote creation is the process by which actor creation occurs on a speci-

fied node other than the node from which creation is being initiated. Remote creation is a basic

facility that can be used in other resource management activities like load-balancing, repli-

cation and migration. By encapsulating the interactions between the application and system

level actors within the remote creation service, we can state requirements that ensure safe and

correct composition of other resource management activities with remote creation. In a real

6

TLAM based implementation, the control activities of remote creation are managed by remote

creation meta-actors (RCM) residing on every node in the system. A remote creation request

has four parameters - a description of the fragment desc to be migrated, the remote node (N),

any initial state the desc has to be set to and the initiating-actor β. The initiator actor β

is maintained by the RCM to ensure composability with other meta-level services [38]. If the

requester needs to know that the request has been completed, or the names of some of the

newly created actors, then this can be achieved by specifying appropriate messages as part of

the requested fragment, and observing their delivery.

Distributed Snapshot Services: Global properties like the number of application-actors,

the current reachability graph of distributed actors, number of messages being processed and

task queue sizes help in making runtime decisions like load balancing, migration and garbage

collection, leading to efficient runtime management of a distributed system. To fully represent

the global state of the distributed system, we need a mechanism for recording the state of all

nodes including the portion of node state being communicated in the network channels [6].

As state information is accessible explicitly only in nodes, a snapshot mechanism must ensure

that node state information in channels are recorded at some node in the system (possibly

the target node itself). The snapshot mechanisms we have devised are such that application-

level computation and system level services proceed concurrently with the snapshot, thereby

preserving application and service semantics. In order to initiate snapshot recording on every

node and force messages in channels to reach a node, we have defined two wave protocols for

message propagation (described below). The starting and finishing points of both waves, the

propagation path and constraints on message propagation are defined elsewhere [38]. Termina-

tion is signaled when the wave is complete, serving as a synchronization point in the snapshot

mechanism. Note that this generalizes the notions of wave and global snapshot discussed in [12]

where the snapshots explicitly do not account for information contained in messages in transit.

Distributed Dissemination: Since delivery latencies can vary significantly in a distributed

system and some components may require different levels of consistency or synchronization,

we have defined mechanisms to propagate information that (1) enforce an adequate level of

synchronization (if needed), (2) help to maintain information consistency and (3) coordinate

7

message propagation used by other core services such as the distributed snapshot service. For

instance, the wave protocols; (a) visit all nodes exactly once, capturing node-resident informa-

tion and (b) traverse all links in the system exactly once forcing messages on channels to reach

nodes (where their state can be recorded).

Directory Services: The challenge here is allowing the middleware to interact with the di-

rectory service in a flexible and efficient manner, enabling developers to effectively create a

variety of directory-enabled applications quickly and easily. This also helps de-couple context

or policy from the application logic, and allows easier handling of interactions between a variety

of general purpose services within the middleware (as oppose to analysis on a per-application

basis). Directory services play an important role in middleware frameworks, and are utilized

in a number of different areas in the CompOSE|Q architecture; (1) resource discovery for load

management and admission control; (2) access control information; (3) system information col-

lection and update and other information that may be used by the QoS brokerage service to

facilitate the use of various middleware services.

3.2 Common Object Services

Implementation of sophisticated policies and mechanisms for QoS management is made possi-

ble by providing support for common services in CompOSE|Q. For instance, object scheduling

mechanisms use the basic remote creation core service to assign newly created objects/actors

on nodes with adequate resources. Using generalized state capture facilities, we are developing

a checkpointing service for capturing causal orders of executions in the system that can be used

for monitoring and debugging distributed computations. A state broadcast mechanism is used

to implement a clock synchronization service, which informs nodes about a global time value

that can be used for time related services.

Migration: We now illustrate how a migration service can be built using the remote cre-

ation core service. By using RC as the basis for migration, we have ensured composability

of migration with other meta-level services [38] such as reachability snapshots and distributed

checkpointing. Migration is the process by which actors move from one node to another. The

migration service allows for relocation of actors for easier access, availability and load balancing.

8

In this section, we describe a generalized migration service and it’s potential implementation

using the remote creation (RC) specification. A migration request is given by a pair (α,ν),

where α is the actor to be migrated, and ν is the destination node. This is interpreted as a

request to move the computation carried out by α to the node ν. In order to state explicitly

invariants maintained by the system during the migration process, we classify the migration

process into 3 phases, associating to each phase the configuration in which it begins. The first

phase is the initiation phase and specifies the state of the system when the migration request

received can be processed. It determines the computation to be migrated by suspending the

computation of the actor and noting its current description. In the second configuration the

actual actor migration is performed using the RC service. The last configuration finalizes the

migration process and establishes transparent access to the migrated actor.

Customized Communication: Although middleware platforms provide some notion of QoS

specification [51], QoS enforcement [47] and mechanisms to enhance application behavior trans-

parently at run-time [29], they do not deal with flexible and safe composition of communication

services. As a result, several (implementation oriented) communication frameworks are not well

suited in distributed and highly dynamic asynchronous environments, where the communication

framework must be able to automatically reconfigure itself in order to respond to application

requirements and/or changes in the environment by adding, removing or composing commu-

nication services dynamically. A communication service is a mechanism that ensures certain

desirable properties about the communication between two or more distributed objects, such as

security or reliability. Communication services are composed to obtain their combined benefits.

However, their service guarantees may crucially depend on the composition order, and a seman-

tic approach to reason about the correct composition of communication services is required.

Furthermore, such composition must be constrained in order to prevent functional interference

with other services of the system that could lead to an inconsistent configuration state, which

may violate the semantics of the basic primitives (safe flexibility). Safe flexibility is required to

protect the system from security threats and failures while ensuring high performance; policies

to enforce this are often hard-wired across different parts of the system.

In order to provide correct composition of communication services to QoS-based applications

in a transparent and scalable fashion, while ensuring correctness of basic middleware services in

9

a meta level architecture for distributed resource management (e.g. garbage collection, remote

creation), the TLAM model is extended with a composable reflective communication framework

(RCF), which customizes the base level communication services among a group of objects as

shown in Fig. 2. Each base level actor has a meta level actor, called messenger, which serves

as the customized and transparent mail queue for that base level actor. There is one commu-

nication manager in every node of the distributed system, which implements and controls the

correct composition of communication services specified by the messenger. The messenger has

four message queues: the up and down queues are used to communicate with its base level actor,

serving as the actor’s send buffer and customized mail queue respectively, while the in and out

queues are used for interaction with the communication manager, requesting communication

services that satisfy QoS constraints. The up and down queues hold raw messages from and

to base level actors, while the out and in queues hold processed messages, which are messages

with the required protocols enforced. Furthermore, the communication manager has a set of

communication protocol actors, each of them implementing a particular communication service

provided by the framework (e.g. reliable protocol, in-order protocol). Communication services

can be added (plugged in) or removed (plugged out) dynamically without side effects. The

above scheme allows us to abstract a core set of communication services and share it between

the different messengers on a node, simplifying the synchronization and composition process,

while encouraging separation of concerns in the process of message transmission and recep-

tion. In order to maintain accurate semantics and provide an efficient implementation of the

architecture, the communication manager implements a set of meta level representatives, called

pool-actors. At any instance, the pool-actor handles the communication services requested by a

messenger for an individual message. In other words, every message requiring communication

services is assigned a pool actor. The pool actor ensures the correct order of composition of

required services and provides a coordination mechanism between the messenger that requires

the services and the protocols that provide it. This concept of reusable pool-actors is an effi-

cient way to handle the service request of each messenger without having to pay the bottleneck

associated with the centralization of the services in the node communication manager.

10

Fig. 2: The RCF model: Each base level actor has a meta level entity; its messenger that interactswith the node communication manager.

3.3 Middleware Services

Middleware provides a standardized way to address interoperability, development, and runtime

support for a wide variety of distributed applications in heterogeneous environments. We pro-

vide a selection of middleware services as part of CompOSE|Q that allow for safe, customizable

operation as well as enabling the implementation of sophisticated policies and mechanisms for

resouce management. Here we present two examples of such services we have developed for

the CompOSE|Q framework, and illustrate some compositional issues between them from the

point of view of the QoS brokerage service. Each of these services addresses a broad area in

the middleware framework, but some specific applications are provided as case studies.

3.3.1 Supporting Real-time in CompOSE|Q

Flexible scheduling mechanisms are critical to the design of customizable and reflective mid-

dleware frameworks. In future, scheduling services have to be customizable and should be able

to dynamically adapt to diverse/interchangeable services (implementing different policies) and

dissimilar workloads at different times. Moreover, systems currently deployed in distributed

environments (e.g. Windows, UNIX etc.) support best-effort, priority schedulers that provide

minimal support for meeting timeliness and dependability requirements of tasks. The growing

popularity of soft real-time payloads with quality of service demands makes the design of ef-

11

fective middleware scheduling services absolutely essential. The overall goals for a distributed

scheduler are to achieve predictable and guaranteed operations for individual tasks and pro-

vide timeliness and real-time guarantees for real-time applications. In addition, the scheduler

should be able to dynamically adapt to changing system and network conditions and improve

the overall performance of the entire distributed system.

The adaptive scheduler in CompOSE|Q extends the notion of “task scheduling” to include

scheduling of tasks within individual nodes (peripheral scheduling) and scheduling of tasks

across nodes (focal scheduling). Soft real-time guarantees are met by exploiting the priority

based native scheduler, to emulate a real-time scheduler. Furthermore, individual messages are

assigned priorities and a conjunctive (task + message) priority scheme is employed to make

scheduling decisions. This innovation enables us to fine tune our scheduler to provide both

coarse and fine grained scheduling. Any customizations made to the focal scheduler would

result in a system-wide change in the scheduling behavior (coarse-grained scheduling); changes

made to the peripheral schedulers exploits the capacity and resource availability of individual

nodes, providing a more fine-grained control over the scheduling.

Case Study: Customizable Scheduling

Within the CompOSE|Q framework, actors communicate with each other via asynchronous

messaging. A single actor can send and receive messages from any number of independent

actors, running either on the same or on different machines. In the absence of our customizable

scheduler, the messages are processed in the order of their arrival at the destination actor and

all messages are assumed to carry the same importance. To provide a more fine grained notion

of priority we introduce priorities into individual messages in the system. The different actors

have priorities assigned to them when they start out and the actor priorities are independent of

the individual message priorities. The scheduling framework provides application actors meth-

ods for setting message priorities and also monitors the assignments so that they conform to

the acceptable priority levels. With the knowledge of priorities of the individual messages and

the destination actors, we combine the message and task priorities to come up with unique

task priority assignments that can be used to optimally schedule tasks (both real-time and

non real-time), delivering a stricter notion of priority scheduling. This approach significantly

12

changes the way we provide adaptability and customizations and introduces a new perspec-

tive to scheduling in distributed environments. The scheduling framework tries to achieve the

overall scheduling goals described above, within a distributed framework that employs active

objects and the messaging based communication paradigm. The model is designed as a two-tier

architecture, with the following two primary components (see Fig. 3):

The Focal Meta Scheduler: The focal scheduler is a system wide global scheduling en-

tity, that is responsible for high level decisions and adaptation that ultimately determine the

scheduling characteristics of the entire distributed system. It schedules the application tasks

onto the various nodes of the distributed system based on overall load and resource availability.

Furthermore, it assigns the various scheduling policies and system wide that invariants that

need to be installed on the distributed nodes.

The Peripheral Meta Schedulers: The peripheral or local meta schedulers are system level

entities that run on the individual nodes (as a part of the node runtime) of the distributed sys-

tem. They schedule the independent tasks on that node and are also responsible for realizing

the conjunctive actor and message scheduling mechanism. The local meta scheduler is logically

divided into a two sub-schedulers for scheduling of the real-time and the non real-time tasks

respectively.

The CompOSE|Q DS forms a communication channel between the focal and the peripheral

schedulers. The focal scheduler commits the policy and other decisions into the DS, which is

used by the peripheral schedulers. Also, the peripheral schedulers periodically update their

state information(system load, current task pool etc.) in the DS as a feedback to the focal

scheduler.

The peripheral scheduler comprises of a soft realtime sub-scheduler (ASRT: Adaptive Soft

Realtime Scheduler) that attempts to provide soft real-time guarantees to such tasks, while

still ensuring that the normal time-sharing tasks within the system do not starve. The native

operating system task priorities are carefully exploited to enforce the correct order in which the

real-time tasks need to be executed. Moreover, a schedule is calculated so that each task gets

adequate resources to finish execution within its specified deadline. Both the Earliest Deadline

First (EDF) and the Rate Monotonic Scheduling (RMS) algorithms are supported and the

13

Monitor

Adapt

ASRT Monitor

Adapt

ASRT

SchedulerNative

SchedulerNative

SchedulerFocal

UDLLFEDFRM

SchedulerPeripheralScheduler

Peripheral

DISTR

BUTED

I

S D

Monitor

Adapt

PolicySelection

Fig. 3: Two Level Meta Scheduler in Compose

ACTOR1

ACTOR2

ACTOR3

ReceivePot

SMAidPriority

Postman

SCHEDULER

BROKER DISP.

1. PeekUpdate Sch

IncreasePriority

Base+Boost(3)

Re-adjustPriority

Base+Boost(1)

level 0 msglevel 2 msg

Fig. 4: Conjunctive Scheduling in CompOSE|Q

adaptation module provides for dynamic switching between the above scheduling algorithms

based on certain predetermined constraints. The real-time sub scheduler always runs at the

highest possible fixed priority and only one real-time task runs at the second highest fixed

priority at any time. The scheduler periodically wakes up and dispatches the next the task

on the schedule. If there is no real-time task waiting, the scheduler suspends itself letting the

native scheduler dispatch one of the timesharing tasks. If there are some real-time tasks waiting

to be scheduled, the PMS changes the current active real-time task, from the running state to

waiting state, and then dispatches another real-time task from the waiting pool based on the

pre-calculated schedule.

The various components of the two level meta scheduler are shown in Fig. 3. Both the

focal and peripheral schedulers have monitoring and adaptation modules built into them. The

monitoring module checks for task compliance. For example, it should be able to detect and

stop a rogue real-time task that does not comply with its resource requirement specification and

tries to hog system resources. In the context of the global scheduler, the monitoring module

systematically observes the transient system load and resource availability. The adaptation

module makes use of this information to make intelligent adjustments that could improve the

overall system performance. The local adaptation module adapts the peripheral scheduler to

changes in the local load and resource availability in a node. Each peripheral scheduler can

be logically considered to have two components for scheduling the real-time and the non real-

time tasks. The proposed architecture handles real time tasks differently from the time-sharing

tasks, and it is therefore reasonable to separate them into different parts.

14

3.3.2 Supporting Security in CompOSE|Q

Security is an obvious concern in distributed environments, however in an object-based archi-

tecture where there exists a high degree of concurrency and mobility, the ability to provide

fine-grained control over the objects in the system at the middleware layer is desirable. As well

as providing security at the application level, access control mechanisms are needed to prevent

service and content providers, as well as unauthorized personnel gaining access to client data

and resources. The dynamic nature and semantic diversity of object-oriented systems makes

it difficult to ensure information security and integrity by the use of traditional access control

mechanisms at either the operating systems or application layer. In addition, session-layer secu-

rity is also an important factor in providing security at the middleware layer. Video streaming

is a typical example, where the ability to map user requirements into security mechanisms and

have those translate to specific protocols while maintaining QoS requirements is a challenge

- particularly with the introduction of real-time constraints. Furthermore, maintaining these

security mechanisms across administrative boundaries (or domains) in the presence of mobility

is an even greater challenge. In corporate networks, firewalls are utilized to filter out ’unwanted’

traffic from the outside world. However, this type of filtering is generally not powerful enough to

detect malicious or anomalous data that can potentially trigger security attacks in OO systems

[16]. We utilize this notion of filtering as well as the concept of partitioning the network into a

set of federated domains (security domains) in our access control framework. Our general focus

is to provide a selection of customizable and safe security solutions at the middleware layer.

Mobility induced by the application or the underlying distributed computing environment

further complicates the management of access rights within domains. The underlying resource

management system can initiate migration of data and computation for load balancing, resource

sharing and localized access to information thereby improving the overall performance and re-

liability of the distributed application. Applications that execute on mobile hosts may hold

classified information and execute classified computation while the mobile hosts enter and leave

domains with varying levels of security. Thus, a target host (receiver of a message) must verify

the access rights of messages received by the transport framework to application components

within the (possibly new) host domain. The target host must also ensure that downloaded exe-

cutable content received in messages (remote programs, applets) do not violate access rights of

15

local data and resources. Similarly the sender of a message must ensure that the recipient of the

message is still within a valid domain to receive information and services contained within the

message. We present an outline of our access control framework in the context of CompOSE|Qand briefly describe domain-based access control (DBAC). In addition, the system supports

techniques for ensuring security in highly mobile environments (both host and object mobility)

as well as mechanisms for delegation and revocation of access rights, details of which can be

found in [48].

Case Study: An Access Control Framework for Mobile, Concurrent Objects

Our reflective model for access control utilizes a meta-level architecture for both enforcement

and interpretation of security policy in the system. This meta-level service manages access

rights for base actors and ensures that the capablities associated with actors accurately reflect

the desired security policies. These management tasks are performed at the meta-level where

meta-actors on their respective nodes carry out access control duties, and are coordinated

(when necessary) via a global security policy. We model application components as mobile

concurrent objects (actors) that are distributed over nodes in the system and provide an access

control framework to give us fine-grained control over what these objects can access under

varying conditions. Here, we assume the communication layer provides the protocols necessary

to establish a secure foundation. In other words, messages cannot be stolen, or tampered with

in transit. The primary component in the architecture is the Meta-level Security Actor (MSA),

which is resident on each node and is responsible for handling messages destined to actors

residing on that node. The global MSA co-ordinates and maintains information that is needed

by the local MSAs, and encapsulates global policies for maintaining system-wide security as well

as handling the secure bootstrapping of the capability objects (classes). However, local access

control enforcement is carried out by MSAs resident on each node, and to a large extent is

independent of the global MSA. An outline of the architecture is depicted in Fig. 5. Finally the

directory service, which is a trusted store of information; includes the access rights information

for each actor in the system, as well as corresponding domain and location information which

is pertinent to the MSAs when carrying out enforcement between nodes.

The state information corresponding to the access rights for objects in the system can only

16

Fig. 5: The AC Meta-architecture: Interaction be-tween the local MSAs with objects on a node (bothtagging and verification of messages), as well as thecoordination duties performed by the global MSA.

Fig. 6: Modelling Domains: Illustrates the interac-tion between various domains in a dynamic mobileenvironment.

be referenced by the MSA; it is typically held within the directory service entry corresponding

to the base-actor and cached at the local node MSA. Each local MSA is responsible for au-

thenticating messages, as well as inserting the appropriate capabilities for outgoing messages

and validating messages on the receiving end. It is also responsible for enforcing delegation

and revocation mechanisms. Details pertaining to these additional access control mechanisms

can be found in more detailed work about the security infrastructure [48]. For domain-based

access control (DBAC) (see Fig. 6), we assume the network is partitioned apriori, encapsulating

nodes of various security levels (in this work we assume four levels, ranging from no security

(L0) to high security (L3)). These domain partitions represent certain equivalence classes and

allow one to control the granularity of the security of the domain. This ’security level’ (SL) of

a domain is represented as a tag associated with a given domain and with each actor in the

system. Base-level actors created in initial domains are assigned a SL which determines the

degree of secrecy they can have (with respect to the state that possess) when moving between

domains. When studying interactions between various services and policies in a distributed

system, objects moving between different domains with varying security requirements should

preserve a set of properties for availability, secrecy, integrity and semantic consistency to be

maintained [48]. These properties should also maintain consistency within domains in regard

to auxiliary object services interacting with or as part of the access control framework, such as

object creation, garbage collection and forwarding.

17

3.4 Compositionality via the QoS Brokerage Service

In order to achieve the desired level of QoS in terms of application requirements and middleware

service specifications, the base level component of the meta-architecture implements the func-

tionality of the distributed session and deals with (a) data, which includes objects of varying

media, types, e.g., video and audio files and (b) requests to access this data via sessions. The

meta-level component deals with the coordination of multiple requests and sharing of existing

resources among multiple requests. To provide coordination at the highest level and perform

admission control for new incoming sessions, we created a meta-level entity called the QoS

broker. The organization of the meta-level services in CompOSE|Q is illustrated in Fig. 7; the

services have been mapped to a specific distributed server architecture [45, 38].

Fig. 7: A sample QoS brokerage for soft real-time services.

The two main functions of the QoS broker are (a) data management and (b) request man-

agement. The data management component decides the placement of data in the distributed

system, i.e. it decides when and where to create additional replicas of data. It also deter-

mines when additional replicas of data actors are no longer needed and can be garbage col-

lected/dereplicated. We implement adaptive admission control mechanisms [45] in the request-

scheduling module that assigns requests to servers and ensures cost-effective utilization of re-

sources. The message-scheduling module ensures QoS constraint satisfaction of requests that

have already been initiated. The data and request management functions in turn require

common object services such as clock synchronization, replication, de-replication and migra-

tion. For instance, (a) Replication: to replicate data and request actors using adaptive and

18

predictive techniques for selecting where, when and how fast replication should proceed; (b)

De-replication: to dereplicate/garbage-collect data or request actors and optimize utilization of

distributed storage based on current system load and expected future demands for the object;

(c) Migration: to migrate data or requests for load balancing, availability and locality. The

interaction of migration with timing based QoS constraints is an interesting issue since it can

introduce playback jitter in MM applications caused by explicit teardown and re-establishment

of network connections.

In order to ensure non-interference among the common object services used to provide

QoS, the specific mechanisms implemented for placement and scheduling must be designed

not to conflict with each other. Currently, placement and dereplication operate on the basis

of a (conservative) snapshot of the current resource allocation and use. The placement and

dereplication services do not consider the exact times at which requests arrive; in contrast, an

adaptive request scheduling process makes decisions based on the exact arrival times of requests.

However, without appropriate constraints on the usage of these services, inconsistencies can

arise due to their interaction. The broker coordinates the service interaction by constraining

the behavior of the placement and scheduling services. For instance, the dereplication service

does not dereplicate a replica that the request scheduling process is making an assignment to.

Furthermore a replica assigned to an active request should not be physically dereplicated. The

broker also ensures that the dereplication and placement metalevel services do not cancel one

another out. While the interaction between dereplication and placement is not a functional

correctness issue, it has to do with cost-effective performance of the overall system. Formal

reasoning of QoS properties within a meta-architectural model, in the presence of other activities

has been discussed in [38].

Another instance of providing safe composition between middleware services is that of se-

cure video streaming. Here, the extra overhead of the security service could possibly introduce

additional latency that will cause violations of real-time constraints. The brokerage must de-

tect such violations caused by the composition of security and real-time video delivery services.

Furthermore, it will exploit flexibility in video transmission quality (i.e. bitrate) to ensure

adequate levels of security while maintaining jitter-free transmission of streaming video. The

broker picks an optimal combination of transmission bitrate and security parameters (i.e. en-

19

cryption key-length, protocol) and the schedulability of the selected composition is verified by

an admission control process at the RT module. Additionally, the broker may further exploit

characteristics of the transmission path to further balance the quality-security tradeoffs. For

example, nodes along the stream supporting flexible transcoding policies allow the bitrate to

adapted, depending on the real-time requirements enforced by the request management mod-

ule. The security framework in turn, allows the video stream (more specifically the packets)

to be transcoded at intermediate points without having to decrypt them (which is handled by

the security module at each node). This in effect, preserves end-to-end security, but is not a

necessity if the intermediate nodes can be trusted (this in turn can be determined based on the

domain-partitioning of the network and respective security levels associated with them).

4 Implementation & Performance of CompOSE|QThe CompOSE|Q middleware consists of a set of runtime kernels that run on the individual

nodes of the distributed system and distributed middleware components that provide the re-

quired distributed services. The node runtime kernel is a substrate on which actors execute [17].

The principal component of the node runtime kernel is a meta-level node manager actor. The

node manager interfaces with a set of runtime components that implement basic services such

as actor scheduling, message communication and actor creation. The distributed middleware

layer of CompOSE|Q contains metalevel services such as remote creation, distributed snapshots,

directory services, migration and a QoS brokerage. The distributed middleware components

can execute on a single controller node or may themselves be distributed across the nodes in

the system, e.g. the QoS broker operates on a single node, whereas the migration meta-actors

are distributed across nodes.

4.1 Implementation of the Node Run Time Architecture

In implementing the run-time, we suitably ’constrain’ the Java programming language to achieve

Actor semantics; this methodology is also taken in systems such as the Actor Foundry [2]. In

order to assist the three core services in achieving their task easily and efficiently, the run-time

system consists of:

20

The NodeManager: Each node running CompOSE|Q has one NodeManager to manage ac-

tors on that node, as well as to start-up and shutdown various other modules of the run-time

system. When a new actor is created it registers itself with the NodeManager and is identified

by an Actor ID (aid). The NodeManager enters the new actor into a local-table which helps

keep track of the actor for activities such as node checkpointing and node shutdown and notifies

the MessageQManager entity, which allocates a message queue that serves as the “in” Queue

for the actor. The MessageQManager is responsible for managing mail queues for all actors on

a node. To start CompOSE|Q on a node, the NodeManager has to be started first. It in turn,

initiates the various other modules and communication components.

The NodeInfoManager: The NodeInfoManager is a repository of information as well as an

interface to the main directory service in the distributed architecture. Currently, the NodeIn-

foManager implements basic functionality to; (1) Register an actor with the directory service

so that it is accessible to all other nodes; (2) Search for a particular actor to find out which

node that actor is currently on and; (3) Search for an actor object given the class name (a rudi-

mentary naming service). The NodeInfoManager has a local-table, which contains references

to all local actors (updated and maintained by the NodeManager) and a remote-actor cache

that contains information about recently accessed remote actors.

The Communication Subsystem: The communication transport layer and the RCF module

(above the transport layer), together compose the node communication subsystem. The mes-

sage transport layer provides a framework for sending the outgoing messages to the appropriate

node (routing) and resolving incoming messages to their appropriate actor queues (message-

resolution). The RCF module is responsible for the implementation of communication services

(and their composition). Separation of these layers allows for independent customization of

protocol implementations and the messaging runtime. This facilitates correct composition of

protocols without interfering with the runtime communication semantics.

The communication transport layer consists of the following components: a Router, a Post-

man and a RemoteMessageReceiver (Fig. 8). The transport layer maintains two message queues

on a node for incoming and outgoing messages (on that node) called SendPot and ReceivePot

respectively. The Router consults the NodeInfoManager to obtain the current location (node)

21

Fig. 8: Schematic of the software design for the CompOSE|Q runtime environment.

of the target actor. If the location of the target actor is local, the Router puts the message

directly into the node’s ReceivePot. Otherwise, it sends the message to the remote node. The

RemoteMessageReceiver (RMR) on the target node extracts the incoming message and puts it

into the node’s ReceivePot. The Postman then picks up the message and adds it to the target

actor’s “in” Queue.

The communication manager is instantiated in each node during system startup. When an

actor is created and protocol composition services are not desired or required, its messenger is

not created and the actor sends and receives raw messages using the transport layer. When

flexible communication is required or desired, an independent messenger is created for every

base level actor and the entire RCF functionality is invoked. The overhead of the RCF module

is minimized in case of communications with no protocols attached. In this scenario, we tunnel

raw messages through the actor’s messenger directly to the transport layer.

Performance Results for the Run Time: Performance metrics of interest include timing

for basic operations, and communication overheads. All execution times presented here are

average execution times over 100 iterations, without JVM induced overheads.

The execution times of the various basic operations are summarized in Tables 1-4 (see

Fig. 9). A locality check is used to determine whether an actor is local or remote and executes

in 31 µs. In order to measure end-to-end message overhead, we measured message transmission

22

Fig. 9: Performance results of the run-time.

and reception of the underlying transport layer, which serve as a reference for the RCF overhead,

then we measured message tunnelling. Finally we measure transmission of processed messages.

A local send/dispatch operation executes in 181 µs. The timings of the remote message send can

vary dramatically depending on whether the location of the target actor exists on the remote

actor cache on a node (cache hit) or not (cache miss). With a cache hit, a remote send operation

executes in 254 µs, whereas a cache miss causes a remote send to take 42,630 µs to execute. The

performance impact of caching (at the socket cache and remote actor cache) can be seen in the

effective roundtrip latencies for an initial message (approx. 3 secs) and subsequent messages

(50ms). The RCF overhead varies between 28.67–237 µs in message reception and between 209-

1296 in message transmission, depending on the number of protocols used (a protocol consumes

approx 45 µs) and whether or not tunneling is applied.

4.2 Implementation of Meta-Level Services

Here we describe the implementation of a few meta-level services being developed for the

CompOSE|Q framework. For further details on the implementation of metalevel services see

[42]. Our initial implementation of the directory service (DS) uses the Lightweight Directory

Access Protocol (LDAP) [49], which is easily adaptable to many commercial directory services

such as NDS (Novell Directory Service). Recent work on efficient implementation of LDAP

query processing includes directory caching and flexible management of lists [19]. We utilize

the OpenLDAP Group’s slapd LDAP server [30] for our implementation. We used the Berkeley

Sleepy Cat Database as the backend for slapd. An abstraction layer was implemented that

23

presented only the necessary functionality via the Netscape Java LDAP API. Following pre-

liminary testing of the DS, tuning was done to ensure faster performance for both access and

updates. Caching was also used, both for the entries and indexing. These parameters were

tuned to the system, and a variety of tests done to determine the optimal settings.

Directory access overhead is substantially high in comparison to the native execution time of

basic operations. We are currently working on removing directory access out of the critical data-

path for frequently used operations. We are also studying scalability issues in directory services.

Our initial attempts at performance optimization have yielded significant improvements over

raw access and update overheads. Adding an actor takes 27,463µs, which is an 80% improvement

over raw access times. Adding an actor attribute and searching for an attribute take 5,875 &

2814µs respectively (90% improvement over raw overheads).

The current implementation of remote creation only allows for the creation of a single actor α

on a remote node ν with an initial state σ. A remote creation request takes the form RC (α, σ,

ν, β). This call creates and starts an actor, α with the specified state, σ on the remote node, ν.

The call is made to the RCM, which then uses the NodeManager (NM) to implement remote

creation. Remote Creation proceeds in two phases. In the first phase the two NMs (caller and

callee) go through a handshake protocol to negotiate loading of classes and execute checks for

security and resource availability. If the handshake is successful, then the caller NM transfers

the actor-state, σ, to the callee node, ν, which then initializes a newly created instance of the

actor-class with that state.

As described in the architecture of CompOSE|Q the migration service is built using the

remote creation core service. Migration behavior is specified by assigning to each node a

migration Meta actor (MM) that handles migration requests for actors on that node. A remote

creation service accessed via a remote creation request RC is used to install the migrating

actor’s state on the remote node. The remote creation request also includes a message to be

sent to the original node containing the address of the newly created actor. To avoid confusion

with other messages to the migrating actor α, a temporary surrogate actor αs is created to

receive this message.

Performance results for the migration and remote creation services are depicted in Table

1 (see Fig. 10). The results are an average over 10 runs. The main overhead in all these

24

operations is message passing between nodes. A round-trip between nodes takes an average

of 41 ms. A Remote Creation call involves at least one round-trip message (phase 1). Thus,

excluding the messaging overhead, remote creation takes only about 7 ms, which is just 1 ms

more than an local actor create. In migration there are more messages being passed around, so

the timings are expectedly higher (82 ms). The Migration and finalization phase includes the

time for a remote creation (48 ms) plus the waiting of the surrogate actor for a message from

the migrated actor.

Fig. 10: Performance results for meta-services.

Scheduling: The peripheral meta scheduler module runs on each individual node in the dis-

tributed system and it schedules locally originating tasks along with the tasks assigned to it

by the focal scheduler. This scheduler performs the conjunctive message and task scheduling

based on the system wide meta-rules established by the focal scheduler. Each peripheral node

is aware of the different message and task priority levels and can therefore generate a unique

priority ordering based on the independent task and message priorities. Any global changes

made by the focal scheduler are reflected in the directory service and the peripheral schedulers

periodically update their local information from the directory service. Therefore, there might

be some delay before the global policies are reflected in all the distributed nodes. Fig. 4 shows

the changes required to the node runtime to achieve conjunctive scheduling. The conjunctive

scheduling rules are used to determine the final task priorities. At this point, it is important

to note that the PMS performs scheduling based on the tasks running on its node and the

messages inbound to them. In order to ascertain the individual message priorities, the sched-

uler needs to peek into each message. Within CompOSE|Q the Postman routes all messages

to their destination queues and is the only entity that touches each message. We therefore

introduce a small scheduling agent into the Postman that reads the individual message prior-

ity. The individual actor message queues are also modified to support priority messages. i.e.

25

messages can be inserted into the message queues based on their priorities (no longer a FIFO

queue). Real time actors are not affected by the conjunctive priority assignments and always

take precedence over the non realtime actors. The sub scheduler has a multi-threaded design

and is composed of the Broker and the Dispatcher.

Performance results for the RT scheduling service are depicted in Table 2 (see Fig. 10).

Dispatch latency is the length of time to switch from the end of one time-slot to the beginning

of the next time-slot (scheduling overhead that should be kept to a minimum). The dispatch

table calculation time is the time taken by Broker to calculate the dispatch table. Finally, the

scheduler response time is the time elapsed between the application base actor contacting the

broker and getting the reply.

Access Control: The prototype security modules have also been implemented in Java to fa-

cilitate portability, flexibility through introspection and type-safety. Additionally, they allow

first-class capability objects to be implemented in an object-oriented manner and allows for

delegation of these capabilities through sub-classing. Capabilities are defined as the follow-

ing triple: ActorID x Access Rights x CapInf and represented as first class objects. ActorID

uniquely identifies an actor within the system (address), the Access Rights define valid oper-

ations for the object. Finally, CapInf allows additional information to be embedded into the

capability (e.g. expiration, where a timestamp is utilized to indicate a period of validity for the

capability). An additional field allows the capability object to be signed cryptographically.

Each (local) MSA implements a caching mechanism on each node, that allows a table of

rights to be maintained in order to minimize directory accesses. In addition, for remote creation

and migration, a secure class loading mechanism was built on top of the existing remote class

loader available in CompOSE|Q [43]. The MSA encapsulates the behavior of the security

service on each node, and is responsible for creation, validation & management of capabilities.

At present, the behavior of the MSA is pre-defined but we are working on adding the ability

to define customized user-policies on each node (MSA). This is not a trivial matter, as inter-

domain coordination of these policies must be dealt with (an ability being implemented within

the global MSA). At present the MSA implements the following functionality: (1) Registration

with the local Node Manager.; (2) Installation of access rights for existing base-actors.; (3)

Message tagging.; (4) Access rights checking (enforcement).

26

During the installation process (2), the MSA queries the Node Manager and (via the name

service) obtains a list of actors present on the given node. Based on the security policy at

the MSA it installs access rights for the actors (default rights if there are none specified),

updates the directory service with the newly defined access control information and caches the

result locally. The MSA also handles installation of rights for newly created actors. Object

creation signals the MSA on the node of creation and the new capabilities are dynamically

added to the rights table. The enforcement (4) is carried out at the CompOSE|Q transport

layer [43], which currently uses TCP sockets for messaging and caches open connections (for re-

use). The CompOSE|Q transport layer depicted in Fig. 8 provides an illustration of the various

components involved in this filtering process (namely the SendPot and ReceivePot where the

message interception occurs).

Performance results for the access control service are depicted in Tables 1-2 (see Fig. 11). In

Table 1, the results are divided into the MSA bootstrapping and the MSA operational overhead.

To determine the startup overhead, we measured the execution time of the MSA initialization

& registration processes. InstallRights represents the time taken 1 for the MSA to obtain a

list of actors (local send & receive RTT) on its node and setup access rights for them (local

store). All times depicted here are for local operations (assuming that the required information

is available in the local cache) and do not include interactions with the directory service. Table

2 illustrates the overheads of the various MSA operations. On message send, we measure the

raw processing overhead (does not include cryptographic overhead of signing the capability)

necessary to tag outgoing messages (i.e. from when the message is intercepted by the MSA to

when it is returned to the regular messaging system) 2.

Fig. 11: Performance results for the access control service.

1measured for 10 actors2Note that these measurements do not include the overhead for message interception and re-insertion into

the communication subsystem.

27

5 Related Work

Commercially available object-based middleware infrastructures including CORBA and DCOM

represent a step toward compositional software architectures but do not deal with interactions

of multiple object services executing at the same time, and the implication of composing object

services. The Electra framework [26] extends CORBA to provide support for fault tolerance

using group-communication facilities; real-time extensions to CORBA [34] to support timing-

based QoS requirements have been proposed. Systems such as dynamicTAO [20] explore ways

to make the various components of an ORB dynamically configurable as well as componentizing

them to achieve minimal footprint for small applications [50]. Other systems such as Infospheres

[7] and Globe [36] use a distributed object model to construct large scale distributed systems.

Globus, a metacomputing framework defines a QoS component called Qualis [22] where low

level QoS mechanisms can be integrated and tested. The QuO project from BBN [51] deals

with the specification, monitoring and control of application level QoS. QualMan [28] is a

QoS aware resource management platform that provides negotiation, admission and reservation

capabilities for sharing end-system resources. To our knowledge, none of the above systems

provides a formal semantics for the composition of middleware services and protocols. The

distinguishing feature of the TLAM architecture on which the CompOSE|Q system is based is

that it is based on formal semantics that supports safe and correct composability of the services

being implemented. This in turn, provides a framework for the dynamic composition and safe

adaptation of middleware services that can provide application level QoS guarantees.

Traditional reflective languages and systems aim at providing a customizable execution of

concurrent systems [32]. The Aspect Oriented Programming paradigm [18] makes it possible

to express programs where design decisions (aspects) can be appropriately isolated permitting

composition and re-use of code. Reflective systems being developed include Apertos [15], 2K

[21] and Broadway [33]. Recent work integrates the OpenORB architecture [5] with the Aster

system [14] to apply ideas from the software architecture community to the field of adaptive

component-based middleware. The distinguishing feature of the CompOSE|Q architecture is

that it is based on formal semantics that ensures safe and correct composability of the services

being implemented. In other reflective models for distributed object computation [31, 5], an

28

object is represented by multiple models allowing behavior to be described at different levels of

abstraction and from different points of view. In the TLAM, each meta actor can examine and

modify the behavior of a group of base level actors – namely those located on the same node.

In former approaches to dynamic installation of communication services, e.g. Broadway

[33], Actor Foundry [2], multiple protocols may be applied to a single component by stacking

metalevel objects, which implement each protocol. The pluggable protocol framework [3] ad-

dresses the lack of support for multiple inter-ORB protocols and deals with integration and use

of multiple ORB messaging and transport protocols, not with the composition of the protocols

itself. Theoretical and formal approaches [27, 4] study modular and adaptable solutions that

deal with the problems posed by composition of distributed communication services. Many such

approaches assume that communication is point-to-point. Specification and formal character-

ization of group communication services has been developed in the context of I/O Automata

[24] and in recent years, several group communication systems have been developed to pro-

vide additional flexibility of communication services by allowing for dynamic composition of

communication protocols [37, 13]. The RCF approach presented in this paper enforces formal

restrictions on the structure of basic communication primitives to facilitate dynamic installation

and composition of communication protocols. CompOSE|Q also facilitates the customization

and composition of protocols and services using a single unified framework.

6 Conclusions

The need for formal semantics and correctness reasoning has generally been underestimated

by system developers. With the growth in distributed computing environments, middleware

frameworks are starting to employ techniques such as computational reflection to manage re-

sources and services under constantly changing conditions. However, while reflection is a very

powerful tool for providing customizability, it can violate desired application semantics and

cause unexpected behavior when used in an unconstrained manner. Interaction constraints

that ensure application and system level correctness should typically be transparent to the

application designer and handled by the middleware framework.

The two-level architecture presented in this paper provides a framework that allows struc-

tured dynamic customization of middleware services in a correctness-preserving manner. The

29

two-level approach naturally extends to multiple levels, with each level manipulating the level

below while being protected from manipulation by lower levels. In practice, however, expressing

a computation in terms of multiple meta-levels becomes unwieldy. A purely reflective architec-

ture provides an unbounded number of meta-levels with a single basic mechanism. The formal

verification of interaction semantics between the different layers in the reflective hierarchy can

be quite complex and requires further investigation.

We are currently implementing a number of extensions to the CompOSE|Q runtime. For

instance, we are actively working on extending CompOSE|Q to support more distributed ser-

vices for scheduling, security, mobility and fault-tolerance. We are developing directory-enabled

mobility management techniques that can support QoS requirements of applications executing

under poor conditions of network connectivity. Power-cognizant middleware management mech-

anisms are being designed to maximize device lifetimes while guaranteeing user QoS. Efficient

support of security in ubiquitous environments will entail the integration of application, net-

work, middleware and OS level security strategies that can be tailored to the needs of the user

and application.

We are also exploring techniques for designing and managing intelligent network infras-

tructures using the basic services in CompOSE|QThese include services to support dynamic

network customizations that deal with routing and network management (information collec-

tion) that adapt to application and system conditions. These services implement decisions

about the degree of network awareness that applications and middleware must possess to en-

sure performance under varying network conditions. Further work is required to provide a

generalized model that captures the architectural resources required in the server and network

to support a session connection. We are further refining the concept of QoS Synchronizers [38]

and developing techniques for their specification, use and implementation.

In general, the dynamic nature of applications under varying network conditions and request

traffic imply that middleware mechanisms must be dynamic and customizable. We believe that

composable and safe middleware frameworks that implement cleanly defined meta-architectures

enable customization of applications, protocols and system services; this will provide a founda-

tion for the evolution of large scale distributed computing.

30

References

[1] G. Agha: Actors: A Model of Concurrent Computation in Distributed Systems. MIT Press,1986.

[2] Mark Astley & Gul Agha: Customization and Composition of Distributed Objects: Middle-ware Abstractions for Policy Management. Intl. Symposium on the Foundation of SoftwareEngineering, 1998.

[3] A.B. Arulanthu, Carlos O’Ryan, Doug Schmidt, Michael Kircher & Jeff Parsons: TheDesign and Performance of a Scalable ORB Architecture for CORBA Asynchronous Mes-saging. Proc. of Middleware, New York, 2000.

[4] Lynne Blair, & Gordon Blair: The Impact of Aspect-Oriented Programming on For-mal Methods. Proceedings of ECOOP Aspect-Oriented Programming Workshop, Brussels,1998.

[5] Gordon Blair, Geoff Coulson, Phillipe Robin & Michael Papathomas: An Architecture forNext Generation Middleware. In Proc. of Middleware’98, Lake District, England, November1998.

[6] Chandy, K. M. & Lamport, L: Distributed Snapshots: determining global states of a dis-tributed system ACM Transactions on Computing Systems, 1985.

[7] K. Chandy, et al.: A Worldwide Distributed System using Java and the Internet Sympo-sium on High Performance Computing (HPDC-5), Syracuse, New York, Aug. 1996.

[8] Benoit Garbinato & Rachid Guerraoui: Using the Strategy Design Pattern to ComposeReliable Distributed Protocols. Usenix Conference on Object-Oriented Technologies andSystems, 1997

[9] Benoit Garbinato & Rachid Guerraoui: Flexible Protocol Composition in BAST IEEEInternational Conference on Distributed Computing Systems, 1998

[10] Erich Gamma, Richard Helm, Ralp Johnson & John Vlissides: Design Patterns: Elementsof Reusable Object-Oriented Software. Addison-Wesley, 1994.

[11] Sebastian Gutierrez-Nolasco & Nalini Venkatasubramanian: Design Patterns for Safe Re-flective Middleware. OOPSLA, Workshop Towards Patterns and Pattern Languages forObject-Oriented Distributed Real-Time and Embedded Systems, 2001.

[12] Helary, J.: Observing global states of asynchronous distributed applications Proceedings ofthe Workshop on Distributed Algorithms, Springer-Verlag, 1989.

[13] Mark Garland Hayden: The Ensemble System PhD Thesis, Department of ComputerScience, Cornell University, 1998.

[14] V. Issarny & C. Bidan: Aster: A Framework for Sound Customization of DistributedRuntime Systems Proc. of ICDCS’96.

31

[15] J. Ichiro Itoh, R. Lea, & Y. Yokote: Using meta-objects to support optimization in theApertos operating system In USENIX COOTS (Conference an Object-Oriented Technology,June 1995).

[16] James B.D. Joshi, Walid G. Aref, Arif Ghafoor, & Eugene H. Spafford: Security Modelsfor Web-Based Applications. Communications of the ACM, Vol 44, No. 2, Feb 2001

[17] Wooyoung Kim: Thal: An Actor System for Efficient and Scalable Concurrent ComputingPhD thesis, UIUC, 1997.

[18] G. Kiezales, et al.: Aspect-Oriented Programming In Proceedings of ECOOP’97, June1997.

[19] Olga Kapitskaia, Raymond T. Ng & Divesh Srivastava: Evolution and Revolutions inLDAP Directory Caches Proc. of the EDBT 2000.

[20] Fabio Kon, et al.: Monitoring, Security, and Dynamic Configuration with the dynamicTAOReflective ORB IFIP/ACM Middleware’2000. New York. April 3-7, 2000.

[21] F.Kon, et al.: 2K: A Reflective, Component-Based Operating System for Rapidly Chang-ing Environments. In Proceedings of ECOOP’98 Workshop on Reflective Object-OrientedProgramming and Systems, Brussels, Belgium, July 1998.

[22] C. Lee et al.: The Quality of Service Component for the Globus Metacomputing SystemProceedings of IWQoS ’98.

[23] R.G. Lavender & Douglas C. Schmidt: Active Object: an Object Behavioral Pattern forConcurrent Programming. Pattern Languages of Program Design, 1996.

[24] Nancy Lynch & M R Tuttle.: An Introduction to Input/Output Automata CWI Quarterly,2(3):219-246, 1989.

[25] Leslie Lamport: Time and clocks and the ordering of events in a distributed system Com-munications of the ACM,21(7):558-565,1978.

[26] S. Maffeis & D. Schmidt: Constructing reliable distributed communication systems withCORBA IEEE Communications, 14(2), February 1997.

[27] Jose Meseguer, Carolyn Talcott & Denker G: Rewriting Semantics of Meta-Objects &Composable Distributed Services 1999.

[28] K.Nahrstedt, H.-H. Chu, & S.Narayan.: QoS aware resource management for distributedmultimedia applications Journal on High-Speed Networking, Special Issue on MultimediaNetworking, 1998

[29] Priya Narasimhan, Luise E. Moser & P.M. Melliar-Smith.: Using Interceptors to EnhanceCORBA IEEE Computer, 1999.

[30] OpenLDAP: http://www.openldap.org/

32

[31] Okamura, H. Ishikawa, Y. & Tokoro, M: A Distributed Programming System with Multi-Model Reflection Framework ACM SIGPLAN, 1992.

[32] B.C. Smith: Reflection and Semantics in a Procedural Language. PhD thesis, Mas-sachusetts Institute of Technology, Jan 1982.

[33] D.Sturman: Modular Specification of Interaction Policies in Distributed Computing PhDthesis, University of Illinois at Urbana-Champaign, May 1996, TR UIUCDCS-R-96-1950

[34] D.C. Schmidt, D.Levine, & S.Mungee: The Design of the TAO Real-time Object RequestBroker. Computer Comm. Special Issue Building Quality of Service into Distributed Sys-tem, 1997.

[35] Ashish Singhai, Aamod Sane & Roy Campbell: Reflective ORBs: supporting robust, time-critical distribution ECOOP 1997.

[36] M. van Steen, A. Tanenbaum, L.Kuz, & H.Sip: A Scalable Middleware Solution for Ad-vanced Wide-area Web Services In Proc. Middleware ’98, The Lake District, UK, 1998.

[37] Robbert van Renesse, Kenneth Birman & Silvano Maffeis: Horus: A Flexible Group Com-munication System Communication of the ACM, 39(4):76-83, 1996.

[38] N. Venkatasubramanian: An Adaptive Resource Management Architecture for Global Dis-tributed Computing. PhD thesis, UIUC, 1998.

[39] Carlos Varela & Gul Agha: Linguistic Support for Actors, First-Class Token-Passing Con-tinuations and Join Continuations. Midwest Society for Programming Languages and Sys-tems Workshop, 1999

[40] Carlos Varela & Gul Agha: Programming Dynamically Reconfigurable Open Systems withSALSA. Object-Oriented Programming, Systems, Languages and Applications (OOP-SLA), Intriguing Technology Track, 2001.

[41] N. Venkatasubramanian, G. Agha, & C.L. Talcott: Scalable distributed garbage collec-tion for systems of active objects. In International Workshop on Memory Management,IWMM92.

[42] N. Venkatasubramanian et al.: Design & Implementation of CompOSE|Q. Technical Re-port, University of California, Irvine, 2000.

[43] N. Venkatasubramanian, M. Deshpande, S. Mohapatra, S. Gutierrez-Nolasco & J Wickra-masuriya: Design and Implementation of a Composable Reflective Middleware Framework.IEEE International Conference on Distributed Computer Systems (ICDCS-21), April 2001.

[44] Robbert van Renesse, Kenneth Birman & Silvano Maffeis: Horus: A Flexible Group Com-munication System. Communications of the ACM, 39(4):76-83, 1996

[45] N. Venkatasubramanian & S. Ramanathan: Effective load management for scalable videoservers. In Proceedings of ICDCS97, May 1997.

33

[46] N. Venkatasubramanian & C.L. Talcott: Reasoning about Meta Level Activities In OpenDistributed Systems. In 14th ACM Symposium on Principles of Distributed Computing,1995.

[47] Victor F Wolfe, Lisa C DiPippo, Roman Ginis, Michael Squadrit, Steven Wohlever, IgorZykh & Russell Johnston: Real-Time CORBA. Department of Computer Science & Statis-tics, University of Rhode Island, TR97-25b, 1997

[48] J. Wickramasuriya & N. Venkatasubramanian: A Middleware Approach to Access Controlfor Mobile, Concurrent Objects. DOA 2002.

[49] M Wahl, T Howes & S. Kille: LDAP (v3). IETF RFC 2251, December 1997.

[50] Manuel Roman et al.: LegORB and Ubiquitous CORBA. Proceedings of the IFIP/ACMWorkshop on Reflective Middleware (RM2000), April 2000.

[51] J. Zinky, D.Bakken & R. Schantz: Architectural Support for Quality of Service for CORBA

Objects. In Theory and Practice of Object Systems, pages 41-49, 1997.

34