A Temporal Foundation for Continuous Queriesover Data Streams

Jurgen Kramer and Bernhard Seeger

Department of Mathematics and Computer SciencePhilipps-University Marburg, Germany

kraemerj,seeger@informatik.uni-marburg.de

ABSTRACTDespite the surge of research in continuous stream process-ing, there is still a semantical gap. In many cases, contin-uous queries are formulated in an enriched SQL-like querylanguage without specifying the semantics of such a queryprecisely enough. To overcome this problem, we present asound and well defined temporal operator algebra over datastreams ensuring deterministic query results of continuousqueries. In analogy to traditional database systems, we dis-tinguish between a logical and physical operator algebra.While our logical operator algebra specifies the semantics ofeach operation in a descriptive way over temporal multisets,the physical operator algebra provides adequate implemen-tations in form of stream-to-stream operators. We showthat query plans built with either the logical or the physicalalgebra produce snapshot-equivalent results. Moreover, weintroduce a rich set of transformation rules that forms a solidfoundation for query optimization, one of the major researchtopics in the stream community. Examples throughout thepaper motivate the applicability of our approach and illus-trate the steps from query formulation to query execution.

1. INTRODUCTIONContinuous queries over data streams have been emerged

as an important type of queries. Their need is motivated bya variety of applications [4, 13, 8, 25, 10, 29] like network andtraffic monitoring. In order to express continuous queries,different query languages have been proposed recently [1,10, 29, 3, 13]. However, most of these languages lack of aformal foundation since they are solely motivated by provid-ing illustrative examples. This causes a semantic gap thatmakes it hard or even impossible to compute a determin-istic output of a continuous query. This observation wasthe starting point of our work. We introduce a well-definedand expressive operator algebra with precise semantics forsupporting continuous queries over data streams.

The most important task of a data stream managementsystem (DSMS) is to support continuous queries over a set ofheterogeneous data sources, mainly data streams. In anal-ogy to traditional database management systems (DBMS),we propose the following well-known steps from query for-mulation to query execution:

1. A query has to be expressed in some adequate querylanguage, e. g. a declarative language with windowingconstructs such as CQL [2].

2. A logical query plan is built from this syntactical queryrepresentation.

3. Based on algebraic transformation rules, the logicalquery plan is optimized according to a specific costmodel.

4. The logical operations in the query plan are replacedby physical operators.

5. The physical query plan is executed.

Due to the fact that in many stream applications, e. g. sen-sor streams, the elements within a data stream are associ-ated with a timestamp attribute, we decided to define andimplement a temporal operator algebra. In this paper, weshow that the above mentioned process from query formula-tion to query execution is also feasible in the context of con-tinuous queries over data streams. While this paper pavesthe way for rule-based optimization of continuous queries,there are many important optimization problems that maybenefit from our approach. Since many queries are long-running, new cost models are required that take stream ratesinto account [28]. Moreover, dynamic query re-optimizationat runtime [30] is required to adapt to changes in the sys-tem load. Eventually, multi-query optimization [20, 19] isof utmost importance to save system resources. All of theseoptimization techniques employ rules for generating equiva-lent query plans and therefore, they require as a prerequisitea precise semantics of the continuous queries.

In this paper, we introduce a temporal semantics for con-tinuous queries and provide a large set of optimization rules.The main contributions of the paper are:

• We define a logical temporal operator algebra for datastreams that extends the well-known semantics of theextended relational algebra [11]. This includes the de-finition of a novel operator to express both, temporal

sliding and fixed windows. This allows us to map con-tinuous queries expressed in a SQL-like query languageto a logical operator plan.

• We outline the implementation concepts and advan-tages of our physical operator algebra, which providesefficient data-driven implementations of the logical op-erators in form of non-blocking stream-to-stream op-erators. Moreover, we employ and extend research re-sults from the temporal database community [22, 23],because stream elements handled in our physical op-erator algebra are associated with time intervals thatmodel their validity independent from the granularityof time. We demonstrate the beneficial usage of thesevalidity information to perform window queries. Thisallows, for example, to unblock originally blocking op-erators such as difference or aggregation. Furthermore,we show that a physical operator produces a result thatis snapshot-equivalent to the result of its logical coun-terpart. This proves the correctness of the physicaloperators and allows to replace a logical operator byits physical counterpart during the query translationprocess.

• We introduce a rich set of transformation rules, whichconsists of conventional as well as temporal transfor-mation rules, forming an excellent foundation for alge-braic query optimization. Since most of our operationsare compliant to the temporal ones proposed by [23],we are able to transfer temporal research results tostream processing. Moreover, we propose a novel kindof physical optimization by introducing two new oper-ators in the stream context, namely coalesce and split.These operators do not have any impact on the seman-tics, but allow to adaptively change the runtime behav-ior of a DSMS with respect to stream rates, memoryconsumption as well as the production of early results.

The rest of this paper is structured as follows. We startwith a motivating example as well as the basic definitionsand assumptions in Section 2. Then, we formalize the se-mantics of our operations in Section 3 by defining the logicaloperator algebra. The main concepts of the physical oper-ator algebra are discussed in Section 4. Section 5 showsthat our approach represents a good foundation for queryoptimization. Thereafter, we compare our approach withrelated ones and conclude finally.

2. PRELIMINARIESThis section motivates our approach by discussing an ex-

ample query, which is first formulated declaratively and thentransformed into an equivalent logical operator plan. There-after, we discuss the integration of external input streamsand their internal stream representation. Thereby, we intro-duce underlying assumptions and give basic definitions.

2.1 A Running ExampleAt first, let us describe our example application scenario

that represents an abstraction from the Freeway Service Pa-trol project. We consider a highway with five lanes whereloop detectors are installed at measuring stations. Eachmeasuring station consists of five detectors, one detector per

lane. Each time a vehicle passes such a sensor, a new recordis generated. This record contains the following informa-tion: lane at which the vehicle passed the detector, the ve-hicle’s speed in meters per second, its length in meters anda timestamp. Hence, each detector generates a stream ofrecords. In our application, the primary goal is to measureand analyze the traffic flow. In the following subsections,we give a brief overview of how we model, express, and exe-cute queries in this use-case using our semantics and streaminfrastructure [17].

2.2 Query FormulationThe focus of this paper is neither on the definition of an

adequate query language for continuous query processingover data streams nor on the translation of language con-structs to logical operator plans. Instead, our goal is toestablish a platform for possible stream query languages bydefining a sound and expressive operator algebra with a pre-cise semantics. In order to illustrate the complete processfrom query formulation to query execution as discussed inthe introduction, we exemplarily express a query in somefictive SQL-like query language using the sliding windowexpressions from CQL [2].

Example: A realistic query in our running examplemight be: ”At which measuring stations of the highway hasthe average speed of vehicles been below 15 m/s over the last15 minutes.” This query may indicate traffic-congested sec-tions of the highway. Let us assume that our query addresses20 measuring stations. Then, the following text representsthe query expressed in our fictive query language:

SELECT sectionIDFROM (SELECT AVG(speed) AS avgSpeed, 1 AS sectionID

FROM HighwayStream1 [Range 15 minutes]UNION ALL...UNION ALLSELECT AVG(speed) AS avgSpeed, 20 AS sectionIDFROM HighwayStream20 [Range 15 minutes]

)WHERE avgSpeed < 15;

2.3 Stream TypesIn analogy to traditional DBMS, we distinguish between

the logical operator algebra and its implementation, thephysical operator algebra. We use the term logical streamsto denote streams processed in the logical operator algebra,whereas physical streams refer to the ones processed in thephysical operator algebra. In addition to logical and phys-ical streams as our internal stream types, we also considerraw input streams as a third type of streams that modelthose arriving at our DSMS.

2.3.1 Raw Input StreamsThe representation of the elements from a raw input

stream depends on the specific application. We assumethat an arbitrary but fixed schema exists for each raw inputstream providing the necessary metadata information aboutthe stream elements. However, this schema is not restrictedto be relational, since our operators are parameterized byarbitrary functions and predicates. Our approach is power-ful enough to support XML streams.

Let Ω be the universe, i. e. the set of all records of anyschema.

Definition 1. (Raw Input Stream) A raw input stream Sr

is a possibly infinite sequence of records e ∈ Ω sharing thesame schema. Sr denotes the set of all raw input streams.

Note that this definition corresponds to the one of a list.Thus, a raw input stream may contain duplicates, and theordering of its elements is significant.

Example: For simplicity reasons, we focus on the follow-ing flat schema in our example:

HighwayStream(short lane, float speed, float length,Timestamp timestamp);

A measuring station might generate the following raw inputstream:

(5; 18.28; 5.27; 03/11/1993 05:00:08)(2; 21.33; 4.62; 03/11/1993 05:01:32)(4; 19.69; 9.97; 03/11/1993 05:02:16)

...

2.3.2 Internal StreamsA physical stream is similar to a raw input stream, but

each record is associated with a time interval modeling itsvalidity. In general, this validity refers to application timeand not to system time. As long as such a stream element isvalid, it is processed by the operators of the physical oper-ator algebra. An element expires when it has no impact onfuture results anymore. Then, it can be removed from thesystem. In a logical stream, we break up the time intervalsof a physical stream element into chronons that correspondto time units at finest time granularity.

In the following, we formalize our notions and representa-tions of logical and physical streams. In particular, we showhow a raw input stream is mapped to our internal represen-tation and provide an equivalence relation for transforminga physical stream into a logical stream and vice versa.

2.4 Basic DefinitionsLet T = (T ;≤) be a discrete time domain as proposed by

[6]. Let I := [ts, te) ∈ T × T | ts < te be the set of timeintervals.

Definition 2. (Physical Stream) A pair Sp = (M,≤ts,te)is a physical stream, if

• M is a potentially infinite sequence of tuples (e, [ts, te)),where e ∈ Ω and [ts, te) ∈ I,

• all elements of M share the same schema,

• ≤ts,te is the order relation over M such that tuples(e, [ts, te)) are lexicographically ordered by timestamps,i. e. primarily by ts and secondarily by te.

Sp denotes the set of all physical streams.

The meaning of a stream tuple (e, [ts, te)) is that a recorde is valid during the half-open time interval [ts, te). Theschema of a physical stream is a combination of the recordschema and a temporal schema that consists of two time at-tributes modeling the start and end timestamps.

Our approach relies on multisets for the following two rea-sons. First, applications may exist where duplicates in a rawinput stream might arise. In our example, this would occurif two vehicles with the same length and speed would passthe same sensor within one second (assuming that the finesttime resolution of the detectors is in seconds). Consequently,this would result in two identical records. Second, operatorslike projection may produce duplicates during runtime, evenif all elements of the raw input stream are unique. In thiscase, the term duplicates has a slightly different meaningand we use value-equivalent stream elements instead.

Definition 3. (Value-equivalence) Let Sp = (M,≤ts,te) ∈Sp be a physical stream. We denote two elements (e, [ts, te)),(e, [ts, te)) ∈ M as value-equivalent, iff e = e.

Note that ordering by ≤ts,te enforces no order within realduplicates, i. e., when the records as well as time intervalsof two elements are equal.

2.5 TransformationNow we describe the transformation of a raw input stream

Sr ∈ Sr into a physical stream Sp ∈ Sp. Especially whensensors are involved, many applications produce a raw in-put stream where the elements are already associated with atimestamp attribute. Typically, these streams are implicitlyordered by their timestamps. This holds for instance in ourrunning example. If streams arrive at a DSMS out of orderand uncoordinated with each other, e. g. due to latenciesintroduced by a network, techniques like the ones presentedin [24] can be applied. It is also possible that a stream doesnot provide any temporal information. In this case, a DSMScan stamp the elements at their arrival by using an internalsystem clock.We then use the start timestamp of each raw input streamelement as the start timestamp of a physical stream tuple.The corresponding end timestamp is set to infinity becauseinitially we assume each record to be valid forever. Thatmeans, we map each element e in Sr to a tuple (e, [ts,∞)) inSp where ts is the explicit timestamp retrieved from e. Thisimplies that the order of Sr is preserved in Sp. The schemaof Sp extends the schema of Sr by two additional timestampattributes modeling the start and end timestamps.

Example: Applying the transformation to the raw inputstream of our running example would produce the followingphysical stream of tuples (record, time interval):

((5; 18.28; 5.27; 03/11/1993 05:00:08),[03/11/1993 05:00:08, ∞))

((2; 21.33; 4.62; 03/11/1993 05:01:32),[03/11/1993 05:01:32, ∞))

((4; 19.69; 9.97; 03/11/1993 05:02:16),[03/11/1993 05:02:16, ∞))

...

2.6 Window OperationsThe usage of windows is a commonly applied technique in

stream processing mainly for the following reasons [13]:

• At any time instant often an excerpt of a stream isonly of interest.

• Stateful operators such as the difference would beblocking in the case of unbounded input streams.

Figure 1: Windowing constructs

• The memory requirements of stateful operators arelimited, e. g. in a join.

• In temporally ordered streams, newly arrived elementsare often more relevant than older ones.

In our logical as well as in our physical operator algebra, wemodel windows by introducing a novel window operator ωthat assigns a finite validity to each stream element. For agiven physical input stream, this is easily achieved by settingthe end timestamp of each incoming stream element, whichis initially set to infinity, to a certain point in time accordingto the type and size of the window.

Let Sp = (M,≤ts,te) ∈ Sp be a physical stream. Letw ∈ T be the window size. By using the window operatorωw : Sp × T → Sp, we are able to perform a variety ofcontinuous window queries involving the following types ofwindows (see Figure 1):

• Sliding windows: In order to retrieve sliding win-dow semantics, the window operator ωw sets the endtimestamp te of each physical stream tuple (e, [ts,∞))∈ M to ts + w. This means that each element e isvalid for w time units starting from its correspondingstart timestamp ts.

• Fixed windows: In the case of fixed windows [25],we divide the time domain T in sections of fixed sizew ∈ T . Hence, each section contains exactly w sub-sequent points in time, where 0 stands for the earli-est time instant. Thus, section i starts at i · w wherei ∈ N0. Fixed window semantics can be obtained, if thewindow operator ωw sets the end timestamp te of eachphysical stream tuple (e, [ts,∞)) ∈ M to the point intime where the next section starts. Consequently, fora given element (e, [ts,∞)) ∈ M , the window operatordetermines the closest point in time te = i · w withts < te.

Note that it is sufficient for performing continuous windowqueries to place a single window operator on each path froma source to a sink in a query plan. These window operatorsare typically located near the sources of a query plan.

Example: Applying a sliding window of 15 minutes tothe physical stream in our example would change the timeintervals as follows:

((5; 18.28; 5.27; 03/11/1993 05:00:08),[03/11/1993 05:00:08, 03/11/1993 05:15:08))((2; 21.33; 4.62; 03/11/1993 05:01:32),

[03/11/1993 05:01:32, 03/11/1993 05:16:32))((4; 19.69; 9.97; 03/11/1993 05:02:16),

[03/11/1993 05:02:16, 03/11/1993 05:17:16))

...

At this point, we want to sketch the basic ideas of ourphysical algebra approach with regard to windowing con-structs: We have physical streams consisting of record/time-interval pairs. The time intervals model the validity of eachrecord which in turn is set via our window operator. Thephysical operators are aware of the time intervals and usethem effectively to guarantee non-blocking behavior as wellas limited memory requirements. Based on these physicaloperators we are able to build query plans that perform con-tinuous window queries over arbitrary data streams whileensuring deterministic semantics.

Before we go into the details of the physical algebra in Sec-tion 4, we will first start the discussion of the logical algebrain the next section. The reason for introducing a logicalalgebra is similar to the approach in a traditional DBMS.The logical algebra abstracts from the physical implemen-tation of the operators, while providing powerful algebraictransformation rules to rearrange operators in a query plan.

3. LOGICAL OPERATOR ALGEBRAThis section formalizes the term logical stream and shows

how a logical stream is derived from its physical counterpart.Then, the basic operators of our logical operator algebra areintroduced by extending the work on multisets [11] towardsa temporal semantics and windowing constructs.

3.1 Logical Streams

Definition 4. (Logical Stream) A logical stream Sl is apossibly infinite multiset of triples (e, t, n) composed of arecord e ∈ Ω, a point in time t ∈ T , and a multiplicityn ∈ N. All records e of a logical stream belong to the sameschema. Moreover, the following condition holds for a logicalstream Sl: ∀ (e, t, n) ∈ Sl. @ (e, t, n) ∈ Sl. e = e ∧ t = t.Let Sl be the set of all logical streams.

The condition in the definition ensures that exactly one ele-ment (e, t, n) exists in Sl for each record e valid at a point intime t. To put it in other words: The projection on the firsttwo attributes of each stream triple in the set representationof a logical stream is unique.A stream triple (e, t, n) has the following semantics: An ele-ment e is valid at time instant t and occurs exactly n times.Since we treat a logical stream as a multiset, we additionallystore the multiplicity of each record in analogy to [11]. Thislogical point of view implies that all records, their validity aswell as their multiplicity are known in advance. We do nottake the order in a logical stream into account. Therefore,it is only relevant in the logical model, when a record e isvalid and how often it occurs at a certain point in time t.The schema of a logical stream is composed of the recordschema and two additional attributes, namely a timestampand the multiplicity.

3.1.1 Transformation: Physical to Logical StreamLet Sp = (M,≤ts,te) ∈ Sp be a physical stream. We define

the transformation τ : ℘(Ω× I) → Sl from a physical streamSp into its logical counterpart as follows:

τ(M) := (e, t, n) ∈ Ω× T × N |n = |(e, [ts, te)) ∈ M | t ∈ [ts, te)|

For each tuple (e, [ts, te)) ∈ M , we split the associated timeinterval into points of time at finest time granularity. Thus,we get all instants in time when the record e is valid. Sincewe allow value-equivalent elements in a physical stream, wehave to add the multiplicity n of a record e at a certain pointin time t.

3.2 Basic OperatorsIn our logical operator algebra, we introduce the following

operations as basic ones since they are minimal and orthogo-nal [23]: filter(σ), map (µ), Cartesian product (×), duplicateelimination (δ), difference (−), group (γ), aggregation (α),union (∪) and window (ω).Appendix A reports the definition of more complex opera-tions derived from the basic ones, e. g. a join.

3.2.1 FilterLet P be the set of all well-defined filter predicates. A

filter σ : Sl ×P → Sl returns all elements of a logical streamSl ∈ Sl that fulfill the predicate p ∈ P with p : (Ω × T ) →true, false. We follow the notation of the extended rela-tional algebra and express the argument predicate as sub-script. Note that our definition also allows temporal filter-ing.

σp(Sl) := (e, t, n) ∈ Sl | p(e, t) (1)

The schema of the logical stream Sl remains unchanged, ifa filter operation is performed.

3.2.2 MapLet Fmap be the set of all mapping functions. The map

operator µf : Sl × Fmap → Sl applies a mapping function fgiven as subscript on the record of each stream element ina logical stream Sl ∈ Sl. Let f ∈ Fmap with f : Ω → Ω.Note, that f can also express an n-ary function due to thedefinition of Ω as a universe of all elements.

µf (Sl) := (e, t, n) | n =P

(e,t,n)∈Sl|f(e)=en (2)

This definition is more powerful than the projection operatorof the relational algebra because the mapping function maygenerate new attributes or even new records. Thus, theschema of the resulting logical stream essentially dependson the mapping function. Note, that the mapping functiondoes not change the timestamp attribute of an element.

3.2.3 Cartesian ProductThe Cartesian product × : Sl × Sl → Sl of two logical

streams Sl1, S

l2 ∈ Sl is defined by:

×(Sl1, S

l2) := ((e1, e2), t, n1 · n2) |

∃ (e1, t, n1) ∈ Sl1 ∧ ∃ (e2, t, n2) ∈ Sl

2(3)

For each pair of elements from Sl1 and Sl

2 valid at the samepoint in time t, a new result is created as concatenation ofboth records by the auxiliary function : Ω× Ω → Ω.The multiplicity of the result is determined by the product ofthe multiplicities of the two qualifying elements. The result-ing schema of the logical output stream is a concatenationof both record schemas, the timestamp, and the multiplicityattribute.

3.2.4 Duplicate EliminationThe duplicate elimination is an unary operation δ : Sl →

Sl that produces for a given logical stream Sl ∈ Sl a setof elements. This implies that each element in Sl occursexactly once.

δ(Sl) := (e, t, 1) | ∃ n. (e, t, n) ∈ Sl (4)

The definition intuitively shows how duplicate eliminationworks, because the multiplicity for each element in Sl issimply set to 1. The schema of a logical stream after aduplicate elimination corresponds to that of the logical inputstream.

3.2.5 DifferenceApplying a difference operation − : Sl × Sl → Sl enforces

that all elements of the second logical stream Sl2 ∈ Sl are

subtracted from the first logical stream Sl1 ∈ Sl in terms

of their multiplicities. Thus, the schema of the differencematches that of Sl

1. Obviously, a difference operation canonly be performed if the schemas of both input streams arecompliant.

−(Sl1, S

l2) := (e, t, n) | ∃ n1. (e, t, n1) ∈ Sl

1

∧ ∃ n2. (e, t, n2) ∈ Sl2 ∧ n = n1 n2 ∧ n > 0)

∨ ((e, t, n) ∈ Sl1 ∧ @ n2. (e, t, n2) ∈ Sl

2)

where n1 n2 :=

n1 − n2 , if n1 > n2

0 , otherwise

(5)

This definition distinguishes between two cases: The firstone assumes that an element of Sl

1 exists that is value-equivalent to one of Sl

2 and both elements are valid at thesame point in time t. Then, the resulting multiplicity isthe subtraction of the corresponding multiplicities. An ele-ment only appears in the output if its resulting multiplicityis greater than 0. In the second case, no element of Sl

2

matches with an element of Sl1. In this case the element is

retained.At the end of this definition, we want to highlight one

major benefit of our descriptive logical algebra approach,namely that the operator semantics can be expressed verycompact and intuitive. For instance, the difference is simplyreduced to the difference in multiplicities, whereas relatedapproaches using the λ-calculus [23] hide this property andturn out to be more complicated.

3.2.6 GroupLet Fgroup be the set of all grouping functions. The group

operation

γf : Sl × Fgroup → (Sl × . . .× Sl| z k times

)

produces a tuple of logical streams. It assigns a group toeach element of a logical stream Sl ∈ Sl based on a groupingfunction f ∈ Fgroup with f : Ω×T → 1, . . . , k. Each groupSl

j represents a new logical stream for j ∈ 1, . . . , k having

the same schema as Sl.

γf (Sl) := (Sl1, . . . , S

lk)

where Slj := (e, t, n) ∈ Sl | f(e, t) = j. (6)

The group operation solely assigns elements to groups with-out modifying them. The j-th group contains all elements

for which the grouping function f returns j. This definitiondiffers from its relational counterpart which includes an ad-ditional aggregation step.We also define a projection operator, which is a map oper-ator

π : (Sl × . . .× Sl| z k times

)× N → Sl

that is typically used in combination with the group opera-tion. For a given index j, π returns the j-th logical outputstream (group): πj(S

l1, . . . , S

lk) := Sl

j .

3.2.7 AggregationLet Fagg be the set of all well-defined aggregation func-

tions. The aggregation operation αf : Sl×Fagg → Sl invokesan aggregation function f ∈ Fagg with f : Sl → Ω on all ele-ments of a logical stream Sl ∈ Sl that are valid at the samepoint in time t:

αf (Sl) := (agg, t, 1) | agg = f((e, t, n) ∈ Sl) (7)

The aggregation eliminates duplicates because an aggregateis computed on all elements valid at the same point in time.Thus, the aggregation operator returns a set. The schema ofa logical stream after an aggregation obviously depends onthe aggregation function, but only the record schema has tobe adopted, while the timestamp and multiplicity attributesof the input schema remain unchanged.Contrary to DBMS, our definitions of group and aggrega-tion additionally offer to use both operations independentlyin query plans. For example, it is possible to apply an ag-gregation to a stream without grouping.

3.2.8 UnionThe union operation ∪+ : Sl × Sl → Sl merges two logical

data streams. Its result contains all elements of Sl1 and

Sl2 ∈ Sl:

∪+(Sl1, S

l2) := (e, t, n1 + n2) |

ni =

n ,∃ (e, t, n) ∈ Sl

i

0 , otherwisefor i ∈ 1, 2 (8)

If an element only occurs in a single input stream, it is di-rectly added to the result. If the same record is containedin both input streams and valid at the same point in time t,both instances are combined to a single element by summingup their multiplicities.Note, that a union can only be performed if both logicalinput streams are schema-compliant. Then, the resultingschema is taken from the more general input schema.

3.2.9 WindowThe window operator ωw : Sl × T → Sl restricts the va-

lidity of each record according to the window type and sizew ∈ T . We assume as a precondition of the input streamthat each record has an infinite validity as already mentionedin Section 2.6.Let Sl be a logical stream whose records have an infinitevalidity. Therefore, the multiplicity of a record in Sl ismonotonically increasing over time. We differ between thefollowing two types of window operations:

1. Sliding Window: Informally, a sliding window ωsw sets

the validity of a record, which is valid for the first time

at a starting point ts ∈ T , to w time units, i. e., theelement is valid from ts to ts + w − 1. However, themultiplicity of a record may change over time. An in-crease in the multiplicity at a certain point in timeindicates that further value-equivalent elements startto be valid at this time instant. Consequently, we alsohave to set the validity of these value-equivalent ele-ments correctly by assigning a validity of w time unitsrelative to their starting points.

ωsw(Sl) := (e, t, n) | ∃ n. (e, t, n) ∈ Sl

∧ [(∃ n. (e, t− w, n) ∈ Sl ∧ n = n− n)∨ (@ n. (e, t− w, n) ∈ Sl ∧ n = n)]

(9)

A sliding window is expressed by setting the multi-plicity n of a record e valid at a time instant t to thedifference of the multiplicities n and n. Here, n and nrefer to the multiplicity of the record e at time t andt−w, respectively. If no record e exists at time instantt− w in Sl, n is set to n.

2. Fixed Window: In the case of a fixed window ωfw, the

time domain T is divided into sections of size w ∈ T .At first, we determine all starting points ts ∈ T ofa record. Then, we determine the start of the nextsection i · w which is larger but temporally closest tots. The validity of each record is set to the start of thenext section.

ωfw(Sl) := (e, t, n) | ∃ n. (e, t, n) ∈ Sl

∧ ∃ i ∈ N0. (i · w ≤ t ∧ ∀ c ∈ N. (c > i ⇒ c · w > t))∧ [(∃ n. (e, (i · w)− 1, n) ∈ Sl ∧ n = n− n)∨ (@ n. (e, (i · w)− 1, n) ∈ Sl ∧ n = n)]

(10)In contrast to the definition of the sliding window, weconsider the multiplicity at time instant (i·w)−1 whichcorresponds to the multiplicity of the record e at thelast point in time belonging to the previous section.The parameter i is chosen such that the start of thesection i ·w is the timely closest start of a section withrespect to t.

The schema of the resulting logical stream after a windowoperator is identical to that of the logical input stream.

3.3 Logical Query PlansA query formulated in some query language is generally

translated into a semantically equivalent algebraic expres-sion. Such an algebraic expression consists of a compositionof logical operators. For our logical operator algebra this canbe achieved similarly to traditional databases where SQL istranslated into a logical operator plan in the extended rela-tional algebra.

Example: The left drawing in Figure 2 depicts the logicalquery plan that results from mapping the query presented inSection 2.2 to the operators in our logical operator algebra.At first, the validity of the stream elements is set to 15minutes, then a map to the attributes speed and sectionID

is performed. Afterwards, the average speed is computedand all streams are merged, followed by a filter operationthat selects all stream elements with an average speed lowerthan 15 m/s. Finally, a projection delivers the IDs of thequalifying sections.

Figure 2: Query plans composed of our operations

4. PHYSICAL OPERATOR ALGEBRAFrom an implementation point of view, it is not sufficient

to process logical streams directly because this would cause asignificant computational overhead. Since a physical streamhas a much compacter representation of the same temporalinformation, we decided to implement the physical opera-tor algebra in PIPES, our infrastructure for data streamprocessing.

To the best of our knowledge, there is no other approachto stream processing which uses time intervals to expressthe validity of stream elements. There are other approaches[3, 14] that are based on a quite similar temporal semantics.But they substantially differ in their implementation as theyemploy so-called positive-negative elements. This howeverhas certain drawbacks as outlined in the following. Whenpositive-negative elements are used, a window operator is re-quired that explicitly controls element expiration. If a datasource emits a new element, the window operator generatesa positive element by decorating the new element with a’+’ which is sent through the query plan afterwards. Thewindow operator stores all incoming elements and creates anegative element if the validity of an element in its bufferexpires according to the window. The negative element,i. e. the element decorated with a ’-’, is pushed through thequery plan and processed. This implies that operators haveto distinguish between positive and negative incoming ele-ments. Furthermore, this approach doubles the number ofelements being processed, since for each stream element ina raw input stream, two stream elements in a physical inputstream are generated. These deficiencies are entirely avoidedin our interval-based approach.

In the following, we describe how we transform a logicalstream into a physical stream. This makes our transforma-tions complete as a logical stream can be transformed into aphysical one and vice versa (see Section 3.1.1). This fact isimportant for query optimization because it offers a seamlessswitching between logical and physical query plans.

4.1 Transformation: Logical to Physical StreamLet Sl ∈ Sl be a logical stream. We transform a logical

stream into a physical stream by two steps:

1. We introduce time intervals by mapping each logicalstream element (e, t, n) ∈ Sl to a triple (e, [t, t+1), n) ∈Ω × I × N. This does not effect our semantics at all,since the time interval [t, t + 1) solely covers a singlepoint in time, namely t. We denote this operation byι : Sl → ℘(Ω× I× N).

2. Then, we merge value-equivalent elements with adja-cent time intervals in order to build larger time in-tervals. This operation termed Coalesce is commonlyused in temporal databases [22].Let M, M ′ be in ℘(Ω × I × N). We define a relationM . M ′ that indicates if M can be coalesced to M ′

with:

M . M ′ :⇔ (∃ m := (e, [ts, te), n) ∈ M,m := (e, [ts, te), n) ∈ M. e = e

∧ te = ts ∧ (∃ M ′′ ∈ ℘(Ω× I× N).M ′′ = (M − m, m) ∪ (e, [ts, te), 1)

∧ M ′′ . M ′)) ∨ M = M ′

where M − m, m := (M \ m, m) ∪((e, [ts, te), n− 1), (e, [ts, te), n− 1) \ Ω× I× 0).

When coalesce merges two triples in M , these elementsare removed from M and the new triple containing themerged time intervals is inserted. Furthermore, themultiplicities have to be adopted.

This definition of coalescing is non-deterministic ifM contains several elements whose start timestampmatches with the end timestamp of an other value-equivalent element. Therefore, coalescing ζ : ℘(Ω× I×N) → ℘(Ω× I× N) produces a set.

ζ(M) := M ′ ∈ ℘(Ω× I× N) | M . M ′ ∧(∀ M ′′ ∈ ℘(Ω× I× N). (M ′ 6 . M ′′) ∨ (M ′ = M ′′))

(11)

For a given logical stream Sl, we obtain a correspondingphysical stream Sp ∈ Sp by ordering the elements of a mul-tiset M ∈ ζ(ι(Sl)) according to ≤ts,te while listing the dupli-cates as separate stream elements. As we will see in Section5.2, our notion of stream equivalence is independent fromthe set chosen from ζ(ι(Sl)).The schema of the physical stream can be derived from thelogical stream by keeping the record schema and decoratingit with the common temporal schema of a physical stream,namely the start and end timestamp attributes.

4.2 Operator ImplementationFor each operation of the logical algebra, PIPES provides

at least one implementation based on physical streams, i. e.,a physical operator takes one or multiple physical streamsas input and produces one or multiple physical streams asoutput. These physical stream-to-stream operators are im-plemented in a data-driven manner assuming that streamelements are pushed through the query plan. This impliesthat a physical operator has to process the incoming ele-ments directly without choosing the input from which thenext element should be consumed.

Another important requirement for the implementation ofphysical operators over data streams is that these operatorsmust be non-blocking. This is due to the potentially infinitelength of the input streams and the request for early re-sults. Our physical operator algebra meets this requirementby employing time intervals and introducing the window op-erator. This technique unblocks blocking operators, e. g. thedifference, while guaranteeing deterministic semantics.

4.2.1 Operator ClassificationThe operators of our physical operator algebra can be

classified in two categories:

• Stateless operators: A stateless operator is able toproduce its results immediately without accessing anykind of internal data structure. Typical stateless oper-ators are: filter, map, group and window. For instance,the filter operation evaluates a user-defined predicatefor each incoming element. If the filter predicate is sat-isfied, the element is appended to the output stream,otherwise it is dropped. Another example is the groupoperation that invokes a grouping function on each in-coming element. The result determines the physicaloutput stream to which the element is appended to.The implementation of stateless operators is straight-forward and fulfills the requirements of data-drivenquery processing.

• Stateful operators: A stateful operator requires somekind of internal data structure for maintaining itsstate. Such a data structure has to support operationsfor efficient insertion, retrieval and reorganization. Weidentify the following physical operators in our algebraas stateful: Cartesian product/join, duplicate elimina-tion, difference, union and aggregation.The implementation of a stateful operator has to guar-antee the ordering of physical streams (see Section4.2.2). Moreover, it should be non-blocking while lim-iting memory usage. Most importantly, the imple-mentation should produce deterministic results (seeSection 4.2.3).

4.2.2 Ordering InvariantA physical operator has to ensure that each of its phys-

ical output streams is ordered by ≤ts,te (see Section 2.4),i. e., the stream elements in an output stream have to bein an ascending order, lexicographically by their start andend timestamps. This invariant of our implementation isassumed to hold for all input as well as output streams of aphysical operator. This may cause delays in the result pro-duction of an operator. In a union for instance, the resultshave to be ordered, e. g. by maintaining an internal heap.This also explains why we consider the union operation asstateful.

This order invariant seems to be very expensive to sat-isfy. However, it is commonly assumed in stream processingthat raw input streams arrive temporally ordered [4, 13] ormechanisms exist that ensure such a temporal ordering [24].Moreover, our efficient algorithms for the reorganization ofthe internal data structures of stateful operators rely on thisordering invariant.

4.2.3 ReorganizationLocal reorganizations are necessary to restrict the mem-

ory usage of stateful physical operators. Such reorganiza-tions are input-triggered, i. e., each time a physical operatorprocesses an incoming element, a reorganization step is per-formed. In this reorganization step, all expired elements areremoved from the internal data structures.

Let Sp1 , . . . , Sp

n ∈ Sp be physical streams, n ∈ N. Foran arbitrary stateful operator with physical input streamsSp

1 , . . . , Spn, the reorganization is performed as follows: We

store the start timestamps tsj for j ∈ 1, . . . , n of the lastincoming element of each physical input stream Sp

j . Then,all elements (e, [ts, te)) can be safely removed from the in-ternal data structures whose end timestamp te is smallerthan mintsj | j ∈ 1, . . . , n or equal. This conditionensures that only expired elements are removed from inter-nal data structures. The correctness results from the or-dering invariant because if a new element (e, [ts, te)) of aninput stream Sp

j arrives, all other elements of this streamprocessed before must have had a start timestamp that isequal or smaller than ts. Furthermore, a result of a statefuloperator is only produced when the time intervals of the in-volved elements overlap. For example, in a binary join twostream elements (e, [ts, te)) ∈ Sp

1 and (e, [ts, te)) ∈ Sp2 qualify

if the join predicate holds for their records, e and e, and thetime intervals, [ts, te) and [ts, te) overlap. The result con-tains the concatenation of the records and the intersectionof the time intervals (see Definition 3.2.3). Hence, the reor-ganization condition specified above solely allows to removean element from an internal data structure if it is guaran-teed that there will be no future stream elements whose timeinterval will overlap with this element.

From this top-level point of view, it seems to be suffi-cient to require that a physical stream is in ascending orderby the start timestamps of its elements. This is becausethe reorganization condition does not make use of the sec-ondary order by end timestamps. However, we maintainthe lexicographical order of physical streams since it gen-erally leads to early results. Our stateful operators addi-tionally link the elements in their internal data structuresaccording to the lexicographical order ≤ts,te . This helps toefficiently run the reorganization phase by simply followingthese links. The reorganization phase can be stopped if alinked element is accessed whose end timestamp is largerthan mintsj | j ∈ 1, . . . , n. Keeping this implemen-tation detail in mind, the secondary order helps to purgeexpired elements earlier.We made an interesting observation during our implemen-tation work. When operations get unblocked by using win-dows, many stateful physical operators produce their resultsduring the reorganization phase. Hence, expired elementsare not only removed from the internal data structures butthey are also appended to the physical output stream.

Input-triggered reorganization is only feasible if each phys-ical input stream continuously delivers elements, which is ageneral assumption in stream processing. However, if one in-put stream totally fails, the minimum of all start timestampsmintsj | j ∈ 1, . . . , n cannot be computed and, conse-quently, no elements can be removed from internal datastructures. This kind of blocking has to be avoided, e. g. by

introducing appropriate timeouts [24]. A similar problemarises if the delays between subsequent elements within aphysical stream are relatively long. In this case, the re-organization phase is triggered seldom, which may lead toan increased memory usage of the internal data structures.Hence, there is a latency-memory tradeoff for stateful oper-ators.

4.2.4 AggregationWe want to sketch the implementation of the aggrega-

tion as a non-trivial stateful operator. Let Sp = (M,≤ts,te)be the physical input stream of the aggregation operator.Since we implemented an incremental aggregation [15], weneed a binary aggregation function f : Ω ∪ ⊥ × M → Ωthat is applied successively to the current aggregation values ∈ Ω ∪ ⊥ and an incoming element s = (e, [ts, te)) ∈ M(see Figure 3). ⊥ is solely used to initialize the aggregate.We first probe the internal data structure for elements whosetime interval overlaps with [ts, te). For the case of a partialoverlap we split the element into maximum subintervals witheither no or full overlap, while keeping the aggregation valuefor each of them. Then, we update the aggregation value ofall overlapping elements s by invoking f on (s, s). For eachmaximum subinterval r of [ts, te) for which no intersectionis found in the internal data structure, we finally insert anelement consisting of an initialized aggregation value f(⊥, s)and the time interval r.Thereafter, we perform the reorganization phase by remov-ing all elements from the internal data structure whose endtimestamp is smaller than ts or equal. Those can efficientlybe determined by additionally linking the elements in theinternal data structure according to ≤ts,te , which corre-sponds to an ordering by end timestamps in this case. Aseach expired element contains the final aggregation valuefor the associated time interval, we append it to the physi-cal output stream. Consequently, the aggregation operatorproduces its results during the reorganization phase.

Example: Let us consider our running example, wherewe compute the average speeds of vehicles. The physicalquery plan is obtained by replacing all logical operations inthe logical query plan (see Section 3.3) by their correspond-ing physical counterparts.The listing below shows the elements within the status ofthe aggregation operator stopped before performing the re-organization phase triggered by the third incoming elementfor the example given in Section 2.6.

((18.280), [03/11/1993 05:00:08, 03/11/1993 05:01:32))((19.805), [03/11/1993 05:01:32, 03/11/1993 05:02:16))((19.766), [03/11/1993 05:02:16, 03/11/1993 05:15:08))((20.510), [03/11/1993 05:15:08, 03/11/1993 05:16:32))((19.690), [03/11/1993 05:16:32, 03/11/1993 05:17:16))

Because the start timestamp 05:02:16 of the third elementis greater than the end timestamp 05:01:32, the reorganiza-tion phase produces the first element of our listing as resultand removes it from the status.

4.2.5 Coalesce and SplitThe coalesce operator merges value-equivalent stream ele-

ments with adjacent time intervals, while the split operatorinverts this operation by splitting a stream element into sev-eral value-equivalent elements with adjacent time intervals.

Figure 3: Aggregation operator

Note that both operations have no impact on the semanticsof a query, since the records are valid at the same points intime and their multiplicities remain unchanged.

However, these operators can effectively be used to con-trol stream rates as well as element expiration adaptively.The latter has direct impact on the memory usage of inter-nal data structures of stateful operators (see Section 4.2.3).Furthermore, earlier element expiration leads to earlier re-sults because most stateful operators produce their resultsduring the reorganization phase. Consequently, the coalesceand split operators can be used for physical optimization.Coalesce generally decreases stream rates at the expense ofa delayed element expiration in internal data structures. Incontrast, split usually leads to earlier results and a reducedmemory consumption in internal data structures at the ex-pense of higher stream rates, which may cause an increase inthe size of intermediate buffers. Hence, coalesce and split of-fer a way to adaptively control the tradeoff between schedul-ing costs and memory usage.

These operators are novel in the stream context and theireffect on the runtime behavior of a DSMS will be investi-gated more detailed in our ongoing work.

4.3 PIPESPIPES (Public Infrastructure for Processing and Explor-

ing Streams) [17] is an infrastructure with fundamentalbuilding blocks that allow the construction of a fully func-tional DSMS tailored to a specific application scenario. Thecore of PIPES is a powerful and generic physical operatoralgebra whose semantics and implementation concepts havebeen presented in this paper. In addition, PIPES providesthe necessary runtime components such as the scheduler,memory manager, and query optimizer to execute physicaloperator plans.

Since PIPES seamlessly extends the Java library XXL [5]towards continuous data-driven query processing over au-tonomous data sources, it has full access to XXL’s queryprocessing frameworks such as the extended relational al-gebra, connectivity to remote data sources or index struc-tures. Therefore, PIPES inherently offers to run queries overstreams and relations [3].

5. QUERY OPTIMIZATIONThe foundation for a logical as well as physical query opti-

mization is a precisely defined semantics. Therefore, we for-mally presented a sound logical operator algebra over datastreams (see Section 3) that is expressive enough to supportstate-of-the-art continuous query processing.

Figure 4: Snapshot reducibility

It remains to provide an equivalence relation that defineswhen two logical query plans are equivalent. Based on thisdefinition, we also derive an equivalence relation for physicalstreams in this section.

5.1 Snapshot-ReducibilityIn order to define snapshot-reducibility, we first introduce

the timeslice operation that generates snapshots from a log-ical stream.

Definition 5. (Timeslice) The timeslice operator is a mapτt : (Sl × T ) → ℘(Ω× N) given by

τt(Sl) := (e, n) ∈ Ω× N | (e, t, n) ∈ Sl (12)

For a given logical stream Sl and a specified point in timet, the timeslice operation returns a non-temporal multisetof all records in Sl that are valid at time instant t. Notethat the argument timestamp is given as subscript. Thecorresponding schema results from a projection to the recordschema and the multiplicity attribute.

Definition 6. (Snapshot-Reducibility) A logical stream op-erator opT is snapshot-reducible to its non-temporal coun-terpart op over multisets, if for any point in time t ∈ T andfor all logical input streams Sl

1, . . . , Sln ∈ Sl, the snapshot

at t of the results of applying opT to Sl1, . . . , S

ln is equal

to the results of applying op to the snapshot R1, . . . , Rn ofSl

1, . . . , Sln at time t.

For example, the duplicate elimination over logical streamsis snapshot-reducible to the duplicate elimination over mul-tisets. Figure 4 gives a commuting diagram that illustratessnapshot-reducibility.

Snapshot-reducibility is a well-known concept from thetemporal database community [22, 7] and guarantees thatthe semantics of a non-temporal operator is preserved inits more complex, temporal counterpart. If we assume therecord schema of a logical or physical stream to be relational,we can show via snapshot-reducibility that our operatorsextend the well-understood semantics of the extended rela-tional algebra. In addition, we introduced novel temporaloperators, like the window operator, in order to provide anadequate basis for temporal continuous query formulationand execution over data streams.

Applying snapshot-reducibility, we can also prove thatour semantics covers the relational approach proposed byArasu et al. [3], while maintaining the advantages of ourimplementation described in Section 4.

5.2 Stream EquivalencesBased on the timeslice operator, we define the following

equivalence relations for schema-compliant logical and phys-ical streams, respectively:

Definition 7. (Logical stream equivalence) We define twological streams Sl

1, Sl2 ∈ Sl to be equal iff all snapshots of

them are equal.

Sl1

.= Sl

2 :⇔ ∀ t ∈ T. τt(Sl1) = τt(S

l2) (13)

Definition 8. (Physical stream equivalence) Let Sp1 = (M1,

≤ts,te), Sp2 = (M2,≤ts,te) ∈ Sp be two physical streams. We

denote two physical streams as snapshot-equivalent iff theircorresponding logical streams are equal.

Sp1 w Sp

2 :⇔ τ(M1).= τ(M2) (14)

Note that snapshot-equivalence over physical streams ab-stracts from their ordering.

We denote two query plans over the same set of inputstreams as equivalent if each output stream of the first queryplan is stream-equivalent and schema-compliant to exactlyone output stream of the second query plan, and vice versa.

5.3 Transformation RulesBased on the previous equivalence relations that rely

on snapshot equivalence over multisets, we can derive aplethora of transformation rules to optimize algebraic ex-pression, i. e. logical query plans. Due to the fact that wedefined most of our operations, except group and window,in compliance with [23], the huge set of conventional andtemporal transformation rules for snapshot-equivalence overmultisets listed in [23] also holds in the stream context. Thisincludes common transformation rules such as join reorder-ing or predicate pushdown, and additional temporal trans-formation rules for duplicate elimination, coalescing etc.

Example: Figure 2 (b) depicts a possible algebraic opti-mization of the query plan in our example query by pushingthe selection down the union operator. There, we apply ageneralized variant of the following transformation rule fortwo logical streams Sl

1, Sl2:

σp(Sl1 ∪+ Sl

2).= σp(Sl

1) ∪+ σp(Sl2)

The physical union operation is a stateful operator thatinternally reorders the incoming elements to ensure the or-dering invariant of the physical output stream. Therefore,this transformation rule generally reduces the memory us-age of the union operator.

The transformation rules for aggregation specified in [23]are only applicable if we combine our group, aggregationand union operator such that we compute the aggregate foreach group and merge the results of the aggregation op-erators. However, we decided to split the group and ag-gregation operations because in the context of continuousstream processing, a group operation that splits an inputstream into multiple output streams (see Section 3.2.6) maybe beneficial for subquery sharing.

These transformation rules are only a first step in staticand dynamic query optimization over data streams and re-lations. Due to continuous queries, DSMS generally run a

large number of queries in parallel. So, it is not sufficient toapply transformation rules solely for a single query plan. In-stead, the complete query graph should be optimized. Thisincludes the sharing of preferably large subqueries as wellas the need for a dynamic re-optimization of subgraphs dur-ing runtime. Currently, we are investigating to what extentresearch results from multi-query optimization [20, 19, 18]can be applied to optimize multiple continuous queries overstreams.

5.3.1 Window Transformation RulesThe window operator is typically placed near the sources

in a query plan because it sets the validity of the streamelements (see Section 2.6). Stateful operators (see Section4.2.1) use the end timestamps of stream elements for reor-ganization. For that reason, the window operator has to beplaced previous to the first stateful operator in a query planwhich means that it is not commutative with stateful opera-tors. However, we can derive some transformation rules forstateless operations. A stateless operator is commutativewith the window operator, if it does not consider the endtimestamp. Then, the following transformation rules holdfor a logical stream Sl ∈ Sl:

σp(ωw(Sl)).= ωw(σp(Sl))

µf (ωw(Sl)).= ωw(µf (Sl))

γf (ωw(Sl)).= (ωw(π(γf (Sl), 1)), . . . , ωw(π(γf (Sl), k)))

The group operation (see Section 3.2.6) produces a tuple of klogical streams since it splits the logical input stream Sl intok groups according to a user-defined function. Therefore,the window operator ωw has to be applied to each logicaloutput stream.

6. RELATED WORKOur work is closely related to multiset (bag) semantics

and algebraic equivalences for the relational algebra [11,12] as well as their temporal extensions [22, 23]. It basi-cally transfers the approach of [23] to continuous queriesover data streams. Thereby, we abstract from relationalschemes, introduce windowing constructs and provide ad-equate non-blocking operator implementations. This leadsto a snapshot-equivalent output to the operations presentedin [23] and therefore, a plethora of transformation rules isapplicable for optimizing continuous queries over streams.Whereas Slivinskas et al. specify their operations from animplementation point of view with the λ-calculus, we presenta suitable temporal logical operator algebra as well, whichdefines the semantics of the operations in a more intuitiveway while abstracting from a particular implementation.In the context of continuous queries over streams, therehas also been considerable research. Tribeca [25] intro-duces fixed and moving window queries over single networkstreams. TelegraphCQ [9] relies on a declarative language toexpress a sequence of windows over a stream, whereas Gigas-cope [10] and [27] try to unblock operations by using streamconstraints instead of windows. Aurora [8, 1] builds a querygraph of stream operators parameterized by functions andpredicates while abstracting from a certain query language,which is similar to our approach. Contrary to PIPES, theoperations in Aurora are defined in a procedural manner and

allow out-of-order elements in streams as well as certain ac-tions which may cause a nondeterministic semantics due toscheduling dependencies. The Tapestry system [26] trans-forms a continuous query into an incremental query thatis run periodically. Tapestry ensures snapshot-reducibilitybut does not support any kind of window queries. [2, 3]propose an abstract semantics for a concrete query languageover streams and relations supporting only sliding windows.From a semantical point of view, our approach is at least asexpressive as [2] due to snapshot-reducibility, whereas ourimplementation significantly benefits from our stream-to-stream operators incorporating time intervals. This avoidsthe drawback of higher stream rates that arise due to send-ing positive and negative tuples through a query plan, inorder to incrementally maintain the internal relations in theSTREAM system correctly. [14] also prefer the positive-negative tuple approach and focus on particular operatorimplementations.In a broader context, our approach is related to sequencedatabases [21] since raw input streams are a temporallyordered sequence of records. Note that the semantics ofsequence languages includes one-time, but not continuousqueries. The chronicle data model [16] provides operatorsover relations and chronicles, which can be considered as araw input stream, but focuses on the space complexity of anincremental maintenance of materialized views over chron-icles. It does not include continuous queries or aspects ofdata-driven processing. We also refer the interested readerto [3, 13] for a broader overview on data stream processing.

7. CONCLUSIONSDue to lack of a formal specification of the semantics of

continuous queries over data streams, we first proposed asound temporal logical operator algebra by exploiting andextending the well-known semantics of the extended rela-tional algebra as well as existing work in temporal data-bases. Second, we described the main implementation issuesof our physical operator algebra that relies on efficient, non-blocking, data-driven, stream-to-stream implementations ofthe logical operations. To the best of our knowledge thisapproach is unique as it assigns stream elements with timeintervals modeling their validity independent from the gran-ularity of time. Due to snapshot-reducibility our approach islogically compliant to related approaches [3], while it doesnot suffer from higher stream rates arising from positive-negative elements used in related approaches to indicate el-ement expiration. We explained why the physical operationsproduce sound results in terms of a snapshot-equivalent out-put to the logical operations. We further showed how toeffectively reorganize stateful operators based on the timeintervals of incoming elements and the ordering invariantassumed for streams. Third, we derived appropriate streamequivalences based on snapshot-multiset equivalence whichallows us to apply most conventional as well as temporaltransformation rules. To support sliding and fixed windowqueries, we furthermore introduced a novel window opera-tor by defining its semantics, describing its implementationand extending the set of transformation rules. Moreover,we motivated a novel kind of physical optimization in thestream context by proposing two physical operators, coa-lesce and split, which can effectively be used to adaptively

influence the runtime behavior of a DSMS with regard tostream rates, memory consumption as well as early results.Consequently, our work forms a solid foundation for queryformulation and optimization in continuous query processingover data streams, while it relies on the common, well-knownsteps from query formulation to query execution establishedin DBMS.

We already proved the feasibility of our approach duringthe development of PIPES [17], our infrastructure for con-tinuous query processing over heterogeneous data sources,where the temporal semantics proposed in this paper wasimplemented.

AcknowledgmentsThis project has been supported by the German ResearchSociety (DFG) under grant no. SE 553/4-1. In addition, weare grateful to Michael Cammert and Christoph Heinz forthe helpful discussions on the semantics of stream opera-tions.

8. REFERENCES[1] D. J. Abadi, D. Carney, U. Cetintemel, M. Cherniack,

C. Convey, S. Lee, M. Stonebraker, N. Tatbul, and S. B.Zdonik. Aurora: A New Model and Architecture for DataStream Management. VLDB Journal, 12(2):120–139, 2003.

[2] A. Arasu, S. Babu, and J. Widom. An Abstract Semanticsand Concrete Language for Continuous Queries overStreams and Relations. In Proc. of the Intl. Conf. on DataBase Programming Languages (DBPL), 2003.

[3] A. Arasu, S. Babu, and J. Widom. The CQL ContinuousQuery Language: Semantic Foundations and QueryExecution. Technical report, Stanford University, 2003.

[4] B. Babcock, S. Babu, M. Datar, R. Motwani, andJ. Widom. Models and Issues in Data Stream Systems. InSymp. on Principles of Database Systems (PODS), pages1–16, 2002.

[5] J. Bercken, B. Blohsfeld, J.-P. Dittrich, J. Kramer,T. Schafer, M. Schneider, and B. Seeger. XXL - A LibraryApproach to Supporting Efficient Implementations ofAdvanced Database Queries. In Proc. of the Conf. on VeryLarge Databases (VLDB), pages 39–48, 2001.

[6] C. Bettini, C. E. Dyreson, W. S. Evans, R. T. Snodgrass,and X. S. Wang. A Glossary of Time Granularity Concepts.In Temporal Databases: Research and Practice, pages406–413. Lecture Notes in Computer Science, 1997.

[7] M. H. Bohlen, R. Busatto, and C. S. Jensen. Point-VersusInterval-Based Temporal Data Models. In Proc. of theIEEE Conference on Data Engineering (ICDE), pages192–200, 1998.

[8] D. Carney, U. Cetintemel, M. Cherniack, C. Convey,S. Lee, G. Seidman, M. Stonebraker, N. Tatbul, and S. B.Zdonik. Monitoring Streams: A New Class of DataManagement Applications. In Proc. of the Conf. on VeryLarge Databases (VLDB), pages 215–226, 2002.

[9] S. Chandrasekaran, O. Cooper, and A. D. et al.TelegraphCQ: Continuous Dataflow Processing for anUncertain World. In Proc. of the Conf. on Innovative DataSystems Research (CIDR), 2003.

[10] C. D. Cranor, T. Johnson, O. Spatscheck, andV. Shkapenyuk. Gigascope: A Stream Database forNetwork Applications. In Proc. of the ACM SIGMOD,pages 647–651, 2003.

[11] U. Dayal, N. Goodman, and R. H. Katz. An ExtendedRelational Algebra with Control Over DuplicateElimination. In Proc. of the ACM SIGMOD, pages117–123, 1982.

[12] H. Garcia-Molina, J. D. Ullman, and J. Widom. DatabaseSystem Implementation. Prentice Hall, 2000.

[13] L. Golab and M. T. Ozsu. Issues in Data StreamManagement. SIGMOD Record, 32(2):5–14, 2003.

[14] M. Hammad, W. Aref, M. Franklin, M. Mokbel, andA. Elmagarmid. Efficient Execution of Sliding WindowQueries over Data Streams. Technical report, PurdueUniversity, 2003.

[15] J. M. Hellerstein, P. J. Haas, and H. Wang. OnlineAggregation. In Proc. of the ACM SIGMOD, pages171–182, 1997.

[16] H. V. Jagadish, I. S. Mumick, and A. Silberschatz. ViewMaintenance Issues for the Chronicle Data Model. In Proc.of the ACM SIGMOD, pages 113–124, 1995.

[17] J. Kramer and B. Seeger. PIPES - A Public Infrastructurefor Processing and Exploring Streams. In Proc. of the ACMSIGMOD, pages 925–926, 2004.

[18] T. Y. C. Leung and R. R. Muntz. Stream Processing:Temporal Query Processing and Optimization. In TemporalDatabases: Theory, Design, and Implementation, pages329–355. Benjamin/Cummings, 1993.

[19] P. Roy, S. Seshadri, S. Sudarshan, and S. Bhobe. Efficientand Extensible Algorithms for Multi Query Optimization.In Proc. of the ACM SIGMOD, pages 249–260, 2000.

[20] T. K. Sellis. Multiple-Query Optimization. ACMTransactions on Database Systems (TODS), 13(1):23–52,1988.

[21] P. Seshadri, M. Livny, and R. Ramakrishnan. The Designand Implementation of a Sequence Database System. InProc. of the Conf. on Very Large Databases (VLDB),pages 99–110, 1996.

[22] G. Slivinskas, C. S. Jensen, and R. T. Snodgrass. QueryPlans for Conventional and Temporal Queries InvolvingDuplicates and Ordering. In Proc. of the IEEE Conferenceon Data Engineering (ICDE), pages 547–558, 2000.

[23] G. Slivinskas, C. S. Jensen, and R. T. Snodgrass. AFoundation for Conventional and Temporal QueryOptimization Addressing Duplicates and Ordering. IEEETransactions on Knowledge and Data Engineering(TKDE), 13(1):21–49, 2001.

[24] U. Srivastava and J. Widom. Flexible Time Management inData Stream Systems. In Symp. on Principles of DatabaseSystems (PODS), pages 263–274, 2004.

[25] M. Sullivan and A. Heybey. Tribeca: A System forManaging Large Databases of Network Traffic. In In Proc.of the USENIX Annual Technical Conference, pages 13–24,1998.

[26] D. B. Terry, D. Goldberg, D. Nichols, and B. M. Oki.Continuous Queries over Append-Only Databases. In Proc.of the ACM SIGMOD, pages 321–330, 1992.

[27] P. A. Tucker, D. Maier, T. Sheard, and L. Fegaras.Exploiting Punctuation Semantics in Continuous DataStreams. Transactions on Knowledge and DataEngineering, 15(3):555–568, 2001.

[28] S. D. Viglas and J. F. Naughton. Rate-based QueryOptimization for Streaming Information Sources. In Proc.of the ACM SIGMOD, pages 37–48, 2002.

[29] H. Wang, C. Zaniolo, and C. Luo. ATLAS: A Small butComplete SQL Extension for Data Mining and DataStreams. In Proc. of the Conf. on Very Large Databases(VLDB), pages 1113–1116, 2003.

[30] Y. Zhu, E. A. Rundensteiner, and G. T. Heineman.Dynamic Plan Migration for Continuous Queries Over DataStreams. In Proc. of the ACM SIGMOD, pages 431–442,2004.

APPENDIX

A. DERIVED OPERATIONSIn this appendix, we shortly adapt some common but

more complex operations known from traditional DBMS to-wards continuous query processing. We do not consider thefollowing operations logically as basic operations, since theycan be derived from the basic ones defined in Section 3.2.Let Sl

1, Sl2 be logical streams.

A.1 Theta-JoinA theta join is a map ./p,f : Sl × P× Fmap → Sl. Let p be

a filter predicate that selects the qualifying join results fromthe Cartesian product. Let f be a mapping function thatcreates the resulting join tuples. We define a theta-join as:

./p,f (Sl1, S

l2) := µf (σp(Sl

1 × Sl2)) (15)

A.2 Semi-JoinA semi-join is a special join operation that returns all

elements of Sl1 that join with an element of Sl

2 according toa join predicate p. For that reason, the mapping functionf in the join definition is replaced by a projection on theschema of Sl

1.

np(Sl1, S

l2) := Sl

1 ./p,µπ(Sl

1)δ(Sl

2) (16)

A.3 IntersectionThe intersection, ∩ : Sl × Sl → Sl, of two logical streams

Sl1 and Sl

2 can be expressed with the help of the differenceoperation.

∩(Sl1, S

l2) := Sl

1 − (Sl1 − Sl

2) (17)

A.4 Max-UnionThe max-union operation, ∪max : Sl × Sl → Sl sets the

multiplicity of an element to its maximum multiplicity inone of the logical input streams Sl

1, Sl2 ∈ Sl.

∪max(Sl1, S

l2) := (e, t, n) | (∃ n1. (e, t, n1) ∈ Sl

1

∧ ∃ n2. (e, t, n2) ∈ Sl2 ∧ n = maxn1, n2)

∨ (∃ nj . (e, t, nj) ∈ Slj ∧ @ nk. (e, t, nk) ∈ Sl

k

∧ n = nj for j, k ∈ 1, 2 ∧ j 6= k)= (Sl

1 − Sl2) ∪+ (Sl

2 − Sl1) ∪+ (Sl

1 ∩ Sl2)

(18)

This definition complies with the one proposed by [23].

A.5 Strict DifferenceDue to multiset semantics we also want to introduce a

strict difference operation which differs from the differencepresented in Subsection 3.2.5 by eliminating duplicates inthe result:

−strict(Sl1, S

l2) := Sl

1 − (Sl1 n= Sl

2) (19)

The semi-join n= determines all elements in Sl1 that are

equal to an element in Sl2.

However, from a implementation point of view it may notbe sufficient to compose these operations of the basic ones.For instance, the join can be implemented much more effec-tively and efficiently from scratch. For that reason, PIPESprovides specific implementations in addition.