12
Approximate Aggregation Techniques for Sensor Databases
Je�rey Considine, Feifei Li, George Kollios, and John Byers
Computer Science Dept., Boston University
fjconsidi, lifeifei, gkollios, [email protected]
Abstract
In the emerging area of sensor-based systems, a sig-ni�cant challenge is to develop scalable, fault-tolerantmethods to extract useful information from the data thesensors collect. An approach to this data managementproblem is the use of sensor database systems, exempli-�ed by TinyDB and Cougar, which allow users to per-form aggregation queries such as MIN, COUNT andAVG on a sensor network. Due to power and range con-straints, centralized approaches are generally impracti-cal, so most systems use in-network aggregation to re-duce network traÆc. However, these aggregation strate-gies become bandwidth-intensive when combined with thefault-tolerant, multi-path routing methods often used inthese environments. For example, duplicate-sensitive ag-gregates such as SUM cannot be computed exactly us-ing substantially less bandwidth than explicit enumera-tion. To avoid this expense, we investigate the use of ap-proximate in-network aggregation using small sketches.Our contributions are as follows: 1) we generalize wellknown duplicate-insensitive sketches for approximatingCOUNTto handle SUM, 2)we present and analyzemeth-ods for using sketches to produce accurate results withlow communication and computation overhead, and 3)we present an extensive experimental validation of ourmethods.
1. Introduction
As computation-enabled devices shrink in scale andproliferate in quantity, a relatively recent research di-rection has emerged to contemplate future applicationsof these devices and services to support them. A canon-ical example of such a device is a sensor mote, a de-vice with measurement, communication, and computa-tion capabilities, powered by a small battery [12]. In-dividually, these motes have limited capabilities, butwhen a large number of them are networked togetherinto a sensor network, they become much more capa-ble. Indeed, large-scale sensor networks are now beingapplied experimentally in a wide variety of areas |some sample applications include environmental mon-itoring, surveillance, and traÆc monitoring.
In a typical sensor network, each sensor producesa stream of sensory observations across one or moresensing modalities. But for many applications and sens-ing modalities, such as reporting temperature readings,it is unnecessary for each sensor to report its entiredata stream in full �delity. Moreover, in a resource-constrained sensor network environment, each messagetransmission is a signi�cant, energy-expending oper-ation. For this reason, and because individual read-ings may be noisy or unavailable, it is natural touse data aggregation to summarize information col-lected by sensors. As a re ection of this, a databaseapproach to managing data collected on sensor net-works has been advocated [15, 20], with particular at-tention paid to eÆcient query processing for aggregatequeries [15, 20, 23].
In the TAG system [15], users connect to the sen-sor network using a workstation or base station directlyconnected to a sensor designated as the sink. Aggregatequeries over the sensor data are formulated using a sim-ple SQL-like language, then distributed across the net-work. Aggregate results are sent back to the worksta-tion over a spanning tree, with each sensor combiningits own data with results received from its children. Ifthere are no failures, this in-network aggregation tech-nique is both e�ective and energy-eÆcient for distribu-tive and algebraic aggregates [11] such as MIN, MAX,COUNT and AVG. However, as we will argue, thistechnique is much less e�ective in sensor network sce-narios with moderate node and link failure rates. Nodefailure is inevitable when inexpensive, faulty compo-nents are placed in a a variety of uncontrolled or evenhostile environments. Similarly, link failures and packetlosses are common across wireless channels because ofenvironmental interference, packet collisions, and lowsignal-to-noise ratios [23].
When a spanning tree approach is used for aggre-gate queries, as in TAG, a single failure results in anentire subtree of values being lost. If this failure is closeto the sink, the change in the resulting aggregate can besigni�cant. Retransmission-based approaches are ex-pensive in this environment, so solutions based uponmulti-path routing were proposed in [15]. For aggre-
gates such as MIN and MAX which are monotonic andexemplary, this provides a fault-tolerant solution. Butfor duplicate-sensitive aggregates such as COUNT orAVG, that give incorrect results when the same valueis counted multiple times, existing methods are not sat-isfactory.
In this paper, we propose a robust and scalablemethod for computing duplicate-sensitive aggregatesacross faulty sensor networks. Guaranteeing exact so-lutions in the face of losses is generally impractical, sowe instead consider approximate methods. These meth-ods are robust against both link and node failures. Ourcontributions can be summarized as follows:
� We extend well-known duplicate insensitivesketches [7] to handle SUM aggregates. Throughanalysis and experiments, we show that the newsketches provide accurate approximations.
� We present a method to combine duplicate insen-sitive sketches with multi-path routing techniquesto produce accurate approximations with low com-munication and computation overhead.
� We provide an analysis of the expected perfor-mance of previous methods as well as our method.
� Finally, we present an extensive experimental eval-uation of our proposed system which we comparewith previous approaches.
Concurrent with our work, Nath and Gibbons [17]independently studied the use of duplicate-insensitivesketches for aggregation in sensor networks. Their workfocused more on the logical decoupling of routing andaggregation, along with the necessary sketch propertiesfor correctness using multipath routing, namely havingan associative, commutative, and idempotent aggrega-tion operation.
The remainder of this paper proceeds as follows.Background material is covered in Section 2. Count-ing sketches, along with theory and new generaliza-tions, are discussed in Section 3. A robust aggrega-tion framework using these sketches is then presentedin Section 4. We validate our methods experimentallyin Section 5 and conclude in Section 6.
2. Background
We now brie y survey the related work of our meth-ods. Sensors and their limitations are described in Sec-tion 2.1. Previous frameworks for processing aggregatesare covered in 2.2, and multipath routing techniquesare covered in 2.3. Finally, the sketches which we useto improve upon these frameworks are introduced inSection 2.4.
2.1. Sensor Devices
Today's sensor motes (e.g. [12]) are full edged com-puter systems, with a CPU, main memory, operat-ing system and a suite of sensors. They are poweredby small batteries and their lifetime is primarily de-pendent on the extent to which battery power is con-served. The power consumption tends to be dominatedby transmitting and receiving messages and most sys-tems try to minimize the number of messages in orderto save power. Also, the communication between sen-sors is wireless and the packet loss rate between nodescan be high. For example, [23] reports on experimentsin which more than 10% of the links su�ered averageloss rate greater than 50%. Another challenge is thatlinks may be asymmetric, both in loss rates and evenreachability. These limitations motivate query evalua-tion methods in sensor networks that are fundamen-tally di�erent from the traditional distributed queryevaluation approaches. First, the query execution planmust be energy eÆcient and second, the process mustbe as robust as possible given the communication lim-itations in these networks.
2.2. In-network Aggregate Query Processing
A simple approach to evaluate an aggregation queryis to route all sensed values to the base station andcompute the aggregate there. Although this approachis simple, the number of messages and the power con-sumption can be large. A better approach is to lever-age the computational power of the sensor devices andcompute aggregates in-network. Aggregates that canbe computed in-network include all decomposable func-tions [15].
De�nition 1 A function f is decomposable, if itcan be computed by another function g as follows:f(v1; v2; :::; vn) = g(f(v1; :::; vk); f(vk+1; :::; vn)).
Using decomposable functions, the value of the ag-gregate function can be computed for disjoint subsets,and these values can be used to compute the aggre-gate of the whole using the merging function g. Ourdiscussion is based on the Tiny Aggregation (TAG)framework used in TinyDB [15]. However, similar ap-proaches are used to compute aggregates in other sys-tems [20, 21, 23, 13].
In TAG, the in-network query evaluation has twophases, the distribution phase and the collection phase.During the distribution phase, the query is ooded inthe network and organizes the nodes into an aggrega-tion tree. The base station broadcasting the query is theroot of the tree. The query message has a counter thatis incremented with each retransmission and counts thehop distance from the root. In this way, each node is
assigned to a speci�c level equal to the node's hop dis-tance from the root. Also, each sensor chooses one ofits neighbors with a smaller hop distance from the rootto be its parent in the aggregation tree.
During the collection phase, each leaf node producesa single tuple and forwards this tuple to its parent. Thenon-leaf nodes receive the tuples of their children andcombine these values. Then, they submit the new par-tial results to their own parents. This process runs con-tinuously and after h steps, where h is the height ofthe aggregation tree, the total result will arrive at theroot. In order to conserve energy, sensor nodes sleepas much as possible during each step where the pro-cessor and radio are idle. When a timer expires or anexternal event occurs, the device wakes and starts theprocessing and communication phases. At this point, itreceives the messages from its children and then sub-mits the new value(s) to its parent. After that, if nomore processing is needed for that step, it enters againinto the sleeping mode [16].
As mentioned earlier, this approach works very wellfor ideal network conditions, but is less satisfactory un-der lossy conditions. To address these issues, Maddenat al. [15] proposed various methods to improve theperformance of their system. One solution is to cacheprevious values and reuse them if newer ones are un-available. Of course, these cached values may re ectlosses at lower levels of the tree.
Another approach considered in [15] takes advan-tage of the fact that a node may select multiple par-ents from neighbors at a higher level. Using this ap-proach, which we refer to as \fractional parents," theaggregate value is decomposed into fractions equal tothe number of parents. Each fraction is then sent to adistinct parent instead of sending the whole value to asingle parent. For example, given an aggregate sum of15 and 2 parents, each parent would be sent the value7:5. It is easy to demonstrate analytically that this ap-proach does not improve the expected value of the es-timate over the single parent approach; it only helps toreduce the variance of the estimated value at the root.Therefore, the problem of losing a signi�cant fractionof the aggregate value due to network failures remains.
2.3. Best Effort Routing in Sensor Networks
Recent years have seen signi�cant work on best-e�ort routing routing in sensor and other wireless net-works. Due to high loss rates and power constraints, acommon approach is to use multi-path routing, wheremore than one copy of a packet is sent to the desti-nation over di�erent paths. For example, directed dif-fusion [13] uses a ood to discover short paths whichsensors would use to send back responses. Various pos-
itive and negative reinforment mechanisms are used toimprove path quality. Braided di�usion [8] builds on di-rected di�usion to use a set of intertwined paths for in-creased resilience. A slightly di�erent approach is usedby GRAB [22], where paths are not explicitly chosen,but the width of the upstream broadcast is controlled.
Our techniques are meant to complement and lever-age any of these routing techniques. We note thatcombining these methods with duplicate-insensitive in-network aggregation will allow some of the overhead ofthese techniques to be amortized and shared amongstdata items from many di�erent sensors.
2.4. Counting Sketches
Counting sketches were introduced by Flajolet andMartin in [7] for the purpose of quickly estimating thenumber of distinct items in a database (or stream) inone pass while using only a small amount of space.Since then, there has been much work developing andgeneralizing counting sketches (e.g. [1, 6, 10, 3, 9, 2]).
It is well known that exact solutions to the distinctcounting problem require (n) space. As shown in [1],�(log n) space is required to approximate the numberof distinct items in a multi-set with n distinct items.The original FM sketches of [7] achieve this bound,though they assume a �xed hash function that appearsrandom, so they are vulnerable to adversarial choicesof inputs. We use these sketches since they are verysmall and accurate in practice, and describe them indetail in Section 3.
A di�erent sketching scheme using linear hash func-tions was proposed in [1]. These sketches are somewhatlarger than FM sketches in practice, although a very re-cent technique [5] extending these methods uses onlyO(log logn) space. We intend to investigate the e�ec-tiveness of these \loglog" sketches for sensor databasesin future work.
3. Sketch Theory
One of the core ideas behind our work is that dupli-cate insensitive sketches will allow us to leverage therobustness typically associated with multi-path rout-ing. We now present some of the theory behind suchsketches and extend it to handle more interesting ag-gregates. First, we present details of the FM sketchesof [7] along with necessary parts of the theory behindthem. Then, we generalize these sketches to handlesummations, and show that they have almost exactlythe same accuracy as FM sketches.
3.1. Counting Sketches
We now describe FM sketches for the distinct count-ing problem.
De�nition 2 Given a multi-set of items M =fx1; x2; x3; : : : g, the distinct counting problem is tocompute n � jdistinct(M)j :
Given a multi-set M , the FM sketch of M , denotedS(M), is a bitmap of length k. The entries of S(M),denoted S(M)[0; : : : ; k� 1], are initialized to zero andare set to one using a random binary hash function happlied to the elements of M . Formally,
S(M)[i] � 1 i� 9x 2M s:t: minfj j h(x; j) = 1g = i:
By this de�nition, each item x is capable of setting asingle bit in S(M) to one { the minimum i for whichh(x; i) = 1. This gives a simple serial implementationwhich is very fast in practice and requires two invoca-tions of h per item on average.
Theorem 1 An element xi can be inserted into an FMsketch in O(1) expected time.
Algorithm 1 CountInsert(S,x)
1: i = 0;2: while hash(x,i) = 0 do3: i = i + 1;4: end while
5: S[i] = 1;
We now describe some interesting properties of thesketches observed in [7].
Property 1 The FM sketch of the union of two multi-sets is the bit-wise OR of their FM sketches. That is,S(M1 [M2)[i] = (S(M1)[i] _ S(M2)[i]):
Property 2 S(M) is entirely determined by the distinctitems ofM .Duplication and ordering donot a�ectS(M).
Property 1 allows each node to compute a sketchof locally held items and send the small sketch for ag-gregation elsewhere. Since aggregation via union oper-ations is cheap, it may be performed in the networkwithout signi�cant computational burden. Property 2allows the use of multi-path routing of the sketches forrobustness without a�ecting the accuracy of the esti-mates. We expand upon these ideas in Section 4.
The next lemma provides key insight into the be-havior of FM sketches and will be the basis of eÆcientimplementations of summation sketches later.
Lemma 1 For i < log2 n � 2 log2 log2 n, S(M)[i] = 1
with probability 1 � O(ne� log22 n). For i � 32 log2 n + Æ,
with Æ � 0, S(M)[i] = 0 with probability 1�O�2�Æpn
�.
Proof: This lemma is proven in [7] and follows frombasic balls and bins arguments.
The lemma implies that given an FM sketch of ndistinct items, one expects an initial pre�x of all onesand a suÆx of all zeros, while only the setting of thebits around S(M)[log2 n] exhibit much variation. Thisgives a bound on the number of bits k required forS(M) in general: k = 3
2 log2 n bits suÆce to repre-sent S(M) with high probability. It also suggests thatjust considering the length of the pre�x of all ones inthis sketch can produce an estimate of n. Formally, letRn � minfi j S(M)[i] = 0g when S(M) is an FMsketch of n distinct items. That is, Rn is a randomvariable marking the location of the �rst zero in S(M).In [7], a method to use Rn as an estimator for n is de-veloped using the following theorems.
Theorem 2 The expected value of Rn for FM sketchessatis�es E(Rn) = log2('n) + P (log2 n) + o(1); wherethe constant ' is approximately 0:775351 and P (u) is aperiodic and continuous function of u with period 1 andamplitude bounded by 10�5.
Theorem 3 The variance of Rn for FM sketches, de-noted �2n, satis�es �
2n = �21 + Q(log2 n) + o(1); where
constant �21 is approximately 1:12127 and Q(u) is a pe-riodic function with mean value 0 and period 1.
Thus, Rn can be used for an unbiased estimator oflog2 n if the small periodic term P (log2 n) is ignored.A much greater concern is that the variance is slightlymore than one, dwar�ng P (log2 n), and implying thatestimates of n will often be o� by a factor of two ormore in either direction. To address this, methods forreducing the variance will be discussed in Section 3.3.
3.2. Summation Sketches
As our �rst theoretical contribution, we generalizeapproximate counting sketches to handle summations.Given a multi-set of items M = fx1; x2; x3; : : : g wherexi = (ki; ci) and ci is a non-negative integer, the dis-tinct summation problem is to calculate
n �X
distinct((ki;ci)2M)
ci:
When ci is restricted to one, this is exactly the distinctcounting problem.
We note that for small values of ci, one mightsimply count ci di�erent items based upon ki andci, e.g. (ki; ci; 1); : : : ; (ki; ci; ci), which we denote sub-items of (ki; ci). Since this is merely ci invocations ofthe counting insertion routine, the analysis for proba-bilistic counting applies. Thus, this approach is equallyaccurate and takes O(ci) expected time. While verypractical for small ci values (and trivially paralleliz-able in hardware), this approach does not scale well
for large values of c. Therefore, we consider more scal-able alternatives for handling large ci values.
Algorithm 2 SumInsert(S,x,c)
1: d = pick threshold(c);2: for i = 0, : : : , d - 1 do3: S[i] = 1;4: end for
5: a = pick binomial(seed=(x, c), c, 1=2d);6: for i = 1, : : : , a do7: j = d;8: while hash(x,c,i,j) = 0 do9: j = j + 1;
10: end while
11: S[j] = 1;12: end for
The basic intuition beyond our more scalable ap-proach is as follows. We intend to set the bits in thesummation sketch as if we had performed ci successiveinsertions to an FM sketch, but we will do so muchmore eÆciently. The method proceeds in two steps:we �rst set a pre�x of the summation sketch bits toall ones, and then set the remaining bits by randomlysampling from the distribution of settings that the FMsketch would have used to set those bits. Ultimately,the distribution of the settings of the bits in the sum-mation sketch will bear a provably close resemblance tothe distribution of the settings of the bits in the equiv-alent FM sketch, and we then use the FM estimator toretrieve the value of the count.
We now describe the method in more detail. First, toset the pre�x, we observe that it follows from Lemma 1,that the �rst Æi = blog2 ci�2 log2 log cic bits of a count-ing sketch are set to one with high probability after ciinsertions. So our �rst step in inserting (ki; ci) into thesummation sketch is to set the �rst Æi bits to one. In theproof of Theorem 2 in [7], the authors prove that thecase where the �rst Æi bits are not all set to one only af-fects the expectation of Rn by O(n�0:49). In practice,we could correct for this small bias, but we disregardit in our subsequent aggregation experiments.
The second step sets the remaining k � Æi bits bydrawing a setting at random from the distribution in-duced by the FM sketch we are emulating. We do soby simulating the insertions of items that set bits Æiand higher in the counting sketch. First, we say aninsertion xi reaches bit z of a counting sketch if andonly if minfj j h(xi; j) = 1g � z. The distributionof the number of items reaching bit z is well-knownfor FM sketches. An item xi reaches bit z if and onlyif 80�j<z(h(xi; j) = 0), which occurs with probabil-ity 2�z. So for a set of ci insertions, the number ofinsertions reaching bit Æi follows a binomial distribu-
tion with parameters ci and 2�Æi . This leads to thefollowing process for setting bits Æi; Æi + 1 : : : k (ini-tialized to zero). First, draw a random sample y fromB(ci; 2
�Æi), and consider each of these y insertions ashaving reached bit Æi. Then use the FM coin- ippingprocess to explicitly set the remaining bits beyond Æi.
The pseudo-code for this approach is shown in Al-gorithm 2, and the analysis of its running time is pre-sented next.
Theorem 4 An element xi = (ki; ci) can be insertedinto a sum sketch in O(log2 ci) expected time.
Proof Sketch: Let �i denote the number of items cho-sen to reach Æi. Setting the �rst Æi bits takes O(Æi)time and simulating the �i insertions takes expectedO(�i) time. The total expected time to insert xi isthen O(Æi + f(�i) + �i), where f(�i) denotes the timeto pick �i. Thus, the time depends on both �i and themethod used to pick �i. By construction,
E(�i) = ci � 2�blog2 ci�2 log2 log cic;so
log2 ci � E(�i) < 2 log2 ci:
Selecting an appropriate method for picking �i re-quires more care. While there exist many eÆcientmethods for generating numbers from a binomial dis-tribution ([14] has a brief survey), these generally re-quire oating point operations or considerable mem-ory for pre-computed tables (linear in ci). Since exist-ing sensor motes often have neither, in Section 4.3.1we describe a space-eÆcient method that uses no oat-ing point operations, uses pre-computed tables of sizeO(c= log2 c), where c is an upper bound on the ci values,and individual insertions take time O(log2 ci). Combin-ing these results give the stated time bound.
We note that for small ci values, it may be faster touse a hybrid implementation combining the naive andscalable insertion functions. Especially for very low civalues, the naive insertion function will be faster.
Theorem 5 The expected value of Rn for sum sketchessatis�es E(Rn) = log2('n) + P (log2 n) + o(1); where 'and P (u) are the same as in Theorem 2.
Proof: The proof of this theorem follows the proof ofTheorem 2 since the sum insertion function approxi-mates repeated use of the count insertion function. Let
cmax = maxfci j (ki; ci) 2Mgand
Æmax = blog2 cmax � log2 log cmaxc:By the insertion method, the bottom Æmax bits ofS�(M) are guaranteed to be set. By construction, the
remaining bits are distributed identically to those of anFM sketch with n distinct items have inserted. Thus,the distribution of Rn is the same except for the caseswhen the FM sketch had one of the �rst Æmax bitsnot set. By Lemma 1, these cases occur with proba-bility O(ne� log2 n), so the di�erence in the expectation
is at most (log2 n � log2 logn) � O(ne� log2 n), whichis bounded (loosely) by O(1=n). Therefore, E(Rn)for summation sketches is within o(1) of that of FMsketches.
Theorem 6 The variance of Rn for sum sketches, alsodenoted �2n, satis�es �
2n = �21+Q(log2 n)+ o(1); where
�21 andQ(u) are as de�ned in Theorem 3.
Proof: The proof of Theorem 3 is adapted in a similarfashion.
3.3. Improving Accuracy
To improve the variance and con�dence of the es-timator, FM sketches can use multiple bitmaps. Thatis, each item is inserted into each of m independentbitmaps to produce m R values, Rh1i; : : : ; Rhmi. Theestimate is then calculated as follows:
n � (1=')2P
iRhii=m:
This estimate is more accurate, with standard errorO(1=
pm), but comes at the cost of increased insertion
times (O(m)). To avoid this overhead, an algorithmcalled Probabilistic Counting with Stochastic Averaging,or PCSA, was proposed in [7]. Instead of inserting eachitem into each of the m bitmaps, each item is hashedand inserted into only one of them. Thus, each of thebitmaps summarizes approximately n=m items. Whilethere is some variation in how many items are assignedto each bitmap, further analysis showed that the stan-dard error of PCSA is roughly 0:78=
pm. Using PCSA,
insertion takes O(1) expected time.PCSA can also be applied to summation sketches,
but greater care must be applied when combiningPCSA to summation sketches. The potential for imbal-ance is much larger with summation sketches - a sin-gle item can contribute an arbitrarily large fraction ofthe total sum. Thus, we employ the following strategy.Each ci value has the form ci = qim+ ri for some inte-gers qi and ri, with 0 � ri < m. We then add ri distinctitems once as in standard PCSA, and then add qi toeach bitmap independently. Thus, we preserve the bal-ance necessary for the improved accuracy and its anal-ysis, but at the cost of O(m log2(ci=m)) for each inser-tion. We will employ PCSA in our experiments.
3.4. Tradeoffs and Other Approaches
In situations where computational resources areseverely constrained, it may be desirable to re-
duce the cost of performing insertion operationswith summation sketches. We now brie y men-tion some tradeo�s in the computational time at thecost of increased communication and decreased accu-racy. While this is unlikely to be desirable in sensornetworks, given the high power costs of commu-nication relative to computation, it may be desir-able in other settings where there are large numbersof items per node.
Suppose that the largest value being inserted isbounded by yx. Insertions with the algorithm describedalready take O(x2 log2 y) time. We can instead use xdi�erent summation sketches, each corresponding to adi�erent digit of the ci's using radix y. To add a civalue, each digit of ci is inserted into the correspond-ing sketch, taking expected O(x log2 y) time, and esti-mates are made by summing the counting sketch esti-mates with the appropriate weights. The accuracy ofthis approach is essentially the same as before, and theincrease in space is bounded by a factor of x.
An alternative approach for reducing the space over-head is to replace FM sketches with the very recentlydeveloped \loglog" sketches of [5]. The reduction ofSection 3.2 can be applied similarly, again with smalle�ects on accuracy. In parallel with our work, thesketches of [1] were adapted to summations in [17], buttheir methods involve both logarithms and exponenti-ation, making them unsuitable for sensor networks.
3.5. Other Aggregates
So far, we have only discussed two aggregates,COUNT and SUM. These techniques can also be ex-tended to other aggregate functions beyond summationand counting. For example, AVG can also be com-puted directly from COUNT and SUM sketches.The second moment can be computed as an aver-age of the squares of the items, and then combinedwith the average of the items, to compute the vari-ance and standard deviation. Finally, we note thatthe sketches themselves are easily generalized to han-dle other data types such as �xed point and signednumbers, and to a certain extent, products (sum-ming logarithms) and oating point.
4. Approximate Estimation of Dupli-
cate Sensitive Aggregates
In this section, we show how to use duplicate insensi-tive sketches to build a robust, loss-resilient frameworkfor aggregation. First, our algorithm for leveraging thebroadcast nature of wireless communication in combi-nation with sketching techniques is described in Sec-tion 4.1. A simple analytic evaluation of the proposed
methods is given in Section 4.2. Finally, practical de-tails of implementations on sensor motes are given inSection 4.3.
4.1. Algorithm and Discussion
Our methods for aggregation leverage two main ob-servations. First, the wireless communication of sensornetworks gives the ability to broadcast a single mes-sage to multiple neighbors simultaneously. Second, theduplicate-insensitive sketches discussed in Section 3 al-low a sensor to combine all of its received sketches intoa single message to be sent. Given proper synchroniza-tion, this will allow us to robustly aggregate data witheach sensor sending just one broadcast.
For simplicity, the remainder of this section will fo-cus on continuous queries (one-shot queries simply ter-minate earlier). Given a new continuous query, thecomputation proceeds in two phases. In the �rst phase,the query is distributed across the sensor network, of-ten using some form of ooding. During this phase,each node also computes its level (i.e. its hop distancefrom the root), and notes the level values of its immedi-ate neighbors. The second phase is divided into a seriesof epochs speci�ed by the query. The speci�ed aggre-gate will be computed once for each epoch.
At the beginning of each epoch, each node con-structs a sketch of its local values for the aggregate.The epoch is then sub-divided into a series of rounds,one for each level, starting with the highest (farthest)level. In each round, the nodes at the correspondinglevel broadcast their sketches, and the nodes at thenext level receive these sketches and combine themwith their sketches in progress. In the last round, thesink receives the sketches of its neighbors, and com-bines them to produce the �nal aggregate.
As an example, we step through a single epoch ag-gregating over the topology of Figure 1. First, eachnode creates a fresh sketch summarizing its own ob-served values. In the �rst round of communication,nodes at level 3 broadcast their sketches, which arethen received by neighboring level 2 nodes and com-bined with the sketches of the level 2 nodes. In the sec-ond round, nodes at level 2 broadcast their sketches,which are then received by neighboring level 1 nodesand combined with the sketches of the level 1 nodes.In the third and last round, nodes at level 1 send theirsketches to the sink, which combines them and extractsthe �nal aggregate value. Note that each node in thistopology except those on the diagonals has multipleshortest paths which are e�ectively used, and a valuewill be included in the �nal aggregate unless all of itspaths su�er from losses.
Nodes at Level 2 Root
Figure 1. Routing topology for 49 node grid.
The tight synchronization described so far is not ac-tually necessary. Our methods can also be applied us-ing gossip-style communication - the main advantageof synchronization and rounds is that better schedul-ing is possible and power consumption can be reduced.However, if a node receives no acknowledgments of itsbroadcast, it may be reasonable in practice to retrans-mit. More generally, loosening the synchronization in-creases the robustness of the �nal aggregate as pathstaking more hops are used to route around failures.This increased robustness comes at the cost of powerconsumption, since nodes broadcast and receive moreoften (due to values arriving later than expected) andincreased time (and variability) to compute the �nalaggregate. As mentioned earlier, this general principleallows us to make use of any best-e�ort routing proto-col (e.g. [13, 8]), with the main performance metric ofinterest being the delivery ratio.
4.2. Analysis
We now analyze the methods discussed so far for arestricted class of regular topologies. We compare theresilience of a single spanning tree against using mul-tiple parents, but only one broadcast per node, as de-scribed in the previous section. For simplicity, we onlyconsider exact COUNT aggregate under independentlink failures; more elaborate analysis for other aggre-gates and failure models is possible. These calculationstend to be \back of the envelope" in nature; they illus-trate the advantages of multipath routing over span-ning trees for resilience. For more detailed analysis, werefer the reader to work such as [22].
In the following, we use p as the probability of (in-dependent) link loss, and h as the maximum numberof hops from the sink.
4.2.1. Fault Resilience of the Spanning TreeFirst, we consider a baseline routing topology in whichaggregates are computed across a single spanning tree.
For simplicity, we assume that we have a completed-ary tree of height h. In general, the probability ofa value from a node at level i to reach the root isproportional to (1 � p)i. The expected value of the
COUNT aggregate is E(count) =Ph
i=0(1 � p)ini,where ni is the number of nodes at level i. This gives
us E(count) =Ph
i=0((1 � p)d)i = (d�pd)h+1�1d�pd�1 . For
h = 10, d = 3 and p = 0:1 (a 10% link loss rate) theexpected fraction of the nodes that will be counted ispoor, only 0:369.
4.2.2. Fault Resilience of Multiple Paths In or-der to analyze the use of multiple paths we now make astronger assumption about the routing topology. Start-ing with the leaves at level 0, we assume that each nodeat level i has exactly d neighbors within its broadcastradius at level i + 1, for all 0 � i � h� 1. From theseneighbors, each node selects k � d of these nodes as itsparents, where k is a fault-resilience parameter, and ittransmits its aggregate value to all k of these nodes. Weuse the pessimistic simpli�cation that only one copy ofa leaf's value reaches a level; while somewhat tighterbounds can be obtained, it suÆces to provide closeagreement with our experimental results. Let Ei de-note the event that a copy of the leaf's value reachedlevel i conditioned on the value having reached leveli + 1. With leaves at level h, these events are well-de�ned for levels 1; 2 : : : h�1. Clearly Pr[Ei] � (1�pk)(from the above simpli�cation), and thus the overallprobability of a message successfully reaching the rootis �i Pr[Ei] � (1 � pk)h. Using the same argumentfor the other levels of the tree we can get the follow-
ing: E(count) � Phi=0(1 � pk)ini =
(d�pkd)h+1�1d�pkd�1 . For
k = 2, p = 0:1 and h = 10 we get E(count) � 0:9n,where n is the total number of nodes. For k = 3 thebound is close to 0:99n, thus we have only a 1% degra-dation in the set of reporting sensors.
4.3. Practical Details
Since our protocols are being developed for use insensor networks, it is important to ensure that they donot exceed the capabilities of individual sensors. Sec-tion 4.3.1 considers the computational costs of gen-erating random numbers for the summation sketchesof 3.2. Section 4.3.2 considers the bandwidth overheadof sending sketches.
4.3.1. Binomial Random Number GenerationExisting sensor motes have a small word size (8 or16 bits), lack oating point hardware and have littleavailable memory for pre-computed tables. For thesereasons, standard methods for drawing from the bino-mial distribution are unsuitable. Here, we outline a ran-domized algorithm which draws from B(n; p) in O(np)
expected running time using O(1=p) space in a pre-computed table and without use of oating point oper-ations. We �rst note the following relationship betweendrawing from the binomial distribution and drawingfrom the geometric distribution, also used in [4].
Fact 1 Suppose we have a method to repeatedly draw atrandom from the geometric distribution G(1 � p). Let dbe the random variable that records the number of drawsfrom G(1 � p) until the sum of the draws exceeds n. Thevalue d � 1 is then equivalent to a random draw fromB(n; p).
The expected number of draws d from the geomet-ric distribution using this method is np, so to bound theexpected running time to draw from B(n; p), we simplyneed to bound the running time to draw from G(1�p).We will make use of the elegant alias method of [19] todo so in O(1) expected time. In [19] Walker demon-strates the following (which has a simple and beautifulimplementation):
Theorem 7 (Walker) For any discrete probabil-ity density function D over a sample space of size k, atable of sizeO(k) can be constructed inO(k) time that en-ables random variables to be drawn from D using twotable lookups.
We can apply this method directly to construct a ta-ble of size n+1 in which the �rst n elements of the pdfDrespectively correspond to the probabilities pi of draw-ing 1 � i � n from the geometric distribution G(1�p),and the �nal element corresponds to the tail probabil-ity of drawing any value strictly larger than n fromG(1�p). Note that for simulating a draw from B(n; p)using the method implicitly de�ned by Fact 1, we nevercare about the exact value of a draw from G(1�p) thatis larger than n. This direct application enables O(1)draws from G(1� p) in O(n) space, thus yields O(np)expected running time to draw from B(n; p).
To achieve O(1=p) space, we make use of the memo-ryless property of the geometric distribution (which thebinomial distribution does not have). Instead of stor-ing the �rst n probabilities pi for the geometric distri-bution, we store only the �rst d1=pe such probabilities,and a �nal element corresponding to the tail probabil-ity of drawing any value strictly larger than d1=pe fromG(1� p). By the memorylessness property, if we selectthe event corresponding to the tail probability, we canrecursively draw again from the table, setting our out-come to d1=pe + x, where x is the result of the recur-sive draw. The recursion terminates whenever one ofthe �rst d1=pe events in the table is selected, or when-ever the accumulated result exceeds n. Since 1=p is theexpectation of G(1�p), this recursion terminates with
constant probability at each round, and thus the ex-pected number of table lookups is O(1). Further re-duction in space is possible, but at the cost of incur-ring a commensurate increase in the expected numberof recursive calls.
Using table sizes of d1=pe and assuming a maxi-mum sensor value of ci � 216 (from a 16 bit wordsize), the lowest value of p used in summation sketcheswill be 162=216 = 1=28. Therefore, we will have ta-bles for p = 1=21; : : : ; 1=28, with 2; : : : ; 28 entries each,respectively. Walker's method utilizes two values foreach entry - the �rst is an index into the table andthe second is a real value used for comparison. The in-dex value only requires one byte since the largest ta-ble size is 28, and a 64 bit �xed-point real value (8bytes) should more than suÆce. This gives a total ta-
ble size ofP8
i=1(2i) � (1 + 8) = 4590 bytes. This can
be improved further by reducing the number of entriesin each table as mentioned before. The smaller tables(e.g. for p = 1=2 and p = 1=4) can also be removed infavor of directly simulating the \coin ips" of the geo-metric distribution, but with negligible space savings.
4.3.2. Sketch Sizes and Compression As men-tioned earlier, the other main limitation of sensor net-works is their limited bandwidth. This limitation isa cause for concern when comparing sketching basedtechniques against the spanning tree strategies of TAG.While 2 bytes of data per packet will generally suÆcefor TAG, a single 16 bit sketch takes the same amountof space and our later experiments will actually be us-ing 20 sketches per packet for a single aggregate. How-ever, as one might guess from Lemma 1, these sketchesare quite compressible. To leverage this, our experi-ments will use the compression techniques of [18]. Inbrief, the sketches are �rst \ attened", enumeratingthe �rst bit of each sketch, then the second bit of eachsketch, and so on, and then the result sequence of bitsis run-length encoded. This reduces the space require-ments to about 30% of uncompressed versions. This issuÆcient for two aggregates to be sketched within oneTinyDB packet (up to 48 bytes).
5. Experimental Evaluation
In this section, we present an evaluation of our meth-ods using the TAG simulator of [15]. Section 5.1 de-scribes the various strategies employed for aggregationand transmission of data and Section 5.2 presents theexperimental results for di�erent scenarios.
5.1. Experimental Setup
We implemented various strategies for aggregationand transmission using the TAG simulator. Under each
Strategy Total MessagesData Bytes Sent Received
TAG1 1800 900 900TAG2 1800 900 2468SKETCH 10843 900 2468LIST 170424 900 2468
Table 1. Communication Cost Comparisons
of these strategies, each node aggregates results re-ceived from its children with its own reading, and thensends the aggregate to one or more of its parents. Anynode within broadcast range which was at a lower level(closer to the root) was considered a candidate parent.In particular, we used the following methods in our ex-periments:
TAG1: The main strategy of [15] (each sensor sendsits aggregate to a single parent).TAG2: The \fractional parents" strategy of [15] de-
scribed in Section 2.2.
LIST: The aggregate consists of an explicit list ofall the items in the aggregate with any duplicates re-moved. These lists are sent to all parents.
SKETCH: The strategy described in Section 4 us-ing duplicate insensitive sketches. The default valuesfor sketches is 20 bitmaps and 16 bits in each bitmapusing the PCSA technique.
For our basic experimental network topology, weused a regular 30 � 30 grid with 900 sensors, whereeach sensor was placed at each grid point. The com-munication radius was
p2 (allowing the nearest eight
grid neighbors to be reached) and the default link lossrate was set at 5%. The root node is always at the cen-ter of the grid. Figure 1 illustrates an example of 7� 7grid.
In all the graphs, we show the average values of 500runs. Also, for each average value, we show the 5th and95th percentiles.
5.2. Experimental Results
First, we evaluate the communication cost of eachapproach. Table 1 shows the total number of bytestransmitted for a single sum query during one epoch,along with the total number of messages sent and re-ceived (assuming no losses). For TAG1 and TAG2, weassume that values are 16 bits each. SKETCH uses 20bitmaps and the compression techniques of [18] withgroup size of 2 (the uncompressed size is 36000 bytes).LIST sends (id; value) pairs as its message format us-ing 32 bits in total for ids and values (two bytes each).As expected, the TAG strategies send the least data,while LIST sends the most and SKETCH is betweenthem. We note that the message size of SKETCH can
0
0.2
0.4
0.6
0.8
1
0 25 50 75 100 125 150 175 200 225 250 275
Ave
rage
Rel
ativ
e E
rror
s
Number of Bits per Message
SUM SKETCH
Figure 2. Number of bits in sketches vs relative
error.
0
200
400
600
800
1000
1200
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35
Ave
rage
Cou
nt
Link Loss Rate
TAG1TAG2
SKETCHLIST
(a) Average counts
0
0.2
0.4
0.6
0.8
1
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35
Ave
rage
Rel
ativ
e E
rror
Link Loss Rate
TAG1TAG2
SKETCH
(b) Average relative error
Figure 3. Performance varying link loss rates.
be tuned further at the cost of accuracy by changingthe number of bitmaps used, as Figure 2 illustrates.Similar results were obtained for count queries.
Figure 3 shows the e�ects of link losses on the per-formance of each strategy for a count query. In Fig-ure 3(a), we see that for all loss rates, the averagecounts returned by LIST and SKETCH are extremelyclose, as are the average counts returned by TAG1and TAG2. As the loss rates increase, the counts re-turned by LIST and SKETCH decrease slowly, whilethe counts returned by TAG1 and TAG2 decrease ata much higher rate. For both pairs, the main di�er-ence is that SKETCH and TAG1 have higher variation
0
200
400
600
800
1000
1200
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35
Ave
rage
Cou
nt
Node Loss Rate
TAG2SKETCH
Figure 4. Performance varying node loss rates.
0
200
400
600
800
1000
1200
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35
Ave
rage
Cou
nt
Link Loss Rate
TAG2SKETCH
Figure 5. Performance with random placement
and communication radius 2p2
than LIST and TAG2 respectively.Figure 3(b) shows the relative errors of SKETCH,
TAG1, and TAG2 compared to LIST. Here, given sam-ple value x and correct value x̂, the relative error is��x�x̂
x̂
��. Again, TAG1 and TAG2 are virtually identi-cal, with quickly growing relative errors. In compar-ison, the relative error of SKETCH only increases asmall amount, but we note that SKETCH has higherrelative errors for very small loss rates. We omit plotsof the relative error for most of the remaining scenariossince they have similar performance trends and are eas-ily extrapolated from the average counts returned. Wealso omit plots for LIST and TAG1 since LIST is in-feasible in practice, and TAG1 is strictly worse thanTAG2.
Figure 4 shows the e�ect of node losses for the samequery. The general trends here look similar to link lossplots in Figure 3(a), but the average counts reporteddrop o� faster, while the average relative error growsmore slowly. Intuitively, a major di�erence here for theLIST and SKETCH strategies is that a value can be\lost" if just the node fails, while all of the links to par-ents must fail to achieve the same loss.
Figure 5 shows the results of placing sensor nodes atrandom grid locations, with the communication rangewas increased to 2
p2 for the random grid placements
to compensate for sparse regions of connectivity. Fig-
0
10000
20000
30000
40000
50000
60000
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35
Ave
rage
Sum
Link Loss Rate
TAG2SKETCH
Figure 6. Performance of sum queries using dif-
ferent link loss rates.
0
1000
2000
3000
4000
5000
10 20 30 40 50 60 70
Ave
rage
Cou
nt
Diameter of Network
TAG2SKETCH
(a) Average counts.
0
0.2
0.4
0.6
0.8
1
10 20 30 40 50 60 70
Ave
rage
Rel
ativ
e E
rror
Network Diameter
TAG2SKETCH
(b) Average relative error.
Figure 7. Performance varying network size.
ure 6 shows the results of using sum sketches, whereeach node chose an integer value uniformly at ran-dom from the range [0; 100], so the expected sum is50 � 900 = 45; 000. The basic trends in both �gureswere essentially the same as when just loss rates werevaried. Results for AVG aggregates, combining summa-tion and count sketches, were similar to those of SUMand were omitted.
Finally, Figure 7 shows the results of varying the net-work size while preserving the grid shape. Despite theloss rate being held constant, the TAG strategies per-form increasingly poorly as the network size increases.Meanwhile, the SKETCH strategy maintains an al-
most constant average relative error around 13 per-cent, though it seems slightly higher for the larger net-work sizes (14 percent).
6. Conclusions and Future Work
We have presented new methods for approximatelycomputing duplicate-sensitive aggregates across dis-tributed datasets. Our immediate motivation comesfrom sensor networks, where energy consumption is aprimary concern, faults occur frequently, and exact an-swers are not required or expected. An elegant buildingblock which enables our techniques are the duplicate-insensitive sketches of Flajolet and Martin, which giveus considerable freedom in our choices of how best toroute data and where to compute partial aggregates. Inparticular, use of this duplicate-insensitive data struc-ture allowed us to make use of dispersity routing meth-ods to provide fault tolerance that would be inappro-priate otherwise.
The implications of these results reach beyond sen-sor networks to other unreliable systems with dis-tributed datasets over which best-e�ort aggregatequeries are posed. Examples include estimating thenumber of subscribers participating in a multicast ses-sion, or counting the number of peers storing a copyof a given �le in a peer-to-peer network. In these set-tings, nodes are less resource-constrained than in sen-sor networks, but the problems are still diÆcult dueto packet loss and frequent node arrivals and depar-tures.
Acknowledgments
John Byers and Je�rey Considine are supported inpart by NSF grants ANI-9986397, ANI-0093296 andANI-0205294. George Kollios and Feifei Li are sup-ported in part by NSF grants IIS-0133825 and IIS-0308213.
The authors would like to thank Phil Gibbons andSuman Nath for valuable feedback on versions of thismanuscript, Sam Madden for providing the simulationcode and answering questions about the code, and JoeHellerstein for valuable discussions.
References
[1] N.Alon,Y.Matias, andM.Szegedy. The space complex-ity of approximating the frequencymoments. Journal ofComputer and System Sciences, 58(1):137{147, 1999.
[2] Z. Bar-Yossef, T.S. Jayram, R. Kumar, D. Sivakumar,and L. Trevisan. Counting distinct elements in a datastream. In Proc. of RANDOM, 2002.
[3] G. Cormode, M. Datar, P. Indyk, and S. Muthukrish-nan. Comparing data streams using hamming norms(how to zero in). InVLDB, 2002.
[4] L. Devroye. Generating the maximum of independentidentically distributed random variables. Computersand Mathematics with Applications, 6:305{315, 1980.
[5] M.Durand andP. Flajolet. Loglog counting of large car-dinalities. In ESA, 2003.
[6] P. Flajolet. On adaptive sampling. COMPUTG: Com-puting, 43, 1990.
[7] P. Flajolet and G. N. Martin. Probabilistic counting al-gorithms for data base applications. Journal of Com-puter and System Sciences, 31, 1985.
[8] D. Ganesan, R. Govindan, S. Shenker, and D. Estrin.Highly-Resilient, Energy-EÆcient Multipath Routingin Wireless Sensor Networks. ACM Mobile Computingand Communications Review, 5(4), 2001.
[9] S. Ganguly, M. Garofalakis, and R. Rastogi. Process-ing set expressions over continuous update streams. InACM SIGMOD, 2003.
[10] P.B. Gibbons and S. Tirthapura. Estimating simplefunctions on the union of data streams. In ACM Sym-posium on Parallel Algorithms and Architectures, pages281{291, 2001.
[11] J.Gray,A.Bosworth,A.Layman,andH.Pirahesh.DataCube: A Relational Aggregation Operator GeneralizingGroup-By,Cross-Tab, andSub-Totals.DataMining andKnowledge Discovery, 1(1):29{53, 1997.
[12] M. Horton, D. Culler, K. Pister, J. Hill, R. Szewczyk,and A. Woo. Mica, the commercialization of microsen-sor motes. 19(4):40{48, April 2002.
[13] C. Intanagonwiwat, R. Govindan, and D. Estrin. Di-rected di�usion: A scalable and robust communicationparadigm for sensor networks. InMobiCOM, 2000.
[14] V. Kachitvichyanukul and B.W. Schmeiser. Binomialrandom variate generation. Communications of theACM, 31(2):216{222, 1988.
[15] S. Madden, M.J. Franklin, J.M. Hellerstein, andW. Hong. TAG: a Tiny AGgregation Service for Ad-Hoc Sensor Networks. InOSDI, 2002.
[16] S. Madden, M.J. Franklin, J.M. Hellerstein, andW. Hong. The design of an acquisitional query proces-sor for sensor networks. InACM SIGMOD, 2003.
[17] S. Nath and P.B. Gibbons. Synopsis Di�usion for Ro-bust Aggregation in Sensor Networks. Technical ReportITR-03-08, Intel Research Pittsburgh, August 2003.
[18] C. Palmer, P. Gibbons, and C. Faloutsos. ANF: A Fastand Scalable Tool for Data Mining in Massive Graphs.In SIGKDD, 2002.
[19] A.J. Walker. An eÆcient method for generatingdiscrete random variables with general distributions.ACMTransactions onMathematical Software (TOMS),3(3):253{256, 1977.
[20] Y. Yao and J. Gehrke. The cougar approach to in-network query processing in sensor networks. SIGMODRecord 31(3), 2002.
[21] Y. Yao and J. Gehrke. Query processing in sensor net-works. In CIDR, 2003.
[22] F. Ye, G. Zhong, S. Lu, and L. Zhang. GRAdient Broad-cast: A Robust Data Delivery Protocol for Large ScaleSensor Networks. ACM Wireless Networks (WINET),11(2), 2005.
[23] J. Zhao, R. Govindan, and D. Estrin. Computing aggre-gates formonitoringwireless sensor networks. In SNPA,2003.
