Tabula in Action: A Sampling Middleware for InteractiveGeospatial Visualization Dashboards

Jia YuArizona State University

jiayu2@asu.edu

Kanchan ChowdhuryArizona State University

kchowdh1@asu.edu

Mohamed SarwatArizona State University

msarwat@asu.edu

ABSTRACTIn this paper, we demonstrate Tabula, a middleware thatsits between the data system and the geospatial visualiza-tion dashboard to increase user interactivity. The proposedsystem adopts a sampling cube approach that stores pre-materialized spatial samples and allows data scientists todefine their own accuracy loss function such that the pro-duced samples can be used for various user-defined visual-ization tasks. The system ensures that the difference be-tween the sample fed into the dashboard and the raw queryanswer never exceeds the user-specified loss threshold. Fordemonstration purposes, we connect Apache Zeppelin, a vi-sualization dashboard, to the system and show how Tab-ula accelerates interactive visualizations on NYC Taxi Tripdata, Yelp review data and San Diego Smart Streetlightsdata.

PVLDB Reference Format:Jia Yu, Kanchan Chowdhury and Mohamed Sarwat. Tabula inAction: A Sampling Middleware for Interactive Geospatial Visu-alization Dashboards. PVLDB, 13(12): 2925-2928, 2020.DOI: https://doi.org/10.14778/3415478.3415510

1. INTRODUCTIONWhen a user explores a spatial dataset using a visual-

ization dashboard, such as Tableau, Apache Zeppelin andArcGIS, this often involves several interactions between thedashboard and the underlying data system. In each inter-action, the dashboard application first issues a query to ex-tract the data of interest from the underlying data system(e.g., PostGIS and Apache Spark SQL), and then runs thevisual analysis task (e.g., heat maps and statistical analysis)on the selected data. Based on the visualization result, theuser may iteratively go through such steps several times toexplore various subsets of the database.

Every interaction between the visualization dashboardand the underlying data system may take a significantamount of time (denoted as data-to-visualization time) to

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. To view a copyof this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. Forany use beyond those covered by this license, obtain permission by emailinginfo@vldb.org. Copyright is held by the owner/author(s). Publication rightslicensed to the VLDB Endowment.Proceedings of the VLDB Endowment, Vol. 13, No. 12ISSN 2150-8097.DOI: https://doi.org/10.14778/3415478.3415510

run, especially over large-scale data because: (1) The data-system query time proportionally increases with the volumeof the underlying data table. Even scalable data systemssuch as Apache Spark still exhibit non-negligible latencyon large scale data. (2) Existing spatial visualization dash-boards such as Tableau, Apache Zeppelin and ArcGIS workwell for small size data but do not scale to large datasets.

To remedy that, one approach that practitioners use is todraw a smaller sample of the entire data table (e.g., 1 milliontuples) and materialize the sample in the database (denotedas SampleFirst). The caveat is that running queries on thesample may lead to inaccurate visualization results since thequery answer may significantly deviate from the actual an-swer especially for some small data populations. There hasbeen a flurry of research papers that addressed the problemby enhancing the accuracy of pre-built stratified samplesfor approximate query processing, such as Sample+Seek,BlinkDB / SnappyData [3]. However, the pre-built strat-ified samples have no deterministic accuracy guarantee andonly apply tailored optimizations on classic OLAP aggre-gate measures such as SUM, COUNT, and AVG. Therefore,they cannot be easily extended to other types of data anal-ysis (e.g., linear regression and most spatial visual effects inFigure 2).

Instead of creating pre-built samples, an alternative ap-proach runs data-system queries over the entire table for ev-ery interaction, draws a sample of the extracted populationand sends it back to the visualization dashboard to shortenthe visualization time. Although this approach (denoted asSampleOnTheFly) can certainly achieve higher accuracy forthe selected population [2], it is prohibitively expensive sinceit has prepared a sample for each user interaction.

In this paper, we demonstrate Tabula [7], a middlewarethat sits between the data system and the geospatial vi-sualization dashboard to increase user interactivity. Theproposed system adopts a materialized sampling cube ap-proach, which pre-materializes sampled answers for a setof potentially unforeseen queries (represented by an OLAPcube cell). In each iteration, the system returns a materi-alized sample for the SQL query, rather than the originalquery answer. The system employs two main strategies tomitigate the initialization time and memory utilization whilestill achieving interactive performance: (1) a partial initial-ization algorithm to only materialize custom-built samplesof those queries for which the global sample (the sampledrawn from the entire dataset) exceeds the required accu-racy loss threshold. (2) a sample selection technique thatfinds similarities among materialized local samples, only per-

2925

Spatial visualizationdashboard

By passenger count& trip distance

By passenger count& payment method

By passenger count

By trip distance

Cash

[0, 5)

[15, 20)

[20, 25)

1 2 3 4DisputeCredit

[5, 10)[10, 15)

All

Bypay

ment me

thod

& tripdist

ance

Bypay

ment me

thod

My accuracyloss func

Partially materialized sampling cube

Initialize

Sampling cubeInitialization

Representativesample selection

Accuracy lossaware Sampling

Visual-awareloss func

Stat. meanloss func

Tabula

Query1:CREATE TABLE SamplingCube ASSELECT D, C, M, SAMPLING(*,θ) AS sampleFROM nyctaxi GROUPBY CUBE(D, C, M)HAVING loss(pickup point, Sam_global ) > θ

Query2:SELECT sampleFROM SamplingCubeWHERE D = [0, 5) AND C = 1

User

Pre-builtsample

Visualizedanalysis result

Data systemFigure 1: Tabula overview. Samples in red cells are material-ized. DCM cuboid is omitted. Example queries use NYC Taxidataset (100GB)[4] which contains 700 million trip records inNYC from year 2009 to 2012. Each record includes dropoff loca-tions, trip distances (D), passenger count (C), payment method(M), itemized fares, and so on.

sists a few representative samples, then uses the represen-tative samples as an answer to many queries. We built aninitial prototype of Tabula 1 inside SparkSQL. In this demo,we tailor Tabula to several visualization tasks such as mapvisualization / linear regression and connect it with a visu-alization dashboard

2. SYSTEM OVERVIEWInitialization query. Figure 1 gives an overview of Tab-

ula. A data scientist must initialize Tabula by providingthe following system parameters to assemble an initializa-tion query: (1) User-defined accuracy loss function (abbr.loss()): This function determines how to calculate the ac-curacy lost due to using the sample as opposed to the origi-nal query answer. (2) Accuracy loss threshold θ: this param-eter decides the acceptable accuracy for all queries processedby Tabula. (3) Cubed attributes: the set of attributes thatwill be used to build the sampling cube (e.g., attributes D,C and M depicted in Figure 1). Data-system SQL querieswill use a subset of these attributes in WHERE clause predi-cates. The data scientist feeds such parameters to Tabulaas follows:

CREATE TABLE [sampling cube name] ASSELECT [cubed attrs], SAMPLING(sampled attr,[θ]) AS sampleFROM [table name]GROUPBY CUBE([cubed attrs])HAVING [loss function name](sampled attr, Samglobal) > [θ]

where Samglobal represents a sample built by Tabula overthe entire table using random sampling. SAMPLING() is aTabula-specific function that takes a dataset represented asa set of tuples and produces a sample of that dataset such

1Tutorial video: https://jiayuasu.github.io/files/video/tabula-demo.mp4

that the accuracy of the produced sample, compared to theoriginal dataset, does not exceed the accuracy loss thresholdθ deterministically. Query1 in Figure 1 is an initializationquery.

Data-system query. Once the sampling cube is initial-ized, the data scientist, via the visualization dashboard, canissue SQL queries to Tabula as follows:

SELECT sample FROM [sampling cube] WHERE [conditions]

After receiving this query, Tabula directly fetches a mate-rialized sample from the sampling cube and returns it backto the visualization dashboard. This way, Tabula signifi-cantly reduces both the data system time and visualizationtime. Besides, the system always guarantees with 100% con-fidence level that the accuracy loss from using the returnedsample, as compared to the original query answer, does notexceed the accuracy loss threshold θ.

User-defined accuracy loss function. The visualanalysis results obtained from a sample should be very closeto the results obtained from the raw data. In this paper, weformalize the difference as accuracy loss. There are manyways to compute accuracy loss, which serves different pur-poses. The accuracy loss highly depends on the type ofvisualization the data scientist plans to perform. The bodyof this function is a user-defined scalar expression over sev-eral aggregate functions. Users can define this function viaCREATE AGGREGATE loss(Raw, Sam).

Such a function takes raw data and sample data as input,then returns a decimal value which is the accuracy loss. Forinstance, consider a visual analysis task which requires alow relative error between the statistical means of the sam-ple and the raw data. This accuracy loss function can be

implemented as follows: BEGIN ABS(AV G(Raw)−AV G(Sam)

AV G(Raw))

END.Tabula requires that the accuracy loss function must be

algebraic [1]. To achieve that, all aggregate functions andmathematical operators involved in calculating loss(Raw,Sam) must be distributive or algebraic. In fact, many com-mon aggregations satisfy this restriction including SUM,COUNT, AVG, MIN, TOP-K, excluding MEDIAN.

Sampling cube initialization algorithm. First, Tab-ula draws a global random sample, called Samglobal, fromthe entire raw dataset. Second, the system builds the sam-pling cube by running a set of GroupBy queries to calculateall cuboids in the cube. Given the grouped raw data of eachcube cell, if applying the global sample to this cell satisfiesthe HAVING condition - loss(cell data, Samglobal) > θ, Tab-ula will identify this cell as an iceberg cell and materializea local sample (called Samlocal) for it. However, the costof the second step increases exponentially with the numberof cubed attributes (2n GroupBy). Tabula avoids that bydividing it into two sub-stages: (1) Dry run for iceberg celllookup: the system identifies all iceberg cuboids (cuboidsthat have iceberg cells) by scanning the raw table data onlyonce; (2) Real run for sampling cube construction: basedon the iceberg cell information learned in the dry run stage,Tabula constructs a sampling cube that only contains ice-berg cuboids. For each iceberg cell in this cuboid, the al-gorithm draws a local sample using the accuracy loss-awaresampling method.

Accuracy loss-aware sampling. The sampling func-tion (i.e., SAMPLING(*,[θ]) in Tabula generates a sample

2926

Visualization front end

Figure 2: Geospatial visualization dashboard powered by Tabula, SampleFirst and SampleOnTheFly (POIsam [2])

for the raw data. It minimizes the sample size while guar-anteeing loss(Raw, Sam) ≤ θ. Finding such sample is NP-complete because there are 2N candidate sample sets to bechecked, where N is the cardinality of raw data. The sam-pling module employs a greedy algorithm which guaranteesthat loss(T, t) ≤ θ, but the sample size may not be minimal.

Sample selection technique. After the cube initial-ization, the partially materialized sampling cube may stillpossess a large memory footprint. To further reduce thememory footprint, Tabula only persists a representative setof local samples, and re-uses the representative samplesin many iceberg cells rather than persisting every individ-ual local sample. SampleA can represent SampleB only ifloss(CellB , SamA) ≤ θ. Tabula first evaluates the represen-tation relationships in a sample representation graph andthen selects the minimal set of samples that can representthe entire graph. This problem is harder than the MinimumDominating Set problem which is known to be NP-Complete(see the proof in [7]), so Tabula develops a greedy algorithmwhich may not always find the minimal set of samples butguarantees the accuracy loss.

3. DEMONSTRATION SCENARIOSFor demonstration purposes, we use three datasets: (1)

NYC Taxi (see Figure 1), (2) denormalized and structuredYelp dataset (50GB) [6]: It contains 6 million Yelp businessreviews. Each review consists of more than 20 columns in-cluding star, comment, coordinate, day of week, user name,business name, category and so on, (3) San Diego SmartStreetlights [5] (100 GB): It contains the city status infor-mation such as temperature, pedestrian, vehicles, reportedby IoT sensors every second.

Demonstration setting. We connect Apache Zeppelin,a visualization dashboard, to Tabula (see Figure 2) in Spark-SQL. All needed sampling cubes are pre-materialized andcached. The dashboard has two panels, one SQL input boxand one visualization window. Attendees can freely inter-act with the dashboard: enter SQL queries with differentWHERE filters in the input box and check visualization resultsof the selected data population in the visualization window.We conduct two types of visual analytics: (1) geospatial dot

map: depicts a map of the target region with a set of geospa-tial objects (2) linear regression: describes a regression lineamong a set of <x, y> data points. We also provide a back-end for the attendees to learn the data-system query plansand execution time (see Figure 3).

Compared approaches. In order to show the advantageof Tabula, we prepare several existing approaches includ-ing SampleFirst and SampleOnTheFly (POIsam [2]) withZeppelin dashboards running on top of them. After everydashboard interaction, the backend system will report twometrics: (1) data-to-visualization time (2) accuracy of theproduced visualization. This way, the attendees will be ableto quantify the performance of compared approaches andverify the superiority of Tabula. Besides, we will also bringSnappyData [3] which implements stratified samples for ag-gregation queries but differentiate it from our approach.

Scenario I: Travel Habits of NYC Residents: NYCTaxi Trip dataset includes detailed traveling informationfor individual trip records and hence brings data scientistsan exclusive opportunity to understand how people arrangetheir travel destinations (in terms of spatial distribution)under different circumstances such as pickup periods (night,morning, afternoon) and passenger count (1-6). We builda Tabula materialized sampling cube on attribute Vendor

name, Payment type, Passenger count, Day of week, andTime of day and draw samples on trip drop-off locations.The demo attendee can give a set of conditions amongthese attributes to specify a particular group of drop-off lo-cations and Tabula will return a pre-materialized samplewhose accuracy loss (explained later) is within the thresh-old. He or she then can immediately see sampled drop-off locations on a dot map. For instance, as shown inthe first panel of Figure 2, he first enters a query as fol-lows, and the dot map shows most people travel to Manhat-tan: SELECT sample FROM nyc dropoff cube WHERE day =

’Monday’ AND passenger count = 1

Then he changes the first condition to day = ’Saturday’

and doesn’t observe significant clusterings. Next, he modi-fies the second condition to passenger_count = 6 and no-tices that many people travel to JFK airport and La Guardiaairport. Finally, the attendee reaches a conclusion that more

2927

SparkSQL backend

Cluster status

Figure 3: Tabula backend monitoring

people travel to Manhattan during weekdays, and they sharerides with others when doing long-distance travel.

To build this sampling cube, we plug a visualization-awareaccuracy loss function into Tabula. This function stems fromrecent work on visualization-aware sampling (POIsam [2]).It uses the average minimum distance between the sam-ple and the raw data to measure the loss, calculated asfollows: BEGIN 1

|Raw|∑

x∈RawMINs∈Sam(loss(x, s)) END,

where loss(x, s) is the Euclidean distance between two drop-off locations.

In addition, we can also construct a Tabula material-ized sampling cube to study the tip percentage at dif-ferent situations. The demo attendee can select differ-ent taxi trip records based on many criteria such as pas-senger count and payment method and then perform lin-ear regression analysis on ¡fare amount, tip amount¿ val-ues. The regression line indicates the trend of tip per-centage. This cube is built upon the same 5 attributesbut with a different accuracy loss function (explainedin [7]). For each query, Tabula returns a set of sampled¡fare amount, tip amount¿ values. As shown in the sec-ond panel of Figure 2, the demo attendee first specifies theconditions such as payment method = ’creditcard’ andpassenger count = 2 in the SQL input box and viewsthe linear regression result immediately in the chart below.Then he can apply new filters to check other data popula-tion. After several iterations, he finally concludes that taxidrivers receive less tips if people share taxis together.

Scenario II: Distribution of Yelp reviews: Yelpreleases its internal dataset once a year for encouragingresearch to discover insights hidden in the data. Sincethis dataset provides 6 million review records with theircoordinates and many categorical attributes, an interestingapplication is to explore spatial distribution of these reviewswith different attributes such as category and stars. Wedenormalize the original tables to a single relational tablefor review records and then build a Tabula sampling cubeon attribute Category, Day of week, Time of day, and

Stars. Similar to that in Scenario I, this sampling cubeis tailored to the dot map visualization task. The demoattendee enters several filters in the SQL input box in thefirst panel of Figure 2 and checks the dot map. For in-stance, he can enter a query like this: SELECT sample FROM

review coordinate cube WHERE category = ’Mexican

food’ AND stars = 5. Then he can remove the secondcondition stars = 5 and re-query the sampling cube. Byvisually comparing the two dot maps, he notices that mostacclaimed Mexican restaurants are located in the southernpart of the US. One possible reason of this phenomenon isthat Mexico borders the US in the south.

Scenario III: San Diego Smart Streetlights Pro-gram: The city of San Diego has installed 4700 smart LEDstreetlights (8000 in total by summer 2020) at differentroad intersections across the entire city [5]. These smartstreetlights equip sensors to monitor the city status includ-ing temperature, pedestrian movement, vehicle movement,and parking activity at every second. With the help ofthe sensors, San Diego now has the world’s largest SmartCity platform which produces massive data every month.A Tabula sampling cube is built on the categorical at-tributes Time of day, Traffic level, Pedestrian level,Temperature level, and Parking level. We plug in thevisualization - aware accuracy loss function explained inScenario I and use dot maps as the visualization effect. Thedemo attendees can freely apply different filters in the SQLinput box and check the city status in a timely manner. Forexample, SELECT sample FROM city sensor cube WHERE

Day of week = ’Monday’ AND Traffic level = ’high’

will return a dot map to show the regions which have heavytraffic on Monday.

4. ACKNOWLEDGMENTThis work is supported by the National Science Founda-

tion (NSF) under Grant 1845789.

5. REFERENCES[1] J. Gray, S. Chaudhuri, A. Bosworth, A. Layman,

D. Reichart, M. Venkatrao, F. Pellow, and H. Pirahesh.Data cube: A relational aggregation operatorgeneralizing group-by, cross-tab, and sub totals. DataMining Knowledge Discovery, 1(1):29–53, 1997.

[2] T. Guo, K. Feng, G. Cong, and Z. Bao. Efficientselection of geospatial data on maps for interactive andvisualized exploration. In SIGMOD, 2018.

[3] J. Ramnarayan, B. Mozafari, S. Wale, S. Menon,N. Kumar, H. Bhanawat, S. Chakraborty, Y. Mahajan,R. Mishra, and K. Bachhav. Snappydata: A hybridtransactional analytical store built on spark. InSIGMOD, 2016.

[4] N. Y. C. Taxi and L. Commission. Nyc taxi records.https://www1.nyc.gov/site/tlc/about/

tlc-trip-record-data.page, 2016.

[5] San diego smart streetlights program, 2019.

[6] Yelp. Yelp Dataset. https://www.yelp.com/dataset,2019.

[7] J. Yu and M. Sarwat. Turbocharging geospatialvisualization dashboards via a materialized samplingcube approach. In ICDE, pages 1165–1176. IEEE, 2020.

2928