Observlets: Empowering Analytical Observations on Web Observatory

Aastha Madaan, Tiropanis ThanassisSrinath Srinivasa, Wendy Hall

01/05/2023 SOCM Workshop 2016 2

• Understanding Web Observatory

– Resources and End-users

• Issues for developing data analytic applications

• Defining Observlets

• Observlets on Web Observatory

• A use-case of Observlets

Outline

01/05/2023 SOCM Workshop 2016 3

Web Observatory

• A global catalogue for sharing distributed datasets and analytic

applications

• Web observatory node includes applications of computational social

science, models of evolution of social machines and big data analytics

• Datasets on a web observatory may include quantitative or qualitative

data, real-time data, multimedia content, open-data, archives, and e-

Science resources.

• It aims to support understanding of web evolution through

observation and experimentation + support user-engagement with

analytic resources

01/05/2023 SOCM Workshop 2016 4

Web Observatory: Resources

WO Portal

WO Datastores WO Apps

WO Portal

WO Datastores WO Apps

WO Portal

WO Datastores WO Apps

Links to other observatoriesEPrints repository, harvested news articles,

patient records

Harvesters, visualizations, analytic applications

01/05/2023 SOCM Workshop 2016 5

Web Observatory: Users

Healthcare Experts

MeteorologistsComputer Scientists

• End-users on a web observatory include individuals, public and private organizations agencies

• Domain experts with limited technical skills. E.g. social scientists, medical experts

• Technical experts including computer scientists and web scientists

01/05/2023 SOCM Workshop 2016 6

The Gap

• Data processing on the web observatory is challenging -

– Data generated from diverse sources in a variety of formats

– Data is owned and shared among different administrative domains

– Data may need to be filtered based on temporal and spatial dimensions

– Complex statistical aggregations are required to study the datasets

• Domain experts are limited in their technical skills and fail to understand

possible data transformations

• Technical users duplicate efforts to build similar analysis for different

datasets hindering building of richer and insightful applications

• Need to enable the users develop and re-use analytic applications

01/05/2023 SOCM Workshop 2016 7

• Formal design patterns for data transformations on the web

observatory

• Provide abstract definitions for intermediate steps of data

analysis

• Support re-use of analytic applications and avoid rebuilding

applications from scratch

• Support share application modules and aggregations

Observlets

01/05/2023 SOCM Workshop 2016 8

Observlets (1): Architecture

Dataset 1 Dataset 2 Dataset 3

Data Harmonization

Spatio-temporal filter(s) Aggregation(s) Visualization(s)

App 1 App 2 App 3

Datastore

Applications

Observlet Inventory

01/05/2023 SOCM Workshop 2016 9

Observlets (2): Data Harmonization

Mongodb MySQL Excel

Data Harmonization

Application

Registered “Asthma” datasets

Data analytic application – Asthma conditions in a given geographical area

Output format: Relational

Input + Metadata

01/05/2023 SOCM Workshop 2016 10

Observlets (3): Spatio-Temporal Filters

IndiaFloods

Spatio-temporal filters

Application

Registered datasets about “floods”in India

Data analytic application – compares disaster responseand analyses micro-climate for floods in different states of India during 2014-15

Subset of original dataset

Query within time window (OR|AND) location attributes

01/05/2023 SOCM Workshop 2016 11

Observlets (4): Aggregation

Aggregation Observlet

Application

Registered datasets about “income and education” of people Delhi

Analyze income trends w.r.t education statistics of people of “Delhi”

Apply selected aggregation for analyses

Schematic definitions of statistical formulae and pseudo-code

01/05/2023 SOCM Workshop 2016 12

Observlets (5): Visualization

Visualization Observlet

Visualization Application

Schematic definitions, pseudo-code of visualizations

Dataset/Aggregated data

01/05/2023 13

Observlet Interactions

SOCM Workshop 2016

01/05/2023 SOCM Workshop 2016 14

References[1] W3c community group for web observatory. www.w3.org/community/webobservatory. Accessed: 2015-11-26.[2] Web observatory schema. https: //www.w3.org/wiki/WebSchemas/WebObsSchema. Accessed: 2015-11-26.[3] Web observatory, university of southampton. http://web-001.ecs.soton.ac.uk/. Accessed: 2015-12-11.[4] I. C. Brown, W. Hall, and L. Harris. Towards a taxonomy for web observatories. In Proceedings of the 23rd International Conference on World Wide Web Companion, WWW Companion '14, pages 1067{1072, Republic and Canton of Geneva, Switzerland, 2014. International World Wide Web Conferences Steering Committee. [5] J. O. Coplien. Software design patterns: Common questions and answers. The Patterns Handbook: Techniques, Strategies, and Applications. Cambridge University Press, NY, pages 311{320, 1998.[6] B. M. Frischmann. Infrastructure: The social value of shared resources. Oxford University Press, 2012.[7] W. Hall and T. Tiropanis. Web evolution and web science. Computer Networks, 56(18):3859{3865, 2012.[8] J. Heer and M. Agrawala. Software design patterns for information visualization. IEEE Transactions on Visualization and Computer Graphics, 12(5):853-860, September 2006.[9] V. Hristidis, S.-C. Chen, T. Li, S. Luis, and Y. Deng. Survey of data management and analysis in disaster situations. J. Syst. Softw., 83(10):1701-1714, Oct. 2010.[10] I. O. Popov, M. M. C. Schraefel, G. Correndo, W. Hall, and N. Shadbolt. Interacting with the web of data through a web of inter-connected lenses. In WWW2012 Workshop on Linked Data on the Web, Lyon, France, 16 April, 2012.[11] C. Pu and M. Kitsuregawa. Big data and disaster management: a report from the JST-NSF joint workshop. Georgia Institute of Technology, CERCS, 2013.[12] T. Tiropanis, W. Hall, N. Shadbolt, D. De Roure, N. Contractor, and J. Hendler. The web science observatory. IEEE Intelligent Systems, (2), pp100-104, 2013.[13] T. Tiropanis, X. Wang, R. Tinati, and W. Hall. Building a connected web observatory: architecture and challenges. 2014.