Converts' rally, Evangelistic Committee of New York City, Carnegie Hall, Sept.14, 1908

Open DataLinkedSix Ingredients

The missing ★

Mix ‘n Mash

Contextualize!

Choose your Grain Size

Lower the Threshold

Repeatable Transformation

1The missing ★

http://give.everything/a/URI

HTTPs URIs only please!(or resolver + URN)

Version information

Version agnostic

Guessable

2Repeatable Transformation

Transformation should be part of routine ...

... manageable and scalable ...

... repeatable ...

Linked Data will not be the official source anytime soon

http://www.w3.org/TR/prov-overview/

Provenance is key

3Choose your Grain Size• The document is the

traditional grain size(dublin core)

• Linked data allows for deep links into data

• Cost versus usefulness

• Are you the right party to provide detailed descriptions?

http://creatingandeducating.blogspot.nl/2011/11/blog-post.html

4 Mix ‘n Mash

• Multiple vocabularies won’t bite

• Multiple identifiers won’t bite

!

• Choose what’s useful for you...

• ... then map to others!

Image © David Sykes 2009 All rights reserved

Good News: the bulk has already been done for you!

5• Information is not always compatible

• Make explicit in which context the information holds ...

• ... and who stated the information, why and how.

Contextualize!

Flat Earth and Square Earth idea courtesy of Szymon Klarman

to2Data Semantics

Semantics for Scientific Data PublishersFrom Data

Photo by Philip Dujardin, http://www.filipdujardin.be

Herkomst en Hergebruik van Open Data

Rinke HoekstraVU University Amsterdam/University of Amsterdam

rinke.hoekstra@vu.nl

Photo by Philip Dujardin, http://www.filipdujardin.be

Definition(Oxford English Dictionary)

• The fact of coming from some particular source or quarter; origin, derivation;

• the history or pedigree of a work of art, manuscript, rare book, etc.;

• concretely, a record of the passage of an item through its various owners.

Making trust judgements

Liability, trust and privacy in open government data

Compliance and auditing of business processes

Licensing and attribution of combined information

Curt Tilmes, Peter Fox, Xiaogang Ma, Deborah L. McGuinness, Ana Pinheiro Privette, Aaron Smith, Anne Waple, Stephan Zednik, Jinguang Zheng: Provenance Representation for the National Climate Assessment in the Global Change Information System. IEEE T. Geoscience and Remote Sensing 51(11): 5160-5168 (2013)

Integrated & Summarized Data

Transparency and Trust

“Provenance is the number one issue that we face when publishing

government data in data.gov.uk”John Sheridan, UK National Archives, data.gov.uk

Provenance?• Provenance = Metadata?

Provenance can be seen as metadata, but not all metadata is provenance

• Provenance = Trust?Provenance provides a substrate for deriving different trust metrics

• Provenance = Authentication?Provenance records can be used to verify and authenticate amongst users

Three Dimensions

• ContentCapturing and representing provenance information

• ManagementStoring, querying, and accessing provenance information

• UseInterpreting and understanding provenance in practice

Three Dimensions

• ContentCapturing and representing provenance information

• ManagementStoring, querying, and accessing provenance information

• UseInterpreting and understanding provenance in practice

recording annotating workflows

Three Dimensions

• ContentCapturing and representing provenance information

• ManagementStoring, querying, and accessing provenance information

• UseInterpreting and understanding provenance in practice

recording annotating workflows

scalability interoperability

Three Dimensions

• ContentCapturing and representing provenance information

• ManagementStoring, querying, and accessing provenance information

• UseInterpreting and understanding provenance in practice

recording annotating workflows

scalability interoperability

trust accountability compliance explanation debugging

Standardization

W3C PROV StandardProvenance is a record

that describes the people, institutions, entities, and

activities, involved in producing, influencing, or delivering a

piece of dataor a thing.

http://www.w3.org/TR/prov-overview

Luc Moreau & Paul Groth

W3C PROV StandardProvenance is a record

that describes the people, institutions, entities, and

activities, involved in producing, influencing, or delivering a

piece of dataor a thing.

http://www.w3.org/TR/prov-overview

http://doc.metalex.eu

http://yasgui.data2semantics.org

Interpretation

Naive Approaches

InProv: Visualizing Provenance Graphs with Radial Layouts and Time-Based Hierarchical Grouping Madelaine D. Boyd - http://www.seas.harvard.edu/sites/default/files/files/archived/Boyd.pdf

Orbiter has several limitations. It does not have capabilities for query subgraph high-lighting, regular expression filters, process grouping, annotations, or programmable views[16].Furthermore, the structure of each summary node, where child nodes are grouped withinparents and are hidden until the parent is expanded, benefits queries earlier in the depen-dency chain. Initial overviews often correspond with system bootup, and appear very similaracross di↵erent traces (time slices of system activity).

Figure 10: In these screenshots of Orbiter, the presence of edges overwhelms the visibility ofnodes. By relying on a node-link graph layout and using spatial location to encode objectrelationships, Orbiter’s graph layout algorithm must draw many long edges to communi-cate node connections. Without edge bundling or opacity variation, the meanings of theserelationships are obscured.

Another one of Orbiter’s weaknesses is its node-link diagram layout. As a result, eachnode’s position in the X-Y plane and the length and angle of connecting lines are wastedattributes. The chosen graph layout algorithm (dot by default) arranges nodes to minimizeedge crossings and total edge lengths. However, depending on the interrelationships amongnodes, it may be impossible to find an optimal layout. In this case, undesirable designs withdense quantities of long edges may emerge, as seen in Figure 10. At the scale of a typicalprovenance graph, related nodes may be drawn far apart. This weakens the e↵ectiveness ofedges as “connections” that show relationships between nodes.

2.4 Large Graph Visualization

While a complete survey of graph and tree visualization is beyond the scope of this paper,I will summarize some notable approaches. See Herman et. al for a more detailed overviewof graphs and information visualization[27], or see Ellis and Dix for an overview of clutterreduction techniques for visualization of large data sets[20].

There is a variety of current e↵orts to visualize large graphs. Many of these tools weredesigned for social network or genomics data sets, for which there is a motivation to seeboth patterns in the data set at large, as well as node-level detail. Visualization attemptsfor large graphs mostly fall within three categories — summary node-link diagrams, tree

17

Figure 11: (Top): A screenshot of the portion of the graph generated by GraphViz for atrace of the third provenance challenge. (Bottom): A zoomed-in view of the same graph.The horizontal black bars across the images are dense collections of edges.

E↵ective large graph visualizations present the user with a summary view that can beexplored, filtered, and expanded interactively.

2.5 Tree Visualization

While trees are a subcategory of graphs, because of their hierarchical composition, tree visu-alization forms its own subfield of research. A survey of over two-hundred tree visualizationsis given at Hans-Jrg Schulz’s treevis.net. Visitors can narrow down by dimensionality(2D, 3D, or mixed), representation (explicit node-link diagram, implicit treemap, or combi-nation), alignment (XY plot, radial layout, or free diagram)[55]. These categories are shownby the icons in Figure 13.

19

Figure 12: Left : Pajek uses various summary node-link and matrix-based representationsdepending on the structure of the supplied data set. Pictured is a main core subgraphextracted from routing data on the Internet. Right : TopoLayout optimizes the choice ofvisualization display depending on the underlying graph structure. The right column isTopoLayout’s output, while the left and middle columns are the outputs of the GRIP andFM graph layout algorithms.

Figure 13: treevis.net defines di↵erent categories for tree maps. Tree maps can be cate-gorized by dimensionality (2D, 3D, or mixed), representation (explicit, implicit, or mixed),or alignment (XY, radial, or spring).

Tree visualizations are either explicit or implicit. Explicit representations resemble node-link diagrams. An example of an implicit representation is a tree map, a diagram where theentire tree is inscribed in a rectangle representing the root node. This root is subdividedhierarchically into more rectangles, which represent child nodes, and each child node issubdivided into more child nodes. Treemaps are excellent for displaying hierarchical orcategorical data[57]. One famous example, shown in Figure 14, is the “Map of the Market”from SmartMoney.com, which displays in red and green the changes in market value ofpublicly-traded companies, grouped by market sector, with cell size proportional to marketcapitalization[64].

TreePlus is an example of a tree-inspired graph visualization tool (Figure 15). It usesthe guiding metaphor of “plant a seed to watch it grow” to summarize navigation of its tree-based large graph visualization tool[42]. The visual interface displays a tree, starting fromthe graph root or a user-specified starting node. Nodes at the same level are listed vertically;parents and children are listed to the left or right. When the user hovers over displayed

20

Width of activities and entities is based on information flow

Activities and entities are extracted from an ego graph

Capturing

We need an intuitive REST-like API to integrated Open Government data. Dealing with all these different formats and identifiers is really taking too much time.

I have all this data, and I want to make (part of) it available for the general public, but haven't a clue how!

Civil Servantwants to publish data

Application Developerswant to consume data

Carrier 12:00 PMPage Title

http://www.domain.com

Google

Apps and applicationsVisual interactions with Open Data. Application specific logics (e.g. 'danger')

CitySDK APIHTTP API to the CitySDKReturns JSON, Turtle, etc.

(includes the Linked Data API of CitySDK)

SPARQL APISPARQL Endpoint to the Linked

Data storage of the ODE

Partial Synchronisation

CitySDK Datastores Linked Data Triplestore

Feed into

Query Orchestrator

Amsterdam Open Data ExchangeHTTP API to `canned queries' across multiple datasets.

Returns JSON-LD, Turtle

Data Integrator

ODE Best PracticesBest practices for publishing Open Data

CitySDK Ingestion Plugins"Standard" adapters part of CitySDK

ODE Ingestion AdaptersIngestion adapters developed within

ODE

Municipal Legacy Systems Excel FilesAmsterdam Open Data CKAN

Amsterdam Open Data CatalogWill point to datasets in the ODE

May provide a direct query interface on top of ODE

Wrapper-based

Workflow-based

Tom de Nies (Ghent University)Sara Magliacane (VU University Amsterdam)

Integrated

to2Data Semantics

Semantics for Scientific Data PublishersFrom Data

The Big Future of Data2 October 2014

Enrich Publish Analyze

Semantic Publication of Data

Publish directly from the cloud

to the cloud

On-the-fly analysis and tag suggestion

Interactive Data Construction via Instrumented IPython Notebook

Integration in popular tool

No “green field”

Visual Exploration of Big Data

Virtualisation

Discover patterns

Interactive visualisation

Sparse and heterogeneous