LarKC Tutorial at ISWC 2009 - Architecture

LarKC Architecture and Technology

Michael Witbrock, Cycorp Europe (+UIBK)

with contributions from all LarKC developers

Realising the Architecture

WorkflowSupportSystem


Plug-in RegistryPlug-in

Registry

Plug-in ManagerPlug-in Manager

Data LayerData Layer

Plug-in APIPlug-in API

Data Layer APIData Layer APIRDF

StoreRDF

Store

2

LarKC Plug-in API: General Plug-in Model

• Plug-ins are assembled into Workflows, to realise a LarKC Experiment or Application

• Plug-ins are identified by a URI (Uniform Resource Identifier)

• Plug-ins provide MetaData about what they do (Functional properties): e.g. type = Selecter

• Plug-ins provide information about their behaviour and needs, including Quality of Service information (Non-functional properties): e.g. Throughput, MinMemory, Cost,…

• Plug-ins can be provided with a Contract that tells them how to behave (e.g. Contract : “give me the next 10 results”) and Context information used to store state between invocations

+ URI getIdentifier()+ QoSInformation getQoSInformation()

+ URI getIdentifier()+ QoSInformation getQoSInformation()

Plug-inPlug-in

Functional propertiesNon-functional propertiesWSDL description

Functional propertiesNon-functional propertiesWSDL description

Plug-in description

3

LarKC Plug-in API: IDENTIFY

• IDENTIFY: Given a query, identify resources that could be used to answer it

• Sindice – Triple Pattern Query RDF Graphs• Google – Keyword Query Natural Language Document• Triple Store – SPARQL Query RDF Graphs

+ Collection<InformationSet> identify(Query theQuery, Contract contract, Context context)


Identifier Identifier

4

LarKC Plug-in API: TRANSFORM (1/2)

• Query TRANSFORM: Transforms a query from one representation to another

• SPARQL Query Triple Pattern Query• SPARQL Query Keyword Query• SPARQL Query SPARQL Query (different abstraction)• SQARQL Query CycL Query

+ Set<Query> transform(Query theQuery, Contract theContract, Context theContext)+ Set<Query> transform(Query theQuery, Contract theContract, Context theContext)

QueryTransformerQueryTransformer

5

LarKC Plug-in API: TRANSFORM (2/2)

• Information Set TRANSFORM: Transforms data from one representation to another

• Natural Language Document RDF Graph• Structured Data Sources RDF Graph

• RDF Graph RDF Graph (e.g. foaf vocabulary to facebook vocabulary)

+ InformationSet transform(InformationSet theInformationSet, Contract theContract, Context theContext)


InformationSetTransformerInformationSetTransformer

6

LarKC Plug-in API: SELECT

• SELECT: Given a set of statements (e.g. a number of RDF Graphs) will choose a selection/sample from this set

– Collection of RDF Graphs Triple Set (Merged)– Collection of RDF Graphs Triple Set (10% of each)– Collection of RDF Graphs Triple Set (N Triples)

+ SetOfStatements select(SetOfStatements theSetOfStatements, Contract contract,

Context context)

+ SetOfStatements select(SetOfStatements theSetOfStatements, Contract contract,

Context context)

SelecterSelecter

7

LarKC Plug-in API: REASON

• REASON: Executes a query against the supplied set of statements

– SPARQL Query Variable Binding (Select)– SPARQL Query Set of statements (Construct)– SPARQL Query Set of statements (Describe)– SPARQL Query Boolean (Ask)

+ VariableBinding sparqlSelect(SPARQLQuery theQuery, SetOfStatements theSetOfStatements, Contract contract, Context context)

+ SetOfStatements sparqlConstruct(SPARQLQuery theQuery, SetOfStatements theSetOfStatements, Contract contract, Context context)

+ SetOfStatements sparqlDescribe(SPARQLQuery theQuery, SetOfStatements theSetOfStatements, Contract contract, Context context)

+ BooleanInformationSet sparqlAsk(SPARQLQuery theQuery, SetOfStatements theSetOfStatements, Contract contract, Context context)





ReasonerReasoner

8

LarKC Plug-in API: DECIDE

• DECIDE: Builds the workflow and manages the control flow– Scripted Decider: Predefined workflow is built and executed– Self-configuring Decider: Uses plug-in descriptions (functional and non-functional

properties) to build the workflow

+ VariableBinding sparqlSelect(SPARQLQuery theQuery, QoSParameters theQoSParameters)

+ SetOfStatements sparqlConstruct(SPARQLQuery theQuery, QoSParameters theQoSParameters)

+ SetOfStatements sparqlDescribe(SPARQLQuery theQuery, QoSParameters theQoSParameters)

+ BooleanInformationSet sparqlAsk(SPARQLQuery theQuery, QoSParameters theQoSParameters)





DeciderDecider

9

Released System: larkc.sourceforge.net

DeciderDecider



QueryTransformer

QueryTransformer



IdentifierIdentifier



Info. SetTransformer




SelecterSelecter



ReasonerReasoner


Plug-in RegistryPlug-in Registry

PipelineSupportSystem


• Open Apache 2.0 license• Previous early adopters

workshop @ ESWC– 20 people attended– participants modified plug-ins,

modified workflows

Standard Open Environment: subversion connection, command line build, or eclipse, netbeans soon?

• Open Apache 2.0 license• Previous early adopters

workshop @ ESWC– 20 people attended– participants modified plug-ins,

modified workflows

Standard Open Environment: subversion connection, command line build, or eclipse, netbeans soon?

10

LarKC Plug-in API

11



Identifier Identifier

+ Set<Query> transform(Query theQuery, Contract theContract, Context theContext)

+ Set<Query> transform(Query theQuery, Contract theContract, Context theContext)

QueryTransformerQueryTransformer



InformationSetTransformerInformationSetTransformer

+ SetOfStatements select(SetOfStatements theSetOfStatements, Contract contract, Context context)

+ SetOfStatements select(SetOfStatements theSetOfStatements, Contract contract, Context context)

SelecterSelecter









ReasonerReasoner









DeciderDecider

• 5 types of plug-ins • Plug-in API enables interoperability (between plug-in and platform and between plug-ins)• Plug-ins I/O abstract data structures of RDF triples => flexibility for assembling plug-ins and for plug-in writers • Compatibility ensured by DECIDER and workflow configurators, based on plug-in description

Data Layer APIData Layer API




Registry

RDFStoreRDF

StoreRDF

StoreRDF

StoreRDF

StoreRDF

StoreRDFDocRDFDoc

RDFDocRDFDoc

RDFDocRDFDoc


DeciderDecider



QueryTransformer

QueryTransformer










SelecterSelecter



ReasonerReasoner


ApplicationApplication

Platform Utility Functionality

APIs

Plug-ins

External systems

External data sources

LarKC Plug-in API

12



Plug-in APIPlug-in API Plug-in APIPlug-in API Plug-in APIPlug-in APIPlug-in APIPlug-in API

LarKC Architecture

DeciderDecider



QueryTransformer

QueryTransformer










SelecterSelecter



ReasonerReasoner





RDFStoreRDF

Store

IdentifierIdentifier Info Set Transformer

Info Set Transformer ReasonerReasoner

DeciderDecider

SelecterSelecterQueryTransformer

QueryTransformer

What does a workflow look like?

13

What Does a Workflow Look Like?

DeciderDecider



QueryTransformer

QueryTransformer










SelecterSelecter



ReasonerReasoner





RDFStoreRDF

Store



DeciderDecider


QueryTransformer

Data LayerData Layer Data LayerData Layer Data LayerData Layer Data LayerData Layer

RDF Graph

RDF Graph

RDF GraphR

DF Graph

RDF Graph

RDF Graph

Default Graph

RDF Graph

RDF Graph

RDF Graph R

DF Graph

RDF Graph

RDF GraphR

DF Graph

14

LarKC Data Model :Transport By Reference

RDF Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

Default Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

Dataset: Collectionof named graphs

Labeled Set: Pointers to data

Current Scale: O(1010) triples

15


DeciderDecider



QueryTransformer

QueryTransformer










SelecterSelecter



ReasonerReasoner





RDFStoreRDF

Store



DeciderDecider


QueryTransformer


RDF Graph

RDF Graph

RDF GraphR

DF Graph

RDF Graph

RDF Graph

Default Graph

RDF Graph

RDF Graph

RDF Graph R

DF Graph

RDF Graph

RDF GraphR

DF Graph

16

What Does a Pipeline Look Like?

DeciderDecider



QueryTransformer

QueryTransformer










SelecterSelecter



ReasonerReasoner



WlorkflowSupportSystem

WlorkflowSupportSystem

RDFStoreRDF

Store



DeciderDecider


QueryTransformer


Info Set Transformer




17

Remote and Heterogeneous Plug-ins

RemotePlug-in Manager

RemotePlug-in Manager

AdaptorAdaptor

External or non-Java Code

External or non-Java Code

TRANSFORMTRANSFORM

SPARQL-CycL

SPARQL-CycL

Research Cyc

Research Cyc

TRANSFORMTRANSFORM

SPARQL- GATE APISPARQL- GATE API

GATEGATE

IDENTIFYIDENTIFY

SPARQLSPARQL

SINDICESINDICE

IDENTIFYIDENTIFY

SPARQLSPARQL

Medical Data

Medical Data


18


DeciderDecider



QueryTransformer

QueryTransformer










SelecterSelecter



ReasonerReasoner





RDFStoreRDF

Store



DeciderDecider


QueryTransformer



Info Set TransformerIdentifierIdentifier


Info Set TransformerReasonerReasoner

19

Decider Using Plug-in Registry to Create Pipeline

Q

TT

II

SS RR

VB

A

Q

TTII

SS RR

VB

B

D 1.3.1Represent Properties• Functional• Non-functional (e.g. QoS)• WSMO-Lite Syntax

Represent Properties• Functional• Non-functional (e.g. QoS)• WSMO-Lite Syntax

Logical Representation• Describes role• Describes Inputs/Outputs• Automatically extracted using API• Decider can use for dynamic configuration

• Rule-based• Fast

Logical Representation• Describes role• Describes Inputs/Outputs• Automatically extracted using API• Decider can use for dynamic configuration

• Rule-based• Fast

20

LarKC Plug-ins


QueryTransformer

QueryTransformer




Plug-in APIPlug-in API• Provide SPARQL end-points• Run in separate threads• Automatically add meta-data to registry when loaded• Communicate RDF data by passing labelled sets or references to labelled sets

• Parallelisation in progressPlug-in ManagerPlug-in Manager

TransformerTransformer





ransformerransformerTransformerTransformer

TransformerTransformer



Plug-in APIPlug-in APIPlug-in ManagerPlug-in Manager




SelectorSelector



SelectorSelector


• Split/Join connectors in progress

21





Registry

RDFStoreRDF

StoreRDF

StoreRDF

StoreRDF

StoreRDF

StoreRDFDocRDFDoc

RDFDocRDFDoc

RDFDocRDFDoc


DeciderDecider



QueryTransformer

QueryTransformer










SelecterSelecter



ReasonerReasoner



LarKC Data Layer

22

Platform Utility Functionality

APIs

Plug-ins

External systems

External data sourcesData Layer APIData Layer API


LarKC Data Layer

23

RDF Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

Default Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

RDF Graph

DatasetLabeled Set

Main goal:• The LarKC Data Layer supports all LarKC

plug-ins with respect to:– storage, retrieval and light-weight inference on top

of large volumes of data– automates the exchange of RDF data by reference

and by value– offers other utility tools to manage data (e.g.

merger)

• The implementation of the data layer was evaluated against– Well-known benchmarks: LUBM (Lehigh Univ. Benchmark) and

BSBM (Berlin SPARQL Benchmark), and

– Two views to the web of linked data used in LarKC: PIKB (Pathway and Interaction Knowledge Base) and LDSR (Linked Data Semantic Repository)

• Loading: – 15B statements at 18 KSt/sec. on $10,000 server

– 1B statements at 66 KSt/sec. on $2,000 desktop

• Reasoning & Materialization: – LUBM: 21 KSt/sec for 1BSt and 10 KSt/sec for 7B expl. statements

– LDSR: 14 KSt/sec for 357M expl. statements

– PIKB: 10 KSt/sec for 1.5B expl. Statements

• Competitive with State of the Art

24

LarKC Data Layer Performance

• Inference with both LDSR and PIKB prove to be much more complex than LUBM, because

– The datasets are much better interconnected

– There are plenty of owl:sameAs links

– OWL vocabulary is used disregarding its formal semantics• E.g. in DBPedia there are skos:broader cycles of categories with length 180

• Optimizations of the handling of owl:sameAs are crucial• PIKB: 1.47B explicit statements + 842M inferred• LDSR loaded in 7 hours on desktop:

– Number of imported statements (NIS): 357M

– Number of new inferred statements: 512M

– Number of stored statements (NSS): 869M

– Number of retrievable statements (NRS): 1.14B• owl:sameAs optimisation allowed reducing the indices by 280M statements

25

LarKC Data Layer Evaluation: Linked Data

Plug-in Architecture Signs of Success




• Platform and Plug-in APIs are useable

• In the twenties of plug-ins already• Plug-ins written with little help from architects• Plug-ins run successfully, and perform together

• Outside plugin-writers:– OKKAM, NeOn, Aberdeen

26

Active and Ready for the Public

• 2170 check-outs

• 1380 commits

• 23 users of code repository – LarKC + Alpha– Plus Early Adopters Workshop branch

• 20 downloads of alpha 1 public release since 30th May 2009.

27

Project Timeline

Surveys (plug-ins, platform) & Requirements (use cases)

Prototype Internal Release Public Release Final Release

420 6 18 3310 14

Plug-insPlug-ins

Use Cases V1

Use Cases V1

Use Cases V2

Use Cases V2

Use Cases V3

Use Cases V3

Data caching

Offer computing resources

Anytime behaviour

Monitoring & instrumentation

28

Rapid Progress, but We’re Not Finished…





Registry

RDFStoreRDF

StoreRDF

StoreRDF

StoreRDF

StoreRDF

StoreRDFDocRDFDoc

RDFDocRDFDoc

RDFDocRDFDoc


DeciderDecider



QueryTransformer

QueryTransformer










SelecterSelecter



ReasonerReasoner



– Data Caching

– Anytime Behaviour

– Plug-in Registration and Discovery

– Plug-in Monitoring and Measurement

– Support for Developers

– Plug-ins

• Classified according to:– Resources

– Heterogeneity

– Usage

– Interoperability

– Parallelization “within plug-ins”

– Distributed/remote execution

– Data Layer

• Sources– Initial Project Objectives (DoW)– LarKC Collider Platform (WP5 discussions)– LarKC Rapid Prototyping– LarKC Use Cases (WP6, WP7a, WP7b)– LarKC Plug-ins (WP2, WP3, WP4)

Detailed information in D5.3.1

Requirements Analysis and

report on lessons learned during

prototyping

Requirements (WP 5)• Optimisation of complex workflows.• Extend meta-data representation for QoS, parallelism and use it.• Concentrate on parallel and distributed execution.• Concentrate on parallel and distributed data layer; caching and data migration.• Support more plug-in needs while maintaining platform integrity (e.g. efficient weight modification for spreading activation)• Data write for persistent transformation (e.g. rumination reasoning in Marvin experiments)

• Optimisation of complex workflows.• Extend meta-data representation for QoS, parallelism and use it.• Concentrate on parallel and distributed execution.• Concentrate on parallel and distributed data layer; caching and data migration.• Support more plug-in needs while maintaining platform integrity (e.g. efficient weight modification for spreading activation)• Data write for persistent transformation (e.g. rumination reasoning in Marvin experiments)

• Support workflows inspired by human cognition (e.g. workflow interruption for optimal stopping)• Support anytime/streaming• Experimental instrumentation and monitoring

• Support workflows inspired by human cognition (e.g. workflow interruption for optimal stopping)• Support anytime/streaming• Experimental instrumentation and monitoring

29

• Distributed Data Layer• Caching, data

warming/cooling• Data Streaming between

remote components• Parallelization and distribution

on different types of environments (high-performance grid, desktop grid, etc.)

• Experimental instrumentation and monitoring

• Distributed Data Layer• Caching, data

warming/cooling• Data Streaming between

remote components• Parallelization and distribution

on different types of environments (high-performance grid, desktop grid, etc.)

• Experimental instrumentation and monitoring

30

Open Issues & Next Steps

• Requirements traceability and update

• Architecture refinement

• Requirements traceability and update

• Architecture refinement

Platform validationPlatform validation

Early Adopters

fin

Date post:	12-May-2015
Category:	Education
Upload:	larkc
View:	953 times
Download:	2 times

LarKC Tutorial at ISWC 2009 - Architecture

Education