Intelligence in the Data Layer

© COPYRIGHT 2015 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.

Intelligence in the Data Layer Bitemporal and Semantics Dave Cassel, Developer Community Manager

BITEMPORAL

© COPYRIGHT 2015 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED. SLIDE: 3

Typical reaction to an explanation of Bitemporal


Bitemporal Timing is everything

Valid

Tim

e

System Time

Valid Time – Real-world time, information “as it actually was”

EVENT 1

EVENT 3

EVENT 2

EVENT 2

System Time – Time it was recorded to the database

Rewind the information “as it actually was” in combination with “as it was recorded” at some point in time

Provides increased insight into your business and mission

Capture evolving schema as the shape of the data changes with changing time, a capability that has prevented relational bitemporal offerings from being widely adopted

Critical for anyone in regulated industries Even better because of Tiered Storage and

Semantics


Scenario: Securities Risk Management Terminology

Position – “The amount of a security either owned (which constitutes a long position) or borrowed (which constitutes a short position) by an individual or by a dealer. In other words, it's a trade an investor currently holds open.” (Investopedia)

Surveillance Objective

Monitor traders current positions and historical (150M LIMIT): Current Positions: Prevent unacceptable risk exposure Effective History: Accurate measurement of historical positions Governance: Identify decision information & provenance


Unitemporal

09:00 12:00 11:00 10:00

ID Trade ID Trade Value Valid Start Valid End

Why is Unitemporal not enough?

1 1 BUY120 09:00 --:-- 2 2 SELL100 10:00 --:-- 3 3 BUY80 11:00 --:--

Trade 1 (BUY120) Trade 3 (BUY80) Trade 2 (SELL100)

120M

--:-- 09:00

20M

--:-- 10:00

100M

--:-- 11:00


Unitemporal

09:00 12:00 11:00 10:00

Trade 1 (BUY120)

120M

--:-- 09:00

Trade 3 (BUY80)

20M

--:-- 10:00

100M

--:-- 11:00

Trade 2 (SELL100)


1 1 BUY120 09:00 --:-- 2 2 SELL100 10:00 --:-- 3 3 BUY80 11:00 --:--

Repudiate 10:30 Cancel 11:30

Trade 2 (SELL100)

4 2 SELL100(C) 11:30 --:-- Why is Unitemporal not enough?


Unitemporal

4 2 SELL100(C) 11:30 --:--


1 1 BUY120 09:00 --:-- 2 2 SELL100 10:00 10:30 3 3 BUY80 11:00 --:--

09:00 12:00 11:00 10:00

Trade 1 (BUY120)

120M

09:00 --:--

Trade 3 (BUY80)

100M

--:-- 11:00

Trade 2 (SELL100)


Trade 2 (SELL100)

--:--



120M 09:00

200M

11:35 Surveillance

Unitemporal

09:00 12:00 11:00 10:00

Trade 1 (BUY120) Trade 3 (BUY80)

--:-- 11:00


4 2 SELL100(C) 11:30 --:--


1 1 BUY120 09:00 --:-- 2 2 SELL100 10:00 10:30 3 3 BUY80 11:00 --:--

Excess Position Capital Limit Violation 35 Minute delay

--:--



The Special Sauce: Two Time Dimensions

Valid Time Captures what the business believed the values were

System Time Tells us when the business believed it


Bitemporal

Trade 1: BUY 120 --:-- 09:00

--:-- 09:00

120M

09:00 12:00 11:00 10:00

1 1 BUY120 09:00 --:-- 09:00 --:--

ID Trade ID Trade Value Valid Start

Valid End System Start System End


20M

09:00 12:00 11:00 10:00

1 1 BUY120 09:00 --:-- 09:00 --:--



Trade 1: BUY 120 --:-- 09:00

--:-- 09:00

--:-- 10:00 --:-- 10:00

Bitemporal

Trade 2: SELL 100

2 2 SELL100 10:00 --:-- 10:00 --:--


09:00 12:00 11:00 10:00 Repudiate

10:30 Cancel 11:30

2 2 SELL100 10:00 --:-- 10:00 --:-- 1 1 BUY120 09:00 --:-- 09:00 --:--



Trade 1: BUY 120 --:-- 09:00

--:-- 09:00

--:-- 10:00 --:-- 10:00

Trade 2: SELL 100

3 2 SELL100(C) 10:30 --:-- 11:30 --:--

Bitemporal

Trade 2: CANCEL

120M


09:00 12:00 11:00 10:00 Repudiate

10:30 Cancel 11:30

2 2 SELL100 10:00 --:-- 10:00 --:-- 1 1 BUY120 09:00 --:-- 09:00 --:--



Trade 1: BUY 120 --:-- 09:00

--:-- 09:00

--:-- 10:00 --:-- 10:00

Trade 2: SELL 100

3 2 SELL100(C) 10:30 --:-- 11:30 --:-- 120M

Bitemporal

--:-- 10:30 --:-- 11:30

Trade 2: CANCEL


09:00 12:00 11:00 10:00 Repudiate

10:30 Cancel 11:30

3 2 SELL100(C) 10:30 --:-- 11:30 --:-- 2 2 SELL100 10:00 --:-- 10:00 11:30 1 1 BUY120 09:00 --:-- 09:00 --:--



Trade 1: BUY 120 --:-- 09:00

--:-- 09:00 Trade 2: SELL 100

3 2 SELL100(C) 10:30 --:-- 11:30 --:-- 120M

--:-- 10:30 --:-- 11:30

Trade 2: CANCEL

Bitemporal

--:-- 10:00 --:-- 10:00 11:30


Trade 1: BUY 120 --:-- 09:00

--:-- 09:00

--:-- 10:00 10:00

Trade 2: SELL 100

--:-- 10:30 --:-- 11:30

Trade 2: CANCEL

11:30

2 2 SELL100 10:00 --:-- 10:00 11:30 1 1 BUY120 09:00 --:-- 09:00 --:--



09:00 12:00 11:00 10:00 Repudiate

10:30 Cancel 11:30

3 2 SELL100(C) 10:30 --:-- 11:30 --:--

Bitemporal Our Position AT 09:30 as we knew it at 11:00? Trade 1 : System Valid +120M Trade 2 : System Valid ----- Trade 2(C) : System Valid -----

120M


Graph Based View

12:00

09:00

10:00

11:00

09:00 12:00 11:00 10:00 Cancel 11:30

Repudiate 10:30 System AXIS

Additional Valid AXIS

Timelines into X(system) -Y(valid) Co-ordinates


12:00

09:00

10:00

11:00

09:00 12:00 11:00 10:00

09:00 --:-- --:-- 09:00

Cancel 11:30

--:--

Graph Based View

TRADE 1 BUY 120

Repudiate 10:30



TRADE 1 BUY 120

12:00

09:00

10:00

11:00

09:00 12:00 11:00 10:00 10:00

TRADE 2 SELL 100

11:30 10:00 --:--

10:00

09:00

--:--

Repudiate 10:30

Graph Based View

TRADE 2 CANCEL

--:--

10:30

--:-- 11:30 --:--

Cancel 11:30



TRADE 1 BUY 120

12:00

09:00

10:00

11:00

09:00 12:00 11:00 10:00 10:00

TRADE 2 SELL 100

11:30 10:00 --:--

10:00

09:00

--:--

Cancel 11:30

Repudiate 10:30

Graph Based View

TRADE 2 CANCEL

--:--

10:30

--:-- 11:30 --:--

Viewed at 11:00 20M

What was our position at 10:30



Use Cases Adverse events in healthcare – It's important to know what the state of the patient was when an adverse event occurred. Bitemporal helps with case management by better tracking a patient’s history over time

Coverage determination with insurance – Bitemporal helps determine what type of coverage a beneficiary had at the point of diagnosis, and how the coverage change based on the new legislation that was enacted retroactively

Gathering intelligence – Bitemporal helps paint a complete picture (predict future events/ understand motives) even when facts are piece-meal (some times before or after the event)

Investigations in criminal law – Bitemporal enables you to go back and ask why you went down a certain path, and is particularly useful when investigations are resurrected from cold case files


Use Cases Documentation in publishing – Companies that process legal documents can see the issue date of the document, what laws were applicable at the time, and then what laws came into effect later over time

Trade reconciliation in financial services – When trades are reconciled with counterparties and closed, updates can occur. Bitemporal helps ensure investment banks can always go back and see when updates occurred


Additional Resources Feature Intro

Temporal Developer's Guide

APIs: JavaScript, XQuery

Fei's demo

http://developer.marklogic.com/features/bitemporal

http://docs.marklogic.com/guide/temporal

http://docs.marklogic.com/js/temporal/temporal

http://docs.marklogic.com/temporal/temporal

https://github.com/fxue/bitemporal-demo

http://docs.marklogic.com/guide/temporal

SEMANTICS


Semantics: Phase 2

Performance! Aggregates Updates

Property Paths

Inference


Inference: Automatically discover new facts

Database

prod

uct

Software is

Registration Data

Industry Data General Data

Software Company isA

Inference

worksAt

Stephen Buxton

MarkLogic

worksAt


Backward-chaining vs Forward-chaining Forward-chaining: At ingest time, insert a new triple

Ingest (and update) is very slow

More disk space

Materialize every possible inferred triple

Implications for security, ACID

.. But queries are fast

Backward-chaining: At query time, return results as if a new triple existed

Fast ingest, less disk space, security and ACID are straightforward, only do the work that's needed

.. But work is done at query-time

John England London isIn

livesIn

livesIn


Property paths SPARQL 1.1 Query – Property Paths

Property path operators added for MarkLogic 8:

– ? – zero or one path

– + – one or more path

– * – zero or more path


Property paths - example ## find papers that cite paper A, and papers that cite papers that cite paper A, and so on

SELECT ?s

WHERE {

?s c:cites*/dc:title "Paper A" .

}

ORDER BY ?s

Note: taken from Bob DuCharme's "Learning SPARQL" For more examples, see http://ea.marklogic.com/wp-content/uploads/2014/08/SPARQL-paths-examplesEA2.pdf

http://ea.marklogic.com/wp-content/uploads/2014/08/SPARQL-paths-examplesEA2.pdf


SPARQL Aggregates Aggregate functionality includes:

GROUP BY

COUNT

SUM

MIN

MAX

SAMPLE

AVG

GROUP_CONCAT

GROUP BY .. HAVING <some aggregate variable>

ORDER BY <some aggregate variable>

GROUP BY <more than one item>


SPARQL aggregates - example ## count how many companies are in each industry sector

SELECT ?industry ( COUNT ( ?company ) AS ?count_companies )

FROM <http://marklogic.com/semantics/sb/COMPANIES100/>

WHERE {

?company demov:industry ?industry .

}

GROUP BY ?industry

For more examples, see http://ea.marklogic.com/wp-content/uploads/2014/09/SPARQL-aggregates-examples-EA2.pdf

http://ea.marklogic.com/wp-content/uploads/2014/09/SPARQL-aggregates-examples-EA2.pdf


SPARQL Update SPARQL Update operations, examples

Managed triples

Graph permissions

Locking


SPARQL UPDATE operations 1. GRAPH MANAGEMENT Manipulate RDF graphs using the SPARQL 1.1 Update language

CREATE – create a graph

DROP – drop a graph and its contents

COPY – make the destination graph into a copy of the source graph; any content in the destination graph before this operation will be removed (think copy/paste)

MOVE – move the contents of the source graph into the destination graph, and remove them from the source graph; any content in the destination graph before this operation will be removed (think cut/paste)

ADD – add the contents of the source graph into the destination graph; keep the source graph intact; keep the initial contents of the destination graph intact


SPARQL UPDATE operations

2. GRAPH UPDATE Delete, insert, and update (delete/insert) triples using the SPARQL 1.1 Update language.

INSERT DATA

DELETE DATA

DELETE .. INSERT WHERE

DELETE WHERE

INSERT WHERE

CLEAR


SPARQL Update APIs Query Console – new mode "SPARQL Update"

REST – POST to /v1/graphs/sparql

Server-side built-ins

– sem:sparql-update()

– sem.sparqlUpdate()


Managed Triples SPARQL Update operations operate over managed triples only

Managed triples are triples loaded into the database using

– mlcp with -input_file_type RDF

– sem:rdf-load()

– sem:rdf-insert()

cf embedded triples

– Triples embedded in XML or JSON documents

– SPARQL Update operations don’t affect embedded triples

[Strictly, a managed triple is a sem:triple element in a document with root element sem:triples]


Graph Permissions Set permissions when you create a graph:

import module namespace sem = http://marklogic.com/semantics at "/MarkLogic/semantics.xqy";

sem:sparql-update(

'CREATE GRAPH <graphs/sb/graph-1>', (),(),(), ( xdmp:permission( "demo-reader", "read" ), xdmp:permission( "demo-writer", "update" ) ) )

Note: arg5 is called $default-permissions , but you should set permissions explicitly

– See also sem:graph-set-permissions()


Graph Permissions Check permissions on the graph you just created:

import module namespace sem = http://marklogic.com/semantics at "/MarkLogic/semantics.xqy";

sem:graph-get-permissions( sem:iri("graphs/sb/graph-1") )

See also

– sem:graph-set-permissions()

– sem:graph-add-permissions()

– sem:graph-remove-permissions()


Graph Permissions Set permissions when you create a graph:

var sem = require('/MarkLogic/semantics');

sem.sparqlUpdate( 'CREATE GRAPH <graphs/sb/graph-2>', [],[],[], ( xdmp.permission( "demo-reader", "read" ), xdmp.permission( "demo-writer", "update" ) ) )

Note: arg5 is called $default-permissions , but you should set permissions explicitly

– See also sem.graphSetPermissions()


Graph Permissions Check permissions on the graph you just created:

var sem = require('/MarkLogic/semantics');

sem.graphGetPermissions( sem.iri("graphs/sb/graph-5") )

See also

– sem:graphSetPermissions()

– sem:graphAddPermissions()

– sem:graphRemovePermissions()


Additional Resources Feature Intro

Semantics Developer's Guide

APIs: JavaScript, XQuery

Charles' demo

http://developer.marklogic.com/features/semantics

http://docs.marklogic.com/guide/semantics

http://docs.marklogic.com/js/sem/semantics

http://docs.marklogic.com/sem/semantics

https://github.com/grechaw/semantics-tutorial

MORE!


There’s a lot more to MarkLogic 8

Management REST API

Incremental Backup

Query-Based Flex-Rep

Enhanced HTTP Server

MARKLOGIC 8: AVAILABLE NOW!!

Learn More about NoSQL and MarkLogic 8

Read nosqlfordummies.com

See It! Meet marklogic.com/training [email protected]

Comment marklogic.com/blog

MarkLogic World Tour 2015 San Francisco | April 13-17

Washington, DC | May 7

Amsterdam | May 12

London | May 14

Boston | May 19

Tokyo | May 21

Chicago | June 2

New York City | June 18

Houston | July 16

Join us! Register today: world.marklogic.com Listen online! mlwonline.marklogic.com

Date post:	30-Jul-2015
Category:	Software
Upload:	marklogic
View:	107 times
Download:	0 times