An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect,...

An overview of InfiniteGraph,the distributed graph database.

Darren WoodChief Architect, InfiniteGraph

For today’s discussion…

• InfiniteGraph– Background / History– Architecture– What we are working on

• Consistency Models• Challenges/Requirements of Distibution• Data model – Schema vs Schema-less

Copyright © InfiniteGraph

Building on Objectivity/DB

• Derived from Objectivity/DB core– Distributed Object Database– Native core (broad platform support) with C++,

Java, C# and Python Bindings– Thousands of deployments, many are 24x7x365– Markets : VLDB, Data Fusion / Metadata, Complex

Object Models, Relationship Analytics


Relationship Analytics Experience

• Relationship Analytics– Large part of Objy/DB Government business– Significant growth in the commercial space– Many OEM opportunities– Mostly based on graph theory– Hand coded algorithms


Needed something more…

• Objectivity Provides– Powerful distributed object storage– Flexible class based persistence– Enterprise DB features

• But….– Lots of custom coding– Complex deployment– No generic algorithm framework


Enter InfiniteGraph

• Simple Graph focused API• Automated distribution and deployment• Mostly configuration driven• Java class based persistence• Property model support• Asynchronous navigation• Indexing framework


Some code…


Vertex alice = myGraph.addVertex(new Person(“Alice”)); Vertex bob = myGraph.addVertex(new Person(“Bob”)); Vertex carlos = myGraph.addVertex(new Person(“Carlos”)); Vertex charlie = myGraph.addVertex(new Person(“Charlie”));

alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob);bob.addEdge(new Call(timestamp), carlos);carlos.addEdge(new Payment(100000.00), charlie);bob.addEdge(new Call(timestamp), charlie);

Alice Carlos CharlieBobMeets Calls Pays

Calls

Basic Architecture


IG Core/API

ConfigurationNavigation Execution

Management Extensions

BlueprintsUser Apps

Objectivity/DB Distributed Database

Session / TX ManagementPlacement


Targeting Large Graphs

• Graphs grow quickly– Billions of phone calls / day in US– Emails, social media events, IP Traffic– Financial transactions

• Some analytics require navigation of large sections of the graph

• Must distribute data and go parallel


Consistency Models

• Trading off full consistency for performance• Relax locking, allow stale reads• Gain 100x edge ingest rates• Great for “social graph” applications• These tend to have relaxed consistency

requirements• API allows choice per operation


Pipelining


IG Core/API

ConfigurationNavigation Execution

Management Extensions

Session / TX Management

Placement(MDP)

Placement(Pipelining)

V1V1

V2V2

V3V3

E12E12

E23E23

Pipeline

Manager

Sta

ging

Con

tain

ers P

ipeline Containers

E(1->2)

E(3->1)

E(2->3)

E(2->1)

E(2->3)E(3->1)

E(1->2)

E(3->2)

E(1->2)

E(2->3)

E(3->1)

E(2->1)

E(2->3)

E(3->1)

E(3->2)

E(1->2)

Distributing Navigation

• Graph algorithms naturally branch• Breaking up the process is relatively simple• Orchestrating it is more challenging


Alice

Carlos CharlieBobMeets Calls Pays

Dave EveChuckCallsLives With

Meets

Distributed API

Application(s)

Partition 1 Partition 3Partition 2 Partition ...n

Processor Processor Processor Processor

Partitioned Graphs are Ugly


Making it scale…

• A distributed data tier supports parallel IO• A distributed cache/processor tier for

processing “navlets”• Ability to deal with remote data reads (fast)• Navlets hit processor/cache instances based

on consistent hashing of their start vertex


Flexible Data Models

• Schema isn’t for everyone– Great for relatively static models– Support for inheritance helps– So does auto-evolution

• Looking to support document style elements– Support for ad-hoc attributes– Ability to mix both approaches


Other Projects

• Blueprints / Gremlin support• Native Visualizer and Management Console• Built in Qualifiers• More Indexing Enhancements• Algorithms• Large scale samples


Thank you !


[email protected]

Twitter - @infinitegraph

Date post:	19-Dec-2015
Category:	Documents
View:	218 times
Download:	0 times

An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect,...

Documents