The Gremlin in the Graph

The Gremlin in the Graph

Marko A. RodriguezGraph Systems Architecthttp://markorodriguez.com

http://twitter.com/twarko

http://slideshare.com/slidarko

First University-Industry Meeting on Graph Databases (UIM-GDB)

Universitat Politecnica de Catalunya – February 8, 2011

February 6, 2011

http://markorodriguez.com

http://twitter.com/twarko

http://slideshare.com/slidarko

Abstract

This tutorial/lecture addresses various aspects of the graph traversallanguage Gremlin. In particular, the presentation focuses on Gremlin 0.7and its application to graph analysis and processing.

GremlinG = (V,E)

http://gremlin.tinkerpop.com


TinkerPop Productions

1

1TinkerPop (http://tinkerpop.com), Marko A. Rodriguez (http://markorodriguez.com), PeterNeubauer (http://www.linkedin.com/in/neubauer), Joshua Shinavier (http://fortytwo.net/), PavelYaskevich (https://github.com/xedin), Darrick Wiebe (http://ofallpossibleworlds.wordpress.com/), Stephen Mallette (http://stephen.genoprime.com/), Alex Averbuch (http://se.linkedin.com/in/alexaverbuch)

http://tinkerpop.com

http://markorodriguez.com

http://www.linkedin.com/in/neubauer

http://fortytwo.net/

https://github.com/xedin

http://ofallpossibleworlds.wordpress.com/

http://ofallpossibleworlds.wordpress.com/

http://stephen.genoprime.com/

http://se.linkedin.com/in/alexaverbuch

http://se.linkedin.com/in/alexaverbuch

Gremlin is a Domain Specific Language

• A domain specific language for graph analysis and manipulation.

• Gremlin 0.7+ uses Groovy has its host language.2

• Groovy is a superset of Java and as such, natively exposes the full JDKto Gremlin.

2Groovy available at http://groovy.codehaus.org/.

http://groovy.codehaus.org/

Gremlin is for Property Graphs

name = "marko"age = 29

1

4

knows

weight = 1.0

name = "josh"age = 32

name = "vadas"age = 27

2

knows

weight = 0.5

created

weight = 0.4

name = "lop"lang = "java"

3created

weight = 0.4

name = "ripple"lang = "java"

5

created

weight = 1.0

name = "peter"age = 35

6

created

weight = 0.2

78

9

11

10

12

Gremlin is for Blueprints-enabled Graph Databases

• Blueprints can be seen as the JDBC for property graph databases.3

• Provides a collection of interfaces for graph database providers to implement.

• Provides tests to ensure the operational semantics of any implementation are correct.

• Provides support for GraphML and other “helper” utilities.

Blueprints

3Blueprints is available at http://blueprints.tinkerpop.com.

http://blueprints.tinkerpop.com

A Blueprints Detour - Basic Example

Graph graph = new Neo4jGraph("/tmp/my_graph");

Vertex a = graph.addVertex(null);

Vertex b = graph.addVertex(null);

a.setProperty("name","marko");

b.setProperty("name","peter");

Edge e = graph.addEdge(null, a, b, "knows");

e.setProperty("since", 2007);

graph.shutdown();

0 1knows

name=marko name=petersince=2007

A Blueprints Detour - Sub-Interfaces

• TransactionalGraph extends Graph

? For transaction based graph databases.4

• IndexableGraph extends Graph

? For graph databases that support the indexing of properties.5

4Graph Transactions https://github.com/tinkerpop/blueprints/wiki/Graph-Transactions.5Graph Indices https://github.com/tinkerpop/blueprints/wiki/Graph-Indices.

https://github.com/tinkerpop/blueprints/wiki/Graph-Transactions

https://github.com/tinkerpop/blueprints/wiki/Graph-Indices

A Blueprints Detour - Implementations

TinkerGraph

A Blueprints Detour - Implementations

• TinkerGraph implements IndexableGraph

• Neo4jGraph implements TransactionalGraph, IndexableGraph6

• OrientGraph implements TransactionalGraph, IndexableGraph7

• RexsterGraph implements IndexableGraph: Implementation of aRexster REST API.8

• SailGraph implements TransactionalGraph: Turns any Sail-basedRDF store into a Blueprints Graph.9

6Neo4j is available at http://neo4j.org.7OrientDB is available at http://www.orientechnologies.com.8Rexster is available at http://rexster.tinkerpop.com.9Sail is available at http://openrdf.org.

http://neo4j.org

http://www.orientechnologies.com

http://rexster.tinkerpop.com

http://openrdf.org

A Blueprints Detour - Ouplementations

JUNGJava Universal Network/Graph Framework

A Blueprints Detour - Ouplementations• GraphSail: Turns any IndexableGraph into an RDF store. 10

• GraphJung: Turns any Graph into a JUNG graph.11 12

10GraphSail https://github.com/tinkerpop/blueprints/wiki/Sail-Ouplementation.11JUNG is available at http://jung.sourceforge.net/.12GraphJung https://github.com/tinkerpop/blueprints/wiki/JUNG-Ouplementation.

https://github.com/tinkerpop/blueprints/wiki/Sail-Ouplementation

http://jung.sourceforge.net/

https://github.com/tinkerpop/blueprints/wiki/JUNG-Ouplementation

Gremlin Compiles Down to Pipes

• Pipes is a data flow framework for evaluating lazy graph traversals.13

• A Pipe extends Iterator, Iterable and can be chained togetherto create processing pipelines.

• A Pipe can be embedded into another Pipe to allow for nestedprocessing.

Pipes13Pipes is available at http://pipes.tinkerpop.com.

http://pipes.tinkerpop.com

A Pipes Detour - Chained Iterators

• This Pipeline takes objects of type A and turns them into objects oftype D.

Pipe1A B Pipe2 C Pipe3 D

Pipeline

A

AA

A

D

DD

D

Pipe<A,D> pipeline =

new Pipeline<A,D>(Pipe1<A,B>, Pipe2<B,C>, Pipe3<C,D>)

A Pipes Detour - Simple Example

“What are the names of the people that marko knows?”

A C

B

D

knows

knows

created

createdname=marko

name=peter

name=pavel

name=gremlin

A Pipes Detour - Simple ExamplePipe<Vertex,Edge> pipe1 = new VertexEdgePipe(Step.OUT_EDGES);

Pipe<Edge,Edge> pipe2= new LabelFilterPipe("knows",Filter.NOT_EQUAL);

Pipe<Edge,Vertex> pipe3 = new EdgeVertexPipe(Step.IN_VERTEX);

Pipe<Vertex,String> pipe4 = new PropertyPipe<String>("name");

Pipe<Vertex,String> pipeline = new Pipeline(pipe1,pipe2,pipe3,pipe4);

pipeline.setStarts(new SingleIterator<Vertex>(graph.getVertex("A"));

A C

B

D

knows

knows

created

createdname=marko

name=peter

name=pavel

VertexEdgePipe(OUT_EDGES)

LabelFilterPipe("knows")

EdgeVertexPipe(IN_VERTEX)

PropertyPipe("name")

name=gremlin

HINT: The benefit of Gremlin is that this Java verbosity is reduced to g.v(‘A’).outE[[label:‘knows’]].inV.name.

A Pipes Detour - Pipes Library

[ FILTERS ]

AndFilterPipe

CollectionFilterPipe

ComparisonFilterPipe

DuplicateFilterPipe

FutureFilterPipe

ObjectFilterPipe

OrFilterPipe

RandomFilterPipe

RangeFilterPipe

[ GRAPHS ]

EdgeVertexPipe

IdFilterPipe

IdPipe

LabelFilterPipe

LabelPipe

PropertyFilterPipe

PropertyPipe

VertexEdgePipe

[ SIDEEFFECTS ]

AggregatorPipe

GroupCountPipe

CountPipe

SideEffectCapPipe

[ UTILITIES ]

GatherPipe

PathPipe

ScatterPipe

Pipeline

...

A Pipes Detour - Creating Pipes

public class NumCharsPipe extends AbstractPipe<String,Integer> {

public Integer processNextStart() {

String word = this.starts.next();

return word.length();

}

}

When extending the base class AbstractPipe<S,E> all that is required isan implementation of processNextStart().

Now onto Gremlin proper...

The Gremlin Architecture

OrientDB TinkerGraphNeo4j

The Many Ways of Using Gremlin

• Gremlin has a REPL to be run from the shell.14

• Gremlin can be natively integrated into any Groovy class.

• Gremlin can be interacted with indirectly through Java, via Groovy.

• Gremlin has a JSR 223 ScriptEngine as well.

14All examples in this lecture/tutorial are via the command line REPL.

The Seamless Nature of Gremlin/Groovy/JavaSimply Gremlin.load() and add Gremlin to your Groovy class.

// a Groovy class

class GraphAlgorithms {

static {

Gremlin.load();

}

public static Map<Vertex, Integer> eigenvectorRank(Graph g) {

Map<Vertex,Integer> m = [:]; int c = 0;

g.V.outE.inV.groupCount(m).loop(3) {c++ < 1000} >> -1;

return m;

}

}

Writing software that mixes Groovy and Java is simple...dead simple.

// a Java class

public class GraphFramework {

public static void main(String[] args) {

System.out.println(GraphAlgorithms.eigenvectorRank(new Neo4jGraph("/tmp/graphdata"));

}

}

Property Graph Model and Gremlin

1

4

knows created

3created

vertex 4 id

vertex 1 out edges

9

8 11

edge 9 labelvertex 3 in edges

edge 9 in vertexedge 9 out vertexedge 9 id


vertex 4 properties

• outE: outgoing edges from a vertex.

• inE: incoming edge to a vertex.

• inV: incoming/head vertex of an edge.

• outV: outgoing/tail vertex of an edge.15

15Documentation on atomic steps: https://github.com/tinkerpop/gremlin/wiki/Gremlin-Steps.

https://github.com/tinkerpop/gremlin/wiki/Gremlin-Steps

Loading a Toy Graph


1

4

knows

weight = 1.0



2

knows

weight = 0.5

created

weight = 0.4


3created

weight = 0.4

name = "ripple"lang = "java"

5

created

weight = 1.0

name = "peter"age = 35

6

created

weight = 0.2

78

9

11

10

12

marko$ ./gremlin.sh

\,,,/

(o o)

-----oOOo-(_)-oOOo-----

gremlin> g = TinkerGraphFactory.createTinkerGraph()

==>tinkergraph[vertices:6 edges:6]

gremlin>

16

16Load up the toy 6 vertex/6 edge graph that comes hardcoded with Blueprints.

Basic Gremlin Traversals - Part 1


1

4

knows



2

knows

created


3created

5

created

6

created

78

9

11

10

12

weight = 1.0

weight = 0.5

weight = 0.4

gremlin> g.v(1)

==>v[1]

gremlin> g.v(1).map

==>name=marko

==>age=29

gremlin> g.v(1).outE

==>e[7][1-knows->2]

==>e[9][1-created->3]

==>e[8][1-knows->4]

gremlin>

17

17Look at the neighborhood around marko.



1

4

knows

2

knows

created


3created

5

created

6

created

78

9

11

10

12

gremlin> g.v(1)

==>v[1]

gremlin> g.v(1).outE[[label:‘created’]].inV

==>v[3]

gremlin> g.v(1).outE[[label:‘created’]].inV.name

==>lop

gremlin>

18

18What has marko created?



1

4

knows



2

knows

created


3created

5

created

6

created

78

9

11

10

12

gremlin> g.v(1)

==>v[1]

gremlin> g.v(1).outE[[label:‘knows’]].inV

==>v[2]

==>v[4]

gremlin> g.v(1).outE[[label:‘knows’]].inV.name

==>vadas

==>josh

gremlin> g.v(1).outE[[label:‘knows’]].inV{it.age < 30}.name

==>vadas

gremlin>

19

19Who does marko know? Who does marko know that is under 30 years of age?


marko

josh

knows

vadas

knows

created

lop

created

ripple

created

peter

created

codeveloper

codeveloper

codeveloper

gremlin> g.v(1)

==>v[1]


==>v[3]


.inE[[label:‘created’]].outV

==>v[1]

==>v[4]

==>v[6]


.inE[[label:‘created’]].outV.except([g.v(1)])

==>v[4]

==>v[6]


.inE[[label:‘created’]].outV.except([g.v(1)]).name

==>josh

==>peter

gremlin>

20

20Who are marko’s codevelopers? That is, people who have created the same software as him, but arenot him (marko can’t be a codeveloper of himeself).


marko

josh

knows

vadas

knows

created

lop

created

ripple

created

peter

created

codeveloper

codeveloper

gremlin> Gremlin.addStep(‘codeveloper’)

==>null

gremlin> c = {_{x = it}.outE[[label:‘created’]].inV

.inE[[label:‘created’]].outV{!x.equals(it)}}

==>groovysh_evaluate$_run_closure1@69e94001

gremlin> [Vertex, Pipe].each{ it.metaClass.codeveloper =

{ Gremlin.compose(delegate, c())}}

==>interface com.tinkerpop.blueprints.pgm.Vertex

==>interface com.tinkerpop.pipes.Pipe

gremlin> g.v(1).codeveloper

==>v[4]

==>v[6]

gremlin> g.v(1).codeveloper.codeveloper

==>v[1]

==>v[6]

==>v[1]

==>v[4]

21

21I don’t want to talk in terms of outE, inV, etc. Given my domain model, I want to talk in terms ofhigher-order, abstract adjacencies — e.g. codeveloper.

Lets explore more complex traversals...

Grateful Dead Concert Graph

The Grateful Dead were an American band that was born out of the San Francisco,

California psychedelic movement of the 1960s. The band played music together

from 1965 to 1995 and is well known for concert performances containing extended

improvisations and long and unique set lists. [http://arxiv.org/abs/0807.2466]

http://arxiv.org/abs/0807.2466

Grateful Dead Concert Graph Schema

• vertices

? song vertices

∗ type: always song for song vertices.

∗ name: the name of the song.

∗ performances: the number of times the song was played in concert.

∗ song type: whether the song is a cover song or an original.

? artist vertices

∗ type: always artist for artist vertices.

∗ name: the name of the artist.

• edges

? followed by (song → song): if the tail song was followed by the head song in

concert.

∗ weight: the number of times these two songs were paired in concert.

? sung by (song→ artist): if the tail song was primarily sung by the head artist.

? written by (song→ artist): if the tail song was written by the head artist.

A Subset of the Grateful Dead Concert Graph

1

type="song"name="Scarlet.."

2

type="song"name="Fire on.."

3

type="song"name="Pass.."

4

type="song"name="Terrap.."

followed_by

followed_by

followed_by

weight=239

weight=1

weight=2

5

type="artist"name="Garcia"

sung_by

sung_by

6

type="artist"name="Lesh"

sung_by

sung_by

7

type="artist"name="Hunter"

written_by

written_by

written_by

Centrality in a Property Graph

Garcia

Dark Star

Terrapin Station

China Doll

Stella Blue

Peggy O

sung_by

...

sung_bysung_by

sung_by

sung_bysung_by

sung_bysung_by

sung_bysung_by

sung_bysung_by

followed_byfollowed_by

followed_by

followed_byfollowed_by

followed_by

...

• What is the most central vertex in this graph? – Garcia.

• What is the most central song? What do you mean “central?”

• How do you calculate centrality when there are numerous ways in which vertices can

be related?

Loading the GraphML Representation

gremlin> g = new TinkerGraph()

==>tinkergraph[vertices:0 edges:0]

gremlin> GraphMLReader.inputGraph(g,

new FileInputStream(‘data/graph-example-2.xml’))

==>null

gremlin> g.V.name

==>OTHER ONE JAM

==>Hunter

==>HIDEAWAY

==>A MIND TO GIVE UP LIVIN

==>TANGLED UP IN BLUE

...

22

22Load the GraphML (http://graphml.graphdrawing.org/) representation of the Grateful Deadgraph, iterate through its vertices and get the name property of each vertex.

http://graphml.graphdrawing.org/

Using Graph Indices

gremlin> v = g.idx(T.v)[[name:‘DARK STAR’]] >> 1

==>v[89]

gremlin> v.outE[[label:‘sung_by’]].inV.name

==>Garcia

23

23Use the default vertex index (T.v) and find all vertices index by the key/value pair name/DARK STAR.Who sung Dark Star?

Emitting Collections

gremlin> v.outE[[label:‘followed_by’]].inV.name

==>EYES OF THE WORLD

==>TRUCKING

==>SING ME BACK HOME

==>MORNING DEW

...

gremlin> v.outE[[label:‘followed_by’]].emit{[it.inV.name >> 1, it.weight]}

==>[EYES OF THE WORLD, 9]

==>[TRUCKING, 1]

==>[SING ME BACK HOME, 1]

==>[MORNING DEW, 11]

...

24

24What followed Dark Star in concert?How many times did each song follow Dark Start in concert?

Eigenvector Centrality – Indexing Vertices

gremlin> m = [:]; c = 0

==>0

gremlin> g.V.outE[[label:‘followed_by’]].inV.groupCount(m)

.loop(4){c++ < 1000}.filter{false}

gremlin> m.sort{a,b -> b.value <=> a.value}

==>v[13]=762

==>v[21]=668

==>v[50]=661

==>v[153]=630

==>v[96]=609

==>v[26]=586

...

25

25Emanate from each vertex the followed by path. Index each vertex by how many times its beentraversed over in the map m. Do this for 1000 times. The final filter{false} will ensure nothing isoutputted.

Eigenvector Centrality – Indexing Names

gremlin> m = [:]; c = 0

==>0

gremlin> g.V.outE[[label:‘followed_by’]].inV.name.groupCount(m)

.back(2).loop(4){c++ < 1000}.filter{false}

gremlin> m.sort{a,b -> b.value <=> a.value}

==>PLAYING IN THE BAND=762

==>TRUCKING=668

==>JACK STRAW=661

==>SUGAR MAGNOLIA=630

==>DRUMS=609

==>PROMISED LAND=586

...

26

26Do the same, index the names of the songs in the map m. However, since you can get the outgoingedges of a string, jump back 2 steps, then loop.

Paths in a Graphgremlin> g.v(89).outE.inV.name

==>EYES OF THE WORLD

==>TRUCKING

==>SING ME BACK HOME

==>MORNING DEW

==>HES GONE

...

gremlin> g.v(89).outE.inV.name.paths

==>[v[89], e[7021][89-followed_by->83], v[83], EYES OF THE WORLD]

==>[v[89], e[7022][89-followed_by->21], v[21], TRUCKING]

==>[v[89], e[7023][89-followed_by->206], v[206], SING ME BACK HOME]

==>[v[89], e[7006][89-followed_by->127], v[127], MORNING DEW]

==>[v[89], e[7024][89-followed_by->49], v[49], HES GONE]

...

27

27What are the paths that are out.inV.name emanating from Dark Star (v[89])?

Paths of Length < 4 Between Dark Star and Drumsgremlin> g.v(89).outE.inV.loop(2){it.loops < 4 & !(it.object.equals(g.v(96)))}[[id:‘96’]].paths

==>[v[89], e[7014][89-followed_by->96], v[96]]

==>[v[89], e[7021][89-followed_by->83], v[83], e[1418][83-followed_by->96], v[96]]












...

28

28Get the adjacent vertices to Dark Star (v[89]). Loop on outE.inV while the amount of loops is lessthan 4 and the current path hasn’t reached Drums (v[96]). Return the paths traversed.

Shortest Path Between Dark Star and Drums

gremlin> [g.v(89).outE.inV

.loop(2){!(it.object.equals(g.v(96)))}[[id:‘96’]].paths >> 1]

==>[v[89], e[7014][89-followed_by->96], v[96]]

gremlin> (g.v(89).outE.inV.loop(2)

{!(it.object.equals(g.v(96)))}[[id:‘96’]].paths >> 1).size() / 3

==>1

29

29Given the nature of the loop step, the paths are emitted in order of length. Thus, just “pop off” thefirst path (>> 1) and that is the shortest path. You can then gets its length if you want to know the lengthof the shortest path. Be sure to divide by 3 as the path has the start vertex, edge, and end vertex.

Integrating the JDK (Java API)

• Groovy is the host language for Gremlin. Thus, the JDK is nativelyavailable in a path expression.

gremlin> v.outE[[label:‘followed_by’]].inV{it.name.matches(‘D.*’)}.name

==>DEAL

==>DRUMS

• Examples below are not with respect to the Grateful Dead schemapresented previously. They help to further articulate the point at hand.

gremlin> x.outE.inV.emit{JSONParser.get(it.uri).reviews}.mean()

...

gremlin> y.outE[[label:‘rated’]].filter{TextAnalysis.isElated(it.review)}.inV

...

The Concept of Abstract Adjacency – Part 1

• Loop on outE.inV.

g.v(89).outE.inV.loop(2){it.loops < 4}

• Loop on followed by.

g.v(89).outE[[label:‘followed_by’]].inV.loop(3){it.loops < 4}

• Loop on cofollowed by.

g.v(89).outE[[label:‘followed_by’]].inV

.inE[[label:‘followed_by’]].outV.loop(6){it.loops < 4}


}

outE

inVoutE

loop(2){..} back(3){it.salary}

friend_of_a_friend_who_earns_less_than_friend_at_work

• outE, inV, etc. is low-level graph speak (the domain is the graph).

• codeveloper is high-level domain speak (the domain is software development).30

30In this way, Gremlin can be seen as a DSL (domain-specific language) for creating DSL’s for yourgraph applications. Gremlin’s domain is “the graph.” Build languages for your domain on top of Gremlin(e.g. “software development domain”).

Explicit Graph

Developer CreatedGraph

Software ImportsGraph

FriendshipGraph

EmploymentGraph

Developer Imports Graph

Friend-Of-A-FriendGraph

Friends at WorkGraph

Developer's FOAF ImportGraph

You need not make derived graphs explicit. You can, at runtime, compute them. Moreover, generate them locally, not globally (e.g. ``Marko's friends from work relations").

This concept is related to automated reasoning and whether reasoned relations are inserted into the explicit graph or computed at query time.


• Gremlin provides fine grained (Turing-complete) control over how yourtraverser moves through a property graph.

• As such, there are as many shortest paths, eigenvectors/PageRanks,clusters, cliques, etc. as there are “abstract adjacencies” in thegraph.31 32

? This is the power of the property graph over standard unlabeled graphs.There are many ways to slice and dice your data.

31Rodriguez M.A., Shinavier, J., “Exposing Multi-Relational Networks to Single-Relational NetworkAnalysis Algorithms, Journal of Informetrics, 4(1), pp. 29–41, 2009. [http://arxiv.org/abs/0806.2274]

32Other terms for abstract adjacencies include domain-specific relations, inferred relations, implicit edges,virtual edges, abstract paths, derived adjacencies, higher-order relations, etc.

http://arxiv.org/abs/0806.2274

In the End, Gremlin is Just Pipes

gremlin> (g.v(89).outE[[label:‘followed_by’]].inV.name).toString()

==>[VertexEdgePipe<OUT_EDGES>, LabelFilterPipe<NOT_EQUAL,followed_by>,

EdgeVertexPipe<IN_VERTEX>, PropertyPipe<name>]

33

33Gremlin is a domain specific language for constructing lazy evaluation, data flow Pipes (http://pipes.tinkerpop.com). In the end, what is evaluated is a pipeline process in pure Java over aBlueprints-enabled property graph (http://blueprints.tinkerpop.com).



http://blueprints.tinkerpop.com

Credits• Marko A. Rodriguez: designed, developed, and documented Gremlin.

• Pavel Yaskevich: heavy development on earlier versions of Gremlin.

• Darrick Wiebe: development on Pipes, Blueprints, and inspired the move to using

Groovy as the host language of Gremlin.

• Peter Neubauer: promoter and evangelist of Gremlin.

• Ketrina Yim: designed the Gremlin logo.

GremlinG = (V,E)


http://groups.google.com/group/gremlin-users



http://groups.google.com/group/gremlin-users


Date post:	08-May-2015
Category:	Technology
Upload:	marko-rodriguez
View:	22,086 times
Download:	2 times

The Gremlin in the Graph

Technology