Preparing for World Domination

A Year of Redis Modules

Dvir Volk@dvirsky

A Quick Recap of Redis

Key

"I'm a Plain Text String!"

{ A: “foo”, B: “bar”, C: “baz” }

Strings / Bitmaps / BitFields

Hash Tables (objects!)

Linked Lists

Sets

Sorted Sets

Geo Sets

HyperLogLog

{ A , B , C , D , E }

[ A → B → C → D → E ]

{ A: 0.1, B: 0.3, C: 100, D: 1337 }

{ A: (51.5, 0.12), B: (32.1, 34.7) }

00110101 11001110 10101010

But Can Redis Do X?

So You Want a New Feature?

● Convince @antirez● Fork Redis● Build Your Own Database!

First Mention of Modules: Redis 1.0

Fast Forward 7 Years...

● Discussion began last Redis Day TLV● Modules were the main feature for 4.0● The key differences this time:

▪ Isolated API ▪ ABI Backwards Compatibility

And So The Journey Began...

● Salvatore started work on the flight back● We had a working module 2 weeks later● And it was already usable!

● It could add commands

● It could call Redis like a Lua script

It Was Awesome!

First iteration of the API

● Secret-ish for 2 months ● Redis command handling● High level, simple, Lua like API● Direct access to string keys● Replication control● Low level, faster API

Initial Experiments

● Some “missing commands”:● HGETSET

● PREPEND

● PDEL

● Etc

● Image Manipulation● PAM Auth● And...

The Obligatory

Starting to Get Serious

● That was very cool, but we wanted more:● Probabilistic Data Structures

● Bloom Filter

● Streaming Top-k

● etc.

● Full-Text Search

RediSearch V1

▪ From-Scratch search index over redis▪ Uses String DMA for holding compressed index data▪ This allows efficient compression of data▪ Includes stemming, exact phrase match, etc.▪ Up to X5 faster than Elastic / Solr

Search in Action> FT.CREATE products SCHEMA title TEXT price NUMERICOK> FT.ADD products id1 1.0 FIELDS title "LG 42'' LCD TV" price 500OK> FT.ADD products id2 1.0 FIELDS title "Toshiba 40'' LCD TV" price 400OK> FT.SEARCH products "lcd tv" FILTER price 450 +inf1) (integer) 12) "id1"3) 1) "title" 2) "LG 42'' LCD TV" 3) "price" 4) "500"

Modules API v2 - Custom Types, Blocking

● Bye Bye Strings, hello data types!● Add any data type to redis at native speeds

● Much greater degree of freedom

● Much more testable

● Block the client, work in the background● Run heavy computation or network in the background

● Let redis handle other calls

● Return when ready!

Working Asynchronously

● In normal operation, each handler blocks Redis● Blocking keeps client waiting but not Redis

Handle Request #1

Background Work

Send ReplyRedis Main Thread

Module Thread

Handle Request #2

This allowed some interesting stuff

● AutoComplete for search● Machine Learning models● Neural-Redis training in the background● Secondary Indexing using efficient structures

Redis-ML

● Database for ML models● Train wherever you want● Store the result in redis● Query models in real time● Scale your models beyond a single machine

Redis Module

Tree Ensembles

Linear Regression

Logistic Regression

Matrix + Vector Operations

More to come...

The Redis-ML Module

Neural Redis

● Developed by Salvatore● Training is done inside Redis● Online continuous training process● Builds Fully Connected NNs

The Trouble With JSON

● Right now people save JSON objects as strings● Changing a property means read, modify, write● Accessing a single property means loading the entire object● This S.U.C.K.S!

● Slow

● Un-atomic

● CPU Wasteful

● Karmically wrong

● Sucks Ass

ReJSON to the rescue

● Store objects as actual object trees● Update elements atomically● Retrieve part of an object

> JSON.SET doc . '{ "foo": "bar", "baz": [1,2] }'> JSON.SET doc .goo true> JSON.NUMINCRBY doc .baz[0] 9> JSON.GET doc .baz"[10,2]"> JSON.GET doc ."{\"foo\":\"bar\",\"baz\":[10,2],\"goo\":true}"

Redis-Secondary

● Schema aware secondary indexes● Integers, floats, booleans, strings, etc

● Either a raw data type or automatic index● Treat HASH keys as objects or table rows● SQL-like WHERE clauses as proxies to redis commands

Secondary Indexing In Action> IDX.CREATE users_name_age TYPE HASH SCHEMA name STRING age INT32

> IDX.INTO users_name_age HMSET user1 name "alice" age 30

> IDX.INTO users_name_age HMSET user2 name "bob" age 25

> IDX.FROM users_name_age WHERE "name LIKE 'b%'" HGET $ name

1) user1

2) "bob"

> IDX.FROM users_name_age WHERE "name >= 'alice' AND age < 31" HGETALL $

1) user1

...

The Start of a Module Ecosystem

● Modules were launched May 2016 at RedisConf● Initial modules started to appear, mostly toy projects● We launched Modules-Hub

Redis Modules Hack 2016

● A global, distributed hackathon● With on-site events in TLV and SF● Some very interesting projects● We gave them cash!

First Place - Redis-Cell

● Author: Brandur Leach● Advanced rate limiting algorithm● Actually written in Rust, not C

CL.THROTTLE user123 15 30 60 1

▲ ▲ ▲ ▲ ▲

| | | | └───── apply 1 token (default if omitted)

| | └──┴─────── 30 tokens / 60 seconds

| └───────────── 15 max_burst

└─────────────────── key "user123"

Notable Mention: Redis-Graph

● Author: Roi Lipman● A fast graph database implemented in Redis● Subset of Cipher (Neo4J QL) implementation● Uses Redis HASH keys as nodes

GRAPH.QUERY presidents

"MATCH (president)-[born]->(state:Hawaii)

RETURN president.name, president.age"

Other Cool Modules

● Time-series aggregation engine● Pyrecks - Python function execution● Statsd reporting of Redis metrics● Zstd value compression● Time decaying popularity ranking● Shared memory support● And many more!

So What’s Next?

Coming In Redise

● To be released soon in Redis Labs’ enterprise cluster● Pre-bundled modules:

● RediSearch

● ReJSON

● Secondary

● Redis-ML

● Use your own modules at their own risk

Stable But Not Done...

● Redis does not yet officially support modules!● 4.0 is due in the coming weeks● The API can be considered 99% stable● But it’s far from complete

TODO: Improved low-level API

● The current API is missing some basic Redis stuff:● Keyspace iteration

● HGETALL

● Geo commands

● etc

TODO: Thread Safe Operations

● Beyond the blocking API● Allow modules to operate on Redis from a bg-thread● Allow modules to schedule periodic timers● Per-key read/write locks - allow real multi threading!

TODO: Hooks

● Get notified when a command is executed● Get notified when keys you are interested change● Allow overriding commands● This opens the door to:

● New data structures for existing commands

● REALLY automatic indexing

● Robust encryption and security models

● etc

TODO: Clustered Modules API

● Allow cluster shards to pass messages automatically● Allow map-reduce operations for multi-shards● Abstract cluster topology away● Allow things like search to REALLY SCALE

● The goal: Disque will become a module!

So You Want to Write a Module?

● It’s really easy, we promise● No special linking required, just one header file● The documentation is excellent● Redis includes a lot of simple examples● Use RedisModulesSDK

Thank You!