REDIS Labcamp

Angelo Simone ScottoCluster Reply

2

• Developed by Salvatore Sanfilippo for his startup Merzia S.r.l.• Aimed as a fast replacement of MySQL for web analytics (LLOOGG)• Released as open source

in 2009.• Sponsored by VMWare

since 2010.• Sponsored by Pivotal

since 2013.• Currently part of backend

of

An italian story

• Twitter• Github• Pinterest• StackOverflow• Flickr• …

3

What is REDIS (REmote DIctionary Service)

Redis is a DSL (Domain Specific Language) that manipulates abstract data types and implemented as a TCP daemon. Commands manipulate a key space where keys are binary-safe strings and values are different kinds of abstract data types.

REDIS Manifesto

4

Available Abstract Data Types

Key-Value Store:• Keys

Printable ASCII• Values

Primitives Strings

Containers (of strings) Hashes Lists Sets Sorted Sets

Other Functionalities Pub/Sub HyperLogLog Bitmaps

5

Where to find Redis

• Obviosuly at redis website : http://redis.io• Redis written in ANSI C and POSIX compliant:

• Linux (supported and recommended)• Mac OS X (supported)• *BSD (supported)• Solaris (support defined «Best effort»)

• Windows is not directly supported by REDIS Project but• Microsoft mantains and support a version at http

://msopentech.com/opentech-projects/redis/• Used by Microsoft in Azure (Azure Redis Cache Service) therefore stable

and usable also on windows.

6

How to Install Redis

• On Windows, very simple:• Download binaries from https://

github.com/MSOpenTech/redis/tree/2.8/bin/release, unzip and run.• On Linux or MacOSX:

• Your preferred Package Manager (apt-get, YUM …) should have it but it’s usually not recommended because often lagging behind current version.

• Recommended way is to recompile binaries directly: http://redis.io/download

7

Strings

Redis Key

Value: Redis String

• Max Value Size: 512 MB• Binary Strings

Commands:• GET, SET• INCR, DECR, INCRBY, DECRBY• GETBIT, SETBIT, BITCOUNT, BITPOS, BITOP• APPEND, GETRANGE, SETRANGE• MGET, MSET

8

Hashes

• Max Value Size: 512 MB• Binary Strings

Redis Key

Value: Redis Hash

Field 1 Value 1

Field 2 Value 2

Field 3 Value 3

Commands:• HGET, HSET• HINCRBY, HINCRBYFLOAT• HLEN, HGETALL, HVALS, HKEYS• HMGET, HMSET

9

Lists

• LPUSH, LPOP (Left, first)• RPUSH, RPOP (Right, last)• Queues are simply lists

when using LPUSH & RPOP• Stack are simply lists when

using LPUSH & LPOP• Index (list) operations are

obviously available (LINDEX)

Redis Key

Value: Redis List

HE

AD

Valu

e 1

Valu

e n

TAIL

Commands:• LPUSH, LPOP• RPUSH, RPOP, RPOPLPUSH• BLPOP, BRPOP, BRPOPLPUSH• LINDEX, LRANGE, LREM

10

Sets• Unsorted set• Unique Values (duplicates

are non inserted) Redis Key

Value: Redis Set

Value 2Value 1

Value 4

Value 5 Value 3

Commands:• SADD, SPOP, SREM• SCARD, SISMEMBER• SUNION, SDIFF, SINTER• SUNIONSTORE, SDIFFSTORE, SINTERSTORE

11

Sorted Sets• Score is a float (double precision

floating number).• Unique Values (duplicates are non

inserted) • Sorted sets are sorted by their score in

ascending way

Redis Key

Value: Redis Sorted Set

Valu

e 1

Sco

re 1

00

Valu

e 2

Sco

re 2

00

Valu

e 3

Sco

re 2

00

Valu

e 4

Sco

re 2

50

Commands:• ZADD, ZREM• ZCARD, ZRANGE, ZREVRANGE• ZSCORE, ZCOUNT, ZINCRBY, ZRANGEBYSCORE• ZLEXCOUNT, ZRANGEBYLEX, ZREVRANGEBYLEX, ZREMRANGEBYLEX• ZUNIONSTORE, ZINTERSTORE

12

Other Functionalities

Bitmaps: Not really a type, just a set of operators on strings.

Publish / Subscribe: Not a type, it implements a fire&forget publish/subscribe engine. PUBLISH, SUBSCRIBE, UNSUBSCRIBE PSUBSCRIBE, PUNSUBSCRIBE

HyperLogLog: Value contains an HyperLogLog algorithm implementation. REDIS implementation uses 12K per key with a std error of 0.81% to

count 2^64 items. PFADD, PFCOUNT, PFMERGE

13

REDIS Architecture

Key points:• Extremely Fast.

• i.e. Data is always in RAM.

The Redis data set, composed of defined key-value pairs, is

primarily stored in the computer's memory. The amount of

memory in all kinds of computers, including entry-level

servers, is increasing significantly each year. Memory is fast,

and allows Redis to have very predictable performance. REDIS Manifesto

14

REDIS Architecture

Key points:• Extremely Fast.

• i.e. Data is always in RAM.• Extremely Simple.

• i.e. REDIS is single-threaded.

We're against complexity. We believe designing systems is a fight

against complexity. We'll accept to fight the complexity when it's

worthwhile but we'll try hard to recognize when a small feature is not

worth 1000s of lines of code. Most of the time the best way to fight

complexity is by not creating it at all.

REDIS Manifesto

15

REDIS Architecture

Key points:• Extremely Fast.

• i.e. Data is always in RAM.• Extremely Simple.

• i.e. REDIS is single-threaded.• Extremely Light.

16

REDIS Architecture

Key points:• Extremely Fast.

• i.e. Data is always in RAM.• Extremely Simple.

• i.e. REDIS is single-threaded.• Extremely Light.• Extremely Robust.

17

REDIS Architecture

Security:«Redis is not optimized for maximum security but for maximum performance and simplicity»• AUTH with password.• BIND to specific network interface.• No encryption (use SSL proxy)• No ACS (each connection is admin)• Possible to RENAME sensible

commands…

18

REDIS Architecture

Persistence:• None.• RDB (Redis DB) single-file dump.• AOF (Append Only File), a log file.

• No fsync (fast, not safe).• Fsync each second.• Fsync each query (slow but safe).

• Both

Memory Usage:• Data is always in memory, you cannot have more data than RAM in REDIS.

• Virtual Memory deprecated.• Disk Store abandoned.

• Several policies to manage overflow:• noeviction: no action, just error.• allkeys-lru: remove older data• allkeys-random: remove rnd data

• volatile-lru: remove expirable• volatile-random: remove expirable• volatile-ttl: remove expirable

19

REDIS Architecture

Transactions:• Not an ACID Transactions.• All commands are serialized and

executed sequentially (MULTI).• Either all commands or no commands

are processed (EXEC – DISCARD).• Keys must be explicitely specified in

Redis transactions (WATCH)• Less useful since lua scripting appeared.

• Commands Available:• WATCH / UNWATCH• MULTI• EXEC / DISCARD

20

REDIS Architecture

Lua Scripting:• Lua is a small, fast, simple embeddable

scripting language.• EVAL command takes script, keys and

arguments:

• Redis calls are done through redis.call or redis.pcall functions:

• Scripts are executed sequentially on the server in atomic way

21

A Swiss Army Knife for Programmers

Redis gives you a small, light, efficient remote datastore with persistence, high-availability, scalability and publish/subscribe functionalities.

Use is limited just by your imagination:• Shared Configuration.• Distributed Lock.• Caching.• Work Queues.• Load Balancers.• Service Discovery.• …

Workgroup and Contest

23

A super-simple blog platform

• Use the frontend and the language you prefer:• Frontend:

• Console Application• Web Application• Form Application

• Language:• List of supported clients: http://redis.io/clients

• 5 Levels of features (10 points):1. User Management (2 points)2. Post Management (2 points)3. Tags / Categories (2 points)4. Votes (2 points)5. New posts notification (2 points)

24

User Management

• System should be able to register new users, login existing users and show registered users.

• Each user should have a unique id (number), a username, a password

• Conventions:• A common convention is to use hierarchical names for keys

separated by ‘:’ such as blog:users:1 or blog:users:1:posts

• Tip:• Use String and INCR to manage unique ids.• Use Hashes to manage user objects• Use Set to guarantee uniqueness of usernames

25

Post Management

• A logged in user should be able to add a blog post.• Each post will have a unique id (number), a content, a timestamp and a

subject.• Given a user the system should return the list of items posted by the

user.

• Tip:• Use String and INCR to manage unique ids.• Use Hashes to manage post objects.• Use Lists to relate posts with owners

26

Tags / Categories

• Modify the previous post structure to keep also a set of tag for each post.

• Given a tag the system should return all posts tagged with it.• Given multiple tags the system should return all posts, if any, tagged

with them.

• Tip:• Add Tags field to post Hash.• Use Sets to manage categories, and SINTER to search for multiple

tags.

27

Votes

• A logged in user should be able to vote posts.• Each user must be able to vote just once for a post.• System should be able to show posts ordered by votes.

• Tip:• Use Sorted Sets to manage post ranks.• Use Sets to enforce one vote per user per post.

28

Push Notifications

• A user could express interest for a category (tag) or another user activity.

• System should notify user if an article is posted in a category he expressed interest into.

• System should notify user if an article is posted by a user he expressed interest into.

• Tip:• Use PubSub engine to manage notifications.• Use one channel per category and one channel per user.

Thanks

Angelo Simone Scottoa.scotto@reply.eu