Introducing redis

transcript

Introducing RedisNuno Caneco

http://netponto.org64ª Reunião Presencial - 24/OUT/2016

Agenda

• Introducing Redis

• Data Structures

•Pipelining and "transactions"

•Publish / Subscriber

• Scripting

Introducing Redis

What is Redis?Redis is an open source (BSD licensed), in-memory data structure store, used as database, cache and message broker. It supports data structures such as strings, hashes, lists, sets,sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

-- www.redis.io

What can it be used for?

• State Manager

• Cache Provider

– Application Data

– Sessions

– …

• Queuing

• Publisher / Subcriber storage point

It’s just a big hash mapKey Value

"pageSize" (String) 10

"onlineUsers" (Set) ["user:10", "user:24", "user:89"]

"user:10" (Hash) [{"name": "John Doe"}, {"login": "jdoe"}]

• Insert: O(1)• Get by key: O(1)• Keys lookups: O(n) n=keys

Redis Keys

• Redis Keys are Strings– All Strings are binary safe

• Some recommendations:– Avoid long keys (> 1024 bytes)– Avoid very short keys (u100 vs user:100)– Stick to a convention (e.g. user:100)

Redis Data Types

String• Simplest basic type

• Can be interpreted as numerics

– Allow add/subtract operations

Documentation

• GET, SET• SETNX• SETEX• MSET, MGET• EXISTS• DEL

For numerics:• INCR, INCRBY• DECR, DECRBY

String: Set and get

# set USD to EUR exchange rate > SET exchangeRate:USD:EUR 0.9107 OK

# Get USD to EUR exchange rate> GET exchangeRate:USD:EUR "0.9107"

# Set multiple values at once> MSET exchangeRate:USD:EUR 0.9107 exchangeRate:EUR:USD 1,0981OK

> MGET exchangeRate:USD:EUR exchangeRate:EUR:USD1) "0.9107"2) "1,0981"

String: If not exists

> SETNX availableSeats 10(integer) 1> GET availableSeats"10"> INCRBY availableSeats -1(integer) 9> INCRBY availableSeats -1(integer) 8> GET availableSeats"7"> SETNX availableSeats 10 #Won't set the key(integer) 0> GET availableSeats"7"

String: Use it as a counter

> INCR visitsCount(integer) 1

> GET visitsCount"1"

> INCRBY visitsCount 5(integer) 6

# It can also decrement a counter> INCRBY visitsCount -2(integer) 4

•Collection of String elements •Sorted according to the order of insertion

A B C DHEAD TAIL

List• Commands are prefixed with "L"• Implemented as Linked List

– Op on Head or Tail: O(1)– Op on Middle: O(n)– Lookup: O(n)

• Enables implementation of Queue

Documentation

> LSET> LINDEX> LLEN> LINSERT> LREM> LTRIM> LRANGE> LPUSH/RPUSH> LPOP/RPOP> LRANGE> RPOPLPUSH

List: Circular List

Use RPOPLPUSH with same source and destination

> RPUSH list "a" "b" "c"(integer) 3> LRANGE list 0 -11) "a"2) "b"3) "c"> RPOPLPUSH list list"c"> LRANGE list 0 -11) "c"2) "a"3) "b"

RPOPLPUSH returns the element that is being exchanged between the lists

List: Reliable Queue

Processing queue

Producer Consumer

LPUSH RPOPLPUSHM5 M4 M3 M2 M1

Processingmessage

NOTE: The consumer can call BRPOPLPUSH if it wishes to be blocked if the queue is empty

Collection of String:• unique, non-repeating String• unsorted

Set• Commands are prefixed

with "S"

Documentation

> SADD> SCARD> SISMEMBER> SMEMBERS> SRANDMEMBER> SREM> SSCAN> SPOP

> SDIFF, SDIFFSTORE> SMOVE> SINTER, SINTERSTORE> SUNION, SUNIONSTORE

Set: Example> SADD prog-lang:alice C Java C# Ruby(integer) 4

# Does Alice knows how to program in C?> SISMEMBER prog-lang:alice C(integer) 1 # Yes

# Does Alice knows how to program in Python?> SISMEMBER prog-lang:alice Python(integer) 0 # No

# So, which programming languages does Alice Know?> SMEMBERS prog-lang:alice1) "C#"2) "C"3) "Ruby"4) "Java"

Set: Multiple Sets Example> SADD prog-lang:bob C Python Ruby(integer) 3

# Which languages both Bob and Alice know about? > SINTER prog-lang:alice prog-lang:bob1) "C"2) "Ruby"

# Which languages could both Bob and Alice work on, if they teamed up?> SUNION prog-lang:alice prog-lang:bob1) "Java"2) "Ruby"3) "Python"4) "C#"5) "C"

# OK, so let's create a team that combines both their skills> SUNIONSTORE prog-lang:team:1 prog-lang:alice prog-lang:bob(integer) 5

Set: Random pick / pop# Add a deck of cards> SADD deck C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 CJ CQ CK D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 DJ DQ DK H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 HJ HQ HK S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 SJ SQ SK(integer) 52

# Pick a random card, without taking it out the deck> SRANDMEMBER deck"H7"

# Create a game with the default deck> SUNIONSTORE game:1:deck deck(integer) 52

# Pick a card from the game deck> SPOP game:1:deck"D9" # A nine of Diamonds

# Now, the game deck has only 51 cards> SCARD game:1:deck(integer) 51

Sorted Set

Collection of Strings• Unique, non-repeating • Sorted by Score (float)

Sorted Set• Commands are prefixed

with "Z"

Documentation

> ZADD> ZRANGE (BYLEX, BYSCORE)> ZREM> ZRANK> ZREMRANGEBYLEX, (BYRANK, BYSCORE)> ZCOUNT> ZREVRANGE (BYLEX, BYSCORE)> ZREVRANK> ZSCAN> ZSCORE> ZINCRBY

> ZINTERSTORE> ZUNIONSTORE

Sorted Set: Using Rank

> ZSCORE set1 E1"3"> ZSCORE set1 E4"5"

> ZRANK set1 E4(integer) 1> ZRANK set1 E1(integer) 0

> ZREVRANK set1 E4(integer) 2> ZREVRANK set1 E1(integer) 3

(*) Score starts at zero

Sorted Set: Example> ZADD prog-lang-rank:alice 40 C 60 Java 80 C# 30 Ruby

# What is Alice's rank of C#> ZREVRANK prog-lang-rank:alice C#(integer) 0 # C# is what she knows best

# Top 2 languages for Alice> ZREVRANGE prog-lang-rank:alice 0 11) "C#"2) "Java"

# What are the programming languages for which Alice scores at least 50?> ZREVRANGEBYSCORE prog-lang-rank:alice 100 501) "C#"2) "Ruby"3) "C"

Sorted Set: Much wow (!)> ZADD prog-lang-rank:bob 30 C 50 Python 50 Ruby

# If we created a team with Alice and Bob, what would be the MAX score for each language?> ZUNIONSTORE prog-lang-rank:team:1 2 prog-lang-rank:alice prog-lang-rank:bob AGGREGATE MAX(integer) 5 # Their skills combined can work on projects using 5 different languages

> ZREVRANGE prog-lang-rank:team:1 0 -1 WITHSCORES 1) "C#" 2) "80" 3) "Java" 4) "60" 5) "Ruby" 6) "50" 7) "Python" 8) "50" 9) "C"10) "40"

Collection of field-value pairs:•Each field is unique•Suitable to represent objects

Field Value

key1 value1

key2 value2

Hash• Commands are prefixed

with "H"

• HINCRBY allows to atomically increment or decrement a value

Documentation

> HSET, HSETNX, HMSET> HGET, HGETALL> HKEYS> HDEL> HEXISTS> HLEN> HVALS

> HINCRBY, HINCREBYFLOAT

Hash: Using it to store objects

> HMSET lang:pt-PT locale "pt-PT" dateFormat "YYYY-MM-dd HH:mm:ss" currencyFormat "#.## EUR"> HMSET lang:en-US locale "en-US" dateFormat "MM-YYYY-dd hh:mm:ss tt" currencyFormat "#.## USD"

> HGET lang:pt-PT dateFormat"YYYY-MM-dd HH:mm:ss"

> HGETALL lang:pt-PT1) "locale"2) "pt-PT"3) "dateFormat"4) "YYYY-MM-dd HH:mm:ss"5) "currencyFormat"6) "#.## EUR"

Hash: Incrementing values

> HMSET comment:1001 text "I just love Redis" votes 0

> HINCRBY comment:1001 votes 1

> HGETALL comment:10011) "text"2) "I just love Redis"3) "votes"4) "1"

HINCRBY performs atomic increments or decrements on an Hash Value

BitmapSet of bitwise instructions implemented over (binary safe) String

0 1 0 1 1 0 0 0 1

> SETBIT> GETBIT

> BITOP (AND, OR, XOR and NOT)

> BITCOUNT

> BITPOS

> BITFIELD

Expiring entries

Entries can be configured to expire automatically

# Cache USD to EUR exchange rate for 1 hour (inline)> SET exchangeRate:USD:EUR 0.9107 EX 3600OK

# Cache an hash/object for 10 minutes> HMSET user:100 name "John Doe" username jdoe OK> EXPIRE user:100 600(integer) 1

Command batch and "transactions"

It DOES NOT ensure that all commands are run as a single atomic set of commands

Pipelining

Most APIs allow to perform command pipeline.

Optimizes network usage, since all commands in a batch or pipeline are sent all at once.

Pipeliningvar batch = redisDb.CreateBatch();

// Commands will be kept pending on the client sidevar counter1Value = batch.StringIncrementAsync("counter:1", 1);var counter2Value = batch.StringIncrementAsync("counter:2", 10);batch.Execute(); // Instructs the batch that all commands are ready

// When accessing the first item of the command, the entire batch is executedConsole.WriteLine("counter1 = {0}", counter1Value.Result);Console.WriteLine("counter2 = {0}", counter2Value.Result);

"INCR" "counter:1""INCRBY" "counter:2" "10"

Performs a set of commands as a single atomic operation

MULTIOKINCR totalVisitsQUEUEDHINCRBY customer:10 totalvisits 1QUEUEDEXEC # Both commands will be executed atomically at this point1) (integer) 2 # The results of the commands are outputed from EXEC2) (integer) 1

Publish/Subscriber

Publish/SubscribeAllows implementation of Publisher / Subscriber pattern.

• Multiple publishers• Multiple subscribers

Documentation

➢ PSUBSCRIBE

➢ PUBLISH

➢ PUBSUB

➢ PUNSUBSCRIBE

➢ SUBSCRIBE

➢ UNSUBSCRIBE

Publish/Subscribe

client1> SUBSCRIBE somefeedReading messages... (press Ctrl-C to quit)1) "subscribe"2) "somefeed"3) (integer) 1

1) "message"2) "somefeed"3) "Hi there!!"

client2> PUBLISH somefeed "Hi there!"(integer) 1

Scripting

Redis allows loading and executing LUA scripts

Scripts are executed atomically.

> EVAL> EVALSHA

> SCRIPT EXISTS> SCRIPT FLUSH> SCRIPT KILL> SCRIPT LOAD

Scripting

# Execute an inline script> eval "return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}" 2 key1 key2 first second1) "key1"2) "key2"3) "first"4) "second"

# Load a script and execute it later> set foo barOK> eval "return redis.call('get','foo')" 0"bar"> evalsha 6b1bf486c81ceb7edf3c093f4c48582e38c0e791 0"bar"

Performance Considerations

Commands performanceO(1) Not O(1)

General Purpose

DELEXISTSEXPIRE / PEXPIRE / PEXPIREATMOVEPTTLRENAME / RENAMENX

KEYSSCAN

String GETSET*GETSETINCRBY*STRLEN

MGETMSET*GETRANGE

Sets SADDSREMSPOPSCARDSISMEMBERSMOVESRANDMEMBER

SDIFFSDIFFSTORESINTERSINTERSTORESSCANSUNIONSUNIONSTORE

Commands performanceO(1) Not O(1)

Sorted Sets ZADDZREMZCARDSISMEMBERSMOVESRANDMEMBER

ZCOUNTZINCRBYZRANGEZRANK (O(log(n))ZINTERSTOREZSCANZUNIONSTORE

Hash Sets HGETHMGETHSET / HSETNXHMSETHINCRBYHSTRLENHLEN

HGETALL (O(size of hash))

HKEYS (O(size of hash))

HKEYSHSCAN

Tips and pitfalls

DO NOT:• Call KEYS or SCAN

– You can keep track of keys on a SET• Call MONITOR

DO:• Use the NX variants instead of calling EXISTS

than SET to initialize keys

Questions?

ReferencesRedis Website

– http://www.redis.io

Redis Commands Documentation– http://redis.io/commands

Redis Documentation– http://redis.io/documentation

Statistics LUA Script– https://gist.github.com/thomasdarimont/60eefb3530f48de3d650

64ª Reunião Presencial @ LISBOADateTime.Parse(”24-09-2016", new CultureInfo("pt-PT"));

http://netponto.org

hashtag #netponto

Patrocinadores “GOLD”

Twitter: @PTMicrosoft http://www.microsoft.com/portugal

Patrocinadores “Silver”

Patrocinadores “Bronze”

http://bit.ly/netponto-aval-63

* Para quem não puder preencher durante a reunião, iremos enviar um email com o link à tarde

Próximas reuniões presenciais

24/09/2016 – Lisboa22/10/2016 – Lisboa19/11/2016 – Lisboa17/12/2016 – LisboaReserva estes dias na agenda! :)

Obrigado!

Nuno Caneconuno.caneco@gmail.com http://twitter.com/ncanecohttps://pt.linkedin.com/in/nunocaneco

Introducing redis

Software