55
Introducing Redis Nuno Caneco http://netpont o.org 64ª Reunião Presencial - 24/OUT/2016
Introducing RedisNuno Caneco
http://netponto.org64ª Reunião Presencial - 24/OUT/2016
Agenda
• Introducing Redis
• Data Structures
•Pipelining and "transactions"
•Publish / Subscriber
• Scripting
What is Redis?Redis is an open source (BSD licensed), in-memory data structure store, used as database, cache and message broker. It supports data structures such as strings, hashes, lists, sets,sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.
-- www.redis.io
What can it be used for?
• State Manager
• Cache Provider
– Application Data
– Sessions
– …
• Queuing
• Publisher / Subcriber storage point
It’s just a big hash mapKey Value
"pageSize" (String) 10
"onlineUsers" (Set) ["user:10", "user:24", "user:89"]
"user:10" (Hash) [{"name": "John Doe"}, {"login": "jdoe"}]
...
• Insert: O(1)• Get by key: O(1)• Keys lookups: O(n) n=keys
Redis Keys
• Redis Keys are Strings– All Strings are binary safe
• Some recommendations:– Avoid long keys (> 1024 bytes)– Avoid very short keys (u100 vs user:100)– Stick to a convention (e.g. user:100)
String• Simplest basic type
• Can be interpreted as numerics
– Allow add/subtract operations
Documentation
• GET, SET• SETNX• SETEX• MSET, MGET• EXISTS• DEL
For numerics:• INCR, INCRBY• DECR, DECRBY
String: Set and get
# set USD to EUR exchange rate > SET exchangeRate:USD:EUR 0.9107 OK
# Get USD to EUR exchange rate> GET exchangeRate:USD:EUR "0.9107"
# Set multiple values at once> MSET exchangeRate:USD:EUR 0.9107 exchangeRate:EUR:USD 1,0981OK
> MGET exchangeRate:USD:EUR exchangeRate:EUR:USD1) "0.9107"2) "1,0981"
String: If not exists
> SETNX availableSeats 10(integer) 1> GET availableSeats"10"> INCRBY availableSeats -1(integer) 9> INCRBY availableSeats -1(integer) 8> GET availableSeats"7"> SETNX availableSeats 10 #Won't set the key(integer) 0> GET availableSeats"7"
String: Use it as a counter
> INCR visitsCount(integer) 1
> GET visitsCount"1"
> INCRBY visitsCount 5(integer) 6
# It can also decrement a counter> INCRBY visitsCount -2(integer) 4
List
•Collection of String elements •Sorted according to the order of insertion
A B C DHEAD TAIL
List
List• Commands are prefixed with "L"• Implemented as Linked List
– Op on Head or Tail: O(1)– Op on Middle: O(n)– Lookup: O(n)
• Enables implementation of Queue
Documentation
> LSET> LINDEX> LLEN> LINSERT> LREM> LTRIM> LRANGE> LPUSH/RPUSH> LPOP/RPOP> LRANGE> RPOPLPUSH
List: Circular List
Use RPOPLPUSH with same source and destination
> RPUSH list "a" "b" "c"(integer) 3> LRANGE list 0 -11) "a"2) "b"3) "c"> RPOPLPUSH list list"c"> LRANGE list 0 -11) "c"2) "a"3) "b"
RPOPLPUSH returns the element that is being exchanged between the lists
List: Reliable Queue
Queue
Processing queue
Producer Consumer
LPUSH RPOPLPUSHM5 M4 M3 M2 M1
LREM
Processingmessage
NOTE: The consumer can call BRPOPLPUSH if it wishes to be blocked if the queue is empty
Set• Commands are prefixed
with "S"
Documentation
> SADD> SCARD> SISMEMBER> SMEMBERS> SRANDMEMBER> SREM> SSCAN> SPOP
> SDIFF, SDIFFSTORE> SMOVE> SINTER, SINTERSTORE> SUNION, SUNIONSTORE
Set: Example> SADD prog-lang:alice C Java C# Ruby(integer) 4
# Does Alice knows how to program in C?> SISMEMBER prog-lang:alice C(integer) 1 # Yes
# Does Alice knows how to program in Python?> SISMEMBER prog-lang:alice Python(integer) 0 # No
# So, which programming languages does Alice Know?> SMEMBERS prog-lang:alice1) "C#"2) "C"3) "Ruby"4) "Java"
Set: Multiple Sets Example> SADD prog-lang:bob C Python Ruby(integer) 3
# Which languages both Bob and Alice know about? > SINTER prog-lang:alice prog-lang:bob1) "C"2) "Ruby"
# Which languages could both Bob and Alice work on, if they teamed up?> SUNION prog-lang:alice prog-lang:bob1) "Java"2) "Ruby"3) "Python"4) "C#"5) "C"
# OK, so let's create a team that combines both their skills> SUNIONSTORE prog-lang:team:1 prog-lang:alice prog-lang:bob(integer) 5
Set: Random pick / pop# Add a deck of cards> SADD deck C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 CJ CQ CK D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 DJ DQ DK H1 H2 H3 H4 H5 H6 H7 H8 H9 H10 HJ HQ HK S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 SJ SQ SK(integer) 52
# Pick a random card, without taking it out the deck> SRANDMEMBER deck"H7"
# Create a game with the default deck> SUNIONSTORE game:1:deck deck(integer) 52
# Pick a card from the game deck> SPOP game:1:deck"D9" # A nine of Diamonds
# Now, the game deck has only 51 cards> SCARD game:1:deck(integer) 51
Sorted Set
Collection of Strings• Unique, non-repeating • Sorted by Score (float)
Set
E13
E26
E45
E39
Sorted Set• Commands are prefixed
with "Z"
Documentation
> ZADD> ZRANGE (BYLEX, BYSCORE)> ZREM> ZRANK> ZREMRANGEBYLEX, (BYRANK, BYSCORE)> ZCOUNT> ZREVRANGE (BYLEX, BYSCORE)> ZREVRANK> ZSCAN> ZSCORE> ZINCRBY
> ZINTERSTORE> ZUNIONSTORE
Sorted Set: Using Rank
Set
E13
E26
E45
E39
> ZSCORE set1 E1"3"> ZSCORE set1 E4"5"
> ZRANK set1 E4(integer) 1> ZRANK set1 E1(integer) 0
> ZREVRANK set1 E4(integer) 2> ZREVRANK set1 E1(integer) 3
(*) Score starts at zero
Sorted Set: Example> ZADD prog-lang-rank:alice 40 C 60 Java 80 C# 30 Ruby
# What is Alice's rank of C#> ZREVRANK prog-lang-rank:alice C#(integer) 0 # C# is what she knows best
# Top 2 languages for Alice> ZREVRANGE prog-lang-rank:alice 0 11) "C#"2) "Java"
# What are the programming languages for which Alice scores at least 50?> ZREVRANGEBYSCORE prog-lang-rank:alice 100 501) "C#"2) "Ruby"3) "C"
Sorted Set: Much wow (!)> ZADD prog-lang-rank:bob 30 C 50 Python 50 Ruby
# If we created a team with Alice and Bob, what would be the MAX score for each language?> ZUNIONSTORE prog-lang-rank:team:1 2 prog-lang-rank:alice prog-lang-rank:bob AGGREGATE MAX(integer) 5 # Their skills combined can work on projects using 5 different languages
> ZREVRANGE prog-lang-rank:team:1 0 -1 WITHSCORES 1) "C#" 2) "80" 3) "Java" 4) "60" 5) "Ruby" 6) "50" 7) "Python" 8) "50" 9) "C"10) "40"
Hash
Collection of field-value pairs:•Each field is unique•Suitable to represent objects
Field Value
key1 value1
key2 value2
Hash• Commands are prefixed
with "H"
• HINCRBY allows to atomically increment or decrement a value
Documentation
> HSET, HSETNX, HMSET> HGET, HGETALL> HKEYS> HDEL> HEXISTS> HLEN> HVALS
> HINCRBY, HINCREBYFLOAT
Hash: Using it to store objects
> HMSET lang:pt-PT locale "pt-PT" dateFormat "YYYY-MM-dd HH:mm:ss" currencyFormat "#.## EUR"> HMSET lang:en-US locale "en-US" dateFormat "MM-YYYY-dd hh:mm:ss tt" currencyFormat "#.## USD"
> HGET lang:pt-PT dateFormat"YYYY-MM-dd HH:mm:ss"
> HGETALL lang:pt-PT1) "locale"2) "pt-PT"3) "dateFormat"4) "YYYY-MM-dd HH:mm:ss"5) "currencyFormat"6) "#.## EUR"
Hash: Incrementing values
> HMSET comment:1001 text "I just love Redis" votes 0
> HINCRBY comment:1001 votes 1
> HGETALL comment:10011) "text"2) "I just love Redis"3) "votes"4) "1"
HINCRBY performs atomic increments or decrements on an Hash Value
BitmapSet of bitwise instructions implemented over (binary safe) String
0 1 0 1 1 0 0 0 1
> SETBIT> GETBIT
> BITOP (AND, OR, XOR and NOT)
> BITCOUNT
> BITPOS
> BITFIELD
Expiring entries
Entries can be configured to expire automatically
# Cache USD to EUR exchange rate for 1 hour (inline)> SET exchangeRate:USD:EUR 0.9107 EX 3600OK
# Cache an hash/object for 10 minutes> HMSET user:100 name "John Doe" username jdoe OK> EXPIRE user:100 600(integer) 1
It DOES NOT ensure that all commands are run as a single atomic set of commands
Pipelining
Most APIs allow to perform command pipeline.
Optimizes network usage, since all commands in a batch or pipeline are sent all at once.
Pipeliningvar batch = redisDb.CreateBatch();
// Commands will be kept pending on the client sidevar counter1Value = batch.StringIncrementAsync("counter:1", 1);var counter2Value = batch.StringIncrementAsync("counter:2", 10);batch.Execute(); // Instructs the batch that all commands are ready
// When accessing the first item of the command, the entire batch is executedConsole.WriteLine("counter1 = {0}", counter1Value.Result);Console.WriteLine("counter2 = {0}", counter2Value.Result);
"INCR" "counter:1""INCRBY" "counter:2" "10"
MULTI
Performs a set of commands as a single atomic operation
MULTIOKINCR totalVisitsQUEUEDHINCRBY customer:10 totalvisits 1QUEUEDEXEC # Both commands will be executed atomically at this point1) (integer) 2 # The results of the commands are outputed from EXEC2) (integer) 1
Publish/SubscribeAllows implementation of Publisher / Subscriber pattern.
• Multiple publishers• Multiple subscribers
Documentation
➢ PSUBSCRIBE
➢ PUBLISH
➢ PUBSUB
➢ PUNSUBSCRIBE
➢ SUBSCRIBE
➢ UNSUBSCRIBE
Publish/Subscribe
client1> SUBSCRIBE somefeedReading messages... (press Ctrl-C to quit)1) "subscribe"2) "somefeed"3) (integer) 1
1) "message"2) "somefeed"3) "Hi there!!"
client2> PUBLISH somefeed "Hi there!"(integer) 1
Scripting
Redis allows loading and executing LUA scripts
Scripts are executed atomically.
> EVAL> EVALSHA
> SCRIPT EXISTS> SCRIPT FLUSH> SCRIPT KILL> SCRIPT LOAD
Scripting
# Execute an inline script> eval "return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}" 2 key1 key2 first second1) "key1"2) "key2"3) "first"4) "second"
# Load a script and execute it later> set foo barOK> eval "return redis.call('get','foo')" 0"bar"> evalsha 6b1bf486c81ceb7edf3c093f4c48582e38c0e791 0"bar"
Commands performanceO(1) Not O(1)
General Purpose
DELEXISTSEXPIRE / PEXPIRE / PEXPIREATMOVEPTTLRENAME / RENAMENX
KEYSSCAN
String GETSET*GETSETINCRBY*STRLEN
MGETMSET*GETRANGE
Sets SADDSREMSPOPSCARDSISMEMBERSMOVESRANDMEMBER
SDIFFSDIFFSTORESINTERSINTERSTORESSCANSUNIONSUNIONSTORE
Commands performanceO(1) Not O(1)
Sorted Sets ZADDZREMZCARDSISMEMBERSMOVESRANDMEMBER
ZCOUNTZINCRBYZRANGEZRANK (O(log(n))ZINTERSTOREZSCANZUNIONSTORE
Hash Sets HGETHMGETHSET / HSETNXHMSETHINCRBYHSTRLENHLEN
HGETALL (O(size of hash))
HKEYS (O(size of hash))
HKEYSHSCAN
Tips and pitfalls
DO NOT:• Call KEYS or SCAN
– You can keep track of keys on a SET• Call MONITOR
DO:• Use the NX variants instead of calling EXISTS
than SET to initialize keys
ReferencesRedis Website
– http://www.redis.io
Redis Commands Documentation– http://redis.io/commands
Redis Documentation– http://redis.io/documentation
Statistics LUA Script– https://gist.github.com/thomasdarimont/60eefb3530f48de3d650
64ª Reunião Presencial @ LISBOADateTime.Parse(”24-09-2016", new CultureInfo("pt-PT"));
http://netponto.org
hashtag #netponto
Patrocinadores “GOLD”
Twitter: @PTMicrosoft http://www.microsoft.com/portugal
Patrocinadores “Silver”
Patrocinadores “Bronze”
http://bit.ly/netponto-aval-63
* Para quem não puder preencher durante a reunião, iremos enviar um email com o link à tarde
Próximas reuniões presenciais
24/09/2016 – Lisboa22/10/2016 – Lisboa19/11/2016 – Lisboa17/12/2016 – LisboaReserva estes dias na agenda! :)
