+

Redis {beyond the basics}

Presented by Steph ☺

+ Outline

■  Revisiting Redis

■  Keeping data safe ■  Persistence ■  Replication ■  Replacing Failed Master ■  Transaction

■  Reducing memory use ■  Ziplist ■  Intlist ■  Sharding

■  Scaling

+ Redis

■  In-memory remote database

■  Advanced key-value store

■  Data-structure server

■  Offers ■  High Performance ■  Replication ■  Unique data model

+Snapshotting

■  Data is taken as it exists and is written to the disk

■  Point-in-time copy of in-memory data

■  Backup & transfer to other server

■  Written to file in “dbfilename” stored in “dir”

■  Until next snapshot is taken, last snapshot can be lost if redis crashes

{Persistence}

+Snapshotting

Initiating Snapshots ■  bgsave ■  SAVE ■  Save lines save 60 10000 ■  SHUTDOWN / TERM ■  SYNC

{Persistence}

+ Snapshotting

■  How often to perform an automatic snapshot

■  Accept writes on failure

■  Snapshot compression

■  What to name the snapshot on the disk

+Append Only File (AOF)

■  Copies incoming write commands as it happens

■  Records data changes at the end of the backup file

■  Data set could be recovered with replaying AOF

■  “append only yes”

■  “appendfsyc always”

■  Limited by disk performance

{Persistence}

+ Append Only File (AOF)

■  Option to use AOF

■  Occurrence of sync writes to disk

■  Option to sync during AOF compaction

■  Occurrence of AOF compaction

+ Replication

■  Method where other servers receive an updated copy of the data as its being written

■  Replicas can service read queries

■  Single master database sends writes out to multiple slave databases

■  Set operations can take seconds to finish

+ Replication

■ Configuring for replication ■  On master, ensure that the path and filename are

writable by redis process ■  Enable slaving : slaveof host port ■  In a running system, redis can be stopped slaving

or connect to a different master ■  New / Transfer connection: slaveof host port ■  Stop data update: SLAVEOF no one

+Replication {SLAVE to MASTER Connection}

+Replication {Redis Master-Slave Replica Tree}

+Replacing Failed Master

■  What will we do in case of system failure?

■  Scenario ■  Machine A – Redis Master, Machine B – Redis Slave ■  Machine A loses network connectivity ■  Machine C has Redis, but no copy of data

■  Solution A ■  Make a fresh snapshot using Machine B using SAVE ■  Copy snapshot to Machine C ■  Start Redis on Machine C ■  Tell Machine B to be a slave of Machine C

{Scenario and Solution}

+Replacing Failed Master {Sequence of Commands}

+Replacing Failed Master

■  What will we do in case of system failure?

■  Solution B ■  Use Machine B (Slave) as Master ■  Create a new Slave (maybe Machine C) ■  Update client configuration to read/write to proper servers ■  (optional) update server configuration if restart is needed

{Scenario and Solution}

+Transactions

■  Begin transaction with MULTI

■  Execute commands with EXEC

■  Delayed execution with multi/exec can improve performance ■  Holds off sending commands until all of them are known ■  When all of the commands are known, MULTI is sent by

client

+Transactions

■  Pipelining ■  Send multiple commands at once ■  Wait for all replies ■  Reduces number of network roundtrips that the client waits

for

+Reducing Memory Use

■  Method of reducing memory use

■  Ziplist – compact storage and unstructured representation of LISTs HASHes and ZSETs

■  Intset – compact representation of SET

■  As structures grow beyond limits, they are converted back to their original data structure type

■  Manipulating compact versions can become slow as they grow

{Short Structures}

+ Ziplist

■  Basic configuration for the 3 data types are similar

■  *-max-ziplist-value – max number of items to be encoded as ziplist

■  If limits are exceeded, redis will convert the list/hash/zset into non-ziplist structure

+ZIPLIST - LIST

+ Intset

+Sharded Structures

■  Sharding – takes data, partitions it to smaller pieces and sends data to different locations depending on which partition the data is assigned to

■  Sharding LISTs – uses LUA scripting

■  Sharding ZSETs – zset operations on shards violate how quickly zsets perform, sharding is not useful on zsets

+Sharded Structures

■  Sharding HASHes ■  Method of partitioning data must be chosen ■  Hash’s keys can be used as source of info for sharding ■  To partition keys: ■  Calculate hash function on the key ■  Calculate number of shards needed depending on number

of keys we want to fit in one shard and the total number of keys

■  Resulting number of shards along with hash value will be used to find out which shard we’ll use

+Scaling

■  In using small structures, make sure max ziplist is not too large

■  Use structures that offer good performance for the types of queries we want to perform

■  Compress large data sent to redis for caching to reduce network reads and writes

■  Use pipelining and connection pooling

{read capacity}

+Scaling

■  Increase total read throughput using read only slave servers ■  Always remember to WRITE TO THE MASTER ■  Writing on SLAVE will cause an error

■  Redis Sentinel ■  Mode where redis server binary doesn’t act like the typical

one ■  Watches behavior and health of master(s) and slave(s) ■  Intended to offer automated failover

{read capacity}

+Scaling

■  Make sure to check all methods to reduce read data volume

■  Make sure larger pieces of unrelated functionality are moved to different servers

■  Aggregate writes in local memory before writing to redis

■  Consider using locks or LUA when limitations such as watch/multi/exec are encountered

■  When using AOF, keep in mind that the disk needs to keep up with the volume we’re writing

{memory capacity}

+Scaling

■  Presharding for growth ■  Run multiple redis servers on your machine (listen on diff.

ports) ■  Use multiple redis database on your database server

{write capability}

+Scaling

■  Scenario : machines have enough memory to hold index, but we need to execute more queries that server can handle

■  Use : SUNIONSTORE, SINTERSTORE, SDIFFSTORE, ZINTERSTORE, and/or ZUNIONSTORE

■  Since we “read” from slave, set : slave-read-only no

{complex queries}

+ Reference

■  Carlson, Josiah. (2013) Redis in Action. Shelter Island, NY: Manning Publications

+

Thank You ☺