Date post: | 14-Jul-2015 |
Category: |
Technology |
Upload: | itamar-haber |
View: | 651 times |
Download: | 0 times |
Benchmarking RedisBy itself and versus other NoSQL databases
@ItamarHaber #RedisConf 2015
About Myself
A Redis Geek and Chief Developers Advocate at
Have you subscribed to the newsletter yet?
https://redislabs.com/redis-watch-archive
define:benchmark
"In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performanceof an object, normally by running a number of standard tests and trials against it."
https://en.wikipedia.org/wiki/Benchmark_(computing)
The Only Truism About Benchmarking
The chances that any benchmark's results will accurately reflect the performance of your specific use case are infinitesimal*
* single exception: benchmarks that you run on your own application <- please wait patiently for my guest speaker
So Why Benchmark Redis at all?
•Quantify Blazing Fast with some ballpark figures
•Regression testing/optimizations verifications/sanity
•Comparative benchmark(et)ing
• Fun & profit
Howto #1: redis-benchmark
• Included with Redis (build it with make)
•Built-in suite of tests and can also run arbitrary operations (the -t switch)
• The perfect tool to profile Redis (and moar) but...• Like Redis, it is single-threaded so generating load is
challenging• Does not report errors - if results are exceptionally high,
there's probably an error in the statement that you're checking
Howto #2: memtier_benchmark
https://github.com/RedisLabs/memtier_benchmark
• Open source traffic generator for Redis (and Memcached)
• We use it for baseline profiling, stress loading and benchmarking
• Notable differences (compared to redis-benchmark):
• Multi-threaded -> a single client can create considerable loads
• Executes n requests or runs for m seconds
• Generates Read:Write traffic patterns (e.g. 80:20)
• Does uniform and Gaussian distributions
• OTOH – only does GET/SET (or GETRANGE/SETRANGE)
Benchmarking With Multiple Clients – Ouch!
redisbenchmarkframework
• https://github.com/RedisLabs/redisbenchmarkframework
• Python & Fabric
• Executes redis-benchmark/memtier_benchmark/other and custom scripts from multiple servers
• Can spawn multiple threads per server
• Collects RPS results and outputs CSV
Benchmarking vs. Other NoSQLs, an i.e.g.
Aerospike vs. Redis benchmark• http://lynnlangit.com/2015/01/28/lessons-learned-benchmarking-nosql-on-
the-aws-cloud-aerospikedb-and-redis/• http://antirez.com/news/85• https://redislabs.com/blog/the-lessons-missing-from-benchmarking-nosql-
on-the-aws-cloud-aerospikedb-and-redis
To test Redis doing GET/SET is like to test a Ferrari checking how good it is at cleaning the mirror when it rains.
"To test Redis doing GET/SET is like to test a Ferrari checking how good it is at cleaning the mirror when it rains."
Salvatore Sanfilippo
Benchmarking vs. Other NoSQLs, i.e.g. #2
Google Compute Engine's How Many Servers Do You Need To Change a Lightbulb 1 Million Writes/Sec Benchmark Series• DataStax – 300 servers: http://googlecloudplatform.blogspot.com/2014/03/cassandra-
hits-one-million-writes-per-second-on-google-compute-engine.html
• Aerospike – 50 servers: http://googlecloudplatform.blogspot.com/2014/12/aerospike-hits-one-million-writes-Per-Second-with-just-50-Nodes-on-Google-Compute-Engine.html
<- We actually needed two, publication pending
"Comparative" Advertising – There Is A War
• The Open Source Redis Project is (and IMO should always remain) pure
• However,• The users want numbers and comparisons• The analysts want numbers and comparisons• The press wants numbers and comparisons• Redis' competitors are providing numbers and comparisons
• Redis is extremely popular, but it may not remain so without the public's attention
• The Redis community and ecosystem is (and IMO should continue to) provide this collateral in form of benchmarks, success stories, case studies, etc…
All Is Fair In Love And War…
And Benchmarking
An i.e.g.: Yahoo Cloud Serving Benchmark
"The goal of the Yahoo Cloud Serving Benchmark (YCSB) project is to develop a framework and common set of workloads for evaluating the performance of different "key-value" and "cloud" serving stores."
http://labs.yahoo.com/news/yahoo-cloud-serving-benchmark/
YCSB Redis client: https://github.com/brianfrankcooper/YCSB/blob/master/redis/src/main/java/com/yahoo/ycsb/db/RedisClient.java
The YCSB Challenge
• The original YCSB client implementation for Redis• Uses a Hash for storing multiple fields per key
• Uses Redis' commands correctly (i.e. HMGET)
• … but is still too slow
• "Weapon-grade", modified YCSB Redis client:• https://gist.github.com/dragnot/06ef5d01df8cb181555a
• Uses GET/SET
• Objects are serialized client-side with MessagePack
• Objects are updated server-side with Lua (cmsgpack)
YCSB Results