Date post: | 27-Jan-2015 |
Category: |
Technology |
Upload: | crashlytics |
View: | 110 times |
Download: | 2 times |
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Redis Analytics
@JeffSeibertCEO, Crashlytics
2
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL3
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL4
Crashlytics for Mac
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Strings
Lists
Hashes
Sets
Sorted Sets
8
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Strings
Lists
Hashes
Sets
Sorted Sets
9
Activity Tracking
Event Tracking
Leader boards
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL10
Active User Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
CREATE TABLE accounts ( id int(11) NOT NULL AUTO_INCREMENT PRIMARY KEY, name varchar(255), email varchar(255), ...
last_active_at datetime);
11
Active User Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
CREATE TABLE events ( id int(11) NOT NULL AUTO_INCREMENT PRIMARY KEY, type varchar(32), account_id int(11), happened_at datetime);
12
Active User Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Active User Tracking
14
0 0 0 0 1 0 0 1
accounts::active
SETBIT key offset value (>= 2.2) O(1) > SETBIT “accounts::active” 4 1> SETBIT “accounts::active” 7 1
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Active User Tracking
15
1 1 1 0 1 0 1 1
accounts::active::2012-10
0 0 1 0 1 0 0 1
accounts::active::2012-10-22
0 0 0 0 1 0 0 1
accounts::active::2012-10-22-00
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Active User Tracking
16
def record_active(obj, t=Time.now.utc) key = "#{obj.class.name.downcase.pluralize}::active::" key << t.year.to_s key << "-" << '%02d' % t.month REDIS.setbit key, obj.id, 1 # accounts::active::2012-10 key << "-" << '%02d' % t.day REDIS.setbit key, obj.id, 1 # accounts::active::2012-10-22 key << "-" << '%02d' % t.hour REDIS.setbit key, obj.id, 1 # accounts::active::2012-10-22-00end
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL17
‣ We want to know…• How many users were active today? This month?
• Was user X active today? This month?
Active User Tracking
BITCOUNT key (>= 2.6) O(N) > BITCOUNT “accounts::active::2012-10-22”(integer) 3> BITCOUNT “accounts::active::2012-10”(integer) 5
GETBIT key index (>= 2.2) O(1) > GETBIT “accounts::active::2012-10-22” 6(integer) 0> GETBIT “accounts::active::2012-10” 6(integer) 1
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL18
‣ Graphs and Heatmaps• Monthly actives over time?
• Over time, when was user X active?
Active User Tracking
> GETBIT “accounts::active::2012-10-22” 6> GETBIT “accounts::active::2012-10-21” 6> GETBIT “accounts::active::2012-10-20” 6> GETBIT “accounts::active::2012-10-19” 6...
> BITCOUNT “accounts::active::2012-07”> BITCOUNT “accounts::active::2012-08”> BITCOUNT “accounts::active::2012-09”> BITCOUNT “accounts::active::2012-10”...
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL19
‣ Advanced Data-Mining: WAU• Computing weekly active users:
•
Active User Tracking
BITOP op destkey srckey [srckeys...] (>= 2.6) O(N) > BITOP OR “accounts::active::2012-W42” \ “accounts::active::2012-10-21” \ “accounts::active::2012-10-20” \ “accounts::active::2012-10-19” \ “accounts::active::2012-10-18” \ “accounts::active::2012-10-17” \ “accounts::active::2012-10-16” \ “accounts::active::2012-10-15”> BITCOUNT “accounts::active::2012-W42”
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL20
‣ Advanced Data-Mining: Retention• What % of users active last week are active this week?
•
Active User Tracking
BITOP op destkey srckey [srckeys...] (>= 2.6) O(N) > BITOP AND “accounts::active::2012-W41+W42” \ “accounts::active::2012-W41” \ “accounts::active::2012-W42”> BITCOUNT “accounts::active::2012-W41”> BITCOUNT “accounts::active::2012-W41+W42”
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL21
‣ Advanced Data-Mining: Churn• Locate accounts that have been inactive for 3 months
•
Active User Tracking
BITOP op destkey srckey [srckeys...] (>= 2.6) O(N) > BITOP OR “accounts::active::2012-Q3” \ “accounts::active::2012-09” \ “accounts::active::2012-08” \ “accounts::active::2012-07”> BITOP NOT “accounts::churned::2012-Q3” \ “accounts::active::2012-Q3”> BITCOUNT “accounts::churned::2012-Q3”
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL22
def record_boolean(obj, topic=:active, t=Time.now.utc) key = "#{obj.class.name.downcase.pluralize}::#{topic}::" key << t.year.to_s key << "-" << '%02d' % t.month REDIS.setbit key, obj.id, 1 # accounts::active::2012-10 key << "-" << '%02d' % t.day REDIS.setbit key, obj.id, 1 # accounts::active::2012-10-22 key << "-" << '%02d' % t.hour REDIS.setbit key, obj.id, 1 # accounts::active::2012-10-22-00end
Active User Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL23
Event Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL24
0 0 0 0 ? 0 0 0
apps::crashes
Event Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL25
apps::crashes { 0 => 34, 1 => 546457, 2 => 1}
HINCRBY key field increment (>= 2.0) O(1) > HINCRBY “apps::crashes” “0” 1> HINCRBY “apps::crashes” “2” 1
Event Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL26
app::0::crash::by_day { 2012-10-22 => 34, 2012-10-21 => 46, 2012-10-20 => 29, ...}
> HINCRBY “app::0::crash::by_day” “2012-10-22” 1
Event Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL27
def record_event(obj, topic=:crash, specificity=:day, t=Time.now.utc) key = "#{obj.class.name.downcase}::#{obj.id}::#{topic}::by_#{specificity}" # e.g. app::0::crash::by_day field = t.year.to_s field << "-" << '%02d' % t.month # 2012-10 REDIS.hincrby key, field, 1 if specificity == :month field << "-" << '%02d' % t.day # 2012-10-22 REDIS.hincrby key, field, 1 if specificity == :day field << "-" << '%02d' % t.hour # 2012-10-22-00 REDIS.hincrby key, field, 1 if specificity == :hour end
Event Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL28
‣ We want to…• Power a graph of crashes over the last week
• “Zoom” the graph to see more detail
HMGET key field1 [...] (>= 2.0) O(N) > HMGET “app::0::crash::by_day” “2012-10-22” \ “2012-10-21” “2012-10-20” “2012-10-19” \
“2012-10-18” “2012-10-17” “2012-10-16”1) ...
> HMGET “app::0::crash::by_hour” “2012-10-22-00” \ “2012-10-22-01” “2012-10-22-02” “2012-10-22-03” \
“2012-10-22-04” “2012-10-22-05” “2012-10-22-06” ...1) ...
Event Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL29
Grouped Event Tracking
“How often has app X crashed on each type of iPad?”
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL30
app::0::crash::iPad1,1 { 2012-10-22 => 34, 2012-10-21 => 46, 2012-10-20 => 29, ...}
app::0::crash::iPad2,1 { 2012-10-22 => 12, 2012-10-21 => 17, 2012-10-20 => 11, ...}
device_models [ “iPad1,1”, “iPad2,1”, ...]
Grouped Event Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL31
app::0::crash::2012-10-22 { ALL => 46, iPad1,1 => 34, iPad2,1 => 12, ...}
HGETALL key (>= 2.0) O(N) > HGETALL “app::0::crash::2012-10-22”(multi-bulk)
Grouped Event Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL32
def record_grouped_event(obj, group, topic=:crash, t=Time.now.utc) key = "#{obj.class.name.downcase}::#{obj.id}::#{topic}::" key = t.year.to_s key << "-" << '%02d' % t.month # app::0::crash::2012-10 REDIS.hincrby key, group, 1 REDIS.hincrby key, 'ALL', 1 field << "-" << '%02d' % t.day # app::0::crash::2012-10-22 REDIS.hincrby key, group, 1 REDIS.hincrby key, 'ALL', 1 field << "-" << '%02d' % t.hour # app::0::crash::2012-10-22-00 REDIS.hincrby key, group, 1 REDIS.hincrby key, 'ALL', 1 end
Grouped Event Tracking
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL33
MongoDB
> Account.first.id=> BSON::ObjectId('507db04798a3340ada000002')
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Sequential ID Generation
34
ZADD key score member (>= 1.2) O(log(N)) > ZADD “sequential_ids::accounts” 10 507db04798a3340ada000002(integer) 1
sequential_ids::accounts { 10 5084bfbb98a33406f0000002, 9 5084bfa798a33406f0000001, 8 507db04798a3340ada000002, ...}
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Sequential ID Generation
35
sequential_ids::accounts { 10 5084bfbb98a33406f0000002, 9 5084bfa798a33406f0000001, 8 507db04798a3340ada000002, ...}
ZCARD key (>= 1.2) O(1) > ZCARD “sequential_ids::accounts”(integer) 9
ZADD key score member (>= 1.2) O(log(N)) > ZADD “sequential_ids::accounts” 10 5084bfbb98a33406f0000002(integer) 1
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Sequential ID Generation
36
sequential_ids::accounts { 10 5084bfbb98a33406f0000002, 9 5084bfa798a33406f0000001, 8 507db04798a3340ada000002, ...}
ZSCORE key member (>= 1.2) O(1) > ZSCORE “sequential_ids::accounts” 5084bfbb98a33406f0000002(integer) 10
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Sequential ID Generation
37
def sequential_id(obj) key = "sequential_keys::#{obj.class.name.downcase.pluralize}" id = obj.id.to_s
# Lua script to atomically determine the score of an id. # If needed, adds it to the set with the next available score. # In the general case, O(1). On add, O(log(N)). Requires Redis >= 2.6 monotonic_zadd = <<LUA local sequential_id = redis.call('zscore', KEYS[1], ARGV[1]) if not sequential_id then sequential_id = redis.call('zcard', KEYS[1]) redis.call('zadd', KEYS[1], sequential_id, ARGV[1]) end
return sequential_idLUA
REDIS.eval(monotonic_zadd, [key], [id]).to_iend
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL38
Redis Analytics Wish List
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL39
‣ MSETBIT, MGETBIT, MBITCOUNT, HMINCRBY• Can already be addressed with scripting
‣ Native support for (insertion-)ordered sets‣ Per-hash-key expiration policies
Redis Analytics Wish List
© 2012. All rights reservedCRASHLYTICS CONFIDENTIAL
Q&A
40
@JeffSeibertCEO, Crashlytics