26
| Date post: | 16-Jan-2017 |
| Category: |
Technology |
| Upload: | redis-labs |
| View: | 233 times |
| Download: | 2 times |
START job "A"
I/Ostdin,
stdout, …
SSHsession "X"
SENDqueue "A:X"
LISTENqueue "X:A"
AUTHORIZE
HTTP SSE
Architecture for SSH I/O streaming
SSH I/O streaming via Redis-based persistent message queue
Mani TadayonRedisConf 2016
About me• @bwsr_sr
• Working in software since ’01
• Using Redis since ’13
<shameless-plug>
• Just finished a book on Ruby testing: RSpec Essentials
• http://amzn.com/1784395900
</shameless-plug>
How to build a message queue with Redis
• Live listener
> subscribe mychannel
• Publisher
> publish mychannel mymessage
And that’s my talk, thanks for listening!
DoDo’s mad!
– http://antirez.com/news/88
“Redis apparently is at the same time the best and the worst system to use like that.”
Demo
START job "A"
I/Ostdin,
stdout, …
SSHhost "X"
SENDqueue "A:X"
LISTENqueue “A:X”
AUTHORIZE
HTTP SSE
Architecture for SSH I/O streaming
How to build a persistent message queue with Redis
• Retrieve persisted messages
> lrange mykey 0 -1
• Publisher
> rpush mykey mymessage
What about a live listener?
The best of both worlds• Live listener
> subscribe mychannel
• Retrieve persisted messages
> lrange mykey 0 -1
• Publisher
> rpush mykey mymessage
> publish mychannel mymessage
That’s pretty much it. But the SSH I/O feature needs a few more features.
• Lookup by session ID (“A:X”)
• Use a simple list of key names
> rpush byjob mykey
> expire byjob 604800
• Lookup by hostname (“A:*”)
• Use a zset with timestamp as score
> zadd byhost 1463012431 mykey
> zremrangebyscore byhost 0 1462407631
• Wrap each message in a transaction
• Protect against excessive memory usage (150GB in 2 hours on the 1st day…)
• Limit number of persisted messages per job
• Expire persisted messages (more aggressively for verbose jobs)
• Stop sending messages above a threshold (handled outside Redis)
> multi
> rpush mykey mymessage
> expire mykey 604800
> ltrim mykey -100 -1
> publish mychannel mymessage
> exec
• Set up indexes for lookup (only once per job)
> rpush byjob mykey
> expire byjob 604800
> zadd byhost 1463012431 mykey
> zremrangebyscore byhost 0 1462407631
Key and channel names
ssh-io:session:event:persisted:$ID:$FQDN
ssh-io:session:event:live:$ID:$FQDN
ssh-io:session:event_lookup:by_job:$ID
ssh-io:session:event_lookup:by_hostname:$FQDN
> multi
> rpush ssh-io:session:event:persisted:b8042948:fakehost.example.com {"session_started":"About to run 4 commands on fakehost.example.com for CustomScript-TestStreamingSimple:b8042948","io_type":"session_started","timestamp":1462407631}
> expire ssh-io:session:event:persisted:b8042948:fakehost.example.com 604800
> ltrim ssh-io:session:event:persisted:b8042948:fakehost.example.com -100 -1
> publish ssh-io:session:event:live:b8042948:fakehost.example.com {"session_started":"About to run 4 commands on fakehost.example.com for CustomScript-TestStreamingSimple:b8042948","io_type":"session_started","timestamp":1462407631}
> exec
> rpush ssh-io:session:event_lookup:by_job:b8042948 ssh-io:session:event:persisted:b8042948:fakehost.example.com
> expire ssh-io:session:event_lookup:by_job:b8042948 604800
> zadd ssh-io:session:event_lookup:by_hostname:fakehost.example.com 1463012431 ssh-io:session:event:persisted:b8042948:fakehost.example.com
> zremrangebyscore ssh-io:session:event_lookup:by_hostname:fakehost.example.com 0 1462407631
> multi
> rpush ssh-io:session:event:persisted:b8042948:fakehost.example.com {"username":"deploy","stdin":"echo "ping number 1"","hostname":"fakehost.example.com","io_type":"stdin","timestamp":1462407631}
> expire ssh-io:session:event:persisted:b8042948:fakehost.example.com 604800
> ltrim ssh-io:session:event:persisted:b8042948:fakehost.example.com -100 -1
> publish ssh-io:session:event:live:b8042948:fakehost.example.com {"username":"deploy","stdin":"echo "ping number 1"","hostname":"fakehost.example.com","io_type":"stdin","timestamp":1462407631}
> exec
# … multi,rpush,expire,ltrim,publish,exec …
> multi
> rpush ssh-io:session:event:persisted:b8042948:fakehost.example.com {"session_finished":"Ran 4 of 4 commands on fakehost.example.com for CustomScript-TestStreamingSimple:b8042948","success":false,"io_type":"session_finished","timestamp":1462407631}
> expire ssh-io:session:event:persisted:b8042948:fakehost.example.com 604800
> ltrim ssh-io:session:event:persisted:b8042948:fakehost.example.com -100 -1
> publish ssh-io:session:event:live:b8042948:fakehost.example.com {"session_finished":"Ran 4 of 4 commands on fakehost.example.com for CustomScript-TestStreamingSimple:b8042948","success":false,"io_type":"session_finished","timestamp":1462407631}
Lookup by hostname
• Lua script that uses the zset index
• Returns all events, in order, per hostname
$ redis-cli zrange ssh-io:session:event_lookup:by_hostname:fakehost.example.com 0 -1 1) "ssh-io:session:event:persisted:0bd6005b-1999-42ab-9b54-2585e0383bcb:fakehost.example.com" 2) "ssh-io:session:event:persisted:84ae935f-55a2-4210-a356-87a0da7c3b58:fakehost.example.com" 3) "ssh-io:session:event:persisted:35696242-a59d-4a1f-b36f-4ee988b558c5:fakehost.example.com" 4) "ssh-io:session:event:persisted:b72488ec-816c-48b9-a48d-ab31cbc41802:fakehost.example.com" 5) "ssh-io:session:event:persisted:5b13b4bd-2800-47fe-a821-a548c90054b6:fakehost.example.com" 6) "ssh-io:session:event:persisted:26078a92-e349-43a7-9375-d0ce511b1dbd:fakehost.example.com" 7) "ssh-io:session:event:persisted:4c3f31f0-dd2d-411f-bfbe-d6591698bf3a:fakehost.example.com" 8) "ssh-io:session:event:persisted:60983166-f580-44d6-9058-3b8bb04c1441:fakehost.example.com" 9) "ssh-io:session:event:persisted:572de4a5-3d15-4160-ab42-006407662ca8:fakehost.example.com"10) "ssh-io:session:event:persisted:3b405df9-3618-4cc2-b245-4dac4b7a203b:fakehost.example.com"11) "ssh-io:session:event:persisted:5968f887-9228-4963-9515-b92eda944063:fakehost.example.com"12) "ssh-io:session:event:persisted:ebcd6068-4a22-48f1-a333-36a312bb78f0:fakehost.example.com"13) "ssh-io:session:event:persisted:d22ae42d-4343-46b8-a477-8fa057bc34b5:fakehost.example.com"14) "ssh-io:session:event:persisted:49c994db-9d3d-4453-ac83-8dbf038a655d:fakehost.example.com"15) "ssh-io:session:event:persisted:2fc5ea10-c8f8-42e7-93d4-073b68826ffa:fakehost.example.com"16) "ssh-io:session:event:persisted:b8042948-63f1-4c6f-afc8-8d2384b3b155:fakehost.example.com"17) "ssh-io:session:event:persisted:5fee9662-af66-4027-9963-ff59716dc7e0:fakehost.example.com"18) "ssh-io:session:event:persisted:c919f801-ab8b-4f31-92f8-3f51d6b7b7dc:fakehost.example.com"19) "ssh-io:session:event:persisted:22f480d9-31cb-4054-9861-51d0275c9468:fakehost.example.com"20) "ssh-io:session:event:persisted:7710e61f-921c-4d6d-bbf4-e9c35d932f1a:fakehost.example.com"
-- finds the keys for all sessions for... -- ...a given hostname using lookup lists,-- then retrieve all events in all keys-- use nested numeric lua tables (i.e. arrays)...-- ...since redis will wipe out string keys-- see: http://redis.io/commands/eval -- (Conversion between Lua and Redis data types)
local keys = redis.call("zrange", KEYS[1], 0, -1) local returner = {}for i, key in ipairs(keys) do returner[i] = { key, redis.call("lrange", key, 0, -1) }endreturn returner
$ redis-cli eval "$(cat values-from-lookup-set.lua)" \ 1 \ ssh-io:session:event_lookup:by_hostname:fakehost.example.com1) 1) "ssh-io:session:event:persisted:0bd6005b-1999-42ab-9b54-2585e0383bcb:fakehost.example.com" 2) 1) "{\"session_started\":\"About to run 4 commands on fakehost.example.com for :0bd6005b-1999-42ab-9b54-2585e0383bcb\",\"io_type\":\"session_started\",\"timestamp\":1462405267}"# … 18 more sessions …20) 1) "ssh-io:session:event:persisted:7710e61f-921c-4d6d-bbf4-e9c35d932f1a:fakehost.example.com" 2) 1) "{\"session_started\":\"About to run 6 commands on fakehost.example.com for CustomScript-TestStreamingSimple:7710e61f-921c-4d6d-bbf4-e9c35d932f1a\",\"io_type\":\"session_started\",\"timestamp\":1462524784}"
Authentication• From browser directly to node.js web service
• Web app creates an auth token
• Token written to Redis…
• …and stored in browser session
• 1 day expiry
• Simple Lua script to create or retrieve token
local token = ''if redis.call('exists', KEYS[1]) == 1 then token = redis.call('get', KEYS[1])else redis.call('setex', KEYS[1], ARGV[1], ARGV[2]) token = ARGV[2] endreturn token
$ redis-cli eval "$(cat stream_auth_token.lua)" \ 1 \ ssh-io:session:stream:auth:myuser \ 86400 \ "7292ed99-da44-42c3-897a-745b6e43ac33"# => "7292ed99-da44-42c3-897a-745b6e43ac33"
$ redis-cli eval "$(cat stream_auth_token.lua)" \ 1 \ ssh-io:session:stream:auth:myuser \ 86400 \ "some-new-random-token"# => "7292ed99-da44-42c3-897a-745b6e43ac33"
START job "A"
I/Ostdin,
stdout, …
SSHhost "X"
SENDqueue "A:X"
LISTENqueue “A:X”
AUTHORIZE
HTTP SSE
Architecture for SSH I/O streaming
Thanks for listening
No Shiba Inus were harmed in the making of this presentation
