March 2018
Spring Session Redis
Oded Shopen
Why, How and Production Considerations
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs2
About Me
▪ Oded Shopen▪ Software Architect @ Amdocs Israel▪ Working on Event-Driven, Cloud Native Microservices
▪🤓 Redis Geek
▪http://odedia.org ▪@odedia on twitter
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs3
Why?
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs4
Why Spring Session?
▪Once upon a time, there was the servlet container. And it was good…
Tomcat 🍪 123-456 = David
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs5
Why Spring Session?
Tomcat
🍪 123-456 = David
Tomcat Tomcat
Load Balancer
🍪 789-111 = Larry
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs6
Why Spring Session?
🍪 123-456 = ?
Tomcat Tomcat
Load Balancer
🍪 789-111 = Larry
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs7
Why Spring Session?
😞
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs8
Why Spring Session?
Tomcat Tomcat
Load Balancer
Tomcat Tomcat
Tomcat Tomcat
Tomcat Tomcat
Tomcat Tomcat
Tomcat Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs9
Why Spring Session?
Jetty Tomcat
Load Balancer
Jetty Tomcat
Tomcat Tomcat
Tomcat Tomcat
Tomcat Tomcat
Jetty Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs10
Why Spring Session?
Tomcat
Session Data
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs11
Why Spring Session?
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat Jetty
Session Data
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs12
Why Spring Session?
▪Replaces the built-in HttpSession with another implementation ▪ Transparent drop-in replacement when using Spring Boot ▪Makes your servers truly stateless ▪ Sessions survive application restarts ▪No need for Load Balancer sticky sessions ▪Conforms to the cloud-native apps 12-factor principals
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs13
Twelve-factor processes are stateless and share-nothing.
Any data that needs to persist must be stored in a stateful backing service, typically a database.
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs14
Why Spring Session Redis?
https://docs.spring.io/spring-session/docs/1.3.1.RELEASE/reference/html5/
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs15
Why Spring Session Redis?
▪ The application requires frequent, fast access to the session ▪A fast database is critical ▪Redis is really fast ▪No user experience degradation by externalizing the session to Redis
▪ Sessions needs to expire after some time ▪Redis expiring keys are a great solution
▪With sharding and clustering, Redis scales when your user base scales
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs16
How?
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs17
Demohttps://github.com/odedia/spring-session-redis-sample
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs18
ProductionConsiderations
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs19
#1 Be Prepared to Scale
▪At the very minimum, use a master/slave setup
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs20
#1 Be Prepared to Scale
▪Consider sharding your sessions in a Redis Cluster ▪RedisLabs + DNS Proxy will hide the cluster topology and let you focus on
a simple configuration on the client side ▪No need to configure sentinels
Node 1
Node 2
Node 3
Proxyspring.redis.url=proxy:6379spring.redis.sentinel.master=MasterNode spring.redis.sentinel.nodes=Node1:6379,Node2:6379,Node3:6379
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs21
#2 Pool Settings▪ Spring Data Redis uses a JedisConnectionFactory with the following defaults:
spring.redis.pool.max-active=8 spring.redis.pool.max-idle=8 spring.redis.pool.max-wait=-1 spring.redis.pool.min-idle=0 spring.redis.timeout=0
▪ Behind the scenes - an Apache Commons GenericObjectPool.
▪ Use max-active based on your tomcat/jetty threads.
▪ Use max-wait based on your setup
▪ Too high - all tomcat threads can get stuck, waiting
▪ Too low - your consumers may get errors
▪ Use min-idle of at least 8 to offset “sudden load” issues (such as mass login on a business day).
▪ Set spring.redis.timeout to a reasonably low number (such as 1000ms)
▪ Too high - your connection pool can become full quickly.
▪ Too low - your consumers may get errors
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs22
#3 Server-Side Metrics
▪ Spring Boot Actuator provides a production-ready monitoring metrics. ▪ Simply add spring-boot-starter-actuator to your gradle/maven dependencies ▪ /metrics endpoint provides valuable monitoring data ▪However, Spring Data Redis is not available… ▪ Luckily, actuator is extensible! ▪https://github.com/nysd/spring-boot-redis-metrics
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs23
#4 Monitor Redis
▪Redislabs monitoring dashboards provide excellent visibility to identify issues, as we’ll soon see.
▪ If you use open source and monitor yourself on-prem, keep in mind that Redis is a single-threaded database. ▪A 2-core virtual machine would report 50%
CPU utilization ▪However, redis itself might be at 100% at that
time ▪Monitor the process, not the VM
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs24
#5 Please Monitor Responsibly
▪Be careful how you monitor… ▪KEYS * runs over all the keys in the DB. Your DB is unresponsive until operation is over! ▪SMEMBERS returns all keys in a given set. Some sets may contain all keys in the system. ▪ In our use case, we kept track of all logins from a certain vendor, which is pretty
much O(n) cardinality, so same as KEYS command ▪SCARD simply returns the size of a given set with cardinality of O(1), so it can be used
safely in production ▪ SCARD “spring:session:index:org.springframework.session.FindByIndexNameSessionRepository.PRINCIPAL_NAME_INDEX_NAME:UI_USERS"
▪ INFO provides valuable information (such as instantaneous_ops_per_sec) and can be safely used in production.
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs25
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs26
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs27
#6 Inside Spring Session Redis
▪What happens when your system is completely idle (0 users)? ▪You would expect redis to be idle as well ▪However…
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs28
#6 Inside Spring Session Redis
▪Multiple logins using: ▪for ((i=1;i<=10000000;i++)); do curl -s -u gateway:password localhost:8080/user >/dev/null; done
▪One would expect consistent load on redis, but in reality, we got:
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs29
#6 Inside Spring Session Redis
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs30
#6 Inside Spring Session Redis
▪ Time to dig into the code… (thanks, open source!) ▪ Inside RedisOperationsSessionRepository:
@Scheduled(cron = "${spring.session.cleanup.cron.expression:0 * * * * *}") public void cleanupExpiredSessions() { this.expirationPolicy.cleanExpiredSessions(); }
▪cleanupExpiredSessions() would loop over expired keys, delete them, and “touch” them to make sure they are immediately deleted.
▪ This means that by default, every minute on the minute, Spring will call delete on every expired key in Redis.
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs31
#6 Inside Spring Session Redis
▪Why? Redis can self-expire keys just fine, thank you very much ▪The main reason: ▪Spring wants to conduct cleanup activities when a session is expired. ▪The API exposes SessionDeletedEvent and SessionExpiredEvent to let the
program cleanup a user’s session (close web sockets, notify SSO server etc.) ▪Spring needs the session details available when the cleanup activity is
performed
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs32
#6 Inside Spring Session Redis
▪To solve this issue, two separate keys are managed for each session ▪An “expiration notification” key expires after 30 minutes ▪The actual session details key expires 5 minutes later
HMSET spring:session:sessions:33fdd1b6-b496-4b33-9f7d-df96679d32fe creationTime 1404360000000 \maxInactiveInterval 1800 \lastAccessedTime 1404360000000 \sessionAttr:attrName someAttrValue \sessionAttr2:attrName someAttrValue2
EXPIRE spring:session:sessions:33fdd1b6-b496-4b33-9f7d-df96679d32fe 2100APPEND spring:session:sessions:expires:33fdd1b6-b496-4b33-9f7d-df96679d32fe ""EXPIRE spring:session:sessions:expires:33fdd1b6-b496-4b33-9f7d-df96679d32fe 1800SADD spring:session:expirations:1439245080000 expires:33fdd1b6-b496-4b33-9f7d-df96679d32feEXPIRE spring:session:expirations1439245080000 2100
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs33
#6 Inside Spring Session Redis
▪Redis guarantees expired keys would be passively cleaned, but makes no guarantees on when
▪Spring Session Redis wants to make sure the “expire” key is expired within the 5 minutes window.
▪For this reason, the cronjob updates the expired keys every minute
▪But, there is a risk here…
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs
#6 Inside Spring Session Redis
34
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat Jetty
Session Data
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs35
#6 Inside Spring Session Redis
Session Data
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat
Tomcat Jetty
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs36
#6 Inside Spring Session Redis
▪ To recap: ▪Every minute, each server tries to delete all “expires” keys. ▪A lot of redundant calls, since only the first instance would actually achieve this
purpose. ▪Redis would then notify every instance that the “expires” key was deleted. ▪Every instance would then connect to Redis to get the session details about the
expiring session to handle the expiration. ▪A little bit of math: ▪100 servers * 2500 expiring sessions = 500,000 access at the same second ▪ The more you scale, the bigger the problem
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs37
#6 Inside Spring Session RedisSolutions: ▪ If you don’t care about session expiration events, set the cron job to an impossible value such as
February 31st: spring.session.cleanup.cron.expression=0 0 5 31 2 ?
▪ You can also disable registering to those expiration and deletion events. Note that all servers need to disable the registering for this to work: @EnableRedisHttpSession
public class RedisOperationsSessionRepositoryConfigNoOp { @Bean public static ConfigureRedisAction configureRedisAction() { return ConfigureRedisAction.NO_OP; } }
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs38
#6 Inside Spring Session Redis
▪If you do care about session expiration events, set only a specific instance(s) to handle those events.
▪On all other servers, disable receiving the events: @EnableRedisHttpSession
public class RedisOperationsSessionRepositoryConfigNoListener { @Bean
public RedisMessageListenerContainer redisMessageListenerContainer( RedisConnectionFactory connectionFactory, RedisOperationsSessionRepository messageListener) {
return new RedisMessageListenerContainer(); } }
Information Security Level 2 – Sensitive© 2017 – Proprietary & Confidential Information of Amdocs39
To Recap
▪Be prepared to scale ▪Monitor well, monitor with caution ▪Open source rocks 🤟. Get involved! Tweak the
framework for your own use case.
Thank you
http://odedia.org