1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
1
Agenda for ISAAC’s APM JourneyAgenda for ISAAC’s APM Journey
Being ISAAC and its expectations
ISAAC’s holistic quality framework
Active Performance Management in ISAAC
APM in production
APM in development
Questions and Answers
2 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
2
Being ISAACBeing ISAAC
• ISAAC is a fully-integrated student lifecycle system
3 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
3
Being ISAAC - expectationsBeing ISAAC - expectations
• ISAAC needs to be better, faster and more effective than the legacy systems
• ISAAC turnaround time for support needs to be good
• ISAAC implementation has to be smooth
• ISAAC needs to be up almost 24 X 7
• ISAAC needs to be error-free
4 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
4
ISAAC – Being AgileISAAC – Being Agile
• ISAAC adopted parts Agile Software Development
– Welcome changing requirements, even late in development. (Agile manifesto)
– Agile processes harness change for the customer's competitive advantage (Agile manifesto)
– Iterative Development Approach
– Just in time documentation
– Evolutionary Architecture
5 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
5
Agile has its challengesAgile has its challenges
– Change requirements -> Change in Design
– Change in design -> Change in architecture
– Change in scope but no change in timeline?
– Things are built iteratively
– Regression testing to be done?
– Integration testing to done?
6 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
6
Agile – How to make senseAgile – How to make sense
• Agile Testing practices
– Sufficient quality assurance measures need to be built
– Test to be automated (critical!)
– Test coverage to be inspected
– Continuous Integration Cycle
7 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
Holistic Quality Framework• Unit Tests
– Unit tests in C#,
– Services, persistence and business logic is functional
• Functional Auto Test
– Automatic UATs, codified test suites
– Browser compatibility
• Performance Tests
– Load test, stress test
– No auto test for performance
– Gap in framework
– What happens when code goes to production?
– How to profile performance in UAT?
• Need an APM tool!
8 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
8
Hunt for the perfect APM toolHunt for the perfect APM tool
• Some tools merely monitor the services
• Some tools are difficult to setup
• Some tools are difficult to use
• Some tools are deployed on the cloud
• Dynatrace is chosen because of
– Ease of installation
– Richness of “deep-dive” information
– Vibrant online community
9 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
9
The old way – reading server logThe old way – reading server log
10 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
10
The new way – APM toolThe new way – APM tool
11 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
11
APM In ISAAC ProductionAPM In ISAAC Production
12 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
12
Case Study 1 – Student Course RegistrationCase Study 1 – Student Course Registration
• ISAAC Course Registration was launched at 2nd Jan 2014
• 2600 student teachers went online to register their courses over 2 days
• First time ISAAC handled a mission critical function for students
• Courses are registered on a first-come-first-serve basis
• Traffic Spikes are expected in the first moments of the window opening
• Not a bidding system but a real-time constraint-checking system
13 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
13
Case Study 1 – Student Course RegistrationCase Study 1 – Student Course Registration
• Major risks identified
– Can system handle the load?
– Will the quota checks be compromised under high load?
– Will the high spikes in traffic affect other ISAAC modules?
• Measures taken to address risk
– Load test done for 200 concurrent users on production environment
– Additional application server added to server farm
14 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
14
Case Study 1 - Purepaths as it happensCase Study 1 - Purepaths as it happens
15 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
15
Case Study 1 - Deep dive into transactionCase Study 1 - Deep dive into transaction
16 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
16
Case Study 2 - Intermittent SlownessCase Study 2 - Intermittent Slowness
• User complain of intermittent slowness in system for some features
• Cannot be reproduced in staging or development servers
• Was working fine for the past week
• No code patches for the past week
17 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
17
Case Study 2 - Finding of root causeCase Study 2 - Finding of root cause
18 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
18
Case Study 3 - BottlenecksCase Study 3 - Bottlenecks
19 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
19
APM In ISAAC DevelopmentAPM In ISAAC Development
20 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
20
Agile Development Process with APMAgile Development Process with APM
21 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
21
Pre-production Load Testing with DynatracePre-production Load Testing with Dynatrace
• ISAAC Course Registration was loaded tested with 200 concurrent users
• Users were simulated using functional testing frameworks
• The following are done using dynatrace
– Performance Profiling, how performance scale with number of concurrent users
• Monitored server resource such as CPU, RAM
• Monitored transaction throughput
– Stress test, increased until system break or performance degrade dramatically
22 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
Performance Profiling of Register Course Function
• Ramp up number of concurrent users
• Check how the timings vs the number concurrent users
• Stress test until performance deteriorate dramatically
• Check Average vs Worst timings for each run
23 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE
Conclusion
• APM is an essential tool in ISAAC’s quality assurance framework
• It allows developers to deep-dive into their code’s performance easily
• It enables real-time insight for production systems
24
25