Date post: | 23-Jan-2015 |
Category: |
Technology |
Upload: | craig-nelson |
View: | 3,261 times |
Download: | 4 times |
Troubleshooting Sitecore Performance
Monitoring & Analyzing Performance IssuesAugust 12, 2013 1
Please review
before viewing this presentationhttp://www.slideshare.net/cwnelson/sitecore-performance-tipshttp://www.slideshare.net/cwnelson/sitecore-mobile-performance-tips
2
Sitecore Enterprise Web CMS
• Popular .NET Web Content Management System for mid-to-large organizations• Provides friendly tools for non-technical Content
Authors and Marketers• Can be customized and extended to meet almost
any conceivable business need
3
Troubleshooting Performance
Steps to troubleshoot performance issues• Create baseline logs of performance metrics• Monitor system when it is offline or off hours• Monitor system during normal operation
• Regularly review performance logs• Become familiar with normal metrics for your system
• Investigate performance metrics that are not normal• Scrutinize counters in areas of unusual performance• Try to narrow the scope of performance problems
• Take action to remove and or resolve bottlenecks4
Performance Monitoring
Monitoring helps locate root causes of issues• Inadequate resources or resource misallocation• Non-optimum coding or use of a legacy API• Missing configuration settings or misconfiguration
Monitoring helps identify resource bottlenecks• Server hardware & operating systems• IIS & ASP.NET • Databases• Sitecore
Monitoring is a recurring process, not an event• Familiarity with the system is a key success factor
5
Why Monitoring is Important
Building business trust• Ensuring performance meets user expectations• Averting business-impacting performance issues
Accomplishing cost-effective site management• Efficiently diagnosing root causes of issues• Perceiving the corrective actions needed to fix issues• Addressing issues instead of upgrading hardware• Accurately budgeting for future growth needs
6
Windows Performance Monitor
Windows Performance Monitor (PerfMon)• Use it to display graphs for all performance logs• System, IIS, ASP.NET, SQL, Sitecore counters, etc.• Computer Mgmt > Performance > Monitoring Tools• Use <Ctrl-h> to toggle highlight of selected counter
7
Understanding Bottlenecks
Performance bottlenecks• Performance of system is limited by a single resource• Bottleneck points are limiting components of system
Bottleneck indicators• One or more resources are over-consumed• Resource requests exceed available resources• Resource requests must be queued until available
• One or more resources are under-consumed• More subtle type of bottleneck that is harder to locate• Resource requests are not at their expected levels• The bottleneck blocking requests is elsewhere
8
Troubleshooting Bottlenecks
Identifying bottlenecks• Create a baseline log when system is offline or at rest• Monitor the system to log the current behavior• Compare the current behavior with the baseline log
Investigating bottlenecks• Review system counters before Sitecore counters• Look for sustained behaviors, not peaks or spikes• Over-consumption or under-consumption of resources• Use broad counters first; % Disk Time (Reads & Writes)• Use detailed counters to investigate suspicious values;
Avg. Disk Bytes Read & Avg. Disk Read Queue Length9
Viewing Server Performance Logs
Inspect server performance logs for issues• Check for signs of memory leaks• Creeping increases in consumption indicate a leak
• Check for signs of bottlenecks• High resource consumption for prolonged periods• Related queue length above 2
• Check for non-optimum metrics• Errors are signs of code issues or misconfiguration• Cache misses are signs of configuration issues
10
Server Bottlenecks
Server Bottlenecks• One or more server resources are over consumed• The server contains one or more bottleneck points
11
Server Bottleneck Types
Memory Processor Disk Network
Server Performance
Troubleshooting server performance issues• Log performance objects that contain key counters • Set logging interval to 15 minutes• Log data over multiple weeks
• Periodically review logged data for key counters • Use detailed counters to investigate suspect counters• Key counters that show signs of a potential problem• Periodically review Sitecore logs for nested exceptions• Rectify issues found from monitoring & Sitecore logs• Periodically review caching statistics & tune caches 12
Key Server Objects & Counters
Memory• Available Bytes• Committed Bytes
Processor • % Processor Time
System• Processor Queue Length
13
Physical Disk• % Disk Time• Avg. Disk Queue Length
Network Interface• Bytes Total/sec• Current Bandwidth• Output Queue Length
Warning Signs for Server Counters
Use performance counters to check warning signs• Memory: Committed Bytes is creeping up long term• Processor: % Processor Time is above 75% long term
and System: Processor Queue Length is above 10• Physical Disk: % Disk Time is above 50% long term and
Avg. Disk Queue Length is above 2 • Network Interface: Bytes Total/sec shows utilization is
above 25% and Output Queue Length is above 214
Memory Performance Counters
Key memory performance counters• Available Bytes - memory available for system use or apps• Value should be at least 10% of physical memory
• Committed Bytes - memory with page file space reserved • Value should be less than 75% of physical memory• A slow rise could indicate a memory leak
• Pages/sec - rate paging is used to resolve hard page faults• Preferred range is 0 to 20. Investigate if over 80
• Pool Nonpaged Bytes - memory for objects not paged• A slow rise could indicate a memory leak
• Pool Paged Bytes - memory for objects paged to disk
15
Processor Performance Counters
Key processor performance counters• % Processor Time – ratio CPU ran non-idle threads • Value should be less than 75% • Check for high value & Processor Queue Length > 10
Key system performance counters• Context Switches/sec - rate CPUs switch threads• A thread is preempted by a higher priority ready thread• CPU switches between user-mode and kernel-mode• High number can indicate high contention in locks
• Processor Queue Length - # threads in the CPU queue• Divide this value by the number of processors• A sustained queue > 10 threads per CPU is a problem
16
Disk Performance Counters
Key physical disk performance counters• % Disk Time - ratio that disk was busy with requests• Value should be less than 50%
• Avg. Disk Queue Length - requests that were queued• Disk delays and bottlenecks are likely when value > 2
Key paging file performance counters*• % Usage - ratio of Page File in use• % Usage Peak - ratio of peak usage of Page File
* Paging File counters are also related to memory counters 17
Network Performance Counters
Key network interface performance counters• Bytes Total/sec - rate of bytes over NICs• Network utilization should be less than 25%
• Output Queue Length – packet queue length• Value should be less than 2
18
Process Performance Counters
Key Sitecore process performance counters • Private Bytes - amount of memory used by process • Memory that cannot be shared with other processes
• Virtual Bytes - virtual address space used by process• Page File Bytes - space process reserved in page file• IO Data Operations/sec - rate process is doing I/O• IO Other Operations/sec - rate of control functions • % Processor Time – ratio process handled instructions
19
ASP.NET Performance Counters
Key ASP.NET performance counters• Application Restarts - # restarts during server uptime• High number indicates problems are causing recycles
• Requests Queued - # requests waiting for service• Increase shows server is at limit of concurrent requests• Raise default of 5,000 in the machine.config file
• Worker Process Restarts - # worker process restarts• Shows unexpected failures and intentional recycles
20
ASP.NET Application Counters
Key ASP.NET Applications performance counters• Errors Total - # of errors executing HTTP requests• Investigate parser, compilation and run-time errors
• Pipeline Instance Count – max # concurrent requests • Shows requests that are being processed for the app• High number signifies the CPU is not well utilized by app
• Requests/sec - current throughput of application• Values should remain within range under constant load
21
.NET Performance Counters
Key .NET CLR Exceptions performance counters• # of Exceps Thrown - # of exceptions thrown• A high value indicates a code or configuration problem• Check Sitecore logs for related application errors
Key .NET CLR Memory performance counters• # Induced GC - # calls to the Garbage Collector• A high value indicates a code or configuration problem• Remove calls to GC.Collect and GC.GetTotalMemory
• % Time in GC - time spent since end of the last cycle• Investigate managed heap when value is 50% or higher 22
IIS Performance Counters
Key Web Service performance counters• The following values should be as high as possible• Bytes Total/sec - # bytes transferred by the service• Total Method Requests/sec - HTTP requests received• Current Connections - # connections to web service
Key Web Service Cache performance counters• The following values should be as high as possible• File Cache Hits % - ratio successful lookups since startup• Kernel:URI Cache Hits % - ratio of cache hits to requests
• The following values should be as low as possible• Kernel:URI Cache Flushes - # files removed from cache• Kernel:URI Cache Misses - # unsuccessful cache lookups
23
SQL Performance Counters
Key SQL Buffer Manager performance counter• Buffer Cache Hit Ratio - % of pages found in buffer• Page did not have to incur a read from disk• Value should be as high as possible
Key SQL Statistics performance counters• Batch Requests/Sec - # received SQL batch requests• Over 1000 batch requests/sec may indicate a problem
• SQL Compilations/Sec - # Transact-SQL compilations • Over 100 compilations/sec may indicate a problem 24
SQL Access Performance Counters
Key SQL Access Methods performance counter• Page Splits/sec – splits from overflowing index pages• An index or data page can split when it becomes full• Page splitting can cause excessive I/O• Increase fill factor to allocate more space
25
Improving SQL Performance
Configuring SQL Server for best performance• Set compatibility level to SQL Server 2008 (100)• Set Auto Close and Auto Shrink properties to false• Set Recovery Model to simple• Check for percentage of fragmentation on indexes• Run the Index Physical Statistics Standard Report• Run report against each of the Sitecore databases• Rebuild indexes based on recommended operations
• Verify a Maintenance Plan exists and runs regularly• Clean up History, PublishQueue & EventQueue tables• Use SQL script in CMS Performance Tuning Guide
26
Sitecore Performance Counters
Key Sitecore Caching performance counters• The following value should be as high as possible• Cache Hits - # of cache hits for all caches
• The following values should be as low as possible• Cache Clearings - times that a cache has been cleared• Cache Misses - # of cache misses for all caches
Key Sitecore Data performance counters• Data.PhysicalReads - # of read operations from DB• Data.PhysicalWrites - # of write operations to DB 27
Sitecore Counters Configuration
Sitecore Performance Counters Configuration• Counters.Enabled – enable/disable counters • Enable performance counters when monitoring• Disable performance counters when not monitoring
• Counters.InstanceName – counter names• Change setting when you monitor multiple web apps
• Counters.ResetOnStartup – resets values to zero (0)
28
The Sitecore Health Monitor
The Sitecore Health Monitor background task• Periodically writes counter values to Sitecore logs• Supports Windows, CLR and Sitecore counters• Web.config contains healthMonitor pipeline settings• Interval specified by HealthMonitorInterval setting• Users can add counters for LogCounterStatus method• instanceType must be specified for added counters
29
The Counter Dump Agent
The Sitecore Counter Dump Agent• Periodically collects data on all Sitecore counters• Writes data to files in XML format • Files contain Sitecore counter values for time ranges• The frequency & filename can be set in web.config <agent type="Sitecore.Tasks.CounterDumpAgent" method="Run" interval="01:00:00"> <DumpFile>$(dataFolder)/diagnostics/counters.{date}.{time}.{processid}.txt</DumpFile> </agent>
30
Sitecore TaskManager Module
Sitecore Task Manager and Performance Monitor• Shared source module available from Marketplace• A web-based tool to monitor background processes• Indexing, scheduled tasks, publishing, etc.
http://marketplace.sitecore.net/Modules/TaskManager
31
Load Testing
Load Testing helps find slow running code paths• Most efficient way to identify code bottlenecks• Run load tests without caching during development• Use ANTS to profile your solution while stress testing
Load Testing helps ensure a system will scale• Creates simulated server loads expected on live site• Anticipates performance with increased traffic loads• A series of test cases are used during the testing• Best to monitor on both the server and client sides
32
Load Testing Tools
Server-based load testing• JMeter is a free stress testing/monitoring tool• Useful for identifying bottlenecks during development• Available at http://jmeter.apache.org/
• Web Capacity Analysis Tool (WCAT) is a free tool• http://www.iis.net/1466/ItemPermalink.ashx
Web-based load testing• Load Tester LITE is provided at no cost• Load Tester PRO can be rented or permanent license• Amazon has rental fees for cloud-based load engines• Available at http://www.webperformance.com/
33
In Summary
• Performance monitoring is a process, not an event• Use PerfMon to log system & Sitecore counters• Log counters for system, IIS, ASP.NET, db, Sitecore, etc.• Monitor performance objects containing key counters• Investigate additional detailed counters as needed• Use Health Monitor to log counters to Sitecore logs• Use Task Manager to monitor background processes• Use load testing to find slow running code paths
34
Caveat Emptor
There are many ways to monitor Sitecore performance• Skilled administrators can provide other solutions• Review your monitoring needs with your Sitecore partner• Review proposed changes with all stakeholders• Implement new configurations on a test system firstThis presentation is intended to be academic• To enhance public knowledge of performance monitoring• To promote better understanding of monitoring SitecoreAuthor cannot guarantee accuracy or fitness for your site• Author makes no representations or warranties• Author disclaims any liability if you act on this information
35
Acknowledgements
Wikimedia Commons Image of ASP.NET Architecture Stack
Wikimedia Commons Image of Network Topology
Wikimedia Commons Image of Warning Sign
Download slide presentation to view URLs
36
More Topics from TIC
All Sitecore-related SlideShares from TIChttp://www.slideshare.net/cwnelson/presentations
How to Assure Success for a New Sitecore Site (full slide deck)http://www.slideshare.net/cwnelson/creating-new-sitecore-sites-full
How to Avoid Shooting Yourself in the SEOhttp://www.slideshare.net/cwnelson/sitecore-seo-tips
Implementing Routine Sitecore Maintenancehttp://www.slideshare.net/cwnelson/sitecore-maintenance-tips
Performance Tips for Sitecore Mobile Usershttp://www.slideshare.net/cwnelson/sitecore-mobile-performance-tips
Sitecore 7 – New Features for Usershttp://www.slideshare.net/cwnelson/sitecore-7-new-features
Sitecore Performance Tips for Visitor Engagementhttp://www.slideshare.net/cwnelson/sitecore-performance-tips
37
The Inside Corner, Inc.
• The Inside Corner, Inc. (TIC) is owned by Craig Nelson• TIC is a Sitecore partner & web consultancy• A U.S. point of contact for teams of Sitecore specialists • We create new Sitecore sites & enhance existing ones
38
Primary Developer - Craig Nelson
• Sitecore Specialist certified for Sitecore 7• 10 years of system administration experience• 10 years as a Microsoft Certified Trainer• 5 years of ASP.NET web development experience• 4 years of Sitecore development experience• 3 years as a Sitecore technical instructor• Sitecore architecture & configuration expert• Performance & SEO optimization specialist• Workflow configuration specialist 39
Get Help with your Sitecore Site
Follow TIC on LinkedIn• http://www.linkedin.com/company/the-inside-corner-inc.
Add TIC to your Circles on Google Plus• http://google.com/+Theinsidecorner
Follow TIC on Twitter • https://twitter.com/TheInsideCorner
Contact Craig Directly• Craig Nelson - Certified 7.0 Developer and Sitecore Partner• cwnelson[at]ticdevs.com• Mobile - 707.319.7446
Visit the TIC website• http://ticdevs.com
40
Bibliography
All about Sitecore performance counters — A real hero
Optimizing Performance in Sitecore
SDN - Working with Sitecore Performance Counters
Sitecore CMS 6.0 -6.5 Diagnostics Guide
Sitecore CMS 6.0 -6.6 Performance Tuning Guide
Troubleshooting Performance Problems in SQL 2008
Download slide presentation to view URLs
41
Thank you for taking the time to view this presentation on
Troubleshooting Sitecore Performance
Please share with others and add a comment if you think it is useful
ThanksAugust 12, 2013
42