ITMS Performance Tuning & Optimization 1
IT Management Suite Design, Tuning & Optimization – Do it Right the First Time
Paul Conover, Brian Sheedy Dir, SQA Engineer, Sr Field Enablement Manager
SYMANTEC VISION 2013
Agenda
ITMS Performance Tuning & Optimization 2
Thinking About Performance 1
Architecture 2
SQL Server 3
Infrastructure & Product Considerations 4
Operating System & IIS 5
SYMANTEC VISION 2013 ITMS Performance Tuning & Optimization 3
Thinking About Performance “Any information regarding pre-release Symantec offerings, future updates or other planned modifications is subject to ongoing evaluation by Symantec and therefore subject to change. This information is provided without warranty of any kind, express or implied. Customers who purchase Symantec offerings should make their purchase decision based upon features that are currently available.”
Expectations: this is a white paper disguised as a presentation
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Thinking About Performance
• Random server errors
• Server timeouts
• Query timeouts
• Slow operations
• Queues filling up
• Caching warnings in the log
• Excessive deadlock retries in the log
• Key Performance Counters (Disk, Wait Stats)
ITMS Performance Tuning & Optimization 4
Obvious Signs for Tuning
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Architecture
• Engage the Right People
– Physical Server & OS Ops
– Virtual Server Ops
– Network Ops
– Storage Ops
– Database Administrators
– Users
– Business Requirement Owners
ITMS Performance Tuning & Optimization 6
Successful Implementation • Common Architecture Failings
– Incorrectly matched architecture for implementation
– Insufficient configuration for architecture
– Rollout didn’t match design
– Excessive non-essential data movement due to incorrect use case analysis
– Site configuration deficiencies
– IO & resource deficiencies
– Ongoing alignment with use cases and business requirements
• Growth
• Security, compliance & regulation
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Architecture
•Recommended Architectures in the Guide – ITMS Server (20 k per server, 120 k per hierarchy, 6 children)
– Site Server (7.5k nodes per Task/Pkg, 5k/PXE, 600 sites per ITMS-Task/pkg/PXE)
– 3000 Sites per hierarchy
ITMS Performance Tuning & Optimization 7
Planning Server Implementation with the Guide
• Meet System Requirements
• Same Data Center Preferred
• Plan Sites
• Plan CEM
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Architecture
• Plan, Design, Configure, and Monitor Sites & Site Health
• Favor Sites, and Org Groups with Security Over Hierarchy
• Package Servers
– Constrain Packages Servers
– Cascade Package Staging – Unconstrained -> Constrained -> Agents
• Task Server
– Limit Task Server Communications on the ITMS Server
– Provides Connection Management & Request Consolidation
• not bandwidth
– Don’t have agents connect to the parent
• Generally - Have at least 1 task server
ITMS Performance Tuning & Optimization 8
Site Design Planning
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Architecture
• Up to 3,000 concurrent connections, 60000 CEM-enabled endpoints
• Hardware requirements:
– Preferably physical box, 8GB RAM, 40GB HDD and dual-core CPU
– VM-based IG – Remember VM Latency/Resources
ITMS Performance Tuning & Optimization 9
ITMS 7.5 CEM Design (See the CEM Sessions for Deep Dive)
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Architecture - Virtual Machines
• Resource Availability to ITMS
• CPU, RAM, Disk, Network
• VM Resource Latency
• Virtual latency will be manifest to users as application latency
• Resource storms
• Vmotion, SnapShots & Defrag/MBR
• Example: Host with 4 CPUs
• ITMS Server configured for 4 CPUs
• Second App server configured for 1 or more CPUs
• Both will have have dramatic and measureable latency on CPU
• Well Balanced VM Infrastructure may still see ~20 less efficiency than physical
ITMS Performance Tuning & Optimization 10
Virtual Environment Considerations
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• Right-size SQL Resources
– Exceed IOPs
– Exceed Memory
– Meet CPU
ITMS Performance Tuning & Optimization 12
Storage IO & Resources
Component POC 100 – 1,000 1,000 – 5,000 5,000 – 10,000 10,000 – 20,000
Processors 1 core 4 cores 8 cores 8-16 cores 16+ cores
Disk Speed (in IOPS)
180 - C: OS,SMP 200 - D: SQL
180 - C: OS,SMP 200 - D: SQL
180 - C: OS, SQL App 300 - D: SQL DB 300 - E: Logs 200 - F: TempDB
180 - C: OS, SQL App 400 - D: SQL DB 400 - E: Logs 300 - F: TempDB
180 - C: OS, SQL App 600 - D: SQL DB 600 - E: Logs 400 - F: TempDB
Disk Capacity 80 GB 80 GB 120GB 250 GB 400 GB
RAM 16 GB 16 GB 16+ GB 24+ GB 32+ GB
• Estimate 6-8MB DB storage per managed endpoint
– Implement a Database that uses the above calculation + 10%
– i.e. 5,000 Endpoints x 6MB = 30GB + 3GB = 33GB CMDB minimum
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• Design for High IOPs – Performance Device
• Storage Selection (SSE/PCIe, SAN, Hybrid, Spindle)
– SSD/SAN: > 1 million IOPS
• High IO Bandwidth
• Low IO Latency
– Single Spindles: 150-200 IOPs
• RAID and multiple channels
– Optimize capacity requirements for Speed
• Say NO to BIG, SLOW storage from SAN admins.
• SSD or PCIe such as FusionIO DUO
– Test the storage subsystem with IOMeter and/or SQLIO
ITMS Performance Tuning & Optimization 13
Storage Subsystem
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• http://www.wesworld.net/raidcalculator.html
• If spindles: fast array of many small disks
• RAID 1+0, RSID 5 - Temp DB Data, CMDB Data, CMDB Logs 1+0, Temp Logs RAID 0
Storage RAID Calculator
Presentation Identifier Goes Here 14
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
ITMS Performance Tuning & Optimization 15
Test the Storage Subsystem - IOMeter/SQLIO
3 Bias Patterns
Large Test Data
64/128/256K Alignment
8/64k Transfer
8/64k, Reply
8/64k Request Size
100% Random Distribution
Burst Length = 1
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• Plan Volume Creation & Data Separation
– Worth the attention -- especially with spindle drives
• SQL Best Practices: http://technet.microsoft.com/en-us/library/cc966534.aspx
• Optimize for I/O parallelism
– Stripe size: 64/128 KB on SQL 2005 or 64/128/256 KB for SQL 2008+
– Spindles: create multiple files, place on separate physical volumes • CMDB Data
• CMDB Logs
• TempDB Data
• TempDB Logs
• High IO SSD: can place files on same volume.
– Again, Test the storage subsystem with IOMeter and/or SQLIO
ITMS Performance Tuning & Optimization 16
File Management
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• Multiple Database Files – Myth vs Reality
– http://blogs.msdn.com/b/psssql/archive/2007/02/21/sql-server-urban-legends-discussed.aspx
– Not for IO thread per CPU.
– For Latency + I/O
• Case for Multiple Data Files
– http://www.sqlskills.com/blogs/paul/benchmarking-multiple-data-files-on-ssds-plus-the-latest-fusion-io-driver
– Single Spindle Drive – less value due to total disk throughput.
– SSD – Sweet spot: # files = # CPUs < 8
– Too many can degrade
• Equal File Size to Avoid Ratio Dist.
ITMS Performance Tuning & Optimization 17
File Management - Case for Multiple Data Files
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• Contention on I/O, syscolumns, sysobjects
• As many files as schedulers
– At least two
– Large Scale Systems, consider # schedulers = # CPUs < 8
• Enable Uniform Allocations: Trace Flag (-T1118) – Avoids contention on SGAM (even on SQL 2005+)
• Monitor SQL IO stats
– PAGELATCH_XX
– PAGEIOLATCH_XX
• Optimize TempDB for Server Restarts (Growth & Size) ITMS Performance Tuning & Optimization 18
File Management–TempDB
SYMANTEC VISION 2013
SQL Helper File – The “Goods”
• Do NOT run all queries at once.
• Do choose to run queries one by one if you desire to use the functionality it provides.
• Do test.
Attached file has queries to review settings & set configuration, and Measure Perf
Presentation Identifier Goes Here 19
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• Use Performance Monitor and/or SQL Server Wait Stats
– Latency: < 20 ms (See query)
– Measure IOPs: http://sqlserverio.com/2011/02/08/gather-virtual-file-statistics-using-t-sql-tsql2sday-15/
• DiskQLength > 1 or 2
– Can be an indicator
– Can be misleading due to SQL asynchronous IO.
– Watch Average Disk Sec/Read Write & Transfer
• Memory
– Page Life Expectancy: > 300*
– Buffer Cache Hit Ratio: > 95-98%
– Free Pages: > 640
ITMS Performance Tuning & Optimization 20
Monitoring IO – How to Check IOPs & Latency
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• ITMS large and complex queries can over-utilize CPUs due to parallelism
– http://blogs.msdn.com/b/jimmymay/archive/2008/11/28/case-study-part-1-cxpacket-wait-stats-max-degree-of-parallelism-option-introduction-to-using-wait-stats-to-identify-remediate-query-parallelism-bottlenecks.aspx
• Default = 0 (change it) – Uses all CPUs
– 2 to 4 on Parent (1x6)
– 1 to 2 on Child
– Never > 8 or # CPUs, whichever is less.
• Use CXPACKET Wait Stats to verify SQL parallelism
– Use wait type query on the next slide
– Or, SSE Tools: http://symantec.com/docs/howto60787
– Consider changing if CXPACKET > 5%
• Increase Cost Threshold for Parallelism to 10
– Reduces chance of simple plans going parallel ITMS Performance Tuning & Optimization 21
Max Degree of Parallelism
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• On x32, use AWE. Don’t use AWE on x64 SQL
– x32 SQL with AWE, /3Gb, /pae, etc
– AWE is deprecated for x64: http://msdn.microsoft.com/en-us/library/ms175581(v=sql.105).aspx
• Enable the option to Optimize for Ad hoc Workloads
– ITMS has high degree of Ad-hoc/dynamic queries that are non-cacheable
– Doesn’t waste caching on Single-Use queries
– Caches on second+ request
• Set Maximum Server Memory value
– In 'Server memory options' page
– leave enough memory for the Operating System and tools such as SQL Management Studio to run effectively
– e.g. around 4-6 GB less than the amount of RAM installed.
ITMS Performance Tuning & Optimization 22
SQL Memory Configurations
NSQLRam = NTotalRam – (NOSRam + NAppRam)
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• Enable Arithmetic Abort in the Server/Instance Properties
– Terminates a query when a overflow or divide-by-zero error occurs.
– http://msdn.microsoft.com/en-us/library/ms190306.aspx
• Enable Read Committed Snapshot mode for improved UI *** • Improve reporting& console performance without blocking other activity
alter database Symantec_CMDB set read_committed_snapshot on with rollback after 30;
• Disable Autoshrink and Autogrow, Control Growth Settings 500 MB
– (default for ITMS Creation).
– Avoids unnecessary file re-allocations latency and increases file fragmentation.
– Avoids unnecessary CPU and IO cycles.
ITMS Performance Tuning & Optimization 23
Other SQL Settings
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization SQL Server
• Update Statistics (nightly)
– Ensures query processor makes “the best” choices it can given ITMS implementation.
– Or, enable Auto Update Statistics Asynchronously • Memory Leak in SQL - get latest update
• Database Properties > Options page.
• Enabling affects your ability to put a database into single-user mode (For Maintenance)
• Regular Database Page Maintenance
– Rebuild Indexes (Monthly)
– Reorganize pages (Weekly)
• Regular backup
– Simple Recovery Mode • Yields Lower IO but you should have an aggressive backup strategy
– Vs. FULL logging with backups
ITMS Performance Tuning & Optimization 24
Maintenance Plan
SYMANTEC VISION 2013 ITMS Performance Tuning & Optimization 25
Infrastructure & Product Considerations
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Infrastructure & Product Considerations
• Frequency & Node Counts
• Tasks & Policies
– Reduce targeting & scheduling, frequency and results
– All types: Custom Inventory, Software, Inventory, Patching, …
• Reports
– Optimize your reports that you run often
– Query lowest level resource views in the resource type hierarchy
• Filters
– Optimize all filters you create that get used
– Query lowest level resource views in the resource type hierarchy
• Inventory Solution
– Limit frequency & Amounts to only what is needed.
ITMS Performance Tuning & Optimization 26
Customizations & Usage
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Infrastructure & Product Considerations
ITMS Performance Tuning & Optimization 27
Product Configuration
Setting < 1000
endpoints 1000 – 5000
5000 – 10,000
10,000 – 15,000
15,000 to 20,000
Agent Configuration Interval Every 1 hour Every 1
hour Every 2 hours
Every 3 hours
Every 4 hours
Full inventory collection schedule Monthly Monthly Monthly Monthly Monthly
Delta inventory collection schedule Weekly Weekly Weekly Weekly Weekly
Complete Resource Membership Update
Daily Daily Daily Daily Daily
Delta Resource Membership schedule
15 min 20 min 30 min 45 min 1 hr
Task Service task update interval 5 min 5 min 15 min 15 min 15 min
Policy Refresh schedule 5 min 10 min 15 min 15-20 min 15-20 min
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Infrastructure & Product Considerations
• Cache size warning in the log
– Source: Altiris.NS.Security.SecurityCache.HandlePermissionCacheFloodedDescription: Core setting SecurityCacheSize is insufficient to handle incoming security checks. Current Size:xxx
• Bump up cache sizes
– CodeBaseCacheSize: NumPkgs * NumPkgServers
– ItemClassCacheSize: 1000 or Number of rows in Class table
– ItemCacheSize: 3000 to 100,000
– SecurityCacheSize: 5000 to 100,000(800,000 on full parent)
– StringCacheSize: 1,000,000 or # Rows in String table
– ResxCacheSize: # Strings in ResourceStrings table
– ClientPolicyCacheMaxSize: NumPolicies + Growth of Policies
ITMS Performance Tuning & Optimization 28
Core Settings – Cache Sizes
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Infrastructure & Product Considerations
• ResourceAssociationTablelockThreshold: 1,000,000
• ReportMaxRowCount from: 50,000 to 500,000
– Generally, set it to 0 unless performance is an issue
– Base on the use case.
• Replication Settings - Latest 7.1 SP2 MP1 update v2+
– ReplicationDataBatchSize: 5,000 – 10,000
– ReplicationEventBatchSize: 5,000 – 10,000
– ItemStateMaxSizeInKB: 128MB
• size of the data container to allow for the increase in the Batch Sizes
ITMS Performance Tuning & Optimization 29
Other Core Settings
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Infrastructure & Product Considerations
• SuppressReportAutorun (ITMS 7.5)
– When enabled reports do not run immediately.
– Affects every report.
• When enabled you must click Refresh to get results.
• Disabled by default
ITMS Performance Tuning & Optimization 30
Other Core Settings - Report Auto Run Control
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Infrastructure & Product Considerations
• Use Saved Searches for frequently used search
• Turn off Search as you type
• Limit result lists
– < 20,000 (consider < 500)
ITMS Performance Tuning & Optimization 31
Enhanced Console View Settings
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Infrastructure & Product Considerations
• Tune it, don’t accept out-of-the-box.
– Don’t schedule full replication (includes re-replication of full security)
• Do FULL rarely/manually (mostly in troubleshooting)
– Leverage Differential Replication Schedules (first differential is a full)
• KB that shows how to do full security in Delta
– Always stagger replications: Child 1 @ 1 am, Child 2 @ 3 am
• Leave time for success (avoid schedule clashes)
• Also stagger with other heavy activities such as full inventory.
• Limit what gets replicated
– Event replication – Only data that you must act on.
– Leverage IT Analytics over replication.
– Avoid extraneous replicated resources & data -- ie Software Files
ITMS Performance Tuning & Optimization 32
Hierarchy
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Infrastructure & Product Considerations
• Change log locations
• Move the event queues
• Don’t compress Event Queue and Replication path
• Use File System junction points to move Replication data folder
• Tune SMP logging & Managed Delivery verbose logging
– Use as needed
• Clear profile trace after each capture, don’t leave it running
• Use history judiciously
– Be cautious what data you keep and the duration that you keep it.
– Consider adding indexes if your history is impacting operations (such as viewing the resource or resource deletion).
ITMS Performance Tuning & Optimization 33
Operating Environment
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Operating System & IIS
• Optimize Performance for Background Services
• Control OS Background Activities
– Restrict schedules
– Limit directories and file types: *.MDF,*.LDF,*.TMP,*.NSE, Replication & Log Folders
– Services to Consider
• Indexing Service
• Encryption
• Compression
• AV Scanning
ITMS Performance Tuning & Optimization 35
Operating System
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Operating System & IIS
• Stop unnecessary services.
ITMS Performance Tuning & Optimization 36
Operating System
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Operating System & IIS
• General .Net Assembly functionality
– Published .Net Assemblies DLLs are digitally signed
– Signatures are validated when assemblies are loaded
• Certificate Revocation List validation at crl.microsoft.com
– 30-60 second timeout delay for each assembly load
• ITMS Symptoms
– Windows services on the Platform computer sometimes fail during startup.
– Some Symantec Management Console pages take a very long time to load.
– The SMP may experience performance issues after being offline for 15 days
– Applies to Site Servers also
• Resolution
– Disable CRL lookups
– You can re-enable CRL lookups easily
– Apply to Remote Site Servers
– Via Hosts File, Registry, or Machine.config
ITMS Performance Tuning & Optimization 37
.Net Framework CRL Validation –Server or Site has no Internet
Edit the x86 & x64 Machine.config Files:
<configuration>
<runtime>
<generatePublisherEvidence enabled="false"/>
</runtime>
</configuration>
SYMANTEC VISION 2013
ITMS Performance Tuning & Optimization Operating System & IIS
• IIS data compression
– Not especially effective since polices are already compressed but, can give some improvements for other activities
• Issues with Kerberos vs NTLM see KB on Browser settings for IE
– http://support.microsoft.com/kb/2749007
– Disable IE Pre-Authentication Headers on Machines accessing the console • OR
– Turn Off Kerberos Authentication in IIS
• HTTP Logging (disable or reduce)
ITMS Performance Tuning & Optimization 38
IIS & Internet Explorer
SYMANTEC VISION 2013
Topic Links
Planning & Implementation Guide
• http://www.symantec.com/business/support/index?page=content&id=DOC4827
SQL IO Tuning
• http://sqlserverio.com/2011/02/08/gather-virtual-file-statistics-using-t-sql-tsql2sday-15/
• http://www.sqlskills.com/blogs/paul/how-to-examine-io-subsystem-latencies-from-within-sql-server/
• http://blogs.msdn.com/b/askjay/archive/2011/07/08/troubleshooting-slow-disk-i-o-in-sql-server.aspx
IOMeter/SQLIO Tools
• http://www.microsoft.com/en-us/download/details.aspx?id=20163
SSE Tools • http://symantec.com/docs/howto60787
References & Resources
Presentation Identifier Goes Here 39
SYMANTEC VISION 2013
Topic Links
Task Server Practices
• http://www.symantec.com/business/support/index?page=content&id=HOWTO48411
Package Staging Practices
• http://www.symantec.com/connect/articles/how-handle-getpackageinfo-request-storms
• http://www.symantec.com/connect/blogs/working-codebasecache-post-getpackageinfo-series
Constrained Pkg Servers
• http://www.symantec.com/business/support/index?page=content&id=HOWTO6907
• http://www.symantec.com/connect/forums/constrained-vs-unconstrained
Package Replication
• http://symantec.com/docs/howto60787
• http://www.symantec.com/business/support/index?page=content&id=HOWTO42304
• http://www.symantec.com/business/support/index?page=content&id=HOWTO44198
References & Resources
Presentation Identifier Goes Here 40
Thank you!
Copyright © 2012 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.
ITMS Performance Tuning & Optimization 41
Paul Conover
801-228-7119
SYMANTEC VISION 2013
How To Print Slides Without a Dark Background
• Conserve printer ink by printing in Grayscale mode. Black backgrounds will be inverted to white, white text to black
• To View how your slides will print in Grayscale mode:
– From the View tab, in the Color/Grayscale group, click Grayscale
– Your Slides will display without color. Check each slide to ensure that all objects are displaying at high contrast (black or gray on white background)
– You can make adjustments for printing purposes. (Note: making changes in Grayscale does not effect the color on your slides)
– Select Back to Color View when finished
Presentation Identifier Goes Here 42
• To adjust Grayscale settings:
– The Grayscale tab includes various settings: Automatic, Grayscale, Light Grayscale, Inverse Grayscale, Gray with White Fill, etc.
– To apply a different grayscale setting, select the object(s) and choose a setting in the Change Selected Object group on the Grayscale tab
• To Print slides in Grayscale mode:
– From the Microsoft Office Button (top left corner), click Print to access the Print dialog box
– Under Color/grayscale: select Grayscale