Site Studio Performance Tuning

Brian “Bex” HuffChief Software Architect

2

Agenda

General web-site tuning tips• The complex basics

High-performance architecture• Tune your system, network, and servers

Tuning Site Studio pages• Oracle UCM specific tips

3

Philosophy of Performance Tuning

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil”

-- Donald Knuth

Need a holistic view of your systems and processes• Use data-flow analysis and diagrams

4

Why Are Network Applications Slow?

Not enough CPU• Needed for text processing, number crunching, cryptography

Not enough memory• Needed to cache temporary data for faster processing

Not enough bandwidth• Needed to support large data transfers, large user bases

Too much latency• Multiple round-trips to servers makes any single request slow

Most web applications are slow because of bandwidth/latency • extra CPU and memory help, but usually are not enough

5

Bandwidth and Latency in Real Life

48 inches wide

Bandwidth:• 2,000,000 barrels per day!

800 miles long

Latency:• 15.8 days!

6

Bandwidth and Latency On The Web

Bandwidth:• Number Of Simultaneous Requests You Can Support

Latency:• Time Added To Each Request Due To Network and Distance

A web page with 50 images means 51 HTTP requests!

High bandwidth and high latency? Pages “feel” slow

80-90% of response time spent in downloading web resources• Images, CSS files, JS files, AJAX requests• Source: http://developer.yahoo.com/performance/rules.html

7

Analyze The Flow Of Data

How does the data get to the client?

How many applications are involved?

Where is the bottleneck?• Latency: ping, traceroute, or tracert• Bandwidth: mrtg, wget, or ask your ISP/IT department• CPU and Memory: top or taskmanager

On the web, latency is usually a bigger problem than bandwidth• Distance and network health between web client and web server• Assume New York to San Francisco latency is ~100 ms• if your page has 100 images, that's 10 seconds for just latency!

In practice, browser caching and multiple downloads help

8

UCM Data Flow Diagram

How do the actual bytes flow from client, to server, and back?

client

Database

File System

Content Server & Web Server

LDAP / Active Directory

9

Reduce CPU Needs of Web Pages

General tip: simplify your web pages so they render faster

Use fewer DOM elements• tables within tables within tables

Put CSS at the top, and JS at the bottom• CSS is rendered immediately• JS must be 100% loaded before it executes

Avoid complex CSS• AlphaImageLoader filter• CSS expressions

10

Reduce Bandwidth Needs of Web Pages

General tip: use smaller web resources

Compress your HTML, CSS, and JS files• minimize their size in general• inline GZIP compression works well

Use small, web-optimized images• avoid scaling them in HTML

Use GET for AJAX

Reduce use and complexity of cookies• cookie state is sent with every request

11

Reduce Latency Of Web Pages

General tip: use fewer web resources• eliminate images when CSS works just as well• consolidate CSS and JS into fewer files• correct any 404 error for missing images

Use CSS SPRITES• Like Google does

12

Reduce Latency Of Web Pages (cont.)

Make fewer network requests• Avoid multiple DNS lookups on the same page• Minimize URL redirects

Serve static files from multiple servers• Browsers download 2 web resources at a time from one domain• So, make more domains: img1.company.com, img2.company.com

Use content delivery network• Amazon Cloudfront : cheap, but mainly North American • Akamai : expensive, but global

Make content “cacheable”

13

High-Performance Architecture

Web Page Cache / Reverse Proxy• Cache rendered web pages to take load off of Site Studio• make sure it understands cookies and GZIP compression

Contribution / consumption environment• Optimize contribution for read/write• Optimize consumption for read (mostly)

Use a dedicated server for each Oracle UCM instance• Do not share with other applications

Schedule resource-intensive behavior in the off-hours• Virus scanners and web spiders• Backups, archives, batchloads

14

High-Performance Architecture (cont.)

Create a cluster• in most cases with 10gr3, significantly improves performance• general best practice for failover, high reliability, and uptime

Minimize latency in local network• keep database, LDAP, and other resources on same subnet• keep filesystem speedy: attached storage, SAN, NAS

Monitor CPU and Memory usage• bandwidth, and latency are usually the culprits• CPU and memory can be a bottleneck for heavy consumption servers

If you use UCM 7.5 with Verity search, CPU can be an issue• if you use database search, database CPU may be an issue

15

Sample High-Performance Architecture

Contribution Cluster

Consumption Cluster

internet

Replication

Reverse Proxy Cluster

DatabaseCluster

SAN

16

Make Static Content Cacheable

Web caches will store static content (JS, CSS, images)• Modest performance gains by keeping them in-memory• Caching dynamic content is better, but harder…

Add Expires or MaxAge HTTP header to content• allows a cache to know how long to store content• for static content, set this in the web server• for dynamic content, set it in Site Studio

Properly configure ETag HTTP headers on clusters• If you're not using them, just remove them

17

Make Dynamic Content Cacheable

Dynamic HTML pages might be different per user• Multi-step forms and personalization are cache-killers

• Different users see different things at the same URL!

• Not all caches work when cookies are present• Need smart caches, and smart personalization

Drive customization with static CSS and JS files• Read cookies from JS, use AJAX to alter pages• Configure cache to ignore cookies, or• Configure web server to omit cookie header

Security checks can be slow• How far away is your user database?• Is it cached locally, or do we need to check ActiveDirectory or LDAP?

18

Site Studio Specifics

UCM performance rule-of-thumb• 20 read-only service calls per second per GHz CPU• 5 read/write service calls per second per GHz CPU

Problem: some Site Studio pages are multiple service calls!• Each calls at least the SS_GET_PAGE service• Many fragments and templates call GET_SEARCH_RESULTS• Many also call DOC_INFO

Not uncommon for one page to have 4 UCM service calls!• Reduces performance to 5 page views per second per GHz CPU

19

UCM Specifics

Minimize number of queries per page• Use static lists instead of dynamic lists when possible

Optimize search queries• Use <matches> or <contains> instead of <substring>• Note: <contains> only works in "zone indexed" metadata fields• Avoid time stamps in queries

Tune Java VM to have as much memory as you can give it• JAVA_MAX_HEAP_SIZE=1024

Optimize the Oracle Database• Install the OracleQueryOptimizer component• Cursor_Sharing=SIMILAR• optimizer_mode=first_rows

20

Finding Slow Pages: Passive

Configure web server logs to measure page creation time• Apache: use mod_log_config and the LogFormat directive• IIS: use W3C extended log file format, and the "Time Taken" field

Enable UCM System Audit Information flags• requestaudit• pagecreation• searchquery• systemdatabase

Parse log files to find slow pages and slow queries

21

Finding Slow Pages: Active

Run a web spider on your system• Caution: this can crash your server!• Use web logs or spider logs to find slow pages

Test slow pages once with auditing turned on• You may be surprised how many database calls there are!

Test slow pages with a load tester• How many users can do this simultaneously?

22

Finding Slow Pages: Active (cont.)

Bandwidth: • How fast can you download a static 1MB HTML file?• How many users can do this simultaneously?

Latency: • How fast can you download a 1k HTML file with a hundred 10k JS files?• How many users can do this simultaneously?

Keep excellent notes!• Only way to measure performance improvement

23

Making Pages Faster

Reduce number of service calls

Reduce number of database calls

Optimize IdocScript• Minimize eval and executeService calls• Instead use setValue, setResourceInclude• Eliminate all IdocScript errors: these are performance drags• Use cacheInclude for slow sections of pages

Optimize JSP/ASP/Java code• In general, same rules apply• Reduce complexity, reduce number of service calls

24

Making Pages Faster (cont.)

Consolidate JS and CSS files• Place them into a global fragment, or global region template• Standard best-practice for maintainable code

Set flags to cache pages• Built-in with the custom section property "ssMaxAge"• Passes the cache control header• Watch out for cookies!

Disable unnecessary JavaScript files• Requires custom component• On consumption environment, disable contributor JS files• Most sites don't need the JS files that contain navigation

25

Review

General web site performance tuning is critical

Data flow diagrams can highlight bottlenecks

Some architectures scale better than others• Size CPU and Memory appropriately• Cluster your servers• Include a reverse proxy• Keep network resources close, and fast!

Understand why specific Site Studio pages are slow• Reduce number of service requests per page• Focus on making “cacheable” content

26

Useful Tools and References

Yahoo YSlow plug-in to Firefox: a must have!• http://developer.yahoo.com/yslow/

JMeter load tester• http://jakarta.apache.org/jmeter/

Heritrix web crawler• http://crawler.archive.org/

Varnish HTTP Accelerator• http://varnish.projects.linpro.no/

Yahoo web site performance tuning best practices• http://developer.yahoo.com/performance/rules.html

Oracle UCM performance tuning guide:• http://download.oracle.com/docs/cd/E10316_01/cs/cs_doc_10/documentati

on/admin/performance_tuning_10en.pdf

27

Now Available…

28

My Company: http://bezzotech.com

My Blog: http://bexhuff.com

My Self: bex@bezzotech.com

Questions?