Recusive Technology LLC

Tracking Memory Performance of Siebel CTI Object managers

With VA2™ for Siebel

Recusive Technology LLC

Background• 7 production Siebel appservers, each 8 CPU 8gb physical memory• ~3000 concurrent users• Critical problem: >90% memory use requires reboots of all

production appservers every 2 days• Memory leak appears to be correlated with a code change in

December 2005, however no historical info on memory use available

• Manual methods not sufficient for identifying which processes are leaking memory

• No historical information available to compare against current performance trends, in areas such as:– Number of Siebel tasks (users)– Performance of individual components– Memory and CPU use patterns per component and process

• VA2 was used to track real time and historical information related to the memory leak problem

Recusive Technology LLC

Point in time: view of # of Siebel tasks per Siebel Appserver

Realtime view for every Siebel

appserver

CTI Component:

440 Tasks7 Processes

Recusive Technology LLC

VA2 - real time memory use

per component

eCommObj_CTI component is using about 2x physical and virtual memory than non CTI. About 5.3 GB of physical memory

Other components have

insignificant memory use

Recusive Technology LLC

Siebel appserver Prod7 process listSome CTI processes are using a huge amount of memory nearly 1 gig!!

Others are using

much less – why??

OS PID + Siebel component, shows which PID relates to each Siebel component

Recusive Technology LLC

CPU Use – CTI Obj Mgr using the most CPU time

Recusive Technology LLC

Historical: Statistics – used for historical reports

100% Customizable historical VA2

“Statistics”.

Unlimited number of KPI (key

performance indicators)

available as Statistics

Recusive Technology LLC

Number of eCommOBJ_CTI Tasks

Historical report on the number of CTI tasks running across all 7 Siebel servers. Consolidate view makes it easier to track total number across all appservers.

Monday and Tuesday the 13th have higher number of tasks than previous days monitored.

Recusive Technology LLC

Drilldown into number of eCommOBJ_CTI Tasks (2)

Drilling into Monday and Tuesday, the number of tasks reaches almost 4000 between 10am and 3PM then starts dropping.

Time

Number of tasks does not drop to 0, but 500 across all app servers-

why??

Recusive Technology LLC

Physical memory does not drop with # of tasks

Physical memory was not released until a reboot Sunday night.

There is an apparent memory leak, as the number of tasks from previous slide does not equal amount of physical memory used. And physical memory use continues to grow.

Recusive Technology LLC

Task report for 1 Appserver: Prod7

Number of tasks for Prod7 follows same trend as # of tasks for all appservers, meaning load balancing is working.

Tops out at about 550 tasks.

Recusive Technology LLC

Physical memory use for Prod7 follows same trend

For particular appserver Prod7, the physical memory use is the same as the larger pattern, of not releasing memory when number of tasks drop.

The total amount of physical memory is very high, showing 5,500,000 KB or 5.5 GB of physical memory used for CTI components alone

Recusive Technology LLC

Virtual memory drops even less than physical with lower tasks

Recusive Technology LLC

Conclusions• Original problem of memory leak confirmed via monitoring – directly

related to eCommObj_CTI Siebel components• After code changes (March 8 2006) and reboot (March 12 2006) ,

memory leak still apparent• Physical and virtual memory not being released with number of

tasks dropping, causing a memory leak• Virtual Memory releases even less memory than physical• Number of CTI tasks never drops to zero even when there should

be 0 users on the system• Siebel Service request should be started • Possibly engage Expert Services sizing review.• System running near capacity at all times – it could be that new load

and not only code changes are now accelerating memory use • Use VA2 data to track trends / changes to the system