Managing Performance and Availability for 25,000 Siebel Contact Center Users with Oracle Real Application Clusters
Roland Höhl, Perot Systems / Deutsche Telekom AG 30rd of July, 2009RAC SIG Webseminar
[email protected]@telekom.de
2
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans
Agenda
3
Who am I / Biography Roland Höhl
46 years old
Studied at the university in Darmstadt / Germany
Perot Systems employee Manager Application Development
Working for Deutsche Telekom as external consultant in the CRM-T project since 2006
In the Deutsche Telekom CRM project responsible for technical architecture infrastructure sizing and planning release management
4
Company overview – Deutsche Telekom
Internationalization: Deutsche Telekom is represented in ~ 50 countries worldwide..
Employees: Approx. 242,000 employees worldwide (December 31, 2007)
Revenue: 62.5 billion euros (2007 financial year)
Figures T-Home 37 Million Fixed Line User (PSTN) No.1 in German DSL-Market
(total amount12, 5 Mio Customer) 150.000 new Customer for IPTV (Entertain) in 2007 1.800 Sale points (Biggest integrated sales- and
Service organization for Fixed and Mobile products)
5
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans
Agenda
6
Facts and figures Project scope
Siebel upgrade from 7.0 to 7.8.2.5 Change from DB2 Host (OS390) to Oracle 10g RAC Unix Reimplementation of customized Siebel business processes Data migration from DB2 to Oracle 10g (based on EIM load) Building a ESB around Siebel CRM (based on MQSeries)
Project staffing: Oracle – Software vendor (Siebel / Database) IBM Global Services – Development Siebel / EAI T-Systems - Operations and hosting
Duration: Project start: 11/2005 First release: 04/2008 Involved employees: ~800
7
Project GoalsStrategic Goals
Improve customersatisfaction and raise call center efficiency
Time reduction to develop new
products and services
Cost reduction for operations,
maintenance and enhancements
Standardization, risk mitigation and
protection of investment
Functional Scale identical with CRM-C V2.130(KM, KKM, OM)
Upgrade to version:Siebel Standard SW 7.8
Align reduction of the application toSiebel Standard
Reduction of interface complexity(Service oriented integration platform)
Hardware change – Change of database platform if applicable
Master function of CRM-T for customer andcontract data (End of parallel operation, KoSi)
Improved application operation, incident and problem management
Operative Goals
8
Project Business Drivers Advantages for the customers:
Improved customer service Increased customer satisfaction
Advantages for the employees: Enhanced graphical user interface Simplification of IT systems Single and completed view of customer information Transparency of processes Decreased administrative complexity
Advantages for the Deutsche Telekom: Simplification of IT processes Improvement of IT stability Increased flexibility through open standards Retirement of legacy systems Achieved saving potentials in operations, support and development 1. Milestone in redesign of the IT architecture
9
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans
Agenda
10
Deutsche Telekom – T-Home - CRM Landscape
2 Data Centers Data Warehouse
Legacy Systems
T-ShopT-Shop
Call CentersCall Centers
PartnerPartner
T-E
SB
11
Scope and usage
89 call center within Germany 20000 named call center users 7000 named T-Shop and partner users
80 million accounts up to 40.000 customer contacts per day
(Inbound items, i.e. Calls, Mails, Docs) ~ up to 10.000 orders / hour ~ 19 TB Data in the Database
~ 40.000.000 user requests / hour ~ 50.000.000 disk reads / hour Up to 7.000.000.000 buffer gets / hour
42 Legacy systems connected several million messages per day transferred
12
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans
Agenda
13
Infrastructure Architecture
14
Web Server Infrastructure
Software Configuration• AIX 5.3 ML8• IBM IHS 2.0• Siebel Web Server Extension 7.8.2.5
Hardware Configuration• 12 x IBM P5 9133-55A
•4 x 1.65 GHz P5 CPUs•24 GB Memory
• CISCO CSS 1150•Frontend and Backend Load balancer
Front End Load
Balancer
Back End Load
Balancer
15
Software Configuration• AIX 5.3 ML8• Siebel 7.8.2.5• WebSphere MQ V6 (on EAI Servers only)• IBM HACMP (Cluster) on EAI Servers• Genesys CTI Driver
Siebel Application InfrastructureHardware Configuration
Users related Servers
• 6 x IBM P595 (eCommunications)•48 x 5 GHz P5 CPUs•336 GB Memory
• 3 x IBM P595 (eCommunications)•32 x 2.1 GHz P5 CPUs•128 GB Memory
• 2 x IBM P590 (Siebel eConfigurator)
EAI Server• 8 x IBM P590 (Active)
•8 x 2.1 GHz CPUs•24 GB Memory
• 8 x IBM P590 (passive – Virtual Servers)•0.1 x 2.1 GHz CPUs•24 GB Memory
Batch/Read Audit Servers• 4 x IBM P590 (active + passive)
•8 x 2.1 GHz CPUs•32 GB Memory
16
Oracle RAC Infrastructure
Software Configuration• AIX 5.3 ML8• ORACLE 10.2.0.3• ASM, Streams
SSDSANSSD
SSDSANSSD
Fibre Channel Switches
Hardware Configuration
Oracle RAC Nodes
• 4 x IBM P595•36 x 2.1 GHz P5 CPUs•312 GB Memory
SAN Storage 160 TB• 10 x IBM DS8300
•RAID 10•16 x 146 GB Disks•2 x 2 GB/sec FC
Solid State Disks• 2 x Texas Memory Systems RamSan-500
•2 TB Flash RAID•2 GB/sec Bandwidth
17
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans
Agenda
18
Session Management for Performance, Availability, and ScalabilitySessions Handling
•Up to 20.000 user session + several hundred sessions from EAI Components will be handled on the Siebel System
•2 Options to handle them on the Database
•Oracle MTS
•Siebel Connection Pooling
EAI
DatabaseService
19
Session Management for Performance, Availability, and ScalabilitySessions Management
Siebel Connection Pooling of 5:1 for User Sessions•Session are connected via DB Services
•Up to now each session will connect via the same service•Multiply Services allow to
•Separate – Distribute – Prioritize - … load over the different RAC Nodes.
Siebel Connection
Pooling
EAI
DatabaseService
20
Session Management for Performance, Availability, and Scalability
Siebel Connection
Pooling
Resource Management
•Siebel implements Transparent Application Failover TAF internal
•Bad SQL statements have been hard to stop
•Solution was Resource Profiles
•CPU limitation
EAI
DatabaseService
DatabaseService
21
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans
Agenda
22
RAC Performance & Availability Aspects
General memory considerations Session size up to 15 MB Keep enough memory for the operating
system 10-20% more memory on RAC environments
Amount of cluster nodes Expected load and hardware limitations
defines the minimum # of nodes More nodes provide better availability / less
failure impact (MTBF, upgrades, …) More nodes allows dedicated resource
assignment Less nodes reduce interconnect traffic,
management overhead
•Interconnect•Dedicated infrastructure•Failover Interfaces / Channel bundling
23
RAC Performance & Availability Aspects II
Disaster Scenarios and Stretch Cluster Limitations 20 – 50 km Max Disaster Recovery
Consider Rolling Upgrades Increase the availability – reduce planned down time
Define database services as soon as possible Group Servers Separate load Control performance and availability
Watch out for Architectural bottlenecks Centralized Tables, Sequences Prevent, Tune, or Invest
24
RAC Performance Configuration
Consider to use Partitioning Try to spread out load but keep the data local Consider that i.e. DB Triggers will be fired on each node
Watch out for High frequently changing DB objects Consider less records per block Less records decreases block concurrency
Less nodes request the same block at a single point in time Use caching
Increase cache size for sequences Pin small tables into the cache
Oracle features ASSM, ASM, Reverse Indexes, Function Based Indexes,
Partitioning Follow best practices
25
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring
System Interfaces Lessons Learned Future plans
Agenda
26
System Monitoring
Joint Team of Admins IBM Tivoli Oracle GRID Control Quest Foglight Experience Monitor Interface monitoring
27
Joint Team of Admins
Joint team of Siebel , EAI and Database Admins Total overview of the system
Database monitoring Siebel view (functional SQL process monitoring) Database view (technical monitoring)
Queue Monitoring (Siebel inbound and outbound)
Technical Monitoring Logs, CPU, Memory, Storage etc.
Propose technical improvements
28
System Monitoring
Joint Team of Admins IBM Tivoli Oracle GRID Control Quest Foglight Experience Monitor Interface monitoring
29
IBM Tivoli
Operational Management System Automation Monitoring (CPU, Memory…)
Workload Scheduling Service Level Advisor
Storage Management: Tivoli Storage Manager SAN Volume Controller
Alerting
30
System Monitoring
Joint Team of Admins IBM Tivoli Oracle GRID Control Quest Foglight Experience Monitor Interface monitoring
31
Oracle GRID Control
RAC monitoring
Long running transactions
Wait States
32
System Monitoring
Joint Team of Admins IBM Tivoli Oracle GRID Control Quest Foglight Experience Monitoring Interface monitoring
33
Quest Foglight Experience MonitorCoverage:
Round about 60 business processes are being monitored (90% coverage)
Various report sets are available to users
Applicable Fields: Monitoring of software quality after
deployment Alarm based control of the
production environment Information base for sizing
recommendations and server consolidation
Acceptance: FxM is accepted as an objective
measuring instrument which reflects “end user reality” and is used to support SLA monitoring and reporting
34
System Monitoring
Joint Team of Admins IBM Tivoli Oracle GRID Control Quest Foglight Experience Monitor Interface Monitoring
35
Interface Monitoring The Telekom Enterprise
Service Bus
36
Interface Monitoring
Evaluation of auditing messages
Monitor traffic and response times on interfaces
37
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans
Agenda
38
Reporting
OA Mirror (Operative analysis)
RO Mirror (Read only)
Logshipment
(arch and redo)
CRM RAC Database
Stage DB
Capture processPropagation process
RO Mirror DBPhysical Standby
read only
Logshipment(arch and
redo)
OA Mirror DB
Apply processes (16)CDC-capture
CDC-apply (8)
Data warehouse
CRM OAReports
39
CRM technical achievements
Reduced Response times Increased Automation Faster time to market and increased flexibility for new products Less development more configuration Service Oriented Architecture is implemented Host platform retired OLAP database with realtime mirroring based on Streams
technology
40
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans
Agenda
41
System Interfaces - ESB
42
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans
Agenda
43
Lessons Learned
Get a 360° view on the system Test with full blown database as early as possible Test the infrastructure with worst case scenarios Test the system infrastructure carefully before roll out Test your migration with real data Always have a look at system and application performance Avoid load mixing of batch and OLTP Keep commit sizes low (Batch Jobs) Keep work on a single node when possible Be prepared for long running queries Slim line tables with highly changing content
44
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans
Agenda
45
Future Plans
Additional customer groups will be integrated and served through the new platform Small business customers Medium business customers
Additional product types will be integrated to process the complete portfolio Internet access and value added services Next generation network services
Service Architecture will be enhanced Additional services will be externalized Further automation improvements Continuous systems consolidation and renewal of IT architecture Upgrades to Oracle 11g and Siebel 8.x
46
Introductions Project Overview & Business Drivers
Scope and Usage of the system Technical Footprint - Handling 25,000 Users
Architecture & Infrastructure Session Management RAC Performance System Monitoring Reporting
System Interfaces Lessons Learned Future plans CRM-T Infrastructure details
Infrastructure in detail
47
CRM-T Infrastructure detailsWeb Server and Reporting
48
CRM-T Infrastructure detailsQuest Monitoring
49
CRM-T Infrastructure detailsObject manager and Batch Server
50
CRM-T Infrastructure detailsEAI Server and ESB
51
CRM-T Infrastructure detailsDatabase
52
CRM-T Infrastructure detailsStorage
53