<Insert Picture Here>
Build an Advanced Oracle E-Business Suite Architecture(S281594 - Technology Track - OOW 2006)
Paul Bramy (Oracle) & Alain Roy (IBM)Oracle/IBM Joint Solutions Center Leaders
This presentation has been built on a real customer case.
It represents a concrete follow up of the JSC presentation delivered last year in OOW ( S1854 - How to implement Oracle E-Business Suite on IBM
eServers )
It aims to present how the Oracle solutions & technology stacks can be leveraged & merged to build an advanced and flexible Oracle E-Business Suite Architecture.
It demonstrates how Thousand Concurrent Users and Massive Reporting with high quality of services has been achieved by optimizing the resources through the virtualization delivered by IBM System p & AIX5L.
This project is also a human story with a strong teaming from Oracle & IBM experts.
Presentation Introduction
<Insert Picture Here>
Agenda
• JSC introduction.
• Project introduction.
• Technical foundations.
• Project presentation.
• Conclusions.
• What is the JSC?
The Oracle/IBM Joint Solutions Center, created in 1998, is a presales team,
permanently staffed by Oracle & IBM, to contribute to the successful delivery of
Oracle/IBM Joint Solutions.
• Why the JSC is unique?
Located inside the PSSC, the IBM’s European Benchmarking and Support
center (PSSC) in Montpellier, France. JSC has access to large amount of
hardware from all the IBM brands (p, x, I, z)
It is staffed by Subject Matter Experts from both IBM & Oracle so we can have
access to the best of our two corporations simultaneously for the benefit of our
joint customers.
Oracle/IBM Joint Solutions CenterJSC Specificities
The living proof of Oracle and IBM’s desire to work closely together
Eight years of experience
What can you do?
Pre Sales Assistance through the whole project/sales cycle
Help me to Convince: Briefings & Conferences (40/year)
Help me to Build: Architecture, Design, Sizing Assistance (300/year)
Help me to Demonstrate: PoC, Benchmarks Assistance (30/year)
Help me to Deliver: hands-on workshops (5/year), Publications (10/year)
Which Products & Solutions are you covering?
Oracle Technologies and Solutions such as Oracle Database & Application
Server (9i, 10g), e-Business Suite 11i on IBM servers System x, System p,
System I & System z.
Expanding since 2006 to PSE, Siebel & JDE. Coming soon Portal Retek.
The living proof of Oracle and IBM’s desire to work closely together
Eight years of experience
Oracle/IBM Joint Solutions CenterJSC Missions
Oracle/IBM Joint Solutions CenterJSC engagement
Q&
A -
Siz
ing
Des
ign
Wor
ksho
psPer
form
ance
Ana
lysi
s
Arc
hite
ctur
e
Impl
emen
tatio
n
Sup
port
Evo
lutio
ns
<Insert Picture Here>
Agenda
• JSC introduction.
• Project introduction.
• Technical foundations.
• Project presentation.
• Conclusions.
Project IntroductionCustomer’s business reasons for change
• Banca Comerciala Romania (BCR) is the biggest Commercial Bank in Romania, having 35% of the Romanian Banking Sector assets andalmost 30% market share. It is the 6th biggest bank in Eastern Europe.
• BCR is part of BCR Financial Group together with BCR Insurance, BCR Asset Management, BCR Financial Investment etc.
• Bank’s business strategy to:
• Remain the no.1 bank on the market by increasing the market share
• Consolidate the capital
• Consolidate the profile of universal bank
• IT Prerequisits to support this strategy:
• Centralized, Integrated
• Technological Consistency
• Client oriented
• Fragmented view of customer , products and accounts information
• Business continuity
• Long cycle “time to market” for new products
• Insular front & back office applications
CHALLENGES
• Centralized Core banking system,meaning:
• Unique customer nomenclature
• Unified products
• Unique account
• 24x7 business availability
• Easy setup of new products
• Capability to setup complex products
• Smooth addition of new channels
WISHES OBJECTIVES
Project IntroductionIT Customer Challenges & Views
• Increased Market Share via new products & delivery channels (Mobile, Internet)
• Increased Customer Satisfaction & Loyalty via High Quality of Services
• Integration of all core solutions
ROLES
• Analysis team leader
• Design and build team
leader
• Analysts
• Technical consultants
RESPONSIBILITIES
• Analyze requirements
• Design, build and unit test
kernel and modules
• Data migration
• System parameterization
Customer Team
Project IntroductionProject teams introduction
ROLES
• Project Director
• System and solution architect
• Business consultant
• Design and build team leader
• Technical consultants
RESPONSIBILITIES
• Design the solution and
technical architecture for the entire application
• Design, build and unit test
kernel and modules
• Install, configure and support Oracle products
Oracle Team
ROLES
• Hardware team leader
• Technical consultants
RESPONSIBILITIES
• Install, configure and
support hardware
equipment
IBM Team
ROLES
• Business requirements
definition team leader
• Business consultants
• Conformity officer
• Training team leader
• Testing team leader
RESPONSIBILITIES
• Gather and formalize business requirements
• Organize training events
• Perform system integration
and acceptance tests
• Produce periodic progress
reports
System Integrator Team
JSC Second Level of Support
<Insert Picture Here>
Agenda
• JSC introduction.
• Project introduction.
• Technical foundations.
• Project presentation.
• Conclusions.
• IBM System p & Power 5
• IBM Storage
• AIX5L
• GPFS
REQUIREMENTS
• 1 to 64 cores
• N°1 on Unix Market
• Performance Leader
• A Clear Roadmap
• Pay as you grow withCapacity Upgrade on Demand
• Virtualization &
Flexibility (CPU, Memory, IO, Network)
• Clustered FileSystemfor Shared Appl_Top
and RAC
PRODUCTS VALUES
Technical foundationsIBM Stacks
• Robust & RecognizedTechnologies
• High Performance
• Scalable Infrastructure
• Resource Optimization
• A Clustered Solution &
FileSystem
Without Virtualization
Jay Kruemcke IBM 2003
• DLPAR with micro-partitioning : Dynamically adjust CPU and memory allocation
• Virtual I/O; Virtual LAN
Test LPAR
Agent
CRM LPAR
Agent
Finance LPAR
Agent
Unbalanced & lost resource
5CPU
6CPU
Agent Agent Agent
Adjusted & Balanced resource
Test LPAR CRM LPAR Finance LPAR
0.5CPU
3CPU
8.5CPU
1CPU
Technical foundationsIBM Stacks: Advanced Virtualization
Key Words :EC for Entitlement Capacity, Capped/Uncapped & Partition Weight
With Virtualization
IBM filesystem designed for cluster use, certified with Oracle RACGPFS is capable of holding the database, redo logs,
control files and archived logs
required by Oracle RAC + Binaries
Data is striped across shared local disks or NSD serversMetadata is maintained by all servers in the cluster
File Locking is distributed across the servers in the cluster
Performing and scalableStriping I/O performance
Direct I/O with close to RAW device performance
Supports terabyte file sizes and multiple terabyte file systems
Highly Available – Fault tolerantSupports RAID subsystems and/or performs own file system replication.
Supports on-line dynamic reconfiguration (add, delete, replace disks and nodes;
rebalance the file system)
HACMP
GPFS
RSCT
GPFS
RSCT
Technical foundationsIBM Stacks: General Parallel File-system (GPFS)
• EBS 11.5.10 (PLSQL & FORMS, WORKFLOW)
• ORACLE 10g DATABASE
• REAL APPLICATION CLUSTER
• DATAGUARD
• XML INTEGRATION
REQUIREMENTS
• UNIFIED DEVELOPMENT FRAMEWORK
• SINGLE DATABASE REPOSITORY
• RELIABILITY & AVAILABILITY
• SECURITY & PERFORMANCES
• QUALITY OF SERVICES
PRODUCTS VALUES
Technical foundationsOracle Stacks
• CORE BANKING & FINANCIALS FOUNDATION
• 11i WORFLOW & MESSAGING
• PARTITIONING
• SCALABILTY, HIGH AVAILABILITY & WORKLOAD ISOLATION
• DISASTER RECOVERY
Technical foundationsOracle Stacks: Oracle E-Business Suite
• 11.5.10.2 with Oracle RAC 10.1.0.4
• Main 11i features used in the project
• Apps Concurrent Management
• Apps Users Management
• Apps Security rules
• 3-tier centralized architecture
• Full use of Oracle Technology stack
• Modular approach ( Financial Modules )
• Exchange of messages in XML format
• Advanced Queue technology
Technical foundationsOracle Stack Highlight : The Specific Banking Kernel
•A Development framework has been built by Oracle Consulting on the top of Oracle E business Suite 11.5.10 foundations
•Kernel Definitions• Ensures a tight integration between
the specific banking softwares, the overall IT system and functional 11i modules.
• Secures the overall security and organizational aspects• Reduces any new definition and/or additional coding.
•Main Kernel features• Kernel Validation Services
Check orders versus Bank’s rules and constraints. • Calculation Services
mostly calculation services for interests, fees, taxes, balances…• Prenotation Services
Create and update prenotations against accounts.• Information Services
Provide information about accounts, balances, prenotations...
Technical foundationsOracle Stack Highlight : The Specific Banking Kernel
Database area
Modules area
Kernel area
Applicationframework
Concurrent Manager Interface
(CM API)Core Kernel Interface
(PL/SQL)
Kernel Service Packages
Transaction Management
General Ledger
Interfaces
Forms Interfaces
Oracle Workflow
Server
Domain Service
Packages
Applications
•Module definitions (packages)
•Data management (tables)
•Transactions Performance
•Security access
•Reliability / Availability
Forms server
Daily Banking activity Official reports ATM integratioB.I activity
GPFS or AMSRAC
SingleSingle
Scalable Secured
Virtualized
Technical foundationsOracle stacks: RAC 10g within EBS environments
Oracle 10G R2 RAC node 1
Oracle 10G R2 Oracle 10G R2
RAC node 1RAC node 1
Oracle EBS11.5.10.2
Prod AS node 1
Oracle EBS11.5.10.2 Oracle EBS11.5.10.2
Prod AS node 1Prod AS node 1
Oracle 10G R2 RAC node 2
Oracle 10G R2 Oracle 10G R2
RAC node 2RAC node 2
Oracle EBS11.5.10.2
Prod AS node 2
Oracle EBS11.5.10.2 Oracle EBS11.5.10.2
Prod AS node 2Prod AS node 2
LPAR# 1.1
LPAR# 1.1
LPAR# 1.2
LPAR# 1.2
LPAR# 2.2
LPAR# 2.2
LPAR# 2.1
LPAR# 2.1
1,7 CPU8 GB
1,7 CPU8 GB
2,5 CPU8 GB
2,5 CPU8 GB
System p5 server 1System p5 server 1
1,1 CPU8 GB
1,1 CPU8 GB
System p5 server 2System p5 server 2
DevDevDev TestTestTestLPAR# 1.3
LPAR# 1.3
1,4 CPU8 GB
1,4 CPU8 GB
LPAR# 2.3
LPAR# 2.3
System p5 server 1System p5 server 1
Oracle 10G R2 RAC node 1
Oracle 10G R2 Oracle 10G R2 Oracle 10G R2
RAC node 1RAC node 1RAC node 1
Oracle EBS11.5.10.2
Prod AS node 1
Oracle EBS11.5.10.2 Oracle EBS11.5.10.2 Oracle EBS11.5.10.2
Prod AS node 1Prod AS node 1Prod AS node 1
LPAR# 1.1
LPARLPARLPAR
# 1.1# 1.1# 1.1
LPAR# 1.2
LPARLPARLPAR
# 1.2# 1.2# 1.2
2,5 CPU8 GB
2,5 CPU2,5 CPU2,5 CPU
8 GB8 GB8 GB
1,1 CPU8 GB
1,1 CPU1,1 CPU1,1 CPU
8 GB8 GB8 GB
DevDevDevDevLPAR# 1.3
LPARLPARLPAR
# 1.3# 1.3# 1.3
Oracle 10G R2 RAC node 2
Oracle 10G R2 Oracle 10G R2
RAC node 2RAC node 2
Oracle EBS11.5.10.2
Prod AS node 2
Oracle EBS11.5.10.2 Oracle EBS11.5.10.2
Prod AS node 2Prod AS node 2LPAR# 2.2
LPAR# 2.2
LPAR# 2.1
LPAR# 2.1
2,7 CPU8 GB
2,7 CPU8 GB
TestTestTest
2,4 CPU8 GB
2,4 CPU8 GB
LPAR# 2.3
LPAR# 2.3
<Insert Picture Here>
Agenda
• JSC introduction.
• 11i Oracle/IBM Customer examples.
• Project introduction.
• Technical foundations.
• Project presentation.
• Conclusions.
Project presentation Project challenges through its Numbers
SOLUTION
Branches: 310
Named users: 9783
Conc.users/day: 6530
Accounts: 25M
Orders/day: 500k
Reports/day : 400k
Database size:2.2 TB
Largest table: 1B rows
16.000 DB session peak
Massive OLTP Massive Online Reporting
Large Database
(Day) Opened hours (8AM – 6PM)
Project presentation Project phases: Most important steps
period
Main events
June 2006 Oct 2006
Database size1.2 TB
Database size2.1 TB
250 branchesConnected
New HeadQuarter
activity
310 branchesConnected
Go Live
PlannedATM
integrationFirst Branch
Pilot
250K ordersPer day
400K ordersPer day
D/R reflexion
New Banking
Applications
Nov 2005 Dec 2005 May 2006
Roll up waves
End
Every weekAddiitional branches
Single Datacenter
Two DP rooms *
Clustered IBM System
p p595 running AIX
IBM Storage models
DS8100
Project presentationDatacenter: Physical layout
Remote Branches
And HeadQuarter
IBM IBM
Global networks
IBM IBM
Production Room D/R Room
Routers Routersfirewalls
WAN
Project presentationBusiness area coverage…
• Three type of performance requirements for Three type of business activities
• The Daily activity: representing the opened period of the bank and the OLTP activity at the application level. This period covers each day of the week starting from 8:30 am to 6:30 pm.
• The Daily Over-Night: representing the batch called « End of day » and some additional tasks ( reports + backups). This period covers each night of the week starting at 6:30 pm to 6:30 am the next day.
• The End of month: representing a specific period of the month (last day of the month) starting from 6:30 pm to 6:30 am the next day.
07:02:28
07:40:29
08:18:30
08:56:31
09:34:32
10:12:33
10:50:33
11:28:34
12:06:35
12:44:36
13:22:36
14:00:37
14:38:38
15:16:39
15:54:39
16:32:40
17:10:41
17:48:42
18:26:42
19:04:43
19:42:44
20:20:45
20:58:46
21:36:47
22:14:47
22:52:48
23:30:49
00:08:50
00:46:51
01:24:52
02:02:52
02:40:53
03:18:54
03:56:55
04:34:55
05:12:56
05:50:57
06:28:57
Pro
ject p
resen
tatio
n24 h
ou
rs S
ys
tem
beh
avio
r… (G
lob
al C
PU
activ
ity )
OLT
P D
aily
activ
ity
Daily
batc
h a
ctiv
ity
Project presentationProduction Products Overview
Partition A Partition B
Partition 1 Partition 2
Partition C Partition D
Partition 3 Partition 4
P595 – 32 Ways (24+8) runningAIX5.3
P595 – 32 Ways (24+8)
IBM DS8100
GPFS
Oracle 10gRAC
DB01 DB02 DB03 DB04
(Forms+
Report)
(Forms) (Forms+
Report)
(Forms)
Datafiles
APP01 APP02(Forms+
Report)
(Forms) (Forms+
Report)
(Forms)
Shared APPL_TOP
APP03 APP04
Project presentationProduction Server Overview
DB03
(Forms)
DB04
(Forms+Report)
P595 – 32 Ways (24+8)
APP04
(Forms +Report)
Partition D
CE = 8VP=18
92GB RAMWeight=250
SMT=ON
APP03
(Forms)
Partition 4
CE = 6VP=12
55GB RAMWeight=128
SMT=ON
Partition 3
CE = 6VP=12
55GB RAMWeight=128
SMT=ON
Partition C
CE = 8VP=18
92GB RAMWeight=250
SMT=ON
Shared Pool of 32 Physical ProcessorsMin, Max and Desired
processing units of
capacity
(Capacity Entitlement)
Total CE = 28
4 CPU not alocated
to a specific partition
Min, Max and
Desired number of Virtual
processors : the whole
number of concurrent operations that
the partition can use.
When a partition is running an uncapped mode an uncapped weight should be indicated.
SMT
Simultaneous MultiThreading
Two threads are opened on a Power 5
processor
Project presentationDaily activity – CPU and memory utilization (APPS tier)
0
100
200
300
400
500
600
700
800
07
:30
07
:56
08
:22
08
:48
09
:14
09
:40
10
:06
10
:32
10
:58
11
:24
11
:50
12
:16
12
:42
13
:08
13
:34
14
:00
14
:26
14
:52
15
:18
15
:44
16
:10
16
:36
17
:02
17
:28
17
:54
18
:20
Production Cluster – Server number 2 – Application Partitions APP03 and APP04
CPU utilization
APP04(Forms + report)
APP03
(Forms)
(C.E)Theorical
Partition Capacity
Activity peak
> 100% CE
Memory consumption(free memory)
0
5
10
15
20
25
30
35
40
45
07:3
0
07:5
4
08:1
8
08:4
2
09:0
6
09:3
0
09:5
4
10:1
8
10:4
2
11:0
6
11:3
0
11:5
4
12:1
8
12:4
2
13:0
6
13:3
0
13:5
4
14:1
8
14:4
2
15:0
6
15:3
0
15:5
4
16:1
8
16:4
2
17:0
6
17:3
0
17:5
4
18:1
8
1650 conc/active usersper application partitions
Average 12MB / user
CM + Report processes = 5GB
0
200
400
600
800
1000
1200
07:3
0
08:0
2
08:3
4
09:0
6
09:3
8
10:1
0
10:4
2
11:1
4
11:4
6
12:1
8
12:5
0
13:2
2
13:5
4
14:2
6
14:5
8
15:3
0
16:0
2
16:3
4
17:0
6
17:3
8
18:1
0
Project presentationDaily activity – CPU and memory utilization (DB tier)
Production Cluster – Server number 2 – Database Partitions DB03 and DB04
CPU utilization
DB04(Forms + report)
DB03
(Forms)
(C.E)Theorical
Partition Capacity
Activity peak
> 100% CE
Memory consumption(free memory)
Average 10 MB / user0
5
10
15
20
25
30
35
07
:30
07
:58
08
:26
08
:54
09
:22
09
:50
10
:18
10
:46
11
:14
11
:42
12
:10
12
:38
13
:06
13
:34
14
:02
14
:30
14
:58
15
:26
15
:54
16
:22
16
:50
17
:18
17
:46
18
:14
Se
rve
r 2( A
pp
03+
AP
P04+
DB
03+
DB
04 )
Glo
ba
l CP
U
us
ag
e:
87
.5 %
Pro
ject p
resen
tatio
nD
aily
activ
ity –
Serv
er 2
-G
lob
al C
PU
utiliz
atio
n
0
200
400
600
800
1000
1200
07:30
08:04
08:38
09:12
09:46
10:20
10:54
11:28
12:02
12:36
13:10
13:44
14:18
14:52
15:26
16:00
16:34
17:08
17:42
18:16
0
500
1000
1500
2000
2500
3000
07:30
08:04
08:38
09:12
09:46
10:20
10:54
11:28
12:02
12:36
13:10
13:44
14:18
14:52
15:26
16:00
16:34
17:08
17:42
18:16
DB
04
DB
03
AP
P04
AP
P03
Project presentationCPU utilization summary : 24 hours period
Partitions CPU ( %user +%sys) Free memory
App01
App02
App03
App04
DB01 75 to 80%
DB02 15% (steady)
DB03 60 + peaks 100%
DB04 90 to 100%
The system is rather flat and low.
No important activity.
More free memory than for the daily activity.
Partitions CPU ( %user +%sys) C.E reached vs affected Free memory
App01 80% 5/6 20 GB
App02 80 to 100% 5 / 6 + peaks > 6 15 GB
App03 80 to 90% 4.5 to 5 / 6 20 GB
App04 95 to 100% 5.5 / 6 15 GB
DB01 75 to 80% 6 to 7 / 8 10 GB
DB02 90% 7 / 8 + peaks > 8 10 GB
DB03 75 to 80% 7/8 15 GB
DB04 90 to 100% 6 to 8 / 8 7 GB
CPU and memory consumption of each partition during the daily activity
CPU and memory consumption of each partition during the End of Day activity
Project description Evolutions and Future integrations
• Integrate Headquarter Workload: +5% active users increase but with very specific & strategic type of business requests ( BI, reports, etc..) -> Done…
• Integrate Card system: a critical part of the global system. Current estimations is +200k orders daily. Proof of concept (POC) proposed to evaluate as precisely as possible the impact of the Card System on the current architecture.
• Add contingency ressources: Add Ressources or Nodes to reduce impact of any system failure.
• A Disaster Recovery Site: Study & Design to be conducted for a D/R solution leveraging Dataguard. Potential Stretched RAC.
Project description Card System integration… ( Study in progress)
DB03 DB04DB02DB01 IBM WebsphereMiddle tier
Kernel
IBM System i
Existing Core banking architecture
DB03 DB04DB02DB01
Kernel
DB05IBM
WebsphereMiddle tier
RAC
DB03 DB04DB02DB01 DB05DB06
IBM
W
eb
sp
he
reM
idd
le tie
r
IBM
W
eb
sp
he
r eM
i dd
le t
ier
RACKernel Kernel
Existing Core banking architecture
Extended Core banking architecture
ATMSystems
<Insert Picture Here>
Agenda
• JSC introduction.
• 11i Oracle/IBM Customer examples.
• Project introduction.
• Technical foundations.
• Project presentation.
• Conclusions.
Oracle is the Applications Market Leader
IBM is the Infrastructure Market Leader
IBM & Oracle in clear and deep Partnership
Oracle & IBM Extended Virtual joint team
Oracle & IBM Long Term Roadmaps
11
Why this solution? — Top 10 Reasons
22
33
44
55
11
22
33
44
55
Oracle E-business suite flexibility
Oracle 10gRAC capabilities
IBM Power 5 features
IBM Advanced virtualization
Oracle and IBM skilled teams
Technology reasons Strategic reasons
<Insert Picture Here>
“Sibcor este unul dintre cele maicompetitive sisteme bancare create
special pentru prima banca de pepiata romaneasca impreuna cu
Oracle; un proiect de mare succes
de care suntem mandri.”
BCR Quote Radu Varduca, IT Manager
Sibcor is one of the most custom developed competitive Core Banking system by Oracle specialy taylored for the no 1 bank in Romania;
we are very proud of our system.
Anca Cont
Catalin BogdanDaniela ZupiniDanut DimaFabienne LepetitFrederic Michiara
Gabriel GheorghiuIonut EnusNadia BendjedouNicoleta MacoveiMihai Teodor Popescu
Octavian Lascu…/…
and all our colleagues involved during this project…
Many thanks to Put it all together…. Again…