Deploying E-Business Suit 11i on Oracle RAC 11g on AIX Oracle OpenWorld September 2010.

Deploying E-Business Deploying E-Business Suit 11i on Oracle RACSuit 11i on Oracle RAC

11g on AIX11g on AIX

Oracle OpenWorldOracle OpenWorldSeptember 2010September 2010

IntroductionIntroduction

AgendaAgenda

• Bunnings at a glance Bunnings at a glance • Our Business Benefits of RAC Our Business Benefits of RAC • Infrastructure Capacity ManagementInfrastructure Capacity Management• Initial Build and Deployment ExperienceInitial Build and Deployment Experience• Our Build and DesignOur Build and Design• Preparation and Preparation and pre-requisitespre-requisites• Patching and Installation Patching and Installation • RAC conversion RAC conversion

– RDBMSRDBMS– Required Mid-tier changes Required Mid-tier changes

• OEM ConfigurationOEM Configuration• Issues and ResolutionIssues and Resolution• Useful References Useful References • Next stepsNext steps• Q&AQ&A

WIDEST RANGE

LOWEST PRICES

BEST SERVICE

0

1000

2000

3000

4000

5000

6000

7000

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Sales

Sales ($m)

Sales = IT ChallengesSales = IT Challenges

Our Business Benefits of RACOur Business Benefits of RAC

• Ensuring business core applications remain and have Ensuring business core applications remain and have the needed Flexibility and Agilitythe needed Flexibility and Agility

• High Availability 24x7 Australia and New Zealand High Availability 24x7 Australia and New Zealand OperationsOperations

• Deliver on Performance Deliver on Performance • Provide a level of Workload SegregationProvide a level of Workload Segregation

““Our Mission critical systems must be agile to support our Our Mission critical systems must be agile to support our business by improving operational effectiveness as well business by improving operational effectiveness as well as supporting decision making which is based on timely, as supporting decision making which is based on timely,

accurate and flexible information”accurate and flexible information”

User UniquenessUser Uniqueness

• PDT – orderingPDT – ordering

• Reporting – DiscovererReporting – Discoverer

• Store SupportStore Support

– FormsForms

– Self-serviceSelf-service

– Custom Forms and Self-serviceCustom Forms and Self-service

• StoresStores

– MC-75 PDA/PDT running self-serviceMC-75 PDA/PDT running self-service

– Touch screen and pen inputTouch screen and pen input

– WIFI and 3G WirelessWIFI and 3G Wireless

• Live reporting via XML Publisher and Discoverer 10gLive reporting via XML Publisher and Discoverer 10g

Our Mobile Procurement SolutionOur Mobile Procurement Solution

Infrastructure Capacity ManagementInfrastructure Capacity ManagementN

o. T

ran

sact

ion

s P

er M

on

th

Transactions Processed

** Chart is actual data from 2002 to 2009, covering total Transactions for Year on Year growth ** Chart is actual data from 2002 to 2009, covering total Transactions for Year on Year growth

Infrastructure Capacity ManagementInfrastructure Capacity Management

• Information Lifecycle Management is part of the solutionInformation Lifecycle Management is part of the solution

• EBS partitioning needs to be done early otherwise EBS partitioning needs to be done early otherwise special issues come up if regular purging cannot be special issues come up if regular purging cannot be undertaken due to business requirementsundertaken due to business requirements

– Inventory, GL, AP, AR, PO can grow quickly Inventory, GL, AP, AR, PO can grow quickly depending on usedepending on use

– Moving data in live systems is trickyMoving data in live systems is tricky

Metalink article 554539.1 (Using Database Partitioning with Oracle E-Business Suite )Has all the information needed to implement partitioning in EBS

Infrastructure Capacity ManagementInfrastructure Capacity Management

• RAC - scales very well… but some caveats!RAC - scales very well… but some caveats!

– OLTP/Warehouse type queries, Global Cache Service OLTP/Warehouse type queries, Global Cache Service (GCS) contention(GCS) contention

– Concurrent gatherstats across nodes can cause lock Concurrent gatherstats across nodes can cause lock upsups

– Any contention in a single node environment Any contention in a single node environment magnified in multi-node RACmagnified in multi-node RAC

Initial Build and Deployment ExperienceInitial Build and Deployment Experience

Full System Configuration:Processors – 58x 1.65GHz P5 (595)Memory – 512Gb RAM

11.5.10Application Server

11i EBS Database

Production Initial Phase :Our Early and Rapid

Deployment

Production Second Phase :

BLA

DE

(2x

3.6

Ghz

EM

64

T4

8G

b R

AM

)

BLA

DE

(2x

3.6

Ghz

EM

64

T4

8G

b R

AM

)

BLA

DE

(2x

3.6

Ghz

EM

64

T4

8G

b R

AM

)

11i EBS Database - RAC Node 3




BLA

DE

(2x

3.6G

hz E

M6

4T

48G

b R

AM

)

BLA

DE

(2x

3.6G

hz E

M6

4T

48G

b R

AM

)

BLA

DE

(2x

3.6G

hz E

M6

4T

48G

b R

AM

)

BLA

DE

(2x

3.6G

hz E

M6

4T

48G

b R

AM

)

Clustered with Jserv& Servlet mode

Clustered with 0C4J

BLA

DE

(2x

3.6

Ghz

EM

64

T4

8G

b R

AM

)

EBS 11i Web Cache 10.1.2.3.0

EBS 11i Forms, Reports & Concurrent Managers



BLA

DE

(2x

3.6G

hz E

M64

T48

Gb

RA

M)

BLA

DE

(2x

3.6G

hz E

M64

T48

Gb

RA

M)

BLA

DE

(2x

3.6G

hz E

M64

T48

Gb

RA

M)

EBS 11i Forms, Reports & Concurrent Managers




BLA

DE

(2x

3.6G

hz E

M64

T48

Gb

RA

M)

BLA

DE

(2x

3.6G

hz E

M64

T48

Gb

RA

M)

BLA

DE

(2x

3.6G

hz E

M64

T48

Gb

RA

M)

BLA

DE

(2x

3.6G

hz E

M64

T48

Gb

RA

M)

Clustered with Jserv& Servlet mode

Clustered with 0C4J

BLA

DE

(2x

3.6G

hz E

M64

T48

Gb

RA

M)

EBS 11i Web Cache 10.1.2.3.0



Current Production – Physical Representation

RAC1

Radware Web Accelerator

Oracle Web Cache Oracle Web Cache

Clustered with OC4J

LBR with cookie

persistence

Form S2 Report S2Forms S1 Report S1

Concurrent Mgr 1 Concurrent Mgr 2

RAC2

(OLTP) (OLTP)

Clustered with Jserv/servlet mode

PCP

Primary/Secondary

Application Layer Linux

RH5

DatabaseLayer

AIX 5.3-08

Application Layer Linux

RH5

RAC3

(Read-Mostly)

Our Build and DesignOur Build and Design


• Our design and build phase experiencesOur design and build phase experiences

– Don’t try to do it all at onceDon’t try to do it all at once• Implement in multiple steps and give time to bed in changes and Implement in multiple steps and give time to bed in changes and

gauge their full impact before moving ongauge their full impact before moving on

– Follow stream deployment and testing strategyFollow stream deployment and testing strategy

• Start in your Sandpit/Test environments to work out the specific Start in your Sandpit/Test environments to work out the specific requirements of your environmentrequirements of your environment

• Most systems are integrated, CIT and SIT testing is next.. Most systems are integrated, CIT and SIT testing is next.. Beware of TNSNAME changes during RAC conversion for DB Beware of TNSNAME changes during RAC conversion for DB LinksLinks



– Test thoroughlyTest thoroughly

• Don’t rush testing and thoroughly regression and load test Don’t rush testing and thoroughly regression and load test the new architecture, some issues only come out under load the new architecture, some issues only come out under load with custom codewith custom code

Testing ApproachTesting Approach

EBS Backend Load TestsEBS Backend Load Tests

•Swingbench ArchitectureSwingbench Architecture

•Custom class extensions to enable Custom class extensions to enable more controlmore control

•Junit integration with Swingbench to Junit integration with Swingbench to allow OA Framework testingallow OA Framework testing

Cluster Controller

Node 1 Node 3Node 2

EBS Environment

Co-ordinator

Workload simulated can include multiple concurrent transactions like TPC, Concurrent jobs and stored

proceedures

• Infrastructure testing using SwingBench bots deployed across our Infrastructure testing using SwingBench bots deployed across our varied network topology to calculate latency and load. varied network topology to calculate latency and load.

• Packeteer was installed at end points to enable granular monitoring of Packeteer was installed at end points to enable granular monitoring of

network utilisation. network utilisation.

NextG

3G

BOT(xN)

BOT(xN)

StoreWiFi

EBS Open Interface

BOT(xN)

BOT(xN)

Existing

Testing ApproachTesting Approach


– First transformation was to move to ASM 11g/CRS11g First transformation was to move to ASM 11g/CRS11g 10g database10g database

• Requires a patch to solve locking issue when changes Requires a patch to solve locking issue when changes happen in ASMhappen in ASM

– Second transformation was to upgrade 10g RDBMS Second transformation was to upgrade 10g RDBMS to 11g RDBMSto 11g RDBMS

• Use best practices, prepare for changes in execution plans, Use best practices, prepare for changes in execution plans, baseline 10g system and compare, make sure you are fully baseline 10g system and compare, make sure you are fully patched up on 11gpatched up on 11g


Metalink document E-Business Suite 11i on RAC (Configuring Database Load balancing & Failover )ID 294652.1] and Metalink note 244366.1.


– Third transformation was to go to RAC with 3 nodesThird transformation was to go to RAC with 3 nodes• More reliability and less impact if loss of one nodeMore reliability and less impact if loss of one node• Better ability to distribute workloadBetter ability to distribute workload• Mixed workload OLTP/ReportingMixed workload OLTP/Reporting• Discoverer and other reporting tools can be segregated to Discoverer and other reporting tools can be segregated to

specific nodesspecific nodes



• Specialise workloads with RAC managed services and E-Specialise workloads with RAC managed services and E-Business suite application profile settings (load direction)Business suite application profile settings (load direction)

– We plan Active/asymmetric service descriptions, 2 We plan Active/asymmetric service descriptions, 2 nodes per service for HAnodes per service for HA

– TNSNAMES and JDBC are used for external access TNSNAMES and JDBC are used for external access such as reporting tools (Discoverer, Cognos, Portal) to such as reporting tools (Discoverer, Cognos, Portal) to isolate workloadsisolate workloads



• Specialise workloads with RAC managed services and E-Specialise workloads with RAC managed services and E-Business suite application profile settings (load direction)Business suite application profile settings (load direction)

–Resource Manager Consumer Groups and Mapping are Resource Manager Consumer Groups and Mapping are used to manage resource utilisation which are then used to manage resource utilisation which are then mapped to servicesmapped to services

–Can go to application level via application profiles, dbc Can go to application level via application profiles, dbc and session_cookie_name settings which can be and session_cookie_name settings which can be changed to match RAC services changed to match RAC services



Preparation and pre-requisitesPreparation and pre-requisites

• The minimum software levels for various EBS modules The minimum software levels for various EBS modules can be found on Metalink. can be found on Metalink.

Key PatchesKey Patches– Minipack 11i.AD.I.5 (233044.1)Minipack 11i.AD.I.5 (233044.1)– 11GR1 APPS Interoperability Patch (6328363) 11GR1 APPS Interoperability Patch (6328363) – 11i.ATG_PF.H RUP6 – (5903765) 11i.ATG_PF.H RUP6 – (5903765) – 11.5.10:SFM Unable to Process Orders in RAC 11.5.10:SFM Unable to Process Orders in RAC

Config (4022732) ( this can be applied before setting Config (4022732) ( this can be applied before setting up Oracle RAC) up Oracle RAC)

Metalink document : Using Oracle Real Application Clusters and Automatic Management with Oracle e-Business Suite Release 11i and Oracle 11g. Metalink note 455398.1Minipack 11i AD.1.5, Metalink note 233044.1

Patching and InstallationPatching and Installation

• Follow best practices for RAC and 11gFollow best practices for RAC and 11g– Separate homes for CRS, ASM, DBSeparate homes for CRS, ASM, DB– Patch up each component to latest patchset available Patch up each component to latest patchset available

on 11g starting with Clusterwareon 11g starting with Clusterware– Check key Metalink documents for known Check key Metalink documents for known

performance patches/issues with EBS 11iperformance patches/issues with EBS 11i– Shared file systems for directory objects (output Shared file systems for directory objects (output

directories), OCFS2, NFS, IBM GPFSdirectories), OCFS2, NFS, IBM GPFS– RAC memory overhead over single instance needs to RAC memory overhead over single instance needs to

be considered (about 10% more for buffer cache and be considered (about 10% more for buffer cache and 15% for shared pool)15% for shared pool)

Review the two key Metalink notes, 950896.1 and 244040.1 for known performance patches/issues with EBS 11i

RAC Conversion RDMBSRAC Conversion RDMBS

• CRS installation on each nodeCRS installation on each node• ASM conversion if requiredASM conversion if required• RAC conversion (follow Metalink)RAC conversion (follow Metalink)• Edit ConvertToRAC.xml Edit ConvertToRAC.xml

– Use RConfig utility and plan a few dry runsUse RConfig utility and plan a few dry runs– NetCA listener reconfiguration for cluster (RDMBS NetCA listener reconfiguration for cluster (RDMBS

level)level)– UNDO and TEMP tablespaces UNDO and TEMP tablespaces

• FLASH area may need adjustingFLASH area may need adjusting

– Enable AutoConfig on all database nodesEnable AutoConfig on all database nodes

EBS to RAC Conversion, Metalink note 455398.1 and 455398.1

RAC Conversion RDMBSRAC Conversion RDMBS

• StandbyStandby– Archive logs generated from each instanceArchive logs generated from each instance– Archive logs name changeArchive logs name change– RMAN scripts may need to be modifiedRMAN scripts may need to be modified

• Cloning implicationsCloning implications– RAC to RAC cloning, multiple to single node RAC or RAC to RAC cloning, multiple to single node RAC or

multiple to multiple adds complexity and will be multiple to multiple adds complexity and will be dependant on sitedependant on site

– Rapid Clone, Oracle Application Management Pack, Rapid Clone, Oracle Application Management Pack, manualmanual

– IBM SVC Metro Mirror and Flash-Copy, block level IBM SVC Metro Mirror and Flash-Copy, block level copying (high speed cloning)copying (high speed cloning)

Cloning Considerations and implications , refer to details on Metalink note 760637.1

RAC Conversion Mid-Tier changesRAC Conversion Mid-Tier changes

• Auto configuration, get ready to run it a few times on all Auto configuration, get ready to run it a few times on all nodes!nodes!

• Implement Load Balancing using context editor by pointing Implement Load Balancing using context editor by pointing to correct balance listenersto correct balance listeners– Tools OH TWO_TASK (s_tools_twotask)Tools OH TWO_TASK (s_tools_twotask)– iAS OH TWO_TASK (s_weboh_twotask)iAS OH TWO_TASK (s_weboh_twotask)– Apps JDBC Connect Alias (s_apps_jdbc_connect_alias)Apps JDBC Connect Alias (s_apps_jdbc_connect_alias)

• Parallel Concurrent Processing configurationParallel Concurrent Processing configuration– In multiple application tier onlyIn multiple application tier only

• Transaction managersTransaction managers

Known Issue, Service Manager Processes Are Not Starting After A Restart - usdsop cannot redirect standard output, Metalink note 733901.1

OEM configurationOEM configuration

• Oracle Enterprise Manager is an essential part of our Oracle Enterprise Manager is an essential part of our administration, control and support structureadministration, control and support structure

• Moving the instance from non-RAC to RAC had some Moving the instance from non-RAC to RAC had some impactsimpacts– Need to re-register the instance as RACNeed to re-register the instance as RAC– User Defined Metrics need to be moved from the User Defined Metrics need to be moved from the

instance level to Cluster level, we have a lot of these instance level to Cluster level, we have a lot of these and process was manualand process was manual

– Challenges in doing live performance management with Challenges in doing live performance management with OEM in RAC contextOEM in RAC context

• need to hone in with SQL scripts looking at session information need to hone in with SQL scripts looking at session information combined with various GV$viewscombined with various GV$views

• drill down in OEM goes from cluster to instance to session view drill down in OEM goes from cluster to instance to session view which makes it difficult to pinpoint cross instance locking => which makes it difficult to pinpoint cross instance locking => identifying library cache pin locks is a good exampleidentifying library cache pin locks is a good example

Issues and ResolutionIssues and Resolution

• Initial issues identified in testing addressed with patchesInitial issues identified in testing addressed with patches• Unique environment challenges Unique environment challenges • Testing stream alignment (UAT/SIT/CIT/SP)Testing stream alignment (UAT/SIT/CIT/SP)

– extra infrastructure required to reproduceextra infrastructure required to reproduce– 1 node RAC can be used for development and 1 node RAC can be used for development and

preliminary test environmentspreliminary test environments– Right sizing test data and environmentsRight sizing test data and environments– UAT must be sized to an adequate ratio to production UAT must be sized to an adequate ratio to production

to outline issues in contention and scalingto outline issues in contention and scaling

Next StepsNext Steps

• Extending EBS Environment ExternallyExtending EBS Environment Externally– OLMOLM– Procure to PayProcure to Pay– Credit Card (own - closed loop)Credit Card (own - closed loop)

• Further Growing Reporting RequirementsFurther Growing Reporting Requirements– Improve PerformanceImprove Performance– PartitioningPartitioning

• Continue with archiving Continue with archiving

Useful ResourcesUseful Resources

Oracle E-Business Suite Release 11i High Availability Documentation Roadmap – Metalink note 1072807.1

Oracle Applications Release 11i with Oracle 11g Release 1 (11.1.0) –Metalink note 452783.1 - Interoperability Notes

Using Oracle Real Application Clusters and Automatic Storage Management with Oracle E-Business Suite Release 11i and Oracle Database 11g – Metalink note 455398.1

John [email protected]

THANK YOUTHANK YOU

Date post:	30-Dec-2015
Category:	Documents
Upload:	sydney-beasley
View:	224 times
Download:	1 times

Deploying E-Business Suit 11i on Oracle RAC 11g on AIX Oracle OpenWorld September 2010.

Documents