Unclassified © 2019 FUJITSUsame time, we are announcing Premier Support for Oracle E-Business Suite...

© 2019 FUJITSUUnclassified

1 © 2019 FUJITSUUnclassified

Large Scale EBS 12.2 Upgrade: Licking the Wounds

Michael Salt , Oracle Solution Architect

UKOUG

Applications Unlimited 2019

2

Safe Harbour Statement

“The views expressed in this presentation are

those of the author and do not necessarily reflect

the views of Fujitsu U.K. nor Oracle Corporation”

UKOUG Applications Unlimited 2019

3

About Me…

24 Years Oracle Experience (v7+)

• Developer

• DBA / E-Biz DBA

• Oracle Solution Architect

• E-Business Suite Consultant

21 Years E-Business Suite Experience (10.5+)

21 Years Consultancy Experience:

• Sun Microsystems

• Fujitsu

DBA 1.5

SIG Attendance:

• RDBMS

• RAC

• Application Server

• E-Biz

Regular Presenter At UKOUG Annual Conference

Fujitsu Distinguished Engineer (2018)


4

About FUJITSU . . . Where we work

Approximately 140,000 Fujitsu colleagues working with customers in over 100 countries (March 2018)

*EMEIA : Europe,Middle East,India and Africa.

Oceania3,000

EMEIA*

27,000

Asia12,000

Japan92,000

Americas6,000


5

About FUJITSU . . . Working with world class partners


6

About FUJITSU . . . Working with world class customers

Public Sector Manufacturing Communications Health

Retail Financial Services Transport & Logistics Utilities


http://www.morphyrichards.co.uk/

© Copyright FUJITSU 20197

Oracle Infrastructure and Applications

Simplify Adopt Innovate

Fujitsu Multi Cloud Services

Oracle IAAS and PAAS Partner of the Year 2019

Oracle ACS and Premier Support Partner Of The Year

2019

Oracle System Partner of the Year 2019

8 UKOUG Applications Unlimited 2019

Introduction & Background

12.2


Possible Reasons for Upgrading to 12.2.X ?

Regulatory Reasons

New Product Features

New Technical Features

Better Look and Feel

Longer Term Support

Integration with Fusion Apps / SaaS

Fun of the Challenge

Applicable to our client Not Applicable to our client

AKA Online Patching!!

Enterprise Command Centres

HTML / Customisable


Roadmap from Oracle (July 2017)

https://www.oracle.com/assets/ebs-roadmap-4739519.pdf


Roadmap from Oracle (October 2018)

http://www.oracle.com/us/products/applications/ebs-suite-12-2-announcement-5172019.pdf


Implications of New Roadmap

http://www.oracle.com/us/products/applications/ebs-suite-12-2-announcement-5172019.pdf

Oracle is today announcing a “Continuous Innovation” release model for Oracle E-Business Suite 12.2,

delivering ongoing applications and underlying technology stack updates without a major upgrade. At the

same time, we are announcing Premier Support for Oracle E-Business Suite 12.2 through at least 2030. This

means that Oracle E-Business Suite customers already running on 12.2 can continue to access new

applications functionality and separately uptake later versions of underlying technology for years to come,

without the time and expense of a major release upgrade.


How it Works . . .


Part 1: The Upgrade Process

(Non-Technical)

12.2


Client Architecture (Production, Simplified)

• FUJITSU M10-4S

• 74 x 3GHz SPARC64™ X cores

• 2TB RAM

• Big SAN (not shown)

• FUJITSU M10-1

• 5 x 2.8GHz SPARC64™ X cores

• 150GB RAM

• ORACLE RDBMS 11.2.0.5

• EBS v12.1.3 Conc Manager Tier

• EBS v12.1.3 “Middle Tier”

• Large User Base


Client Upgrade Path

Developer 10.1.2

COMMON_TOP

APPL_TOP

INST_TOP

AS 10.1.3

Applications Tier

v12.1.3

Developer 10.1.2

COMMON_TOP

APPL_TOP

INST_TOP

Applications Tier

V12.2.X

Oracle HTTP Server (OHS)

WebLogic Server (WLS)

Fusion Middleware 11g

DB UPGRADE?


Upgrade Approach: Phase A (Proof of Concept)

Iteration A1

Take Recent Clone of Production

Follow the Oracle Upgrade Documentation

Document as you go!

Fix / Resolve Issues as you go!

Iteration A2 . . An

Take a new Clone of Production

Follow your own documentation

Fix / Resolve Issues as you go!

Refine Documentation

Hand-over for Functional Testing

Formulate List of Patches

Update Master Documentation

Our Appendix!!


Upgrade Approach: Phase B (Refinement & Timing)

Review Oracle Updates

New Patches & Documents

Iteration B1

Take new Clone of Production

Use Production-Like Hardware

Use Different Resources

Follow & Refine own documentation

Conduct Performance Testing

Iteration B2 . . Bn

Repeat of above until clean runs achieved

Add all timing information to documentation

Tune the migration process!!

Hand-over for Functional Testing


Upgrade Approach: Phase C (Dress Rehearsals)

Plan for Production-Like Upgrade Schedule

Iteration C1

Take new Clone of Production

Use Production-Like Hardware

Use Multiple Resources

Follow your own Documentation

Verify Timings in Documentation

Refine documentation

Iteration C2 . . Cn (optional)

Repeat of above until clean runs achieved

Establish Final Baseline (no more patches)

Last iteration should be done 24x7

Hand-over for Final Functional Testing


Upgrade Approach: Phase D (The Real Thing)

Perform Pre-Upgrade Steps in Production

Shut down Production

Back up Production

Perform Upgrade Steps

Use Multiple Resources (24x7 Tag Teams)

Follow your own documentation

Internal Testing & Backup

Technical & Functional

Take Backup / Snapshot

External User Testing

Small set of Functional Product Champions

Go / No-Go!


Upgrade Gotchas Part 1

Process Can Take Many Months

Oracle Things Change Frequently

AD/TXK DELTA Patches

Release Update Packs (RUPs)

Upgrade Patches

Database Patches

Application Patches

Etc etc

Project Scope Creep

New Products Added

Database Upgrade

Project Resources

OBSOLETE!

OBSOLETE!

OBSOLETE!

OBSOLETE!

OBSOLETE!

CLIENT DECISION


Upgrade Gotchas Part 2

Staging Software Area – Invalid Symbolic Link

References to “/d1/R12/ab/apps/V122X39”

Fixed manually

Too Much Memory (yes, I said too much!!)

If >2TB Free Space under “/tmp” it causes problems!

Problem with old 10.1 Oracle Universal Installer, used by:

• Rapid Wizard Installer (fails after several hours)

• Adop “fs_clone” Operations (fails after an hour)

applmgr> $ORACLE_HOME/oui/bin/runInstaller -help

Preparing to launch Oracle Universal Installer from /tmp/OraInstall . . . Please wait ...

applmgr> echo $?

255

applmgr> export TEMP=/var/tmp ## Required to fix 10.1 OUI

### Now re-check OUI

applmgr> $ORACLE_HOME/oui/bin/runInstaller -help

Preparing to launch Oracle Universal Installer from /tmp/OraInstall . . . Please wait ...

Oracle Universal Installer, Version 10.1.0.6.0 Production

. . .

applmgr> echo $?

0

Problem!

OK


Moving Goalposts – Some Things to watch for . . .

Changes To Watch For . . .Migration Phase

A B C D

Upgrade Documentation (1320300.1 / 133070x.1 / 1349240.1 / E48839-nn / etc) ?

Latest JDK Updates (1530033.1 / 1439822.1)

Consolidated Update Patch (CUPn – Section 3 of 1320300.1) ?

Pre-Install Patches Report [Quarterly] (1448102.2) ?

Latest Release Update Pack (RUP 12.2.n - Section 2 of 1320300.1) ? ?

Latest AD/TXK Delta Rollup Patches (1583092.1 / 1617461.1)

Database PSUs & Overlay Patches (1147107.1 / ETCC Patch 17537119)

Latest Patches and Technology Bug Fixes (1594274.1 / ETCC Patch 17537119)

VERY COOL UTILITY!


Upgrade Process Tuning

Determine Target

How much downtime is acceptable to client?

Target Timeline (Bank Holiday Weekends)

How much testing time to be built into plan?

First Iterations (Early Learning)

Fix-on-Fail vs Raise Calls with Oracle

Tasks Run Serially

Focus on Long-Running tasks (30mins+)

Look out for Performance Patches

Later Iterations (Get Smarter)

Determine Steps which can be done in Advance

Determine Steps which can be run in Parallel

Test, Time, Tune, Repeat . . .


Upgrade Process Tuning Example 1 (bsu.sh)

WebLogic Patching Utility

During Upgrade, needs to be run many times *

Remove Operations: 8 + Install Operations: 7

Before Tuning: 5 mins per operation (approx)

Time Taken: 75 mins!!

Done in Preparation Phase, but. . . future?

Two Steps For Tuning

Edit “bsu.sh” & Increase Heap Size: 512MB 2GB+

Apply Patch 27238412 (replaced 26664545) [Doc ID 2271366.1]

After Tuning:

Operations now take approx 1min

Total Time Reduced to 15 mins (Saving 1 hour!!)

* Numbers may vary: This was from our upgrade!


Upgrade Process Tuning Example 2 (Scale-up)

We will see later that we need multiple “OACORE” JVMs

Method for adding these is cumbersome (see below)

Developed Scripted Approach

Also re-used the same script for Cloning (Post Go-Live)

Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle EBS Release 12.2 (Doc ID 1905593.1)

1). Logon to first target node

2). Run:$ perl <AD_TOP>/patch/115/bin/adProvisionEBS.pl \

ebs-create-managedserver -contextfile=<CONTEXT_FILE> \

-managedsrvname=oacore_server2 -servicetype=oacore \

-managedsrvport=7203 -

logfile=<APPLRGF>/TXK/addMS_oacoreserver2.log

3). Wait a few mins for start-up / shutdown of managed server

4). Run:$ perl <FND_TOP>/patch/115/bin/txkSetAppsConf.pl -

contextfile=<CONTEXT_FILE> \

-configoption=addMS -oacore=testserver.example.com:7203

5). Repeat for each managed server (x42 for us)

1). From Primary Application Tier Node, Run:

applmgr> cd /stage/R12.2/XTRA_SCRIPTS

applmgr> add_new_managed_servers.sh 7 <node1> <node2> <node3>

<node4> <node5> <node6>

. . .

ManagedServer oacore_server8 created successfully.



. . .


Complete, check logfiles to ensure success.


Final Upgrade Process: Prepare Phase (High Level)

Download and Prepare All Software into Staging Area

DB Upgrade: Preparation Phase

Install 12c RDBMS Software

Apply Latest PSU and Associated Patches

Prepare 12c Init.ora Files (Upgrade & Live Running)

Apps Upgrade: Preparation Phase

Prepare Apps Staging Area

Run Pre-Upgrade Tasks

Create Upgrade Filesystems

Patch the Apps Tech Stacks

Merge AD12.2 Upgrade Driver

Merge Consolidated Upgrade Patches

Prepare Online Patching Enablement Patch and Report


Final Upgrade Process: Execute Phase (High Level)

System Close & Backup/Snap

DB Upgrade: Execute Phase

Run Pre-Upgrade Tool

Run “dbua” Upgrade Assistant

Run Manual Upgrade Post-Steps

Backup/Snap

Apps Upgrade: Execute Phase

Run Pre-Upgrade Tasks (Tech and Functional)

Close Down App

Apply Manual Work-Arounds

Run 12.2 Upgrade Steps (inc AD/TXK + R.U.P. etc)

Run Customisation Tasks

Run Post-Upgrade Tasks (inc Scale-out and Scale-up)


Final Upgrade Process: Hand-Over Phase (High Level)

System Close

Full System Backup / Snapshot

Open System for Internal Testing

Open System for Limited End-User Testing

Open System Fully

Take cover . . . !


Upgrade Summary

What Worked Well What Didn’t Work Well

Fully Documented Process Relatively Early Adopters (Bugs & Changes)

Iterative Structured Approach Project Scope Creep (DB Upgrade)

Performance Testing* & Tuning Release Update Pack 12.2.6 vs 12.2.7

Functional Testing Managing Resources

Upgrade Completed On Time On Cut-Over Go Live Issues! (More on this later!!)

Automated / Scripted Many Steps No Party!!


Part 2: EBS 12.2 Tuning (Technical)

12.2


E-Business Suite R12.2: Application Tuning

Test Harness

Fully Sized Database

Production-Like Hardware

Load Simulator (Silk Performer)

Key Tuneable Parameters

Database Parameters

OHS Parameters

Number of “OACORE” JVMs

Connection Pool Settings

WebLogic Logging

JVM Parameters

Other Items For Tuning

Inefficient “Profiles” (later slide)

12.2


EBS Tuning: Database Parameters

Based on the Previous “init.ora”

Changes made for 11g 12c DB Upgrade

Changes made for 12.1.3 12.2.X App Upgrade

Refer to notes: [396009.1] & [174605.1]

Use “bde_chk_cbo.sql” provided in [174605.1]

Some changes needed just for Upgrade Period (see below)

WebLogic Connection Pool reduced backend connections

Enables Reduction to “Processes” & “Sessions” vs 12.1.3

See later slide on this!

12.2

ora[12c]$ cd $ORACLE_HOME/dbs

ora[12c]$ vi init$ORACLE_SID.ora

#############################################################################

# Added For 12.2 Upgrade - Remove Afterwards

#############################################################################

log_buffer=100M

job_queue_processes=30

parallel_max_servers = 60

max_dump_file_size = UNLIMITED # Prevents delays in adop enab (Doc ID 1946877.1)


EBS Tuning: OHS Parameters

12.2

vi $FMW_HOME/webtier/instances/*OHS[2-7]/config/OHS/*/httpd.conf

under worker MPM. . . set:

PARAMETER NEW VALUE DEFAULT VALUE COMMENTS

ServerLimit 8 25

ThreadLimit 128 unset <--- This is needed to support ThreadsPerChild

StartServers 8 5

MaxClients 1024 625

MinSpareThreads 100 25 <--- Gives warning when close to max-clients (<100 free)

MaxSpareThreads 1024 75 <--- Will prevent any server shutdowns

ThreadsPerChild 128 25

MaxRequestsPerChild 0 0 <--- No sign of memory leak on httpd.worker processes, so OK!

KeepAliveTimeout 15 15 <--- Default (unchanged)

MaxKeepAliveRequests 0 0 <--- Default (unchanged)

Note: Also need to change "AcceptMutex fcntl" to AcceptMutex pthread

And comment out the LockFile entries!


EBS Tuning: “OACORES” & DB Connection Pool

Trials Revealed ~300 Concurrent Users* per JVM was OK

Found 300 JDBC Connections per JVM worked well

Connection Pool Harvesting (tweaked)

Do the maths . .

12.2

*What’s a concurrent user? Your mileage will vary!


EBS Tuning: Load Balancer “Fencing”

By Default, Oracle will use software load balancing

Incoming connection to OHS redirects to any managed server!

Can result in a lot of cross-tier network traffic

Not necessary if using Hardware Load Balancer

Less control in some maintenance scenarios

Our solution was to “fence” . . .

After scale-out / scale-up . . .

12.2

applmgr> vi $FMW_HOME/webtier/instances/*OHS[2-7]/config/OHS/*/mod_wl_ohs.conf

[Ensure OHS2 File has only WebLogicCluster commands referring to <Apps Node 1>]



etc

applmgr> vi $FMW_HOME/webtier/instances/*OHS[2-7]/config/OHS/*/apps.conf

[Ensure OHS2 File has only BalancerMember commands referring to <Apps Node 1>]



etc


EBS Tuning: WebLogic Logging

By Default, WebLogic Logging is Verbose and Duplicated:

12.2

[OACORE Server Logs – Standard Out]

applmgr> wc -l oacore_serverX.out

1831 oacore_serverX.out

applmgr> egrep -c 'at |Truncated.|Caused By:|java:' oacore_serverX.out

1282

[OACORE Server Logs – Normal Log File]

applmgr> wc -l oacore_serverX.log

17895 oacore_serverX.log

applmgr> egrep -c ' at |java:|Caused By:' oacore_serverX.log

3639

applmgr> grep -c "<Info>" oacore_serverX.log

13741

[AdminServer Logs – Domain Log File]

applmgr> wc -l EBS_domain_XXXXX.log

59713 EBS_domain_XXXXX.log

applmgr> grep -c "<Error>" EBS_domain_XXXXX.log

872

applmgr> egrep -c ' at |ServletException|java:|Caused By:' EBS_domain_XXXXX.log

55491

Majority (70%) of Standard Out

Messages are Truncated Stack Dumps

Majority (75%) of Standard Log

Messages are “INFO” Level

Majority (94%) of Domain Log Messages

are “ERROR” and Stack Dumps


EBS Tuning: WebLogic Logging (2)

Here’s how to reduce this overhead:

12.2

Change From

“Info” to “Notice”

Select

Select

Change From

“Trace” to “Notice”


EBS Tuning: WebLogic Logging (3)

Here’s how to reduce this overhead (cont):

12.2

Change From

“Notice” to “Warning”

Change From

“Notice” to “Critical”

Reduce From

10 to 1

UNTICK!


EBS Tuning: JVM Parameters (OACORE)

During Trials, experimented with many JVM parameters:

Increased Heap Size 1GB 3GB

Used Garbage First (G1) Collector

Set “DevPollDisabled” (doesn’t scale without this on some platforms)

Used “Tiered Compilation” (more on this later)

12.2


EBS Tuning: Changing JVM Parameters - Gotcha

In 12.1.3:

Change JVM Parameters via “oacore_jvm_start_options”

Run Auto-Config

In 12.2:

We have same parameter – yay!

However, only used when creating a new managed server!

Therefore need to do this before scale-out & scale-up!

To Change in 12.2 after Managed Server Created:

Edit in WebLogic Console under Config Server Start Arguments

Or, edit the “$EBS_DOMAIN_HOME/config/config.xml” file

• Be Careful! – Take backup first!

Or, use the “adProvisionEBS.pl ebs-set-managedsrvproperty” utility

• Quite Messy, all arguments listed in full, refer to Doc ID 1905593.1

12.2


First Encounter with Problematic Profiles

Discovered During Load Testing

Good Performance for first few hours of test

Performance Rapidly Declined

Observed the following:

Increasing size of result cache, 200MB 500MB did not fix issue!

12.2Latch Name Tot Wait (s) Get Requests Miss % SLEEPS

------------------------------------ ------------ ------------ ------ ---------

Result Cache: RC Latch 4000631.1 1268891679 125.4 59713271

row cache objects 76299.2 7095624871 13.9 50778810

cache buffers chains 1570.9 1.3298E+11 0.2 4289651

In memory undo latch 752.9 643026407 0.8 1069567

parameter table management 479.2 95434614 5.9 479927

NAMESPACE STATUS CACHE_ID

---------- --------- ------------------------------

NAME

--------------------------------------------------------------------------------

NUM_RESULTS AVG_SCAN_PER_ENTRY MAX_SCAN_PER_ENTRY TOT_SCAN_CNT TOT_BLK_CNT

----------- ------------------ ------------------ ------------ -----------

PLSQL Published a77mb1r8mfppd0ktc3712ykmt2

"APPS"."FND_PROFILE"::11."GET_USER_VALUE"#4b60a496cb9dc4ee #505

206195 22 638031 4486880 206195

ICX: Language

ICX:Session Timeout

FND: Override Directory

PA: Tasks to Display for Expenditure Entry

etc etc


Problematic Profiles – Root Cause (1)

Oracle’s Code Doesn’t Scale!

Here’s how “FND_PROFILE.VALUE” function works:

As of: $Header: AFPFPROB.pls 120.43.12020000.7 2016/06/23

Moves from User Responsibility Application . . . Site

Each Sub-Function Uses RESULT_CACHE

• get_user_value

• get_resp_value

• get_appl_value

• etc

12.2

** The algorithm is such:

** 1. Using the profile option information acquired at declaration,

** determine if the profile exists in FND_PROFILE_OPTIONS. If it does

** not, return NULL.

** 2. If it exists and the user_enabled_flag is Y, get user level value.

** 3. If there was no user-level value, proceed to check the applicable

** levels based on the profile option hierarchy.

** 4. If no value was obtained in all the other levels, check the site

** level if it is enabled for the profile option.

*Refer to Bug 20198791 & Doc ID 2099303.1 for some background information



Problem for Systems with many Active Users / Resps

In R12.2 We Have Around 10,000 Profile Options

Even if only 50 of these were actually enabled at user / resp level

By ½ of all users and ½ of all responsibilities

That would be over 2 Million Result Cache entries (blocks)

Each Result Cache Block is 1kb in size, so that’s . .

12.2



Problem is not:

Number of Profile Options that are Set

Paradoxically, better if set at User / Resp levels!

Problem is:

Number of profile options that have ability to be set

. . . as NULL results still stored in result cache

. . . so what to do?

Initial Solution was simple:

Disable Result Cache for “User Level” Lookups

Not documented*

12.2

ora[12c]$ vi $ORACLE_HOME/dbs/init$ORACLE_SID.ora

#############################################################################

# Added For 12.2 - Enable Only After DB Upgrade Complete Inc Patch 17305006

#############################################################################

_result_cache_black_list=(a77mb1r8mfppd0ktc3712ykmt2)

*The black_list feature has some refs, see bug 17305006, PSU merge patches needed before DB 12.2


Part 3: Post Go-Live (Technical)

12.2


Day 1 – Slow System & Complete Panic


Day 1 – Main Issues

Integration Issues

Some custom static files needed amending

GOTCHA: Files changed under “OA_HTML” Need WebLogic Bounce!

Lots of Performance Bugs

Especially

• Email Centre – Stuck Threads on Apps Servers

• Timecards – High “Log File Sync” due to excessive logging

None of these appeared during testing!

Load Test Suite focused on “Majority” Transactions

Some “esoteric features” did not scale!

Return of the “Profiles” Issue

This was “Major Issue” for us!

Engaged with Senior Levels in Oracle

12.2


Week 1 – Example Issue: Timecards*

Lots of messages appearing in “FND_LOG_MESSAGES”

Piece of very bad code located (after manual search)

Even though mobile timecards not in use!

Common Code (clearly shows debugging):

No Debug Check! So executed for every timesheet

AND, “write_log” function had Explicit Commit!

12.2

$Header: hxclockapi.pkb 120.6.12020000.4 2014/01/30 09:54:17 asrajago ship

-- Bug 18113601

hxc_mob_utils.write_log('p_resource_id :'||p_resource_id);

hxc_mob_utils.write_log('p_process_locker_type :'||p_process_locker_type);

hxc_mob_utils.write_log('p_start_time : '||p_start_time);

hxc_mob_utils.write_log('p_stop_time : '||p_stop_time);

hxc_mob_utils.write_log('p_time_building_block_id : '||p_time_building_block_id);

hxc_mob_utils.write_log('p_time_building_block_ovn : '||p_time_building_block_ovn);

hxc_mob_utils.write_log('p_expiration_time : '||p_expiration_time);

hxc_mob_utils.write_log('p_row_lock_id : '||p_row_lock_id);

-- Bug : Timecard Mobile

-- Check if this timecard is locked in In Transit Tables.

hxc_mob_utils.write_log(dbms_utility.format_call_Stack);

hxc_mob_utils.write_log(p_process_locker_type);

hxc_mob_utils.write_log('Checking mobile lock ');

hxc_mob_utils.write_log('Checking mobile lock '||p_start_time||'-'||TO_CHAR(p_stop_time,'dd-MON-YYYY HH24:MI'));

*This issue was present as of 12.2.7 Codeline!


Week 1 – Example Issue: Timecards* (2)

We Told Oracle of the Fault in their Code!

Fujitsu Proposed Interim Solution:

2 Lines of Code

Big Difference!12.2

IF g_debug THEN-- Bug 18113601

hxc_mob_utils.write_log('p_resource_id :'||p_resource_id);

hxc_mob_utils.write_log('p_process_locker_type :'||p_process_locker_type);

hxc_mob_utils.write_log('p_start_time : '||p_start_time);

hxc_mob_utils.write_log('p_stop_time : '||p_stop_time);

hxc_mob_utils.write_log('p_time_building_block_id : '||p_time_building_block_id);

hxc_mob_utils.write_log('p_time_building_block_ovn : '||p_time_building_block_ovn);

hxc_mob_utils.write_log('p_expiration_time : '||p_expiration_time);

hxc_mob_utils.write_log('p_row_lock_id : '||p_row_lock_id);



hxc_mob_utils.write_log(dbms_utility.format_call_Stack);

hxc_mob_utils.write_log(p_process_locker_type);

hxc_mob_utils.write_log('Checking mobile lock ');

hxc_mob_utils.write_log('Checking mobile lock '||p_start_time||'-'||TO_CHAR(p_stop_time,'dd-MON-YYYY HH24:MI'));

END IF;



Week 1 – Example Issue: Timecards* (3)

Oracle’s Final Solution:

Moved away from problematic “write_log”

Included the “debug” check first!12.2

$Header: hxclockapi.pkb 120.6.12020000.7 2018/01/04 16:59:42 asrajago ship $

IF g_debug THENhr_utility.trace('p_resource_id :'||p_resource_id);

hr_utility.trace('p_process_locker_type :'||p_process_locker_type);

hr_utility.trace('p_start_time : '||p_start_time);

hr_utility.trace('p_stop_time : '||p_stop_time);

hr_utility.trace('p_time_building_block_id : '||p_time_building_block_id);

hr_utility.trace('p_time_building_block_ovn : '||p_time_building_block_ovn);

hr_utility.trace('p_expiration_time : '||p_expiration_time);

hr_utility.trace('p_row_lock_id : '||p_row_lock_id);



hr_utility.trace(dbms_utility.format_call_Stack);

hr_utility.trace(p_process_locker_type);

hr_utility.trace('Checking mobile lock ');

hr_utility.trace('Checking mobile lock '||p_start_time||'-'||TO_CHAR(p_stop_time,'dd-MON-YYYY HH24:MI'));

END IF;



Month 1 – Main Issue: Profiles

Still issues with Result Cache (esp for “RESP”)

Added “RESP” and “SITE” into Black List

Result Cache Performance Improved

System Just About Coped, But . . .

12.2

Elapsed Elapsed Time

Time (s) Executions per Exec (s) %Total %CPU %IO SQL Id

---------------- -------------- ------------- ------ ------ ------ -------------

250,607.2 113 2,217.76 21.5 25.9 .0 6sc8jn1bssj0c

select count(*) from per_workforce_x hr,per_phones pho,hr_lookups hrl,hr_locati

ons loc,hr_territories_v tr WHERE hr.person_id = pho.parent_id(+) AND pho.phon

e_type(+) = 'W1' AND pho.parent_table(+) = 'PER_ALL_PEOPLE_F' AND sysdate BETWEE

N pho.date_from(+) AND nvl(pho.date_to(+), sysdate) AND pho.phone_type = hrl

149,828.2 129,441,219 0.00 12.9 8.0 .0 7qwsx7rw0s3a5

SELECT PROFILE_OPTION_VALUE FROM FND_PROFILE_VALUES_USER_V WHERE PROFILE_OPTION_

ID = :B3 AND APPLICATION_ID = :B2 AND LEVEL_VALUE = :B1 AND PROFILE_OPTION_VALUE

IS NOT NULL

54,806.2 156,747,772 0.00 4.7 25.3 .0 gj84g2yx56u2q

SELECT PROFILE_OPTION_VALUE FROM FND_PROFILE_VALUES_SITE_V WHERE PROFILE_OPTION_

ID = :B2 AND APPLICATION_ID = :B1 AND PROFILE_OPTION_VALUE IS NOT NULL

49,114.6 108,659,530 0.00 4.2 25.1 .0 9k7gd9pn661pj

SELECT PROFILE_OPTION_VALUE FROM FND_PROFILE_VALUES_RESP_V WHERE PROFILE_OPTION_

ID = :B4 AND APPLICATION_ID = :B3 AND LEVEL_VALUE = :B2 AND LEVEL_VALUE_APPLICAT

ION_ID = :B1 AND PROFILE_OPTION_VALUE IS NOT NULL


Month 1 – Main Issue: Profiles (2)

During peak times, system went into meltdown!

Huge Mutex Issues – Massive ramp up in CPU

Needed Complete Bounce to resolve!

Oracle Advised “Beef Up” RC & remove black lists!

We were reluctant, but client insisted!

So we did it in the evening . . . but . . .

12.2

Total Wait Wait % DB Wait

Event Waits Time (sec) Avg(ms) time Class

------------------------------ ----------- ---------- ---------- ------ --------

latch free 54,089,624 181.9K 3.36 77.3 Other

DB CPU 39.2K 16.7

db file sequential read 2,273,797 9988.5 4.39 4.2 User I/O

log file sync 79,076 2933.1 37.09 1.2 Commit

Pct Avg Wait Pct

Get Get Slps Time NoWait NoWait

Latch Name Requests Miss /Miss (s) Requests Miss

------------------------ -------------- ------ ------ ------ ------------ ------

Result Cache: RC Latch 550,997,581 26.2 0.4 2.E+05 1 0.0

PL/SQL Entry Subprogram % Activity

----------------------------------------------------------------- ----------

APPS.PAY_ARCHIVE.PROCESS_EMPLOYEE 55.13

APPS.FND_PROFILE.VALUE 53.07





Oracle keen to get to bottom of Result Cache Issue!

1st Problem: Invalidations

Single change to any profile value invalidates ALL profile caches!

But, we have many changes every day . . .

12.2

SQL> select APPLICATION_ID,PROFILE_OPTION_ID,LEVEL_ID,LEVEL_VALUE,

2 to_char(LAST_UPDATE_DATE,'HH24:MI:SS')

3 from apps.fnd_profile_option_values

4 where LAST_UPDATE_DATE>sysdate-(8/24)

5* order by LAST_UPDATE_DATE;

APPLICATION_ID PROFILE_OPTION_ID LEVEL_ID LEVEL_VALUE TO_CHAR(

-------------- ----------------- ---------- ----------- --------

0 13791 10004 315365 08:43:12

690 4306 10004 359689 09:24:39

178 7646 10004 359689 09:24:39

690 4306 10004 283772 09:31:48

. . .

690 4306 10004 365973 11:00:46

690 4292 10004 365973 11:00:46

. . .

178 3798 10004 481853 14:07:23

0 3773 10004 481853 14:07:35

178 4108 10004 481853 14:08:55

800 1204 10004 241598 15:22:18

49 rows selected.

Show all profile

changes in last 8 hours

Almost all changes are

“USER” Level (10004)

Sometimes changes

are bunched together!

Lots of changes!



How to cut-down on Invalidations?

Can’t stop users from making changes!

However, found a “Password Sync” job:

• Synchronising between SSO and E-biz

• Developed by Oracle Consulting

• Ran Every 15 minutes and made profile changes!!

• Changed to run once in the evening

• Improved situation, but ad hoc user changes still prevalent

12.2



I Suggested Design Change to Oracle:

Replace “FND_PROFILE_OPTION_VALUES” Table with:

• FND_PROFILE_OPTION_VALUES_USER

• FND_PROFILE_OPTION_VALUES_RESP

• FND_PROFILE_OPTION_VALUES_SITE

• etc

Add “FND_PROFILE_OPTION_VALUES” View

• Based on “UNION ALL” of all of the above

• For backwards compatibility

Change “FND_PROFILE” RC functions:

• Each would select from their own Specific Table

Would give Result Cache independence

Even better* option would be Partitioned Table!

Oracle said . . . NO! Too much change for them!

12.2

*Need to check result cache invalidation based on partition?



2nd Problem: Too many Result Cache Entries

Remember: NULL Entries are held in cache

Remember: User Responsibility Application . . . Site

Problem gets worse as you fix first problem!

Major Performance Problems if Result Cache gets full

Oracle’s solution was to increase RC size

Poor Performance when lots of entries in Result Cache

Especially when invalidations occurring!

Due to long hash chain length.

12.2



How to reduce number of Result Cache entries?

Check which Profiles hitting cache hardest

Check Profiles not in use at various levels

Where not in use DISABLE

12.2ICX: Language disable RESP/USER

ICX:Session Timeout disable USER only

FND: Override Directory disable RESP/USER

PA: Tasks to Display for Expenditure Entry disable RESP/USER

RCV: Fail All ASN Lines if One Line Fails disable RESP/USER

Achieved 10% Reduction

in RC Entries with just

these 5!



How to manage Result Cache when getting full?

Manual Methods:

Fujitsu Developed Automatic Solution:

• Monitor Result Cache on periodic basis (e.g. 5mins)

• Invalidate Specific Cache Entries when >50,000 (approx 50MB)

12.2

SQL> exec dbms_result_cache.flush(); Whole Cache

SQL> exec dbms_result_cache.INVALIDATE_OBJECT(<cache_id>); Single Entry

Wed Apr 18 16:23:13 2018

FUJITSU RC MONITOR

Invalidating large (63796) cache id: a77mb1r8mfppd0ktc3712ykmt2

Wed Apr 18 16:34:16 2018

Thread 1 cannot allocate new log, sequence 651

Private strand flush not complete

Beginning log switch checkpoint up to RBA [0x28b.2.10], SCN: 287205410218

Wed Apr 18 16:34:19 2018

Thread 1 advanced to log sequence 651 (LGWR switch)

Wed Apr 18 16:38:14 2018

FUJITSU RC MONITOR


Wed Apr 18 16:39:22 2018

Completed checkpoint up to RBA [0x28b.2.10], SCN: 287205410218

Wed Apr 18 16:53:15 2018

FUJITSU RC MONITOR




So what happened next?

Fujitsu Spent Time Looking at “Mutex Meltdown”

Issue suspected as “hot objects” [Ref: 2194448.1]

Three things done for “USER” and “RESP” Queries:

1). Make SQLs as fast as possible with Indexing & Baselines

2). Set “Hot Object Copies” parameter to 16

3). Add SQLs as “HOT” during database start-up:

12.2

### Run MarkHot Procedure To Ensure Multiple Copies Of USER/RESP Busy SQL

sqlplus -s / as sysdba <<EOF

set heading off

select APPS.FND_PROFILE.VALUE_SPECIFIC('BOM:STANDARD_ITEM_ACCESS',1234,1234)

from dual;

begin

sys.dbms_shared_pool.markhot (hash=>'cb73920f7392208c7b731d3df80c0d45',

namespace=>0, global=>true);

sys.dbms_shared_pool.markhot (hash=>'39966b9bcb73f953991ded4d686306b1',

namespace=>0, global=>true);

end;

/

exit;

EOF



New Solution Implemented

Black Lists added again for “USER” and “RESP” Queries

Good, Consistent and Stable Performance

Hidden Snares

Non-Standard Solution (we are using 2 sticking plasters)

Relies on “black_list” Procedure

• Not standard until DB12.2

• Have to request new merge patch with each PSU

• Changes in Oracle code could generate new signatures!

Relies on “markhot” Procedure

• Beware manual DB bounces without using script

• Changes in Oracle code could generate new signatures!

Code is still Inefficient

• Exasperated with use of “HR_SECURITY”

12.2


Future Plans For Profiles Issue

Move back towards Result Cache

Trim down as many unnecessary profiles as possible

60% of hits already removed!

Keep pushing Oracle for Enhanced Profiles Solution

Use new Fujitsu RC Monitor to maintain stability

Keep tuning “HR Security” problematic queries

12.2


Online Patching

12.2

HEY – Wait a minute!

This is a 12.2 Presentation and we have

NOT spoken about online patching!!!


Much Later – Online Patching Issue

12.2

NOTE:

Logarithmic Scale!

*Bug 22568797 - latch: row cache objects contention with editions.

*Bug 20161956 - Adop:prepare Should Throw Warning If No. Of Editions Exceed Threshold Limit


Much Later – Online Patching Issue (2)

Oracle’s internal guides say . . .

We need to actualize when we have >20 Editions

However, for busy systems need lower threshold!

Patch 22568797 Should Help, but . . .

We now “actualize_all” after every patching session!

12.2

SQL> select count(*) from dba_editions;

COUNT(*)

----------

5

SQL> select count(*) from dba_objects;

COUNT(*)

----------

459168

SQL> select count(*) from obj$;

COUNT(*)

----------

1203883


COUNT(*)

----------

1


COUNT(*)

----------

459164


COUNT(*)

----------

460160



How we patch with “actualize_all”

12.2

$ adop phase=prepare

$ adop phase=apply patches=xxx,xxx...

$ adop phase=finalize

[bring down system manually]

$ adop phase=cutover mtrestart=no

$ . /home/$LOGNAME/.profile

$ adop phase=cleanup


$ adop phase=actualize_all

$ adop phase=finalize finalize_mode=full



$ adop phase=cleanup cleanup_mode=full

SQL> ?/rdbms/admin/utlrp

[bring up system manually]

SQL> exec dbms_stats.gather_fixed_objects_stats();

SQL> exec dbms_stats.gather_dictionary_stats();

Can take over 2 hours (fs_clone)

10mins 25mins

~10mins (much faster than “mtrestart”)

5mins 10mins

10mins++ (depending on number of patches)

30mins+

35mins+

10mins+

25mins+

Can take over 3 hours!

10mins 30mins

~15mins

~8mins

~30mins

~10mins



How we patch with “actualize_all”

12.2


$ adop phase=apply patches=xxx,xxx...

$ adop phase=finalize

[bring down system manually]



$ adop phase=cleanup


$ adop phase=actualize_all

$ adop phase=finalize finalize_mode=full



$ adop phase=cleanup cleanup_mode=full

SQL> ?/rdbms/admin/utlrp

[bring up system manually]

SQL> exec dbms_stats.gather_fixed_objects_stats();

SQL> exec dbms_stats.gather_dictionary_stats();

~6 Hours for Cut-Over

Rather than ~1 Hour

without Actualize!



Patches Required before using “ACTUALIZE_ALL”:

25099339:

• Trigger On Editioning View Not Firing When Updated Via Inline View

27997875

• Regression Of Base Bug 25919622 (Invalid Objects Observed In Ebs

Environment After Drop Edition ORA$BASE)

Beware, you may see:

But*, if you check the logs:

12.2

The cleanup phase completed successfully.

[ERROR] ORA-04045: errors during recompilation/revalidation of APPS.xxxobjectxxx

ORA-04052: error occurred when looking up remote object MYOBJECT@XXXXXXXXX

ORA-00604: error occurred at recursive SQL level 3

ORA-12541: TNS:no listener, SQL[V_20180920_1105]: drop PACKAGE "APPS"."xxxobject"

[ERROR] Could not drop covered object [V_20180920_1105] APPS.xxxobjectxxx

(PACKAGE): ORA-04045: errors during recompilation/revalidation of APPS.xxxobjectxxx

*Many possible reasons why clean-up may silently fail, example above is reference to remote object



Belt and Braces Check:

12.2


COUNT(*)

----------

5


COUNT(*)

----------

459168


COUNT(*)

----------

1203883

SQL> select * from dba_editions;EDITION_NAME PARENT_EDITION_NAME usable

--------------------------------------------------------------------------------

V_20180920_1105 V_20180630_1205 YES

V_20180922_1132 V_20180920_1105 YES

V_20180630_1205 YES


COUNT(*)

----------

1


COUNT(*)

----------

459164


COUNT(*)

----------

460160


Much Later – JVM Tuning

Remember this from slide 38?

We hadn’t noticed . . 12.2Warnings reported in oacore_serverX.out

Java HotSpot(TM) 64-Bit Server VM warning:

CodeCache is full. Compiler has been disabled.

Java HotSpot(TM) 64-Bit Server VM warning:

Try increasing the code cache size using -XX:ReservedCodeCacheSize=


Much Later – JVM Tuning (2)

Fair Re-Trial, Compare Apples with Apples

Non-Tiered (Default) Compilation is Best!

But, you may need to set -XX:ReservedCodeCacheSize=nnn

We also needed to increase our heap size 3GB 4GB

12.2


Much Later – Quarterly PSUs

Great idea to maintain latest PSUs (eg. DB and WLS)

But, be careful not to go “too recent”*

Example Ref: 2443497.1

Download Latest ETCC Patch 17537119

Simple check of mappings file . . .

E.g. ETCC Patch: 29th Nov 2018:

12.2

applmgr> cd $INSTALL_BASE/fs_ne/etcc/mw

applmgr> grep WLS_HOME txk_R1220_MT_mappings.xml | tail -1

<Technology name="WLS_HOME" version="10.3.6.0.181016">

applmgr> cd $INSTALL_BASE/fs_ne/etcc/db/onprem

applmgr> grep "DB_HOME" txk_R1220_DB_mappings.xml | grep 12.1 | tail -1

<Technology name="DB_HOME" version="12.1.0.2.181016ProactiveBP">

applmgr> cd $INSTALL_BASE/fs_ne/etcc/db/onprem

applmgr> grep "DB_HOME" txk_R1220_DB_mappings.xml | grep 12.1 | grep -v Pro | tail -1

<Technology name="DB_HOME" version="12.1.0.2.181016">

*Wait ~1 month after release! Example Ref: 2443497.1 – Download latest EBS Technology Codelevel Checker


Part 4: Wrap Up!

12.2


Be Aware (bigger software stack / more ports)


Cool Features of R12.2

WebLogic Server

Good visibility of System Performance

Even better when integrated with OEM!

Faster Start-up / Shutdown!

Better connection pooling!


WebLogic Admin Console (Summary of Servers)

QUICK HEALTH VIEW

USEFUL!


Dealing with Stuck Threads

12.2


Thanks For Your Time!

Any Questions?

Contact: [email protected]

mailto:[email protected]


Date post:	13-Mar-2020
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times